GPU Passtrough to Openstack

README.md

Goals

Can passtrough GPU from Host to VM on Openstack

Set /etc/default/grub

GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt rd.driver.pre=vfio-pci"

Reboot

grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

Configure Vfio

cat /etc/modules-load.d/vfio-pci.conf
---
vfio

## Dump Address for GPU
lspci -nn | grep -i nvidia #for Example our GPU code are 10de:128b

nano /etc/modprobe.d/gpu-vfio.conf
---
options vfio-pci ids=10de:128b

Openstack Configurar

Configure on /etc/kolla/nova-compute/nova.conf

[PCI]
passthrough_whitelist = { "vendor_id": "10de", "product_id": "128b" }
alias: { "vendor_id":"10de", "product_id":"128b", "device_type":"type-PCI", "name":"gt-710" }

Configure on /etc/kolla/nova-api/nova.conf

[PCI]
alias: { "vendor_id":"10de", "product_id":"128b", "device_type":"type-PCI", "name":"gt-710" }

Configure on /etc/kolla/nova-scheduler/nova.conf

[filter_scheduler]
enabled_filters = PciPassthroughFilter

Create flavor

openstack flavor create --vcpus 1 --ram 2048 --disk 10 --property "pci_passthrough:alias"="gt-710:1"  gpu1.small

Verify on VM

lspci | grep -i nvidia

Test Benchmark GPU on VM

apt-get update
sudo ubuntu-drivers autoinstall && sudo reboot
nvidia-smi

Some Failure & Resolving

• If you have an error like :
  • Exhausted all hosts available for retrying build failures for instance
  • VFIO: ERROR GROUP 15 IS NOT VIABLE, PLEASE ENSURE ALL DEVICES WITHIN THE IOMMU_GROUP
  • Please ensure all devices within the iommu_group are bound to their vfio bus driver.
  • vfio 0000:01:00.0: group 12 is not viable
    

You can check on lspci -vvs 01:00.0 or lspci -vvs 01:00.1. Make sure they are Kernel driver in use: vfio-pci. If no, you can set it with :

#### Unbind kernel use
echo 0000:01:00.1 > /sys/bus/pci/drivers/mlx4_core/unbind

#### Bind to Vfio
echo 10de 128b > /sys/bus/pci/drivers/vfio-pci/new_id 

#### 10de 128b change to your environment, you can check address on 
lspci -nn | grep -i nvidia

Clue -> Make sure controller & audio your GPU use vfio-pci.

Reference :

• https://www.ibm.com/docs/en/linux-on-systems?topic=setup-pci
• https://satishdotpatel.github.io/gpu-passthrough-for-openstack/
• https://iaas-support.virtuozzo.com/hc/en-us/articles/6407271999633-Can-t-attach-a-GPU-to-a-VM-via-offline-resize-PCI-alias-gpu-is-not-defined
• https://askubuntu.com/questions/1211666/kvm-gpu-passthrough-group-15-is-not-viable-please-ensure-all-devices-within-th