This video demonstrates effective use of virtual machines with direct GPU access to get the most out of your Talos™ machine.

OpenPOWER fully supports advanced virtualization technologies, including PCI device passthrough when used with a PowerNV system, KVM, and the HV module. We demonstrate the power of this virtualization support below by launching two separate virtual machines, each with its own dedicated AMD Radeon GPU attached, and putting both virtual machines and GPUs under moderate load simultaneously.

Demonstration

For the best viewing experience, expand the video to full screen mode on your device. A screen resolution of at least 1920 x 1080 pixels is recommended. This video is available in native 4K resolution for those with 4K-compatible devices. The version below is scaled down and some detail, such as text, may have been lost in the scaling process.

Notes

• This radeon driver patch is required for proper GPU start on non-x86 platforms, and was applied to the radeon driver used in this demonstration.
• This radeon driver patch is required for QEMU VFIO support on all platforms, and was applied to the radeon driver used in this demonstration.
• The Radeon R9 290X GPUs (Hawaii class) used in this demonstration contain a hardware reset bug that precludes stable long term use in virtualized environments. Specifically, they do not respond to the PCIe reset signal, and therefore cannot be reinitialized without a host power cycle. This affects all architectures including x86. Subsequent AMD cards do not have this issue.

System Configuration

VFIO Configuration and QEMU Launch

The following commands will isolate the GPU at PCI bus location 0000:01:00 (GPU core [1002:67b0], HDMI audio [1002:aac8]), then prepare it for attachment to a virtual machine:

modprobe vfio-pci
echo "1002 67b0" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "1002 aac8" > /sys/bus/pci/drivers/vfio-pci/new_id
echo 0000:01:00.0 > /sys/bus/pci/devices/0000\:00\:00.0/0000\:01\:00.0/driver/unbind
echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:00\:00.0/0000\:01\:00.1/driver/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind
echo 0000:01:00.1 > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 67b0" > /sys/bus/pci/drivers/vfio-pci/remove_id
echo "1002 aac8" > /sys/bus/pci/drivers/vfio-pci/remove_id

Once the GPU has been prepared, it can be attached to a newly started virtual machine via the "-device vfio-pci" arguments to QEMU:

qemu-system-ppc64 --enable-kvm -M pseries -cpu host -m 32G -device vfio-pci,host=0000:01:00.0 -device vfio-pci,host=0000:01:00.1 qemu-pcie-passthrough-vm-1.qcow2