Enabling IOMMU to directly connect a physical card to a guest used to be a painful and error prone task. I remember in the past having to play with the Access Control Services to get anything to work. Now most of the Intel and AMD devices support it – at least partially.

I recently started building my virtual homelab and needed to have a network accelerated guest to handle my VM traffic. This guest would be running on the host along with the other VMs but would have direct access to the second network card in my PC. This way – virtual traffic is routed out of one access card and the host’s traffic is routed out of a different card. Physical separation – at least in theory. In another post I’ll discuss the VLAN I configured at the physical router’s end to enable physical separation between the two network cards.

Before continuing, I recommend you read the Errata for your CPU (see mine: Intel Xeon E3-1200) to see if there’s any known (and most likely unfixable) issues that will impact the use of physical devices instead virtual guests. I checked my IOMMU status using lspci -vv and looked for the ACSCap line under my PCI Express Root.

Using this handy guide on InstallGentoo I ran the following to list my IOMMU groups which I would then be isolating and handing off to the VM.

$ for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); \ 
do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); \ 
do echo -n $'\t'; lspci -nns "$device"; done; done

Here my network card was located under IOMMU group 18 (and my graphics card under IOMMU group 1 but that’s for another day :)).

There were a few options for the next step – either I could have written a small systemd service that runs before the network is up or I could add an entry to my modprobe config in /etc to bind the dummy driver to the interface – I opted for the latter, using the device ID from the above scriptlet.

IOMMU group 18
05:00.0 Ethernet controller [0200]: Intel Corporation I2210 Gigabit Network Connection [XXXX:YYYY] (rev ZZ)

# echo "options vfio-pci ids=XXXX:YYYY" | tee -a /etc/modprobe.d/vfio.conf

You can either modprobe vfio-pci to have this happen (you should see the kernel driver in use be vfio-pci when using lspci -v) or unbind using /sys.

First list the device directory in /sys/bus/pci/devices/<domain:bus:device:fun>, checking for the driver directory.

# ls /sys/bus/pci/devices/0000\:05\:00.0/ 
[...] driver [...]

Then issue an unbind command to the driver.

# echo XXXX:YYYY > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind

Now your Ethernet controller is ready to be handed to the VM.  Simply add -device vfio-pci,host=05:00.0 to your QEMU command line.

More information

• Heiko Sieger’s post on IOMMU: IOMMU Groups – What You Need to Consider