Enabling IOMMU to directly connect a physical card to a guest used to be a painful and error prone task. I remember in the past having to play with the Access Control Services to get anything to work. Now most of the Intel and AMD devices support it – at least partially.
I recently started building my virtual homelab and needed to have a network accelerated guest to handle my VM traffic. This guest would be running on the host along with the other VMs but would have direct access to the second network card in my PC. This way – virtual traffic is routed out of one access card and the host’s traffic is routed out of a different card. Physical separation – at least in theory. In another post I’ll discuss the VLAN I configured at the physical router’s end to enable physical separation between the two network cards.
Before continuing, I recommend you read the Errata for your CPU (see mine: Intel Xeon E3-1200) to see if there’s any known (and most likely unfixable) issues that will impact the use of physical devices instead virtual guests. I checked my IOMMU status using lspci -vv and looked for the ACSCap line under my PCI Express Root.
Using this handy guide on InstallGentoo I ran the following to list my IOMMU groups which I would then be isolating and handing off to the VM.
$ for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); \ do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); \ do echo -n $'\t'; lspci -nns "$device"; done; done
Here my network card was located under IOMMU group 18 (and my graphics card under IOMMU group 1 but that’s for another day :)).
There were a few options for the next step – either I could have written a small systemd service that runs before the network is up or I could add an entry to my modprobe config in /etc to bind the dummy driver to the interface – I opted for the latter, using the device ID from the above scriptlet.
Over the last couple of months, I have been nagging myself to fix several of the issues with my Linux installation, in particular I need to:
update my kernel (currently 2.6.27-rc9)
get the wacom input device working with the newest xorg-server
add some snaz to my desktop configuration (openbox, dmenu…bleh)
So, sparing you the `wget`ing and `tar -xf`ing, I upgraded my kernel to 2.6.29-rc2 and enabled the kernel modesetting. This enables me to switch to the console with no delay, instead of the current 2-3 second delay. Good news: the upgrade worked well.
Additionally, I upgraded my xorg-server to 1.6.1: the only problem was wacom tablet didn’t work, so I installed the development version of linuxwacom (version 8.3-3).
Everything works well so far except rendering terminals. When I try to open alsamixer with uxterm or resize it, the terminal behaves as if it has a really low refresh rate. I am looking into the issue and I will post a fix if I find one.
As for the ‘added snaz,’ I’ll deal with that another time. Right now I’m just happy that my fps in stepmania has increased by about 20!
8 December 2008
The latest Xorg video driver for Intel chipsets will probably cause a slowdown in rendering due to a switch from TTM to GEM. GEM is only supported in kernel 2.6.28 and above so it’s recommended that you stick with drivers released before 2.4.3.
More information: http://bugs.freedesktop.org/show_bug.cgi?id=13922
Upgrading to catalyst-8.7
9 August 2008
Choosing to run the latest kernel means choosing to deal with errors. Luckily, my mileage with 2.6.27-rc2 has been pretty good. Here are a few pointers (and an experimental patch) to use when setting up catalyst-8.7 kernel 2.6.27-rc2.
I created the patch from various components incl.:
Also, note that I do not have any working knowledge of C nor do I take credit for this patch.
Changes to Kernel API (that I’m aware of, at least)
smp_call_function. See commit 5e374fb62621aca9522f76c2317c9acda75a8e88 by Jens Axboe to the file arch/x86/kernel/smp.c
find_task_by_pid. See commit 5cd204550b1a006f2b0c986b0e0f53220ebfd391 by Pavel Emelyanov to the file kernel/pid.c
NOPAGE_SIGBUS. See commit 3c18ddd160d1fcd46d1131d9ad6c594dd8e9af99 by Nick Piggin to the file include/linux/mm.h
Other things to be aware of
If you have paravirtualised guest support enabled in the kernel, the compilation will fail with FATAL: modpost: GPL-incompatible module fglrx.ko uses GPL-only symbol ‘pv_lock_ops’. Disable it: PARAVIRT_GUEST=n
The catalyst-8.7 driver uses deprecated symbols so if they kernel does not export them (in particular init_mm) the compilation will fail. Enable them: UNUSED_SYMBOLS=y
catalyst-utils-8.7 needs to be installed with this driver
The commands (to set up catalyst-8.7, I’m sure you can setup catalyst-utils by yourself). Note: If you’re not using the kernel that will be running with the drivers, then replace `uname -r` with the correct kernel version/name.
~ % mkdir proj/catalyst
~ % cd proj/catalyst
catalyst % wget www2.ati.com/drivers/linux/ati-driver-installer-8-7-x86.x86_64.run -O catalyst-8.7.run
catalyst % wget http://antonyat.googlepages.com/firegl_public.diff
catalyst % /bin/sh ./catalyst-8.7.run --extract archive_files
I use x86. x86_64 users should substitute x86 with x86_64 and x710 with x710_64a respectively.
catalyst % cp archive_files/arch/x86/* ./
catalyst % cp archive_files/common/* ./
catalyst % cp archive_files/x710/* ./
catalyst % cp firegl_public.diff lib/modules/fglrx/build_mod/
catalyst % cd lib/modules/fglrx/build_mod/
catalyst % patch -p1 < firegl_public.diff
catalyst % cp 2.6.x/Makefile .
catalyst % make -C /lib/modules/`uname -r`/build SUBDIRS="`pwd`" ARCH=i386 modules
catalyst % su
pts/3:root@GENTOO /home/antony/proj/catalyst/lib/modules/fglrx/build_mod # install -m 644 -D fglrx.ko /lib/modules/`uname -r`/video/fglrx.ko
You should also copy the licenses (in archive_files to /usr/share/licenses)