Jiff Slater
🤔 About
✍️ Contact
14 Jun 2021

Connecting a physical NIC to a Qemu Linux Guest
12 November 2020

Enabling IOMMU to directly connect a physical card to a guest used to be a painful and error prone task. I remember in the past having to play with the Access Control Services to get anything to work. Now most of the Intel and AMD devices support it – at least partially.

I recently started building my virtual homelab and needed to have a network accelerated guest to handle my VM traffic. This guest would be running on the host along with the other VMs but would have direct access to the second network card in my PC. This way – virtual traffic is routed out of one access card and the host’s traffic is routed out of a different card. Physical separation – at least in theory. In another post I’ll discuss the VLAN I configured at the physical router’s end to enable physical separation between the two network cards.

Before continuing, I recommend you read the Errata for your CPU (see mine: Intel Xeon E3-1200) to see if there’s any known (and most likely unfixable) issues that will impact the use of physical devices instead virtual guests. I checked my IOMMU status using lspci -vv and looked for the ACSCap line under my PCI Express Root.

Using this handy guide on InstallGentoo I ran the following to list my IOMMU groups which I would then be isolating and handing off to the VM.

$ for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); \ 
do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); \
do echo -n $'\t'; lspci -nns "$device"; done; done

Here my network card was located under IOMMU group 18 (and my graphics card under IOMMU group 1 but that’s for another day :)).

There were a few options for the next step – either I could have written a small systemd service that runs before the network is up or I could add an entry to my modprobe config in /etc to bind the dummy driver to the interface – I opted for the latter, using the device ID from the above scriptlet.

IOMMU group 18
05:00.0 Ethernet controller [0200]: Intel Corporation I2210 Gigabit Network Connection [XXXX:YYYY] (rev ZZ)
# echo "options vfio-pci ids=XXXX:YYYY" | tee -a /etc/modprobe.d/vfio.conf

You can either modprobe vfio-pci to have this happen (you should see the kernel driver in use be vfio-pci when using lspci -v) or unbind using /sys.

First list the device directory in /sys/bus/pci/devices/<domain:bus:device:fun>, checking for the driver directory.

# ls /sys/bus/pci/devices/0000\:05\:00.0/ 
[...] driver [...]

Then issue an unbind command to the driver.

# echo XXXX:YYYY > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind

Now your Ethernet controller is ready to be handed to the VM.  Simply add -device vfio-pci,host=05:00.0 to your QEMU command line.

More information

Squashing the PITA!
29 April 2009

Over the last couple of months, I have been nagging myself to fix several of the issues with my Linux installation, in particular I need to:

  • update my kernel (currently 2.6.27-rc9)
  • get the wacom input device working with the newest xorg-server
  • add some snaz to my desktop configuration (openbox, dmenu…bleh)

So, sparing you the `wget`ing and `tar -xf`ing, I upgraded my kernel to 2.6.29-rc2 and enabled the kernel modesetting. This enables me to switch to the console with no delay, instead of the current 2-3 second delay. Good news: the upgrade worked well.

Additionally, I upgraded my xorg-server to 1.6.1: the only problem was wacom tablet didn’t work, so I installed the development version of linuxwacom (version 8.3-3).

Everything works well so far except rendering terminals. When I try to open alsamixer with uxterm or resize it, the terminal behaves as if it has a really low refresh rate. I am looking into the issue and I will post a fix if I find one.

As for the ‘added snaz,’ I’ll deal with that another time.  Right now I’m just happy that my fps in stepmania has increased by about 20!

Compiling using the vanilla kernel tree sources
15 November 2008

I’m simply going to list the steps.

wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-
tar xf linux-
cd linux-
zcat /proc/config.gz > .config
make menuconfig
make modules_install
mkinitcpio -k `make kernelrelease` -g /boot/kernel-`make kernelrelease`.img
cp System.map /boot/System.map-`make kernelrelease`
cp arch/i386/boot/bzImage /boot/vmlinuz-`make kernelrelease`
install -D -m644 .config /boot/kconfig-`make kernelrelease`
/sbin/depmod -A -v `make kernelrelease`
vi /boot/grub/menu.lst
Upgrading to linux-2.6.27-rc3
19 August 2008

rc3 is another solid release.  Follow the same procedure documented in upgrading to linux-2.6.27-rc2.

The catalyst patch works with this release.

Upgrading to linux-2.6.27-rc2
8 August 2008

git makes it incredibly easy to upgrade my kernel to the latest (but not necessarily greatest). I’ve decided that I’ll stick to tagged releases instead of the latest git pull.

I’ll skip the prologue and just show the commands.

Download the latest commits.

linux-git %   git pull  (Note:  Make sure you are in the master branch 🙂 [git checkout master].)

Download the latest tags.

linux-git % git fetch –tags git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

List the current tags.

linux-git % git tag -l “v2.6.27*”

Duplicate the 2.6.27-rc2 branch, call it aj2.6.27-rc2, and switch to it.

linux-git % git checkout -b aj2.6.27-rc2 v2.6.27-rc2

Copy the current config to the kernel directory.

linux-git % zcat /proc/config > .config

Configure any options that changed in the latest version.

linux-git % make oldconfig

Go through the usual compilation steps. See my post on upgrading to 2.6.27-rc1 for more information.

linux-git % make | tee make.log
linux-git % su
pts/2:root@GENTOO /home/antony/proj/linux-git # make modules_install | tee modules.log
pts/2:root@GENTOO /home/antony/proj/linux-git # mkinitcpio -k `make kernelrelease` -g /boot/kernel-2.6.27-rc2.img
pts/2:root@GENTOO /home/antony/proj/linux-git # cp System.map /boot/System.map-2.6.27-rc2
pts/2:root@GENTOO /home/antony/proj/linux-git # cp arch/i386/boot/bzImage /boot/vmlinuz-2.6.27-rc2
pts/2:root@GENTOO /home/antony/proj/linux-git # install -D -m644 .config /boot/kconfig-2.6.27-rc2
pts/2:root@GENTOO /home/antony/proj/linux-git # /sbin/depmod -A -v `make kernelrelease`
pts/2:root@GENTOO /home/antony/proj/linux-git # vim /boot/grub/menu.lst
pts/2:root@GENTOO /home/antony/proj/linux-git # reboot


Difficulty in setting up catalyst-8.7. See my post on upgrading to catalyst-8.7.

Notes: My kernel config


Upgrading to linux-2.6.27-rc1
4 August 2008

I recently upgraded my kernel to linux-2.6.27-rc1.

This was the first time I compiled a kernel on this distro (but not the first time I have compiled one).  Thankfully, the compilation had no problematic errors.  This write up is more for my benefit than for yours; nevertheless, I hope it will help you out.

What works

ipw3945->iwl3945, catalyst, usb, sound, cpufreq (with new cpufreq stats output)

Extra features (that I choose to compile in)

watchdog, 9p filesystem support




00:00.0 Host bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express Memory Controller Hub (rev 03)
00:01.0 PCI bridge: Intel Corporation Mobile 945GM/PM/GMS, 943/940GML and 945GT Express PCI Express Root Port (rev 03)
00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 01)
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 01)
00:1c.3 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 4 (rev 01)
00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 (rev 01)
00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev e1)
00:1f.0 ISA bridge: Intel Corporation 82801GBM (ICH7-M) LPC Interface Bridge (rev 01)
00:1f.2 IDE interface: Intel Corporation 82801GBM/GHM (ICH7 Family) SATA IDE Controller (rev 01)
00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev 01)
01:00.0 VGA compatible controller: ATI Technologies Inc Radeon Mobility X1400
03:00.0 Ethernet controller: Broadcom Corporation BCM4401-B0 100Base-TX (rev 02)
03:01.0 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller
03:01.1 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter (rev 19)
03:01.2 System peripheral: Ricoh Co Ltd R5C592 Memory Stick Bus Host Adapter (rev 0a)
03:01.3 System peripheral: Ricoh Co Ltd xD-Picture Card Controller (rev 05)
0b:00.0 Network controller: Intel Corporation PRO/Wireless 3945ABG Network Connection (rev 02)


(If you haven’t) Clone the current stable git repository.

proj %   git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2-6

Initialise git variables / configuration (set username, email, enable colour, and improve performance for multicore systems)

proj %   git config --global user.name "Antony Jepson"
proj %   git git config --global user.email "email@gmail.com"
proj %   git config --global color.ui "auto"
proj %   git config --global pack.threads "0"
proj %   cd linux-2-6

Update local tree and make note of which changeset you are using.

linux-2-6 %   git pull
Already up-to-date.
linux-2-6 %  git whatchanged

I’m using commit 9cb7117fa4858468014f76bd996076985111e955.

Clean the tree.

linux-2-6 %   make mrproper && make clean

If you make a mistake it’s best to start fresh:

linux-2-6 %   git reset --hard origin/master

HEAD is now at 8f616cd Merge branch ‘for_linus’ of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4

Configure compilation options.

linux-2-6 %   vim Makefile
 223 HOSTCXX      = g++
 224 HOSTCFLAGS   = -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -march=prescott -mtune=generic -pipe

Configure the kernel

linux-2-6 %   zcat /proc/config.gz > .config
linux-2-6 %   make menuconfig

Apply patches and add them to git.

<apply patches, change files, etc.>

linux-2-6 %   git add <changed file(s)>
linux-2-6 %   git commit -a

Compile the kernel, install modules, build ramdisk, copy the new Systemmap, kernel, and .config, and rebuild module dependencies.

linux-2-6 %  make | tee 27rc1.log
linux-2-6 %  su
pts/4:root@GENTOO /home/antony/proj/linux-2-6 # make modules_install | tee 27rc1mod.log

Compilations within a git checkout will generate a long kernel versionstring, use `make kernelrelease` to make it easier.

mkinitcpio is distro specific (for Arch Linux).  Substitute your distro’s program here.

pts/4:root@GENTOO /home/antony/proj/linux-2-6 # mkinitcpio -k `make kernelrelease` -g /boot/kernel-2.6.27-rc1.img

See http://dirac.org/linux/system.map/ for more info.

pts/4:root@GENTOO /home/antony/proj/linux-2-6 # cp System.map /boot/System.map-2.6.27-rc1
pts/4:root@GENTOO /home/antony/proj/linux-2-6 # cp arch/i386/boot/bzImage /boot/vmlinuz-2.6.27-rc1

The next step (copying the config) is optional as you should enable the option to create a copy at /proc/config.gz (IKCONFIG=y and IKCONFIG_PROC=y).

pts/4:root@GENTOO /home/antony/proj/linux-2-6 # install -D -m644 .config /boot/kconfig-2.6.27-rc1
pts/4:root@GENTOO /home/antony/proj/linux-2-6 # /sbin/depmod -A -v `make kernelrelease`

Configuring X.org.

Although Catalyst (8.5) worked out of the box, you may need to adjust your /etc/X11/xorg.conf.  I recommend using Xorg -configure and working from there.

pts/4:root@GENTOO /home/antony # mv /etc/X11/xorg.conf /etc/X11/xorg.conf_bak
pts/4:root@GENTOO /home/antony # Xorg -configure > /etc/X11/xorg.conf

Configuring Grub.

pts/4:root@GENTOO /home/antony/proj/linux-2-6 # vim /boot/grub/menu.lst
 36 title Arch Linux (2.6.27-rc1)
 37 root   (hd0,4)
 38 kernel /vmlinuz-2.6.27-rc1 root=/dev/sda7 ro vga=771
 39 initrd /kernel-2.6.27-rc1.img

Testing the kernel.

After you have copied the kernel to /boot, reboot and test it.  If it does not work, just boot your fallback kernel (you have one, right?) and try to identify the cause of the problem.

If the kernel works (well, might I add), clean the tree then commit your changes to the git repository.

linux-2-6 %   make clean
linux-2-6 %   git add .
linux-2-6 %   git commit -a


After a while, the flashing LED indicator for the iwlwifi driver becomes annoying.  Stop the blinking with:

pts/1:root@GENTOO /sys/class/leds 17:09 # echo 0 > iwl-phy0:{RX,TX,assoc}/brightness
Built with Wordpress and Vim
© 2008 to 2021 Jiff Slater