logo
Jiff Slater
🤔 About
✍️ Contact
📚Knowledge
03 Dec 2020
 

Connecting a physical NIC to a Qemu Linux Guest
12 November 2020

Enabling IOMMU to directly connect a physical card to a guest used to be a painful and error prone task. I remember in the past having to play with the Access Control Services to get anything to work. Now most of the Intel and AMD devices support it – at least partially.

I recently started building my virtual homelab and needed to have a network accelerated guest to handle my VM traffic. This guest would be running on the host along with the other VMs but would have direct access to the second network card in my PC. This way – virtual traffic is routed out of one access card and the host’s traffic is routed out of a different card. Physical separation – at least in theory. In another post I’ll discuss the VLAN I configured at the physical router’s end to enable physical separation between the two network cards.

Before continuing, I recommend you read the Errata for your CPU (see mine: Intel Xeon E3-1200) to see if there’s any known (and most likely unfixable) issues that will impact the use of physical devices instead virtual guests. I checked my IOMMU status using lspci -vv and looked for the ACSCap line under my PCI Express Root.

Using this handy guide on InstallGentoo I ran the following to list my IOMMU groups which I would then be isolating and handing off to the VM.

$ for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); \ 
do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); \
do echo -n $'\t'; lspci -nns "$device"; done; done

Here my network card was located under IOMMU group 18 (and my graphics card under IOMMU group 1 but that’s for another day :)).

There were a few options for the next step – either I could have written a small systemd service that runs before the network is up or I could add an entry to my modprobe config in /etc to bind the dummy driver to the interface – I opted for the latter, using the device ID from the above scriptlet.

IOMMU group 18
05:00.0 Ethernet controller [0200]: Intel Corporation I2210 Gigabit Network Connection [XXXX:YYYY] (rev ZZ)
# echo "options vfio-pci ids=XXXX:YYYY" | tee -a /etc/modprobe.d/vfio.conf

You can either modprobe vfio-pci to have this happen (you should see the kernel driver in use be vfio-pci when using lspci -v) or unbind using /sys.

First list the device directory in /sys/bus/pci/devices/<domain:bus:device:fun>, checking for the driver directory.

# ls /sys/bus/pci/devices/0000\:05\:00.0/ 
[...] driver [...]

Then issue an unbind command to the driver.

# echo XXXX:YYYY > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind

Now your Ethernet controller is ready to be handed to the VM.  Simply add -device vfio-pci,host=05:00.0 to your QEMU command line.

More information

Installing Gentoo in 2020
27 September 2020

I had my once a year conundrum recently – install Gentoo or Ubuntu. This year I chose Gentoo and was immediately awash with nostalgia from seeing this screen.

The last time I ran Gentoo seriously was back in 2008 and now to see this in all its glory really brings me back.

In memory of those good times, I wanted to document the installation process on a modern 2020 PC.  Overall, I would say not much has changed except the availability of systemd as a init option.

Initial setup

I began by downloading the Gentoo minimal install CD. For AMD64 it’s available here.  As with all Gentoo installations it’s wise to have the handbook nearby for support.  It’s available here for AMD64 here.

I burned the CD image to a USB stick using Etcher by Balena.  It’s not my favourite tool but it’s reliably despite the ads.  dd is also another alternative.

Next, I rebooted in the the USB stick via the UEFI boot menu and chose the default Grub menu.  If you have a wired network configured then in most cases the network will already be working and configured with DHCP.

Configuring disks

The hardest part about configuring a fresh installation of Linux is deciding on the storing configuration.  The usual deciding factors are tolerance for risk and available storage devices.  In my case, I have a SSD solely dedicated to Linux and a pair of HDDs for longer term storage.

livecd ~ # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 390.1M 1 loop /mnt/livecd
sda 8:0 0 1.8T 0 disk 
└─md156 9:126 0 3.8T 0 raid0 
├─md156p1 259:0 0 128M 0 part 
└─md156p2 259:1 0 3.8T 0 part 
sdb 8:16 0 1.8T 0 disk 
└─md156 9:126 0 3.8T 0 raid0 
├─md156p1 259:0 0 128M 0 part 
└─md156p2 259:1 0 3.8T 0 part 
sdc 8:32 0 460.8G 0 disk 
├─sdc1 8:33 0 512M 0 part 
├─sdc2 8:34 0 244M 0 part 
└─sdc3 8:35 0 460G 0 part 
sde 8:64 1 14.6G 0 disk 
├─sde1 8:65 1 427M 0 part /mnt/cdrom
└─sde2 8:66 1 6.4M 0 part 
sr0 11:0 1 1024M 0 rom

Here you can see I have two 2TB drives already in RAID from an existing Linux installation, the USB stick mounted at /mnt/cdrom and the rootfs for the livecd mounted at /mnt/livecd.

For the filesystem, I usually select btrfs for the SSD and btrfs in RAID1 for the HDDs.  This gives me the best balance of recoverability, reliability, and performance for my usecases.  However, because I’m curious to try some of the latest technology this time round I’m going btrfs across all my drives.

Here’s the storage configuration:

  • SSD: btrfs on LUKS with GPT
    1. EFI partition, 512M
    2. boot partition, 512M
    3. swap partition, 32G
    4. root partition, remaining space
  • HDD: btrfs on LUKS (hdd1 & hdd2) with GPT
    • EFI partition, 512M, empty
    • root partition, remaining disk

Configuring the SSD

First before partitioning I usually erase the SSD.  I have more than normal faith in the ATA secure erase or NVMe secure erase and commonly use it before partitioning drives.  hdparm -I /dev/<drive> quickly tells me what my options are.   In case I’m told my drive is frozen, a quick hotplug (while the computer is on) will unfreeze the device so you can send the secure erase command.  I also had to make sure that the BIOS had the drives connected using AHCI.

Next I set up the partition table.  MBR with DOS disklabel is deprecated (and has been for a while) so I’ll use GPT.  I want the disk to be bootable in any case so I usually use a /boot partition that’s unencrypted, along with EFI and rootfs partitions.  This setup can be changed later to a completely encrypted drive with boot USB.

After the partition table was created, it looked something like this:

livecd ~ # fdisk -l /dev/sdc
Disk /dev/sdc: WW GiB, XX bytes, YY sectors
Disk model: WDC WXXXXXXXXX
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: ZZ-ZZ-ZZ-ZZ-ZZ

Device Start End Sectors Size Type
/dev/sdc1 2048 1050623 1048576 512M EFI System
/dev/sdc2 1050624 1550335 499712 244M Linux filesystem
/dev/sdc3 X Y Z 450G Linux filesystem
/dev/sdb4 X Y Z 32G Swap

mkfs.fat -F32 /dev/sdc1 (for the EFI system partition)

mkfs.ext2 /dev/scdc2 (for the boot partition)

Next I’ll turn the last partition into a LUKS container by using the cryptsetup command.  Read more about LUKS here.

First the performance of the disk is evaluated by using cryptsetup benchmark.  From the results and my risk profile, I can choose the configuration that makes the most sense for you.

cryptsetup --key-size 512 --hash whirlpool --iter-time 5000 --use-random --cipher aes-xts-plain64 --pbkdf-memory=4194304 --pbkdf=argon2id luksFormat /dev/sdc3
cryptsetup luksOpen /dev/sdc3 xtsroot

Next, I’ll set up btrfs on the encrypted root filesystem.

mkfs.btrfs -L broot /dev/mapper/xtsroot

I created a btrfs block device with label broot.  Next, I mounted it with light compression turned on along with disk quotas.  I made sure to have the rootfs and storage directories beneath the toplevel directory for ease of snapshots.

mkdir /toplevel
mount -o compress=lzo /dev/mapper/xtsroot /toplevel
# btrfs quota enable /toplevel # we will not be enabling this yet.
btrfs subvolume create /toplevel/rootfs
btrfs subvolume create /toplevel/rootfs/gentoo
btrfs subvolume create /toplevel/savestate
btrfs subvolume create /toplevel/storage
btrfs subvolume create /toplevel/storage/home
btrfs subvolume create /toplevel/storage/home/user
mount -o subvol=/rootfs/gentoo,compress=lzo /dev/mapper/xtsroot /mnt/gentoo

Finally, I turned on swap from the filesystem I made earlier.

mkswap /dev/sdc4
swapon /dev/sdc4

Installing Gentoo

Before continuing I made sure the date is correct – date.  In most cases, it will already be set correctly.  Next, I downloaded the multilib tarball from the Gentoo’s website.

wget https://bouncer.gentoo.org/fetch/root/all/releases/amd64/autobuilds/20200920T214503Z/stage3-amd64-systemd-20200920T214503Z.tar.xz \
wget https://bouncer.gentoo.org/fetch/root/all/releases/amd64/autobuilds/current-stage3-amd64/stage3-amd64-20200923T214503Z.tar.xz.CONTENTS.gz \
wget https://bouncer.gentoo.org/fetch/root/all/releases/amd64/autobuilds/current-stage3-amd64/stage3-amd64-20200923T214503Z.tar.xz.DIGESTS \
wget https://bouncer.gentoo.org/fetch/root/all/releases/amd64/autobuilds/current-stage3-amd64/stage3-amd64-20200923T214503Z.tar.xz.DIGESTS.asc
tar xpvf stage3-amd64-systemd-20200920T214503Z.tar.xz --xattrs-include='*.*' --numeric-owner
nano -w /mnt/gentoo/etc/portage/make.conf

For my compilation options, I chose the following: MAKEOPTS=”-j2″; EMERGE_DEFAULT_OPTS=”–jobs=3″; and COMMON_FLAGS=”-Ofast -flto -pipe -march=native -funroll-loops”.

Next, I continued following the guide by selecting mirrors and the ebuild repository.  Finally, I chrooted into the new directory as per the Gentoo handbook.

For my profile, I opted to use default/linux/amd64/17.1/systemd. Next I set my default Python interpreter to python 3.7 by adding the lines below to /etc/portage/package.use

*/* PYTHON_TARGETS: python3_7
*/* PYTHON_SINGLE_TARGET: -* python3_7

# For systemd, I added some specific use flags +cryptsetup +homed +pkcs11 +policykit and for python I added sqlite as a dependency.

After this I issued “emerge –ask –verbose –update –deep –newuse @world“.  I commonly check the default USE flags for the profile before going further – it’s stored in /var/db/repos/gentoo/profiles/* and more information here.

Furthermore, when I need to check the current Portage configuration it’s just a “emerge –info“ away.  The Gentoo wiki page also serves as a helpful reference.

For the kernel, I followed the handbook and made sure I had the following functionality:

  • CONFIG_X86_CHECK_BIOS_CORRUPTION=y
  • CONFIG_WATCHDOG=y

I also added the following to my bootflags kernel command line:

  • resume=/dev/root

Next, for the initramfs I opted to use genkernel and I used the following configuration.

genkernel --lvm --mdadm --bcache  --btrfs --e2fsprogs --dmraid --bootloader=grub2 --luks --busybox --install --kernel-config=/usr/src/linux/.config initramfs

For configuring systemd and grub2 you’ll need to add the following to /etc/default/grub

GRUB_CMDLINE_LINUX="init=/lib/systemd/systemd crypt_root=<device> dobtrfs"

Remember to specify the UUID of the real_root in GRUB_DEVICE and to uncomment GRUB_TERMINAL=console if you want to see errors during bootup.

I usually compile two kernels – one custom configured by me and the other by genkernel in case I get stuck.  Also in the modern day and age, there’s no region to configure any framebuffers, so at best I only set the EFI framebuffer and simple framebuffer kernel option.

genkernel --kernel-append-localversion=-custom all
grub-mkconfig -o /boot/grub/grub.cfg

Once finished, I rebooted, selected the new entry in Grub, entered my password for my encrypted root drive, and immediately started emerging Firefox.

Welcome home!

Debugging ebuild failure on Gentoo for lxml
22 April 2019

While setting up a fresh installation of Gentoo to deploy to the cloud for remote dev work, I kept running into compilation errors with lxml.

The referenced snippet of code shown was the standard compilation line: “${@}” || die “${die_args[@]}” which means run the command and associated arguments or quit and show the associated arguments.

I went into the build directory to debug further.

# cd /var/tmp/portage/dev-python/lxml-4.3.3/lxml-4.3.3-python2_7/work/lxml-4.3.3-python2_7
# i686-pc-linux-gnu-gcc -O2 -march=native -pipe -fno-strict-aliasing -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -I/usr/include/libxml2 -Isrc -Isrc/lxml/includes -I/usr/include/python2.7 -c src/lxml/etree.c -o /var/tmp/portage/dev-python/lxml-4.3.3/work/lxlm-4.3.3-python2_7/build/temp.linux-x86_64-2.7/src/lxml/etree.o -w

Which compiled without problems.

I emerged screen and then ran emerge –resume and managed to capture the error.

> Unknown psuedo op .str

I found a similar thread on the Gentoo forums and created a bigger swapfile as I was running out of memory.
# fallocate /swapfile -l 1GiB
# chmod 600 !$
# mkswap !$
# swapon !$

This solved the compilation issue.

Installing Windows XP in VirtualBox
18 July 2009

Not too difficult, here are a few tips to make sure that your experience is flawless.

When configuring the virtual machine, make sure that you

  • enable APIC and ACPI in Settings –> System.
  • create a fixed sized harddisk to improve disk I/O speeds.

After installing Windows XP, make sure that you

  • download the driver for the Intel Pro 1000 network adapater and change the default adapter from PCNet to Intel Pro 1000 in Settings –> Network. For me, this improved network access speed and reduced DNS lookup delays.
  • install VirtualBox Guest Editions which enables seamless mouse integration and experimental Direct3D support by mounting the Guest Editions ISO.
  • if, in the event that Windows Update fails to work, register the Wups2.dll as outlined in the Microsoft Knowledge Base <http://support.microsoft.com/kb/943144>

I extend my thanks to the VirtualBox team for making this so effortless. By the way, anything between 5G – 10G is a good size for your virtual harddisc.

Squashing the PITA!
29 April 2009

Over the last couple of months, I have been nagging myself to fix several of the issues with my Linux installation, in particular I need to:

  • update my kernel (currently 2.6.27-rc9)
  • get the wacom input device working with the newest xorg-server
  • add some snaz to my desktop configuration (openbox, dmenu…bleh)

So, sparing you the `wget`ing and `tar -xf`ing, I upgraded my kernel to 2.6.29-rc2 and enabled the kernel modesetting. This enables me to switch to the console with no delay, instead of the current 2-3 second delay. Good news: the upgrade worked well.

Additionally, I upgraded my xorg-server to 1.6.1: the only problem was wacom tablet didn’t work, so I installed the development version of linuxwacom (version 8.3-3).

Everything works well so far except rendering terminals. When I try to open alsamixer with uxterm or resize it, the terminal behaves as if it has a really low refresh rate. I am looking into the issue and I will post a fix if I find one.

As for the ‘added snaz,’ I’ll deal with that another time.  Right now I’m just happy that my fps in stepmania has increased by about 20!

xf86-video-intel-2.4.3
8 December 2008

The latest Xorg video driver for Intel chipsets will probably cause a slowdown in rendering due to a switch from TTM to GEM. GEM is only supported in kernel 2.6.28 and above so it’s recommended that you stick with drivers released before 2.4.3.

More information: http://bugs.freedesktop.org/show_bug.cgi?id=13922

http://www.x.org/wiki/ttm

Compiling 2.6.27.6 using the vanilla kernel tree sources
15 November 2008

I’m simply going to list the steps.

wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.27.6.tar.bz2
tar xf linux-2.6.27.6.tar.bz2
cd linux-2.6.27.6
zcat /proc/config.gz > .config
make menuconfig
make
su
make modules_install
mkinitcpio -k `make kernelrelease` -g /boot/kernel-`make kernelrelease`.img
cp System.map /boot/System.map-`make kernelrelease`
cp arch/i386/boot/bzImage /boot/vmlinuz-`make kernelrelease`
install -D -m644 .config /boot/kconfig-`make kernelrelease`
/sbin/depmod -A -v `make kernelrelease`
vi /boot/grub/menu.lst
reboot
Upgrading to linux-2.6.27-rc3
19 August 2008

rc3 is another solid release.  Follow the same procedure documented in upgrading to linux-2.6.27-rc2.

The catalyst patch works with this release.

Upgrading to catalyst-8.7
9 August 2008

Choosing to run the latest kernel means choosing to deal with errors. Luckily, my mileage with 2.6.27-rc2 has been pretty good. Here are a few pointers (and an experimental patch) to use when setting up catalyst-8.7 kernel 2.6.27-rc2.

I created the patch from various components incl.:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=485793

Also, note that I do not have any working knowledge of C nor do I take credit for this patch.

Changes to Kernel API (that I’m aware of, at least)

  • smp_call_function.  See commit 5e374fb62621aca9522f76c2317c9acda75a8e88 by Jens Axboe to the file arch/x86/kernel/smp.c
  • find_task_by_pid.  See commit 5cd204550b1a006f2b0c986b0e0f53220ebfd391 by Pavel Emelyanov to the file kernel/pid.c
  • NOPAGE_SIGBUS.  See commit 3c18ddd160d1fcd46d1131d9ad6c594dd8e9af99 by Nick Piggin to the file include/linux/mm.h

Other things to be aware of

  • If you have paravirtualised guest support enabled in the kernel, the compilation will fail with FATAL: modpost: GPL-incompatible module fglrx.ko uses GPL-only symbol ‘pv_lock_ops’. Disable it: PARAVIRT_GUEST=n
  • The catalyst-8.7 driver uses deprecated symbols so if they kernel does not export them (in particular init_mm) the compilation will fail.  Enable them: UNUSED_SYMBOLS=y
  • catalyst-utils-8.7 needs to be installed with this driver

The commands (to set up catalyst-8.7, I’m sure you can setup catalyst-utils by yourself).  Note: If you’re not using the kernel that will be running with the drivers, then replace `uname -r` with the correct kernel version/name.

~ %   mkdir proj/catalyst
~ %   cd proj/catalyst
catalyst %   wget www2.ati.com/drivers/linux/ati-driver-installer-8-7-x86.x86_64.run -O catalyst-8.7.run
catalyst %   wget http://antonyat.googlepages.com/firegl_public.diff
catalyst %   /bin/sh ./catalyst-8.7.run --extract archive_files
I use x86.  x86_64 users should substitute x86 with x86_64 and x710 with x710_64a respectively.
catalyst %   cp archive_files/arch/x86/* ./
catalyst %   cp archive_files/common/* ./
catalyst %   cp archive_files/x710/* ./
catalyst %   cp firegl_public.diff lib/modules/fglrx/build_mod/
catalyst %   cd lib/modules/fglrx/build_mod/
catalyst %   patch -p1 < firegl_public.diff
catalyst %   cp 2.6.x/Makefile .
catalyst %   make -C /lib/modules/`uname -r`/build SUBDIRS="`pwd`" ARCH=i386 modules
catalyst %   su
pts/3:root@GENTOO /home/antony/proj/catalyst/lib/modules/fglrx/build_mod # install -m 644 -D fglrx.ko /lib/modules/`uname -r`/video/fglrx.ko

You should also copy the licenses (in archive_files to /usr/share/licenses)

As usual, your mileage may vary.

Upgrading to linux-2.6.27-rc2
8 August 2008

git makes it incredibly easy to upgrade my kernel to the latest (but not necessarily greatest). I’ve decided that I’ll stick to tagged releases instead of the latest git pull.

I’ll skip the prologue and just show the commands.

Download the latest commits.

linux-git %   git pull  (Note:  Make sure you are in the master branch 🙂 [git checkout master].)

Download the latest tags.

linux-git % git fetch –tags git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

List the current tags.

linux-git % git tag -l “v2.6.27*”

Duplicate the 2.6.27-rc2 branch, call it aj2.6.27-rc2, and switch to it.

linux-git % git checkout -b aj2.6.27-rc2 v2.6.27-rc2

Copy the current config to the kernel directory.

linux-git % zcat /proc/config > .config

Configure any options that changed in the latest version.

linux-git % make oldconfig

Go through the usual compilation steps. See my post on upgrading to 2.6.27-rc1 for more information.

linux-git % make | tee make.log
linux-git % su
pts/2:root@GENTOO /home/antony/proj/linux-git # make modules_install | tee modules.log
pts/2:root@GENTOO /home/antony/proj/linux-git # mkinitcpio -k `make kernelrelease` -g /boot/kernel-2.6.27-rc2.img
pts/2:root@GENTOO /home/antony/proj/linux-git # cp System.map /boot/System.map-2.6.27-rc2
pts/2:root@GENTOO /home/antony/proj/linux-git # cp arch/i386/boot/bzImage /boot/vmlinuz-2.6.27-rc2
pts/2:root@GENTOO /home/antony/proj/linux-git # install -D -m644 .config /boot/kconfig-2.6.27-rc2
pts/2:root@GENTOO /home/antony/proj/linux-git # /sbin/depmod -A -v `make kernelrelease`
pts/2:root@GENTOO /home/antony/proj/linux-git # vim /boot/grub/menu.lst
pts/2:root@GENTOO /home/antony/proj/linux-git # reboot

Problems

Difficulty in setting up catalyst-8.7. See my post on upgrading to catalyst-8.7.

Notes: My kernel config

kconfig-2.6.27-rc2

Built with Wordpress and Vim
© 2008 to 2020 Antony Jepson