logo
Jiff Slater
🤔 About
✍️ Contact
📚Knowledge
28 Jul 2021
 

Best btrfs + luks configuration for longevity with new hard drives
29 June 2021

I’m a big fan of btrfs as the filesystem of choice not only for backups but also for the root filesystem.  However, as I move around with my backups sometimes I want to restore them to different machines and I always hesitate when faced with the prospect of formatting, encrypting, and partitioning, and mounting a new drive.  I wanted to document how I do it so I can have some consistency as I move from hard drive to hard drive, SSD to SSD.

Selecting

In general, when I purchase hard drives I purchased two of the same kind.  I don’t usually like to go smaller than 2TB and recently I’ve been harbouring a certain liking for 4TB hard drives when I can get my hands on them.  I’m a big fan of Western Digital and also of Seagate, especially at these larger ranges.  You can view the WD Red series prices here (UK).

BackBlaze has a great website where they list the failure rates of their hard drives (link) so you can start here to prime your instincts.

Formatting

This will take a while.  Just like onboarding new RAM with memtestx86, I’ll test my hard drives for badblocks with, funnily enough, the badblocks tool.  Arch Linux has a great wiki article about it here.  Here are my default steps for new drives.

Run the smartctl self-tests.

smartctl -t short /dev/sda

smartctl -t conveyance /dev/sda # Used to check that the drive wasn’t damaged in transportation

And view the results

smartctl -A /dev/sda

Next, run the badblocks command.  You can speed this up by figuring out the block size of your hard drive.

hdparm -I /dev/sda | grep Sector # Get the block size, usually around 4096

badblocks -b 4096 -ws -o badblocks-dev-sda.txt /dev/sda

From this mailing list post (link), it seems that modern drives will automatically notice a bad block and re-allocate to another location.  So, before cursing the wind if you get a bad block, it’s probably valuable to run it again and see if the error heals itself.

At the expense of write speed, you can use hdparm to turn on the Write-Read-Verify support.  This should also result in a lower likelihood of being bitten by writing corrupted data.

Finally, we can use the wipefs tool to make sure there’s no filesystem signatures remaining.

wipefs -a /dev/sda

As shown in the encrypting section, if you really want to wipe the drive you can open the drive in raw dm-crypt mode and wipe it.

cryptsetup open --type plain -d /dev/urandom /dev/sda sda_luks

Partitioning

Usually, there’s no need to do anything fancy here.  If the drive is a root drive, I add an EFI partition, a swap partition, a root partition, and leave a small amount of space at the end of the hard drive unpartitioned.  The encryption step will be on the root partition (typically /dev/mapper/sda_luks3).  If it’s a data drive, I typically encrypt the entire drive and then make a single partition to use for the data.

parted /dev/sda

mklabel gpt

mkpart "EFI system partition" fat32 1MiB 301MiB

set 1 esp on

mkpart "EFI system partition" btrfs 301MiB -40960MiB

mkpart "Swap space" linux-swap -40959MiB -8192MiB

Encrypting

Depending on if the drive is a root drive or a data drive, the order of the this and the partitioning step can be swapped – especially if you need an EFI partition.

I’ve been using LUKS (Linux Unified Key Setup) for over a decade and I’ve found it rock-solid for hard drive encryption.  My settings of choice have evolved over the years and today I use a very modern set of options.  Cryptsetup, the administration interface to dmcrypt and LUKS, has a lot of options that let you really (shoot yourself in the foot) customise the configuration.  In general, for data drives I use keyfiles instead of passphrases and store an encrypted copy of the keyfile in a different building (virtual or otherwise) than the one in which the drive lives.  For root drives, I use a passphrase and store a copy of the LUKS header in a different building.

More recently, the dm-integrity (GitLab; Kernel; presentation) component enables even more assurance that your data is not silently becoming corrupted and I’ve started to enable that as well.  It’s noted as experimental so I use a very minor configuration that hasn’t (touch wood) caused problems to date.

One quick thing to note before diving into encryption – it’s important to establish your threat model when using encryption – for me, I’m simply trying to avoid any hiccups that could happen in the event that a drive is stolen.

First, I encrypt the drive using luksFormat.  I add a few custom commands for my particular risk tolerance of encryption strength.  If speed is a concern, you can first use cryptsetup benchmark and see which algorithms are most performant for your set up.  On my machine, argon2id + XTS was the best for reducing number of iterations on a key (if someone is trying to brute force your key) and for performance when reading and writing encrypted data.

Before doing the below, if you’re really paranoid, as per the cryptsetup FAQ (GitLab) you can over-write the hard drive with random data before getting started.

cryptsetup open --type plain -d /dev/urandom /dev/sda sda_luks

dd if=/dev/zero of=/dev/sda_luks oflag=direct status=progress

cryptsetup close sda_luks

cryptsetup --key-size 512 --hash whirlpool --iter-time 5000 --use-random --cipher aes-xts-plain64 --pbkdf-memory=4194304 --pbkdf=argon2id --integrity hmac-sha256 luksFormat --type luks2 /dev/sda

If it’s a data drive, I’ll generate a key-file and use that.

dd if=/dev/random of=./sda-keyfile bs=512

cryptsetup --key-size 512 --key-file ./sda-keyfile --hash whirlpool --iter-time 5000 --use-random --cipher aes-xts-plain64 --pbkdf-memory=4194304 --pbkdf=argon2id --integrity hmac-sha256 luksFormat --type luks2 /dev/sda

Next, I immediately back up the header and store it somewhere safe.

cryptsetup luksHeaderBackup --header-backup-file /root/sda_header.luksHeader /dev/sda

Now I can open the device and begin the partitioning and setting up the filesystem.

cryptsetup open --type luks2 /dev/sda sda_luks

Creating the filesystem

The final stage of this abstraction sandwich is to configure btrfs.  This is my favourite part: 1) because we’re almost done; and 2) because it’s the abstraction level I deal with the most.

A unique component of btrfs is that is presents a subvolume filesystem on top of the filesystem you see at the command line.  I usually don’t build the filesystem structure where the root filesystem is at top of the hierarchy, instead I create a tree like so:

/toplevel # the parent of all subvolumes

/toplevel/savestate # this is where snapshots are stored

/toplevel/rootfs # where the rootfs will go

/toplevel/data # where data is stored, in the case of a data drive

One thing I’ve learned from using btrfs is that if you don’t configure quotas, you will rarely know how much space you’ll save by deleting a given snapshots.  So when I usually enable that option when configuring the initial structure.

mkfs.btrfs --csum xxhash /dev/mapper/sda_luks
mkdir /toplevel
mount -o autodefrag=on,compress=lzo /dev/mapper/sda_luks /toplevel
btrfs quota enable /toplevel
btrfs subvolume create /toplevel/rootfs
btrfs subvolume create /toplevel/rootfs/gentoo
btrfs subvolume create /toplevel/savestate
btrfs subvolume create /toplevel/storage

Summary

And there you have it, a very simple but practical way of managing the longevity of your system with a systematic approach to formatting, partitioning, encrypting, and creating filesystems on your hard drives.

[Note] Playing videos on a Raspberry Pi Zero W
15 January 2021

Just a quick note – to play videos on a Raspberry Pi – use omxplayer instead of mplayer for maximum performance.

Configuring Wireguard on the Pinebook Pro in Manjaro Linux
1 February 2020

I recently (Twitter) ordered and received a Pinebook Pro and wanted to share how I got Wireguard working. Wireguard is a VPN that uses modern cryptography while still being easy to configure for various environments.  Unfortunately, even though the kernel module has been merged upstream Manjaro Linux still requires a custom module to be built.  Because the kernel sources aren’t included with the distribution as of now, installing the wireguard-dkms package will fail.  This post shows how I got the userspace wireguard-go program to work in lieu of the kernel module.

Before I continue, if you’re using the default Debian install that came with the device, you should be able to follow this tutorial which uses Cloudflare’s boringtun Rust implementation.  I couldn’t get this tutorial to work so here is an alternative that uses the official Wireguard Go language reference implementation.

Installing the compiler

The Go compiler should be available in all distributions so install it before continuing.  On Manjaro Linux you can do so by typing `sudo pamac install go`.

Cloning the repository

You’ll need to clone to source code from the Wireguard repo: `git clone https://git.zx2c4.com/wireguard-go`.

Building the tool

Once cloning has completed, enter the directory and issue `make`.  After it completes, you should have ./wireguard-go executable in the same directory.

Launching the tool

Open two terminal windows.  In the first, issue sudo LOG_LEVEL=debug ./wireguard-go -f wg0.  This will launch the userspace implementation and create an interface called wg0 which you can see by typing `ip a`.

Configuring and bringing up the Wireguard interface

Bringing up the interface is almost as simple as presented in the docs but because we’re running Manjaro Linux we’ll need to make sure it works well with NetworkManager.  The first step is mark the interface along with any similarly named interfaces as unmanaged.  Create the following file and restart NetworkManager.

/etc/NetworkManager/conf.d/wireguard-unmanaged.conf

[keyfile]
unmanaged-devices=interface-name:wg*

# systemctl restart NetworkManager

In a new terminal window, issue the following commands, taking into account your configuration.  Before continuing you’ll also need to have a valid /etc/wireguard/wg0.conf that uses `wg` syntax not wg-quick syntax.  Check the manpage for wg to confirm.  Note that CLIENT_IP_ADDRESS and PEER_IP_ADDRESS_OR_RANGE refers to the address Wireguard interface address space.

# ip address add dev wg0 CLIENT_IP_ADDRESS peer PEER_IP_ADDRESS_OR_RANGE
# wg setconf wg0 /etc/wireguard/wg0.conf
# ip link set mtu 1420 up dev wg0
# ip route add PEER_IP_ADDRESS_OR_RANGE dev wg0

Finally, as per Thaller’s post on the GNOME blogs, if you don’t issue the last command we’ll need to let NetworkManager know about the new route.  List your current connections with nmcli conn show and copy the UUID for your current connection below.  Replace GATEWAY and WIREGUARD_ENDPOINT with the actual IP addressses.

nmcli connection modify UUID +ipv4.routes "WIREGUARD_ENDPOINT/32 GATEWAY"

This should be sufficient to set up the VPN.  You’ll see the handshake initiated and completed in the other terminal window.

Let me know if this worked for you.  DNS resolution is still problematic because NetworkManager doesn’t adjust resolvconf to accomodate the new route.  If you manage to get that working correctly, please let me know on Twitter.

ifconfig Output on macOS Mojave
28 November 2018

ifconfig is a venerable tool that’s been around since 2008, according to the manpage on macOS. If you have a recent device, you’ll discover a much larger number of devices than on Linux.

Here’s the simplified output from my computer.

$ ifconfig | sed -E 's/[[:space:]:].*//;/^$/d' | sort
VHC128
XHC0
XHC1
XHC20
ap1
awdl0
bridge0
en0
en1
en2
en3
en4
en5
gif0
lo0
p2p0
stf0
utun0
utun2
utun3

Let’s walk through them individually.

lo0 is the loopback interface. This is used for the machine to refer to itself.

gif0 is the software network interface.

stf0 is the IPv6 to IPv4 interface.

gif0 is a tunnel interface for IPv4 to IPv6.

p2p0 is airdrop.

en0 is the WiFi interface.

en1 through en4 are the Thunderbolt interfaces, first through fourth.

bridge0 is the Thunderbolt bridge. typically for transfering files over cable between two Macs.

awdl0 is the Apple Wireless Direct Link, typically used for Hotspot functionality with iOS devices and your Apple computer.

ap1 is probably related to the above but I can’t confirm.

en5 is iBridge adapter for the TouchBar.

utunN are related to the sharing of information between devices on the same iCloud account. They can also be created by any VPN interfaces you’ve added.

XHC20, XHC0, XHC1, and VHC128 are something I’ve never seen before with an ifconfig output. I assume they are related to USB controllers. Let’s look at the IOUSB registry plane for more details.

$ ioreg -p IOUSB
+-o Root [...]
 +-o AppleUSBVHCIBCE Root Hub Simulation@80000000 [...]
 +-o AppleUSBXHCI Root Hub Simulation@14000000 [...]

Here we can see two virtual USB hub simulators.

On the first, a simulated virtual USB Host Controller Interface, we can see many iBridge devices connected, including the display, ambient light sensor, camera, microphone, keyboard/trackpad, and something called a DFR brightness. The closest related acronym I could find was digital feedback reducer. In any case, all the Apple devices are exposed via this virtual USB hub. The address of the AppleUSBVHCIBCE is 128 so that explains the VHC128 interface on the ifconfig output.

On the second, an eXtensible Host Controller Inteface (HCI) for Universal Serial Bus (USB), will show any devices you have connected via the ports on your Mac. If we dig deeper into the AppleUSBHostController,

$ ioreg -w0 -rc AppleUSBHostController

[… output truncated …]

We can see one AppleIntelCNLUSBXHCI client which shares the same PCI bus with the four AppleUSB20XHCITypeCPort clients. Furthermore, we can see two AppleUSBXHCITR , each of which having two of the AppleUSB30XHCITypeCPort class, likely the USB 3.0 controllers. Important to note is that the addresses of the clients with the AppleUSBXHCITR class: XHC1@14, XHC2@00, and XHC3@01, giving us the three devices we see in the ifconfig output: XHC20, XHC0, and XHC1, if you we take the high byte from the addresses. It’s likely the numbers after XHC are their sequential locations on the bus.

From these we can conclude that these are merely the USB debugging interfaces, both for the internal clients and external clients.

Using ioreg to dig deeper into the relationship between entities wasn’t as easy as I’m used to on Linux by using lsusb. This leads me to believe that perhaps ioreg is deprecated and there’s a better utility available or some of this information is not exposed completely in publically available tools.

In any case, knowing the source of these interfaces certainly leaves me less perturbed.

Sources

https://apple.stackexchange.com/questions/47477/unexpected-interfaces-in-ifconfig

https://github.com/RehabMan/OS-X-USB-Inject-All

https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/extensible-host-controler-interface-usb-xhci.pdf

https://www.intel.com/content/www/us/en/support/products/65855/software/chipset-software/intel-usb-3-0-extensible-host-controller-driver.html

https://github.com/Dunedan/mbp-2016-linux/issues/71

https://forums.developer.apple.com/thread/95380

https://developer.apple.com/library/archive/documentation/DeviceDrivers/Conceptual/IOKitFundamentals/Features/Features.html

https://developer.apple.com/library/archive/documentation/DeviceDrivers/Conceptual/IOKitFundamentals/TheRegistry/TheRegistry.html

https://www.ifixit.com/Teardown/MacBook+Pro+13-Inch+Touch+Bar+2018+Teardown/111384

https://duo.com/blog/apple-imac-pro-and-secure-storage

Budget Intel Gaming Rig ($500)
25 June 2009

Recently I built my first gaming rig (also my first PC build) with a measly
budget of $500.

The specs:

  • Case: Antec 300
  • CPU: Intel Pentium E6300 Wolfdale
  • PSU: Antec EA650 650W
  • GPU: Sapphire Radeon HD 4830
  • HDD: Western Digital Cavier SE 320GB
  • Mobo: Gigabyte GA-G31-ES2l LGA 775 Intel µATX
  • RAM: OCZ SLI-Ready 240-pin DDR2 SDRAM DDR2 800

Everything was purchased from Newegg and thanks to a couple combo discounts this build was less than $500 including shipping and applicable tax. It was a tough decision. For example, I switched from a NVIDIA GeForce 9800 GT to a NVIDIA GeForce 8600 GTS, finally settling with a AMD Radeon HD 4830. Fortunately, most of the components are overclockable; with the addition of an aftermarket cooler, I will be flying in no time. As an added bonus, the CPU supports VT which will come in handy for running paravirtualised guests under Linux and Windows XP under Windows 7.

Some thoughts for the future: In the next build, the OCZ memory will be replaced by the Corsair XMS brand and I’m definitely going to consider the Antec Sonata III case.

Update: My build is complete! Although I haven’t installed Linux on it yet, I do get a WEI score of 5.5 (highest is 5.9). I’m quite impressed with the performance of this build as well as the ventilation of the case. Temperatures range from the low thirties (celsius) (for both the CPU and the GPU) under a light load to no higher than 50 degrees for a high load.

Arch Linux and the HP Compaq 2710p
28 October 2008

I recently received a HP Compaq 2710p and I’m quite surprised by the performance of this convienent tablet.

After using it for a few weeks, I can say that it works well with GNU/Linux (Arch Linux in particular).  There are a few issues, however:

  • Use of the stylus occasionally conflicts with the default mouse.
  • Brightness of the screen is fixed (unchangeable) upon resume from suspend (both from disk and RAM).

Here is some information about the devices within the computer.

[antony@ARCH ~]$ lspci
00:00.0 Host bridge: Intel Corporation Mobile PM965/GM965/GL960 Memory Controller Hub (rev 0c)
00:02.0 VGA compatible controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c)
00:02.1 Display controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c)
00:19.0 Ethernet controller: Intel Corporation 82566MM Gigabit Network Connection (rev 03)
00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #5 (rev 03)
00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 03)
00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2 (rev 03)
00:1c.2 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 3 (rev 03)
00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f3)
00:1f.0 ISA bridge: Intel Corporation 82801HBM (ICH8M-E) LPC Interface Controller (rev 03)
00:1f.1 IDE interface: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) IDE Controller (rev 03)
02:09.0 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller (rev 05)
02:09.1 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter (rev 22)
02:09.2 System peripheral: Ricoh Co Ltd R5C843 MMC Host Controller (rev 12)
10:00.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN Network Connection (rev 61)

[antony@ARCH ~]$ lsusb
Bus 007 Device 001: ID 1d6b:0002
Bus 002 Device 001: ID 1d6b:0001
Bus 006 Device 001: ID 1d6b:0001
Bus 004 Device 003: ID 0461:4d17 Primax Electronics, Ltd
Bus 004 Device 002: ID 08ff:2580 AuthenTec, Inc.
Bus 004 Device 001: ID 1d6b:0001
Bus 005 Device 001: ID 1d6b:0001
Bus 003 Device 004: ID 049f:0081 Compaq Computer Corp.
Bus 003 Device 003: ID 04b4:6560 Cypress Semiconductor Corp. CY7C65640 USB-2.0 "TetraHub"
Bus 003 Device 001: ID 1d6b:0002
Bus 001 Device 004: ID 03f0:171d Hewlett-Packard
Bus 001 Device 001: ID 1d6b:0001 
[antony@ARCH ~]$ uname -a
Linux ARCH 2.6.27-rc9 #2 SMP PREEMPT Thu Oct 9 00:58:03 EDT 2008 i686 Intel(R) Core(TM)2 Duo CPU U7600 @ 1.20GHz GenuineIntel GNU/Linux
[antony@ARCH ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5              15G  2.3G   12G  17% /
none                 1009M     0 1009M   0% /dev/shm
/dev/sda2              92M   15M   73M  17% /boot
/dev/sda6              15G  1.3G   13G  10% /home

I’ve kept the system relatively lightweight. Some applications that work well with this computer are:

  • mutt: imap gmail access
  • audacious: music
  • pidgin: instant messenger
  • wget: download manager
  • uxterm: terminal
  • firefox: browser
  • dosbox: games
  • dwm: window manager
  • bash: shell
  • netcfg: network manager

This computer is really convienent to carry around because it is so lightweight.  Soon, I will upgrade to Xorg 7.4 and see if the wacom driver has improved.

Built with Wordpress and Vim
© 2008 to 2021 Jiff Slater