Best btrfs + luks configuration for longevity with new hard drives
29 June 2021
I’m a big fan of btrfs as the filesystem of choice not only for backups but also for the root filesystem. However, as I move around with my backups sometimes I want to restore them to different machines and I always hesitate when faced with the prospect of formatting, encrypting, and partitioning, and mounting a new drive. I wanted to document how I do it so I can have some consistency as I move from hard drive to hard drive, SSD to SSD.
In general, when I purchase hard drives I purchased two of the same kind. I don’t usually like to go smaller than 2TB and recently I’ve been harbouring a certain liking for 4TB hard drives when I can get my hands on them. I’m a big fan of Western Digital and also of Seagate, especially at these larger ranges. You can view the WD Red series prices here (UK).
BackBlaze has a great website where they list the failure rates of their hard drives (link) so you can start here to prime your instincts.
This will take a while. Just like onboarding new RAM with memtestx86, I’ll test my hard drives for badblocks with, funnily enough, the badblocks tool. Arch Linux has a great wiki article about it here. Here are my default steps for new drives.
Run the smartctl self-tests.
smartctl -t short /dev/sda
smartctl -t conveyance /dev/sda # Used to check that the drive wasn’t damaged in transportation
And view the results
smartctl -A /dev/sda
Next, run the badblocks command. You can speed this up by figuring out the block size of your hard drive.
hdparm -I /dev/sda | grep Sector # Get the block size, usually around 4096
From this mailing list post (link), it seems that modern drives will automatically notice a bad block and re-allocate to another location. So, before cursing the wind if you get a bad block, it’s probably valuable to run it again and see if the error heals itself.
At the expense of write speed, you can use hdparm to turn on the Write-Read-Verify support. This should also result in a lower likelihood of being bitten by writing corrupted data.
Finally, we can use the wipefs tool to make sure there’s no filesystem signatures remaining.
wipefs -a /dev/sda
As shown in the encrypting section, if you really want to wipe the drive you can open the drive in raw dm-crypt mode and wipe it.
cryptsetup open --type plain -d /dev/urandom /dev/sda sda_luks
Usually, there’s no need to do anything fancy here. If the drive is a root drive, I add an EFI partition, a swap partition, a root partition, and leave a small amount of space at the end of the hard drive unpartitioned. The encryption step will be on the root partition (typically /dev/mapper/sda_luks3). If it’s a data drive, I typically encrypt the entire drive and then make a single partition to use for the data.
mkpart "EFI system partition" fat32 1MiB 301MiB
set 1 esp on
mkpart "EFI system partition" btrfs 301MiB -40960MiB
mkpart "Swap space" linux-swap -40959MiB -8192MiB
Depending on if the drive is a root drive or a data drive, the order of the this and the partitioning step can be swapped – especially if you need an EFI partition.
I’ve been using LUKS (Linux Unified Key Setup) for over a decade and I’ve found it rock-solid for hard drive encryption. My settings of choice have evolved over the years and today I use a very modern set of options. Cryptsetup, the administration interface to dmcrypt and LUKS, has a lot of options that let you really (shoot yourself in the foot) customise the configuration. In general, for data drives I use keyfiles instead of passphrases and store an encrypted copy of the keyfile in a different building (virtual or otherwise) than the one in which the drive lives. For root drives, I use a passphrase and store a copy of the LUKS header in a different building.
More recently, the dm-integrity (GitLab; Kernel; presentation) component enables even more assurance that your data is not silently becoming corrupted and I’ve started to enable that as well. It’s noted as experimental so I use a very minor configuration that hasn’t (touch wood) caused problems to date.
One quick thing to note before diving into encryption – it’s important to establish your threat model when using encryption – for me, I’m simply trying to avoid any hiccups that could happen in the event that a drive is stolen.
First, I encrypt the drive using luksFormat. I add a few custom commands for my particular risk tolerance of encryption strength. If speed is a concern, you can first use cryptsetup benchmark and see which algorithms are most performant for your set up. On my machine, argon2id + XTS was the best for reducing number of iterations on a key (if someone is trying to brute force your key) and for performance when reading and writing encrypted data.
Before doing the below, if you’re really paranoid, as per the cryptsetup FAQ (GitLab) you can over-write the hard drive with random data before getting started.
cryptsetup open --type plain -d /dev/urandom /dev/sda sda_luks
Now I can open the device and begin the partitioning and setting up the filesystem.
cryptsetup open --type luks2 /dev/sda sda_luks
Creating the filesystem
The final stage of this abstraction sandwich is to configure btrfs. This is my favourite part: 1) because we’re almost done; and 2) because it’s the abstraction level I deal with the most.
A unique component of btrfs is that is presents a subvolume filesystem on top of the filesystem you see at the command line. I usually don’t build the filesystem structure where the root filesystem is at top of the hierarchy, instead I create a tree like so:
/toplevel # the parent of all subvolumes
/toplevel/savestate # this is where snapshots are stored
/toplevel/rootfs # where the rootfs will go
/toplevel/data # where data is stored, in the case of a data drive
One thing I’ve learned from using btrfs is that if you don’t configure quotas, you will rarely know how much space you’ll save by deleting a given snapshots. So when I usually enable that option when configuring the initial structure.
And there you have it, a very simple but practical way of managing the longevity of your system with a systematic approach to formatting, partitioning, encrypting, and creating filesystems on your hard drives.
Configuring Wireguard on the Pinebook Pro in Manjaro Linux
1 February 2020
I recently (Twitter) ordered and received a Pinebook Pro and wanted to share how I got Wireguard working. Wireguard is a VPN that uses modern cryptography while still being easy to configure for various environments. Unfortunately, even though the kernel module has been merged upstream Manjaro Linux still requires a custom module to be built. Because the kernel sources aren’t included with the distribution as of now, installing the wireguard-dkms package will fail. This post shows how I got the userspace wireguard-go program to work in lieu of the kernel module.
Before I continue, if you’re using the default Debian install that came with the device, you should be able to follow this tutorial which uses Cloudflare’s boringtun Rust implementation. I couldn’t get this tutorial to work so here is an alternative that uses the official Wireguard Go language reference implementation.
Installing the compiler
The Go compiler should be available in all distributions so install it before continuing. On Manjaro Linux you can do so by typing `sudo pamac install go`.
Cloning the repository
You’ll need to clone to source code from the Wireguard repo: `git clone https://git.zx2c4.com/wireguard-go`.
Building the tool
Once cloning has completed, enter the directory and issue `make`. After it completes, you should have ./wireguard-go executable in the same directory.
Launching the tool
Open two terminal windows. In the first, issue sudo LOG_LEVEL=debug ./wireguard-go -f wg0. This will launch the userspace implementation and create an interface called wg0 which you can see by typing `ip a`.
Configuring and bringing up the Wireguard interface
Bringing up the interface is almost as simple as presented in the docs but because we’re running Manjaro Linux we’ll need to make sure it works well with NetworkManager. The first step is mark the interface along with any similarly named interfaces as unmanaged. Create the following file and restart NetworkManager.
In a new terminal window, issue the following commands, taking into account your configuration. Before continuing you’ll also need to have a valid /etc/wireguard/wg0.conf that uses `wg` syntax not wg-quick syntax. Check the manpage for wg to confirm. Note that CLIENT_IP_ADDRESS and PEER_IP_ADDRESS_OR_RANGE refers to the address Wireguard interface address space.
# ip address add dev wg0 CLIENT_IP_ADDRESS peer PEER_IP_ADDRESS_OR_RANGE
# wg setconf wg0 /etc/wireguard/wg0.conf
# ip link set mtu 1420 up dev wg0
# ip route add PEER_IP_ADDRESS_OR_RANGE dev wg0
Finally, as per Thaller’s post on the GNOME blogs, if you don’t issue the last command we’ll need to let NetworkManager know about the new route. List your current connections with nmcli conn show and copy the UUID for your current connection below. Replace GATEWAY and WIREGUARD_ENDPOINT with the actual IP addressses.
This should be sufficient to set up the VPN. You’ll see the handshake initiated and completed in the other terminal window.
Let me know if this worked for you. DNS resolution is still problematic because NetworkManager doesn’t adjust resolvconf to accomodate the new route. If you manage to get that working correctly, please let me know on Twitter.
ifconfig Output on macOS Mojave
28 November 2018
ifconfig is a venerable tool that’s been around since 2008, according to the manpage on macOS. If you have a recent device, you’ll discover a much larger number of devices than on Linux.
On the first, a simulated virtual USB Host Controller Interface, we can see many iBridge devices connected, including the display, ambient light sensor, camera, microphone, keyboard/trackpad, and something called a DFR brightness. The closest related acronym I could find was digital feedback reducer. In any case, all the Apple devices are exposed via this virtual USB hub. The address of the AppleUSBVHCIBCE is 128 so that explains the VHC128 interface on the ifconfig output.
On the second, an eXtensible Host Controller Inteface (HCI) for Universal Serial Bus (USB), will show any devices you have connected via the ports on your Mac. If we dig deeper into the AppleUSBHostController,
$ ioreg -w0 -rc AppleUSBHostController
[… output truncated …]
We can see one AppleIntelCNLUSBXHCI client which shares the same PCI bus with the four AppleUSB20XHCITypeCPort clients. Furthermore, we can see two AppleUSBXHCITR , each of which having two of the AppleUSB30XHCITypeCPort class, likely the USB 3.0 controllers. Important to note is that the addresses of the clients with the AppleUSBXHCITR class: XHC1@14, XHC2@00, and XHC3@01, giving us the three devices we see in the ifconfig output: XHC20, XHC0, and XHC1, if you we take the high byte from the addresses. It’s likely the numbers after XHC are their sequential locations on the bus.
From these we can conclude that these are merely the USB debugging interfaces, both for the internal clients and external clients.
Using ioreg to dig deeper into the relationship between entities wasn’t as easy as I’m used to on Linux by using lsusb. This leads me to believe that perhaps ioreg is deprecated and there’s a better utility available or some of this information is not exposed completely in publically available tools.
In any case, knowing the source of these interfaces certainly leaves me less perturbed.
Recently I built my first gaming rig (also my first PC build) with a measly
budget of $500.
Case: Antec 300
CPU: Intel Pentium E6300 Wolfdale
PSU: Antec EA650 650W
GPU: Sapphire Radeon HD 4830
HDD: Western Digital Cavier SE 320GB
Mobo: Gigabyte GA-G31-ES2l LGA 775 Intel µATX
RAM: OCZ SLI-Ready 240-pin DDR2 SDRAM DDR2 800
Everything was purchased from Newegg and thanks to a couple combo discounts this build was less than $500 including shipping and applicable tax. It was a tough decision. For example, I switched from a NVIDIA GeForce 9800 GT to a NVIDIA GeForce 8600 GTS, finally settling with a AMD Radeon HD 4830. Fortunately, most of the components are overclockable; with the addition of an aftermarket cooler, I will be flying in no time. As an added bonus, the CPU supports VT which will come in handy for running paravirtualised guests under Linux and Windows XP under Windows 7.
Some thoughts for the future: In the next build, the OCZ memory will be replaced by the Corsair XMS brand and I’m definitely going to consider the Antec Sonata III case.
Update: My build is complete! Although I haven’t installed Linux on it yet, I do get a WEI score of 5.5 (highest is 5.9). I’m quite impressed with the performance of this build as well as the ventilation of the case. Temperatures range from the low thirties (celsius) (for both the CPU and the GPU) under a light load to no higher than 50 degrees for a high load.
Arch Linux and the HP Compaq 2710p
28 October 2008
I recently received a HP Compaq 2710p and I’m quite surprised by the performance of this convienent tablet.
After using it for a few weeks, I can say that it works well with GNU/Linux (Arch Linux in particular). There are a few issues, however:
Use of the stylus occasionally conflicts with the default mouse.
Brightness of the screen is fixed (unchangeable) upon resume from suspend (both from disk and RAM).
Here is some information about the devices within the computer.
[antony@ARCH ~]$ lspci
00:00.0 Host bridge: Intel Corporation Mobile PM965/GM965/GL960 Memory Controller Hub (rev 0c)
00:02.0 VGA compatible controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c)
00:02.1 Display controller: Intel Corporation Mobile GM965/GL960 Integrated Graphics Controller (rev 0c)
00:19.0 Ethernet controller: Intel Corporation 82566MM Gigabit Network Connection (rev 03)
00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #5 (rev 03)
00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 03)
00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2 (rev 03)
00:1c.2 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 3 (rev 03)
00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev f3)
00:1f.0 ISA bridge: Intel Corporation 82801HBM (ICH8M-E) LPC Interface Controller (rev 03)
00:1f.1 IDE interface: Intel Corporation 82801HBM/HEM (ICH8M/ICH8M-E) IDE Controller (rev 03)
02:09.0 FireWire (IEEE 1394): Ricoh Co Ltd R5C832 IEEE 1394 Controller (rev 05)
02:09.1 SD Host controller: Ricoh Co Ltd R5C822 SD/SDIO/MMC/MS/MSPro Host Adapter (rev 22)
02:09.2 System peripheral: Ricoh Co Ltd R5C843 MMC Host Controller (rev 12)
10:00.0 Network controller: Intel Corporation PRO/Wireless 4965 AG or AGN Network Connection (rev 61)
[antony@ARCH ~]$ lsusb
Bus 007 Device 001: ID 1d6b:0002
Bus 002 Device 001: ID 1d6b:0001
Bus 006 Device 001: ID 1d6b:0001
Bus 004 Device 003: ID 0461:4d17 Primax Electronics, Ltd
Bus 004 Device 002: ID 08ff:2580 AuthenTec, Inc.
Bus 004 Device 001: ID 1d6b:0001
Bus 005 Device 001: ID 1d6b:0001
Bus 003 Device 004: ID 049f:0081 Compaq Computer Corp.
Bus 003 Device 003: ID 04b4:6560 Cypress Semiconductor Corp. CY7C65640 USB-2.0 "TetraHub"
Bus 003 Device 001: ID 1d6b:0002
Bus 001 Device 004: ID 03f0:171d Hewlett-Packard
Bus 001 Device 001: ID 1d6b:0001
[antony@ARCH ~]$ uname -a
Linux ARCH 2.6.27-rc9 #2 SMP PREEMPT Thu Oct 9 00:58:03 EDT 2008 i686 Intel(R) Core(TM)2 Duo CPU U7600 @ 1.20GHz GenuineIntel GNU/Linux
[antony@ARCH ~]$ df -h
Filesystem Size Used Avail Use% Mounted on/dev/sda5 15G 2.3G 12G 17% /none 1009M 0 1009M 0% /dev/shm/dev/sda2 92M 15M 73M 17% /boot/dev/sda6 15G 1.3G 13G 10% /home
I’ve kept the system relatively lightweight. Some applications that work well with this computer are:
mutt: imap gmail access
pidgin: instant messenger
wget: download manager
dwm: window manager
netcfg: network manager
This computer is really convienent to carry around because it is so lightweight. Soon, I will upgrade to Xorg 7.4 and see if the wacom driver has improved.