logo
Jiff Slater
🤔 About
✍️ Contact
📚Knowledge
📄Posts
📁Archive
Updated: 7 November 2021
<p>Creating privileged Firefox containers using Linux containers</p>

Unprivileged containers

Setting up containers using LXC on Gentoo

What follows are mostly my unformatted notes from getting unprivileged containers to work on an OpenRC host.

Installation

emerge app-emulation/lxc sys-apps/shadow sys-auth/pambase net-firewall/nftables
rc-update add nftables default

Enabling unprivileged containers

Make sure that you have subuids and subgids for your current user.

grep $USER /etc/sub*
/etc/subgid:jiff:0:0
/etc/subuid:jiff:0:0

Here we don't have the mappings so we'll create them by putting the below in both /etc/sub{uid,gid}

jiff:100000:65536

We'll allow the user to create up to 16 veth (virtual Ethernet) devices without root access. Note that each container needs two veth devices. See man lxc-usernet for more info.

Setting up the network

First, we'll turn off automatic creation of bridges and manage it ourselves.

/etc/default/lxc
USE_LXC_BRIDGE="false"

Next, we'll create a basic bridge used for networking. Any connections on the host that we want to handle will need to be forwarded used nftables.

echo '1' | sudo tee -a /proc/sys/net/ipv4/ip_forward
ip link add name xctcontainerbr0 type bridge
ip link set xctcontainerbr0 up
ip addr add 10.10.10.1/24 dev xctcontainerbr0

  1. stateless mostly copied from lxc-net

nft add table inet lxc
nft 'add chain inet lxc postrouting { type nat hook postrouting priority 100; policy accept; }'
nft add rule inet lxc postrouting ip saddr 10.10.10.0/24 ip daddr != 10.10.10.0/24 counter masquerade

  1. example for port forwarding where 10.10.10.3 is the address of the container
 nft 'add chain inet lxc input { type filter hook input priority 0; policy accept; }'
 nft 'add rule inet lxc input iifname xctcontainerbr0 tcp dport { 80, 443 } accept;'
 nft 'add chain inet lxc forward { type filter hook forward priority 100; policy accept; }'
 nft 'add rule inet lxc forward iifname xctcontainerbr0 ip daddr 10.10.10.3 tcp dport {80, 443} ct state new accept'
 nft 'add chain inet lxc prerouting { type nat hook prerouting priority 0; policy accept; }'
 nft 'add rule inet lxc prerouting iifname xctcontainerbr0 ip saddr != 10.10.10.0/24 tcp dport 80 dnat ip to 10.10.10.3:80'
 nft 'add rule inet lxc prerouting iifname xctcontainerbr0 ip saddr != 10.10.10.0/24 tcp dport 443 dnat ip to 10.10.10.3:443'

Save the configuration changes.

# rc-service nftables save

If you make a mistake you view your rules and their associated handles with.

nft --handle --numeric list ruleset
nft delete rule ip nat postrouting handle 4

If the nft commands don't work you'll need to add the right modules to your kernel. See https://wiki.gentoo.org/wiki/Nftables and https://developers.redhat.com/blog/2018/10/22/introduction-to-linux-interfaces-for-virtual-networking for more insight.

Tell LXC which bridge to use as the default.

/etc/lxc/lxc-usernet jiff veth xctcontainerbr0 16

Make the network config permanent. I'm using netifrc https://gitweb.gentoo.org/proj/netifrc.git/tree/doc/net.example.Linux.in

And add it to the default runlevel.

ln -s /etc/init.d/net.lo /etc/init.d/net.xctcontainerbr0
rc-service net.xctcontainerbr0 start
rc-update add net.xctcontainerbr0 default

Next we'll create the configuration file for the guest by setting up a virtual Ethernet pair connected to a bridge. We'll use static IPs for ease of configuration.

 ~/.config/lxc/firefox.conf
 lxc.net.0.type = veth
 lxc.net.0.flags = up
 lxc.net.0.link = xctcontainerbr0
 lxc.net.0.name = eth0
 lxc.net.0.ipv4.gateway = 10.10.10.1
 lxc.net.0.ipv4.address = 10.10.10.3/24
 lxc.idmap = u 0 100000 165536
 lxc.idmap = g 0 100000 165536

Downloading the container

Try downloading the first container with this command. (Try btrfs with -B)

lxc-create -t download -n firefox -B btrfs -f ~/.config/lxc/firefox.conf -- -d alpine -r 3.14 -a amd64
(or) lxc-create -t download -n firefox -f ~/.config/lxc/firefox.conf -- -d gentoo -r current -a amd64
(or) lxc-create -t download -n firefox -B btrfs -f ~/.local/config/lxc/firefox.conf -- -d debian -r bullseye -a amd64
-- if errors with this inside the container use source /etc/profile then env-update
-- if errors also append Ubuntu's keyserver --key-server hkp://keyserver.ubuntu.com in the download options
-- make sure the btrfs root is mounted with user_subvol_rm_allowed
-- if you're using systemd, use lxc-unpriv-attach and lxc-unpriv-start instead

LXC will need access to your local container directories so we'll add an ACL to permit this. You should have execute permission set for the share directories that stores the root directory of the containers. Using the base uid from above that will represent root in the container. This doesn't feel portable at all but isn't too crazy. shiftfs is supposed to make this easier. https://lists.ubuntu.com/archives/kernel-team/2019-March/099623.html

getfacl /home/jiff/.local/share/lxc
setfacl -m u:100000:x /home/jiff/.local/share/lxc

Starting the container

lxc-start -n firefox -d

The container will be running as shown by lxc-ls on the host.

 lxc-info -n firefox
 Name:           firefox
 State:          RUNNING
 PID:            6228
 Link:           veth1000_KD5N
  TX bytes:      858 bytes
  RX bytes:      1.20 KiB
  Total bytes:   2.03 KiB

Look at the connection status.

lxc-attach --clear-env -n firefoxi -- /bin/sh -c "ip addr"

You may get locale errors.

Get a shell in the container with lxc-attach.

lxc-attach -n firefox

Launching Firefox in the container

 # Install all the fonts
 # Remember to set up /etc/resolv.conf for DNS resolution.
 # echo 'nameserver 192.168.1.1' > /etc/resolv.conf'
 apk add font-bh-100dpi font-sun-misc font-bh-lucidatypewriter-100dpi font-adobe-utopia-type1 font-schumacher-misc font-daewoo-misc font-adobe-utopia-75dpi font-bitstream-100dpi font-xfree86-type1 font-bitstream-75dpi font-bh-ttf font-arabic-misc font-dec-misc font-misc-ethiopic font-micro-misc font-isas-misc font-bh-lucidatypewriter-75dpi font-winitzki-cyrillic font-jis-misc font-bitstream-type1 font-mutt-misc font-misc-misc font-adobe-100dpi font-bh-type1 font-bh-75dpi font-sony-misc font-ibm-type1 font-bitstream-speedo font-adobe-utopia-100dpi font-adobe-75dpi font-misc-meltho font-cursor-misc
 # apk add mesa-dri-intel hicolor-icon-theme alsa-lib ca-certificates ttf-dejavu ttf-freefont xorg-server firefox mesa-gles libva-glx mesa-gl mesa-dri-gallium mesa-demos pciutils-dev ffmpeg 
 # apk add 
(or https://download.mozilla.org/?product=firefox-latest-ssl&os=linux64&lang=en-US)
 # Run the command by using invocating Firefox.  You can view the APKBUILD file here https://git.alpinelinux.org/aports/tree/community/firefox/APKBUILD?h=3.14-stable
 # lxc-attach -n firefox -- su -l jiff -c '/usr/bin/firefox --display=:0' 
 ref: https://bugzilla.mozilla.org/show_bug.cgi?id=1676883

And finally, stop the container with lxc-stop.

Getting connectivity inside the container

 lxc-attach -n firefox -- /bin/sh -c "ip show"
 ip -4 addr add 10.10.10.3/24 dev eth0
 ip -4 route add default via 10.10.10.1 dev eth0
 echo 'nameserver $NAMESERVER' > /etc/resolv.conf

If you get ABORTED errors, you can check the logs with. lxc-start -n alpha -o ./log.txt -l TRACE -- /bin/bash

Getting sound in the container

Easiest way to do this is with pipewire. Install pipewire on the host and use it as the PULSE_SERVER by exporting the variable.

Supporting cgroups for systemd guests inside OpenRC hosts

Finally, we'll create the userspace cgroups for the container to run in. This is critically important if you're running these containers on a none OpenRC host.

 sudo emerge -av lxcfs
 sudo rc-service lxcfs start
 sudo rc-update add lxcfs default
 sudo mkdir /sys/fs/cgroup/systemd
 sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,none,name=systemd cgroup /sys/fs/cgroup/systemd
 sudo chmod g+rwX /sys/fs/cgroup/systemd
 sudo chown root:100000 /sys/fs/cgroup/systemd