SystemD killing local user processes by default

Or, how I spent three hours debugging why mosh and screen weren't working.

I've used my Pinebook Pro in a variety of places now. It's a really great and portable computing device. I've also used it with a variety of connection types, 4G, WiFi, Ethernet, you name it. Unfortunately, while I've been a huge fan of Mosh in the past, it didn't seem to work when connecting to my personal machine over Wireguard. SSH worked admirably but I kept getting an error with Mosh that the connection server never started.

I spent a long time (and drank a lot of tea) debugging routers, port forwarding settings, firewalls, NAT configuration, and even routes. No dice. A seemingly unrelated issue that was also irritating was that I found my screen sessions on the same computer ended as soon as my SSH session ended. After a while, I managed to reliably reproduce this issue by running commands in screen and disconnecting SSH using the hang up command (~.)

It soon became clear that my login sessions weren't being persisted after disconnecting and something was killing them. This was why mosh wasn't working because after mosh did the initial connection over SSH and started mosh-server, it closed the SSH connection and mosh-server was terminated.

The resolution for both issues is as follows. SystemD changed a default configuration to kill processes after a user session ends. This means by default, nohup, terminal multiplexers, or any long-running command with & will no longer work.:

$ loginctl enable-linger
# systemctl restart systemd-logind
$ mosh --server="systemd-run --user --scope mosh-server" user@host
$ systemd-run --user --scope screen

While I'm glad I've resolved the issue, I'm not sure who's at fault here. Is it SystemD for changing the defaults? Is it the distro maintainers for going forward with the change? Is it me for not being more aware of what happened back in 2016? You can read more posts from the angry community here, here, here, here, here, and here.