It's Pi all the way down...

One of Jammy's more "radioactive" wallpapers; a series of waves fading from a bright purple to a positively eye-searing orange. I rather like it!

It’s now several months after jammy’s release, and thus long past time since I should’ve posted something about it. I’ve been holding back on this because, despite being an LTS (in fact, the first Ubuntu LTS desktop for the Pi), I can’t say I’m happy with it.


Still, that’s why we don’t enable upgrades for LTS users until the first point release (planned for August 4th at the time of writing). So let’s go through the painful bits and see what early adopters can rectify before I get my nose back to the grindstone and try and fix all the bits I’ve messed up!

In some ways, this may be considered a spiritual successor to my earlier post (and even earlier post) on the Pi desktops. Or to put it another way, consider this the current “Dave recommends you run your Ubuntu Pi this way” post (until I supersede it again)!


I’m going to take the unusual step of stating quite explicitly that the following (like everything on this site) is my personal opinion. This is my recommendation for running your Ubuntu Pi, not anyone else’s (including Canonical).

So, with disclaimers thoroughly disclaimed, let us begin …


It’s the serial complaint: the default browser (Firefox) is now a snap. I’m not going to get into the fiery debate over snaps here (more beer is required for that!). Nor am I going to address the start-up speed issues (they’ve been furiously hacked upon and thoroughly covered elsewhere; in fact I was a small cog in the team that did some of that work). Besides, I’m more concerned with the performance of the browser after it’s started (a browser isn’t something I fire up more than once a day), and in that regard I’ve no particular complaints about the performance of the snapped version over the deb version.

And yet, complaints I have …

There are currently issues with the Firefox snap which, personally, I consider deal-breakers. In particular LP: #1741074 “chrome-gnome-shell extension fails to detect native host connector” (aka the “native messaging issue”). The bug’s title may sound relatively innocuous but in practice this translates to “I can’t use KeepassXC” (my password database of choice) with my browser.

I would fallback to KeepassXC’s awesome auto-type feature, but that doesn’t work under Wayland (to be clear: this isn’t exactly a defect, more a design choice; the linked thread has more details). Hence, at least with the version of jammy as released, my only option to use my (now pretty extensive, and certainly indispensable) password database is to copy and paste usernames and passwords via the clipboard … which just about any process can view. Obviously, this is a fairly major step backwards in security and not one I’m willing to put up with.

I should stress this is another issue that is top of the list and being furiously attacked with many sharp objects. Once it’s fixed, I’ll have no specific qualms about returning to the snap (after all, Mozilla seem to prefer it as a means of distribution). But what to do in the meantime?

Switch to the flatpak? No good: the aforementioned bug applies to any confined (snap or otherwise) browser. Besides … the Firefox flatpak is only available for amd64!

Switch to Chromium? It’s snapped too. That said, I’m not sure I would even if I could: uBlock Origin (an extension I consider damned near mandatory for anyone wanting to browse the modern web on anything less than a beefy Ryzen with 32GB of RAM) is better in Firefox than in Chromium.

What about full-fat Chrome? Not an option; Google only distribute Chrome packages for amd64 too.


Rant: frankly, this is one of my biggest misgivings about snap, flatpak, and all these distribution methods that purport to put “distribution in the hands of the developers”: they all suck at providing for anything other than PC architectures leaving stuff like the Pi (or RISC-V, or any new architecture) high and dry.

So, until this is fixed, let’s just use the deb:

$ sudo add-apt-repository ppa:mozillateam/ppa
... lots of output, press Enter ...
$ cat << EOF | sudo tee /etc/apt/preferences.d/firefox
Package: firefox*
Pin: release=LP-PPA-mozillateam
Pin-Priority: 501

Package: firefox*
Pin: release o=Ubuntu
Pin-Priority: -1
$ sudo apt purge firefox
$ sudo snap remove firefox
$ sudo apt install firefox


Another cereal complaint (I make no excuses for the puns here): the introduction of systemd-oomd! In fact, I was rather glad of this one but I also think it’s less of an issue on the Pi than it is the PC, for reasons I’ll go into …


This section is partly redundant because, while writing it, one of my colleagues (Nick, who despite being fresh-faced and new, jumped headlong into the fray) pushed a new configuration that substantially solves things. See LP: #1972159 for the full story.

However, there’s still some useful stuff in here for users on smaller memory systems (including non-Pis!) so I’ve left it in.

Firstly, a little context. For those not aware, systemd-oomd is a new facility for killing memory hogging processes (now activated by default on our desktop images). Previously, this was solely the domain of the dreaded OOM killer in the Linux kernel. While this certainly did the job of killing memory hogs, it was typically only capable of doing so after the system had descended into “swap hell”. In other words, only once something ridiculously fat had pushed everything else on the system to page itself out to disk, did the kernel OOM killer wake up and stalk the process table looking for potential victims.

A frame from Futurama's "Hell is Other Robots" depicting the Beastie Boys scratching a hard drive. A not so subtle allusion to "swap hell".

An artist’s impression of “swap hell”

This typically meant that, while things did get remedied eventually, it took several minutes of the machine “thrashing” for that to happen. In the noughties, I might listen for the audible cue of an HDD thrashing its heads to have a clue this was going on, but this is now and SSDs (or SD cards) don’t provide quite the same feedback!

As a rough rule of thumb, people will assume something has crashed after roughly 5 seconds of the UI appearing to freeze. If that sounds short, consider how many seconds it would take before you assumed a crash if even your mouse pointer is frozen (or jumpy to the point that someone might reasonably assume something was “wrong”). In practice this typically meant that, long before the kernel OOM killer got involved, the user’s itchy power-button finger got involved first.

To be fair, systemd-oomd does fix several issues with the kernel’s OOM killer. The most notable being that a modern application may have many “processes” and (being process-oriented) the kernel’s OOM killer would only kill the heaviest individual process (regardless of whether it was a mere component in a larger system). That potentially left the rest of the application in a unknown / unstable state. By contrast, systemd-oomd will kill the whole cgroup consistently (cgroup in this context can be read as “application” … sort of).

It also acts before the system ever gets near “swap hell”, basing its actions on the system reaching some (fairly arbitrary) thresholds like 90% of both RAM and swap being utilized (to activate), and pressure stall information (to determine what to act upon).

Unfortunately there are some major issues:

  • Most browsers (presumably in acknowledgement of how much fat is dripping off the silk of the web these days) already have mechanisms to “unload” their heavier processes (usually individual tabs) when they detect the system is “under load”. However, being fine-tuned to the individual application, these usually kick in later than systemd-oomd will (or to put it another way, by the time they would kick in, systemd-oomd has already killed the whole browser).
  • systemd-oomd has no user interaction or even notification. One second you’re working in your browser and the next: blip. It’s gone. No warning, no chance to intervene, and not so much as an apology after the fact!

On the upside, it all happens very quickly: there’s no wading through the treacle of swap-thrashing. But it feels … arbitrary, unpredictable, and frankly “flaky”.

There is a simple solution: get more RAM! On my machines with 8GB (or more) of RAM, I simply haven’t encountered the issue. It’s only on machines with 4GB or less that this seems to be a serious issue (at least, that’s my personal experience — I’m sure others with different workloads may find differently). Still, that includes the Pi 400 I use as a desktop, and also my tiddly laptop (which has 4GB of RAM and a Celeron processor — I like my machines small and cheap!).

And this is where things (surprisingly!) turned out better on the Pi than on the PC:

On my little 4GB laptop (which doesn’t really get much use, only when I’m away from the house), I noticed Firefox frequently getting killed especially when working in Google Docs. Curiously, I didn’t notice the same on the Pi 400 (one of my two main workhorses that gets used a lot more than the laptop).

Then I recalled that we’d activated zswap (see earlier post) by default on jammy for the Pi desktop images. My PC laptop had the same RAM size (4GB), the same swap size (1GB), but while zswap was active on the Pi, it wasn’t on the laptop. That’s not to say the Pi was perfect: there were two occasions when Firefox got killed, but that was in stark contrast to the laptop on which it happened half a dozen times in a single day.

Obviously the first thing I did was activate zswap on my laptop and, after a reboot suddenly things were much better. I did still manage to get Firefox killed during a subsequent test run, but after then expanding the swap to 2GB on both machines, I managed a full day of work on the laptop without a single systemd-oomd activation.

So (unusually for this site), here’s a recommendation for any jammy desktop users on small PCs out there (Pi users, you can ignore this bit):

$ sudo -i
# sed -i \
> -e '/GRUB_CMDLINE_LINUX_DEFAULT/ s/splash/splash zswap.enabled=1 zswap.compressor=zstd zswap.zpool=z3fold/' \
> /etc/default/grub
# echo zstd >> /etc/initramfs-tools/modules
# echo z3fold >> /etc/initramfs-tools/modules
# update-initramfs -u
# update-grub

The above will activate zswap by adding zswap.enabled=1 zswap.compressor=zstd zswap.zpool=z3fold to the kernel’s command line in the GRUB configuration, and adding the necessary modules to the initramfs.

Next, I’d also recommend (to both small PC users and Pi users), to bump the default swap-file up to 2GB. After doing this, I haven’t suffered a single systemd-oomd kill on my Pi 400 (again, for my particular workload — bear in mind I spend most of my time in lightweight terminal things, so adjust accordingly). The following should be run on a freshly booted system as the first thing we’re going to do is disable the existing swap before expanding it:

$ sudo swapoff /swapfile
$ sudo fallocate -l 2G /swapfile
$ sudo mkswap /swapfile
$ sudo chmod 600 /swapfile
$ sudo swapon /swapfile

What have we actually accomplished with all this?

Let’s take the example of my 4GB laptop. The defaults for zswap’s configuration will reserve 20% of all RAM for compressed pages, which is roughly 800MB. Those compressed pages will (with the z3fold allocator) store up to 3 uncompressed pages each. So that 800MB of RAM can act as up to 2.4GB of (suspiciously fast) swap. And beyond that we also have 2GB of (typical, slow) disk-based swap.

Visually what we’ve done is this:

Three bars illustrating the original state of the memory layout (4GB RAM, 1GB swap), the new physical layout (3.2GB RAM, 800MB zswap, 2GB disk swap), and the new “effective” layout (3.2GB RAM, 2.4GB zswap, 2GB disk swap)

Don’t be fooled: we’ve reduced the amount of available RAM from 4GB to 3.2GB. This isn’t good, but it’s the trade-off you make with compressed RAM systems of any sort. We’ve also sacrificed another gig of disk space to swap. However, we’ve increased the available swap from 1GB (by default) to about 4.4GB, of which more than half is very fast compressed-memory swap and all for a (fairly) minimal memory cost.

Furthermore, we’ve also moved from a configuration where there’s 4GB of very fast pages, and 1GB of really slow pages to one where there’s 3.2GB of very fast pages, 2.4GB of fairly quick pages, and 2GB of really slow pages. In other words, we’ve now got a “smoother” progression of memory the kernel can stick pages into. This helps a system under load go a bit slower while remaining reasonably responsive, instead of suddenly hitting a brick wall of swap.

Soggy Cornflakes

One mea-culpa I should offer here: I managed to forget something rather important on the Ubuntu Pi desktop images for jammy.

While zswap is activated, and the swap-file is created (now on boot by the mkswap.service unit so we no longer ship a gigabyte of “nothing” in the image itself) … I managed to forget to add the “z3fold” and “zstd” modules to the initramfs modules list. As a result, zswap is still working on the Pi desktop images, but it’s currently falling back to the default “zbud” (2-page) allocator, and “lzo” compression.

LP: #1977764 is tracking this and I’ll get it fixed as soon as I reasonably can (unfortunately, while it sounds trivial it’s actually a messy one — read the bug if you want the full story).


That about covers things for the desktop side. What about the server image? Overall I’m actually pretty happy with the server side of things, but one thing has been irking me whenever I’ve played with the lovely little Pi Zero 2W. There’s precious little memory available at runtime, particularly on the arm64 images (which it seems the vast majority of our users prefer).

There’s been some effort recently to identify things installed by default which shouldn’t be, with the goal of reducing the “base” memory footprint of our images and I thought I’d have a look at what’s running by default on the Pi server images to see if there’s anything we could cull there.

First, let’s see how much free memory we have on a fresh boot:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           414Mi       152Mi        33Mi       3.0Mi       228Mi       242Mi
Swap:             0B          0B          0B

Hmmm, only 242MB available. Okay, what’s eating it?


A quick clarification about “free” vs “available”. Ignore “free”. This is the amount of RAM actually unused, but unused RAM is wasted RAM. The kernel has a duty to minimize free RAM by filling anything unused with disk (and other useful) caches which can be evicted trivially in the case that more RAM is required. The “available” metric indicates the amount of RAM that is available “on demand” should userland processes request it. It’s roughly (but not precisely) equal to “free” + “buff/cache”.

Or to put it another way: “available” tells you how much RAM is available for use; “free” is the metric which tells you how much RAM you’ve wasted your money on ;)

We’ll use the ps command exclude everything under PID 2 which is the kernel (--ppid 2 -p 2 -N). We’ll turn on the hierarchical view (-H) to get a clue of what belongs to what (this won’t show every inter-process dependency, but it can be a rough guide), and the “full” output (-F) so we can get the memory usage info (amongst other things; I’ll exclude some of the columns below for the sake of text wrapping):

$ ps --ppid 2 -p 2 -NHF
UID          PID    PPID     SZ   RSS PSR TTY   CMD
root           1       0  41786 11364   2 ?     /sbin/init fixrtc splash
root         374       1  12020 12572   0 ?       /lib/systemd/systemd-journald
root         416       1  72416 25672   3 ?       /sbin/multipathd -d -s
root         429       1   5998  6332   2 ?       /lib/systemd/systemd-udevd
root         576       1   4087  9732   3 ?       /sbin/wpa_supplicant -c /run/netpla
systemd+     588       1  22160  6292   3 ?       /lib/systemd/systemd-timesyncd
systemd+     634       1   4097  7704   1 ?       /lib/systemd/systemd-networkd
systemd+     636       1   6270 12224   1 ?       /lib/systemd/systemd-resolved
message+     668       1   2280  4220   3 ?       @dbus-daemon --system --address=sys
root         673       1  20492  3344   2 ?       /usr/sbin/irqbalance --foreground
root         677       1   8217 17916   1 ?       /usr/bin/python3 /usr/bin/networkd-
root         678       1  59035  8220   3 ?       /usr/libexec/polkitd --no-debug
syslog       680       1  55508  4392   3 ?       /usr/sbin/rsyslogd -n -iNONE
root         684       1  366205 27060  0 ?       /usr/lib/snapd/snapd
root         687       1   6002  7164   2 ?       /lib/systemd/systemd-logind
root         691       1  98379 11280   3 ?       /usr/libexec/udisks2/udisksd
root         695       1   3813  3440   0 ?       /sbin/wpa_supplicant -u -s -O /run/
root         712       1   1727  2456   0 ?       /usr/sbin/cron -f -P
root         726       1  61827 12368   1 ?       /usr/sbin/ModemManager
root         736       1   1409   808   1 ttyS0   /sbin/agetty -o -p -- \u --keep-bau
root         744       1  27480 19884   0 ?       /usr/bin/python3 /usr/share/unatten
root         755       1   3787  8428   1 ?       sshd: /usr/sbin/sshd -D [listener]
root        1660     755   4530  9716   0 ?         sshd: ubuntu [priv]
ubuntu      1804    1660   4661  7592   1 ?           sshd: ubuntu@pts/0
ubuntu      1805    1804   2152  4852   2 pts/0         -bash
ubuntu    355020    1805   2495  2844   3 pts/0           ps --ppid 2 -p 2 -NHF
root         881       1    560   168   2 ?       /usr/bin/hciattach /dev/serial1 bcm
root         899       1   2386  4468   3 ?       /usr/lib/bluetooth/bluetoothd
ubuntu      1004       1   4559  9924   2 ?       /lib/systemd/systemd --user
ubuntu      1005    1004  42795  5800   2 ?         (sd-pam)
root        1766       1   1398   812   2 tty6    /sbin/agetty -o -p -- \u --noclear
root        1777       1   2417  3976   1 tty1    /bin/login -p --
ubuntu    227377    1777   2174  4856   2 tty1      -bash

Okay, lots to go through here. RSS (Resident Set Size) is the most interesting column here (it’s not wholly accurate but good enough for our purposes). What’s the fattest thing in RAM? Ah … snapd (at ~26MB). On my fat 8GB Pi 4 I’m happy to leave that because I use the LXD snap (a lot!). However, this is a little Zero 2W and I only need debs on this, so out it goes:

$ sudo apt purge snapd
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
Use 'sudo apt autoremove' to remove it.
The following packages will be REMOVED:
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 84.1 MB disk space will be freed.
Do you want to continue? [Y/n]
... lots of complaints about read-only stuff ...
rm: cannot remove '/snap/core20/1408/var/log': Read-only file system
rm: cannot remove '/snap/core20/1408/var/mail': Read-only file system
rm: cannot remove '/snap/core20/1408/var/opt': Read-only file system
rm: cannot remove '/snap/core20/1408/var/snap': Read-only file system
rm: cannot remove '/snap/core20/1408/var/spool/mail': Read-only file system
rm: cannot remove '/snap/core20/1408/var/tmp': Read-only file system
rm: cannot remove '/snap/core20/1408/writable': Read-only file system
Cannot remove directory /snap
Removing extra snap-confine apparmor rules
Removing snapd cache
Removing snapd state
dpkg: warning: while removing snapd, directory '/snap' not empty so not removed
$ sudo apt autoremove --purge
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 408 kB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 100500 files and directories currently installed.)
Removing squashfs-tools (1:4.5-3build1) ...
Processing triggers for man-db (2.10.2-1) ...

At this point, given it couldn’t remove everything under /snap I’d just reboot to clear up any remaining mounts and sudo rm -fr /snap (I don’t recall having to do this before; I’ll try and remember to investigate this further at some point).

$ sudo reboot
... wait to get back to the login ...
$ sudo rm -fr /snap

Now, what’s next? “multipathd” (~25MB). This is a piece of the default server installation for dealing with multipath disk IO. However, unless you’re running a Compute Module with some fairly serious storage attached to it, this is likely pointless on your Pi. Unfortunately, removing this is probably not a good idea:

$ sudo apt purge multipath-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  kpartx libsgutils2-2 liburcu8 sg3-utils sg3-utils-udev
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  multipath-tools* ubuntu-server* ubuntu-server-raspi*
0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded.
After this operation, 1179 kB disk space will be freed.
Do you want to continue? [Y/n] n

Notice how it wanted to remove the “ubuntu-server” package? That’s usually not a good sign! However, perhaps we can disable it somehow:

$ systemctl cat multipathd.service
# /lib/systemd/system/multipathd.service
Description=Device-Mapper Multipath Device Controller
Before=iscsi.service iscsid.service lvm2-activation-early.service blk-availability.service
After=multipathd.socket systemd-remount-fs.service

... etc ...

... etc ...

Ah ha! Adding nompath or the slightly more obvious multipath=off to the kernel command line should disable this service. Well, that’s easily accomplished:

$ sudo sed -i -e 's/$/ multipath=off/' /boot/firmware/cmdline.txt
$ sudo reboot

I’m sorely tempted to add this by default to the Pi images, but I need to dig a bit further into whether multipath has any genuine use-cases on the Pi.

Next on the list? The Python-based “unattended-upgrades-shutdown” process (~19MB). It’s not a great idea to disable this (it ensures the computer doesn’t shut down in the middle of unattended-upgrades). However, it’s annoying that it hangs around the whole time eating RAM and only actually does anything at shutdown. Still, it’s the subject of an existing ticket, LP: #1955084 so we’ll just leave that one for now.

Next? Another Python process: the “networkd-dispatcher” daemon (~17MB). If you like your WiFi connection, leave this one alone! Next up, it’s “systemd-journald” (~12MB). That’s also doing useful work, so we’ll leave that one alone too.

Then … “ModemManager”. Hang on? Modem Manager? Am I back in the 90s?! Oh, it’s for GSM modems. Still, I’m not actually using a GSM modem on this Pi so why is that there? In fact, it’s the subject of another ticket, LP: #1981109 (you may note it’s another one filed by Steve, another colleague who has been ruthlessly hunting down the cruft of late). Anyway, given the details in that ticket this one can be safely excised (it’s not even a real dependency, just a “recommendation” of another package)!

$ sudo apt purge modemmanager --autoremove
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  libtcl8.6* modemmanager* tcl* tcl8.6* usb-modeswitch* usb-modeswitch-data*
0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.
After this operation, 8595 kB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 100491 files and directories currently installed.)
Removing usb-modeswitch (2.6.1-3ubuntu2) ...
Removing tcl (8.6.11+1build2) ...
Removing tcl8.6 (8.6.12+dfsg-1build1) ...
Removing libtcl8.6:arm64 (8.6.12+dfsg-1build1) ...
Removing modemmanager (1.18.6-1) ...
Unknown option: runtime
Removing usb-modeswitch-data (20191128-4) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for dbus (1.12.20-2ubuntu4) ...
Processing triggers for libc-bin (2.35-0ubuntu3) ...
(Reading database ... 100105 files and directories currently installed.)
Purging configuration files for tcl8.6 (8.6.12+dfsg-1build1) ...
Purging configuration files for usb-modeswitch (2.6.1-3ubuntu2) ...
Purging configuration files for modemmanager (1.18.6-1) ...
Processing triggers for dbus (1.12.20-2ubuntu4) ...

Next up, “systemd-resolved” (~12MB). We shouldn’t touch that one (DNS resolution and caching is generally useful).

Next? “udisks2” (~11MB). This is a useful service, handling things like auto-mounting USB storage devices that have been inserted. However, I’m not going to be doing that on this Pi Zero 2W so I don’t need it running all the time. I’m not going to uninstall it as it’s potentially useful and, even after disabling it (which will prevent it auto-starting), the udisksctl command can implicitly start it back up again, should I find I need it.

$ sudo systemctl stop udisks2
$ sudo systemctl disable udisks2
Removed /etc/systemd/system/

At this point we’re into diminishing returns so let’s reboot to a clean start and see what free says now.

$ sudo reboot
... wait to get back to the login ...
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           414Mi       103Mi       165Mi       2.0Mi       146Mi       299Mi
Swap:             0B          0B          0B

That’s looking better! Nearly 300MB free now. Finally, another tweak we can use to grab a bit more memory. This one is only useful if you have no intention of using GPU functionality (no camera, no video decoding / encoding acceleration, etc). However, I don’t need any graphics on this particular Pi (in fact there won’t be any display plugged into it) so let’s reduce the GPU memory split all the way down to the minimum (16MB).

$ sudo -i
$ echo "gpu_mem=16" >> /boot/firmware/config.txt
$ sudo reboot
... wait to get back to the login ...
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           462Mi       101Mi       214Mi       2.0Mi       146Mi       348Mi
Swap:             0B          0B          0B

And there we have it; over 100MB extra available memory from our starting point. Bear in mind we have disabled some functionality to achieve this, so these aren’t all changes that could (or should!) be made universally. However, from the tickets we’ve encountered along the way there’s definitely some improvements that can still be made in the base image.