It’s now several months after jammy’s release, and thus long past time since I should’ve posted something about it. I’ve been holding back on this because, despite being an LTS (in fact, the first Ubuntu LTS desktop for the Pi), I can’t say I’m happy with it.
Yet.
Still, that’s why we don’t enable upgrades for LTS users until the first point release (planned for August 4th at the time of writing). So let’s go through the painful bits and see what early adopters can rectify before I get my nose back to the grindstone and try and fix all the bits I’ve messed up!
In some ways, this may be considered a spiritual successor to my earlier post (and even earlier post) on the Pi desktops. Or to put it another way, consider this the current “Dave recommends you run your Ubuntu Pi this way” post (until I supersede it again)!
Note
I’m going to take the unusual step of stating quite explicitly that the following (like everything on this site) is my personal opinion. This is my recommendation for running your Ubuntu Pi, not anyone else’s (including Canonical).
So, with disclaimers thoroughly disclaimed, let us begin …
Snap!
It’s the serial complaint: the default browser (Firefox) is now a snap. I’m not going to get into the fiery debate over snaps here (more beer is required for that!). Nor am I going to address the start-up speed issues (they’ve been furiously hacked upon and thoroughly covered elsewhere; in fact I was a small cog in the team that did some of that work). Besides, I’m more concerned with the performance of the browser after it’s started (a browser isn’t something I fire up more than once a day), and in that regard I’ve no particular complaints about the performance of the snapped version over the deb version.
And yet, complaints I have …
There are currently issues with the Firefox snap which, personally, I consider deal-breakers. In particular LP: #1741074 “chrome-gnome-shell extension fails to detect native host connector” (aka the “native messaging issue”). The bug’s title may sound relatively innocuous but in practice this translates to “I can’t use KeepassXC” (my password database of choice) with my browser.
I would fallback to KeepassXC’s awesome auto-type feature, but that doesn’t work under Wayland (to be clear: this isn’t exactly a defect, more a design choice; the linked thread has more details). Hence, at least with the version of jammy as released, my only option to use my (now pretty extensive, and certainly indispensable) password database is to copy and paste usernames and passwords via the clipboard … which just about any process can view. Obviously, this is a fairly major step backwards in security and not one I’m willing to put up with.
I should stress this is another issue that is top of the list and being furiously attacked with many sharp objects. Once it’s fixed, I’ll have no specific qualms about returning to the snap (after all, Mozilla seem to prefer it as a means of distribution). But what to do in the meantime?
Switch to the flatpak? No good: the aforementioned bug applies to any confined (snap or otherwise) browser. Besides … the Firefox flatpak is only available for amd64!
Switch to Chromium? It’s snapped too. That said, I’m not sure I would even if I could: uBlock Origin (an extension I consider damned near mandatory for anyone wanting to browse the modern web on anything less than a beefy Ryzen with 32GB of RAM) is better in Firefox than in Chromium.
What about full-fat Chrome? Not an option; Google only distribute Chrome packages for amd64 too.
Note
Rant: frankly, this is one of my biggest misgivings about snap, flatpak, and all these distribution methods that purport to put “distribution in the hands of the developers”: they all suck at providing for anything other than PC architectures leaving stuff like the Pi (or RISC-V, or any new architecture) high and dry.
So, until this is fixed, let’s just use the deb:
$ sudo add-apt-repository ppa:mozillateam/ppa ... lots of output, press Enter ... $ cat << EOF | sudo tee /etc/apt/preferences.d/firefox Package: firefox* Pin: release=LP-PPA-mozillateam Pin-Priority: 501 Package: firefox* Pin: release o=Ubuntu Pin-Priority: -1 EOF $ sudo apt purge firefox $ sudo snap remove firefox $ sudo apt install firefox
Crackle!
Another cereal complaint (I make no excuses for the puns here): the introduction of systemd-oomd! In fact, I was rather glad of this one but I also think it’s less of an issue on the Pi than it is the PC, for reasons I’ll go into …
Note
This section is partly redundant because, while writing it, one of my colleagues (Nick, who despite being fresh-faced and new, jumped headlong into the fray) pushed a new configuration that substantially solves things. See LP: #1972159 for the full story.
However, there’s still some useful stuff in here for users on smaller memory systems (including non-Pis!) so I’ve left it in.
Firstly, a little context. For those not aware, systemd-oomd is a new facility for killing memory hogging processes (now activated by default on our desktop images). Previously, this was solely the domain of the dreaded OOM killer in the Linux kernel. While this certainly did the job of killing memory hogs, it was typically only capable of doing so after the system had descended into “swap hell”. In other words, only once something ridiculously fat had pushed everything else on the system to page itself out to disk, did the kernel OOM killer wake up and stalk the process table looking for potential victims.
An artist’s impression of “swap hell”
This typically meant that, while things did get remedied eventually, it took several minutes of the machine “thrashing” for that to happen. In the noughties, I might listen for the audible cue of an HDD thrashing its heads to have a clue this was going on, but this is now and SSDs (or SD cards) don’t provide quite the same feedback!
As a rough rule of thumb, people will assume something has crashed after roughly 5 seconds of the UI appearing to freeze. If that sounds short, consider how many seconds it would take before you assumed a crash if even your mouse pointer is frozen (or jumpy to the point that someone might reasonably assume something was “wrong”). In practice this typically meant that, long before the kernel OOM killer got involved, the user’s itchy power-button finger got involved first.
To be fair, systemd-oomd does fix several issues with the kernel’s OOM killer. The most notable being that a modern application may have many “processes” and (being process-oriented) the kernel’s OOM killer would only kill the heaviest individual process (regardless of whether it was a mere component in a larger system). That potentially left the rest of the application in a unknown / unstable state. By contrast, systemd-oomd will kill the whole cgroup consistently (cgroup in this context can be read as “application” … sort of).
It also acts before the system ever gets near “swap hell”, basing its actions on the system reaching some (fairly arbitrary) thresholds like 90% of both RAM and swap being utilized (to activate), and pressure stall information (to determine what to act upon).
Unfortunately there are some major issues:
- Most browsers (presumably in acknowledgement of how much fat is dripping off the silk of the web these days) already have mechanisms to “unload” their heavier processes (usually individual tabs) when they detect the system is “under load”. However, being fine-tuned to the individual application, these usually kick in later than systemd-oomd will (or to put it another way, by the time they would kick in, systemd-oomd has already killed the whole browser).
- systemd-oomd has no user interaction or even notification. One second you’re working in your browser and the next: blip. It’s gone. No warning, no chance to intervene, and not so much as an apology after the fact!
On the upside, it all happens very quickly: there’s no wading through the treacle of swap-thrashing. But it feels … arbitrary, unpredictable, and frankly “flaky”.
There is a simple solution: get more RAM! On my machines with 8GB (or more) of RAM, I simply haven’t encountered the issue. It’s only on machines with 4GB or less that this seems to be a serious issue (at least, that’s my personal experience — I’m sure others with different workloads may find differently). Still, that includes the Pi 400 I use as a desktop, and also my tiddly laptop (which has 4GB of RAM and a Celeron processor — I like my machines small and cheap!).
And this is where things (surprisingly!) turned out better on the Pi than on the PC:
On my little 4GB laptop (which doesn’t really get much use, only when I’m away from the house), I noticed Firefox frequently getting killed especially when working in Google Docs. Curiously, I didn’t notice the same on the Pi 400 (one of my two main workhorses that gets used a lot more than the laptop).
Then I recalled that we’d activated zswap (see earlier post) by default on jammy for the Pi desktop images. My PC laptop had the same RAM size (4GB), the same swap size (1GB), but while zswap was active on the Pi, it wasn’t on the laptop. That’s not to say the Pi was perfect: there were two occasions when Firefox got killed, but that was in stark contrast to the laptop on which it happened half a dozen times in a single day.
Obviously the first thing I did was activate zswap on my laptop and, after a reboot suddenly things were much better. I did still manage to get Firefox killed during a subsequent test run, but after then expanding the swap to 2GB on both machines, I managed a full day of work on the laptop without a single systemd-oomd activation.
So (unusually for this site), here’s a recommendation for any jammy desktop users on small PCs out there (Pi users, you can ignore this bit):
$ sudo -i # sed -i \ > -e '/GRUB_CMDLINE_LINUX_DEFAULT/ s/splash/splash zswap.enabled=1 zswap.compressor=zstd zswap.zpool=z3fold/' \ > /etc/default/grub # echo zstd >> /etc/initramfs-tools/modules # echo z3fold >> /etc/initramfs-tools/modules # update-initramfs -u # update-grub
The above will activate zswap by adding zswap.enabled=1 zswap.compressor=zstd zswap.zpool=z3fold to the kernel’s command line in the GRUB configuration, and adding the necessary modules to the initramfs.
Next, I’d also recommend (to both small PC users and Pi users), to bump the default swap-file up to 2GB. After doing this, I haven’t suffered a single systemd-oomd kill on my Pi 400 (again, for my particular workload — bear in mind I spend most of my time in lightweight terminal things, so adjust accordingly). The following should be run on a freshly booted system as the first thing we’re going to do is disable the existing swap before expanding it:
$ sudo swapoff /swapfile $ sudo fallocate -l 2G /swapfile $ sudo mkswap /swapfile $ sudo chmod 600 /swapfile $ sudo swapon /swapfile
What have we actually accomplished with all this?
Let’s take the example of my 4GB laptop. The defaults for zswap’s configuration will reserve 20% of all RAM for compressed pages, which is roughly 800MB. Those compressed pages will (with the z3fold allocator) store up to 3 uncompressed pages each. So that 800MB of RAM can act as up to 2.4GB of (suspiciously fast) swap. And beyond that we also have 2GB of (typical, slow) disk-based swap.
Visually what we’ve done is this:
Don’t be fooled: we’ve reduced the amount of available RAM from 4GB to 3.2GB. This isn’t good, but it’s the trade-off you make with compressed RAM systems of any sort. We’ve also sacrificed another gig of disk space to swap. However, we’ve increased the available swap from 1GB (by default) to about 4.4GB, of which more than half is very fast compressed-memory swap and all for a (fairly) minimal memory cost.
Furthermore, we’ve also moved from a configuration where there’s 4GB of very fast pages, and 1GB of really slow pages to one where there’s 3.2GB of very fast pages, 2.4GB of fairly quick pages, and 2GB of really slow pages. In other words, we’ve now got a “smoother” progression of memory the kernel can stick pages into. This helps a system under load go a bit slower while remaining reasonably responsive, instead of suddenly hitting a brick wall of swap.
Soggy Cornflakes
One mea-culpa I should offer here: I managed to forget something rather important on the Ubuntu Pi desktop images for jammy.
While zswap is activated, and the swap-file is created (now on boot by the mkswap.service unit so we no longer ship a gigabyte of “nothing” in the image itself) … I managed to forget to add the “z3fold” and “zstd” modules to the initramfs modules list. As a result, zswap is still working on the Pi desktop images, but it’s currently falling back to the default “zbud” (2-page) allocator, and “lzo” compression.
LP: #1977764 is tracking this and I’ll get it fixed as soon as I reasonably can (unfortunately, while it sounds trivial it’s actually a messy one — read the bug if you want the full story).
Pop!
That about covers things for the desktop side. What about the server image? Overall I’m actually pretty happy with the server side of things, but one thing has been irking me whenever I’ve played with the lovely little Pi Zero 2W. There’s precious little memory available at runtime, particularly on the arm64 images (which it seems the vast majority of our users prefer).
There’s been some effort recently to identify things installed by default which shouldn’t be, with the goal of reducing the “base” memory footprint of our images and I thought I’d have a look at what’s running by default on the Pi server images to see if there’s anything we could cull there.
First, let’s see how much free memory we have on a fresh boot:
$ free -h total used free shared buff/cache available Mem: 414Mi 152Mi 33Mi 3.0Mi 228Mi 242Mi Swap: 0B 0B 0B
Hmmm, only 242MB available. Okay, what’s eating it?
Note
A quick clarification about “free” vs “available”. Ignore “free”. This is the amount of RAM actually unused, but unused RAM is wasted RAM. The kernel has a duty to minimize free RAM by filling anything unused with disk (and other useful) caches which can be evicted trivially in the case that more RAM is required. The “available” metric indicates the amount of RAM that is available “on demand” should userland processes request it. It’s roughly (but not precisely) equal to “free” + “buff/cache”.
Or to put it another way: “available” tells you how much RAM is available for use; “free” is the metric which tells you how much RAM you’ve wasted your money on ;)
We’ll use the ps command exclude everything under PID 2 which is the kernel (--ppid 2 -p 2 -N). We’ll turn on the hierarchical view (-H) to get a clue of what belongs to what (this won’t show every inter-process dependency, but it can be a rough guide), and the “full” output (-F) so we can get the memory usage info (amongst other things; I’ll exclude some of the columns below for the sake of text wrapping):
$ ps --ppid 2 -p 2 -NHF UID PID PPID SZ RSS PSR TTY CMD root 1 0 41786 11364 2 ? /sbin/init fixrtc splash root 374 1 12020 12572 0 ? /lib/systemd/systemd-journald root 416 1 72416 25672 3 ? /sbin/multipathd -d -s root 429 1 5998 6332 2 ? /lib/systemd/systemd-udevd root 576 1 4087 9732 3 ? /sbin/wpa_supplicant -c /run/netpla systemd+ 588 1 22160 6292 3 ? /lib/systemd/systemd-timesyncd systemd+ 634 1 4097 7704 1 ? /lib/systemd/systemd-networkd systemd+ 636 1 6270 12224 1 ? /lib/systemd/systemd-resolved message+ 668 1 2280 4220 3 ? @dbus-daemon --system --address=sys root 673 1 20492 3344 2 ? /usr/sbin/irqbalance --foreground root 677 1 8217 17916 1 ? /usr/bin/python3 /usr/bin/networkd- root 678 1 59035 8220 3 ? /usr/libexec/polkitd --no-debug syslog 680 1 55508 4392 3 ? /usr/sbin/rsyslogd -n -iNONE root 684 1 366205 27060 0 ? /usr/lib/snapd/snapd root 687 1 6002 7164 2 ? /lib/systemd/systemd-logind root 691 1 98379 11280 3 ? /usr/libexec/udisks2/udisksd root 695 1 3813 3440 0 ? /sbin/wpa_supplicant -u -s -O /run/ root 712 1 1727 2456 0 ? /usr/sbin/cron -f -P root 726 1 61827 12368 1 ? /usr/sbin/ModemManager root 736 1 1409 808 1 ttyS0 /sbin/agetty -o -p -- \u --keep-bau root 744 1 27480 19884 0 ? /usr/bin/python3 /usr/share/unatten root 755 1 3787 8428 1 ? sshd: /usr/sbin/sshd -D [listener] root 1660 755 4530 9716 0 ? sshd: ubuntu [priv] ubuntu 1804 1660 4661 7592 1 ? sshd: ubuntu@pts/0 ubuntu 1805 1804 2152 4852 2 pts/0 -bash ubuntu 355020 1805 2495 2844 3 pts/0 ps --ppid 2 -p 2 -NHF root 881 1 560 168 2 ? /usr/bin/hciattach /dev/serial1 bcm root 899 1 2386 4468 3 ? /usr/lib/bluetooth/bluetoothd ubuntu 1004 1 4559 9924 2 ? /lib/systemd/systemd --user ubuntu 1005 1004 42795 5800 2 ? (sd-pam) root 1766 1 1398 812 2 tty6 /sbin/agetty -o -p -- \u --noclear root 1777 1 2417 3976 1 tty1 /bin/login -p -- ubuntu 227377 1777 2174 4856 2 tty1 -bash
Okay, lots to go through here. RSS (Resident Set Size) is the most interesting column here (it’s not wholly accurate but good enough for our purposes). What’s the fattest thing in RAM? Ah … snapd (at ~26MB). On my fat 8GB Pi 4 I’m happy to leave that because I use the LXD snap (a lot!). However, this is a little Zero 2W and I only need debs on this, so out it goes:
$ sudo apt purge snapd Reading package lists... Done Building dependency tree... Done Reading state information... Done The following package was automatically installed and is no longer required: squashfs-tools Use 'sudo apt autoremove' to remove it. The following packages will be REMOVED: snapd* 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. After this operation, 84.1 MB disk space will be freed. Do you want to continue? [Y/n] ... lots of complaints about read-only stuff ... rm: cannot remove '/snap/core20/1408/var/log': Read-only file system rm: cannot remove '/snap/core20/1408/var/mail': Read-only file system rm: cannot remove '/snap/core20/1408/var/opt': Read-only file system rm: cannot remove '/snap/core20/1408/var/snap': Read-only file system rm: cannot remove '/snap/core20/1408/var/spool/mail': Read-only file system rm: cannot remove '/snap/core20/1408/var/tmp': Read-only file system rm: cannot remove '/snap/core20/1408/writable': Read-only file system Cannot remove directory /snap Removing extra snap-confine apparmor rules Removing snapd cache Removing snapd state dpkg: warning: while removing snapd, directory '/snap' not empty so not removed $ sudo apt autoremove --purge Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be REMOVED: squashfs-tools* 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. After this operation, 408 kB disk space will be freed. Do you want to continue? [Y/n] (Reading database ... 100500 files and directories currently installed.) Removing squashfs-tools (1:4.5-3build1) ... Processing triggers for man-db (2.10.2-1) ...
At this point, given it couldn’t remove everything under /snap I’d just reboot to clear up any remaining mounts and sudo rm -fr /snap (I don’t recall having to do this before; I’ll try and remember to investigate this further at some point).
$ sudo reboot ... wait to get back to the login ... $ sudo rm -fr /snap
Now, what’s next? “multipathd” (~25MB). This is a piece of the default server installation for dealing with multipath disk IO. However, unless you’re running a Compute Module with some fairly serious storage attached to it, this is likely pointless on your Pi. Unfortunately, removing this is probably not a good idea:
$ sudo apt purge multipath-tools Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages were automatically installed and are no longer required: kpartx libsgutils2-2 liburcu8 sg3-utils sg3-utils-udev Use 'sudo apt autoremove' to remove them. The following packages will be REMOVED: multipath-tools* ubuntu-server* ubuntu-server-raspi* 0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded. After this operation, 1179 kB disk space will be freed. Do you want to continue? [Y/n] n Abort.
Notice how it wanted to remove the “ubuntu-server” package? That’s usually not a good sign! However, perhaps we can disable it somehow:
$ systemctl cat multipathd.service # /lib/systemd/system/multipathd.service [Unit] Description=Device-Mapper Multipath Device Controller Before=iscsi.service iscsid.service lvm2-activation-early.service Before=local-fs-pre.target blk-availability.service shutdown.target Wants=systemd-udevd-kernel.socket After=systemd-udevd-kernel.socket After=multipathd.socket systemd-remount-fs.service DefaultDependencies=no Conflicts=shutdown.target ConditionKernelCommandLine=!nompath ConditionKernelCommandLine=!multipath=off ConditionVirtualization=!container [Service] ... etc ... [Install] ... etc ...
Ah ha! Adding nompath or the slightly more obvious multipath=off to the kernel command line should disable this service. Well, that’s easily accomplished:
$ sudo sed -i -e 's/$/ multipath=off/' /boot/firmware/cmdline.txt $ sudo reboot
I’m sorely tempted to add this by default to the Pi images, but I need to dig a bit further into whether multipath has any genuine use-cases on the Pi.
Next on the list? The Python-based “unattended-upgrades-shutdown” process (~19MB). It’s not a great idea to disable this (it ensures the computer doesn’t shut down in the middle of unattended-upgrades). However, it’s annoying that it hangs around the whole time eating RAM and only actually does anything at shutdown. Still, it’s the subject of an existing ticket, LP: #1955084 so we’ll just leave that one for now.
Next? Another Python process: the “networkd-dispatcher” daemon (~17MB). If you like your WiFi connection, leave this one alone! Next up, it’s “systemd-journald” (~12MB). That’s also doing useful work, so we’ll leave that one alone too.
Then … “ModemManager”. Hang on? Modem Manager? Am I back in the 90s?! Oh, it’s for GSM modems. Still, I’m not actually using a GSM modem on this Pi so why is that there? In fact, it’s the subject of another ticket, LP: #1981109 (you may note it’s another one filed by Steve, another colleague who has been ruthlessly hunting down the cruft of late). Anyway, given the details in that ticket this one can be safely excised (it’s not even a real dependency, just a “recommendation” of another package)!
$ sudo apt purge modemmanager --autoremove Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be REMOVED: libtcl8.6* modemmanager* tcl* tcl8.6* usb-modeswitch* usb-modeswitch-data* 0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded. After this operation, 8595 kB disk space will be freed. Do you want to continue? [Y/n] (Reading database ... 100491 files and directories currently installed.) Removing usb-modeswitch (2.6.1-3ubuntu2) ... Removing tcl (8.6.11+1build2) ... Removing tcl8.6 (8.6.12+dfsg-1build1) ... Removing libtcl8.6:arm64 (8.6.12+dfsg-1build1) ... Removing modemmanager (1.18.6-1) ... Unknown option: runtime Removing usb-modeswitch-data (20191128-4) ... Processing triggers for man-db (2.10.2-1) ... Processing triggers for dbus (1.12.20-2ubuntu4) ... Processing triggers for libc-bin (2.35-0ubuntu3) ... (Reading database ... 100105 files and directories currently installed.) Purging configuration files for tcl8.6 (8.6.12+dfsg-1build1) ... Purging configuration files for usb-modeswitch (2.6.1-3ubuntu2) ... Purging configuration files for modemmanager (1.18.6-1) ... Processing triggers for dbus (1.12.20-2ubuntu4) ...
Next up, “systemd-resolved” (~12MB). We shouldn’t touch that one (DNS resolution and caching is generally useful).
Next? “udisks2” (~11MB). This is a useful service, handling things like auto-mounting USB storage devices that have been inserted. However, I’m not going to be doing that on this Pi Zero 2W so I don’t need it running all the time. I’m not going to uninstall it as it’s potentially useful and, even after disabling it (which will prevent it auto-starting), the udisksctl command can implicitly start it back up again, should I find I need it.
$ sudo systemctl stop udisks2 $ sudo systemctl disable udisks2 Removed /etc/systemd/system/graphical.target.wants/udisks2.service
At this point we’re into diminishing returns so let’s reboot to a clean start and see what free says now.
$ sudo reboot ... wait to get back to the login ... $ free -h total used free shared buff/cache available Mem: 414Mi 103Mi 165Mi 2.0Mi 146Mi 299Mi Swap: 0B 0B 0B
That’s looking better! Nearly 300MB free now. Finally, another tweak we can use to grab a bit more memory. This one is only useful if you have no intention of using GPU functionality (no camera, no video decoding / encoding acceleration, etc). However, I don’t need any graphics on this particular Pi (in fact there won’t be any display plugged into it) so let’s reduce the GPU memory split all the way down to the minimum (16MB).
$ sudo -i $ echo "gpu_mem=16" >> /boot/firmware/config.txt $ sudo reboot ... wait to get back to the login ... $ free -h total used free shared buff/cache available Mem: 462Mi 101Mi 214Mi 2.0Mi 146Mi 348Mi Swap: 0B 0B 0B
And there we have it; over 100MB extra available memory from our starting point. Bear in mind we have disabled some functionality to achieve this, so these aren’t all changes that could (or should!) be made universally. However, from the tickets we’ve encountered along the way there’s definitely some improvements that can still be made in the base image.