It's Pi all the way down... - Dave Jones

Pull yourself up by your bootstraps

2025-07-09T00:00:00+01:00

Updated: Minor punctuation changes, and a small clarification of why the current fallback probably isn’t a fallback at all

The way Ubuntu boots on the Raspberry is changing in questing. Here’s the story behind the changes (along with my usual copious tangents, and finishing up with some details on how to avoid these changes if you really, really need to).

Boots an’ all

Our current boot setup is… far from optimal. To understand it we should first have a look at how the Pi boots, and how Ubuntu uses this (skip this section if you’re familiar with the Pi’s bootloader!).

The Pi’s native bootloader is split into three parts which we’ll simply call Stages 1, 2, and 3.

Stage 1 is always on the SoC (on all models). It’s main job (as far as we’re concerned) as to find and load stage 2. Its configuration is the autoboot.txt file. If this is present, it must be on the first partition of the boot media. Due to stage 1’s very limited resources (there’s no RAM available at this stage), this file is limited to 512 bytes, and only understands a few directives. The most important is boot_partition which tells it which partition contains the rest of the boot media.

Prior to the Pi 4, stage 2 was the bootcode.bin executable. From the Pi 4 onwards, this is part of the boot EEPROM. I’m not fully clear on all the things stage 2 does, but I do know this stage handles bringing up some more bits of hardware to load and execute the (much bigger) third stage. For instance, when network booting, it’s this stage that brings the ethernet port up, performs DHCP, and starts the TFTP process for the other boot assets.

The second stage also reads at least some of the config.txt configuration file, as this can be used to customize which binary is loaded for the third stage.

Stage 3 is the real “meat” of the bootloader. On the Pi 3 and earlier it was named start.elf; on the Pi 4 this became start4.elf file; on the Pi 5 this is another part of the boot EEPROM [1]. There were 4 historical variants of this stage:

start.elf / start4.elf: The default third stage
start_cd.elf / start4cd.elf: A “cut down” version of the bootloader with minimal facilities [2]
start_db.elf / start4db.elf: The “debug” build of the bootloader
start_x.elf / start4x.elf: The bootloader incorporating the legacy camera firmware [3]

Each of these binaries has a corresponding fixup*.dat (fixup.dat, fixup4.dat, etc.) file which contains the relocation information [5] for each binary.

This stage is responsible for loading [4] the device-tree and customizing it for the hardware detected, loading the binaries containing the Linux kernel, and (optionally) the initramfs. It parses all of config.txt which has many options that can be used to customize the booted state. It also reads cmdline.txt which contains the Linux kernel’s command line. Importantly, this also defines the eventual rootfs location.

Finally, this stage is responsible for bringing the ARM cores online and starting the kernel (passing it the addresses of the fixed up device-tree, the optional initramfs, and the kernel command line).

Summarizing, the following table shows the location of the various stages on the various generations of Pi:

Generation	SoC	EEPROM	SD / USB / NVMe
Pi 1, 2, 3, and Zero	1		2, 3
Pi 4, 400	1	2	3
Pi 5, 500	1	2, 3

One other important thing to note is that the various stages can only read the FAT file-system [6], so all the following assets must be placed on FAT partition(s) somewhere:

Any bootloader assets that don’t live in EEPROM for the generation of Pis you need to support
The base device-tree(s) of all models you need to support
All the device-tree overlays wanted (optional)
The Linux kernel
The initramfs (optional)

Die with your boots on

What does Ubuntu’s current boot set up look like, particularly with regard to safety? Erm…

On the plus side we always keep two sets of boot assets around on the boot partition. But I’m afraid that’s it for the good news:

There is no fallback facility. If your boot configuration is corrupted, if we release a bad kernel update, etc. your boot will fail and it’s up to you to pick the pieces
The boot assets are spread across many files (the aforementioned bootloader files, the kernel, the initramfs, the base device trees, and a few hundred overlays). But the backup files aren’t in easy-to-swap directories: they’re simply next to their original files with a “.bak” suffix.
If you know enough shell scripting you might figure out that for f in $(find /boot/firmware -type f -name "*.bak"); do cp "$f" "${f%.bak}"; done is what you want [7].
Even if you figure this out, the old boot files probably aren’t there. If flash-kernel has run more than once since your last boot (this is extremely likely given it runs in response to initramfs rebuilds, kernel updates, flash-kernel upgrades, and so on) the “.bak” files will simply be copies of the new boot assets!

This is Bad with a capital B. No automatic fallback, the manual fallback is extremely painful, and is not even likely to work.

Puttin’ the boot in

A little-known [8] facility of the Pi’s bootloader is the “tryboot” facility. This is a rather neat system designed to provide a robust means to implement A/B booting. Classically, A/B booting involves having two separate copies of all your critical boot assets, and having some mechanism to switch between them.

Let’s say you booted from set “A”. A new kernel is released, and gets installed into “B”, and the bootloader is set to boot from “B” instead. The bootloader then needs some means of noticing if the “B” boot fails, falling back to “A”. In some cases this is done by having the bootloader record state itself in some persistent location, but the Pi’s mechanism for this is quite ingenious: it uses ephemeral state [9] to track that it should be trying “B” so that any failure results in it falling back to “A”.

How does this work in practice? A typical boot-flow reads the following files:

Stage 1 reads boot_partition from autoboot.txt (if it exists)
Reads stage 2 (bootcode.bin) from the aforementioned boot_partition (the first by default, but from EEPROM on the Pi 4 and 5)
Reads stage 3 (start*.elf) from the aforementioned boot_partition (from EEPROM on the Pi 5)
Reads config.txt to determine the rest of the configuration, which specifies the locations of all remaining assets
Reads the kernel, initramfs, device-trees, overlays, etc. from the boot_partition

However, if you reboot the Pi in “tryboot” mode this changes to:

Stage 1 reads boot_partition from autoboot.txt (if it exists), but optionally reads a [tryboot] section that may override boot_partition (potentially directing it to a different partition)
Reads stage 2 (bootcode.bin) from the (potentially overridden) boot_partition (from EEPROM on the Pi 4 and 5)
Reads stage 3 (start*.elf) from the (potentially overridden) boot_partition (from EEPROM on the Pi 5)
Reads tryboot.txt (instead of config.txt) which specifies the locations of all remaining assets
Reads the kernel, initramfs, device-trees, overlays, etc. from the (potentially overridden) boot_partition

The “tryboot” mode is initiated using the following command to reboot the Pi:

$ sudo reboot '0 tryboot'

This leads to a couple of typical designs for a tryboot-enabled boot implementation on the Pi…

Full ABs

A “full” A/B boot implementation consists of the following partition layout:

Partition 1 (FAT)

Just contains autoboot.txt with the following content [10]:

[all]
boot_partition=2

[tryboot]
tryboot_a_b=1
boot_partition=3

Partition 2 (FAT)

Contains the “A” set of boot assets. This includes the stage 2 and 3 binaries (if support of older Pi models is required), the kernel, initramfs, device-trees, overlays, etc. No directories required; everything’s in the root.

Boot configuration is stored in config.txt. The kernel command line in cmdline.txt points to partition 4 as the rootfs.

Partition 3 (FAT)

Contains the “B” set of boot assets. Identical layout to partition 2, but cmdline.txt points to partition 5 as the rootfs.

Partition 4

The “A” rootfs

Partition 5

The “B” rootfs

Note

Note the (highlighted) tryboot_a_b=1 line in autoboot.txt. This causes the third stage to read config.txt instead of tryboot.txt because the assumption is that, in this layout, the boot configuration will be precisely duplicated (both boot partitions have config.txt).

To switch between the A and B sets, autoboot.txt on the first partition is updated (preferably atomically [11]) with swapped values of the boot_partition= lines.

Typically the first partition is some smaller variant of FAT (FAT-12/16) while partitions two and three are FAT-32. Partitions 4 and 5 are whatever you use as a rootfs (commonly ext4, but can be anything your kernel and initramfs combination can support).

The benefits of this layout are pretty obvious:

It switches everything including all the bootloader assets (at least on the Pi models without any EEPROM).
It’s also capable of switching the entire rootfs, although this is optional (the “B” cmdline.txt could equally point at partition 4 to have a single rootfs).

The drawbacks are:

You pretty much have to design this “up front” into your image; it’s great for fresh installations, but almost impossible to safely migrate existing installations with a more basic layout to it.
This design is also more complex for users who wish to edit the boot configuration as they now need to figure out which partition is mounted and edit the boot files on the other partition [12]. That said, I suspect most uses of this layout try and ensure that users have no ability to mess with the boot configuration at all.

The beer belly version

A much simpler configuration uses the traditional partition layout but with some minor tweaks to the files on the boot partition:

Partition 1 (FAT-32)

No autoboot.txt (as per usual)
Bootloader assets (bootcode.bin, start*.elf) are placed in the root of this partition.
The a/ directory contains the remainder of the boot assets: device-trees, Linux kernel, initramfs, overlays, cmdline.txt (which points at partition 2 as the rootfs).
Another directory, b/ contains a second set of these boot assets.

config.txt contains something like the following:

[all]
os_prefix=a/
kernel=vmlinuz
initramfs initrd.img followkernel
...

tryboot.txt contains something like:

[all]
os_prefix=b/
kernel=vmlinuz
initramfs initrd.img followkernel
...

Partition 2

The rootfs

To switch between A and B, either config.txt and tryboot.txt are exchanged, or the a/ and b/ directories are switched (preferably atomically [13]).

The advantages to this layout are primarily simplicity.

It’s fairly easy to migrate an existing classical layout to this, and do so in a “safe” manner that guarantees a “known good” boot configuration is present at every step.
It’s also easier for users who know all their boot configuration will be present on a single partition (they don’t need to go potentially mounting things).

The disadvantages are:

The bootloader assets (bootcode.bin, start*.elf) cannot be switched by this method as they cannot be read from sub-directories. However, bear in mind that these are only used by older models of Pi and thus are unlikely to change much in the coming years anyway.
Changing the boot configuration becomes a little more complex. Let’s say you want to try a new overlay. You add the dtoverlay= line to tryboot.txt and initiate “tryboot”. Your OS boots successfully, and some service switches tryboot.txt and config.txt. However, now you need to remember to make the same edit to the new tryboot.txt (your former config.txt) or the next switch will lose the change.
If you want to edit the kernel command line, you still have to query config.txt to figure out which cmdline.txt to fiddle with (a/cmdline.txt or b/cmdline.txt).

Boot camp

As you can probably tell from the above, I don’t think either of the two designs presented is quite what we want for questing.

The “full” A/B design is far too complex to migrate to, and I don’t think there’s any way of doing it safely. This is especially the case given that we may have upgraders who have inherited a minimal 256MB boot partition from earlier releases of Ubuntu.

The “lite” version is a good basis, but the whole “a” and “b” thing strikes me as a little obscure. I also don’t particularly like the duplication of config.txt into tryboot.txt. There are an absolute ton of scripts out there that assume they can blindly append configuration to config.txt (which is why it’s usually important to make sure your configuration ends with an [all] section). While this isn’t “good practice”, the swapping of tryboot.txt and config.txt would break people who rely on these scripts.

After experimenting with a few designs over the last few weeks, here’s the design I’ve come up with, which should be landing in questing as I post this:

As in the “lite” A/B boot design, we still have our regular old FAT boot partition, and one rootfs partition. The bootloader assets (bootcode.bin, start*.elf, and so on) still live in the root of the boot partition (because they have to). All other assets move into three directories:

current: This directory always exists, and always contains the current boot assets of the booted system [14]. Thus, by definition, it will always contain “known good” boot assets.
old: If this directory exists, it contains the formerly “known good” boot assets. We keep this around when possible just in case users need it; consider a boot configuration change that isn’t fatal (the boot still succeeds), but which results in some undesirable side effect later at runtime. The new service should make it trivial to switch “old” to “current” when desired.
new: If this directory exists, it either contains new untested boot assets, or it contains boot assets that were tested, but failed.

Why keep failed boot assets around? It may be useful to see the configuration that failed for debugging purposes, but more importantly we can only assume that we have space for two sets of boot assets on the boot partition. Therefore, before “new” is created, “old” is always deleted first. This is safe because “current” is always “known good”.

So, we have the following “states” in our new boot layout:

stable: “current” exists; “old” may exist (if it does, contains “known good” assets); “new” does not exist
untested: “current” exists; “old” does not exist; “new” exists and contains untested boot assets
trying: “current” exists; “old” does not exist; “new” exists and contains boot assets we’re about to try (this state is entered immediately before rebooting into the “tryboot” mode)
failed: “current” exists; “old” does not exist; “new” exists and contains boot assets marked as having failed

The state transition diagram is as follows:

A simple static transition diagram showing “stable” goes to “untested”; “untested” can go to “untested” or “trying”; “trying” goes back to “stable” or to “bad”; and “bad” can only go to “untested”.

The “loop” on “untested” exists because (as mentioned earlier) it’s perfectly valid to have a long running system where flash-kernel has been called multiple times (a kernel update, an initramfs rebuild, etc.), overwriting the new boot assets each time. However, in this scenario the current boot assets are never touched by this.

The boot configuration is stored in config.txt and looks something like this:

[all]
os_prefix=current/

[tryboot]
os_prefix=new/

[all]
kernel=vmlinuz
initramfs initrd.img followkernel

# The rest of config.txt
# ...

Finally, autoboot.txt also exists, and just contains:

[all]
tryboot_a_b=1

This ensures the bootloader always reads config.txt, and we simply use a [tryboot] filter within that file to redirect our boot to the “new” directory. The advantages of this layout are:

We get all the advantages of the “lite” A/B boot setup: a reliable fallback, and a configuration that we can migrate to safely.
The boot configuration lives in a single file (config.txt) that doesn’t get arbitrarily over-written; existing scripts that append configuration to config.txt continue to work without configuration lines silently disappearing.
There’s no confusion over which boot assets are current, which are old, and which are new: the directory names tell you everything.

Booty! Yarrr!

This all sounds a bit too good to be true! What are the drawbacks?

The most obvious one is the fact that the “tryboot” mode can never be entered from a cold boot. It requires a reboot. This means some service needs to notice (during a normal boot) that new untested boot assets are present, interrupt the boot and restart in “tryboot” mode. This will mean that, each time flash-kernel is run for whatever reason (new kernel, initramfs rebuild, etc.) the next boot will be a double boot. This will probably be a bit jarring to people at first (“why did I see the rainbow screen twice?!”), and also means that boot will take roughly twice as long (obviously).

However, I don’t see a way around this and, to be frank, it’s a small price to pay for the reliability that this mechanism should bring.

Tough as old boots

I don’t mean all this to sound like a fait accompli [15]. Some people do weird things with their Pi. Some people may be relying on all their boot assets being in the root of their boot partition. Some may have weird hardware that breaks horribly if reboots occur too close together (or which doesn’t reset quickly enough for the second boot).

In short, there must be a fallback mechanism. To that end, I’m introducing these changes as a new “method” in flash-kernel, but keeping the old one in place for those that really need it.

All boards that use flash-kernel to write their boot assets (including the Raspberry Pi under Ubuntu) have entries in the flash-kernel database (/usr/share/flash-kernel/db/all.db) which look something like this:

Machine: Raspberry Pi 3 Model B+
Machine: Raspberry Pi 3 Model B Plus
Machine: Raspberry Pi 3 Model B Plus Rev 1.3
Machine: Raspberry Pi 3 Model B Plus Rev *
Kernel-Flavors: raspi
Method: pi-try
DTB-Id: bcm2710-rpi-3-b-plus.dtb

The important setting for our purposes is the Method: line (highlighted). Prior to questing this reads Method: pi on all the (relevant) Raspberry Pi entries. From questing onwards this will read Method: pi-try. When flash-kernel encounters the pi-try method, and finds the old boot configuration on the boot partition, it will attempt to migrate it to the new pi-try layout, and re-write the config.txt configuration accordingly.

However, the older pi method (no A/B booting, write everything to the root of the boot partition) is still present in flash-kernel, and if you absolutely need to, you can switch back to it.

Warning

Using the old “pi” mechanism leaves you in the situation described earlier: your fallback “.bak” files may not really be a fallback at all, and even if they are, it’s a major pain to revert to them. Still, if you’re absolutely determined…

Firstly, override the flash-kernel database entry by copying the relevant entry to /etc/flash-kernel/db and adjusting the Method: line.
Then, remove the old/ or new/ folders from your boot partition.
Run flash-kernel to re-copy the boot assets to the root of your boot partition.
Remove the os_prefix lines from your boot configuration.
Finally remove the current/ directory and autoboot.txt.

Done in this specific order, this procedure should be reasonably safe and leave you with a bootable system at all steps.

By way of an example, in the case of the Pi 3 entry above, the following should be sufficient to revert to the old boot method (pay close attention to the highlighted lines):

$ sudo -i
Password:
# cat << EOF >> /etc/flash-kernel/db
Machine: Raspberry Pi 3 Model B+
Machine: Raspberry Pi 3 Model B Plus
Machine: Raspberry Pi 3 Model B Plus Rev 1.3
Machine: Raspberry Pi 3 Model B Plus Rev *
Kernel-Flavors: raspi
Method: pi
DTB-Id: bcm2710-rpi-3-b-plus.dtb
EOF
# rm /boot/firmware/old
# rm /boot/firmware/new
# flash-kernel
# sync
# sed -i -e '/^os_prefix/d' /boot/firmware/config.txt
# rm -rf /boot/firmware/current
# rm /boot/firmware/autoboot.txt

Anyway, that’s enough “boot” puns from me, for now. If you do anything even slightly weird with your boot configuration, I would strongly encourage you to try out the questing dailies on a spare SD card:

To test the new implementation from a questing daily, add ppa:waveform/flash-kernel and install the flash-kernel-piboot package:

$ sudo add-apt-repository ppa:waveform/flash-kernel
$ sudo apt install flash-kernel-piboot

This will also upgrade flash-kernel and migrate your boot partition to the new layout. In theory the migration is entirely safe, i.e. it can fail at any point and you should be left with a bootable system (I’ve tested this a couple of times by yanking the power-cord in the middle of it!). However, if you can find a way to break it (from a configuration we’d actually support), then please let me know!

If you find any issues, please file a bug against the flash-kernel package in Launchpad, and tag it raspi-image to bring it to my attention.

If you do find a genuine need to fall back to the old boot configuration, again please let me know! I’d be really curious to find out what these circumstances are; I’m sure they exist, but they’ll be things I haven’t thought of (yet), and if there’s a way I can tweak the new design to accommodate them, I’d prefer to do that rather than have people forcing themselves back into the old (fundamentally unsafe) configuration.

Boots of Spanish Leather

The alternative to falling back to the old mechanism is to keep the A/B facility, but manage the tryboot mode somewhat manually…

The piboot-try command is the new script that manages all the stuff mentioned above (which will be provided by the new flash-kernel-piboot package, hopefully in Ubuntu questing by next week. When called with the --test flag this will exit with status code 0 (“true” in shell parlance) in the event there are new, untested boot assets. Otherwise, it will exit with status code 1 (“false” or “error” in shells).

When called with --reboot, if new untested boot assets are present, it will immediately re-write their status to “trying” and reboot into the “tryboot” mode.

Hence, if you know flash-kernel has run, and new boot assets are present, and you’re happy to reboot immediately you can simply run sudo piboot-try --reboot to reboot and immediately try the new boot assets (no double boot necessary). If you’re thinking of scripting this and want to query the state of the boot assets, piboot-try --test (no sudo necessary) will tell you whether sudo piboot-try --reboot would reboot (if the exit code is 0).

If you want to learn more about the command, I’ll be including a full man-page for it, man piboot-try.

[1]	On the Pi 5, none of the bootloader resides on the FAT partition anymore. In certain configurations, you can even get away with no configuration file, so just shoving a Linux kernel on the FAT partition is enough!

[2]	Intended to maximize the RAM available at runtime (the default bootloader typically reserved 64MB of RAM for facilities it provided at runtime; the cut down variant only needed 16MB).

[3]	One particularly odd aspect of the Pi’s boot process is that it runs entirely on the GPU (well … VPU?), rather than the ARM CPU. The legacy camera firmware basically ran its own RTOS over on the GPU, hence incorporating it into the bootloader made perfect sense.

[4]	Actually, I’m not entirely clear if stage 3 loads the base device tree. That might be stage 2, but stage 3 handles some of the customization of the device-tree, loading overlays, and so forth.

[5] I’m not entirely clear on what fixup.dat really does either. I know each fixup file corresponds to each start.elf file and that it has something to do with the CPU/GPU memory split. The start.elf binary will operate without fixup.dat being present, but the wrong amount of RAM is reported by the mailbox interface in this case. I’ve also heard fixup has to do with “relocation”, and I do know that the CPU/GPU memory split has the CPU (ARM portion) at the “low” end and the GPU firmware at the “high” end so my vaguely educated guess is that start.elf gets loaded at a too-low, but definitely safe location, then fixup.dat is used to relocate it to the highest possible point it can safely sit in RAM to maximize the amount available to the ARM cores. But that’s just my guess.

[6]	The stages all understand FAT-12, FAT-16, FAT-32, and the VFAT long filename extensions.

[7]	Maybe… This is just off the top of my head; I haven’t tested it and give no warranty that this will (or won’t!) unbrick your boot!

[8]	Maybe, given some interactions I’ve had online?

[9]	The state is stored in the PM_RSTS register on the PMIC. PMIC registers (generally) survive reset (obviously not power off though), but this particular one is also reset-on-read so it is guaranteed subsequent boots will fall back if the tryboot one fails for any reason.

[10]	The `autoboot.txt` file has a size limit of 512 bytes (presumably because stage 1 has to be incredibly basic, and can’t read FAT chains, so the file is limited to one sector). In other words don’t include extraneous comments or line breaks!

[11]

Bear in mind that FAT is not a journalling file-system. Updates to any boot configuration should generally be done atomically by writing the new content to a temporary file on the target FAT partition, then renaming the temporary file over the original. The rename operation is atomic so anything reading the file-system should either see the original content or the new content, but no partially written file.

[12]	Under the assumption that you shouldn’t ever be fiddling with your “known good” boot configuration. All experiments should be performed in the “alternate” boot assets which are then tested with tryboot before being made the new “known good” set.

[13]	With GNU coreutils 9.5 and above, simply: `mv --exchange config.txt tryboot.txt`!

[14]	This is a slight lie. There are some brief transitional states where current exists, but contains former assets. These should always be rapidly corrected, as we’ll see.

[15]	Though it kinda is… I do control the boot process on your Pi after all! Mwuhahahaha!

D0!

2025-07-04T00:00:00+01:00

Updated: To include the picture I forgot, and fix a bunch of typos

Dear Reader,

Please be forewarned that this is a rambling mess of a post that will dart off on tangents a-plenty. This was originally planned to be posted before noble’s .2 release, but too many work things go in the way and it’s laid in the drafts folder until I noticed it again while writing something else!

Anyway, this may at least be an entertaining distraction into the effort that goes in to dealing with hardware changes in Ubuntu.

Buckle up!

Background

First of all… what’s a stepping? Put simply, it’s a specific version of a particular chip. A new stepping might be released to fix a bug, or improve yields, or make other improvements to a design, but it isn’t a fundamentally new design. Think of it like a bump in a minor version number.

There have been new steppings of Raspberry Pi SoCs in the past, but you may not have noticed. For example, the Pi 4 uses the Broadcom 2711 as its SoC. At launch, this was the B0 stepping. However, when the Compute Module 4 launched later, it moved to the C0 stepping, which corrected an issue in the boot loader’s verification mechanism. I think later versions of the Pi 4 moved to this stepping as well, but looking at my range of Pi 4 boards here, they’re all B0s.

What about the Pi 5? The 4GB and 8GB models initially launched, shipped with the C1 stepping of the Broadcom 2712 SoC. However, Broadcom’s initial design of the 2712 included several features which were never used in the Pi 5, increasing the cost and the power draw of the chip without any benefit. So much so, that initially the planned 2GB variant of the Pi 5 was uneconomic to produce.

The new 2712 D0 stepping stripped out this “dark silicon“, reportedly as much as 30% of the die, making the 2GB possible to produce. It was also used in the subsequent launches of the Compute Module 5, and later the 16GB Pi 5 variant launched in January this year. The assumption is that at some point the D0 will become the standard silicon on the 4GB and 8GB models too.

So far, so good. But also, so what? Why does the OS care about the stepping?

The D0 stepping introduced another change: one of the data structures associated with the shaders grew some new fields in the middle of the structure, shuffling the locations of other fields. Stripping out some details of the structures to make the diff a bit easier to read:

-<struct name="GL Shader State Record" min_ver="71">
+<struct name="GL Shader State Record Draw Index" min_ver="71">
     <field name="Point size in shaded vertex data" size="1" type="bool"/>
     <field name="Enable clipping" size="1" type="bool"/>

     <field name="Vertex ID read by coordinate shader" size="1" type="bool"/>
     <field name="Instance ID read by coordinate shader" size="1" type="bool"/>
     <field name="Base Instance ID read by coordinate shader" size="1" type="bool"/>
+    <field name="cs_basevertex" size="1" type="bool"/>
+    <field name="cs_drawindex" size="1" type="bool"/>
+
     <field name="Vertex ID read by vertex shader" size="1" type="bool"/>
     <field name="Instance ID read by vertex shader" size="1" type="bool"/>
     <field name="Base Instance ID read by vertex shader" size="1" type="bool"/>
+    <field name="vs_basevertex" size="1" type="bool"/>
+    <field name="vs_drawindex" size="1" type="bool"/>

     <field name="Fragment shader does Z writes" size="1" type="bool"/>
     <field name="Turn off early-z test" size="1" type="bool"/>

This is a breaking change, requiring changes in mesa (the library that provides various graphical APIs including OpenGL and Vulkan).

So, how can you tell if you’ve got a C1 or a D0 stepping of a 2712? You could take the cooler off your Pi 5 and look at the lid of the SoC. The stepping is just near the end of the long code.

It’s spot the difference time! The easy stepping difference has been pointed out for you. Can you find the other two differences? No cheating by looking at the PCN

Alternatively, you could try and use Ubuntu 24.04.1 (noble’s .1) Desktop image. If you’ve got a C1, it works! If you’ve got a D0 … erm …

RaspiOS

How do we fix this? First let’s look at RaspiOS. Obviously mesa needs updating, with the requisite changes. RaspiOS added these in version 23.2.1-1~bpo12+rpt3 with the following slightly unceremonious changelog:

mesa (23.2.1-1~bpo12+rpt3) bookworm; urgency=medium

  * Update patches

To be fair, this may have been a matter of trying to keep the existence of the new stepping relatively quiet (though anyone constructing a debdiff would see the 2712D0 mentioned quite prominently). The release date on that package is January 2024, and the Pi 5 2GB model featuring the D0 didn’t hit the market until August 2024.

Still, this is basically all that happens in RaspiOS [1]. New mesa, job done.

Ubuntu

Over in Ubuntu land, things are a bit more complex. As it turned out, the fix was already in the development version (sync’d from upstream), and even Oracular (24.10 which is about to go EOL as I write this) had a sufficiently up to date mesa to incorporate the fix.

However, the interim versions are not what most Ubuntu users are interested in. The vast majority of Ubuntu users (something like 99%) stick with the LTS releases which at the time of writing is noble (24.04). In order to fix things in an existing stable release, we needed an SRU…

SRU

A quick tangent on Stable Release Updates, or SRUs as we refer to them (because there’s nothing more developers love than another TLA). Actually, first another tangent on stability of Linux distros.

Linux distributions can be roughly divided into two categories: stable distros and rolling distros. Stable distros grew out of the frustrations of server operators who wanted a guarantee of stability above all else: once the system was installed and working the only updates they wished to see were those that fixed bugs. They had precisely zero interest in new features and even minor bug fixes were probably more risk than they were worth. The only time risk was introduced into the equation was when the OS as a whole was updated (a “dist-upgrade” in Debian terms), and because this was a known risk, it would be planned for, tested, and implemented in a controlled manner.

Classic “stable” distros include Red Hat, Debian, and of course Ubuntu.

The rolling distros by contrast, have no concept of an OS “release”. They simply package the latest versions of all things at all times. Much of the time, this does work reasonably well, but it does mean that at certain times there fairly major changes that happen in a more or less “adhoc” fashion. Historically, consider the move to systemd, or transitions between Qt versions.

Classic “rolling” distros include Gentoo and Arch.

Some distros incorporate both models. Alpine and Debian are examples of these. While Debian has a “stable” release (currently “bookworm”, soon to be “trixie”), it also has the “unstable” branch called “sid” which is basically a rolling distro in itself. Alpine likewise makes six monthly releases, but has a rolling “edge” branch.

While the “stable” distro model is widely accepted (and very reasonable, in this writer’s opinion), it does cause some tension, particularly with the rise of desktop Linux usage. Desktop users often do want the fancy new features in their applications, don’t have the same level of risk-aversion as the typical hard-boiled sysadmin [2], and thus aren’t willing to wait 2 or 3 years for the next release of their stable distro to get them. This is part of the drive behind the rise of “alternate” packaging systems on various distros (Flatpaks, Snaps, AppImages, etc).

On Ubuntu, updates to stable releases (outside of the realm of snaps) have to follow the Stable Release Update process which incorporates numerous checks and balances to try and avoid regressions. Test plans must be written, reviewed, and agreed upon before anything is even uploaded. Updates must be minimally invasive (no full version backports — those have to follow a separate process which doesn’t automatically update existing installs). After upload, verification is carried out, and only then is the fix allowed into the release.

But not right away…

Phasing

Even after an update to a stable release has made it through review, verification, and upload, such updates go through a “phasing” procedure. This is where the update is slowly trickled out to larger and larger proportions of the userbase. The idea is that, if an update does have a catastrophic bug in it, the damage can be limited to the portion of the userbase that (randomly or by choice) got the update early. In the event of a nasty bug appearing in a stable update, phasing can be halted and the issue investigated before either rolling back the update, or deciding it’s a false-positive and resuming the phasing.

Incidentally, the stable release updates report, and the phasing report are both public if you want to peak at what’s in the pipeline of your stable install! You can read more about phasing in the server documentation.

Noble

Okay, back to the story. Mesa was up to date in oracular, but we needed an SRU to noble to avoid new Pi 5 owners getting a full blown crash the second anything graphical like a login screen dared show its face on the monitor.

Time for a noble SRU. The SRU was written (LP: #2082072), the update sponsored, verification performed, phasing completed, and by the middle of October everything was fixed.

mesa (24.0.9-0ubuntu0.2) noble; urgency=medium

  * Add support for Pi 2712D0 stepping (LP: #2082072)

…

I said, everything was fixed!

…

Right?

…

Oh, ****!

Snaps

One thing I’d forgotten about in this whole process was that there’s no longer just one version of mesa on most Ubuntu desktop systems. Nowadays, a fair number of applications, including some vaguely important ones like Firefox, are shipped as snaps, at least partly in order to alleviate the aforementioned tension between stable distros and desktop users demanding the latest and greatest versions.

There are many ways to handle distributing the “latest” versions of things on a stable distro, but one of the things that must be dealt with is the conflict between the versions of libraries on the stable distro, and the libraries that the “latest” thing expects.

One method is to simply “bundle everything”; include all the libraries that your app expects within its package. Obviously this results in a rather bloated package which isn’t going to share any libraries with anything else on the system, and thus takes more time to load and more memory to run. It also means dealing with all the path nonsense to ensure your application only loads the versions of libraries from its own location. This doesn’t solve all the bundling issues, but it does solve most and it is at least simple.

Another method is “statically compile everything”. This is a close cousin to bundling everything, and still results in a fairly huge download but a little slimmer than bundling all the libraries (and still shares basically nothing at runtime). This is becoming more common with the rise of things like go and rust, which compile this way by default [3].

Snaps take a rather interesting (and different) approach here.

All snaps are built with a “base” snap which represents a stable release of Ubuntu (core22 represents a jammy base, core24 a noble base, and so on). The “base” snap provides the “common” libraries that the snap would expect to find on such a stable release of Ubuntu.

The snap can still statically compile, or bundle, whatever it likes, but this means at least some of its dependencies can be shared (from the base snap). Moreover, the method by which snaps are mounted means that multiple versions of these core snaps can co-exist so it’s fine to be running a snap based on core22, and another based on core24 (obviously they won’t be sharing their core libraries in this case, but that’s the trade-off).

The libraries the core snaps include are fairly limited and, notably, don’t include mesa. However, there are also “content” snaps that provide commonly used sets of libraries above and beyond the core snaps. One of these is the “gnome” content snap which provides (surprise surprise), Gnome, GTK, and, you guessed it, mesa. The gnome content snap itself comes in several versions to accomodate different “base” snaps:

gnome-42-2204 is the gnome content snap using core22 (jammy) as a base
gnome-46-2404 is the gnome content snap using core24 (noble) as a base

Now… guess which gnome content snap Firefox is (still, at the time of writing) using?

Jammy

Shortly after the middle of October, and the noble SRU landing, I noticed we had another problem. The noble daily would happily boot to the login prompt on the new D0 Pi 5, and I could login and run many things. But not Firefox. Starting that, or anything else on the core22 snap, froze the entire desktop. Not good.

So, back to the SRU queue. The original ticket (LP: #2082072) was re-targeted for jammy, the upload was made, and a bunch of other packages regressed. Damn! Various retries later and the package was ready for verification.

But… now we had a problem. This was only an update to the mesa deb package, in Ubuntu jammy (22.04). This wasn’t an update to the gnome content snap; that would have to be done after verification of the SRU. But the Pi 5 was fundamentally not supported in jammy.

Rather than trying to get a jammy image into a state it could boot (and run Gnome) on a Pi 5, I opted to rebuild the gnome content snap locally to verify things instead. This took a few days to work out as it’s an almost entirely undocumented process and I had to figure out that the gnome-sdk snap needed rebuilding first (i.e. the chain goes: mesa deb -> gnome-sdk snap -> gnome snap).

To cut a long story short (or at least… less long), eventually the verification was done, and the jammy mesa update was released.

mesa (23.2.1-1ubuntu3.1~22.04.3) jammy; urgency=medium

  [ Timo Aaltonen ]
  * Add support for Pi 2712D0 stepping (LP: #2082072)

  [ Alessandro Astone ]
  * patches: Backport patch for green artifacting and GPU crash on
    radeonsi with kernel >= 6.10 (LP: #2083538)

Phasing was then completed, then the gnome-sdk snap was rebuilt, then gnome content snap, and finally Firefox was working on the D0 stepping.

By the time all this was done, it was the 31st of Jan. To give some context, the .2 release of noble was scheduled for the 13th of Feb, and I was adamant that we had to get the mesa fix in for that otherwise we’d have new Pi 5 owners unable to run Firefox “out of the box” on our current LTS. You could argue: “users can still login, they can always get the release from updates?”. However, the app-store is also a core22 snap, and hence also crashed the desktop. While there was still the sudo snap refresh route, I’m loathe to tell desktop users they have to use the command line.

Suffice to say, it was butt-clenching time in the run up to noble’s .2 release.

A heavy chain (of dependencies)

What’s the moral of the story?

I’m not entirely sure. Half the problem is that everything above has good reason for being the way it is. The SRU process is long and complicated because it’s borne of the pain of unexpected regressions. It is the way it is, because the alternative is actually more painful.

The snap eco-system is the way it is to avoid the more painful issues with bundling absolutely everything.

Still, what we have here is a dependency chain with many links, and I can’t help but think some of those links may be unnecessary (or at least warrant some consideration).

Should a content snap rely on another snap (the gnome-sdk snap in the middle of the chain, here)?
Should the gnome content snap rely on the jammy debs at all, or should it be able to pull in patches independently of them?
Should we mandate that seeded snaps always have (useful) contact information and well-documented procedures for building?

That said, there are trade-offs in every direction here. At the moment, the gnome content snap can reasonably rely on the stability (and quality) guarantees of the underlying deb. Do we really want to move away from that? Should we be pushing seeded applications to use later base snaps, so that SRUs don’t have to go back as far in time?

I don’t know the answers to all this yet. But I do know taking 3+ months to get a fix to the place it needs to be is no fun!

[1]	This is categorically not true. New device-trees were also added and probably a whole host of other changes I haven’t noticed, but this post is going to be waffley enough without me going off track with every detail!

[2]	I originally wrote “paranoia of the sys-admin”, but frankly it’s not paranoia. This level of risk-aversion is usually the result of hard-learned lessons!

[3]

Which is not to say I think it’s any better. Both static compilation and bundling of everything do “challenge” the security / bug-fix model of classic distributions by essentially ignoring updates to some / all base libraries. Provided they actually stay ahead of those base libraries, that’s not an issue…

Loading the Deck

2025-02-27T00:00:00+00:00

I mentioned in my last post on this topic that fake SD cards have been an issue in the past.

Buy from well-known outlets, not minor vendors on whatever variant of flea-bay you frequent. Just because it’s on Amazon, doesn’t mean it’s not a knock-off; check the actual seller.

For that post, I also purchased a variety of SD cards of various models for testing purposes, some from Amazon. I also stated…

I haven’t had any issues for the last few years, but during them I’ve stuck to buying strictly from either SanDisk or Samsung […]

Guess how that’s going!

Another One Bites the Dust

Another month rolls by, and another Ubuntu release rolls off the production line. In this case it was Noble’s .2 release (24.04.2).

At each release, we go through a round of so-called “ISO tests” [1]. This invariably involves grabbing a handful of SD cards, flashing the release candidate image to them, booting them on various models of Pi model and running through a battery of tests. Once a board has passed or failed, pull the card, reflash, bung it in another board, rinse and repeat.

This means the life of an SD card in my possession boils down to one of two possibilities:

If it’s lucky, it gets stuck in a “long running” Pi serving something in my house. This means 24x7 duty for many years, but usually relatively light duty.
If it’s unlucky, it goes in the drawer and for much of the year it stays there. But, before each release, it’ll get hauled out and for a few days be brutally tortured with tens of gigabytes written to it over and over again.

If I have a card fail on me, it’s inevitably during a release. This time was no different, and I had a fancy gold-coloured “SanDisk Extreme” die. Rather than one of the usual failure modes (fails to read entirely, or silently ignores writes), this one wrote the candidate image happily but then reported tons of I/O errors during boot. A little odd, but not unprecedented.

What was strange was that this card was barely more than 1½ years old. This is an unusually short life-span, in my experience. Brutal though I am to these test cards, they usually last 4 or more years. There was no time to investigate during the release, so I tossed it in the “dead cards” drawer and moved on. However, the release is now done, and I’ve taken another look.

Aaaa!

When you insert an SD card into a Pi, if you’re watching the kernel log (and I often am in some window) you’ll typically see something like the following appear:

[344239.702588] mmc0: cannot verify signal voltage switch
[344239.771565] mmc0: new ultra high speed SDR104 SDHC card at address 59b4
[344239.771898] mmcblk0: mmc0:59b4 USD00 29.5 GiB
[344239.773131]  mmcblk0: p1 p2
[344239.773418] mmcblk0: mmc0:59b4 USD00 29.5 GiB

What does all this mean?

mmc0: cannot verify signal voltage switch: Basically noise; just a warning that the signal voltage was switched but we can’t be sure it was successful (hardware limitation).
mmc0: new ultra high speed SDR104 SDHC card at address 59b4: The interface has noticed a new ultra high speed (ooooh!) card which supports SDR104 signalling at “address 59b4”. Typically an interface only has a single card slot, but technically it can support several. This is the “address” of the card within that interface.
mmcblk0: mmc0:59b4 USD00 29.5 GiB: Mostly a repeat of the prior information. This says the kernel’s created a new “mmcblk0” device representing the card with address “59b4” on the “mmc0” interface, and it’s about 29.5GB in capacity…
mmcblk0: p1 p2: … and it’s got two partitions.

Incidentally, the above is from a Raspberry Pi branded SD card. Let’s see what the broken SanDisk Extreme reports:

[341496.190074] mmc0: cannot verify signal voltage switch
[341496.266122] mmc0: new ultra high speed SDR104 SDXC card at address aaaa
[341496.266891] mmcblk0: mmc0:aaaa SN64G 59.5 GiB
[341496.268624]  mmcblk0: p1 p2
[341496.269098] mmcblk0: mmc0:aaaa SN64G 59.5 GiB (quirks 0x00004000)

Huh. That “aaaa” address looks a bit odd.

This may be complete coincidence, but having come across fakes in the past, sometimes dodgy cards have “suspicious” (read: low entropy) fields in their registers. Serial numbers filled with zeros and the like. Querying other things (CID, manufacturers ID, etc) under /sys/block/mmcblk0/device yields nothing terribly suspicious here, but nonetheless, this feels a little odd so let’s dig a bit further.

Rather than throwing my dead cards in the bin, in recent years I’ve taken to keeping them in a drawer. Most still have some function, and I wondered if I could spot any patterns in their failure modes.

In the drawer I’ve got a “SanDisk Ultra” card (this is their “bog standard” range with grey rather than gold colouring on the case) which failed a year ago after a suspiciously short life-span. What does it report on insertion?

[341732.375548] mmc0: cannot verify signal voltage switch
[341732.448272] mmc0: new ultra high speed SDR104 SDHC card at address aaaa
[341732.448560] mmcblk0: mmc0:aaaa SD32G 29.7 GiB
[341732.450888]  mmcblk0: p1 p2
[341732.451108] mmcblk0: mmc0:aaaa SD32G 29.7 GiB (quirks 0x00004000)

Hmm. Another “aaaa”. Still, maybe that’s just the address that SanDisk cards use generally? Let’s try another SanDisk Ultra branded card which has so far worked reliably:

[345859.140220] mmc0: cannot verify signal voltage switch
[345859.209840] mmc0: new ultra high speed SDR104 SDXC card at address 59b4
[345859.210384] mmcblk0: mmc0:59b4 SD64G 59.4 GiB
[345859.212228]  mmcblk0: p1 p2
[345859.212795] mmcblk0: mmc0:59b4 SD64G 59.4 GiB (quirks 0x00004000)

So, the “aaaa” address appears common to two cards both purportedly from SanDisk, both of which failed suspiciously early.

Aaaamazon

Being more curious now I went and dug out my order history to see when and where these were purchased. I’m not 100% certain of which batch the 32GB “Ultra” card came in. However, I can say with confidence that both were purchased from Amazon, both from the “SanDisk Store” on there, and at least 6 months elapsed between the purchase of the 32GB “Ultra” card (latest February 2023) and the 64GB “Extreme” card (July 2023).

What to do?

Well, not much. It’s another fake from Amazon (which more and more resembles an online tat-store in my experience). Still, I figured in the interests of being a conscientious consumer, I should leave a review cautioning others that even buying from the official “SanDisk Store” is not a cast-iron guarantee of quality. I spent 5 minutes throwing together a quick précis of the above (“aaaa reported by kernel”, “substantially shorter lifespan than other SanDisk cards”, etc. etc.)

And I would’ve left it at that. But for the email that just arrived…

Please edit and resubmit your review

Hello Dave Jones, We could not post your review because it does not meet our community guidelines. Please edit and resubmit your review. Before you do, make sure it meets all of our guidelines.

Intriguing. I wonder which “community guidelines” I’ve violated. Unfortunately, while they’ve linked to their guidelines (which are pretty much as one would expect) they haven’t specified which one I violated, and the review is apparently gone. So I’ve no clue.

Naturally, this leaves me feeling less-than-confident in the positively stellar review ratings alongside this product.

Though, again, hilariously skewed reviews are hardly a new feature of Amazon either. More importantly: can I still stand by my former advice? Probably not.

eBay, Amazon, Alibaba, whoever. If it’s a large marketplace, I’ve had fakes from it (certainly from the aforementioned big three). Ideally, buy from your local electronics vendor (Pimoroni or The Pi Hut for me) but failing that, look for the larger vendors on the big marketplaces.

[1]	ISO because this used to be limited to ISO CD images. Though we still make ISO images for various platforms, we’re just testing raw disk images for the Pi.

How Noble in reason

2025-02-02T00:00:00+00:00

Noble’s release is coming up (as I edit this, it’s now many months in the past and even noble’s .1, and oracular’s releases are long gone), so it’s high time I write something about this most difficult of births. Still, you may have to forgive my use of the future tense…

Welcome back, pointers

What’s new? For Pi users the big news is that this is an LTS, and it’ll be the first Ubuntu LTS that supports the new hardware of the day: the Pi 5. For those not familiar, that means 5 years of free support (and another 5 paid, or free at various tiers of usage).

All well and good, but what’s new in the software itself? Not quite as much as I’d hoped. The final part of my series on NBD (part 3 following on from part 1, part 2) won’t operate until the first second point release (24.04.2) [1] scheduled for February next this year, as I didn’t manage to shepherd the nbd-client package through into main in time. Sadly, there wasn’t time to address libcamera either and so, at present, it’s completely broken and won’t even list the Pi’s camera modules. This will be a priority to fix in the next (oracular) next +1 (plucky) cycle (for eventual SRU to noble). In fact, those are just the tip of a rather long list of stuff I need to fix.

But enough doom and gloom. One piece of very good news is that the Firefox snap (or more precisely the GNOME content snap) finally got updated to mesa version 23.2. This brings full hardware acceleration on the Pi 5 to the browser, so instead of the hilariously pathetic 2fps we got with the classic Aquarium benchmark in mantic, it’s more like 40-60fps 30-40fps in noble (I managed 60fps at 1080p full-screen when first testing this pre-release, but by the time we got to release it seems something else in mesa changed and we were back down to 30-40 — still, better than 2!).

GNOME’s also been updated to version 46. Amongst many changes, this fixed a long standing and, as we’ll see, quite pertinent issue with its notifications. As you may have noticed in prior versions, desktop notifications could have “actions” associated with them which would appear as buttons beneath the notification. For example, when plugging in a storage device the action on the associated notification would be “Open with Files”.

But if you happened to miss the notification, and it wound up in the tray, the actions were gone…

As the linked ticket showed, this has been a pretty long standing bug in the desktop, but GNOME 46 finally squashes it, and just in time for one of the new Pi-specific bits in noble. The Pi 5’s new PMIC includes the ability to inform users when the previous reboot was caused by an undervolt (a “brownout” because the power-supply isn’t up to snuff). The RaspiOS desktop helpfully informs users via a desktop notification when this happens, and now so does Ubuntu.

As you can see, the Ubuntu version also includes actions to silence the warning, and to get more information. But when the warning disappears into the system tray, with previous versions of GNOME there were no actions and, worse still, the notification was truncated. Now you can expand it in the tray to see everything you need.

What a piece of work…

Another major piece of news is a new official flavour of Ubuntu for the Raspberry Pi: Edubuntu have spun up a pre-installed Raspberry Pi desktop image. There are a wealth of applications pre-installed here, including several I was previously unaware of like Marble, a fantastic globe explorer including some wonderful ancient map modes (ah, the island of California!), Kig, a constructive geometry application, and laby, a very nice gamified coding tutor supporting several languages.

How infinite in faculty

The other big news is that, while armhf (32-bit ARM) as an architecture is still supported for 24.04’s lifetime, we will not be producing Raspberry Pi images for it. This means there is no upgrade path for users currently on (for example) 22.04 armhf images. Those users will need to re-image with the arm64 image.

Why?

There are multiple reasons here, but the primary reason is a combination of the Pi 5’s limited support for 32-bit architecture, and a difference in the way RaspiOS and Ubuntu support multiple models of Raspberry Pi with a small number of images.

Traditionally (and even today in some parts of the SBC space), OS vendors produced an OS image per board. Imagine this in the context of Raspberry Pi. There would be an image for the Raspberry Pi 2B, another for the Raspberry Pi 3B, possibly another for the 3B+, certainly another for the 4B, and so on, each tailored to their specific board. There might be further variations on top of this, like armhf and arm64 images for the 3B, two more for the 4B, maybe different desktop variants, the list goes on.

The result would be an immensely confusing proliferation of images, with the user having to know exactly which board they had, and typically not being able to move eminently moveable storage (like an SD card) between boards.

RaspiOS (or Raspbian formerly) was quite the pioneer here. Early on, as the Pi family grew from the B+ to the 2B and beyond, it was recognized that asking teachers (particularly, given the educational focus) to know exactly which board(s) they had, and figure out which image they needed to download, was simply not practical. This is why there’s still a single RaspiOS image that works on absolutely all Raspberry Pi boards: keep it simple for the user.

Obviously there’s been some proliferation (a lite variant without a desktop, and architecture variations), and the situation of having a single image that works universally probably can’t last forever, but ultimately the goal of minimizing the number of images is still sought.

What about Ubuntu?

When I first joined the Foundations team several (nearly six?) years ago, the “image per board” model was being followed. There was an Ubuntu image for the 2B, another for the 3B, and they were thinking about another for the (at the time, forthcoming) 4B [2], assuming they could drop the 2B image in due course as it ran out of support.

The first thing I had to correct was the notion that the 2B was going to drop out of support anytime soon. Raspberry Pi have always promised 10 years of production on all models, and so far they’ve not disappointed: as of writing this the 2B is still available. This also meant that following the “image per board” model was completely unsustainable. The resulting number of images would be insane (not to mention the support burden). So I switched Ubuntu to a “minimal images” model, specifically one image for armhf and another arm64 [3].

How?

While RaspiOS and Ubuntu both follow the “minimal images” model, we both do so in quite different ways. What matters here is the separation between “kernel” and “userland”, and what architectures are supported by each SoC. In userland, things are relatively simple:

SoC	ARMv6	ARMv7	ARMv8
2835 (Pi 1, Pi Zero)	✓
2836 (Pi 2)	✓	✓
2837 (Pi 3, 3+, Zero 2)	✓	✓	✓
2711 (Pi 4, 400)	✓	✓	✓
2712 (Pi 5, 500)	✓	✓	✓

As you can see, ARMv6 is a “universal” architecture for userland, so compile everything except the kernel for ARMv6 and your image will work on all boards. Great! But what about the kernel…

SoC	ARMv6	ARMv7	ARMv8
2835 (Pi 1, Pi Zero)	✓
2836 (Pi 2)		✓
2837 (Pi 3, 3+, Zero 2)		✓	✓
2711 (Pi 4, 400)		✓	✓
2712 (Pi 5, 500)			✓

Now things are more complicated. For a truly universal image, you require separate kernels. This is the route RaspiOS has taken on their 32-bit images. The userland is ARMv6, but there are four [4] separate kernels on the image, with the bootloader selecting the appropriate one for the board.

At Ubuntu, we don’t have the resources to maintain four separate kernel builds for the Raspberry Pi [5]. However, we also never (officially) supported the Pi 1 models. With no need for ARMv6 support, the route we took in Ubuntu was a simpler “pure architecture” model: the armhf images use an ARMv7 kernel and userland; the arm64 images use an ARMv8 kernel and userland.

To be clear, I don’t think either model is perfect. There are advantages, disadvantages, and compromises all round here. This is just how things are as a result of history and the limitations and resources surrounding each distro. Still, the goal of minimizing the number of images is a worthwhile one, however it is achieved.

Good for you, it’s a growth industry…

There’s a problem. Note that the 2712 on the Pi 5 cannot boot an ARMv7 kernel. So with noble we were in the position of having an armhf image that could boot on some modern Pi models, but not all. The options available were:

Retro-fit an arm64 kernel onto the armhf userland images and have the bootloader select the right one, a bit like RaspiOS. Ultimately this was too complex to support (several systems assume there is “one kernel” on the system, not multiple that need to be kept in lock-step)
Make two armhf images, one with an armhf kernel and one with an arm64 kernel. Given this would only be of use to Pi 5 users who wanted an armhf userland, this seemed far too niche to expend time and effort on (you’ll see just how niche in a bit!).
Make the armhf image, but explicitly state it’s not supported on the Pi 5, and likely any future Raspberry Pi boards.
Drop the armhf image, leaving it as a legacy architecture only for old releases.

Personally, I would have preferred the third option. However, armhf has always had minimal (and waning) interest from Ubuntu users. A few years ago, for every one armhf download there were 5 arm64 downloads. Today, that ratio is 1:30 and rising.

Given that arm64 support covers all boards with the exception of the Raspberry Pi 1 (which we never supported) and the Raspberry Pi 2 (which is easily classified as a “legacy” board at this point, i.e. people should not be planning new products based on this board), the decision was made to simplify things in this release, and drop armhf for the Raspberry Pi. The resulting arm64 images available for noble (and onwards) work on every Raspberry Pi board we’ve ever supported except the Pi 2.

The support matrix for all models of Raspberry Pi, plotted against all currently supported versions of Ubuntu can be found in the Raspberry Pi section of the new Ubuntu Boards documentation project. This aims to replace the out of date (and rather variable) Ubuntu wiki pages with a single source of information for installing Ubuntu on all manner of single board computers (including Raspberry Pi, naturally).

Let’s have a drink and forget the whole damned thing

To say it’s been a challenging year would be an understatement of frankly comic proportions.

Between the extensive work on the solution to the Year 2038 problem (i.e. the 64-bit time_t transition on armhf), the challenges involved in the xz-utils hack, some considerable churn in personnel, and the mess we found ourselves in over the D0 stepping (more on that in the next post!), 2024 has been … tough. Still, I’m vaguely hoping 2025, and 25.04 (plucky) might be less “eventful” [6].

I’ve got a new helper on the Pi side of Foundations, with the result that we might actually get the camera working this cycle (at least, that’s the plan). Various long-standing architectural issues might resolve this cycle (he says, looking at the desktop installer). And there’s a fair chance we won’t crash into any more hardware incompatibilities until … oh, at least the middle of the year.

I hope.

[1]	We’ll see if the nbd MIR makes it in time. Pragyansh (my new helper in Ubuntu on Pi) has done fantastic work on this, but even that may not be enough and unfortunately there are even more pressing things to get in. But that’s for the next post…

[2]	There was no image for the B+ (or prior) boards because Ubuntu never (officially) supported the ARMv6 architecture on those boards

[3]	Both were server images only at the time — we didn’t have a desktop offering as the Ubuntu desktop was far too heavy to run on the 3B+, which was the latest model at the time

[4]	There are two ARMv7 kernels: one for 2837 boards (Pi 3 era), one for the 2711 (Pi 4). By default, the 2711 boots the ARMv8 64-bit kernel, but can be forced back to 32-bit for various reasons, in which case it boots the `kernel7l.img` rather than `kernel7.img`

[5]	More precisely: we maintain dozens of different kernel variations but rarely more than one or two configurations for a specific board or vendor

[6]	Well, 2025 got off to a crap start. RIP Steve.

I ain’t got NBD

2024-12-03T00:00:00+00:00

Last time we set up an NBD server, and the associated DHCP machinery to netboot our Pi. We then discovered a rather serious problem: when the Pi updated the content of the boot partition, the TFTP wouldn’t notice and would serve old files because the mount of the boot partition on the server didn’t realize it was “shared” with the NBD server.

Come and take a chance with me?

As mentioned in the last post, in trying to solve this I ventured through a myriad of hacky or simple solutions, all attempting to work around the “caching” of pages by the file-system driver on the server side. Yet again, caching rears its head as one of the two “hard problems” of computer science [1], though the root of the issue here is the assumption that the file-system driver has exclusive access to the underlying block device (the cache is merely a consequence of this).

In hindsight, I should’ve looked at the pieces and concluded what I eventually did: just write your own TFTP server! One which can read files directly out of the boot partition of the OS image. This solution sounds like a sledgehammer to crack a nut, but actually it’s less complex than it seems. All it’s got to handle is simple partitioning (MBR or GPT), reading a FAT file-system , and TFTP.

I may not be selling this as “simple”, but let’s break down what each of these actually entails:

MBR

Master Boot Record partitioning is an ancient method of partitioning a floppy disk hard drive USB stick small storage medium, but while it’s very difficult to accurately write (in such a way that it’ll actually … boot things), it’s not hard to read (other than the minor mess that is logical partitioning, intended to get around the original 4-partition limit). A few hours work. Maybe.

GPT

GUID Partition Tables are the modern, and frankly trivial (by comparison to MBR) means of partitioning storage media. No more than an hour needed here.

FAT

File Allocation Tables are another positively ancient piece of computing history, used for storing files on just about every piece of computing media with the possible exception of the tape drive (and I wouldn’t put it past someone to have tried that). The file-system isn’t complicated in basis (in fact I’ve written a library for something very similar before [2]); the complexity comes from the myriad implementations that use the same structures, but disagree subtly (and not so subtly) about the meaning or even position of certain fields within them.

Still, a few evenings to get the basics done, then several more to work out the rough edges (FAT-12, long filename handling, etc. etc.), then a few more to build a nice Path-like API on top of it, and we’re good!

TFTP

Trivial File Transfer Protocol is, tautologically, trivial. Getting the basic protocol implemented took only a couple of hours (including testing with several implementations). Then it was onto the extensions; I wound up only bothering with those that the Pi bootloader attempts to negotiate and one more that’s useful for testing (tsize, blocksize, and timeout) though hopefully I’ve structured things suitably for a future addition of the windowsize option. Still, only a few more hours on that.

All in all, despite a few headaches (mostly around FAT’s obscure history), it wasn’t difficult to throw all this together, stir it around a bit, and produce what I needed (would’ve been nice to make it into a “learn Rust” project, but I was already waaay overdue on this post series, so I wound up going with my comfortable old Python).

The result is nobodd: a TFTP boot-server that will read files directly out of the FAT partition of an image without mounting it.

NBD! NBD!

Let’s walk through a set up using nobodd and nbd-server. The client side is exactly the same as before so I’ll just refer you to the start of the last post for that part. Go through requirements (a Pi and a server), configuring the Pi for netboot, adding linux-modules-extra- and nbd-client, and transferring the regenerated initramfs and identity to the server, and shutting down the Pi [3].

Now, on the server we’re going to do things a bit differently. This time we’re going to configure a “template” image that we can copy to “instance” images any time we want a new install. And this time we’re going to use the fancy little nobodd-prep tool from nobodd to do it.

Start as we did before by downloading the image and verifying it:

$ sudo -i
Password:
# mkdir /srv/images
# cd /srv/images
# wget http://cdimage.ubuntu.com/releases/22.04.3/release/ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz
 ...
# wget http://cdimage.ubuntu.com/releases/22.04.3/release/SHA256SUMS
 ...
# sha256sum --check --ignore-missing SHA256SUMS
ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz: OK
# rm SHA256SUMS

Now, we simply use nobodd-prep to customize it. This tool can be used to resize the image, re-write the cmdline.txt parameters, and copy files onto the boot partition (including the customized initrd.img and a cloud-init user-data file), all in one go. It can only operate on uncompressed images though, so we still need to do that:

# unxz ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz
# mv ubuntu-22.04.3-preinstalled-server-arm64+raspi.img jammy-template.img
# apt install nobodd-tools
 ...
# cat << EOF >> user-data
package_update: true
packages:
- avahi-daemon
- nbd-client
- linux-modules-extra-raspi
# cp jammy-template.img jammy.img
# nobodd-prep --copy user-data --copy initrd.img --size 16GB jammy.img

And that’s it! My hope is that, in noble (24.04) most of these steps will not be necessary. In that release, it should be possible to download the image unpack it, give it a nice name, run nobodd-prep to customize it, and go from there.

Hi, Future Dave here! Well, the MIR for nbd-client, which I had hoped would be a simple affair, erm, wasn’t [4]. It didn’t make it into noble (24.04), or oracular (24.10). It might make it into plucky (25.04), thanks largely to the efforts of my new side-kick [5]. But this update is so overdue, I’m just going to post it anyway, and hope it’ll be of use to Future You if/when the relevant things finally make it into the image.

—Future Dave

Next, we’ll install and configure the required daemons. As before, we’ll use dnsmasq as a DHCP proxy, and nbd-server. However, this time instead of dnsmasq also handling TFTP duties, we’ll use nobodd-tftpd for that. First up, install the packages:

# apt install dnsmasq nbd-server nobodd-tftpd

Next, configure dnsmasq:

# cat << EOF >> /etc/dnsmasq.conf
interface=eth0
bind-interfaces
dhcp-range=192.168.255.255,proxy
pxe-service=0,"Raspberry Pi Boot"
# systemctl restart dnsmasq

Note

Adjust the reference to eth0 if your Ethernet NIC is named something else. If your network’s mask is not 192.168.255.255, adjust this accordingly.

Now the NBD server:

# chown nbd:nbd jammy.img
# ls -lh jammy.img
-rw-r--r-- 1 nbd  nbd 16.0G Dec  3 13:48 jammy.img
# cat << EOF >> /etc/nbd-server/conf.d/jammy.conf
[jammy]
exportname = /srv/images/jammy.img
EOF
# systemctl restart nbd-server

And finally the TFTP server:

# cat ~ubuntu/pi-ident.txt
Serial          : 1000000089025d75
# piserial=$(sed -e '1s/^Serial.*\([0-9a-f]\{8\}\)$/\1/' ~ubuntu/pi-ident.txt)
# echo $piserial
89025d75
# cat << EOF >> /etc/nobodd/conf.d/jammy.conf
[board:$piserial]
image = /srv/images/jammy.img
partition = 1
EOF
# systemctl reload nobodd-tftpd

The final config piece (/etc/nobodd/conf.d/jammy.conf) ties the Pi’s board serial number to the image that we want to actually boot. The cmdline.txt on the boot partition of that image needs to point to the NBD server serving that same image. This was actually handled implicitly by the nobodd-prep command above. It assumes:

The NBD server is the machine it is running on
The NBD share will be named after the stem of the image’s filename (in other words it transforms “jammy.img” to “jammy”)
The root partition is the first non-FAT-type partition in the image (partition 2 in this case)

Hence, in this case it will have automatically inserted the following in the cmdline.txt of the image:

ip=dhcp nbdroot=ubuntu/jammy root=/dev/nbd0p2 ...

When the end comes, I know

At this point, you should be ready to go. Power up the Pi, and if everything is working correctly, it should netboot into your jammy image, applying the custom cloud-init along the way.

How do you add more boards? Now you’ve got your template image, the process should be as simple as:

Find your next board’s serial number
Create another copy of jammy-template.img
Customize it with nobodd-prep
Add the NBD server configuration, reload the NBD server
Add the TFTPD server configuration, reload the TFTPD server

It’s worth noting that nobodd-prep is technically capable of generating the required configurations for the last two items. However, I’m a bit unhappy with the design of it at the moment. It’s become a horribly convoluted “big” tool, and I think things would be much more flexible if it was broken up into a series of much simpler tools that emulate the functionality of cat, cp, rm, ls, and providing something that uses those pieces to achieve the customization of the current tool.

Another aspect of this setup worth noting is that we haven’t used mount anywhere. Neither nbd-server nor nobodd-tftpd require mounts or loop devices to operate. Why is this important? Because now this entire set up can run in an unprivileged container, which makes testing things much simpler. There are still hoops to jump through [6] if you want to go this route, which are beyond the scope of this post, but it is at least possible.

[1]	Alongside naming things and off-by-one errors …

[2]	I’ve written stuff very similar to a FAT file system driver in Python before.

[3]	In other words, follow the last post until “Why’d you have to be so good?”. Incidentally, I make no apologies for the musical puns in my titles!

[4]	After several years in Foundations, you’d think I’d have learned by now, wouldn’t you? If it looks hard, it is. If it looks easy, it isn’t.

[5]	Yes! It’s not just me anymore! Say “hi” to r41k0u who’s also doing some sterling work on the camera stack this cycle.

[6]	Specifically this involves getting your container to appear directly on your network, rather than hiding behind NAT, so that it can proxy DHCP requests

NBD Does it Better

2023-10-30T00:00:00+00:00

In the prior post we looked at the issues with using NFS as our netboot root, and some of the alternatives. This time we’ll go through setting up the server and client of an NBD netboot system. If you want to play along at home you’ll need:

A Raspberry Pi. I’ll be using a Pi 4B [1].
A real [2] server running Ubuntu. I’m going to be using another Pi for this, but any old PC should be fine too (whatever server you use will benefit from as much IO bandwidth, both disk and network, as you can get).
An ethernet network. You can’t netboot a Pi over wifi, so you’ll need an ethernet set up for this.

Before we begin you should ensure your server is up and running, and that you have remote SSH access to it. In the following instructions I’m assuming that your server’s hostname is server, and that you have an ubuntu user on it which can sudo to root. Adjust according to your setup!

We’re going to start with the client side of things, oddly, but there is method in the madness. Firstly, we need to ensure that the Pi’s bootloader is configured to attempt bootloading (which none are by default). And secondly, we need to do some surgery on the initramfs for later.

Some Kind of Magic

We’re going to be using the current Ubuntu LTS (“jammy”, 22.04.3) throughout this guide, both on the server later, and for the Pi’s client image, to keep things relatively simple. Fire up rpi-imager and flash Ubuntu 22.04 server onto an SD card, then boot that SD card on your chosen Pi.

Warning

Do not be tempted to upgrade packages at this point. Specifically, the kernel package must not be upgraded yet.

Now we need to ensure that the Pi is configured to attempt network booting. This is a one-time change which will be stored in the EEPROM of the Pi in question. On the Pi, extract the current boot configuration from the EEPROM, modify the existing BOOT_ORDER= line (or append a new one if none is present), and apply the modified configuration:

$ sudo rpi-eeprom-config > boot.conf
$ cat boot.conf
[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf41
$ sed -i -e '/^BOOT_ORDER=/d' boot.conf
$ echo BOOT_ORDER=0xf21 >> boot.conf
$ cat boot.conf
[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf21
$ sudo rpi-eeprom-config --apply boot.conf
Updating bootloader EEPROM
 ...
$ sudo reboot

Obviously, feel free to fire up your favourite editor and just change the BOOT_ORDER= line yourself, instead of messing with sed. The mysterious 0xf21 value is explained fully in the BOOT_ORDER documentation on the Raspberry Pi website, but simply means try the SD card first (1), followed by the network (2), instead of USB boot (4) previously, and if both fail then repeat (f) [3]. The digits are specified in reverse order for $reasons.

The reboot at the end is required to apply the new configuration to the boot EEPROM. You can run sudo rpi-eeprom-config after rebooting to check the newly applied configuration.

Next, we need to install the linux-modules-extra-raspi package for the currently running kernel version. The reason is that the nbd kernel module was moved out of the default linux-modules-raspi package for efficiency. We specifically need the version matching the running kernel version because installing this package will regenerate the initramfs (initrd.img). We’ll be copying that regenerated file into the image we’re going to netboot and it must match the kernel version in that image. This is why it was important not to upgrade any packages after the first boot.

We also need to install the NBD client package. This will add the nbd-client executable to the initramfs, along with some scripts to call it if the kernel command line specifies an NBD device as root [4]:

$ sudo apt install linux-modules-extra-$(uname -r) nbd-client

We need to gather one piece of information about the client Pi for use later on the server: its serial number. We’ll store this in a file and copy it and the initrd.img to the server. Finally, we’ll shut down the Pi and move to the server side of things:

$ grep Serial /proc/cpuinfo > pi-ident.txt
$ cat pi-ident.txt
Serial          : 1000000089025d75
$ scp -q pi-ident.txt ubuntu@server:
$ scp -q /boot/firmware/initrd.img ubuntu@server:
$ sudo poweroff

Why’d you have to be so good?

The first thing to do on the server is get the image [5] we want to serve, and do a little surgery on it. We flashed Ubuntu 22.04.3 so we set up a directory under /srv to hold the image, wget it, and check the SHA256 checksum. Note that we’re going to perform most of these steps as root:

$ sudo -i
Password:
# mkdir /srv/images
# cd /srv/images
# wget http://cdimage.ubuntu.com/releases/22.04.3/release/ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz
 ...
# wget http://cdimage.ubuntu.com/releases/22.04.3/release/SHA256SUMS
 ...
# sha256sum --check --ignore-missing SHA256SUMS
ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz: OK
# rm SHA256SUMS

Now we’re going to unpack the image (it’s no good mounting something that’s XZ compressed), rename it to something more manageable , and expand the image file to the full size of SD card we want to emulate (I’m using 8GB here, but change the fallocate command accordingly):

# unxz ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz
# mv ubuntu-22.04.3-preinstalled-server-arm64+raspi.img jammy.img
# ls -lh
-rw-rw-r-- 1 root root 4.0G Oct  5 14:12 jammy.img
# fallocate -l 8G jammy.img
# ls -lh
-rw-rw-r-- 1 root root 8.0G Oct  5 14:51 jammy.img

We’ve expanded the image, but naturally the partitions inside it haven’t been changed in size, and nor have the file-systems inside those partitions. However, that’s fine. This is exactly what a freshly flashed 8GB SD card looks like. The device (in this case the image file) is 8GB, but the root partition inside is a mere 3.7 GB. We can use fdisk to see this:

# fdisk -l jammy.img
Disk jammy.img: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x542d34fa

Device     Boot  Start     End Sectors  Size Id Type
jammy.img1 *      2048  526335  524288  256M  c W95 FAT32 (LBA)
jammy.img2      526336 8320243 7793908  3.7G 83 Linux

On first boot of a 8GB SD card, the cloud-init service checks its configuration, sees that it should expand the root, re-writes the root partition, and expands the file-system. Exactly the same will happen here when we first boot this image.

Next, we need to overwrite the initrd.img in the boot partition of this image, with the one we generated on our client (and which we copied to the server earlier). In order to do so we need to mount this image. Normally, if the image file contained only the file-system we wanted to manipulate, this would be as trivial as running mount -o loop jammy.img some-path. But this image contains partitions, so the file-system we wish to mount isn’t at the start of the image.

To get around this, instead of having a loop device created implicitly (with mount’s -o loop option), we need to make our own loop-device and tell the kernel to scan it for partitions. Then we’ll create a mount-point and mount the first partition there. Finally, we’ll copy our customized initrd.img into the mount-point:

# losetup --find --show --partscan jammy.img
/dev/loop5
# ls -l /dev/loop5*
brw-rw---- 1 root disk   7, 5 Oct  5 21:20 /dev/loop5
brw-rw---- 1 root disk 259, 0 Oct  5 21:20 /dev/loop5p1
brw-rw---- 1 root disk 259, 1 Oct  5 21:20 /dev/loop5p2
# mkdir boot
# mkdir boot/jammy
# mount /dev/loop5p1 /srv/images/boot/jammy
# cp initrd.img boot/jammy/

Warning

The loop device on your system will likely have a different number; adjust /dev/loop5 references accordingly.

Next, we should also customize the cloud-init initial configuration to ensure the image installs the same packages that we installed on the client earlier:

# cat << EOF >> boot/jammy/user-data
package_update: true
packages:
- avahi-daemon
- nbd-client
- linux-modules-extra-raspi
EOF

If we don’t do this, the next time our netbooted client refreshes its initramfs, it would generate it without the NBD client (and would naturally fail at the next reboot).

Now we need to edit the kernel command line to tell it that its root device is an NBD share. The kernel command line is one long line of text with space-separated portions. We’re going to change those space-separated bits into individual lines to make it easier to manipulate, remove the existing root=LABEL=writable portion [6], and insert the following portions instead:

ip=dhcp — we need an IP address to find the root device, and that it should obtain it via DHCP
nbdroot=server/jammy — set up an NBD client and connect to the jammy share on the host server (adjust this to match your server’s name or IP address)
root=/dev/nbd0p2 — find the actual root on the second partition of the connected NBD device

Instead of doing my usual confusing one-liner, we’ll step through the actions below, but feel free to fire up your favourite text editor and just edit cmdline.txt directly if you find that easier:

# cat boot/jammy/cmdline.txt
console=serial0,115200 dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash
# cat boot/jammy/cmdline.txt | tr ' ' '\n' > /tmp/cmdline.txt
# cat /tmp/cmdline.txt
console=serial0,115200
dwc_otg.lpm_enable=0
console=tty1
root=LABEL=writable
rootfstype=ext4
rootwait
fixrtc
quiet
splash
# sed -i -e '/^root=/ s@=.*$@=/dev/nbd0p2@' /tmp/cmdline.txt
# sed -i -e '/^root=/ i ip=dhcp' /tmp/cmdline.txt
# sed -i -e '/^root=/ i nbdroot=server/jammy' /tmp/cmdline.txt
# cat /tmp/cmdline.txt
console=serial0,115200
dwc_otg.lpm_enable=0
console=tty1
ip=dhcp
nbdroot=server/jammy
root=/dev/nbd0p2
rootfstype=ext4
rootwait
fixrtc
quiet
splash
# paste -s -d ' ' /tmp/cmdline.txt > boot/jammy/cmdline.txt
# cat boot/jammy/cmdline.txt
console=serial0,115200 dwc_otg.lpm_enable=0 console=tty1 ip=dhcp nbdroot=server/jammy root=/dev/nbd0p2 rootfstype=ext4 rootwait fixrtc quiet splash

Now it’s time to configure the DHCP proxy and NBD server we talked about in the last article. We’ll start with the packages we’re going to need: the NBD server itself, and the ubiquitous dnsmasq daemon which will be handling DHCP, and TFTP for our netbooting clients.

Note

Don’t worry if you’ve already got a DHCP server on your network. I’ve assumed that you almost certainly do and will be configuring the DHCP portion of dnsmasq in DHCP “proxy” mode where it simply steps in to augment the options transmitted by the authoritative DHCP server.

To put it another way: you shouldn’t have to dismantle or reconfigure your network to play along!

Install the required packages:

# apt install dnsmasq nbd-server

During this installation you may see several warnings about dnsmasq being unable to start due to the address already being in use. This is normal and occurs because systemd-resolved is already listening on port 53 (the DNS port) for the loopback address, so it can cache DNS requests. We now configure dnsmasq to only listen on port 53 of the ethernet NIC, to act as a DHCP proxy, and TFTP server:

# cat << EOF >> /etc/dnsmasq.conf
interface=eth0
bind-interfaces
dhcp-range=192.168.255.255,proxy
pxe-service=0,"Raspberry Pi Boot"
enable-tftp
tftp-root=/srv/images/boot
EOF
# systemctl restart dnsmasq

Note

Adjust the reference to eth0 if your Ethernet NIC is named something else. If your network’s mask is not 192.168.255.255, adjust this accordingly.

Next up is the NBD server, which simply needs to point the share “jammy” at our “jammy.img”. However, we also need to remember to change ownership of our images so the unprivileged “nbd” user can write to it:

# chown nbd:nbd jammy.img
# ls -lh
total 8.1G
drwxr-xr-x 3 root root 4.0K Oct 30 16:49 boot
-rw-r--r-- 1 nbd  nbd  8.0G Oct 30 16:55 jammy.img
# cat << EOF >> /etc/nbd-server/conf.d/jammy.conf
[jammy]
exportname = /srv/images/jammy.img
EOF
# systemctl restart nbd-server

Finally, we link our Pi’s serial number (or more precisely, the last 8 digits of it, if it’s longer than that) with the mounted boot partition.

# cat ~ubuntu/pi-ident.txt
Serial          : 1000000089025d75
# piserial=$(sed -e '1s/^Serial.*\([0-9a-f]\{8\}\)$/\1/' ~ubuntu/pi-ident.txt)
# echo $piserial
89025d75
# ln -s jammy boot/$piserial
# ls -l boot
total 3
lrwxrwxrwx 1 root root    5 Oct 30 16:49 89025d75 -> jammy
drwxr-xr-x 3 root root 2560 Jan  1  1970 jammy

Keeps Me From Runnin’

You’re now at a point where you can try netbooting your client Pi. Remove its SD card, and plug it in. You should see the “rainbow” boot screen appear fairly quickly, but there’ll be a long pause on that screen. The reason is that your Pi is transferring initrd.img (which is now much larger than normal due to our installation of linux-modules-extra) over TFTP which is not an efficient protocol without certain extensions, which the Pi’s bootloader doesn’t implement. However, eventually you should be greeted by the typical Linux kernel log scrolling by and reach a typical “booted” state the same as you would with an SD card.

If you hit any snags here, the following things are worth checking:

Pay attention to any errors shown on the Pi’s bootloader screen. In particular, you should be able to see the Pi obtaining an IP address via DHCP and various TFTP request attempts.
Run journalctl -f --unit dnsmasq.service on your server to follow the dnsmasq log output. Again, if things are working, you should be seeing several TFTP requests here. If you see nothing, double check the network mask is specified correctly in the dnsmasq configuration, and that any firewall on the server is permitting inbound traffic to port 69 (the TFTP port).
You will see numerous “Early terminate” TFTP errors in the dnsmasq log output. This is normal, and appears to be how the Pi’s bootloader operates (my guess would be it’s attempting to determine the size of a file with the tsize extension, terminating the transfer, allocating RAM for the file, then starting the transfer again).
If cloud-init’s final phase running apt update and apt install avahi-daemon linux-modules-extra-raspi nbd-client fails (it seems to randomly on my test Pi), just login and run them manually.

At this point you should have a fully booted system with a block device as the root. All is good! We can use anything we would on a regular Pi with an SD card, including snapd, docker, or anything else. But there’s a rather serious problem waiting silently for us. If things have worked correctly, nbd-client and linux-modules-extra-raspi will have been installed. This will have re-built the initramfs, and likely also have upgraded the kernel package. If you attempt to reboot at this point, you’ll likely find the next boot fails on the rainbow screen.

Spot the problem? Two things are accessing the boot partition’s block device. First, the TFTP server (dnsmasq) is reading the boot partition via a loop device. Second, the nbd-server is serving the boot partition directly from the image.

Recall that mounts of block devices assume they have exclusive access to the underlying block device. But here, the vfat driver on the server does not have exclusive access to the boot partition. What happens if we change the boot partition on the client. Does the server notice?

Try the following experiment:

On the client: sudo touch /boot/firmware/foo
On the server: ls /srv/images/boot/jammy/foo

You should find that, even if you sudo sync on the client the foo file simply won’t show up on the server’s boot mount. The boot mount needs remounting to make it re-read the necessary blocks from the underlying image. This bodes badly for anything that writes to the boot partition, as happens when installing a new kernel, or anything that causes the initramfs to be rebuilt.

Thus, to resurrect your netbooting Pi do the following on the server:

# umount /srv/images/boot/jammy
# mount /dev/loop5p1 /srv/images/boot/jammy

Well, this sucks. Solutions?

Use NFS for the boot partition mount instead of NBD. This will work, but there’re several issues here. Firstly … it requires an NFS server! Secondly, you need to customize the fstab in the image before first boot (urgh). Thirdly you need to extract the content of the boot partition to an exported directory and point TFTP to that (which wastes space) or you can stick with the looped mount, but you must be absolutely certain that you mount the boot partition with NFS on the client, not NBD.
Try and remount the boot partition when the client reboots. This is risky as it’s almost impossible to guarantee in practice. If you think you can get clever with DHCP hook scripts in dnsmasq to remount just as the machine is booting (because DHCP comes before the first TFTP request), you’re wrong [7]. dnsmasq scripts run asynchronously and you can’t unmount a “busy” partition. It’s still possible to do this manually, but that’s boring! [8]
Write our own TFTP server that directly accesses the image without ever mounting it! Hmmm…

Join me next time for the conclusion of this series where … well, you can probably guess where this is going …

[1]	The Pi 2B, 3B, and 3B+ can also netboot, but the instructions differ on each, and this article is long enough as it is. See Network boot for more information, and drop me a question in the comments if you need more!

[2]	Your server needs to be a “real” server, or at the very least a full virtual machine, not a container. Specifically, you need to be able to create loop devices and mount images (both of which are not typically possible in a container environment).

[3] If you just want to add network boot instead of replacing the USB boot option, that’s fine too — you can use 0xf421 or 0xf241 here too. Also, the reason we’ve left SD card boot in the order is simply for safety (there are recovery methods available even if you remove it but it’s simpler to just leave it there).

[4] In mantic, the nbd module moved back to linux-modules-raspi, so you can skip installing linux-modules-extra there if you want, but you’ll still need nbd-client. In noble, I’m intending to seed nbd-client in the image, so no modifications at all should be required. It’s a pretty small install (172KB) so it constitutes little bloat, and the ability to netboot out of the box (with an appropriately configured Pi) seems to justify the change to me.

[5] If you’re cunning, you can avoid downloading the image again by extracting it from the rpi-imager cache. On Linux at least, this is under ~/.cache/Raspberry Pi/Imager/lastdownload.cache. Because the filename has lost it’s extension I usually use file to determine what it was (e.g. if it’s “XZ compressed data, checksum CRC64” I know it was .img.xz).

[6]	Unfortunately, the initramfs isn’t intelligent enough to go searching for a label-based root in an NBD device, even though technically this is valid.

[7]	Ask me how I know this!

[8]	Before anyone asks, no `-o remount` isn’t sufficient. It needs to be completely unmounted and re-mounted.

NBD Knows

2023-10-27T00:00:00+01:00

This is part 1 of an exploration of netbooting Raspberry Pis, with an emphasis on NBD over the more traditional NFS. While obviously we’re going to be looking at Ubuntu in this series, I’m hoping this should be generic enough to be useful to be useful on a variety of distributions.

The trouble I’ve seen

The obvious question is “why NBD?” or put another way “what’s wrong with NFS?”. After all, NFS is the current “best practice” for booting Raspberry Pis, and widely used, including by some very large installations (e.g. Mythic Beasts’ fantastic Pi cloud).

Ultimately, the reason is that the Network File System … is a file-system.

What a ridiculous objection, of course it’s a file-system! The clue’s in the name! And surely you want a file-system? More typically, though, you actually have a root block device, which your kernel will then transform into a file-system, rather than being given a file-system directly.

What problems arise from the lack of (access to) an underlying block device? The crux of the issue is what the kernel can assume about the file-system, most particularly whether it can assume it has exclusive access to it, and whether it can trivially access the underlying blocks of certain files.

Consider the common “root is a block device” case:

The kernel on your machine runs the transformation (the file system driver) that converts that block device into a file-system. Crucially, it can assume that it is the only entity accessing this block device (that it has exclusive access), and that for certain operations it can (with some jiggery-pokery) bypass the file-system and treat a file as a block device [1], which makes certain things nice and simple.

Need to allocate a large contiguous file, for example as a swap file? No problem! The file-system driver implements this [2], and can even give the kernel the contiguous blocks for use in the swap system (remember how block devices “make storage look like RAM”? Foreshadowing!).

Need to lock a file? The kernel knows nothing else is mounting file systems from that block device, so it knows all the locks that exist on it and doesn’t need to coordinate with anything else. Likewise, caching is simple [3]. The kernel can assume it has absolute knowledge of which blocks are dirty and need writing back because nothing else can be producing file systems from that block device.

Need some temporary file space, private to your process? Create a unique temporary file and delete it, leaving the file handle open, then use the file as a temporary store. The space won’t be reclaimed because, even though there are no links to the file, the kernel knows there’s still an open file handle.

Now the “root is a file-system” case:

In this case, there’s a block device somewhere, but your kernel has no visibility of it, and must assume that other entities can change its blocks without notification.

This interferes with several of the scenarios above. Historically, it was fatal to several of them (fallocate for cheap allocation of large files, and swap over NFS did not work). Whilst these features do work today, it’s notable that kernel support needed to be added for swap [4], and the NFS protocol extended specifically for fallocate.

Caching becomes tricky because files are complicated and messy things that have data and “meta-data”. Here’s an excerpt from the NFS mount options:

ac / noac

Selects whether the client may cache file attributes. If neither option is specified (or if ac is specified), the client caches file attributes.

To improve performance, NFS clients cache file attributes. Every few seconds, an NFS client checks the server’s version of each file’s attributes for updates. Changes that occur on the server in those small intervals remain undetected until the client checks the server again. The noac option prevents clients from caching file attributes so that applications can more quickly detect file changes on the server.

Yikes.

The practice of using unlinked temporary files also works over NFS but again … required workarounds because NFS is stateless [1] so open file handles locally do not correspond to open file handles on the server.

All this doesn’t matter too much when the portion of the virtual file-system being mounted over NFS is relatively limited, as in the common case of mounting user home directories over NFS. However, when the entire root file-system is NFS, quite a few applications can start having “difficulty”. One of the more prominent, on Ubuntu especially, is snapd which doesn’t like running on an NFS root. Like it or not, it’s a pretty integral part of the Ubuntu eco-system at this point (providing Firefox on the desktop and LXD on the server), so it would be nice to have a netboot system that can support it too [5].

Okay, we’ve established there are some issues with running NFS as root. What’re the alternatives?

Sometimes I’m up

There are several daemons and protocols that support serving block devices over the network, and they differ in some quite interesting ways:

iSCSI

iSCSI can be trivially summarized as: SCSI commands over TCP/IP. This is by far the most popular method of serving block devices. It has the advantages that it doesn’t require expensive equipment (unlike Fibre Channel, its major competitor in the enterprise), and can be routed over multiple networks (as it’s built on IP). However, it’s not entirely trivial to configure (it is very flexible, but for our purposes here I wanted something simpler to start out with).

AoE

ATA over Ethernet. Like iSCSI, the name says it all. The client simply passes ATA disk commands over Ethernet to the server. Note that this does indeed run over Ethernet only (layer 2), not TCP/IP so it’s not routable over the Internet, only local networks. It’s very simple to set up (moreso than iSCSI), but two things made me skip it here. The first is that routing over layer 2 may be perfectly sufficient for many use-cases, but there’s several others where it’ll be a limiting factor.

The second is that aoetools, the package for AoE in Ubuntu, is an extremely “mature” package. Specifically, the upstream version in Ubuntu hasn’t changed since Xenial (16.04, 7 years ago at the time of writing). That isn’t to say it’s bad or entirely unmaintained, but there’s no active work on it as best as I can tell (and that usually doesn’t bode too well from a security point of view).

NBD

Network Block Devices differs in that doesn’t implement the commands from an existing disk protocol (SCSI or ATA). Instead, it uses its own protocol which, at it’s core, is almost laughably simple.

It also operates over TCP/IP, avoiding the layer 2 limitations of AoE, and is an actively maintained project which optionally includes facilities for TLS encryption. I won’t be using those here, but you’ll have some options for better security down the line.

Simplicity is another argument in favour of serving block devices instead of file-systems. Compare the two scenarios:

They don’t look that different but consider what serving files over a network means, versus serving block devices. What operations can be performed against a file? Opening, closing, reading, writing, truncating, locking, linking, touching, the list goes on. All this must be handled by the protocol to implement even a bare-bones network file-system. The bare-bones case for block devices (as noted above) is radically simpler.

The baseline portion of the NBD Protocol consists of commands to “read some bytes”, “write some bytes”, and “disconnect”. That’s it. There are some other commands which may optionally tell the server to trim, flush, or cache blocks, and some other messages for option negotiation at connection time, but the core of the protocol really is that simple. I like simple.

Sometimes I’m down

So far, we’ve looked at how the root file-system will be handled when netbooting. However, the root file-system only matters after the Linux kernel has started. How do we obtain the Linux kernel itself (and other sundry boot resources) at system start? This involves neither NFS nor NBD.

When netbooting, the Pi first requests an IPv4 address from the local network via DHCP. The local router responds with a DHCP offer, and our netboot server tacks a (minimal) PXE boot menu on the end suggesting where the client may find a TFTP server for booting.

This is typical jargon-laden nonsense, so let’s translate a bit:

Jargon	Raspberry Pi	Router	Netboot Server
DHCP DISCOVER	“Hello? Can anybody give me an IPv4 address? By the way, I’m a netboot client”
DHCP OFFER		“Sure, would you like to be 192.168.0.200?”
DHCP Proxy Option 43 PXE Boot Menu			“By the way, for ‘Raspberry Pi Boot’ see TFTP server at 192.168.0.4”
DHCP REQUEST	“Okay, I’d like to be 192.168.0.200, please?”
DHCP ACK		“Right, you are 192.168.0.200 for the next 12 hours, see me again after that”
TFTP RRQ	“Can you send me the content of `SERIAL/start.elf`, please?”
TFTP OACK			“Sure, it’s going to be 225065 bytes long, and I’ll send it in chunks of 1468 bytes”

The important things to note here are as follows:

We need a DHCP server. This is pretty much taken-as-read on any network these days.
We need a netboot server with a DHCP proxy and a TFTP server. This is fairly simple. Any Ubuntu server can install dnsmasq (if it hasn’t already) to obtain this.
We need the Raspberry Pi’s serial number.

This last point may seem a bit strange, but it’s because TFTP is, as the name suggests, trivial. The protocol provides no means for the client to identify itself to the server, so how are we to know which boot partition we should read files from?

Identification by MAC address is one possibility [6], but that’s not an option for us (unsupported by the TFTP server in dnsmasq). Instead we rely upon the Pi identifying itself by the sequence of files it initially attempts to request. When netbooting, a Pi (more specifically a Pi 4 or later) will attempt the following sequence of files [1]:

SERIAL/start4.elf, e.g. 1234abcd/start4.elf
SERIAL/start.elf, e.g. 1234abcd/start.elf
start.elf

If the bootloader finds files with the SERIAL/ [7] prefix, all subsequent requests will also have that prefix, allowing us to easily determine which OS image files should be served from.

The Pi then proceeds to request (over TFTP):

The rest of the tertiary bootloader (e.g. start4.elf, fixup4.dat, and its configuration files like config.txt).
The device-tree for the specific board (e.g. bcm2711-rpi-4-b.dtb) and any overlays required by the configuration, or by devices that are plugged in (e.g. overlays/dwc2.dtbo, overlays/ov5647.dtbo).
The kernel and initramfs requested by the configuration, and its command line (e.g. vmlinuz, initrd.img, cmdline.txt).

With all this loaded into appropriate locations in memory, the bootloader hands over to the Linux kernel, which mounts the initramfs as its initial root file-system, and launches the /init binary within it.

In the case of Ubuntu, this is the usual initramfs you’ll find on pretty much any Ubuntu installation. It’ll search the kernel command line for the “real” root device, attempt to mount it, and “pivot” the root that mount.

For the NFS case, the kernel command line would include something like nfsroot=server:/exports/ubuntu-jammy root=/dev/nfs. For the NBD case, the kernel command line would include something like nbdroot=server/ubuntu-jammy root=/dev/nbd0p2.

At this point you should have a basic understanding of the Pi’s netboot process. Let’s explore what the server side configuration for TFTP, NFS, and NBD can look like from a high level. We’ll get into specifics in the next post; this is just to give you an idea of the considerations and possibilities involved.

Glory, hallelujah!

A typical server TFTP configuration (whether subsequently NFS, NBD, or anything else) is to have the boot partition of an OS image unpacked or mounted under a particular path, and then make a symlink from the Raspberry Pi’s serial number to that path. Re-writing the symlink is then enough to switch which Pi boots which image.

While this much does not differ between the NFS and NBD cases, there is the question of how to make the boot files available.

In the case of NFS, as the server is serving a file-system it is typical to simply unpack the entire OS image, both the root and boot file-systems, into a directory and call that the “image” that is served. The symlink for the Pi’s serial number then points to the /boot/firmware directory within the unpacked image.

For example, if we have two OS images, ubuntu-jammy and ubuntu-mantic, and two Raspberry Pis with serial numbers 1234abcd and 4567cdef we might lay out our files like so:

/
├─ …
├─ srv/
│  ├─ ubuntu-jammy/
│  │  ├─ bin/
│  │  ├─ boot/
│  │  │  ├─ firmware/
│  │  │  └─ …
│  │  └─ …
│  ├─ ubuntu-mantic/
│  │  ├─ bin/
│  │  ├─ boot/
│  │  │  ├─ firmware/
│  │  │  └─ …
│  │  └─ …
│  └─ boot/
│     ├─ 1234abcd->/srv/ubuntu-jammy/boot/firmware
│     └─ 4567cdef->/srv/ubuntu-mantic/boot/firmware
└─ …

The two images are completely unpacked under /srv/ubuntu-jammy and /srv/ubuntu-mantic, then symlinks under /srv/boot point to the /boot/firmware directories of the unpacked images. Our TFTP server is configured to serve /srv/boot, and our NFS server to export /srv.

The advantages are that it’s a relatively simple setup, requiring no special mounts on the server side, and that (assuming all unpacked images are in the same file-system on the server), all available space is shared between all netbooting Pis. The disadvantage (other than it being an NFS boot setup) is that all available space is shared between all netbooting Pis so one Pi can use up all available space for everyone, unless additional steps like quotas are taken.

In the case of NBD, which we’ll explore more in the next post, I would suggest a simple setup is to leave the OS image as it is (in a file) and simply expand the file to the desired size. A loop device can be created for the image file, with a partition scan to find the boot partition, which can then be mounted. I would still recommend using a symlink to point to the mount for ease of changing later.

Let’s consider the scenario from before (two images, jammy and mantic, two Pis with known serial numbers) in this setup:

/
├─ …
├─ srv/
│  ├─ ubuntu-jammy.img
│  ├─ ubuntu-mantic.img
│  ├─ mnt/
│  │  ├─ ubuntu-jammy/  (mount of ubuntu-jammy.img partition 1)
│  │  └─ ubuntu-mantic/  (mount of ubuntu-mantic.img partition 1)
│  └─ boot/
│     ├─ 1234abcd->/srv/mnt/ubuntu-jammy
│     └─ 4567cdef->/srv/mnt/ubuntu-mantic
└─ …

The two images are simply placed under /srv. Loop devices are created of each, and the first partition mounted under appropriate directories under /srv/mnt. Symlinks under /srv/boot link Pi serial numbers to the appropriate mount. Our TFTP server is configured to serve /srv/boot as before, and our NBD server is configured to export each /srv/*.img file as a block device.

The advantages are that this is a trivial setup (the image doesn’t even need unpacking), and that each image is necessarily limited to its own storage. The disadvantage is that storage isn’t shared between images. However, we saw in the prior post that there’s all manner of things we can do with block devices, so we’ll explore some possibilities there in a future post too.

I should warn there’s also a nicely hidden, but quite serious issue with this set up which we’ll look at next time when we actually build it (and then fix it … obviously). Anyway, that’s all for now!

[1]	(1, 2, 3) This is a gross over-simplification (or an outright lie), but serves to make the point.

[2]	Alright, some of them don’t or only recently added the functionality, or it’s still experimental (cough btrfs), but they’re usually the file-systems that have their own rules for block mapping.

[3]	Caching is never simple.

[4]	No other file system has a “make swap work on this file-system” kernel config item

[5]	I’m told Docker overlays also have issues with NFS but it’s not something I’ve played with directly and I’ve not had the time to verify it that for this article.

[6]	Though not for “security” (MACs can be trivially spoofed)

[7]	The `SERIAL` portion is the serial number of the Pi (which can be found at the end of the output of `cat /proc/cpuinfo`), in lower-case hexidecimal format. If the serial number is longer than 8 characters, only the last 8 characters are used.

Building Blocks

2023-10-24T00:00:00+01:00

In an earlier series of posts (Playing with Blocks 1, 2, 3, and 4) we had a look at customizing Ubuntu images for the Raspberry Pi by playing around with various block device transformations. However, we never covered what a block device actually is. This is going to be a vaguely important topic in an upcoming series of posts on NBD booting of Ubuntu, so I figured some brief coverage of the topic is warranted.

Brief.

Who am I kidding?

Blockage

In the UNIX world a block device is a very simple concept: it’s a storage device which provides access to its contents as an ordered sequence of identically sized blocks. That’s it. No files, no directories, no permissions, nothing more than a numbered sequence of blocks all with the same size. Typically, the blocks are some whole multiple of an underlying device’s sector size, which over most of computing history has been 512 bytes. However, the user may access block devices byte-by-byte if they wish:

For instance, a read of a few bytes (illustrated on the left in green) may read a full 512-byte sector, throw away the bytes outside the requested range, and return that which was requested [1]. A write of a few bytes (illustrated in the middle in red) is a bit more complex; the kernel needs to read the full sector from the underlying device, change those bytes we requested to write in the middle, then write the whole sector back to the underlying device [2].

Crucially, block devices are random access. We can access any block, anywhere, at any time, without having to access all blocks prior to it. In other words, block devices make storage look and “feel” like RAM. This is as opposed to the “byte-stream” view of files which owes more to the tape drives of old (“seeking” the head to a particular location, then reading or writing from that point).

Transform!

The simplicity of block devices leads to them being wonderfully flexible things that can often be converted into new block devices in useful ways. Probably the most basic block device transformation that everyone is familiar with is “partitioning”, which simply sub-divides one block device into multiple new ones (minus some header space). This transformation effectively just re-numbers the blocks but doesn’t change their content or ordering.

More complex transformations are also available.

The LVM sub-system (explored previously) gathers multiple block devices into a “volume group”. The volume group can then be used to produce other block devices called “logical volumes”. These are similar to partitions, but have no need to be contiguous on the underlying block devices, and can even span multiple underlying devices.

A drive (sda) is divided into two partitions (sda1, sda2) the latter of which is further divided into three block devices with LVM (root, home, and swap). The home device has been expanded at some point in its history as its storage is non-contiguous

Speaking of multiple underlying devices, RAID systems (like mdadm) consume multiple block devices and produce a single block device. The blocks in the produced device are duplicated to the underlying block devices (in the simplest mirror case; more complex transforms are involved in things like RAID5). Corruption or even loss of one of the underlying block devices, as in the failure of a drive, can be entirely hidden from the consumer of the produced block device.

Four drives (sda, sdb, sdc, and sdd) are partitioned such that each has a small initial partition (sda1 etc.) and a larger partition (sda2 etc.); the small initial partitions are combined as a RAID1 mirror into a boot block device. The larger partitions are combined as a RAID10 device (a stripe of mirrors) as a root device.

The LUKS encryption mechanism (also explored in previous posts) consumes one block device and produces a slightly smaller one (minus the size of the LUKS header) that scrambles (and unscrambles) the content of blocks as they are written (and read).

The Hierarchy

A file-system is just another kind of transformation of a block device; one which converts the flat sequence of blocks into a tree-like hierarchy of files (optionally with attributes like ownership, groups, permissions, and all the rest of that gubbins).

File systems can be relatively simple affairs, like the ancient but ubiquitous FAT which has no concept of file ownership, UNIX modes, and no symlinks. Alternatively they can be complex, like the common ext4 file-system which has journaling data recovery, the full suite of UNIX ownership, attributes, and linkage, and numerous tunable options.

Ultimately, the file-systems produced from various block devices are all grafted together (“mounted” in UNIX parlance) onto a “virtual” file-system which is what all userland processes (and you, the user) generally consider “the file-system”; the place you find all the files on your computer.

Two partitions (sda1 and sda2) are formatted as FAT and ext4 respectively. The vfat driver converts the sda1 block device into a typical boot file-system for the Pi (containing bootcode.bin, start4.elf, config.txt, etc). The ext4 driver converts the sda2 block device into a typical Linux root file-system (containing bin, etc, lib, usr, sbin directories). The virtual file-system then “mounts” the ext4 hierarchy under / and the FAT hierarchy under /boot/firmware

A common Linux storage layout is shown above. The drive (/dev/sda) has two partitions. The first partition (/dev/sda1) is formatted as FAT, which the kernel’s vfat driver transforms into a file-system. The second partition (/dev/sda2) is formatted as ext4, which the ext4 driver transforms into another file-system. The kernel has mounted the ext4 file-system as the root of the “virtual” file-system, and the FAT file-system under the /boot/firmware mount-point.

Holy CoW!

Some modern file-systems provide a copy-on-write facility (most notably btrfs and zfs). This allows rapid (near instantaneous) cloning of an entire file-system hierarchy. The clone doesn’t initially take any storage of its own, and reads of the clone actually go to the original files. However, writes to the clone allocate new storage for the changed content.

However, the facility is not limited to modern file-systems. LVM also provides such a facility, initially intended to provide snapshots of file-systems at a point in time (for backup, testing, or other purposes).

The “root” block device contains a number of filled blocks in green, while the “snap” block device has an equivalent number of “empty” blocks. Two reads, indicated by arrows, are made against the “snap” device but pass through to the underlying “root” device. Below this an equivalent layout is show, but now one of the “filled” blocks is “red” indicating a change in content. Above it, the equivalent block in the “snap” device is green indicating it contains the original content. A read against this block comes directly from the “snap” device instead of passing through to the underlying “root” device.

The illustration above shows a “root” block device, presumably containing a root file-system. We create a snapshot of the “root” block device called “snap”. Initially, the “snap” block device is empty, and reads to it pass through to the underlying “root” block device. However, when a block is the “root” device is written to, its original content is first copied to the “snap” device. Subsequent reads of this block in the “snap” device will go to this copied block instead. Further writes to the same block in the “root” device are ignored because the “snap” device only cares about the content at the point in time it was created.

Note that one of the beauties of this system is that we don’t care what file-system is on the “root” device. It could be FAT, ext4, XFS, even (redundantly) one of the more modern systems that supports this internally. However, a limitation of this snapshot system is that we need to allocate space for it up front [4]. Can we come up with something in which the snapshot allocates blocks dynamically?

At the top, the “image” block device (read-only) contains a number of filled blocks in green. Above it, a clone, “clone1”, exists with the same number of blocks but all are empty. A read of two blocks in “clone1”, indicated by arrows, passes through to the corresponding blocks in “image”. Below, the one of the blocks in “clone1” is now red indicating it has been changed from the original content in “image”. Another arrow, indicating a read of that block now comes straight from “clone1” and doesn’t pass through. Another clone, “clone2”, is below it with entirely empty blocks. A read of the same block which is changed in “clone1”, passes through “clone2” to the underlying “image” device.

The illustration above shows “thinly provisioned” snapshots in LVM. This operates a little like the regular snapshots in reverse. An underlying block device called “image” presumably contains an OS image. We have created a “thinly provisioned” snapshot of this called “clone1”. Reads of an unchanged block pass through to the underlying (read-only) “image”. Writes to blocks in “clone1” occur within its storage, with subsequent reads to that block no longer passing through. Finally, we create another clone, “clone2”, which reads the original block from the underlying “image”.

It’s worth noting that most of the blocks in the clones are empty. Provided they stay that way (i.e. the clones don’t change too many blocks of the underlying image), there’s little sense in actually allocating them. This is the “thin” in “thinly provisioned”. The blocks of the clones are allocated on demand so a clone initially takes up no space. This allocate-on-demand facility can implement one form of “over provisioning” of storage. Each clone thinks it has all the storage allocated to the original image, but there’s no need to have all the storage available (unless the clones grow to require it, naturally).

Hopefully this gives some idea of how modern cloud systems can spin up clones of an OS image nearly instantaneously, and how they can seemingly provide many gigabytes of storage to containers or VMs, without necessarily having installed all the storage they apparently provide.

Layer Cake

Finally, a brief illustration of just how silly this can get. Here’s a diagram of a storage layout I’ve used on servers in the past:

A spiders web of storage. Four boxes represent four hard at the bottom. Above this a larger box represents the RAID device built from the drives. Above this a hexagonal volume group is built from the large RAID device. From the volume group spring numerous block devices, for “tmp”, “root”, “home”, and “images”. From the first three spring regular file-systems, but from “images” spring several more block devices named “base” and from that “munin”, “docs”, and “git” (OS images running in containers).

Four hard drives exist, each containing a small boot partition and a large partition occupying the rest of the capacity
The four boot partitions are combined with a RAID1 mirror into a boot block device, “md1” [3].
The “md1” block device is formatted as FAT and contains the “/boot” hierarchy.
The four large partitions are combined with RAID5 into a large block device, “md2” (RAID5 implies it has the capacity of n-1, i.e. three, of the underlying devices, and can survive the loss of any one device).
The “md2” device is consumed by the “raidvg” volume group from which are produced various block devices: the “root”, “home”, “tmp”, and “images” volumes.
“root”, “home”, and “tmp” are formatted with a mix of file-systems, ext4 and XFS.
The “images” volume is a “thin pool” from which the “base” volume is derived.
The “munin”, “docs”, and “git” volumes are all thinly provisioned snapshots of the “base” volume.

This may seem like a complicated stack of transformations, but consider that in essence most amount to little more than re-numbering and re-ordering of blocks (the RAID5 transform is a little more involved, admittedly). Hence, for very little performance cost, we’ve got redundancy (RAID), flexible volume creation system with snapshotting (LVM), and copy-on-write cloning for VMs or containers launched on the server (thin provisioning).

By this point you should have a reasonable understanding of what a block device is, and what can be accomplished with the various transforms that are available for them under Linux. Next time we’ll take a look at the issues with netbooting from NFS and how NBD (or block devices in general) can mitigate some of these issues.

[1]	In practice it’s unlikely the bytes will actually be thrown away; the sector requested will be stored in a cache, in case a future read requests more bytes from the same sector (very likely when reading through a file sequentially).

[2]	Again, the changed block will wind up in a cache so future modifications can skip the initial read step.

[3]

Simple RAID1 mirroring is used to ensure that the BIOS can use any of the drives as its initial boot device without caring about the RAID layout (which it doesn’t understand). Note: This carries risks as it means the BIOS and bootloader must treat the boot partition as strictly read-only (and that’s not necessarily guaranteed).

[4]	The snapshot doesn’t require all the space to be allocated, as the diagram suggests. In fact, assuming the underlying device changes infrequently, it’s common to only allocate ~5% of the origin’s storage. However, you do still need to provide some up front.

Making Mantic Magic

2023-10-12T00:00:00+01:00

Meet the Minotaur!

It’s around this time of year that a new Ubuntu release comes out (including images for the Raspberry Pi), and shortly afterwards I pen something on the subject of how best to tweak it to get the most out of it on your Pi. Past subjects have included adding a swap file … but that’s now done out of the box. Or enabling the “zswap” facility … but that’s now on by default too. Erm … let’s see, how to flash the image to an SSD? Nope, that’s as easy as an SD card these days, thanks to rpi-imager. Disabling multipath to gain a bit more RAM on the Pi Zero? Nope, default now too.

The list is indeed shrinking fast, but it’s not completely empty yet and the arrival of the Pi 5 has certainly left a little opportunity for post-release tweakage!

Speaking of the 5 …

Presto!

What’s it like? How is the Pi 5, and how is Ubuntu on the Pi 5?

Simply put, it’s an awesome piece of Pi. I’ll go so far as to say it’s the first Pi where, if your computing needs are fairly typical, it will meet them as well as any cheap PC.

I use Pis for pretty much all my work. All my coding, packaging, emailing, talking to colleagues online, and much of my browsing is done on a Pi 4. But there it is: “much of”. Not “all of”. Playing videos on Youtube? Painful. A video call involving a dozen colleagues? Out of the question. For those I turn to a PC. Often at home, the honking huge thing which doubles as a gaming platform, but out on the road my little laptop does these duties.

When I call my laptop “little”, I’m not kidding. I like my diminutive, and above all cheap, computers. I really don’t like lugging around a thousand quid’s worth of easily breakable or nickable gear. So my current laptop is an Acer Travelmate Spin B1 which has these positively mind-blowing specs:

Component	Specification
CPU	Intel Pentium N4200 (ooooh!)
Graphics	Integrated Intel Graphics
RAM	4GB (yes, my Pi has more)
Storage	64GB eMMC (again, the Pi has a bigger, faster SSD)
Display	11.6” 1080p (you guessed it, my Pi has a bigger monitor)

Yet, for all its diminutive capacity, it still breezes past my main Pi 4 at boot time, and is an eminently more capable desktop machine [1]. In fact, I run two Pi 4s for my work; one runs the Ubuntu desktop and the other a customized Ubuntu server image with kmscon as the console. The vast majority of my work (including all email) is done on the latter because the console is so much faster than the desktop.

So … enough waffling and delay. What about the Pi 5?

It beats the laptop. Hands down. I know my laptop is nothing to write home about, but this Pi is, being the first Pi to roundly beat one of my PCs at being a “good desktop”. It boots faster (about 20 seconds from cold for Ubuntu; we’re not at the blazing 7 seconds that RaspiOS manages, but Ubuntu’s a rather beefier desktop environment). It runs faster (everything launches faster, and generally “feels” faster than on the laptop). It’s considerably more expandable (remember the storage on my laptop is a soldered-on eMMC module!). And yet, it’s cheaper.

The “cheaper” comparison obviously sounds unfair; the laptop includes a keyboard, touchpad, battery, screen and so on, and moreover it’s older technology (newer technology is typically cheaper per unit of performance than older). When I bought it refurbished several years ago, it still cost about 300 quid, so let’s tot up my Raspberry Pi equivalent:

The Pi 5 I’ve been testing on is the 8GB model (kindly provided a couple of months ago by the folk at Raspberry Pi), which retails for 80 quid
Add in a good power supply (12 quid)
Reasonable keyboard (20 quid [2])
Mouse (10 quid)
Monitor (80 quid for a 24” FHD)
Good SD card (15 quid for a 128GB SanDisk)

217 of your pounds sterling. Yes, definitely “cheaper”, and recall we’re comparing to a refurbished laptop, not a new one. It’s not as portable as the laptop, but 83 quid probably buys a fair sized power-bank! Alright, that’s still not a fair comparison but I’m going to make it anyway because my Pis really do accompany me on the road :-)

Dave’s “portable” setup.

Abracadabra!

What should you do to get the most out of Ubuntu on your Pi 5? Honestly, not a great deal. Anyone who’s used Ubuntu desktop on the Pi 4 extensively will be yelling things like “active cooling!” and “SSD boot for performance!” at the screen, but frankly these are optional on the 5 and absolutely unnecessary to have a system that works “well”.

The pre-release Pi 5 sent to me a few months back (thanks Gordon!) didn’t include the fancy new cooling fan [3]. The board does get warm, particularly under load, and it’s not too difficult to get it to throttle (watching a video fullscreen in Firefox will do the job). But do I notice it throttling? No! And once the video finishes (or the package finishes building, or what have you) and I go back to doing less computationally intensive things, the board’s back to an unthrottled temperature within a couple of seconds … without any active cooling.

The other thing the Pi 5 didn’t arrive with was the official power supply [4]. This is a rather more important piece as, in order to reach its full potential, the Pi 5 does have rather higher power demands. Specifically, if you want the USB ports to deliver enough juice to run external storage (and thus to boot from an SSD drive), you need a supply capable of delivering 5 amps at 5 volts. The vast majority of USB-PD compatible supplies don’t manage this (certainly none of the ones I owned could). There are a few third-party power supplies that do (for example, the Radxa Power PD 30W) but my recommendation to most would be to simply buy a Raspberry Pi supply.

I’ll write a bit more about this in a future article but an important aspect here is that the official supply has a “captive cable”. This means that at the “business end” (that sticks into your Pi), it will deliver 5V. If your supply requires an external cable, it’ll presumably deliver its rated 5V at its socket, but that doesn’t guarantee it’ll still be 5V at the business end of the cable you’re inserting. Furthermore, to carry 5A, USB cables need an e-marker chip.

Anyway, suffice it to say, I’m still running my 5 off my old 3A capable supply (an official power supply is on order!), and even though it may not power anything beefy off the USB ports, it’s still managed everything else without any brown-outs.

Cool as a very odd cucumber.

What about storage? Much the same applies. Yes, I run my Pi 4 on an SSD, and for doing serious work on the Pi 5 I would do exactly the same. But whereas on the Pi 4 this was about gaining capacity and performance, on the Pi 5 it’s really just down to extra capacity (and possibly reliability). The SD card interface is so much faster on the 5 (and the IO bandwidth to other components like memory so much bigger), that there really isn’t a hard requirement to have an SSD for good desktop performance. A decent SD card is perfectly sufficient.

In summary, a throttled, under-powered, SD-card driven Pi 5 is still a considerably more comfortable and responsive machine than an unthrottled, fully-powered, SSD-booted Pi 4. And it’s still capable of beating my laptop!

Ala Peanut Butter Sandwiches?

So … there’s nothing to do? No hardware to add, no software to tinker with to get the most out of Ubuntu on your shiny new Pi? Perish the thought! Where would we be without the need of a bit of tinkering? [5]

There’s a few major things that we couldn’t get into the release in time. The first is the upstream patches for libcamera (this is the reason for the camera module plugged into the board in the shot above — still testing!). This is intended to be delivered as an SRU in Mantic, which isn’t too bad given that libcamera isn’t included in the base image so users need to download it anyway.

The second is the mesa patches for full graphical performance. These did make it into the base image, so most applications on the desktop do indeed get the full benefit of the new Vulkan-compliant drivers. But some desktop applications (and one rather notable one in particular) don’t use the system’s mesa drivers because they’re shipped as snaps, so they use the libraries included in the corresponding content or base snaps (most of which are still from the 22.04 era).

Obviously this includes Firefox.

He’s not helping.

For the most part, this doesn’t matter too much. Browsing the web and even playing video is still fine because the CPU’s capable of just powering through (the Pi 5 really does have a beast of a processor!). However, fire up the WebGL aquarium sample and it becomes blindingly obvious. It’ll barely manage a paltry 2fps, if you’re lucky. Replace the Firefox snap with a deb so it’s using the system mesa libraries (I won’t go over the instructions again here, but you can derive them from this post if you really want to) and all of a sudden that jumps up to a much more respectable 40fps or so (at least on my 1080p monitor).

Naturally, the plan is to include the updated mesa drivers in the base snaps as soon as we’re able after launch, so hopefully this should be a short-lived state of affairs. Also, as noted, this doesn’t appear to affect the vast majority of web browsing, only rather niche things like WebGL.

The other missing bit is a power monitoring daemon, which is something I’ll be prioritising in the next cycle. The new power management chip on the Pi 5 allows the OS to determine whether the previous reboot was due to a power brownout. This is fantastic as brownout issues are notoriously variable and difficult to diagnose, so having the hardware tell you that this was definitely the cause, should deal with a whole pile of Heisenbugs.

Finally, the other major change with the Pi 5 is the means by which GPIO pins are controlled. No longer is the SoC itself driving the GPIOs. The new RP1 “southbridge” is now in charge instead. This means that the traditional GPIO libraries like RPi.GPIO no longer work. I’ve written about this previously and thankfully Ubuntu’s pretty well placed to weather the storm here. We switched the GPIO Zero back-end over back in hirsute (21.04) and added rpi-lgpio (a RPi.GPIO compatible shim) back in kinetic (22.10) to ease the transition.

These pieces needed a few updates [6] to deal with the Pi 5’s new hardware, but that’s all included in mantic. With a bit of luck most users either won’t notice the change, or won’t have much difficulty in getting their circuits to work on their shiny new Pi 5.

That’s all for now. Go grab your copy of Mantic Minotaur (it should be in rpi-imager by the time I publish this) and have fun!

[1]	Running Windows on it was as awful as you might imagine, but Ubuntu is still light enough to be perfectly useable on it

[2]	At this point, anybody who knows me is spluttering their tea over the screen; “twenty quid? Sure you’re not missing a zero there, Dave?”

[3]	There weren’t enough to give one to each alpha tester at that point, and it wasn’t necessary for Ubuntu’s compatibility work

[4]

Again, there weren’t enough to go around. However, I should add that they did very kindly (and usefully!) ship a Pico Probe with the board which was really useful in debugging things, especially since it attaches to the separate debug port now, meaning I didn’t have to unplug things from the GPIO pins just to get a serial console

[5]	That’s rhetorical :)

[6]	Try `pinout` on the new Pi 5!

Dealing the (SD) Cards

2023-09-12T00:00:00+01:00

Updated to remove large outlet recommendations, and fix footnote

It came to my attention recently that many people don’t quite know what to look for in a micro-SD card for use in a Pi. As mercilessly torturing micro-SD cards is something of a hobby round here I thought I’d write a quick guide to what to look for.

A small selection of the SD cards that have graced my desk over the years, ranging from the rubbish to the really rather good.

Vegas Standard

There have been many standards for SD cards to conform to over the years, but the vast majority of them are concerned with sequential transfer rates. These started with the so-called speed classes, where Class 4 implied that a card could sustain a write speed of 4MB/sec to the card, Class 6 implied 6MB/sec and so on.

Later there were the UHS speed classes, where U1 was equivalent to the older Class-10, a sustained write speed of 10MB/sec to the card. U3 implied 30MB/s sustained write speed. Why a new class system? Because UHS also required a specific bus speed (confusingly designated UHS-I, then UHS-II, etc). Hence, a U1 card must manage 10MB/sec minimum but the corresponding bus speed could top out at 104MB/s allowing considerably faster read speeds (depending on bus compatibility).

Later still came the video speed classes. These still use the UHS bus interfaces, but demand higher minimum sequential write speeds. V10 requires 10MB/s minimum, V60 requires 60MB/s minimum and so on. Here’s a nice little table outlining the minimums required by each of these classes.

Minimum write performance	Speed Class	UHS Speed Class	Video Speed Class
2MB/s	The SD Class 2 logo, a “2” within a stylized “C”	—	—
4MB/s	The SD Class 4 logo, a “4” within a stylized “C”	—	—
6MB/s	The SD Class 6 logo, a “6” within a stylized “C”	—	The SD video 6 logo, a stylized “V” almost like a square root symbol followed by a “6”
10MB/s	The SD Class 10 logo, a “10” within a stylized “C”	The SD UHS 1 logo, a “1” within a stylized “U”	The SD video 10 logo, a stylized “V” almost like a square root symbol followed by a “10”
30MB/s	〃	The SD UHS 3 logo, a “3” within a stylized “U”	The SD video 30 logo, a stylized “V” almost like a square root symbol followed by a “30”
60MB/s	〃	〃	The SD video 60 logo, a stylized “V” almost like a square root symbol followed by a “60”
90MB/s	〃	〃	The SD video 90 logo, a stylized “V” almost like a square root symbol followed by a “90”

Got all that? Good. Now you can forget it.

As I mentioned above all of this is to do with sequential access. This is very useful in a camera where you’re recording video frame after video frame (or still image cameras that do … erm … exactly the same thing only much slower with considerably larger frames!). However, this doesn’t remotely resemble the access pattern of an operating system on an SD card which will typically entail hundreds of small writes to small files spread across a multitude of sectors.

In other words these classes will tell you next to nothing about the performance of your card in a Raspberry Pi. For that you need guarantees about the minimum random access performance. Luckily, the recent SD specifications have a standard for that too: the application performance classes.

Poker Face

At the time of writing, two application classes are defined: A1 which is fairly common on decent cards, and A2 which is still somewhat “premium”. Their minimum guarantees are as follows:

Minimum read performance	Minimum write performance	Application class
1500 IOPS	500 IOPS	The SD video application-performance logo, a stylized “A” followed by a “1” with “app performance” in capitals below
4000 IOPS	2000 IOPS	The SD video application-performance logo, a stylized “A” followed by a “2” with “app performance” in capitals below

Note that these guarantee both write and read minimums. Further, they’re not concerned with bytes transferred, but with IOPS, or I/O operations per second. The exact meaning of IOPS is rather loose across storage generally but here it means the number of block sized (4KB) read or write commands that can be carried out per second. Even this is relatively fuzzy because the controller in an SD card is free to batch operations together. Erasure is almost always batched into “extents” of blocks, and writes can be generally too.

In addition to the above requirements, A1 and A2 both require a minimum of 10MB/s sequential write speed (equivalent to the old Class 10 cards). This may appear excessive. After all, if A1 only demands that a card handle 500x 4KB writes per second, that only equates to 2MB/s? However, remember that cards are free to batch operations together. Cards that do not conform to one of the application classes may well display 10MB/s sequential writes to one large file by batching together smaller writes that are all in sequence. However, if asked to perform many small 4KB/s writes to a multitude of blocks, the performance suddenly drops off a cliff.

In my experience, any reputable card with an A1 mark performs “well” on a Pi. A2 cards from a given vendor perform better than A1 cards from that same vendor, but I’ve also had A1 cards from vendor A beat A2 cards from vendor B. But when I say “beats”, I mean in performance benchmarks. I cannot say I ever noticed a difference in daily usage.

To put it another way: A1 is the sweet spot right now, and A2 is probably a waste of money at this point unless you’re quite sure you absolutely need the performance.

Shooting Craps

The astute reader may have noticed my careful use of the word “reputable” just before “card” in the prior section. Fakes are a problem, or have been in my experience (though we’re going back several years here).

Buy from well-known outlets, not minor vendors on whatever variant of flea-bay you frequent [1]. Just because it’s on Amazon, doesn’t mean it’s not a knock-off; check the actual seller.

Cheap, bulk deals are especially to be avoided. There’s enough competition in the micro-SD card market that there’s usually little that can be (legitimately) saved by buying in bulk; if the unit price is substantially below what you could get the individual cards for, be wary!

To summarize:

Buy well-known brands,
from reputable vendors,
avoiding bulk (or suspiciously cheap) deals.
Look for A1 or A2 on the casing [3]. Any modern card without those marks is likely to be rubbish for use in an SBC like a Raspberry Pi.

The Long Game

I haven’t had any issues for the last few years, but during them I’ve stuck to buying strictly from either SanDisk or Samsung (also had a Toshiba card that lasted many years without issue, but it’s a singleton sample), and avoiding any cheap-looking bulk deals.

This is not to say these cards last forever. But I have several Pis that run 24x7, and “good” cards seem quite capable of lasting at least 5 years in them, which is “good enough” for my purposes (I’ve had hard drives last less than that before!).

The failure modes of micro-SD cards seem to fall into two camps:

completely dead — the card can be detected in a reader but fails to mount at all
going “read-only” — the card will happily mount, and read pretty much anything but all writes are silently thrown away

The latter sounds preferable, and in some ways it is because if you have any data you want to recover from it, you can do so. But it’s also worse in certain circumstances.

Consider a board where the card “goes read-only”. It may appear to continue operating quite happily (if it’s serving a static site for instance), but all writes to logs are being thrown away. It’s also quite confusing to diagnose if you’re not specifically looking for it because the card may appear to be operating normally.

When dealing with a card I know to be old one of the first things I’ll try (assuming I can mount it at all), is touch foo, remove and re-mount the card, followed by ls just to see if an empty “foo” file shows up. If it doesn’t, the card’s almost certainly gone read-only and it’s time to make a fresh one.

[1]	eBay, Amazon, Alibaba, whoever. If it’s a large marketplace, I’ve had fakes from it (certainly from the aforementioned big three). Ideally, buy from your local electronics vendor (Pimoroni [2] or The Pi Hut for me) but failing that, look for the larger vendors on the big marketplaces.

[2]	Unfortunately, at the time of writing, Pimoroni’s SD card offering doesn’t include shots of the front of the card. Being a reputable company, I’d be extremely surprised if they weren’t good quality A1 cards, but it’d be nice to see this at least suggested visually! Oooh, they’re A2 cards!

[3]	Assuming you can make it out from the cacophany of logos. Is it just me, or are micro-SD card cases beginning to look like the side of a Formula 1 car?

The Ubuntu Desktop

2023-08-09T00:00:00+01:00

See Also: The intro post which links to all the other flavour posts.

In this post, we’ll be comparing the official Ubuntu desktop release with all the flavours we’ve been baking from cloud-init configurations, on a Raspberry Pi.

Contents

Ubuntu
Sort it out!
Popping and locking
Verdict

Ubuntu

Finally we come to the comparison with the official desktop. Obviously I don’t need to include any cloud-init bits here because … you just install the desktop image on your card and that’s it.

The official desktop has a few other simplicity tricks up its sleeve too: no need to explicitly select a Wayland-based desktop, you get one by default. The first-time setup wizard also handles configuring things like the keyboard layout and initial user without having to deal with YAML. So … points for simplicity?

The default applications are a mix of the brilliant (the aforementioned Rhythmbox) and the … not so brilliant. In the screenshot above you can see Totem failing to play anything (LP: #1998782 is the Ubuntu bug, but the upstream bug has more details), so it was back to sudo apt install vlc. The incorrect audio selection bug (LP: #1993347) also popped up but is trivial as ever to deal with.

Everything else is mostly fine: good terminal, fine web-browser, stuff opens happily even on unmounted network shares, though (as noted before) dragging files from an archive to extract them doesn’t work (LP: #1923033). Though this is ultimately due to the Wayland transition, it’s notable that it does work in Kubuntu, even when running Plasma under Wayland.

One other difference worth noting is that on this distribution I stuck with using the snap version of Firefox because we may as well do a comparison of stuff “as it ships” here (which we can’t really do with the other flavours). Despite being a snap, the Wayland support works (once the MOZ_ENABLE_WAYLAND=1 setting is exported), and since LP: #1974064 was fixed [1] opening links from a standalone PDF reader works too.

Speaking of Wayland, all the desktop animations are generally smooth and don’t lag. With the exception of the start menu.

Oh dear, it’s rant time …

Sort it out!

The full-screen zoom-out animation is just too large to do smoothly even under Wayland (sometimes it manages it after a few attempts). However, this doesn’t bother me too much because, once it’s deigned to appear, I can never find stuff in it anyway.

Some flavours have a categorical start menu: Lubuntu, Xubuntu, Kubuntu, and Studio. It’s a bit dated, but it works: find the category you want, scroll through the alphabetical list, and pick the application you want. Others take a more smart-phone-y approach and just have a big alphabetical list of everything installed to scroll through: Budgie, Unity … GNOME? GNOME has one big list. But it’s not alphabetical:

It’s not in most-recently-used order either. I have fired up Videos here, but not GVim for instance. Maybe something weird like installation order? Let’s install something and see. I’ve picked GIMP which, after installation, appears at the end of the list on the second screen, after all the other apparently alphabetically arranged items:

But, if we logout and login again, our start menu has re-arranged and now GIMP is before GVim, in apparently alphabetical order. But no! “Cheese” is now before “Additional Drivers” although prior to the reboot it came later in the list!

Okay, I give up. I’ve no idea what the order is here. Maybe it’s “whatever you want it to be”? You can re-order things manually by dragging them around, or drag them over each other to form groups (such as the pre-grouped “Utilities”). But this means that if I want a sensible layout (which the default … isn’t), I have to laboriously drag things around until they’re “just so”. I have to put work in (something no other flavour has demanded). Or if you just like them alphabetical then simply … erm … install an extension?

In essence the starter is a strange mash-up of “smart-phone home screen” and “smart-phone all-applications list”, made all the more smart-phone-y by scrolling sideways because … well, just because. The thing is … this isn’t a bug. This is genuinely GNOME’s idea of “how a start menu should work”!

The mitigation to this is that most flavours these days, including GNOME (but even some which favour a categorised menu like Kubuntu) allow you to search by just opening the start menu, typing a few letters of what you want, then hitting Enter when it’s the first in the list. This is how I tend to use the start menu on GNOME because it’s the only way it makes any sense. But, for this purpose I don’t care about the icons, and that in turns means I don’t care about giant full-screen zooming animations that lag the entire menu.

Okay, rant over.

Popping and locking

Oh, and the screen-saver? Blanks and locks the screen, but fails to suspend the monitor. I’ve dug into this before (LP: #1998716) and all I came away with was a new-found appreciation for how bloody complicated something as apparently simple as a screen-saver is, once you begin to understand all that it does.

Verdict

It’s not lightweight, but it is pretty, reasonably responsive, with a decent selection of apps, and despite a few Wayland transition issues (like drag and drop) it’s mostly functional … except for that start menu. Still, it can’t be that bad as it has been my main desktop for the last few years.

The problem is … now I’ve used Kubuntu …

[1]	during the writing of this series in fact — this is another bit I’ve had to go back and re-write!

The Ubuntu Studio

2023-08-08T00:00:00+01:00

See Also: The intro post which links to all the other flavour posts. Skip down here for the configurations used.

In this post, we’ll be looking at creating an Ubuntu Studio desktop with a cloud-init configuration, on a Raspberry Pi.

Contents

Ubuntu Studio
Verdict
Configuration

Ubuntu Studio

Originally, this was not a flavour I was planning to cover. After all Ubuntu Studio is intended for multimedia creation which is some of the heaviest-duty work you can put any computer through. However, I was also aware they’d recently switched to Plasma (from KDE) as a basis, and that meant Wayland was supported. After Kubuntu’s striking performance, I was intrigued to see how (or if!) it’d work on a Pi.

For Ubuntu Studio, the packages section in user-data contains an apparent multitude of packages:

packages:
  - ubuntustudio-desktop
  - ubuntustudio-audio
  - ubuntustudio-graphics
  - ubuntustudio-photography
  - ubuntustudio-video
  - ubuntustudio-publishing
  - ubuntu-raspi-settings-desktop
  - firefox

The various ubuntustudio-* packages other than ubuntustudio-desktop are all optional depending on what you want to use. For example, if you’re only interested in audio work, leave out the graphics, photography, video, and publishing meta-packages. I would recommend leaving out video because doing video-editing on a Pi is … “optimistic” to say the least. However, one oddity of Ubuntu Studio is that it does “assume” a certain set of installed packages in the default quick launcher. If you exclude certain meta-packages you may find a number of blank “?” icons in the quick launch bar because of “missing” applications. You can un-pin these items without consequence.

One other note on default applications: Ubuntu Studio typically ships with FreeShow (which seems to be a dead link at the time of writing) from the corresponding snap. Unfortunately we can’t include that here as it’s another snap that’s only built for amd64.

There’s plenty in these meta-packages that won’t work on a Pi even if it will install without complaint. For instance, OBS Studio and Blender both require a GPU with OpenGL 3.3 support (the Pi’s only supports up to 2.1). You can attempt to override this with MESA_GL_VERSION_OVERRIDE=3.3 in your environment, but don’t expect much in the way of performance (and also don’t be surprised if several things, like the surfaces in the Blender preview, don’t display at all, assuming you’re patient enough to wait for it to start).

That said, we should talk about the performance because, once again, we’re dealing with a Plasma desktop running under Wayland (assuming you selected the Wayland option on the login screen, which I highly recommend). And, once again, the performance (for a Pi) is great. Despite it being an environment replete with little UI animations, they all seem to play smoothly. The interface never feels laggy in responding to clicks or movements. Sure, there’s some delay when launching things but that’s not the desktop interface; that’s the SD card.

In use Ubuntu Studio is a bit different. For starters, it was the only flavour here to correctly select the HDMI audio output by default. Perhaps this is something to do with the rather more complex audio stack? This is no more than a guess on my part, however.

Studio is by far the largest of the flavours here. With all the meta-packages included above it weighs in at a positively obese 17GB on the SD card, so don’t even think about trying this on anything less than a 32GB card (unless you leave out some meta-packages). Speaking of storage, the partitions on the SD card showed up as removable in the file browser (same as in Kubuntu).

As on Kubuntu, the terminal was the the typically excellent and feature-full Konsole, and the file-browser is again Dolphin. Still, it feels a bit churlish to look at my usual set of applications for Ubuntu Studio. No-one’s going to be installing Studio just to play music or fiddle with the terminal; it’s a flavour for creators. So I scoured the house for all the interesting gear I could find to plug in and see how things went …

First off, given we’re dealing with a novel display stack (Wayland), how about trying to colour-calibrate the monitor? I dug out my trusty old ColorHug2 and strapped it to the screen, fired up DisplayCAL and got to work. No issues generating the profile: hardware was recognized, the screen-saver was suppressed for the duration of calibration, and although the calculations took a while (it is a Pi), it was ultimately successful at generating a profile. Unfortunately, I couldn’t get the profile to install at the end of it (complaints about a timeout, though after a bit of digging I think there’s something else going on). Nor, for that matter, did the swapped-red-and-blue test profile appear to work, so there’s something else missing here. Oh well.

What about drawing? I helped myself to the ancient Wacom Intuos 3 tablet attached to my other half’s Mac Mini, fired up Krita and drew some bits that 6 year-old me would’ve been truly proud of. Ahem. Needless to say, the tablet Just Worked, including all the fun configuration options for optional hot-keys, pressure response curves and tool switching depending on the pen tip. I can’t say this is terribly surprising, though, as it’s always worked on every Linux system I’ve tried it on for the last 10 years or so.

How about music composition? I nicked my daughter’s Yamaha keyboarad, found the one remaining USB-B cable in the house (that wasn’t busy serving a UPS), and hooked it up as a MIDI source. After failing to figure out Ardour, I managed to rekindle some of my LMMS skills and, with some awkward fumbling around the sound output configuration, got it recording from (and playing back through) the keyboard. My formerly formidable piano skills have sadly bit-rotted over the last 30 years. But I still managed to record a passable rendition of the opening of Scott Joplin’s The Entertainer … treble only, at a positively sedate 45bpm, then the bass line on a separate track, then sped it all up to 70 to pretend I could still play.

On the publishing side of things I gave the excellent MuseScore a whirl, including the keyboard as MIDI input. It performed happily and I managed to crank out a vaguely acceptable score for the first bit of The Entertainer again [1]. Printing remotely worked happily, as did scanning (once I’d configured SANE to look on the right host for the scanner).

I gave the video editing and rendering bits a brief try, but they were pretty much as one would expect: basically unusable without much beefier hardware.

Finally, I tried out Scribus as well and was rather surprised to find it very “laggy” when dragging things out on the canvas. I pondered if it’s one of the applications currently suffering from Wayland transition issues and ran it instead with QT_QPA_PLATFORM=xcb and that largely fixed things. Still, I didn’t dig any further (as I’d already burned far too much time playing with this flavour).

But now for the burning question: what about the screen-saver?

It managed to lock the screen (no suspend), and then crashed. It didn’t crash back to the desktop (good marks for security), but also didn’t display a “here’s how to get out of this” message like Kubuntu. I managed to manually use the loginctl trick from Kubuntu to figure out an unlock from another console, but I can imagine an ordinary user resorting to the three-finger shuffle here. Ouch.

Verdict

This isn’t your regular flavour, and it’s certain overkill for many. But if you’re the creative type it may well be the flavour for you. Provided your expectations of the platform are reasonable (this is not the box for Blender, OBS Studio, or kdenlive) it’s useable for basic music creation, composition, writing or art.

Some things aren’t responsive in all circumstances (when drawing with a tablet you have to be a bit careful of what’s visible vs the complexity of the brushes), but with a bit of patience and effort it works! Studio also has a really nice menu detailing where to get further help, from documentation to community support.

It looks like there’s still some kinks to work out (particularly around the Wayland transition) but yet again I’m very impressed by a KDE-based flavour.

Configuration

For Ubuntu Studio, the configurations I used were as follows. For the boot configuration, highlighted lines are those changed from the lunar defaults:

config.txt

 1[all]
 2kernel=vmlinuz
 3cmdline=cmdline.txt
 4initramfs initrd.img followkernel
 5
 6[pi4]
 7max_framebuffers=2
 8arm_boost=1
 9
10[all]
11# Enable the audio output, I2C and SPI interfaces on the GPIO header. As these
12# parameters related to the base device-tree they must appear *before* any
13# other dtoverlay= specification
14dtparam=audio=on
15dtparam=i2c_arm=on
16dtparam=spi=on
17
18# Comment out the following line if the edges of the desktop appear outside
19# the edges of your display
20disable_overscan=1
21
22# If you have issues with audio, you may try uncommenting the following line
23# which forces the HDMI output into HDMI mode instead of DVI (which doesn't
24# support audio output)
25#hdmi_drive=2
26
27# Enable the serial pins
28#enable_uart=1
29
30# Autoload overlays for any recognized cameras or displays that are attached
31# to the CSI/DSI ports. Please note this is for libcamera support, *not* for
32# the legacy camera stack
33camera_auto_detect=1
34display_auto_detect=1
35
36# Config settings specific to arm64
37arm_64bit=1
38dtoverlay=dwc2
39
40[cm4]
41# Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into
42# such a board)
43dtoverlay=dwc2,dr_mode=host
44
45[all]
46dtoverlay=vc4-kms-v3d

For the kernel command line, the entire file must consist of a single line of text so I would suggest simply copying this wholesale:

cmdline.txt

1zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=zstd dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash

For the cloud-init configuration, the highlighted lines are those that you may wish to change after copying the content. As mentioned above, you may wish to trim the list of meta-packages highlighted:

user-data

 1#cloud-config
 2
 3hostname: studio-pi
 4
 5locale: en_GB.UTF-8
 6timezone: Europe/London
 7
 8keyboard:
 9  model: pc105
10  layout: gb
11
12user:
13  name: "dave"
14  lock_passwd: false
15  gecos: "Dave Jones"
16  plain_text_passwd: raspberry
17  sudo: "ALL=(ALL:ALL) ALL"
18
19ssh_import_id:
20  - lp:waveform
21
22apt:
23  sources:
24    mozillateam:
25      source: 'ppa:mozillateam/ppa'
26
27write_files:
28  - path: /etc/apt/preferences.d/firefox
29    content: |
30      Package: firefox*
31      Pin: release o=LP-PPA-mozillateam
32      Pin-Priority: 501
33
34      Package: firefox*
35      Pin: release o=Ubuntu
36      Pin-Priority: -1
37  - path: /home/dave/.profile
38    append: true
39    defer: true
40    content: |
41      export MOZ_ENABLE_WAYLAND=1
42
43package_update: true
44package_upgrade: true
45package_reboot_if_required: true
46
47packages:
48  - ubuntustudio-desktop
49  - ubuntustudio-audio
50  - ubuntustudio-graphics
51  - ubuntustudio-photography
52  - ubuntustudio-video
53  - ubuntustudio-publishing
54  - ubuntu-raspi-settings-desktop
55  - firefox

Finally, the networking configuration. I used Ethernet for my experiments, and given the number of packages that need installing I’d generally recommend that too. Please see the intro post for important information on adjusting the network configuration to Network Manager post installation:

network-config

1network:
2  version: 2
3  ethernets:
4    eth0:
5      dhcp4: true
6      optional: true

[1]	Hang on, if you’re so gung-ho about hating WYSIWYG, shouldn’t you be using LilyPond here? Yes, but that’s not part of the Ubuntu Studio audio seed :)

The Kubuntu Flavour

2023-08-05T00:00:00+01:00

See Also: The intro post which links to all the other flavour posts. Skip down here for the configurations used.

In this post, we’ll be looking at creating a Kubuntu desktop with a cloud-init configuration, on a Raspberry Pi.

Contents

Kubuntu
Verdict
Configuration

Kubuntu

Kubuntu is the KDE-based spin of Ubuntu. For those unfamiliar, KDE is the other “big” desktop player besides GNOME, and is based on the Qt toolkit. KDE’s always had a more “traditional” feel in its desktop (almost staid, some might say). The start bar (with a classical categorised start menu) goes at the bottom, followed by the application switcher, and system tray. Applications all have a reasonably common look and feel, and all applications have every configuration option you could ever hope for scattered throughout their numerous pages of preferences.

This is in fairly stark contrast to GNOME which, of late, has been more experimental and has pursued a deliberate path of “making it simple”. To put it another way, if you sat your elderly mother in front of a modern GNOME system, it might take them a while to figure out how to get going. But on KDE, they’d be up and running right out the gate [1].

The last time I tried Kubuntu on a Pi, it was laughably slow, and I was expecting no different this time. I could not have been more wrong …

Kubuntu is notably the only other desktop environment here that supports Wayland out of the box. It defaults to X11, but you can easily select “Desktop Session: Plasma (Wayland)” from the bottom of the KDM login page, and this selection will persist for future logins. Speaking of logins, there’s a bizarrely long delay before the desktop shows up after login. This appears peculiar to KDE but if you put up with it, the wait is worthwhile!

When running under Wayland, the environment is buttery-smooth. All the little animations from the fade-in of the desktop, to the opening slide of the start menu proceed without jitters or jerks. Don’t expect any speed up in opening large applications. This is still a Pi, and I’m running these experiments off SD cards which top out at 20Mb/s transfer on a Pi 4. Nonetheless, I was astonished to find that the desktop I expected to be the heavy-weight laggard was up there with the best of them in responsiveness!

The bundled applications largely work well. The file browser is the capable Dolphin which was happy browsing and opening things from unmounted network shares, and has a nifty “multiple-selection” feature (look for the “plus” when hovering over items). One oddity is that, as under Lubuntu, it considered the partitions on the SD card “removable” (though it mercifully didn’t offer an eject button for the root partition). One other thing it notably managed was to extract files from an archive with drag’n’drop. This has been broken on GNOME (and thus Ubuntu, LP: #1923033) for years now.

As before, the default audio output is incorrectly selected, but again that’s a trivial work-around. Video playback is handled by the Haruna media player. Unfortunately this wasn’t happy playing things from unmounted network shares (despite them being accessible from the Open File menu). Otherwise, it proved one of the more capable bundled players, handling chapter markers and subtitles with aplomb. The bundled music player is Elisa which is a simple, attractive, and intuitive affair, but sadly doesn’t support playing (or indexing) anything other than a mounted file-system.

Firefox worked well, but now we finally come to something I alluded to at the start. When installed as a snap it doesn’t seem able to use the “full” Wayland mode (reverting to XWayland protocol instead). This is the reason for installing the deb, and also the reason for me re-writing a whole pile of stuff here [2]!

The screen-saver works … then sometimes crashes. At this point I was getting rather used to flakiness in screen-savers but what surprised me was that, having crashed, it brought up a nice and detailed set of instructions on securely unlocking the desktop from another terminal. This … seems like an improvement?

Verdict

What a difference from last time!

Great performance (after a slow initial login), combined with KDE’s legendary configurability. Some reliability issues still (screen-saver, not-so-removable storage!), but this is a genuinely useful desktop on a Pi now.

There’s also a ton of nice little UI tweaks that made it genuinely pleasant to use for several days. The start menu encapsulates all styles (categorical, one big list, and searchable), the system tray has myriad capabilities that make it far more useful than most (e.g. selecting the audio output, or bluetooth devices straight from the tray widgets without launching other things).

Configuration

For Kubuntu, the configurations I used were as follows. For the boot configuration, highlighted lines are those changed from the lunar defaults:

config.txt

 1[all]
 2kernel=vmlinuz
 3cmdline=cmdline.txt
 4initramfs initrd.img followkernel
 5
 6[pi4]
 7max_framebuffers=2
 8arm_boost=1
 9
10[all]
11# Enable the audio output, I2C and SPI interfaces on the GPIO header. As these
12# parameters related to the base device-tree they must appear *before* any
13# other dtoverlay= specification
14dtparam=audio=on
15dtparam=i2c_arm=on
16dtparam=spi=on
17
18# Comment out the following line if the edges of the desktop appear outside
19# the edges of your display
20disable_overscan=1
21
22# If you have issues with audio, you may try uncommenting the following line
23# which forces the HDMI output into HDMI mode instead of DVI (which doesn't
24# support audio output)
25#hdmi_drive=2
26
27# Enable the serial pins
28#enable_uart=1
29
30# Autoload overlays for any recognized cameras or displays that are attached
31# to the CSI/DSI ports. Please note this is for libcamera support, *not* for
32# the legacy camera stack
33camera_auto_detect=1
34display_auto_detect=1
35
36# Config settings specific to arm64
37arm_64bit=1
38dtoverlay=dwc2
39
40[cm4]
41# Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into
42# such a board)
43dtoverlay=dwc2,dr_mode=host
44
45[all]
46dtoverlay=vc4-kms-v3d

For the kernel command line, the entire file must consist of a single line of text so I would suggest simply copying this wholesale:

cmdline.txt

1zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=zstd dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash

For the cloud-init configuration, the highlighted lines are those that you may wish to change after copying the content:

user-data

 1#cloud-config
 2
 3hostname: kubuntu-pi
 4
 5locale: en_GB.UTF-8
 6timezone: Europe/London
 7
 8keyboard:
 9  model: pc105
10  layout: gb
11
12user:
13  name: "dave"
14  lock_passwd: false
15  gecos: "Dave Jones"
16  plain_text_passwd: raspberry
17  sudo: "ALL=(ALL:ALL) ALL"
18
19ssh_import_id:
20  - lp:waveform
21
22apt:
23  sources:
24    mozillateam:
25      source: 'ppa:mozillateam/ppa'
26
27write_files:
28  - path: /etc/apt/preferences.d/firefox
29    content: |
30      Package: firefox*
31      Pin: release o=LP-PPA-mozillateam
32      Pin-Priority: 501
33
34      Package: firefox*
35      Pin: release o=Ubuntu
36      Pin-Priority: -1
37  - path: /home/dave/.profile
38    append: true
39    defer: true
40    content: |
41      export MOZ_ENABLE_WAYLAND=1
42
43package_update: true
44package_upgrade: true
45package_reboot_if_required: true
46
47packages:
48  - kubuntu-desktop
49  - ubuntu-raspi-settings-desktop
50  - firefox

network-config

1network:
2  version: 2
3  ethernets:
4    eth0:
5      dhcp4: true
6      optional: true

[1]	I leave it for the reader to judge what this implies about GNOME’s drive for simplicity …

[2]

I’d started off this series doing everything “by the book” with Firefox from a snap, then after discovering that only the deb-based install supported this, wound up re-doing everything partly to ensure all the flavours had the best performance I could give them (full Wayland support if available without too much hacking), and secondly that they were on a level playing field (all snaps or all debs for Firefox).

The Ubuntu Unity Flavour

2023-08-04T00:00:00+01:00

See Also: The intro post which links to all the other flavour posts. Skip down here for the configurations used.

In this post, we’ll be looking at creating an Ubuntu Unity desktop with a cloud-init configuration, on a Raspberry Pi.

Contents

Ubuntu Unity
Verdict
Configuration

Ubuntu Unity

A blast from the past! This was one I didn’t manage to test last time, but I’ve a major soft spot for good ol’ Unity, having spent many years with it as my desktop on the PC back in the pre-Bionic days. So let’s give it a spin on the Pi!

For those unfamiliar with Unity, it has some quite different ideas of its own: the start menu and quick launcher are on the left of the screen (rather than the traditional top or bottom). If I recall correctly (and I’m sure someone will correct me in the comments if I’m not) this was because most screens have a wide-screen aspect ratio, so if you’re going to “waste” space with non-application stuff, better to do it horizontally than vertically (which sounds pretty reasonable to me).

In later iterations, and indeed in Unity 7 that we’re using here, it also borrows a few tid-bits from MacOS X: the window icons (minimize, maximize, close) are on the top left, and application menus appear in the bar at the top of the window rather than within the application window. Again, I think I’m right in stating this was (at least partially) in aid of not wasting vertical real-estate.

The selection of bundled applications was excellent. VLC for video playing, and Rhythmbox for music. Couldn’t ask for better. The file browser is the excellent Nemo from the Cinnamon desktop. Everything was generally reliable and worked … eventually.

Unfortunately, this brings us to the first major issue: performance. As with all the alternative desktops so far there’s no Wayland support yet, but even discounting that performance was painfully slow. I did note that, as under Budgie, cups-browsed was eating 100% of a CPU core again. But even after killing that, things still felt very sluggish, especially after the snappy responses of Lubuntu and Xubuntu.

Integration was also spotty. The menus for most things appeared in the top bar. But not Firefox (arguably the most important application on the desktop). One might argue that Firefox doesn’t have a menu bar any-more, that it’s succumbed to the modern obsession with hiding everything behind a “burger” button but the menus are still there: hit Alt+F and you’ll see (in fact several of the menus aren’t available from the “burger” button, which is probably why).

However, the screen-saver? Works! This is the only other flavour to, out of the box, reliably suspend my monitor and lock the desktop with no crashes (Lubuntu being the other).

Verdict

I’d really love to recommend Unity. Its interface ideas may be novel and quite different to most other flavours, but they make a good deal of sense (far more than GNOME’s do, to me anyway). However, while the reliability is great, the performance just isn’t good enough for use on a small platform.

The theme (whether light or dark) made it rather hard to distinguish window edges of inactive applications. Looking at the screenshot above, can you see where the file-browser ends, and the music player or the browser begins? There are also some strange edges to menu corners as if the environment wanted to draw shadows there, but forgot:

This all gave me an impression that things were not rendering “as intended”. Is there meant to be a shadow border around windows?

Configuration

For Ubuntu Unity, the configurations I used were as follows. For the boot configuration, highlighted lines are those changed from the lunar defaults:

config.txt

 1[all]
 2kernel=vmlinuz
 3cmdline=cmdline.txt
 4initramfs initrd.img followkernel
 5
 6[pi4]
 7max_framebuffers=2
 8arm_boost=1
 9
10[all]
11# Enable the audio output, I2C and SPI interfaces on the GPIO header. As these
12# parameters related to the base device-tree they must appear *before* any
13# other dtoverlay= specification
14dtparam=audio=on
15dtparam=i2c_arm=on
16dtparam=spi=on
17
18# Comment out the following line if the edges of the desktop appear outside
19# the edges of your display
20disable_overscan=1
21
22# If you have issues with audio, you may try uncommenting the following line
23# which forces the HDMI output into HDMI mode instead of DVI (which doesn't
24# support audio output)
25#hdmi_drive=2
26
27# Enable the serial pins
28#enable_uart=1
29
30# Autoload overlays for any recognized cameras or displays that are attached
31# to the CSI/DSI ports. Please note this is for libcamera support, *not* for
32# the legacy camera stack
33camera_auto_detect=1
34display_auto_detect=1
35
36# Config settings specific to arm64
37arm_64bit=1
38dtoverlay=dwc2
39
40[cm4]
41# Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into
42# such a board)
43dtoverlay=dwc2,dr_mode=host
44
45[all]
46dtoverlay=vc4-kms-v3d

For the kernel command line, the entire file must consist of a single line of text so I would suggest simply copying this wholesale:

cmdline.txt

1zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=zstd dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash

For the cloud-init configuration, the highlighted lines are those that you may wish to change after copying the content:

user-data

 1#cloud-config
 2
 3hostname: unity-pi
 4
 5locale: en_GB.UTF-8
 6timezone: Europe/London
 7
 8keyboard:
 9  model: pc105
10  layout: gb
11
12user:
13  name: "dave"
14  lock_passwd: false
15  gecos: "Dave Jones"
16  plain_text_passwd: raspberry
17  sudo: "ALL=(ALL:ALL) ALL"
18
19ssh_import_id:
20  - lp:waveform
21
22apt:
23  sources:
24    mozillateam:
25      source: 'ppa:mozillateam/ppa'
26
27write_files:
28  - path: /etc/apt/preferences.d/firefox
29    content: |
30      Package: firefox*
31      Pin: release o=LP-PPA-mozillateam
32      Pin-Priority: 501
33
34      Package: firefox*
35      Pin: release o=Ubuntu
36      Pin-Priority: -1
37  - path: /home/dave/.profile
38    append: true
39    defer: true
40    content: |
41      export MOZ_ENABLE_WAYLAND=1
42
43package_update: true
44package_upgrade: true
45package_reboot_if_required: true
46
47packages:
48  - ubuntu-unity-desktop
49  - ubuntu-raspi-settings-desktop
50  - firefox

network-config

1network:
2  version: 2
3  ethernets:
4    eth0:
5      dhcp4: true
6      optional: true

The Ubuntu Budgie Flavour

2023-08-03T00:00:00+01:00

See Also: The intro post which links to all the other flavour posts. Skip down here for the configurations used.

In this post, we’ll be looking at creating an Ubuntu Budgie desktop (with Pi-specific customizations!) with a cloud-init configuration.

Contents

Ubuntu Budgie
Verdict
Configuration

Ubuntu Budgie

The Budgie desktop is (was?) based on GNOME but with a more “traditional” desktop feel. The Ubuntu Budgie remix is styled with a brief nod towards MacOS X with a Dock-like launcher (Plank) centred at the bottom of the screen. However, that’s as far as the Mac-allusions go. There’s no application menu in the bar at the top, and the window icons are at the top-right instead of top-left.

Despite the GNOME heritage, there’s no Wayland support yet (although a bit of googling does suggest that it’s a high priority for the next release). On first login, I did need to select “Budgie Desktop” in the login manager (GDM) instead of the “Ubuntu Desktop” default, which was a little odd, but it turned out there was a reason for that which we’ll come back to later …

The specific package used to install the Ubuntu Budgie remix (ubuntu-budgie-desktop-raspi) also includes a nice little Pi-specific customization tool that launches on first login. However, I think some of its options may be redundant at this point (for instance, when using the “full” KMS overlay there’s no need to specify a GPU memory size, even if you are using a camera via the newer libcamera stack).

Most of the terminals in the various flavours are much of a muchness; all support multiple tabs, all have scroll-back, and all are essentially decent at their job. However, Budgie’s pick for a terminal (Tilix) stands out as being the only GTK-based terminal to support tiled panes, that I’m aware of [1]. Last time I tried it (in 2020) the terminal (which may or may not have been Tilix, I can’t recall) crashed pretty reliably, but this time around it was solid as any of the rest, and a stand-out feature of the environment.

Budgie seems to have a bit of a thing about tiling generally. The desktop’s got corner tiling by default (useful on a big monitor), but also has some fascinating shortcuts for ⅖-⅗ tiling, custom (shortcut activated) layouts, and arrangement of windows in arbitrarily sized grids. On my little monitor, I don’t really need anything more than side-by-side tiling but I can imagine this being very handy on large-scale high-DPI monitors.

Sadly, the reliability of the console wasn’t reflected elsewhere. Things felt a bit sluggish at times and eventually I figured out cups-browsed was sitting around eating 100% of one of the CPU cores. This might be a known issue (LP: #2018504), but that’s marked “fix released” on Lunar and this was running on a fully up-to-date system. I’ll need to try and reproduce it to see if that bug needs more attention.

Video playback also proved tricky to test; Budgie bundles the same default video player as Xubuntu, Parole, but this crashed the Budgie window manager process on the couple of occasions I tried it (it happily kept on playing the video, but the desktop behind flickered out of existence and then restarted!). The bundled Music player is Lollypop and, while it seems a capable player, it didn’t seem to have any support for any of the music sharing protocols supported on the home server, so I wound up mounting an NFS share for the purposes of testing. This worked, but it struggled a bit with the sheer quantity of albums, eating a good half a gig of RAM by the time it was done indexing. The audio output default also needed adjusting (as on the other desktops).

And the screen-saver? Didn’t appear to do anything at all at first (screen never blanked, no suspend, no lock). This turned out to be an issue with gnome-shell-common being installed. With that package purged (and … ubuntu-budgie-desktop along with it!), the screen-saver managed to lock the desktop correctly, but still failed to suspend the monitor (same as the official GNOME desktop). Interestingly, this also uninstalled the GDM3 login greeter, and replaced it with the slick greeter (familiar to users of Unity), which resolved having to select “Budgie Desktop” at login.

Still, this begs the question why gnome-shell-common is a transitive dependency of ubuntu-budgie-desktop (purging it does remove the meta-package)? Is there a reason for this, or is it a bug?

Verdict

It’s attractive (nice start menu!), reasonably responsive for a modern desktop, and has some novel applications (nice to see some Pi specific customizations). However, it needs a bit of tweaking out of the box for better reliability. I don’t like removing seeds that supposedly define the system!

The tiling capabilities look very interesting for those on large monitors, though.

Configuration

For Ubuntu Budgie, the configurations I used were as follows. For the boot configuration, highlighted lines are those changed from the lunar defaults:

config.txt

 1[all]
 2kernel=vmlinuz
 3cmdline=cmdline.txt
 4initramfs initrd.img followkernel
 5
 6[pi4]
 7max_framebuffers=2
 8arm_boost=1
 9
10[all]
11# Enable the audio output, I2C and SPI interfaces on the GPIO header. As these
12# parameters related to the base device-tree they must appear *before* any
13# other dtoverlay= specification
14dtparam=audio=on
15dtparam=i2c_arm=on
16dtparam=spi=on
17
18# Comment out the following line if the edges of the desktop appear outside
19# the edges of your display
20disable_overscan=1
21
22# If you have issues with audio, you may try uncommenting the following line
23# which forces the HDMI output into HDMI mode instead of DVI (which doesn't
24# support audio output)
25#hdmi_drive=2
26
27# Enable the serial pins
28#enable_uart=1
29
30# Autoload overlays for any recognized cameras or displays that are attached
31# to the CSI/DSI ports. Please note this is for libcamera support, *not* for
32# the legacy camera stack
33camera_auto_detect=1
34display_auto_detect=1
35
36# Config settings specific to arm64
37arm_64bit=1
38dtoverlay=dwc2
39
40[cm4]
41# Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into
42# such a board)
43dtoverlay=dwc2,dr_mode=host
44
45[all]
46dtoverlay=vc4-kms-v3d

For the kernel command line, the entire file must consist of a single line of text so I would suggest simply copying this wholesale:

cmdline.txt

1zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=zstd dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash

For the cloud-init configuration, the highlighted lines are those that you may wish to change after copying the content:

user-data

 1#cloud-config
 2
 3hostname: budgie-pi
 4
 5locale: en_GB.UTF-8
 6timezone: Europe/London
 7
 8keyboard:
 9  model: pc105
10  layout: gb
11
12user:
13  name: "dave"
14  lock_passwd: false
15  gecos: "Dave Jones"
16  plain_text_passwd: raspberry
17  sudo: "ALL=(ALL:ALL) ALL"
18
19ssh_import_id:
20  - lp:waveform
21
22apt:
23  sources:
24    mozillateam:
25      source: 'ppa:mozillateam/ppa'
26
27write_files:
28  - path: /etc/apt/preferences.d/firefox
29    content: |
30      Package: firefox*
31      Pin: release o=LP-PPA-mozillateam
32      Pin-Priority: 501
33
34      Package: firefox*
35      Pin: release o=Ubuntu
36      Pin-Priority: -1
37  - path: /home/dave/.profile
38    append: true
39    defer: true
40    content: |
41      export MOZ_ENABLE_WAYLAND=1
42
43package_update: true
44package_upgrade: true
45package_reboot_if_required: true
46
47packages:
48  - ubuntu-budgie-desktop-raspi
49  - ubuntu-raspi-settings-desktop
50  - firefox

network-config

1network:
2  version: 2
3  ethernets:
4    eth0:
5      dhcp4: true
6      optional: true

[1]	I specify GTK-based because that tittering you can hear is from KDE’s Konsole users, who’ve had this feature for yonks.

The Xubuntu Flavour

2023-08-02T00:00:00+01:00

See Also: The intro post which links to all the other flavour posts. Skip down here for the configurations used.

In this post, we’ll be looking at creating a Xubuntu desktop with a cloud-init configuration, on a Raspberry Pi.

Contents

Xubuntu
Verdict
Configuration

Xubuntu

Xubuntu is another lightweight flavour of Ubuntu (similar to Lubuntu in this respect), but based on XFCE (a GTK based environment). Subjectively, I couldn’t discern much difference in performance between Lubuntu and Xubuntu. Other than the positioning of the application bar at the top of the screen (compared to Lubuntu’s traditional bottom placement), things felt pretty similar.

On the functionality side of things, the incorrect audio output appeared as in Lubuntu, but was trivial to work around. Out of the box, music playback from the network worked thanks to the inclusion of the excellent Rhythmbox [1]. Video playback worked well with the Parole player, but unfortunately it didn’t seem able to playback from an SMB network share unless it was mounted.

This is one of those “integration” things; the file browser (Thunar), and most file-open dialogs, will happily browse to an SMB share and will appear to open the file. But then an error appears or, worse, nothing happens. Of course, the solution in this case is simple: sudo apt install vlc and it Just Works.

On the subject of bugs, remember I mentioned screen-savers? The lock-screen would happily kick in and blank the monitor, but wouldn’t suspend it (presumably LP: #1998716 again?). However, attempting to unlock (wiggling the mouse or hitting a key to bring up the login screen) gave the poor thing conniptions. The password prompt would briefly flicker into life, then the screen would blank again. Then flicker into life, then blank. Password entry wouldn’t work while all this was going on (something crashing?). After this happened several (4 or 5?) times, it seemed to get a grip on itself and allow login once more, but it’s rather broken! I’ll try and find some time to dig out some more useful info for a bug report (like whether this is specific to my monitor).

Everything else was pretty much as expected: I could read PDFs, print things, open links, browse the web, manipulate the file-system, use the terminal, all without issue, and all reasonably smoothly.

One amusing stand-out I did notice: the choice of bundled desktop games! On most desktops, this is pretty bland fare. Some variation of Klondike solitaire, maybe a Minesweeper clone, perhaps a Sudoku implementation. But not here! No, Xubuntu bundles one single game package [2], but it’s packed with more fiendishly fun entertainment than all the other default desktop gaming packages put together: Simon Tatham’s Puzzles (if you haven’t played these before, you owe it to yourself to sudo apt install sgt-launcher and give them a whirl).

Verdict

It’s pretty (rather more so that Lubuntu in my opinion), lightweight, responsive and reasonably functional. However, it’s got a few more bugs than Lubuntu (the screen-saver particularly) so given the choice I’d probably go with Lubuntu for a light-weight choice. Still … a brilliant selection of games!

Configuration

For Xubuntu, the configurations I used were as follows. For the boot configuration, highlighted lines are those changed from the lunar defaults:

config.txt

 1[all]
 2kernel=vmlinuz
 3cmdline=cmdline.txt
 4initramfs initrd.img followkernel
 5
 6[pi4]
 7max_framebuffers=2
 8arm_boost=1
 9
10[all]
11# Enable the audio output, I2C and SPI interfaces on the GPIO header. As these
12# parameters related to the base device-tree they must appear *before* any
13# other dtoverlay= specification
14dtparam=audio=on
15dtparam=i2c_arm=on
16dtparam=spi=on
17
18# Comment out the following line if the edges of the desktop appear outside
19# the edges of your display
20disable_overscan=1
21
22# If you have issues with audio, you may try uncommenting the following line
23# which forces the HDMI output into HDMI mode instead of DVI (which doesn't
24# support audio output)
25#hdmi_drive=2
26
27# Enable the serial pins
28#enable_uart=1
29
30# Autoload overlays for any recognized cameras or displays that are attached
31# to the CSI/DSI ports. Please note this is for libcamera support, *not* for
32# the legacy camera stack
33camera_auto_detect=1
34display_auto_detect=1
35
36# Config settings specific to arm64
37arm_64bit=1
38dtoverlay=dwc2
39
40[cm4]
41# Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into
42# such a board)
43dtoverlay=dwc2,dr_mode=host
44
45[all]
46dtoverlay=vc4-kms-v3d

For the kernel command line, the entire file must consist of a single line of text so I would suggest simply copying this wholesale:

cmdline.txt

1zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=zstd dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash

For the cloud-init configuration, the highlighted lines are those that you may wish to change after copying the content:

user-data

 1#cloud-config
 2
 3hostname: xubuntu-pi
 4
 5locale: en_GB.UTF-8
 6timezone: Europe/London
 7
 8keyboard:
 9  model: pc105
10  layout: gb
11
12user:
13  name: "dave"
14  lock_passwd: false
15  gecos: "Dave Jones"
16  plain_text_passwd: raspberry
17  sudo: "ALL=(ALL:ALL) ALL"
18
19ssh_import_id:
20  - lp:waveform
21
22apt:
23  sources:
24    mozillateam:
25      source: 'ppa:mozillateam/ppa'
26
27write_files:
28  - path: /etc/apt/preferences.d/firefox
29    content: |
30      Package: firefox*
31      Pin: release o=LP-PPA-mozillateam
32      Pin-Priority: 501
33
34      Package: firefox*
35      Pin: release o=Ubuntu
36      Pin-Priority: -1
37  - path: /home/dave/.profile
38    append: true
39    defer: true
40    content: |
41      export MOZ_ENABLE_WAYLAND=1
42
43package_update: true
44package_upgrade: true
45package_reboot_if_required: true
46
47packages:
48  - xubuntu-desktop
49  - ubuntu-raspi-settings-desktop
50  - firefox

network-config

1network:
2  version: 2
3  ethernets:
4    eth0:
5      dhcp4: true
6      optional: true

[1]

Rhythmbox is probably my favourite part of the entire GNOME project. It’s genuinely superb: supports every protocol and format I’ve ever needed (and several I haven’t), has a UI that’s minimal but fantastically functional, and most of all fast. Want to navigate your way through ~8000 songs spread over ~500 albums? No problem, even on a Pi! It’s one of the four applications that I reliably fire up every single day.

[2]	That I could see; the seed seems to suggest it ships the GNOME implementations of Mines and Sudoku too but they didn’t appear in the start menu.

The Lubuntu Flavour

2023-08-01T00:00:00+01:00

See Also: The intro post which links to all the other flavour posts. Skip down here for the configurations used.

In this post, we’ll be looking at creating a Lubuntu desktop with a cloud-init configuration, on a Raspberry Pi.

Contents

Lubuntu
Verdict
Configuration

Lubuntu

Lubuntu is a lightweight flavour of Ubuntu, based on LXQt (which is the desktop environment that RaspiOS is based upon too). It’s certainly the lightest in terms of resource usage from all the desktops I’m comparing, and one of the most responsive too.

It’s a bit prettier than when I last saw it 3 years ago, but still doesn’t have the polish of the “full” desktop suites like GNOME or KDE. The interface feels very responsive but it’s notable that there’s no Wayland support yet so the performance of graphically heavy things is not entirely smooth.

One issue immediately jumped out, which was the incorrect selection of the default audio output (the 3.5mm jack output which wasn’t connected, instead of HDMI). This is a known issue (LP: #1993347), and one that occurs on almost every other flavour too. Thankfully it’s a trivial work-around (just pick the HDMI output instead) and the selection persists across reboots, so it’s a one-time fix.

The applications on Lubuntu are understandably minimal, but surprisingly good! Most desktop environments try and bundle “native” applications for specific purposes, like a video player. By “native” I mean applications that stick strictly within that desktop’s base toolkit (whether that’s GTK or Qt, and their corresponding media bits), so that the distribution of that desktop doesn’t bloat itself out with additional libraries (or whole frameworks) for a single application. This means you’ll see things like the Parole or Totem video players … right before you open a command line, sudo apt install vlc, and never look back.

Lubuntu, by contrast, just installs VLC saving you a whole step! That said, VLC is all you get for media playback here and, capable though VLC is with music, it’s not set up as an album browser. I’m too used to having something to do just that. The distro seed giveth, and the distro seed … leaveth stuff out.

There were a couple of minor oddities. The two partitions on the SD card show up as “removable” (which they fairly obviously … aren’t) on the system tray widget. After one system upgrade and a reboot, the wallpaper disappeared inexplicably. However, the screen-saver worked perfectly. Locked the screen and put the monitor in power-saving mode. Why do I mention this? Firstly, I’ve had a long-standing issue with the official GNOME desktop not putting my monitor into power-saving mode (LP: #1998716). Secondly … screen-saver breakage became a bit of a theme the further I went on with this series …

Otherwise, things mostly worked well. The included file browser is PCManFM, which is capable (and fast!). It’ll also be familiar to users of RaspiOS as it’s the default file browser there too. No issues opening things, including over unmounted network shares, no problems printing or scanning over the network, and Firefox worked as normal.

Verdict

It’s not the prettiest, but it is lightweight, responsive, and has a nice choice of default apps. In practice it feels functional enough that I’d be happy to use it as my desktop, but I’d definitely miss the ability to easily “tile” applications (maybe there’s a shortcut for this that I didn’t find!).

Configuration

For Lubuntu, the configurations I used were as follows. For the boot configuration, highlighted lines are those changed from the lunar defaults:

config.txt

 1[all]
 2kernel=vmlinuz
 3cmdline=cmdline.txt
 4initramfs initrd.img followkernel
 5
 6[pi4]
 7max_framebuffers=2
 8arm_boost=1
 9
10[all]
11# Enable the audio output, I2C and SPI interfaces on the GPIO header. As these
12# parameters related to the base device-tree they must appear *before* any
13# other dtoverlay= specification
14dtparam=audio=on
15dtparam=i2c_arm=on
16dtparam=spi=on
17
18# Comment out the following line if the edges of the desktop appear outside
19# the edges of your display
20disable_overscan=1
21
22# If you have issues with audio, you may try uncommenting the following line
23# which forces the HDMI output into HDMI mode instead of DVI (which doesn't
24# support audio output)
25#hdmi_drive=2
26
27# Enable the serial pins
28#enable_uart=1
29
30# Autoload overlays for any recognized cameras or displays that are attached
31# to the CSI/DSI ports. Please note this is for libcamera support, *not* for
32# the legacy camera stack
33camera_auto_detect=1
34display_auto_detect=1
35
36# Config settings specific to arm64
37arm_64bit=1
38dtoverlay=dwc2
39
40[cm4]
41# Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into
42# such a board)
43dtoverlay=dwc2,dr_mode=host
44
45[all]
46dtoverlay=vc4-kms-v3d

For the kernel command line, the entire file must consist of a single line of text so I would suggest simply copying this wholesale:

cmdline.txt

1zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=zstd dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash

For the cloud-init configuration, the highlighted lines are those that you may wish to change after copying the content:

user-data

 1#cloud-config
 2
 3hostname: lubuntu-pi
 4
 5locale: en_GB.UTF-8
 6timezone: Europe/London
 7
 8keyboard:
 9  model: pc105
10  layout: gb
11
12user:
13  name: "dave"
14  lock_passwd: false
15  gecos: "Dave Jones"
16  plain_text_passwd: raspberry
17  sudo: "ALL=(ALL:ALL) ALL"
18
19ssh_import_id:
20  - lp:waveform
21
22apt:
23  sources:
24    mozillateam:
25      source: 'ppa:mozillateam/ppa'
26
27write_files:
28  - path: /etc/apt/preferences.d/firefox
29    content: |
30      Package: firefox*
31      Pin: release o=LP-PPA-mozillateam
32      Pin-Priority: 501
33
34      Package: firefox*
35      Pin: release o=Ubuntu
36      Pin-Priority: -1
37  - path: /home/dave/.profile
38    append: true
39    defer: true
40    content: |
41      export MOZ_ENABLE_WAYLAND=1
42
43package_update: true
44package_upgrade: true
45package_reboot_if_required: true
46
47packages:
48  - lubuntu-desktop
49  - ubuntu-raspi-settings-desktop
50  - firefox

network-config

1network:
2  version: 2
3  ethernets:
4    eth0:
5      dhcp4: true
6      optional: true

We Serve All Flavours!

2023-07-31T00:00:00+01:00

See Also: The prior post in which we learned about the wonders of automated configuration with cloud-init.

It’s that time of the decade, again: desktop round-up time! As of writing this introductory paragraph, I’ve now finished poking, prodding, and reviewing each of the Ubuntu flavours. However, the article turned into yet another small-novella sized piece and the only sensible way I can think to publish this is by making this one an introductory piece, with a few final cloud-init hints, and an index to the other articles which will be published once a day over the coming week (and a bit).

My apologies, dear reader, for keeping you on tenterhooks again …

Contents

Adding Sprinkles
99 Flakes
The Taste Test
The Flavours

Adding Sprinkles

In the interests of “integration”, and because I wish to demonstrate yet more capabilities of cloud-init, I’m going to pre-emptively work around a long-standing bug (LP: #1974064) which I frequently run into, preventing links in PDFs from opening in the browser. But only if that browser is a snap … as Firefox is. Furthermore (without giving too much away from future posts in this series) there still appear to be some integration issues between the snapped Firefox and Wayland based environments in some of the desktops. So, in the interests of a level playing field for everyone we’re going to get cloud-init to install Firefox from a deb on first boot.

We need to add an apt source (the Firefox stable PPA), and add some configuration to pin its items higher than the archive’s. Thankfully, this is pretty simple to derive from all those guides out there showing how to do this on an already running system:

apt:
  sources:
    mozillateam:
      source: 'ppa:mozillateam/ppa'

write_files:
  - path: /etc/apt/preferences.d/firefox
    content: |
      Package: firefox*
      Pin: release o=LP-PPA-mozillateam
      Pin-Priority: 501

      Package: firefox*
      Pin: release o=Ubuntu
      Pin-Priority: -1

Now we can ditch that weird snap section in our configuration, and just stick firefox in the list of packages to install.

One other tip I’m going to add here, but which I won’t be adding to the experimental configurations as published, because the vast majority of people won’t want this, is configuration for an apt proxy (specifically apt-cacher-ng). To be clear, for most people I don’t recommend running this (a couple of times a month it sends my server into a bit of a tizz and needs kicking), but if you’re going to be repeatedly installing the same large packages (as I frequently do for packaging work, or in this case when installing several desktop packages which have similar bases) it can save a huge amount of time and bandwidth. Anyway, this is what the relevant snippet looks like:

apt:
  conf: |
    Acquire::http { Proxy "http://myproxy.local:3142"; }

Obviously, replace myproxy.local with the address of your local proxy.

Below is the full configuration we’ll be using (minus the apt-cacher configuration), with a fairly obvious placeholder for the desktop package. I’ll publish full configurations (including the config.txt, cmdline.txt, and network-config, largely derived from the prior post) in each section in case you want to play along at home. Just place these on the boot partition of an Ubuntu Server for Pi (23.04 or later) card [1] and adjust the highlighted lines to your tastes:

user-data

 1#cloud-config
 2
 3hostname: miss-piggy
 4
 5locale: en_GB.UTF-8
 6timezone: Europe/London
 7
 8keyboard:
 9  model: pc105
10  layout: gb
11  options: ctrl:nocaps
12
13user:
14  name: "dave"
15  lock_passwd: false
16  gecos: "Dave Jones"
17  plain_text_passwd: raspberry
18  sudo: "ALL=(ALL:ALL) ALL"
19
20ssh_import_id:
21  - lp:waveform
22
23apt:
24  sources:
25    mozillateam:
26      source: 'ppa:mozillateam/ppa'
27
28write_files:
29  - path: /etc/apt/preferences.d/firefox
30    content: |
31      Package: firefox*
32      Pin: release o=LP-PPA-mozillateam
33      Pin-Priority: 501
34
35      Package: firefox*
36      Pin: release o=Ubuntu
37      Pin-Priority: -1
38  - path: /home/dave/.profile
39    append: true
40    defer: true
41    content: |
42      export MOZ_ENABLE_WAYLAND=1
43
44package_update: true
45package_upgrade: true
46package_reboot_if_required: true
47
48packages:
49  - YOUR-DESKTOP-PACKAGE-HERE
50  - ubuntu-raspi-settings-desktop
51  - firefox

Bear in mind your first boot is going to install a lot of things, and it will take at least an hour (more like an hour and a half) even on a good A1-compliant SD card.

99 Flakes

One wrinkle I forgot to mention in the prior post is that of networking. The server images we’re basing this on use netplan, and start off by having it “render” the network configuration via networkd (part of systemd).

However, every Ubuntu desktop flavour ultimately winds up using Network Manager for its network configuration. While netplan is capable of rendering a configuration via Network Manager, that’s probably not what you ultimately want as your network configuration wouldn’t be editable via the Network Manager GUI (only the netplan YAML configuration files).

Why not change the initial netplan configuration to render through Network Manager instead? This won’t work because at first boot, the card is just Ubuntu Server for Pi and doesn’t contain Network Manager. That only gets installed along with the chosen desktop flavour, but that act requires a network to already be configured. Dependency cycles!

The fix is simple enough. Once the desktop has booted, and you’re all snugly logged in for the first time, run the following to delete the generated netplan configuration and re-apply the (now empty) settings:

$ sudo rm /etc/netplan/50-cloud-init.yaml
$ sudo netplan apply

At this point networkd will stop “managing” the interfaces and Network Manager will take them over. In the case of a simple DHCP-based Ethernet connection, it should reconfigure automatically and you’ll be back up and running in seconds. However, if you performed your initial set up over WiFi you’ll now need to go into the GUI network settings for your flavour (wherever that is — it can usually be found from the system-tray though) and re-enter your WiFi credentials.

This is one of the ways that this Frankenstein-like desktop setup differs from the official desktop releases. There are a few others (dual seeds, user creation differences, root-fs sizing, probably a few others that I’ve forgotten!), and if anyone’s interested I can go through them, but this is the only one that’s affected the use of the desktop during my explorations.

The Taste Test

What should I look for in a desktop? I can’t define this universally, so I’ll just go with what works for me. My personal workflow is probably a strange mix of deeply anachronistic, and semi-modern. On the one hand, I spend the vast majority of my time in a terminal window. On the other, I eschew office applications in favour of doing office-y work in a web-browser. Google Sheets for any spreadsheet work (though I have been known to dip into sc for trivial bits), and when I absolutely have to touch a word-processor [2], Google Docs.

I could almost get away with working on a Chromebook, but not quite. If I have PDFs to deal with (and between Canonical’s hiring push, and various hardware data-sheets, I get through quite a few of these), I use an honest-to-goodness standalone PDF reader, instead of the (usually horribly slow) built-in PDF readers in browsers. Sometimes I even … brace yourself … print them [3]!

I also play most of my music, served from the home server, in a standalone application, and the same goes for most films I watch (there are some good web-based systems for this, but browsers are heavy and I work on a Pi so I tend to try and limit my use of them to text-based duties). Most of my file manipulation is done from the command line, but sometimes for simpler stuff (especially involving external storage) I’ll resort to the file browser.

So, I’ll be looking for:

A decent terminal — this is pretty much a given; it’d hardly be a Linux distro without a decent terminal!
A decent browser — again, pretty much a given as we’ll be installing Firefox anyway.
Some decent media applications — music player, video player, a standalone PDF viewer, preferably a good file browser for managing it all.
Integration — I want to be able to click a link from my terminal, or in a PDF, and have it open in the web-browser. I’d like to be able to open files, archives, and videos straight from the file-browser, even from network shares (ideally without messing around with network mounts). The home server has all sorts of services for this (NFS, SMB, DLNA, etc. all advertised over avahi) so there should be plenty of opportunity here. I also want to be able to print stuff without jumping through tons of hoops (again, the printers here are network-accessible and advertised over avahi).
Performance — it’s a Pi, so I can’t expect the world here, but the video player should handle 720p or 1080p sources in full-screen, smoothly. The music player should play without stuttering, and generally things shouldn’t feel (too) sluggish. The browser is an exception here: modern browsers are ridiculously (over-)complex beasts and I’ll allow some lee-way for this.
Reliability — The more bugs I run into, the more points I’ll dock. Erm … are we doing points? Well, any bugs I run into I’ll point out. And possibly report, if I can figure out what to file ‘em against!

Now, with all that extra gubbins out of the way, on with the tests!

The Flavours

These are the flavours I’m intending to cover, which will become links to the relevant article:

Lubuntu — the LXQt flavour
Xubuntu — the XFCE flavour
Ubuntu Budgie — the Budgie flavour
Ubuntu Unity — the Unity flavour
Kubuntu — the KDE flavour
Ubuntu Studio — the creator’s flavour
Ubuntu — comparison to the official GNOME-based image

[1]	Side note: theoretically this should work on either armhf or arm64, however I’m only testing arm64 here as it’s all we officially support on the desktop images anyway.

[2]

In case it’s not obvious, I am firmly of the opinion that WYSIWYG is the work of the devil, and the One True Way is that of the markup language, beautifully processed by 100% natural, hand-crafted, hypo-allergenic, non-genetically modified, organically-certified algorithms into whatever output format you may prefer.

[3]	I have rarely felt as old as I did a few weeks ago when a younger colleague casually inquired, “does anyone still have a printer?” Yes! In my utter extravagance I even have two. One of them does colour!

Desktop on a Stick

2023-06-30T00:00:00+01:00

Updated to include the per-release quirks channel for the snap.

TL;DR: Skip down here for the final configurations.

It’s that time of the decade, again: desktop round-up time! But thanks to a genius notion from Oliver (one of our product managers), I’m going to be doing it a bit differently. Previously, my procedure for checking out the various Ubuntu desktops on a Raspberry Pi looked something like this:

Flash a server image (go make coffee)
Boot the server image (twiddle thumbs for a minute or two while it sorts out the first time setup)
apt install lubuntu-desktop (go make more coffee)
Reboot, play with it a bit, install a bunch of stuff (yet more coffee)

To avoid the inevitable caffeine poisoning that results from repeating this procedure for Lubuntu, Xubuntu, Kubuntu, and Budgie, this time we’re going to use … drum roll please …

The (original) cloud-init logo with the words “cloud-init” adjacent. The logo is the usual Ubuntu-orange circle, within which appears a very geometrically simplified representation of the fingers of the left hand. The index finger is extended and appears to be touching the edge of a large white spark (which in no way suggests someone electrocuting themselves by touching a live surface). Or it could be a worm with a large anime hair-do. I’m not sure.

If that sounds like a bit of an anti-climax, that’s just because you haven’t played with cloud-init enough. Yes, its one-shot nature makes it painful to iterate when you get things wrong. Yes, the documentation is painfully obscure in places. However, once you get a grip on it, it’s quite astonishingly flexible. Hopefully, by reading this you can bypass some of that pain and start from a more useful point!

But before we get to the fun bits, we need to get the basics done …

Ingredients

The first step is to get a Pi to test with. You’ll need a Pi 4, Pi 400, or CM4 with at least 4GB of RAM. At the time of writing, that’s a slightly tricky proposition given the supply shortages. The vital rpilocator can help here, but it would appear (at least according to Eben, who probably knows what he’s talking about!) that the shortage should begin to clear during the second half of 2023.

The second step is to get an image we can play with. I’m going to be using the recently released Ubuntu 23.04 (Lunar Lobster) Server for Raspberry Pi image, specifically the arm64 variant. You can use rpi-imager to flash this to a spare SD card, or even SSD drive attached to a USB adapter.

Warning

A word of caution: don’t try this with Jammy (22.04). There’s a reason I’m using Lunar (23.04) for this experiment. If you don’t care about the reasons, skip to the next section. Otherwise, read on …

Historically, Ubuntu images (including the Ubuntu for Raspberry Pi images) have been built with a system called livecd-rootfs. This could charitably be described as a rough collection of dozens of hooks (and hacks) in the form of shell scripts, all of them plastered onto the already crufty base of livecd-rootfs, itself a gigantic mess of shell script.

The result is exactly the hellish nightmare of obscure regex-loaded nonsense that you might imagine. Worse still, many of the hacks inject configuration files into the resulting image that are unowned. That is to say, no Debian package owns these injected files, and that causes all sorts of fun when it comes to upgrades.

My friend William, before he left, made a valiant start on sorting out this mess by re-writing the old ubuntu-image tool. In concert with this effort, my small contribution was to start migrating hacks from livecd-rootfs into the ubuntu-settings package so that in Lunar (and beyond), as much configuration as possible would be directly under the control of the package management system (this contribution made a screeching U-turn when, at the last minute, it was decided we were going to use livecd-rootfs for lunar after all, but that’s another story!).

This is one of the major reasons it’s now much simpler to spin up a desktop image with cloud-init. Instead of having a write a ton of configuration to add all the necessary tweaks livecd-rootfs used to do, we can just include one extra package and (mostly) be done.

Preparing your kitchen

Once your SD card (or drive) is flashed, eject it, then re-insert it. Regardless of the OS you’re using (Windows, Mac OS, Linux), you should see the boot partition (labelled “system-boot”) appear [1]. We’re going to be tweaking several configuration files, specifically:

config.txt — this is the Pi bootloader configuration
cmdline.txt — this is the Linux kernel command line
user-data — this is the main cloud-init configuration file
network-config — this is the cloud-init network configuration

We’ll start with config.txt because the changes here are universal: they’ll be required no matter which desktop we’re intending to use:

Find the enable_uart=1 line (it’s on line 28 by default) and comment it out by inserting # at the start of the line. This simply disables the serial console which isn’t strictly necessary, but if the serial console is enabled, the Pi’s GPU clock is locked to low speed and obviously we want the GPU clock to float higher for our desktop.
At the end of the file add dtoverlay=vc4-kms-v3d to load the “full” KMS device-tree overlay (it should be in the [all] section at the end of the file)

Next we need to tweak the kernel command line in cmdline.txt. Again, all the changes here are common to all desktops. This file consists of a single line of text; don’t introduce any line breaks when editing it:

Delete console=serial0,115200 at the start. This just removes the tty associated with the serial console which we disabled above in config.txt.
Insert zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=zstd at the start. This activates the zswap system that we use on the desktop.

These changes more or less boil down to the base differences between the Ubuntu Pi server and desktop images. Most of the other differences (since Lunar, as discussed above) come down to the selection of packages that are installed …

Picking your flavour

It’s time to learn what we can do with cloud-init. The most obvious thing we want is to have a particular desktop package installed. This is pretty simple, and achieved by adding the following block to the user-data file on the boot partition:

packages:
  - lubuntu-desktop
  - ubuntu-raspi-settings-desktop

This tells cloud-init that, on first boot, we want it to try and install two extra packages:

lubuntu-desktop is the basis of the Lubuntu desktop images; we’ll substitute other desktop packages here later
ubuntu-raspi-settings-desktop contains all the little tweaks for the Pi desktop (ensuring there’s a swap-file, disabling suspend in Gnome, inserting the correct modules for zswap into the initrd, setting appropriate permissions for I2C and SPI interfaces, etc.)

In addition to these we should also add the following settings (these aren’t mandatory, but highly recommended):

package_update: true
package_upgrade: true
package_reboot_if_required: true

The first line tells cloud-init we want it to run apt update to refresh the set of available packages. This is the most important line as, without it, cloud-init will likely fail at some point because the apt index shipped on the image will be out of date. The next line demands that apt upgrade should be run to ensure all packages are up to date. The final line indicates that, if any package indicates a reboot is required, we should carry one out (this is absolutely the case when installing a desktop, as a display manager will be installed at the very least).

Next, while some of the desktops available install a browser by default (by depending on the Firefox apt package, which in turns installs the Firefox snap), several don’t as their official images include the Firefox (or Chromium) snap directly without the apt “wrapper”. To avoid winding up with a desktop without a browser, we’ll add another section to install the Firefox snap explicitly:

snap:
  commands:
    - snap install --channel=latest/stable/ubuntu-23.04 firefox

I’m not entirely sure why the snap section is so different from packages, or why the full snap install <package> form is required. It does appear from the documentation that the snap.commands section simply allows arbitrary commands to be specified (canonical-livepatch is included in the examples), but in that case why call it a snap section?

The exceedingly long channel name causes the snap to follow a branch of a channel that may include per-release quirks. Or … something like that. I’m still not clear on the whole “channel” concept in snaps.

Locale-ly sourced

On the official Ubuntu Desktop for Raspberry Pi images, we have a first time setup process [2] which guides the user through locale selection and initial user creation. We don’t have that here, so we’d better get cloud-init to handle that instead. Firstly the locale, timezone, and keyboard settings:

locale: en_GB.UTF-8
timezone: Europe/London
keyboard:
  model: pc105
  layout: gb
  options: ctrl:nocaps

Note

You may want to leave off that “options:” line. I include it to put Ctrl in the right place (i.e. where Caps Lock is) and because I have no need of a Caps Lock key.

To find out the valid values for these settings, you can use the following commands on an existing Ubuntu desktop (I say desktop specifically because the available locales on server installations tend to be extremely minimal) [3]:

locale:

localectl list-locales

timezone:

timedatectl list-timezones

keyboard:

model:: localectl list-x11-keymap-models
layout:: localectl list-x11-keymap-layouts
variant:: localectl list-x11-keymaps-variants [layout]
options:: localectl list-x11-keymaps-option

Adding personality

Next we need to create, and customize, the initial user. By default, cloud-init will create a user named ubuntu with a locked password, which also has password-less sudo access, and a bunch of default group memberships (for things like direct video and audio access, etc). The default group memberships are fine, but I’d like a slightly more personalized username, and one with password-required sudo rights. This can be accomplished with the following block:

user:
  name: "dave"
  gecos: "Dave Jones"
  plain_text_passwd: "raspberry"
  lock_passwd: false
  sudo: "ALL=(ALL:ALL) ALL"
  ssh_import_id:
    - lp:waveform

This changes the default username to “dave”, and uses my name in the gecos field (which traditionally recorded a whole bunch of things like display name, office number, telephone number, and so forth but these days tends to just be used for the display name). The sudo setting specifically lacks the NOPASSWD flag, requiring a password for use. The ssh_import_id setting lists users from which to import public SSH keys (use “lp:” prefix for Launchpad usernames, and “gh:” for Github).

The block also sets the initial password to the plain-text string “raspberry” and ensures password login is possible (lock_passwd is true by default which means the only way to login is with SSH or another auxiliary mechanism, but that’s not so useful on the desktop!). This is horribly insecure! We could do something slightly more secure here by providing a hashed password instead. First we’d run mkpasswd to generate the hash:

$ mkpasswd --method=SHA-512 --rounds=4096
Password:
$6$rounds=4096$JZRfK1tM.xiWZtR5$XpMvuj2reJr.....lI0T4Z/

Then we would use this with hashed_passwd instead of plain_text_passwd:

user:
  name: "dave"
  gecos: "Dave Jones"
  hashed_passwd: "$6$rounds=4096$JZRfK1tM.xiWZtR5$XpMvuj2reJr.....lI0T4Z/"
  lock_passwd: false
  sudo: "ALL=(ALL:ALL) ALL"
  ssh_import_id:
    - lp:waveform

However, don’t be lulled into a false sense of security. As the cloud-init documentation notes:

While hashed_password is better than plain_text_passwd, using [a password] in user-data represents a security risk as user-data could be accessible by third-parties depending on your cloud platform

To be clear: user-data is a world-readable file on the Ubuntu Pi images (it has to be because it’s on a FAT partition). There is no absolutely secure method of setting the password via cloud-init (at least on the Ubuntu Pi images); the secure thing to do here is login and then change your password. Consider this password initial and temporary.

Baking in hacks

Finishing off our user-data file we’re going to make one final rather hacky (but useful) change. At the time of writing, Firefox starts under the XWayland layer. This leads to rather poor performance under certain circumstances. To force it to load under Wayland directly, we’d need to set the MOZ_ENABLE_WAYLAND environment variable. Ideally we would do this only in our user’s environment, not system wide. This could be done by appending a line to our user’s ~/.profile script. However, this doesn’t exist initially (and won’t until cloud-init creates our user). Thankfully, this presents no problem!

write_files:
  - path: /home/dave/.profile
    defer: true
    append: true
    content: |
      export MOZ_ENABLE_WAYLAND=1

The powerful (but obviously slightly dangerous) write_files key allows us to write to arbitrary files on first boot. Here we specify we’d like to write to /home/dave/.profile (adjust for your custom username as necessary). We also inform cloud-init that the write should be deferred which means this will take place after user creation. Further, we specify that we wish to append (not overwrite), and finally we provide the lines we’d like to append.

Going shopping

Naturally, in order to obtain the packages we’ve requested we’re going to need some network connectivity. If your Pi is going to be connected by Ethernet, you’re good to go; the default configuration will just use your Ethernet connection. However, if you’re going to be relying on WiFi instead we need to be a bit of surgery in the last file we mentioned earlier, network-config.

Firstly we’ll delete the ethernets section, then uncomment the wifis section and specify the local access point and password:

network:
  version: 2
  wifis:
    wlan0:
      dhcp4: true
      optional: true
      access-points:
        "my-wifi-ssid":
          password: "my-wifi-password"

It is also useful to specify the regulatory-domain here (note that this needs to be specified under the wlan0 key, which is the reason for the inclusion of the parent keys below):

network:
  wifis:
    wlan0:
      regulatory-domain: GB

The valid regulatory domains can be found in the Linux kernel source Specifically, the two-letter code after the “country” line, which are almost entirely the ISO-3166 standard two-letter country codes. The special “00” domain (the “world regulatory domain”) is the default, but relying on this can lead to poor WiFi performance, particularly in 5GHz setups where many channels have restrictions in the world domain.

The first byte is with the eye

At the end of all this tinkering, you should have files that look something like the following on your boot partition (lines you may wish to pay specific attention to are highlighted):

config.txt

 1[all]
 2kernel=vmlinuz
 3cmdline=cmdline.txt
 4initramfs initrd.img followkernel
 5
 6[pi4]
 7max_framebuffers=2
 8arm_boost=1
 9
10[all]
11# Enable the audio output, I2C and SPI interfaces on the GPIO header. As these
12# parameters related to the base device-tree they must appear *before* any
13# other dtoverlay= specification
14dtparam=audio=on
15dtparam=i2c_arm=on
16dtparam=spi=on
17
18# Comment out the following line if the edges of the desktop appear outside
19# the edges of your display
20disable_overscan=1
21
22# If you have issues with audio, you may try uncommenting the following line
23# which forces the HDMI output into HDMI mode instead of DVI (which doesn't
24# support audio output)
25#hdmi_drive=2
26
27# Enable the serial pins
28#enable_uart=1
29
30# Autoload overlays for any recognized cameras or displays that are attached
31# to the CSI/DSI ports. Please note this is for libcamera support, *not* for
32# the legacy camera stack
33camera_auto_detect=1
34display_auto_detect=1
35
36# Config settings specific to arm64
37arm_64bit=1
38dtoverlay=dwc2
39
40[cm4]
41# Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into
42# such a board)
43dtoverlay=dwc2,dr_mode=host
44
45[all]
46dtoverlay=vc4-kms-v3d

cmdline.txt

1zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=zstd dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash

Please note that cmdline.txt must consist of a single line of text. Ignore any wrapping of text above.

user-data

 1#cloud-config
 2
 3hostname: lubuntu
 4
 5locale: en_GB.UTF-8
 6timezone: Europe/London
 7
 8keyboard:
 9  model: pc105
10  layout: gb
11  options: ctrl:nocaps
12
13user:
14  name: "dave"
15  lock_passwd: false
16  gecos: "Dave Jones"
17  plain_text_passwd: "raspberry"
18  sudo: "ALL=(ALL:ALL) ALL"
19  ssh_import_id:
20    - lp:waveform
21
22write_files:
23  - path: /home/dave/.profile
24    append: true
25    defer: true
26    content: |
27      export MOZ_ENABLE_WAYLAND=1
28
29package_update: true
30package_upgrade: true
31package_reboot_if_required: true
32
33packages:
34  - lubuntu-desktop
35  - ubuntu-raspi-settings-desktop
36
37snap:
38  commands:
39    - snap install firefox

You may be wondering if the ordering of these blocks matters. The answer is “no”. The ordering of actions in cloud-init does not depend on the ordering of the configuration file (and nor should it). The system is intended to be declarative; we are describing the state we wish to attain, not how to obtain it.

This is why the write_files section has options like defer which adjust the timing of these actions. By default, write_files entries will always occur early on so that written files can be available to other actions.

network-config

 1network:
 2  version: 2
 3  wifis:
 4    wlan0:
 5      regulatory-domain: GB
 6      dhcp4: true
 7      optional: true
 8      access-points:
 9        "my-wifi-ssid":
10          password: "my-wifi-password"

With all those written, safely eject your card (or SSD drive), plug it into your Pi and let it boot. This will take a … very … very … long … time. Installing an entire hierarchy of desktop packages in this manner is not a quick process, and the first boot will probably take at least 1 hour (I didn’t time every installation during this test, but the cloud-init logs for the Lubuntu run indicated setup on the microSD card took roughly 1½ hours).

However, once it’s complete, it should automatically reboot and you should find yourself at a shiny new graphical desktop login! There’s still some horror to sort out here (in particular the clash of networking configuration stacks), but we’ll deal with that next time when we’ll also take a look at comparing the current crop of desktops on Ubuntu, and see what’s changed in the last few years. We’ll also compare them to the official Gnome desktop, and learn a few more cloud-init tricks …

[1]	If you’re using Linux, you’ll also see the root partition (“writable”) appear, but you can ignore this

[2]	`oem-config`, derived from the `ubiquity` installer. There are moves afoot to replace this at some point (given `ubiquity` is being replaced with `subiquity`), but nothing concrete as yet

[3]

For those violently allergic to systemd, the equivalent commands the listing locales, timezones, and keyboard settings are:

locale:: locale -a
timezone:: (cd /usr/share/zoneinfo/posix; find -type f -or -type l -printf '%P\n')
keyboard:: Read various sections in /usr/share/X11/xkb/rules/base.lst

4K 60Hz on a Pi

2023-06-23T00:00:00+01:00

TL;DR: this is a long rambling semi-rant of the sort I occasionally go off on. If you just want to get 4k 60Hz working under Ubuntu on the Pi, skip down here.

Your humble correspondent is, dear reader, something of a luddite when it comes to display technology. On my desk is a (by modern standards) positively pedestrian 1080p monitor attached to my PC, and another one attached to my main development Pi. Not for me, a high-DPI dual- or triple-monitor setup!

I’ve never found a particular need for multiple monitors on a machine and, every time I’ve tried it, the OS in question (whether Linux or Windows) has found some novel way to screw things up, whether it be with full-screen applications, moving things between monitors, or even positioning a centered dialog box.

The more recent advances in high-DPI displays have also done little to enthuse me. My little laptop with its 11.6” screen (no, I don’t want a huge 15”+ luggable with a numeric keypad, it’s meant to be portable!) has a high-DPI 1080p screen (~190dpi). As a result, the curves and lines of fonts on the screen are beautifully smooth … and too small to be legible to my aging eyes without cranking up the display scale [1]. So when the Pi 4 was released with 4K display capability! and dual micro-HDMI ports! my interest was not exactly piqued until I got to dual USB3 ports! and more than 1GB of RAM!

Linux-oleum

Several people have commented, upon learning of my solo-screened predilections, “how do you cope with having to switch between applications? I need two screens so I can be working on one while looking something up on the other.” This is a reasonable concern. I always need a couple of windows open: the terminal I’m working in, and the web-browser I’m reading something from. But the answer has existed for a good long time, and is available in every modern OS and windowing system: tiling.

I stick my terminal on the left, my browser on the right and now, not only do I enjoy two windows together on the screen, they both have a pleasingly A4-paper-esque aspect ratio which is greatly preferable for reading [2]. Naturally, there are downsides. The sheer number of websites which purport to have a responsive design but casually fall off the right of the screen is infuriating [3].

The Unbearable Lightness of Daytime TV

Thus I remain firmly unconvinced by the benefits of having multiple monitors. My friend Ben did, however, come up with a setup which I confess I would merrily steal were it not for a distinct lack of space on my desk … buy a TV.

Now, I don’t just mean any mere kitchen-suitable model, barely capable of the 1080p of yesteryear. No! A full-throated, red-blooded, 40 diagonal inches of ultra HD 4K-ready LED TV, ready to blast screaming rainbows at your reluctant retinas!

Ben swears this is a “monitor” [4]. I can only apologise for the lack of screaming rainbows …

This makes use of 4K display outputs without producing text that requires squinting through a loupe to read. Furthermore, it doesn’t cost an arm and a leg like most multi-monitor setups (my frugality in computing extends beyond the mere computer!).

It sounds expensive, and if you’re purchasing a 40” monitor … it really is. You’ll be hard pushed to find anything at the 40” scale in monitors for less than 700 quid. If you do find something for less than that … you’re likely to find it’s actually just a TV being sold as a monitor [5]. TVs, on the other hand, can be found at this scale for well under 400 of your finest pounds sterling.

Why the price difference? Surely TVs do a lot more than a monitor? They’ve all got speakers (unlike many monitors), they’ve all got tuners, and these days at this scale they’ll all have some “smart” functionality (i.e. some rubbish menu system to run their “apps”), not to mention some IR / Bluetooth remote thingy. Is it all a gigantic con? Slap “monitor” instead of “TV” on a screen and you can instantly sell it for twice the price?

The Need for Speed

Monitors at this scale all boast of their response time, and it’s usually something under 10 milliseconds. From the signal coming out of your computer, to it being beamed to your eager eyeballs, is typically less than one frame’s duration at a 60Hz refresh rate. But for a TV it doesn’t matter at all if it takes one (or more) frame’s time between Disney+ pumping a signal down your broadband connection, to The Bob’s Burgers Movie infesting your childrens’ brains.

This (allegedly) matters for gamers.

Bob and his family look at the price of “gaming” monitor.

Being a gamer myself (though not the sort of twitchy adrenaline junkie who runs around pwning people in CoD … wandering around admiring the stark-yet-beautiful scenery in Fallout is more my speed), I suppose it ought to matter to me but I can’t say I’ve ever given it much thought. Then again, I’ve always used a monitor and not a TV so perhaps I’ve never had a display where it mattered?

But, hold fire! What about these “console” things? I’m reliably informed these are even more popular than a PC for gaming [6], yet … they typically plug into a TV. Are the inputs of console gamers all stuck several frames in the past? The lag ruining their reactions? Are they unable to compete with the might of the PC Gamer?

Erm … no. Or if they are, I certainly can’t notice it on my daughter’s Switch running Minecraft on the TV downstairs. Having said that, we’ve already established the acuity of my visual system is far from perfect. Still, the point stands. For my purposes (and thus presumably a few others): yes, it’s all a giant con and one can get a screen of twice the size for half the price simply by searching for “TV” instead of “monitor” when shopping.

This’ll Hertz Me More …

The Pi 4 is capable of running 4K at 60Hz, but it won’t do it out of the box, and it requires some gentle cajoling to get it in the mood for 60Hz operation. The reason for this is simple: 4K at 60Hz is pushing the absolute limits of the Pi 4’s display hardware and as a result there are limitations when using it.

Firstly, ensure you have an adequate cable. Specifically, you need an HDMI 2.0 or later cable in order to support 4K at 60Hz. Confusingly, HDMI 1.4 cables support the 1080p resolution at 60Hz, and the 4K resolution but only up to 30Hz. You need HDMI 2.0 (or later) cabling to support the 4K resolution at 60Hz. If, like me, you’ve been stuck in 1080p land for a while you may have amassed a small hoard of spare HDMI cables, only to find that none of them support HDMI 2.0. Sorry!

Next, use the HDMI-0 port on the Pi; the HDMI-1 port cannot be used when operating at 4K 60Hz. The HDMI-0 port is nearest the USB-C power port on the Pi 4, and nearest the micro-SD card slot on the Pi 400.

Now it’s time for some configuration changes. Firstly, add the following line to config.txt on the boot partition [7]:

hdmi_enable_4kp60=1

The line should appear under an [all] section (or I suppose more correctly a [pi4] section which would mean it only applies to a Pi 4, 400, CM4, or CM4S — however, it won’t cause any issues on older models as the bootloader will simply ignore the line on those).

Finally, reboot your Pi to apply these changes. When you’re back to your desktop you may note, if you are already in a 4K resolution, that you’re still operating at 30Hz (your mouse motion will not look smooth!). This is because you still need to tell your desktop to switch to the higher refresh rate. In Gnome, this is as simple as going to Settings, selecting “Screen Display” on the left, and then picking “60Hz” under the “Refresh Rate” option.

After your screen does its mode-switching dance, you should have buttery smooth mouse motion!

No hotel TV is safe …

Be warned that 4K at 60Hz is not exactly a paradise on the Pi. This resolution requires significantly more memory for many things and, in particular, I’ve found that several things cannot operate full-screen. Notably, Firefox crashed quite painfully when I was attempting to test it playing a video full-screen. However, when Firefox was forced to run purely under Wayland [8] (as opposed to the XWayland server it defaults to currently), it did succeed in working (not smoothly, but at least it didn’t burst into flames).

The Pi will also run hotter and demand more power at this resolution and framerate. If you experience crashes, double-check you have a good power supply that isn’t dropping voltage at higher loads (even if you’ve never had problems before, that’s no guarantee you won’t with this configuration).

That’s all for this time! Next, I’ll be having a look at spinning up a custom desktop from cloud-init.

[1]	thank goodness Wayland makes fractional scaling on Gnome fast; 200% was too big, but 150% is perfect

[2]	have you noticed how many websites limit their horizontal width? There’s a reason for this …

[3]	GitHub recently fixed this after being broken for nearly a decade, meanwhile Google now frequently falls off the right after spending the last decade being perfect … oh well, you win some, you lose some

[4]	… with a remote

[5]	Oh look, it’s a “monitor” with … a remote for Netflix, Disney+ and Prime Video. Amazon, you really are the new flea-bay

[6]	How this came to be when they lack even the most basic sort of mouse I have no idea

[7]	This is typically mounted on `/boot/firmware` on Ubuntu, or `/boot` under RaspiOS

[8]	This can be done by adding `export MOZ_ENABLE_WAYLAND=1` to your `~/.profile` script and re-logging in. Hopefully this mode will become the default in the near future, as Firefox operates visibly more smoothly under pure Wayland

Playing with Blocks - ReLUKS

2023-04-20T00:00:00+01:00

See Also: LUKS, LVM+LUKS.

We’ve looked at using LUKS on the Ubuntu Pi desktop images before but it was a rather painful process primarily as it involved shrinking the root partition. This in turn meant that a separate machine (or at the very least a Pi booting from something other than its SD card) was required to perform the changes.

Can we do better? In particular, can we reduce the requirements to just a single machine (the target Raspberry Pi) with a single storage medium? Thanks to a relatively minor change in the just-released Ubuntu Lunar (23.04) Pi desktop image, I think we can!

When you wanna go do it

You’ll need the following for any scenario:

a Raspberry Pi 4 or 400
a micro-SD card (or USB-attached boot drive)
a monitor
a keyboard and a mouse

If you’re trying to do this on a single machine (not flashing your storage device on a separate machine), you’ll also need:

a relatively recent boot firmware
an Ethernet connection

Newly purchased Pis (as of 2023) should all have sufficiently up to date firmwares. Older Pis may require an EEPROM update which can be accomplished under Ubuntu (or RaspiOS) by running sudo rpi-eeprom-update -a and rebooting if an update is shown as available.

Hit me with those laser beams

Start up your Raspberry Pi. The monitor should show the red and white Raspberry Pi bootloader screen, with the prompt “Press and hold <SHIFT> key to stop boot and start net install”. Follow this instruction and it should change to “If not done so already, insert ethernet cable” (this capability only works over Ethernet, not WiFi as the bootloader can only configure the former).

After a short download, you should find yourself at a full-screen version of the rpi-imager. From here you can select Ubuntu as you normally would in the desktop version of the software:

Click “Choose OS”
Select “Other general-purpose OS”
Select “Ubuntu”
Select “Ubuntu Desktop 23.04 (64-bit)”
Back at the main screen, click “Choose Storage” and pick your boot medium (which should be the only drive connected to the Pi at this point)
Click “Write”

The utility should prompt to ensure you want to overwrite everything and anything on the storage device and, once confirmed, you should find it writing to the storage medium.

Note

Do not be alarmed if there is an extremely long pause in the writing process. The Ubuntu images are heavily compressed, and the “slack space” baked into the image (which may be excessive) compresses extremely well. This can lead to a potentially long pause in the progress bar while a few bytes in the compressed image expand into gigabytes of written data.

Make making it your intention

Once the OS is written to the boot medium (and verified), the Pi should reboot into Ubuntu Desktop. You’ll need to run through the initial setup wizard, configuring your locale and user details.

After the initial setup wizard, login to your new user. We now need to take a series of steps which may seem premature and in some cases a bit odd. Start a terminal and perform the following:

$ sudo -i
# apt install cryptsetup-initramfs
# echo "rootfs  $(findmnt -n -o SOURCE /)  none  luks,discard" >> /etc/crypttab
# sed -i -e 's,^LABEL=writable,/dev/mapper/rootfs,' /etc/fstab
# sed -i -e 's,LABEL=writable,/dev/mapper/rootfs,' /boot/firmware/cmdline.txt
# reboot

In order, these commands:

Switch to the root user (everything we’re going to do requires root access and I can’t be bothered re-typing sudo).
Re-install cryptsetup. I say “re-” because this package is actually part of the base image, but the initial setup decides it isn’t required and removes it at the end of the setup. Doh!
Create the crypttab entry for the rootfs.
Re-write the source of the root mount in fstab.
… and the same in the kernel command line.
Reboot the machine.

If you want to double check things, assuming your boot device is an SD card (/dev/mmcblk0) your crypttab, fstab and cmdline.txt files should look like this:

root@miss-piggy:~# cat /etc/crypttab
# <target name> <source device>         <key file>      <options>
rootfs  /dev/mmcblk0p2  none  luks,discard
root@miss-piggy:~# cat /etc/fstab
/dev/mapper/rootfs      /       ext4    discard 0       1
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1
root@miss-piggy:~# cat /boot/firmware/cmdline.txt
zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=zstd dwc_otg.lpm_enable=0 console=tty1 root=/dev/mapper/rootfs rootfstype=ext4 rootwait fixrtc quiet splash

Hit me, hit me!

At this point you may notice that the boot seems to be taking rather a long time. In fact, it’s going to fail because the rootfs device we told the kernel to go looking for doesn’t exist yet. But that’s okay! Be patient and let it fail; the initramfs still gets loaded and it’s got all the tools we need to create it.

Once you find yourself at a command prompt (with a notice complaining that it couldn’t mount the root file-system), do the following:

(initramfs) rootdev=$(readlink -f /dev/disk/by-label/writable)
(initramfs) cryptsetup reencrypt --encrypt --reduce-device-size 16M --cipher xchacha12,aes-adiantum-plain64 $rootdev
(initramfs) cryptsetup open $rootdev rootfs
(initramfs) exit

Create the rootfs device. This will prompt you to enter “YES” (for confirmation you really want to do this), and for the password to unlock the encryption key for the device (twice); don’t forget this!.
Go make coffee; or twiddle your thumbs for an excessively long period of time while it encrypts the entire device, including all the empty space. On the plus side, after a minute it should give you a reasonably accurate estimate of how long it’s going to take to encrypt the device, so you can set a timer and come back when it’s done.
Open the newly created rootfs device (this will prompt you once more for the password) and exit to continue the boot.

Re-LUKS

Once you’re logged into the desktop once more, you have one final task. Re-generate the initramfs now that the rootfs device exists:

$ sudo update-initramfs -u

Done! At this point all future boots should prompt you for the password to unlock the encrypted root file-system.

Obviously in an ideal world, this could all be accomplished from the installer, but there’s a slight wrinkle in that with the lunar release. The first time setup application on the Pi desktop images is oem-config which is derived from the venerable ubiquity installer. As you may have noticed in the lunar release notes the installer is being replaced by a new one based on subiquity.

However, this only applies to those platforms which need the “full fat” installer, i.e. the PC where you boot off some medium and then install to another (the internal drive). It doesn’t apply to Ubuntu’s pre-installed desktop images (like the Pi ones) yet. There seemed little point in pouring effort into adding this to ubiquity this cycle, only to see it thrown out during the next cycle when the subiquity based installer comes to replace oem-config.

Hopefully in a future cycle we can integrate all this and have a slightly friendlier experience, but for now at least it’s achievable without requiring a separate machine.

Wherefore UART Thou?

2022-10-21T00:00:00+01:00

MicroPython is one of the things in modern computing that I still find somewhat miraculous and slightly surreal. The idea that a microcontroller, i.e. the sort of processor I grew up using, could run something as heavyweight as an interpreted language like Python, and could do it sufficiently fast as to be genuinely useful still amazes me to some degree.

Of course, my judgment is faulty: the RP2040 is a positive behemoth compared to the Z80 that powered many of the machines of my youth. Suffice it to say, I love playing around with MicroPython on micro-controllers, particularly on the Raspberry Pi Pico, and various boards using the Espressif ESP32. This post covers some of the ways I work with MicroPython from Ubuntu, and some of the recent additions to the Ubuntu archive that ease this.

Grab a favoured beverage, and settle down with your Pico to “play along” …

All things are ready, if our packages be so

Pretty much all communication with MicroPython is done via serial link. In more advanced cases this involves hooking up wires to the serial pins on the Pico itself, but that’s not the focus of this post. We’re just going to take the easy route of talking to a serial port emulated over the Micro-USB connection. The things you’ll need to follow along:

A computer running Ubuntu 22.10 (Kinetic). This can be a PC, or a Raspberry Pi. Basically anything with a USB port.
A Raspberry Pi Pico. This can be the Pico or Pico W; the examples assume the latter but will work equally well on the former (I’ll touch on the W’s WiFi facilities in a future post). For that matter, these examples should be easily adaptable to any other RP2040 or ESP32-based boards.
A micro-USB cable to connect your Ubuntu computer to your Pico.

We start by installing a MicroPython firmware on your Pico. The Raspberry Pi MicroPython documentation covers this perfectly but a quick overview would be:

Download the firmware (a file with a “.uf2” extension) for your specific board; the Pico and Pico W have separate builds.
Hold down the BOOTSEL button (the only button) on your Pico, while connecting it to your computer with the micro-USB cable.
Release the BOOTSEL button once it’s connected.
After a few seconds you should see a USB storage drive called “RPI-RP2” appear on your computer. Open this device in your file explorer.
Copy the “.uf2” file you downloaded over to the RPI-RP2 drive.
After a few seconds the “RPI-RP2” drive should disappear; this indicates the Pico has flashed the firmware and rebooted into the MicroPython prompt.

Now that the Pico’s set, let’s install some new toys on your Ubuntu machine. The rshell tool was new in jammy (22.04), but mpremote is new to kinetic (22.10):

$ sudo apt install pyboard-rshell micropython-mpremote

You may be wondering why I’ve excluded “thonny” here. Partly that’s because the Raspberry Pi documentation covers it perfectly already. However, it’s also because these two command line tools have some interesting facilities unique to them, and they’re how I tend to work with MicroPython.

The fault … is not in our cable, but in our file modes

Both these tools communicate over a serial connection with MicroPython running on the microcontroller, so the first thing we need to establish is … which serial device? The vast majority of the time, it will be /dev/ttyACM0 but you can double check this by starting a terminal window and running:

$ sudo dmesg -w

This will dump the kernel ring buffer (basically the kernel log) and “follow” it, printing out new messages as they occur. Now plug the Pico back in again, and you should see something like the following appear:

[24893.432219] usb 1-2.4: new full-speed USB device number 13 using xhci_hcd
[24893.759502] usb 1-2.4: New USB device found, idVendor=2e8a, idProduct=0005, bcdDevice= 1.00
[24893.759506] usb 1-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[24893.759508] usb 1-2.4: Product: Board in FS mode
[24893.759509] usb 1-2.4: Manufacturer: MicroPython
[24893.759510] usb 1-2.4: SerialNumber: 1234567890abcdef
[24893.778722] cdc_acm 1-2.4:1.0: ttyACM0: USB ACM device

That last line is the important one, indicating that the new serial device is called ttyACM0, so the device path is simply /dev/ttyACM0. Next we should check that your user has access to this serial device:

$ ls -l /dev/ttyACM0
crw-rw---- 1 root dialout 166, 0 Oct 19 16:01 /dev/ttyACM0
$ groups
ubuntu adm dialout cdrom floppy sudo audio dip video plugdev netdev lxd

Note that the serial device belongs to user root and group dialout, and that our user also belongs to the dialout group. If you have any issues talking to MicroPython, this is the first thing to check. On server Ubuntu images, the default user belongs to the dialout group but sadly this isn’t the case on Ubuntu desktop images (LP: #1923363). If your user does not belong to the dialout group, run the following:

$ sudo adduser $USER dialout

Afterwards, log out and log back in again (group membership changes are only evaluated at login time).

… and make a heaven of rshell

The first thing to try is just to get to the MicroPython REPL (the interactive prompt). We’ll export RSHELL_PORT so we don’t have to continually specify the port on the command line (if your port is /dev/ttyACM0, that’s the default anyway but it never hurts to be certain):

$ export RSHELL_PORT=/dev/ttyACM0
$ rshell repl
Using buffer-size of 32
Connecting to /dev/ttyACM0 (buffer-size 32)...
Trying to connect to REPL  connected
Retrieving sysname ... rp2
Testing if ubinascii.unhexlify exists ... Y
Retrieving root directories ...
Setting time ... Oct 19, 2022 17:42:43
Evaluating board_name ... pyboard
Retrieving time epoch ... Jan 01, 1970
Entering REPL. Use Control-X to exit.
>
MicroPython v1.19.1 on 2022-10-14; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
>>>
>>> import sys
>>> sys.version_info
(3, 4, 0)

Several things to note here. Firstly, the message “Entering REPL. Use Control-X to exit”. Now you know how to get back to your command line! Secondly, that the current version of MicroPython is (mostly) compatible with the Python 3.4 language specification. Actually, it includes some features from later versions too (like f-strings from version 3.6), but these aren’t feature-complete yet:

>>> f"Compatible with Python {'.'.join(str(i) for i in sys.version_info)}"
'Compatible with Python 3.4.0'

Now, hit Ctrl-X to quit the MicroPython REPL, and let’s explore the Pico’s file-system. rshell implements a number of common file operations including ls, cp, rm, and cat but the most important detail is that it considers the Pico’s file-system to exist under the path /pyboard/. If we list that path currently we’ll find it’s empty:

$ rshell --quiet ls /pyboard/

Note

We used --quiet to suppress all that “connecting” gubbins that rshell tends to spit out.

Let’s create a basic script for MicroPython to run on startup, copy it to the board and run it on boot. The script we’ll create is the hardware equivalent of the classic “Hello, world!” script: blinking an LED. Fire up your favourite editor and enter the following script:

main.py

1 from machine import Pin
2 from time import sleep
3 
4 led = Pin('LED', Pin.OUT)
5 while True:
6     led.toggle()
7     sleep(1)

This script should work on a Pico W, but on a Pico the built-in LED is wired slightly differently. If you’ve got a Pico, change 'LED' to 25 (which is the GPIO pin the internal LED is connected to).

Save this script as main.py and then copy it to your connected Pico like so:

$ rshell --quiet cp main.py /pyboard/
Copying '/home/dave/main.py' to '/pyboard/main.py' ...
$ rshell --quiet ls /pyboard/
main.py

So far, so good, but nothing seems to be happening yet. That’s because we need to reset the Pico to get the script running. There’s a few ways to do this: the simplest is to unplug and then re-connect the Pico. That works but is rather unsatisfying as it also reset the serial connection which doesn’t help with debugging. The other method is to start the REPL and reset MicroPython with the common Ctrl+D shortcut:

$ rshell --quiet repl
Entering REPL. Use Control-X to exit.
>
MicroPython v1.19.1 on 2022-10-14; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
>>>
>>> ^D
MPY: soft reboot

The ^D above indicates where I pressed Ctrl+D. At this point your LED should be happily blinking away happily but … what happened to our REPL? It’s still there, but it’s sat in our while loop. In other words, we won’t get our prompt back until we break out of the loop. Press Ctrl+C to do so and you should get back to the regular MicroPython prompt:

^C
Traceback (most recent call last):
  File "main.py", line 7, in <module>
KeyboardInterrupt:
MicroPython v1.19.1 on 2022-10-14; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
>>>

Of course, having done that our program has also stopped (the LED won’t be blinking any more), but usefully the REPL is still running in the context of our (terminated) main.py script. In other words, we can poke around its globals and even manipulate things:

>>> led
Pin(WL_GPIO0, mode=OUT)
>>> led.off()
>>> led.on()

Generally speaking, this is how I handle the vast majority of my debugging needs in MicroPython. Either a few carefully inserted print statements in my program (which can then be watched from the REPL), or hitting Ctrl+C at some appropriate point and poking around the program’s state are generally sufficient.

How about debugging a more complex example? The following is a simple script that blinks the LED with a morse-code message:

main.py

 1 from machine import Pin
 2 from time import sleep
 3 
 4 def morse(msg, dit=0.1):
 5     dah = dit * 3
 6     intra_char = dit
 7     inter_char = dah
 8     inter_word = dit * 7
 9     codes = {'S': '...', 'O': '---'}
10 
11     for word in msg.split():
12         sleep(inter_word)
13         for char in word.upper():
14             sleep(inter_char)
15             for bit in codes[char]:
16                 sleep(intra_char)
17                 led.on()
18                 if bit == '.':
19                     sleep(dit)
20                 elif bit == '-':
21                     sleep(dah)
22                 else:
23                     assert False, 'invalid morse char!'
24                 led.off()
25 
26 led = Pin('LED', Pin.OUT)
27 morse('SOS')
28 morse('PICO')

If we upload this as main.py and run it, we find the second call to morse will fail because we haven’t got enough characters in our codes dictionary:

$ rshell --quiet cp main.py /pyboard/
Copying '/home/dave/main.py' to '/pyboard/main.py' ...
$ rshell --quiet repl
Entering REPL. Use Control-X to exit.
>
MicroPython v1.19.1 on 2022-10-14; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
>>>
>>> ^D
MPY: soft reboot
Traceback (most recent call last):
  File "main.py", line 28, in <module>
  File "main.py", line 15, in morse
KeyError: P
MicroPython v1.19.1 on 2022-10-14; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
>>>

How could we determine this if we didn’t understand the exception that’s thrown? The experienced Python user may be tempted to throw something like print(repr(locals())) in the morse function to dump the local variables after they’re set up. However, MicroPython has some subtle differences to CPython and one of them is that locals doesn’t really work.

This is one of the reasons a lot of MicroPython scripts use more globals than might typically be considered “good practice” in regular Python. Another is that MicroPython scripts (necessarily) tend to be a lot simpler, often fitting in a single module so spilling state into globals isn’t quite such an egregious habit there.

Hence, in this case if I were confused about the KeyError I might move most of the locals in morse into the global namespace and dump the state from within the function. In this case the top of the updated script looks something like this:

main.py

 1 from machine import Pin
 2 from time import sleep
 3 from logging import fatal, info
 4 
 5 dit = 0.1
 6 dah = dit * 3
 7 intra_char = dit
 8 inter_char = dah
 9 inter_word = dit * 7
10 codes = {'S': '...', 'O': '---'}
11 
12 def morse(msg):
13     for word in msg.split():
14         sleep(inter_word)

At this point, I could query codes from the REPL after the failure and probably figure things out. This approach can even be advantageous to your script anyway as it guarantees that these variables aren’t being recalculated on each run of morse (programming for a micro-controller sometimes requires a somewhat different mindset to programming for a “full-blown” computer).

Under baud’s heavy burden do I rsync

What about investigation of something “after the fact”, when the Pico’s not plugged into a serial console? In this case there’s two methods I tend to turn to:

Take a leaf out of Linux’s book: log stuff to text files!
Take a hint from the Pi’s bootloader (and the script above): blink the LED to indicate different error conditions

We have to be bit careful with the first point: we don’t have the practically infinite logging space that we’d have on a PC. For that reason I tend to have my error logging routines start their file from scratch on each run. I also combine a fatal routine (which should log an exception) with an infinite blinking routine.

The following is a slightly cut-down version of a logging.py module I often use:

logging.py

 1 import os
 2 import io
 3 import sys
 4 import time
 5 from machine import Pin
 6 from time import sleep
 7 
 8 log_filename = 'log.txt'
 9 log_file = None
10 
11 def format_msg(level, msg, ts=None):
12     if ts is None:
13         ts = time.localtime()
14     return f'{ts[0]:04d}-{ts[1]:02d}-{ts[2]:02d}T{ts[3]:02d}:{ts[4]:02d}:{ts[5]:02d} {level:-8s} {msg}'
15 
16 def log(level, msg):
17     global log_file
18     if log_file is None:
19         log_file = open(log_filename, 'w')
20     s = format_msg(level, msg)
21     print(s)
22     log_file.write(s)
23     log_file.write('\n')
24 
25 def debug(msg):
26     log('debug', msg)
27 
28 def info(msg):
29     log('info', msg)
30 
31 def warn(msg):
32     log('warning', msg)
33 
34 def error(msg):
35     log('error', msg)
36 
37 def exception(exc):
38     with io.StringIO() as buf:
39         sys.print_exception(exc, buf)
40         buf.seek(0)
41         for line in buf:
42             log('error', line.rstrip())
43 
44 def fatal(exc, blink=3, pin=None):
45     exception(exc)
46     if pin is None:
47         machine = os.uname().machine
48         if 'Pico W' in machine:
49             pin = 'LED'
50         elif 'Pico' in machine:
51             pin = 25
52         else:
53             error(f'Cannot blink: unknown board {machine}')
54     if pin is not None:
55         led = Pin(pin, Pin.OUT)
56         while True:
57             for i in range(count):
58                 led.on()
59                 sleep(0.2)
60                 led.off()
61                 sleep(0.2)
62             sleep(1)

Note

There is an official port of the CPython logging module for MicroPython but personally I consider that overkill for most MicroPython logging needs.

And here’s our updated morse script (still broken!) with some sensible logging calls, and a catch-all at the end which will dump the (otherwise unhandled) exception to the log file and enter an infinite blinking loop to warn us that something’s gone horribly wrong:

main.py

 1 from machine import Pin
 2 from time import sleep
 3 
 4 import logging
 5 
 6 dit = 0.1
 7 dah = dit * 3
 8 intra_char = dit
 9 inter_char = dah
10 inter_word = dit * 7
11 codes = {'S': '...', 'O': '---'}
12 
13 def morse(msg):
14     logging.info(f'Blinking {repr(msg)}')
15     for word in msg.split():
16         sleep(inter_word)
17         for char in word.upper():
18             sleep(inter_char)
19             for bit in codes[char]:
20                 sleep(intra_char)
21                 led.on()
22                 if bit == '.':
23                     sleep(dit)
24                 elif bit == '-':
25                     sleep(dah)
26                 else:
27                     assert False, 'invalid morse char!'
28                 led.off()
29 
30 try:
31     led = Pin('LED', Pin.OUT)
32     morse('SOS')
33     morse('PICO')
34 except Exception as exc:
35     logging.fatal(exc, blink=3)

We’re now getting into multiple files that need to be kept up to date between our computer and the Pico. We can’t exactly run git on the Pico, but rshell still has some tricks up its sleeve to make maintaining this a bit easier in the form of a (very basic) rsync implementation:

$ rshell --quiet rsync . /pyboard/
Checking /pyboard/main.py
Checking /pyboard/logging.py
/home/dave/projects/home/pico/morse/logging.py is newer than /pyboard/logging.py - copying

This makes development cycles of Pico setups with multiple files much faster. Unfortunately there’s currently an issue with this when the host’s clock isn’t set to UTC. Personally I just set my Pi’s clock to UTC to work around this, but I should take a proper look when I get a second!

Now, once we reset the board we get the initial “SOS” blink, then a steady triple-blink on the internal LED when things have failed, and we can query the log file like so:

$ rshell --quiet cat /pyboard/log.txt
2022-10-19T22:58:24 info     Blinking 'SOS'
2022-10-19T22:58:28 info     Blinking 'PICO'
2022-10-19T22:58:29 error    Traceback (most recent call last):
2022-10-19T22:58:29 error      File "main.py", line 33, in <module>
2022-10-19T22:58:29 error      File "main.py", line 19, in morse
2022-10-19T22:58:29 error    KeyError: P

We could change the final except clause to blink differently for differing exceptions, e.g. 4 blinks for a socket issue, 3 blinks for other IO errors, 2 blinks for all other exceptions.

That’s enough on rshell for now. The package includes a full man-page, rshell(1), which I would strongly recommend reading for all the other features it includes.

Mount, mount, my soul!

This brings us to the new stuff in Ubuntu Kinetic (22.10): the mpremote utility. Unlike rshell which is a third-party tool (though still one of my favourites for interfacing with MicroPython), mpremote is straight from the MicroPython developers themselves.

The first thing to deal with is how we specify the serial port for the MicroPython board in mpremote. The full syntax is connect <device>, for example:

$ mpremote connect /dev/ttyACM0 repl
Connected to MicroPython at /dev/ttyACM0
Use Ctrl-] to exit this shell

>>>

Note

The exit sequence under mpremote is Ctrl+[ rather than Ctrl+X.

However, this is a bit of a mouthful (fingerful?) to type so mpremote ships with a whole load of built-in aliases for common serial devices:

$ mpremote --help
mpremote -- MicroPython remote control
See https://docs.micropython.org/en/latest/reference/mpremote.html

List of commands:
  connect     connect to given device
  disconnect  disconnect current device
  edit        edit files on the device
  eval        evaluate and print the string
  exec        execute the string
  fs          execute filesystem commands on the device
  help        print help and exit
  mip         install packages from micropython-lib or third-party sources
  mount       mount local directory on device
  repl        connect to given device
  resume      resume a previous mpremote session (will not auto soft-reset)
  run         run the given local script
  soft-reset  perform a soft-reset of the device
  umount      unmount the local directory
  version     print version and exit

List of shortcuts:
  --help
  --version
  a0          connect to serial port "/dev/ttyACM0"
  a1          connect to serial port "/dev/ttyACM1"
  a2          connect to serial port "/dev/ttyACM2"
  a3          connect to serial port "/dev/ttyACM3"
  bootloader  make the device enter its bootloader
  c0          connect to serial port "COM0"
  c1          connect to serial port "COM1"
  c2          connect to serial port "COM2"
  c3          connect to serial port "COM3"
  cat
  cp
  devs        list available serial ports
  df
  ls
  mkdir
  reset       reset the device after delay
  rm
  rmdir
  setrtc
  touch
  u0          connect to serial port "/dev/ttyUSB0"
  u1          connect to serial port "/dev/ttyUSB1"
  u2          connect to serial port "/dev/ttyUSB2"
  u3          connect to serial port "/dev/ttyUSB3"

Here we can see that a0 is an alias for connect /dev/ttyACM0 so we can use this instead:

$ mpremote a0 repl
Connected to MicroPython at /dev/ttyACM0
Use Ctrl-] to exit this shell

>>>

The mpremote utility includes many of the same facilities as rshell like ls, cp, rm, and cat but with a different idea of the location of the remote file-system. Unlike rshell which pretends the Pico is mounted under /pyboard/, mpremote prefixes MicroPython paths with a colon:

$ mpremote a0 ls :
ls :
     327 log.txt
    1387 logging.py
     807 main.py
$ mpremote a0 cp *.py :
cp logging.py :
cp main.py :
$ mpremote a0 cat :log.txt
2022-10-19T22:58:24 info     Blinking 'SOS'
2022-10-19T22:58:28 info     Blinking 'PICO'
2022-10-19T22:58:29 error    Traceback (most recent call last):
2022-10-19T22:58:29 error      File "main.py", line 33, in <module>
2022-10-19T22:58:29 error      File "main.py", line 19, in morse
2022-10-19T22:58:29 error    KeyError: P

However, there’s one particular facility that’s unique to mpremote: the mount sub-command. This allows you to mount a directory on your computer on the MicroPython board under the /remote directory:

$ mpremote a0 mount .
Local directory . is mounted at /remote
Connected to MicroPython at /dev/ttyACM0
Use Ctrl-] to exit this shell
>
MicroPython v1.19.1 on 2022-10-14; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
>>> import os
>>> os.listdir('.')
['main.py', 'logging.py']
>>> os.listdir('/')
['remote', 'log.txt', 'logging.py', 'main.py']

Note

The “mount” only persists as long as mpremote is running.

How does this help us? You can edit your MicroPython code on your computer, and have your MicroPython board run it straight from there without constantly copying stuff back and forth! When you run mount . the MicroPython REPL that’s launched is already in the /remote path (as you can see in the snippet above where listing “.” shows only main.py and logging.py.

In this case we almost certainly don’t want the MicroPython interpreter running a main.py script when it starts; we just want it to drop to the REPL and let us run our script manually. So the first thing to do is remove :main.py which is the copy of main.py on the board. Then we’ll mount our directory on the board, and manually run our script by importing main. As expected, it’ll blink SOS, then fail but afterwards (once we kill the script with Ctrl+C, then exit the REPL with Ctrl+]) we should have a “log.txt” on the computer:

$ mpremote a0 rm main.py
rm :main.py
$ mpremote a0 ls :
ls :
     327 log.txt
    1394 logging.py
$ ls
logging.py  main.py
$ mpremote a0 mount .
Local directory . is mounted at /remote
Connected to MicroPython at /dev/ttyACM0
Use Ctrl-] to exit this shell
>
MicroPython v1.19.1 on 2022-10-14; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
>>> import main
2022-10-21T10:10:45 info     Blinking 'SOS'
2022-10-21T10:10:50 info     Blinking 'PICO'
2022-10-21T10:10:51 error    Traceback (most recent call last):
2022-10-21T10:10:51 error      File "main.py", line 33, in <module>
2022-10-21T10:10:51 error      File "main.py", line 19, in morse
2022-10-21T10:10:51 error    KeyError: P
^C
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "main.py", line 35, in <module>
  File "logging.py", line 59, in fatal
KeyboardInterrupt:
>>> ^]
$ ls
logging.py  log.txt  main.py
$ cat log.txt
2022-10-21T10:10:45 info     Blinking 'SOS'
2022-10-21T10:10:50 info     Blinking 'PICO'
2022-10-21T10:10:51 error    Traceback (most recent call last):
2022-10-21T10:10:51 error      File "main.py", line 33, in <module>
2022-10-21T10:10:51 error      File "main.py", line 19, in morse
2022-10-21T10:10:51 error    KeyError: P

We could now edit our scripts on the computer (in another terminal) and then immediately retry them on the board without any copying!

Warning

One word of caution here: the remote mount effectively performs file-system operations over the UART. Mostly this is not significantly different but there are some cases where performance or behaviour is substantially changed. For instance, lots of small reads or writes are likely to be slower than on the local flash storage, and os.statvfs currently raises an exception when called on the remote mount.

Exit, pursued by Baudelaire

That’s quite enough waffling from me! I blame the editor (who he? —Ed).

There’s plenty more detail in the man-pages of both rshell and mpremote, and an absolute ton of MicroPython capable boards out there to explore, not to mention the myriad things you can attach to them.

Have fun!

Hyping pixels

2022-10-18T00:00:00+01:00

One of the features I’ve been taking a look at in the Ubuntu Kinetic cycle is getting as many different display types working out of the box (or as nearly out of the box as is reasonable) on Ubuntu. The official Raspberry Pi touch display is still a work in progress (it’s working with the FKMS overlay, but that’s not sustainable for the future, and there’s still issues using it with KMS under Ubuntu).

There’s some interesting e-Ink displays which I’m also working on, but this post is about Pimoroni’s awesome little Hyperpixel range of displays.

Pinhead hasn’t quite understood “using all the pins”

In the beginning

In which Dave, yet again, buries the lede …

Some may be aware that the graphics stack on the Raspberry Pi has undergone some upheaval of late. For those not following along, things are quite rapidly (by the standards of these things) moving from the original (largely closed-source) stack to an open one.

In the old stack (referred to as “fkms” which, ever so slightly confusingly, means “fake” KMS) most graphical operations were handled by a closed-source firmware, running on the GPU which Linux interacted with via the VC4 “mailbox”. In the new stack (referred to as “full” KMS, but the “full” is definitely not “fake” so it’s just “kms” for short), the Linux kernel is in charge of stuff directly.

Ultimately the idea is that, without all the special cases sloshing around in closed-source blobs stuff should Just Work™. Naturally, the real world isn’t quite so simple but in the case of the Hyperpixel it really is pretty damned close to this. Or at least, it turned out to be considerably easier than I expected!

Darkness moving over the water

The nicest thing about the new stack is that the Hyperpixel can be treated as a regular display, and there are no extra dependencies. In other words, it can be made to work from first boot … sort of. We’ll get into the short-comings a bit further on, but for now here’s what you do to get going.

The Hyperpixel uses DPI (Display Parallel Interface) to receive its imagery. Unfortunately this requires a lot of pins. Nearly all the GPIO pins in fact.

The default Ubuntu configuration reserves some of the GPIO pins for interfaces like I²C, SPI, and serial UART. We need to disable all that so the display can use them instead:

We’ll start with a freshly flashed, unbooted card containing Ubuntu 22.10 (Kinetic). Either the desktop or the server version is fine.
Stick the card in another machine (running any OS) with an SD card reader so we can edit the configuration.
Open config.txt on the boot partition (which will be the only partition that shows up if you’re doing this on Mac OS X or Windows, or the one labelled “system-boot” otherwise)
Disable the I²C and SPI interfaces. This is simply a matter of deleting the following lines (or placing a “#” in front of them to comment them out):
```
dtparam=i2c_arm=on
dtparam=spi=on
```
If you are using the desktop image, skip to step 8.
Delete (or comment out) the line enabling the serial console. Again, this uses pins that the Hyperpixel needs:
```
enable_uart=1
```
Open cmdline.txt and remove console=serial0,115200 from the start of the line (you cannot comment out bits of the kernel command line, so just delete this bit). This is the only change required in cmdline.txt, so close it and switch back to config.txt.
Add the following line at the end of the file:
```
dtoverlay=vc4-kms-v3d
```
If you have a rectangular Hyperpixel display, add the following line at the end of the file:
```
dtoverlay=vc4-kms-dpi-hyperpixel4
```
If you have a square Hyperpixel display, add the following line instead:
```
dtoverlay=vc4-kms-dpi-hyperpixel4sq
```
If you need to rotate the display, add the appropriate parameters; the rotate parameter rotates the display clockwise, while the touchscreen parameters transpose or invert the coordinates reported by the touchscreen which allows you to configure an equivalent rotation:
No rotation

For non-square Hyperpixel displays this will be a portrait orientation with the top of the display by the USB and Ethernet ports (if any). No extra configuration required.

90° rotation
For non-square Hyperpixel displays this will be a landscape orientation with the left of the display by the USB and Ethernet ports (if any). Add the following lines:
```
dtparam=rotate=90
dtparam=touchscreen-swapped-x-y,touchscreen-inverted-y
```
180° rotation
For non-square Hyperpixel displays this will be a portrait orientation with the bottom of the display by the USB and Ethernet ports (if any). Add the following lines:
```
dtparam=rotate=180
dtparam=touchscreen-inverted-x,touchscreen-inverted-y
```
270° rotation
For non-square Hyperpixel displays this will be a portrait orientation with the right of the display by the USB and Ethernet ports (if any). Add the following lines:
```
dtparam=rotate=270
dtparam=touchscreen-swapped-x-y,touchscreen-inverted-x
```

For example, if you started with an Ubuntu Server image, a rectangular Hyperpixel, and you want 180° rotation, you should wind up with a config.txt that looks like this:

config.txt

 1[all]
 2kernel=vmlinuz
 3cmdline=cmdline.txt
 4initramfs initrd.img followkernel
 5
 6[pi4]
 7max_framebuffers=2
 8arm_boost=1
 9
10[all]
11# Enable the audio output, I2C and SPI interfaces on the GPIO header. As these
12# parameters related to the base device-tree they must appear *before* any
13# other dtoverlay= specification
14dtparam=audio=on
15#dtparam=i2c_arm=on
16#dtparam=spi=on
17
18# Comment out the following line if the edges of the desktop appear outside
19# the edges of your display
20disable_overscan=1
21
22# If you have issues with audio, you may try uncommenting the following line
23# which forces the HDMI output into HDMI mode instead of DVI (which doesn't
24# support audio output)
25#hdmi_drive=2
26
27# Enable the serial pins
28#enable_uart=1
29
30# Autoload overlays for any recognized cameras or displays that are attached
31# to the CSI/DSI ports. Please note this is for libcamera support, *not* for
32# the legacy camera stack
33camera_auto_detect=1
34display_auto_detect=1
35
36# Config settings specific to arm64
37arm_64bit=1
38dtoverlay=dwc2
39
40[cm4]
41# Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into
42# such a board)
43dtoverlay=dwc2,dr_mode=host
44
45[all]
46dtoverlay=vc4-kms-v3d
47dtoverlay=vc4-kms-dpi-hyperpixel4
48dtparam=rotate=180
49dtparam=touchscreen-inverted-x,touchscreen-inverted-y

And your cmdline.txt should look like this:

dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash

Let there be rainbows!

If you’re using the desktop edition, you’ll need a monitor plugged in for the first time setup. Not because the Hyperpixel won’t work … but because the setup daemon won’t fit on the Hyperpixel. Unfortunately this will become a bit of a theme as we explore further.

For that matter, you’ll probably want a “proper” keyboard with the desktop edition too because the on-screen keyboard is … painfully small!

Boot up your Pi, and you should see the plymouth “whirly” loading screen appear on both the monitor and the Hyperpixel. When the initial setup window loads it will only appear on the monitor. You can try dragging it off the right edge to appear on the Hyperpixel but it won’t fit (in either portrait or landscape orientations).

Once through the initial setup, don’t detach the monitor just yet. You’ll need it to deal with the first login greeter which … also doesn’t fit (despite being mostly empty space). That said, a major plus point in Kinetic is that the Online Accounts system finally works (because the embedded webkit component was fixed). Once through the first login greeter you can now shut down the Pi, remove the monitor, and start it up again.

Note that everything in the boot-process still works, just on the Hyperpixel, from the plymouth “whirly” screen, to the login screen (where you can try the on-screen keyboard … though personally I’d recommend sticking to a “real” keyboard!), to the desktop itself.

I’d recommend firstly starting up the Settings application. Note how nicely this fits even in tiny resolution with an odd aspect ratio (well done, those developers!). Go to “Ubuntu Desktop”, and switch on “Auto-hide the dock” to get a little bit more precious horizontal space!

Explore around a little more, however, and you may find that this ability to reform an application to fit a small space … is far from universal. The file browser and calculator are good:

Unfortunately the application selector is effectively unreadable, and the help page falls off the edge of the screen although the help application itself does manage to maximize correctly. The Mines game, unfortunately, doesn’t (and won’t) reshape itself to the Hyperpixel:

Firefox works tolerably well though (although the Settings page also falls off the right side):

Kivy Kivy Kivy A Man …

If you’re using the server edition, you can just plug in and go but you will need a keyboard as there’s no on-screen keyboard you can use at the Linux console. You should find rotation of the screen is honoured … for now!

First of all we’ll try one of the more popular Python-based frameworks for touchscreen interfaces: Kivy. This is already in the Ubuntu archive, and so are its examples. However, we do need to make a couple of tweaks to get the most out of it:

Firstly, add the default user to the input group. Without this change, the touchscreen won’t work for console-based applications:
```
$ sudo adduser ubuntu input
```
For this to take effect, you’ll need to logout and login again (group membership is only evaluated at login time; you can use sg to switch groups but this gets confusing to follow with all the sub-shells!)
Next we install Kivy, Kivy’s examples, and the multi-touch library libmtdev1 (also necessary to take advantage of certain examples — I suspect this is a missing dependency / recommendation of the examples package):
```
$ sudo apt install libmtdev1 python3-kivy python-kivy-examples
```
Kivy installs its examples under /usr/share/kivy-examples. Feel free to browse this directory and its contents, trying things for yourself. By way of example, we’ll start with a look at Kivy’s multi-touch and basic graphics capabilities:
```
$ cd /usr/share/kivy-examples
$ python3 demo/pictures/main.py
$ python3 tutorials/pong/main.py
```
Warning

If you enabled rotation, you may note several things at this point: firstly that the screen rotation is ignored completely. And secondly that the touchscreen rotation … isn’t. Which results in some difficulty!

What about slightly more advanced rendering?

$ python3 3Drendering/main.py
$ python3 shader/plasma.py

And how about UI widgets?
```
$ python3 settings/main.py
```

I’d recommend trying other demos under /usr/share/kivy-examples, but be aware that not all of them work: some of the demos are rather out of date and require Python 2.x, others have extra dependencies, or require OS facilities or hardware that isn’t necessarily available (e.g. the Android demos). The list of demos I failed to get (reliably) running includes:

RST_Editor/main.py
android/*
camera/main.py
demo/multistroke/main.py
frameworks/twisted/twisted_app.py
keyboard/main.py
kinect/kinectviewer.py
svg/*
widgets/actionbar.py
widgets/colorpicker.py
widgets/unicode_textinput.py

Furthermore, as noted above, screen rotation appears to be ignored by Kivy, but touchscreen rotation isn’t.

What a Qt!

Another popular UI framework is Qt. You may be familiar with this from the KDE desktop, but Qt is also perfectly capable of running outside a full windowing environment. This makes it an ideal framework for embedded applications as you can develop and test within the desktop, but then run the same code at the console outside the windowed environment, without the “weight” of the full desktop.

Again, you’ll need a keyboard here as there’s no on-screen keyboard in the server edition. And once again, we need a couple of tweaks and some new packages installed:

If you’re following on from the Kivy examples you’ll have done this already but just in case, add the default user to the input group. Without this change, the touchscreen won’t work for console-based applications:
```
$ sudo adduser ubuntu input
```
For this to take effect, you’ll need to logout and login again (group membership is only evaluated at login time)
Install the Qt5 examples. No need to install anything extra here; the examples are pre-compiled and the dependencies are all correct (e.g. libmtdev1 is included):
```
$ sudo apt install qtbase5-examples
```
To tell Qt that you want to render direct to the screen you need to set an environment variable:
```
$ export QT_QPA_PLATFORM=eglfs
```
The Qt documentation has more information on platform plugins for embedded devices.
Qt’s examples are stored under /usr/lib/$GNU_ARCH/qt5/examples. We’ll switch to that directory and try a simple demo that includes controls for exiting it:
```
$ cd /usr/lib/aarch64-linux-gnu/qt5/examples
$ ./widgets/touch/fingerpaint/fingerpaint
```

There are many other examples under the qt5/examples directory, but I would caution against trying them immediately. Firstly, set up SSH to your Pi from another machine. Most of the Qt examples assume they’re in a windowing environment with a convenient “Close” button on the owning window. But that’s not the case when running from the console, and unlike the Kivy examples you can’t simply Escape or Ctrl+C your way out of them. If you find yourself in a demo you can’t quit, you can either reboot with the classic Ctrl+Alt+Del shuffle, or if you have SSH set up you can login from another machine and killall on the demo you’re running.

As with Kivy, if you play around with rotation you’ll notice that screen rotation is ignored. This can be worked around by using the QT_QPA_EGLFS_ROTATION environment variable to tell Qt to handle the rotation itself but there are caveats:

270° rotates incorrectly for some reason (bug?)
Rotation is anti-clockwise, the opposite direction to the overlay’s sense of rotation

Hence, if you wish the touchscreen coordinates to match you’ll need to apply opposite measures:

A strange disagreement in coordinate systems!

But rotation can be achieved with a little work

That’s all for now, folks. Have fun!

The One Where Dave Breaks Stuff

2022-10-10T00:00:00+01:00

Updated with rpi-lgpio instructions

This is a follow-on from the prior post about the ever so slightly dodgy state of userspace GPIO on the Raspberry Pi. If you want the full detail, I’d strongly recommend going and reading that article first. However, it’s in my typically waffley style, so here’s a quick-ish TL;DR for those that don’t want to wade through my excessive verbiage:

Previously On…

GPIO handling on Pi can be roughly split into “traditional” libraries and “new” style libraries.

“Traditional” libraries bang directly on the GPIO registers for most stuff, by-passing the kernel and hoping not to tread on other GPIO users’ toes. They also use the (long deprecated) sysfs GPIO controls for some things (usually edge-detection). “Traditional” libraries include the extremely popular RPi.GPIO, WiringPi, pigpio, the legacy RPIO, and a few others.

“New” style libraries (which include libgpiod, and the excellent lgpio) operate the GPIO pins via the kernel’s gpiochip devices (/dev/gpiochip*). This is a bit more cumbersome as applications must “reserve” the pins they want to use, and it’s slightly slower than banging on the hardware registers directly. However, it guarantees two applications can’t tread on each others toes by using the same GPIO; GPIO resources are automatically cleaned up on process termination (because the kernel knows which process is using which GPIO), and at least until now it’s also backwards compatible with the sysfs GPIO interface.

Oh, and it’s not deprecated and scheduled for removal…

The Bell Tolls

Actually, it’s been tolling for a couple of years now, but I don’t think most of the people that’ll be directly affected have actually noticed. Take a look at the section on sysfs-gpio under ABI obsolete symbols in the Linux kernel documentation and you’ll find the following:

This ABI is deprecated and will be removed after 2020. It is replaced with the GPIO character device.

Just to hammer the message home, the top of the GPIO Sysfs Interface for Userspace documentation has the following SHOUTY WARNING:

THIS ABI IS DEPRECATED, THE ABI DOCUMENTATION HAS BEEN MOVED TO Documentation/ABI/obsolete/sysfs-gpio AND NEW USERSPACE CONSUMERS ARE SUPPOSED TO USE THE CHARACTER DEVICE ABI. THIS OLD SYSFS ABI WILL NOT BE DEVELOPED (NO NEW FEATURES), IT WILL JUST BE MAINTAINED.

Though as noted above, it won’t be maintained and will be removed … erm … at some point after two years ago. As you’ve no doubt guessed it hasn’t been removed yet (because things like RPi.GPIO still operate happily), but it is nonetheless “living on borrowed time”.

Playing the Piper

As I mentioned in LP: #1918583, I didn’t think it was a good idea to go actively breaking stuff before an LTS release (Jammy, 22.04). However, now we’re coming up to the first interim release (Kinetic, 22.10) and I’m keen to see just how much pain we’re in for in the future.

To that end, the kernel team are going to disable the GPIO sysfs interface for the forthcoming 5.19 kernel. To be absolutely clear:

This will not affect prior releases; all currently working GPIO libraries will continue to work in Jammy (22.04) and all releases before that
This will not affect Ubuntu Core; it only uses kernels from LTS releases so again, no change (yet)
This will break the traditional libraries (RPi.GPIO, WiringPi, et al) under Kinetic (22.10)
This should not break gpiozero because that already uses one of the new-style libraries (specifically lgpio) by default on Ubuntu (although it falls back to RPi.GPIO if it can’t find lgpio, or if it’s been told explicitly to use another backend)

Something Must Be Done

What if it’s all a disaster, and so many things break that Something Must Be Done? There’s obvious and slightly less obvious “somethings” that we can turn to in that case:

The most obvious is “undo it”. This is just a configuration option in the kernel, and as it’s still there we could just flip it back on for 23.04 and carry on our merry way. However, that’s likely not a long term option…
Slightly less obvious: make compatibility shims. For example, if most breakage turns out to be in applications relying on RPi.GPIO, make a library with the RPi.GPIO interface which uses lgpio or libgpiod underneath.

The issue here is we need to know which libraries to shim (RPi.GPIO? WiringPi? pigpio? All of the above?), and that the shim will never be 100% perfect (if two processes rely on accessing the same GPIO, their assumptions will still be broken, and they will not work with the shim).
Even less obvious: fix the applications. If only a minority of applications break (or all that break use different pin libraries) it may be better to simply migrate those applications to a new-style library. GPIO handling is usually a minority of most applications that use it, so the patches shouldn’t be too long / hard to generate.

Obviously it’s premature to jump into some of these options, and the end result may well be a combination of some of them.

Still, I am anticipating that RPi.GPIO is likely to be one of the major sources of breakage. To that end, I’ve already worked on a simplistic shim which transforms the lgpio library into a compatible interface. It’s called rpi-lgpio and will be landing in Kinetic as the python3-rpi-lgpio package:

$ sudo apt install python3-rpi-lgpio
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  rpi.gpio-common
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  liblgpio1 python3-lgpio
The following packages will be REMOVED:
  python3-rpi.gpio
The following NEW packages will be installed:
  liblgpio1 python3-lgpio python3-rpi-lgpio
0 upgraded, 3 newly installed, 1 to remove and 0 not upgraded.
Need to get 108 kB of archives.
After this operation, 365 kB of additional disk space will be used.
Do you want to continue? [Y/n]

Installing this package will remove the python3-rpi.gpio package (because both define an RPi.GPIO module and so cannot co-exist) but should permit most existing applications that rely on RPi.GPIO to continue operating normally. The python3-rpi-lgpio package also “provides” (in apt parlance) python3-rpi.gpio so it can satisfy any package that “depends” on python3-rpi.gpio.

This Is Something

In the meantime, what can you do? If you’re relying on GPIO connected devices on your Pi, I would caution you not to upgrade production systems to Kinetic (22.10) when it’s released without testing them first. The simplest way to test a Raspberry Pi based setup is simply to clone the SD card and try upgrading the backup:

Shut down your Pi
Duplicate the SD card to another one of equal / greater size:
- Assuming you have a single SD card reader at your disposal, with the device name /dev/mmcblk0, place the “production” card in the reader
- Run sudo umount /dev/mmcblk0p1 and sudo umount /dev/mmcblk0p2 to unmount any auto-mounted partitions
- Run sudo dd if=/dev/mmcblk0 of=backup.img bs=16M status=progress
- Remove the production card
- Insert the blank card
- Run sudo dd if=backup.img of=/dev/mmcblk0 bs=16M status=progress
- Run sudo sync just to make sure and remove the cloned card
Plug the duplicated SD card back into your Pi and boot it up
Run through the upgrade to Kinetic: sudo do-release-upgrade --devel
See what breaks!
If nothing breaks, great!
If something breaks, file bugs!
- File against the application itself if it’s from an Ubuntu package
- File against the GPIO library (e.g. file a bug against RPi.GPIO on Ubuntu) it’s using if the application isn’t from an Ubuntu package, and you happen to know which GPIO library is being used
- File against Ubuntu generally if the application isn’t from an Ubuntu package, and you’ve no idea which GPIO library is being used
In any of the cases above, tag the bug raspi-image and it should get my attention

In the meantime, I’m keen to hear thoughts and/or suggestions about the forthcoming GPIO apocalypse changes.

Playing with Blocks - It’s All Connected

2022-09-08T00:00:00+01:00

In the last two articles we looked at LVM and LUKS for some enhancement of the basic storage of the standard Ubuntu Pi images. This time around we’ll combine both of those previous articles to make an image in the same way that the Ubuntu Desktop for PC images install when they’re given free reign to overwrite the entire contents of the drive.

Specifically, this means we aim to wind up with:

A FAT boot partition (no difference there), 512 MB in size
A second partition taking up the rest of the card containing
- An encrypted LUKS container, containing
  - An LVM physical volume (PV) belonging to
    - An LVM volume group (VG) called “pivg” containing
      - An LVM logical volume (LV) called “root” taking up some reasonable portion of the disk space (8 GB in this example)

A diagram illustrating the SD card layout showing: a large bar representing the whole SD card /dev/mmcblk0 (N GB). Beneath that, two bars: a small one on the left for the system-boot partition /dev/mmcblk0p1 (512 MB), and a large one on the right for the remaining partition /dev/mmcblk0p2 (N GB - 512 MB). Beneath the large right-hand bar, a bar representing the LUKS encrypted container (again, N GB - 512 MB). Beneath that, two more bars: a tiny one on the left representing the LUKS header (16 MB), and a much larger one on the right representing the LVM PV /dev/mapper/rootfs within the LUKS container (N GB - 512 MB - 16 MB). Lower down, a dotted outline of the same size representing the LVM VG /dev/pivg. Finally, beneath that, a smaller bar representing the LVM LV /dev/mapper/pivg-root (8 GB).

Note

One thing to note is that I’ve only tested this with the Ubuntu 22.04 (jammy) server for pi image. Notably, jammy has cryptsetup within the default initramfs on the image. I would not necessarily be confident that this works on earlier releases.

Three (?) Rowdy Layers

Rather than go through all the individual commands, which would result in this post looking like a foreword followed by the dump of a shell script, I’ll go through the high level procedure because the detail is almost exactly the same as the original articles but for a few minor differences and some deliberate omissions.

Firstly, we’ll deal with the content of the first article and set up LVM:

Follow the first article instructions until it’s time to create the PV (with pvcreate):
- Plug in the SD card
- Unmount any auto-mounted partitions
- Repartition with gdisk

At this point we need to create the PV with 16MB less space than is available in the second partition, to leave a little room for the LUKS header around the PV:

From the gdisk session, you should have a printout of the new partition table; if you don’t, run sudo gdisk -l /dev/mmcblk0 (assuming /dev/mmcblk0 is your SD card)

For the second partition, calculate the number of sectors it occupies (which is simply end sector minus start sector) and then subtract 32768 (each sector is 512 bytes, so 32768 is 16MB). For example, on a 32GB SD card:

$ sudo gdisk -l /dev/mmcblk0
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/mmcblk0: 62333952 sectors, 29.7 GiB
Model: STORAGE DEVICE
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): A9BDFD97-F2A5-4419-9A83-B0160820FDBB
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 62333918
Partitions will be aligned on 2048-sector boundaries
Total free space is 4061 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1048576   511.0 MiB   0700  system-boot
   2         1050624        62333918   29.2 GiB    8E00  lvm
$ PV_SECTORS=$((62333918 - 1050624 - 32768))
$ echo $PV_SECTORS
61250526

Proceed to creating the logical volume, but explicitly specify the number of sectors to use when creating the PV with --setphysicalvolumesize and the number of sectors with the suffix “s”:

$ sudo pvcreate --setphysicalvolumesize ${PV_SECTORS}s /dev/mmcblk0p2
Physical volume "/dev/mmcblk0p2" successfully created.
$ sudo vgcreate pivg /dev/mmcblk0p2
Volume group "pivg" successfully created
$ sudo lvcreate --size 8G --name root pivg
Logical volume "root" created.

Finish the first article’s instructions:
- Unpacking the image
- Creating a loop device for the image
- Transferring the content of the image partitions to the card
- Updating the boot command line and fstab to point to the “new” root device
- Unmount the root and boot partitions, and disable the VG
- Don’t eject the SD card, we still need to do some surgery!

LUKS Dujour

Now it’s time to move on to the second article and set up the encrypted LUKS container around the physical volume (PV):

Skip everything up to the encryption step:
- No need to download another image
- No need to boot the image yet
- No need to unmount stuff (we already did that)
- No need to shrink partitions (the root partition hasn’t been auto-expanded because we haven’t booted anything, and we’ve pre-shrunk the PV containing the root)

Now it’s time to encrypt the second partition containing the PV:

First the encryption; this is exactly as in the second article:

$ sudo cryptsetup reencrypt --encrypt --reduce-device-size 16M --cipher xchacha12,aes-adiantum-plain64 /dev/mmcblk0p2

WARNING!
========
This will overwrite data on LUKS2-temp-a823bd67-a632-41cb-955e-8776b8b3fb5b.new irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for LUKS2-temp-a823bd67-a632-41cb-955e-8776b8b3fb5b.new:
Verify passphrase:
Finished, time 36:53.687, 29915 MiB written, speed  13.5 MiB/s

Next we open the encrypted container:

$ sudo cryptsetup open /dev/mmcblk0p2 rootfs
Enter passphrase for /dev/mmcblk0p2:

Then we re-scan for physical volumes (PVs) and re-activate the pivg volume group (VG) we created earlier; this isn’t in the original article but is necessary as we’ve altered the mapper configuration:

$ sudo pvscan
  PV /dev/mapper/rootfs   VG pivg            lvm2 [29.20 GiB / 21.20 GiB free]
  Total: 1 [29.20 GiB] / in use: 1 [29.20 GiB] / in no VG: 0 [0   ]
$ sudo vgchange -ay pivg
  1 logical volume(s) in volume group "pivg" now active

We skip over all the resizing bits because we’ve already established the size of the root volume, and just mount the root file-system (we don’t need to mount the boot file-system). This is, again, slightly different to the original article because the root file-system is a mapper device, pivg-root as in the first article:
```
$ sudo mkdir -p /mnt/root
$ sudo mount /dev/mapper/pivg-root /mnt/root
```

Then a bit (but only a bit) of configuration manipulation:

Add our entry to /etc/crypttab. Please heed the advice in the original article regarding the source device here; this is the source device as seen by the Pi:

$ cat /mnt/root/etc/crypttab
# <target name> <source device>         <key file>      <options>
$ echo "rootfs  /dev/mmcblk0p2  none  luks,discard" >> /mnt/root/etc/crypttab
$ cat /mnt/root/etc/crypttab
# <target name> <source device>         <key file>      <options>
rootfs  /dev/mmcblk0p2  none  luks,discard

Skip the bits about editing fstab and the boot command line; we’ve already done everything we need to for those. Just unmount root file-system, and close the encrypted container:
```
$ sudo umount /mnt/root
$ sudo cryptsetup close rootfs
```

Now just follow the second article to the end:
- Eject the card and boot it on your Pi
- There’s a long delay while the initial root mount fails
- In the emergency prompt that appears, open the encrypted device and exit the prompt
- Once booted, update the initramfs
- Reboot to make sure everything works!

Future Cases

Congratulations! At this point you should have a setup eerily similar to the one I use on my main development Pi, albeit that one boots off a large SSD instead of an SD card. Future things worth exploring with this setup:

How to get plymouth prompting nicely for the password; plymouth’s always been on the Pi server images but for some reason it doesn’t work properly (unlike the Pi desktop images where it’s happy). I had a cursory look into this a couple of cycles ago, but didn’t get anywhere at that time.
How about ditching password encryption and using a key on removable storage. For instance, if booting off SD card, how about using a USB key with the encryption key. Or for my favoured setup where I boot off a USB3 attached SSD drive, how about using an SD card with the key?
With this configuration I usually set up LXD with an LVM thin-pool for its default storage. This usually requires taking a crow-bar to LXD and forcing it to use the thin-pool (with lvm.vg.force_reuse; LXD doesn’t like working with a non-empty VG for reasons I’ve never full understood, but at least it’s possible). Let me know in the comments if it would be useful to have another article covering this!

Making Jammy less Dodgy

2022-07-22T00:00:00+01:00

It’s now several months after jammy’s release, and thus long past time since I should’ve posted something about it. I’ve been holding back on this because, despite being an LTS (in fact, the first Ubuntu LTS desktop for the Pi), I can’t say I’m happy with it.

Yet.

Still, that’s why we don’t enable upgrades for LTS users until the first point release (planned for August 4th at the time of writing). So let’s go through the painful bits and see what early adopters can rectify before I get my nose back to the grindstone and try and fix all the bits I’ve messed up!

In some ways, this may be considered a spiritual successor to my earlier post (and even earlier post) on the Pi desktops. Or to put it another way, consider this the current “Dave recommends you run your Ubuntu Pi this way” post (until I supersede it again)!

Note

I’m going to take the unusual step of stating quite explicitly that the following (like everything on this site) is my personal opinion. This is my recommendation for running your Ubuntu Pi, not anyone else’s (including Canonical).

So, with disclaimers thoroughly disclaimed, let us begin …

Snap!

It’s the serial complaint: the default browser (Firefox) is now a snap. I’m not going to get into the fiery debate over snaps here (more beer is required for that!). Nor am I going to address the start-up speed issues (they’ve been furiously hacked upon and thoroughly covered elsewhere; in fact I was a small cog in the team that did some of that work). Besides, I’m more concerned with the performance of the browser after it’s started (a browser isn’t something I fire up more than once a day), and in that regard I’ve no particular complaints about the performance of the snapped version over the deb version.

And yet, complaints I have …

There are currently issues with the Firefox snap which, personally, I consider deal-breakers. In particular LP: #1741074 “chrome-gnome-shell extension fails to detect native host connector” (aka the “native messaging issue”). The bug’s title may sound relatively innocuous but in practice this translates to “I can’t use KeepassXC” (my password database of choice) with my browser.

I would fallback to KeepassXC’s awesome auto-type feature, but that doesn’t work under Wayland (to be clear: this isn’t exactly a defect, more a design choice; the linked thread has more details). Hence, at least with the version of jammy as released, my only option to use my (now pretty extensive, and certainly indispensable) password database is to copy and paste usernames and passwords via the clipboard … which just about any process can view. Obviously, this is a fairly major step backwards in security and not one I’m willing to put up with.

I should stress this is another issue that is top of the list and being furiously attacked with many sharp objects. Once it’s fixed, I’ll have no specific qualms about returning to the snap (after all, Mozilla seem to prefer it as a means of distribution). But what to do in the meantime?

Switch to the flatpak? No good: the aforementioned bug applies to any confined (snap or otherwise) browser. Besides … the Firefox flatpak is only available for amd64!

Switch to Chromium? It’s snapped too. That said, I’m not sure I would even if I could: uBlock Origin (an extension I consider damned near mandatory for anyone wanting to browse the modern web on anything less than a beefy Ryzen with 32GB of RAM) is better in Firefox than in Chromium.

What about full-fat Chrome? Not an option; Google only distribute Chrome packages for amd64 too.

Note

Rant: frankly, this is one of my biggest misgivings about snap, flatpak, and all these distribution methods that purport to put “distribution in the hands of the developers”: they all suck at providing for anything other than PC architectures leaving stuff like the Pi (or RISC-V, or any new architecture) high and dry.

So, until this is fixed, let’s just use the deb:

$ sudo add-apt-repository ppa:mozillateam/ppa
... lots of output, press Enter ...
$ cat << EOF | sudo tee /etc/apt/preferences.d/firefox
Package: firefox*
Pin: release=LP-PPA-mozillateam
Pin-Priority: 501

Package: firefox*
Pin: release o=Ubuntu
Pin-Priority: -1
EOF
$ sudo apt purge firefox
$ sudo snap remove firefox
$ sudo apt install firefox

Crackle!

Another cereal complaint (I make no excuses for the puns here): the introduction of systemd-oomd! In fact, I was rather glad of this one but I also think it’s less of an issue on the Pi than it is the PC, for reasons I’ll go into …

Note

This section is partly redundant because, while writing it, one of my colleagues (Nick, who despite being fresh-faced and new, jumped headlong into the fray) pushed a new configuration that substantially solves things. See LP: #1972159 for the full story.

However, there’s still some useful stuff in here for users on smaller memory systems (including non-Pis!) so I’ve left it in.

Firstly, a little context. For those not aware, systemd-oomd is a new facility for killing memory hogging processes (now activated by default on our desktop images). Previously, this was solely the domain of the dreaded OOM killer in the Linux kernel. While this certainly did the job of killing memory hogs, it was typically only capable of doing so after the system had descended into “swap hell”. In other words, only once something ridiculously fat had pushed everything else on the system to page itself out to disk, did the kernel OOM killer wake up and stalk the process table looking for potential victims.

An artist’s impression of “swap hell”

This typically meant that, while things did get remedied eventually, it took several minutes of the machine “thrashing” for that to happen. In the noughties, I might listen for the audible cue of an HDD thrashing its heads to have a clue this was going on, but this is now and SSDs (or SD cards) don’t provide quite the same feedback!

As a rough rule of thumb, people will assume something has crashed after roughly 5 seconds of the UI appearing to freeze. If that sounds short, consider how many seconds it would take before you assumed a crash if even your mouse pointer is frozen (or jumpy to the point that someone might reasonably assume something was “wrong”). In practice this typically meant that, long before the kernel OOM killer got involved, the user’s itchy power-button finger got involved first.

To be fair, systemd-oomd does fix several issues with the kernel’s OOM killer. The most notable being that a modern application may have many “processes” and (being process-oriented) the kernel’s OOM killer would only kill the heaviest individual process (regardless of whether it was a mere component in a larger system). That potentially left the rest of the application in a unknown / unstable state. By contrast, systemd-oomd will kill the whole cgroup consistently (cgroup in this context can be read as “application” … sort of).

It also acts before the system ever gets near “swap hell”, basing its actions on the system reaching some (fairly arbitrary) thresholds like 90% of both RAM and swap being utilized (to activate), and pressure stall information (to determine what to act upon).

Unfortunately there are some major issues:

Most browsers (presumably in acknowledgement of how much fat is dripping off the silk of the web these days) already have mechanisms to “unload” their heavier processes (usually individual tabs) when they detect the system is “under load”. However, being fine-tuned to the individual application, these usually kick in later than systemd-oomd will (or to put it another way, by the time they would kick in, systemd-oomd has already killed the whole browser).
systemd-oomd has no user interaction or even notification. One second you’re working in your browser and the next: blip. It’s gone. No warning, no chance to intervene, and not so much as an apology after the fact!

On the upside, it all happens very quickly: there’s no wading through the treacle of swap-thrashing. But it feels … arbitrary, unpredictable, and frankly “flaky”.

There is a simple solution: get more RAM! On my machines with 8GB (or more) of RAM, I simply haven’t encountered the issue. It’s only on machines with 4GB or less that this seems to be a serious issue (at least, that’s my personal experience — I’m sure others with different workloads may find differently). Still, that includes the Pi 400 I use as a desktop, and also my tiddly laptop (which has 4GB of RAM and a Celeron processor — I like my machines small and cheap!).

And this is where things (surprisingly!) turned out better on the Pi than on the PC:

On my little 4GB laptop (which doesn’t really get much use, only when I’m away from the house), I noticed Firefox frequently getting killed especially when working in Google Docs. Curiously, I didn’t notice the same on the Pi 400 (one of my two main workhorses that gets used a lot more than the laptop).

Then I recalled that we’d activated zswap (see earlier post) by default on jammy for the Pi desktop images. My PC laptop had the same RAM size (4GB), the same swap size (1GB), but while zswap was active on the Pi, it wasn’t on the laptop. That’s not to say the Pi was perfect: there were two occasions when Firefox got killed, but that was in stark contrast to the laptop on which it happened half a dozen times in a single day.

Obviously the first thing I did was activate zswap on my laptop and, after a reboot suddenly things were much better. I did still manage to get Firefox killed during a subsequent test run, but after then expanding the swap to 2GB on both machines, I managed a full day of work on the laptop without a single systemd-oomd activation.

So (unusually for this site), here’s a recommendation for any jammy desktop users on small PCs out there (Pi users, you can ignore this bit):

$ sudo -i
# sed -i \
> -e '/GRUB_CMDLINE_LINUX_DEFAULT/ s/splash/splash zswap.enabled=1 zswap.compressor=zstd zswap.zpool=z3fold/' \
> /etc/default/grub
# echo zstd >> /etc/initramfs-tools/modules
# echo z3fold >> /etc/initramfs-tools/modules
# update-initramfs -u
# update-grub

The above will activate zswap by adding zswap.enabled=1 zswap.compressor=zstd zswap.zpool=z3fold to the kernel’s command line in the GRUB configuration, and adding the necessary modules to the initramfs.

Next, I’d also recommend (to both small PC users and Pi users), to bump the default swap-file up to 2GB. After doing this, I haven’t suffered a single systemd-oomd kill on my Pi 400 (again, for my particular workload — bear in mind I spend most of my time in lightweight terminal things, so adjust accordingly). The following should be run on a freshly booted system as the first thing we’re going to do is disable the existing swap before expanding it:

$ sudo swapoff /swapfile
$ sudo fallocate -l 2G /swapfile
$ sudo mkswap /swapfile
$ sudo chmod 600 /swapfile
$ sudo swapon /swapfile

What have we actually accomplished with all this?

Let’s take the example of my 4GB laptop. The defaults for zswap’s configuration will reserve 20% of all RAM for compressed pages, which is roughly 800MB. Those compressed pages will (with the z3fold allocator) store up to 3 uncompressed pages each. So that 800MB of RAM can act as up to 2.4GB of (suspiciously fast) swap. And beyond that we also have 2GB of (typical, slow) disk-based swap.

Visually what we’ve done is this:

Three bars illustrating the original state of the memory layout (4GB RAM, 1GB swap), the new physical layout (3.2GB RAM, 800MB zswap, 2GB disk swap), and the new “effective” layout (3.2GB RAM, 2.4GB zswap, 2GB disk swap)

Don’t be fooled: we’ve reduced the amount of available RAM from 4GB to 3.2GB. This isn’t good, but it’s the trade-off you make with compressed RAM systems of any sort. We’ve also sacrificed another gig of disk space to swap. However, we’ve increased the available swap from 1GB (by default) to about 4.4GB, of which more than half is very fast compressed-memory swap and all for a (fairly) minimal memory cost.

Furthermore, we’ve also moved from a configuration where there’s 4GB of very fast pages, and 1GB of really slow pages to one where there’s 3.2GB of very fast pages, 2.4GB of fairly quick pages, and 2GB of really slow pages. In other words, we’ve now got a “smoother” progression of memory the kernel can stick pages into. This helps a system under load go a bit slower while remaining reasonably responsive, instead of suddenly hitting a brick wall of swap.

Soggy Cornflakes

One mea-culpa I should offer here: I managed to forget something rather important on the Ubuntu Pi desktop images for jammy.

While zswap is activated, and the swap-file is created (now on boot by the mkswap.service unit so we no longer ship a gigabyte of “nothing” in the image itself) … I managed to forget to add the “z3fold” and “zstd” modules to the initramfs modules list. As a result, zswap is still working on the Pi desktop images, but it’s currently falling back to the default “zbud” (2-page) allocator, and “lzo” compression.

LP: #1977764 is tracking this and I’ll get it fixed as soon as I reasonably can (unfortunately, while it sounds trivial it’s actually a messy one — read the bug if you want the full story).

Pop!

That about covers things for the desktop side. What about the server image? Overall I’m actually pretty happy with the server side of things, but one thing has been irking me whenever I’ve played with the lovely little Pi Zero 2W. There’s precious little memory available at runtime, particularly on the arm64 images (which it seems the vast majority of our users prefer).

There’s been some effort recently to identify things installed by default which shouldn’t be, with the goal of reducing the “base” memory footprint of our images and I thought I’d have a look at what’s running by default on the Pi server images to see if there’s anything we could cull there.

First, let’s see how much free memory we have on a fresh boot:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           414Mi       152Mi        33Mi       3.0Mi       228Mi       242Mi
Swap:             0B          0B          0B

Hmmm, only 242MB available. Okay, what’s eating it?

Note

A quick clarification about “free” vs “available”. Ignore “free”. This is the amount of RAM actually unused, but unused RAM is wasted RAM. The kernel has a duty to minimize free RAM by filling anything unused with disk (and other useful) caches which can be evicted trivially in the case that more RAM is required. The “available” metric indicates the amount of RAM that is available “on demand” should userland processes request it. It’s roughly (but not precisely) equal to “free” + “buff/cache”.

Or to put it another way: “available” tells you how much RAM is available for use; “free” is the metric which tells you how much RAM you’ve wasted your money on ;)

We’ll use the ps command exclude everything under PID 2 which is the kernel (--ppid 2 -p 2 -N). We’ll turn on the hierarchical view (-H) to get a clue of what belongs to what (this won’t show every inter-process dependency, but it can be a rough guide), and the “full” output (-F) so we can get the memory usage info (amongst other things; I’ll exclude some of the columns below for the sake of text wrapping):

$ ps --ppid 2 -p 2 -NHF
UID          PID    PPID     SZ   RSS PSR TTY   CMD
root           1       0  41786 11364   2 ?     /sbin/init fixrtc splash
root         374       1  12020 12572   0 ?       /lib/systemd/systemd-journald
root         416       1  72416 25672   3 ?       /sbin/multipathd -d -s
root         429       1   5998  6332   2 ?       /lib/systemd/systemd-udevd
root         576       1   4087  9732   3 ?       /sbin/wpa_supplicant -c /run/netpla
systemd+     588       1  22160  6292   3 ?       /lib/systemd/systemd-timesyncd
systemd+     634       1   4097  7704   1 ?       /lib/systemd/systemd-networkd
systemd+     636       1   6270 12224   1 ?       /lib/systemd/systemd-resolved
message+     668       1   2280  4220   3 ?       @dbus-daemon --system --address=sys
root         673       1  20492  3344   2 ?       /usr/sbin/irqbalance --foreground
root         677       1   8217 17916   1 ?       /usr/bin/python3 /usr/bin/networkd-
root         678       1  59035  8220   3 ?       /usr/libexec/polkitd --no-debug
syslog       680       1  55508  4392   3 ?       /usr/sbin/rsyslogd -n -iNONE
root         684       1  366205 27060  0 ?       /usr/lib/snapd/snapd
root         687       1   6002  7164   2 ?       /lib/systemd/systemd-logind
root         691       1  98379 11280   3 ?       /usr/libexec/udisks2/udisksd
root         695       1   3813  3440   0 ?       /sbin/wpa_supplicant -u -s -O /run/
root         712       1   1727  2456   0 ?       /usr/sbin/cron -f -P
root         726       1  61827 12368   1 ?       /usr/sbin/ModemManager
root         736       1   1409   808   1 ttyS0   /sbin/agetty -o -p -- \u --keep-bau
root         744       1  27480 19884   0 ?       /usr/bin/python3 /usr/share/unatten
root         755       1   3787  8428   1 ?       sshd: /usr/sbin/sshd -D [listener]
root        1660     755   4530  9716   0 ?         sshd: ubuntu [priv]
ubuntu      1804    1660   4661  7592   1 ?           sshd: ubuntu@pts/0
ubuntu      1805    1804   2152  4852   2 pts/0         -bash
ubuntu    355020    1805   2495  2844   3 pts/0           ps --ppid 2 -p 2 -NHF
root         881       1    560   168   2 ?       /usr/bin/hciattach /dev/serial1 bcm
root         899       1   2386  4468   3 ?       /usr/lib/bluetooth/bluetoothd
ubuntu      1004       1   4559  9924   2 ?       /lib/systemd/systemd --user
ubuntu      1005    1004  42795  5800   2 ?         (sd-pam)
root        1766       1   1398   812   2 tty6    /sbin/agetty -o -p -- \u --noclear
root        1777       1   2417  3976   1 tty1    /bin/login -p --
ubuntu    227377    1777   2174  4856   2 tty1      -bash

Okay, lots to go through here. RSS (Resident Set Size) is the most interesting column here (it’s not wholly accurate but good enough for our purposes). What’s the fattest thing in RAM? Ah … snapd (at ~26MB). On my fat 8GB Pi 4 I’m happy to leave that because I use the LXD snap (a lot!). However, this is a little Zero 2W and I only need debs on this, so out it goes:

$ sudo apt purge snapd
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  squashfs-tools
Use 'sudo apt autoremove' to remove it.
The following packages will be REMOVED:
  snapd*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 84.1 MB disk space will be freed.
Do you want to continue? [Y/n]
... lots of complaints about read-only stuff ...
rm: cannot remove '/snap/core20/1408/var/log': Read-only file system
rm: cannot remove '/snap/core20/1408/var/mail': Read-only file system
rm: cannot remove '/snap/core20/1408/var/opt': Read-only file system
rm: cannot remove '/snap/core20/1408/var/snap': Read-only file system
rm: cannot remove '/snap/core20/1408/var/spool/mail': Read-only file system
rm: cannot remove '/snap/core20/1408/var/tmp': Read-only file system
rm: cannot remove '/snap/core20/1408/writable': Read-only file system
Cannot remove directory /snap
Removing extra snap-confine apparmor rules
Removing snapd cache
Removing snapd state
dpkg: warning: while removing snapd, directory '/snap' not empty so not removed
$ sudo apt autoremove --purge
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  squashfs-tools*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 408 kB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 100500 files and directories currently installed.)
Removing squashfs-tools (1:4.5-3build1) ...
Processing triggers for man-db (2.10.2-1) ...

At this point, given it couldn’t remove everything under /snap I’d just reboot to clear up any remaining mounts and sudo rm -fr /snap (I don’t recall having to do this before; I’ll try and remember to investigate this further at some point).

$ sudo reboot
... wait to get back to the login ...
$ sudo rm -fr /snap

Now, what’s next? “multipathd” (~25MB). This is a piece of the default server installation for dealing with multipath disk IO. However, unless you’re running a Compute Module with some fairly serious storage attached to it, this is likely pointless on your Pi. Unfortunately, removing this is probably not a good idea:

$ sudo apt purge multipath-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  kpartx libsgutils2-2 liburcu8 sg3-utils sg3-utils-udev
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  multipath-tools* ubuntu-server* ubuntu-server-raspi*
0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded.
After this operation, 1179 kB disk space will be freed.
Do you want to continue? [Y/n] n
Abort.

Notice how it wanted to remove the “ubuntu-server” package? That’s usually not a good sign! However, perhaps we can disable it somehow:

$ systemctl cat multipathd.service
# /lib/systemd/system/multipathd.service
[Unit]
Description=Device-Mapper Multipath Device Controller
Before=iscsi.service iscsid.service lvm2-activation-early.service
Before=local-fs-pre.target blk-availability.service shutdown.target
Wants=systemd-udevd-kernel.socket
After=systemd-udevd-kernel.socket
After=multipathd.socket systemd-remount-fs.service
DefaultDependencies=no
Conflicts=shutdown.target
ConditionKernelCommandLine=!nompath
ConditionKernelCommandLine=!multipath=off
ConditionVirtualization=!container

[Service]
... etc ...

[Install]
... etc ...

Ah ha! Adding nompath or the slightly more obvious multipath=off to the kernel command line should disable this service. Well, that’s easily accomplished:

$ sudo sed -i -e 's/$/ multipath=off/' /boot/firmware/cmdline.txt
$ sudo reboot

I’m sorely tempted to add this by default to the Pi images, but I need to dig a bit further into whether multipath has any genuine use-cases on the Pi.

Next on the list? The Python-based “unattended-upgrades-shutdown” process (~19MB). It’s not a great idea to disable this (it ensures the computer doesn’t shut down in the middle of unattended-upgrades). However, it’s annoying that it hangs around the whole time eating RAM and only actually does anything at shutdown. Still, it’s the subject of an existing ticket, LP: #1955084 so we’ll just leave that one for now.

Next? Another Python process: the “networkd-dispatcher” daemon (~17MB). If you like your WiFi connection, leave this one alone! Next up, it’s “systemd-journald” (~12MB). That’s also doing useful work, so we’ll leave that one alone too.

Then … “ModemManager”. Hang on? Modem Manager? Am I back in the 90s?! Oh, it’s for GSM modems. Still, I’m not actually using a GSM modem on this Pi so why is that there? In fact, it’s the subject of another ticket, LP: #1981109 (you may note it’s another one filed by Steve, another colleague who has been ruthlessly hunting down the cruft of late). Anyway, given the details in that ticket this one can be safely excised (it’s not even a real dependency, just a “recommendation” of another package)!

$ sudo apt purge modemmanager --autoremove
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  libtcl8.6* modemmanager* tcl* tcl8.6* usb-modeswitch* usb-modeswitch-data*
0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.
After this operation, 8595 kB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 100491 files and directories currently installed.)
Removing usb-modeswitch (2.6.1-3ubuntu2) ...
Removing tcl (8.6.11+1build2) ...
Removing tcl8.6 (8.6.12+dfsg-1build1) ...
Removing libtcl8.6:arm64 (8.6.12+dfsg-1build1) ...
Removing modemmanager (1.18.6-1) ...
Unknown option: runtime
Removing usb-modeswitch-data (20191128-4) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for dbus (1.12.20-2ubuntu4) ...
Processing triggers for libc-bin (2.35-0ubuntu3) ...
(Reading database ... 100105 files and directories currently installed.)
Purging configuration files for tcl8.6 (8.6.12+dfsg-1build1) ...
Purging configuration files for usb-modeswitch (2.6.1-3ubuntu2) ...
Purging configuration files for modemmanager (1.18.6-1) ...
Processing triggers for dbus (1.12.20-2ubuntu4) ...

Next up, “systemd-resolved” (~12MB). We shouldn’t touch that one (DNS resolution and caching is generally useful).

Next? “udisks2” (~11MB). This is a useful service, handling things like auto-mounting USB storage devices that have been inserted. However, I’m not going to be doing that on this Pi Zero 2W so I don’t need it running all the time. I’m not going to uninstall it as it’s potentially useful and, even after disabling it (which will prevent it auto-starting), the udisksctl command can implicitly start it back up again, should I find I need it.

$ sudo systemctl stop udisks2
$ sudo systemctl disable udisks2
Removed /etc/systemd/system/graphical.target.wants/udisks2.service

At this point we’re into diminishing returns so let’s reboot to a clean start and see what free says now.

$ sudo reboot
... wait to get back to the login ...
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           414Mi       103Mi       165Mi       2.0Mi       146Mi       299Mi
Swap:             0B          0B          0B

That’s looking better! Nearly 300MB free now. Finally, another tweak we can use to grab a bit more memory. This one is only useful if you have no intention of using GPU functionality (no camera, no video decoding / encoding acceleration, etc). However, I don’t need any graphics on this particular Pi (in fact there won’t be any display plugged into it) so let’s reduce the GPU memory split all the way down to the minimum (16MB).

$ sudo -i
$ echo "gpu_mem=16" >> /boot/firmware/config.txt
$ sudo reboot
... wait to get back to the login ...
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           462Mi       101Mi       214Mi       2.0Mi       146Mi       348Mi
Swap:             0B          0B          0B

And there we have it; over 100MB extra available memory from our starting point. Bear in mind we have disabled some functionality to achieve this, so these aren’t all changes that could (or should!) be made universally. However, from the tickets we’ve encountered along the way there’s definitely some improvements that can still be made in the base image.

The Pi Zero 2

2021-10-28T00:00:00+01:00

A quick run-down of the new Raspberry Pi Zero 2, and how to run Ubuntu on it (should you wish to!).

Small, but perfectly formed

While browsing the news last week I was particularly amused to come across several comments on Eben’s post about the first price increase of a Raspberry Pi in history (TL;DR “global silicon shortage affects everyone, even Raspberry Pi”), each of which either speculated on designs for an updated Pi Zero, and/or the potential time-frames we might be looking at for such a thing. I’m sure each of the erstwhile Cassandra’s won’t be disappointed to learn how far out their predictions were! :)

I had to chuckle because for the better part of this year, I’ve had a production prototype of the Zero 2 lying somewhere on my desk for testing (and have had to be careful to move it out of shot during certain video calls!).

The initial plan was to release Ubuntu Impish with support for the Zero 2. However, as the global silicon shortage started to bite, the release date for the Zero 2 slipped beyond Impish’s release date, and the Impish release had to go ahead without support for it on the images.

Speaking of which … what’s actually required for support of the Zero 2? To understand that, we should first take a look at the hardware.

Zero History

First, a quick visual tour of the Pi Zero’s history:

Top left is the original Pi Zero (board rev 1.2) with its 1GHz ARM6 SoC, and 512MB of RAM (the RAM is pretty much the only thing that hasn’t changed in terms of specs!)
Top right is the Pi Zero rev 1.3 which added the mini CSI connector for the camera interface
Bottom left is the Pi Zero W which added the triangular Proant WiFi/BT antenna and the WiFi chip (the shiny IC to the right of the SoC)
Finally, bottom right is the new Pi Zero 2 which replaced the venerable SoC with the new quad-core “RP3A0”

The form factor is obviously based on previous Raspberry Pi Zero models. The major difference is that huge (for the Pi Zero) SoC labelled the “RP3A0” (and, slightly confusingly “2041” which is the manufacturing date and nothing to do with RP2040 on the Pico). It’s the good ol’ 2710 (aka 2837) SoC that powered the Raspberry Pi 3B, 3A+, and 3B+ packaged together with its RAM in Raspberry Pi branded silicon.

That’s pretty much the major headline right there: instead of the old single-core ARM6 chip that the Pi Zero shipped with, the new Zero 2 has a full quad-core arm64-capable SoC. That also means it’s the first Pi in a Zero form-factor that Ubuntu’s compatible with (in either armhf or arm64 flavours).

There’s also the notable WiFi “can” sitting to the right of the SoC, like the 3A+ and 3B+. However, it’s worth noting that it doesn’t use the same WiFi chip as the 3A+/3B+ (Broadcom 43455 compatible); instead it uses a WiFi interface similar to the Pi 3B/Zero W (Broadcom 43430 compatible). Which just means: don’t go expecting 5GHz (it’s 2.4GHz only).

If you were to think of it as a cheaper, smaller version of the 3B … well, you wouldn’t be far wrong.

Ubuntu Impish

Ubuntu Impish armhf (the 32-bit image) will support the Zero 2 “out of the box” so you can just flash’n’go!

What about arm64? For that there’s a wrinkle (on which, more details below): you need an extra device-tree on the boot partition. On (or the day after? TBC) release day, we should have a kernel update that includes the new Zero 2 device-tree, so the “official” method of upgrading a card to work on the Zero 2 should simply be:

Boot your Ubuntu card on a supported Raspberry Pi (other than the Zero 2).
sudo apt update
sudo apt upgrade
Shut down the Pi. Put the card in your new Zero 2 and boot it!

What if you don’t have a spare non-Zero 2 Raspberry Pi? In that case there’s a bit of a hack you can use instead (which I’ve been using as a quick work-around during testing):

Flash a fresh Ubuntu Impish image to your choice of storage media (SD card, SSD, etc). Probably the easiest means of doing this is the Raspberry Pi Imaging utility (you’ll find Ubuntu under “Other general purpose OS”).
Once complete, safely remove the medium and re-insert. The boot partition (and potentially the root partition if you’re on Linux) will auto-mount.
Open the boot partition in your file-browser and find the bcm2710-rpi-3-b.dtb file. Make a copy of this file and rename it to bcm2710-rpi-zero-2.dtb. The trick here is simply that the Zero 2’s hardware is close enough to the 3B (minus the ethernet port, some RAM, and the DSI port) that the 3B’s device-tree works.
You’re done! You can now boot the card on your new Zero 2. Once you’ve done that, go through the usual sudo apt update, sudo apt upgrade dance to get the “proper” device-tree installed.

Ubuntu Focal

Ubuntu Focal, as the current LTS, will be receiving support for the Pi Zero 2 in the next kernel SRU cycle which is due out on Monday, November 8th. Once that lands, you can do the sudo apt update, sudo apt upgrade dance on another (non-Zero 2) Raspberry Pi to upgrade your card for compatibility with the Zero 2.

What if you don’t have another Pi? That gets a bit more tricky. First you have to get rid of u-boot from the boot sequence, then copy the 3B dtb as for Impish above. Once that’s done Focal seems to work reasonably happily (though I haven’t had time to do extensive testing, just make sure the basics like USB and WiFi are operating).

Pieces of 8²

Why isn’t all this copying of device trees necessary on the armhf image for Ubuntu Impish? It’s a funny story …

In order to get the Raspberry Pi Zero 2 to boot successfully (or any Pi for that matter), you need a device-tree for the board. That’s what all those “.dtb” files are on the boot partition. When the Pi starts up, the boot firmware checks what board it’s on, finds the appropriate device-tree file and loads it. The device-tree in turn tells the kernel what devices exist, what interfaces they’re connected to, bus addresses, etc. etc.

Obviously the device-tree for the Zero 2 has to be named bcm2710-rpi-zero-2.dtb. But that file doesn’t exist on the armhf image, so how can it boot?

In order to permit early testing, the Raspberry Pi developers sneaked in a hack of their own to the boot firmware. If the firmware finds itself running on a Zero 2, and also finds a bcm2708-rpi-zero-w.dtb (for the Raspberry Pi Zero W), it loads that then patches it to be equivalent to the appropriate Zero 2 device-tree. As it happens, despite not supporting the Zero W (which has an ARM6 processor; Ubuntu only supports the ARM7 arch and upward), our armhf images have always shipped with a (redundant) bcm2708-rpi-zero-w.dtb on the boot partition so by happy coincidence they “just work”.

However, the arm64 images didn’t ship with a (redundant) Zero W device-tree. And naturally, adding one would’ve looked … somewhat suspicious (“so Dave, why are you adding a device-tree for a board you definitely don’t support on any architecture?”), and obviously we couldn’t just shove a device-tree named bcm2710-rpi-zero-2.dtb on the boot partition without giving the whole thing away (and probably violating various NDAs)!

Fry from Futurama looks suspiciously at the new device-tree on the boot partition

Anyway, the 3B device-tree is a reasonably close match to the Zero 2’s hardware, and is sufficient for it to boot without further alteration. The only thing I’ve noticed that’s “off” when the 3B device-tree is in use is that /proc/asound/cards reports both the HDMI audio output, and the auxiliary headphone socket (the latter doesn’t exist on the Zero 2, and doesn’t appear when the “official” device-tree is in use).

And that’s all folks! Now, go treat yourself to the newest diminutive member of the ever-growing Pi family, and have some fun!

Playing with Blocks - LUKS

2021-10-19T00:00:00+01:00

See Also: LVM+LUKS

Last time we had a look at using LVM on our root storage. This time around it’s LUKS‘ turn: we’re going to make an Ubuntu Desktop for Raspberry Pi card, with the root file-system encrypted with LUKS. Thankfully this is much simpler affair as LUKS can encrypt, decrypt, or for that matter, re-encrypt a block device non-destructively (even with interruptions in the middle!).

As Easy As …

We get off to the simplest possible start. You’ll need a machine running Ubuntu and some form of Pi-compatible storage (an SD card, or some form of USB mass storage).

Install the Raspberry Pi imaging utility (or use sudo snap install rpi-imager for the snap version if you’re on an architecture other than amd64)
Grab an SD card or some form of USB storage (if you’re intending to use the desktop image, I’d recommend something fast like an SSD)
Install the Ubuntu 21.10 desktop image (substitute server if you wish) on the storage with the imaging utility (I won’t bother detailing this process because, frankly, the pi’s imaging utility is about as simple as you can possibly get!)
Boot the installed image on your Pi

Wot, no cryptsetup?

It might seem a bit odd that we’re jumping straight to booting the image, but there is a reason for this: on earlier versions of Ubuntu, the cryptsetup-initramfs package isn’t installed by default. On Impish it is installed by default, but the initramfs that ships with the image still doesn’t incorporate the cryptsetup binary that’s actually necessary. Hence, we need to boot it at least once and re-generate the initramfs to include it.

Get to a usable terminal (if you’re on the desktop image just go through the initial setup), then install cryptsetup-initramfs (as mentioned, on Impish this shouldn’t be necessary but best to make sure) and re-generate the initramfs.

$ sudo apt install cryptsetup-initramfs
... usual apt stuff ...
$ sudo update-initramfs -u
... lots of flash-kernel output ...

That last step will take several minutes; here’s some hints to read first if you want to speed things up a bit. Now shut down the Pi and plug the SD card (or USB drive) back into your original machine.

Setec Astronomy

Time to perform the actual encryption. Below I’m assuming that your SD card (or whatever target storage you’re using) appears as the /dev/mmcblk0 on your machine. Adjust these instructions according to your setup!

We’re going to unmount the partitions of the SD card (or other storage) you’ve just inserted (this is only necessary if your Ubuntu machine is a desktop machine which will auto-mount the card), force a file-system check of the root partition, and shrink that partition to its minimal size:

$ sudo umount /dev/mmcblk0p1
$ sudo umount /dev/mmcblk0p2
$ sudo fsck -f /dev/mmcblk0p2
fsck from util-linux 2.34
e2fsck 1.45.5 (07-Jan-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
writable: 154688/557520 files (0.2% non-contiguous), 1507014/2229303 blocks
$ sudo resize2fs -M /dev/mmcblk0p2
resize2fs 1.45.5 (07-Jan-2020)
Resizing the filesystem on /dev/sda2 to 1881115 (4k) blocks.
The filesystem on /dev/sda2 is now 1881115 (4k) blocks long.

Now for the actual encryption. First, pick a decent passphrase, then pick a decent encryption cipher. Below I’m going to use the Adiantum cipher as this performs reasonably well on the Pi. It’s well above the speed of most SD cards, but be aware it will probably hurt the performance of SSDs. Use cryptsetup benchmark --cipher xchacha12,aes-adiantum if you want an idea of its performance; compare this to the output of cryptsetup benchmark which runs through several of the more common algorithms.

At any rate, expect this next bit to take some considerable time!

$ sudo cryptsetup reencrypt --encrypt --reduce-device-size 16M --cipher xchacha12,aes-adiantum-plain64 /dev/mmcblk0p2

WARNING!
========
This will overwrite data on LUKS2-temp-dd54103b-d96c-4d1d-9e23-e3c375c81ecb.new irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for LUKS2-temp-dd54103b-d96c-4d1d-9e23-e3c375c81ecb.new:
Verify passphrase:
Progress:   1.8%, ETA 09:51,  160 MiB written, speed  14.4 MiB/s
...
Progress:  17.7%, ETA 08:05, 1544 MiB written, speed  14.8 MiB/s
...
Finished, time 09:45.528, 8700 MiB written, speed  14.8 MiB/s
$ sudo cryptsetup open /dev/mmcblk0p2 rootfs
Enter passphrase for /dev/mmcblk0p2:

Rooting Around

At this point our encrypted container on /dev/mmcblk0p2 is open as a decrypted device called /dev/mapper/rootfs. We can now expand the partition back to its “normal” size (minus the 16M we shrunk it for metadata), and mount it (and the boot partition) at some appropriate location.

$ sudo resize2fs /dev/mapper/rootfs
resize2fs 1.45.5 (07-Jan-2020)
Resizing the filesystem on /dev/mapper/rootfs to 2227255 (4k) blocks.
The filesystem on /dev/mapper/rootfs is now 2227255 (4k) blocks long.
$ sudo mkdir -p /mnt/boot /mnt/root
$ sudo mount /dev/mmcblk0p1 /mnt/boot
$ sudo mount /dev/mapper/rootfs /mnt/root

Now we need to perform some surgery on the file-system tables. Firstly, we need to tell the system how to set up the “rootfs” node we just opened. This is the job of the /etc/crypttab file. This should look relatively familiar to anyone used to messing with /etc/fstab, but bear in mind that the target and source devices are “backwards” (target first, then source).

Another important thing to remember here is that in this file, /dev/mmcblk0p2 is the device for the encrypted partition from the Pi’s perspective. In other words, if your SD card appears as /dev/sdf on your PC and you’ve been using /dev/sdf2 as the card’s root partition in the instructions above you should not do that here: just use /dev/mmcblk0p2. However, if you’re using an SSD via a USB adapter as your storage, you may need to adjust the line below to use /dev/sda2 as that’s how the Pi will be referring to this partition at boot time.

$ cat /mnt/root/etc/crypttab
# <target name> <source device>         <key file>      <options>
$ echo "rootfs  /dev/mmcblk0p2  none  luks,discard" >> /mnt/root/etc/crypttab
$ cat /mnt/root/etc/crypttab
# <target name> <source device>         <key file>      <options>
rootfs  /dev/mmcblk0p2  none  luks,discard

Note

You may or may not wish to include the “discard” option above. It’s the default in Debian and Ubuntu for encrypted partitions but may carry security risks. See man crypttab for full details.

With the /dev/mapper/rootfs device configured, all that remains is to tell the kernel and systemd to look there for its root device via the /etc/fstab table, and the cmdline.txt file on the boot partition:

$ cat /mnt/root/etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
LABEL=writable    /     ext4    defaults,x-systemd.growfs    0 0
/swapfile         none  swap    sw    0 0
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1
$ sudo sed -i -e 's,^LABEL=writable,/dev/mapper/rootfs,' /mnt/root/etc/fstab
$ cat /mnt/root/etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/rootfs    /     ext4    defaults,x-systemd.growfs    0 0
/swapfile         none  swap    sw    0 0
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1
$ cat /mnt/boot/cmdline.txt
dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash
$ sudo sed -i -e 's,LABEL=writable,/dev/mapper/rootfs,' /mnt/boot/cmdline.txt
$ cat /mnt/boot/cmdline.txt
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mapper/rootfs rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash

Note

Again, feel free to use an editor here instead of messing around with sed; I’ve illustrated the changes necessary by showing the before and after states of the file affected.

Fail! (and then succeed)

Time to clean up and test our card.

$ sudo umount /mnt/root /mnt/boot
$ sudo cryptsetup close rootfs

Eject the card (or mass storage) properly, shove it in your Pi and start it booting. This first boot will appear to take a very long time (several minutes, while it times out waiting for the root device to appear) and will then fail, dropping to the initramfs prompt. The reason for this is that, even though we re-generated the initramfs to include the cryptsetup tool, it still isn’t configured to actually unlock the correct partition. We’ll need to re-generate it again to do that, but first we need to get the system booted.

At the (initramfs) prompt run the following to get the system booting (if you’re booting an SSD drive, adjust /dev/mmcblk0p2 below to /dev/sda2):

(initramfs) cryptsetup open /dev/mmcblk0p2 rootfs
Enter passphrase for /dev/mmcblk0p2:
(initramfs) exit

Once the system has booted successfully, login and re-generate the initramfs one last time before rebooting to test your new setup:

$ sudo update-initramfs -u
$ sudo reboot

This time the initramfs should prompt you nicely to unlock your device before proceeding with the rest of the boot.

That’s all for this time. Let me know below if you run into any issues with this!

Playing with Blocks - LVM

2021-10-18T00:00:00+01:00

See Also: LUKS encryption

This is the first of a few posts I’ll be doing on customizing the storage of Ubuntu for Raspberry Pi images.

One of the first things I did when I moved to using a Pi as my primary workstation was set up LXD to enable “clean” build environments to help with debugging things. My favoured LXD storage backend is lvm …

(cue gasps of horror, followed by overlapping cries of “but zfs rules!” and “what about btrfs?”, before the two sides descend into several parallel flame wars over licensing and reliability and performance and … well, you get the point)

Yes, I like lvm. Which means I’d really like lvm on my Pi’s SSD, preferably for everything except the boot partition (which can’t be lvm because the bootloader doesn’t handle esoterica like that).

I’m going to demonstrate the necessary steps with a fresh Ubuntu 21.10 (Impish Indri) image on an SD card, but you can adapt the steps trivially to target an SSD drive on USB (just substitute /dev/sdX wherever you see /dev/mmcblkX below). With a teensy bit more effort, you could also operate on an existing installation rather than a fresh image (but again, you’ll need separate target storage — you can’t trivially convert an existing installation to lvm on the same block device).

Tabula Rasa

Start by grabbing your fresh SD card (or whatever storage you prefer). A word of (frankly obvious) warning here: we’re going to wipe everything on this storage so make sure you’re not going to mind losing anything on there first.

Instead of flashing an entire image to it, we’re going to set it up with our own partition table, create an LVM layout within that, then flash each partition of the image to it separately (all two of them, that is).

First things first: creating the new partition table. Plug in your storage, in my case an SD card. If you’re on an Ubuntu desktop (as I am) you’ll need to unmount any partitions that auto-mount from this storage before proceeding to (re-)partition it.

$ umount /dev/mmcblk0p1
$ umount /dev/mmcblk0p2
... repeat for any more auto-mounted partitions ...

Now we’re going to partition our storage with GPT (plus protective MBR), set up the first partition as FAT (for the bootloader bits), and the second partition as LVM. The commands we’ll execute are as follows:

(p)rint the current state of the partition table (not necessary, but just to be sure we’re looking at the “right” disk)
(o)verwrite the partition table (to wipe it and start again)
(n)ew partition, with the default number (1), default start sector (2048), a size of 512MB (actually larger than the source image which is 256MB, but that always feels a bit tight to me), and partition type 0700 (Microsoft basic data)
(n)ew partition again, with the default number (2), default start sector (1050624), default size (rest of the storage medium, however large that is), and partition type of 8e00 (Linux LVM)
(c)hange partition 1’s name to “system-boot”
(c)hange partition 2’s name to “lvm”
(p)rint the current state of the partition table again (not necessary, but just to confirm things “look right”)
(w)rite the new partition table and exit (confirming we want to proceed)

Here’s a transcript of me doing this with a 16GB SD card:

$ sudo gdisk /dev/mmcblk0
GPT fdisk (gdisk) version 1.0.5

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***************************************************************


Command (? for help): p
Disk /dev/mmcblk0: 31116288 sectors, 14.8 GiB
Model: STORAGE DEVICE
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 30D2AB17-E36C-4915-AF37-2C94E8FFE5C4
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 31116254
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          526335   256.0 MiB   0700  Microsoft basic data
   2          526336        31116254   14.6 GiB    8300  Linux filesystem

Command (? for help): o
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): y

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-31116254, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-31116254, default = 31116254) or {+-}size{KMGTP}: 512M
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 0700
Changed type of partition to 'Microsoft basic data'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-31116254, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-31116254, default = 31116254) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8e00
Changed type of partition to 'Linux LVM'

Command (? for help): c
Partition number (1-2): 1
Enter name: system-boot

Command (? for help): c
Partition number (1-2): 2
Enter name: lvm

Command (? for help): p
Disk /dev/mmcblk0: 31116288 sectors, 14.8 GiB
Model: STORAGE DEVICE
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): C202E4F2-59CD-402D-9720-3C8D7D0E84AF
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 31116254
Partitions will be aligned on 2048-sector boundaries
Total free space is 4061 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1048576   511.0 MiB   0700  system-boot
   2         1050624        31116254   14.3 GiB    8E00  lvm

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/mmcblk0.
The operation has completed successfully.

Note

You may note we’re using GPT partitioning above. This is okay provided you’re using a reasonably modern version of the Pi’s bootloader which has supported GPT partitioning since mid-2020. In other words, this should work with post-Focal images (including later Focal point-releases) but don’t try it with anything older than that.

More (logical) volume!

Now we set up the logical volume(s) on our GPT partitioned card.

$ sudo pvcreate /dev/mmcblk0p2
Physical volume "/dev/mmcblk0p2" successfully created.
$ sudo vgcreate pivg /dev/mmcblk0p2
Volume group "pivg" successfully created
$ sudo lvcreate --size 8G --name root pivg
Logical volume "root" created.

You might want to make the root LV bigger than 8G, especially if you’re using the desktop image. I’m specifically not using all the available storage (16GB in this case) because I assume you want some spare for … whatever it is you want lvm (in my case, a thin pool for LXD containers).

Going loopy

Time to grab a source image. Below, I’m getting a copy of the Ubuntu 21.10 pre-installed server image for the Pi (substitute the desktop image here if you like, there’s no real difference in the procedure), and unpacking it with the unxz utility:

$ sudo apt install -y xz-utils
... all the usual apt stuff ...
$ wget http://cdimage.ubuntu.com/releases/impish/release/ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz
$ unxz ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz

We need to access the individual partitions on the original image as we need to copy its boot partition to our new boot partition, then its root partition to our new root logical volume. To do this we set up a loop-device which treats our unpacked image as a disk in its own right, and check that it finds the right partitions:

$ sudo losetup --read-only --find --show --partscan ubuntu-21.10-preinstalled-server-arm64+raspi.img
/dev/loop34
$ ls -l /dev/loop34*
brw-rw---- 1 root disk   7, 34 Oct 15 14:01 /dev/loop34
brw-rw---- 1 root disk 259,  6 Oct 15 14:01 /dev/loop34p1
brw-rw---- 1 root disk 259,  7 Oct 15 14:01 /dev/loop34p2

In the example above (your loop-device will likely have a different number), the “loop34p1” and “loop34p2” block devices represent the first (boot) partition of the image, and the second (root) partition. We can now use good ol’ disk-destroyer to perform the block transfers (as ever with dd, be damned sure you get the “of=” device argument correct):

$ sudo dd if=/dev/loop34p1 of=/dev/mmcblk0p1 bs=16M status=progress
16+0 records in
16+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 15.8294 s, 17.0 MB/s
$ sudo dd if=/dev/loop34p2 of=/dev/mapper/pivg-root bs=16M status=progress
2030043136 bytes (2.0 GB, 1.9 GiB) copied, 1 s, 2.0 GB/s
226+1 records in
226+1 records out
3798978560 bytes (3.8 GB, 3.5 GiB) copied, 246.436 s, 15.4 MB/s

At this point, we can clean up the loop-device as we don’t need anything else from the source image.

$ sudo losetup -d /dev/loop34

Root Cause

Now mount the new boot partition so we can do a bit of surgery on where it should look for the root device (in our new logical volume):

$ sudo mkdir /mnt/boot
$ sudo mount /dev/mmcblk0p1 /mnt/boot
$ ls /mnt/boot
bcm2710-rpi-2-b.dtb       bootcode.bin  fixup_x.dat     start_db.elf
bcm2710-rpi-3-b.dtb       boot.scr      initrd.img      start.elf
bcm2710-rpi-3-b-plus.dtb  cmdline.txt   meta-data       start_x.elf
bcm2710-rpi-cm3.dtb       config.txt    network-config  uboot_rpi_3.bin
bcm2711-rpi-400.dtb       fixup4cd.dat  overlays        uboot_rpi_4.bin
bcm2711-rpi-4-b.dtb       fixup4.dat    README          uboot_rpi_arm64.bin
bcm2711-rpi-cm4.dtb       fixup4db.dat  start4cd.elf    user-data
bcm2837-rpi-3-a-plus.dtb  fixup4x.dat   start4db.elf    vmlinuz
bcm2837-rpi-3-b.dtb       fixup_cd.dat  start4.elf
bcm2837-rpi-3-b-plus.dtb  fixup.dat     start4x.elf
bcm2837-rpi-cm3-io3.dtb   fixup_db.dat  start_cd.elf
$ cat /mnt/boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash
$ sudo sed -i -e 's,root=[^ ]*,root=/dev/mapper/pivg-root,' /mnt/boot/cmdline.txt
$ cat /mnt/boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mapper/pivg-root rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash

Note

Feel free to use an editor here instead of messing around with sed; I’ve illustrated the changes necessary by showing the before and after states of the file affected.

Some (very similar) surgery is also required on the /etc/fstab file on our new root volume:

$ sudo mkdir -p /mnt/root
$ sudo mount /dev/mapper/pivg-root /mnt/root
$ cat /mnt/root/etc/fstab
LABEL=writable  /        ext4   discard,errors=remount-ro       0 1
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1
$ sudo sed -i -e 's,^LABEL=writable,/dev/mapper/pivg-root,' /mnt/root/etc/fstab
$ cat /mnt/root/fstab
/dev/mapper/pivg-root  /        ext4   discard,errors=remount-ro       0 1
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1

Finally, clean up all our mounts, deactivate the volume group containing our root volume, and remove all the temporary mount-points we created:

$ sudo umount /mnt/root
$ sudo umount /mnt/boot
$ sudo vgchange -an pivg
$ sudo rmdir /mnt/root
$ sudo rmdir /mnt/boot

Now eject the SD card (or whatever storage you’re using) and you should find it boots happily on your Pi, with the root storage as LVM. Let me know below if you run into any issues with this!

Making Impish Impier

2021-10-15T00:00:00+01:00

One of the things that constantly annoys me on Ubuntu (and therefore, one of the things I’m intending to fix this cycle) is how bloody long it takes for kernel upgrades to install! Most of this turns out to be down to two components (which … ahem … my team is responsible for): update-initramfs which spends nearly two minutes re-building the initrd.img for the boot partition, and flash-kernel which spends another two minutes copying everything to the boot partition.

Under (Com)pressure

At some point last week when I was testing a lot of different kernel bits (see the prior post for context!), this annoyed me to the point that I sat down to dig into it. The first bit update-initramfs was a bit tricky to analyze: there’s no really good means of profiling shell scripts sadly, so I wound up using the old PS4-calling-date trick and writing a quick Python script to analyze the results. Once this was done, however, one line stood out like a sore thumb; the compression of the resulting image.

Impish has switched to using zstd as its compression algorithm and, while this certainly compresses things better than gzip, it takes ages to run on an ARM processor. Specifically, even on an overclocked Pi 400 running at 2GHz it was taking 85 seconds (!) to compress the image. A quick change to /etc/initramfs-tools/initramfs.conf and we can chop that down to 10 seconds with lz4:

50 #
51 # COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ]
52 #
53 
54 #COMPRESS=zstd
55 COMPRESS=lz4

Admittedly the resulting image is much larger (~35MB with lz4 vs ~22MB with zstd) and that will affect boot speed. However, even at SD card read speeds that’s only one second lost per boot vs 75 seconds per kernel install (and lately I’ve been doing about one kernel install per boot!). Anyway, consider whether you do more or less than 75 boots per kernel install, and adjust accordingly!

Flash, ahh ahh!

On Ubuntu (and for that matter, Debian), the flash-kernel tool is charged with installing the bootloader, device-tree, kernel, and initrd (basically everything needed to get the system started) to … wherever they need to be on a given board. Sometimes that’s some NVRAM, sometimes a special ext4 partition, sometimes (as in the Pi’s case) that’s a FAT partition on some storage medium.

The Pi is quite an unusual case here inasmuch as its root storage is fully expected to go walkies (such as when upgrading an SD card on an old Pi and moving it to a new Pi model). Due to this, flash-kernel (in Ubuntu) copies the device-tree files for all Pi models (not just the one it finds itself on) to the boot partition every time the kernel is updated (along with everything else it usually copies). Unfortunately, the flash-kernel function that handles each transfer is quite slow and has a lot of overhead.

For most boards, this doesn’t matter as they copy (maybe) five files: a bootloader, a kernel, an initrd, a device-tree, and possibly some firmware or other scripts, or more often a single file which is some amalgam of the aforementioned pieces. But for the Pi there’s well over 200 individual files to copy and the result is a major slow-down.

Once I dug into the code the cause was pretty obvious. In the /usr/share/flash-kernel/functions script, the backup_and_install function on line 642 is the function we’re interested in:

642backup_and_install() {
643    local source="$1"
644    local dest="$2"
645    local do_dot_bak=$(get_dot_bak_preference)
646    local mtd_backup_dir=$(get_mtd_backup_dir)
647    if [ -e "$dest" ]; then
648        if [ -n "$do_dot_bak" ]; then
649            echo "Taking backup of $(basename "$dest")." >&2
650            mv "$dest" "$dest.bak"
651        else
652            echo "Skipping backup of $(basename "$dest")." >&2
653        fi
654    fi
655    # If we are installing to a filesystem which is not normally mounted
656    # then take a second copy in /var/backups, where they can e.g. be
657    # backed up.
658    if [ -n "$boot_mnt_dir" ] && [ -n "$mtd_backup_dir" ] ; then
659        local bak="$mtd_backup_dir/"$(basename "$dest")
660        #echo "Saving $boot_device:"$(basename "$source")" in $bak"
661        mkdir -p "$mtd_backup_dir"
662        cp "$source" "$bak"
663    fi
664    echo "Installing new $(basename "$dest")." >&2
665    mv "$source" "$dest"
666    maybe_defrag "$dest"
667}

Yes, it’s convoluted, but it needs to be for certain of the copying cases, and mostly it’s pretty quick. However, it’s the last line that’s interesting. That leads us to the “maybe_defrag” function on line 471:

471maybe_defrag() {
472    local file="$1"
473    local field="Bootloader-Has-Broken-Ext4-Extent-Support"
474    local broken_fw
475
476    if ! broken_fw=$(get_machine_field "$machine" "$field"); then
477        return
478    fi
479    if [ "$broken_fw" != "yes" ]; then
480        return
481    fi
482    if [ "$(df --output=fstype ${file} | sed -e 1d)" != "ext4" ]; then
483        return
484    fi
485    if ! command -v e4defrag > /dev/null; then
486        error "e4defrag command not found, unable to defrag $file"
487    fi
488    if ! e4defrag "$file" > /dev/null 2>&1; then
489        error "e4defrag of $file failed. Try freeing up space in /boot and re-executing flash-kernel"
490    fi
491}

From working on flash-kernel in the past, I know that “get_machine_field” calls are quite expensive (it’s essentially using shell-script to parse a text-file database). That’s why there’s a ton of calls caching these lookups around line 1008. Ultimately the “proper” fix is to cache the result of that call too. However, on the Pi the “maybe_defrag” function never does anything (and will never do anything) anyway, so a quick hack is to simply comment out that final line in backup_and_install:

642backup_and_install() {
643    local source="$1"
644    local dest="$2"
645    local do_dot_bak=$(get_dot_bak_preference)
646    local mtd_backup_dir=$(get_mtd_backup_dir)
647    if [ -e "$dest" ]; then
648        if [ -n "$do_dot_bak" ]; then
649            echo "Taking backup of $(basename "$dest")." >&2
650            mv "$dest" "$dest.bak"
651        else
652            echo "Skipping backup of $(basename "$dest")." >&2
653        fi
654    fi
655    # If we are installing to a filesystem which is not normally mounted
656    # then take a second copy in /var/backups, where they can e.g. be
657    # backed up.
658    if [ -n "$boot_mnt_dir" ] && [ -n "$mtd_backup_dir" ] ; then
659        local bak="$mtd_backup_dir/"$(basename "$dest")
660        #echo "Saving $boot_device:"$(basename "$source")" in $bak"
661        mkdir -p "$mtd_backup_dir"
662        cp "$source" "$bak"
663    fi
664    echo "Installing new $(basename "$dest")." >&2
665    mv "$source" "$dest"
666    #maybe_defrag "$dest"
667}

Once I’d put these two changes in place, I found that kernel installs went from taking more than 4 minutes, to slightly less than 2. Okay, still not fantastic but I am operating on an SD card, and a halving of the time for two trivial changes is not bad!

I’ll get the “proper” fixes in place during the next development cycle.

Common Sense (HAT)

2021-10-14T00:00:00+01:00

One of the main new features in Ubuntu for the Raspberry Pi is the addition of the packages to support the Raspberry Pi Sense HAT. Specifically, the packages you’re most likely to be interested in are:

sense-hat: The main Sense HAT package which depends on all the other bits you’ll need (like the Python library detailed below) and a basic configuration with some reasonable calibration figures
python3-sense-hat: The official Python library for interfacing to the Sense HAT (note: there’s no corresponding python-sense-hat library for Python 2.x on Ubuntu because Python 2.x is no longer supported)
sense-emu-tools: The Sense HAT desktop emulator package which includes the GUI desktop emulator, and the sense_rec and sense_play tools

Fake HATs

We’ll start with the Sense HAT desktop emulator package for a couple of reasons (full disclosure: these reasons have nothing whatsoever to do with my having written it … no, sir!). Firstly you can play with it even if you don’t have a Sense HAT, but secondly if you do have a Sense HAT, then you can use all the demo scripts that come with it!

Let’s get started with installing the HAT. This is my (at this point positively ancient) Sense HAT, mounted on a Pimoroni Black HAT Hacker board (because it makes it far easier to wiggle around without worrying about HDMI cables and such):

When you switch on the Pi you should see a rainbow image on the Sense HAT (after an initially blinding flash, if you happen to be on a Pi 4 — don’t worry, this is normal!). Once the kernel starts, the rainbow on the Sense HAT will go blank.

Note

If you’re using the Ubuntu Desktop image, you may see the LEDs on the Sense HAT show an orange strip on a white background which I think is plymouth trying to show an Ubuntu logo on a very low-res screen (you don’t see this on the Server image, which has no “pretty” boot screen).

Once you’re logged in and at a terminal prompt, install the packages we want:

$ sudo apt install sense-emu-tools sense-hat

Next, install a decent code editor. Below I’m using Geany (anyone who knows me is now crying “Wot! No vim?”, but that’s not something I’d foist upon the unsuspecting):

$ sudo apt install geany

Next, start the desktop emulator, which you should find under Gnome’s application menu (or you can hit the Super key and just type “Sense”; it should find it):

Finally (because I’m silly and forgot this bit), configure it to use geany (or your-choice-of-editor-here) because the default, thonny, isn’t in the Ubuntu archive … yet:

While you’re at it, I’d suggest activating the “Simulate environment sensors” and “Simulate inertial measurement unit” options too, as these make the emulation of these sensors more “realistic” (on those platforms with enough computing power, which is pretty much any modern computer with the exception of the Pi Zero).

Note

The Sense HAT emulator runs on any machine, not just the Raspberry Pi (you can install it on your Ubuntu PC running Impish), but we’re using on a Pi here partly to show how to use the emulator’s demo scripts with the real Sense HAT.

Let’s open up the humidity.py demo script. Go to “File / Open example / Simple / humidity.py” in the menus. This should fire up your chosen editor with a copy of this script, written to your home directory with a timestamp appended so that you can edit it as much as you’d like and subsequent selections of that entry will generate a “pristine” copy of the original:

If you’re using Geany (as suggested), there’s one more thing we need to before we can directly run this script. Geany still defaults to trying to call python rather than python3 when running Python scripts. Since Ubuntu no longer ships Python 2.x, the only interpreter is python3 (unless you’ve installed the python-is-python3 package), so we need to tell it to use this instead. Select “Build / Set Build Commands” from the Geany menu and change the Execute command to python3:

Finally, we can run our demo script. Select “Build / Execute” from the menu, or click the cogs icon in the toolbar. You should see the display on the emulated Sense HAT turn partly white and partly green. The number of green elements indicates the current humidity. If you drag the humidity slider up and down you should see the number of green elements change accordingly.

Note

If you activated environment simulation earlier, there may be some delay between you dragging the slider and the display updating. This is because the real sensor typically displays a certain delay in responding to changing humidity which the simulation emulates.

When you want to stop the script, press Ctrl+C in the (apparently empty) terminal window that appeared when you ran the script (that terminal actually contains the Python interpreter running your script). Feel free to modify the script and see what difference it makes when you run it. Simple changes could be modifying the colours used, more complex ones could change the orientation or appearance of the “graph”.

Real HATs

What about running our humidity script on the real Sense HAT though? This is pretty easy too. Firstly, edit the script to import from sense_hat instead of sense_emu. This is the only code change that’s necessary:

Next, we need to ensure our script starts as “root”. This is because the I²C bus that the Sense HAT uses is only accessible to root (by default, see below) under Ubuntu currently (this is something I need to correct — but carefully — before the next release!). Again, bring up Geany’s “Build / Set Build Commands” dialog, and change the Execute command to include a “sudo” prefix (i.e. it should read sudo python3 "%f":

Once again, select “Build / Execute” from the menu. This time, the terminal that appears will prompt for your password (which sudo needs to run things as root). Once entered you should see the script running on the actual Sense HAT on your Pi:

In the image above the reading is quite high as I’ve just breathed on the HAT. You can try the same and watch the reading spike up before falling back down (assuming your ambient relative humidity is fairly low that is!)

At this point, I’d suggest having a play with the other demo scripts available in the emulator. There’s plenty in there that deal with the various environmental sensors, the display, the joystick, and the IMU on the HAT. Have fun!

Rootless

If you want to use the Sense HAT without resorting to sudo or root, you need to add some udev rules to permit access to regular users. Add the following content to a file named /etc/udev/rules.d/99-user-sense.rules (you will need to be root to create/edit this file):

SUBSYSTEM=="i2c-dev", KERNEL=="i2c-[0123456]", GROUP="plugdev", MODE="0660"
SUBSYSTEM=="input", ENV{LIBINPUT_DEVICE_GROUP}=="*:rpi-sense-joy", GROUP="plugdev", MODE="0660"
SUBSYSTEM=="graphics", ENV{ID_PATH}=="*-rpi-sense-fb", GROUP="plugdev", MODE="0660"

Note

If you’re typing this instead of copying and pasting, pay close attention to the subtle distinctions between “==” and “=”. The first entries on a line use “==” as they’re matching events; the latter entries use “=” as they’re assigning values to the device.

After creating/editing this file, reboot the machine and you should find that you can access your Sense HAT without “sudo”.

When HDMI freezes over

2021-10-13T00:00:00+01:00

Well, it’s time for another Ubuntu release, and so time for another slew of blog posts from me! In this particular case though, I need to start with a mea culpa on the state of the Ubuntu Desktop for Raspberry Pi release.

The bug, the bodge, and the deadline

Shortly before release, a bug in HDMI output handling (initially part of another bug) was discovered which vexed us right up to the release itself. The symptoms ranged from the annoying (display output freezes after a certain amount of use, usually a couple of hours in my case) to the critical (display output freezes during boot, in the case of a colleague).

The determining factor eventually turned out to be something to do with high refresh-rate monitors. With my eye-sight, I’ve never wanted or needed anything beyond good ol’ 1080p at 60Hz … and so that’s what all my monitors are. But younger colleagues (with better eye-sight) have fancier displays with higher frame-rates and on these the issue was most definitely critical.

We quickly found a work-around (which is in the release notes) which was simply to replace the “kms” overlay in config.txt with the “fkms” overlay, but then the race was on to fix things before release.

The kernel team worked tirelessly to try and come up with a fix, but it quickly became apparent this was an upstream issue too, and that it was unlikely we’d find a fix on our own. Eventually it was obvious that the bug was going to make it into the release images, and so the debate switched to whether to release with the fkms workaround in place.

To bodge, or not to bodge

As will be obvious to anyone who’s tried the desktop release with a high-refresh rate monitor, we released without the workaround in place (and that’s the royal we, because ultimately this was my decision so if you’re going to yell at anyone, yell at me — comments below!).

By way of (lengthy) explanation, here was my reasoning:

We have regular meetings with the Pi Foundation (or more precisely the engineers at Raspberry Pi Trading Ltd. — just to appease Ben’s nagging of my constant conflation of these entities!). My understanding from these meetings is the “kms” overlay is considered the “future direction of development”. As such, it is under active development in the upstream kernel.
The “fkms” overlay is considered legacy, and while it is supported in the kernel in use in Raspberry Pi OS, if it happens to break in the development kernel (which is where the Ubuntu kernel patches are sourced from), that’s acceptable.
We have a mechanism for modifying the boot configuration during a release upgrade, but no such mechanism currently exists for kernel upgrades. The modifications performed during release upgrades are not perfect (there are edge cases we cannot reasonably account for), but a release upgrade is a sufficiently major operation that a certain amount of breakage might be considered reasonable in the case of esoteric configurations. The same cannot be said of kernel upgrades.
Other (minor) points against “fkms”: audio doesn’t operate correctly (choppy) under “fkms” without the “tsched=0” workaround, and video displays screen-tearing. These are obviously non-critical, but it’s a sub-par experience compared to fixing “kms” properly.

To summarise: we could switch people to the “fkms” overlay during the impish release upgrade (and obviously modify the release image to use “fkms” too) but that would leave us in a position where a future kernel upgrade during the release broke things (in fact, our testing with a few interim upstream kernels suggested this was already the case), and there was no good mechanism for fiddling with the boot configuration during a regular apt update, apt upgrade procedure.

Alternatively, we could release with the “kms” overlay knowing full well that would break badly on certain high-fps monitors, but with a release noted work-around that wasn’t terribly complex (editing one line in a text file, which was accessible on all platforms), and the hope that we could fix the kernel in a (quickly released) update. Obviously this would still entail pain for people having to use the workaround to even boot (in order to install such an upgrade). But it would also mean that such people would have experience of installing the workaround and thus would have little difficulty in removing it themselves, without having to bodge a postinst script into the kernel to try removing it (probably imperfectly).

Conclusion … ?

As it turns out, we may have a fix quicker than I’d hoped: as I write this final section, I’m just waiting on another SD card to flash to play with a testing kernel which may fix at least part of the issue (I wasn’t kidding when I said the kernel team were working tirelessly on this!). There are also a couple of patches that appear to stabilize things completely, at least on my old 1080p 60Hz monitor.

Anyway, to all those affected, all I can say for now is “sorry” that this release isn’t everything I’d hoped it to be, and that the release includes this issue. That’s on me: I could’ve made the call to switch to “fkms” for the release, but I genuinely think that although this is obviously the more painful course, it’s the right one for a better experience in the end.

You boot — no, u-boot first!

2021-05-05T00:00:00+01:00

A frequently asked question on the various #ubuntu-* channels I lurk around on FreeNode is that of getting Ubuntu Focal (20.04) booting from USB on a Pi. The major issue here is that Focal still uses U-Boot and that doesn’t work with USB boot on the Pi (our U-Boot build does now support both the USB2 and USB3 ports on the Pi 4, but the MSD support doesn’t work for some reason).

However, U-Boot isn’t critical to booting Ubuntu. In fact, on the armhf images it’s basically redundant. On the arm64 images though it does have one very important job: decompressing the gzip’d kernel.

How come the kernel doesn’t decompress itself? Because arm64 kernels can’t. For whatever reason, the Linux kernel includes self-extraction code for x86 and armhf … but not arm64 where “that’s the bootloader’s job”. Around the time of Focal’s release, the Pi’s native bootloader didn’t support any kind of extraction, so we shipped a U-Boot script to handle decompressing the gzip’d kernel.

Later on in 2020, the Pi’s bootloader got an upgrade to support decompressing gzip’d kernels but Focal’s boot sequence still uses U-Boot because forcing a boot sequence change on existing installations is not something I’m keen to do unless absolutely necessary (that said, it’s something that’ll need tackling for the upgrade to the next LTS).

On a Fresh Install

First let’s tackle installing a completely new Focal image on a USB SSD and adapting it so it’ll boot from USB. Bear in mind your Pi 4 will already need to be configured to boot from USB so do that first if you haven’t already (this is one-time setup though so if you can already boot your Pi 4 from USB, you’re all set).

Flash the Focal arm64 image (which must be point release 2 or later) straight to your SSD drive. The easiest way to do this is with the Raspberry Pi imaging utility:

Slot your SSD drive into a USB3 enclosure of some sort (personally I find the Inateck FE2004 works nicely) and plug that into your PC
Start up the imaging utility
Select “CHOOSE OS”, “Other general purpose OS”, “Ubuntu”, “Ubuntu Server 20.04.2 LTS (RPi3/4/400)” (if you’re reading this from the future, any later point release like .3 will be fine too)
Select “CHOOSE STORAGE” and pick your SSD drive (if you’ve no other removable storage attached it’ll probably be the only thing listed)
Select “WRITE” and twiddle thumbs / make coffee while everything finishes

Now, jump to the “Boot Surgery” section below!

From an Existing Install

Converting an existing SD card installation is a very similar procedure. First we’ll deal with transferring the content of the SD card to the SSD. I’m assuming here that your SSD is at least as large as your SD card. If that isn’t the case you’ll need to go and read about shrinking partitions and such-like but it sounds rather unlikely so I’m going to skip all that here.

The first step is simply to make sure that your installation is up to date. You must have an updated bootloader for this to work, so boot your SD card and run a typical sudo apt update, sudo apt upgrade cycle. Then shut the Pi down, and get a machine running Linux other than your Pi with an SD card reader and a USB port.

Note

It is possible to do this on a Pi, especially one that’s already USB booted where you can use the internal SD reader as a source, and a spare USB port as a target, but that’s rather circular so for the purposes of this article let’s assume you’ve got an Ubuntu PC lying around!

On your PC, insert your SD card and your SSD in some appropriate USB enclosure. I’m going to assume your SD card appears as /dev/mmcblk0 and your USB enclosed SSD as /dev/sdf below. If you want to be certain, before inserting the devices run sudo dmesg -w and watch the output as you insert them. The device names should be apparent from the output.

First, ensure nothing on the SD card or SSD is mounted: sudo umount /dev/mmcblk0p1, sudo umount /dev/mmblk0p2, …, sudo umount /dev/sdf1, sudo umount /dev/sdf2. Check the output of mount to make sure no partitions from these devices are still mounted.
Next transfer everything from the source to the target with good ol’ disk-destroyer: sudo dd if=/dev/mmcblk0 of=/dev/sdf bs=16M status=progress; be damned sure you get the of= device name correct here!
Be warned this transfer will take much longer than just writing a normal image as it’ll include all the blank unused blocks on the SD card too, so prepare for much thumb twiddling / coffee drinking!
Run sudo sync (probably unnecessary but no harm in a little paranoia)

Now continue with …

Boot Surgery

Now for some surgery on the boot configuration. Unplug the SSD’s enclosure from your PC and then plug it back in so that the freshly written boot partition (labelled system-boot) auto-mounts (if your PC is running Linux, the writable partition will likely auto-mount too, but system-boot should auto-mount on everything including Windows and Mac OS X).

On the system-boot partition that’s mounted, find the config.txt file and open it in a text editor (I think Notepad can handle UNIX line-endings these days; Windows users please let me know if this isn’t the case!). It should look like this:

config.txt

 1 # Please DO NOT modify this file; if you need to modify the boot config, the
 2 # "usercfg.txt" file is the place to include user changes. Please refer to
 3 # the README file for a description of the various configuration files on
 4 # the boot partition.
 5 
 6 # The unusual ordering below is deliberate; older firmwares (in particular the
 7 # version initially shipped with bionic) don't understand the conditional
 8 # [sections] below and simply ignore them. The Pi4 doesn't boot at all with
 9 # firmwares this old so it's safe to place at the top. Of the Pi2 and Pi3, the
10 # Pi3 uboot happens to work happily on the Pi2, so it needs to go at the bottom
11 # to support old firmwares.
12 
13 [pi4]
14 kernel=uboot_rpi_4.bin
15 max_framebuffers=2
16 
17 [pi2]
18 kernel=uboot_rpi_2.bin
19 
20 [pi3]
21 kernel=uboot_rpi_3.bin
22 
23 [all]
24 arm_64bit=1
25 device_tree_address=0x03000000
26 
27 # The following settings are "defaults" expected to be overridden by the
28 # included configuration. The only reason they are included is, again, to
29 # support old firmwares which don't understand the "include" command.
30 
31 enable_uart=1
32 cmdline=cmdline.txt
33 
34 include syscfg.txt
35 include usercfg.txt

First off, ignore that warning at the top (I wrote it, so take it from me that you can ignore it!). I’ve covered why it’s there and why it was wrong before, so go read that if you’re interested, but for now all that’s important is that you can happily modify this file without breaking stuff (if you’re reasonably careful). In fact, feel free to delete that comment :-)

Now, comment out all the kernel= lines pointing to U-boot, and the device_tree_address= line. Then under the [all] section add kernel=vmlinuz, and initramfs initrd.img followkernel. Your file should wind up looking something like this:

config.txt

 1# The unusual ordering below is deliberate; older firmwares (in particular the
 2# version initially shipped with bionic) don't understand the conditional
 3# [sections] below and simply ignore them. The Pi4 doesn't boot at all with
 4# firmwares this old so it's safe to place at the top. Of the Pi2 and Pi3, the
 5# Pi3 uboot happens to work happily on the Pi2, so it needs to go at the bottom
 6# to support old firmwares.
 7
 8[pi4]
 9#kernel=uboot_rpi_4.bin
10max_framebuffers=2
11
12[pi2]
13#kernel=uboot_rpi_2.bin
14
15[pi3]
16#kernel=uboot_rpi_3.bin
17
18[all]
19arm_64bit=1
20#device_tree_address=0x03000000
21kernel=vmlinuz
22initramfs initrd.img followkernel
23
24# The following settings are "defaults" expected to be overridden by the
25# included configuration. The only reason they are included is, again, to
26# support old firmwares which don't understand the "include" command.
27
28enable_uart=1
29cmdline=cmdline.txt
30
31include syscfg.txt
32include usercfg.txt

Obviously if you want to undo the changes you can simply comment out the lines you’ve added and uncomment the ones you commented out (one of the joys of textual configuration being that leaving obsolete stuff commented out for reversion is trivial!).

Save your changes, and safely remove your USB enclosure however that’s done in your particular OS (don’t just unplug it, you need to make sure everything’s unmounted cleanly).

And you’re done! Connect the enclosure to a free USB3 port on your Pi 4 and you should be good to go.

The Pins They Are A-Changin’

2021-04-20T00:00:00+01:00

See Also: Breaking GPIO(ish)

Some fairly major (and frankly rather overdue) changes are coming in the way that users access the GPIO pins on the Raspberry Pi. The shift will (hopefully) be fairly gradual and, for users of certain libraries, entirely invisible. However, for others they may be quite disruptive. As forewarned is forearmed, hopefully this little article may be of some use to those affected!

I really must stop writing “little” or “quick” in a desperate attempt to persuade my writing tendencies toward brevity … it never works … my humble apologies, dear reader, for the extended ramble you are about to endure.

Ancient History

Our story begins in antiquity, which is to say Linux kernel version 3.16, circa 2014. The Raspberry Pi is growing in popularity and everyone is keen to use these new (to the PC world anyway) things called “pins”.

The simple, neat theory of hardware access is that a kernel mediates all access to your hardware, and your program interacts with kernel APIs of some form to indirectly use that hardware. For the GPIO pins on the Pi, this meant the GPIO sysfs interface. This is still found today under the /sys/class/gpio path in the file-system. The two important paths under here are export and unexport.

Let’s say you have an LED on GPIO 26 that you want to control, like so:

Breadboard diagram illustrating an LED and a 330Ω resistor connected in series between GPIO 26 (on pin 37) and ground (on pin 39)

You can do so via simple file-system operations (the following assumes you have write access to this area of the file-system, which is the case for the default user on RaspiOS, but not Ubuntu so switch to root if you want to try it there):

$ cd /sys/class/gpio
$ ls
export  gpiochip0  gpiochip504  unexport
$ echo 26 > export
$ ls
export  gpio26  gpiochip0  gpiochip504  unexport
$ ls gpio26
active_low  device  direction  edge  power  subsystem  uevent  value

As we can see, writing “26” to export has magically created a new gpio26 path that has various files under it which we can use to manipulate the GPIO:

$ cat gpio26/direction
in
$ echo out > gpio26/direction
$ cat gpio26/value
0
$ echo 1 > gpio26/value

Presto! At this point, your LED will be lit.

As you may guess, you can change the GPIO back into an input by writing “in” to gpio26/direction. You can read gpio26/value to determine the value of the input. You can perform edge detection by writing values to the gpio26/edge file, and so on.

This may look simple and neat, but in practice there’s a lot of problems with it. At the root of most of those problems is the issue that the file-system is a global resource, so this API provides no concept of “ownership”.

If process A and process B both have write access to the GPIO sysfs interface, both can export GPIO 26. What’s stopping them both from accessing GPIO 26 at the same time?

Nothing.

Can this be worked around with file-system permissions? Sadly, it’s not that simple: what process sets those permissions? How do you avoid race conditions when changing permissions of multiple file-system entries?

For instance, on RaspiOS, to permit GPIO access to ordinary users, rules were added that change the ownership of these files to permit members of the “gpio” group to access them. However, that change isn’t instantaneous when the gpio26 path is created, so all GPIO libraries relying on it have hacks that look something like:

Check if gpio26 exists; if it does, exit
Write “26” to gpio/export
Check gpio26 exists
Check gpio26/direction is writable
If not, wait a few milliseconds and try again
If after, say, 1 second we still can’t write to it, assume we’ll never be able to and throw an error

Urgh! You can see what this looks like in gpiozero’s “native” driver under the GPIOFS.export method (full disclosure: this part of gpiozero is largely my fault).

It gets worse …

Due to that first check, if the GPIO is already exported, our only option is to go ahead and use it. Maybe it’s left over from something that crashed before it could un-export it? Maybe it’s currently in use by another process? We can’t know, so we just barge in and use it; a library could throw a warning but there’s precious little a user could do in response to that warning.

And there’s another problem: this interface is slow. Sufficiently slow that back in the Pi 1 days, it was considered effectively useless for Pulse Width Modulation (PWM). This is a technique of rapidly pulsing a pin which is commonly used to provide variable speed control on motors, position information for servos, etc.

What’s faster than sysfs?

How The Wave Was Done

If the sysfs interface was too slow, and the Linux kernel (prior to 4.8) provided no alternative … how did userspace (that is, “not kernel”) processes manage PWM back in the Pi 1 days? There were two options:

Do things the Right Way™, with hardware! The Pi provides a hardware driven PWM port capable of fantastic speeds. Because it’s hardware driven it also doesn’t use any CPU time and is wonderfully stable as it’s not interrupted by things like I/O. Perfect!

Unfortunately it’s only available on a single pin (actually on modern Pis there’s a choice of 4 pins, but there’s still only two hardware PWM channels you can use).
Get hacky! Just bang on the hardware registers from userspace! If you’re using a compiled language like C, you can still go pretty damned fast, and you can use whichever and however many pins you want (provided the CPU can keep up).

Take a wild guess which one got used!

To be clear, I don’t blame anyone for this choice. The limitation of a single pin for hardware PWM was a major problem. What if you have two servos to control (pitch and yaw, say)? The “correct” answer is “go buy more hardware like an I²C servo driver” but writing code is often cheaper, and faster than waiting for the postman.

But hang on a tick … doesn’t the kernel control the hardware? How does a userspace application get to bang on hardware registers?! Simple! Get root privileges, find the relevant area for the GPIO registers in /dev/mem (a “special file” that represents all the physical memory in your machine) and write to it!

This is, shall we say, not what userspace programs are meant to do. Things got slightly more respectable along the way, of course (they had to!). Running everything as root was a problem that was solved by producing a device called /dev/gpiomem which simply gave access to the portion of memory that contained the GPIO registers and nothing else. And numerous libraries for numerous languages sprung up to paper over these horrors and present a nice user-friendly GPIO interface. Still, if all this sounds incredibly hacky … well, that’s because it was.

Or rather is.

Yes, dear reader, this is still how the vast majority of GPIO control on the Pi is done on at the time of writing. This is how wiringPi, pigpio, RPi.GPIO, and gpiozero’s “native” driver all operate (yes kids, the latter is pure Python banging directly on hardware registers; you don’t have to learn C just to abuse stuff!).

Doesn’t stomping all over the registers under the kernel’s nose cause any issues? Especially considering the kernel’s also trying to control those same pins via the sysfs interface, using those same registers?

Well … yes …

The Bleeding Edge

Almost from the start it became apparent that attempting to perform edge detection (nothing to do with images; this is simply responding to the rising or falling “edge” of a pulse on a pin) didn’t work with the registers. This was one area where the Linux kernel exerted absolute control.

You could configure pins and drive them high and low quite happily without the kernel complaining, but mess with the edge detection registers and stuff started crashing.

Ultimately this meant that most libraries wound up being rather schizophrenic. They’d “play nice” and use the sysfs interface for edge detection, but then stomp all over the hardware registers whenever they wanted to do anything else (reading the value of a GPIO, setting its direction, writing a value, etc.).

And until very recently, this has worked happily …

As Neat as a Pin

Time rolls on, the seasons change, and Linux kernel 4.8 is released.

This kernel introduced a new “gpiochip” device file (e.g. /dev/gpiochip0). Such files have a set of ioctls (“special” commands for special files) that provide the means to manipulate the GPIOs, and this is the basis of the new API for userspace control of the pins.

By far the most important change is that now there’s a concept of “ownership”. Before your process can do anything with a GPIO it has to request access to it (via an ioctl). In doing so, the kernel will track your process’ use of that GPIO, and won’t permit any other process to use it simultaneously. Should your process terminate without releasing it, the kernel will notice and free up the ownership automatically.

So far, so good.

A userspace library (libgpiod) exists to provide a friendly interface to all the ioctls for the gpiochip device, and it comes with a few nice command line utilities like gpioinfo which can be used to determine what GPIOs exist and whether they are currently in use by anything. You can get all this by installing the “gpiod” package. For example:

$ sudo apt install gpiod
... all the usual apt stuff ...
$ gpioinfo
gpiochip0 - 58 lines:
        line   0:     "ID_SDA"       unused   input  active-high
        line   1:     "ID_SCL"       unused   input  active-high
        line   2:       "SDA1"       unused   input  active-high
        line   3:       "SCL1"       unused   input  active-high
        line   4:  "GPIO_GCLK"       unused   input  active-high
        line   5:      "GPIO5"       unused   input  active-high
        line   6:      "GPIO6"       unused   input  active-high
        line   7:  "SPI_CE1_N"   "spi0 CS1"  output   active-low [used]
        line   8:  "SPI_CE0_N"   "spi0 CS0"  output   active-low [used]
        line   9:   "SPI_MISO"       unused   input  active-high
        line  10:   "SPI_MOSI"       unused   input  active-high
        line  11:   "SPI_SCLK"       unused   input  active-high
        line  12:     "GPIO12"       unused   input  active-high
        line  13:     "GPIO13"       unused  output  active-high
        line  14:       "TXD1"       unused   input  active-high
        line  15:       "RXD1"       unused   input  active-high
        line  16:     "GPIO16"       unused   input  active-high
        line  17:     "GPIO17"       unused   input  active-high
        line  18:     "GPIO18" "gpio-fan@0"  output  active-high [used]
        line  19:     "GPIO19"       unused   input  active-high
        line  20:     "GPIO20"       unused   input  active-high
        line  21:     "GPIO21"       unused   input  active-high
        line  22:     "GPIO22"       unused   input  active-high
        line  23:     "GPIO23"       unused   input  active-high
        line  24:     "GPIO24"       unused   input  active-high
        line  25:     "GPIO25"       unused   input  active-high
        line  26:     "GPIO26"      "sysfs"   input  active-high [used]
        line  27:     "GPIO27"       unused   input  active-high
        line  28: "RGMII_MDIO"       unused   input  active-high
        line  29:  "RGMIO_MDC"       unused   input  active-high
        line  30:       "CTS0"       unused   input  active-high
        line  31:       "RTS0"       unused   input  active-high
        line  32:       "TXD0"       unused   input  active-high
        line  33:       "RXD0"       unused   input  active-high
        line  34:    "SD1_CLK"       unused   input  active-high
        line  35:    "SD1_CMD"       unused   input  active-high
        line  36:  "SD1_DATA0"       unused   input  active-high
        line  37:  "SD1_DATA1"       unused   input  active-high
        line  38:  "SD1_DATA2"       unused   input  active-high
        line  39:  "SD1_DATA3"       unused   input  active-high
        line  40:  "PWM0_MISO"       unused   input  active-high
        line  41:  "PWM1_MOSI"       unused   input  active-high
        line  42: "STATUS_LED_G_CLK" "led0" output active-high [used]
        line  43: "SPIFLASH_CE_N" unused input active-high
        line  44:       "SDA0"       unused   input  active-high
        line  45:       "SCL0"       unused   input  active-high
        line  46: "RGMII_RXCLK" unused input active-high
        line  47: "RGMII_RXCTL" unused input active-high
        line  48: "RGMII_RXD0"       unused   input  active-high
        line  49: "RGMII_RXD1"       unused   input  active-high
        line  50: "RGMII_RXD2"       unused   input  active-high
        line  51: "RGMII_RXD3"       unused   input  active-high
        line  52: "RGMII_TXCLK" unused input active-high
        line  53: "RGMII_TXCTL" unused input active-high
        line  54: "RGMII_TXD0"       unused   input  active-high
        line  55: "RGMII_TXD1"       unused   input  active-high
        line  56: "RGMII_TXD2"       unused   input  active-high
        line  57: "RGMII_TXD3"       unused   input  active-high
gpiochip1 - 8 lines:
        line   0:      "BT_ON"       unused  output  active-high
        line   1:      "WL_ON"       unused  output  active-high
        line   2: "PWR_LED_OFF" "led1" output active-low [used]
        line   3: "GLOBAL_RESET" unused output active-high
        line   4: "VDD_SD_IO_SEL" "vdd-sd-io" output active-high [used]
        line   5:   "CAM_GPIO"       unused  output  active-high
        line   6:  "SD_PWR_ON" "sd_vcc_reg"  output  active-high [used]
        line   7:    "SD_OC_N"       unused   input  active-high

That’s a whole lotta output! But, hold on! If you look over the list, you’ll see that GPIO 26 is apparently “used” by “sysfs”. Yup, the old interface is still there, but now it’s implemented as something on top of the new gpiochip device(s).

This is actually good news: it means nothing breaks immediately (sort of). The bad news is: remember how things using sysfs can leave things exported? If anything does, the user needs to manually unexport them before anything using the new interface can use them. To be fair, there’s no good way around this, and I think this is probably the best compromise possible in the circumstances.

How would controlling our LED look with libgpiod under Python?

>>> import gpiod
>>> chip = gpiod.Chip('0', gpiod.Chip.OPEN_BY_NUMBER)
>>> line = chip.get_line(26)
>>> line.request('my-script')
>>> line.set_direction_output(1)

All this looks very promising; the problems with the sysfs interface are dealt with and there’s now a simple, neat solution to controlling GPIO pins from userspace in the form of libgpiod. Which begs the question: if all this came along in kernel version 4.8, circa 2016, why are the vast majority of userspace GPIO libraries on the Pi still banging on registers?

Two reasons: firstly, it’s still slower than banging on registers. No real surprise there, but it’s not much slower. However, a bigger problem is that (in the initial version of this new interface, at least), there was still something missing …

Pulling the Pin

GPIO pins are remarkably versatile things. We’ve seen them acting as an output. But how about as an input? Imagine we have a push-button connected between GPIO 13 and ground, like so:

Breadboard diagram adding a momentary push-button connected between GPIO 13 (on pin 33) and ground (on pin 39) to the earlier circuit

How do we go about detecting when this button is pressed?

We need the GPIO to be an input. No problem there.
We need to wait for a falling edge when the button connects the GPIO to ground. Again, no problem.
We need the GPIO to be pulled up to ensure there’s a potential to fall (or at least enough to register a difference on a GPIO which would otherwise be floating).

Go back and have a look at the sysfs interface we saw earlier. See any pseudo-file for configuring a pull?

No, and this was another big reason why all the libraries resorted to banging on the registers; the kernel interface of the time didn’t support a fairly major piece of functionality, and one that is useful in the most basic of operations like detecting when a switch closes.

Sadly, the brand new gpiochip interface, also omitted this functionality in its first incarnation (and all the way up to the version that shipped with Debian and RaspiOS “Buster”), so even if the myriad GPIO libraries had wanted to move to it as an underlying layer, they’d still have to bang on registers to implement some of their existing interface.

C’est la vie …

Pins and Needles

Allow me a brief interlude, dear reader, to explore userspace GPIO access from the perspective of the kernel.

Accessing GPIOs from userspace is considered an aberration that should never be tolerated in any end-user product. That statement may sound rather strong, but here’s a quote from the kernel’s GPIO sub-system documentation:

The userspace ABI is intended for one-off deployments. […] Do not under any circumstances abuse the GPIO userspace ABI to cut corners in any product development projects. If you use it for prototyping, then do not productify [sic] the prototype: rewrite it using proper kernel drivers. Do not under any circumstances deploy any uniform products using GPIO from userspace.

If the Kernel Gods hold my work in disfavour, I can only plead that gpiozero is a library for educators and makers, not mass-produced products.

Furthermore, I have some considerable sympathy with this view. GPIOs are a means to an end, and userspace should be concerned with the “end”, not the “means”. In fact this is a large part of gpiozero’s philosophy: that by concentrating on the things attached to pins (like LEDs and buttons) rather than the pins themselves, everything becomes much simpler (I’d love to claim credit for this stroke of genius, but it was another of Ben’s ideas).

Still, gpiozero is a userspace GPIO library, so perhaps I’m damned regardless. In the course of my background research for this article, I did come across a presentation (PDF) covering the introduction of the gpiochip interface which included “The Rules of Linux Userspace GPIO”:

You do not access GPIOs from userspace

YOU DO NOT ACCESS GPIOS FROM USERSPACE

Read Documentation/gpio/drivers-on-gpio.txt

Use the character device [/dev/gpiochipN]

And now I’m picturing Tyler Durden railing against the kids at a Raspberry Jam for breaking rule #1 :-)

Bang Up To Date

Anyhow, that brings us to the present day. Ubuntu Hirsute Hippo is releasing shortly (a couple of days away as I write this), and will ship with Linux kernel version 5.11. This has a gpiochip device (and libgpiod) which does support setting pulls, and is therefore (mostly) capable of doing everything the traditional GPIO libraries do.

Hang on … “mostly”?

Unfortunately, software PWM is still missing from this. But fret not! The creator of one of the traditional GPIO libraries (pigpio) has produced another GPIO library built upon the new gpiochip device called “lg“. This is a well and truly feature-complete library covering all the GPIO operations discussed above (including a very full-featured PWM implementation), and covering the SPI, I²C, and UART kernel devices.

This will be include in Ubuntu Hirsute with the packages:

liblgpio1 — the C library
python3-lgpio — the Python 3 bindings for liblgpio

It will also be included in RaspiOS Buster shortly (I can only apologise I haven’t sent the packaging for this yet, but last minute release debugging has swamped me! I will get to it shortly!), and is a fine interface to migrate your GPIO projects to if you’re currently using one of the traditional libraries like RPi.GPIO.

Why should you migrate?

The sysfs interface has been deprecated for a long time and will disappear at some point.
Fighting the kernel for control of the GPIO registers is not guaranteed to work.
There are a few more changes coming to the gpiochip interface, and we can’t guarantee that libraries that entirely bypass the kernel will keep working in the face of those changes.

Migration is always a pain, but lg has good documentation (considerably more than libgpiod does as far as I can tell), and plenty of examples. For some more lg examples, you should also head over to my colleague William’s site where he’s put together a nice lgpio tutorial post using lgpio on Hirsute (and say “hi” — he’s new to the team, but he’s been a huge help during the Hirsute cycle!).

However, if you’re using gpiozero …

You shouldn’t need to migrate at all. The gpiozero library itself has traditionally defaulted to using RPi.GPIO as its underlying “pin driver”. In Ubuntu Hirsute (and probably RaspiOS Bullseye when it releases) that underlying implementation will default to lg instead (falling back to RPi.GPIO if lg isn’t found), so you shouldn’t need to do anything (but obviously if you do notice any issues, please open a ticket).

Finally, for those users of pigpio’s remote socket support, it’s worth noting that the “lg” project also includes:

rgpiod — a daemon using the lgpio library to provide a (potentially remote) socket based interface to the GPIO header
librgpio1 — the C client library for rgpiod (largely the same interface as lgpio but talks to an rgpiod instance instead of the local gpiochip device)
python3-rgpio — the Python 3 bindings for librgpio
rgpio-tools — includes “rgs”, a shell utility for talking to rgpiod

6 Months with the Pi Desktop

2021-04-17T00:00:00+01:00

See Also: A look at the Jammy desktop image.

In a deep movie trailer voice: “Previously on Dave’s blog“… “Oh no! My PC is no more! Whatever shall I do?!” “Oooh, a Pi 400 …”

Well, it’s been about 6 months since I started dog-fooding the Ubuntu desktop on a Raspberry Pi (and since I last managed to write a blog post … ahem). I’d originally intended to go back to using a big, fat, Watt-guzzling PC as my main development machine at some point in this cycle. However, a combination of global silicon shortages (if the damned crypto-miners could let me buy a single decent graphics card that’d be … nice?), and some major improvements in the Pi desktop meant I wound up spending the entire hirsute cycle using the little Pi 400 as my primary desktop.

The Hippopotamus in the Room

The biggest issue with this, as discussed in the previous installment, was video playback. We didn’t have video acceleration on the Ubuntu Desktop for Pi and this meant that, at the start of the hirsute development cycle, I still had to resort to my little laptop (a tiny Celeron powered thing) for video conferencing. However, at some point (a couple of months ago? I forget the precise timing as all sorts of work-arounds have been required at one stage or another) things came together, the planets aligned, the Gods of the Desktop smiled upon their lowly worshippers and Wayland+KMS started working.

Suddenly … things got much better.

For the past several weeks I’ve been using my little Pi 400 as my sole desktop. The laptop has been ditched entirely. HD YouTube playback works happily. Video conferencing over Google Meet works perfectly (no Zoom, but that’s because there’s no ARM builds of the client, ho hum), and the desktop generally feels responsive and slick.

There’s definitely a few kinks still to work out, but nothing show-stopping that’s made me dig the laptop out of storage or prevented me from Getting Stuff Done.

Making Hippos Fly

Above is a shot of my little desktop setup. Some optimizations to note:

I don’t have a fancy monitor. It’s bog-standard old 1080p HDMI thing. Personally, I can’t tell the difference between 1080p and 4K without squinting through my glasses anyway, but from a performance perspective having a 1080p monitor means no scaling is required to make the desktop readable. 200% scaling is fast under Gnome, but the result is frankly too big for my liking, and fractional scaling causes a noticeable performance hit. Choose your screen wisely!
Use the HDMI0 port. On the Pi 4, it’s next to the USB-C power port. On the Pi 400, it’s the one next to the SD card slot. In certain configurations, audio playback through HDMI only works from HDMI0.
It’s booting off an SSD connected to one of the USB3 ports (that little black box sat on the base of the monitor). We introduced support for booting Ubuntu over USB in groovy, and hirsute should work just as well. Simply flash the image to your USB drive and you’re good to go. But do note that you may need to enable USB boot in your Pi’s boot order first (this is one-time setup so if your Pi can already boot off USB, you’re good to go).
You can’t attach a Pi Camera Module to the Pi 400, so I’ve got an old Logitech C920 attached to one of the USB ports.
The Pi 400 only has 4GB of RAM. For lightweight desktops like LXDE this is okay, but Gnome shell is quite a bit heavier. The Ubuntu Desktop image for Pi does come with a 1GB swap file but honestly you probably want to extend this and I highly recommend using zswap (more on that below).
One advantage the Pi 400 does have over the Pi 4 is it’s much better at running cool. In turn, this means overclocking it is entirely viable even without any active cooling. At present I have the following lines added to my config.txt file on the boot partition, and the Pi 400 has been running with no heat / throttling issues for months:
```
arm_freq=2000
over_voltage=6
```
You might get away with these settings on a Pi 4, but you’d likely want some active cooling for that setup.

Swap, swap, glorious zswap!

As a rough rule of thumb, all systems should have swap. It doesn’t matter if your machine is a beast with 128GB of RAM, having some swap (not much, just a gig or two, say) is still useful as it permits the kernel to ditch pages that are never (or extremely rarely) used and use the RAM saved for disk cache (and there’s no such thing as too much of that!).

However, on a Pi 400 running a hefty desktop environment under 4GB of RAM it’s downright vital. As mentioned above, the images ship with a 1GB swap file (some may argue it would be preferable to create it on first boot, but it compresses trivially in the image, and this ensures that when people flash it they effectively “reserve” room for the necessary swap space). Ideally though (unless you’re on a beast like the Pi 4 8GB model), you probably want to expand this to something a bit bigger.

This can be easily done from the command line but I’d caution to do this when you’ve first booted the system and before you’ve launched anything else as the first step is to (temporarily) disable the swap. The following commands will add 1GB of size to the default swap-file on the desktop image:

$ sudo swapoff /swapfile
$ sudo fallocate -l 2g /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile

Note

You could trivially add a second swap-file and this wouldn’t necessitate disabling the existing one, but that then means getting into editing the /etc/fstab file to add it and this is a bit easier to document. Use whichever means you are comfortable with, but the important thing here is that 1GB of swap probably isn’t quite enough on a system with 4GB of RAM.

Next, I would highly recommend enabling “zswap“. This is one of several forms of virtual memory compression provided by the Linux kernel. The other commonly used one is “zram“. I’m not enough of an expert to opine on the fine detail of these systems, but my rough rule of thumb is as follows:

If you have no swap-file at all, zram is preferable. It provides a pseudo swap-file which is actually a compressed area of RAM and hence requires no swap-file backing it.
However, if you have a swap-file, zswap is preferable. It acts as a write-back cache for the swap-file and can in certain circumstances avoid pages ever hitting the disk at all.

Don’t worry about the computational overhead of these systems. CPU speed and memory bandwidth have grown considerably faster than bulk storage IO bandwidth has. As a result, memory compression systems that (to some degree) prevent or reduce disk IO are almost always a win.

Anyway, because we definitely have a swap-file, we’re going to use zswap here. It’s trivial to enable, but we’re going to customize it a bit to get some more out of it. To enable zswap you simply need to add the parameter zswap.enabled=1 to your kernel command line:

$ sudo sed -i -e 's/$/ zswap.enabled=1/' /boot/firmware/cmdline.txt

This is sufficient to enable zswap; if you were to reboot at this point it would already be working. If you don’t want to get into things that are a bit more complex, then feel free to stop at this point — you’ll already have plenty of benefit.

Still here? Okay, zswap’s defaults are, if anything, a bit conservative. The default allocator (zbud) is limited to 2 compressed objects per page. A superior allocator (z3fold) permits 3 objects per page boosting the typical compression ratio. Furthermore, the default compression algorithm is “lzo” which is nice and fast, but doesn’t provide great compression. Other options include “lz4”, “lz4hc” (LZ4 with “high compression”), and “zstd”. Personally, I’ve gone for “lz4”; “zstd” is also tempting but remember we’re balancing compression with speed here, and despite zstd being better at compressing things, lz4 is very fast.

Unfortunately, we can’t “just” enable these as we enabled zswap above. We need to make sure that the modules for these features are included in the initramfs at boot time:

$ sudo -i
# echo lz4 >> /etc/initramfs-tools/modules
# echo z3fold >> /etc/initramfs-tools/modules
# update-initramfs -u

Then simply add some more parameters to your kernel command line:

# sed -i -e 's/$/ zswap.compressor=lz4/' /boot/firmware/cmdline.txt
# sed -i -e 's/$/ zswap.zpool=z3fold/' /boot/firmware/cmdline.txt

Now reboot and you should have your super-whizzy compressed RAM setup complete! To check everything is set correctly (after rebooting) you can run the following:

$ grep -R . /sys/module/zswap/parameters
/sys/module/zswap/parameters/same_filled_pages_enabled:Y
/sys/module/zswap/parameters/enabled:Y
/sys/module/zswap/parameters/max_pool_percent:20
/sys/module/zswap/parameters/compressor:lz4
/sys/module/zswap/parameters/zpool:z3fold
/sys/module/zswap/parameters/accept_threshold_percent:90

If things aren’t right, have a look at /boot/firmware/cmdline.txt in a text editor.

To see how zswap is performing:

$ sudo grep -R . /sys/kernel/debug/zswap
/sys/kernel/debug/zswap/same_filled_pages:3418
/sys/kernel/debug/zswap/stored_pages:242039
/sys/kernel/debug/zswap/pool_total_size:353652736
/sys/kernel/debug/zswap/duplicate_entry:0
/sys/kernel/debug/zswap/written_back_pages:0
/sys/kernel/debug/zswap/reject_compress_poor:368
/sys/kernel/debug/zswap/reject_kmemcache_fail:0
/sys/kernel/debug/zswap/reject_alloc_fail:0
/sys/kernel/debug/zswap/reject_reclaim_fail:0
/sys/kernel/debug/zswap/pool_limit_hit:0

I’ve even tried this on a Pi 4 2GB model and, while it’s not as nice as a Pi 400 (in particular due to the SoC running hotter), it makes it a pretty viable desktop at least for light usage. It was perfectly capable of firing up Firefox and playing an HD YouTube video, while also running LibreOffice Writer, and a terminal with vim, all on a $35 2GB Pi!

Note

I should note we don’t officially support using the desktop image on a 2GB Pi, but personally I think this little optimization makes it a reasonable platform for the Ubuntu desktop.

Hirsute Hippo’s due for release on April 22nd, and the release notes are definitely worth a quick peek.

A Week with the Pi Desktop

2020-10-22T00:00:00+01:00

In a previous post I mentioned I’d be using a Pi as my exclusive desktop for a week, mostly to validate that it actually worked (and that our guesstimate that 4Gb RAM models were a reasonable “base”), but partly just for the experience!

Here’s a blow-by-blow account of how it went (there’s a TL;DR at end if you just want to scroll there :)…

Monday

I’d spent a few hours of the prior week setting up the system as I needed for work so, come this morning, everything was pretty much as my desktop normally is. Just on a Pi.

The Pi in question was a model 4B with 4Gb of RAM. It was booting off an ancient 120Gb SSD I had lying around from an old PC (in that small black case on the floor) and the eagle-eyed will note that there’s absolutely no active cooling or so much as a heat-sink installed on it in this picture. This is deliberate: I was reasonably sure it was going to thermally throttle at some point in the day, but I wanted to see how often that happened and how bad it was in practice. In other words, trying to answer the question: is this usable on an otherwise “naked” Pi?

Other than that, it was plugged into all the stuff I usually have plumbed into my beefy (but aging, and now somewhat dead) Core i7.

First tweaks:

Audio from rhythmbox was choppy. I’d encountered this before and the hints from the “Audio” section of my earlier notes on the Pi desktop soon sorted this out. I’ll see what we can do about fixing this in the final image but given where it is I don’t think we can layer another file on top of it to implement this just for the Pi. This will probably be something I just have to stick in the release notes.
The wrong audio output device was always selected on boot (should be “Analog Output - Built-in Audio”, but it was always “Multichannel Output - Built-in Audio”). On a hunch, I tried disabling the “module-switch-on-connect” lines in that same file (/etc/pulse/default.pa) and the problem was solved, although later I discovered this did mean that audio devices plugged into USB weren’t automatically selected. However, in my case, that’s ideal as I didn’t want them selected. Again, this may wind up being a release note.
No swap. Not an issue initially, but I figured I’d be needing it sooner or later so I set up a 1Gb swap file and some monitoring scripts to regularly report things like system load, swap used, SoC temperature, etc. then added them to my steadily growing tmux configuration. Will see what can be done about fixing this for the release.

The first real test of the day: a meeting in Google Meet! I didn’t have any camera plumbed in at this point as I hadn’t succeeded in getting the camera module working properly with it yet, but I plugged in my nice USB microphone (that’s the USB audio device I mentioned above, which I don’t want to use for output, just input), fired up Chromium (installed from the snap-store) and somewhat nervously connected…

My group were evidently feeling somewhat shy (!) and everyone had their camera feed off, so it was just audio for this first challenge. Still, there were 5 people in the call so this was a reasonable first test. My audio apparently worked perfectly, and I had no issues hearing anyone at my end. All pretty smooth so far!

The Pi had been idling at ~60°C but that temperature slowly began to climb during the call. Within about 10 minutes it was at the 78°C mark (the Pi 4 throttles at 80°C) and I was starting to get a bit nervous. It took quite a while to get actually hit 80°C (maybe another 5 minutes) and at that point there was an occasional drop-out from the audio when others were speaking but otherwise the thermal throttling worked remarkably smoothly and everything stayed connected and working! So far, so good.

After the call was over, I dove into some other work and wound up compiling a couple of packages, writing up some notes and answering a few e-mails. The compiling all went fine, no swapping, no throttling, but the e-mail… This proved to be the next big challenge. Within minutes of firing up a GMail tab, swap space was being eaten into. It really is quite the memory hog!

Next tweaks:

I activated zswap (appended zswap.enabled=1 to /boot/firmware/cmdline.txt) to see if that alleviated the 50% of my 1Gb swap space that’d been eaten. It seemed to (hovered around 20-30% after loading everything back up after rebooting to check my config), but these things are difficult to measure. YMMV.

Now, the next big challenge of the day. Another Google Meet call, but this time one that usually had 10+ participants and was very likely to involve video streams. I opted to try this one in Firefox instead (just to see if it would work). Within a minute of connecting it was obvious the Pi was struggling; the SoC temperature hit 80°C (the throttle point) within the first minute, and the video streams were mostly frozen. That said, at all points audio from the call was playing smoothly. With a fair bit of patience (very slow UI!), I disabled the incoming video feeds so the Pi just had audio to deal with.

Then the next challenge: using a shared Google Doc in another tab with the call still going. I was highly dubious the Pi would manage this given it was already into thermal throttling and swapping but I opened the tab anyway. It wasn’t quick to load but to my amazement it worked! I could see others editing away and could make a few edits myself. It wasn’t smooth, it did have a fair bit of cursor lag, but at all points the audio of the call kept playing mostly smoothly (a few crackles here and there, but nothing to affect comprehension), and I could actually work semi-productively. Remember, this was on the smallest of the Pis supported, with no thermal management at all and no shutting down anything I was already running.

Impressions at the end of the day: it’s slightly uncanny. Several times during the day I had to remind myself that I wasn’t on Ammy (my old Core i7). To be clear: I’m not claiming they’re in any way comparable on performance because they’re not. It’s trivial to find something that shows a massive performance difference (like Google Meet calls, or using GMail), but for most of the time, playing some music in the background while I hacked away at something in vim under a terminal window there was effectively no difference.

Then I’d need to flash a fresh image onto an SD card to test something, and suddenly I’d find myself reaching down to stick the card in the flash-reader that used to be in one of Ammy’s 5.25” bays and realize: nope, I’m on the Pi, and I need to use that nice little card-reader-slash-usb-hub I’d stuck on the desk instead.

To my genuine surprise, there wasn’t one thing the Pi failed at all day. Bear in mind this was an uncooled, more or less “stock” 4Gb Pi. Sure, the Google Meet calls weren’t great, and the GMail UI was hilariously laggy in some places … but they worked! Nothing crashed, nothing hung, and I had a day about as productive as any normal day.

If there’s one major difference that I’m just noticing it’s that, as I write this at 8 o’clock in the evening, my fingers are cold.

This may not sound too surprising: my office is in an unheated loft extension, it’s mid-October, and I’m in the North of the UK, so it’s pretty chilly outside. But normally Ammy, with her mammoth heat-sink, myriad fans, and frankly silly graphics card, has done a pretty good job of keeping things toasty and warm up here. She’s usually more than capable of raising the temperature in this room up to 30°C (in summer, things get ridiculously hot up here!). But the Pi has nothing like that thermal output. I might have to get some gloves…

Tuesday

Played around a bit with the configuration today to test some extra interfaces:

Unplugged my usual Microsoft Intellimouse and tried out a bluetooth-based mouse instead. After some considerable pain getting it paired (which may be because it was previously paired to another machine), it’s working happily.
Paired the Pi with my mobile phone and tried out bluetooth-based tethering. Totally uneventful: just worked.
Also fitted a Pimoroni fan-shim to see what difference that made. Thankfully since this image doesn’t have U-Boot there were no problems with the fan-shim interrupting it.

To configure the temperature limit on the fan-shim, I simply added the following line to /boot/firmware/config.txt:

dtoverlay=gpio-fan,gpiopin=18,temp=65000

The only thing that brought the Pi into throttling territory yesterday were the calls in Google Meet, so I was interested to see what difference that made today. For the first call of the day there were only a few people with cameras on, but that was enough to quickly push the Pi’s temperature up.

Sure enough, the fan-shim kept things nicely below 65°C throughout the call. However, while this meant the audio was perfect throughout, I still wound up having to switch off most of the incoming video streams to keep things operating smoothly (Google Meet has a nice option to show a single standard-def stream at a time which seemed to work well). On this call, I was still audio only.

Then it was back to some compiling and documentation work, no problems here and everything went smoothly.

Next call: at this point while I’d managed to get a camera module installed, there was a problem in that the cable isn’t that long and I didn’t think people would particularly enjoy a view from under my desk (ahem). So, instead I plugged in the USB web-cam that normally sits on my monitor. I knew this was likely to give the Pi a hard time as USB web-cams typically chuck out an MJPEG stream and it would be up to Chromium to convert that to VP8 (or VP9?) which it wouldn’t use the GPU for. Sure enough, within a few seconds of connecting it was obvious the Pi was struggling (apparently my audio was choppy and dropping out), so I disabled my video feed and carried on as normal.

Finally: needed to modify some bits in JIRA (which we use for organizing the team’s work). JIRA was … hilariously slow. Nearly a full minute to just load the web-page. That said, JIRA is hilariously, painfully, slow on just about anything I’ve used (including Ammy) so I shouldn’t be surprised there. It was useable with a good dose of patience, but I’m thankful it’s not something I spend all my time in (can’t see my boss using it with a Pi :).

Verdict for the day: bluetooth was fine; the fan-shim is probably worthwhile for heavy workloads; USB web-cams are not going to be happy on video calls - use a camera module if at all possible (though cable length may be an issue).

Wednesday

Switched up to a Pi4 8Gb today. I was satisfied with the 4Gb’s performance with a decent dose of swap-file but was now interested in whether the 8Gb would swap at all and whether the (hypothetical) lack of swap would lead to any serious performance difference.

I transferred the fan-shim to this Pi so I could compare just the difference in RAM rather than anything else. Everything else was just unplugged from the old Pi and plugged into the new Pi. This is one of the things I love about the Pi: transferring all your storage between them is trivial (whether SD card or USB hard-drive based, it’s a lot easier than messing around inside a PC tower!).

There was some small faff getting the EEPROM configured to handle USB boot, but I had an SD card already setup to deal with that so it didn’t take more than a minute or two, then we were off.

The day was frankly uneventful. The usual mix of meets, compiling, emailing, etc. Only one thing of note really: no swapping. With a judicious set of JIRA, Gmail, and other hogs open I did manage to push the Pi to ~70% RAM usage but never managed to get it to actually swap (beyond trivial levels).

However, I can’t claim I actually noticed any performance difference. It felt pretty much exactly the same as the 4Gb model - the only difference being that I could see the swap monitor sat resolutely at zero all day.

Verdict for the day: the 8Gb is certainly a technical improvement, and for some that extra memory may be crucial (especially if you’re heavily into containers). Personally, I could live happily with a 4Gb plus some swap. Incidentally, even on an 8Gb I’d still keep some swap (say, 1Gb) around. The myths surrounding things being faster without swap are generally just that: myths.

Thursday

Still on the Pi4 8Gb today. I did cheat on my Google Meet calls today, and used my little laptop instead, largely because I knew one of the calls was going to involve screen-sharing and at this point I was pretty confident I knew the Pi’s limits and that I couldn’t smoothly deal with that, and ~10 people on the call too (on Google Meet, if you opt for “Audio only” it really means that and doesn’t permit screen-sharing; unsurprising but slightly annoying today).

Come the evening I had to “cheat” on another meeting call but for different reasons. I was due to give a quick talk for the Python North West group, which was using Zoom and, about 30 minutes before I was due to connect, discovered there’s no Zoom client for ARM. I wouldn’t have minded going audio-only for this one - I didn’t have any slides, just a few notes for a quick 10 minute segment. Still, it was back to the laptop again.

Everything else was basically uneventful:

Needed to do some printing for my kid (colour laser printer on the network); worked without a hitch as it usually did on Ammy
Needed to scan some pages for my partner (USB connected scanner); also worked without a hitch
Did some quick image tweaks in Gimp (for this post!) with my partner’s Wacom tablet. No problems there either (just plug’n’play), though I’m pretty sure the extra RAM of the 8Gb model helped

Slight hassle plugging and unplugging all these USB things, but necessary. The Pi’s power supply isn’t a 750W beast like Ammy’s and doesn’t like having all my USB gadgets plugged in at once. Further, given I’m also booting off a USB attached device it’d be ill-advised to try and starve that of power! Might be time to invest in a powered USB hub…

Verdict for the day: if only Chromium could use the GPU for video encode and decode here. We’ll have to see what can be done about that (this is easier said than done; I have some inkling of what the Pi Foundation have gone through to get GPU decode into their version of Chromium, and it’s non-trivial).

Also, more stuff needs compiling for ARM. After the lack-of-Zoom-client discovery I went checking around for a few things. There’s no Chrome for ARM (well, outside of Chromebooks, so it’s obviously possible), no Zoom (as noted already), and no Skype. Signal and Telegram are available, but there’s definitely some prodding to be done here…

Side note: it was amusing to note the number of sites that happily offered me an amd64 deb!

Friday

Still on the Pi4 8Gb. Cheated on the Google Meet calls again with the laptop because, well, I might as well at this point. Everything else still on the Pi4, and uneventful. Watched a few YouTube videos; slightly jerky startup for the first second, and then played entirely smoothly while I was still working in another window.

Looks like it’s happy dealing with 1 video stream (even with it just doing CPU decoding), but pushing it to do more than that (e.g. during conferencing) isn’t viable without hardware acceleration.

Wound up doing some work in the evening on the piwheels website. The Firefox and Chromium developer tools are certainly a bit sluggish, but still useable. Once used to the slower refresh rate, it didn’t slow me down.

TL;DR / Conclusion

It’s so close to being a fully viable desktop replacement. The thing holding it back for me is video acceleration on Google Meet calls. If we had that, I could effectively switch to using this for the vast majority of my work, firing up the big PC (when it’s fixed :) only for seriously heavy duty things like video editing, or PC gaming, which are pretty rare activities for me, and which it really wouldn’t be realistic to expect a Pi to accomplish.

For others? Very much depends on typical uses and software. If you need Skype or Zoom, for instance, you’ll be stuck using the web-based versions (which’ll have the same issues as I did with Google Meet) because there’s no ARM compiled native clients (yet). But if you don’t need video conferencing (okay, that’s probably a fairly major ask in the midst of a pandemic where everyone in front of a computer is probably home-working), it’s a capable replacement.

Don’t go expecting Ryzen-like performance because you’re not going to get it … obviously. But I’d hardly call it sluggish either. Running off a USB-attached SSD everything starts up quickly and runs pretty smoothly. I was also impressed that, even on the 4Gb model, despite it eating into swap a bit, nothing ever felt like it was overwhelming the system. Though you might want to invest in a powered USB-hub!

Finally, for all the talk of the Pi 4 “running hot”, I only ever needed the fan-shim for video conferencing. My other workloads failed to push it into thermal throttling (though a few got close). The board certainly gets toasty, but it’s nothing like the small fan heater that Ammy was. If only it could heat my fingers up a bit!

Final note: any issues we haven’t sorted out by release, with any known workarounds, should be mentioned in the Groovy Release Notes.

Groovy Boot Modes

2020-10-21T00:00:00+01:00

By default, Ubuntu Groovy Pi images (both Server and Desktop) will ship with a boot sequence that involves the Pi’s native bootloader only. In other words, the new boot sequence is simply:

Pi Bootloader --> Linux Kernel --> Initrd

Previous versions of Ubuntu on the Pi have included U-Boot in the sequence. In other words, in Focal (and prior) versions, the boot sequence was:

Pi Bootloader --> U-Boot --> Linux Kernel --> Initrd

Users upgrading to Groovy will continue using the legacy U-Boot sequence but are free to switch to the new “native only” boot sequence by following the instructions below. Conversely, new users of Groovy may also switch to a U-Boot based sequence if they so wish (the U-Boot binaries will still be installed on the boot partition even when they’re not used to enable switching between the two choices).

Note

Be aware that the U-Boot based sequence does not support native USB boot, or the Pi’s native netboot (U-Boot on the Pi 3 and earlier does have its own PXE-like netboot system, however our build of U-Boot currently lacks ethernet drivers for the Pi 4).

Also note that the U-Boot based sequence is now deprecated. If you are still relying on U-Boot for some reason, please let me know what it is (open a ticket against u-boot on Launchpad, leave a comment below, or pester me on Twitter)!

Switching from U-Boot to Native

The only file that needs editing is /boot/firmware/config.txt. There may be warnings at the top of the warning you not to edit it. You can ignore these. I know this, because I wrote them (you can read the sordid tale of those warnings and why they’re now redundant in this post).

Remove any existing lines that start with kernel=, initramfs, or devicetree_address= from this file. The add the following lines under the [all] section:

kernel=vmlinuz
initramfs initrd.img followkernel

That’s all there is to it! If you want to try this from pibootctl (with some extra lines to save your current boot configuration so you can switch back to it if you wish), the following commands should do the trick:

$ sudo pibootctl save uboot
$ sudo pibootctl set boot.kernel.filename=vmlinuz \
    boot.initramfs.filename=initrd.img boot.devicetree.address=
$ sudo pibootctl save native

Now you can simply do the following to switch back to uboot if you wish:

$ sudo pibootctl load uboot

Switching from Native to U-Boot

Switching back to U-Boot from the native boot sequence is a little more complex but not much so. Again, in /boot/firmware/config.txt remove all lines that start with kernel= or initramfs. Next, determine whether you’re using a 32-bit or 64-bit:

$ dpkg --print-architecture
arm64

If you’re on arm64 (64-bit), you want a config.txt that includes the following (by “includes” I mean don’t replace your whole configuration with this; either append this to your current configuration or merge it into the existing sections - either should be fine):

[pi4]
kernel=uboot_rpi_4.bin
max_framebuffers=2

[pi2]
kernel=uboot_rpi_2.bin

[pi3]
kernel=uboot_rpi_3.bin

[all]
arm_64bit=1
device_tree_address=0x03000000

If you’re on armhf (32-bit), you want a config.txt that includes the following:

[pi4]
kernel=uboot_rpi_4_32b.bin
max_framebuffers=2

[pi2]
kernel=uboot_rpi_2.bin

[pi3]
kernel=uboot_rpi_3_32b.bin

[all]
device_tree_address=0x03000000

(if you’re curious about the strange ordering of the sections, this isn’t strictly required on Groovy but is just there for historical reasons)

You can try this doing this via pibootctl, but it currently only has options for modifying boot configuration sections relevant to the machine you’re running on (e.g. if you’re on a Pi 4, it can modify the [pi4] and [all] sections but won’t modify [pi2] or [pi3] sections). So, if you’re on a Pi 4 running an arm64 kernel, you could do the following (with some extra lines to save your current boot configuration so you can switch back to it if you wish):

$ sudo pibootctl save native
$ sudo pibootctl set --this-model boot.kernel.filename=uboot_rpi_4.bin
$ sudo pibootctl set --no-backup boot.initramfs.filename= \
    boot.devicetree.address=0x3000000
$ sudo pibootctl save uboot

Now you can simply do the following to switch back to uboot if you wish:

$ sudo pibootctl load native

Installing the new Pi Ubuntu Desktop

2020-10-12T00:00:00+01:00

Now that Martin’s given the game away on Twitter, it’s probably a good idea for me to post a quick guide on installing the new Ubuntu Desktop for the Pi on various media. So, here goes!

Stuff you’ll need

A Raspberry Pi Model 4B with 4Gb of RAM or higher. Theoretically, the desktop can run on less but we’re only officially supporting it on these models because with less than 4Gb of RAM it’s not a particularly “useful” experience!
The Raspberry Pi Imager tool for whatever platform you’re on.
The Ubuntu Desktop for Raspberry Pi image, for now anyway. Once Groovy’s released it’ll just appear in the menu in the imager tool, but for now you can grab the pre-release beta from:

http://cdimage.ubuntu.com/ubuntu/daily-preinstalled/current/

You need the groovy-preinstalled-desktop-arm64+raspi.img.xz image from there. You may note there’s no armhf image and the simple reason for that is we’re not providing one at this time; it’s arm64 only.

Now, continue with one of the sections below…

SD Card

To install on an SD card is pretty much simplicity itself:

Start the Imager Tool
Click “Choose OS”, select “Use custom”, and select the file you downloaded earlier; you don’t need to decompress it (after release you’ll be able to pick an entry under “Ubuntu”)
Click “Choose SD Card”, select the card you want to flash
Click “Write”

Then you can boot the card on your Pi 4.

USB HDD / SSD

In Groovy we’re (finally!) adding support for booting Ubuntu directly from a USB attached hard-drive or SSD with no SSD card involved. The first thing to ensure is that you’ve got an up to date EEPROM version on your Pi 4 (as USB booting was added after the Pi 4’s release). This is easy enough to accomplish under either Raspbian or Ubuntu:

sudo apt install rpi-eeprom - only strictly necessary on Ubuntu as it’s installed by default on Raspbian, but it won’t do any harm there
sudo vcgencmd bootloader_config > bootconf.txt - extract the current bootloader configuration to a text file
sed -i -e '/^BOOT_ORDER=/ s/=.*$/=0xf41/' bootconf.txt - set the BOOT_ORDER option to 0xf41 (meaning attempt SD card, then USB mass-storage device, then repeat; see pi4 bootloader configuration for more information). Alternatively vim bootconf.txt and make the edits yourself if you don’t trust my sed-hacking :)
rpi-eeprom-config --out pieeprom-new.bin --config bootconf.txt /lib/firmware/raspberrypi/bootloader/critical/pieeprom-2020-09-03.bin - generate a copy of the EEPROM with the updated configuration
sudo rpi-eeprom-update -d -f ./pieeprom-new.bin - set the system to flash the new EEPROM firmware on the next boot
sudo reboot - to apply any changes (the EEPROM is only updated during the early stages of boot)

Note

Don’t worry about having to do this on every EEPROM upgrade because you don’t have to. Each EEPROM upgrade extracts the current EEPROM configuration and merges it with the updated EEPROM binary before flashing it.

Now that the EEPROM’s dealt with, how do we get the image onto a hard drive?

Read the SD Card instructions above
Pretend your USB-attached hard-drive is a very fat SD card and just select it instead when you click on “Choose SD Card”
erm … that’s it

You should now be able to boot your flashed drive on your Pi 4 off your USB-attached hard-drive without any SD card inserted. Be aware that some drives have issues being used to boot the Pi. In particular:

Spinning hard-disks required a lot more power than SSDs and will very likely required a powered USB hub. If you have a stonkingly huge hard-drive, see the next section too…
Hubs themselves can cause compatibility issues, so frankly you’re much better off with an SSD to boot off (typically no need for a hub and no spin-up time issues).

There’s lots of good information on both the Pi forums and various GitHub issues for debugging boot issues; here’s a selection of links in a rough “look at this first” order:

Stonkingly huge HDD / SSD

What if you have a hard-drive larger than 2Tb (or SSD if you’re richer than Croesus) and want to actually use all that space? The default MBR partitioning on the image won’t be much use as it can’t address anything beyond 2Tb. Thankfully, the Pi’s bootloader now supports GPT partitioning too so this isn’t a big deal and just requires a minor tweak to the partition table.

The following instructions are Linux-specific. I’m sure it’s possible to do this on Windows and Mac OS too, but you’ll have to Google that if you need it. I would advise doing this with the drive attached to another machine (i.e. when you’re not booted from it). It might be possible to do this on a “live” system, but I haven’t tried that so YMMV!

sudo apt install gdisk - just to make sure you’ve got gdisk (the GPT partitioning tool) installed
sudo gdisk /dev/sdX - where sdX is whatever your hard-drive is called (consult the end of dmesg for a hint)
If gdisk complains about a valid MBR and a corrupt GPT, don’t worry (this can happen when re-imaging drives that’ve had a GPT partition table before). Just select 1 to use the valid MBR and continue
gdisk should’ve converted your MBR partition table to GPT in RAM (not written to the disk yet). Enter p to print the table and make sure it looks relatively sane:
- two partitions,
- a small one with type 0700 (“Microsoft basic data”) which is the boot partition, and
- a larger one with type 8300 (“Linux filesystem”) which is the root partition
Enter w to write the new partition table to the disk and exit.
Safely eject the drive and boot from it!

Note that you can do this before the first boot, or you can do it later on. However, if you do this after the first boot you may need to manually resize the root partition to take up the full disk size (if that’s what you want). If you do this before the first boot, the image should expand the root partition to fill the disk (I’ve checked resizing works under GPT, but I haven’t got a spare HD >2Tb to actually try that with! :).

Have you actually tested this?

For the last week I’ve been using the desktop image on and off on a Pi 4 with 4Gb of RAM (I’ve got an 8Gb model here, but obviously the point is to try and validate that 4Gb is “sufficient”), with my little laptop as a backup. However, this week I’m embarking on something a little more adventurous: using it full time as my only desktop (laptop strictly for emergency backup only, and no desktop either because … well, it sort of blew up).

The specific configuration I’m using is: booting off a small 120Gb SSD I had spare. Anyway, if I come up with any tweaks along the way I’ll post a write-up later this week. Wish me luck!

Boot configuration with pibootctl

2020-09-16T00:00:00+01:00

In the previous post, we looked at reading the boot status with pibootctl (in several formats). This time we’ll look at how we can use pibootctl to change the boot configuration, and the side-effects of doing so.

First though, a brief interlude on why this post took a lot longer than intended (feel free to skip this section if you’re only interested in how to set stuff with pibootctl; it’s mostly self-flagellation anyway! :).

On What Dave Should’ve Known Sooner

When it started out, I had a notion of pibootctl’s manipulation logic that was based on certain assumptions, which in turn were based on how I’d structured the boot configuration for Ubuntu.

On Raspbian (oops, that’s now Raspberry Pi OS - funny how so much can change in a few months), the configuration structure is simple: everything goes in “config.txt”.

On Ubuntu, I’d altered the configuration structure to be rather more complex: everything starts in “config.txt” (as usual), then continues in the included “syscfg.txt”, and finishes in the included “usercfg.txt”. The intention was as follows:

“config.txt” was to be under the control of the package manager. Its one and only job would be to get the Linux kernel going and then include the other files. The idea was we could clobber that from the package manager with impunity, not worrying about arbitrary modifications to it.
“syscfg.txt” was to be under the control of pibootctl (and thus anything built on top of it). The pibootctl utility would be free to rewrite the whole file according to whatever the user requested.
Finally “usercfg.txt” was the “free for all” file in which we migrated existing settings from config.txt in the transition to the split. Users were free to do whatever they wanted in this file, with the understanding “you break it, you get to keep the pieces”.

Well, that’s all fine and dandy but for one little problem: some boot configuration settings only work in “config.txt” and get ignored if they’re in included files. In my defense, while this is now documented in the excellent configuration documentation it wasn’t at the time I started writing it (yes, I tracked down that commit after wondering “how the hell did I miss this?!”).

However, the case for the prosecution is: most of the directives affected (e.g. start_x, start_debug, gpu_mem) are pretty obvious if you understand the separation between “bootcode.bin” (or the EEPROM on the Pi 4), and “start*.elf” … which I absolutely did. What I’d missed was that, while many of the conditional sections are supported by both “bootcode.bin” and “start*.elf”, only the latter implements the “include” command.

Anyway, this meant that my oh-so-carefully separated boot config was a bust. The “config.txt” file must be modifiable by pibootctl (or anything else), otherwise certain rather important settings (e.g. enabling the camera module) just aren’t possible. This in turn defied several of the assumptions baked into pibootctl’s manipulation of the boot configuration:

I had assumed that pibootctl could re-write a file (e.g. syscfg.txt) completely and thus shouldn’t need to concern itself with, for example, preserving commentary in that file. This is obviously not the case if it needs to manipulate “config.txt” too.
I had assumed there needn’t be any provision for commenting / uncommenting lines of configuration. We could just wipe that file and start fresh. However, in “config.txt” there are traditionally comments preceding various lines (esp. on Raspbian) and those comments should remain with their corresponding configuration lines (á la raspi-config).
I had assumed that the included file we’d be re-writing could reasonably be assumed to be in an “[all]” context so we wouldn’t need to worry about the context of removed / commented lines, or the context of added lines. But, if you’re messing with “config.txt” that also goes out of the window. Both Raspbian and Ubuntu use conditional sections in both of their default configurations.

All the assumptions above were now broken and needed to be worked around. The tool needed to understand the conditional context applicable to each line, the conditional context applicable to newly generated lines, and the set-based relationship between conditional contexts. For example, given the following configuration:

[pi2]
kernel=uboot_rpi_2.bin

[pi3]
kernel=uboot_rpi_3.bin

[pi4]
kernel=uboot_rpi_4.bin

If I want to set the kernel’s filename to “vmlinuz” (bypassing u-boot), on all models, the tool needs to know that the conditional context of each “kernel” line above is a subset of the requested conditional context (“[all]”), and thus that all the existing “kernel” lines should be removed / commented; that a new “[all]” section needs to be added at the end and the new line placed in that section.

Alternatively, if we want to make the same manipulation, but only for the current model (let’s assume that’s a Pi 3), the tool needs to know that the “[pi2]” section is a disjoint set with “[pi3]” so the first kernel line needs to be left alone; the second is equal so that kernel line can be removed / commented then replaced, and the final “[pi4]” section is also a disjoint set with the requested context.

Anyway to cut an already way too long interlude short … that’s what the tool now does. It’s fully aware of the conditional sections, how they relate, which commands are only recognized in “config.txt” (and therefore should be ignored for effect in other contexts), how to generate new conditional sections to transform one conditional context into another, how to comment out lines (optionally), and how to search for and uncomment lines in relevant contexts.

So, on with the show …

Stored Configurations

One major feature of pibootctl is that it’s capable of backing up and restoring boot configurations. It’s relatively important to understand how it does this given that, as a relatively low level tool designed to mess with your boot configuration, it’s entirely possible to use it to break your boot configuration!

The Pi’s boot partition is a simple FAT partition at the start of the image, and the boot configuration is (mostly) held in a series of straight-forward text files. The great benefit of this is universal support: FAT is the “universal file-system”; everything knows how to read and write it from Windows to Mac to Linux, and even numerous embedded systems. Likewise, text files are the “universal format”; every operating system has some facility for editing text files (which is the reason it was selected as the original format for the throughly worthy Project Gutenberg).

Ultimately this means that, should the worst happen, and you screw up your boot configuration, you can stick your SD card in another machine and edit it back to a workable state. However, it might be easier if you know what a workable state is!

pibootctl strives to be equally accessible in its backup strategy: boot configurations are stored as ZIP archives of the original (mostly) text files, on the FAT partition (in a sub-directory). ZIP is another pretty-much-universal format (Windows, Mac OS, and most Linux distros have built-in support for opening ZIP files), and by storing the backups on the FAT partition we ensure that they remain accessible on all platforms too.

Assuming you’ve installed pibootctl on Ubuntu from the PPA. you can create a backup of your current boot configuration using pibootctl like so:

$ sudo pibootctl save original

Why is sudo required? Simply because we’re writing that backup to the boot partition which only root has access to. After this you should be able to list the available stored configurations like so:

$ pibootctl ls
┌──────────┬────────┬─────────────────────┐
│ Name     │ Active │ Timestamp           │
├──────────┼────────┼─────────────────────┤
│ original │ ✓      │ 2020-09-15 12:27:02 │
└──────────┴────────┴─────────────────────┘

Note that sudo isn’t required here as we’re not writing anything to the boot partition, just reading things. Also note that pibootctl has compared the current boot configuration to the stored “original” configuration and noticed they’re exactly the same, hence the tick-mark in the “Active” column. Where exactly is this stored?

In the pibootctl package in my PPA, pibootctl is configured to create a “store” directory on the boot partition which will contain the stored configurations:

$ ls /boot/firmware/store
original.zip

You can install an appropriate archiving tool to take a look in this archive if you want. Personally, I favour “atool” which provides generic commands for working with a variety of archives (tar.gz, tar.xz, zip, etc. etc.):

$ sudo apt install -y atool
$ als /boot/firmware/store/original.zip
Archive:  /boot/firmware/store/nouboot.zip
pibootctl:0:e5bf92a57556aa24d4dd8c94ba63a755a21fdcc3

Do not edit the content of this archive; the line above is a hash of the
content which will not match after manual editing. Please use the pibootctl
tool to manipulate stored boot configurations
  Length      Date    Time    Name
---------  ---------- -----   ----
     1116  2020-09-15 12:27   config.txt
      327  2020-09-01 07:02   syscfg.txt
      200  2020-09-01 07:02   usercfg.txt
      127  2020-09-01 12:27   cmdline.txt
---------                     -------
     1770                     4 files

Several points of interest here:

There’s a strange “pibootctl:0:xxxxxxx” line at the top of the ZIP archive’s comment. This is a hash of the boot configuration in parsed order, and pibootctl uses it to quickly determine which stored configurations match the current boot configuration.
For this reason, there’s a comment straight after the hash line warning users not to modify the content of the archive, or the hash will no longer match. At the moment, the tool doesn’t use the hash to validate the archive contents as I’m on the fence as to whether permitting users to create their own archives manually is a good or bad idea but for now the warning stands.
After this you can see the various files that make up the boot configuration: config.txt, syscfg.txt, usercfg.txt, and cmdline.txt (which is referenced from config.txt). Note that other files referenced by the boot configuration like the kernel (vmlinuz) and the initrd (initrd.img) are not included in the archive. The assumption is that these things are under the control of the package manager and shouldn’t be restored with the boot configuration itself.

Manipulation

Now let’s edit our boot configuration using pibootctl. We’ll enable the camera module and set GPU memory to 128Mb:

$ sudo pibootctl set camera.enabled=on gpu.mem=128

Note once again we use “sudo”. Because we’re changing files on the boot partition we need root privileges. We can check the “status” output to see these are definitely now set:

$ pibootctl status
┌─────────────────────────┬──────────────────────┐
│ Name                    │ Value                │
├─────────────────────────┼──────────────────────┤
│ audio.enabled           │ on                   │
│ boot.devicetree.address │ 50331648 (0x3000000) │
│ boot.initramfs.filename │ ['initrd.img']       │
│ boot.kernel.64bit       │ on                   │
│ boot.kernel.cmdline     │ 'cmdline.txt'        │
│ boot.kernel.filename    │ 'vmlinuz'            │
│ camera.enabled          │ on                   │
│ gpu.mem                 │ 128 (Mb)             │
│ i2c.enabled             │ on                   │
│ serial.enabled          │ on                   │
│ spi.enabled             │ on                   │
│ video.framebuffer.max   │ 2                    │
│ video.overscan.enabled  │ off                  │
└─────────────────────────┴──────────────────────┘

In fact, the tool ensures that the written configuration matches the desired configuration as part of its generation logic. Consider the following configuration:

config.txt

1 [pi4]
2 max_framebuffers=2
3 
4 [all]
5 arm_64bit=1
6 cmdline=cmdline.txt
7 
8 include usercfg.txt

usercfg.txt

1 dtoverlay=disable-bt

In this case “usercfg.txt”, a file that pibootctl is not permitted to edit, disables the bluetooth module. What happens if we explicitly try to enable it?

$ sudo pibootctl set --no-backup bluetooth.enabled=on
Failed to set 3 setting(s)
Expected bluetooth.enabled to be True, but was False after being overridden
by usercfg.txt line 1
Expected serial.uart to be 1, but was 0 with no valid lines; this usually
means a setting like start_x or gpu_mem is in a file other than config.txt
Expected serial.enabled to be False, but was True with no valid lines; this
usually means a setting like start_x or gpu_mem is in a file other than
config.txt

pibootctl has noticed that it failed to set bluetooth.enabled, and warns the user that the setting is “overridden by usercfg.txt line 1” (the other errors come from the fact that disabling the bluetooth module implicitly enables the serial port and places it on the PL011 UART; these also don’t match pibootctl’s expectations for the resulting configuration and so they get reported too).

Restoration

How do we restore a saved configuration using pibootctl? The opposite of the “save” command is “load”:

$ sudo pibootctl load nouboot
Backed up current configuration in backup-20200915-133659

Once again, we need “sudo” as we’re manipulating the content of the boot partition. One additional thing to note above is that pibootctl automatically saved a copy of the current boot configuration. It did this because, before modifying the current configuration, it checked a copy of it existed in the stored configurations. In this case no copy existed (since we changed it above), so it automatically saved one for us.

You can suppress this behaviour with the --no-backup switch, or permanently by modifying the tool’s configuration in “/etc/pibootctl.conf”.

Don’t Panic!

Let’s assume the worst has happened and you’ve broken your boot configuration. However, you’ve got a stored configuration you know is good in one of these ZIP archives. How do you restore it if you can no longer boot your Pi and access pibootctl?

Simply place the SD card in another machine, find the ZIP archive storing the boot configuration you want (which will be under the “store” directory we explored above), and extract it over the top of current boot configuration. You’ll almost certainly be prompted to replace the existing files (config.txt, etc.) but that’s fine: permit the replacement and you’ve restored your boot configuration! It really is that simple.

Addendum: Why ZIPs?

I did debate just using sub-directories instead of ZIP archives and storing configurations “raw” so users could simply copy them back without bothering with unzip tools. This is certainly an attractive method, but:

It makes it much easier for users to manipulate stored configurations in place. As mentioned above, I’m not (yet) convinced this is a good or bad idea, so for now I’ve opted for the method which makes it harder.
It makes it easy to store meta-data alongside the stored configuration (the aforementioned hash). We could do this in an auxilliary file in the stored configuration but then we must be sure that could never exist in a legitimate stored config. Or it could be in a file above the directory storing the configuration, but then that has to be kept in sync with the stored directories (which, as mentioned above, are easy to manipulate in their own right).

Next Time

That’s quite enough on this subject. Next time (hopefully less than months away!) we’ll take a look at the “diff” command, and how to integrate pibootctl with more friendly tools on top of it.

Using pibootctl

2020-05-20T00:00:00+01:00

In the previous post, we looked at why I’ve been working on pibootctl. But if you just want to use it, look no further! (Or, do go and read the fine manual if you prefer).

Installing it on your pi is pretty simple:

$ sudo add-apt-repository ppa:waveform/pibootctl
$ sudo apt install pibootctl

Side note: in future, this should be included by default on Ubuntu images, but for now it lives in my PPA.

Help!

The utility includes several sub-commands, but the first you ought to learn (or even guess) is “help”:

$ pibootctl help
usage: pibootctl [-h] [--version]
                 {help,?,status,dump,get,set,save,load,diff,show,cat,list,
                 ls,remove,rm,rename,mv} ...

pibootctl is a tool for querying and modifying the boot configuration of
the Raspberry Pi.

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

commands:
  {help,?,status,dump,get,set,save,load,diff,show,cat,list,ls,remove,rm,
  rename,mv}
    help (?)            Displays help about the specified command or setting
    status (dump)       Output the current boot time configuration
    get                 Query the state of one or more boot settings
    set                 Change the state of one or more boot settings
    save                Store the current boot configuration for later use
    load                Replace the boot configuration with a saved one
    diff                Show the differences between boot configurations
    show (cat)          Show the specified stored configuration
    list (ls)           List the stored boot configurations
    remove (rm)         Remove a stored boot configuration
    rename (mv)         Rename a stored boot configuration

You can also ask this sub-command for help on any of the other sub-commands … including itself:

$ pibootctl help help
usage: pibootctl help [-h] [command-or-setting]

With no arguments, displays the list of pibootctl commands. If a command
name is given, displays the description and options for the named command.
If a setting name is given, displays the description and default value for
that setting.

positional arguments:
  command-or-setting  The name of the command or setting to output help for

optional arguments:
  -h, --help          show this help message and exit

If you fancy good old man-pages, there’s those too. One for the tool itself, and one for each sub-command. For example:

$ man pibootctl
$ man pibootctl help

These include some additional notes and examples which may be worth a look (they’re also included in the online manual).

First Steps

What about actual usage? The first thing you’ll probably want to do is look at the current state of your boot configuration. This is what Ubuntu 20.04’s looks like in pibootctl “out of the box”:

$ pibootctl status
┌─────────────────────────┬──────────────────────┐
│ Name                    │ Value                │
├─────────────────────────┼──────────────────────┤
│ audio.enabled           │ on                   │
│ boot.devicetree.address │ 50331648 (0x3000000) │
│ boot.kernel.64bit       │ on                   │
│ boot.kernel.cmdline     │ 'cmdline.txt'        │
│ boot.kernel.filename    │ 'uboot_rpi_3.bin'    │
│ i2c.enabled             │ on                   │
│ serial.enabled          │ on                   │
│ spi.enabled             │ on                   │
└─────────────────────────┴──────────────────────┘

Several things to note above:

Oooh, pretty!: The default output mode is designed to be human-readable and, if your locale includes “UTF-8” then unicode line-drawing characters will be used for tables.
You’re using arm64 on a Pi3?: Right now, yes. It was the first card and Pi combination that came to hand!
There’s not many settings there?: Yes, only modified settings are listed by default because otherwise there’s … a lot (and most of them you probably don’t care about). You can list all the settings with the --all option, but be warned there’s more than a page’s worth!
I don’t recognize those setting names!: Yes. This is probably going to be one of the more contentious decisions in the design of pibootctl. I could’ve stuck with the actual commands used in the Pi’s boot configuration, e.g. “dtparam=spi=on” or “uart_enable=1” or “disable_overscan=1”, but I didn’t. This probably needs justifying as it’s going to confuse anyone familiar with the existing settings (including me!)…

Setting Names

The names for settings in the Pi’s bootloader have grown more or less organically over time and occasionally change as new interfaces emerge (e.g. “display_rotate” being deprecated in favour of “display_lcd_rotate” and “display_hdmi_rotate”). Moreover, the settings are for use in a fairly low-level tool (the bootloader) so there’s a general preference for defaults to be “0”, hence the occasional setting with reverse setting logic (“disable_overscan=1”).

The lack of a common naming standard (while understandable) does make it tricky to query groups of settings common to an area. For example, how would you query all the video related settings (remembering that “disable_overscan” setting)? However, if a naming scheme is imposed on the settings, it becomes easy:

ubuntu@pitest:~$ pibootctl status --all video.*
┌──────────────────────────────┬──────────┬────────────────────────────────┐
│ Name                         │ Modified │ Value                          │
├──────────────────────────────┼──────────┼────────────────────────────────┤
│ video.cec.enabled            │          │ on                             │
│ video.cec.init               │          │ on                             │
│ video.cec.name               │          │ 'Raspberry Pi'                 │
│ video.dispmanx.offline       │          │ off                            │
│ video.dpi.clock              │          │ off                            │
│ video.dpi.enabled            │          │ off                            │
│ video.dpi.format             │          │ 1 (9-bit RGB666; unsupported)  │
│ video.dpi.group              │          │ 0 (auto from EDID)             │
│ video.dpi.hsync.disabled     │          │ off                            │
│ video.dpi.hsync.phase        │          │ off                            │
│ video.dpi.hsync.polarity     │          │ off                            │
...

The usual “*” and “?” wildcards can be used in the setting pattern. In the above output it’s worth noting that explanatory text is included for certain setting values (e.g. “auto from EDID” for “video.dpi.group”). A more minimal example is all the settings for the TV output:

ubuntu@pitest:~$ pibootctl status --all video.tv.*
┌─────────────────────┬──────────┬──────────┐
│ Name                │ Modified │ Value    │
├─────────────────────┼──────────┼──────────┤
│ video.tv.aspect     │          │ 1 (4:3)  │
│ video.tv.colorburst │          │ on       │
│ video.tv.enabled    │          │ on       │
│ video.tv.mode       │          │ 0 (NTSC) │
└─────────────────────┴──────────┴──────────┘

This scheme also means that simple sorting by name groups related settings together in the output. However, as mentioned above it is going to confuse anyone who’s familiar with the underlying setting names. To try and mitigate this a bit, the help output for all settings includes the underlying command name (or overlay and parameter name as appropriate), and the “help” command itself will also accept such names:

$ pibootctl help disable_overscan
      Name: video.overscan.enabled
   Default: on
Command(s): disable_overscan

When enabled (the default), if a group 1 (CEA) HDMI mode is selected
(automatically or otherwise), the display output will include black borders
to align the edges of the output with a typical TV display.

There are cases where an underlying command is represented (or affected by) several settings (and vice versa). In this case, asking for help on such a setting will tell you the names of all settings that can affect it:

$ pibootctl help start_file
start_file is affected by the following settings:

boot.debug.enabled
boot.firmware.filename
camera.enabled

Scripting

We’ve seen the pretty-printed output of multiple settings above. What about retrieving an individual setting for scripting purposes? The “get” command is used for this (there’s very little point using this command outside a script):

$ pibootctl get serial.enabled
on

While this is simple enough, it’s not very efficient when it comes to obtaining the value of lots of settings. To accommodate usage from scripts, the majority of commands in pibootctl (including “get” and “status”) accept the --json, --yaml, or --shell switches to produce output in JSON, YAML, or a shell-compatible style respectively:

$ pibootctl status --json --all "video.hdmi*"
{"video.hdmi.edid.override": false, "video.hdmi.safe": false,
"video.hdmi.4kp60": false, "video.hdmi.edid.ignore": false,
"video.hdmi.edid.contenttype": 0, "video.hdmi0.edid.filename": "edid.dat",
"video.hdmi.edid.parse": true, "video.hdmi0.enabled": null,
"video.hdmi.edid.3d": false, "video.hdmi0.audio": null,
"video.hdmi0.boost": 5, "video.hdmi0.group": 0, "video.hdmi0.timings": [],
"video.hdmi.powersave": false, "video.hdmi0.mode": 0,
"video.hdmi0.mode.force": false, "video.hdmi0.encoding": 0,
"video.hdmi0.drive": 0, "video.hdmi0.rotate": 0, "video.hdmi0.flip": 0,
"video.hdmi1.boost": 5, "video.hdmi1.enabled": null, "video.hdmi1.audio":
null, "video.hdmi1.edid.filename": "edid.dat", "video.hdmi1.timings": [],
"video.hdmi1.group": 0, "video.hdmi1.mode": 0, "video.hdmi1.mode.force":
false, "video.hdmi1.encoding": 0, "video.hdmi1.drive": 0,
"video.hdmi1.rotate": 0, "video.hdmi1.flip": 0}
$ pibootctl get --json serial.enabled
true

Writing?

That’s enough for this post; next time we’ll have a look at how pibootctl modifies the boot configuration, and the steps it takes to try and do so safely!

Introducing pibootctl

2020-04-27T00:00:00+01:00

For the last few months I’ve been working on a little tool called pibootctl. The hope was to include that in the Focal release, but it couldn’t quite make the cut while so much work was going on with the boot sequence. Still, it’s available from a PPA (ppa:waveform/pibootctl) now for those that wish to try it out, and the documentation is online.

This is the first in a short series of posts explaining some of the background behind the tool, and what its future (hopefully!) is.

So what is it?

It’s a command line tool for managing the Pi’s boot configuration.

So what is it?

It’s a utility for saving, restoring, and editing the various text files on the boot partition.

So what is it?

It’s a less friendly raspi-config … which doesn’t do networking either.

Only Joking!

Well, sort of. To be clear: it’s not intended to be a replacement for the excellent raspi-config tool, however I do intend it to form part of the underlying kit for something vaguely similar.

Why?!

Or more precisely “why not just use or port raspi-config?”. I’ve been asked this pretty frequently so it’s probably best to have somewhere I can point to with all my arguments in a row, so here goes …

Firstly, let’s be clear: raspi-confgi is great. It does exactly what it says on the tin in a friendly, user-accessible way, and Ubuntu certainly needs it, or something very much like it. So let’s begin with why we can’t “just use it”:

Ubuntu Server is pretty tightly wedded to netplan for its networking configuration. Whether that’s via the subiquity installer on regular server images, or via cloud-init on our cloud and Pi images.
Raspbian is more traditional in its networking approach, and raspi-config knows nothing about netplan. Now, that’s not to say it won’t work: it will set up your WPA config, but it won’t keep the netplan configuration in sync so the next time you run netplan apply it’s overwritten. Not an ideal user experience.

Well, that doesn’t sound so complex? We could patch raspi-config to affect the netplan configuration instead couldn’t we?

Yes we could, but now we’re carrying a delta to the upstream raspi-config tool. Either someone needs to maintain that delta as raspi-config evolves, or we need to get that delta applied upstream. The former is undesirable, and the latter is highly unlikely to happen (unless Raspbian wants to switch over to using netplan).

Besides, it’s not the only problem to overcome:

Ubuntu’s boot sequence on the Pi is in some ways more complex and in some ways simpler than Raspbian’s, but either way it’s pretty different. One notable facet is that our “config.txt” file includes other files by default.
Unfortunately, raspi-config doesn’t handle includes in the boot configuration, either for parsing or writing. In the face of includes, it either doesn’t know the state of existing settings (from includes), or will fail to notice if its written settings are overridden in later includes.

Solving this means writing a fairly comprehensive parser for the “config.txt” file (and its includes) and some logic to compare a “desired” configuration to the “effective” configuration after writing changes to determine whether settings have been overridden in later included files.

Funnily enough, that’s exactly what pibootctl does!

Ah ha! Could pibootctl be used by raspi-config to fix these issues? Potentially; I’ve ensured there’s absolutely nothing Ubuntu-specific in the tool, and it’s pure Python so there’s nothing preventing it from running on architectures we don’t support like the Pi Zero, but we’re getting ahead of ourselves.

Let’s finish off the issues with porting raspi-config:

Setting up the boot options for desktop or CLI: we don’t include a desktop on our images, so that’s out (or at the very least, becomes conditional).
Setting up auto-login and splash-screen. We don’t have those either (on classic, and there’s … difficulties with adding a splash screen that are sufficiently intricate I won’t bore you with them here).
VNC. Raspbian includes the extremely slick (but proprietary) Real-VNC, which we don’t have on Ubuntu. Side note: personally, the one feature I’d really like to see ported to one of the open source VNC implementations is dispmanx capture so that things like the camera preview can pass over VNC, as on Real-VNC.
Remote GPIO. Nope, we haven’t got pigpio into the archive yet either.
Read-only FS via overlay. Again, there’s … intricate issues here.

You get the picture. Even if we did port raspi-config, tweak it to use netplan, and slot in pibootctl to handle the configuration re-writing, we’d still wind up with something inferior. Something the user would fire up and immediately notice several options missing. Something that, even if we included the VNC option, would work quite as well as on Raspbian.

Not an ideal result, and we’d still be carrying a fat delta which would require on-going maintenance every time upstream changed.

Addendum

There’s a flip-side to the above. While there’s several things we’d need to drop from a ported raspi-config, there’s also plenty of stuff raspi-config doesn’t do that we could trivially add given the capabilities of the underlying tools on Ubuntu:

With netplan, we could have full Ethernet configuration options, IPv4 and IPv6 settings, and more.
We tend to favour non-password SSH configurations; should we not add facilities to import a user’s SSH keys?
There’s some facilities of pibootctl (which we’ll be seeing in forthcoming posts) that’d be interesting to include too!

So … how then?

That’s enough argument about why porting raspi-config over to Ubuntu is not as trivial as most think (or in some cases even desirable). But it’s obvious to everyone (even me!) that we do need something user friendly for Pi configuration in Ubuntu.

This is something I’ll be working on this cycle, building on top of the tools we’ve got in Ubuntu. Which brings us (finally!) back to pibootctl. When I was looking at the tools we’ve got to play with for system configuration on Ubuntu, which I could build upon for a raspi-config-esque tool, one major piece of the jigsaw was missing: something to deal with “config.txt”.

Not terribly surprising given it’s entirely specific to the Pi but we did need something to fill that gap. Initially, I’d considered grabbing bits from raspi-config but when I checked it there were a couple of problems:

Firstly, it uses lua for re-writing the configuration. Nothing expressly wrong with that, but lua isn’t in “main” on Ubuntu and anything included by default on our images (as such a raspi-config-esque tool eventually would) is meant to be in “main” (not “universe”).

Moving stuff to main involves a pretty thorough review process and wasn’t something I thought we could realistically do for a full-blown programming language when it was only going to be used for a single thing. That already necessitated re-writing a chunk of raspi-config with something in “main”, most likely python.

It can’t handle configurations using includes, and for better or worse I’d already moved us over to a configuration that used them (in Eoan). Moving back would be a nightmare of detection and manipulation logic in some package’s post-inst scripts (more complex than moving to the split configuration in the first place and I remember how nasty that was). Not to mention, I’m still of the opinion that the split configuration brings certain benefits (though that’s involved enough that it should probably be its own post).

Not now?

It feels like I’ve been wittering on for way too long in this post, so I’m going to stop here, and leave the first play-through for next time! In the meantime, anyone that wants to have a play, go have a look at the PPA or the documentation.

Ubuntu Desktops on the Pi

2020-04-17T00:00:00+01:00

See Also: A look at the Hirsute desktop image, Jammy desktop image, and Making desktops with cloud-init.

One of the things I’ve been looking at recently is the relative performance of our desktop packages on the Pi (a lot of people seem to be trying to use the Pi 4 as a desktop work-horse - and it is reasonably capable of this with the right selections).

Anyway, here’s some quick notes from my experiments. Firstly, the various desktop packages and some comments (just in the order I tried them):

ubuntu-desktop: A bit sticky here and there (particularly during login and application selection - basically anywhere Gnome tries to do any animation), but tolerable.
xubuntu-desktop: A bit slicker than the default on the Pi; could actually be a reasonable choice for a default desktop.
lubuntu-desktop: It’s not pretty, but damn this flies! No real surprise there given it’s LXQt which is the basis of Raspbian too. With a bit of layout tweaking this would definitely be my go to choice of desktop under Ubuntu on the Pi.
kubuntu-desktop: Ouch, this is painful. Don’t misunderstand me - I love Kubuntu generally (I’m one of those old stick-in-the-muds that bemoans all the options that’ve disappeared from Gnome over the years and loves that they’re all still there on KDE). However, it’s undeniably heavier and on the Pi, even on a 4Gb Pi 4, it’s just unusable.
ubuntu-budgie-desktop: Oooh, pretty! Sadly, not that functional. The start menu is better than the default but (shock! horror!) the terminal won’t start, so that’s this experiment prematurely ended (yes, I could switch tty and fiddle around but the Mac-like prettiness hasn’t tempted me that much :).

Compatibility notes

I was only testing the arm64 images; I’d expect similar results under armhf though. Several things are worth adding to improve compatibility or enable hardware (these apply to pretty much all the desktops above):

Graphics

Add dtoverlay=vc4-fkms-v3d to /boot/firmware/syscfg.txt to enable the “fake” KMS frame-buffer. Lots of GUI instability without this. For all those that want this to be the default: so do I, but there’s issues with doing this that need solving first (breaks u-boot’s video console, and breaks boot on the 3A+).

Audio

From Focal (20.04) onwards, dtparam=audio=on should be the default in the boot configuration so audio should at least be present. However, if you’re installing a desktop environment you also want to tweak a pulse-audio parameter to improve its scheduling and avoid choppy audio. Edit /etc/pulse/default.pa and find the line:

load-module module-udev-detect

And append tsched=0 so that the line reads:

load-module module-udev-detect tsched=0

(thanks to BranLoux896 on the Raspberry Pi forums for mentioning this!)

Login managers

Most of the desktop packages provide a selection of display managers (gdm3, lightdm, ssdm, etc.) on installation. I attempted most selections and found that lightdm doesn’t work for some reason I haven’t tried to figure out yet, but gdm3 and ssdm work fine.

However, one issue with gdm3: it always defaults to the “ubuntu” desktop even if you’ve installed, say, lubuntu-desktop. After selecting the user to login as, click the gears at the bottom right and select the actual desktop you installed before completing the login :)

Wireless & Bluetooth

Install the “iw” package (which’ll pull in “crda”) and then run sudo iw reg set GB (if you’re in the UK like me; substitute GB for your region code otherwise) to set the WiFi region correctly (usually enables a few channels that aren’t enabled in the default “world region”). I’ll endeavour to get this in the image by default in Focal, but you’ll still need to set the region code for now.

Install the “pi-bluetooth” package to enable Bluetooth. This “just works” once the package is installed.

Remote desktops

I ran a quick experiment in Terminal Services-style remote desktops with xrdp under lubuntu-desktop (the only one I figured would be light enough to be worthwhile for such endeavours). To try this:

install the xrdp package
add the xrdp user to the ssl-cert group
reboot

Then you should be able to connect remotely using an RDP client (like remmina) providing you specify the username and password in the connection settings.

Another quick experiment in screen-scraping style remote desktop was semi-successful with the tigervnc-scraping-server package but a further experiment to do TS-style remote desktop with VNC wouldn’t work (some issue with GDM I didn’t have time to dig into).

Power saving

Power saving is … rather pointless. I forget which desktop I saw this on, but one of them decided to put the Pi to sleep on idle timeout and … that just effectively switched the Pi off! So, turn off all those suspend timers :)

Package configuration

2020-04-16T00:00:00+01:00

In the previous post we saw how to configure networking on first boot with cloud-init. If your Pi has a reliable network connection during that first boot (preferably Ethernet, given the current wifi issue described in LP: #1870346) you can also opt to have cloud-init install whatever packages you’re going to need (and configure them too!) so your system is completely ready for you when you login.

Updates

First, it’s useful to learn how to get the local package index up to date (especially if you’re dealing with an old image). This can be done by adding the following line to the user-data file (that one again!) on the boot partition:

package_update: true

If you also want to upgrade the system to the latest version of all installed packages on the initial boot, you can add:

package_upgrade: true

Installation

As you can tell, on Ubuntu these effectively run the “apt update” and “apt upgrade” commands, but cloud-init also supports several other distros, hence the rather generic naming of these options. How about installing extra packages? This is also simple to arrange:

# Install some things that Dave probably ought to consider including on
# the images by default...
packages:
- pi-bluetooth
- iw

You can specify a particular version of a package by using a list, but this generally isn’t a good idea (there’s no guarantee that version is going to be available unless you’re installing from, say, a repo you control like a PPA):

packages:
- [nginx-light, 1.17.9-0ubuntu3]

Speaking of which: what if you want to include a PPA as a source?

apt:
  sources:
    my-ppa:
      source: "ppa:waveform/pibootctl"  # quote the source

This may seem like an overly convoluted series of structures (“source” within a label within “sources” within “apt”) but the reason for this is that there are a great many options for apt, and a number of different means for specifying third party repos (including importing of signing keys, and templates for deriving new repos from existing ones).

Configuration

A couple more features are worth introducing in the context of installing packages. Firstly, the ability to write arbitrary files. This is particularly useful for providing configuration for packages you’ve installed:

write_files:
- path: /etc/default/keyboard
  content: |
    # KEYBOARD CONFIGURATION FILE
    XKBMODEL="pc105"
    XKBLAYOUT="gb"
    XKBVARIANT=""
    XKBOPTIONS="ctrl: nocaps"
    BACKSPACE="guess"

And finally, the ultimate blunt tool of “runcmd” which effectively lets you run arbitrary commands as root on the first boot; use with caution! In this case we can use it to set the wireless regulatory domain:

runcmd:
- iw reg set GB

Putting it all together

Here’s a user-data file I use when testing Pi daily images. It sets up the Pi so I can access it over SSH, importing my keys from GitHub, installs some extra packages for WiFi and Bluetooth, configures the WiFi regulatory domain, adds the PPA for pibootctl (a little project I’ve been working on for boot configuration manipulation):

user-data

 1 #cloud-config
 2 
 3 chpasswd:
 4   expire: false
 5   list:
 6   - ubuntu:ubuntu
 7 
 8 hostname: pitest
 9 
10 apt:
11   sources:
12     my-ppa:
13       source: "ppa:waveform/pibootctl"  # the quotes are mandatory
14 
15 package_update: true
16 package_upgrade: true
17 packages:
18 - pi-bluetooth
19 - iw
20 - pibootctl
21 
22 runcmd:
23 - iw reg set GB
24 
25 ssh_pwauth: false
26 
27 ssh_import_id:
28 - gh:waveform80
29 
30 write_files:
31 - path: /etc/default/keyboard
32   content: |
33     # KEYBOARD CONFIGURATION FILE
34     XKBMODEL="pc105"
35     XKBLAYOUT="gb"
36     XKBVARIANT=""
37     XKBOPTIONS="ctrl: nocaps"
38     BACKSPACE="guess"
39 
40 runcmd:
41 - reboot

At the moment this file lives on my desktop and, after flashing a fresh image, the first thing I do is copy that file onto the boot partition, replacing the existing user-data. Then I boot the Pi and leave it for a few minutes to sort everything out!

Hopefully that gives you some ideas for playing with. And remember, you can throw these same configurations at various cloud instances too (which can be quite handy if you ever need to scale things up temporarily!)

Coming Up

Next I’m going to take a short break and see if I can set up some form of comments here (probably via the lazy disqus route), and then I might delve into a series on pibootctl or something.

Network configuration

2020-04-10T00:00:00+01:00

In the previous post we saw how to configure multiple different users via user-data, one of the cloud-init configuration files. This time, we’ll turn our attention to network-config which is probably the most important configuration file aside from user-data as it tells Netplan what to do when setting up the network.

Note

What’s Netplan? It’s a tool used on Ubuntu used to present a common interface to whatever happens to manage the network. For instance, on desktops this is typically Network Manager, while servers tend to use systemd’s networkd service. Netplan can “render” a configuration to work with either of these systems.

network-config

This file is basically just a typical Netplan configuration minus the root “network:” value. By default it contains the following on the Pi images:

network-config

 1 version: 2
 2 ethernets:
 3   eth0:
 4     dhcp4: true
 5     optional: true
 6 #wifis:
 7 #  wlan0:
 8 #    dhcp4: true
 9 #    optional: true
10 #    access-points:
11 #      myhomewifi:
12 #        password: "S3kr1t"
13 #      myworkwifi:
14 #        password: "correct battery horse staple"
15 #      workssid:
16 #        auth:
17 #          key-management: eap
18 #          method: peap
19 #          identity: "me@example.com"
20 #          password: "passw0rd"
21 #          ca-certificate: /etc/my_ca.pem

On line 1, “version:” just declares the version of the Netplan schema that the file conforms to (this should be “2”).

On line 2, the “ethernets:” value starts which sets up the “eth0” interface to use DHCP for IPv4 (we haven’t specified anything for IPv6) and marks the interface optional (indicating we shouldn’t wait for it to come up on boot, just in case there’s no cable plugged in).

On line 6 onwards is a big “wifis:” section which is commented out (the “#” prefix), and which contains several examples of common WiFi configurations. We’ll explore this in the next section.

What should we do if we wanted to use a static IP configuration for eth0 instead of DHCP? The following is an example block which should be reasonably obvious to anyone familiar with IP networking.

version: 2
ethernets:
  eth0:
    addresses:
      - 192.168.0.20/24
    gateway4: 192.168.0.1
    nameservers:
      search: [mydomain]
      addresses: [192.168.0.1, 8.8.8.8]
    optional: true

It is important to remember that “addresses:” takes a list of addresses (as an interface can have multiple).

wifi configuration

This is much the same as ethernet (as you can probably see from the commented out example in the default file). The only major difference is the extra “access-points” sub-value. Under this, you can list all the wifi access points you regularly connect to, and the credentials required for each.

For example, let’s assume your home router is called “BT-1234” with password “a very long password”, but you also have an extender called “wifi_ex” which has the (slightly less secure) password “passw0rd”. In this case you might use the following configuration:

version: 2
wifis:
  wlan0:
    dhcp4: true
    optional: true
    access-points:
      "BT-1234":
        password: "a very long password"
      "wifi_ex":
        password: "passw0rd"
ethernets:
  eth0:
    dhcp4: true
    optional: true

Note that it doesn’t matter what order the top-level values appear in. I tend to write “version” first, followed by “ethernets” but it really doesn’t matter. The same goes for sub-values of values; place them in whatever order you want.

Later changes

What does cloud-init do with the network-config file? Roughly speaking, it copies it (with an additional root “network:” value) to /etc/netplan/50-cloud-init.yaml (the Netplan configuration consists of all YAML files under /etc/netplan) and then applies it.

Note

Unfortunately, there’s currently a bug here. While cloud-init will happily copy your WiFi configuration to Netplan, it won’t (currently) apply it automatically. To do that, you need to reboot your Pi afterwards (or run sudo netplan apply from the command prompt, assuming you have a keyboard attached).

The issue is being tracked under LP: #1870346 for those interested.

For users of Raspbian this probably feels similar to its facility of copying a file named wpa_supplicant.conf on the boot partition. The main difference is that this deals with all interfaces rather than just wifi.

Remember that cloud-init only deals with the very first boot. What happens if you want to change your network configuration later? In this case simply edit the file cloud-init created, /etc/netplan/50-cloud-init.yaml, and run sudo netplan apply (or reboot).

Back to user-data

While we’re looking at network related stuff, there’s a couple of settings in user-data we should cover. First is “hostname” which can be used to set the hostname of a freshly minted machine. It’s as simple as specifying this at the top level:

hostname: waldorf

Whether this is honoured by the local DNS server will depend on its configuration. If you use dnsmasq as your local DHCP/DNS server (as I do) then it should work just fine.

System time can often be an issue on Pis, which lack a RTC. NTP is the usual solution to this for any Pi connected to a network. Often, the DHCP server will supply the address of an NTP server, or the NTP’s default configuration will be sufficient. However in some cases you may need to configure the NTP servers to query manually. Again, you can use cloud-init to set NTP up:

ntp:
  enabled: true
  servers:
  - ntp.server.local
  - ntp.ubuntu.com
  - 192.168.23.2

Cloud-init knows how to configure a reasonable variety of NTP clients and can adapt the configuration above to their particular format. But how to get such a client installed on the very first boot? That’s a topic we’ll leave for next time…

Setting up users

2020-04-08T00:00:00+01:00

In the previous post we had a quick look at the user-data configuration file for cloud-init. In this post, we’ll have a more extensive look at some of the other options for configuring users in this file.

Different Passwords

We’ve seen how cloud-init is told to set the initial user password, but there’s a few more possibilities there we haven’t seen yet. Have a look at the following example:

1 chpasswd:
2   expire: false
3   list:
4   - ubuntu:ubuntu
5   - root:$1$1dE3dcLz$8iWGkDxx9SCWdQfkPXbCE/
6   - pi:RANDOM

Here, as last time, “expire” is false so you won’t be prompted to change the password on first login. We’ve also specified a password for a pre-existing user (root, not that this is necessarily encouraged on a system with “sudo”). Finally, the special value “RANDOM” tells cloud-init to generate a completely random password for the “pi” user. This will be output on the console, and in the /var/log/cloud-init-output.log file.

Warning

You may want to erase (or at least restrict access to) the cloud-init-output.log file if you choose to use RANDOM passwords.

It is also worth noting (as the cloud-init docs do) that providing hashes of passwords is not necessarily secure. You may want to try running the hash above through something like John the Ripper to convince yourself of this!

Different Users

We’ve already seen how cloud-init creates a default “ubuntu” user, and for many purposes that’s enough. What would you do if you wanted to create other users besides this, and can we import different sets of SSH keys to each user? Here’s another example user-data:

 1 groups:
 2 - robot: [robot]
 3 - robotics: [robot]
 4 - pi
 5 
 6 users:
 7 - default
 8 - name: robot
 9   gecos: Mr. Robot
10   primary_group: robot
11   groups: [users]
12   ssh_import_id: gh:example
13   passwd: $5$hkui88$nvZgIle31cNpryjRfO9uArF7DYiBcWEnjqq7L1AQNN3

The “groups” value lists the groups to create, and optionally the members they should have. Here we set up three groups named “robot”, “robotics”, and “pi”. The “robot” and “robotics” groups will each have a single member, the “robot” user, while the “pi” group will start off with no members.

The “users” value lists all the users to create, along with their details. Here we create two users: the default “ubuntu” user which is simply represented by the “default” entry at the top. You can remove this if you don’t want the default user.

Then there’s the aforementioned “robot” user. Various fields give this user an initial password (in the form of a hash under “passwd”), tell the ssh-import-id utility to import SSH keys from the “example” user on GitHub, and additionally places the user in the “users” group.

There’s an absolute pile of options that be specified for users; have a look at the Users and Groups section of the cloud-init docs for more information.

Coming Up

Next, we’ll take a break from user-data to look at the other important configuration file for cloud-init: network-config.

Securing your Ubuntu Pi

2020-04-02T00:00:00+01:00

One of the neat features of Raspbian is the ability to set up SSH out of the box by placing an ssh file on the boot partition. This is a security feature to ensure that your Pi isn’t booted up with SSH for remote access enabled, plus a widely known default username and password (pi / raspberry) which has full “sudo” rights before you’ve had a chance to change it.

On the surface, Ubuntu on the Pi has some similarities here, and some differences:

As with Raspbian, there’s a default username and password (ubuntu / ubuntu).
As with Raspbian, the default user has full “sudo” rights.
Unlike Raspbian, SSH is enabled by default. This is an obvious security problem given the above points.
To mitigate this, unlike Raspbian, Ubuntu forces you to change your password on first login.

Still, doesn’t this leave a (hopefully brief) period of time when your Pi is potentially network connected, with an active SSH daemon, and a widely known default username and password combination with full root access? Yes … it does. Can we do better than this? Yes, with …

cloud-init

First, we’ll see what I usually stick in the user-data configuration file on a freshly flashed SD card, then we’ll dissect it to see what’s going on and what else you can do with it!

Note

The file’s name is literally user-data with no extension. It’s essentially a text file though, so feel free to open it in Notepad or any other plain text editor.

This file is part of the configuration for cloud-init and you’ll find it on the boot partition. This is the first partition on the card and, if you’re sticking the card in a Windows or Mac OS X machine, the only partition that’ll appear (because it’s formatted FAT).

Without further ado, here’s the TL;DR version:

 1 ssh_pwauth: false
 2 
 3 ssh_import_id:
 4 - lp:waveform
 5 
 6 chpasswd:
 7   expire: false
 8   list:
 9   - ubuntu:ubuntu

What does all this YAML mean?

Line 1 configures SSH to disable password authentication, so that’s our security hole fixed (this happens before the default user is created). The username and password may be well known but they’re now useless for remote access. Well that was easy, all done and … oh, wait a minute. How should I access the Pi remotely now?

Lines 3-4 take care of this. These tell cloud-init to use the incredibly useful ssh-import-id utility to grab my public SSH keys from Launchpad (that’s the “lp:” prefix, you can also use “gh:” for GitHub) and install them as authorized keys for all users that cloud-init creates.

Lines 6-9 tell cloud-init that the “ubuntu” user (which it will create by default) should have the password “ubuntu” and that I don’t want it expired (so it won’t prompt me to change it on first login). Half the time I’m dealing with an ephemeral image I won’t be keeping around anyway, so I don’t much care about changing the password, and on the occassions that I do want to keep the image, I don’t want to be bugged about the password on that first login.

And that’s it! I boot my Pi for the first time (making sure it has an Ethernet connection), wait a bit while cloud-init does its thing but, after about a minute, my Pi has an “ubuntu” user I can access over SSH with public-key authentication and no gaping security holes.

Note

cloud-init performs initial set up. That is to say, it only performs set up on the very first boot. If you want to configure your Pi with cloud-init you need to make changes to the configuration (detailed below) before booting your Pi for the first time.

If you wish to re-run cloud-init at a later time, you can do so with the following command but be aware that some settings may not work as expected (because users will already have been created, files already written, etc. etc.):

$ sudo cloud-init clean --logs --reboot

In general, consider this guide relevant for first boot situations only.

YAML Refresher

In case you’re wondering what the format of the file above is, it’s YAML. It may be useful to have a quick refresher course in YAML as that’s what user-data (and all the other cloud-init configuration files) are written in. Here’s the file we saw above again, with some slight alterations for the purposes of demonstration:

 1 ssh_pwauth: false
 2 
 3 ssh_import_id:
 4 - lp:waveform
 5 - gh:foobar
 6 
 7 chpasswd:
 8   expire: false
 9   list: [ubuntu:ubuntu, pi:raspberry]

There are three top-level values: “ssh_pwauth”, “ssh_import_id”, and “chpasswd”
“ssh_pwauth” is a simple value set to “false”. In all cases, the name of a value is written prior to a colon (“:”)
Next “ssh_import_id” is a list value. There’s two ways of writing lists in YAML and this is the “long-hand” variant: “-” prefixed items, one per line below the name of the value:
```
mylist:
- Item 1
- Item 2
- Item 3
```
The other way of writing lists is the “short-hand” variant: comma-separated values enclosed in square brackets. The list above could equally well be written like this:
```
mylist: [Item 1, Item 2, Item 3]
```
Finally “chpasswd” is a compound value, containing two other values. This is indicating by the indentation before the names of the two sub-values, “expire” and “list”

Warning

You must not mix indentation styles from one line to the next (as in Python). When editing this file you are strongly recommended to stick to spaces for indentation; do not use the Tab key (unless you’re certain your editor is configured to expand tabs to spaces)!
As with lists, there’s a short-hand variant for compound values too: command-separated elements enclosed in curly braces. The following two declarations are equivalent in YAML:
```
character: {name: Zaphod Beeblebrox, age: 42}

character:
  name: Zaphod Beeblebrox
  age: 42
```

List items can be simple values, as we’ve seen above or they can be compound values themselves. For example:

characters:
- name: Oberon
  description: King of the fairies
- name: Titania
  description: Queen of the fairies
- name: Bottom
  description: a weaver
- name: Robin Goodfellow
  description: a mischievous sprite

Coming Up

In the next post we’ll take a look in more detail at the user-data file and see how to create multiple different users, assigning them different SSH keys.