D0!

SRU

A quick tangent on Stable Release Updates, or SRUs as we refer to them (because there’s nothing more developers love than another TLA). Actually, first another tangent on stability of Linux distros.

Linux distributions can be roughly divided into two categories: stable distros and rolling distros. Stable distros grew out of the frustrations of server operators who wanted a guarantee of stability above all else: once the system was installed and working the only updates they wished to see were those that fixed bugs. They had precisely zero interest in new features and even minor bug fixes were probably more risk than they were worth. The only time risk was introduced into the equation was when the OS as a whole was updated (a “dist-upgrade” in Debian terms), and because this was a known risk, it would be planned for, tested, and implemented in a controlled manner.

Classic “stable” distros include Red Hat, Debian, and of course Ubuntu.

The rolling distros by contrast, have no concept of an OS “release”. They simply package the latest versions of all things at all times. Much of the time, this does work reasonably well, but it does mean that at certain times there fairly major changes that happen in a more or less “adhoc” fashion. Historically, consider the move to systemd, or transitions between Qt versions.

Classic “rolling” distros include Gentoo and Arch.

Some distros incorporate both models. Alpine and Debian are examples of these. While Debian has a “stable” release (currently “bookworm”, soon to be “trixie”), it also has the “unstable” branch called “sid” which is basically a rolling distro in itself. Alpine likewise makes six monthly releases, but has a rolling “edge” branch.

While the “stable” distro model is widely accepted (and very reasonable, in this writer’s opinion), it does cause some tension, particularly with the rise of desktop Linux usage. Desktop users often do want the fancy new features in their applications, don’t have the same level of risk-aversion as the typical hard-boiled sysadmin [2], and thus aren’t willing to wait 2 or 3 years for the next release of their stable distro to get them. This is part of the drive behind the rise of “alternate” packaging systems on various distros (Flatpaks, Snaps, AppImages, etc).

On Ubuntu, updates to stable releases (outside of the realm of snaps) have to follow the Stable Release Update process which incorporates numerous checks and balances to try and avoid regressions. Test plans must be written, reviewed, and agreed upon before anything is even uploaded. Updates must be minimally invasive (no full version backports — those have to follow a separate process which doesn’t automatically update existing installs). After upload, verification is carried out, and only then is the fix allowed into the release.

But not right away…

Phasing

Even after an update to a stable release has made it through review, verification, and upload, such updates go through a “phasing” procedure. This is where the update is slowly trickled out to larger and larger proportions of the userbase. The idea is that, if an update does have a catastrophic bug in it, the damage can be limited to the portion of the userbase that (randomly or by choice) got the update early. In the event of a nasty bug appearing in a stable update, phasing can be halted and the issue investigated before either rolling back the update, or deciding it’s a false-positive and resuming the phasing.

Incidentally, the stable release updates report, and the phasing report are both public if you want to peak at what’s in the pipeline of your stable install! You can read more about phasing in the server documentation.

Noble

Okay, back to the story. Mesa was up to date in oracular, but we needed an SRU to noble to avoid new Pi 5 owners getting a full blown crash the second anything graphical like a login screen dared show its face on the monitor.

Time for a noble SRU. The SRU was written (LP: #2082072), the update sponsored, verification performed, phasing completed, and by the middle of October everything was fixed.

mesa (24.0.9-0ubuntu0.2) noble; urgency=medium

  * Add support for Pi 2712D0 stepping (LP: #2082072)

…

I said, everything was fixed!

…

Right?

…

Oh, ****!

Snaps

One thing I’d forgotten about in this whole process was that there’s no longer just one version of mesa on most Ubuntu desktop systems. Nowadays, a fair number of applications, including some vaguely important ones like Firefox, are shipped as snaps, at least partly in order to alleviate the aforementioned tension between stable distros and desktop users demanding the latest and greatest versions.

There are many ways to handle distributing the “latest” versions of things on a stable distro, but one of the things that must be dealt with is the conflict between the versions of libraries on the stable distro, and the libraries that the “latest” thing expects.

One method is to simply “bundle everything”; include all the libraries that your app expects within its package. Obviously this results in a rather bloated package which isn’t going to share any libraries with anything else on the system, and thus takes more time to load and more memory to run. It also means dealing with all the path nonsense to ensure your application only loads the versions of libraries from its own location. This doesn’t solve all the bundling issues, but it does solve most and it is at least simple.

Another method is “statically compile everything”. This is a close cousin to bundling everything, and still results in a fairly huge download but a little slimmer than bundling all the libraries (and still shares basically nothing at runtime). This is becoming more common with the rise of things like go and rust, which compile this way by default [3].

Snaps take a rather interesting (and different) approach here.

All snaps are built with a “base” snap which represents a stable release of Ubuntu (core22 represents a jammy base, core24 a noble base, and so on). The “base” snap provides the “common” libraries that the snap would expect to find on such a stable release of Ubuntu.

The snap can still statically compile, or bundle, whatever it likes, but this means at least some of its dependencies can be shared (from the base snap). Moreover, the method by which snaps are mounted means that multiple versions of these core snaps can co-exist so it’s fine to be running a snap based on core22, and another based on core24 (obviously they won’t be sharing their core libraries in this case, but that’s the trade-off).

The libraries the core snaps include are fairly limited and, notably, don’t include mesa. However, there are also “content” snaps that provide commonly used sets of libraries above and beyond the core snaps. One of these is the “gnome” content snap which provides (surprise surprise), Gnome, GTK, and, you guessed it, mesa. The gnome content snap itself comes in several versions to accomodate different “base” snaps:

• gnome-42-2204 is the gnome content snap using core22 (jammy) as a base
• gnome-46-2404 is the gnome content snap using core24 (noble) as a base

Now… guess which gnome content snap Firefox is (still, at the time of writing) using?

Jammy

Shortly after the middle of October, and the noble SRU landing, I noticed we had another problem. The noble daily would happily boot to the login prompt on the new D0 Pi 5, and I could login and run many things. But not Firefox. Starting that, or anything else on the core22 snap, froze the entire desktop. Not good.

So, back to the SRU queue. The original ticket (LP: #2082072) was re-targeted for jammy, the upload was made, and a bunch of other packages regressed. Damn! Various retries later and the package was ready for verification.

But… now we had a problem. This was only an update to the mesa deb package, in Ubuntu jammy (22.04). This wasn’t an update to the gnome content snap; that would have to be done after verification of the SRU. But the Pi 5 was fundamentally not supported in jammy.

Rather than trying to get a jammy image into a state it could boot (and run Gnome) on a Pi 5, I opted to rebuild the gnome content snap locally to verify things instead. This took a few days to work out as it’s an almost entirely undocumented process and I had to figure out that the gnome-sdk snap needed rebuilding first (i.e. the chain goes: mesa deb -> gnome-sdk snap -> gnome snap).

To cut a long story short (or at least… less long), eventually the verification was done, and the jammy mesa update was released.

mesa (23.2.1-1ubuntu3.1~22.04.3) jammy; urgency=medium

  [ Timo Aaltonen ]
  * Add support for Pi 2712D0 stepping (LP: #2082072)

  [ Alessandro Astone ]
  * patches: Backport patch for green artifacting and GPU crash on
    radeonsi with kernel >= 6.10 (LP: #2083538)

Phasing was then completed, then the gnome-sdk snap was rebuilt, then gnome content snap, and finally Firefox was working on the D0 stepping.

By the time all this was done, it was the 31st of Jan. To give some context, the .2 release of noble was scheduled for the 13th of Feb, and I was adamant that we had to get the mesa fix in for that otherwise we’d have new Pi 5 owners unable to run Firefox “out of the box” on our current LTS. You could argue: “users can still login, they can always get the release from updates?”. However, the app-store is also a core22 snap, and hence also crashed the desktop. While there was still the sudo snap refresh route, I’m loathe to tell desktop users they have to use the command line.

Suffice to say, it was butt-clenching time in the run up to noble’s .2 release.