Well, dear reader, another couple of years have passed and it’s time for another LTS release. These are always somewhat nerve-wracking exercises given they’re the choice that something like 99% of our users stick to, so we really want things to go smooth.
“It never goes smooth”
Leaving aside fresh installs, which seem to be “okay” at the moment, upgrades are the subject of this story. At the time of writing, there is one supported upgrade path and a couple of definitely not supported ones.
The supported path is to use ubuntu-release-upgrader by running sudo do-release-upgrade. This gates upgrades on a number of things:
- Firstly, it will only offer those upgrades configured by /etc/update-manager/release-upgrades. If this contains Prompt=normal then upgrades for all releases, including interims like questing (25.10) will be considered. But for most people this contains Prompt=lts, in which case only upgrades from LTS to LTS will be considered.
- For LTS to LTS upgrades, these are only offered once the “point one” release comes out, not at initial release. In the case of resolute (26.04), this means upgrades for LTS users will only be officially switched on after August 27th (the current planned release date for 26.04.1).
- Secondly, do-release-upgrade gates on all updates for the current release being applied; it won’t proceed until all are installed and the latest kernel for that release is active (i.e. if you have a pending kernel upgrade you’ll need to reboot before it will proceed).
- Thirdly, it will not permit releases to be “skipped” during an upgrade. For example, if Prompt=normal and your current install is jammy (22.04), it will ensure you upgrade to noble (24.04) first, then to questing (25.10), and only then to resolute (26.04). You cannot skip straight from jammy to resolute.
That’s the supported path… What about the unsupported routes?
- First, there’s the --devel option to do-release-upgrade. This is intended for testing upgrades to the “development” release (currently stonking, 26.10) prior to its release. However, if Prompt=lts on your system, then resolute appears to be the development release (from the point of view of the upgrader), because it’s the next “unreleased” LTS (unreleased because it’s prior to the “point one” release).
- Second, there’s the good ol’ dist-upgrade route: re-write all your apt sources to point to the new release (e.g. sed -i -e 's/noble/resolute/g' /etc/apt/sources.list.d/*.sources) then run sudo apt update; sudo apt dist-upgrade, and hope for the best!
I’ll stress these are unsupported routes and you shouldn’t use them. Of course, that’s not stopping anyone… [1]
Unfortunately, due to some of the changes that I introduced in resolute (and one big mistake), if you try the unsupported route on a Raspberry Pi you’re very likely to run into some trouble. This post explains some of the issues you’re going to run into [3] and how to avoid them [4].
“She makes things not be smooth”
There are four problems (at least) occurring more or less simultaneously at the moment.
Out of date EEPROM
You’re on a Pi 4 or a Pi 5, and you didn’t read the release notes before dist-upgrading [5], particularly the part about having an up to date EEPROM (specifically one later than 2022-11-25 on the Pi 4, or 2025-02-11 on the Pi 5). Now you’ve rebooted and found the bootloader can’t (or won’t) load a Linux kernel at version 7.
This is partly my fault for not getting the updated EEPROM into noble before resolute’s release. In mitigation, it is in questing, so if you were following the official upgrade route, you wouldn’t run into this. The updated EEPROM package has now been back-ported to noble, and the official release upgrader was bumped [7] to check the EEPROM level and refuse to proceed unless it’s up to date.
Still, if you’ve dist-upgraded and now your card won’t boot at all (because you’ve got this problem and the next one), flash another card with some reasonably up to date Ubuntu (anything back to noble should now work), or RaspiOS. Boot it, run through the typical upgrades, then sudo rpi-eeprom-update -a and reboot. Afterwards, re-run sudo rpi-eeprom-update to verify the EEPROM is now up to date, and you should find your Ubuntu install works too.
Half-complete dist-upgrade
LP: #2151216 (flash-kernel task)
Your dist-upgrade didn’t quite upgrade everything. In particular, it didn’t upgrade the meta-package (ubuntu-server-raspi or ubuntu-desktop-raspi depending), but did upgrade the flash-kernel package. Unfortunately, the flash-kernel package in resolute doesn’t know anything about the Pi and now apt is crashing out every time something tries to run flash-kernel.
If you’re in this situation you’ll get an error every time apt tries to call flash-kernel because the version in resolute knows nothing about the Pi (specifically the error will be something like Unsupported platform 'Raspberry Pi 4 Model B Rev 1.2', substitute your model of Pi here).
This also doesn’t occur via the official route because ubuntu-release-upgrader ensures that meta-packages are in the upgrade set but ultimately this is my fault for forgetting a “Breaks” clause in flash-kernel. That’s currently being dealt with (but is currently waiting on another SRU to land before I upload this fix).
The workaround here is to sudo apt install piboot-try, and accept that it will remove flash-kernel (in this case the piboot-try package will provide its own stripped down version of flash-kernel that only knows about the Pi). After this you may need to sudo apt -f install to “fix” any pending kernel updates.
Failure to upgrade with armhf as foreign arch
This one is a consequence of the move of the arm64 packages from ports.ubuntu.com to archive.ubuntu.com. The default source configuration for apt in resolute looks something like this:
Types: deb deb-src
URIs: http://archive.ubuntu.com/ubuntu
Suites: resolute resolute-updates resolute-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Types: deb deb-src
URIs: http://security.ubuntu.com/ubuntu
Suites: resolute-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Things to note:
- Whereas previously the URI would be http://ports.ubuntu.com/ubuntu-ports it’s now http://archive.ubuntu.com/ubuntu
- There’s no Architecture limit on these stanzas; they’ll attempt to pick up whatever architectures dpkg is configured to use
If you’re on noble, and you’ve added armhf as a foreign architecture using sudo dpkg --add-architecture armhf, which is a supported configuration, you’ll now encounter an error if you try and upgrade to resolute. Actually, you’ll encounter a similar problem if you’re on a fresh resolute install and you try and add armhf as an architecture. The architecture is registered with dpkg, but now apt will try and download armhf package information from http://archive.ubuntu.com. Unfortunately, while arm64 packages moved there, armhf ones did not; they’re still on http://ports.ubuntu.com. So apt reports an error and (in the upgrade case), ubuntu-release-upgrader simply quits.
In the dist-upgrade case, I’m not sure what happens (haven’t tested this yet because that’s not a supported route anyway — any fix will be going into ubuntu-release-upgrader).
The workaround for now (and what will probably be automatically done by the release upgrader in the future) is to split the apt configuration. One file (presumably the default configuration in ubuntu.sources) contains a limit for the primary architecture (arm64):
Types: deb deb-src
Architectures: arm64
URIs: http://archive.ubuntu.com/ubuntu
Suites: resolute resolute-updates resolute-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Types: deb deb-src
Architectures: arm64
URIs: http://security.ubuntu.com/ubuntu
Suites: resolute-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
And a separate one is made for armhf, pointing at the older address:
Types: deb deb-src
Architectures: armhf
URIs: http://ports.ubuntu.com/ubuntu-ports
Suites: resolute resolute-updates resolute-backports
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Types: deb deb-src
Architectures: armhf
URIs: http://security.ubuntu.com/ubuntu
Suites: resolute-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Failure to boot latest kernel on older models
This is the issue that’s particularly concerning me at the moment, and mostly because I’ve been unable to reliably reproduce it thus far. Several reports (more than just the two bugs listed) have come in reporting that piboot-try has fallen back and marked new boot assets “bad”. The preponderance of these reports concern kernel 7.0 revision 1011, but I’m still trying to corroborate this is the specific revision concerned in a couple of cases.
On the plus side: nobody has complained of a non-booting system, so at least we can say “piboot-try is doing its job”. On the downside, everyone’s very confused that they get a report complaining about bad boot assets on every single login.
For the avoidance of doubt: this is intended, and not harmful. The piboot-try implementation tries to be cautious in its implementation: it warns on every login because firstly we can’t assume you read all the output on every login [8], and secondly having a login hook that writes as well as reads (to record that a warning has been presented) is generally a bad idea (assumes every login user has a writable home directory, assumes there will be free space to perform a write, etc).
Unfortunately at the moment, there is no piboot-try --yes-i-know-stfu option to silence the warning. I’ll stick it on my TODO list for the future, but I’m afraid it won’t be a terribly high priority.
Still, at the moment the focus is coming up with a reliable reproducer for this, and/or narrowing down all reports to a single kernel revision. After that, it’ll be on figuring out which component is actually responsible (my gut feeling at the moment is this may be a kernel issue).
“Why does it never go smooth?”
I was planning a longer post on this, but for now this will have to do. If there’s one thing I’ve gleaned from all this though: the documentation on piboot-try, and the login messages it produces clearly need some revision.
It may be more useful to point people at an expanded man-page rather than encourage everyone to go filing bugs (also I need to update the message to point to the piboot-try package; flash-kernel is strictly wrong in resolute now it’s been split out). Also, the various switches like --reset-new appear to have a lot of people confused. I should include some worked examples in the man-page illustrating the scenarios each switch is intended for use under.
| [1] | Yes, I am also guilty of using these (in fact it’s what I tend to do at some point in a cycle when it’s time to start testing the new release). But I know what I’m doing. [2] |
| [2] | Most of the time. Okay, some of the time. Well, now and then, I guess. |
| [3] | Or have already |
| [4] | Or fix your system now it’s utterly borked |
| [5] | Tut tut! Repeat after me: the release notes no longer suck [6]; I shall read the release notes before upgrading in future |
| [6] | No, really! Instead of a massive, unnavigable, unmanageable post in Discourse’s horrible interface, they’re now a Sphinx document with a proper multi-level table of contents… and search! The organisation is a bit messy, but it’s still a massive improvement |
| [7] | Actually it was bumped a couple of times. The Pi 4 case was done before release, but the Pi 5 case was just after because they have different root causes. On the Pi 4 the recent EEPROM is required to run the A/B boot system (so we picked that up back in questing). The Pi 5’s EEPROMs are all capable of A/B boot out of the box, but on that platform the earlier EEPROMs won’t boot the 7.0 kernel. Unfortunately, that kernel landed quite late in development so we didn’t notice this issue until just after release. |
| [8] | I certainly don’t! |