Boot configuration with pibootctl

On What Dave Should’ve Known Sooner

When it started out, I had a notion of pibootctl’s manipulation logic that was based on certain assumptions, which in turn were based on how I’d structured the boot configuration for Ubuntu.

On Raspbian (oops, that’s now Raspberry Pi OS - funny how so much can change in a few months), the configuration structure is simple: everything goes in “config.txt”.

On Ubuntu, I’d altered the configuration structure to be rather more complex: everything starts in “config.txt” (as usual), then continues in the included “syscfg.txt”, and finishes in the included “usercfg.txt”. The intention was as follows:

• “config.txt” was to be under the control of the package manager. Its one and only job would be to get the Linux kernel going and then include the other files. The idea was we could clobber that from the package manager with impunity, not worrying about arbitrary modifications to it.
• “syscfg.txt” was to be under the control of pibootctl (and thus anything built on top of it). The pibootctl utility would be free to rewrite the whole file according to whatever the user requested.
• Finally “usercfg.txt” was the “free for all” file in which we migrated existing settings from config.txt in the transition to the split. Users were free to do whatever they wanted in this file, with the understanding “you break it, you get to keep the pieces”.

Well, that’s all fine and dandy but for one little problem: some boot configuration settings only work in “config.txt” and get ignored if they’re in included files. In my defense, while this is now documented in the excellent configuration documentation it wasn’t at the time I started writing it (yes, I tracked down that commit after wondering “how the hell did I miss this?!”).

However, the case for the prosecution is: most of the directives affected (e.g. start_x, start_debug, gpu_mem) are pretty obvious if you understand the separation between “bootcode.bin” (or the EEPROM on the Pi 4), and “start*.elf” … which I absolutely did. What I’d missed was that, while many of the conditional sections are supported by both “bootcode.bin” and “start*.elf”, only the latter implements the “include” command.

Anyway, this meant that my oh-so-carefully separated boot config was a bust. The “config.txt” file must be modifiable by pibootctl (or anything else), otherwise certain rather important settings (e.g. enabling the camera module) just aren’t possible. This in turn defied several of the assumptions baked into pibootctl’s manipulation of the boot configuration:

• I had assumed that pibootctl could re-write a file (e.g. syscfg.txt) completely and thus shouldn’t need to concern itself with, for example, preserving commentary in that file. This is obviously not the case if it needs to manipulate “config.txt” too.
• I had assumed there needn’t be any provision for commenting / uncommenting lines of configuration. We could just wipe that file and start fresh. However, in “config.txt” there are traditionally comments preceding various lines (esp. on Raspbian) and those comments should remain with their corresponding configuration lines (á la raspi-config).
• I had assumed that the included file we’d be re-writing could reasonably be assumed to be in an “[all]” context so we wouldn’t need to worry about the context of removed / commented lines, or the context of added lines. But, if you’re messing with “config.txt” that also goes out of the window. Both Raspbian and Ubuntu use conditional sections in both of their default configurations.

All the assumptions above were now broken and needed to be worked around. The tool needed to understand the conditional context applicable to each line, the conditional context applicable to newly generated lines, and the set-based relationship between conditional contexts. For example, given the following configuration:

[pi2]
kernel=uboot_rpi_2.bin

[pi3]
kernel=uboot_rpi_3.bin

[pi4]
kernel=uboot_rpi_4.bin

If I want to set the kernel’s filename to “vmlinuz” (bypassing u-boot), on all models, the tool needs to know that the conditional context of each “kernel” line above is a subset of the requested conditional context (“[all]”), and thus that all the existing “kernel” lines should be removed / commented; that a new “[all]” section needs to be added at the end and the new line placed in that section.

Alternatively, if we want to make the same manipulation, but only for the current model (let’s assume that’s a Pi 3), the tool needs to know that the “[pi2]” section is a disjoint set with “[pi3]” so the first kernel line needs to be left alone; the second is equal so that kernel line can be removed / commented then replaced, and the final “[pi4]” section is also a disjoint set with the requested context.

Anyway to cut an already way too long interlude short … that’s what the tool now does. It’s fully aware of the conditional sections, how they relate, which commands are only recognized in “config.txt” (and therefore should be ignored for effect in other contexts), how to generate new conditional sections to transform one conditional context into another, how to comment out lines (optionally), and how to search for and uncomment lines in relevant contexts.

So, on with the show …

Stored Configurations

One major feature of pibootctl is that it’s capable of backing up and restoring boot configurations. It’s relatively important to understand how it does this given that, as a relatively low level tool designed to mess with your boot configuration, it’s entirely possible to use it to break your boot configuration!

The Pi’s boot partition is a simple FAT partition at the start of the image, and the boot configuration is (mostly) held in a series of straight-forward text files. The great benefit of this is universal support: FAT is the “universal file-system”; everything knows how to read and write it from Windows to Mac to Linux, and even numerous embedded systems. Likewise, text files are the “universal format”; every operating system has some facility for editing text files (which is the reason it was selected as the original format for the throughly worthy Project Gutenberg).

Ultimately this means that, should the worst happen, and you screw up your boot configuration, you can stick your SD card in another machine and edit it back to a workable state. However, it might be easier if you know what a workable state is!

pibootctl strives to be equally accessible in its backup strategy: boot configurations are stored as ZIP archives of the original (mostly) text files, on the FAT partition (in a sub-directory). ZIP is another pretty-much-universal format (Windows, Mac OS, and most Linux distros have built-in support for opening ZIP files), and by storing the backups on the FAT partition we ensure that they remain accessible on all platforms too.

Assuming you’ve installed pibootctl on Ubuntu from the PPA. you can create a backup of your current boot configuration using pibootctl like so:

$ sudo pibootctl save original

Why is sudo required? Simply because we’re writing that backup to the boot partition which only root has access to. After this you should be able to list the available stored configurations like so:

$ pibootctl ls
┌──────────┬────────┬─────────────────────┐
│ Name     │ Active │ Timestamp           │
├──────────┼────────┼─────────────────────┤
│ original │ ✓      │ 2020-09-15 12:27:02 │
└──────────┴────────┴─────────────────────┘

Note that sudo isn’t required here as we’re not writing anything to the boot partition, just reading things. Also note that pibootctl has compared the current boot configuration to the stored “original” configuration and noticed they’re exactly the same, hence the tick-mark in the “Active” column. Where exactly is this stored?

In the pibootctl package in my PPA, pibootctl is configured to create a “store” directory on the boot partition which will contain the stored configurations:

$ ls /boot/firmware/store
original.zip

You can install an appropriate archiving tool to take a look in this archive if you want. Personally, I favour “atool” which provides generic commands for working with a variety of archives (tar.gz, tar.xz, zip, etc. etc.):

$ sudo apt install -y atool
$ als /boot/firmware/store/original.zip
Archive:  /boot/firmware/store/nouboot.zip
pibootctl:0:e5bf92a57556aa24d4dd8c94ba63a755a21fdcc3

Do not edit the content of this archive; the line above is a hash of the
content which will not match after manual editing. Please use the pibootctl
tool to manipulate stored boot configurations
  Length      Date    Time    Name
---------  ---------- -----   ----
     1116  2020-09-15 12:27   config.txt
      327  2020-09-01 07:02   syscfg.txt
      200  2020-09-01 07:02   usercfg.txt
      127  2020-09-01 12:27   cmdline.txt
---------                     -------
     1770                     4 files

Several points of interest here:

• There’s a strange “pibootctl:0:xxxxxxx” line at the top of the ZIP archive’s comment. This is a hash of the boot configuration in parsed order, and pibootctl uses it to quickly determine which stored configurations match the current boot configuration.
• For this reason, there’s a comment straight after the hash line warning users not to modify the content of the archive, or the hash will no longer match. At the moment, the tool doesn’t use the hash to validate the archive contents as I’m on the fence as to whether permitting users to create their own archives manually is a good or bad idea but for now the warning stands.
• After this you can see the various files that make up the boot configuration: config.txt, syscfg.txt, usercfg.txt, and cmdline.txt (which is referenced from config.txt). Note that other files referenced by the boot configuration like the kernel (vmlinuz) and the initrd (initrd.img) are not included in the archive. The assumption is that these things are under the control of the package manager and shouldn’t be restored with the boot configuration itself.

Manipulation

Now let’s edit our boot configuration using pibootctl. We’ll enable the camera module and set GPU memory to 128Mb:

$ sudo pibootctl set camera.enabled=on gpu.mem=128

Note once again we use “sudo”. Because we’re changing files on the boot partition we need root privileges. We can check the “status” output to see these are definitely now set:

$ pibootctl status
┌─────────────────────────┬──────────────────────┐
│ Name                    │ Value                │
├─────────────────────────┼──────────────────────┤
│ audio.enabled           │ on                   │
│ boot.devicetree.address │ 50331648 (0x3000000) │
│ boot.initramfs.filename │ ['initrd.img']       │
│ boot.kernel.64bit       │ on                   │
│ boot.kernel.cmdline     │ 'cmdline.txt'        │
│ boot.kernel.filename    │ 'vmlinuz'            │
│ camera.enabled          │ on                   │
│ gpu.mem                 │ 128 (Mb)             │
│ i2c.enabled             │ on                   │
│ serial.enabled          │ on                   │
│ spi.enabled             │ on                   │
│ video.framebuffer.max   │ 2                    │
│ video.overscan.enabled  │ off                  │
└─────────────────────────┴──────────────────────┘

In fact, the tool ensures that the written configuration matches the desired configuration as part of its generation logic. Consider the following configuration:

1 [pi4]
2 max_framebuffers=2
3 
4 [all]
5 arm_64bit=1
6 cmdline=cmdline.txt
7 
8 include usercfg.txt

1 dtoverlay=disable-bt

In this case “usercfg.txt”, a file that pibootctl is not permitted to edit, disables the bluetooth module. What happens if we explicitly try to enable it?

$ sudo pibootctl set --no-backup bluetooth.enabled=on
Failed to set 3 setting(s)
Expected bluetooth.enabled to be True, but was False after being overridden
by usercfg.txt line 1
Expected serial.uart to be 1, but was 0 with no valid lines; this usually
means a setting like start_x or gpu_mem is in a file other than config.txt
Expected serial.enabled to be False, but was True with no valid lines; this
usually means a setting like start_x or gpu_mem is in a file other than
config.txt

pibootctl has noticed that it failed to set bluetooth.enabled, and warns the user that the setting is “overridden by usercfg.txt line 1” (the other errors come from the fact that disabling the bluetooth module implicitly enables the serial port and places it on the PL011 UART; these also don’t match pibootctl’s expectations for the resulting configuration and so they get reported too).

Restoration

How do we restore a saved configuration using pibootctl? The opposite of the “save” command is “load”:

$ sudo pibootctl load nouboot
Backed up current configuration in backup-20200915-133659

Once again, we need “sudo” as we’re manipulating the content of the boot partition. One additional thing to note above is that pibootctl automatically saved a copy of the current boot configuration. It did this because, before modifying the current configuration, it checked a copy of it existed in the stored configurations. In this case no copy existed (since we changed it above), so it automatically saved one for us.

You can suppress this behaviour with the --no-backup switch, or permanently by modifying the tool’s configuration in “/etc/pibootctl.conf”.

Don’t Panic!

Let’s assume the worst has happened and you’ve broken your boot configuration. However, you’ve got a stored configuration you know is good in one of these ZIP archives. How do you restore it if you can no longer boot your Pi and access pibootctl?

Simply place the SD card in another machine, find the ZIP archive storing the boot configuration you want (which will be under the “store” directory we explored above), and extract it over the top of current boot configuration. You’ll almost certainly be prompted to replace the existing files (config.txt, etc.) but that’s fine: permit the replacement and you’ve restored your boot configuration! It really is that simple.

Addendum: Why ZIPs?

I did debate just using sub-directories instead of ZIP archives and storing configurations “raw” so users could simply copy them back without bothering with unzip tools. This is certainly an attractive method, but:

• It makes it much easier for users to manipulate stored configurations in place. As mentioned above, I’m not (yet) convinced this is a good or bad idea, so for now I’ve opted for the method which makes it harder.
• It makes it easy to store meta-data alongside the stored configuration (the aforementioned hash). We could do this in an auxilliary file in the stored configuration but then we must be sure that could never exist in a legitimate stored config. Or it could be in a file above the directory storing the configuration, but then that has to be kept in sync with the stored directories (which, as mentioned above, are easy to manipulate in their own right).

Next Time

That’s quite enough on this subject. Next time (hopefully less than months away!) we’ll take a look at the “diff” command, and how to integrate pibootctl with more friendly tools on top of it.