It's Pi all the way down... - pibootctlhttps://waldorf.waveform.org.uk/2023-07-04T09:26:38+01:00Groovy Boot Modes2020-10-21T00:00:00+01:002023-07-04T09:26:38+01:00Dave Jonestag:waldorf.waveform.org.uk,2020-10-21:/2020/groovy-boot-modes.html<p class="first last">Adding or removing U-Boot in Ubuntu Groovy</p>
<p>By default, Ubuntu Groovy Pi images (both Server and Desktop) will ship with a
boot sequence that involves the Pi’s native bootloader <em>only</em>. In other words,
the new boot sequence is simply:</p>
<pre class="code literal-block">
Pi Bootloader --> Linux Kernel --> Initrd
</pre>
<p>Previous versions of Ubuntu on the Pi have included U-Boot in the sequence.
In other words, in Focal (and prior) versions, the boot sequence was:</p>
<pre class="code literal-block">
Pi Bootloader --> U-Boot --> Linux Kernel --> Initrd
</pre>
<p>Users <em>upgrading</em> to Groovy will continue using the legacy U-Boot sequence but
are free to switch to the new “native only” boot sequence by following the
instructions below. Conversely, new users of Groovy may also switch to a U-Boot
based sequence if they so wish (the U-Boot binaries will still be installed on
the boot partition even when they’re not used to enable switching between the
two choices).</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p>Be aware that the U-Boot based sequence does <em>not</em> support native <span class="caps">USB</span> boot,
or the Pi’s native netboot (U-Boot on the Pi 3 and earlier does have its
own <span class="caps">PXE</span>-like netboot system, however our build of U-Boot currently lacks
ethernet drivers for the Pi 4).</p>
<p class="last">Also note that the U-Boot based sequence is now deprecated. If you are
still relying on U-Boot for some reason, please let me know what it is
(open a ticket against <a class="reference external" href="https://launchpad.net/ubuntu/+source/u-boot">u-boot on Launchpad</a>, leave a comment below, or
pester me on <a class="reference external" href="https://twitter.com/waveform80/">Twitter</a>)!</p>
</div>
<div class="section" id="switching-from-u-boot-to-native">
<h2>Switching from U-Boot to Native</h2>
<p>The only file that needs editing is <tt class="docutils literal">/boot/firmware/config.txt</tt>. There
may be warnings at the top of the warning you not to edit it. You can ignore
these. I know this, because I wrote them (you can read the sordid tale of those
warnings and why they’re now redundant in <a class="reference external" href="https://waldorf.waveform.org.uk/2020/boot-configuration-with-pibootctl.html">this post</a>).</p>
<p>Remove any existing lines that start with <tt class="docutils literal">kernel=</tt>, <tt class="docutils literal">initramfs</tt>, or
<tt class="docutils literal">devicetree_address=</tt> from this file. The add the following lines under the
<code class="ini"><span class="k">[all]</span></code> section:</p>
<pre class="code literal-block">
kernel=vmlinuz
initramfs initrd.img followkernel
</pre>
<p>That’s all there is to it! If you want to try this from <a class="reference external" href="https://pibootctl.readthedocs.io/">pibootctl</a> (with some
extra lines to save your current boot configuration so you can switch back to
it if you wish), the following commands should do the trick:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo pibootctl save uboot
<span class="gp">$ </span>sudo pibootctl <span class="nb">set</span> boot.kernel.filename<span class="o">=</span>vmlinuz <span class="se">\
</span> boot.initramfs.filename<span class="o">=</span>initrd.img boot.devicetree.address<span class="o">=</span>
<span class="gp">$ </span>sudo pibootctl save native
</pre>
<p>Now you can simply do the following to switch back to uboot if you wish:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo pibootctl load uboot
</pre>
</div>
<div class="section" id="switching-from-native-to-u-boot">
<h2>Switching from Native to U-Boot</h2>
<p>Switching back to U-Boot from the native boot sequence is a little more complex
but not much so. Again, in <tt class="docutils literal">/boot/firmware/config.txt</tt> remove all lines
that start with <tt class="docutils literal">kernel=</tt> or <tt class="docutils literal">initramfs</tt>. Next, determine whether you’re
using a 32-bit or 64-bit:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>dpkg --print-architecture
<span class="go">arm64</span>
</pre>
<p>If you’re on arm64 (64-bit), you want a <tt class="docutils literal">config.txt</tt> that includes the
following (by “includes” I mean don’t replace your whole configuration with
this; either append this to your current configuration or merge it into the
existing sections - either should be fine):</p>
<pre class="code ini literal-block">
<span class="k">[pi4]</span><span class="w">
</span><span class="na">kernel</span><span class="o">=</span><span class="s">uboot_rpi_4.bin</span><span class="w">
</span><span class="na">max_framebuffers</span><span class="o">=</span><span class="s">2</span><span class="w">
</span><span class="k">[pi2]</span><span class="w">
</span><span class="na">kernel</span><span class="o">=</span><span class="s">uboot_rpi_2.bin</span><span class="w">
</span><span class="k">[pi3]</span><span class="w">
</span><span class="na">kernel</span><span class="o">=</span><span class="s">uboot_rpi_3.bin</span><span class="w">
</span><span class="k">[all]</span><span class="w">
</span><span class="na">arm_64bit</span><span class="o">=</span><span class="s">1</span><span class="w">
</span><span class="na">device_tree_address</span><span class="o">=</span><span class="s">0x03000000</span>
</pre>
<p>If you’re on armhf (32-bit), you want a <tt class="docutils literal">config.txt</tt> that includes the following:</p>
<pre class="code ini literal-block">
<span class="k">[pi4]</span><span class="w">
</span><span class="na">kernel</span><span class="o">=</span><span class="s">uboot_rpi_4_32b.bin</span><span class="w">
</span><span class="na">max_framebuffers</span><span class="o">=</span><span class="s">2</span><span class="w">
</span><span class="k">[pi2]</span><span class="w">
</span><span class="na">kernel</span><span class="o">=</span><span class="s">uboot_rpi_2.bin</span><span class="w">
</span><span class="k">[pi3]</span><span class="w">
</span><span class="na">kernel</span><span class="o">=</span><span class="s">uboot_rpi_3_32b.bin</span><span class="w">
</span><span class="k">[all]</span><span class="w">
</span><span class="na">device_tree_address</span><span class="o">=</span><span class="s">0x03000000</span>
</pre>
<p>(if you’re curious about the strange ordering of the sections, this isn’t
strictly required on Groovy but is just there for historical reasons)</p>
<p>You can try this doing this via <a class="reference external" href="https://pibootctl.readthedocs.io/">pibootctl</a>, but it currently only has options
for modifying boot configuration sections relevant to the machine you’re
running on (e.g. if you’re on a Pi 4, it can modify the <code class="ini"><span class="k">[pi4]</span></code> and
<code class="ini"><span class="k">[all]</span></code> sections but won’t modify <code class="ini"><span class="k">[pi2]</span></code> or <code class="ini"><span class="k">[pi3]</span></code>
sections). So, if you’re on a Pi 4 running an arm64 kernel, you could do the
following (with some extra lines to save your current boot configuration so you
can switch back to it if you wish):</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo pibootctl save native
<span class="gp">$ </span>sudo pibootctl <span class="nb">set</span> --this-model boot.kernel.filename<span class="o">=</span>uboot_rpi_4.bin
<span class="gp">$ </span>sudo pibootctl <span class="nb">set</span> --no-backup boot.initramfs.filename<span class="o">=</span> <span class="se">\
</span> boot.devicetree.address<span class="o">=</span>0x3000000
<span class="gp">$ </span>sudo pibootctl save uboot
</pre>
<p>Now you can simply do the following to switch back to uboot if you wish:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo pibootctl load native
</pre>
</div>
Boot configuration with pibootctl2020-09-16T00:00:00+01:002023-06-30T01:02:44+01:00Dave Jonestag:waldorf.waveform.org.uk,2020-09-16:/2020/boot-configuration-with-pibootctl.html<p class="first last">Changing stuff with pibootctl</p>
<p>In the <a class="reference external" href="https://waldorf.waveform.org.uk/2020/using-pibootctl.html">previous post</a>, we looked at reading the boot status with pibootctl
(in several formats). This time we’ll look at how we can use pibootctl to
change the boot configuration, and the side-effects of doing so.</p>
<p>First though, a brief interlude on why this post took a lot longer than
intended (feel free to skip this section if you’re only interested in how to
set stuff with pibootctl; it’s mostly self-flagellation anyway! :).</p>
<div class="section" id="on-what-dave-should-ve-known-sooner">
<h2>On What Dave Should’ve Known Sooner</h2>
<p>When it started out, I had a notion of pibootctl’s manipulation logic that was
based on certain assumptions, which in turn were based on how I’d structured
the boot configuration for Ubuntu.</p>
<p>On Raspbian (oops, that’s now Raspberry Pi <span class="caps">OS</span> - funny how so much can change in
a few months), the configuration structure is simple: everything goes in “config.txt”.</p>
<p>On Ubuntu, I’d altered the configuration structure to be rather more complex:
everything starts in “config.txt” (as usual), then continues in the included
“syscfg.txt”, and finishes in the included “usercfg.txt”. The intention was as follows:</p>
<ul class="simple">
<li><span class="dquo">“</span>config.txt” was to be under the control of the package manager. Its one and
only job would be to get the Linux kernel going and then include the other
files. The idea was we could clobber that from the package manager with
impunity, not worrying about arbitrary modifications to it.</li>
<li><span class="dquo">“</span>syscfg.txt” was to be under the control of pibootctl (and thus anything
built on top of it). The pibootctl utility would be free to rewrite the whole
file according to whatever the user requested.</li>
<li>Finally “usercfg.txt” was the “free for all” file in which we migrated
existing settings from config.txt in the transition to the split. Users were
free to do whatever they wanted in this file, with the understanding “you
break it, you get to keep the pieces”.</li>
</ul>
<p>Well, that’s all fine and dandy but for one little problem: some boot
configuration settings only work in “config.txt” and get ignored if they’re in
included files. In my defense, while this is now documented in the excellent
<a class="reference external" href="https://www.raspberrypi.org/documentation/configuration/config-txt/misc.md">configuration documentation</a> it wasn’t <a class="reference external" href="https://github.com/raspberrypi/documentation/commit/b385eec9911ec0aab697c23997406d50ddca0625">at the time</a> I started writing it
(yes, I tracked down that commit after wondering “how the hell did I miss this?!”).</p>
<p>However, the case for the prosecution is: most of the directives affected (e.g.
start_x, start_debug, gpu_mem) are pretty obvious if you understand the
separation between “bootcode.bin” (or the <span class="caps">EEPROM</span> on the Pi 4), and “start*.elf”
… which I absolutely did. What I’d missed was that, while many of the
conditional sections are supported by both “bootcode.bin” and “start*.elf”,
only the latter implements the “include” command.</p>
<p>Anyway, this meant that my oh-so-carefully separated boot config was a bust.
The “config.txt” file <em>must</em> be modifiable by pibootctl (or anything else),
otherwise certain rather important settings (e.g. enabling the camera module)
just aren’t possible. This in turn defied several of the assumptions baked into
pibootctl’s manipulation of the boot configuration:</p>
<ul class="simple">
<li>I had assumed that pibootctl could re-write a file (e.g. syscfg.txt)
completely and thus shouldn’t need to concern itself with, for example,
preserving commentary in that file. This is obviously not the case if it
needs to manipulate “config.txt” too.</li>
<li>I had assumed there needn’t be any provision for commenting / uncommenting
lines of configuration. We could just wipe that file and start fresh.
However, in “config.txt” there are traditionally comments preceding various
lines (esp. on Raspbian) and those comments should remain with their
corresponding configuration lines (á la <tt class="docutils literal"><span class="pre">raspi-config</span></tt>).</li>
<li>I had assumed that the included file we’d be re-writing could reasonably be
assumed to be in an “[all]” context so we wouldn’t need to worry about the
context of removed / commented lines, or the context of added lines. But, if
you’re messing with “config.txt” that also goes out of the window. Both
Raspbian and Ubuntu use conditional sections in both of their default configurations.</li>
</ul>
<p>All the assumptions above were now broken and needed to be worked around. The
tool needed to understand the conditional context applicable to each line, the
conditional context applicable to newly generated lines, and the set-based
relationship between conditional contexts. For example, given the following configuration:</p>
<pre class="code ini literal-block">
<span class="k">[pi2]</span><span class="w">
</span><span class="na">kernel</span><span class="o">=</span><span class="s">uboot_rpi_2.bin</span><span class="w">
</span><span class="k">[pi3]</span><span class="w">
</span><span class="na">kernel</span><span class="o">=</span><span class="s">uboot_rpi_3.bin</span><span class="w">
</span><span class="k">[pi4]</span><span class="w">
</span><span class="na">kernel</span><span class="o">=</span><span class="s">uboot_rpi_4.bin</span>
</pre>
<p>If I want to set the kernel’s filename to “vmlinuz” (bypassing u-boot), on all
models, the tool needs to know that the conditional context of each “kernel”
line above is a subset of the requested conditional context (“[all]”), and thus
that all the existing “kernel” lines should be removed / commented; that a new
“[all]” section needs to be added at the end and the new line placed in that section.</p>
<p>Alternatively, if we want to make the same manipulation, but only for the
current model (let’s assume that’s a Pi 3), the tool needs to know that the
“[pi2]” section is a disjoint set with “[pi3]” so the first kernel line needs
to be left alone; the second is equal so that kernel line can be removed /
commented then replaced, and the final “[pi4]” section is also a disjoint set
with the requested context.</p>
<p>Anyway to cut an already way too long interlude short … that’s what the tool
now does. It’s fully aware of the conditional sections, how they relate, which
commands are only recognized in “config.txt” (and therefore should be ignored
for effect in other contexts), how to generate new conditional sections to
transform one conditional context into another, how to comment out lines
(optionally), and how to search for and uncomment lines in relevant contexts.</p>
<p>So, on with the show …</p>
</div>
<div class="section" id="stored-configurations">
<h2>Stored Configurations</h2>
<p>One major feature of pibootctl is that it’s capable of backing up and restoring
boot configurations. It’s relatively important to understand <em>how</em> it does this
given that, as a relatively low level tool designed to mess with your boot
configuration, it’s entirely possible to use it to <em>break</em> your boot configuration!</p>
<p>The Pi’s boot partition is a simple <span class="caps">FAT</span> partition at the start of the image,
and the boot configuration is (mostly) held in a series of straight-forward
text files. The great benefit of this is universal support: <span class="caps">FAT</span> is the
“universal file-system”; everything knows how to read and write it from Windows
to Mac to Linux, and even numerous embedded systems. Likewise, text files are
the “universal format”; every operating system has <em>some</em> facility for editing
text files (which is the reason it was selected as the original format for the
throughly worthy <a class="reference external" href="https://www.gutenberg.org/">Project Gutenberg</a>).</p>
<p>Ultimately this means that, should the worst happen, and you screw up your boot
configuration, you can stick your <span class="caps">SD</span> card in another machine and edit it back
to a workable state. However, it might be easier if you know what a workable
state is!</p>
<p>pibootctl strives to be equally accessible in its backup strategy: boot
configurations are stored as <span class="caps">ZIP</span> archives of the original (mostly) text files,
on the <span class="caps">FAT</span> partition (in a sub-directory). <span class="caps">ZIP</span> is another pretty-much-universal
format (Windows, Mac <span class="caps">OS</span>, and most Linux distros have built-in support for
opening <span class="caps">ZIP</span> files), and by storing the backups on the <span class="caps">FAT</span> partition we ensure
that they remain accessible on all platforms too.</p>
<p>Assuming you’ve <a class="reference external" href="https://waldorf.waveform.org.uk/2020/using-pibootctl.html">installed pibootctl on Ubuntu</a> from the <span class="caps">PPA</span>. you can create a
backup of your current boot configuration using pibootctl like so:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo pibootctl save original
</pre>
<p>Why is sudo required? Simply because we’re writing that backup to the boot
partition which only root has access to. After this you should be able to list
the available stored configurations like so:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>pibootctl ls
<span class="go">┌──────────┬────────┬─────────────────────┐
│ Name │ Active │ Timestamp │
├──────────┼────────┼─────────────────────┤
│ original │ ✓ │ 2020-09-15 12:27:02 │
└──────────┴────────┴─────────────────────┘</span>
</pre>
<p>Note that sudo isn’t required here as we’re not writing anything to the boot
partition, just reading things. Also note that pibootctl has compared the
current boot configuration to the stored “original” configuration and noticed
they’re exactly the same, hence the tick-mark in the “Active” column. Where
exactly is this stored?</p>
<p>In the pibootctl package in my <span class="caps">PPA</span>, pibootctl is configured to create a “store”
directory on the boot partition which will contain the stored configurations:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>ls /boot/firmware/store
<span class="go">original.zip</span>
</pre>
<p>You can install an appropriate archiving tool to take a look in this archive if
you want. Personally, I favour “atool” which provides generic commands for
working with a variety of archives (tar.gz, tar.xz, zip, etc. etc.):</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo apt install -y atool
<span class="gp">$ </span>als /boot/firmware/store/original.zip
<span class="go">Archive: /boot/firmware/store/nouboot.zip
pibootctl:0:e5bf92a57556aa24d4dd8c94ba63a755a21fdcc3
Do not edit the content of this archive; the line above is a hash of the
content which will not match after manual editing. Please use the pibootctl
tool to manipulate stored boot configurations
Length Date Time Name
--------- ---------- ----- ----
1116 2020-09-15 12:27 config.txt
327 2020-09-01 07:02 syscfg.txt
200 2020-09-01 07:02 usercfg.txt
127 2020-09-01 12:27 cmdline.txt
--------- -------
1770 4 files</span>
</pre>
<p>Several points of interest here:</p>
<ul class="simple">
<li>There’s a strange “pibootctl:0:xxxxxxx” line at the top of the <span class="caps">ZIP</span> archive’s
comment. This is a hash of the boot configuration in parsed order, and
pibootctl uses it to quickly determine which stored configurations match the
current boot configuration.</li>
<li>For this reason, there’s a comment straight after the hash line warning users
not to modify the content of the archive, or the hash will no longer match.
At the moment, the tool doesn’t use the hash to validate the archive contents
as I’m on the fence as to whether permitting users to create their own
archives manually is a good or bad idea but for now the warning stands.</li>
<li>After this you can see the various files that make up the boot configuration:
config.txt, syscfg.txt, usercfg.txt, and cmdline.txt (which is referenced
from config.txt). Note that other files referenced by the boot configuration
like the kernel (vmlinuz) and the initrd (initrd.img) are <em>not</em> included in
the archive. The assumption is that these things are under the control of the
package manager and shouldn’t be restored with the boot configuration itself.</li>
</ul>
</div>
<div class="section" id="manipulation">
<h2>Manipulation</h2>
<p>Now let’s edit our boot configuration using pibootctl. We’ll enable the camera
module and set <span class="caps">GPU</span> memory to 128Mb:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo pibootctl <span class="nb">set</span> camera.enabled<span class="o">=</span>on gpu.mem<span class="o">=</span><span class="m">128</span>
</pre>
<p>Note once again we use “sudo”. Because we’re changing files on the boot
partition we need root privileges. We can check the “status” output to see
these are definitely now set:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>pibootctl status
<span class="go">┌─────────────────────────┬──────────────────────┐
│ Name │ Value │
├─────────────────────────┼──────────────────────┤
│ audio.enabled │ on │
│ boot.devicetree.address │ 50331648 (0x3000000) │
│ boot.initramfs.filename │ ['initrd.img'] │
│ boot.kernel.64bit │ on │
│ boot.kernel.cmdline │ 'cmdline.txt' │
│ boot.kernel.filename │ 'vmlinuz' │
│ camera.enabled │ on │
│ gpu.mem │ 128 (Mb) │
│ i2c.enabled │ on │
│ serial.enabled │ on │
│ spi.enabled │ on │
│ video.framebuffer.max │ 2 │
│ video.overscan.enabled │ off │
└─────────────────────────┴──────────────────────┘</span>
</pre>
<p>In fact, the tool ensures that the written configuration matches the desired
configuration as part of its generation logic. Consider the following configuration:</p>
<div class="topic">
<p class="topic-title">config.txt</p>
<pre class="code ini literal-block">
<span class="ln">1 </span><span class="k">[pi4]</span><span class="w">
</span><span class="ln">2 </span><span class="w"></span><span class="na">max_framebuffers</span><span class="o">=</span><span class="s">2</span><span class="w">
</span><span class="ln">3 </span><span class="w">
</span><span class="ln">4 </span><span class="w"></span><span class="k">[all]</span><span class="w">
</span><span class="ln">5 </span><span class="w"></span><span class="na">arm_64bit</span><span class="o">=</span><span class="s">1</span><span class="w">
</span><span class="ln">6 </span><span class="w"></span><span class="na">cmdline</span><span class="o">=</span><span class="s">cmdline.txt</span><span class="w">
</span><span class="ln">7 </span><span class="w">
</span><span class="ln">8 </span><span class="w"></span><span class="na">include usercfg.txt</span>
</pre>
</div>
<div class="topic">
<p class="topic-title">usercfg.txt</p>
<pre class="code ini literal-block">
<span class="ln">1 </span><span class="na">dtoverlay</span><span class="o">=</span><span class="s">disable-bt</span>
</pre>
</div>
<p>In this case “usercfg.txt”, a file that pibootctl is not permitted to edit,
disables the bluetooth module. What happens if we explicitly try to enable it?</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo pibootctl <span class="nb">set</span> --no-backup bluetooth.enabled<span class="o">=</span>on
<span class="go">Failed to set 3 setting(s)
Expected bluetooth.enabled to be True, but was False after being overridden
by usercfg.txt line 1
Expected serial.uart to be 1, but was 0 with no valid lines; this usually
means a setting like start_x or gpu_mem is in a file other than config.txt
Expected serial.enabled to be False, but was True with no valid lines; this
usually means a setting like start_x or gpu_mem is in a file other than
config.txt</span>
</pre>
<p>pibootctl has noticed that it failed to set bluetooth.enabled, and warns the
user that the setting is “overridden by usercfg.txt line 1” (the other errors
come from the fact that disabling the bluetooth module implicitly enables the
serial port and places it on the <span class="caps">PL011</span> <span class="caps">UART</span>; these also don’t match pibootctl’s
expectations for the resulting configuration and so they get reported too).</p>
</div>
<div class="section" id="restoration">
<h2>Restoration</h2>
<p>How do we restore a saved configuration using pibootctl? The opposite of the
“save” command is “load”:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo pibootctl load nouboot
<span class="go">Backed up current configuration in backup-20200915-133659</span>
</pre>
<p>Once again, we need “sudo” as we’re manipulating the content of the boot
partition. One additional thing to note above is that pibootctl automatically
saved a copy of the current boot configuration. It did this because, before
modifying the current configuration, it checked a copy of it existed in the
stored configurations. In this case no copy existed (since we changed it
above), so it automatically saved one for us.</p>
<p>You can suppress this behaviour with the <tt class="docutils literal"><span class="pre">--no-backup</span></tt> switch, or permanently
by modifying the tool’s configuration in “/etc/pibootctl.conf”.</p>
</div>
<div class="section" id="don-t-panic">
<h2>Don’t Panic!</h2>
<p>Let’s assume the worst has happened and you’ve broken your boot configuration.
However, you’ve got a stored configuration you know is good in one of these <span class="caps">ZIP</span>
archives. How do you restore it if you can no longer boot your Pi and access pibootctl?</p>
<p>Simply place the <span class="caps">SD</span> card in another machine, find the <span class="caps">ZIP</span> archive storing the
boot configuration you want (which will be under the “store” directory we
explored above), and extract it over the top of current boot configuration.
You’ll almost certainly be prompted to replace the existing files (config.txt,
etc.) but that’s fine: permit the replacement and you’ve restored your boot
configuration! It really is that simple.</p>
</div>
<div class="section" id="addendum-why-zips">
<h2>Addendum: Why ZIPs?</h2>
<p>I did debate just using sub-directories instead of <span class="caps">ZIP</span> archives and storing
configurations “raw” so users could simply copy them back without bothering
with unzip tools. This is certainly an attractive method, but:</p>
<ul class="simple">
<li>It makes it much easier for users to manipulate stored configurations in
place. As mentioned above, I’m not (yet) convinced this is a good or bad
idea, so for now I’ve opted for the method which makes it harder.</li>
<li>It makes it easy to store meta-data alongside the stored configuration (the
aforementioned hash). We <em>could</em> do this in an auxilliary file in the stored
configuration but then we must be sure that could never exist in a legitimate
stored config. Or it could be in a file above the directory storing the
configuration, but then that has to be kept in sync with the stored
directories (which, as mentioned above, are easy to manipulate in their own right).</li>
</ul>
</div>
<div class="section" id="next-time">
<h2>Next Time</h2>
<p>That’s quite enough on this subject. Next time (hopefully less than months
away!) we’ll take a look at the “diff” command, and how to integrate pibootctl
with more friendly tools on top of it.</p>
</div>
Using pibootctl2020-05-20T00:00:00+01:002021-04-24T01:38:29+01:00Dave Jonestag:waldorf.waveform.org.uk,2020-05-20:/2020/using-pibootctl.html<p class="first last">The how of pibootctl</p>
<p>In the <a class="reference external" href="https://waldorf.waveform.org.uk/2020/introducing-pibootctl.html">previous post</a>, we looked at <em>why</em> I’ve been working on pibootctl. But
if you just want to use it, look no further! (Or, do go and read the fine
<a class="reference external" href="https://pibootctl.readthedocs.io/">manual</a> if you prefer).</p>
<p>Installing it on your pi is pretty simple:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>sudo add-apt-repository ppa:waveform/pibootctl
<span class="gp">$ </span>sudo apt install pibootctl
</pre>
<p>Side note: in future, this should be included by default on Ubuntu images, but
for now it lives in my <span class="caps">PPA</span>.</p>
<div class="section" id="help">
<h2>Help!</h2>
<p>The utility includes several sub-commands, but the first you ought to learn (or
even guess) is “help”:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>pibootctl <span class="nb">help</span>
<span class="go">usage: pibootctl [-h] [--version]
{help,?,status,dump,get,set,save,load,diff,show,cat,list,
ls,remove,rm,rename,mv} ...
pibootctl is a tool for querying and modifying the boot configuration of
the Raspberry Pi.
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
commands:
{help,?,status,dump,get,set,save,load,diff,show,cat,list,ls,remove,rm,
rename,mv}
help (?) Displays help about the specified command or setting
status (dump) Output the current boot time configuration
get Query the state of one or more boot settings
set Change the state of one or more boot settings
save Store the current boot configuration for later use
load Replace the boot configuration with a saved one
diff Show the differences between boot configurations
show (cat) Show the specified stored configuration
list (ls) List the stored boot configurations
remove (rm) Remove a stored boot configuration
rename (mv) Rename a stored boot configuration</span>
</pre>
<p>You can also ask this sub-command for help on any of the other sub-commands …
including itself:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>pibootctl <span class="nb">help</span> <span class="nb">help</span>
<span class="go">usage: pibootctl help [-h] [command-or-setting]
With no arguments, displays the list of pibootctl commands. If a command
name is given, displays the description and options for the named command.
If a setting name is given, displays the description and default value for
that setting.
positional arguments:
command-or-setting The name of the command or setting to output help for
optional arguments:
-h, --help show this help message and exit</span>
</pre>
<p>If you fancy good old man-pages, there’s those too. One for the tool itself,
and one for each sub-command. For example:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>man pibootctl
<span class="gp">$ </span>man pibootctl <span class="nb">help</span>
</pre>
<p>These include some additional notes and examples which may be worth a look
(they’re also included in the online <a class="reference external" href="https://pibootctl.readthedocs.io/">manual</a>).</p>
</div>
<div class="section" id="first-steps">
<h2>First Steps</h2>
<p>What about actual usage? The first thing you’ll probably want to do is look at
the current state of your boot configuration. This is what Ubuntu 20.04’s looks
like in pibootctl “out of the box”:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>pibootctl status
<span class="go">┌─────────────────────────┬──────────────────────┐
│ Name │ Value │
├─────────────────────────┼──────────────────────┤
│ audio.enabled │ on │
│ boot.devicetree.address │ 50331648 (0x3000000) │
│ boot.kernel.64bit │ on │
│ boot.kernel.cmdline │ 'cmdline.txt' │
│ boot.kernel.filename │ 'uboot_rpi_3.bin' │
│ i2c.enabled │ on │
│ serial.enabled │ on │
│ spi.enabled │ on │
└─────────────────────────┴──────────────────────┘</span>
</pre>
<p>Several things to note above:</p>
<dl class="docutils">
<dt>Oooh, pretty!</dt>
<dd>The default output mode is designed to be human-readable and, if your
locale includes “<span class="caps">UTF</span>-8” then unicode line-drawing characters will be used
for tables.</dd>
<dt>You’re using arm64 on a Pi3?</dt>
<dd>Right now, yes. It was the first card and Pi combination that came to hand!</dd>
<dt>There’s not many settings there?</dt>
<dd>Yes, only modified settings are listed by default because otherwise there’s
… a lot (and most of them you probably don’t care about). You can list
<em>all</em> the settings with the <tt class="docutils literal"><span class="pre">--all</span></tt> option, but be warned there’s more
than a page’s worth!</dd>
<dt>I don’t recognize those setting names!</dt>
<dd>Yes. This is probably going to be one of the more contentious decisions in
the design of pibootctl. I could’ve stuck with the actual commands used in
the Pi’s boot configuration, e.g. “dtparam=spi=on” or “uart_enable=1” or
“disable_overscan=1”, but I didn’t. This probably needs justifying as it’s
going to confuse anyone familiar with the existing settings (including me!)…</dd>
</dl>
<div class="section" id="setting-names">
<h3>Setting Names</h3>
<p>The names for settings in the Pi’s bootloader have grown more or less
organically over time and occasionally change as new interfaces emerge (e.g.
“display_rotate” being deprecated in favour of “display_lcd_rotate” and
“display_hdmi_rotate”). Moreover, the settings are for use in a fairly
low-level tool (the bootloader) so there’s a general preference for defaults to
be “0”, hence the occasional setting with reverse setting logic (“disable_overscan=1”).</p>
<p>The lack of a common naming standard (while understandable) does make it tricky
to query groups of settings common to an area. For example, how would you query
all the video related settings (remembering that “disable_overscan” setting)?
However, if a naming scheme is imposed on the settings, it becomes easy:</p>
<pre class="code console literal-block">
<span class="gp">ubuntu@pitest:~$ </span>pibootctl status --all video.*
<span class="go">┌──────────────────────────────┬──────────┬────────────────────────────────┐
│ Name │ Modified │ Value │
├──────────────────────────────┼──────────┼────────────────────────────────┤
│ video.cec.enabled │ │ on │
│ video.cec.init │ │ on │
│ video.cec.name │ │ 'Raspberry Pi' │
│ video.dispmanx.offline │ │ off │
│ video.dpi.clock │ │ off │
│ video.dpi.enabled │ │ off │
│ video.dpi.format │ │ 1 (9-bit RGB666; unsupported) │
│ video.dpi.group │ │ 0 (auto from EDID) │
│ video.dpi.hsync.disabled │ │ off │
│ video.dpi.hsync.phase │ │ off │
│ video.dpi.hsync.polarity │ │ off │
...</span>
</pre>
<p>The usual “*” and “?” <a class="reference external" href="https://en.wikipedia.org/wiki/Wildcard_character">wildcards</a> can be used in the setting pattern. In the
above output it’s worth noting that explanatory text is included for certain
setting values (e.g. “auto from <span class="caps">EDID</span>” for “video.dpi.group”). A more minimal
example is all the settings for the <span class="caps">TV</span> output:</p>
<pre class="code console literal-block">
<span class="gp">ubuntu@pitest:~$ </span>pibootctl status --all video.tv.*
<span class="go">┌─────────────────────┬──────────┬──────────┐
│ Name │ Modified │ Value │
├─────────────────────┼──────────┼──────────┤
│ video.tv.aspect │ │ 1 (4:3) │
│ video.tv.colorburst │ │ on │
│ video.tv.enabled │ │ on │
│ video.tv.mode │ │ 0 (NTSC) │
└─────────────────────┴──────────┴──────────┘</span>
</pre>
<p>This scheme also means that simple sorting by name groups related settings
together in the output. However, as mentioned above it is going to confuse
anyone who’s familiar with the underlying setting names. To try and mitigate
this a bit, the help output for all settings includes the underlying command
name (or overlay and parameter name as appropriate), and the “help” command
itself will also accept such names:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>pibootctl <span class="nb">help</span> disable_overscan
<span class="go"> Name: video.overscan.enabled
Default: on
Command(s): disable_overscan
When enabled (the default), if a group 1 (CEA) HDMI mode is selected
</span><span class="gp-VirtualEnv">(automatically or otherwise)</span><span class="go">, the display output will include black borders
to align the edges of the output with a typical TV display.</span>
</pre>
<p>There are cases where an underlying command is represented (or affected by)
several settings (and vice versa). In this case, asking for help on such a
setting will tell you the names of all settings that can affect it:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>pibootctl <span class="nb">help</span> start_file
<span class="go">start_file is affected by the following settings:
boot.debug.enabled
boot.firmware.filename
camera.enabled</span>
</pre>
</div>
</div>
<div class="section" id="scripting">
<h2>Scripting</h2>
<p>We’ve seen the pretty-printed output of multiple settings above. What about
retrieving an individual setting for scripting purposes? The “get” command is
used for this (there’s very little point using this command outside a script):</p>
<pre class="code console literal-block">
<span class="gp">$ </span>pibootctl get serial.enabled
<span class="go">on</span>
</pre>
<p>While this is simple enough, it’s not very efficient when it comes to obtaining
the value of <em>lots</em> of settings. To accommodate usage from scripts, the
majority of commands in pibootctl (including “get” and “status”) accept
the <tt class="docutils literal"><span class="pre">--json</span></tt>, <tt class="docutils literal"><span class="pre">--yaml</span></tt>, or <tt class="docutils literal"><span class="pre">--shell</span></tt> switches to produce output in <span class="caps">JSON</span>,
<span class="caps">YAML</span>, or a shell-compatible style respectively:</p>
<pre class="code console literal-block">
<span class="gp">$ </span>pibootctl status --json --all <span class="s2">"video.hdmi*"</span>
<span class="go">{"video.hdmi.edid.override": false, "video.hdmi.safe": false,
"video.hdmi.4kp60": false, "video.hdmi.edid.ignore": false,
"video.hdmi.edid.contenttype": 0, "video.hdmi0.edid.filename": "edid.dat",
"video.hdmi.edid.parse": true, "video.hdmi0.enabled": null,
"video.hdmi.edid.3d": false, "video.hdmi0.audio": null,
"video.hdmi0.boost": 5, "video.hdmi0.group": 0, "video.hdmi0.timings": [],
"video.hdmi.powersave": false, "video.hdmi0.mode": 0,
"video.hdmi0.mode.force": false, "video.hdmi0.encoding": 0,
"video.hdmi0.drive": 0, "video.hdmi0.rotate": 0, "video.hdmi0.flip": 0,
"video.hdmi1.boost": 5, "video.hdmi1.enabled": null, "video.hdmi1.audio":
null, "video.hdmi1.edid.filename": "edid.dat", "video.hdmi1.timings": [],
"video.hdmi1.group": 0, "video.hdmi1.mode": 0, "video.hdmi1.mode.force":
false, "video.hdmi1.encoding": 0, "video.hdmi1.drive": 0,
"video.hdmi1.rotate": 0, "video.hdmi1.flip": 0}
</span><span class="gp">$ </span>pibootctl get --json serial.enabled
<span class="go">true</span>
</pre>
</div>
<div class="section" id="writing">
<h2>Writing?</h2>
<p>That’s enough for this post; next time we’ll have a look at how pibootctl
modifies the boot configuration, and the steps it takes to try and do so safely!</p>
</div>
Introducing pibootctl2020-04-27T00:00:00+01:002021-04-24T01:38:29+01:00Dave Jonestag:waldorf.waveform.org.uk,2020-04-27:/2020/introducing-pibootctl.html<p class="first last">The what and the why of pibootctl</p>
<p>For the last few months I’ve been working on a little tool called pibootctl.
The hope was to include that in the Focal release, but it couldn’t quite make
the cut while so much work was going on with the boot sequence. Still, it’s
available from a <a class="reference external" href="https://launchpad.net/~waveform/+archive/ubuntu/pibootctl"><span class="caps">PPA</span></a> (<tt class="docutils literal">ppa:waveform/pibootctl</tt>) now for those that wish to
try it out, and <a class="reference external" href="https://pibootctl.readthedocs.io/">the documentation</a> is online.</p>
<p>This is the first in a short series of posts explaining some of the background
behind the tool, and what its future (hopefully!) is.</p>
<div class="section" id="so-what-is-it">
<h2>So what is it?</h2>
<p>It’s a command line tool for managing the Pi’s boot configuration.</p>
<div class="section" id="id1">
<h3>So what is it?</h3>
<p>It’s a utility for saving, restoring, and editing the various text files on the
boot partition.</p>
</div>
<div class="section" id="id2">
<h3><a class="reference external" href="https://www.youtube.com/watch?v=TxWN8AhNER0">So what is it?</a></h3>
<p>It’s a less friendly <tt class="docutils literal"><span class="pre">raspi-config</span></tt> … which doesn’t do networking either.</p>
</div>
<div class="section" id="only-joking">
<h3>Only Joking!</h3>
<p>Well, sort of. To be clear: it’s <em>not</em> intended to be a replacement for the
excellent <tt class="docutils literal"><span class="pre">raspi-config</span></tt> tool, however I <em>do</em> intend it to form part of the
underlying kit for something vaguely similar.</p>
</div>
</div>
<div class="section" id="why">
<h2>Why?!</h2>
<p>Or more precisely “why not just use or port <tt class="docutils literal"><span class="pre">raspi-config</span></tt>?”. I’ve been asked
this pretty frequently so it’s probably best to have somewhere I can point to
with all my arguments in a row, so here goes …</p>
<p>Firstly, let’s be clear: <tt class="docutils literal"><span class="pre">raspi-confgi</span></tt> is great. It does exactly what it
says on the tin in a friendly, user-accessible way, and Ubuntu certainly needs
it, or something very much like it. So let’s begin with why we can’t “just use it”:</p>
<ul class="simple">
<li>Ubuntu Server is pretty tightly wedded to <a class="reference external" href="https://netplan.io/">netplan</a> for its networking
configuration. Whether that’s via the <a class="reference external" href="https://github.com/CanonicalLtd/subiquity">subiquity</a> installer on regular
server images, or via <a class="reference external" href="https://cloudinit.readthedocs.io/">cloud-init</a> on our cloud and Pi images.</li>
<li>Raspbian is more traditional in its networking approach, and <tt class="docutils literal"><span class="pre">raspi-config</span></tt>
knows nothing about <a class="reference external" href="https://netplan.io/">netplan</a>. Now, that’s not to say it won’t work: it will
set up your <span class="caps">WPA</span> config, but it won’t keep the netplan configuration in sync
so the next time you run <tt class="docutils literal">netplan apply</tt> it’s overwritten. Not an ideal
user experience.</li>
</ul>
<p>Well, that doesn’t sound so complex? We could patch <tt class="docutils literal"><span class="pre">raspi-config</span></tt> to affect
the netplan configuration instead couldn’t we?</p>
<p>Yes we could, but now we’re carrying a delta to the upstream <tt class="docutils literal"><span class="pre">raspi-config</span></tt>
tool. Either someone needs to maintain that delta as <tt class="docutils literal"><span class="pre">raspi-config</span></tt> evolves,
or we need to get that delta applied upstream. The former is undesirable, and
the latter is highly unlikely to happen (unless Raspbian wants to switch over
to using netplan).</p>
<p>Besides, it’s not the only problem to overcome:</p>
<ul class="simple">
<li>Ubuntu’s boot sequence on the Pi is in some ways more complex and in some
ways simpler than Raspbian’s, but either way it’s pretty <em>different</em>. One
notable facet is that our “config.txt” file includes other files by default.</li>
<li>Unfortunately, <tt class="docutils literal"><span class="pre">raspi-config</span></tt> doesn’t handle includes in the boot
configuration, either for parsing or writing. In the face of includes, it
either doesn’t know the state of existing settings (from includes), or will
fail to notice if its written settings are overridden in later includes.</li>
</ul>
<p>Solving this means writing a fairly comprehensive parser for the “config.txt”
file (and its includes) and some logic to compare a “desired” configuration to
the “effective” configuration after writing changes to determine whether
settings have been overridden in later included files.</p>
<p>Funnily enough, that’s exactly what pibootctl does!</p>
<p>Ah ha! Could pibootctl be used by <tt class="docutils literal"><span class="pre">raspi-config</span></tt> to fix these issues?
Potentially; I’ve ensured there’s absolutely nothing Ubuntu-specific in the
tool, and it’s pure Python so there’s nothing preventing it from running on
architectures we don’t support like the Pi Zero, but we’re getting ahead of ourselves.</p>
<p>Let’s finish off the issues with porting <tt class="docutils literal"><span class="pre">raspi-config</span></tt>:</p>
<ul class="simple">
<li>Setting up the boot options for desktop or <span class="caps">CLI</span>: we don’t include a desktop on
our images, so that’s out (or at the very least, becomes conditional).</li>
<li>Setting up auto-login and splash-screen. We don’t have those either (on
classic, and there’s … difficulties with adding a splash screen that are
sufficiently intricate I won’t bore you with them here).</li>
<li><span class="caps">VNC</span>. Raspbian includes the extremely slick (but proprietary) Real-<span class="caps">VNC</span>, which
we don’t have on Ubuntu. <em>Side note:</em> personally, the one feature I’d
<em>really</em> like to see ported to one of the open source <span class="caps">VNC</span> implementations is
dispmanx capture so that things like the camera preview can pass over <span class="caps">VNC</span>, as
on Real-<span class="caps">VNC</span>.</li>
<li>Remote <span class="caps">GPIO</span>. Nope, we haven’t got pigpio into the archive yet either.</li>
<li>Read-only <span class="caps">FS</span> via overlay. Again, there’s … intricate issues here.</li>
</ul>
<p>You get the picture. Even if we <em>did</em> port <tt class="docutils literal"><span class="pre">raspi-config</span></tt>, tweak it to use
<a class="reference external" href="https://netplan.io/">netplan</a>, and slot in <a class="reference external" href="https://pibootctl.readthedocs.io/">pibootctl</a> to handle the configuration re-writing,
we’d still wind up with something inferior. Something the user would fire up
and immediately notice several options missing. Something that, even if we
included the <span class="caps">VNC</span> option, would work <em>quite</em> as well as on Raspbian.</p>
<p>Not an ideal result, and we’d <em>still</em> be carrying a fat delta which would
require on-going maintenance every time upstream changed.</p>
<div class="section" id="addendum">
<h3>Addendum</h3>
<p>There’s a flip-side to the above. While there’s several things we’d need to
drop from a ported <tt class="docutils literal"><span class="pre">raspi-config</span></tt>, there’s also plenty of stuff
<tt class="docutils literal"><span class="pre">raspi-config</span></tt> doesn’t do that we could trivially add given the capabilities
of the underlying tools on Ubuntu:</p>
<ul class="simple">
<li>With netplan, we could have full Ethernet configuration options, IPv4 and
IPv6 settings, and more.</li>
<li>We tend to favour non-password <span class="caps">SSH</span> configurations; should we not add
facilities to import a user’s <span class="caps">SSH</span> keys?</li>
<li>There’s some facilities of pibootctl (which we’ll be seeing in forthcoming
posts) that’d be interesting to include too!</li>
</ul>
</div>
</div>
<div class="section" id="so-how-then">
<h2>So … how then?</h2>
<p>That’s enough argument about why porting <tt class="docutils literal"><span class="pre">raspi-config</span></tt> over to Ubuntu is not
as trivial as most think (or in some cases even desirable). But it’s obvious to
everyone (even me!) that we do need <em>something</em> user friendly for Pi
configuration in Ubuntu.</p>
<p>This is something I’ll be working on this cycle, building on top of the tools
we’ve got in Ubuntu. Which brings us (finally!) back to pibootctl. When I was
looking at the tools we’ve got to play with for system configuration on Ubuntu,
which I could build upon for a raspi-config-esque tool, one major piece of the
jigsaw was missing: something to deal with “config.txt”.</p>
<p>Not terribly surprising given it’s entirely specific to the Pi but we did need
something to fill that gap. Initially, I’d considered grabbing bits from
<tt class="docutils literal"><span class="pre">raspi-config</span></tt> but when I checked it there were a couple of problems:</p>
<p>Firstly, it uses lua for re-writing the configuration. Nothing expressly wrong
with that, but lua isn’t in “main” on Ubuntu and anything included by default
on our images (as such a raspi-config-esque tool eventually would) is meant to
be in “main” (not “universe”).</p>
<p>Moving stuff to main involves <a class="reference external" href="https://wiki.ubuntu.com/MainInclusionProcess">a pretty thorough review</a> process and wasn’t
something I thought we could realistically do for a full-blown programming
language when it was only going to be used for a single thing. That already
necessitated re-writing a chunk of <tt class="docutils literal"><span class="pre">raspi-config</span></tt> with something in “main”,
most likely python.</p>
<p>It can’t handle configurations using includes, and for better or worse I’d
already moved us over to a configuration that used them (in Eoan). Moving back
would be a nightmare of detection and manipulation logic in some package’s
post-inst scripts (more complex than moving <em>to</em> the split configuration in the
first place and I remember how nasty that was). Not to mention, I’m still of
the opinion that the split configuration brings certain benefits (though that’s
involved enough that it should probably be its own post).</p>
</div>
<div class="section" id="not-now">
<h2>Not now?</h2>
<p>It feels like I’ve been wittering on for way too long in this post, so I’m
going to stop here, and leave the first play-through for next time! In the
meantime, anyone that wants to have a play, go have a look at the <a class="reference external" href="https://launchpad.net/~waveform/+archive/ubuntu/pibootctl"><span class="caps">PPA</span></a> or
<a class="reference external" href="https://pibootctl.readthedocs.io/">the documentation</a>.</p>
</div>