It's Pi all the way down... - net

I ain’t got NBD

2024-12-03T00:00:00+00:00

Last time we set up an NBD server, and the associated DHCP machinery to netboot our Pi. We then discovered a rather serious problem: when the Pi updated the content of the boot partition, the TFTP wouldn’t notice and would serve old files because the mount of the boot partition on the server didn’t realize it was “shared” with the NBD server.

Come and take a chance with me?

As mentioned in the last post, in trying to solve this I ventured through a myriad of hacky or simple solutions, all attempting to work around the “caching” of pages by the file-system driver on the server side. Yet again, caching rears its head as one of the two “hard problems” of computer science [1], though the root of the issue here is the assumption that the file-system driver has exclusive access to the underlying block device (the cache is merely a consequence of this).

In hindsight, I should’ve looked at the pieces and concluded what I eventually did: just write your own TFTP server! One which can read files directly out of the boot partition of the OS image. This solution sounds like a sledgehammer to crack a nut, but actually it’s less complex than it seems. All it’s got to handle is simple partitioning (MBR or GPT), reading a FAT file-system , and TFTP.

I may not be selling this as “simple”, but let’s break down what each of these actually entails:

MBR

Master Boot Record partitioning is an ancient method of partitioning a floppy disk hard drive USB stick small storage medium, but while it’s very difficult to accurately write (in such a way that it’ll actually … boot things), it’s not hard to read (other than the minor mess that is logical partitioning, intended to get around the original 4-partition limit). A few hours work. Maybe.

GPT

GUID Partition Tables are the modern, and frankly trivial (by comparison to MBR) means of partitioning storage media. No more than an hour needed here.

FAT

File Allocation Tables are another positively ancient piece of computing history, used for storing files on just about every piece of computing media with the possible exception of the tape drive (and I wouldn’t put it past someone to have tried that). The file-system isn’t complicated in basis (in fact I’ve written a library for something very similar before [2]); the complexity comes from the myriad implementations that use the same structures, but disagree subtly (and not so subtly) about the meaning or even position of certain fields within them.

Still, a few evenings to get the basics done, then several more to work out the rough edges (FAT-12, long filename handling, etc. etc.), then a few more to build a nice Path-like API on top of it, and we’re good!

TFTP

Trivial File Transfer Protocol is, tautologically, trivial. Getting the basic protocol implemented took only a couple of hours (including testing with several implementations). Then it was onto the extensions; I wound up only bothering with those that the Pi bootloader attempts to negotiate and one more that’s useful for testing (tsize, blocksize, and timeout) though hopefully I’ve structured things suitably for a future addition of the windowsize option. Still, only a few more hours on that.

All in all, despite a few headaches (mostly around FAT’s obscure history), it wasn’t difficult to throw all this together, stir it around a bit, and produce what I needed (would’ve been nice to make it into a “learn Rust” project, but I was already waaay overdue on this post series, so I wound up going with my comfortable old Python).

The result is nobodd: a TFTP boot-server that will read files directly out of the FAT partition of an image without mounting it.

NBD! NBD!

Let’s walk through a set up using nobodd and nbd-server. The client side is exactly the same as before so I’ll just refer you to the start of the last post for that part. Go through requirements (a Pi and a server), configuring the Pi for netboot, adding linux-modules-extra- and nbd-client, and transferring the regenerated initramfs and identity to the server, and shutting down the Pi [3].

Now, on the server we’re going to do things a bit differently. This time we’re going to configure a “template” image that we can copy to “instance” images any time we want a new install. And this time we’re going to use the fancy little nobodd-prep tool from nobodd to do it.

Start as we did before by downloading the image and verifying it:

$ sudo -i
Password:
# mkdir /srv/images
# cd /srv/images
# wget http://cdimage.ubuntu.com/releases/22.04.3/release/ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz
 ...
# wget http://cdimage.ubuntu.com/releases/22.04.3/release/SHA256SUMS
 ...
# sha256sum --check --ignore-missing SHA256SUMS
ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz: OK
# rm SHA256SUMS

Now, we simply use nobodd-prep to customize it. This tool can be used to resize the image, re-write the cmdline.txt parameters, and copy files onto the boot partition (including the customized initrd.img and a cloud-init user-data file), all in one go. It can only operate on uncompressed images though, so we still need to do that:

# unxz ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz
# mv ubuntu-22.04.3-preinstalled-server-arm64+raspi.img jammy-template.img
# apt install nobodd-tools
 ...
# cat << EOF >> user-data
package_update: true
packages:
- avahi-daemon
- nbd-client
- linux-modules-extra-raspi
# cp jammy-template.img jammy.img
# nobodd-prep --copy user-data --copy initrd.img --size 16GB jammy.img

And that’s it! My hope is that, in noble (24.04) most of these steps will not be necessary. In that release, it should be possible to download the image unpack it, give it a nice name, run nobodd-prep to customize it, and go from there.

Hi, Future Dave here! Well, the MIR for nbd-client, which I had hoped would be a simple affair, erm, wasn’t [4]. It didn’t make it into noble (24.04), or oracular (24.10). It might make it into plucky (25.04), thanks largely to the efforts of my new side-kick [5]. But this update is so overdue, I’m just going to post it anyway, and hope it’ll be of use to Future You if/when the relevant things finally make it into the image.

—Future Dave

Next, we’ll install and configure the required daemons. As before, we’ll use dnsmasq as a DHCP proxy, and nbd-server. However, this time instead of dnsmasq also handling TFTP duties, we’ll use nobodd-tftpd for that. First up, install the packages:

# apt install dnsmasq nbd-server nobodd-tftpd

Next, configure dnsmasq:

# cat << EOF >> /etc/dnsmasq.conf
interface=eth0
bind-interfaces
dhcp-range=192.168.255.255,proxy
pxe-service=0,"Raspberry Pi Boot"
# systemctl restart dnsmasq

Note

Adjust the reference to eth0 if your Ethernet NIC is named something else. If your network’s mask is not 192.168.255.255, adjust this accordingly.

Now the NBD server:

# chown nbd:nbd jammy.img
# ls -lh jammy.img
-rw-r--r-- 1 nbd  nbd 16.0G Dec  3 13:48 jammy.img
# cat << EOF >> /etc/nbd-server/conf.d/jammy.conf
[jammy]
exportname = /srv/images/jammy.img
EOF
# systemctl restart nbd-server

And finally the TFTP server:

# cat ~ubuntu/pi-ident.txt
Serial          : 1000000089025d75
# piserial=$(sed -e '1s/^Serial.*\([0-9a-f]\{8\}\)$/\1/' ~ubuntu/pi-ident.txt)
# echo $piserial
89025d75
# cat << EOF >> /etc/nobodd/conf.d/jammy.conf
[board:$piserial]
image = /srv/images/jammy.img
partition = 1
EOF
# systemctl reload nobodd-tftpd

The final config piece (/etc/nobodd/conf.d/jammy.conf) ties the Pi’s board serial number to the image that we want to actually boot. The cmdline.txt on the boot partition of that image needs to point to the NBD server serving that same image. This was actually handled implicitly by the nobodd-prep command above. It assumes:

The NBD server is the machine it is running on
The NBD share will be named after the stem of the image’s filename (in other words it transforms “jammy.img” to “jammy”)
The root partition is the first non-FAT-type partition in the image (partition 2 in this case)

Hence, in this case it will have automatically inserted the following in the cmdline.txt of the image:

ip=dhcp nbdroot=ubuntu/jammy root=/dev/nbd0p2 ...

When the end comes, I know

At this point, you should be ready to go. Power up the Pi, and if everything is working correctly, it should netboot into your jammy image, applying the custom cloud-init along the way.

How do you add more boards? Now you’ve got your template image, the process should be as simple as:

Find your next board’s serial number
Create another copy of jammy-template.img
Customize it with nobodd-prep
Add the NBD server configuration, reload the NBD server
Add the TFTPD server configuration, reload the TFTPD server

It’s worth noting that nobodd-prep is technically capable of generating the required configurations for the last two items. However, I’m a bit unhappy with the design of it at the moment. It’s become a horribly convoluted “big” tool, and I think things would be much more flexible if it was broken up into a series of much simpler tools that emulate the functionality of cat, cp, rm, ls, and providing something that uses those pieces to achieve the customization of the current tool.

Another aspect of this setup worth noting is that we haven’t used mount anywhere. Neither nbd-server nor nobodd-tftpd require mounts or loop devices to operate. Why is this important? Because now this entire set up can run in an unprivileged container, which makes testing things much simpler. There are still hoops to jump through [6] if you want to go this route, which are beyond the scope of this post, but it is at least possible.

[1]	Alongside naming things and off-by-one errors …

[2]	I’ve written stuff very similar to a FAT file system driver in Python before.

[3]	In other words, follow the last post until “Why’d you have to be so good?”. Incidentally, I make no apologies for the musical puns in my titles!

[4]	After several years in Foundations, you’d think I’d have learned by now, wouldn’t you? If it looks hard, it is. If it looks easy, it isn’t.

[5]	Yes! It’s not just me anymore! Say “hi” to r41k0u who’s also doing some sterling work on the camera stack this cycle.

[6]	Specifically this involves getting your container to appear directly on your network, rather than hiding behind NAT, so that it can proxy DHCP requests

NBD Does it Better

2023-10-30T00:00:00+00:00

In the prior post we looked at the issues with using NFS as our netboot root, and some of the alternatives. This time we’ll go through setting up the server and client of an NBD netboot system. If you want to play along at home you’ll need:

A Raspberry Pi. I’ll be using a Pi 4B [1].
A real [2] server running Ubuntu. I’m going to be using another Pi for this, but any old PC should be fine too (whatever server you use will benefit from as much IO bandwidth, both disk and network, as you can get).
An ethernet network. You can’t netboot a Pi over wifi, so you’ll need an ethernet set up for this.

Before we begin you should ensure your server is up and running, and that you have remote SSH access to it. In the following instructions I’m assuming that your server’s hostname is server, and that you have an ubuntu user on it which can sudo to root. Adjust according to your setup!

We’re going to start with the client side of things, oddly, but there is method in the madness. Firstly, we need to ensure that the Pi’s bootloader is configured to attempt bootloading (which none are by default). And secondly, we need to do some surgery on the initramfs for later.

Some Kind of Magic

We’re going to be using the current Ubuntu LTS (“jammy”, 22.04.3) throughout this guide, both on the server later, and for the Pi’s client image, to keep things relatively simple. Fire up rpi-imager and flash Ubuntu 22.04 server onto an SD card, then boot that SD card on your chosen Pi.

Warning

Do not be tempted to upgrade packages at this point. Specifically, the kernel package must not be upgraded yet.

Now we need to ensure that the Pi is configured to attempt network booting. This is a one-time change which will be stored in the EEPROM of the Pi in question. On the Pi, extract the current boot configuration from the EEPROM, modify the existing BOOT_ORDER= line (or append a new one if none is present), and apply the modified configuration:

$ sudo rpi-eeprom-config > boot.conf
$ cat boot.conf
[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf41
$ sed -i -e '/^BOOT_ORDER=/d' boot.conf
$ echo BOOT_ORDER=0xf21 >> boot.conf
$ cat boot.conf
[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf21
$ sudo rpi-eeprom-config --apply boot.conf
Updating bootloader EEPROM
 ...
$ sudo reboot

Obviously, feel free to fire up your favourite editor and just change the BOOT_ORDER= line yourself, instead of messing with sed. The mysterious 0xf21 value is explained fully in the BOOT_ORDER documentation on the Raspberry Pi website, but simply means try the SD card first (1), followed by the network (2), instead of USB boot (4) previously, and if both fail then repeat (f) [3]. The digits are specified in reverse order for $reasons.

The reboot at the end is required to apply the new configuration to the boot EEPROM. You can run sudo rpi-eeprom-config after rebooting to check the newly applied configuration.

Next, we need to install the linux-modules-extra-raspi package for the currently running kernel version. The reason is that the nbd kernel module was moved out of the default linux-modules-raspi package for efficiency. We specifically need the version matching the running kernel version because installing this package will regenerate the initramfs (initrd.img). We’ll be copying that regenerated file into the image we’re going to netboot and it must match the kernel version in that image. This is why it was important not to upgrade any packages after the first boot.

We also need to install the NBD client package. This will add the nbd-client executable to the initramfs, along with some scripts to call it if the kernel command line specifies an NBD device as root [4]:

$ sudo apt install linux-modules-extra-$(uname -r) nbd-client

We need to gather one piece of information about the client Pi for use later on the server: its serial number. We’ll store this in a file and copy it and the initrd.img to the server. Finally, we’ll shut down the Pi and move to the server side of things:

$ grep Serial /proc/cpuinfo > pi-ident.txt
$ cat pi-ident.txt
Serial          : 1000000089025d75
$ scp -q pi-ident.txt ubuntu@server:
$ scp -q /boot/firmware/initrd.img ubuntu@server:
$ sudo poweroff

Why’d you have to be so good?

The first thing to do on the server is get the image [5] we want to serve, and do a little surgery on it. We flashed Ubuntu 22.04.3 so we set up a directory under /srv to hold the image, wget it, and check the SHA256 checksum. Note that we’re going to perform most of these steps as root:

$ sudo -i
Password:
# mkdir /srv/images
# cd /srv/images
# wget http://cdimage.ubuntu.com/releases/22.04.3/release/ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz
 ...
# wget http://cdimage.ubuntu.com/releases/22.04.3/release/SHA256SUMS
 ...
# sha256sum --check --ignore-missing SHA256SUMS
ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz: OK
# rm SHA256SUMS

Now we’re going to unpack the image (it’s no good mounting something that’s XZ compressed), rename it to something more manageable , and expand the image file to the full size of SD card we want to emulate (I’m using 8GB here, but change the fallocate command accordingly):

# unxz ubuntu-22.04.3-preinstalled-server-arm64+raspi.img.xz
# mv ubuntu-22.04.3-preinstalled-server-arm64+raspi.img jammy.img
# ls -lh
-rw-rw-r-- 1 root root 4.0G Oct  5 14:12 jammy.img
# fallocate -l 8G jammy.img
# ls -lh
-rw-rw-r-- 1 root root 8.0G Oct  5 14:51 jammy.img

We’ve expanded the image, but naturally the partitions inside it haven’t been changed in size, and nor have the file-systems inside those partitions. However, that’s fine. This is exactly what a freshly flashed 8GB SD card looks like. The device (in this case the image file) is 8GB, but the root partition inside is a mere 3.7 GB. We can use fdisk to see this:

# fdisk -l jammy.img
Disk jammy.img: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x542d34fa

Device     Boot  Start     End Sectors  Size Id Type
jammy.img1 *      2048  526335  524288  256M  c W95 FAT32 (LBA)
jammy.img2      526336 8320243 7793908  3.7G 83 Linux

On first boot of a 8GB SD card, the cloud-init service checks its configuration, sees that it should expand the root, re-writes the root partition, and expands the file-system. Exactly the same will happen here when we first boot this image.

Next, we need to overwrite the initrd.img in the boot partition of this image, with the one we generated on our client (and which we copied to the server earlier). In order to do so we need to mount this image. Normally, if the image file contained only the file-system we wanted to manipulate, this would be as trivial as running mount -o loop jammy.img some-path. But this image contains partitions, so the file-system we wish to mount isn’t at the start of the image.

To get around this, instead of having a loop device created implicitly (with mount’s -o loop option), we need to make our own loop-device and tell the kernel to scan it for partitions. Then we’ll create a mount-point and mount the first partition there. Finally, we’ll copy our customized initrd.img into the mount-point:

# losetup --find --show --partscan jammy.img
/dev/loop5
# ls -l /dev/loop5*
brw-rw---- 1 root disk   7, 5 Oct  5 21:20 /dev/loop5
brw-rw---- 1 root disk 259, 0 Oct  5 21:20 /dev/loop5p1
brw-rw---- 1 root disk 259, 1 Oct  5 21:20 /dev/loop5p2
# mkdir boot
# mkdir boot/jammy
# mount /dev/loop5p1 /srv/images/boot/jammy
# cp initrd.img boot/jammy/

Warning

The loop device on your system will likely have a different number; adjust /dev/loop5 references accordingly.

Next, we should also customize the cloud-init initial configuration to ensure the image installs the same packages that we installed on the client earlier:

# cat << EOF >> boot/jammy/user-data
package_update: true
packages:
- avahi-daemon
- nbd-client
- linux-modules-extra-raspi
EOF

If we don’t do this, the next time our netbooted client refreshes its initramfs, it would generate it without the NBD client (and would naturally fail at the next reboot).

Now we need to edit the kernel command line to tell it that its root device is an NBD share. The kernel command line is one long line of text with space-separated portions. We’re going to change those space-separated bits into individual lines to make it easier to manipulate, remove the existing root=LABEL=writable portion [6], and insert the following portions instead:

ip=dhcp — we need an IP address to find the root device, and that it should obtain it via DHCP
nbdroot=server/jammy — set up an NBD client and connect to the jammy share on the host server (adjust this to match your server’s name or IP address)
root=/dev/nbd0p2 — find the actual root on the second partition of the connected NBD device

Instead of doing my usual confusing one-liner, we’ll step through the actions below, but feel free to fire up your favourite text editor and just edit cmdline.txt directly if you find that easier:

# cat boot/jammy/cmdline.txt
console=serial0,115200 dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash
# cat boot/jammy/cmdline.txt | tr ' ' '\n' > /tmp/cmdline.txt
# cat /tmp/cmdline.txt
console=serial0,115200
dwc_otg.lpm_enable=0
console=tty1
root=LABEL=writable
rootfstype=ext4
rootwait
fixrtc
quiet
splash
# sed -i -e '/^root=/ s@=.*$@=/dev/nbd0p2@' /tmp/cmdline.txt
# sed -i -e '/^root=/ i ip=dhcp' /tmp/cmdline.txt
# sed -i -e '/^root=/ i nbdroot=server/jammy' /tmp/cmdline.txt
# cat /tmp/cmdline.txt
console=serial0,115200
dwc_otg.lpm_enable=0
console=tty1
ip=dhcp
nbdroot=server/jammy
root=/dev/nbd0p2
rootfstype=ext4
rootwait
fixrtc
quiet
splash
# paste -s -d ' ' /tmp/cmdline.txt > boot/jammy/cmdline.txt
# cat boot/jammy/cmdline.txt
console=serial0,115200 dwc_otg.lpm_enable=0 console=tty1 ip=dhcp nbdroot=server/jammy root=/dev/nbd0p2 rootfstype=ext4 rootwait fixrtc quiet splash

Now it’s time to configure the DHCP proxy and NBD server we talked about in the last article. We’ll start with the packages we’re going to need: the NBD server itself, and the ubiquitous dnsmasq daemon which will be handling DHCP, and TFTP for our netbooting clients.

Note

Don’t worry if you’ve already got a DHCP server on your network. I’ve assumed that you almost certainly do and will be configuring the DHCP portion of dnsmasq in DHCP “proxy” mode where it simply steps in to augment the options transmitted by the authoritative DHCP server.

To put it another way: you shouldn’t have to dismantle or reconfigure your network to play along!

Install the required packages:

# apt install dnsmasq nbd-server

During this installation you may see several warnings about dnsmasq being unable to start due to the address already being in use. This is normal and occurs because systemd-resolved is already listening on port 53 (the DNS port) for the loopback address, so it can cache DNS requests. We now configure dnsmasq to only listen on port 53 of the ethernet NIC, to act as a DHCP proxy, and TFTP server:

# cat << EOF >> /etc/dnsmasq.conf
interface=eth0
bind-interfaces
dhcp-range=192.168.255.255,proxy
pxe-service=0,"Raspberry Pi Boot"
enable-tftp
tftp-root=/srv/images/boot
EOF
# systemctl restart dnsmasq

Note

Adjust the reference to eth0 if your Ethernet NIC is named something else. If your network’s mask is not 192.168.255.255, adjust this accordingly.

Next up is the NBD server, which simply needs to point the share “jammy” at our “jammy.img”. However, we also need to remember to change ownership of our images so the unprivileged “nbd” user can write to it:

# chown nbd:nbd jammy.img
# ls -lh
total 8.1G
drwxr-xr-x 3 root root 4.0K Oct 30 16:49 boot
-rw-r--r-- 1 nbd  nbd  8.0G Oct 30 16:55 jammy.img
# cat << EOF >> /etc/nbd-server/conf.d/jammy.conf
[jammy]
exportname = /srv/images/jammy.img
EOF
# systemctl restart nbd-server

Finally, we link our Pi’s serial number (or more precisely, the last 8 digits of it, if it’s longer than that) with the mounted boot partition.

# cat ~ubuntu/pi-ident.txt
Serial          : 1000000089025d75
# piserial=$(sed -e '1s/^Serial.*\([0-9a-f]\{8\}\)$/\1/' ~ubuntu/pi-ident.txt)
# echo $piserial
89025d75
# ln -s jammy boot/$piserial
# ls -l boot
total 3
lrwxrwxrwx 1 root root    5 Oct 30 16:49 89025d75 -> jammy
drwxr-xr-x 3 root root 2560 Jan  1  1970 jammy

Keeps Me From Runnin’

You’re now at a point where you can try netbooting your client Pi. Remove its SD card, and plug it in. You should see the “rainbow” boot screen appear fairly quickly, but there’ll be a long pause on that screen. The reason is that your Pi is transferring initrd.img (which is now much larger than normal due to our installation of linux-modules-extra) over TFTP which is not an efficient protocol without certain extensions, which the Pi’s bootloader doesn’t implement. However, eventually you should be greeted by the typical Linux kernel log scrolling by and reach a typical “booted” state the same as you would with an SD card.

If you hit any snags here, the following things are worth checking:

Pay attention to any errors shown on the Pi’s bootloader screen. In particular, you should be able to see the Pi obtaining an IP address via DHCP and various TFTP request attempts.
Run journalctl -f --unit dnsmasq.service on your server to follow the dnsmasq log output. Again, if things are working, you should be seeing several TFTP requests here. If you see nothing, double check the network mask is specified correctly in the dnsmasq configuration, and that any firewall on the server is permitting inbound traffic to port 69 (the TFTP port).
You will see numerous “Early terminate” TFTP errors in the dnsmasq log output. This is normal, and appears to be how the Pi’s bootloader operates (my guess would be it’s attempting to determine the size of a file with the tsize extension, terminating the transfer, allocating RAM for the file, then starting the transfer again).
If cloud-init’s final phase running apt update and apt install avahi-daemon linux-modules-extra-raspi nbd-client fails (it seems to randomly on my test Pi), just login and run them manually.

At this point you should have a fully booted system with a block device as the root. All is good! We can use anything we would on a regular Pi with an SD card, including snapd, docker, or anything else. But there’s a rather serious problem waiting silently for us. If things have worked correctly, nbd-client and linux-modules-extra-raspi will have been installed. This will have re-built the initramfs, and likely also have upgraded the kernel package. If you attempt to reboot at this point, you’ll likely find the next boot fails on the rainbow screen.

Spot the problem? Two things are accessing the boot partition’s block device. First, the TFTP server (dnsmasq) is reading the boot partition via a loop device. Second, the nbd-server is serving the boot partition directly from the image.

Recall that mounts of block devices assume they have exclusive access to the underlying block device. But here, the vfat driver on the server does not have exclusive access to the boot partition. What happens if we change the boot partition on the client. Does the server notice?

Try the following experiment:

On the client: sudo touch /boot/firmware/foo
On the server: ls /srv/images/boot/jammy/foo

You should find that, even if you sudo sync on the client the foo file simply won’t show up on the server’s boot mount. The boot mount needs remounting to make it re-read the necessary blocks from the underlying image. This bodes badly for anything that writes to the boot partition, as happens when installing a new kernel, or anything that causes the initramfs to be rebuilt.

Thus, to resurrect your netbooting Pi do the following on the server:

# umount /srv/images/boot/jammy
# mount /dev/loop5p1 /srv/images/boot/jammy

Well, this sucks. Solutions?

Use NFS for the boot partition mount instead of NBD. This will work, but there’re several issues here. Firstly … it requires an NFS server! Secondly, you need to customize the fstab in the image before first boot (urgh). Thirdly you need to extract the content of the boot partition to an exported directory and point TFTP to that (which wastes space) or you can stick with the looped mount, but you must be absolutely certain that you mount the boot partition with NFS on the client, not NBD.
Try and remount the boot partition when the client reboots. This is risky as it’s almost impossible to guarantee in practice. If you think you can get clever with DHCP hook scripts in dnsmasq to remount just as the machine is booting (because DHCP comes before the first TFTP request), you’re wrong [7]. dnsmasq scripts run asynchronously and you can’t unmount a “busy” partition. It’s still possible to do this manually, but that’s boring! [8]
Write our own TFTP server that directly accesses the image without ever mounting it! Hmmm…

Join me next time for the conclusion of this series where … well, you can probably guess where this is going …

[1]	The Pi 2B, 3B, and 3B+ can also netboot, but the instructions differ on each, and this article is long enough as it is. See Network boot for more information, and drop me a question in the comments if you need more!

[2]	Your server needs to be a “real” server, or at the very least a full virtual machine, not a container. Specifically, you need to be able to create loop devices and mount images (both of which are not typically possible in a container environment).

[3] If you just want to add network boot instead of replacing the USB boot option, that’s fine too — you can use 0xf421 or 0xf241 here too. Also, the reason we’ve left SD card boot in the order is simply for safety (there are recovery methods available even if you remove it but it’s simpler to just leave it there).

[4] In mantic, the nbd module moved back to linux-modules-raspi, so you can skip installing linux-modules-extra there if you want, but you’ll still need nbd-client. In noble, I’m intending to seed nbd-client in the image, so no modifications at all should be required. It’s a pretty small install (172KB) so it constitutes little bloat, and the ability to netboot out of the box (with an appropriately configured Pi) seems to justify the change to me.

[5] If you’re cunning, you can avoid downloading the image again by extracting it from the rpi-imager cache. On Linux at least, this is under ~/.cache/Raspberry Pi/Imager/lastdownload.cache. Because the filename has lost it’s extension I usually use file to determine what it was (e.g. if it’s “XZ compressed data, checksum CRC64” I know it was .img.xz).

[6]	Unfortunately, the initramfs isn’t intelligent enough to go searching for a label-based root in an NBD device, even though technically this is valid.

[7]	Ask me how I know this!

[8]	Before anyone asks, no `-o remount` isn’t sufficient. It needs to be completely unmounted and re-mounted.

NBD Knows

2023-10-27T00:00:00+01:00

This is part 1 of an exploration of netbooting Raspberry Pis, with an emphasis on NBD over the more traditional NFS. While obviously we’re going to be looking at Ubuntu in this series, I’m hoping this should be generic enough to be useful to be useful on a variety of distributions.

The trouble I’ve seen

The obvious question is “why NBD?” or put another way “what’s wrong with NFS?”. After all, NFS is the current “best practice” for booting Raspberry Pis, and widely used, including by some very large installations (e.g. Mythic Beasts’ fantastic Pi cloud).

Ultimately, the reason is that the Network File System … is a file-system.

What a ridiculous objection, of course it’s a file-system! The clue’s in the name! And surely you want a file-system? More typically, though, you actually have a root block device, which your kernel will then transform into a file-system, rather than being given a file-system directly.

What problems arise from the lack of (access to) an underlying block device? The crux of the issue is what the kernel can assume about the file-system, most particularly whether it can assume it has exclusive access to it, and whether it can trivially access the underlying blocks of certain files.

Consider the common “root is a block device” case:

The kernel on your machine runs the transformation (the file system driver) that converts that block device into a file-system. Crucially, it can assume that it is the only entity accessing this block device (that it has exclusive access), and that for certain operations it can (with some jiggery-pokery) bypass the file-system and treat a file as a block device [1], which makes certain things nice and simple.

Need to allocate a large contiguous file, for example as a swap file? No problem! The file-system driver implements this [2], and can even give the kernel the contiguous blocks for use in the swap system (remember how block devices “make storage look like RAM”? Foreshadowing!).

Need to lock a file? The kernel knows nothing else is mounting file systems from that block device, so it knows all the locks that exist on it and doesn’t need to coordinate with anything else. Likewise, caching is simple [3]. The kernel can assume it has absolute knowledge of which blocks are dirty and need writing back because nothing else can be producing file systems from that block device.

Need some temporary file space, private to your process? Create a unique temporary file and delete it, leaving the file handle open, then use the file as a temporary store. The space won’t be reclaimed because, even though there are no links to the file, the kernel knows there’s still an open file handle.

Now the “root is a file-system” case:

In this case, there’s a block device somewhere, but your kernel has no visibility of it, and must assume that other entities can change its blocks without notification.

This interferes with several of the scenarios above. Historically, it was fatal to several of them (fallocate for cheap allocation of large files, and swap over NFS did not work). Whilst these features do work today, it’s notable that kernel support needed to be added for swap [4], and the NFS protocol extended specifically for fallocate.

Caching becomes tricky because files are complicated and messy things that have data and “meta-data”. Here’s an excerpt from the NFS mount options:

ac / noac

Selects whether the client may cache file attributes. If neither option is specified (or if ac is specified), the client caches file attributes.

To improve performance, NFS clients cache file attributes. Every few seconds, an NFS client checks the server’s version of each file’s attributes for updates. Changes that occur on the server in those small intervals remain undetected until the client checks the server again. The noac option prevents clients from caching file attributes so that applications can more quickly detect file changes on the server.

Yikes.

The practice of using unlinked temporary files also works over NFS but again … required workarounds because NFS is stateless [1] so open file handles locally do not correspond to open file handles on the server.

All this doesn’t matter too much when the portion of the virtual file-system being mounted over NFS is relatively limited, as in the common case of mounting user home directories over NFS. However, when the entire root file-system is NFS, quite a few applications can start having “difficulty”. One of the more prominent, on Ubuntu especially, is snapd which doesn’t like running on an NFS root. Like it or not, it’s a pretty integral part of the Ubuntu eco-system at this point (providing Firefox on the desktop and LXD on the server), so it would be nice to have a netboot system that can support it too [5].

Okay, we’ve established there are some issues with running NFS as root. What’re the alternatives?

Sometimes I’m up

There are several daemons and protocols that support serving block devices over the network, and they differ in some quite interesting ways:

iSCSI

iSCSI can be trivially summarized as: SCSI commands over TCP/IP. This is by far the most popular method of serving block devices. It has the advantages that it doesn’t require expensive equipment (unlike Fibre Channel, its major competitor in the enterprise), and can be routed over multiple networks (as it’s built on IP). However, it’s not entirely trivial to configure (it is very flexible, but for our purposes here I wanted something simpler to start out with).

AoE

ATA over Ethernet. Like iSCSI, the name says it all. The client simply passes ATA disk commands over Ethernet to the server. Note that this does indeed run over Ethernet only (layer 2), not TCP/IP so it’s not routable over the Internet, only local networks. It’s very simple to set up (moreso than iSCSI), but two things made me skip it here. The first is that routing over layer 2 may be perfectly sufficient for many use-cases, but there’s several others where it’ll be a limiting factor.

The second is that aoetools, the package for AoE in Ubuntu, is an extremely “mature” package. Specifically, the upstream version in Ubuntu hasn’t changed since Xenial (16.04, 7 years ago at the time of writing). That isn’t to say it’s bad or entirely unmaintained, but there’s no active work on it as best as I can tell (and that usually doesn’t bode too well from a security point of view).

NBD

Network Block Devices differs in that doesn’t implement the commands from an existing disk protocol (SCSI or ATA). Instead, it uses its own protocol which, at it’s core, is almost laughably simple.

It also operates over TCP/IP, avoiding the layer 2 limitations of AoE, and is an actively maintained project which optionally includes facilities for TLS encryption. I won’t be using those here, but you’ll have some options for better security down the line.

Simplicity is another argument in favour of serving block devices instead of file-systems. Compare the two scenarios:

They don’t look that different but consider what serving files over a network means, versus serving block devices. What operations can be performed against a file? Opening, closing, reading, writing, truncating, locking, linking, touching, the list goes on. All this must be handled by the protocol to implement even a bare-bones network file-system. The bare-bones case for block devices (as noted above) is radically simpler.

The baseline portion of the NBD Protocol consists of commands to “read some bytes”, “write some bytes”, and “disconnect”. That’s it. There are some other commands which may optionally tell the server to trim, flush, or cache blocks, and some other messages for option negotiation at connection time, but the core of the protocol really is that simple. I like simple.

Sometimes I’m down

So far, we’ve looked at how the root file-system will be handled when netbooting. However, the root file-system only matters after the Linux kernel has started. How do we obtain the Linux kernel itself (and other sundry boot resources) at system start? This involves neither NFS nor NBD.

When netbooting, the Pi first requests an IPv4 address from the local network via DHCP. The local router responds with a DHCP offer, and our netboot server tacks a (minimal) PXE boot menu on the end suggesting where the client may find a TFTP server for booting.

This is typical jargon-laden nonsense, so let’s translate a bit:

Jargon	Raspberry Pi	Router	Netboot Server
DHCP DISCOVER	“Hello? Can anybody give me an IPv4 address? By the way, I’m a netboot client”
DHCP OFFER		“Sure, would you like to be 192.168.0.200?”
DHCP Proxy Option 43 PXE Boot Menu			“By the way, for ‘Raspberry Pi Boot’ see TFTP server at 192.168.0.4”
DHCP REQUEST	“Okay, I’d like to be 192.168.0.200, please?”
DHCP ACK		“Right, you are 192.168.0.200 for the next 12 hours, see me again after that”
TFTP RRQ	“Can you send me the content of `SERIAL/start.elf`, please?”
TFTP OACK			“Sure, it’s going to be 225065 bytes long, and I’ll send it in chunks of 1468 bytes”

The important things to note here are as follows:

We need a DHCP server. This is pretty much taken-as-read on any network these days.
We need a netboot server with a DHCP proxy and a TFTP server. This is fairly simple. Any Ubuntu server can install dnsmasq (if it hasn’t already) to obtain this.
We need the Raspberry Pi’s serial number.

This last point may seem a bit strange, but it’s because TFTP is, as the name suggests, trivial. The protocol provides no means for the client to identify itself to the server, so how are we to know which boot partition we should read files from?

Identification by MAC address is one possibility [6], but that’s not an option for us (unsupported by the TFTP server in dnsmasq). Instead we rely upon the Pi identifying itself by the sequence of files it initially attempts to request. When netbooting, a Pi (more specifically a Pi 4 or later) will attempt the following sequence of files [1]:

SERIAL/start4.elf, e.g. 1234abcd/start4.elf
SERIAL/start.elf, e.g. 1234abcd/start.elf
start.elf

If the bootloader finds files with the SERIAL/ [7] prefix, all subsequent requests will also have that prefix, allowing us to easily determine which OS image files should be served from.

The Pi then proceeds to request (over TFTP):

The rest of the tertiary bootloader (e.g. start4.elf, fixup4.dat, and its configuration files like config.txt).
The device-tree for the specific board (e.g. bcm2711-rpi-4-b.dtb) and any overlays required by the configuration, or by devices that are plugged in (e.g. overlays/dwc2.dtbo, overlays/ov5647.dtbo).
The kernel and initramfs requested by the configuration, and its command line (e.g. vmlinuz, initrd.img, cmdline.txt).

With all this loaded into appropriate locations in memory, the bootloader hands over to the Linux kernel, which mounts the initramfs as its initial root file-system, and launches the /init binary within it.

In the case of Ubuntu, this is the usual initramfs you’ll find on pretty much any Ubuntu installation. It’ll search the kernel command line for the “real” root device, attempt to mount it, and “pivot” the root that mount.

For the NFS case, the kernel command line would include something like nfsroot=server:/exports/ubuntu-jammy root=/dev/nfs. For the NBD case, the kernel command line would include something like nbdroot=server/ubuntu-jammy root=/dev/nbd0p2.

At this point you should have a basic understanding of the Pi’s netboot process. Let’s explore what the server side configuration for TFTP, NFS, and NBD can look like from a high level. We’ll get into specifics in the next post; this is just to give you an idea of the considerations and possibilities involved.

Glory, hallelujah!

A typical server TFTP configuration (whether subsequently NFS, NBD, or anything else) is to have the boot partition of an OS image unpacked or mounted under a particular path, and then make a symlink from the Raspberry Pi’s serial number to that path. Re-writing the symlink is then enough to switch which Pi boots which image.

While this much does not differ between the NFS and NBD cases, there is the question of how to make the boot files available.

In the case of NFS, as the server is serving a file-system it is typical to simply unpack the entire OS image, both the root and boot file-systems, into a directory and call that the “image” that is served. The symlink for the Pi’s serial number then points to the /boot/firmware directory within the unpacked image.

For example, if we have two OS images, ubuntu-jammy and ubuntu-mantic, and two Raspberry Pis with serial numbers 1234abcd and 4567cdef we might lay out our files like so:

/
├─ …
├─ srv/
│  ├─ ubuntu-jammy/
│  │  ├─ bin/
│  │  ├─ boot/
│  │  │  ├─ firmware/
│  │  │  └─ …
│  │  └─ …
│  ├─ ubuntu-mantic/
│  │  ├─ bin/
│  │  ├─ boot/
│  │  │  ├─ firmware/
│  │  │  └─ …
│  │  └─ …
│  └─ boot/
│     ├─ 1234abcd->/srv/ubuntu-jammy/boot/firmware
│     └─ 4567cdef->/srv/ubuntu-mantic/boot/firmware
└─ …

The two images are completely unpacked under /srv/ubuntu-jammy and /srv/ubuntu-mantic, then symlinks under /srv/boot point to the /boot/firmware directories of the unpacked images. Our TFTP server is configured to serve /srv/boot, and our NFS server to export /srv.

The advantages are that it’s a relatively simple setup, requiring no special mounts on the server side, and that (assuming all unpacked images are in the same file-system on the server), all available space is shared between all netbooting Pis. The disadvantage (other than it being an NFS boot setup) is that all available space is shared between all netbooting Pis so one Pi can use up all available space for everyone, unless additional steps like quotas are taken.

In the case of NBD, which we’ll explore more in the next post, I would suggest a simple setup is to leave the OS image as it is (in a file) and simply expand the file to the desired size. A loop device can be created for the image file, with a partition scan to find the boot partition, which can then be mounted. I would still recommend using a symlink to point to the mount for ease of changing later.

Let’s consider the scenario from before (two images, jammy and mantic, two Pis with known serial numbers) in this setup:

/
├─ …
├─ srv/
│  ├─ ubuntu-jammy.img
│  ├─ ubuntu-mantic.img
│  ├─ mnt/
│  │  ├─ ubuntu-jammy/  (mount of ubuntu-jammy.img partition 1)
│  │  └─ ubuntu-mantic/  (mount of ubuntu-mantic.img partition 1)
│  └─ boot/
│     ├─ 1234abcd->/srv/mnt/ubuntu-jammy
│     └─ 4567cdef->/srv/mnt/ubuntu-mantic
└─ …

The two images are simply placed under /srv. Loop devices are created of each, and the first partition mounted under appropriate directories under /srv/mnt. Symlinks under /srv/boot link Pi serial numbers to the appropriate mount. Our TFTP server is configured to serve /srv/boot as before, and our NBD server is configured to export each /srv/*.img file as a block device.

The advantages are that this is a trivial setup (the image doesn’t even need unpacking), and that each image is necessarily limited to its own storage. The disadvantage is that storage isn’t shared between images. However, we saw in the prior post that there’s all manner of things we can do with block devices, so we’ll explore some possibilities there in a future post too.

I should warn there’s also a nicely hidden, but quite serious issue with this set up which we’ll look at next time when we actually build it (and then fix it … obviously). Anyway, that’s all for now!

[1]	(1, 2, 3) This is a gross over-simplification (or an outright lie), but serves to make the point.

[2]	Alright, some of them don’t or only recently added the functionality, or it’s still experimental (cough btrfs), but they’re usually the file-systems that have their own rules for block mapping.

[3]	Caching is never simple.

[4]	No other file system has a “make swap work on this file-system” kernel config item

[5]	I’m told Docker overlays also have issues with NFS but it’s not something I’ve played with directly and I’ve not had the time to verify it that for this article.

[6]	Though not for “security” (MACs can be trivially spoofed)

[7]	The `SERIAL` portion is the serial number of the Pi (which can be found at the end of the output of `cat /proc/cpuinfo`), in lower-case hexidecimal format. If the serial number is longer than 8 characters, only the last 8 characters are used.