It's Pi all the way down... - luks

Building Blocks

2023-10-24T00:00:00+01:00

In an earlier series of posts (Playing with Blocks 1, 2, 3, and 4) we had a look at customizing Ubuntu images for the Raspberry Pi by playing around with various block device transformations. However, we never covered what a block device actually is. This is going to be a vaguely important topic in an upcoming series of posts on NBD booting of Ubuntu, so I figured some brief coverage of the topic is warranted.

Brief.

Who am I kidding?

Blockage

In the UNIX world a block device is a very simple concept: it’s a storage device which provides access to its contents as an ordered sequence of identically sized blocks. That’s it. No files, no directories, no permissions, nothing more than a numbered sequence of blocks all with the same size. Typically, the blocks are some whole multiple of an underlying device’s sector size, which over most of computing history has been 512 bytes. However, the user may access block devices byte-by-byte if they wish:

For instance, a read of a few bytes (illustrated on the left in green) may read a full 512-byte sector, throw away the bytes outside the requested range, and return that which was requested [5]. A write of a few bytes (illustrated in the middle in red) is a bit more complex; the kernel needs to read the full sector from the underlying device, change those bytes we requested to write in the middle, then write the whole sector back to the underlying device [6].

Crucially, block devices are random access. We can access any block, anywhere, at any time, without having to access all blocks prior to it. In other words, block devices make storage look and “feel” like RAM. This is as opposed to the “byte-stream” view of files which owes more to the tape drives of old (“seeking” the head to a particular location, then reading or writing from that point).

Transform!

The simplicity of block devices leads to them being wonderfully flexible things that can often be converted into new block devices in useful ways. Probably the most basic block device transformation that everyone is familiar with is “partitioning”, which simply sub-divides one block device into multiple new ones (minus some header space). This transformation effectively just re-numbers the blocks but doesn’t change their content or ordering.

More complex transformations are also available.

The LVM sub-system (explored previously) gathers multiple block devices into a “volume group”. The volume group can then be used to produce other block devices called “logical volumes”. These are similar to partitions, but have no need to be contiguous on the underlying block devices, and can even span multiple underlying devices.

A drive (sda) is divided into two partitions (sda1, sda2) the latter of which is further divided into three block devices with LVM (root, home, and swap). The home device has been expanded at some point in its history as its storage is non-contiguous

Speaking of multiple underlying devices, RAID systems (like mdadm) consume multiple block devices and produce a single block device. The blocks in the produced device are duplicated to the underlying block devices (in the simplest mirror case; more complex transforms are involved in things like RAID5). Corruption or even loss of one of the underlying block devices, as in the failure of a drive, can be entirely hidden from the consumer of the produced block device.

Four drives (sda, sdb, sdc, and sdd) are partitioned such that each has a small initial partition (sda1 etc.) and a larger partition (sda2 etc.); the small initial partitions are combined as a RAID1 mirror into a boot block device. The larger partitions are combined as a RAID10 device (a stripe of mirrors) as a root device.

The LUKS encryption mechanism (also explored in previous posts) consumes one block device and produces a slightly smaller one (minus the size of the LUKS header) that scrambles (and unscrambles) the content of blocks as they are written (and read).

The Hierarchy

A file-system is just another kind of transformation of a block device; one which converts the flat sequence of blocks into a tree-like hierarchy of files (optionally with attributes like ownership, groups, permissions, and all the rest of that gubbins).

File systems can be relatively simple affairs, like the ancient but ubiquitous FAT which has no concept of file ownership, UNIX modes, and no symlinks. Alternatively they can be complex, like the common ext4 file-system which has journaling data recovery, the full suite of UNIX ownership, attributes, and linkage, and numerous tunable options.

Ultimately, the file-systems produced from various block devices are all grafted together (“mounted” in UNIX parlance) onto a “virtual” file-system which is what all userland processes (and you, the user) generally consider “the file-system”; the place you find all the files on your computer.

Two partitions (sda1 and sda2) are formatted as FAT and ext4 respectively. The vfat driver converts the sda1 block device into a typical boot file-system for the Pi (containing bootcode.bin, start4.elf, config.txt, etc). The ext4 driver converts the sda2 block device into a typical Linux root file-system (containing bin, etc, lib, usr, sbin directories). The virtual file-system then “mounts” the ext4 hierarchy under / and the FAT hierarchy under /boot/firmware

A common Linux storage layout is shown above. The drive (/dev/sda) has two partitions. The first partition (/dev/sda1) is formatted as FAT, which the kernel’s vfat driver transforms into a file-system. The second partition (/dev/sda2) is formatted as ext4, which the ext4 driver transforms into another file-system. The kernel has mounted the ext4 file-system as the root of the “virtual” file-system, and the FAT file-system under the /boot/firmware mount-point.

Holy CoW!

Some modern file-systems provide a copy-on-write facility (most notably btrfs and zfs). This allows rapid (near instantaneous) cloning of an entire file-system hierarchy. The clone doesn’t initially take any storage of its own, and reads of the clone actually go to the original files. However, writes to the clone allocate new storage for the changed content.

However, the facility is not limited to modern file-systems. LVM also provides such a facility, initially intended to provide snapshots of file-systems at a point in time (for backup, testing, or other purposes).

The “root” block device contains a number of filled blocks in green, while the “snap” block device has an equivalent number of “empty” blocks. Two reads, indicated by arrows, are made against the “snap” device but pass through to the underlying “root” device. Below this an equivalent layout is show, but now one of the “filled” blocks is “red” indicating a change in content. Above it, the equivalent block in the “snap” device is green indicating it contains the original content. A read against this block comes directly from the “snap” device instead of passing through to the underlying “root” device.

The illustration above shows a “root” block device, presumably containing a root file-system. We create a snapshot of the “root” block device called “snap”. Initially, the “snap” block device is empty, and reads to it pass through to the underlying “root” block device. However, when a block is the “root” device is written to, its original content is first copied to the “snap” device. Subsequent reads of this block in the “snap” device will go to this copied block instead. Further writes to the same block in the “root” device are ignored because the “snap” device only cares about the content at the point in time it was created.

Note that one of the beauties of this system is that we don’t care what file-system is on the “root” device. It could be FAT, ext4, XFS, even (redundantly) one of the more modern systems that supports this internally. However, a limitation of this snapshot system is that we need to allocate space for it up front [8]. Can we come up with something in which the snapshot allocates blocks dynamically?

At the top, the “image” block device (read-only) contains a number of filled blocks in green. Above it, a clone, “clone1”, exists with the same number of blocks but all are empty. A read of two blocks in “clone1”, indicated by arrows, passes through to the corresponding blocks in “image”. Below, the one of the blocks in “clone1” is now red indicating it has been changed from the original content in “image”. Another arrow, indicating a read of that block now comes straight from “clone1” and doesn’t pass through. Another clone, “clone2”, is below it with entirely empty blocks. A read of the same block which is changed in “clone1”, passes through “clone2” to the underlying “image” device.

The illustration above shows “thinly provisioned” snapshots in LVM. This operates a little like the regular snapshots in reverse. An underlying block device called “image” presumably contains an OS image. We have created a “thinly provisioned” snapshot of this called “clone1”. Reads of an unchanged block pass through to the underlying (read-only) “image”. Writes to blocks in “clone1” occur within its storage, with subsequent reads to that block no longer passing through. Finally, we create another clone, “clone2”, which reads the original block from the underlying “image”.

It’s worth noting that most of the blocks in the clones are empty. Provided they stay that way (i.e. the clones don’t change too many blocks of the underlying image), there’s little sense in actually allocating them. This is the “thin” in “thinly provisioned”. The blocks of the clones are allocated on demand so a clone initially takes up no space. This allocate-on-demand facility can implement one form of “over provisioning” of storage. Each clone thinks it has all the storage allocated to the original image, but there’s no need to have all the storage available (unless the clones grow to require it, naturally).

Hopefully this gives some idea of how modern cloud systems can spin up clones of an OS image nearly instantaneously, and how they can seemingly provide many gigabytes of storage to containers or VMs, without necessarily having installed all the storage they apparently provide.

Layer Cake

Finally, a brief illustration of just how silly this can get. Here’s a diagram of a storage layout I’ve used on servers in the past:

A spiders web of storage. Four boxes represent four hard at the bottom. Above this a larger box represents the RAID device built from the drives. Above this a hexagonal volume group is built from the large RAID device. From the volume group spring numerous block devices, for “tmp”, “root”, “home”, and “images”. From the first three spring regular file-systems, but from “images” spring several more block devices named “base” and from that “munin”, “docs”, and “git” (OS images running in containers).

Four hard drives exist, each containing a small boot partition and a large partition occupying the rest of the capacity
The four boot partitions are combined with a RAID1 mirror into a boot block device, “md1” [7].
The “md1” block device is formatted as FAT and contains the “/boot” hierarchy.
The four large partitions are combined with RAID5 into a large block device, “md2” (RAID5 implies it has the capacity of n-1, i.e. three, of the underlying devices, and can survive the loss of any one device).
The “md2” device is consumed by the “raidvg” volume group from which are produced various block devices: the “root”, “home”, “tmp”, and “images” volumes.
“root”, “home”, and “tmp” are formatted with a mix of file-systems, ext4 and XFS.
The “images” volume is a “thin pool” from which the “base” volume is derived.
The “munin”, “docs”, and “git” volumes are all thinly provisioned snapshots of the “base” volume.

This may seem like a complicated stack of transformations, but consider that in essence most amount to little more than re-numbering and re-ordering of blocks (the RAID5 transform is a little more involved, admittedly). Hence, for very little performance cost, we’ve got redundancy (RAID), flexible volume creation system with snapshotting (LVM), and copy-on-write cloning for VMs or containers launched on the server (thin provisioning).

By this point you should have a reasonable understanding of what a block device is, and what can be accomplished with the various transforms that are available for them under Linux. Next time we’ll take a look at the issues with netbooting from NFS and how NBD (or block devices in general) can mitigate some of these issues.

[5]	In practice it’s unlikely the bytes will actually be thrown away; the sector requested will be stored in a cache, in case a future read requests more bytes from the same sector (very likely when reading through a file sequentially).

[6]	Again, the changed block will wind up in a cache so future modifications can skip the initial read step.

[7]

Simple RAID1 mirroring is used to ensure that the BIOS can use any of the drives as its initial boot device without caring about the RAID layout (which it doesn’t understand). Note: This carries risks as it means the BIOS and bootloader must treat the boot partition as strictly read-only (and that’s not necessarily guaranteed).

[8]	The snapshot doesn’t require all the space to be allocated, as the diagram suggests. In fact, assuming the underlying device changes infrequently, it’s common to only allocate ~5% of the origin’s storage. However, you do still need to provide some up front.

Playing with Blocks - ReLUKS

2023-04-20T00:00:00+01:00

See Also: LUKS, LVM+LUKS.

We’ve looked at using LUKS on the Ubuntu Pi desktop images before but it was a rather painful process primarily as it involved shrinking the root partition. This in turn meant that a separate machine (or at the very least a Pi booting from something other than its SD card) was required to perform the changes.

Can we do better? In particular, can we reduce the requirements to just a single machine (the target Raspberry Pi) with a single storage medium? Thanks to a relatively minor change in the just-released Ubuntu Lunar (23.04) Pi desktop image, I think we can!

When you wanna go do it

You’ll need the following for any scenario:

a Raspberry Pi 4 or 400
a micro-SD card (or USB-attached boot drive)
a monitor
a keyboard and a mouse

If you’re trying to do this on a single machine (not flashing your storage device on a separate machine), you’ll also need:

a relatively recent boot firmware
an Ethernet connection

Newly purchased Pis (as of 2023) should all have sufficiently up to date firmwares. Older Pis may require an EEPROM update which can be accomplished under Ubuntu (or RaspiOS) by running sudo rpi-eeprom-update -a and rebooting if an update is shown as available.

Hit me with those laser beams

Start up your Raspberry Pi. The monitor should show the red and white Raspberry Pi bootloader screen, with the prompt “Press and hold <SHIFT> key to stop boot and start net install”. Follow this instruction and it should change to “If not done so already, insert ethernet cable” (this capability only works over Ethernet, not WiFi as the bootloader can only configure the former).

After a short download, you should find yourself at a full-screen version of the rpi-imager. From here you can select Ubuntu as you normally would in the desktop version of the software:

Click “Choose OS”
Select “Other general-purpose OS”
Select “Ubuntu”
Select “Ubuntu Desktop 23.04 (64-bit)”
Back at the main screen, click “Choose Storage” and pick your boot medium (which should be the only drive connected to the Pi at this point)
Click “Write”

The utility should prompt to ensure you want to overwrite everything and anything on the storage device and, once confirmed, you should find it writing to the storage medium.

Note

Do not be alarmed if there is an extremely long pause in the writing process. The Ubuntu images are heavily compressed, and the “slack space” baked into the image (which may be excessive) compresses extremely well. This can lead to a potentially long pause in the progress bar while a few bytes in the compressed image expand into gigabytes of written data.

Make making it your intention

Once the OS is written to the boot medium (and verified), the Pi should reboot into Ubuntu Desktop. You’ll need to run through the initial setup wizard, configuring your locale and user details.

After the initial setup wizard, login to your new user. We now need to take a series of steps which may seem premature and in some cases a bit odd. Start a terminal and perform the following:

$ sudo -i
# apt install cryptsetup-initramfs
# echo "rootfs  $(findmnt -n -o SOURCE /)  none  luks,discard" >> /etc/crypttab
# sed -i -e 's,^LABEL=writable,/dev/mapper/rootfs,' /etc/fstab
# sed -i -e 's,LABEL=writable,/dev/mapper/rootfs,' /boot/firmware/cmdline.txt
# reboot

In order, these commands:

Switch to the root user (everything we’re going to do requires root access and I can’t be bothered re-typing sudo).
Re-install cryptsetup. I say “re-” because this package is actually part of the base image, but the initial setup decides it isn’t required and removes it at the end of the setup. Doh!
Create the crypttab entry for the rootfs.
Re-write the source of the root mount in fstab.
… and the same in the kernel command line.
Reboot the machine.

If you want to double check things, assuming your boot device is an SD card (/dev/mmcblk0) your crypttab, fstab and cmdline.txt files should look like this:

root@miss-piggy:~# cat /etc/crypttab
# <target name> <source device>         <key file>      <options>
rootfs  /dev/mmcblk0p2  none  luks,discard
root@miss-piggy:~# cat /etc/fstab
/dev/mapper/rootfs      /       ext4    discard 0       1
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1
root@miss-piggy:~# cat /boot/firmware/cmdline.txt
zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=zstd dwc_otg.lpm_enable=0 console=tty1 root=/dev/mapper/rootfs rootfstype=ext4 rootwait fixrtc quiet splash

Hit me, hit me!

At this point you may notice that the boot seems to be taking rather a long time. In fact, it’s going to fail because the rootfs device we told the kernel to go looking for doesn’t exist yet. But that’s okay! Be patient and let it fail; the initramfs still gets loaded and it’s got all the tools we need to create it.

Once you find yourself at a command prompt (with a notice complaining that it couldn’t mount the root file-system), do the following:

(initramfs) rootdev=$(readlink -f /dev/disk/by-label/writable)
(initramfs) cryptsetup reencrypt --encrypt --reduce-device-size 16M --cipher xchacha12,aes-adiantum-plain64 $rootdev
(initramfs) cryptsetup open $rootdev rootfs
(initramfs) exit

Create the rootfs device. This will prompt you to enter “YES” (for confirmation you really want to do this), and for the password to unlock the encryption key for the device (twice); don’t forget this!.
Go make coffee; or twiddle your thumbs for an excessively long period of time while it encrypts the entire device, including all the empty space. On the plus side, after a minute it should give you a reasonably accurate estimate of how long it’s going to take to encrypt the device, so you can set a timer and come back when it’s done.
Open the newly created rootfs device (this will prompt you once more for the password) and exit to continue the boot.

Re-LUKS

Once you’re logged into the desktop once more, you have one final task. Re-generate the initramfs now that the rootfs device exists:

$ sudo update-initramfs -u

Done! At this point all future boots should prompt you for the password to unlock the encrypted root file-system.

Obviously in an ideal world, this could all be accomplished from the installer, but there’s a slight wrinkle in that with the lunar release. The first time setup application on the Pi desktop images is oem-config which is derived from the venerable ubiquity installer. As you may have noticed in the lunar release notes the installer is being replaced by a new one based on subiquity.

However, this only applies to those platforms which need the “full fat” installer, i.e. the PC where you boot off some medium and then install to another (the internal drive). It doesn’t apply to Ubuntu’s pre-installed desktop images (like the Pi ones) yet. There seemed little point in pouring effort into adding this to ubiquity this cycle, only to see it thrown out during the next cycle when the subiquity based installer comes to replace oem-config.

Hopefully in a future cycle we can integrate all this and have a slightly friendlier experience, but for now at least it’s achievable without requiring a separate machine.

Playing with Blocks - It’s All Connected

2022-09-08T00:00:00+01:00

In the last two articles we looked at LVM and LUKS for some enhancement of the basic storage of the standard Ubuntu Pi images. This time around we’ll combine both of those previous articles to make an image in the same way that the Ubuntu Desktop for PC images install when they’re given free reign to overwrite the entire contents of the drive.

Specifically, this means we aim to wind up with:

A FAT boot partition (no difference there), 512 MB in size
A second partition taking up the rest of the card containing
- An encrypted LUKS container, containing
  - An LVM physical volume (PV) belonging to
    - An LVM volume group (VG) called “pivg” containing
      - An LVM logical volume (LV) called “root” taking up some reasonable portion of the disk space (8 GB in this example)

A diagram illustrating the SD card layout showing: a large bar representing the whole SD card /dev/mmcblk0 (N GB). Beneath that, two bars: a small one on the left for the system-boot partition /dev/mmcblk0p1 (512 MB), and a large one on the right for the remaining partition /dev/mmcblk0p2 (N GB - 512 MB). Beneath the large right-hand bar, a bar representing the LUKS encrypted container (again, N GB - 512 MB). Beneath that, two more bars: a tiny one on the left representing the LUKS header (16 MB), and a much larger one on the right representing the LVM PV /dev/mapper/rootfs within the LUKS container (N GB - 512 MB - 16 MB). Lower down, a dotted outline of the same size representing the LVM VG /dev/pivg. Finally, beneath that, a smaller bar representing the LVM LV /dev/mapper/pivg-root (8 GB).

Note

One thing to note is that I’ve only tested this with the Ubuntu 22.04 (jammy) server for pi image. Notably, jammy has cryptsetup within the default initramfs on the image. I would not necessarily be confident that this works on earlier releases.

Three (?) Rowdy Layers

Rather than go through all the individual commands, which would result in this post looking like a foreword followed by the dump of a shell script, I’ll go through the high level procedure because the detail is almost exactly the same as the original articles but for a few minor differences and some deliberate omissions.

Firstly, we’ll deal with the content of the first article and set up LVM:

Follow the first article instructions until it’s time to create the PV (with pvcreate):
- Plug in the SD card
- Unmount any auto-mounted partitions
- Repartition with gdisk

At this point we need to create the PV with 16MB less space than is available in the second partition, to leave a little room for the LUKS header around the PV:

From the gdisk session, you should have a printout of the new partition table; if you don’t, run sudo gdisk -l /dev/mmcblk0 (assuming /dev/mmcblk0 is your SD card)

For the second partition, calculate the number of sectors it occupies (which is simply end sector minus start sector) and then subtract 32768 (each sector is 512 bytes, so 32768 is 16MB). For example, on a 32GB SD card:

$ sudo gdisk -l /dev/mmcblk0
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/mmcblk0: 62333952 sectors, 29.7 GiB
Model: STORAGE DEVICE
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): A9BDFD97-F2A5-4419-9A83-B0160820FDBB
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 62333918
Partitions will be aligned on 2048-sector boundaries
Total free space is 4061 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1048576   511.0 MiB   0700  system-boot
   2         1050624        62333918   29.2 GiB    8E00  lvm
$ PV_SECTORS=$((62333918 - 1050624 - 32768))
$ echo $PV_SECTORS
61250526

Proceed to creating the logical volume, but explicitly specify the number of sectors to use when creating the PV with --setphysicalvolumesize and the number of sectors with the suffix “s”:

$ sudo pvcreate --setphysicalvolumesize ${PV_SECTORS}s /dev/mmcblk0p2
Physical volume "/dev/mmcblk0p2" successfully created.
$ sudo vgcreate pivg /dev/mmcblk0p2
Volume group "pivg" successfully created
$ sudo lvcreate --size 8G --name root pivg
Logical volume "root" created.

Finish the first article’s instructions:
- Unpacking the image
- Creating a loop device for the image
- Transferring the content of the image partitions to the card
- Updating the boot command line and fstab to point to the “new” root device
- Unmount the root and boot partitions, and disable the VG
- Don’t eject the SD card, we still need to do some surgery!

LUKS Dujour

Now it’s time to move on to the second article and set up the encrypted LUKS container around the physical volume (PV):

Skip everything up to the encryption step:
- No need to download another image
- No need to boot the image yet
- No need to unmount stuff (we already did that)
- No need to shrink partitions (the root partition hasn’t been auto-expanded because we haven’t booted anything, and we’ve pre-shrunk the PV containing the root)

Now it’s time to encrypt the second partition containing the PV:

First the encryption; this is exactly as in the second article:

$ sudo cryptsetup reencrypt --encrypt --reduce-device-size 16M --cipher xchacha12,aes-adiantum-plain64 /dev/mmcblk0p2

WARNING!
========
This will overwrite data on LUKS2-temp-a823bd67-a632-41cb-955e-8776b8b3fb5b.new irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for LUKS2-temp-a823bd67-a632-41cb-955e-8776b8b3fb5b.new:
Verify passphrase:
Finished, time 36:53.687, 29915 MiB written, speed  13.5 MiB/s

Next we open the encrypted container:

$ sudo cryptsetup open /dev/mmcblk0p2 rootfs
Enter passphrase for /dev/mmcblk0p2:

Then we re-scan for physical volumes (PVs) and re-activate the pivg volume group (VG) we created earlier; this isn’t in the original article but is necessary as we’ve altered the mapper configuration:

$ sudo pvscan
  PV /dev/mapper/rootfs   VG pivg            lvm2 [29.20 GiB / 21.20 GiB free]
  Total: 1 [29.20 GiB] / in use: 1 [29.20 GiB] / in no VG: 0 [0   ]
$ sudo vgchange -ay pivg
  1 logical volume(s) in volume group "pivg" now active

We skip over all the resizing bits because we’ve already established the size of the root volume, and just mount the root file-system (we don’t need to mount the boot file-system). This is, again, slightly different to the original article because the root file-system is a mapper device, pivg-root as in the first article:
```
$ sudo mkdir -p /mnt/root
$ sudo mount /dev/mapper/pivg-root /mnt/root
```

Then a bit (but only a bit) of configuration manipulation:

Add our entry to /etc/crypttab. Please heed the advice in the original article regarding the source device here; this is the source device as seen by the Pi:

$ cat /mnt/root/etc/crypttab
# <target name> <source device>         <key file>      <options>
$ echo "rootfs  /dev/mmcblk0p2  none  luks,discard" >> /mnt/root/etc/crypttab
$ cat /mnt/root/etc/crypttab
# <target name> <source device>         <key file>      <options>
rootfs  /dev/mmcblk0p2  none  luks,discard

Skip the bits about editing fstab and the boot command line; we’ve already done everything we need to for those. Just unmount root file-system, and close the encrypted container:
```
$ sudo umount /mnt/root
$ sudo cryptsetup close rootfs
```

Now just follow the second article to the end:
- Eject the card and boot it on your Pi
- There’s a long delay while the initial root mount fails
- In the emergency prompt that appears, open the encrypted device and exit the prompt
- Once booted, update the initramfs
- Reboot to make sure everything works!

Future Cases

Congratulations! At this point you should have a setup eerily similar to the one I use on my main development Pi, albeit that one boots off a large SSD instead of an SD card. Future things worth exploring with this setup:

How to get plymouth prompting nicely for the password; plymouth’s always been on the Pi server images but for some reason it doesn’t work properly (unlike the Pi desktop images where it’s happy). I had a cursory look into this a couple of cycles ago, but didn’t get anywhere at that time.
How about ditching password encryption and using a key on removable storage. For instance, if booting off SD card, how about using a USB key with the encryption key. Or for my favoured setup where I boot off a USB3 attached SSD drive, how about using an SD card with the key?
With this configuration I usually set up LXD with an LVM thin-pool for its default storage. This usually requires taking a crow-bar to LXD and forcing it to use the thin-pool (with lvm.vg.force_reuse; LXD doesn’t like working with a non-empty VG for reasons I’ve never full understood, but at least it’s possible). Let me know in the comments if it would be useful to have another article covering this!

Playing with Blocks - LUKS

2021-10-19T00:00:00+01:00

See Also: LVM+LUKS

Last time we had a look at using LVM on our root storage. This time around it’s LUKS‘ turn: we’re going to make an Ubuntu Desktop for Raspberry Pi card, with the root file-system encrypted with LUKS. Thankfully this is much simpler affair as LUKS can encrypt, decrypt, or for that matter, re-encrypt a block device non-destructively (even with interruptions in the middle!).

As Easy As …

We get off to the simplest possible start. You’ll need a machine running Ubuntu and some form of Pi-compatible storage (an SD card, or some form of USB mass storage).

Install the Raspberry Pi imaging utility (or use sudo snap install rpi-imager for the snap version if you’re on an architecture other than amd64)
Grab an SD card or some form of USB storage (if you’re intending to use the desktop image, I’d recommend something fast like an SSD)
Install the Ubuntu 21.10 desktop image (substitute server if you wish) on the storage with the imaging utility (I won’t bother detailing this process because, frankly, the pi’s imaging utility is about as simple as you can possibly get!)
Boot the installed image on your Pi

Wot, no cryptsetup?

It might seem a bit odd that we’re jumping straight to booting the image, but there is a reason for this: on earlier versions of Ubuntu, the cryptsetup-initramfs package isn’t installed by default. On Impish it is installed by default, but the initramfs that ships with the image still doesn’t incorporate the cryptsetup binary that’s actually necessary. Hence, we need to boot it at least once and re-generate the initramfs to include it.

Get to a usable terminal (if you’re on the desktop image just go through the initial setup), then install cryptsetup-initramfs (as mentioned, on Impish this shouldn’t be necessary but best to make sure) and re-generate the initramfs.

$ sudo apt install cryptsetup-initramfs
... usual apt stuff ...
$ sudo update-initramfs -u
... lots of flash-kernel output ...

That last step will take several minutes; here’s some hints to read first if you want to speed things up a bit. Now shut down the Pi and plug the SD card (or USB drive) back into your original machine.

Setec Astronomy

Time to perform the actual encryption. Below I’m assuming that your SD card (or whatever target storage you’re using) appears as the /dev/mmcblk0 on your machine. Adjust these instructions according to your setup!

We’re going to unmount the partitions of the SD card (or other storage) you’ve just inserted (this is only necessary if your Ubuntu machine is a desktop machine which will auto-mount the card), force a file-system check of the root partition, and shrink that partition to its minimal size:

$ sudo umount /dev/mmcblk0p1
$ sudo umount /dev/mmcblk0p2
$ sudo fsck -f /dev/mmcblk0p2
fsck from util-linux 2.34
e2fsck 1.45.5 (07-Jan-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
writable: 154688/557520 files (0.2% non-contiguous), 1507014/2229303 blocks
$ sudo resize2fs -M /dev/mmcblk0p2
resize2fs 1.45.5 (07-Jan-2020)
Resizing the filesystem on /dev/sda2 to 1881115 (4k) blocks.
The filesystem on /dev/sda2 is now 1881115 (4k) blocks long.

Now for the actual encryption. First, pick a decent passphrase, then pick a decent encryption cipher. Below I’m going to use the Adiantum cipher as this performs reasonably well on the Pi. It’s well above the speed of most SD cards, but be aware it will probably hurt the performance of SSDs. Use cryptsetup benchmark --cipher xchacha12,aes-adiantum if you want an idea of its performance; compare this to the output of cryptsetup benchmark which runs through several of the more common algorithms.

At any rate, expect this next bit to take some considerable time!

$ sudo cryptsetup reencrypt --encrypt --reduce-device-size 16M --cipher xchacha12,aes-adiantum-plain64 /dev/mmcblk0p2

WARNING!
========
This will overwrite data on LUKS2-temp-dd54103b-d96c-4d1d-9e23-e3c375c81ecb.new irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for LUKS2-temp-dd54103b-d96c-4d1d-9e23-e3c375c81ecb.new:
Verify passphrase:
Progress:   1.8%, ETA 09:51,  160 MiB written, speed  14.4 MiB/s
...
Progress:  17.7%, ETA 08:05, 1544 MiB written, speed  14.8 MiB/s
...
Finished, time 09:45.528, 8700 MiB written, speed  14.8 MiB/s
$ sudo cryptsetup open /dev/mmcblk0p2 rootfs
Enter passphrase for /dev/mmcblk0p2:

Rooting Around

At this point our encrypted container on /dev/mmcblk0p2 is open as a decrypted device called /dev/mapper/rootfs. We can now expand the partition back to its “normal” size (minus the 16M we shrunk it for metadata), and mount it (and the boot partition) at some appropriate location.

$ sudo resize2fs /dev/mapper/rootfs
resize2fs 1.45.5 (07-Jan-2020)
Resizing the filesystem on /dev/mapper/rootfs to 2227255 (4k) blocks.
The filesystem on /dev/mapper/rootfs is now 2227255 (4k) blocks long.
$ sudo mkdir -p /mnt/boot /mnt/root
$ sudo mount /dev/mmcblk0p1 /mnt/boot
$ sudo mount /dev/mapper/rootfs /mnt/root

Now we need to perform some surgery on the file-system tables. Firstly, we need to tell the system how to set up the “rootfs” node we just opened. This is the job of the /etc/crypttab file. This should look relatively familiar to anyone used to messing with /etc/fstab, but bear in mind that the target and source devices are “backwards” (target first, then source).

Another important thing to remember here is that in this file, /dev/mmcblk0p2 is the device for the encrypted partition from the Pi’s perspective. In other words, if your SD card appears as /dev/sdf on your PC and you’ve been using /dev/sdf2 as the card’s root partition in the instructions above you should not do that here: just use /dev/mmcblk0p2. However, if you’re using an SSD via a USB adapter as your storage, you may need to adjust the line below to use /dev/sda2 as that’s how the Pi will be referring to this partition at boot time.

$ cat /mnt/root/etc/crypttab
# <target name> <source device>         <key file>      <options>
$ echo "rootfs  /dev/mmcblk0p2  none  luks,discard" >> /mnt/root/etc/crypttab
$ cat /mnt/root/etc/crypttab
# <target name> <source device>         <key file>      <options>
rootfs  /dev/mmcblk0p2  none  luks,discard

Note

You may or may not wish to include the “discard” option above. It’s the default in Debian and Ubuntu for encrypted partitions but may carry security risks. See man crypttab for full details.

With the /dev/mapper/rootfs device configured, all that remains is to tell the kernel and systemd to look there for its root device via the /etc/fstab table, and the cmdline.txt file on the boot partition:

$ cat /mnt/root/etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
LABEL=writable    /     ext4    defaults,x-systemd.growfs    0 0
/swapfile         none  swap    sw    0 0
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1
$ sudo sed -i -e 's,^LABEL=writable,/dev/mapper/rootfs,' /mnt/root/etc/fstab
$ cat /mnt/root/etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
/dev/mapper/rootfs    /     ext4    defaults,x-systemd.growfs    0 0
/swapfile         none  swap    sw    0 0
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1
$ cat /mnt/boot/cmdline.txt
dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash
$ sudo sed -i -e 's,LABEL=writable,/dev/mapper/rootfs,' /mnt/boot/cmdline.txt
$ cat /mnt/boot/cmdline.txt
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mapper/rootfs rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash

Note

Again, feel free to use an editor here instead of messing around with sed; I’ve illustrated the changes necessary by showing the before and after states of the file affected.

Fail! (and then succeed)

Time to clean up and test our card.

$ sudo umount /mnt/root /mnt/boot
$ sudo cryptsetup close rootfs

Eject the card (or mass storage) properly, shove it in your Pi and start it booting. This first boot will appear to take a very long time (several minutes, while it times out waiting for the root device to appear) and will then fail, dropping to the initramfs prompt. The reason for this is that, even though we re-generated the initramfs to include the cryptsetup tool, it still isn’t configured to actually unlock the correct partition. We’ll need to re-generate it again to do that, but first we need to get the system booted.

At the (initramfs) prompt run the following to get the system booting (if you’re booting an SSD drive, adjust /dev/mmcblk0p2 below to /dev/sda2):

(initramfs) cryptsetup open /dev/mmcblk0p2 rootfs
Enter passphrase for /dev/mmcblk0p2:
(initramfs) exit

Once the system has booted successfully, login and re-generate the initramfs one last time before rebooting to test your new setup:

$ sudo update-initramfs -u
$ sudo reboot

This time the initramfs should prompt you nicely to unlock your device before proceeding with the rest of the boot.

That’s all for this time. Let me know below if you run into any issues with this!