It's Pi all the way down...


In the last two articles we looked at LVM and LUKS for some enhancement of the basic storage of the standard Ubuntu Pi images. This time around we’ll combine both of those previous articles to make an image in the same way that the Ubuntu Desktop for PC images install when they’re given free reign to overwrite the entire contents of the drive.

Specifically, this means we aim to wind up with:

  • A FAT boot partition (no difference there), 512 MB in size
  • A second partition taking up the rest of the card containing
    • An encrypted LUKS container, containing
      • An LVM physical volume (PV) belonging to
        • An LVM volume group (VG) called “pivg” containing
          • An LVM logical volume (LV) called “root” taking up some reasonable portion of the disk space (8 GB in this example)
A diagram illustrating the SD card layout showing: a large bar representing the whole SD card /dev/mmcblk0 (N GB). Beneath that, two bars: a small one on the left for the system-boot partition /dev/mmcblk0p1 (512 MB), and a large one on the right for the remaining partition /dev/mmcblk0p2 (N GB - 512 MB). Beneath the large right-hand bar, a bar representing the LUKS encrypted container (again, N GB - 512 MB). Beneath that, two more bars: a tiny one on the left representing the LUKS header (16 MB), and a much larger one on the right representing the LVM PV /dev/mapper/rootfs within the LUKS container (N GB - 512 MB - 16 MB). Lower down, a dotted outline of the same size representing the LVM VG /dev/pivg. Finally, beneath that, a smaller bar representing the LVM LV /dev/mapper/pivg-root (8 GB).


One thing to note is that I’ve only tested this with the Ubuntu 22.04 (jammy) server for pi image. Notably, jammy has cryptsetup within the default initramfs on the image. I would not necessarily be confident that this works on earlier releases.

Three (?) Rowdy Layers

Rather than go through all the individual commands, which would result in this post looking like a foreword followed by the dump of a shell script, I’ll go through the high level procedure because the detail is almost exactly the same as the original articles but for a few minor differences and some deliberate omissions.

Firstly, we’ll deal with the content of the first article and set up LVM:

  1. Follow the first article instructions until it’s time to create the PV (with pvcreate):

    • Plug in the SD card
    • Unmount any auto-mounted partitions
    • Repartition with gdisk
  2. At this point we need to create the PV with 16MB less space than is available in the second partition, to leave a little room for the LUKS header around the PV:

    • From the gdisk session, you should have a printout of the new partition table; if you don’t, run sudo gdisk -l /dev/mmcblk0 (assuming /dev/mmcblk0 is your SD card)

    • For the second partition, calculate the number of sectors it occupies (which is simply end sector minus start sector) and then subtract 32768 (each sector is 512 bytes, so 32768 is 16MB). For example, on a 32GB SD card:

      $ sudo gdisk -l /dev/mmcblk0
      GPT fdisk (gdisk) version 1.0.8
      Partition table scan:
        MBR: protective
        BSD: not present
        APM: not present
        GPT: present
      Found valid GPT with protective MBR; using GPT.
      Disk /dev/mmcblk0: 62333952 sectors, 29.7 GiB
      Sector size (logical/physical): 512/512 bytes
      Disk identifier (GUID): A9BDFD97-F2A5-4419-9A83-B0160820FDBB
      Partition table holds up to 128 entries
      Main partition table begins at sector 2 and ends at sector 33
      First usable sector is 34, last usable sector is 62333918
      Partitions will be aligned on 2048-sector boundaries
      Total free space is 4061 sectors (2.0 MiB)
      Number  Start (sector)    End (sector)  Size       Code  Name
         1            2048         1048576   511.0 MiB   0700  system-boot
         2         1050624        62333918   29.2 GiB    8E00  lvm
      $ PV_SECTORS=$((62333918 - 1050624 - 32768))
      $ echo $PV_SECTORS
    • Proceed to creating the logical volume, but explicitly specify the number of sectors to use when creating the PV with --setphysicalvolumesize and the number of sectors with the suffix “s”:

      $ sudo pvcreate --setphysicalvolumesize ${PV_SECTORS}s /dev/mmcblk0p2
      Physical volume "/dev/mmcblk0p2" successfully created.
      $ sudo vgcreate pivg /dev/mmcblk0p2
      Volume group "pivg" successfully created
      $ sudo lvcreate --size 8G --name root pivg
      Logical volume "root" created.
  3. Finish the first article’s instructions:

    • Unpacking the image
    • Creating a loop device for the image
    • Transferring the content of the image partitions to the card
    • Updating the boot command line and fstab to point to the “new” root device
    • Unmount the root and boot partitions, and disable the VG
    • Don’t eject the SD card, we still need to do some surgery!

LUKS Dujour

Now it’s time to move on to the second article and set up the encrypted LUKS container around the physical volume (PV):

  1. Skip everything up to the encryption step:

    • No need to download another image
    • No need to boot the image yet
    • No need to unmount stuff (we already did that)
    • No need to shrink partitions (the root partition hasn’t been auto-expanded because we haven’t booted anything, and we’ve pre-shrunk the PV containing the root)
  2. Now it’s time to encrypt the second partition containing the PV:

    • First the encryption; this is exactly as in the second article:

      $ sudo cryptsetup reencrypt --encrypt --reduce-device-size 16M --cipher xchacha12,aes-adiantum-plain64 /dev/mmcblk0p2
      This will overwrite data on irrevocably.
      Are you sure? (Type 'yes' in capital letters): YES
      Enter passphrase for
      Verify passphrase:
      Finished, time 36:53.687, 29915 MiB written, speed  13.5 MiB/s
    • Next we open the encrypted container:

      $ sudo cryptsetup open /dev/mmcblk0p2 rootfs
      Enter passphrase for /dev/mmcblk0p2:
    • Then we re-scan for physical volumes (PVs) and re-activate the pivg volume group (VG) we created earlier; this isn’t in the original article but is necessary as we’ve altered the mapper configuration:

      $ sudo pvscan
        PV /dev/mapper/rootfs   VG pivg            lvm2 [29.20 GiB / 21.20 GiB free]
        Total: 1 [29.20 GiB] / in use: 1 [29.20 GiB] / in no VG: 0 [0   ]
      $ sudo vgchange -ay pivg
        1 logical volume(s) in volume group "pivg" now active
    • We skip over all the resizing bits because we’ve already established the size of the root volume, and just mount the root file-system (we don’t need to mount the boot file-system). This is, again, slightly different to the original article because the root file-system is a mapper device, pivg-root as in the first article:

      $ sudo mkdir -p /mnt/root
      $ sudo mount /dev/mapper/pivg-root /mnt/root
  3. Then a bit (but only a bit) of configuration manipulation:

    • Add our entry to /etc/crypttab. Please heed the advice in the original article regarding the source device here; this is the source device as seen by the Pi:

      $ cat /mnt/root/etc/crypttab
      # <target name> <source device>         <key file>      <options>
      $ echo "rootfs  /dev/mmcblk0p2  none  luks,discard" >> /mnt/root/etc/crypttab
      $ cat /mnt/root/etc/crypttab
      # <target name> <source device>         <key file>      <options>
      rootfs  /dev/mmcblk0p2  none  luks,discard
    • Skip the bits about editing fstab and the boot command line; we’ve already done everything we need to for those. Just unmount root file-system, and close the encrypted container:

      $ sudo umount /mnt/root
      $ sudo cryptsetup close rootfs
  4. Now just follow the second article to the end:

    • Eject the card and boot it on your Pi
    • There’s a long delay while the initial root mount fails
    • In the emergency prompt that appears, open the encrypted device and exit the prompt
    • Once booted, update the initramfs
    • Reboot to make sure everything works!

Future Cases

Congratulations! At this point you should have a setup eerily similar to the one I use on my main development Pi, albeit that one boots off a large SSD instead of an SD card. Future things worth exploring with this setup:

  • How to get plymouth prompting nicely for the password; plymouth’s always been on the Pi server images but for some reason it doesn’t work properly (unlike the Pi desktop images where it’s happy). I had a cursory look into this a couple of cycles ago, but didn’t get anywhere at that time.
  • How about ditching password encryption and using a key on removable storage. For instance, if booting off SD card, how about using a USB key with the encryption key. Or for my favoured setup where I boot off a USB3 attached SSD drive, how about using an SD card with the key?
  • With this configuration I usually set up LXD with an LVM thin-pool for its default storage. This usually requires taking a crow-bar to LXD and forcing it to use the thin-pool (with; LXD doesn’t like working with a non-empty VG for reasons I’ve never full understood, but at least it’s possible). Let me know in the comments if it would be useful to have another article covering this!