It's Pi all the way down...

Playing with Blocks - LVM

Mon 18 October 2021
by Dave Jones

This is the first of a few posts I’ll be doing on customizing the storage of Ubuntu for Raspberry Pi images.

One of the first things I did when I moved to using a Pi as my primary workstation was set up LXD to enable “clean” build environments to help with debugging things. My favoured LXD storage backend is lvm …

(cue gasps of horror, followed by overlapping cries of “but zfs rules!” and “what about btrfs?”, before the two sides descend into several parallel flame wars over licensing and reliability and performance and … well, you get the point)

Yes, I like lvm. Which means I’d really like lvm on my Pi’s SSD, preferably for everything except the boot partition (which can’t be lvm because the bootloader doesn’t handle esoterica like that).

I’m going to demonstrate the necessary steps with a fresh Ubuntu 21.10 (Impish Indri) image on an SD card, but you can adapt the steps trivially to target an SSD drive on USB (just substitute /dev/sdX wherever you see /dev/mmcblkX below). With a teensy bit more effort, you could also operate on an existing installation rather than a fresh image (but again, you’ll need separate target storage — you can’t trivially convert an existing installation to lvm on the same block device).

Tabula Rasa

Start by grabbing your fresh SD card (or whatever storage you prefer). A word of (frankly obvious) warning here: we’re going to wipe everything on this storage so make sure you’re not going to mind losing anything on there first.

Instead of flashing an entire image to it, we’re going to set it up with our own partition table, create an LVM layout within that, then flash each partition of the image to it separately (all two of them, that is).

First things first: creating the new partition table. Plug in your storage, in my case an SD card. If you’re on an Ubuntu desktop (as I am) you’ll need to unmount any partitions that auto-mount from this storage before proceeding to (re-)partition it.

$ umount /dev/mmcblk0p1
$ umount /dev/mmcblk0p2
... repeat for any more auto-mounted partitions ...

Now we’re going to partition our storage with GPT (plus protective MBR), set up the first partition as FAT (for the bootloader bits), and the second partition as LVM. The commands we’ll execute are as follows:

  • (p)rint the current state of the partition table (not necessary, but just to be sure we’re looking at the “right” disk)
  • (o)verwrite the partition table (to wipe it and start again)
  • (n)ew partition, with the default number (1), default start sector (2048), a size of 512MB (actually larger than the source image which is 256MB, but that always feels a bit tight to me), and partition type 0700 (Microsoft basic data)
  • (n)ew partition again, with the default number (2), default start sector (1050624), default size (rest of the storage medium, however large that is), and partition type of 8e00 (Linux LVM)
  • (c)hange partition 1’s name to “system-boot”
  • (c)hange partition 2’s name to “lvm”
  • (p)rint the current state of the partition table again (not necessary, but just to confirm things “look right”)
  • (w)rite the new partition table and exit (confirming we want to proceed)

Here’s a transcript of me doing this with a 16GB SD card:

$ sudo gdisk /dev/mmcblk0
GPT fdisk (gdisk) version 1.0.5

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***************************************************************


Command (? for help): p
Disk /dev/mmcblk0: 31116288 sectors, 14.8 GiB
Model: STORAGE DEVICE
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 30D2AB17-E36C-4915-AF37-2C94E8FFE5C4
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 31116254
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          526335   256.0 MiB   0700  Microsoft basic data
   2          526336        31116254   14.6 GiB    8300  Linux filesystem

Command (? for help): o
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): y

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-31116254, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-31116254, default = 31116254) or {+-}size{KMGTP}: 512M
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 0700
Changed type of partition to 'Microsoft basic data'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-31116254, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-31116254, default = 31116254) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8e00
Changed type of partition to 'Linux LVM'

Command (? for help): c
Partition number (1-2): 1
Enter name: system-boot

Command (? for help): c
Partition number (1-2): 2
Enter name: lvm

Command (? for help): p
Disk /dev/mmcblk0: 31116288 sectors, 14.8 GiB
Model: STORAGE DEVICE
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): C202E4F2-59CD-402D-9720-3C8D7D0E84AF
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 31116254
Partitions will be aligned on 2048-sector boundaries
Total free space is 4061 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1048576   511.0 MiB   0700  system-boot
   2         1050624        31116254   14.3 GiB    8E00  lvm

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/mmcblk0.
The operation has completed successfully.

Note

You may note we’re using GPT partitioning above. This is okay provided you’re using a reasonably modern version of the Pi’s bootloader which has supported GPT partitioning since mid-2020. In other words, this should work with post-Focal images (including later Focal point-releases) but don’t try it with anything older than that.

More (logical) volume!

Now we set up the logical volume(s) on our GPT partitioned card.

$ sudo pvcreate /dev/mmcblk0p2
Physical volume "/dev/mmcblk0p2" successfully created.
$ sudo vgcreate pivg /dev/mmcblk0p2
Volume group "pivg" successfully created
$ sudo lvcreate --size 8G --name root pivg
Logical volume "root" created.

You might want to make the root LV bigger than 8G, especially if you’re using the desktop image. I’m specifically not using all the available storage (16GB in this case) because I assume you want some spare for … whatever it is you want lvm (in my case, a thin pool for LXD containers).

Going loopy

Time to grab a source image. Below, I’m getting a copy of the Ubuntu 21.10 pre-installed server image for the Pi (substitute the desktop image here if you like, there’s no real difference in the procedure), and unpacking it with the unxz utility:

$ sudo apt install -y xz-utils
... all the usual apt stuff ...
$ wget http://cdimage.ubuntu.com/releases/impish/release/ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz
$ unxz ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz

We need to access the individual partitions on the original image as we need to copy its boot partition to our new boot partition, then its root partition to our new root logical volume. To do this we set up a loop-device which treats our unpacked image as a disk in its own right, and check that it finds the right partitions:

$ sudo losetup --read-only --find --show --partscan ubuntu-21.10-preinstalled-server-arm64+raspi.img
/dev/loop34
$ ls -l /dev/loop34*
brw-rw---- 1 root disk   7, 34 Oct 15 14:01 /dev/loop34
brw-rw---- 1 root disk 259,  6 Oct 15 14:01 /dev/loop34p1
brw-rw---- 1 root disk 259,  7 Oct 15 14:01 /dev/loop34p2

In the example above (your loop-device will likely have a different number), the “loop34p1” and “loop34p2” block devices represent the first (boot) partition of the image, and the second (root) partition. We can now use good ol’ disk-destroyer to perform the block transfers (as ever with dd, be damned sure you get the “of=” device argument correct):

$ sudo dd if=/dev/loop34p1 of=/dev/mmcblk0p1 bs=16M status=progress
16+0 records in
16+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 15.8294 s, 17.0 MB/s
$ sudo dd if=/dev/loop34p2 of=/dev/mapper/pivg-root bs=16M status=progress
2030043136 bytes (2.0 GB, 1.9 GiB) copied, 1 s, 2.0 GB/s
226+1 records in
226+1 records out
3798978560 bytes (3.8 GB, 3.5 GiB) copied, 246.436 s, 15.4 MB/s

At this point, we can clean up the loop-device as we don’t need anything else from the source image.

$ sudo losetup -d /dev/loop34

Root Cause

Now mount the new boot partition so we can do a bit of surgery on where it should look for the root device (in our new logical volume):

$ sudo mkdir /mnt/boot
$ sudo mount /dev/mmcblk0p1 /mnt/boot
$ ls /mnt/boot
bcm2710-rpi-2-b.dtb       bootcode.bin  fixup_x.dat     start_db.elf
bcm2710-rpi-3-b.dtb       boot.scr      initrd.img      start.elf
bcm2710-rpi-3-b-plus.dtb  cmdline.txt   meta-data       start_x.elf
bcm2710-rpi-cm3.dtb       config.txt    network-config  uboot_rpi_3.bin
bcm2711-rpi-400.dtb       fixup4cd.dat  overlays        uboot_rpi_4.bin
bcm2711-rpi-4-b.dtb       fixup4.dat    README          uboot_rpi_arm64.bin
bcm2711-rpi-cm4.dtb       fixup4db.dat  start4cd.elf    user-data
bcm2837-rpi-3-a-plus.dtb  fixup4x.dat   start4db.elf    vmlinuz
bcm2837-rpi-3-b.dtb       fixup_cd.dat  start4.elf
bcm2837-rpi-3-b-plus.dtb  fixup.dat     start4x.elf
bcm2837-rpi-cm3-io3.dtb   fixup_db.dat  start_cd.elf
$ cat /mnt/boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash
$ sudo sed -i -e 's,root=[^ ]*,root=/dev/mapper/pivg-root,' /mnt/boot/cmdline.txt
$ cat /mnt/boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mapper/pivg-root rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash

Note

Feel free to use an editor here instead of messing around with sed; I’ve illustrated the changes necessary by showing the before and after states of the file affected.

Some (very similar) surgery is also required on the /etc/fstab file on our new root volume:

$ sudo mkdir -p /mnt/root
$ sudo mount /dev/mapper/pivg-root /mnt/root
$ cat /mnt/root/etc/fstab
LABEL=writable  /        ext4   discard,errors=remount-ro       0 1
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1
$ sudo sed -i -e 's,^LABEL=writable,/dev/mapper/pivg-root,' /mnt/root/etc/fstab
$ cat /mnt/root/fstab
/dev/mapper/pivg-root  /        ext4   discard,errors=remount-ro       0 1
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1

Finally, clean up all our mounts, deactivate the volume group containing our root volume, and remove all the temporary mount-points we created:

$ sudo umount /mnt/root
$ sudo umount /mnt/boot
$ sudo vgchange -an pivg-root
$ sudo rmdir /mnt/root
$ sudo rmdir /mnt/boot

Now eject the SD card (or whatever storage you’re using) and you should find it boots happily on your Pi, with the root storage as LVM. Let me know below if you run into any issues with this!