See Also: LUKS encryption
This is the first of a few posts I’ll be doing on customizing the storage of Ubuntu for Raspberry Pi images.
One of the first things I did when I moved to using a Pi as my primary workstation was set up LXD to enable “clean” build environments to help with debugging things. My favoured LXD storage backend is lvm …
(cue gasps of horror, followed by overlapping cries of “but zfs rules!” and “what about btrfs?”, before the two sides descend into several parallel flame wars over licensing and reliability and performance and … well, you get the point)
Yes, I like lvm. Which means I’d really like lvm on my Pi’s SSD, preferably for everything except the boot partition (which can’t be lvm because the bootloader doesn’t handle esoterica like that).
I’m going to demonstrate the necessary steps with a fresh Ubuntu 21.10 (Impish Indri) image on an SD card, but you can adapt the steps trivially to target an SSD drive on USB (just substitute /dev/sdX wherever you see /dev/mmcblkX below). With a teensy bit more effort, you could also operate on an existing installation rather than a fresh image (but again, you’ll need separate target storage — you can’t trivially convert an existing installation to lvm on the same block device).
Tabula Rasa
Start by grabbing your fresh SD card (or whatever storage you prefer). A word of (frankly obvious) warning here: we’re going to wipe everything on this storage so make sure you’re not going to mind losing anything on there first.
Instead of flashing an entire image to it, we’re going to set it up with our own partition table, create an LVM layout within that, then flash each partition of the image to it separately (all two of them, that is).
First things first: creating the new partition table. Plug in your storage, in my case an SD card. If you’re on an Ubuntu desktop (as I am) you’ll need to unmount any partitions that auto-mount from this storage before proceeding to (re-)partition it.
$ umount /dev/mmcblk0p1 $ umount /dev/mmcblk0p2 ... repeat for any more auto-mounted partitions ...
Now we’re going to partition our storage with GPT (plus protective MBR), set up the first partition as FAT (for the bootloader bits), and the second partition as LVM. The commands we’ll execute are as follows:
- (p)rint the current state of the partition table (not necessary, but just to be sure we’re looking at the “right” disk)
- (o)verwrite the partition table (to wipe it and start again)
- (n)ew partition, with the default number (1), default start sector (2048), a size of 512MB (actually larger than the source image which is 256MB, but that always feels a bit tight to me), and partition type 0700 (Microsoft basic data)
- (n)ew partition again, with the default number (2), default start sector (1050624), default size (rest of the storage medium, however large that is), and partition type of 8e00 (Linux LVM)
- (c)hange partition 1’s name to “system-boot”
- (c)hange partition 2’s name to “lvm”
- (p)rint the current state of the partition table again (not necessary, but just to confirm things “look right”)
- (w)rite the new partition table and exit (confirming we want to proceed)
Here’s a transcript of me doing this with a 16GB SD card:
$ sudo gdisk /dev/mmcblk0 GPT fdisk (gdisk) version 1.0.5 Partition table scan: MBR: MBR only BSD: not present APM: not present GPT: not present *************************************************************** Found invalid GPT and valid MBR; converting MBR to GPT format in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by typing 'q' if you don't want to convert your MBR partitions to GPT format! *************************************************************** Command (? for help): p Disk /dev/mmcblk0: 31116288 sectors, 14.8 GiB Model: STORAGE DEVICE Sector size (logical/physical): 512/512 bytes Disk identifier (GUID): 30D2AB17-E36C-4915-AF37-2C94E8FFE5C4 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 31116254 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector) End (sector) Size Code Name 1 2048 526335 256.0 MiB 0700 Microsoft basic data 2 526336 31116254 14.6 GiB 8300 Linux filesystem Command (? for help): o This option deletes all partitions and creates a new protective MBR. Proceed? (Y/N): y Command (? for help): n Partition number (1-128, default 1): First sector (34-31116254, default = 2048) or {+-}size{KMGTP}: Last sector (2048-31116254, default = 31116254) or {+-}size{KMGTP}: 512M Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): 0700 Changed type of partition to 'Microsoft basic data' Command (? for help): n Partition number (2-128, default 2): First sector (34-31116254, default = 1050624) or {+-}size{KMGTP}: Last sector (1050624-31116254, default = 31116254) or {+-}size{KMGTP}: Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): 8e00 Changed type of partition to 'Linux LVM' Command (? for help): c Partition number (1-2): 1 Enter name: system-boot Command (? for help): c Partition number (1-2): 2 Enter name: lvm Command (? for help): p Disk /dev/mmcblk0: 31116288 sectors, 14.8 GiB Model: STORAGE DEVICE Sector size (logical/physical): 512/512 bytes Disk identifier (GUID): C202E4F2-59CD-402D-9720-3C8D7D0E84AF Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 31116254 Partitions will be aligned on 2048-sector boundaries Total free space is 4061 sectors (2.0 MiB) Number Start (sector) End (sector) Size Code Name 1 2048 1048576 511.0 MiB 0700 system-boot 2 1050624 31116254 14.3 GiB 8E00 lvm Command (? for help): w Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!! Do you want to proceed? (Y/N): y OK; writing new GUID partition table (GPT) to /dev/mmcblk0. The operation has completed successfully.
Note
You may note we’re using GPT partitioning above. This is okay provided you’re using a reasonably modern version of the Pi’s bootloader which has supported GPT partitioning since mid-2020. In other words, this should work with post-Focal images (including later Focal point-releases) but don’t try it with anything older than that.
More (logical) volume!
Now we set up the logical volume(s) on our GPT partitioned card.
$ sudo pvcreate /dev/mmcblk0p2 Physical volume "/dev/mmcblk0p2" successfully created. $ sudo vgcreate pivg /dev/mmcblk0p2 Volume group "pivg" successfully created $ sudo lvcreate --size 8G --name root pivg Logical volume "root" created.
You might want to make the root LV bigger than 8G, especially if you’re using the desktop image. I’m specifically not using all the available storage (16GB in this case) because I assume you want some spare for … whatever it is you want lvm (in my case, a thin pool for LXD containers).
Going loopy
Time to grab a source image. Below, I’m getting a copy of the Ubuntu 21.10 pre-installed server image for the Pi (substitute the desktop image here if you like, there’s no real difference in the procedure), and unpacking it with the unxz utility:
$ sudo apt install -y xz-utils ... all the usual apt stuff ... $ wget http://cdimage.ubuntu.com/releases/impish/release/ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz $ unxz ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz
We need to access the individual partitions on the original image as we need to copy its boot partition to our new boot partition, then its root partition to our new root logical volume. To do this we set up a loop-device which treats our unpacked image as a disk in its own right, and check that it finds the right partitions:
$ sudo losetup --read-only --find --show --partscan ubuntu-21.10-preinstalled-server-arm64+raspi.img /dev/loop34 $ ls -l /dev/loop34* brw-rw---- 1 root disk 7, 34 Oct 15 14:01 /dev/loop34 brw-rw---- 1 root disk 259, 6 Oct 15 14:01 /dev/loop34p1 brw-rw---- 1 root disk 259, 7 Oct 15 14:01 /dev/loop34p2
In the example above (your loop-device will likely have a different number), the “loop34p1” and “loop34p2” block devices represent the first (boot) partition of the image, and the second (root) partition. We can now use good ol’ disk-destroyer to perform the block transfers (as ever with dd, be damned sure you get the “of=” device argument correct):
$ sudo dd if=/dev/loop34p1 of=/dev/mmcblk0p1 bs=16M status=progress 16+0 records in 16+0 records out 268435456 bytes (268 MB, 256 MiB) copied, 15.8294 s, 17.0 MB/s $ sudo dd if=/dev/loop34p2 of=/dev/mapper/pivg-root bs=16M status=progress 2030043136 bytes (2.0 GB, 1.9 GiB) copied, 1 s, 2.0 GB/s 226+1 records in 226+1 records out 3798978560 bytes (3.8 GB, 3.5 GiB) copied, 246.436 s, 15.4 MB/s
At this point, we can clean up the loop-device as we don’t need anything else from the source image.
$ sudo losetup -d /dev/loop34
Root Cause
Now mount the new boot partition so we can do a bit of surgery on where it should look for the root device (in our new logical volume):
$ sudo mkdir /mnt/boot $ sudo mount /dev/mmcblk0p1 /mnt/boot $ ls /mnt/boot bcm2710-rpi-2-b.dtb bootcode.bin fixup_x.dat start_db.elf bcm2710-rpi-3-b.dtb boot.scr initrd.img start.elf bcm2710-rpi-3-b-plus.dtb cmdline.txt meta-data start_x.elf bcm2710-rpi-cm3.dtb config.txt network-config uboot_rpi_3.bin bcm2711-rpi-400.dtb fixup4cd.dat overlays uboot_rpi_4.bin bcm2711-rpi-4-b.dtb fixup4.dat README uboot_rpi_arm64.bin bcm2711-rpi-cm4.dtb fixup4db.dat start4cd.elf user-data bcm2837-rpi-3-a-plus.dtb fixup4x.dat start4db.elf vmlinuz bcm2837-rpi-3-b.dtb fixup_cd.dat start4.elf bcm2837-rpi-3-b-plus.dtb fixup.dat start4x.elf bcm2837-rpi-cm3-io3.dtb fixup_db.dat start_cd.elf $ cat /mnt/boot/cmdline.txt dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash $ sudo sed -i -e 's,root=[^ ]*,root=/dev/mapper/pivg-root,' /mnt/boot/cmdline.txt $ cat /mnt/boot/cmdline.txt dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mapper/pivg-root rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash
Note
Feel free to use an editor here instead of messing around with sed; I’ve illustrated the changes necessary by showing the before and after states of the file affected.
Some (very similar) surgery is also required on the /etc/fstab file on our new root volume:
$ sudo mkdir -p /mnt/root $ sudo mount /dev/mapper/pivg-root /mnt/root $ cat /mnt/root/etc/fstab LABEL=writable / ext4 discard,errors=remount-ro 0 1 LABEL=system-boot /boot/firmware vfat defaults 0 1 $ sudo sed -i -e 's,^LABEL=writable,/dev/mapper/pivg-root,' /mnt/root/etc/fstab $ cat /mnt/root/fstab /dev/mapper/pivg-root / ext4 discard,errors=remount-ro 0 1 LABEL=system-boot /boot/firmware vfat defaults 0 1
Finally, clean up all our mounts, deactivate the volume group containing our root volume, and remove all the temporary mount-points we created:
$ sudo umount /mnt/root $ sudo umount /mnt/boot $ sudo vgchange -an pivg $ sudo rmdir /mnt/root $ sudo rmdir /mnt/boot
Now eject the SD card (or whatever storage you’re using) and you should find it boots happily on your Pi, with the root storage as LVM. Let me know below if you run into any issues with this!