It's Pi all the way down... - lvm

Building Blocks

2023-10-24T00:00:00+01:00

In an earlier series of posts (Playing with Blocks 1, 2, 3, and 4) we had a look at customizing Ubuntu images for the Raspberry Pi by playing around with various block device transformations. However, we never covered what a block device actually is. This is going to be a vaguely important topic in an upcoming series of posts on NBD booting of Ubuntu, so I figured some brief coverage of the topic is warranted.

Brief.

Who am I kidding?

Blockage

In the UNIX world a block device is a very simple concept: it’s a storage device which provides access to its contents as an ordered sequence of identically sized blocks. That’s it. No files, no directories, no permissions, nothing more than a numbered sequence of blocks all with the same size. Typically, the blocks are some whole multiple of an underlying device’s sector size, which over most of computing history has been 512 bytes. However, the user may access block devices byte-by-byte if they wish:

For instance, a read of a few bytes (illustrated on the left in green) may read a full 512-byte sector, throw away the bytes outside the requested range, and return that which was requested [1]. A write of a few bytes (illustrated in the middle in red) is a bit more complex; the kernel needs to read the full sector from the underlying device, change those bytes we requested to write in the middle, then write the whole sector back to the underlying device [2].

Crucially, block devices are random access. We can access any block, anywhere, at any time, without having to access all blocks prior to it. In other words, block devices make storage look and “feel” like RAM. This is as opposed to the “byte-stream” view of files which owes more to the tape drives of old (“seeking” the head to a particular location, then reading or writing from that point).

Transform!

The simplicity of block devices leads to them being wonderfully flexible things that can often be converted into new block devices in useful ways. Probably the most basic block device transformation that everyone is familiar with is “partitioning”, which simply sub-divides one block device into multiple new ones (minus some header space). This transformation effectively just re-numbers the blocks but doesn’t change their content or ordering.

More complex transformations are also available.

The LVM sub-system (explored previously) gathers multiple block devices into a “volume group”. The volume group can then be used to produce other block devices called “logical volumes”. These are similar to partitions, but have no need to be contiguous on the underlying block devices, and can even span multiple underlying devices.

A drive (sda) is divided into two partitions (sda1, sda2) the latter of which is further divided into three block devices with LVM (root, home, and swap). The home device has been expanded at some point in its history as its storage is non-contiguous

Speaking of multiple underlying devices, RAID systems (like mdadm) consume multiple block devices and produce a single block device. The blocks in the produced device are duplicated to the underlying block devices (in the simplest mirror case; more complex transforms are involved in things like RAID5). Corruption or even loss of one of the underlying block devices, as in the failure of a drive, can be entirely hidden from the consumer of the produced block device.

Four drives (sda, sdb, sdc, and sdd) are partitioned such that each has a small initial partition (sda1 etc.) and a larger partition (sda2 etc.); the small initial partitions are combined as a RAID1 mirror into a boot block device. The larger partitions are combined as a RAID10 device (a stripe of mirrors) as a root device.

The LUKS encryption mechanism (also explored in previous posts) consumes one block device and produces a slightly smaller one (minus the size of the LUKS header) that scrambles (and unscrambles) the content of blocks as they are written (and read).

The Hierarchy

A file-system is just another kind of transformation of a block device; one which converts the flat sequence of blocks into a tree-like hierarchy of files (optionally with attributes like ownership, groups, permissions, and all the rest of that gubbins).

File systems can be relatively simple affairs, like the ancient but ubiquitous FAT which has no concept of file ownership, UNIX modes, and no symlinks. Alternatively they can be complex, like the common ext4 file-system which has journaling data recovery, the full suite of UNIX ownership, attributes, and linkage, and numerous tunable options.

Ultimately, the file-systems produced from various block devices are all grafted together (“mounted” in UNIX parlance) onto a “virtual” file-system which is what all userland processes (and you, the user) generally consider “the file-system”; the place you find all the files on your computer.

Two partitions (sda1 and sda2) are formatted as FAT and ext4 respectively. The vfat driver converts the sda1 block device into a typical boot file-system for the Pi (containing bootcode.bin, start4.elf, config.txt, etc). The ext4 driver converts the sda2 block device into a typical Linux root file-system (containing bin, etc, lib, usr, sbin directories). The virtual file-system then “mounts” the ext4 hierarchy under / and the FAT hierarchy under /boot/firmware

A common Linux storage layout is shown above. The drive (/dev/sda) has two partitions. The first partition (/dev/sda1) is formatted as FAT, which the kernel’s vfat driver transforms into a file-system. The second partition (/dev/sda2) is formatted as ext4, which the ext4 driver transforms into another file-system. The kernel has mounted the ext4 file-system as the root of the “virtual” file-system, and the FAT file-system under the /boot/firmware mount-point.

Holy CoW!

Some modern file-systems provide a copy-on-write facility (most notably btrfs and zfs). This allows rapid (near instantaneous) cloning of an entire file-system hierarchy. The clone doesn’t initially take any storage of its own, and reads of the clone actually go to the original files. However, writes to the clone allocate new storage for the changed content.

However, the facility is not limited to modern file-systems. LVM also provides such a facility, initially intended to provide snapshots of file-systems at a point in time (for backup, testing, or other purposes).

The “root” block device contains a number of filled blocks in green, while the “snap” block device has an equivalent number of “empty” blocks. Two reads, indicated by arrows, are made against the “snap” device but pass through to the underlying “root” device. Below this an equivalent layout is show, but now one of the “filled” blocks is “red” indicating a change in content. Above it, the equivalent block in the “snap” device is green indicating it contains the original content. A read against this block comes directly from the “snap” device instead of passing through to the underlying “root” device.

The illustration above shows a “root” block device, presumably containing a root file-system. We create a snapshot of the “root” block device called “snap”. Initially, the “snap” block device is empty, and reads to it pass through to the underlying “root” block device. However, when a block is the “root” device is written to, its original content is first copied to the “snap” device. Subsequent reads of this block in the “snap” device will go to this copied block instead. Further writes to the same block in the “root” device are ignored because the “snap” device only cares about the content at the point in time it was created.

Note that one of the beauties of this system is that we don’t care what file-system is on the “root” device. It could be FAT, ext4, XFS, even (redundantly) one of the more modern systems that supports this internally. However, a limitation of this snapshot system is that we need to allocate space for it up front [4]. Can we come up with something in which the snapshot allocates blocks dynamically?

At the top, the “image” block device (read-only) contains a number of filled blocks in green. Above it, a clone, “clone1”, exists with the same number of blocks but all are empty. A read of two blocks in “clone1”, indicated by arrows, passes through to the corresponding blocks in “image”. Below, the one of the blocks in “clone1” is now red indicating it has been changed from the original content in “image”. Another arrow, indicating a read of that block now comes straight from “clone1” and doesn’t pass through. Another clone, “clone2”, is below it with entirely empty blocks. A read of the same block which is changed in “clone1”, passes through “clone2” to the underlying “image” device.

The illustration above shows “thinly provisioned” snapshots in LVM. This operates a little like the regular snapshots in reverse. An underlying block device called “image” presumably contains an OS image. We have created a “thinly provisioned” snapshot of this called “clone1”. Reads of an unchanged block pass through to the underlying (read-only) “image”. Writes to blocks in “clone1” occur within its storage, with subsequent reads to that block no longer passing through. Finally, we create another clone, “clone2”, which reads the original block from the underlying “image”.

It’s worth noting that most of the blocks in the clones are empty. Provided they stay that way (i.e. the clones don’t change too many blocks of the underlying image), there’s little sense in actually allocating them. This is the “thin” in “thinly provisioned”. The blocks of the clones are allocated on demand so a clone initially takes up no space. This allocate-on-demand facility can implement one form of “over provisioning” of storage. Each clone thinks it has all the storage allocated to the original image, but there’s no need to have all the storage available (unless the clones grow to require it, naturally).

Hopefully this gives some idea of how modern cloud systems can spin up clones of an OS image nearly instantaneously, and how they can seemingly provide many gigabytes of storage to containers or VMs, without necessarily having installed all the storage they apparently provide.

Layer Cake

Finally, a brief illustration of just how silly this can get. Here’s a diagram of a storage layout I’ve used on servers in the past:

A spiders web of storage. Four boxes represent four hard at the bottom. Above this a larger box represents the RAID device built from the drives. Above this a hexagonal volume group is built from the large RAID device. From the volume group spring numerous block devices, for “tmp”, “root”, “home”, and “images”. From the first three spring regular file-systems, but from “images” spring several more block devices named “base” and from that “munin”, “docs”, and “git” (OS images running in containers).

Four hard drives exist, each containing a small boot partition and a large partition occupying the rest of the capacity
The four boot partitions are combined with a RAID1 mirror into a boot block device, “md1” [3].
The “md1” block device is formatted as FAT and contains the “/boot” hierarchy.
The four large partitions are combined with RAID5 into a large block device, “md2” (RAID5 implies it has the capacity of n-1, i.e. three, of the underlying devices, and can survive the loss of any one device).
The “md2” device is consumed by the “raidvg” volume group from which are produced various block devices: the “root”, “home”, “tmp”, and “images” volumes.
“root”, “home”, and “tmp” are formatted with a mix of file-systems, ext4 and XFS.
The “images” volume is a “thin pool” from which the “base” volume is derived.
The “munin”, “docs”, and “git” volumes are all thinly provisioned snapshots of the “base” volume.

This may seem like a complicated stack of transformations, but consider that in essence most amount to little more than re-numbering and re-ordering of blocks (the RAID5 transform is a little more involved, admittedly). Hence, for very little performance cost, we’ve got redundancy (RAID), flexible volume creation system with snapshotting (LVM), and copy-on-write cloning for VMs or containers launched on the server (thin provisioning).

By this point you should have a reasonable understanding of what a block device is, and what can be accomplished with the various transforms that are available for them under Linux. Next time we’ll take a look at the issues with netbooting from NFS and how NBD (or block devices in general) can mitigate some of these issues.

[1]	In practice it’s unlikely the bytes will actually be thrown away; the sector requested will be stored in a cache, in case a future read requests more bytes from the same sector (very likely when reading through a file sequentially).

[2]	Again, the changed block will wind up in a cache so future modifications can skip the initial read step.

[3]

Simple RAID1 mirroring is used to ensure that the BIOS can use any of the drives as its initial boot device without caring about the RAID layout (which it doesn’t understand). Note: This carries risks as it means the BIOS and bootloader must treat the boot partition as strictly read-only (and that’s not necessarily guaranteed).

[4]	The snapshot doesn’t require all the space to be allocated, as the diagram suggests. In fact, assuming the underlying device changes infrequently, it’s common to only allocate ~5% of the origin’s storage. However, you do still need to provide some up front.

Playing with Blocks - It’s All Connected

2022-09-08T00:00:00+01:00

In the last two articles we looked at LVM and LUKS for some enhancement of the basic storage of the standard Ubuntu Pi images. This time around we’ll combine both of those previous articles to make an image in the same way that the Ubuntu Desktop for PC images install when they’re given free reign to overwrite the entire contents of the drive.

Specifically, this means we aim to wind up with:

A FAT boot partition (no difference there), 512 MB in size
A second partition taking up the rest of the card containing
- An encrypted LUKS container, containing
  - An LVM physical volume (PV) belonging to
    - An LVM volume group (VG) called “pivg” containing
      - An LVM logical volume (LV) called “root” taking up some reasonable portion of the disk space (8 GB in this example)

A diagram illustrating the SD card layout showing: a large bar representing the whole SD card /dev/mmcblk0 (N GB). Beneath that, two bars: a small one on the left for the system-boot partition /dev/mmcblk0p1 (512 MB), and a large one on the right for the remaining partition /dev/mmcblk0p2 (N GB - 512 MB). Beneath the large right-hand bar, a bar representing the LUKS encrypted container (again, N GB - 512 MB). Beneath that, two more bars: a tiny one on the left representing the LUKS header (16 MB), and a much larger one on the right representing the LVM PV /dev/mapper/rootfs within the LUKS container (N GB - 512 MB - 16 MB). Lower down, a dotted outline of the same size representing the LVM VG /dev/pivg. Finally, beneath that, a smaller bar representing the LVM LV /dev/mapper/pivg-root (8 GB).

Note

One thing to note is that I’ve only tested this with the Ubuntu 22.04 (jammy) server for pi image. Notably, jammy has cryptsetup within the default initramfs on the image. I would not necessarily be confident that this works on earlier releases.

Three (?) Rowdy Layers

Rather than go through all the individual commands, which would result in this post looking like a foreword followed by the dump of a shell script, I’ll go through the high level procedure because the detail is almost exactly the same as the original articles but for a few minor differences and some deliberate omissions.

Firstly, we’ll deal with the content of the first article and set up LVM:

Follow the first article instructions until it’s time to create the PV (with pvcreate):
- Plug in the SD card
- Unmount any auto-mounted partitions
- Repartition with gdisk

At this point we need to create the PV with 16MB less space than is available in the second partition, to leave a little room for the LUKS header around the PV:

From the gdisk session, you should have a printout of the new partition table; if you don’t, run sudo gdisk -l /dev/mmcblk0 (assuming /dev/mmcblk0 is your SD card)

For the second partition, calculate the number of sectors it occupies (which is simply end sector minus start sector) and then subtract 32768 (each sector is 512 bytes, so 32768 is 16MB). For example, on a 32GB SD card:

$ sudo gdisk -l /dev/mmcblk0
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/mmcblk0: 62333952 sectors, 29.7 GiB
Model: STORAGE DEVICE
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): A9BDFD97-F2A5-4419-9A83-B0160820FDBB
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 62333918
Partitions will be aligned on 2048-sector boundaries
Total free space is 4061 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1048576   511.0 MiB   0700  system-boot
   2         1050624        62333918   29.2 GiB    8E00  lvm
$ PV_SECTORS=$((62333918 - 1050624 - 32768))
$ echo $PV_SECTORS
61250526

Proceed to creating the logical volume, but explicitly specify the number of sectors to use when creating the PV with --setphysicalvolumesize and the number of sectors with the suffix “s”:

$ sudo pvcreate --setphysicalvolumesize ${PV_SECTORS}s /dev/mmcblk0p2
Physical volume "/dev/mmcblk0p2" successfully created.
$ sudo vgcreate pivg /dev/mmcblk0p2
Volume group "pivg" successfully created
$ sudo lvcreate --size 8G --name root pivg
Logical volume "root" created.

Finish the first article’s instructions:
- Unpacking the image
- Creating a loop device for the image
- Transferring the content of the image partitions to the card
- Updating the boot command line and fstab to point to the “new” root device
- Unmount the root and boot partitions, and disable the VG
- Don’t eject the SD card, we still need to do some surgery!

LUKS Dujour

Now it’s time to move on to the second article and set up the encrypted LUKS container around the physical volume (PV):

Skip everything up to the encryption step:
- No need to download another image
- No need to boot the image yet
- No need to unmount stuff (we already did that)
- No need to shrink partitions (the root partition hasn’t been auto-expanded because we haven’t booted anything, and we’ve pre-shrunk the PV containing the root)

Now it’s time to encrypt the second partition containing the PV:

First the encryption; this is exactly as in the second article:

$ sudo cryptsetup reencrypt --encrypt --reduce-device-size 16M --cipher xchacha12,aes-adiantum-plain64 /dev/mmcblk0p2

WARNING!
========
This will overwrite data on LUKS2-temp-a823bd67-a632-41cb-955e-8776b8b3fb5b.new irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for LUKS2-temp-a823bd67-a632-41cb-955e-8776b8b3fb5b.new:
Verify passphrase:
Finished, time 36:53.687, 29915 MiB written, speed  13.5 MiB/s

Next we open the encrypted container:

$ sudo cryptsetup open /dev/mmcblk0p2 rootfs
Enter passphrase for /dev/mmcblk0p2:

Then we re-scan for physical volumes (PVs) and re-activate the pivg volume group (VG) we created earlier; this isn’t in the original article but is necessary as we’ve altered the mapper configuration:

$ sudo pvscan
  PV /dev/mapper/rootfs   VG pivg            lvm2 [29.20 GiB / 21.20 GiB free]
  Total: 1 [29.20 GiB] / in use: 1 [29.20 GiB] / in no VG: 0 [0   ]
$ sudo vgchange -ay pivg
  1 logical volume(s) in volume group "pivg" now active

We skip over all the resizing bits because we’ve already established the size of the root volume, and just mount the root file-system (we don’t need to mount the boot file-system). This is, again, slightly different to the original article because the root file-system is a mapper device, pivg-root as in the first article:
```
$ sudo mkdir -p /mnt/root
$ sudo mount /dev/mapper/pivg-root /mnt/root
```

Then a bit (but only a bit) of configuration manipulation:

Add our entry to /etc/crypttab. Please heed the advice in the original article regarding the source device here; this is the source device as seen by the Pi:

$ cat /mnt/root/etc/crypttab
# <target name> <source device>         <key file>      <options>
$ echo "rootfs  /dev/mmcblk0p2  none  luks,discard" >> /mnt/root/etc/crypttab
$ cat /mnt/root/etc/crypttab
# <target name> <source device>         <key file>      <options>
rootfs  /dev/mmcblk0p2  none  luks,discard

Skip the bits about editing fstab and the boot command line; we’ve already done everything we need to for those. Just unmount root file-system, and close the encrypted container:
```
$ sudo umount /mnt/root
$ sudo cryptsetup close rootfs
```

Now just follow the second article to the end:
- Eject the card and boot it on your Pi
- There’s a long delay while the initial root mount fails
- In the emergency prompt that appears, open the encrypted device and exit the prompt
- Once booted, update the initramfs
- Reboot to make sure everything works!

Future Cases

Congratulations! At this point you should have a setup eerily similar to the one I use on my main development Pi, albeit that one boots off a large SSD instead of an SD card. Future things worth exploring with this setup:

How to get plymouth prompting nicely for the password; plymouth’s always been on the Pi server images but for some reason it doesn’t work properly (unlike the Pi desktop images where it’s happy). I had a cursory look into this a couple of cycles ago, but didn’t get anywhere at that time.
How about ditching password encryption and using a key on removable storage. For instance, if booting off SD card, how about using a USB key with the encryption key. Or for my favoured setup where I boot off a USB3 attached SSD drive, how about using an SD card with the key?
With this configuration I usually set up LXD with an LVM thin-pool for its default storage. This usually requires taking a crow-bar to LXD and forcing it to use the thin-pool (with lvm.vg.force_reuse; LXD doesn’t like working with a non-empty VG for reasons I’ve never full understood, but at least it’s possible). Let me know in the comments if it would be useful to have another article covering this!

Playing with Blocks - LVM

2021-10-18T00:00:00+01:00

See Also: LUKS encryption

This is the first of a few posts I’ll be doing on customizing the storage of Ubuntu for Raspberry Pi images.

One of the first things I did when I moved to using a Pi as my primary workstation was set up LXD to enable “clean” build environments to help with debugging things. My favoured LXD storage backend is lvm …

(cue gasps of horror, followed by overlapping cries of “but zfs rules!” and “what about btrfs?”, before the two sides descend into several parallel flame wars over licensing and reliability and performance and … well, you get the point)

Yes, I like lvm. Which means I’d really like lvm on my Pi’s SSD, preferably for everything except the boot partition (which can’t be lvm because the bootloader doesn’t handle esoterica like that).

I’m going to demonstrate the necessary steps with a fresh Ubuntu 21.10 (Impish Indri) image on an SD card, but you can adapt the steps trivially to target an SSD drive on USB (just substitute /dev/sdX wherever you see /dev/mmcblkX below). With a teensy bit more effort, you could also operate on an existing installation rather than a fresh image (but again, you’ll need separate target storage — you can’t trivially convert an existing installation to lvm on the same block device).

Tabula Rasa

Start by grabbing your fresh SD card (or whatever storage you prefer). A word of (frankly obvious) warning here: we’re going to wipe everything on this storage so make sure you’re not going to mind losing anything on there first.

Instead of flashing an entire image to it, we’re going to set it up with our own partition table, create an LVM layout within that, then flash each partition of the image to it separately (all two of them, that is).

First things first: creating the new partition table. Plug in your storage, in my case an SD card. If you’re on an Ubuntu desktop (as I am) you’ll need to unmount any partitions that auto-mount from this storage before proceeding to (re-)partition it.

$ umount /dev/mmcblk0p1
$ umount /dev/mmcblk0p2
... repeat for any more auto-mounted partitions ...

Now we’re going to partition our storage with GPT (plus protective MBR), set up the first partition as FAT (for the bootloader bits), and the second partition as LVM. The commands we’ll execute are as follows:

(p)rint the current state of the partition table (not necessary, but just to be sure we’re looking at the “right” disk)
(o)verwrite the partition table (to wipe it and start again)
(n)ew partition, with the default number (1), default start sector (2048), a size of 512MB (actually larger than the source image which is 256MB, but that always feels a bit tight to me), and partition type 0700 (Microsoft basic data)
(n)ew partition again, with the default number (2), default start sector (1050624), default size (rest of the storage medium, however large that is), and partition type of 8e00 (Linux LVM)
(c)hange partition 1’s name to “system-boot”
(c)hange partition 2’s name to “lvm”
(p)rint the current state of the partition table again (not necessary, but just to confirm things “look right”)
(w)rite the new partition table and exit (confirming we want to proceed)

Here’s a transcript of me doing this with a 16GB SD card:

$ sudo gdisk /dev/mmcblk0
GPT fdisk (gdisk) version 1.0.5

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***************************************************************


Command (? for help): p
Disk /dev/mmcblk0: 31116288 sectors, 14.8 GiB
Model: STORAGE DEVICE
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 30D2AB17-E36C-4915-AF37-2C94E8FFE5C4
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 31116254
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          526335   256.0 MiB   0700  Microsoft basic data
   2          526336        31116254   14.6 GiB    8300  Linux filesystem

Command (? for help): o
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): y

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-31116254, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-31116254, default = 31116254) or {+-}size{KMGTP}: 512M
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 0700
Changed type of partition to 'Microsoft basic data'

Command (? for help): n
Partition number (2-128, default 2):
First sector (34-31116254, default = 1050624) or {+-}size{KMGTP}:
Last sector (1050624-31116254, default = 31116254) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 8e00
Changed type of partition to 'Linux LVM'

Command (? for help): c
Partition number (1-2): 1
Enter name: system-boot

Command (? for help): c
Partition number (1-2): 2
Enter name: lvm

Command (? for help): p
Disk /dev/mmcblk0: 31116288 sectors, 14.8 GiB
Model: STORAGE DEVICE
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): C202E4F2-59CD-402D-9720-3C8D7D0E84AF
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 31116254
Partitions will be aligned on 2048-sector boundaries
Total free space is 4061 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1048576   511.0 MiB   0700  system-boot
   2         1050624        31116254   14.3 GiB    8E00  lvm

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/mmcblk0.
The operation has completed successfully.

Note

You may note we’re using GPT partitioning above. This is okay provided you’re using a reasonably modern version of the Pi’s bootloader which has supported GPT partitioning since mid-2020. In other words, this should work with post-Focal images (including later Focal point-releases) but don’t try it with anything older than that.

More (logical) volume!

Now we set up the logical volume(s) on our GPT partitioned card.

$ sudo pvcreate /dev/mmcblk0p2
Physical volume "/dev/mmcblk0p2" successfully created.
$ sudo vgcreate pivg /dev/mmcblk0p2
Volume group "pivg" successfully created
$ sudo lvcreate --size 8G --name root pivg
Logical volume "root" created.

You might want to make the root LV bigger than 8G, especially if you’re using the desktop image. I’m specifically not using all the available storage (16GB in this case) because I assume you want some spare for … whatever it is you want lvm (in my case, a thin pool for LXD containers).

Going loopy

Time to grab a source image. Below, I’m getting a copy of the Ubuntu 21.10 pre-installed server image for the Pi (substitute the desktop image here if you like, there’s no real difference in the procedure), and unpacking it with the unxz utility:

$ sudo apt install -y xz-utils
... all the usual apt stuff ...
$ wget http://cdimage.ubuntu.com/releases/impish/release/ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz
$ unxz ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz

We need to access the individual partitions on the original image as we need to copy its boot partition to our new boot partition, then its root partition to our new root logical volume. To do this we set up a loop-device which treats our unpacked image as a disk in its own right, and check that it finds the right partitions:

$ sudo losetup --read-only --find --show --partscan ubuntu-21.10-preinstalled-server-arm64+raspi.img
/dev/loop34
$ ls -l /dev/loop34*
brw-rw---- 1 root disk   7, 34 Oct 15 14:01 /dev/loop34
brw-rw---- 1 root disk 259,  6 Oct 15 14:01 /dev/loop34p1
brw-rw---- 1 root disk 259,  7 Oct 15 14:01 /dev/loop34p2

In the example above (your loop-device will likely have a different number), the “loop34p1” and “loop34p2” block devices represent the first (boot) partition of the image, and the second (root) partition. We can now use good ol’ disk-destroyer to perform the block transfers (as ever with dd, be damned sure you get the “of=” device argument correct):

$ sudo dd if=/dev/loop34p1 of=/dev/mmcblk0p1 bs=16M status=progress
16+0 records in
16+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 15.8294 s, 17.0 MB/s
$ sudo dd if=/dev/loop34p2 of=/dev/mapper/pivg-root bs=16M status=progress
2030043136 bytes (2.0 GB, 1.9 GiB) copied, 1 s, 2.0 GB/s
226+1 records in
226+1 records out
3798978560 bytes (3.8 GB, 3.5 GiB) copied, 246.436 s, 15.4 MB/s

At this point, we can clean up the loop-device as we don’t need anything else from the source image.

$ sudo losetup -d /dev/loop34

Root Cause

Now mount the new boot partition so we can do a bit of surgery on where it should look for the root device (in our new logical volume):

$ sudo mkdir /mnt/boot
$ sudo mount /dev/mmcblk0p1 /mnt/boot
$ ls /mnt/boot
bcm2710-rpi-2-b.dtb       bootcode.bin  fixup_x.dat     start_db.elf
bcm2710-rpi-3-b.dtb       boot.scr      initrd.img      start.elf
bcm2710-rpi-3-b-plus.dtb  cmdline.txt   meta-data       start_x.elf
bcm2710-rpi-cm3.dtb       config.txt    network-config  uboot_rpi_3.bin
bcm2711-rpi-400.dtb       fixup4cd.dat  overlays        uboot_rpi_4.bin
bcm2711-rpi-4-b.dtb       fixup4.dat    README          uboot_rpi_arm64.bin
bcm2711-rpi-cm4.dtb       fixup4db.dat  start4cd.elf    user-data
bcm2837-rpi-3-a-plus.dtb  fixup4x.dat   start4db.elf    vmlinuz
bcm2837-rpi-3-b.dtb       fixup_cd.dat  start4.elf
bcm2837-rpi-3-b-plus.dtb  fixup.dat     start4x.elf
bcm2837-rpi-cm3-io3.dtb   fixup_db.dat  start_cd.elf
$ cat /mnt/boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash
$ sudo sed -i -e 's,root=[^ ]*,root=/dev/mapper/pivg-root,' /mnt/boot/cmdline.txt
$ cat /mnt/boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mapper/pivg-root rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash

Note

Feel free to use an editor here instead of messing around with sed; I’ve illustrated the changes necessary by showing the before and after states of the file affected.

Some (very similar) surgery is also required on the /etc/fstab file on our new root volume:

$ sudo mkdir -p /mnt/root
$ sudo mount /dev/mapper/pivg-root /mnt/root
$ cat /mnt/root/etc/fstab
LABEL=writable  /        ext4   discard,errors=remount-ro       0 1
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1
$ sudo sed -i -e 's,^LABEL=writable,/dev/mapper/pivg-root,' /mnt/root/etc/fstab
$ cat /mnt/root/fstab
/dev/mapper/pivg-root  /        ext4   discard,errors=remount-ro       0 1
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1

Finally, clean up all our mounts, deactivate the volume group containing our root volume, and remove all the temporary mount-points we created:

$ sudo umount /mnt/root
$ sudo umount /mnt/boot
$ sudo vgchange -an pivg
$ sudo rmdir /mnt/root
$ sudo rmdir /mnt/boot

Now eject the SD card (or whatever storage you’re using) and you should find it boots happily on your Pi, with the root storage as LVM. Let me know below if you run into any issues with this!