It's Pi all the way down... - reliability

Loading the Deck

2025-02-27T00:00:00+00:00

I mentioned in my last post on this topic that fake SD cards have been an issue in the past.

Buy from well-known outlets, not minor vendors on whatever variant of flea-bay you frequent. Just because it’s on Amazon, doesn’t mean it’s not a knock-off; check the actual seller.

For that post, I also purchased a variety of SD cards of various models for testing purposes, some from Amazon. I also stated…

I haven’t had any issues for the last few years, but during them I’ve stuck to buying strictly from either SanDisk or Samsung […]

Guess how that’s going!

Another One Bites the Dust

Another month rolls by, and another Ubuntu release rolls off the production line. In this case it was Noble’s .2 release (24.04.2).

At each release, we go through a round of so-called “ISO tests” [1]. This invariably involves grabbing a handful of SD cards, flashing the release candidate image to them, booting them on various models of Pi model and running through a battery of tests. Once a board has passed or failed, pull the card, reflash, bung it in another board, rinse and repeat.

This means the life of an SD card in my possession boils down to one of two possibilities:

If it’s lucky, it gets stuck in a “long running” Pi serving something in my house. This means 24x7 duty for many years, but usually relatively light duty.
If it’s unlucky, it goes in the drawer and for much of the year it stays there. But, before each release, it’ll get hauled out and for a few days be brutally tortured with tens of gigabytes written to it over and over again.

If I have a card fail on me, it’s inevitably during a release. This time was no different, and I had a fancy gold-coloured “SanDisk Extreme” die. Rather than one of the usual failure modes (fails to read entirely, or silently ignores writes), this one wrote the candidate image happily but then reported tons of I/O errors during boot. A little odd, but not unprecedented.

What was strange was that this card was barely more than 1½ years old. This is an unusually short life-span, in my experience. Brutal though I am to these test cards, they usually last 4 or more years. There was no time to investigate during the release, so I tossed it in the “dead cards” drawer and moved on. However, the release is now done, and I’ve taken another look.

Aaaa!

When you insert an SD card into a Pi, if you’re watching the kernel log (and I often am in some window) you’ll typically see something like the following appear:

[344239.702588] mmc0: cannot verify signal voltage switch
[344239.771565] mmc0: new ultra high speed SDR104 SDHC card at address 59b4
[344239.771898] mmcblk0: mmc0:59b4 USD00 29.5 GiB
[344239.773131]  mmcblk0: p1 p2
[344239.773418] mmcblk0: mmc0:59b4 USD00 29.5 GiB

What does all this mean?

mmc0: cannot verify signal voltage switch: Basically noise; just a warning that the signal voltage was switched but we can’t be sure it was successful (hardware limitation).
mmc0: new ultra high speed SDR104 SDHC card at address 59b4: The interface has noticed a new ultra high speed (ooooh!) card which supports SDR104 signalling at “address 59b4”. Typically an interface only has a single card slot, but technically it can support several. This is the “address” of the card within that interface.
mmcblk0: mmc0:59b4 USD00 29.5 GiB: Mostly a repeat of the prior information. This says the kernel’s created a new “mmcblk0” device representing the card with address “59b4” on the “mmc0” interface, and it’s about 29.5GB in capacity…
mmcblk0: p1 p2: … and it’s got two partitions.

Incidentally, the above is from a Raspberry Pi branded SD card. Let’s see what the broken SanDisk Extreme reports:

[341496.190074] mmc0: cannot verify signal voltage switch
[341496.266122] mmc0: new ultra high speed SDR104 SDXC card at address aaaa
[341496.266891] mmcblk0: mmc0:aaaa SN64G 59.5 GiB
[341496.268624]  mmcblk0: p1 p2
[341496.269098] mmcblk0: mmc0:aaaa SN64G 59.5 GiB (quirks 0x00004000)

Huh. That “aaaa” address looks a bit odd.

This may be complete coincidence, but having come across fakes in the past, sometimes dodgy cards have “suspicious” (read: low entropy) fields in their registers. Serial numbers filled with zeros and the like. Querying other things (CID, manufacturers ID, etc) under /sys/block/mmcblk0/device yields nothing terribly suspicious here, but nonetheless, this feels a little odd so let’s dig a bit further.

Rather than throwing my dead cards in the bin, in recent years I’ve taken to keeping them in a drawer. Most still have some function, and I wondered if I could spot any patterns in their failure modes.

In the drawer I’ve got a “SanDisk Ultra” card (this is their “bog standard” range with grey rather than gold colouring on the case) which failed a year ago after a suspiciously short life-span. What does it report on insertion?

[341732.375548] mmc0: cannot verify signal voltage switch
[341732.448272] mmc0: new ultra high speed SDR104 SDHC card at address aaaa
[341732.448560] mmcblk0: mmc0:aaaa SD32G 29.7 GiB
[341732.450888]  mmcblk0: p1 p2
[341732.451108] mmcblk0: mmc0:aaaa SD32G 29.7 GiB (quirks 0x00004000)

Hmm. Another “aaaa”. Still, maybe that’s just the address that SanDisk cards use generally? Let’s try another SanDisk Ultra branded card which has so far worked reliably:

[345859.140220] mmc0: cannot verify signal voltage switch
[345859.209840] mmc0: new ultra high speed SDR104 SDXC card at address 59b4
[345859.210384] mmcblk0: mmc0:59b4 SD64G 59.4 GiB
[345859.212228]  mmcblk0: p1 p2
[345859.212795] mmcblk0: mmc0:59b4 SD64G 59.4 GiB (quirks 0x00004000)

So, the “aaaa” address appears common to two cards both purportedly from SanDisk, both of which failed suspiciously early.

Aaaamazon

Being more curious now I went and dug out my order history to see when and where these were purchased. I’m not 100% certain of which batch the 32GB “Ultra” card came in. However, I can say with confidence that both were purchased from Amazon, both from the “SanDisk Store” on there, and at least 6 months elapsed between the purchase of the 32GB “Ultra” card (latest February 2023) and the 64GB “Extreme” card (July 2023).

What to do?

Well, not much. It’s another fake from Amazon (which more and more resembles an online tat-store in my experience). Still, I figured in the interests of being a conscientious consumer, I should leave a review cautioning others that even buying from the official “SanDisk Store” is not a cast-iron guarantee of quality. I spent 5 minutes throwing together a quick précis of the above (“aaaa reported by kernel”, “substantially shorter lifespan than other SanDisk cards”, etc. etc.)

And I would’ve left it at that. But for the email that just arrived…

Please edit and resubmit your review

Hello Dave Jones, We could not post your review because it does not meet our community guidelines. Please edit and resubmit your review. Before you do, make sure it meets all of our guidelines.

Intriguing. I wonder which “community guidelines” I’ve violated. Unfortunately, while they’ve linked to their guidelines (which are pretty much as one would expect) they haven’t specified which one I violated, and the review is apparently gone. So I’ve no clue.

Naturally, this leaves me feeling less-than-confident in the positively stellar review ratings alongside this product.

Though, again, hilariously skewed reviews are hardly a new feature of Amazon either. More importantly: can I still stand by my former advice? Probably not.

eBay, Amazon, Alibaba, whoever. If it’s a large marketplace, I’ve had fakes from it (certainly from the aforementioned big three). Ideally, buy from your local electronics vendor (Pimoroni or The Pi Hut for me) but failing that, look for the larger vendors on the big marketplaces.

[1]	ISO because this used to be limited to ISO CD images. Though we still make ISO images for various platforms, we’re just testing raw disk images for the Pi.

Dealing the (SD) Cards

2023-09-12T00:00:00+01:00

Updated to remove large outlet recommendations, and fix footnote

It came to my attention recently that many people don’t quite know what to look for in a micro-SD card for use in a Pi. As mercilessly torturing micro-SD cards is something of a hobby round here I thought I’d write a quick guide to what to look for.

A small selection of the SD cards that have graced my desk over the years, ranging from the rubbish to the really rather good.

Vegas Standard

There have been many standards for SD cards to conform to over the years, but the vast majority of them are concerned with sequential transfer rates. These started with the so-called speed classes, where Class 4 implied that a card could sustain a write speed of 4MB/sec to the card, Class 6 implied 6MB/sec and so on.

Later there were the UHS speed classes, where U1 was equivalent to the older Class-10, a sustained write speed of 10MB/sec to the card. U3 implied 30MB/s sustained write speed. Why a new class system? Because UHS also required a specific bus speed (confusingly designated UHS-I, then UHS-II, etc). Hence, a U1 card must manage 10MB/sec minimum but the corresponding bus speed could top out at 104MB/s allowing considerably faster read speeds (depending on bus compatibility).

Later still came the video speed classes. These still use the UHS bus interfaces, but demand higher minimum sequential write speeds. V10 requires 10MB/s minimum, V60 requires 60MB/s minimum and so on. Here’s a nice little table outlining the minimums required by each of these classes.

Minimum write performance	Speed Class	UHS Speed Class	Video Speed Class
2MB/s	The SD Class 2 logo, a “2” within a stylized “C”	—	—
4MB/s	The SD Class 4 logo, a “4” within a stylized “C”	—	—
6MB/s	The SD Class 6 logo, a “6” within a stylized “C”	—	The SD video 6 logo, a stylized “V” almost like a square root symbol followed by a “6”
10MB/s	The SD Class 10 logo, a “10” within a stylized “C”	The SD UHS 1 logo, a “1” within a stylized “U”	The SD video 10 logo, a stylized “V” almost like a square root symbol followed by a “10”
30MB/s	〃	The SD UHS 3 logo, a “3” within a stylized “U”	The SD video 30 logo, a stylized “V” almost like a square root symbol followed by a “30”
60MB/s	〃	〃	The SD video 60 logo, a stylized “V” almost like a square root symbol followed by a “60”
90MB/s	〃	〃	The SD video 90 logo, a stylized “V” almost like a square root symbol followed by a “90”

Got all that? Good. Now you can forget it.

As I mentioned above all of this is to do with sequential access. This is very useful in a camera where you’re recording video frame after video frame (or still image cameras that do … erm … exactly the same thing only much slower with considerably larger frames!). However, this doesn’t remotely resemble the access pattern of an operating system on an SD card which will typically entail hundreds of small writes to small files spread across a multitude of sectors.

In other words these classes will tell you next to nothing about the performance of your card in a Raspberry Pi. For that you need guarantees about the minimum random access performance. Luckily, the recent SD specifications have a standard for that too: the application performance classes.

Poker Face

At the time of writing, two application classes are defined: A1 which is fairly common on decent cards, and A2 which is still somewhat “premium”. Their minimum guarantees are as follows:

Minimum read performance	Minimum write performance	Application class
1500 IOPS	500 IOPS	The SD video application-performance logo, a stylized “A” followed by a “1” with “app performance” in capitals below
4000 IOPS	2000 IOPS	The SD video application-performance logo, a stylized “A” followed by a “2” with “app performance” in capitals below

Note that these guarantee both write and read minimums. Further, they’re not concerned with bytes transferred, but with IOPS, or I/O operations per second. The exact meaning of IOPS is rather loose across storage generally but here it means the number of block sized (4KB) read or write commands that can be carried out per second. Even this is relatively fuzzy because the controller in an SD card is free to batch operations together. Erasure is almost always batched into “extents” of blocks, and writes can be generally too.

In addition to the above requirements, A1 and A2 both require a minimum of 10MB/s sequential write speed (equivalent to the old Class 10 cards). This may appear excessive. After all, if A1 only demands that a card handle 500x 4KB writes per second, that only equates to 2MB/s? However, remember that cards are free to batch operations together. Cards that do not conform to one of the application classes may well display 10MB/s sequential writes to one large file by batching together smaller writes that are all in sequence. However, if asked to perform many small 4KB/s writes to a multitude of blocks, the performance suddenly drops off a cliff.

In my experience, any reputable card with an A1 mark performs “well” on a Pi. A2 cards from a given vendor perform better than A1 cards from that same vendor, but I’ve also had A1 cards from vendor A beat A2 cards from vendor B. But when I say “beats”, I mean in performance benchmarks. I cannot say I ever noticed a difference in daily usage.

To put it another way: A1 is the sweet spot right now, and A2 is probably a waste of money at this point unless you’re quite sure you absolutely need the performance.

Shooting Craps

The astute reader may have noticed my careful use of the word “reputable” just before “card” in the prior section. Fakes are a problem, or have been in my experience (though we’re going back several years here).

Buy from well-known outlets, not minor vendors on whatever variant of flea-bay you frequent [1]. Just because it’s on Amazon, doesn’t mean it’s not a knock-off; check the actual seller.

Cheap, bulk deals are especially to be avoided. There’s enough competition in the micro-SD card market that there’s usually little that can be (legitimately) saved by buying in bulk; if the unit price is substantially below what you could get the individual cards for, be wary!

To summarize:

Buy well-known brands,
from reputable vendors,
avoiding bulk (or suspiciously cheap) deals.
Look for A1 or A2 on the casing [3]. Any modern card without those marks is likely to be rubbish for use in an SBC like a Raspberry Pi.

The Long Game

I haven’t had any issues for the last few years, but during them I’ve stuck to buying strictly from either SanDisk or Samsung (also had a Toshiba card that lasted many years without issue, but it’s a singleton sample), and avoiding any cheap-looking bulk deals.

This is not to say these cards last forever. But I have several Pis that run 24x7, and “good” cards seem quite capable of lasting at least 5 years in them, which is “good enough” for my purposes (I’ve had hard drives last less than that before!).

The failure modes of micro-SD cards seem to fall into two camps:

completely dead — the card can be detected in a reader but fails to mount at all
going “read-only” — the card will happily mount, and read pretty much anything but all writes are silently thrown away

The latter sounds preferable, and in some ways it is because if you have any data you want to recover from it, you can do so. But it’s also worse in certain circumstances.

Consider a board where the card “goes read-only”. It may appear to continue operating quite happily (if it’s serving a static site for instance), but all writes to logs are being thrown away. It’s also quite confusing to diagnose if you’re not specifically looking for it because the card may appear to be operating normally.

When dealing with a card I know to be old one of the first things I’ll try (assuming I can mount it at all), is touch foo, remove and re-mount the card, followed by ls just to see if an empty “foo” file shows up. If it doesn’t, the card’s almost certainly gone read-only and it’s time to make a fresh one.

[1]	eBay, Amazon, Alibaba, whoever. If it’s a large marketplace, I’ve had fakes from it (certainly from the aforementioned big three). Ideally, buy from your local electronics vendor (Pimoroni [2] or The Pi Hut for me) but failing that, look for the larger vendors on the big marketplaces.

[2]	Unfortunately, at the time of writing, Pimoroni’s SD card offering doesn’t include shots of the front of the card. Being a reputable company, I’d be extremely surprised if they weren’t good quality A1 cards, but it’d be nice to see this at least suggested visually! Oooh, they’re A2 cards!

[3]	Assuming you can make it out from the cacophany of logos. Is it just me, or are micro-SD card cases beginning to look like the side of a Formula 1 car?