Weeknotes 256

24th May, 2026

“DEGRADED state”

Hot innit.
Investigating unauthorized access to GitHub-owned repositories
Procrastination in it’s purest form raised it’s head this week as I decided to fix a Game Boy project that’s been sitting on the side, in plain view, for months. It is still not finished.
GDS weighs in on the NHS’s decision to retreat from Open Source
I made Focaccia, which was going swimmingly until I realised I’d left out one of the key ingredients after having folded and proved it four times over the course of a couple of hours. Still, I baked it anyway, and it was nice as all freshly baked bread tends to be. I will do it again some time given how it easy it was.
We continued the comedy this week with John Kearns, who was excellent. His support act, Pat Cahill, was also great and not someone who I’d heard of before, so that was a bonus.
I started reading Dungeon Crawler Carl.
I feel like every time I use an em dash now people are going to think I’ve used an LLM to write text, but the fact of the matter is that I was quite partial to the em dash before!

Some fun and games with my ZFS pool this week. I just happened upon the fact that ZFS thought the pool was in a DEGRADED state. Mild panic. (Incidentally, I really need to setup some sort of notifications for such events.)

$ zpool status
  pool: storage
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
  scan: resilvered 424K in 00:00:01 with 0 errors on Tue Apr 14 11:33:06 2026
config:

        NAME                      STATE     READ WRITE CKSUM
        storage                   DEGRADED     0     0     0
          mirror-0                DEGRADED     0     0     0
            sdf                   ONLINE       0     0     0
            10150489895668176262  FAULTED      0     0     0  was /dev/sdc1
          mirror-1                DEGRADED     1     0     0
            sdb                   ONLINE       0     0     2
            1079001000417129426   FAULTED      0     0     0  was /dev/sdd1
          mirror-2                ONLINE       0     0     0
            sda                   ONLINE       0     0     0
            sde                   ONLINE       0     0     0

It seems something had caused the disks to change names causing ZFS to react like “where my disks at?”. Understandable from ZFS’ point of view.

Then I realised that the disks which were found by ZFS where being referenced using traditional device names like /dev/sd* whereas I had setup the pool using the disk IDs from /dev/disk/by-id/ which is meant to bypass this exact kind of thing.

So why have the device names changed and why was my pool DEGRADED now? Well, I ran some Ubuntu updates by accident trying to update a single package. (I’ve done this before. Whether this is user error, or poor UX, or a combination of both is left up to you to decide. Just never update software.)

I’m not 100% sure what caused the issue, but this seems like a Linux device naming issue rather than a ZFS thing to me, and the udev was one of the packages updated. I’m not interested enough to investigate further.

After trying various things to get the disks online I figured the best way out of this mess was the classic turn it off and on again. I’d noticed that the system updates had also generated a new kernel image and if we haven’t rebooted we’re not running that image. I wondered if there was some Kernel vs userland incompatibility, and a reboot is the quickest way out of that.

I rebooted, and the missing disks were found, and ZFS kicked off the resilvering process automatically. It said it was going to take over a day to complete, but it was done in minutes instead.

I was left with a working pool, but some checksum errors so I decided to run a pool scrub, which ran overnight, and found no issues.

$ zpool status -P
  pool: storage
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub repaired 0B in 14:38:04 with 0 errors on Wed May 20 23:25:59 2026
config:

        NAME           STATE     READ WRITE CKSUM
        storage        ONLINE       0     0     0
          mirror-0     ONLINE       0     0     0
            /dev/sda1  ONLINE       0     0     0
            /dev/sdd1  ONLINE       0     0     1
          mirror-1     ONLINE       0     0     0
            /dev/sde1  ONLINE       0     0     0
            /dev/sdf1  ONLINE       0     0     0
          mirror-2     ONLINE       0     0     0
            /dev/sdc1  ONLINE       0     0     0
            /dev/sdb1  ONLINE       0     0     0

errors: No known data errors

I then cleared the errors with zpool clear storage and now I just have a message about not having access to the latest ZFS features – that’s for another day.

$ zpool status -P
  pool: storage
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 14:38:04 with 0 errors on Wed May 20 23:25:59 2026
config:

        NAME           STATE     READ WRITE CKSUM
        storage        ONLINE       0     0     0
          mirror-0     ONLINE       0     0     0
            /dev/sda1  ONLINE       0     0     0
            /dev/sdd1  ONLINE       0     0     0
          mirror-1     ONLINE       0     0     0
            /dev/sde1  ONLINE       0     0     0
            /dev/sdf1  ONLINE       0     0     0
          mirror-2     ONLINE       0     0     0
            /dev/sdc1  ONLINE       0     0     0
            /dev/sdb1  ONLINE       0     0     0

errors: No known data errors

I’m still running the pool using device names rather than the IDs I had originally setup. I would like to fix that. And I will one day.

This incident has actually increased my confidence in ZFS. It should be noted that the ZFS pool stayed online during all of this. Some disks went missing, and ZFS still maintained a working pool. This doesn’t seem like it was a ZFS problem.

I finished reading Say Nothing and started watching Say Nothing.
We went to see The Mandalorian and Grogu. It’s a long episode of the TV series, and I didn’t mind a bit. I’ll take all the Baby Yoda ya got!