Skip to content

Raspberrypi2 stability issues

Status: unresolved. This document records the symptoms, the theories we tested, and the current mitigations, so we don't lose progress between episodes.

The raspberrypi2 host (Raspberry Pi 4 8GB, Argon One M.2 case, powered USB3 hub with a Coral USB accelerator, a Western Digital MyBook disk, and a Sonoff Zigbee dongle) freezes or becomes unreachable at irregular intervals, over multiple years, power supplies, and peripheral configurations.

Symptoms

  • The network stack becomes unavailable: the host stops responding on the network.
  • The Argon One M.2 case (USB to SATA bridge) appears to disconnect.

  • USB resets in the kernel log:

    Apr 19 08:53:54 raspberrypi2 kernel: sd 0:0:0:0: [sda] tag#26 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD
Apr 19 08:54:02 raspberrypi2 kernel: sd 0:0:0:0: [sda] tag#26 CDB: opcode=0x85 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
Apr 19 08:54:02 raspberrypi2 kernel: sd 0:0:0:0: [sda] tag#18 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD OUT
Apr 19 08:54:02 raspberrypi2 kernel: sd 0:0:0:0: [sda] tag#18 CDB: opcode=0x2a 2a 00 00 20 33 28 00 00 20 00
Apr 19 08:54:02 raspberrypi2 kernel: scsi host0: uas_eh_device_reset_handler start
Apr 19 08:54:02 raspberrypi2 kernel: usb 2-2: reset SuperSpeed USB device number 3 using xhci_hcd
Apr 19 08:54:02 raspberrypi2 kernel: scsi host0: uas_eh_device_reset_handler success
Apr 19 08:54:02 raspberrypi2 udisksd[432]: Error performing housekeeping for drive /org/freedesktop/UDisks2/drives/KINGSTON_SA400M8240G_50026B768544D6F8: Error updating SMART data: Error sending ATA command CHECK POWER MODE: Unexpected sense data (g-io-error-quark, 0)

  • Metrics behavior is inconsistent across episodes: the original analysis observed Prometheus still collecting samples during an episode, and inferred from that that the file systems were still mounted. Later episodes show holes in metrics collection, so neither continued scraping nor the mounted-file-systems inference can be assumed. Record the metrics behavior of each episode as a data point.

Theories tested

  • Excessive USB power draw from the Raspberry Pi 4 USB hub: the Coral USB accelerator can draw a peak current of 900 mA (needs at least 500 mA), and the Kingston SSD draws up to 123 mA.
    • Tried a 4 A power supply: it didn't solve the issue.
    • Deployed a powered USB hub: the issue keeps occurring.
  • usb-persist is enabled, as expected.
  • Open lead: check whether the Argon One case has the extra pin to get power from the 5 V rail directly.

Similar experiences:

The 2026-09-13 freeze

The host hard-locked between 13:39 and 13:42 local time, with no kernel, undervoltage, thermal, or memory precursors in the logs or in the Prometheus history, and required a manual power cycle. The investigation is recorded in the manual changes diary and was distilled into the unresponsive host runbook.

The 2026-10-06 root storage stall

A partial variant: the host kept answering pings while everything that needed the root filesystem hung. Timeline (local time):

  • 09:27: a UAS device reset on the Argon One bridge (usb 2-2, scsi host0: uas_eh_device_reset_handler start ... success), logged as successful. The only precursor; memory, temperature, and load stayed normal.
  • 12:15: last journal entry of the boot. The journal could not write afterwards.
  • 12:18-12:19: the stall, visible in the last metrics the host's node exporter delivered: load average 28, 29 processes blocked on I/O, the root SSD (sda, behind the bridge) busy 477 ms per second while the data disk was idle, 5.8 GB of memory available, no OOM kill, CPU at 50 degrees Celsius.
  • 12:19 onwards: sshd reset every connection at the key exchange (from three different hosts, so not a per-source ban); node exporter, cAdvisor, Grafana, the media containers, and the Zigbee pipeline died over the next ten minutes. Containers already resident in memory (Prometheus, Alertmanager, blackbox exporter, restic exporter) kept serving: Prometheus kept appending samples throughout.
  • 12:42: manual power cycle. The root filesystem replayed its journal with one orphan inode.

Metrics behavior for this episode: scraping continued (the host's Prometheus kept running) but host-level collectors died, so the hl01 replica holds the usable history. Side effect: the smart plug series used for power measurements went offline on both replicas for 25 minutes, because the Zigbee plugs reach Home Assistant through Zigbee2MQTT on this host.

The armed hardware watchdog did not fire and could not have: systemd keeps petting it as long as PID 1 runs, and PID 1 needs no disk I/O to do so. A root I/O stall is a different failure mode from the hung kernel the watchdog guards against; the mitigation that matches it is tracked in the specs todo list.

The 2026-10-07 root storage stall

The same failure one day later, caught earlier in its course thanks to the previous episode's analysis. Timeline (local time):

  • 15:35: last journal entry of the boot. Nothing in the kernel log before it points at the storage: the last kernel lines are Docker network noise from 01:03.
  • 16:05: the host's Prometheus stopped answering; its data directory is on the root SSD.
  • 16:49: node exporter stopped answering scrapes. By the time the host was inspected it returned "Limit of concurrent requests reached (40)": every scrape since then hung inside its filesystem and disk collectors. Its last delivered sample showed the root SSD busy 477 ms per second with load and blocked processes still low, so the stall was recorded at its onset.
  • Until the power cycle: ping answered; sshd timed out from two different hosts (so not a per-source ban); cAdvisor, Grafana, Alertmanager, and part of the media stack kept serving from memory; the Zigbee pipeline was down, so the smart plug series stopped on both monitoring replicas.
  • 18:54: manual power cycle. The root filesystem replayed its journal with one orphan inode again.

Two findings:

  • The first thing to fail is the write path of the root filesystem, silently: journald stops persisting about an hour before anything is visible from the outside, and whatever the kernel reports about the device afterwards never reaches the disk. Two episodes in a row have an empty kernel log where the cause should be.
  • The root SSD sits behind the Argon One M.2 case's USB-to-SATA bridge, an ASMedia device (174c:1156, usb 2-2), bound to the uas driver. The WD data disk on the same USB 3 hub uses the plain usb-storage driver, and the Coral accelerator shares that hub.

Leading suspect

The UAS binding of the Argon One bridge is the first thing to change. The Raspberry Pi forum's reference guide for USB 3 storage on the Raspberry Pi 4 prescribes usb-storage.quirks=<vendor>:<product>:u in cmdline.txt for bridges with non-compliant UAS implementations, naming "frequent disconnects-reconnects" among the symptoms; the kernel documents the u flag as IGNORE_UAS (don't bind to the uas driver). The cost is throughput (150-200 MB/s instead of a 350 MB/s peak), irrelevant for this host. It has never been tried here: until 2026-10-07 the open item was to verify that UAS was enabled. For this bridge the line is usb-storage.quirks=174c:1156:u.

The outdated kernel remains the other plausible culprit, and the fix for it is the same either way: the host runs Debian 11 (bullseye), past LTS end of life, with an April 2023 kernel. The operating system upgrade is tracked in the specs todo list: a re-image with current Raspberry Pi OS, which is the most supported path for the Raspberry Pi 4 hardware (the Raspberry Pi OS documentation strongly discourages in-place upgrades, recommending a re-image instead). A NixOS migration was considered and deferred: reconsider it after the planned container migration to hl01 shrinks this host's role.

Current mitigations

  • The systemd hardware watchdog is armed: RuntimeWatchdogSec=15 (the maximum the BCM2711 platform watchdog supports), configured by the ferrarimarco_home_lab_node role, so a hard lockup should now self-reboot the host within 15 seconds. The end-to-end validation with a deliberate kernel crash is still pending, tracked in the specs todo list. It does not cover a root I/O stall (see the 2026-10-06 and 2026-10-07 episodes above): a complementary stall watchdog was designed and reviewed on 2026-10-12 (a resident probe service with a systemd failure action, plus kernel hung-task sysctls) and postponed until the OS re-image and a way to trigger the stall deliberately exist, since nothing today can show that the probe sees the real failure; the design is kept in the same list.

Other open follow-ups

Tracked in the specs todo list: the SMART long self-test on the WD30EZRX 3TB data disk, and the operating system upgrade. Additional ideas from the original investigation:

  • Monitor the USB power consumption (reference).
  • The Argon One case extra-pin check above.