Manual changes diary

The design of this home lab is to have a completely declarative configuration, whenever possible. At times, this is not possible yet because of an architectural or tooling limitation, so we perform manual changes. The goal is to eventually get to a state where we don't need this file anymore.

2026-10-11

pve1: destroyed the empty rpool-usb-1/media dataset (zfs destroy rpool-usb-1/media, 120 KB of directory skeleton, no snapshots, no remaining mount or share reference) now that the media library lives on rpool-usb-2/media; the dataset's inventory declaration was dropped in the same change, since the setup-disks role would otherwise recreate it.

2026-10-10

pve1: created the rpool-usb-2 ZFS pool on the 2 TB LaCie USB disk for replaceable media (pool layout decided 2026-09-26; the setup_disks role asserts pools and documents the command but never creates them):
```
zpool create -f -o ashift=12 rpool-usb-2 \
  /dev/disk/by-id/usb-LaCie_P9230_0000000035741a1320ed-0:0
```
-f was needed for the disk's leftover Windows partition table. The rpool-usb-2/media dataset and its ownership came from the setup-disks playbook, and the pool's Proxmox storage registration from Terraform.
pve1: re-homed the NAS container's media-usb bind mount from /rpool-usb-1/media to /rpool-usb-2/media with pct set 200 -mp2 /rpool-usb-2/media,mp=/mnt/shared/media-usb and one container reboot, the fast-forward the NAS spec prescribes for mount point changes; the next Terraform run then planned no changes. The old, empty rpool-usb-1/media dataset is still declared and present, pending cleanup.

2026-10-05

pve2: plugged back in (BMC up at its DHCP lease) and powered on for the second round of power experiments; the node exporter was deployed with the monitoring playbook, which resolved the InstanceDown and HTTP probe alerts for the host. Manual runtime changes, none persisted:
- PCIe ASPM policy set to powersave again; it lasts until the next boot.
- PCI runtime PM set to auto on 99 devices for a measured window, then restored to on by its own 20-minute timer.
- SATA med_power_with_dipm on host0 was refused by the kernel: the firmware flags every port hot-plug-capable and both controllers lack the aggressive link power management capability (details in the host power guide).
- Supermicro Update Manager was copied to /root/sum, run in-band once to export the BIOS configuration (GetCurrentBiosCfg), and removed again.

2026-10-04

pve2: powered on for the power reduction investigation (goal: run the host 24/7 once its idle draw comes down; see the specs todo list). Findings that extend or correct the 2026-08-25 entry:
- The smart plug series in Home Assistant (scraped by Prometheus) confirms the baseline across three power-on windows: ~24 W with pve2 off, 67-71 W with pve2 on and idle with the nas-pve2 guest running, ~110 W at boot.
- RAM is 128 GB (8 x 16 GB DDR4-2133 RDIMM at 1866 MT/s), not 256 GB.
- PSU: OCZ ModXStream Pro 600 W (OCZ600MXSP), a 2008-era 80 Plus unit with no PMBus: the BMC's DCMI power reading and the kernel's ACPI power meter both read 0 W.
- BMC readings via ipmitool (now installed through the bootstrap role, has_bmc host flag): fan mode already Optimal, fans at 300-1100 RPM (FAN1 reads below its lower threshold), CPU 36 degrees Celsius, PCH 50, DIMMs 32-37. The fan lever is negligible.
- Settled idle: RAPL package 7.6 W, DRAM 3.5 W, cores 99.8% in C6. The untapped levers from the 2026-08-25 entry are still untouched.
- The cable is on nic0 (link up) while the inventory assigns the reserved address to nic1 (link down); not a power issue, to reconcile.
pve2: ran pvecm updatecerts by hand to restore the Proxmox authorized_keys symlink after a bootstrap playbook run replaced it with a regular file. The bootstrap role now converges the link target and declares the node's own root key, so the shared file stays stable across runs and reboots (see the Ansible development guide); the two hardware security keys that the file carried were dropped on purpose.
pve2: set the PCIe ASPM policy to powersave at runtime (19:17 local) as the first power tuning experiment, with a 20-minute self-reverting timer that was cancelled after the health checks passed. Both i210 links moved to L0s/L1 with no errors; the plug median stayed at 68 W over 3.3 hours, so the gain is within the plug's 1 W resolution. The setting lasts until the next boot; nothing persists it yet. The host was shut down the same evening; the plug read 25 W with it off and 20 W with it unplugged, so its standby draw is 5 W. Measurements and the query recipe are in the host power guide.

2026-09-13

raspberrypi2: manually power-cycled after the host froze (hard lockup between 13:39 and 13:42 local time; no kernel, undervoltage, thermal, or memory precursors in logs or Prometheus history). The investigation was distilled into the unresponsive host runbook; follow-ups (hardware watchdog, data-disk SMART test, OS upgrade) are tracked in the specs todo list.

2026-08-25

pve2: measured the idle power baseline (Supermicro X10SRL-F, Xeon E5-2683 v4, 256 GB RAM, kernel 6.17.2-1-pve, no VMs or containers running):
- Wall power: ~45 W (smart plug reads 67 W with pve2 idle and 22 W with pve2 off; other devices share the plug, so 22 W is the plug baseline). At 0.099 EUR/kWh, 45 W around the clock costs about 39 EUR/year.
- RAPL package-0 ~8.7 W, DRAM ~3.3 W; package C6 residency 88.5%, cores ~99% cc6. The remainder is HDDs, platform, fans, and PSU losses.
- Untapped levers (powertop): governor performance on intel_cpufreq (passive), all 10 SATA hosts at max_performance link power policy, PCIe ASPM policy default, most PCI devices with runtime PM on. Realistic tuned idle is ~30-35 W (~8-13 EUR/year saved). HDD spin-down is excluded by policy: pve2 keeps its platters spinning to avoid start/stop wear.
- Duty-cycle reduction (shutdown plus wake-on-LAN) is the bigger lever if pve2 does not need to run around the clock.
- ipmitool is not installed on pve2; lm-sensors only shows x86_pkg_temp. The ASPEED BMC may expose DCMI power readings if the PSU has PMBus.
- The short hostname pve2 does not resolve from the workstation; use pve2.edge.lab.ferrari.how.

2026-08-12

pve1: updated the Beelink BIOS from N95V104 to N95V107 to fix the Eaton UPS (Eaton Ellipse PRO 1600 DIN, USB ID 0463:ffff, usbhid-ups driver for nut-driver@eaton-ellipse-pro-1600-din-1; NUT runs on pve1) failing USB enumeration at cold boot (device descriptor read/64, error -71), which previously only a physical replug recovered. If it recurs:
- Software recovery on the root port is not possible: the hub has no per-port power switching, and the self-powered UPS only resets its USB state machine on VBUS loss. Cycling usb1-port1/disable in sysfs and the old-scheme enumeration quirk were both tried and failed.
- Fallback plan: place a PPPS-capable USB 2.0 hub (verified with uhubctl) between the UPS and the host, enabling a true software replug via uhubctl -a cycle.
- Keep the Toshiba USB disk (backing hl01's rpool-usb-1 885G volume, same xHCI controller) directly on the root port, and never unbind the xHCI driver while hl01 is running.

2026-08-11

pve1: cleaned the clogged heatsink fins and replaced the degraded thermal paste after the host (Beelink EQ-series, Intel N100) throttled to its 200 MHz floor, pinning VM hl01 at 100% CPU and pushing the Coral PCIe TPU past its 94.8 degrees Celsius trip point. Diagnostics learned:
- Trust platform_coretemp_0 / x86_pkg_temp readings; the acpitz thermal zone on this board reads a bogus ~27.8 degrees Celsius regardless of load.
- The fans are EC-controlled and invisible to the OS (no hwmon fan sensors), so "fans look fine in monitoring" carries no information on this host.
- Coral PCIe TPU health signatures: healthy inference is ~5-10 ms (Frigate /api/stats); ~200 ms means thermal throttling; the coral_pci_temperature_celsius metric reads -89.7 when the TPU has crashed. A crashed TPU can also fail to initialize at the next boot, with the apex driver logging "RAM did not enable within timeout (12000 ms)" and "Couldn't initialize interrupts: -12".
- Check CPU throttling directly via /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq.
- Preventive maintenance: clean the heatsink fins yearly, in early summer.

2026-04-23

pve2: manually created (through the Proxmox GUI) two new ZFS pools: tank-hdd-scratch and tank-ssd-scratch. They are intended to be used as scratch space for temporary files. Rationale: ZFS Ansible modules are not idempotent, and I didn't find a suitable ZFS Terraform provider.

2026-02-28

pve2: updated the BMC firmware to 3.93, and configured the lower thresholds for the FAN1 and FAN2 sensors. The hardware operations guide describes the procedure.

2025-09-15

gateway: updated the Asus RT-AX86U firmware to 3.0.0.4.388_24386.

2025-04-03

Notes to create the 100 VM

The following notes describe the manual creation of the 100 VM on pve1. These instructions are a reference that I used as a base to define the 100 VM Terraform configuration.

Create VM:

qm create 100 --name hl01 --net0 \
  virtio=BC:24:11:D4:F6:64,bridge=vmbr0,firewall=1 \
  --scsihw virtio-scsi-single \
  --machine q35 --ostype l26

Configure CPU and memory:

qm set 100 --cpu host --cores 2 --memory 8192

Configure the VM to start at boot:
```
qm set 100 --onboot 1
```
Enable QEMU guest agent:
```
qm set 100 --agent enabled=1
```

Import raw disk:

qm disk import 100 \
/var/lib/vz/template/raw/debian-12-generic-amd64-20240211-1654.raw local-zfs

Attach and configure disk to the vm:

qm set 100 -scsi0 \
  local-zfs:vm-100-disk-0,discard=on,iothread=1,size=2G,ssd=1,aio=io_uring

Resize:
```
qm disk resize 100 scsi0 8G
```
Set boot order:
```
qm set 100 --boot order=scsi0
```

Attach and configure a second disk from the rpool-sata pool:

qm set 100 -scsi1 \
  rpool-sata:vm-100-disk-0,discard=on,iothread=1,size=113G,ssd=1,aio=io_uring

Attach and configure a third disk from the rpool-usb-1 pool:

qm set 100 -scsi1 \
  rpool-usb-1:vm-100-disk-1,discard=on,iothread=1,size=885G,ssd=1,aio=io_uring

Enable UEFI and create a UEFI disk volume:
```
qm set 100 --bios ovmf
```
Configure UEFI disk volume:
```
qm set 100 --efidisk0 local-zfs:0,efitype=4m
```
If you need to enable Secure Boot, add the pre-enrolled-keys=1 option.

Configure cloud-init datasource:

qm set 100 --cicustom
"user=local:snippets/cloud-init-hl01-user-data.yaml,network=local:snippets/cloud-init-hl01-network.yaml"

Configure cloud-init drive:

qm set 100 --ide2 local-zfs:cloudinit,media=cdrom

Pass the Coral PCIe module to the VM, mark it as a PCIe device:
```
qm set 100 --hostpci0 0000:03:00,pcie=1
```
Pass the iGPU to the VM, mark it as a PCIe device, make the firmware ROM visible to the guest, set it as the primary GPU:
```
qm set 100 -hostpci1 0000:00:02.0,pcie=on,rombar=on,x-vga=on
```
Start the VM:
```
qm start 100
```

2025

Exact date unknown.

Fury 2: installed BIOS version 3801.

2023-05-03

gateway: fixed the 2.4 GHz Wi-Fi control channel to 1, trying to address the ESP32 disconnection issue. The issue still occurs after this change.

2023-04-27

gateway: disabled WMM APSD on the 2.4 GHz Wi-Fi network, trying to address the ESP32 disconnection issue.

2023-04-26

gateway: disabled target wake time on the 2.4 GHz Wi-Fi network, trying to address the ESP32 disconnection issue.

2023-03-30

gateway: updated the Asus RT-AX86U firmware to 3.0.0.4.388_22525. The exact date is approximate.

2023-02

Exact days unknown.

gateway: on the 2.4 GHz Wi-Fi network, trying to address the ESP32 disconnection issue:
- Disabled 802.11ax.
- Disabled Wi-Fi Agile Multiband.
- Disabled Protected Management Frames.

2023

Exact date unknown.

gateway: disabled Smart Connect for both the 2.4 GHz and 5 GHz Wi-Fi bands, to fix cam-1 disconnecting from the network after a power loss (the camera stopped responding to pings).

2022-05-16

gateway: updated the Asus RT-AX86U firmware to 3.0.0.4.386_46061 at 9:30.