Design Spec: Monitoring Alerting

Implementation Status

Component / Feature	Status	Details
Alertmanager Service	Fully Implemented	`prom/alertmanager` service in the monitoring backend compose template; deployed and healthy on raspberrypi2 (§3).
Alertmanager Configuration	Fully Implemented	Severity-aware routing and the Telegram receiver; end-to-end delivery verified with a synthetic alert (§4, §5).
Prometheus Alerting Wiring	Fully Implemented	Rule file loading, the Alertmanager target, and the Alertmanager scrape job; scrape target healthy (§3.2, §7).
Alert Rules: Availability	Fully Implemented	`InstanceDown` deployed; surfaced real down targets on first evaluation (§6.1).
Alert Rules: Node Health	Fully Implemented	Unexpected reboots, textfile staleness, filesystem space and inodes, failed systemd units, clock skew, and memory errors (§6.2).
Alert Rules: Temperature	Fully Implemented	Generic CPU temperature, Coral TPU temperature, and Coral sensor failure (§6.3).
Alert Rules: Backups	Fully Implemented	Restic backup staleness, repository identity, structural and data-read checks, and the monthly restore drill (§6.4).
Alert Rules: Blackbox Probes	Fully Implemented	ICMP, DNS, and HTTP probe failures (§6.5).
Alert Rules: Frigate	Fully Implemented	Frigate metrics scrape job plus camera stream, capture rate, and detector latency rules (§6.6).
Alert Rules: Containers	Fully Implemented	Restart-loop and OOM-kill detection on the cadvisor metrics (§6.7).
Alert Rules: Syncthing	Fully Implemented	Scrape job and rules deployed; target up and rules loaded on both replicas (2026-10-03, §6.8).
Alert Rules: SMART Disk Health	Fully Implemented	SMART health verdict and sector-growth rules on the smartmon textfile metrics (§6.9).
Alert Rules: UPS	Fully Implemented	On-battery and low-battery rules on the NUT flag-encoded status (§6.10).
Alert Rules: NixOS Deployments	Fully Implemented	comin fetch and deployment failure rules on the nix-comin job (§6.11).
Alert Rules: Monitoring Pipeline	Fully Implemented	Notification delivery, configuration reload, rule evaluation, and Alertmanager cluster health (§6.12).
Restart Policy Migration	Fully Implemented	All four monitoring backend services run with `restart: unless-stopped`, verified via `docker inspect` after deployment (§8).
HA Pair: Prometheus Replicas	Fully Implemented	Deployed on hl01 and raspberrypi2; cross-scrapes verified up from both replicas, hl01 seeded from the raspberrypi2 TSDB with history queryable (§3.3).
HA Pair: Alertmanager Cluster	Fully Implemented	Gossip cluster healthy over host networking; silence replication both ways and exactly-once Telegram delivery verified (§3.3).
HA Pair: Per-Host Grafana and Blackbox	Fully Implemented	Grafana and the Blackbox exporter run on both replica hosts from identical templates; all backend targets up (§3.3).

1. Goal

Add alerting to the existing Prometheus-based monitoring stack so that known failure modes are pushed to the operator instead of waiting to be noticed on a Grafana dashboard. The design deploys Prometheus Alertmanager next to the existing Prometheus instance, defines a first catalogue of alert rules covering the gaps identified during past incidents (thermal events, silent hardware-watchdog recoveries, stale exporters, backup failures, unreachable targets), and routes notifications to Telegram.

2. Rationale

2.1 Why Alertmanager

Prometheus already scrapes every signal the known gaps need; what is missing is evaluation and delivery. Alertmanager is the Prometheus-native component for that role: rules are evaluated by Prometheus itself from version-controlled rule files that live and deploy alongside the scrape configuration, while Alertmanager owns deduplication, grouping, silencing, and delivery.

Rejected alternatives:

Grafana-managed alerts: rejected because alert definitions would live in Grafana's database (or a second provisioning pipeline) instead of the same templated configuration that defines scraping, and because delivery would depend on the dashboard layer being healthy.
Home Assistant automations on Prometheus data: rejected because Home Assistant is itself one of the monitored (and historically flaky) workloads; the alerting path must not depend on a monitored service.
Uptime Kuma or similar standalone watchers: rejected for this scope because they duplicate checks Prometheus already performs; Uptime Kuma remains separately tracked in the specs index as a possible complement for external, outside-in checks.

2.2 Scope

This spec covers the alerting pipeline and the first rules catalogue. It does not cover new exporters or new metrics: every rule in §6 evaluates data the stack already collects.

3. Architecture

3.1 Deployment Model

Alertmanager runs as an additional service in the monitoring backend Docker Compose stack, on the same host as Prometheus and Grafana. It is enabled by the same mechanism as the rest of the stack (the monitoring backend enablement flag): a host that runs the monitoring backend runs Alertmanager, with no separate enablement flag. The hosts that enable the flag form the backend replica set: the stack deploys identically to each of them, and every generated reference to the backend iterates the replica set instead of assuming a single instance (§3.3).

Key properties:

Pinned image: the Alertmanager container image is pinned in the dependency-updates helper Dockerfile like every other image in the stack, so Renovate manages its updates.
Persistent state: Alertmanager stores silences and notification state in a dedicated data directory under the monitoring backend runtime data directory, so container recreation neither drops active silences nor re-sends already-delivered notifications.
Host port exposure: the Alertmanager API and UI are published on host port 9093, like the Prometheus API on 9090. These ports, together with the Alertmanager cluster port 9094, are LAN-reachable between the backend replica hosts, which the pair requires for cross-scraping, alert delivery to every replica, and cluster gossip (§3.3). Operator access stays over SSH during incident investigation, for amtool operations, and for managing silences.
Restart policy: restart: unless-stopped, together with the rest of the stack (§8).

3.2 Data Flow

Each Prometheus replica loads the identical alerting rules from a dedicated, version-controlled rule file rendered by the same configuration mechanism that renders prometheus.yaml, and evaluates them on its global evaluation interval (1 minute).
Each replica sends its firing alerts to every Alertmanager replica by host FQDN, with the replica-identifying label stripped so the copies deduplicate (§3.3).
The Alertmanager cluster groups, deduplicates, and routes them to the Telegram receiver (§4): one notification per alert, regardless of how many replicas fired it.
Each Prometheus replica also scrapes every Prometheus and Alertmanager instance in the replica set, so a dead or unhealthy backend component surfaces as a down target on the surviving replica (§7).

3.3 High Availability Pair

The monitoring backend runs as a two-replica pair on hl01 and raspberrypi2. The replica hosts sit in different failure domains: hl01 is a VM on pve1, raspberrypi2 is standalone hardware, so a backend that ran only on hl01 would die together with the pve1 workloads it watches. The pair also lets either host be taken down for maintenance (notably the planned raspberrypi2 re-image) without a monitoring blind spot.

Prometheus replicas: both instances scrape the same generated target lists, which include every Prometheus and Alertmanager replica by host FQDN; the configuration contains no localhost or Compose-network self-references. Each instance carries a replica-identifying external label (replica: <hostname>), which is stripped from outgoing alerts through alert relabeling so the Alertmanager cluster deduplicates the copies. There is no state replication between the replicas: each accumulates its own TSDB, and divergence between them is accepted.
TSDB seeding: a replica joining the pair is seeded from a consistent copy of an existing replica's TSDB (the snapshot API, or a copy taken while the source instance is stopped), so the metrics history collected before the pair existed stays queryable from every replica. After seeding, the copies evolve independently. The same pattern seeds Grafana's local database (user accounts, preferences) from an existing replica, because the Grafana credentials are not yet declaratively managed; making them declarative is tracked in the specs index.
Alertmanager cluster: the two instances form a gossip cluster over port 9094 between the replica hosts. The cluster deduplicates notifications and replicates silences, so a silence survives the loss of either replica. During a network partition between the replicas the deduplication degrades and Telegram may briefly receive duplicate notifications: accepted as the cost of keeping notification delivery free of a single point of failure.
Grafana: one instance per replica host, provisioned identically from the same templates, each using its local Prometheus replica as its datasource. Grafana is not in the alerting path, so no cross-host coupling is introduced.
Blackbox exporter: one instance per replica host, giving the probes two vantage points; duplicate probe alerts deduplicate in the Alertmanager cluster like every other alert.

Rejected alternatives:

Primary/standby notification delivery (only one Alertmanager notifies unless the primary is down): rejected because it reintroduces the asymmetry and failover logic the pair exists to remove; occasional duplicates during partitions are the cheaper failure mode.
A global-view layer (Thanos, Mimir, VictoriaMetrics) merging the replicas behind one query endpoint with long-term object storage: rejected as operational weight a two-node lab does not need; revisit if long-term retention or a merged query view becomes a goal.
Remote read between the replicas to paper over history divergence: rejected because it couples the replicas at query time, making each depend on the other's availability for historical queries.

4. Notification Channel

Notifications are delivered to Telegram through Alertmanager's native Telegram integration, using a dedicated bot and chat.

Rejected alternatives:

Webhook into Home Assistant (reusing its existing Telegram bot): rejected because it makes alert delivery depend on Home Assistant, a monitored service with known restart and DNS issues; the alerting path must stay independent of the systems it watches.
SMTP email: rejected because the lab has no existing mail infrastructure; it would add an external dependency and another credential for a slower notification channel.
Self-hosted push services (ntfy, Gotify): rejected because they add a new always-on service to operate, and the managed alternative (Telegram) is reliable and already in use in the lab.

4.1 Secrets

The bot token and chat identifier are secrets and follow the repository secrets policy: they are stored as vaulted variables in the untracked group-scoped Ansible vault (group_vars/all) and referenced from the Alertmanager configuration template:

vault_monitoring_backend_alertmanager_telegram_bot_token
vault_monitoring_backend_alertmanager_telegram_chat_id

The rendered configuration file on the host contains the token, matching the existing posture of the rendered Prometheus configuration, which embeds the Home Assistant bearer token.

5. Routing and Severities

Every rule carries a severity label with one of two values:

critical: conditions that risk data loss, hardware damage, or an ongoing outage (host down, overheating, failed backup integrity check).
warning: degradations that need attention but not immediately (stale textfiles, aging backups, failing HTTP probes).

Both severities route to the same Telegram receiver; the severity changes the re-notification cadence, so chronic warnings do not drown urgent alerts:

Grouping: by alert name and instance, with a short group wait (~30 s) and a group interval of ~5 minutes.
Repeat interval: ~4 hours for critical, ~24 hours for warning.

5.1 Planned Downtime

Hosts that are deliberately powered off (for example a Proxmox node shut down on purpose) will fire availability alerts. Planned downtime is handled with Alertmanager silences (via amtool over SSH or the UI), not with configuration changes.

Rejected alternative: a per-host "intermittent" classification in the inventory that would route availability alerts for such hosts at lower severity. Rejected for now as premature complexity in a single-operator lab where creating a silence is a single command; revisit if planned power cycles become frequent enough that missing silences produce recurring noise.

6. Alert Rules Catalogue

Rules are grouped by theme. Conditions are stated as the contract each rule implements; thresholds and durations are the initial values and may be tuned with operational experience.

6.1 Availability

Alert	Severity	Condition	Duration	Rationale
`InstanceDown`	critical	`up == 0`	10 min	An exporter or host stopped answering scrapes; covers unreachable targets and dead exporters.

6.2 Node Health

Alert	Severity	Condition	Duration	Rationale
`UnexpectedReboot`	warning	`max_over_time(node_boot_time_seconds[1h]) - min_over_time(node_boot_time_seconds[1h]) > 10`	—	Reboots (including hardware-watchdog recoveries) are noticed instead of silently absorbed. The boot time derives from the wall clock, so an NTP correction moves it by a second (raspberrypi2 paged for an hour on 2026-10-12 without rebooting); a reboot moves it by the previous uptime, so only a spread over ten seconds counts, and a crash loop shorter than that is `InstanceDown` territory.
`AptUpdateStale`	warning	`time() - apt_package_cache_timestamp_seconds > 2 * 86400`	—	The apt collector reports the time of the last successful `apt-get update` (the roles deploy an apt hook that touches the success stamp, and the current `apt_info.py` reads it; older versions reported the last attempt). Two missed daily runs mean a dead source or a stopped job; the Coral repository outage of 2026-09-29 went unnoticed for two weeks without this.
`AptMetricsMissing`	warning	`node_os_info{id="debian"} unless on (instance) apt_package_cache_timestamp_seconds`	1 h	A Debian host without apt metrics: the collector is not deployed (hl01 was never converged with the stack until 2026-10-12) or its timer stopped.
`NodeTextfileStale`	warning	`time() - node_textfile_mtime_seconds{file!~".\\.static\\.prom"} > 26 3600`	—	A textfile collector stopped updating. 26 h covers the slowest producer (the daily apt job); per-collector tuning is future work. Files named `*.static.prom` are rendered by Ansible with no producer timer and are exempt.
`NodeFilesystemSpaceLow`	warning	`node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.15` (tmpfs and ramfs excluded)	30 min	Early signal that a filesystem is filling up, with time to react; motivated by the September 2026 incident where the hl01 root filesystem silently reached 99%.
`NodeFilesystemSpaceCritical`	critical	`node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.05` (tmpfs and ramfs excluded)	10 min	Services writing to the filesystem are about to fail; pages before workloads (databases, recordings) start erroring.
`NodeFilesystemInodesLow`	warning	`node_filesystem_files_free / node_filesystem_files < 0.10` (tmpfs and ramfs excluded)	30 min	Inode exhaustion fails writes even with free space left; symmetric with the space rules.
`ExpectedMountMissing`	critical	`node_mount_expected == 1 unless on (instance, mountpoint) node_filesystem_size_bytes`	10 min	A mount the host's inventory declares (`disks_to_mount`, rendered into `expected-mounts.static.prom`) is not mounted. Mounts carry `nofail`, so a dead disk or a dropped share leaves a bare directory that every consumer keeps writing into on the root disk: the media share on hl01, a backup share.
`SystemdUnitFailed`	warning	`node_systemd_unit_state{state="failed"} == 1`	15 min	A failed unit sits between a dead host (`InstanceDown`) and service-specific rules; the fleet starts from zero failed units (openipmi cleared 2026-10-04).
`NodeClockSkewed`	warning	`abs(node_timex_offset_seconds) > 0.05`	15 min	A skewed clock breaks TLS validation and metric ordering; 50 ms is far above the NTP steady state (all hosts within 10 ms when introduced).
`EdacUncorrectableErrors`	critical	`increase(node_edac_uncorrectable_errors_total[1h]) > 0`	—	Uncorrectable memory errors mean a failing module and possible data corruption; only hosts exposing EDAC (pve1) produce the metric.

6.3 Temperature

Alert	Severity	Condition	Duration	Rationale
`HostHighCpuTemperature`	critical	`node_hwmon_temp_celsius > 85`	5 min	Generic across all scraped hosts, so new hosts are covered automatically; threshold from the pve1 incident.
`CoralTpuHighTemperature`	critical	`coral_pci_temperature_celsius > 90`	5 min	Coral TPU threshold identified during the August 2026 thermal incident.
`CoralTpuSensorFailure`	warning	`coral_pci_temperature_celsius < 0`	15 min	The exporter reports `-1` on failed sysfs reads, and a hung device reports implausible negative values.

6.4 Backups

Alert	Severity	Condition	Duration	Rationale
`ResticExporterDown`	warning	`up{job="restic"} == 0`	30 min	The backup rules below are expressions over the exporter's series: while the exporter is down, a stale or misplaced backup raises nothing else (raspberrypi2 backed up to its own root disk unnoticed, 2026-10-10).
`ResticMetricsMissing`	warning	`up{job="restic"} == 1 unless on (instance) restic_backup_timestamp`	30 min	The exporter answers scrapes but exposes no snapshot series: its collection failed and it retries only every 12 hours (raspberrypi2's exporter started before its repository existed, 2026-10-11), so the rules below cannot fire.
`ResticRepositoryChanged`	critical	`count by (instance, backup_job) (count_over_time(restic_repository_id[3d])) > 1`	—	The job wrote more than one repository identity (from `restic cat config`, exported by the job to a textfile) in three days: it was pointed at another repository or a new one was initialized under its path. The entrypoint refuses implicit initialization, so this is the observed check behind that guard.
`ResticBackupStale`	warning	`time() - restic_backup_timestamp > 2 * 86400`	—	No successful backup for two days: early signal before it becomes a real gap.
`ResticBackupMissing`	critical	`time() - restic_backup_timestamp > 4 * 86400`	—	No successful backup for four days: standing data-loss exposure.
`ResticRestoreDrillStale`	warning	`time() - restic_job_step_last_success_timestamp_seconds{step="restore-drill"} > 40 * 86400`	—	The monthly drill restores one bounded subtree of the latest snapshot into a scratch directory on the backup share with `--verify`, compares the file count with the snapshot listing, and removes it: it proves the restore path (container, password, share mount, space guard, file verification), while the nightly pack reads prove the data. 40 days covers one missed month.
`ResticRestoreDrillMissing`	warning	`restic_repository_id unless on (instance, backup_job) restic_job_step_last_success_timestamp_seconds{step="restore-drill"}`	1 h	A job that reports its repository but has never completed a drill: the correct state for a new job until its first drill, and the signal that a drill unit is missing.
`ResticCheckFailed`	critical	`restic_check_success == 0`	—	The exporter's own structural `restic check` (every 12 hours) failed: indices, trees, or snapshot references are damaged, and the repository must be repaired before any restore. It does not read the pack files; the job's nightly read cycle does.
`ResticReadDataCheckStale`	warning	`time() - restic_job_step_last_success_timestamp_seconds{step="check-read-data"} > 3 * 86400`	—	The backup job reads one group of pack files every night (`restic check --read-data-subset=n/t`, groups fixed by pack identifier, so each pack is read once per cycle of `t` nights, 7 by default per job); no completed read for three days means the job is failing or the read was disabled, and corrupt packs go unnoticed until a restore.

6.5 Blackbox Probes

Alert	Severity	Condition	Duration	Rationale
`BlackboxProbeFailed`	critical	`probe_success == 0` on ICMP and DNS probe jobs	10 min	A host does not answer pings, or a DNS record does not resolve as declared.
`BlackboxHttpProbeFailed`	warning	`probe_success == 0` on HTTP probe jobs	10 min	An HTTP endpoint stopped answering with the expected status.

Syncthing probe: the Syncthing check targets the instance's dedicated unauthenticated health endpoint (/rest/noauth/health on the nas-pve1 GUI port) over HTTPS, with certificate verification skipped in the http_2xx module because internal HTTPS endpoints serve self-signed certificates (Syncthing is currently the only HTTPS target; split the module into a verifying one when a certificate-validated target appears). Probing the GUI root was rejected because it requires authentication, and embedding the API key in the Blackbox exporter configuration was rejected because the health endpoint exists exactly for this purpose. Verified passing on both replicas (2026-10-05).

6.6 Frigate

These rules consume Frigate's native Prometheus metrics endpoint (/api/metrics), scraped by a dedicated frigate job whose target list is generated from the hosts that enable Frigate (the configure_frigate flag). The container-level Docker healthcheck and the HTTP probe of the Frigate UI stay green while the per-camera capture pipeline fails, so these rules watch the pipeline itself. The capture rate rule encodes the September 2026 incident, where a corrupted camera stream was ingested at roughly 20 times the configured detect rate and crashed the hardware decoder about 200 times per day for weeks without surfacing anywhere.

Alert	Severity	Condition	Duration	Rationale
`FrigateCameraStreamDown`	critical	`frigate_camera_fps == 0`	10 min	A camera produces no frames: the camera is offline or the capture process fails beyond the watchdog's recovery.
`FrigateCameraCaptureRateAbnormal`	warning	`frigate_skipped_fps > 10`	15 min	A healthy stream has no skipped frames; sustained skipping is the corrupted-stream signature.
`FrigateDetectorSlow`	warning	`frigate_detector_inference_speed_seconds > 0.1`	10 min	The Coral infers in well under 25 ms; sustained slowness means degradation or a CPU fallback.

6.7 Containers

This rule consumes the container_start_time_seconds metric that cadvisor already exports for every container on the Docker hosts. A container stuck in a restart loop keeps its Docker healthcheck irrelevant (it never lives long enough to report) while the workload is effectively down, as in the September 2026 incident where Jellyseerr crash looped for a month without surfacing anywhere.

Alert	Severity	Condition	Duration	Rationale
`ContainerRestartLooping`	warning	`changes(container_start_time_seconds{name!=""}[30m]) > 3`	—	More than 3 restarts in 30 minutes ignores deploys and upgrades but catches sustained crash loops.
`ContainerOomKilled`	warning	`increase(container_oom_events_total{name!=""}[30m]) > 0`	—	An OOM kill is silent when the container restarts cleanly; the memory limit or the workload needs attention.

6.8 Syncthing

These rules consume Syncthing's native metrics endpoint (/metrics on the GUI port), scraped by a dedicated syncthing job over HTTPS (self-signed GUI certificate, so certificate verification is skipped) authenticating with a vault-backed API key as a bearer token (vault_syncthing_prometheus_api_key). The endpoint refuses unauthenticated requests, and exposing it without authentication (Syncthing's metricsWithoutAuth) was rejected: the metrics carry device IDs and personal folder names as labels — private material the GUI port would otherwise serve LAN-wide. Targets are gated by the monitoring_scrape_syncthing host variable (nas-pve1; the retiring raspberrypi2 instance is deliberately not scraped). The identifying labels flow into the TSDB and the Telegram notifications, both private, but the committed rule expressions stay generic: the only identifier a rule names is the public nas-pve1 hostname, which excludes the instance's own device entry from the disconnection rule (a device always reports zero connections to itself). The device join matches on (device, instance) so the two Prometheus replicas' series pair correctly.

Alert	Severity	Condition	Duration	Rationale
`SyncthingFolderError`	critical	`syncthing_model_folder_state == 8`	30 min	State 8 is `FolderError` (verified in the v2.0.12 source): the folder stopped operating, so the backup transport is broken.
`SyncthingFolderOutOfSync`	warning	`sum by (folder, instance) (syncthing_model_folder_summary{scope="need"}) > 0`	24 h	Steady-state incremental syncs clear within hours; a day of unmet need means the sync is stuck (the initial seed took ~14 h).
`SyncthingDeviceDisconnected`	warning	`syncthing_connections_active == 0 and on (device, instance) syncthing_config_device_info{name!="nas-pve1",paused="false"}`	72 h	The offsite peer is intermittently online by design; three days unseen means backups stopped flowing. Paused devices stay quiet.

6.9 SMART Disk Health

These rules consume the smartmon textfile collector metrics that the node exporter already exposes on the hosts with local disks. The textfile staleness rule (§6.2) covers the collector itself; these rules cover the disks it reports on.

Alert	Severity	Condition	Duration	Rationale
`SmartDeviceUnhealthy`	critical	`smartmon_device_smart_healthy == 0`	—	The drive's own overall health verdict; by the time it flips, replacement is overdue.
`SmartSectorsDegrading`	warning	`delta(smartmon_current_pending_sector_raw_value[24h]) > 0 or delta(smartmon_reallocated_sector_ct_raw_value[24h]) > 0`	—	Growth in pending or reallocated sectors is the classic pre-failure signal; the delta form ignores historic nonzero counts.

6.10 UPS

These rules consume the Network UPS Tools exporter metrics (the ups scrape job). The NUT status is flag-encoded: one series per status flag, valued zero or one. Delivery of these alerts during a real outage depends on the notification path staying powered; brownouts and battery degradation are the primary cases.

Alert	Severity	Condition	Duration	Rationale
`UpsOnBattery`	warning	`network_ups_tools_ups_status{flag="OB"} == 1`	2 min	Mains power lost or browning out; 2 minutes ignores micro-cuts and self-tests.
`UpsLowBattery`	critical	`network_ups_tools_ups_status{flag="LB"} == 1`	—	The UPS itself signals imminent shutdown; no duration, every scrape counts.

6.11 NixOS Deployments

These rules consume the comin exporter metrics (the nix-comin job, every NixOS host). GitOps is the only deployment path for these hosts, so a failed fetch or activation is silent configuration drift until alerted.

Alert	Severity	Condition	Duration	Rationale
`CominDeploymentFailed`	warning	`comin_last_eval_failed == 1 or comin_last_build_failed == 1 or comin_last_deployment_failed == 1`	30 min	The host keeps running its previous generation; 30 minutes rides out transient build issues.
`CominFetchFailed`	warning	`comin_last_fetch_failed == 1`	2 h	The host stopped following the repository; 2 hours tolerates forge and network blips.

6.12 Monitoring Pipeline

The cross-scrape design (§7) catches dead pipeline components via InstanceDown; these rules catch the components that are up but silently broken. Every rule evaluates on both replicas against both instances' metrics, so a replica that cannot alert on itself is covered by the other.

Alert	Severity	Condition	Duration	Rationale
`AlertmanagerNotificationsFailing`	critical	`increase(alertmanager_notifications_failed_total[15m]) > 0`	—	The Telegram delivery leg failing is the one pipeline fault nothing else can report.
`PrometheusNotificationsErroring`	warning	`increase(prometheus_notifications_errors_total[15m]) > 0`	—	The Prometheus-to-Alertmanager leg; warning because the HA pair tolerates one failing replica.
`ConfigReloadFailed`	warning	`prometheus_config_last_reload_successful == 0 or alertmanager_config_last_reload_successful == 0`	10 min	A rejected configuration leaves the previous one running indefinitely while the files on disk lie.
`PrometheusRuleEvaluationFailures`	warning	`increase(prometheus_rule_evaluation_failures_total[15m]) > 0`	—	A broken rule silently stops the alerts it defines.
`AlertmanagerClusterDegraded`	warning	`alertmanager_cluster_members < 2`	15 min	Both Alertmanagers up but not gossiping breaks deduplication: duplicate or missing notifications.

7. Alerting Pipeline Health

The pipeline must not fail silently:

Every Prometheus replica scrapes every Prometheus and Alertmanager instance in the replica set (§3.3), so the death of any single backend component — including a whole replica host — raises InstanceDown from a surviving replica.
The monitoring pipeline rule group (§6.12) covers the components that are up but silently broken: failed notification delivery, rejected configuration reloads, failing rule evaluations, and a degraded Alertmanager cluster.
A full dead-man's-switch (an always-firing heartbeat alert delivered through an independent channel, catching the case where Prometheus or the whole host is down) is deliberately out of scope for this iteration and tracked in the specs index.

8. Restart Policy Migration

All services in the monitoring backend Compose stack (Prometheus, Grafana, the Blackbox exporter, and the new Alertmanager) use restart: unless-stopped instead of restart: always, so a service stopped deliberately (for example during a migration or an incident) stays stopped across daemon restarts. This completes the monitoring backend part of the repository-wide restart policy migration tracked in the specs index.

9. Verification

A deployment of this spec is verified with read-only checks:

The rendered rule file and Alertmanager configuration validate with the promtool and amtool checkers from the same pinned container images the stack runs.
The Alertmanager and Prometheus health endpoints report healthy on every replica, and the Prometheus rules API lists every group in the catalogue (§6) on every replica.
The Alertmanager cluster status reports both peers, and a silence created on one replica is visible on the other.
A synthetic alert posted through the Alertmanager API is delivered to the Telegram chat, proving the full routing and credential path — exactly once, also when both Prometheus replicas fire it, proving the deduplication.