Observability is not an afterthought in this homelab — it’s a first-class concern. Every workload is monitored, every log line is indexed, and every alert has a defined routing policy. The stack is designed to answer three questions: What is the system doing right now? What happened in the past? What should I be notified about?
The stack runs entirely in the monitoring namespace and is composed of well-established open-source projects, deployed via Helm.
Three layers, and the direction matters: collectors never talk to Grafana, and Grafana
never talks to a workload. Everything meets in Prometheus and Loki — which is also why
Argo Rollouts can gate a deployment on exactly the data an operator would look at.
Prometheus is the time-series metrics database at the heart of the stack. It scrapes metrics from all configured targets on a pull model and stores them with a 5-day retention window in a 10 GiB PVC backed by local-path storage.
Log aggregation is handled by the Loki + Alloy combination. Loki stores logs in a compact, indexed format queryable with LogQL. Alloy is the modern replacement for Promtail — a fully programmable pipeline agent.
Labels are the whole design: Loki indexes them and nothing else, so a query is cheap when
it filters on namespace, pod or unit and expensive
when it greps the message body. loki-canary continuously writes entries and
reads them back, which is the only way to tell an idle pipeline from a broken one.
Alloy runs as a DaemonSet ensuring it’s present on every node. The loki-canary DaemonSet is a companion service that continuously writes test log entries to Loki and verifies they can be read back — providing a live health check of the log ingestion pipeline.
A fully custom dashboard tracks the backup and storage posture of the cluster. It ingests metrics pushed to Pushgateway by all four backup layers — Velero + Kopia, restic PVC data, pg_dumpall, and host configuration — and surfaces backup status, duration trends, PostgreSQL dump size over time, and OCI Object Storage usage against the 20 GB free tier limit.
Critical alerts are sent immediately to both a Telegram bot (musashi_homelab_bot) and a Gmail address. The dual-channel approach ensures notifications are received even if one channel is unavailable.
The observability stack doesn’t just monitor — it actively gates deployments. During every canary rollout of Backstage, Argo Rollouts queries Prometheus for real-time application health metrics derived from Traefik’s ServiceMonitor:
The analysis runs while traffic is already split, so the queries measure the canary under
real load rather than a synthetic check. Every thirty seconds all three must hold; one
failure ends the rollout and returns traffic to the stable version without anyone being
asked.
This turns Prometheus from a passive observer into an active safety gate, automatically protecting production from bad deployments.