Longhorn Storage Observability for a k3s Homelab

A replica on the MacBook Pro had been in an error state for weeks. The volume wasn't degraded — two of three replicas were healthy, and Longhorn considers that acceptable. The UI showed nothing worth investigating. The replica compliance panel showed it immediately: one volume with a running count of 2, highlighted red.

That's what running Longhorn for four months without dashboards costs you. Not data loss, but a quiet erosion of the redundancy you thought you had.

The Longhorn UI is adequate for confirming that volumes exist. It tells you almost nothing about trends — whether storage is filling, whether replica distribution matches what you configured, whether any volume is doing something unusual with I/O. This is the observability stack I built on top of it.

The Cluster

Six nodes, k3s v1.33, all Ubuntu 24.04:

• k3s-control-01 — dedicated control plane, small box, no storage
• k3s-worker-01 / k3s-worker-02 — HP EliteBook 840 G3 laptops (acquired for free)
• k3s-worker-03 — a 2016 MacBook Pro, running headless
• k3s-worker-04 — a Lenovo T400 that is older than some of my houseplants
• k3s-worker-05 — an Asus tower, the only node with real storage

Longhorn v1.10.1 handles persistent storage across all of them. Every volume gets three replicas spread across nodes. In practice the replicas land mostly on the EliteBooks and the Asus tower — the nodes with the most disk. The T400 participates in scheduling but has the smallest disk in the cluster, which turned out to be worth knowing.

One Block to Start Scraping

Longhorn exposes Prometheus metrics at port 9500 on the longhorn-backend service. I was already running the grafana/k8s-monitoring Helm chart — it manages several Alloy instances in an alloy namespace for cluster metrics, logs, and traces. Adding Longhorn was one block in alloy-metrics.extraConfig:

prometheus.scrape "longhorn" {
  targets = [{
    __address__ = "longhorn-backend.longhorn-system.svc.cluster.local:9500",
  }]
  job_name        = "integrations/longhorn"
  scrape_interval = "30s"
  forward_to      = [prometheus.remote_write.grafana_cloud_metrics.receiver]
}

Helm upgrade, wait a minute, metrics start appearing in Grafana Cloud.

One thing to catch before building dashboards: there is a duplicate scraping situation that is easy to miss. The alloy-metrics pod scrapes the longhorn-backend ClusterIP (one target, correct), but the chart was also pulling in six additional series from individual pod IPs under a different job label. Those pod-level series had no cluster label and no pvc_namespace enrichment, making them less useful and inconsistent with everything else. Adding job="integrations/longhorn" to every dashboard query eliminates the ambiguity and keeps everything sourced from the correct scrape target.

What Longhorn Actually Exposes

Longhorn emits 54 metrics. Before building anything, it is worth understanding how they are structured — several are not shaped the way you would expect, and community dashboards found online are often broken because of this.

Label names are non-obvious. Volume names are in a label called volume, not name. PVC namespace is pvc_namespace, not pvcNamespace. Any dashboard query using {name=~"..."} will return nothing.

Replica health is a state matrix, not a count. There is no longhorn_replica_count_total gauge. Instead, longhorn_replica_state carries a state label (running, stopped, error, starting, stopping, unknown) with a value of 1 if the replica is in that state. Counting running replicas per volume:

count by (volume) (longhorn_replica_state{state="running"} == 1)

Volume state is numeric. 1=creating, 2=attached, 3=detaching, 4=attaching, 5=detached. Both volumes in my cluster showed state=2 while their pods were running correctly — worth verifying against live data rather than guessing.

Node status conditions are lowercase strings. condition="schedulable" and condition="ready", not Schedulable and Ready.

Backup metrics may not exist. I had written panels expecting longhorn_backup_state. They showed no data in Longhorn v1.10.1. Replaced them with replica error tracking, which is more operationally useful anyway.

The full metrics reference is in the repo README.

Four Dashboards

Four dashboards, each scoped to a different set of questions.

SRE Overview

The day-to-day operational view. Top row is a health snapshot: total volumes, availability percentage (robustness=healthy count divided by total), degraded count, faulted count, attached count, and total provisioned capacity. Below that, robustness and state trends over time.

The second half is I/O: aggregate throughput and IOPS across all volumes, then per-volume breakdowns for the top 10 by read and write throughput. Latency panels with threshold lines at 5ms and 10ms — Longhorn reports latency in nanoseconds, so the panels stay in nanoseconds and the threshold lines do the translation. Those thresholds are probably too strict for the EliteBooks running spinning disks; the intent is to start conservative and relax based on real data rather than the reverse.

Sysadmin

Capacity planning. Cluster-wide storage totals at the top, then per-node and per-disk breakdowns as bar gauges. The nodes have uneven storage — the Asus tower has significantly more than the EliteBooks — so the per-node view makes it obvious which nodes are filling up faster. The Lenovo T400 (k3s-worker-04) consistently shows the highest utilization of any node. It has the smallest disk in the cluster. The bar gauge makes that visible every time the dashboard loads; before it, the T400's situation was a vague known unknown.

Below the capacity panels: a full volume inventory table showing robustness, state, provisioned size, and actual usage side by side, and a replica count table for spotting any volume running with fewer replicas than configured.

Kubernetes Admin

PVC-centric. The main panel maps every volume to its PVC name and namespace with robustness color-coded. Below that, storage consumption broken down by namespace as donut charts — useful for knowing which workloads are consuming the most storage without digging through kubectl.

Also includes node status conditions (schedulable, ready, allowScheduling) and running replica count per volume as a bar gauge, where anything below 3 turns red.

Security and Compliance

The pessimist's dashboard. The premise is that something is probably wrong; the job is to surface it.

Risk summary at the top: faulted volumes, degraded volumes, volumes with fewer than three running replicas, replicas in error state, nodes not schedulable, worst-case node storage utilization. Everything the other dashboards might not put front-and-center.

The replica compliance table is the main panel. Every volume, running replica count, sorted ascending. Anything below 3 is immediately visible. This is the panel that caught the MacBook Pro replica — the volume appeared healthy in every other view because the other two replicas were covering for it. The compliance panel sorts by replica count, puts the worst case at the top, and makes it impossible to overlook.

Below that, I/O anomaly detection: top-5 write and read throughput over time, and a table of volumes with write throughput more than three standard deviations above the cluster average. The statistical threshold is tunable; the intent is to surface runaway processes, not alert on expected backup or migration traffic.

Alerts

Thirteen rules across three tiers.

Critical — requires immediate attention:

Warning — needs attention soon:

Compliance — weekly review, not urgent:

The latency thresholds will probably need relaxing once there is baseline data from the EliteBook spinning disks. They are set where they are deliberately — a strict threshold with known false positives is more useful during the first few weeks than a permissive one that never fires.

Running It

Everything is in github.com/colinedwardwood/longhorn-o11y.

Add the Alloy snippet to your alloy-metrics.extraConfig Helm values:

# Copy the snippet
cat alloy/extraConfig-snippet.alloy

# Paste into your k8s-monitoring values, then upgrade
helm upgrade k8s-monitoring grafana/k8s-monitoring \
  -n alloy -f your-values.yaml

Import the dashboards via Grafana UI (Dashboards → Import) or the provisioning API. All panels use a ${datasource} template variable — no hardcoded UIDs, no find-and-replace for datasource names. The only thing to update if you use a different Alloy job label: find-and-replace integrations/longhorn in the JSON before importing.

# Verify metrics are flowing
kubectl exec -n alloy deploy/alloy-metrics -- \
  wget -qO- http://longhorn-backend.longhorn-system.svc.cluster.local:9500/metrics \
  | grep longhorn_volume_robustness

A healthy response returns one series per volume with a value of 1 (healthy), 2 (degraded), or 3 (faulted).

The replica that had been sitting in error state for weeks on the MacBook Pro was not causing a visible problem. The volume was covered. But covered is not the same as resilient, and there is no way to know the difference without tracking it.

Grafana Cloud free tier handles the metric volume from a six-node homelab cluster without issue.

Dashboards and the Alloy scrape config are in colinedwardwood/longhorn-o11y.