Drive Health Monitoring with SMART and Grafana Cloud

Hard drive prices surged 46–60% between late 2025 and early 2026. If you're running a homelab or self-hosted infrastructure with spinning rust — or SSDs accumulating write cycles — a failed drive you didn't see coming now costs you twice: the inflated replacement price, plus whatever recovery effort the data loss requires.

SMART data has been telling you about impending failures for decades. Most people only read it after something breaks.

This is the monitoring stack that reads it continuously.

What SMART Actually Tells You

SMART (Self-Monitoring, Analysis and Reporting Technology) is firmware built into virtually every modern drive — HDD and SSD. The drive maintains a table of internal counters and thresholds. Some are pass/fail. Most are raw numbers that mean nothing in isolation but are significant when they change.

Five attributes predict failure better than the rest. Backblaze, who runs hundreds of thousands of drives in production and publishes their failure data, identified these as the strongest predictors:

Any non-zero value for attributes 197 or 198 is a back-up-immediately signal. Non-zero attribute 5 or increasing attribute 188 are plan-a-replacement signals. The drive's own overall health status — the SMART pass/fail result — is the last line of defense, and by the time it fails, the situation is already serious.

The Stack

Drives (HDD/SSD) — SMART firmware
    └── smartctl_exporter  :9633  Prometheus metrics
          └── Grafana Alloy  remote_write → Grafana Cloud

smartctl_exporter is a Prometheus exporter from the prometheus-community. It calls smartctl on a configurable interval, reads every SMART attribute from every attached drive, and exposes them as labeled Prometheus metrics. One exporter per host. One scrape target per host. No per-drive configuration needed.

Grafana Alloy scrapes it and ships to Grafana Cloud. If you're already running Alloy for node metrics, adding drive health is one additional scrape target per host.

Deploying It

Everything is in github.com/colinedwardwood/smartmon-o11y — dashboards, alerts, and the Ansible role that deploys the exporter.

Step 1 — Install smartmontools and configure smartd

ansible-playbook ansible/smartmon.yml -K

This installs smartmontools and enables smartd with a DEVICESCAN config that runs short self-tests weekly and long self-tests on Saturdays, with temperature thresholds at 45°C (warn) and 55°C (critical).

Step 2 — Deploy smartctl_exporter

ansible-playbook ansible/smartmon-exporter.yml

This deploys smartctl_exporter v0.14.0 across all hosts (Raspberry Pis excluded — no SATA drives). Per host, the role:

• Creates a dedicated smartctl-exporter system user added to the disk group
• Downloads the correct architecture binary (amd64, arm64, or armv7) with checksum verification
• Installs a hardened systemd unit running as that user with CAP_SYS_RAWIO — raw disk access without running as root
• Starts the exporter on port 9633 with --smartctl.device-exclude=^/dev/(loop|dm|sr) to filter out loop devices, device mapper volumes, and optical drives

The systemd unit includes ProtectSystem=strict, NoNewPrivileges=true, PrivateTmp=true, and related hardening — the exporter gets exactly the capabilities it needs and nothing else.

Step 3 — Add the scrape target to Alloy

Add a prometheus.scrape block to your Alloy config pointing at localhost:9633. Include a relabeling rule to strip the port from the instance label so drive metrics join cleanly with your existing node metrics:

prometheus.scrape "smartctl_exporter" {
  targets  = [{"__address__" = "localhost:9633"}]
  job_name = "smartctl-exporter"

  relabeling {
    source_labels = ["__address__"]
    regex         = "([^:]+)(?::\d+)?"
    target_label  = "instance"
    replacement   = "${1}"
  }

  forward_to = [prometheus.remote_write.metrics_service.receiver]
}

Verify the exporter is running and exposing data:

curl http://<host>:9633/metrics | grep smartctl_device_info

A healthy response includes one smartctl_device_info line per drive with model, serial, capacity, and firmware version as labels.

Three Dashboards

Fleet Overview

The fleet dashboard answers operational questions across all drives on all hosts.

Top row: total drives monitored, SMART failures, drives with pending sectors, drives with reallocated sectors. These four stats should all read as expected at-a-glance — if "SMART Failures" or "With Pending Sectors" is non-zero, investigate immediately.

Below that: a drive health table with one row per drive — host, device name, model, health status, temperature, and power-on hours. Sorted by health status so unhealthy drives float to the top.

The bottom panel — Backblaze 5 Fleet Trend — is the most useful for long-term monitoring. It plots the five failure-predictive attributes as stacked time series across the whole fleet. A flat line is healthy. A rising line on any of the five means something is changing.

Drive Detail

The per-drive dashboard gives full visibility into one drive at a time. Select host and device from the dropdowns.

Key panels:
- SMART Health stat — pass/fail from smartctl_device_smart_status
- Temperature stat with color thresholds (green < 40°C, yellow < 50°C, red ≥ 55°C)
- Power-On Hours stat — total lifetime runtime
- Backblaze 5 — Failure Predictors — the five critical attributes as individual time series. A drive that has been at zero on all five for years and then shows any movement gets investigated.
- Temperature Trend — useful for identifying thermal patterns tied to workload or ambient temperature changes
- Disk I/O panels — IOPS, throughput, and I/O wait time from node exporter, correlated with drive health metrics

Security & Anomaly

The third dashboard covers failure modes that the other two don't catch.

Write Rate per Drive — Anomaly Monitor plots sustained write throughput per drive. In a homelab, a drive that's writing at 200+ MB/s sustained for ten minutes is unusual. The alert that fires on this pattern (configured at 200MB/s for 10 minutes) was specifically designed to catch one thing: ransomware encryption activity. When malware encrypts a filesystem, it reads every file and writes the encrypted version — sustained, sequential, high-throughput writes across multiple drives simultaneously. This panel makes that visible. If it fires during normal working hours with no backup job running, investigate immediately.

UDMA CRC Errors (SMART 199) — CRC errors on the SATA/SAS interface often indicate a bad cable or a failing controller, not the drive itself. Worth distinguishing from drive media errors.

Power Cycle Count — unusual patterns here (e.g. a drive cycling power repeatedly without intentional reboots) can indicate hardware instability.

The Alert Rules

Ten rules. All targeting noDataState: Alerting where a missing exporter is itself an emergency, and NoData where the absence of a metric just means the attribute isn't exposed by that drive model.

Three critical alerts that require immediate action:

SMART Health Check Failed — smartctl_device_smart_status < 1 for 5 minutes. The drive has crossed its own failure threshold. Back up everything and replace the drive. noDataState: Alerting so a crashed exporter doesn't silently hide a failing drive.

Current Pending Sectors — smartctl_device_attribute{attribute_name="Current_Pending_Sector"} > 0. Unstable sectors exist. Reads from those sectors may return corrupt data right now. This isn't "it might fail soon." It's "data may already be lost."

Uncorrectable Sectors — attribute 198 > 0. Media damage that couldn't be corrected. Data in those sectors is gone. Replace the drive.

Warning-level alerts worth having:

Reallocated Sectors — attribute 5 > 0. The drive has remapped bad sectors to spare. A handful in an older drive is normal; any new increase means the spare area is being consumed. Monitor for increases.

Drive Age Warning — power-on hours > 43,800 (5 years continuous). This is a planning alert sourced from Backblaze's published failure rate curves — consumer drives fail at significantly higher rates past this threshold. Not "replace now," but "source a replacement and schedule a swap."

Command Timeout Increasing — attribute 188 rising over 1 hour. Could be a failing controller, a bad cable, or a struggling drive. Check cables before assuming drive failure.

Temperature Critical / High — >55°C fires immediately; >45°C sustained for 30 minutes fires a warning. Both route to alert when the exporter goes silent, because a drive that's too hot and then disappears from metrics is worth knowing about.

Anomalous Write Rate — >200MB/s for 10 minutes. See the ransomware discussion above. Tune the threshold if you have high-throughput workloads — the alert annotation includes that note explicitly.

Exporter Stale — no data from a host in 15 minutes. The monitoring is blind. A crashed smartctl-exporter is a security and reliability gap.

Deploying Dashboards and Alerts

GRAFANA_URL=https://yourinstance.grafana.net \
GRAFANA_API_KEY=<token> \
bash grafana/deploy.sh

The deploy script creates a "Drive Health" folder, pushes all three dashboards with overwrite: true, and uses the provisioning API to push the alert rules as a rule group. Idempotent — safe to re-run after making changes.

Cost

Zero. Grafana Cloud's free tier metrics quota covers the scrape volume from a homelab fleet. The total metric count per drive is modest — a few dozen labeled attributes. A homelab with 20 drives across 10 hosts generates well under the free tier threshold.

Drive health monitoring is one of the highest signal-to-noise monitoring investments you can make in a homelab. The failure modes it catches — pending sectors, reallocated sectors, temperature excursions — are slow and predictable. They don't fail silently if you're watching. A drive that's about to take your NAS offline typically shows symptoms for days or weeks before the final failure. This stack surfaces those symptoms.

The price of a replacement drive is 46–60% higher than it was a year ago. The price of catching the failure early is the same as it's always been.

Dashboards, alert rules, and Ansible role are in colinedwardwood/smartmon-o11y.