Backup Monitoring and Alerting

Every backup job reports its status to a central monitoring system — Prometheus with custom exporters, Datadog, or CloudWatch. We track: job start time, duration, data size, success/failure, and error messages. Failed jobs trigger immediate alerts via PagerDuty with the error context needed to diagnose the issue. A backup job that silently stops running is detected within one missed schedule window.

We monitor backup age — the time since the last successful backup — for every protected system. A PostgreSQL database with 1-hour RPO fires an alert if the latest backup is older than 90 minutes. S3 bucket policies or Lambda functions check object timestamps against expected schedules. Freshness monitoring catches the failure modes that job monitoring misses: the job ran but produced a zero-byte file, or WAL archiving stopped silently.

Dashboards show backup storage consumption by system, retention tier, and provider. Growth trends identify systems with increasing backup sizes before they blow storage budgets. Cost metrics break down spending by backup type and retention tier. Anomaly detection flags unexpected changes: a database backup that doubled in size, a system that stopped producing backups, or a lifecycle rule that failed to transition objects to cold storage.

A single dashboard shows backup compliance status across all systems: last successful backup, last verified restore, backup encryption status, offsite copy status, and retention policy compliance. Red/yellow/green indicators make it easy to spot issues. The dashboard generates a weekly report for stakeholders and an on-demand compliance report for auditors. Every protected system must be green — there is no acceptable level of backup non-compliance.