Building reliable backup operations

Almost every team has backups. Far fewer can state, for a given machine, when it was last successfully backed up, how long a restore would take, and whether that restore has been tested. The gap between those two positions is operational, not technological — and it comes down to four practices.

1. Reconcile against what should have run

Most backup dashboards show the results of jobs that ran. That is a different question from whether every asset that should have been protected was protected. A job that never started produces no failure — it produces silence, and silence reads as green.

Reconciliation inverts the question. Start from the asset inventory, derive the set of assets that should have a successful copy within the window, then subtract the assets that actually do. Whatever remains is your real exposure.

This matters most at the boundaries: machines provisioned yesterday, machines renamed last week, machines moved between clusters. Those are exactly the assets that fall out of a job definition and never report a failure.

2. Give failures a follow-up, not a notification

A failed backup job that sends an email has not been handled. It has been announced. Handling means a defined policy: how many retries, with what backoff, under which conditions, and what happens when retries are exhausted.

Distinguish transient from structural failures. A target that was briefly unreachable deserves a retry. A repository that is out of space, a credential that has expired, or a source that no longer exists will fail identically on every retry, and retrying them just delays the notification.

• Transient — network timeout, target temporarily unavailable, lock contention. Retry with backoff.
• Structural — authentication failure, no space on target, source missing, permission denied. Escalate immediately; retries waste the window.
• Partial — job completed with warnings or skipped files. Treat as a failure for reconciliation purposes until a person decides otherwise.

An unresolved failure should stay open and visible until it succeeds or someone explicitly accepts it, with the acceptance recorded. Failures that disappear from view at the end of the day are how estates accumulate assets with no recent copy.

3. Treat retention as policy, not per-job configuration

Retention set per job drifts. Someone extends one job for a migration, someone shortens another to reclaim space, and two years later the retention actually in force bears little relation to the retention you believe you have.

Express retention as policy bound to an asset class or tenant, apply it to jobs from that policy, and audit the resulting state against the policy on a schedule. The audit is the part that is usually missing, and it answers two different questions: are we keeping what we are required to keep, and are we deleting what we are required to delete.

The second question is easy to forget and increasingly consequential. Data you were supposed to remove and did not is a liability, not an abundance of caution.

4. Test restores on a rotation

A backup is a hypothesis about a future restore. Testing converts it into a measurement. The objection to restore testing is always effort, which is exactly why it should be a scheduled workflow rather than a project.

A workable rotation restores a small number of assets per week to an isolated target, runs checks appropriate to the asset type, records how long it took, and tears the target down. Over a quarter, you cover a representative sample of every asset class.

Define the checks per asset class rather than generically:

• Filesystem — restore completes, expected paths present, checksums match for a sampled set.
• Relational database — restore completes, integrity check passes, a known query returns the expected row count.
• Application — restore completes, the service starts, a health endpoint responds, a read-only transaction succeeds.
• Object storage — object count and a sampled content comparison against the source manifest.

Record the duration every time. Restore duration is the number your recovery time objective actually depends on, and it is the number that quietly grows as data volumes grow. A measured trend gives you warning; an estimate does not.

Restores need tighter controls than backups

Running a backup is safe. Restoring over live data is one of the most destructive operations available in an operations toolkit, and the permissions should reflect that asymmetry.

• Separate the permission to run a backup from the permission to restore.
• Require named approval for any in-place restore, with the target and the chosen recovery point displayed at approval time.
• Default restores to a new location rather than over the original, and make overwriting an explicit choice.
• Record every restore — who requested it, who approved it, which recovery point, and what verification was performed afterwards.

What to report

A monthly backup report that is useful rather than decorative answers four questions: which assets had no successful copy in the period and why; which failures are still open; whether retention matches policy; and what restore tests were performed, with measured durations.

If the report is generated from the same run history your team works from, it costs nothing to produce and it is hard to argue with. If it is assembled by hand from several consoles, it will be produced late, infrequently, and optimistically.