Problems faced
- A successful backup does not by itself prove that a service can be restored.
- Without a shared recovery sequence, an incident can lead to slow or inconsistent decisions.
- Occasional tests do not provide enough evidence to show that the procedure is still usable.
- Undefined RTO and RPO values make it impossible to set priorities and success criteria.
Implemented solution
A runbook documents incident activation, containment, recovery, and closure. Each service has a priority, an RTO, and an RPO; test restores run in an isolated environment on a defined schedule, with outcomes, timings, and anomalies recorded.
Results
- The procedure can be followed under pressure and says who decides, what comes back first, and when recovery is complete.
- Backups are checked both for integrity and by booting and testing real samples.
- Every test leaves comparable evidence: date, source, procedure version, duration, outcome, and corrective actions.
- Infrastructure changes trigger a runbook review and a targeted test.
Technical description
A disaster recovery (DR) plan is more than a list of backups. It is an operating procedure for returning services to a working state after the loss of a node, storage, configuration, or the entire site. This document defines the recovery flow and the test programme that proves it works.
Objectives and criteria
Each service gets two objectives:
- RPO (Recovery Point Objective): the maximum amount of data that may be lost, expressed as a time interval.
- RTO (Recovery Time Objective): the maximum time before the service must be available again.
The recovery priority must be explicit. In general, network, DNS, and authentication come first, followed by storage and the virtualization platform, then application services, and finally non-essential services. The actual sequence must follow the documented dependencies in the inventory.
Warning
RPO and RTO values are objectives to validate through testing, not promises inferred from the theoretical speed of a backup.
When to activate the procedure
The runbook is activated when an event affects service availability or integrity: hardware failure, storage corruption, human error, a failed update, node loss, or suspected ransomware. The operator immediately records:
- detection time and affected services;
- observed symptoms and the latest changes;
- available backups and their latest verified outcome;
- people involved and the coordination channel.
Before restoring, preserve evidence and contain the failure: isolate suspected systems, stop jobs that could overwrite data, and do not connect a restored machine to the network until it has been checked.
Recovery procedure
1. Assess and choose the recovery point
Identify the likely cause, confirm the scope of the incident, and choose the latest recovery point before the event. If the latest copy is questionable, use an earlier copy and record why. Do not modify the original backups during analysis.
2. Prepare the environment
Check the destination platform, storage, and isolated network. Verify free space, emergency credentials, repository access, and service dependencies. Restore with a temporary identifier and without conflicting addresses or DNS records.
3. Perform the technical restore
For a virtual machine, the minimum sequence looks like this. The values are examples: replace them with the inventory and repository used in the incident.
qmrestore local-pbs:backup/vm/105/<snapshot> 9100 --storage local-zfs --unique 1
qm set 9100 --delete net0
qm start 9100
qm agent 9100 pingRestore configuration, data, and required secrets in the dependency order. Only after the isolated checks pass should the required connectivity be enabled and the production service updated.
4. Validate and return to service
Recovery does not end when the VM boots. At minimum, verify that the operating system responds, the service listens on the expected port, sample data is readable, and monitoring and logs show no unexpected errors. For a critical service, also run a non-destructive transaction or test query.
Once stable, record the return time, compare RTO and RPO with their objectives, re-enable backups, and confirm that the new state is protected again. Remove the temporary machine and test artefacts only after preserving the required logs.
Periodic test plan
| Frequency | Test | Success criterion | Evidence |
|---|---|---|---|
| Daily | Job results, capacity, errors, and alerts | No critical job failed without an owner | Report or closed alert |
| Weekly | Backup verification and replication check | Integrity confirmed and copy up to date | Verify and replication logs |
| Monthly | Restore a VM or container on an isolated network | Boot, guest agent, and application check pass | Record with timing and outcome |
| Quarterly | Restore a critical service end to end | RTO/RPO met and dependencies work | Signed report and corrective actions |
| After every major change | Targeted test of the changed part | No regression in the recovery chain | Change record linked to test |
Monthly samples rotate across service classes so the same machine is not always tested. At least one quarterly exercise uses the off-site copy, or an equivalent path for a loss-of-primary-site scenario.
Recording the outcome
Each test produces a record containing at least: date and operator, simulated scenario, backup source and timestamp, runbook version, destination environment, start and end time, checks performed, result, anomalies, and corrective-action owner. A failed test stays open until a successful rerun; it is not marked “pass” merely because the next backup succeeded.
Review the runbook after every incident, failed test, or change to storage, networking, authentication, orchestration, or critical services. Keep the version used for each exercise with its result so the exact procedure tested can be reconstructed.
Link to the backup strategy
This procedure complements the 3-2-1 backup strategy with automated restore verification, but does not replace it: redundant copies reduce the risk of data loss, while tests prove that those copies can become a working service.