m4nifest0-tech

Search the site

Storage and backup

3-2-1 backup strategy with Proxmox Backup Server and a NAS

How to apply the 3-2-1 rule with Proxmox Backup Server: datastore, retention, verification, encryption, off-site copy, test restores and a final checklist.
Published on Updated on 7 min readLevel: Intermediate

A backup that lives on the same system as the data, or that nobody has ever tried to restore, is just a hypothesis. Proxmox Backup Server (PBS) offers incremental, deduplicated and verifiable backups, but real protection depends on how you organize the copies and how often you prove they are usable. The 3-2-1 design described here combines PBS, a NAS and a remote site, for Proxmox VE 8 and 9.

The 3-2-1 rule

The rule calls for at least three copies of the data (the original and two backups), on two different types of media, with one copy off-site. Each constraint covers a different failure: more copies protect against an error on one of them, different media avoid a common defect (controller, firmware, administrative credentials), and the remote copy covers fire, theft and flooding. A common extension, 3-2-1-1-0, adds an offline or immutable copy against ransomware and zero errors after verification.

Architecture

The design goes beyond the minimum: original data, a local PBS datastore, a copy on the NAS and an off-site copy.

Data flow
Proxmox VE nodes (VMs and containers)              copy 1, original data
        |  backup job, client-side encryption
        v
Local PBS, datastore on the server's disks         copy 2, media A
        |  sync job
        +-----> NAS (second PBS)                   copy 3, media B
        |  sync job over VPN
        +-----> Remote PBS at another site         copy 4, off-site

PBS should be installed on hardware separate from the nodes it protects: the documentation advises against installing it on the hypervisor, and a VM in the same cluster would go down along with the data. The datastore uses a local file system (ext4, XFS or ZFS) on fast disks; a network share is not among the supported configurations.

The NAS, which is the second medium, can be integrated in two ways:

  • A second PBS on the NAS, if it can host a VM or a container: it receives the data with a sync job and keeps deduplication, verification and encryption.
  • An NFS or SMB share, if the NAS only offers files: it is added in Proxmox VE as storage for backup files and the nodes write to it with a second vzdump job. The archives are full, without deduplication, and take up more space, but they are independent of PBS.

Datastore and backup jobs

The datastore is created from the web interface (Administration, Storage / Disks) or from the command line, on a path that is already mounted.

On the PBS
proxmox-backup-manager datastore create datastore-principale /mnt/datastore/principale

In Proxmox VE the PBS storage is added from Datacenter, Storage, Add, Proxmox Backup Server, giving the address, a dedicated user, the datastore and the certificate fingerprint. The job is created from Datacenter, Backup: snapshot mode, daily schedule, notification on error.

Retention, garbage collection and verification

These are three distinct operations, to be scheduled together:

  • Prune applies the retention and removes the metadata of excess snapshots, without freeing space.
  • Garbage collection deletes blocks that are no longer referenced. As a safety measure it treats as removable only blocks unused for more than about 24 hours, so space is freed with a delay.
  • Verify rereads the blocks and checks their integrity. The documentation recommends reverifying all backups at least once a month; the job is created in the datastore's Verify Jobs tab.
On the PBS
proxmox-backup-manager prune-job create prune-giornaliero --store datastore-principale --schedule daily --keep-daily 7 --keep-weekly 4 --keep-monthly 6
proxmox-backup-manager datastore update datastore-principale --gc-schedule 'sat 03:00'

The keep-* rules apply in order and each one considers only backups older than those already covered by the previous ones: keep-daily 7 keeps the latest backup of each of the last seven days on which one exists. The values are a starting point to adapt. It is best to define the retention in a single place, preferably in the PBS prune job: different rules on Proxmox VE and on PBS make it unclear what will be deleted.

Client-side encryption

PBS supports client-side AES-256-GCM encryption: data leaves the node already encrypted and the server never knows the key. It is enabled in the PBS storage in Proxmox VE (Encryption tab), generating an automatic key; Proxmox VE keeps it in /etc/pve/priv/storage/<ID-STORAGE>.enc, readable only by root.

Caution

Without the key, encrypted backups are unrecoverable. A key that exists only on the cluster is lost along with the cluster, which is exactly when you need it. Right after creating it, copy it off the nodes: password manager, USB stick in a safe place and a paper copy generated with proxmox-backup-client key paperkey. An optional RSA master key adds an encrypted copy of the key to every backup; keep the private part offline.

Encrypted backups stay encrypted in sync jobs too: the remote PBS holds opaque data, so a stolen disk or a compromised off-site server does not expose the contents.

Sync to a remote PBS

The sync job copies backups from one datastore to another. The remote PBS is registered as a Remote (Configuration, Remotes) with address, user and certificate fingerprint. Since version 3.3 the job can run in push mode: the source PBS connects to the remote one, so the main site does not have to accept inbound connections. In pull mode the destination connects to the source.

On the local PBS (push mode, Remote already registered)
proxmox-backup-manager sync-job create sync-fuori-sede --store datastore-principale --remote pbs-remoto --remote-store datastore-remoto --sync-direction push --schedule daily

Important

The PBS API (port 8007) must not be exposed to the Internet: the sync goes through the VPN tunnel between the sites, as described in WireGuard: remote access and site-to-site VPN. The Remote's user should have only the privileges needed for the sync: a dedicated user, not the administrator.

Warning

The --remove-vanished option propagates deletions: if it is on, a backup deleted on the source, by mistake or by an attacker, disappears from the destination as well. For the off-site copy it is better to leave it off and give the remote retention its own prune job.

The test restore

An untested backup is not a backup. The test checks not only the integrity of the data but also keys, credentials, documentation and timing. Three levels, with indicative schedules:

  • Single file (monthly). In the Backups tab of the PBS storage in Proxmox VE, the File Restore button lets you browse a backup and download a file.
  • Full VM (quarterly). Restore a backup with a new ID and unique network addresses, then check boot and application consistency.
  • Disaster scenario (yearly). Starting only from the off-site copy, the key and the documentation, rebuild an empty node and restore the critical services, noting the times.
On a test node
pvesm list <STORAGE_PBS> --content backup
qmrestore <STORAGE_PBS>:backup/vm/<VMID_ORIGINE>/<TIMESTAMP> <NUOVO_VMID> --storage <STORAGE_DESTINAZIONE> --unique 1

The first command lists the snapshots, the second restores one. Before the first boot, disconnect the copy's network card (the Disconnect option in the Hardware tab) to avoid address conflicts with the original VM. The outcome of every test should be recorded: date, backup used, time taken, problems encountered.

Final checklist

  • There are at least three copies of the data, on two different media, one of them off-site.
  • PBS runs on hardware separate from the protected nodes.
  • Backup jobs report errors; retention, garbage collection and verify jobs are scheduled, with the retention defined in a single place.
  • The encryption key is copied outside the cluster and its location is documented.
  • The off-site sync goes through a VPN, with a dedicated user and remove-vanished turned off.
  • Free space on the datastores is monitored (see Prometheus and Grafana in an LXC container).
  • Container bind mounts, which vzdump backups exclude, have separate protection (see Unprivileged LXC containers and bind mounts).
  • Test restores are on the calendar and their outcome is recorded.

Related articles