If your production server can write to the backup NAS, an attacker who owns that server owns your backups too. That is the gap the classic 3-2-1 backup rule never closed. It counts copies and places them on different media and sites, but it says nothing about who can delete them, and ransomware crews go after the backup repository before they encrypt anything.
Here we compare the three strategies you will meet in vendor docs and audits: plain 3-2-1, the 3-2-1-1-0 extension, and immutable copies (S3 Object Lock, append-only servers, offline media). We cover what each layer protects against, what the latest ransomware surveys say about recovery from backups, and how to pick a scheme for your own estate. Everything in the lab section was built in October 2026 on two Ubuntu 26.04 VMs with restic 0.18.1 and an append-only rest-server, so the error messages and timings are real.
The 3-2-1 backup rule and what it protects
The 3-2-1 backup rule says you keep three copies of any important data (the primary plus two backups), on two different media types, with one copy off site. The US-CERT paper Data Backup Options (2012) states it in exactly those terms and credits it to photographer Peter Krogh’s The DAM Book (2009). It was written for hardware failure, fire and theft, which is why every digit is about placement and none is about access.
In practice, each digit maps to a failure the rule was designed around:
- Three copies: one failed disk or one corrupted backup job never leaves you with zero good copies.
- Two media types: a firmware bug or a bad batch of drives does not hit both copies at once. A ZFS mirror is one media type, however many disks it holds, as our ZFS RAID levels guide explains.
- One off site: a flood, fire or stolen rack still leaves a copy somewhere else.
None of those three answers the first thing an incident responder asks, which is whether the intruder could delete it. A cloud bucket that the backup server writes with a key that can also delete objects is off site and on different media, so it satisfies 3-2-1, but a stolen key empties it in one API call. That is the single weakness everything else on this page is built to fix.
Ransomware recovery numbers from backups
Backups decide whether a ransomware victim pays, and the surveys show that the copies often are not there when needed. Sophos’ 2026 State of Ransomware report found that 66% of organisations whose data was encrypted restored it from backups, up from 54% in the 2025 edition, which was the lowest rate in six years. Ransom payment fell to 48% over the same period, and the median payment dropped from $1M to $769K.
Look at the trend rather than any single year. Backup recovery peaked at 73% in the 2022 report and then slid for three editions. Over the same stretch, the share that paid and got data back rose to 56% in the 2024 report. In the 2025 report, 38% of the organisations that paid more than the attacker first demanded gave the same reason: “Our backups failed or were malfunctioning.” The backups existed on paper, and the restore did not work when it counted.
Veeam’s survey data points the same way. Its 2024 Ransomware Trends report found that attacks compromised 41% of data on average and that only 57% of that data came back. Of the surveyed organisations, 81% paid, and one in three of those still could not recover. Veeam’s 2026 Data Trust and Resilience report adds a confidence gap, with 90% of organisations saying they could recover but only 28% of those whose operations or data were hit by ransomware getting all affected data back. CISA’s #StopRansomware Guide gives the reason in one line: “many ransomware variants attempt to find and subsequently delete or encrypt accessible backups.”
The 3-2-1-1-0 rule: one immutable copy, zero errors
The 3-2-1-1-0 rule keeps everything in 3-2-1 and adds two conditions: at least one copy that is immutable, air-gapped or offline, and zero errors when you verify a restore. Veeam describes it as its own extension of the rule (“At Veeam, we extended the rule to 3-2-1-1-0”). The digits read in order:
- 3 copies of the data, production included.
- 2 media types, so one hardware fault cannot take both.
- 1 copy off site for site loss.
- 1 copy immutable or offline, which production credentials cannot change or delete.
- 0 errors after an automated or scheduled restore test.
Digit four answers the “who can delete it” question, and digit five answers the Sophos finding above, because a backup that has never been restored is an assumption rather than a copy. You should treat the zero as a schedule, not a checkbox. A monthly restore of a sample into a scratch VM, with hashes compared against the source, catches the silent failures (expired credentials, a changed path, an excluded mount) long before an incident does.
Immutable, append-only and offline copies
“Immutable” covers several mechanisms with very different trust models, and the difference decides who can still destroy the copy. For each one, you need to know which account can still delete data before the retention date, and what an attacker would have to compromise to use it.
| Mechanism | Who can delete before expiry | What it costs you |
|---|---|---|
| S3 Object Lock, compliance mode (AWS, Backblaze B2, Wasabi) | Nobody, including the AWS root user; on AWS only closing the account removes it | You pay for every locked byte until the date passes; a wrong retention period can’t be shortened |
| S3 Object Lock, governance mode | Users granted the bypass permission | Protects against stolen backup keys, not against a stolen admin account |
| Append-only server (restic rest-server, borg serve) | The backup server’s own admin, never the client | Pruning must run from a separate trusted host |
| Azure Blob immutable storage | Nobody while a locked time-based policy applies, and a legal hold lasts until it is cleared | Same storage-cost trade-off as compliance mode |
| Offline tape or rotated disk | Whoever holds the media physically | Manual handling, slowest restores |
Compliance mode is the strongest lock most teams can buy. It also sets a trap. CISA calls it out directly: “Use immutable storage with caution as it does not meet compliance criteria for certain regulations and misconfiguration can impose significant cost.” A 3,000-day retention set by mistake on Backblaze B2, which allows 1 to 3,000 days, is a bill you cannot cancel. Set the retention period a little longer than the time it takes you to notice an intrusion, test it on a scratch bucket first, and put a quota on the bucket.
Append-only servers are cheaper and fit self-hosted stacks, but the guarantee covers only the client. BorgBackup’s docs say it plainly: in append-only mode borg delete and borg prune still run, they just can’t free space, and the server admin can roll the repository back through its transaction log. That rollback has to happen first, because the deletions a compromised client logged take effect the next time the admin runs a write operation such as borg compact without append-only mode. Proxmox Backup Server’s “protected” snapshot flag is another tool in this group. It stops pruning and manual removal of that snapshot, but it is a flag on the snapshot, not a storage-level lock, so pair PBS with an offline tape or an Object Lock bucket if you need the fourth digit. Our Proxmox Backup Server box build covers the datastore side.
A lab where the immutable copy survives a wipe
To show the difference between a mutable copy and an append-only one, we built two VMs. The app host, app-eu-west-1a, held 324 MB of sample data (9,051 files) and backed it up with restic to two repositories: a local one at /backup/local-repo and a remote one on backup-vault-1b, which ran rest-server v0.14.0 in append-only mode. The attacker gets root on the app host, the repository password file and both repository URLs, which is what real ransomware finds when it reads a backup script.
All of the append-only behaviour lives in the rest-server unit on the vault. Before it starts, you need the rest-server binary from its GitHub release in /usr/local/bin, a restic system user owning /srv/restic and a bcrypt password file at /srv/restic/.htpasswd, which we created with htpasswd -B for a user named app01 to match the repository path. rest-server turns authentication on by default and reads that file. Open the unit file:
sudo vim /etc/systemd/system/rest-server.serviceWe ran this unit as written. --private-repos limits each htpasswd user to the repository path matching its name, and the hardening lines keep the service writing only under /srv/restic. The lab served plain HTTP on a private network; outside a lab, add --tls with --tls-cert and --tls-key so the basic-auth password never crosses the wire in clear text:
[Unit]
Description=restic rest-server (append-only)
After=network-online.target
Wants=network-online.target
[Service]
User=restic
Group=restic
ExecStart=/usr/local/bin/rest-server --path /srv/restic --listen :8000 --append-only --private-repos
Restart=on-failure
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/srv/restic
PrivateTmp=true
[Install]
WantedBy=multi-user.targetBoth repositories took the same initial backup in under nine seconds (8.9 s local, 6.8 s remote, 284 MiB stored after compression). A second run that picked up one changed file and one new 5 MiB file took 1.5 s and 2.6 s. Each repository then held two snapshots. On the app host, the attacker encrypted the database files with openssl, deleted everything else, and went after the backups. The capture below is an excerpt; in the root shell, LOCAL and REMOTE hold the two repository locations and RESTIC_PASSWORD_FILE points at the stolen password:
One rm -rf and the local repository is gone, which is the fate of every backup that lives where production credentials can reach it. The remote forget --prune gets 403 Forbidden on its snapshot DELETE and carries on to plan and repack. It then dies with exit code 1 when its index DELETE is refused too, and both snapshots are still listed afterwards. Raw HTTP did no better: a curl -X DELETE on a snapshot or a data pack returned 403, a POST overwriting an existing snapshot object returned 403, and a DELETE on the repository root returned 405 Method Not Allowed.
restic 0.18 forget exits 0 after “403 Forbidden”
One result will bite anyone who scripts retention. With restic 0.18.1, the Ubuntu 26.04 package, a plain restic forget <id> against the append-only repository printed the 403 and “unable to remove”, yet returned exit code 0, while forget --prune returned 1. The restic 0.19.0 changelog fixes this: forget now returns exit code 3 when it fails to remove snapshots. This single-snapshot run shows it:
restic -r $REMOTE forget b69a9d65; echo exit=$?You get the error on screen, but the shell sees success:
[0:00] 0.00% 0 / 1 files deleted
Remove(<snapshot/b69a9d650b>) failed: unexpected HTTP response (403): 403 Forbidden
unable to remove snapshot/b69a9d650b9a10fb7c2bc85753e08a3d7f6cb33d689809f77fc4e15858236b60 from the repository
exit=0On 0.18.x and older, a cron job that checks only the exit code would report a successful retention run that deleted nothing, so the repository keeps growing until the disk fills. If you stay on that version, grep the output for “unable to remove”. If you upgrade, treat exit 3 as a failure.
Either way, retention belongs on a trusted host. On the vault itself, a forget --keep-last 1 --prune --dry-run against the local path /srv/restic/app01 worked and planned to remove one snapshot, but that count-based policy was only a smoke test. restic’s own docs warn that an attacker who can add garbage snapshots to an append-only repository can push the real ones out of a --keep-last count. Your real policy there should use --keep-within. Run it as the restic service user, so any pack and index files a real prune writes stay owned by the account rest-server runs as. That user needs its own copy of the password file and a cache directory outside /srv/restic, which is rest-server’s data path. We created both on the vault, feeding the repository password in on standard input:
sudo install -d -o restic -g restic -m 700 /var/cache/restic-maint
sudo install -o restic -g restic -m 600 /dev/stdin /etc/restic-maint.passWith those in place, the retention dry run works as the service user:
sudo -u restic RESTIC_PASSWORD_FILE=/etc/restic-maint.pass restic -r /srv/restic/app01 --cache-dir /var/cache/restic-maint forget --keep-within 30d --prune --dry-runBoth lab snapshots are minutes old, so the time-based policy keeps them and plans nothing to delete:
Applying Policy: keep all snapshots within 30d of the newest
keep 2 snapshots:
ID Time Host Tags Reasons Paths Size
------------------------------------------------------------------------------------------------
0e1d9c6c 2026-10-31 03:09:34 app-eu-west-1a within 30d /srv/appdata 297.795 MiB
b69a9d65 2026-10-31 03:09:44 app-eu-west-1a within 30d /srv/appdata 302.795 MiB
------------------------------------------------------------------------------------------------
2 snapshotsNote that restic measures --keep-within from the newest snapshot’s timestamp, as its output says, so check the snapshot list for odd timestamps before any real prune. Append-only also does not stop writes: the failed prune still repacked one pack and wrote a new index file before its DELETE was refused, which left duplicate index entries behind.
The restore and the zero-error check
Recovery came from the vault copy. We restored the latest snapshot into a scratch path and compared every file against a SHA-256 manifest taken before the attack:
restic -r $REMOTE restore latest --target /srv/restore
cd /srv/restore/srv/appdata && sha256sum -c --quiet /root/appdata.sha256; echo mismatches=$?All 9,052 files came back in 4.5 seconds of wall time, and the manifest check found no mismatches:
restoring snapshot b69a9d65 of [/srv/appdata] at 2026-10-31 03:09:44.912865153 +0000 UTC by root@app-eu-west-1a to /srv/restore
Summary: Restored 12665 files/dirs (302.795 MiB) in 0:03
mismatches=0The restore above is the “0” in 3-2-1-1-0. You prove the other half with a repository check that reads every pack back, not only the index. restic check --read-data took 2.8 seconds here, and on a multi-terabyte repository --read-data-subset spreads the reading across scheduled runs. Ours first listed 21 “pack contained in several indexes” warnings left by the attacker’s failed prune, then finished clean:
Duplicate packs are non-critical, you can run `restic repair index' to correct this.
check all packs
check snapshots, trees and blobs
[0:00] 100.00% 2 / 2 snapshots
read all data
[0:01] 100.00% 22 / 22 packs
no errors were foundThose warnings, as restic 0.18.1 prints them, are useful evidence in their own right: duplicate indexes on a repository nobody pruned can be a sign that a client tried to, although an interrupted prune on the trusted host leaves the same trace. For the Object Lock and Borg append-only versions of this test, see our immutable backups on Linux guide.
3-2-1 vs 3-2-1-1-0 vs a single immutable copy
Where the three schemes really differ is the attacker with admin credentials. This table maps each one against the failures it survives, based on the mechanisms above and the lab results:
| Failure | 3-2-1 | 3-2-1-1-0 | One immutable copy only |
|---|---|---|---|
| Disk or array failure | Survives | Survives | Survives, with a slow restore |
| Site loss (fire, flood) | Survives | Survives | Survives if the copy is off site |
| Accidental delete | Survives | Survives | Survives |
| Attacker with production and backup credentials | Usually lost | Survives on the immutable copy | Survives |
| Silent backup failure | Found during the incident | Found by the restore test | Found during the incident |
| Restore speed | Fast (local copy) | Fast (local copy) | Limited by the remote link |
Choose by the threat you are most likely to face and the restore time you can tolerate:
- Homelab or a single workstation: 3-2-1 with an external disk you unplug is close to 3-2-1-1-0 already, because an unplugged disk is offline. Our external drives for Linux backups roundup covers that leg.
- A small business with one server room: 3-2-1-1-0, with the immutable copy in an Object Lock bucket, so nobody has to rotate media by hand.
- Regulated data (finance, health): 3-2-1-1-0 with compliance-mode retention, which AWS and Azure have had assessed by Cohasset against SEC 17a-4.
- Only budget for one extra copy: make it the immutable one, then add the fast local copy when you can.
We’d skip the “one immutable copy only” design for anything with a restore-time target under a day, because pulling terabytes back over a WAN link is where recoveries stall. A local copy for speed plus an immutable copy for survival is the layout that covers both.
Tools in this space
These products cover one or more digits of 3-2-1-1-0. Each line states what the tool does, not a ranking; only restic and rest-server were run in this lab, and the Borg vs restic vs Kopia benchmark has measured numbers for the open-source three.
- restic and rest-server: open-source deduplicating backup client (latest release v0.19.1) with an HTTP server whose
--append-onlyflag “prevents deletion and modification of existing backups”. - BorgBackup: deduplicating backup over SSH (stable 1.4.5);
borg serve --append-onlykeeps committed data on the server even when a client runs prune. See our BorgBackup with Borgmatic guide. - Kopia: open-source backup client (v0.23.1) that can set S3 Object Lock retention on its blobs with
--retention-modeand--retention-period; set up in our Kopia with Backblaze B2 guide. - Proxmox Backup Server: deduplicated VM and container backups for Proxmox VE (4.2 line), with verify jobs, protected snapshots, tape support and an S3 datastore backend that left technology preview in 4.2.
- Veeam Backup & Replication: the backup engine of Veeam Data Platform (v13.1 on its product page), with immutable repositories and SureBackup for automated restore verification.
- Amazon S3 Object Lock: WORM storage with governance and compliance modes; compliance-mode objects can’t be deleted by any user before the retention date.
- Backblaze B2: S3-compatible object storage with Object Lock in governance or compliance mode, retention of 1 to 3,000 days.
- Wasabi: S3-compatible storage that supports the S3 Object Lock API with no added API request charges, and lists verification for Veeam Backup & Replication.
Whichever product holds the immutable copy, give it credentials the production estate never sees, because the lab shows that the copy survives only as long as the credentials that can delete it stay out of the attacker’s reach.
A quarterly restore drill
A zero in 3-2-1-1-0 only holds if someone proves it on a schedule. We would run this drill every quarter against the immutable copy, timed, so the next incident has a real recovery-time number behind it:
- Restore from the immutable copy into a scratch VM with no route to production, as if the local copy were gone.
- Compare hashes or row counts against a manifest taken at backup time, the same way the lab used
sha256sum -c. - Run the full integrity check (
restic check --read-data, a PBS verify job or Veeam SureBackup) and read every warning. - Try a delete with the production credentials and confirm it fails, because a lock that was never tested is another assumption.
- Record the wall-clock time from start to verified data, and compare it to your recovery-time target.
If step 4 succeeds, the copy is not immutable and the scheme is still plain 3-2-1, whatever the vendor brochure calls it. For filesystem-consistent sources to feed into the drill, LVM snapshots give the backup tool a frozen view of a busy server.
Sources
- Ruggiero and Heckathorn, Data Backup Options, US-CERT, 2012 (3-2-1 wording and the Krogh attribution).
- Veeam, 3-2-1 Backup Rule Explained (3-2-1-1-0 definition and SureBackup).
- Sophos, The State of Ransomware 2026 (66% backup recovery, 48% paid, $769K median payment).
- Sophos, The State of Ransomware 2025 (54% backup recovery, 38% backup failure among over-payers, 2020 to 2023 chart values).
- Sophos, The State of Ransomware 2024 (68% backup recovery, 56% paid).
- Veeam, 2024 Ransomware Trends Report press release (41% compromised, 57% recovered, 81% paid).
- Veeam, Data Trust and Resilience Report 2026 press release (90% confident, 28% full recovery).
- CISA, #StopRansomware Guide (offline, encrypted backups and the immutable storage caution).
- AWS, Locking objects with Object Lock; Backblaze, Object Lock; Microsoft, Azure Blob immutable storage; Wasabi, S3 Object Lock.
- restic rest-server README and restic v0.19.1 release, v0.19.0 release (forget exit code 3) and removing snapshots (append-only security considerations); BorgBackup append-only notes; Kopia ransomware protection.
- Proxmox, Proxmox Backup Server documentation and roadmap; Veeam, Backup & Recovery product page.
- Lab: restic 0.18.1 (Ubuntu 26.04 package) and rest-server v0.14.0 on two Ubuntu 26.04.1 VMs, October 2026; captures kept with the article source.
