Ransomware crews go after backups first, because a victim with a clean restore point doesn’t pay. If anyone holding root on the backup client can delete your snapshots, those snapshots were never safe. Immutable backups close that gap: once data lands, the storage layer refuses to change or delete it until a retention window expires, no matter which credentials the attacker stole.
This guide builds two immutable targets on Linux and attacks both. The first is Restic writing to a RustFS bucket with S3 Object Lock in COMPLIANCE mode. The second is Borg in append-only mode over SSH. Earlier versions used MinIO, but its community downloads now answer 410 Gone and the upstream repository was archived in April 2026, so RustFS takes its place. Every command below was run on Ubuntu 26.04 LTS and Rocky Linux 10 in October 2026, including the failed delete attempts and the recoveries.
Choose the Immutability Layer
An immutable backup means the backup client can add data but can’t destroy what is already stored. You get that from one of two places: the storage API (S3 Object Lock) or the backup server process (Borg’s append-only mode). A third layer, a backup key that has no delete permission, stops most attacks before either of the first two is tested. The lab showed what each layer really blocks:
| Attacker has | Restic + backup-only S3 key | Restic + RustFS admin key | Borg append-only |
|---|---|---|---|
| What the delete does | Refused: Access Denied | Succeeds on paper, leaves delete markers | Succeeds on paper, logged as a transaction |
| Locked data survives? | Yes, nothing was touched | Yes, COMPLIANCE versions refuse deletion | Yes, segments stay on disk |
| Recovery work after a delete attack | None | Remove delete markers, restic unlock | Roll back the transaction log on the server |
| Root on the storage host | Can wipe /data/rustfs | Can wipe /data/rustfs | Can wipe the repository |
| Automatic expiry | Yes, retention days | Yes, retention days | No, manual |
Read the last two rows carefully. Object Lock is enforced by the S3 API, so it protects you from anyone who talks to RustFS over the network, including the admin account. It doesn’t protect against root on the storage server, who can delete the files under /data/rustfs directly. That is why the backup server has to sit on its own credentials, its own network segment and ideally its own admin team. With that in place, run both targets: Restic with Object Lock as the primary and Borg as an independent second copy.
Prerequisites
The lab uses two machines, and the same split works in production. Keep the backup server off the domain or directory that manages your production fleet, because a shared admin account turns “immutable” back into “deletable”.
- backup01: Ubuntu 26.04 LTS, 10.0.1.50, runs RustFS and the Borg server
- app01: Rocky Linux 10 with SELinux enforcing, 10.0.1.51, runs Restic and the Borg client
- 2 vCPU and 4 GB RAM on backup01 (RustFS idled at 57 MB)
- Disk on backup01 sized for the full retention window, not just one backup
- sudo access on both hosts
Tested versions were RustFS 1.0.4, the RustFS rc client 0.1.36, AWS CLI 2.39.1, Restic 0.19.1, Borg 1.4.4 on Ubuntu and Borg 1.4.5 from EPEL on Rocky. The credentials in this guide are lab throwaways; generate your own.
Install RustFS on the Backup Server
If you followed the old MinIO steps, this is where they broke. The download URL now answers HTTP 410 instead of serving a binary, so the old wget step stops with an error before anything is installed. Fetch the URL yourself and the server explains why.
curl -s https://dl.min.io/server/minio/release/linux-amd64/minioMinIO’s notice covers mc and every hotfix build too, so there is no older URL to fall back on. Pinning an old binary from a mirror would also mean running object storage that gets no security fixes, which is the wrong trade for the one server meant to survive an attack.
410 Gone
The open-source MinIO Server, MinIO Client (mc) and MinIO KES projects are
archived and no longer maintained. MinIO does not provide product support,
security updates, or security advisories for them, and does not accept or
process vulnerability reports concerning them.
These files are no longer served from this site. This applies to all
community releases and to all hotfix builds for them.We picked RustFS as the replacement because its own documentation spells out the behaviour this guide depends on: Object Lock on a versioned bucket, a default retention rule, and COMPLIANCE mode that blocks deletion even with a governance bypass. It ships as a single static binary under the Apache 2.0 licence, and it speaks the same S3 API Restic already uses. Garage is out because it has no bucket versioning, so it can’t do Object Lock at all. If you would rather keep MinIO’s command set, the community fork from Pigsty (SILO) still publishes builds, but we did not test it here.
Download and Verify the Binary
RustFS is at 1.0.x and moves quickly, so pin the release you tested instead of pulling “latest” on every rebuild. The GitHub release ships a SHA256SUMS file, which the docs’ latest.zip link does not give you. Install unzip, then fetch both files and check them:
sudo apt install -y unzip
VER=1.0.4 # pin: tested release, RustFS 1.0.x moves fast; https://github.com/rustfs/rustfs/releases
mkdir -p ~/rustfs && cd ~/rustfs
curl -fsSLO https://github.com/rustfs/rustfs/releases/download/$VER/rustfs-linux-x86_64-musl-v$VER.zip
curl -fsSLO https://github.com/rustfs/rustfs/releases/download/$VER/SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMSOnly continue when the zip reports OK. A mismatch here means a broken download or a tampered file, and you don’t want either one guarding your backups.
rustfs-linux-x86_64-musl-v1.0.4.zip: OKUnpack it and install the binary. The zip holds a single 276 MB file named rustfs:
unzip -o rustfs-linux-x86_64-musl-v$VER.zip
sudo install -m 0755 rustfs /usr/local/bin/rustfs
rustfs --versionThe version banner should match the tag you pinned. Note the git commit too, because it’s the quickest way to match this binary against a RustFS security advisory later:
rustfs 1.0.4
build time : 2026-10-25 13:46:05 +00:00
build profile: release
build os : linux-x86_64
build target : x86_64-unknown-linux-musl
rust version : rustc 1.99.1 (58cf8f9c3 2026-10-13)
rust channel : stable-x86_64-unknown-linux-musl
git branch :
git commit : 46337942d55f3a8e6a97b147ca98d9ea8115555d
git tag : 1.0.4
git status :RustFS should never run as root. A bug in a network-facing storage daemon running as root hands the attacker the box, and with it the files under the bucket. Create a system account with no login shell, plus the data and log directories it owns:
sudo groupadd -r rustfs
sudo useradd -M -r -g rustfs -s /usr/sbin/nologin rustfs
sudo mkdir -p /data/rustfs /var/log/rustfs
sudo chown -R rustfs:rustfs /data/rustfs /var/log/rustfs
sudo chmod -R 750 /data/rustfs /var/log/rustfsPut /data/rustfs on its own disk in production. When the retention window is 30 days, the volume has to hold 30 days of backups plus everything Restic tried to prune, and a full root filesystem takes the whole server down with it.
Create the Environment File
RustFS reads its settings from environment variables. Open the file the systemd unit will load:
sudo vim /etc/default/rustfsAdd the admin credentials, the data path and the two listeners. Port 9000 is the S3 API and 9001 is the web console:
RUSTFS_ACCESS_KEY=backupadmin
RUSTFS_SECRET_KEY=Ch4ngeMe-Lab-Only-2026
RUSTFS_VOLUMES="/data/rustfs"
RUSTFS_ADDRESS=":9000"
RUSTFS_CONSOLE_ENABLE=true
RUSTFS_CONSOLE_ADDRESS=":9001"
RUSTFS_OBS_LOGGER_LEVEL=error
RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"That file holds the one key that can create delete markers, so only root and the service group should read it:
sudo chown root:rustfs /etc/default/rustfs
sudo chmod 640 /etc/default/rustfsTreat the admin key as a break-glass credential. Production hosts never get it, because the attack section shows exactly what it can do in the wrong hands; they get the backup-only user created later in this guide. Store a copy in your password vault so you can still run a recovery if backup01 itself needs rebuilding.
Create the Systemd Service
The unit below is the one from the RustFS docs, with the user, group and data path changed to match this server. Create it:
sudo vim /etc/systemd/system/rustfs.servicePaste the full unit. The hardening options at the bottom are worth keeping, since this process holds every backup you have:
[Unit]
Description=RustFS Object Storage Server
Documentation=https://rustfs.com/docs/
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
NotifyAccess=main
User=rustfs
Group=rustfs
WorkingDirectory=/usr/local
EnvironmentFile=-/etc/default/rustfs
ExecStart=/usr/local/bin/rustfs $RUSTFS_VOLUMES
LimitNOFILE=1048576
LimitNPROC=32768
TasksMax=infinity
Restart=always
RestartSec=10s
OOMScoreAdjust=-1000
SendSIGKILL=no
TimeoutStartSec=120s
TimeoutStopSec=30s
NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectClock=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictRealtime=true
StandardOutput=append:/var/log/rustfs/rustfs.log
StandardError=append:/var/log/rustfs/rustfs-err.log
[Install]
WantedBy=multi-user.targetType=notify means systemd only reports the service as started once RustFS says it’s ready, and TimeoutStartSec=120s gives it two minutes to get there. If the start times out, read /var/log/rustfs/rustfs-err.log before anything else. Start it and confirm both ports are listening:
sudo systemctl daemon-reload
sudo systemctl enable --now rustfs
systemctl status rustfs --no-pager
ss -tlnp | grep -E ':900[01]'You want active (running) with the status line reading Running, plus listeners on 9000 and 9001:
● rustfs.service - RustFS Object Storage Server
Loaded: loaded (/etc/systemd/system/rustfs.service; enabled; preset: enabled)
Active: active (running) since Thu 2026-10-29 17:05:08 UTC; 5s ago
Invocation: 91dac4f51dbe49cebdc05440de7da5b4
Docs: https://rustfs.com/docs/
Main PID: 32997 (rustfs)
Status: "Running"
Tasks: 15
Memory: 57.5M (peak: 57.6M)
CPU: 161ms
CGroup: /system.slice/rustfs.service
└─32997 /usr/local/bin/rustfs /data/rustfs
Oct 29 17:05:08 backup01 systemd[1]: Starting rustfs.service - RustFS Object Storage Server...
Oct 29 17:05:08 backup01 systemd[1]: Started rustfs.service - RustFS Object Storage Server.
LISTEN 0 4096 *:9001 *:*
LISTEN 0 4096 *:9000 *:* The Ubuntu 26.04 cloud image ships with ufw inactive. If you enable ufw, allow SSH first (Borg and your own logins use it, and skipping it locks you out), then the S3 port, and keep 9001 closed unless you reach the console from a management network. The lab only ran the S3 rule, because ufw stayed inactive there:
sudo ufw allow 9000/tcpThe production servers only need port 9000 for Restic and port 22 for Borg. Nothing else on backup01 should be reachable from them, so if you have a network firewall between the segments, write the rule there as well. A host firewall alone is one ufw disable away from open, and an attacker who reaches backup01 as root would run exactly that.
Create the Object-Locked Bucket
Create the bucket with Object Lock from the start, so versioning and the lock are both on before Restic’s first write. RustFS can also add a lock configuration to an existing versioned bucket, but anything written before that stays unlocked. Restic’s init creates a plain bucket, so you create this one yourself first. We used the AWS CLI for this because it maps one to one onto the S3 API calls the RustFS docs describe, and because you’ll need the same tool later to inspect object versions during a recovery.
Install the Official AWS CLI
Skip the awscli package from the Ubuntu 26.04 archive. Version 2.31.35 installs fine, but every s3api list-* call we ran with it died before sending a request:
badly formed help stringThat breaks list-object-versions, which is exactly the call you need during recovery. The official AWS build bundles its own Python and doesn’t have the problem. Remove the distro package if you installed it, then install the official one:
sudo apt remove -y awscli
cd ~
curl -fsSLO https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip
unzip -qo awscli-exe-linux-x86_64.zip
sudo ./aws/install --update
hash -r
aws --versionThe exe/x86_64 part of the version string marks the bundled build:
aws-cli/2.39.1 Python/3.14.8 Linux/7.0.0-15-generic exe/x86_64.ubuntu.26Save the RustFS admin key as a named profile, and keep the endpoint in a variable so the following commands stay short. The CLI insists on a region, and us-east-1 worked fine against RustFS:
aws configure set aws_access_key_id backupadmin --profile rustfs
aws configure set aws_secret_access_key Ch4ngeMe-Lab-Only-2026 --profile rustfs
aws configure set region us-east-1 --profile rustfs
export AWS_PROFILE=rustfs
S3=http://10.0.1.50:9000Add the export AWS_PROFILE and S3= lines to ~/.bashrc on backup01 if you’ll come back to this shell later. Every aws command from here on assumes both are set.
Create the Bucket with a 30-Day COMPLIANCE Default
Create the bucket with the lock flag. A successful create-bucket prints nothing, so we check versioning right after it.
aws --endpoint-url $S3 s3api create-bucket --bucket immutable-backups --object-lock-enabled-for-bucket
aws --endpoint-url $S3 s3api get-bucket-versioning --bucket immutable-backupsVersioning comes back enabled even though you never asked for it. Object Lock depends on it, and it’s also the reason deletes later turn into delete markers instead of real removals. If this reads Suspended or prints nothing, the lock flag didn’t take, so delete the empty bucket and create it again.
{
"Status": "Enabled"
}Now set a default rule so every object Restic writes is locked for 30 days in COMPLIANCE mode. Restic sends no retention headers of its own, so without this rule its objects would be versioned but not locked:
aws --endpoint-url $S3 s3api put-object-lock-configuration --bucket immutable-backups \
--object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":30}}}'
aws --endpoint-url $S3 s3api get-object-lock-configuration --bucket immutable-backupsThe rule reads back exactly as written:
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 30
}
}
}
}Pick the number of days before the first backup lands. COMPLIANCE retention can’t be shortened later, not even by the admin account, so a 365 typed by mistake means a year of storage you can’t reclaim. Thirty days covers the dwell time of most ransomware incidents and keeps the space cost predictable.
Create a Backup-Only User for Restic
Object Lock saves the data, but a key with delete rights still lets an attacker hide it behind delete markers and force you into a recovery. A key with no delete rights stops the attempt outright. Restic does need to remove its own lock files under locks/, so the policy allows DeleteObject on that prefix and nowhere else. Install the RustFS admin client, rc, from its release page and verify its checksum:
cd ~/rustfs
RCVER=v0.1.36 # pin: tested release; https://github.com/rustfs/cli/releases
curl -fsSLO https://github.com/rustfs/cli/releases/download/$RCVER/rustfs-cli-linux-amd64-$RCVER.tar.gz
curl -fsSLO https://github.com/rustfs/cli/releases/download/$RCVER/rustfs-cli-linux-amd64-$RCVER.tar.gz.sha256
sha256sum -c rustfs-cli-linux-amd64-$RCVER.tar.gz.sha256
tar xzf rustfs-cli-linux-amd64-$RCVER.tar.gz
sudo install -m 0755 rc /usr/local/bin/rc
rc alias set rustfs http://10.0.1.50:9000 backupadmin Ch4ngeMe-Lab-Only-2026The checksum line has to read OK and the alias step confirms with a tick. The alias stores the admin key in your home directory on backup01, which is acceptable there and nowhere else.
rustfs-cli-linux-amd64-v0.1.36.tar.gz: OK
✓ Alias 'rustfs' configured successfully.Write the policy to a file. RustFS takes AWS-style IAM policy JSON, so the actions and ARNs below read exactly as they would in an S3 bucket policy:
vim ~/restic-app01-policy.jsonAdd this document. Listing, reading and writing cover restic backup, check and restore; the third statement is the only delete it gets:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket", "s3:GetBucketLocation"],
"Resource": ["arn:aws:s3:::immutable-backups"]
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": ["arn:aws:s3:::immutable-backups/*"]
},
{
"Effect": "Allow",
"Action": ["s3:DeleteObject"],
"Resource": ["arn:aws:s3:::immutable-backups/locks/*"]
}
]
}Create the policy, create a user per production host, and attach the policy to it. We named ours after the host it serves.
rc admin policy create rustfs restic-app01-write ~/restic-app01-policy.json
rc admin user add rustfs restic-app01 Restic-App01-Lab-Only-2026
rc admin policy attach rustfs restic-app01-write --user restic-app01
rc admin user info rustfs restic-app01The user should show as enabled with exactly one policy attached. If you see more than one policy here, the user inherits the union of all of them, so a broader policy left over from testing quietly hands the delete right back:
✓ Policy 'restic-app01-write' created successfully.
✓ User 'restic-app01' created successfully.
✓ Policy 'restic-app01-write' attached to user 'restic-app01' successfully.
Access Key: restic-app01
Status: enabled
Policies: restic-app01-write
Groups: (none)One user per host also tells you which machine was compromised when the RustFS logs show refused deletes, which a shared key never will. It also lets you disable a single host’s key during an incident while every other server keeps backing up. This lab ran a single client. For a fleet, give each host its own prefix in the bucket (for example immutable-backups/app01) and its own policy scoped to that prefix, because hosts sharing one repository can read each other’s data, and the exclusive lock taken by restic check makes a second host’s backup fail while it runs.
Know what this key still allows, because we only tested deletes. PutObject lets a stolen key write a new current version over config, a key file or a snapshot; the locked older version survives, but Restic reads the newest one, so you would have to promote the old version back by hand. Every junk write also inherits the 30-day COMPLIANCE lock, so a stolen key can fill /data/rustfs with data nobody can delete. Watch bucket growth on backup01 and size the volume with headroom.
Configure Restic on app01
Everything in this section runs on app01. Restic needs root to read system files, and Rocky Linux 10 leaves /usr/local/bin out of root’s PATH, so open a root shell and add it:
sudo -i
export PATH=$PATH:/usr/local/binWithout that export, every restic command below answers command not found even though the binary is installed. Ubuntu’s root PATH includes /usr/local/bin, so this step is specific to the RHEL family. Stay in this root shell until the Borg section, because the environment you load next lives only in it.
Install Restic from GitHub
EPEL carries Restic, but it lags the upstream releases. The Rocky cloud image also ships only vi, so the first line adds vim-enhanced for the file edits that follow. Look up the latest tag, download the binary with its checksum file, and verify it before installing:
dnf install -y bzip2 vim-enhanced
VER=$(curl -sL https://api.github.com/repos/restic/restic/releases/latest | grep tag_name | head -1 | sed 's/.*"v\([^"]*\)".*/\1/')
echo $VER
cd /tmp
curl -fsSLO https://github.com/restic/restic/releases/download/v$VER/restic_${VER}_linux_amd64.bz2
curl -fsSLO https://github.com/restic/restic/releases/download/v$VER/SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
bunzip2 -f restic_${VER}_linux_amd64.bz2
install -m 0755 restic_${VER}_linux_amd64 /usr/local/bin/restic
restic versionWe got 0.19.1 and a passing checksum. The lookup always pulls the newest release, so pin the version you tested in your own automation, because a new release could change prune behaviour under you:
0.19.1
restic_0.19.1_linux_amd64.bz2: OK
restic 0.19.1 compiled with go1.26.4 on linux/amd64Restic is a single static binary, so you have nothing else to install on app01.
Store the Credentials and Initialize the Repository
Keep the repository password in a root-only file, and back it up somewhere off this host. Without it the encrypted repository is unreadable, Object Lock or not:
openssl rand -base64 24 > /root/.restic-pass
chmod 600 /root/.restic-passThe same environment file serves your interactive shell now and the systemd timer later, so the key only lives in one place:
vim /etc/restic-envUse the backup-only user, never the RustFS admin key. Anything in this file is readable by whoever gets root on app01, so it should only ever hold credentials you are prepared to see stolen:
AWS_ACCESS_KEY_ID=restic-app01
AWS_SECRET_ACCESS_KEY=Restic-App01-Lab-Only-2026
RESTIC_REPOSITORY=s3:http://10.0.1.50:9000/immutable-backups
RESTIC_PASSWORD_FILE=/root/.restic-passLock the file down, load it into the shell, and initialize the repository inside the bucket we created on backup01.
chmod 600 /etc/restic-env
set -a; . /etc/restic-env; set +a
restic initRestic writes its config and key objects into the bucket, and both are locked from this moment. Copy /root/.restic-pass to your password vault now, because the locked objects are useless without it and app01 is the machine you are planning to lose.
created restic repository a6d0f0f7be at s3:http://10.0.1.50:9000/immutable-backups
Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.The backup-only key handled init without trouble, which confirms the policy grants enough for normal use before you rely on it. If init fails with Access Denied instead, compare the policy’s ListBucket and PutObject statements against the bucket name, so you find the gap now rather than during the first nightly run.
Run the First Backup
Generate 50 files with random content to stand in for business data, then back them up and record their checksums. The checksums are how you’ll prove the restore is byte-for-byte correct, not just “some files came back”:
mkdir -p /srv/important-data
for i in $(seq 1 50); do echo "Critical business document $i $(date '+%F %T') $(openssl rand -hex 32)" > /srv/important-data/document_$i.txt; done
restic backup /srv/important-data
sha256sum /srv/important-data/*.txt > /root/original-checksums.txt
restic snapshotsThe snapshot ID in the last table is the one the attacker goes after next, and yours will differ from bae9711f. Note that 5.6 KiB of files became 13.2 KiB stored, because Restic’s pack and index overhead dominates on tiny test data; on real data the ratio flips:
no parent snapshot found, will read all files
Files: 50 new, 0 changed, 0 unmodified
Dirs: 2 new, 0 changed, 0 unmodified
Added to the repository: 29.002 KiB (13.238 KiB stored)
processed 50 files, 5.606 KiB in 0:00
snapshot bae9711f saved
ID Time Host Tags Paths Size
-------------------------------------------------------------------------------------
bae9711f 2026-10-29 17:07:13 app01 /srv/important-data 5.606 KiB
-------------------------------------------------------------------------------------
Timestamps shown in local time
1 snapshotsEvery object behind that snapshot now carries a COMPLIANCE lock that runs 30 days from the moment it was written. You can confirm it on any object with s3api get-object-retention from backup01, as the next section does before it tries to delete one.
Test the Restic Repository Against Deletion
Assume the attacker has root on app01. They can read /etc/restic-env and the password file, so they have everything Restic has. We ran the attack three ways, from the weakest key to the strongest, and the results differ in ways the old version of this guide got wrong.
Delete with the Backup-Only Key
First the obvious move: forget the snapshot and prune the data behind it, using the credentials already on the box. Substitute your own snapshot ID here and in the next command:
restic forget bae9711f --pruneRustFS refuses the very first delete, and Restic gives up without touching the data packs. The Go stack trace at the end looks alarming, but it’s only Restic reporting where the error surfaced:
[0:00] 0.00% 0 / 1 files deleted
Remove(<snapshot/bae9711f59>) failed: client.RemoveObject: Access Denied
unable to remove snapshot/bae9711f5930c8d97b04753f675d0a4ca753f8a699aa13bb820a8252105a08ab from the repository
failed to remove one or more snapshots
main.init
/restic/cmd/restic/cmd_forget.go:67
runtime.doInit1
/usr/local/go/src/runtime/proc.go:8103
runtime.doInit
/usr/local/go/src/runtime/proc.go:8070
runtime.main
/usr/local/go/src/runtime/proc.go:258
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1771A follow-up restic snapshots still lists bae9711f. This is the outcome you want on every production host, and it’s why the backup-only user matters more than the lock itself for day-to-day safety: the attack fails at the first call and leaves nothing to clean up. The side effect is that you can’t run forget from app01 either, so retention cleanup moves to the backup server, where the admin key lives. That’s the right place for it anyway, because the host that decides what to delete should not be the host most likely to be compromised.
Delete with Stolen Admin Keys
Now assume the attacker also found the RustFS admin key, say in a forgotten script or a password manager export. Override the two key variables and run the same command:
AWS_ACCESS_KEY_ID=backupadmin AWS_SECRET_ACCESS_KEY=Ch4ngeMe-Lab-Only-2026 restic forget bae9711f --pruneThis time Restic reports a clean prune. Every pack and index is listed as deleted, because every S3 delete call came back successful:
[0:00] 100.00% 1 / 1 files deleted
1 snapshots have been removed, running prune
loading all snapshots...
finding data that is still in use for 0 snapshots
[0:00] 0 snapshots
searching used packs...
collecting packs for deletion and repacking
[0:00] 100.00% 2 / 2 packs processed
to repack: 0 blobs / 0 B
this removes: 0 blobs / 0 B
to delete: 53 blobs / 11.116 KiB
total prune: 53 blobs / 11.116 KiB
remaining: 0 blobs / 0 B
unused size after prune: 0 B ( of remaining size)
rebuilding index
[0:00] 100.00% 1 / 1 indexes processed
[0:00] 100.00% 1 / 1 old indexes deleted
removing 2 old packs
[0:00] 100.00% 2 / 2 files deleted
doneRun restic snapshots afterwards and it prints nothing at all. From the client’s point of view the backups are gone, and this is the point where an untested setup panics. What happened is that Restic sent ordinary deletes without a version ID. On a versioned bucket those don’t remove data; they stack a delete marker on top of each object, which hides it from normal listings. Check from backup01:
aws --endpoint-url $S3 s3api list-object-versions --bucket immutable-backups --prefix snapshots/Read the output from the bottom up. The delete marker has IsLatest: true, which is why Restic’s normal listing can’t see the snapshot anymore. The real object sits above it with the same key, its own version ID and its full 355 bytes, so nothing was actually lost:
{
"Versions": [
{
"ETag": "\"a8bcb5a535687766443c1ac43d332372\"",
"Size": 355,
"StorageClass": "STANDARD",
"Key": "snapshots/bae9711f5930c8d97b04753f675d0a4ca753f8a699aa13bb820a8252105a08ab",
"VersionId": "6d4767aa-4bd6-4c52-b035-9abab6708674",
"IsLatest": false,
"LastModified": "2026-10-29T17:07:14.031000+00:00"
}
],
"DeleteMarkers": [
{
"Key": "snapshots/bae9711f5930c8d97b04753f675d0a4ca753f8a699aa13bb820a8252105a08ab",
"VersionId": "eb1a960d-d78a-4f68-bf00-ebb5e524b81f",
"IsLatest": true,
"LastModified": "2026-10-29T17:07:39.745000+00:00"
}
],
"RequestCharged": null,
"Prefix": "snapshots/"
}Across the whole bucket the attack left 11 delete markers: 2 on data packs, 1 on the index, 1 on the snapshot and 7 on lock files. Every one of them can be removed, which is the whole recovery.
Try to Remove a Locked Version
A smarter attacker with the admin key skips Restic and deletes the versions themselves. Try it on backup01, setting K and V to the key and version ID from your own listing above. First read the retention on that version:
K=snapshots/bae9711f5930c8d97b04753f675d0a4ca753f8a699aa13bb820a8252105a08ab
V=6d4767aa-4bd6-4c52-b035-9abab6708674
aws --endpoint-url $S3 s3api get-object-retention --bucket immutable-backups --key $K --version-id $VThe default rule stamped it with COMPLIANCE mode and a date 30 days after the backup ran:
{
"Retention": {
"Mode": "COMPLIANCE",
"RetainUntilDate": "2026-11-28T17:07:14.031000+00:00"
}
}Now delete that exact version, once plainly and once with the governance bypass flag:
aws --endpoint-url $S3 s3api delete-object --bucket immutable-backups --key $K --version-id $V
aws --endpoint-url $S3 s3api delete-object --bucket immutable-backups --key $K --version-id $V --bypass-governance-retentionBoth fail, and the error names the date the lock expires. The bypass flag makes no difference, because it only applies to GOVERNANCE mode and these objects were written under the COMPLIANCE default:
aws: [ERROR]: An error occurred (AccessDenied) when calling the DeleteObject operation: Object is under COMPLIANCE retention and cannot be deleted until 2026-11-28 17:07:14.031276215 +00:00:00
aws: [ERROR]: An error occurred (AccessDenied) when calling the DeleteObject operation: Object is under COMPLIANCE retention and cannot be deleted until 2026-11-28 17:07:14.031276215 +00:00:00The remaining tricks fail the same way. Shortening the retention date, or switching the version to GOVERNANCE so a bypass would work, returns AccessDenied from PutObjectRetention. Deleting the bucket returns BucketNotEmpty, because every locked version still counts as content. We also tried the RustFS client’s forced removal, rc bucket remove --force, on a second locked test bucket; it stopped at the first locked version with Access Denied and left the bucket in place. Every network path we tested ends the same way: the data stays until the 30 days run out, so an attacker who wants a payout has to stay hidden in your network for a month first.
Encrypt the Production Data
Back on app01, finish the simulation the way ransomware does. It encrypts every file in place and deletes the original, so the only clean copy left is the one in the backups you just watched it fail to destroy.
cd /srv/important-data
for f in *.txt; do openssl enc -aes-256-cbc -salt -pbkdf2 -in "$f" -out "${f}.encrypted" -pass pass:ransom_key_2026 && rm -f "$f"; done
ls /srv/important-data | head -5All 50 documents now carry the .encrypted suffix, and the directory still counts 50 entries. Real ransomware would also drop a ransom note here:
document_10.txt.encrypted
document_11.txt.encrypted
document_12.txt.encrypted
document_13.txt.encrypted
document_14.txt.encryptedThe Rocky cloud image doesn’t include xxd; dnf install -y vim-common adds it. A hex dump shows OpenSSL’s Salted__ header followed by ciphertext:
xxd /srv/important-data/document_1.txt.encrypted | head -3Nothing readable is left in the file, and without the attacker’s key there is no way back from app01 itself. Recovery has to come from the backups:
00000000: 5361 6c74 6564 5f5f c656 b6dc 0fa9 195d Salted__.V.....]
00000010: c809 f832 7099 0c43 8e0e d477 8d1d 6ccd ...2p..C...w..l.
00000020: 66fa efbf 1159 43a8 8d64 c6f9 9f6e 3303 f....YC..d...n3.At this point app01 holds only ciphertext, and the attacker has had full use of both keys against the backups. From the client side it looks hopeless: the snapshot list is empty and the files are gone. The recovery starts on backup01, because removing the markers needs the admin key and that key must never touch app01. In a real incident you would rebuild app01 before restoring onto it; the lab restores in place to keep the test short.
Restore Restic from the Delete Markers
Delete markers carry no retention, so the admin key can remove them, and each removal makes the locked version underneath current again. Do this from backup01, never from the compromised host. Save a small script that finds every marker that is currently on top and deletes it by version ID:
sudo vim /usr/local/bin/s3-undelete.shThe bucket and endpoint are arguments with defaults, so the same script works for other buckets. It only touches markers that are currently on top, which means older versions and the locked data are never part of the loop:
#!/bin/bash
set -euo pipefail
BUCKET=${1:-immutable-backups}
EP=${2:-http://10.0.1.50:9000}
export AWS_PROFILE=${AWS_PROFILE:-rustfs}
n=0
while read -r key vid; do
[ -z "$key" ] && continue
aws --endpoint-url "$EP" s3api delete-object --bucket "$BUCKET" --key "$key" --version-id "$vid" >/dev/null
n=$((n+1))
done < <(aws --endpoint-url "$EP" s3api list-object-versions --bucket "$BUCKET" \
--query 'DeleteMarkers[?IsLatest==`true`].[Key,VersionId]' --output text | grep -v '^None')
echo "removed $n delete markers"Make it executable and run it as the user whose ~/.aws holds the rustfs profile. Run it from backup01 only, because the admin key it needs must never be typed on a host you are still cleaning up. It prints a single count when it finishes:
sudo chmod 755 /usr/local/bin/s3-undelete.sh
s3-undelete.shIt removed all 11 markers the attack created. Every marker removal is a normal S3 call, so the locks never have to be bypassed for this to work:
removed 11 delete markersThis blanket undo is right when the only deletes since your last good state came from the attacker. If you also run legitimate retention cleanup, restrict the script to markers newer than the start of the incident instead of removing everything. The full recovery runs in this order:
- Remove the delete markers from backup01
- Clear stale locks with
restic unlock - Confirm with
restic snapshotsandrestic check - Restore to a staging directory and compare checksums
Skipping the check in step 3 is tempting when the snapshot list looks right, but a list only proves the snapshot object is back. The check compares the index against the packs the backend lists and reports any that are missing, so it catches a data object still hidden behind a marker you missed. Add --read-data if you also want every pack downloaded and verified.
Error: “repository is already locked exclusively”
Back on app01, your first snapshot list fails. Prune took an exclusive lock, its delete of that lock file became one more marker, and removing the markers brought the stale lock back with everything else:
repo already locked, waiting up to 0s for the lock
unable to create lock in backend: repository is already locked exclusively by PID 52637 on app01 by root (UID 0, GID 0)
lock was created at 2026-10-29 17:07:39 (3m2.602982006s ago)
storage ID 3b43c386
the `unlock` command can be used to remove stale locksThe backup-only key is allowed to delete under locks/, so restic unlock works with the normal credentials. Clear the locks, then list and check the repository:
restic unlock
restic snapshots
restic checkThe snapshot the attacker “deleted” is back, and the full check finds no damage. The unlock line reports 7 locks, matching the 7 lock-file markers from the attack. On a server that has run for weeks, expect far more: every backup and check deletes its own lock, so the blanket undo also revives old locks, and restic unlock clears the stale ones. A lock younger than 30 minutes from another host is not considered stale; restic unlock --remove-all removes those too:
successfully removed 7 locks
ID Time Host Tags Paths Size
-------------------------------------------------------------------------------------
bae9711f 2026-10-29 17:07:13 app01 /srv/important-data 5.606 KiB
-------------------------------------------------------------------------------------
Timestamps shown in local time
1 snapshots
using temporary cache in /tmp/restic-check-cache-2726532435
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00% 1 / 1 snapshots
no errors were foundOnly restore once check reports no errors. A failed check after undeleting usually means a marker was missed, so run the script again and check that list-object-versions shows no DeleteMarkers left before you touch production data.
Restore and Verify the Files
Move the encrypted files aside instead of deleting them, restore to a staging directory, copy the data back and compare checksums against the ones you saved before the attack. The quarantine copy is removed only after the checksums match:
cd /
mv /srv/important-data /root/quarantine-encrypted && mkdir /srv/important-data
restic restore latest --target /tmp/restic-restore
cp -a /tmp/restic-restore/srv/important-data/. /srv/important-data/
sha256sum /srv/important-data/*.txt > /tmp/restored-checksums.txt
diff /root/original-checksums.txt /tmp/restored-checksums.txt && echo MATCH && rm -rf /root/quarantine-encrypted
ls /srv/important-data/*.txt | wc -lAn empty diff followed by MATCH means every file is identical to the original. The restore count of 52 is the 50 files plus the two parent directories, so do not read it as two extra files:
restoring snapshot bae9711f of [/srv/important-data] at 2026-10-29 17:07:13.314529877 +0000 UTC by root@app01 to /tmp/restic-restore
Summary: Restored 52 files/dirs (5.606 KiB) in 0:00
MATCH
50Restoring to a staging directory and keeping the quarantine until MATCH is deliberate. If the restore is incomplete you still have the encrypted copy, and some ransomware families get public decryptors later. If diff prints anything, the quarantine stays in /root/quarantine-encrypted for you to investigate. In the lab, latest is the clean snapshot. In a real incident the nightly timer may already have backed up the ciphertext, so pick the last snapshot from before the compromise in restic snapshots and restore it by ID instead. Without a checksum list like ours, spot-check the restored files by opening them.
Set Up Borg Append-Only Mode
Borg gives you a second, unrelated target. There is no object store here: the backup server runs borg serve through a forced SSH command, and the --append-only flag makes it keep every segment it ever wrote. It’s simpler to stand up than RustFS, and an attack that works against one tool rarely works against both. If you’re still choosing tools, our Borg, Restic and Kopia comparison covers the trade-offs beyond immutability.
Install BorgBackup on Both Hosts
Ubuntu carries Borg in its universe component, which cloud images enable by default. On backup01:
sudo apt install -y borgbackup
borg --versionUbuntu 26.04 installed 1.4.4. The server side never needs anything newer than the clients it serves, so you can leave this package on the distro’s update cadence:
borg 1.4.4On Rocky Linux, Borg comes from EPEL because the base repositories don’t carry it. Run this on app01:
sudo dnf install -y epel-release
sudo dnf install -y borgbackup
borg --versionEPEL is one point release ahead of Ubuntu. That mismatch caused no trouble in the lab, since the 1.4.5 client worked against the 1.4.4 server for every step below, but keep both ends on the same 1.x series. Borg 2 changes the repository format, so don’t upgrade one side alone:
borg 1.4.5With both ends installed, the rest of the setup is SSH configuration. The append-only guarantee lives entirely in the server’s authorized_keys line, so that file is the one to protect and audit. Anyone who can edit it can drop the flag and turn the next client connection into a full-access one.
Restrict the SSH Key to Append-Only
On backup01, create a dedicated account and the repository directory. The account exists only to run borg serve, and we never log in to it interactively.
sudo useradd -m -s /bin/bash borgbackup
sudo mkdir -p /backup/borg-repo
sudo chown borgbackup:borgbackup /backup/borg-repo
sudo install -d -m 700 -o borgbackup -g borgbackup /home/borgbackup/.sshOn app01, generate a key used for nothing but backups. It has no passphrase because a timer has to use it unattended, which is acceptable only because the server side limits it to one command:
sudo ssh-keygen -t ed25519 -f /root/.ssh/borg_backup_key -N "" -C borg-backup-client
sudo cat /root/.ssh/borg_backup_key.pubCopy the public key it prints. Don’t use ssh-copy-id for this one, because it would install the key without the restriction below and give the client a full shell. Open the borgbackup account’s authorized_keys on backup01 instead:
sudo vim /home/borgbackup/.ssh/authorized_keysPrefix the key with the forced command. Whatever the client asks for, sshd runs exactly this, so even a stolen key can only ever speak the Borg protocol. Replace the key part with the one from your app01:
command="borg serve --restrict-to-path /backup/borg-repo --append-only",restrict ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL8Ti/K4g9HuAYU42RtQpILYyrehHu4qzep6xnKEdPNI borg-backup-client--append-only keeps every written segment, --restrict-to-path pins the client to one directory, and restrict turns off port forwarding, agent forwarding and PTY allocation. Set the ownership and mode sshd expects:
sudo chown borgbackup:borgbackup /home/borgbackup/.ssh/authorized_keys
sudo chmod 600 /home/borgbackup/.ssh/authorized_keysWith its default StrictModes, sshd ignores an authorized_keys file that is group- or world-writable, and the client then can’t log in with the key at all. That fails a scheduled job silently until someone reads the logs.
Initialize the Repository and Back Up
Back in the root shell on app01, tell Borg which key to use and set the passphrase. The key has a non-default name, so without BORG_RSH ssh never offers it. The accept-new option records backup01’s host key on first contact instead of stopping at a yes/no prompt, which would hang a timer:
export BORG_RSH="ssh -i /root/.ssh/borg_backup_key -o StrictHostKeyChecking=accept-new"
export BORG_PASSPHRASE='Borg-Lab-Only-2026'
export BORG_REPO=ssh://[email protected]/backup/borg-repo
borg init --encryption=repokey-blake2Borg reminds you that the passphrase and the key both matter. In repokey mode the key lives inside the repository on backup01, so if that server’s disk dies you need your exported copy to read the second replica:
IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
Key storage location depends on the mode:
- repokey modes: key is stored in the repository directory.
- keyfile modes: key is stored in the home directory of this user.
For any mode, you should:
1. Export the borg key and store the result at a safe place:
borg key export REPOSITORY encrypted-key-backup
borg key export --paper REPOSITORY encrypted-key-backup.txt
borg key export --qr-html REPOSITORY encrypted-key-backup.html
2. Write down the borg key passphrase and store it at safe place.Take that advice and export the key to somewhere outside both machines. Then record checksums, create the first archive and list it:
sha256sum /srv/important-data/*.txt > /root/original-checksums-borg.txt
borg create --stats ::app01-{now} /srv/important-data
borg listThe archive name carries the timestamp, which you’ll need for the restore. The {now} placeholder is expanded by Borg, so each scheduled run gets a unique name without any shell date logic:
------------------------------------------------------------------------------
Repository: ssh://[email protected]/backup/borg-repo
Archive name: app01-2026-10-29T17:12:00
Archive fingerprint: eef5f5f64432ae8a2dad642e4e4cb53b499e038471bf32e233f86c5ee074d413
Time (start): Thu, 2026-10-29 17:12:00
Time (end): Thu, 2026-10-29 17:12:00
Duration: 0.04 seconds
Number of files: 50
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 5.74 kB 7.89 kB 7.89 kB
All archives: 5.74 kB 7.89 kB 11.57 kB
Unique chunks Total chunks
Chunk index: 52 52
------------------------------------------------------------------------------
app01-2026-10-29T17:12:00 Thu, 2026-10-29 17:12:00 [eef5f5f64432ae8a2dad642e4e4cb53b499e038471bf32e233f86c5ee074d413]Keep the archive fingerprint from this output, because it’s how you’ll confirm later that the restored archive is the original. The 7.89 kB stored against 5.74 kB of files is metadata overhead on tiny test data, so do not use it to size anything real.
Test Borg Against Deletion and Roll It Back
Now play the attacker on app01, with the same environment you just used to back up. Delete the archive (use the name your borg list printed), try to destroy the whole repository, then compact it to free the space:
borg delete ::app01-2026-10-29T17:12:00
borg list
borg delete --force
borg compactThe archive delete exits 0 and borg list comes back empty, so the attacker believes it worked. The repository delete is the one call the server refuses outright, with a traceback that contains this line:
ValueError: /backup/borg-repo is in append-only modeborg compact also exits 0, but in append-only mode it frees nothing. Every segment is still on disk, and the server logs each operation as a numbered transaction. Read the log on backup01:
sudo -u borgbackup cat /backup/borg-repo/transactionsTransaction 5 is the backup, 9 is the delete and 12 is the compact. The timestamps line up with the attacker’s commands to the second, so the last good state is 5. On a busy repository you would read this list against your incident timeline instead:
transaction 5, UTC time 2026-10-29T17:12:00.708332
transaction 9, UTC time 2026-10-29T17:12:11.526014
transaction 12, UTC time 2026-10-29T17:12:13.576524Match those times against when you believe the compromise began. On a real server you’ll see a long run of nightly backups and then a burst of deletes from the attacker’s session, and everything before the burst is safe to keep.
Roll Back to the Last Good Transaction
The Borg append-only notes describe the rollback: remove the hints, index and integrity files, which Borg can always rebuild, then remove every segment written after the good transaction. Take a hard-linked copy of the repository first, so you keep the evidence and can undo a wrong cut. The segments in that copy never change, but the transactions log is shared through the hard link and keeps growing, so save a separate copy of that file if you need it for the incident report:
sudo cp -al /backup/borg-repo /backup/borg-repo.compromised
sudo -u borgbackup bash -c "cd /backup/borg-repo && rm -v hints.12 index.12 integrity.12 && rm -v data/0/{6..12}"The suffix on the three files is the latest transaction (12 here), and the segments start one after the good transaction (6). Then let Borg rebuild its index from the segments that remain:
sudo -u borgbackup borg check --repository-only /backup/borg-repoIt replays the segments and exits 0. Afterwards the repository holds fresh index files with the suffix 5, which confirms Borg now considers transaction 5 the latest state:
Replaying segments. Please wait, this can take a while...The server side is now back at transaction 5, but the client doesn’t know that yet. It cached the repository state after transaction 12, and Borg treats any repository that moves backwards as a possible replay attack, so the next command from app01 refuses to run.
Error: “Cache, or information obtained from the security directory is newer than repository”
The client still remembers the repository as it was after transaction 12, and refuses to talk to an older copy. borg list on app01 fails with:
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)Here you rolled the repository back yourself, so the warning is expected. The append-only notes and the Borg FAQ give the same fix: delete only the manifest-timestamp file for this repository and drop the local cache. The FAQ calls this an unsafe, unsupported path, which is fair, since it switches off a protection on purpose. Leave the rest of the security directory alone, because its nonce file is the encryption counter, and losing it risks nonce reuse. Get the repository ID on backup01:
sudo -u borgbackup borg config /backup/borg-repo idIt prints the 64-character ID, which is also the directory name under /root/.config/borg/security/ on the client:
3f4f96f805514bb2cc0dae1899930c42039dd56a803cf9968d5ff8bd8bc26333Back on app01, substitute your own ID in the path, then clear the cache and list again:
rm /root/.config/borg/security/3f4f96f805514bb2cc0dae1899930c42039dd56a803cf9968d5ff8bd8bc26333/manifest-timestamp
borg delete --cache-only
borg list
borg checkThe deleted archive is listed again, and borg check exits 0 without output. The fingerprint matches the one from the original borg create, so this is the same archive and not a rebuilt one:
app01-2026-10-29T17:12:00 Thu, 2026-10-29 17:12:00 [eef5f5f64432ae8a2dad642e4e4cb53b499e038471bf32e233f86c5ee074d413]Borg rewrites manifest-timestamp on that first successful run, and the nonce file is still in place next to it. Only do this after you have rolled back yourself. If that error shows up when nobody touched the server, treat it as the attack Borg is warning you about.
Extract and Verify the Borg Archive
Extract into a scratch directory, again with your own archive name, and compare against the checksums taken before the attack. Borg extracts relative to the current directory, so strip that prefix from the paths before diffing:
mkdir /tmp/borg-restore && cd /tmp/borg-restore
borg extract ::app01-2026-10-29T17:12:00
sha256sum /tmp/borg-restore/srv/important-data/*.txt | sed "s# /tmp/borg-restore# #" > /tmp/borg-checksums.txt
diff /root/original-checksums-borg.txt /tmp/borg-checksums.txt && echo MATCH
ls /tmp/borg-restore/srv/important-data | wc -lAll 50 files match the originals, so the rollback lost nothing that was written before the attack:
MATCH
50The Borg recovery takes more hands-on work than the S3 one, and it depends on the server keeping an honest transaction log. That’s the gap Object Lock closes, and the reason Borg is the second copy here, not the only one.
Run Restic on a Systemd Timer
A systemd timer catches up after downtime with Persistent=true and logs every run to the journal, which cron doesn’t (the systemd timers guide covers the syntax). The unit reuses /etc/restic-env from earlier. Create the service on app01:
sudo vim /etc/systemd/system/restic-backup.serviceThe two cache lines are not optional. Our first version left them out, and the very first timed run failed:
[Unit]
Description=Restic backup to RustFS (Object Lock bucket)
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
EnvironmentFile=/etc/restic-env
CacheDirectory=restic
Environment=RESTIC_CACHE_DIR=/var/cache/restic
ExecStart=/usr/local/bin/restic backup /srv/important-data /etc
ExecStartPost=/usr/local/bin/restic check
Nice=19
IOSchedulingClass=idleWithout CacheDirectory and RESTIC_CACHE_DIR, the journal shows this and the unit exits with status 1:
restic[54238]: unable to open cache: unable to locate cache directory: neither $XDG_CACHE_HOME nor $HOME are definedSystem services run without $HOME, and Restic has nowhere to put its cache. CacheDirectory=restic makes systemd create /var/cache/restic, and the environment line points Restic at it. Now the timer:
sudo vim /etc/systemd/system/restic-backup.timerIt fires at 02:00 with up to 15 minutes of jitter, so a fleet of hosts does not hit the backup server at the same second. Persistent=true runs a missed backup at the next boot, which matters for anything that gets shut down at night:
[Unit]
Description=Nightly restic backup
[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=900
Persistent=true
[Install]
WantedBy=timers.targetEnable the timer, then trigger one run by hand so you see a failure today instead of at 2 AM. A oneshot unit blocks systemctl start until the backup and the check finish, so the command returns only when the run is done:
sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup.timer
sudo systemctl start restic-backup.service
journalctl -u restic-backup.service -o cat --no-pager | tail -12The backup and the post-run check both succeed, and systemd reports the unit finished cleanly. SELinux stayed enforcing, and ausearch -m avc -ts today found no denials, because the binary in /usr/local/bin carries the standard bin_t label and needs no custom policy:
processed 606 files, 20.389 MiB in 0:00
snapshot 2398a50d saved
using temporary cache in /var/cache/restic/restic-check-cache-96503601
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:00] 100.00% 3 / 3 snapshots
no errors were found
restic-backup.service: Deactivated successfully.
Finished restic-backup.service - Restic backup to RustFS (Object Lock bucket).
restic-backup.service: Consumed 1.035s CPU time, 46.9M memory peak.The first timed run in our lab added 20.959 MiB of /etc and test data as 4.685 MiB stored after compression and deduplication. Leave forget out of this unit on purpose: the backup-only key can’t run it, and retention belongs on the backup server.
Keep the Backup Server Out of Reach
The lab proves the locks hold. These are the gaps it leaves open for you, in the order we’d close them:
- Separate admin accounts for backup01, with no SSO or directory trust shared with production
- Firewall rules that let production reach only 9000/tcp and 22/tcp on backup01
- The RustFS admin key kept off production hosts, with one backup-only user per host
- An alert on stale snapshots
- A monthly restore drill with checksum comparison
The first item matters most because of the root-on-storage row in the table at the top. If the same compromised directory account can log in to backup01, the attacker skips the S3 API and runs rm -rf /data/rustfs, and no retention setting survives that. A storage host that production admins can’t reach is what makes COMPLIANCE mode mean anything. Give backup01 its own local accounts, its own SSH keys held by a small group, and console access through a separate management network, so that losing your production identity provider does not also lose your backups.
For the alert, a snapshot-age check is enough. This script reads the same environment file as the timer and exits non-zero when the newest snapshot is older than 26 hours, which leaves two hours of slack on a daily schedule. Save it on app01:
sudo vim /usr/local/sbin/restic-snapshot-age.shRocky 10 already ships python3, which parses the JSON. Python does the date maths because Restic records snapshot times with the client’s UTC offset, and comparing them as timezone-aware values keeps the age right on servers that do not run in UTC. The script loads the same backup-only key as the timer, so it needs no extra credentials:
#!/bin/bash
set -a; . /etc/restic-env; set +a
export PATH=$PATH:/usr/local/bin RESTIC_CACHE_DIR=/var/cache/restic
restic snapshots --json --latest 1 | python3 -c '
import sys, json, re, datetime
t = json.load(sys.stdin)[-1]["time"]
t = re.sub(r"(\.\d{6})\d+", r"\1", t).replace("Z", "+00:00")
ts = datetime.datetime.fromisoformat(t)
age = int((datetime.datetime.now(datetime.timezone.utc) - ts).total_seconds() // 3600)
print(f"latest={ts.isoformat()} age_hours={age}")
sys.exit(0 if age < 26 else 1)
' && echo OK || { echo STALE; exit 1; }Make it executable and run it once. Our capture comes from an earlier timer run in the same lab, so its timestamp and the snapshot count in the journal above are one run apart from a straight read of this guide.
sudo chmod 755 /usr/local/sbin/restic-snapshot-age.sh
sudo /usr/local/sbin/restic-snapshot-age.shRun soon after a backup, it reports zero hours. On a healthy daily schedule the number climbs to about 24 and drops back each night, so anything above 26 is worth a page:
latest=2026-10-29T17:14:13.367190+00:00 age_hours=0
OKWire the exit code into whatever already pages you, whether that’s Zabbix, Prometheus’ textfile collector or a plain mail on failure. Run the check from a different host if you can. A compromised app01 can fake its own health check, but it can’t stop backup01 from noticing that no new objects arrived.
Budget storage for the lock window. Nothing Restic prunes is freed until its retention ends, and the deletes add markers, so the bucket only grows during the window. After our short test it held 23 object versions in 5.3 MB, against a few kilobytes of live data. For a 100 GB dataset changing 5% a day, plan on roughly 100 GB plus 30 days of 5 GB increments, about 250 GB, before deduplication helps. Run forget and prune from backup01 with the admin key, then remove noncurrent versions once their lock has expired. We did not test RustFS lifecycle rules for that cleanup, so check them against the RustFS docs before you rely on them.
Finally, keep the two targets apart. Restic to RustFS on backup01 plus Borg to a second machine in another room or another provider means one stolen key, one bad disk or one rogue admin can’t take both. Take LVM snapshots for consistent backups of busy databases before Restic reads them. If you need a plain S3 target without the lock, the Restic to S3 storage guide covers retention and pruning. For whole-machine images next to these file-level copies, UrBackup fills the gap.
Hi
I just found this tutorial and wanted to try it out. Seems that opensource-min.io is discontinued, so could you show up an alternative storage backend?
Thanks a lot!
You’re right, MinIO’s community builds are gone. The guide now uses RustFS for the object-locked bucket, with the same restic repository, deletion test and restore steps on top. If you’d rather keep MinIO’s command set, Pigsty’s SILO fork still publishes builds, but we haven’t tested it here.