At 9:30 on a Friday, a team I worked with kicked off what they called a “routine” restore drill. They expected to prove a 45-minute RTO. Instead, the test blew up in under six minutes.
The backup catalog looked healthy. Jobs were green. Alerts were quiet. But one critical bucket had no immutable retention policy, another had recent versions overwritten by automation, and the “gold” vault was still in a changeable window nobody remembered setting.
That drill changed how they thought about resilience. Backups are not a checkbox, they are a control system. If you want ransomware resilience and true recovery confidence, you need immutable cloud backups plus repeatable restore testing, not just “successful backup jobs”.
This guide is the runbook I wish we had before that Friday incident, built around AWS S3 Object Lock, AWS Backup Vault Lock, and practical restore testing.
1) Start with failure assumptions, not architecture diagrams
Most teams design backup pipelines as if the only risk is hardware failure. Real incidents look different:
- An IAM credential leak deletes snapshots.
- A misconfigured lifecycle policy expires the wrong data tier.
- A rushed script overwrites object versions during migration.
- A restore procedure exists, but nobody has run it in months.
That is why immutable cloud backups matter. You are designing for situations where privileged users, automation, or malware attempt destructive changes.
If this sounds familiar, it connects with the reliability lessons in this backend partial-failure playbook. Green dashboards can still hide systemic risk.
2) Build the immutable storage layer first (S3 Object Lock)
From AWS docs, two details are non-negotiable:
- Object Lock requires versioning.
- Retention is enforced per object version, not by object key name.
That means your policy has to protect versions, and your restore runbook must know which version to recover.
resource "aws_s3_bucket" "immutable_backup" {
bucket = "prod-immutable-backup-123456"
object_lock_enabled = true
}
resource "aws_s3_bucket_versioning" "immutable_backup" {
bucket = aws_s3_bucket.immutable_backup.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_object_lock_configuration" "immutable_backup" {
bucket = aws_s3_bucket.immutable_backup.id
rule {
default_retention {
mode = "GOVERNANCE" # move to COMPLIANCE when process is mature
days = 30
}
}
}
Tradeoff: Governance mode gives controlled bypass paths for authorized users. Compliance mode is stricter, but recovery and deletion operations become operationally harder if you misconfigure retention. Many teams start in governance mode while they validate runbooks, then move high-value datasets to compliance mode.
3) Lock the vault policy plane (AWS Backup Vault Lock)
S3 immutability protects object versions. Vault Lock protects backup governance at the vault level, including retention enforcement against early deletion.
AWS Backup supports governance and compliance modes here too. Compliance mode includes a grace period, then lock settings become immutable. That is powerful, but unforgiving if you rushed your retention design.
# Compliance mode example (includes a changeable grace period)
aws backup put-backup-vault-lock-configuration \
--backup-vault-name prod-immutable-vault \
--changeable-for-days 7 \
--min-retention-days 35 \
--max-retention-days 3650
# Verify lock state and effective limits
aws backup describe-backup-vault \
--backup-vault-name prod-immutable-vault \
--query '{Locked:Locked,LockDate:LockDate,MinRetentionDays:MinRetentionDays,MaxRetentionDays:MaxRetentionDays}'
Important operational warning: if you allow indefinite retention accidentally, locked data can become a permanent cost anchor. Review lifecycle values with finance and security together before lock activation. We touched related cost-governance patterns in this cloud cost control guide.
4) Define restore testing as an engineering ritual, not an audit task
AWS Backup restore testing exists for a reason. Teams overfocus on backup completion and underfocus on recovery viability.
In practice, a useful cadence looks like this:
- Weekly: restore one low-risk service dataset and verify application-level integrity.
- Monthly: restore one business-critical dataset into an isolated account/environment.
- Quarterly: run full incident simulation with RTO/RPO scoring and executive sign-off.
Do not stop at “restore job completed.” Validate the thing your business cares about: can the service actually boot, read, and serve correct data?
This is where runbook quality matters. If your procedures drift between teams, you get brittle incident response. We covered that failure mode in this operational memory article.
5) A practical reference workflow for small platform teams
If your team is under 10 engineers, keep it simple and enforceable:
- Classify datasets by impact (tier-0, tier-1, tier-2).
- Apply S3 Object Lock defaults per tier (shorter for dev, longer for prod).
- Create dedicated backup vaults per environment, then apply Vault Lock.
- Run restore testing plans with clear pass/fail criteria.
- Track evidence in Git (test IDs, restore times, exception notes).
Also, treat IAM as part of backup architecture. If broad admin roles can bypass controls casually, your immutability posture is mostly theoretical.
Provenance and policy checks in CI can help here too, similar to what we discussed in this GitHub attestation runbook.
Troubleshooting: what commonly breaks and how to fix it
Problem 1: “Object Lock isn’t working, deletes still appear to succeed”
Cause: Teams often test with simple DELETE (without version ID), which can create delete markers while protected versions still exist.
Fix: Inspect object versions explicitly and test deletion behavior with version IDs. Confirm retention/hold state on the target version, not just the key.
Problem 2: Vault lock enabled, but restores fail in tests
Cause: Restore metadata gaps (network, subnet, IAM role, or target parameter mismatch), not backup corruption.
Fix: Standardize restore templates per workload and run test restores in the same account structure you use during incidents.
Problem 3: Backup jobs fail after setting min/max retention
Cause: Plan lifecycle settings conflict with vault lock bounds.
Fix: Align backup plan retention values to vault limits before enforcing lock. Add pre-deploy checks in IaC pipelines to block mismatch.
Problem 4: Costs spike after immutability rollout
Cause: Overly long default retention, duplicated copies, and no cold-tier lifecycle design.
Fix: Review tier policy per dataset class. Keep immutable does not mean infinite. Design retention windows from legal, business, and cost requirements together.
FAQ
1) Should we use governance mode or compliance mode first?
For most teams, start governance mode during rollout so you can recover from policy mistakes. Move critical datasets to compliance mode once restore procedures and retention choices are proven.
2) Is immutable backup enough for ransomware readiness?
No. You still need identity hardening, segmentation, incident playbooks, and tested restore paths. Immutability reduces destructive blast radius, but it does not replace broader security controls.
3) How often should we run restore testing?
At minimum, monthly for critical systems. If your RTO targets are aggressive, test weekly on representative datasets. The key is consistency and measurable pass/fail evidence.
Actionable takeaways
- Adopt immutable cloud backups as a policy system, not a storage feature.
- Enable S3 Object Lock with version-aware restore procedures.
- Apply AWS Backup Vault Lock only after retention values are validated cross-team.
- Run restore testing on a schedule and score real RTO/RPO outcomes.
- Store evidence and exceptions in Git so incident response is auditable and repeatable.
Closing note
The team from that Friday drill did one thing right after the incident: they stopped calling backups “done” and started calling recovery “proved.”
That mindset shift is the difference between passing compliance paperwork and surviving a real outage.
Leave a Reply