Immutability

This page covers the immutability feature: what it is, why it matters, and how to set up the native on-prem service.

info

If your backup repository is an S3-compatible bucket, immutability is provided by the storage itself and behaves quite differently. See S3 Object Lock instead: the operational advice on this page does not transpose to it.

What is immutability?

In backup systems, immutability means that once a backup is created, it cannot be altered or deleted for a set period. This safeguards your backups against ransomware attacks, accidental deletion, or data corruption.

Here is how it works: you designate a backup repository (BR) that Xen Orchestra can write to, but cannot modify during the specified immutability period. Even if an attacker compromises XO, your backups remain untouched. The goal is to ensure that your backup data cannot be deleted, encrypted, or tampered with, unless someone with direct physical or root access to the storage intervenes.

Immutability can be implemented using external services like AWS S3 Object Lock, but Xen Orchestra also offers a native, on-premises solution that doesn't rely on third-party infrastructure.

Why it matters

Immutability guarantees that your backups stay secure and verifiable over time. It acts as a critical defense against ransomware, human mistakes, or intentional tampering. Additionally, it helps organizations comply with legal requirements by ensuring that data remains unaltered for mandatory retention periods. In essence, immutability turns your BR into a write-once, read-many (WORM) archive, protecting your data when it matters most.

Immutability approaches

Immutability offers two distinct approaches to data protection, to meet different security and operational needs.

Object storage

The first approach leverages object storage solutions like Amazon S3, where immutability is enforced using features such as Object Lock. In this setup, the storage provider itself manages the retention period, and it applies it to every object as soon as it is uploaded.

Object Lock protects object versions rather than making files read-only, so XO's own operations keep succeeding on a locked bucket and the protection works differently from what the rest of this page describes. Its setup, its compatible backup settings and its cost model are covered in S3 Object Lock.

On-prem immutability

The second model, and the subject of the rest of this page, is an on-premises immutable repository. It uses a lightweight package, installed on the BR host itself. This service monitors the repository and enforces immutability at the filesystem level, ensuring that XO cannot delete or overwrite existing backups before the end of the protection period.

The protection follows a governance model: the local root account of the BR host can still lift immutability. This is by design, so that an administrator with direct access to the storage always keeps control.

tip

The immutability engine was rewritten in 2026 on native filesystem APIs. It uses roughly 7 times less memory and locks and unlocks files much faster on large repositories, so it scales to BRs holding many VMs and disks. The new engine shipped with the XO 6.3 release.

Configuring on-prem immutability

To set up immutability on your local infrastructure, follow these steps:

1. Install the required package

1. Make sure Node.js (version 20 or later) is installed on your BR machine
2. Install the dedicated XO package globally:

On the backup repository host

# npm install -g @xen-orchestra/immutable-backups

The host must also meet these requirements:

• chattr and lsattr must be available on the system.
• The underlying filesystem must support the immutable attribute (ext4, btrfs, and other common Linux filesystems do).
• The service must run as root, or as a user with the CAP_LINUX_IMMUTABLE capability.

2. Configure the immutability settings

Create a configuration file at /etc/xo-immutable-backups/config.toml, with the following structure:

liftEvery = "1h"

[remotes.remote1]
root = "/mnt/ssd/vhdblock/"
immutabilityDuration = "7d"

Mandatory parameters

• liftEvery: Defines how often the service checks for expired protection and lifts immutability (e.g., 1h for every hour).
• Per backup repository (the remotes key in the configuration file keeps the legacy name):
  • root: Absolute path to the directory where Xen Orchestra stores backups.
  • immutabilityDuration: Defines how long files remain protected from deletion (e.g., 7d for 7 days). The minimum accepted value is 1d.

You can declare as many [remotes.<id>] sections as you have repositories to protect on the host.

3. Start the service

Launch the xo-immutable-remote daemon and check its logs (with journalctl, for example) to confirm it is running correctly. For persistent operation, configure it as a systemd service so it starts at boot and keeps running: the daemon must stay up to protect new backups as they are written.

How it works

Once active, any backups written by Xen Orchestra to this repository will be protected for the specified duration. Even if Xen Orchestra is compromised, the immutability configuration remains secure, as it is managed entirely on the backup host. On startup, the service verifies that the filesystem actually supports the immutable attribute, then writes a locked immutability.json marker at the repository root before it starts watching for new backups.

A backup's life on an immutable BR: written, locked for immutabilityDuration, lifted, then rotated out by retentionwritten, still mutablelocked backupdeleted by retentionliftEvery scan

Locking

The service watches the backup directory tree in real time. When a backup completes, it waits until the final file (the metadata .json, always written last by XO) has been fully written to disk, then locks all files belonging to that backup run with the Linux chattr +i attribute. This prevents any process, including Xen Orchestra, from modifying or deleting them.

Lifting

Protection is released automatically once the immutabilityDuration has elapsed. The service periodically (every liftEvery interval) scans the backup tree, computes the age of each backup from the datetime encoded in its filename, and removes the immutable attribute from any backup that has expired.

The expiry reference is the datetime in the filename, not the file's modification time. XO periodically rewrites metadata files for cache updates and reconciliation, which would reset mtime and indefinitely defer expiry if that were used instead.

On the first scan after the service starts, all files are checked unconditionally regardless of their current state, to catch anything that may have been missed (e.g. after a crash or restart).

Working with immutable backups

When setting up backup jobs in Xen Orchestra, select your configured immutable backup repository. Define your retention and rotation policies as you normally would. Immutability ensures that existing backups cannot be deleted or altered before their protection period expires, while still allowing new backups to be added.

Retention and immutability are two independent mechanisms, and they only coexist peacefully within a specific range of settings. The retention calculator models both, and shows what a given combination of retention, full backup interval and immutability duration actually protects. Run it before enabling immutability on a job.

Best practices

Only enable on stable, healthy backup jobs

warning

Immutability should only be enabled on backup jobs that are already running correctly and whose retention policy is fully settled. Nothing will be lost, but there will be a lot of EACCESS/EPERM errors in the backup log.

The points below describe the on-prem service, where a blocked operation surfaces as a permission error. On an object-locked bucket the same conflicts exist but produce no error at all, only silent storage growth: see S3 Object Lock.

Immutability and retention are two independent mechanisms, and they can conflict if a backup job is not in a clean state:

• Jobs that run more than once per schedule (accidental duplicates, misconfigured triggers) will accumulate extra backups that XO cannot clean up while they are protected. Those backups count against storage but cannot be removed until their immutability duration expires.
• Backups in an incorrect or partial state (failed mid-run, inconsistent chain) will be locked in place for the full immutability duration. The normal cleanup scripts cannot remove them, because any attempt to delete or overwrite a protected file raises a permission error (EPERM). Those errors are logged, but the files stay.
• Retention and immutability durations must be aligned. If the immutability duration is longer than the retention window, XO will keep trying, and failing, to delete backups it considers expired. Set immutabilityDuration to be at most equal to the retention period, so that files are only released after XO has already rotated them out.
• Do not use Long Term Retention (LTR) with immutability. GFS long-term retention selects and removes intermediate backups from within a chain, for example to keep only one backup per month. If any of those intermediate files are still immutable, the deletion fails with EPERM. The backup chain is left in an inconsistent state that XO cannot repair until the immutability duration expires. The retention calculator refuses the combination for that reason.
• The immutability duration must cover a whole backup chain. A restore point is only usable if its base full backup is still there. If the duration is shorter than the span of a full backup interval, the oldest backups protected at any given time are deltas whose base full has already been merged away: the files are locked, but they no longer restore anything. Either set immutabilityDuration to at least the time covered by one full backup interval, or set the full backup interval to 1 so that every chain is a single restore point.
• Keep the immutability duration shorter than the retention window. If the duration covers the entire retention window, the oldest restore point is never mutable when retention wants to rotate it out, so the job can never advance and backups pile up.
• A broken chain root blocks cleanup of the rest of the chain. In delta backup chains, removing any backup requires starting from the root. If the chain root is in a bad state (missing, corrupted, or partially written) and the remaining files are immutable, the cleanup script cannot remove them either: it encounters EPERM on each attempt and leaves the orphaned files in place until they age out naturally.
• Disks are not protected during upload. The immutable attribute is applied only after all disk images for a given backup run have been fully written. During that window, those files can still be modified or deleted. Coupling immutability with XO's at-rest encryption reduces this exposure, since an encrypted file is useless even if tampered with before locking. Object storage does not have this window: the bucket applies its retention at upload time.

In short: make sure your backup jobs are stable and producing clean results before adding immutability. Applying it to a job that already has problems will lock those problems in place.

Define a clear immutability policy

Align the lock duration with your data retention strategy, compliance requirements, and disaster recovery goals. Make sure the policy reflects both legal obligations and operational needs.

Secure your encryption keys

Store encryption keys separately from your backup data.

warning

Losing encryption keys will render backups permanently unrecoverable.

Maintain independence from Xen Orchestra

The immutability enforcement mechanism must operate independently of Xen Orchestra. This ensures that even if an attacker compromises XO, they cannot delete or alter existing backups.

Monitor storage and test recovery

• Track storage capacity closely, as immutability prevents immediate deletion of older backups.
• Plan for additional space to accommodate protected backups over time.
• Test recovery procedures regularly, to make sure immutability does not disrupt your ability to restore data when needed.

Limitations

The on-prem service depends on the immutable attribute of the underlying filesystem, so the repository must sit on a local filesystem that supports it. It cannot protect a repository mounted over NFS or SMB, because the attribute is enforced by the filesystem hosting the data, not by the client writing to it.

Cloud-based solutions such as S3 Object Lock depend on your provider's implementation; not all S3-compatible systems behave identically, so always test before production use. See S3 Object Lock for the specifics, and the supported object storage providers list for support tiers.

Troubleshooting

For troubleshooting, refer to the package README on GitHub. It covers the most common situations: files still immutable after the duration expired, making a repository or a single VM mutable again for manual cleanup, changing the immutability duration, and why incremental backups are only marked as protected in XO once their whole chain is inside the immutability window.

Conclusion

Immutability adds a critical layer of protection to your backups in Xen Orchestra. Whether you use Amazon S3 or an on-premises repository, the core principle is simple: once a backup is created, you cannot change or delete it until the protection period expires.

By integrating immutability with strong encryption, proactive monitoring, and regular testing, you ensure that your backup data stays secure, reliable, and recoverable, even in the face of ransomware, human error, or malicious attacks. This approach transforms your backup strategy into a robust defense against data loss and corruption.