Security considerations in append-only mode

Note: TL;DR: With append-only repositories, one should specifically use the --keep-within option of the forget command when removing snapshots.

To prevent a compromised backup client from deleting its backups (for example due to a ransomware infection), a repository service/backend can serve the repository in a so-called append-only mode. This means that the repository is served in such a way that it can only be written to and read from, while delete and overwrite operations are denied. rustic’s rest-server features an append-only mode, but few other standard backends do. To support append-only with such backends, one can use rclone as a complement in between the backup client and the backend service.

To remove snapshots and recover the corresponding disk space, the forget and prune commands require full read, write and delete access to the repository. If an attacker has this, the protection offered by append-only mode is naturally void. The usual and recommended setup with append-only repositories is therefore to use a separate and well-secured client whenever full access to the repository is needed, e.g. for administrative tasks such as running forget, prune and other maintenance commands.

However, even with append-only mode active and a separate, well-secured client used for administrative tasks, an attacker who is able to add garbage snapshots to the repository could bring the snapshot list into a state where all the legitimate snapshots risk being deleted by an unsuspecting administrator that runs the forget command with certain --keep-* options, leaving only the attacker’s useless snapshots.

For example, if the forget policy is to keep three weekly snapshots, and the attacker adds an empty snapshot for each of the last three weeks, all with a timestamp (see the backup command’s --time option) slightly more recent than the existing snapshots (but still within the target week), then the next time the repository administrator (or a scheduled job) runs the forget command with this policy, the legitimate snapshots will be removed (since the policy will keep only the most recent snapshot within each week). Even without running prune, recovering data would be messy and some metadata lost.

To avoid this, forget policies applied to append-only repositories should use the --keep-within option, as this will keep not only the attacker’s snapshots but also the legitimate ones. Assuming the system time is correctly set when forget runs, this will allow the administrator to notice problems with the backup or the compromised host (e.g. by seeing more snapshots than usual or snapshots with suspicious timestamps). This is, of course, limited to the specified duration: if forget --keep-within 7d is run 8 days after the last good snapshot, then the attacker can still use that opportunity to remove all legitimate snapshots.

Last change: 2024-02-14, commit: 5720dd8