working notes for borg history manifest.md

borg history

New command borg history --manifest will list out all segment entries that are PUTs of an all-zeroes key, along with their decrypted values.

If a repo has been maintained strictly in an append-only manner, and no deletions attempted by an adversary with append-only access, then this will provide a listing of all archives that were ever created. If such an adversary attempts to delete an archive, they'll have to add a new manifest which omits an archive -- and that can be detected by comparing it to older, shadowed manifests.

The invariant can be maintained across authorized deletions if the rightful owner fully compacts after deletions. Process below.

Safe deletion of archives in an append-only repo

• Verify integrity of recent backups via spot-checks or other situation-specific criteria
• Perform a borg check
• With lock:
  • Read the local list of known-deleted archive IDs (initially empty)
  • Run the new borg history --manifest command
  • From the history output, confirm that the archive list only ever grows from one version of the manifest to the next, or that any dropped archive IDs are in the known-deleted list
    • If there are any unexpected archive removals, log and and exit with an error -- this may be the sign of an attack
  • Log this history locally for reference (since it will be lost)
  • Choose which archives to prune
  • Add them to the known-deleted list
  • Perform the borg delete action
  • Perform borg compact with 0% threshold so that all DELETEs are compacted

Afterwards, there will only be a single manifest in the history, thanks to the compaction.

The latter recipe is much easier if you do it like this:

• Lock out any access to the repo other than from the known secure client/key to be used for non-append-only operation.
• Ensure borg check --verify-data shows no errors
• Ensure borg history --manifest shows no deletions other that checkpoint archives to detect trivial manipulations
• Verify integrity of all archives against existing integrity data to be kept to detect advanced manipulations
• Porform borg delete or borg prune, if desired
• Drop append-only mode
• Perform borg compact with 0% threshold so that all DELETEs are compacted
• Reinstate append-only mode

With any intended deletions only benig done after the manifest history, you do not have to filter them from its output.

I was thinking that if the process dies after the borg delete step but before compaction happens, it will have the appearance of an attack. Even if it signals an error and says "hey, this automated pruning job failed and needs attention", there needs to be some kind of escape hatch to say "don't worry about those archive deletions".

But... it occurs to me now that as long as deletions are deterministic, I can move the "Choose which archives to prune" step to before the check on the archive list. If the "attacker" (or a previous botched run) only tries to delete a subset of the archives I was going to delete anyhow... well, that's not so bad, is it?

(Towers of Hanoi backup rotation seems to be appropriate here, as long as I can trust timestamps.)

I was thinking that if the process dies after the borg delete step but before compaction happens, it will have the appearance of an attack. Even if it signals an error and says "hey, this automated pruning job failed and needs attention", there needs to be some kind of escape hatch to say "don't worry about those archive deletions".

I never ever thought of nor ever wanted any of the recipe above to be automatic in any way. This is something for an admin to do manually, starting with disabling e.g. SSH key access to the repo to perform the "lock out any other access" step, setting up a fresh machine from trusted media to temporarily entrust with the repo keys for the remaining steps, etc.
Additionally, the "Verify integrity of all archives against existing integrity data" is the crucial and by far biggest step that is external anyway.

Ideas for a the other things a plain borg history without the --manifest limitation should:

• Iterate over all segments
  • record chunk IDs of all PUTs, report any PUT for a chunk ID other than 0 that has been PUT before as an issue
    • additionally (later) handle the following things by reading any such multiple chunks and downgrade the issue to a warning if all copies do have identical (decompressed/decrypted) and correct according to data content:
      • borg recreate doing recompression (which makes little sense on an append-only repo in the first place and would rather be done with apend-only dropped after running this command as part of auditing the repo before doing so)
      • aborted compaction that eventually left around an old uncompacted segment along with the new compacted copy. Compaction however only happens with append-only mode disabled, thus in any practical use-case after this command has been used to audit the repository before disabling append-only mode and can be expected to be completed/any results of it being aborted to be cleaned up/repaired before continuing actual use of the repository that ultimately may lead to another audit of its history.
    • more tricky cases, these can and should not be ignored, because an attacker could PUT zeroad out chunks before then putting tampered ones to masquerade the tampering as a repair, and the presence of both repaired and unrepaired (last PUT zeroad) is still noteworthy to point out in the repository history when legitimate:
      • borg check --repair PUTing zeroed chunks, which due to conflicting with append-only mode would just like above operation only be done after audit & disabling of append-only mode.
      • borg create after a borg check --repair PUTing chunks that heal zeroed ones. This may happen lateron, and if the chunks to heal are not taken from other backups through the secure mangemant/temporary system used for the audit, but from the potentially hacked client, it actually should happen with append-only mode enabled. borg history pointinng out any files repaired the latter way would then be used in the next audit&repair cycle to specifically audit the contents (checksums/hashes) against records of those at time of backup before letting the repair finally heal them.
  • report any DELETEs other than chunk ID 0 (handled by the manifests part) or those referring to an archive that is a checkpoint
    • additionally (later) come up with a way to detect archives being renamed using borg rename (I'd assume those to show as DELETE(s) of the meta data of the old archive and PUT(s) of meta data for a new archive differing only in the encoded name)

I like the idea of locking out access at the SSH level rather than using borg's lock. I think there would still need to be either A) some persistence of "which archives did I intend to delete" or B) a deterministic method of archive-to-delete selection that will remain stable (or only grow) in the face of more archives having been created since a previous, aborted run. But that would all be external to borg anyhow.