Skip to content

Instantly share code, notes, and snippets.

@vrdhn

vrdhn/dedupe.md

Last active November 22, 2021 04:32
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save vrdhn/7f76c68d1a966eaea121764460c1b9dd to your computer and use it in GitHub Desktop.
Save vrdhn/7f76c68d1a966eaea121764460c1b9dd to your computer and use it in GitHub Desktop.
Download ZIP
Raw
dedupe.md

Flow

  • A 'Master' store : which all the files are to be moved to permanently
  • Several 'Source' stores : where all the files are currently, and need to be empties

The Master store is external, cloud or removable storage, so it's not scanned completely. By construction, there will be no duplicated files on it, and there will be a check ( which can be costly with cloud storage ).

The master store will have a db storing size, sample-hash , full-hash etc. The tool will move only new files to Master Store, and update db

There will be option to transform the directory path while mocinv/copynig/hardlinking the files into master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment