Skip to content

Instantly share code, notes, and snippets.

@liejuntao001

liejuntao001/diff lots of files

Last active November 30, 2019 05:10
Show Gist options
  • Save liejuntao001/a572bf31d3af0fa259b73ffc4b90173a to your computer and use it in GitHub Desktop.
Save liejuntao001/a572bf31d3af0fa259b73ffc4b90173a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
diff lots of files
# this is to handle a case of comparing a volume of 90G against its backup
# there are some huge files inside it (3G, 20G)
# https://unix.stackexchange.com/questions/111251/compare-massive-directories-with-progress-report
# find out the total file count
find dir1 -type f | wc -l
# comapre and show progress. filecount is the number from above step
diff -rqs dir1 dir2 | pv -l -s filecount | tee logfile | grep -v "^Files .* identical$"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment