Skip to content

Instantly share code, notes, and snippets.

@phikal
Last active May 18, 2017 20:04
Show Gist options
  • Save phikal/d128bed2079e2e96b121be8fc54c4599 to your computer and use it in GitHub Desktop.
Save phikal/d128bed2079e2e96b121be8fc54c4599 to your computer and use it in GitHub Desktop.
Delete duplicate files (ie. same hash) within a directory.
#!/bin/bash
MDF="/tmp/deldup.md5"
DUP="/tmp/deldup.list"
if [ $MDF -nt . ]; then
echo found cached hash list in $MDF
else
echo -n "hashing files... "
find -O3 . -type f -print0 $@ | xargs -L 1 -P 2 -0 md5sum > $MDF
echo finished
fi
cut -f1 -d" " < $MDF | sort | uniq -d > $DUP
echo Found $(cat $DUP | wc -l) duplicate files
for dh in $(cat $DUP); do
echo
echo All files with hash $dh:
grep -a $dh $MDF | cut -f2- -d" " | {
readarray -t -O 1 FILES
C=1 # counter
for line in $(seq ${#FILES[@]}); do
printf "%4d\t%s\n" $C "${FILES[$C]}"
C=$((C+1))
done
echo -n 'Select to delete (eg. "1", "2,3,4", "1,3-5"): '
IFS=',' read -ra RANGE <&1;
for i in "${RANGE[@]}"; do
if [[ $i =~ ^([0-9]+)-([0-9]+)$ ]]; then
START=${BASH_REMATCH[1]}
END=${BASH_REMATCH[2]}
for x in $(seq $START $END); do
rm -v "$(echo ${FILES[$x]} | xargs -0 echo)"
done
elif [[ $i =~ ^[0-9]+$ ]]; then
rm -v "$(echo ${FILES[$i]} | xargs -0 echo)"
else
echo Invalid range or number \"$i\"
fi
done
}
done
echo
read -p "delete cache [y/N]? " DC
[[ $DC =~ ^[yY] ]] && rm -v $MDF && rm -v $DUP
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment