Skip to content

Instantly share code, notes, and snippets.

@ssstonebraker
Last active December 15, 2018 07:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ssstonebraker/90a37a1ff5faac28d44ec76ba8ccfd79 to your computer and use it in GitHub Desktop.
Save ssstonebraker/90a37a1ff5faac28d44ec76ba8ccfd79 to your computer and use it in GitHub Desktop.
Linux dedupe compare files
#!/bin/bash
# Filename: showdupes.sh
# source: http://brakertech.com/compare-two-files-and-print-lines-that-match/
# this file takes two text files as input
# sorts them and outputs lines from
# file 2 that match file 1
if [ -f "$1" ] && [ -f "$2" ]
then
awk 'NR==FNR{arr[$0];next} $0 in arr' $1.tmp $2.tmp;
else
echo "Usage: dedupe.sh file1 file2";
echo "where file1 is the 'master' file";
echo "and file2 is the file possibly containing duplicates";
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment