Skip to content

Instantly share code, notes, and snippets.

@linxlunx

linxlunx/add_column_and_remove_duplicate.sh

Created July 13, 2020 12:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save linxlunx/42fa74f0669140edb4387063dad2cecd to your computer and use it in GitHub Desktop.
Save linxlunx/42fa74f0669140edb4387063dad2cecd to your computer and use it in GitHub Desktop.
Download ZIP
Raw
add_column_and_remove_duplicate.sh
# add to first column for matching purpose (lower char), $3 is key to be matched
cat first.csv | awk -F"," 'BEGIN { OFS = "," }; {$1=tolower($3) OFS $1; print}' > firstWithLow.csv
# remove match line by first column
awk -F"," 'NR==FNR{a[$1]++;next} !(a[$1])' exclude_list.csv master_file.csv > outfile.csv
# remove first column
cat outfile.csv | cut -d"," -f2- > outfileclean.csv
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment