Skip to content

Instantly share code, notes, and snippets.

@mturilin
Created March 11, 2021 00:34
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mturilin/a8d4865850496546ec86c97e5a3eb66b to your computer and use it in GitHub Desktop.
Save mturilin/a8d4865850496546ec86c97e5a3eb66b to your computer and use it in GitHub Desktop.
CVS dedupe
import csv
uniques = {}
with open('original.csv') as csv_file:
reader = csv.DictReader(csv_file)
for row in reader:
if row['Password'] == '':
if row['Group'] == 'Root':
row['Group'] = 'Root/Non_Login'
key = ("Other", row['Title'])
else:
key = ("Login", row['Title'], row['Username'])
if key in uniques:
print("Dup!", key)
else:
uniques[key] = row
with open('no_dupes.csv', "w") as csv_export:
writer = csv.DictWriter(csv_export, fieldnames=reader.fieldnames)
writer.writeheader()
for row in uniques.values():
writer.writerow(row)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment