Skip to content

Instantly share code, notes, and snippets.

@ncole458

ncole458/Python remove duplicates from CSV

Last active February 26, 2016 00:28
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ncole458/883a03092c4f613ed329 to your computer and use it in GitHub Desktop.
Save ncole458/883a03092c4f613ed329 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Python remove duplicates from CSV
import csv
reader=csv.reader(open('UserstoImport.csv', 'r'), delimiter=',')
writer=csv.writer(open('UserstoImport_withoutduplicates.csv', 'w'), delimiter=',')
entries = set()
for row in reader:
key = (row[1], row[2]) # CSV rows by num
if key not in entries:
writer.writerow(row)
entries.add(key)
@TylerNakamura
Copy link

awesome, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment