danielrobertson/RemoveDuplicates.py

Created March 19, 2015 17:48

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/danielrobertson/8d92a8a547dcc5d59155.js"></script>
Save danielrobertson/8d92a8a547dcc5d59155 to your computer and use it in GitHub Desktop.

Download ZIP

Remove duplicates in CSV

Raw

RemoveDuplicates.py

	import pandas as pd
	toclean = pd.read_csv('fileWithDuplicates.csv')
	deduped = toclean.drop_duplicates('columnName')
	deduped.to_csv('fileWithoutDuplicates.csv')

Author

danielrobertson commented Mar 19, 2015

For xlsx files, use ssconvert to convert them to csv's first:
bash ssconvert myFile.xlsx myNewFile.csv

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment