Skip to content

Instantly share code, notes, and snippets.

@igaray

igaray/deduplicate.py

Created February 28, 2017 15:48
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save igaray/2579f1863b3d0fac1afacefbad72327a to your computer and use it in GitHub Desktop.
Save igaray/2579f1863b3d0fac1afacefbad72327a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
deduplicate.py
with open("LINKS", "r") as linksfile:
with open("LINKS2", "r") as duplicatefile:
with open("LINKS3", "w") as dedpulicatedfile:
links = frozenset(linksfile)
for line in duplicatefile.readlines():
if line not in links:
dedpulicatedfile.write(line)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment