Skip to content

Instantly share code, notes, and snippets.

@lukasmartinelli

lukasmartinelli/unique_jobs.py

Created May 11, 2016 14:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lukasmartinelli/e0d89507234cbd30ec7253d7001cc05c to your computer and use it in GitHub Desktop.
Save lukasmartinelli/e0d89507234cbd30ec7253d7001cc05c to your computer and use it in GitHub Desktop.
Download ZIP
Filter JSON lines jobs by unique id
Raw
unique_jobs.py
import json
ids = {}
for line in open('failed-jobs.json'):
doc = json.loads(line)
if not doc['id'] in ids:
ids[doc['id']] = True
print(json.dumps(doc))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment