Skip to content

Instantly share code, notes, and snippets.

View mikemccabe's full-sized avatar

Mike McCabe mikemccabe

  • San Francisco, CA
View GitHub Profile
@mikemccabe
mikemccabe / parallel_md_get.py
Created September 25, 2012 22:28
Parallel archive.org metadata fetching using python and gevent
# This demonstrates doing multiple metadata fetches in parallel.
# It seems to be fast enough that the json decoding cost becomes
# a significant proportion of the execution time.
# It requires gevent; see http://www.gevent.org/intro.html#installation
# This is callable from the command line; call with --help for a summary.
# If you use it is a library, the main entry point is
# metadata_record_iterator(); see main() for an example.
import json
import urllib
import collections
def get_iterable(x):
return (x,) if not isinstance(x, (tuple, list)) else x
# if isinstance(x, collections.Iterable):
# return x
# else:
# return (x,)