Skip to content

Instantly share code, notes, and snippets.

View justincsi's full-sized avatar

justincsi

  • Clear Science, Inc.
View GitHub Profile
@justincsi
justincsi / crawler.py
Created August 11, 2016 13:28 — forked from jmoiron/crawler.py
Simple gevent/httplib2 web crawler.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Simple async crawler/callback queue based on gevent."""
import traceback
import logging
import httplib2
import gevent