Skip to content

Instantly share code, notes, and snippets.

@nonamenix
nonamenix / crawler.py
Created January 22, 2017 13:01 — forked from jmoiron/crawler.py
Simple gevent/httplib2 web crawler.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Simple async crawler/callback queue based on gevent."""
import traceback
import logging
import httplib2
import gevent