Sebastian Nagel sebastian-nagel
sebastian-nagel
/ common-crawl-cdx.py
Last active
May 16, 2016 13:20
— forked from tfmorris/common-crawl-cdx.py
Analyze Common Crawl index - http://index.commoncrawl.org/
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # -*- coding: utf-8 -*- | |
| """ | |
| common-crawl-cdx.py | |
| A simple example program to analyze the Common Crawl index. | |
| This is implemented as a single stream job which accesses S3 via HTTP, | |
| so that it can be easily be run from any laptop, but it could easily be | |
| converted to an EMR job which processed the 300 index files in parallel. |