The following gist is an extract of the article Building a simple crawler. It allows crawling from a URL and for a given number of bounce.
from crawler import Crawler
crawler = Crawler()
crawler.crawl('http://techcrunch.com/')
2013-07-01 | 16650 | |
---|---|---|
2013-07-02 | 22745 | |
2013-07-03 | 21864 | |
2013-07-04 | 22326 | |
2013-07-05 | 21842 | |
2013-07-06 | 20467 | |
2013-07-07 | 20477 | |
2013-07-08 | 21615 | |
2013-07-09 | 26641 | |
2013-07-10 | 25732 |
The following gist is an extract of the article Building a simple crawler. It allows crawling from a URL and for a given number of bounce.
from crawler import Crawler
crawler = Crawler()
crawler.crawl('http://techcrunch.com/')