Chingis cdgz
cdgz
/ crawl.py
Created
October 25, 2011 15:14
— forked from jonhurlock/crawl.py
Python Web Crawler - jonhurlock
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python | |
| """ | |
| Simple Indexer | |
| ================================= | |
| Author: Jon Hurlock, October 2011 | |
| This script basically crawls a domain (not just a page) and | |
| then extracts all links <a href=""></a>, and finds all links | |
| on that domain it also is able extract different file types |