digitalstem

## crawler.md

      
        
          
            
              
              3 files
            
          
          
            
              
              0 forks
            
          
          
            
              
              0 comments
            
          
          
            
              
              0 stars
            
          
        
        
          
              
          
          
            
                digitalstem
                / crawler.md
            
            
              Created
              March 24, 2018 08:59
                — forked from typehorror/crawler.md
            
              
                Simple Website Crawler (in python)
              
          
        
      
        
  
      
    Simple Website Crawler

The following gist is an extract of the article Building a simple crawler. It allows crawling from a URL and for a given number of bounce.
Basic Usage

from crawler import Crawler
crawler = Crawler()
crawler.crawl('http://techcrunch.com/')

displays the urls