Skip to content

Instantly share code, notes, and snippets.

Avatar

artforlife

  • Washington, DC
View GitHub Profile
@artforlife
artforlife / crawler.md
Created Jul 23, 2017 — forked from debrice/crawler.md
Simple Website Crawler (in python)
View crawler.md

Simple Website Crawler

The following gist is an extract of the article Building a simple crawler. It allows crawling from a URL and for a given number of bounce.

Basic Usage

from crawler import Crawler
crawler = Crawler()
crawler.crawl('http://techcrunch.com/')
# displays the urls
You can’t perform that action at this time.