Skip to content

Instantly share code, notes, and snippets.

artforlife

  • Washington, DC
Block or report user

Report or block artforlife

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@artforlife
artforlife / crawler.md
Created Jul 23, 2017 — forked from debrice/crawler.md
Simple Website Crawler (in python)
View crawler.md

Simple Website Crawler

The following gist is an extract of the article Building a simple crawler. It allows crawling from a URL and for a given number of bounce.

Basic Usage

from crawler import Crawler
crawler = Crawler()
crawler.crawl('http://techcrunch.com/')
# displays the urls
You can’t perform that action at this time.