lukekarrys

Node Crawler to find all domain links on a site and run a function on them

Linked to from http://lukecod.es/2012/11/18/random-problem-of-the-night/

What

This is a node.js crawler that will crawl an entire site (using crawl) to find all internal links in the entire site. It will then test each unique internal link for the presence of an optional string and then the query string into an object. All values with the same key from the query string will be pushed to an array for that key.

Usage

ritch

I'm getting tired of writing documents to then parse and rebuild in JavaScript. HTML is not a markup language for complex interactive applications. rml is. Chekk-it.

// html
<img src="http://google.com/logo.png" />

Same thing in simple rml.

// rml
img('http://google.com/logo.png');