Skip to content

Instantly share code, notes, and snippets.

@penk
Last active August 15, 2018 13:29
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save penk/6265528 to your computer and use it in GitHub Desktop.
Save penk/6265528 to your computer and use it in GitHub Desktop.
Parallel crawler example using PhantomJS and Aysnc.js
// usage: phantomjs async-phantom.js
var async = require('async'),
links = [
"http://google.com",
"http://yahoo.com",
"http://duckduckgo.com",
"http://bing.com",
];
function crawler(url, callback) {
var page = require('webpage').create();
page.open(url, function (status) {
console.log( page.evaluate(function(){ return document.title }) );
page.close();
callback.apply();
});
}
async.each(links, crawler, function (err) {
if (err) console.log(err);
phantom.exit();
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment