Skip to content

Instantly share code, notes, and snippets.

@oodavid
Last active September 19, 2016 11:22
Show Gist options
  • Save oodavid/5eb9755461079eee5fe0c054b08fd999 to your computer and use it in GitHub Desktop.
Save oodavid/5eb9755461079eee5fe0c054b08fd999 to your computer and use it in GitHub Desktop.
Scraper Example
{
"name": "scraper",
"version": "1.0.0",
"description": "Scraper",
"main": "scraper.js",
"dependencies": {
"node-horseman": "^3.1.1",
"random-useragent": "^0.3.0"
}
}
'use strict';
var Horseman = require('node-horseman');
var random_useragent = require('random-useragent');
var horseman = new Horseman({
injectJquery: true,
timeout: 10000,
loadImages: false,
// proxy: '36.250.186.100:8888'
// proxyType: ...
// proxyAuth: ...
});
var url = 'http://oodavid.com/';
horseman
.userAgent(random_useragent.getRandom())
.open(url)
.html()
.then(function(body) {
console.log(body);
return horseman.close();
});
@oodavid
Copy link
Author

oodavid commented Sep 15, 2016

npm install
then
node scraper.js

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment