Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
basic scrape website
const scrape = require('website-scraper');
const websiteUrl = 'Website url you want to scrap';
scrape({
urls: [websiteUrl],
urlFilter: function (url) {
return url.indexOf(websiteUrl) === 0;
},
recursive: true,
maxDepth: 50,
prettifyUrls: true,
filenameGenerator: 'bySiteStructure',
directory: './website'
}).then((data) => {
console.log("Entire website succesfully downloaded");
}).catch((err) => {
console.log("An error ocurred", err);
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment