Skip to content

Instantly share code, notes, and snippets.

@jancurn
Created September 24, 2018 14:03
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save jancurn/3cc2f1c4c61ab2c99315d44147cfc071 to your computer and use it in GitHub Desktop.
Save jancurn/3cc2f1c4c61ab2c99315d44147cfc071 to your computer and use it in GitHub Desktop.
Apify SDK hello world example
const Apify = require('apify');
Apify.main(async () => {
const requestQueue = await Apify.openRequestQueue();
await requestQueue.addRequest(new Apify.Request({ url: 'https://www.iana.org/' }));
const pseudoUrls = [new Apify.PseudoUrl('https://www.iana.org/[.*]')];
const crawler = new Apify.PuppeteerCrawler({
requestQueue,
handlePageFunction: async ({ request, page }) => {
const title = await page.title();
console.log(`Title of ${request.url}: ${title}`);
await Apify.utils.puppeteer.enqueueLinks(page, 'a', pseudoUrls, requestQueue);
},
maxRequestsPerCrawl: 100,
maxConcurrency: 10,
});
await crawler.run();
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment