Skip to content

Instantly share code, notes, and snippets.

👨‍💻
Probably behind computer

Jan Čurn jancurn

👨‍💻
Probably behind computer
Block or report user

Report or block jancurn

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@jancurn
jancurn / spider.py
Created Sep 4, 2019
Scrapy Executor code example
View spider.py
import scrapy
import apify
class MySpider(scrapy.Spider):
name = 'apifySpider'
def start_requests(self):
urls = [
'https://apify.com',
'https://apify.com/store',
@jancurn
jancurn / hello_world.js
Created Sep 24, 2018
Apify SDK hello world example
View hello_world.js
const Apify = require('apify');
Apify.main(async () => {
const requestQueue = await Apify.openRequestQueue();
await requestQueue.addRequest(new Apify.Request({ url: 'https://www.iana.org/' }));
const pseudoUrls = [new Apify.PseudoUrl('https://www.iana.org/[.*]')];
const crawler = new Apify.PuppeteerCrawler({
requestQueue,
handlePageFunction: async ({ request, page }) => {
@jancurn
jancurn / example.js
Created Aug 15, 2018
Apify actor that demonstrates usage of Puppeteer live view
View example.js
const Apify = require('apify');
Apify.main(async () => {
const browser = await Apify.launchPuppeteer({ liveView: true });
const page = await browser.newPage();
await page.goto('https://news.ycombinator.com/');
const pageTitle = await page.title();
console.log(`Page loaded: ${pageTitle}`);
@jancurn
jancurn / Dockerfile
Last active Mar 5, 2019
Example of an Apify actor stored in a GitHub Gist.
View Dockerfile
# Here you choose the base Docker image for the actor. Apify provides the following images:
# apify/actor-node-basic
# apify/actor-node-chrome
# apify/actor-node-puppeteer
# However, you can use any other image from Docker Hub.
# For more information, see https://apify.com/docs/actor#base-images
FROM apify/actor-node-basic
# Copy all files and directories from the directory to the Docker image
COPY . ./
@jancurn
jancurn / puppeteer-proxy-page-authenticate.js
Last active Nov 9, 2019
Puppeteer's page.authenticate() does not work for proxy authorization!
View puppeteer-proxy-page-authenticate.js
const puppeteer = require('puppeteer');
(async() => {
const proxyUrl = 'http://proxy.example.com:8000';
const username = 'bob';
const password = 'password123';
const browser = await puppeteer.launch({
args: [`--proxy-server=${proxyUrl}`],
headless: false,
@jancurn
jancurn / proxy-chain-example.js
Last active Jun 26, 2019
Example showing how to use the proxy-chain NPM package to let headless Chrome use a proxy server with username and password
View proxy-chain-example.js
const puppeteer = require('puppeteer');
const proxyChain = require('proxy-chain');
(async() => {
const oldProxyUrl = 'http://bob:password123@proxy.example.com:8000';
const newProxyUrl = await proxyChain.anonymizeProxy(oldProxyUrl);
// Prints something like "http://127.0.0.1:45678"
console.log(newProxyUrl);
View hello-world.js
const Apify = require('apify');
Apify.main(async () => {
// Get input of your act and print it
const input = await Apify.getValue('INPUT');
console.log('My input:');
console.dir(input);
// Save the output
const output = {
You can’t perform that action at this time.