Skip to content

Instantly share code, notes, and snippets.

@elog08
elog08 / reasearch-proxy.md
Created Jul 15, 2018
Research: Proxy w/ HTTP Auth Upstream
View reasearch-proxy.md

Proxy w/ HTTP Auth Upstream

Research: Find Proxy Servers that support upstream forwarding to HTTP proxies with authentication.

Problems

  1. I am trying to use paid proxy X but it requires either IP-based or HTTP Proxy-Auth based authentication, I want a proxy with no auth requirements that works well with browsers.

Projects Explored

Glider

@elog08
elog08 / scrape.js
Created Apr 17, 2018
Scrape with Puppeteer
View scrape.js
const puppeteer = require('puppeteer')
const script = require('./script');
const { writeFileSync } = require("fs");
function save(raw) {
writeFileSync('results.json', JSON.stringify(raw));
}
const URL = 'https://www.quora.com/search?q=meaning%20of%20life&type=answer';
@elog08
elog08 / script.js
Created Apr 17, 2018
ScrapeInfiniteList.js
View script.js
module.exports = function() {
return new Promise((resolve, reject) => {
// Class for Individual Thread
const C_THREAD = '.pagedlist_item:not(.pagedlist_hidden)';
// Class for threads marked for deletion on subsequent loop
const C_THREAD_TO_REMOVE = '.pagedlist_item:not(.pagedlist_hidden) .TO_REMOVE';
// Class for Title
const C_THREAD_TITLE = '.title';
// Class for Description
@elog08
elog08 / asyncRecurse.js
Created Jan 31, 2017
A recursive function to process nested JSON objects asynchronously
View asyncRecurse.js
/*
* Modifies the values of a JSON object recursively and asynchronously.
*/
// Wrapper for async.Map that persists the keys of the object
// https://github.com/caolan/async/issues/374#issuecomment-59191322
async.objectMap = async.objectMap || function(obj, func, cb) {
var i, arr = [],
keys = Object.keys(obj);
for (i = 0; i < keys.length; i += 1) {
You can’t perform that action at this time.