Skip to content

Instantly share code, notes, and snippets.

Michael Keller mhkeller

  • csv soundsytem, NY Times
  • New York, NY
Block or report user

Report or block mhkeller

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
mhkeller / count-string-occurrences.sql
Created Oct 15, 2019
count occurrences of a string across rows of a pgsql table
View count-string-occurrences.sql
--useful for seeing if a css class exists across all instances of scraped pages in a pgsql database
--adapted from here
select sum(array_length(string_to_array(html, 'SELECTOR'), 1) - 1) from TABLE_NAME
--should equal rows in the table
--or this to get occurrences per row
select (array_length(string_to_array(html, 'SELECTOR'), 1) - 1) as ct from TABLE_NAME ORDER BY ct DESC
mhkeller / throttle.js
Created Apr 17, 2019
Throttle requests
View throttle.js
const sleep = ms => new Promise(f => setTimeout(f, ms));
async function do_things() {
for (const thing of things) {
await Promise.all([
mhkeller / gist:01f408c1a4e11e0577eb052e35f526bf
Created Mar 23, 2019
use python 2.7 with a node install for node-gyp
View gist:01f408c1a4e11e0577eb052e35f526bf
`npm install --python=python2.7` and `npm config set python python2.7`
mhkeller / deps.R
Created Mar 20, 2019
Loading some sweet R packages
View deps.R
pkgs <- c("packages")
check <- sapply(pkgs, require, warn.conflicts = TRUE, character.only = TRUE)
if(any(!check)) {
pkgs.missing <- pkgs[!check]
check <- sapply(pkgs.missing, require, warn.conflicts = TRUE, character.only = TRUE)
rm(pkgs, check)
mhkeller / index.js
Last active May 1, 2019
Basic puppeteer setup
View index.js
const puppeteer = require('puppeteer');
async function main () {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto(``, { waitUntil: 'load' });
mhkeller / pg-connect.js
Created Feb 26, 2019
boilerplate connect to database
View pg-connect.js
const { Pool } = require('pg');
const connectionString = require('./connection-string.json');
const pool = new Pool(connectionString);
async function main () {
const x = await pool.query('Select count(*) from my_table');
mhkeller / wget.js
Created Feb 6, 2019
promisified wget
View wget.js
async function downloadPage (matches) {
for (let match of matches) {
const requestUrl = `tktk`;
const html = await wget(requestUrl);
async function wget (requestUrl) {
return new Promise((resolve, reject) => {
const cmd = `wget -q -O - '${requestUrl}'`;
mhkeller /
Created Dec 17, 2018
Ways to get in touch
View query-to-csv.psql
\copy (SELECT * FROM persons) to 'C:\tmp\persons_client.csv' with (format csv, header)
You can’t perform that action at this time.