Skip to content

Instantly share code, notes, and snippets.

View ns-mkusper's full-sized avatar
👋

Mark Kusper ns-mkusper

👋
  • Chicago
View GitHub Profile
@juanpampliega
juanpampliega / gist:f7b68c3546d921154ac9eaabf06a8911
Created June 2, 2018 21:46
Install OpenX Hive JSON SerDe in Amazon EMR to use it with Presto
# Do this on every node of the cluster
curl -O http://www.congiu.net/hive-json-serde/1.3.8/hdp23/json-serde-1.3.8-jar-with-dependencies.jar
sudo cp json-serde-1.3.8-jar-with-dependencies.jar /usr/lib/presto/plugin/hive-hadoop2/
sudo chown presto:presto /usr/lib/presto/plugin/hive-hadoop2/json-serde-1.3.8-jar-with-dependencies.jar
#restart presto
sudo restart presto-server
@fasiha
fasiha / csv2sql.sh
Last active July 30, 2021 09:20
Dump the contents of an Anki collection.anki2 SQLite database to a bunch of CSV files (one for each table)
#!/bin/bash
# USAGE: csv2sql.sh NO_HEADER.csv TABLE_NAME
# Delete all rows from TABLE_NAME and import NO_HEADER.csv into it.
sqlite3 collection.anki2 <<EOF
DELETE FROM $2;
.mode csv
.import $1 $2
EOF
@nezihyigitbasi
nezihyigitbasi / presto-rest.md
Last active September 14, 2021 11:54
Presto REST Services

/v1/node
/v1/node/failed
/v1/execute
/v1/statement
/v1/query
/v1/query/{query_id}
/v1/thread
/ui/thread
/v1/task
/v1/task/{task_id}

@robinkraft
robinkraft / s3bucketsize.py
Last active October 24, 2021 11:43
Simple python script to calculate size of S3 buckets
import sys
import boto
# based on http://www.quora.com/Amazon-S3/What-is-the-fastest-way-to-measure-the-total-size-of-an-S3-bucket
# assumes you've already configured your access id & secret key
s3 = boto.connect_s3()
@t3rmin4t0r
t3rmin4t0r / tez2graph.py
Last active April 22, 2022 02:28
Convert Hive Tez explains into images for debugging
import re, sys
NX = True
try:
import networkx as nx
except:
NX = False
sys.stderr.write("Could not import nx\npip install networkx, please\n")
plan39 = """
Map 1 <- Map 5 (BROADCAST_EDGE), Map 6 (BROADCAST_EDGE), Map 8 (BROADCAST_EDGE), Reducer 7 (BROADCAST_EDGE), Reducer 9 (BROADCAST_EDGE)
@jespada
jespada / compile-pgloader-ccl.sh
Created May 18, 2017 10:53
pgloader-ccl-compile
# run as root
git clone git@github.com:dimitri/pgloader.git
cd pgloader
mkdir -p /opt/src/pgloader
cp -R * /opt/src/pgloader
apt-get update
apt-get install -y wget curl make git bzip2 time libzip-dev libssl1.0.0 openssl
apt-get install -y patch unzip libsqlite3-dev gawk freetds-dev subversion
@tiagoengel
tiagoengel / noise-cancellation.sh
Last active May 12, 2023 20:41
Hiss / White / Static noise cancellation on Linux using Pulseaudio and Sox
#!/bin/bash
# You'll need to have sox, pavucontrol and alsa-utils installed, and the snd_aloop kernel module loaded.
# You can configure your system to load it on startup or load it manually with "sudo modprobe snd_aloop"
# Once this is script is running, you need to start recording audio in the application of your
# preference, open pavucontrol, go to the recording tab and change the recording source of that application
# to "Monitor of Loopback ..."
time=5
@justinhj
justinhj / git-add-dired.el
Created July 7, 2013 21:28
Allows you to use dired to add multiple files to a git repository. This is useful since staging files in magit is sluggish and takes a long time when you have a lot of files.
(defun git-add-files(files)
"Run git add with the input file"
(interactive)
(shell-command (format "git add %s" files)))
(defun dired-git-add-marked-files()
"For each marked file in a dired buffer add it to the index"
(interactive)
(if (eq major-mode 'dired-mode)
(let ((filenames (dired-get-marked-files))
@skylock
skylock / ReadMe.md
Last active September 11, 2023 13:51 — forked from devinrhode2/README.md
How to Change Open Files Limit on OS X and macOS Sierra (10.8 - 10.12)

How to Change Open Files Limit on OS X and macOS

To check the current limits on your Mac OS X system, run in terminal:

launchctl limit maxfiles
ulimit -a

Steps