Skip to content

Instantly share code, notes, and snippets.

View drj42's full-sized avatar

Will Jones drj42

6 followers · 2 following
View GitHub Profile
@drj42
drj42 / org-mode-reference-in.org
Created February 6, 2012 23:53
This is a cheat sheet for Emacs org-mode... in org-mode format!
@drj42
drj42 / luigi_sftp.py
Last active January 18, 2022 17:47
sftp remote target for luigi
"""sftp.py - PySftp connections wrapped up in a luigi.Target.
TODO: get rid of the redundant stuff, write some tests, contribute to luigi
upstream.
"""
# -*- coding: utf-8 -*-
import io
import os
import random
@drj42
drj42 / types_and_roles_demo.sql
Created September 5, 2018 22:24 — forked from levlaz/types_and_roles_demo.sql
Create Types and Roles If Not Exist in PostgreSQL
BEGIN;
DO $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'task_status') THEN
create type task_status AS ENUM ('todo', 'doing', 'blocked', 'done');
END IF;
END
$$;
CREATE TABLE IF NOT EXISTS
@drj42
drj42 / postgres_queries_and_commands.sql
Last active August 29, 2018 20:17 — forked from rgreenjr/postgres_queries_and_commands.sql
Useful PostgreSQL Queries and Commands
-- long running queries
SELECT
pid,
now() - pg_stat_activity.query_start AS duration,
query,
state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes'
-- show running queries (9.2)
@drj42
drj42 / kube-registry.yaml
Created November 22, 2017 19:00 — forked from coco98/kube-registry.yaml
Docker registry on minikube
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-registry-v0
namespace: kube-system
labels:
k8s-app: kube-registry
version: v0
spec:
replicas: 1
@drj42
drj42 / async_worker_pool.py
Created January 30, 2017 21:06 — forked from thehesiod/async_worker_pool.py
Asynchronous Worker Pool
import asyncio
from datetime import datetime, timezone
import os
def utc_now():
# utcnow returns a naive datetime, so we have to set the timezone manually <sigh>
return datetime.utcnow().replace(tzinfo=timezone.utc)
class Terminator:
pass
@drj42
drj42 / asyncio_producer_consumer.py
Created January 30, 2017 19:47 — forked from akrylysov/asyncio_producer_consumer.py
Python 3 asyncio basic producer / consumer example
import asyncio
import random
q = asyncio.Queue()
async def producer(num):
while True:
await q.put(num + random.random())
await asyncio.sleep(random.random())
@drj42
drj42 / s3.py
Created January 27, 2017 18:56 — forked from stalkerg/s3.py
Async Tornado S3 uploader with AWS4 sign
import hashlib
import hmac
import mimetypes
import binascii
from calendar import timegm
from datetime import datetime
import time
from email.utils import formatdate
from urllib.parse import quote, urlparse
@drj42
drj42 / df_reduce_by_key.py
Created March 15, 2016 18:45
PySpark Helper Function - perform reduceByKey on a dataframe
# Removes a lot of the boiler plate involved in converting a pyspark dataframe
# to and from an rdd, in order to do a reduceByKey operation.
#
# Lifted from:
# - http://codereview.stackexchange.com/questions/115082/generic-reduceby-or-groupby-aggregate-functionality-with-spark-dataframe
from pyspark.sql import Row
from pyspark.sql.functions import struct
from pyspark.sql import DataFrame
@drj42
drj42 / gist:7221275
Created October 29, 2013 19:44
Mojolicious ingress cemetery portal name generator
#!/usr/bin/env perl
use Mojolicious::Lite;
# Instructions:
# $ cpanm Mojolicious
# $ morbo grave_names.pl
#
# browse to http://localhost:3000
# Documentation browser under "/perldoc"