Skip to content

Instantly share code, notes, and snippets.

jolynch /
Last active December 10, 2023 08:05
Upload a directory to S3 as fast as humanly possible

Upload files to S3 as fast as Possible

Often I hear people claiming S3 is slow, when in fact S3 is very fast but the user space program you are using is slow!

This gist provides a tool: which will upload the directory you run the tool in to a provided S3 bucket, as fast as humanly possible. This tool will saturate your NICs, will saturate your CPUS, and will upload data to S3 faster than basically anything else. s3://some-path/in-s3
jolynch /
Created August 18, 2017 18:31
Making Pretty Graphs
#!/usr/bin/env bash
# Utility for compiling gnuplot scripts
# to pdf and png images containing latex text
# requires: awk, latex, dvips, convert, gnuplot
# convert is a part of "Imagemagick" suite for image manipulation
# usage: figlatex (format)
# -*- coding: utf-8 -*-
import logging
import gevent
import gevent.event
import time
logger = logging.getLogger(__name__)
require 'synapse/config_generator/base'
require 'fileutils'
class Synapse::ConfigGenerator
class Nginx < BaseGenerator
include Synapse::Logging
NAME = 'nginx'.freeze
jolynch /
Created January 25, 2016 23:00
A RLock implementation with a timeout.
import threading
import time
class TRLock(threading._RLock):
"""Implemention of RLock with a timeout
Works around
def __init__(self, default_timeout_s=None, *args, **kwargs):
super(TRLock, self).__init__(*args, **kwargs)
self.default_timeout_s = default_timeout_s
jolynch / Dockerfile
Created January 18, 2016 21:15
An example for how to run Synapse in Docker (not encouraged)
FROM ubuntu:trusty
# Prep
RUN echo "deb trusty main restricted" > /etc/apt/sources.list
RUN echo "deb-src trusty main restricted" >> /etc/apt/sources.list
RUN echo "deb trusty-updates main restricted" >> /etc/apt/sources.list
RUN echo "deb-src trusty-updates main restricted" >> /etc/apt/sources.list
RUN echo "deb trusty universe" >> /etc/apt/sources.list
RUN echo "deb trusty-updates universe" >> /etc/apt/sources.list
RUN echo "deb trusty-backports main restricted universe multiverse" >> /etc/apt/sources.list
jolynch /
Last active August 29, 2015 14:18
Hacky script to convert an apache benchmark data file into a CDF
# Usage
# ./ ab_output.dat ab_output_cdf.dat
import csv
import sys
times = []
just_times = []
infile = sys.argv[1]
outfile = sys.argv[2]
jolynch /
Created April 7, 2015 20:43
Shows how to manipulate a plug qdisc manually using pyroute2
import pyroute2
from pyroute2 import IPRoute
from pyroute2.iproute import transform_handle
from pyroute2.netlink import NLM_F_ACK
from pyroute2.netlink import NLM_F_REQUEST
from pyroute2.netlink.rtnl.tcmsg import tcmsg
PLUG_CLASS = '1:4'
PLUG_QDISC = '40:'