Skip to content

Instantly share code, notes, and snippets.

@jolynch
jolynch / S3_fast_upload.md
Last active Sep 27, 2018
Upload a directory to S3 as fast as humanly possible
View S3_fast_upload.md

Upload files to S3 as fast as Possible

Often I hear people claiming S3 is slow, when in fact S3 is very fast but the user space program you are using is slow!

This gist provides a tool: rapid_s3_upload.sh which will upload the directory you run the tool in to a provided S3 bucket, as fast as humanly possible. This tool will saturate your NICs, will saturate your CPUS, and will upload data to S3 faster than basically anything else.

rapid_s3_upload.sh s3://some-path/in-s3
@jolynch
jolynch / figlatex.sh
Created Aug 18, 2017
Making Pretty Graphs
View figlatex.sh
#!/usr/bin/env bash
# Utility for compiling gnuplot scripts
# to pdf and png images containing latex text
#
# requires: awk, latex, dvips, convert, gnuplot
# convert is a part of "Imagemagick" suite for image manipulation
#
# usage: figlatex script.gp (format)
#
View speculate.py
# -*- coding: utf-8 -*-
import logging
import gevent
import gevent.event
import time
logger = logging.getLogger(__name__)
View nginx.rb
require 'synapse/config_generator/base'
require 'fileutils'
class Synapse::ConfigGenerator
class Nginx < BaseGenerator
include Synapse::Logging
NAME = 'nginx'.freeze
@jolynch
jolynch / TRLock.py
Created Jan 25, 2016
A RLock implementation with a timeout.
View TRLock.py
import threading
import time
class TRLock(threading._RLock):
"""Implemention of RLock with a timeout
Works around http://bugs.python.org/issue13697
"""
def __init__(self, default_timeout_s=None, *args, **kwargs):
super(TRLock, self).__init__(*args, **kwargs)
self.default_timeout_s = default_timeout_s
@jolynch
jolynch / Dockerfile
Created Jan 18, 2016
An example for how to run Synapse in Docker (not encouraged)
View Dockerfile
FROM ubuntu:trusty
# Prep
RUN echo "deb http://archive.ubuntu.com/ubuntu/ trusty main restricted" > /etc/apt/sources.list
RUN echo "deb-src http://archive.ubuntu.com/ubuntu/ trusty main restricted" >> /etc/apt/sources.list
RUN echo "deb http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted" >> /etc/apt/sources.list
RUN echo "deb-src http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted" >> /etc/apt/sources.list
RUN echo "deb http://archive.ubuntu.com/ubuntu/ trusty universe" >> /etc/apt/sources.list
RUN echo "deb http://archive.ubuntu.com/ubuntu/ trusty-updates universe" >> /etc/apt/sources.list
RUN echo "deb http://archive.ubuntu.com/ubuntu trusty-backports main restricted universe multiverse" >> /etc/apt/sources.list
@jolynch
jolynch / convert_to_cdf.py
Last active Aug 29, 2015
Hacky script to convert an apache benchmark data file into a CDF
View convert_to_cdf.py
#!/usr/bin/python
# Usage
# ./convert_to_cdf.py ab_output.dat ab_output_cdf.dat
import csv
import sys
times = []
just_times = []
infile = sys.argv[1]
outfile = sys.argv[2]
@jolynch
jolynch / manipulate_qdisc.py
Created Apr 7, 2015
Shows how to manipulate a plug qdisc manually using pyroute2
View manipulate_qdisc.py
import pyroute2
from pyroute2 import IPRoute
from pyroute2.iproute import transform_handle
from pyroute2.netlink import NLM_F_ACK
from pyroute2.netlink import NLM_F_REQUEST
from pyroute2.netlink.rtnl.tcmsg import tcmsg
PLUG_CLASS = '1:4'
PLUG_QDISC = '40:'