Skip to content

Instantly share code, notes, and snippets.

View kgriffs's full-sized avatar

Kurt Griffiths kgriffs

View GitHub Profile
@kgriffs
kgriffs / example.py
Created August 26, 2015 18:37
Python: Wrap an error
raise MyError, e.args, sys.exc_info()[2]
@kgriffs
kgriffs / gist:8695688
Created January 29, 2014 19:56
Double a value per iteration
N = 5
V = 0.001
for i in range(N):
print V * (2 ** i)
@kgriffs
kgriffs / gist:8831610
Last active August 29, 2015 13:56
conflict or complimentary?

Appendix D: Conflict or Complimentary?

Web Frameworks

  • Django — Flask
  • Rails — Sinatra
  • Hapi — Express
  • Pecan — Falcon

Libraries

@kgriffs
kgriffs / thread_local.py
Last active August 29, 2015 13:56
Thread local example
# solom/__init__.py
THREAD_LOCAL = threading.local()
# In some pecan helper module...
def before_hook(self):
# Use thread-local storage so we don't have to
# add two extra params to virtuall every single
# function in our app!
solum.THREAD_LOCAL.ctx = context.RequestContext(...)
@kgriffs
kgriffs / producer.py
Last active August 29, 2015 14:01
Producer POC for benchmarking Marconi
from __future__ import division
import multiprocessing as mp
import argparse
import random
import time
from gevent import monkey as curious_george
curious_george.patch_all(thread=False, select=False)
@kgriffs
kgriffs / scaling-queues.md
Last active August 29, 2015 14:05
Notes on Scaling Queues

Using timestamps as IDs

Helps scale writes:

  • For a single master, means don't have to query to find last inserted ID.
  • Collisions can be practically impossible
  • Avoids having to retry using an optimistic algorithm, in the case that a parallel request beat me to the punch and I have a collision of IDs.

Caveats:

  • Timestamp must be granular enough to be unique for expected max message insertion rate per queue. For example, given a goal of 100,000 writes/sec to a specific queue, to avoid collisions the timestamps would need to be stored with at least 100 ms granularity. However, one must also allow for outliers, where occasionally the stars will align and several requests will arrive and request timestamps at the same moment. [wWth that in mind, what is a the practical granularity required? microseconds? 100-nanoseconds?]
  • Requires specialized hardware to avoid significant drift. However, NTP may be "good enough" if ~1 ms drift is acceptable.
░░░░░░░░░▄░░░░░░░░░░░░░░▄░░░░
░░░░░░░░▌▒█░░░░░░░░░░░▄▀▒▌░░░
░░░░░░░░▌▒▒█░░░░░░░░▄▀▒▒▒▐░░░
░░░░░░░▐▄▀▒▒▀▀▀▀▄▄▄▀▒▒▒▒▒▐░░░
░░░░░▄▄▀▒░▒▒▒▒▒▒▒▒▒█▒▒▄█▒▐░░░
░░░▄▀▒▒▒░░░▒▒▒░░░▒▒▒▀██▀▒▌░░░
░░▐▒▒▒▄▄▒▒▒▒░░░▒▒▒▒▒▒▒▀▄▒▒▌░░
░░▌░░▌█▀▒▒▒▒▒▄▀█▄▒▒▒▒▒▒▒█▒▐░░
░▐░░░▒▒▒▒▒▒▒▒▌██▀▒▒░░░▒▒▒▀▄▌░
░▌░▒▄██▄▒▒▒▒▒▒▒▒▒░░░░░░▒▒▒▒▌░
@kgriffs
kgriffs / gist:d3f08afc39f1f4311aa0
Created September 23, 2014 21:20
JSON pretty-printing on the CLI with Mac OS X
$ pbpaste | python -m json.tool | pbcopy
@kgriffs
kgriffs / gist:fc6c726fb8b432b66dde
Last active August 29, 2015 14:07
Parse a host header
def parse_host(host, default_port=None):
# NOTE(kgriff): The value from the Host header may
# contain a port, so check that and strip it if
# necessary. This is complicated by the fact that
# a hostname may be specified either as an IP address
# or as a domain name, and in the case of IPv6 there
# may be multiple colons in the string.
if host.startswith('['):
# IPv6 address with a port