Skip to content

Instantly share code, notes, and snippets.

View dan-blanchard's full-sized avatar

Dan Blanchard dan-blanchard

View GitHub Profile
@juliantaylor
juliantaylor / pygil.stp
Created September 22, 2015 23:04
python gil stap
# stap -g pygil.stp <pid> <minlocktime-in-mus>
global init = 0
global gpid = $1
global lock_start = 0
global mintime = $2
# may need to be libpython on some systems
probe process("/usr/bin/python-dbg").function("PyThread_acquire_lock")
{
if (init == 0) {
@topiaruss
topiaruss / pytest.ini
Last active June 21, 2016 15:57
Experimenting with minimal test frameworks for spout and bolt
[pytest]
python_paths = src/bolts src/spouts
testpaths = src/tests
@dan-blanchard
dan-blanchard / fix_timestamps.py
Created September 29, 2013 16:38
Script to fix timestamps of files based on filenames. Default filename regular expression looks for filenames named by CameraSync. I developed this to fix an annoying issue where timestamps were wrong for files uploaded from my iPhone, which made Aperture/iPhoto sort them incorrectly (because videos don't have EXIF data).
#!/usr/bin/env python
'''
Little script to set timestamps based on filenames for files uploaded to
Dropbox by CameraSync.
:author: Dan Blanchard
:date: September, 2013
'''
from __future__ import print_function, unicode_literals
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser(description="""
I never freaking remember argparse syntax and the docs are so all over the place
that I need this for an example.
#!/usr/bin/env python
from bs4 import BeautifulSoup
from twilio.rest import TwilioRestClient
import json
import os
import re
import requests
url = 'https://postmates.com/los-angeles'
@saranrapjs
saranrapjs / default.vcl
Last active February 15, 2018 19:22
Varnish + JSONP example
vcl 4.0;
backend default {
.host = "127.0.0.1";
.port = "80";
}
sub vcl_recv {
unset req.http.callback;
@dan-blanchard
dan-blanchard / .1.miniconda.md
Last active December 11, 2019 22:38
Quicker Travis builds that rely on numpy and scipy using Miniconda

For ETS's SKLL project, we found out the hard way that Travis-CI's support for numpy and scipy is pretty abysmal. There are pre-installed versions of numpy for some versions of Python, but those are seriously out of date, and scipy is not there are at all. The two most popular approaches for working around this are to (1) build everything from scratch, or (2) use apt-get to install more recent (but still out of date) versions of numpy and scipy. Both of these approaches lead to longer build times, and with the second approach, you still don't have the most recent versions of anything. To circumvent these issues, we've switched to using Miniconda (Anaconda's lightweight cousin) to install everything.

A template for installing a simple Python package that relies on numpy and scipy using Miniconda is provided below. Since it's a common s

@letmaik
letmaik / README.md
Last active November 23, 2020 04:37
Using travis_retry inside shell scripts together with set -e

If you want to use travis_retry from within your own shell script files then you first have to make the travis_retry shell function available by sourcing the travis_retry.sh file, otherwise you just get a "command not found" error. See example.sh for a full example.

Note that the original function as found in the travis-ci/travis-build repository was slightly modified to allow it to be used in a shell context where set -e is enabled.

For reference, a tweet by Travis CI saying that you should copy the travis_retry code as I've done here: https://twitter.com/plexus/status/499194992632811520

@xofer
xofer / 0-storm-proxy-readme.md
Last active June 28, 2021 11:43
Storm UI Nginx conf for Nimbus with logviewer proxy

We have a setup that I assume is quite common: A publicly accessible Nimbus running Storm UI. The worker nodes can only be accessed from the Nimbus (via the LAN). All the nodes have internal DNS names (i.e. node.lan.example.com), which is set in the configuration files; they use these DNS names to reach each other. The Nimbus has an external DNS name (storm.example.com) for public access. The Nimbus's UI is behind an Nginx proxy, which provides HTTP Auth and HTTPS.

Because of this setup, the logviewer links in the UI do not work. In order to fix this, we employ an elaborate hack shown in the conf file below. It uses ngx_http_substitutions_filter_module to rewrite content returned by the Storm UI and some complicated URL rewrite tricks to proxy the workers' logviewers through through the Nimbus.