Skip to content

Instantly share code, notes, and snippets.

View Dieterbe's full-sized avatar

Dieter Plaetinck Dieterbe

View GitHub Profile
@Dieterbe
Dieterbe / gensim interfaces.py; work with tuple directly. just as slow
Created February 24, 2011 14:51
attempt to make SimilarityABC __getitem__ faster
def __getitem__(self, doc):
import time
# get similarities of doc to all documents in the corpus
b = time.time()
if self.normalize:
doc = matutils.unitVec(doc)
allSims = self.getSimilarities(doc)
a = time.time()
# return either all similarities as a list, or only self.numBest most similar, depending on settings from the constructor
@Dieterbe
Dieterbe / create_crowbar_swift.sh
Created April 25, 2012 14:47
create a numbered virtualbox VM with the right properties to serve as Swift node.
#!/bin/bash
if [ -z "$1" ]; then
echo "pass a number as \$1 (storage nodes are numbered, i.e. 1 and 2)"
exit 2
fi
name="crowbar_swift_$1"
# Create and Register the VM with VirtualBox
VBoxManage createvm --register --name "$name"
@Dieterbe
Dieterbe / statsd.init.upstart
Created June 4, 2012 19:30
upstart initscript for statsd
description "Statsd upstart initscript"
author "Dieter Plaetinck <dieter@vimeo.com>"
# based on http://newcome.wordpress.com/2012/02/26/running-programs-as-linux-daemons-using-upstart/
# Stanzas
#
# Stanzas control when and how a process is started and stopped
# See a list of stanzas here: http://upstart.ubuntu.com/wiki/Stanzas#respawn
@Dieterbe
Dieterbe / gist:2938061
Created June 15, 2012 18:34
authenticate-swauth.js
// Dieter Plaetinck <dieter@vimeo.com>
// based on https://github.com/feedhenry/openstack-storage
var request = require('request');
var async = require('async');
var url = require('url');
var defaults = {
openstackIdentityTokens: "/auth/v1.0"
};
@Dieterbe
Dieterbe / monitoring.txt
Created July 7, 2012 09:18
Monitoring thoughts
## graphing/trending dashboards (not alerting)
i see 3 main uses cases:
* interactively selecting datasources to satisfy a specific need ("i want to find out what went wrong yesterday 4-5PM on our memcache cluster, so i'm gonna have a look at different graphs in this timerange, each graph containing 1-N datapoints from various sources")
* trend analysis, prediction and anomaly detection
* more "fixed" dashboards, like amount of storage available/used on storage cluster, the past month. this point becomes less important if you can set up good alerting (based on datapoints or on predictions) but is still useful to have.
### must haves
* date/time range selection through UI widgets (datetime pickers)
@Dieterbe
Dieterbe / find-updated-wsps.sh
Created October 5, 2012 15:58
a tool to detect updated wsp files, useful for finding statsd daemons still writing metrics to graphite
#!/bin/bash
# find-updated-wsp.sh: a tool to detect updated wsp files, useful for finding statsd daemons still writing metrics to graphite;
# especially useful when you tell statsd to write to new graphite servers, and you want to make sure nothing is writing data to the old graphite machine.
# why this way:
# * you can't just check for network traffic, statsd will stay connected even when it has no data to send
# * plain mtime of wsp files is not sufficient: statsd will keep writing the last known value to gauges, and 0's for inactive counters, this data should be ignored.
# * both statsd and graphite write stats of their own, updates of these files can be ignored.
# this script has two modes of operation, based on $1:
@Dieterbe
Dieterbe / swift-signed-url.py
Created October 29, 2012 20:33
generate tempURL for openstack swift
#!/usr/bin/python2
import hmac
import sys
from hashlib import sha1
from time import time
method = 'GET'
base = sys.argv[1]
path = sys.argv[2]
key = sys.argv[3]
expires = int(time() + int(sys.argv[4]))
@Dieterbe
Dieterbe / swift-signed-url.php
Created October 29, 2012 20:49
generate tempURL for openstack swift
#!/usr/bin/php
<?php
$method = 'GET';
$base = $argv[1];
$path = $argv[2];
$key = $argv[3];
$expires = time() + $argv[4];
$hmac_body = "$method\n$expires\n$path";
$sig = hash_hmac('sha1', $hmac_body, $key);
echo "$base$path?temp_url_sig=$sig&temp_url_expires=$expires\n";
@Dieterbe
Dieterbe / swift-1.7.4-statsd-sampling.patch
Created December 14, 2012 18:17
swift 1.7.4 statsd sampling fix
26c26
< from random import shuffle
---
> from random import random, shuffle
363a364
> self.random = random
380c381,384
< parts.append('@%s' % (sample_rate,))
---
> if self.random() < sample_rate:
@Dieterbe
Dieterbe / slides-autoadvance.sh
Created December 19, 2012 23:01
auto advance html slides using xdotool
#!/bin/bash
# dependencies: xdotool
# auto-advance html slides
# <dieter@vimeo.com>
html_file="$1"
BROWSER=chromium
slide=1
right () {
slide=$((slide+1))