Skip to content

Instantly share code, notes, and snippets.

@mjwillson
mjwillson / gist:675cc0259e4291d97104
Created September 7, 2015 17:55 — forked from benanne/gist:1759022
Theano AdvancedSubtensor memory leak
import theano.tensor as T
import theano
import numpy as np
import gc
def freemem():
gc.collect()
gc.collect()
gc.collect()
return theano.sandbox.cuda.cuda_ndarray.cuda_ndarray.mem_info()[0] / 1024**2
@mjwillson
mjwillson / iterable.py
Last active August 29, 2015 14:27
Decorate a generator function (or other iterator-returning function) as a multi-shot iterable. A fix for many Python gotchas relating to use of one-shot iterators
class iterable(object):
"""Decorates a generator function (or any other iterator-returning
function) as something which implements the iterable protocol and
can be safely passed to other code which may iterate over it
multiple times.
Usage:
@iterable
def foo():
@mjwillson
mjwillson / ngrams_via_striding.py
Last active August 1, 2017 04:19
Matrix of sliding window ngrams without any copying via numpy striding tricks
from numpy.lib.stride_tricks import as_strided
def ngrams_via_striding(array, order):
itemsize = array.itemsize
assert array.strides == (itemsize,)
return as_strided(array, (max(array.size + 1 - order, 0), order), (itemsize, itemsize))
In [71]: a = numpy.arange(10)
In [72]: ngrams_via_striding(a, 4)
Out[72]:
@mjwillson
mjwillson / ann.rb
Last active December 29, 2015 16:09
ann -- ultra-basic console-based multiclass text annotation tool
#!/usr/bin/env ruby
require 'optparse'
OPTIONS = {}
PARSER = OptionParser.new do |opts|
opts.banner = "Usage: #{$0} [OPTIONS] INPUT_FILE [HOTKEY OUTPUT_FILE]..."
opts.separator(<<END
#{$0} -- ultra-basic console-based multiclass text annotation tool
@mjwillson
mjwillson / scoped_streams.clj
Last active December 17, 2015 14:59
scoped streams in clojure
(ns streams.core
(:require [clojure.java.io :as io]))
(def END (Object.))
(defprotocol Stream
(with-generator [_ callback]
"Should call callback with a generator function, finally closing any
resources associated with the stream after the callback returns.
@mjwillson
mjwillson / sexpr-html-templating.clj
Last active December 17, 2015 12:49
Proof of concept: S-expressions for HTML templating in clojure
(defprotocol ToHtml
(to-html [x]))
(extend-protocol ToHtml
String
(to-html [s]
(clojure.string/escape s {\< "&lt;" \> "&gt;" \" "&quot;" \& "&amp;"}))
clojure.lang.IPersistentMap
(to-html [attrs]
@mjwillson
mjwillson / Encoding.java
Created April 30, 2013 16:33
convert between float arrays and byte arrays
package randomindexing;
/* Helpers for encoding various primitive arrays as byte arrays.
(Why this isn't in the stdlib I have no idea! feel free to replace
with some existing library implementation...)
All encodings use little-endian byte order.
I benchmarked using
ByteBuffer.wrap(b).order(LITTLE_ENDIAN).asFloatBuffer().get(f),
but it's around 3 times slower even on big input.
@mjwillson
mjwillson / mapreduce.clj
Last active December 16, 2015 01:19
Clojure-like psuedo-code for something closer to the actual structure of a Hadoop MapReduce computation
;; I'm sure I'm wrong here -- please correct me!
;; (Although note I'm not trying to capture the exact algorithms and dataflow complexities of hadoop here, just the logical structure of MapReduce computations and a rough sketch of how they're distributed.)
;; not just
(->> data
(pmap mapper)
(reduce reducer))
;; but something more like this.
@mjwillson
mjwillson / multimethod-matrix-multiplication.clj
Last active December 10, 2015 17:58
An example of how multi-method-based dispatch might work for a binary operation like matrix multiplication. Illustrates how a variety of coercion-based defaults can be specified to make life easy on the implementer, while still easily allowing for dispatch to optimal implementation-specific routines whenever it's desired.
;; First some dummy matrix types and operations for us to play with later:
(deftype GenericDense [])
(deftype FooMatrix [])
(deftype BarMatrix [])
(defn generic-multiply [x y] "generic-multiply")
(defn foo-multiply [x y] "foo-multiply")
(defn bar-multiply [x y] "bar-multiply")
(defn foo-generic-multiply [x y] "foo-generic-multiply")
(defn generic-foo-multiply [x y] "generic-foo-multiply")
(defn bar-generic-multiply [x y] "bar-generic-multiply")
@mjwillson
mjwillson / tax_bands.rb
Created October 29, 2009 22:41
Same thing in ruby
Infinity = 1.0/0
SAVINGS_INCOME_TAX_200910 = [[0, 6475], [0.1, 8915], [0.2, 43875], [0.4, Infinity]]
INCOME_TAX_200910 = [[0, 6475], [0.2, 43875], [0.4, Infinity]]
def tax(income, bands=INCOME_TAX_200910)
rate, threshold = bands.first
if income <= threshold
rate * income
else
lowered_remaining_bands = bands[1..-1].map {|r, t| [r, t - threshold]}