Skip to content

Instantly share code, notes, and snippets.

View bnyeggen's full-sized avatar

Bryce Nyeggen bnyeggen

View GitHub Profile
@bnyeggen
bnyeggen / multiprocess_with_instance_methods.py
Created July 16, 2011 14:17
Example showing how to use instance methods with the multiprocessing module
from multiprocessing import Pool
from functools import partial
def _pickle_method(method):
func_name = method.im_func.__name__
obj = method.im_self
cls = method.im_class
if func_name.startswith('__') and not func_name.endswith('__'): #deal with mangled names
cls_name = cls.__name__.lstrip('_')
func_name = '_' + cls_name + func_name
@bnyeggen
bnyeggen / MMapper.java
Last active May 31, 2021 12:06
Mmap more than 2GB of a file in Java
import java.io.RandomAccessFile;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.nio.channels.FileChannel;
import sun.nio.ch.FileChannelImpl;
import sun.misc.Unsafe;
@SuppressWarnings("restriction")
public class MMapper {
@bnyeggen
bnyeggen / upgradablelock.go
Last active July 25, 2019 03:31
Upgradable read -> write locks in Go
package main
import (
"fmt"
"runtime"
"sync"
)
type UpgradableLock struct {
uglMutex sync.RWMutex
@bnyeggen
bnyeggen / clojure_hive_jdbc.clj
Created December 13, 2011 15:12
Clojure to Hive via JDBC
(comment Add [org.clojure/java.jdbc "0.1.1"] to project dependencies)
(ns myproject.core
(:use [clojure.java.jdbc :only [with-connection, with-query-results]]))
(let [db-host "MyHost"
db-port 10000
db-name "default"]
(def db {:classname "org.apache.hadoop.hive.jdbc.HiveDriver" ; must be in classpath
:subname (str "//" db-host ":" db-port "/" db-name)
@bnyeggen
bnyeggen / geoull.cl
Created December 16, 2013 04:24
Geospatial recommendation kernel
/*
OpenCL code for a geospatial recommender. The underlying algorithm is:
- Each user has a set of XY points
- Similarity between user A and user B is defined as the inverse mean minimum distance to each of user A's points, from any of user B's points. This is not symmetric. E.g., if user A has points at (0,0) and (10,10), and user B has a point at (4,4), similarity of B to A will be 1 / ((sqrt(4^2 + 4^2) + sqrt(6^2 + 6^2)) / 2), and similarity of A to B will be 1 / (sqrt(4^2 + 4^2)). It is possible to define other similarity measures along the same lines (e.g. inverse mean minimum squared distance) that will give different results. We're all about the heuristics.
- For evaluating the predicted strength at a new XY location for a given main user,
- Calculate an inverse-square dropoff from every point (as if one was calculating gravity in an N-body simulation). Again, one could use some other exponent with better empirical foundation. This can be optimized by excluding points beyond a certa
@bnyeggen
bnyeggen / core.clj
Last active December 27, 2015 14:29
Running a Compojure app with a main function via java -jar
(ns myproject.core
(:use compojure.core)
(:require [compojure.route :as route]
[compojure.handler :as handler]
[ring.adapter.jetty :as ring-jetty])
(:gen-class))
(defroutes my-routes
(GET "/" [] "<h1>Hello World</h1>")
(route/not-found "<h1>Page not found</h1>"))
@bnyeggen
bnyeggen / RandomElementProvider.java
Created October 6, 2013 04:10
Threadsafe pseudo-shuffle - each caller gets a random element of the backing array, with the guarantee that each element will be selected at most once. The tradeoff is that as we approach all elements selected, we hit a slowdown as we're forced to scan for the remaining elements. For that reason, this is best suited to situations where we need a…
import java.util.Random;
import java.util.concurrent.atomic.AtomicIntegerArray;
public class RandomElementProvider {
//Flag value - not valid as an element of the backing array.
public static final int INVALID = Integer.MIN_VALUE;
//After this many failed random selections, devolve to a scan.
//Corresponds to being able to handle on average (1 - 1/RETRY_THRESHOLD) of elements
//being selected already before scanning
public static final int RETRY_THRESHOLD = 4;
@bnyeggen
bnyeggen / hinttype.clj
Created February 2, 2013 21:13
Macro-generated deftype with type-hinted fields.
(defmacro hinttype
[type-name & hint-symbols]
`(deftype ~type-name
[~@(for [s hint-symbols]
(with-meta (gensym "a")
{:tag (case s
:int 'int
:long 'long
:float 'float
:double 'double
@bnyeggen
bnyeggen / SizeOf.java
Last active December 10, 2015 13:08
Unsafe-based sizeof
// A simpler method than reflection-based traversal, using Unsafe
public static Unsafe getUnsafe() {
Field f = Unsafe.class.getDeclaredField("theUnsafe");
f.setAccessible(true);
return (Unsafe) f.get(null);
}
public static long sizeOf(Object o) {
Unsafe u = getUnsafe();
@bnyeggen
bnyeggen / Crypt.java
Created November 29, 2015 22:04
Stream based en/decryption in Java 7
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.spec.InvalidKeySpecException;
import java.security.spec.KeySpec;