Skip to content

Instantly share code, notes, and snippets.

💭
boiling the ocean

Zach Tellman ztellman

💭
boiling the ocean
Block or report user

Report or block ztellman

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View boolean.js
var text = new PointText({
position: view.center + [0, 200],
fillColor: 'black',
justification: 'center',
fontSize: 20
});
var originals = new Group({ insert: false }); // Don't insert in DOM.
var square = new Path.Circle({
View quasi-quote-macro.clj
(defmacro if-in-str [haystack & clauses]
`(do
~@(map
(fn [clause]
(let [needle (first clause)
is-present-form (second clause)
is-not-present-form (nth clause 2 nil)]
`(if (> (.indexOf ~haystack ~needle) -1)
~is-present-form
~is-not-present-form)))
View kafka-manifold.clj
(defn kafka-sink []
(let [s (s/stream)]
(s/consume-async
(fn [msg]
(d/future (synchronous-put-to-kafka msg))))
s))
View top-k-threshold.md

This relates to work I'm doing on a system for distributed outlier detection, as described here.

In this system, the count-min sketch is used as a distributed filter. Over some interval, the CMS is populated on all the edge nodes, sent to a server for merging, and the merged CMS is broadcast back out to the edges. On the following interval, each new key is run through the "global" CMS, and if it's over some threshold, it is added to the set of potential outliers, which are precisely counted.

This is, as described so far, an adaptation of this 2003 paper from single-node to multi-node. However, that paper assumes that the threshold is known ahead of time.

In most cases, though, what constitutes an "outlier" is a moving target. We'd like to adapt as the distribution changes. Previously, I was using a consistent uniform sample of the keys to identify the threshold, but we'd have to take a very

View async.clj
(require '[manifold.deferred :as d])
(defn wrap-async-handler [f]
(fn [req]
(let [rsp (d/deferred)]
(f req #(d/success! rsp %) #(d/error! rsp %))
rsp)))
View Netty.java
public final class Netty {
private Netty() {
}
private static final Field QUEUE;
private static final Class BITS;
private static final Class VM;
View cljx.clj
(defn- queue []
#+clj clojure.lang.PersistentQueue/EMPTY
#+cljs cljs.core/PersistentQueue.EMPTY)
(defn matching-inputs
"Returns a lazy sequence of input sequences which the automaton will match."
[fsm]
(let [fsm (-> fsm ->dfa final-minimize)
accept? (set (accept fsm))
q (atom (conj (queue) [(start fsm) []]))]
View cmsketch.lua
local function delta(t0, t1)
return ((t1[1] - t0[1]) * 1000000) + (t1[2] - t0[2])
end
local t0,t1
local s1 = redis.call("get", KEYS[1])
local s2 = ARGV[1]
if s1 == false then
redis.call("set", KEYS[1], s2)
View FPC.java
package compression;
import java.lang.Exception;
import java.nio.ByteBuffer;
import java.nio.DoubleBuffer;
public class FPC {
// adapted from http://users.ices.utexas.edu/~burtscher/papers/tr08.pdf
private static final int PREDICT_TABLE_SIZE = 32768;
View virgil.clj
(ns virgil
(:require
[clojure.java.io :as io]
[clojure.string :as str]
[filevents.core :as fe])
(:import
[javax.tools
DiagnosticCollector
ForwardingJavaFileManager
JavaCompiler
You can’t perform that action at this time.