Skip to content

Instantly share code, notes, and snippets.

Avatar

Erik Osheim non

  • Providence / Philly
View GitHub Profile
anonymous
anonymous / private_deps.md
Created Jun 12, 2015
Better Dependency Hygiene With Private Dependencies On JVM
View private_deps.md

Better Dependency Hygiene With Private Dependencies On JVM

A common pain when working with large projects is the diamond dependency. Consider a commonly used library such as ASM. One wants to build a big application reusing many powerful libraries, but unfortunately many of my desired dependencies themselves depend on different and incompatible versions of ASM. While compiling my code, since ASM does not appear in any APIs I touch, everything compiles fine, but at runtime the JVM only includes one version of classes of a given name leading to runtime binary errors.

OSGI Bundles are related to solving this problem, but it appears that is a heavy solution that has proven to be too cumbersome to actually use. Here we propose a lighter weight approach that benefits each incremental project that adopts this method.

Private dependencies are implemented by a build tool plug-in. In the build where one declares dependencies, one can label a jar dependency to be a private dependency. A private dependency means tha

@johnynek
johnynek / gist:8961994
Last active Aug 29, 2015
Some Questions with Sketch Monoids
View gist:8961994

Unifying Sketch Monoids

As I discussed in Algebra for Analytics, many sketch monoids, such as Bloom filters, HyperLogLog, and Count-min sketch, can be described as a hashing (projection) of items into a sparse space, then using two different commutative monoids to read and write respectively. Finally, the read monoids always have the property that (a + b) <= a, b and the write monoids has the property that (a + b) >= a, b.

##Some questions:

  1. Note how similar CMS and Bloom filters are. The difference: bloom hashes k times onto the same space, CMS hashes k times onto a k orthogonal subspaces. Why the difference? Imagine a fixed space bloom that hashes onto k orthogonal spaces, or an overlapping CMS that hashes onto k * m length space. How do the error asymptotics change?
  2. CMS has many query modes (dot product, etc...) can those generalize to other sketchs (HLL, Bloom)?
  3. What other sketch or non-sketch algorithms can be expressed in this dual mo
@johnynek
johnynek / hdfsio.scala
Created Aug 25, 2013
IO Monad for the HDFS FileSystem: http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/fs/FileSystem.html Make your HdfsIO[T] object (for example op), set an implicit FileSystem, then call HdfsIO.run(op) TOTALLY UNTESTED! (but it does compile)
View hdfsio.scala
import java.io.{FileNotFoundException, IOException}
import org.apache.hadoop.fs.{
FileSystem,
FileStatus,
FSDataInputStream,
FSDataOutputStream,
Path
}
@robinsloan
robinsloan / langoliers.rb
Last active Jun 11, 2021
The Langoliers, a tweet deletion script
View langoliers.rb
require "rubygems"
require "twitter"
require "json"
# things you must configure
TWITTER_USER = "your_username"
MAX_AGE_IN_DAYS = 1 # anything older than this is deleted
# get these from dev.twitter.com
CONSUMER_KEY = "your_consumer_key"
@eed3si9n
eed3si9n / kind.scala
Created Sep 3, 2012
calculates a type's kind
View kind.scala
// requires Scala 2.10.0-M7
def kind[A: scala.reflect.TypeTag]: String = {
import scala.reflect.runtime.universe._
def typeKind(sig: Type): String = sig match {
case PolyType(params, resultType) =>
(params map { p =>
typeKind(p.typeSignature) match {
case "*" => "*"
case s => "(" + s + ")"
}
@tbuktu
tbuktu / BigInteger.java.phase3
Created Jan 7, 2012
Patched BigInteger using efficient algorithms for multiplication and division
View BigInteger.java.phase3
/*
* Copyright (c) 1996, 2013, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License version 2 only, as
* published by the Free Software Foundation. Oracle designates this
* particular file as subject to the "Classpath" exception as provided
* by Oracle in the LICENSE file that accompanied this code.
*