Skip to content

Instantly share code, notes, and snippets.

@sorenmacbeth
sorenmacbeth / ambrose.clj
Last active August 29, 2015 13:55
cascalog ambrose integration
(defn ambrose?-
[& bindings]
(let [[name bindings] (flow/parse-exec-args bindings)
bindings (mapcat (partial apply normalize-sink-connection)
(partition 2 bindings))
flow (-> (apply compile-flow name bindings)
flow/flow-def
flow/compile-hadoop)
server (EmbeddedAmbroseCascadingNotifier.)]
(.addListener flow server)
tell application "System Preferences"
reveal anchor "keyboardTab" of pane "com.apple.preference.keyboard"
tell application "System Events"
click checkbox 1 of tab group 1 of window 1 of application process "System Preferences"
end tell
end tell
if application "System Preferences" is running then
tell application "System Preferences" to quit
end if
(ns hbase.cascalog.core
(:require [cascalog.workflow :as w])
(:import [cascading.hbase HBaseTap HBaseScheme ByteHolder]
[cascading.tuple Fields]
org.apache.hadoop.hbase.util.Bytes))
(defn hbase-tap [table-name key-field column-family & value-fields]
(let [scheme (HBaseScheme. (w/fields key-field) column-family (w/fields value-fields))]
(HBaseTap. table-name scheme)))
@martintrojer
martintrojer / queries.clj
Created July 16, 2012 12:16
Datomic queries in core.logic
(ns queries
(:refer-clojure :exclude [==])
(:use [clojure.core.logic])
(:use [datomic.api :only [q]]))
(defn query [rule xs]
(let [prule (prep rule)]
(map #(binding-map* prule (prep %)) xs)))
;; ---
@erasmas
erasmas / WriteToParquetExample.java
Last active March 29, 2017 05:47
Cascalog workflow to copy data from CSV to Parquet. How do I fix this so that schema fields are not prepended with '?' ?
/**
* Same workflow but using Cascading, output fields in Parquet file are obviously fine and not prepended with '?'
*/
package cascading.sandbox;
import cascading.flow.Flow;
import cascading.flow.FlowDef;
import cascading.flow.hadoop.HadoopFlowConnector;
import cascading.pipe.Pipe;
import cascading.property.AppProps;
@jneen
jneen / variant-multimethods.clj
Created November 22, 2014 20:55
Multimethods with variants
; it's a bit cumbersome to set up and there's the unfortunate need to ignore the tag
; in the individual methods, but this allows you to leave the interpretation of open
; variants, well, *open* for extension by multimethod.
; dispatch off the first argument, which will be the tag
(defmethod command-multi (fn [tag & data] tag))
; the first argument to the *method* is still the tag
(defmulti command-multi :print [_ val] (println val))
(defmulti command-multi :read [_ fname] (slurp fname))
@davegurnell
davegurnell / TypeclassDemo.scala
Created October 6, 2015 14:53
Example of the type class pattern in Scala
object TypeclasseDemo {
// The parts of the type class pattern are:
//
// 1. the "type class" itself -- a trait with a single type parameter;
//
// 2. type class "instances" for each type we care about,
// each marked with the `implicit` keyword;
//
// 3. an "interface" to the type class -- one or more methods
@jcromartie
jcromartie / arcc.clj
Created October 31, 2012 14:57
arc challenge in Clojure
(ns arc.core
(:use compojure.core)
(:require [compojure.route :as route]
[compojure.handler :as handler]
[ring.util.response :as response]
[ring.adapter.jetty :as jetty]))
(def route-map (ref {}))
(def ^:dynamic *params* nil)
@MLnick
MLnick / HyperLogLogStoreUDAF.scala
Last active March 16, 2022 05:31
Experimenting with Spark SQL UDAF - HyperLogLog UDAF for distinct counts, that stores the actual HLL for each row to allow further aggregation
class HyperLogLogStoreUDAF extends UserDefinedAggregateFunction {
override def inputSchema = new StructType()
.add("stringInput", BinaryType)
override def update(buffer: MutableAggregationBuffer, input: Row) = {
// This input Row only has a single column storing the input value in String (or other Binary data).
// We only update the buffer when the input value is not null.
if (!input.isNullAt(0)) {
if (buffer.isNullAt(0)) {
@msgodf
msgodf / kiczales-oopsla94-black-boxes-reuse.md
Last active March 28, 2022 22:23
Gregor Kiczales "Why are black boxes so hard to reuse?"

This talk was given by Gregor Kiczales of Xerox PARC at OOPSLA ’94, 10/26/94. © 1994, University Video Communications. A transcript, with point- and-click retrieval of the slides, is available at http:/www.xerox.com/PARC/spl/eca/oi/gregor-invite/gregor- transcript.html

Why are black boxes so hard to reuse?

I think our field will go through a revolution. We will fundamentally change the way we think about and use abstraction in the engineering of software.

The goal of this talk is to summarize the need for and the basic nature of this abstraction framework.

The change is not new problems or new systems, but a new way of thinking about existing problems and existing systems.