Skip to content

Instantly share code, notes, and snippets.

View eslick's full-sized avatar

Ian Eslick eslick

View GitHub Profile
@eslick
eslick / gist:840745
Created February 23, 2011 17:18
clojure-hadoop source component example
(define-source :hbase [table constraints]
:input-format TableInputFormat
:map-reader hbase-table-reader-latest
:configure (fn [job]
(configure-hbase job table constraints)))
@eslick
eslick / gist:840748
Created February 23, 2011 17:20
clojure-hadoop shuffle component example
(define-shuffle :clojure []
:map-writer wrap/clojure-writer
:reduce-reader wrap/clojure-reduce-reader)
@eslick
eslick / gist:840750
Created February 23, 2011 17:20
clojure-hadoop shuffle component example
(define-sink :null []
:reduce-writer wrap/clojure-writer
:output-format NullOutputFormat
:output-key Text
:output-value Text)
@eslick
eslick / gist:840823
Created February 23, 2011 17:56
clojure-hadoop shuffle component example
;; Pseudocode example for guards in flows
(def *guard-time* [:days 7])
(defmacro with-resource-time-guard
[[name timespec inhibit?] &body ]
`(let [resource# (get-resource name)]
(cond (and ~inhibit? (resource-value resource#))
(resource-value resource#)
(or (resource-expired? resource# ~timespec)
@eslick
eslick / step-example.clj
Created February 23, 2011 21:26
clojure-hadoop shuffle component example
;; hadoop -jar compass.jar -step count-users
;; hadoop -jar compass.jar -flow count-user-flow
(defn map-count-users [k fmap]
(ctx/increment-counter "Compass" "rows")
(ctx/increment-counter "Compass" (format "src=%s" (:src (:userinfo fmap))))
[])
(defn count-users-total [job]
(job-counter-value job "Compass" "rows"))
@eslick
eslick / gist:855663
Created March 4, 2011 20:34
An example custom HBase Sink for Flume written in Clojure
;;
;; Flume HBase User Sink
;;
(gen-class
:name compass.flume.UserSink
:extends "com.cloudera.flume.core.EventSink$Base"
:prefix "sink-"
:main false
:constructors {[String] []
@eslick
eslick / election.clj
Created October 30, 2012 06:27
A simple election mechanism for Datomic peers
;; This is a relatively simple election mechanism for a set of
;; processes or datomic peers where a processing step requires a
;; global master. The elector entity is used to connect a set of peer
;; entities labeled as :election/master. Using the algorithm,
;; there will only ever be one :election/master in the :election/peer
;; group defined on the elector.
;;
;; The user API is simple a guarded transaction which calls a txn-fn
;; if the entity is the current master. If not it holds an election
;; if the timeout has expired.
@eslick
eslick / reserve.clj
Created November 21, 2012 02:05
Simple reservation for multiple worker peers using Datomic
(ns experiment.infra.election
(:use [datomic.api :only [q] :as d])
(:require [experiment.system :as sys]
[experiment.infra.data :as data]
[experiment.infra.protocols :as p]))
;;
;; Reserve - Simple peer coordination mechanism
;;
;; Reservations support multiple peers trying to distribution operations
@eslick
eslick / profiles.clj
Last active December 17, 2015 09:48
ritz middleware in profiles.clj
{:user {:plugins [[lein-ritz "0.7.0"]]
:dependencies [[nrepl-inspect "0.3.0"]
[ritz/ritz-nrepl-middleware "0.7.0"]]
:repl-options
{:nrepl-middleware
[inspector.middleware/wrap-inspect
ritz.nrepl.middleware.javadoc/wrap-javadoc
ritz.nrepl.middleware.apropos/wrap-apropos]}}}
@eslick
eslick / Vagrantfile
Created May 20, 2013 18:00
Vagrant file for a test Ubuntu Box
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant::Config.run do |config|
config.vm.box = "Ubuntu64vb"
config.vm.boot_mode = :gui
config.vm.network :bridged
end