Skip to content

Instantly share code, notes, and snippets.

@eslick
Created February 23, 2011 21:26
Show Gist options
  • Save eslick/841219 to your computer and use it in GitHub Desktop.
Save eslick/841219 to your computer and use it in GitHub Desktop.
clojure-hadoop shuffle component example
;; hadoop -jar compass.jar -step count-users
;; hadoop -jar compass.jar -flow count-user-flow
(defn map-count-users [k fmap]
(ctx/increment-counter "Compass" "rows")
(ctx/increment-counter "Compass" (format "src=%s" (:src (:userinfo fmap))))
[])
(defn count-users-total [job]
(job-counter-value job "Compass" "rows"))
(define-step count-users []
:source (:hbase :users (make-constraints
(f/project :columns [[:userinfo :src]])))
:map map-count-users
:sink :null
:reduce :none
:post-hook count-users-total)
(define-flow count-user-flow []
(do-step count-users))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment