Skip to content

Instantly share code, notes, and snippets.

@eightysteele
Forked from robinkraft/gist:3035807
Created July 2, 2012 21:26
Show Gist options
  • Save eightysteele/3035811 to your computer and use it in GitHub Desktop.
Save eightysteele/3035811 to your computer and use it in GitHub Desktop.
(use 'forma.hadoop.jobs.scatter)
;; put stuff in pail
(let [pail-path "/Users/robin/delete/tmp/newpail"
vcf-src [(thrift/DataChunk* "vcf"
(thrift/ModisPixelLocation* "500" 28 8 0 0)
25
"00")]
gadm-src [(thrift/DataChunk* "gadm"
(thrift/ModisPixelLocation* "500" 28 8 0 0)
16435
"00")]
vcf-query (<- [?dc]
(vcf-src ?dc))
gadm-query (<- [?dc]
(gadm-src ?dc))]
;; Data appear in pail no problem:
(to-pail pail-path vcf-query)
(to-pail pail-path gadm-query))
(defn unpack-long
"Unpack long value from static source"
[dv]
(.getLongVal dv))
;; get values out of pail
;; data comes out when there's one source, but joining on sres h v s l produces nothing
;; This works:
(let [pail-path "/Users/robin/delete/tmp/newpail"
vcf-src (split-chunk-tap pail-path ["vcf"])]
(<- [?sres ?h ?v ?s ?l ?vcf]
(vcf-src _ ?vcf-obj)
(thrift/unpack ?vcf-obj :> _ ?vcf-loc ?vcf-dv _ _)
(thrift/unpack ?vcf-loc :> ?sres ?h ?v ?s ?l)
(unpack-long ?vcf-dv :> ?vcf)))
;; Works but no results:
(let [pail-path "/Users/robin/delete/tmp/newpail"
vcf-src (split-chunk-tap pail-path ["vcf"])
gadm-src (split-chunk-tap pail-path ["gadm"])
]
(<- [?sres ?h ?v ?s ?l ?vcf ?gadm]
(vcf-src _ ?vcf-obj)
(thrift/unpack ?vcf-obj :> _ ?vcf-loc ?vcf-dv _ _)
(thrift/unpack ?vcf-loc :> ?sres ?h ?v ?s ?l)
(unpack-long ?vcf-dv :> ?vcf)
(gadm-src _ ?gadm-obj)
(thrift/unpack ?gadm-obj :> _ ?gadm-loc ?gadm-dv _ _)
(thrift/unpack ?gadm-loc :> ?sres ?h ?v ?s ?l)
(unpack-long ?gadm-dv :> ?gadm)))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment