Skip to content

Instantly share code, notes, and snippets.

@robinkraft
Last active August 29, 2015 14:21
Show Gist options
  • Save robinkraft/8f538a0e30b08ccdd047 to your computer and use it in GitHub Desktop.
Save robinkraft/8f538a0e30b08ccdd047 to your computer and use it in GitHub Desktop.
convert FORMA 250m data into common data model suitable for GlobalForestWatch.org website
(use 'forma.hadoop.jobs.postprocess)
(in-ns 'forma.hadoop.jobs.postprocess)
(defn f250->cdm
[src t-res zoom]
(let [epoch (date/datetime->period t-res "2000-01-01")]
(<- [?x ?y ?z ?min-period]
(src ?line)
(clojure.string/split ?line #"\t" :> ?lat-str ?lon-str ?date-str)
(read-string ?lat-str :> ?lat)
(read-string ?lon-str :> ?lon)
(date/datetime->period t-res ?date-str :> ?raw-period)
(- ?raw-period epoch :> ?period)
(min-period ?period :> ?min-period)
(latlon->tile ?lat ?lon zoom :> ?x ?y ?z))))
(defn f250->website
"Do full prep of FORMA data for the website, generating all zoom levels."
[src t-res zoom min-zoom]
(let [cdm-src (f250->cdm src t-res zoom)
zoom-src (gen-all-zooms cdm-src min-zoom)]
(agg-periods-counts zoom-src)))
(let [src (hfs-textline "s3n://forma250/forma-filtered-1-text")
t-res "16"
zoom 18
min-zoom 6
sink (hfs-textline "s3n://forma250/gfw-site-1" :sinkmode :replace)]
(?- sink (f250->website src t-res zoom min-zoom)))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment