Skip to content

Instantly share code, notes, and snippets.

Created May 17, 2015 22:07
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save raspasov/7c9d8f2872d6065b2145 to your computer and use it in GitHub Desktop.
example of blocking core.async pipeline
;add (:require [clojure.core.async :refer [chan close! go >! <! <!! >!! go-loop put! thread alts! alts!! timeout pipeline pipeline-blocking pipeline-async]])
(def to-ch (chan))
(def from-ch (chan))
;data looks like {url some-data, etc}
(def my-map (atom {}))
(defn fetch-from-url [{:keys [url]}]
(println "going to fetch from " url)
(let [data (slurp url)]
{:url url :data data}))
;save to atom one at a time
(go (loop []
(let [{:keys [url data]} (<! to-ch)]
(swap! my-map assoc url data)
;pipeline with 10 parallelism
(def pipeline-1 (pipeline-blocking 10 to-ch (map fetch-from-url) from-ch))
(def urls [""
;start the processing
(doseq [url urls]
(>!! from-ch {:url url}))
Copy link

Great stuff!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment