Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Use Clojure agents to concurrently filter out elements from a collection which requires calling a remote API to know if the element should be filtered or not.
(defn api-pred [e]
(Thread/sleep 100)
(even? e))
(let [coll-to-process (range 1000)
concurrency 100
agents (repeatedly concurrency #(agent []))]
(doseq [[i agnt] (map vector coll-to-process (cycle agents))]
(send-off agnt
#(if (api-pred i)
(conj % i)
%)))
(apply await agents)
(->> (mapv deref agents)
(reduce into)))
;; We spawn concurrency number of agents (so 100 in this example). And then we round-robin
;; sending them the task of calling the api-pred function for each collection item and if
;; true we conj the item over the batch the agent is handling. Then we wait for all of
;; them to be done, and we reduce over their results.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment