Skip to content

Instantly share code, notes, and snippets.

  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
What would you like to do?
Example to control Clojure pmap concurrency level using the chunk size of lazy-seqs.
;; From:
(defn re-chunk [n xs]
(when-let [s (seq (take n xs))]
(let [cb (chunk-buffer n)]
(doseq [x s] (chunk-append cb x))
(chunk-cons (chunk cb) (re-chunk n (drop n xs)))))))
;; Simply wrap the collection or seq/lazy-seq with re-chunk and the configured chunk size will make pmap's concurrency level grow to the size of the chunk.
(time (dorun (pmap (fn[e] (Thread/sleep 100) (inc e)) (re-chunk 100 (range 1000)))))
"Elapsed time: 1038.57 msecs"
Copy link

didibus commented Jan 5, 2021

In Clojure, pmap has a concurrency level (parallelism level) equal to numberOfCpuCores + 2 or the size of the chunk of the lazy-seq if it is passed a lazy-seq as collection. That means you can take advantage of the fact its concurrency level is the size of lazy-seq by passing it a lazy-seq of a custom chunk size. Lucky for us, there is a convenient function to re-chunk a lazy-seq and thus let us control pmap's parallelization level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment