Example to control Clojure pmap concurrency level using the chunk size of lazy-seqs.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
;; From: https://clojuredocs.org/clojure.core/chunk#example-5c9cebc3e4b0ca44402ef6ec | |
(defn re-chunk [n xs] | |
(lazy-seq | |
(when-let [s (seq (take n xs))] | |
(let [cb (chunk-buffer n)] | |
(doseq [x s] (chunk-append cb x)) | |
(chunk-cons (chunk cb) (re-chunk n (drop n xs))))))) | |
;; Simply wrap the collection or seq/lazy-seq with re-chunk and the configured chunk size will make pmap's concurrency level grow to the size of the chunk. | |
(time (dorun (pmap (fn[e] (Thread/sleep 100) (inc e)) (re-chunk 100 (range 1000))))) | |
"Elapsed time: 1038.57 msecs" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In Clojure,
pmap
has a concurrency level (parallelism level) equal tonumberOfCpuCores + 2
or the size of the chunk of thelazy-seq
if it is passed alazy-seq
as collection. That means you can take advantage of the fact its concurrency level is the size oflazy-seq
by passing it alazy-seq
of a custom chunk size. Lucky for us, there is a convenient function to re-chunk a lazy-seq and thus let us controlpmap
's parallelization level.