Last active
April 6, 2022 09:14
-
-
Save JacobNinja/5c98496a632e1a466cbf to your computer and use it in GitHub Desktop.
Clojure core.async pipeline example
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(require '[clojure.core.async :as async] | |
'[clj-http.client :as client] | |
'[clojure.data.json :as json]) | |
(def concurrency 5) | |
(let [in (async/chan) | |
out (async/chan) | |
request-handler (fn [url out*] | |
(async/go | |
(println "Making request:" url) | |
(let [response (client/get url) | |
body (json/read-str (:body response))] | |
(doseq [repo (body "items")] | |
(async/>! out (repo "clone_url")))) | |
; Finally close the channel to signal finished processing | |
(async/close! out*)))] | |
; Process `in` messages concurrently | |
(async/pipeline-async concurrency out request-handler in) | |
; Push URLs to process | |
(async/go | |
(doseq [url (for [page (range 10)] (str "https://api.github.com/search/repositories?q=language:clojure&page=" | |
(inc page)))] | |
(async/>! in url))) | |
; Print results of processing | |
(async/go-loop [] | |
(println (async/<! out)) | |
(recur))) | |
; `in` can be backed by a redis queue | |
(comment | |
(async/go-loop [] | |
(if-let [message (pop-redis-queue)] | |
(async/>! in message) | |
; Sleep if no messages available | |
(async/<! (async/timeout 1000))) | |
(recur))) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have tried to implement the code using redis, to try it out and it works once.
Trying to understand where I am going wrong:
After the first processing, the out channel is closed.
That blocks any other future processing.
I can push to redis multiple times and I would like a continuous process.
Any idea what am I missing?