Skip to content

Instantly share code, notes, and snippets.

@duanebester
Created October 28, 2021 19:50
Show Gist options
  • Save duanebester/db182ca4003ae3e609968191db511a3f to your computer and use it in GitHub Desktop.
Save duanebester/db182ca4003ae3e609968191db511a3f to your computer and use it in GitHub Desktop.
Web Scraping w/ Clojure Cache
(require '[taoensso.carmine :as car])
;; Redis Setup
(def redis-conn {:pool {} :spec {:uri "redis://localhost:6379/"}})
(defmacro wcar* [& body] `(car/wcar redis-conn ~@body))
(defn clear-site
[^String site]
(wcar* (car/del site)))
(defn get-count
[^String site]
(wcar* (car/llen site)))
(require '[clojure.string :refer [includes?]])
(defn product->cache [site mmap]
(when-let [url (:loc mmap)]
(when (includes? url "/products")
(wcar* (car/rpush site mmap)))))
(defn cache->product [site] (wcar* (car/lpop site)))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment