Skip to content

Instantly share code, notes, and snippets.

@tie-rack
Last active August 29, 2015 14:15
Show Gist options
  • Save tie-rack/b7056482699d7e1e2865 to your computer and use it in GitHub Desktop.
Save tie-rack/b7056482699d7e1e2865 to your computer and use it in GitHub Desktop.
Potential function for splitting a sequence of maps into groups to feed to a function that has a limit on the total number of keys it can handle in any given sequence of maps (SQLite statement parameter limit). (Naming suggestions welcome!)
(defn chunk-rows
"Given a seq of maps, split them into groups, such that no group
have more than n total keys across its maps."
[rows n]
(lazy-seq
(loop [this-chunk []
this-chunk-size 0
to-go rows]
(if (empty? to-go)
(list this-chunk)
(let [next-row (first to-go)
size-with-next-row (+ this-chunk-size (count next-row))]
(cond
(< n (count next-row)) (throw (ex-info "Map too large" {:map next-row}))
(< n size-with-next-row) (cons this-chunk (chunk-rows to-go n))
:else (recur (conj this-chunk next-row)
size-with-next-row
(rest to-go))))))))
;; A sequence of maps read from crew.csv, ready to be inserted as rows
;; into a SQLite database
(def rows
[{:name "Picard", :shirt "red", :rank "Captain"}
{:name "Riker", :shirt "red", :rank "Commander"}
{:name "Data", :shirt "yellow", :rank "Lt. Commander"}
{:name "Worf", :shirt "yellow", :rank "Lt."}
{:name "Crusher"}
{:name "Troi", :shirt "teal", :rank "Lt. Commander"}])
;; If SQLite had a statement parameter limit of less than 16, trying
;; to pass all these rows at once would fail. Here, chunk-rows lets us
;; break this up into groups it can handle.
(chunk-rows rows 15)
;; ([{:name "Picard", :shirt "red", :rank "Captain"}
;; {:name "Riker", :shirt "red", :rank "Commander"}
;; {:name "Data", :shirt "yellow", :rank "Lt. Commander"}
;; {:name "Worf", :shirt "yellow", :rank "Lt."}
;; {:name "Crusher"}]
;; [{:name "Troi", :shirt "teal", :rank "Lt. Commander"}])
;; A sequence of sequences, each with a maximum of 15 total keys in
;; each group. The first group has 13 total keys and the second group
;; has 3. Each group can successfully be passed to a function to
;; insert them into the database.
;; Other examples:
(chunk-rows rows 7)
;; ([{:name "Picard", :shirt "red", :rank "Captain"}
;; {:name "Riker", :shirt "red", :rank "Commander"}]
;; [{:name "Data", :shirt "yellow", :rank "Lt. Commander"}
;; {:name "Worf", :shirt "yellow", :rank "Lt."}
;; {:name "Crusher"}]
;; [{:name "Troi", :shirt "teal", :rank "Lt. Commander"}])
(chunk-rows rows 5)
;; ([{:name "Picard", :shirt "red", :rank "Captain"}]
;; [{:name "Riker", :shirt "red", :rank "Commander"}]
;; [{:name "Data", :shirt "yellow", :rank "Lt. Commander"}]
;; [{:name "Worf", :shirt "yellow", :rank "Lt."}
;; {:name "Crusher"}]
;; [{:name "Troi", :shirt "teal", :rank "Lt. Commander"}])
;; (chunk-rows rows 2) ;; <- that will blow up, because there's no
;; hope for that insert anyway.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment