Proposed solution to problem of a future in a lazy-seq not being run if never deref'd.

reads_example.clj

(defn generic-insert [query]
  (try
    (with-mongo (get *conns* (get query "server_id" "default"))
      (with-db (query "database")
        (insert!
          (query "collection")
          (query "insert"))))
    (catch Exception e
      (error e "Mongo generic-insert failed."))))

(defn queryblock [query]
  (for [action query]
    (condp = (action "action")
      "query"  {(action "label") (generic-read action)}
      "update" {(action "label") (generic-update action)}
      "remove" {(action "label") (generic-remove action)}
      "insert" {(action "label") (generic-insert action)})))


(defn exec-query 
  "Example of exec-query"
  [type cfg]
  (condp = type
    "mysql" (mysql/queryblock (cfg "queryblock"))
    "mongo" (mongo/queryblock (cfg "queryblock"))
    {:error (str "Unknown 'type' " type ".")}))


(defn create-tasks
  "Transform named input into a sorted vector of hash-maps.
  Each hash-map contains a future with associated meta."
  [input timeout]
  (sort compare-timeout
    (vec
      (for [cfg input]
        {:id (cfg "label")
         :future (future (exec-query (cfg "type") cfg))
         :start (System/currentTimeMillis)
         :timeout (get-number (cfg "timeout") timeout)}))))

(defn original-dps-read
  [input & {:keys [timeout]
            :or {timeout dps-default-timeout}}]
  "Parallelized processing of input config."
  (into {}
    (map
      #(vector
        (% :id)
         (hash-map
          ;; remove the following, future is never executed.
          :result (deref (% :future) (% :timeout) nil)
          :time (- (System/currentTimeMillis) (% :start))))
      (create-tasks input timeout))))

(defn new-dps-read
  [input & {:keys [timeout]
            :or {timeout dps-default-timeout}}]
  "Parallelized processing of input config."
        (into {} (for [tasks (create-tasks input timeout)
                       :let [f (future (deref (tasks :future) (tasks :timeout) nil))]]
          (do 
            (doall (deref f))
            (vector 
              (tasks :id) 
              (hash-map
                ;; remove the following, exec-query-future is executed 
                :result (deref f)
                :time (- (System/currentTimeMillis) (tasks :start))))))))

The goal is to do some sort of fire-and-forget solution with our data store: create a future, return without derefing.

input to *-dps-read is just k/v pairs of what requests to run. Create-tasks is creating a vector of maps, the future is in those maps.

In original-dps-read, i remove the deref, the future thread never runs. doseq around the created map doesn't help. new-dps-read is my attempt at a solution, but it doesn't really seem like a future around a future just to initialize the original future is the right way to do this.