Clojure Syntactic Piplines with a while-> Threading Macro

core_ext.clj

;; I got thinking about this after reading Stuart Sierra's blog post on syntactic
;; pipelines, which while a bit old by now is still good reading for anybody
;; applying clojure to larger workflows. You can find it here:
;; http://stuartsierra.com/2012/05/16/syntactic-pipelines
;;
;; The point of this gist is an alternative way to organize synchronouse pipelines
;; that may otherwise be implemented as heavily nested branch expresssions, which
;; can be hard to read. The final proposal is a very minimal addition that fits closely
;; with existing clojure threading macors, but I wasn't able to find anything online
;; where this was already discussed.

;; By way of example, let's look at how `some->` helps clarify intent. Given two "step"
;; fns, `bar` and `baz`, each of which has some chance of returning nil. Here, a rand
;; is a simple stand in for what might be a network failure, missing resource, or some
;; other dependency failure.
(defn bar [m] 
  (when (< (rand) 0.3)
    (assoc m :bar 1)))
  
(defn baz [m] 
  (when (< (rand) 0.7)
    (assoc m :baz 1)))

;; Implementing a fn that pipes a value through bar, baz, and other custom
;; logic using `if` or `when` results in deeply nested logic, which is more
;; verbose and difficult to read in non-contrived situations. The assumption
;; is that if one step fails, we do not want to continue to subsequent steps.
(defn foo1 [m]
  (when m
    (when-let [m (bar m)]
      (when-let [m (baz m)]
        (assoc m :complete 1)))))

;; The `some->` thread operator allows us to more succinctly convey the same
;; intent:
(defn foo2 [m]
  (some-> m
    (bar)
    (baz)
    (assoc :complete true)))

;; So that's useful, but what if the continuation condition isn't nil? That's where
;; this proposal steps in. I initially thought that reduce/reduced might be useful,
;; which might look something like:

(defn foo3 [m]
  (reduce (fn [m step]
            (let [m (step m)]
              ;; success? is whatever test makes sense (some? would emulate some->)
              (if (success? m) 
                m
                (reduced m))))
          m
          [bar baz #(assoc % :complete true)]))

;; That works, and representing the pipeline as a collection conveniently allows
;; applying different pipelines at runtime, in a way a threading operator wouldn't.
;; But, if the operations are a fixed set, then it's certainly not as clear as some->
;; and perhaps less clear than the initial nested branch structure.

;; So, what if we had a threading macro like some->, but that allowed the caller
;; to specify the continuation condition? Well, it might look like this:
(defmacro while->
  "When test? is true for expr, threads expr into the first form (via ->),
  and when test? is true for that result, through the next etc"
  [test? expr & forms]
  (let [g (gensym)
        pstep (fn [step] `(if (test? ~g) (-> ~g ~step) ~g))]
    `(let [~g ~expr
           ~@(interleave (repeat g) (map pstep forms))]
       ~g)))

;; And, a similar version for consistency with first vs last argument threading:
(defmacro while->>
  "When test? is true for expr, threads it into the first form (via ->>),
  and when test? is true for that result, through the next etc"
  [test? expr & forms]
  (let [g (gensym)
        pstep (fn [step] `(if (test? ~g) (->> ~g ~step) ~g))]
    `(let [~g ~expr
           ~@(interleave (repeat g) (map pstep forms))]
       ~g)))
     
;; The caller-provided continuation predicate allows creating workflow pipelines
;; with custom continuation/termination other than nil, and that return the value 
;; at the time of termination. This behaves similarly to `(reduced v)` in a reduce, 
;; except using threading, which when used appropriately can more clearly convey
;; intention.

;; Here's a contrived example:

(defn maybe-get-foo []
  ;; Assume some chance of returning a resource or nil, could be for 
  ;; any resason.
  )

(defn setup 
  "Performs setup, gets some foo resouce and adds it to m. If foo is
  unavailable, returns an error."
  [m]
  (if-let [foo (maybe-get-foo)]
    (assoc m :foo  foo)
    :no-foo))

(defn get-data [m]
  "Tries to get some data. If fails for some reason, returns error why."
  (if-let [foo-val (use-foo (:foo m))]
    (assoc m :foo-val foo-val)
    :no-foo-val))

(defn transform [m]
  (update m :foo-val transform-foo))

;; Here, we'ere starting with an initial state map. Each of the steps obeys a contract
;; to return a map on success; a non-success value could be anything else, here I'm using
;; keywords to indicate the cause of failure. This allows me to pipe a state map through
;; various fns, and allows each of them to declare a failure that halts the processing
;; of further steps in the pipeline and propagates the error back to the caller.
(while-> map? {}
  (setup)
  (get-data)
  (transform)
  :foo-val)

;; Given this pipeline, the result will either be some `:foo-val` from the map, or if
;; anything along the way failed, a keyword indicating the error. 

;; A benefit of this approach to building syntactic pipelines is the breaking down of
;; pipeline steps into smaller fns, each of which is easier to comprehend in isolation,
;; and easier to unit test individual steps.

;; Note: another way to do this could be with exceptions, if that's your bag.

Just stumbled across another approach to a similar problem: https://github.com/egamble/let-else