An alternative solution to didibus' business process using missionary.

complex_business_process_example_missionary.clj

(ns complex-business-process-example-missionary
  "A stupid example of a more complex business process to implement as a flowchart."
  (:require [missionary.core :as m])
  (:import missionary.Cancelled))

;;;; Config

(def config
  "When set to :test will trigger the not-boosted branch which won't write to db.
   When set to :prod will trigger the boosted branch which will try to write to the db."
  {:env :prod})

(def chance-of-failure
  "The chances that any db read or write 'times out'.
   1 means there is a 1 in 1 chance a read or write times out, always does.
   2 means there is a 1 in 2 chance a read or write times out.
   3 means there is 1 in 3 chance a read or write times out.
   X means there is 1 in X chance a read or write times out."
  3)

;;;; Fake Databases

(def prod-db
  "Our fake prod db, we pretend it has first-name -> last-name
   mappings and some total of some business thing which is supposed
   to reflect the total of all 'boost' events logged in boost-records.
   This means boost-records and total must always reconcile.
   eg: {john mayer
        jane queen
        :boost-records [{:first-name john, :last-name mayer, :boost 12}]
        :total 12}"
  (atom {"john" "mayer"
         "jane" "queen"
         :boost-records []
         :total 0}))

(def test-db
  "Same as prod db, but is for non prod environments.
   Based on the rules and our example input, it should not be written too, but will
   be read from"
  (atom {"john" "doe"
         "jane" "doee"
         :boost-records []
         :total 0}))

;;;; Utils

(defn prod?
  "[pure] Returns true if end is :prod, false otherwise."
  [env]
  (boolean (#{:prod} env)))

;;;; Validation

(defn valid-bar-input?
  "[pure] Returns true if input is valid for bar processing, false otherwise."
  [input]
  (every? number? (vals input)))

;;;; Pure business logic

(defn apply-credit
  "[pure] Applies given credit using ridiculous business stakeholder requirements."
  [credit env]
  (if (prod? env)
    (inc credit)
    (dec credit)))

(defn apply-bonus-over-credit
  "[pure] Applies given bonus over credit using ridiculous business stakeholder requirements."
  [credit bonus env]
  (if (prod? env)
    (+ 10 credit bonus)
    (- credit bonus 10)))

(defn apply-generous-bonus-over-credit
  "[pure] Applies given generous bonus generously over credit using ridiculous business stakeholder requirements."
  [credit bonus env]
  (if (prod? env)
    (+ 100 credit bonus)
    (- credit bonus 100)))

(defn boost->first-name
  "[pure] Given a boost amount, returns the first-name that should dictate boosting based on
   ridiculous business stakeholder requirements."
  [boost env]
  (if (prod? env)
    (if (pos? boost)
      "john"
      "jane")
    (if (neg? boost)
      "john"
      "jane")))

(defn boost?
  "[pure] Returns true if we should boost based on ridiculous business stakeholder requirements."
  [last-name]
  (if (#{"mayer"} last-name)
    true
    false))

(defn query-get-last-name
  "[pure] Task getting last-name from db for given first-name.
   Can throw based on chance-of-failure setting."
  [db first-name]
  (m/sp
    (when (zero? (rand-int chance-of-failure))
      (throw (ex-info "Timed out getting last-name from db" {:first-name first-name})))
    (m/? (m/sleep 1000))
    (get @db first-name)))

(defn query-get-total
  "[pure] Task getting total from db.
   Can throw based on chance-of-failure setting."
  [db]
  (m/sp
    (when (zero? (rand-int chance-of-failure))
      (throw (ex-info "Timed out getting total from db" {})))
    (m/? (m/sleep 1000))
    (get @db :total)))

(defn query-get-boost-records
  "[pure] Task getting boost records from db.
   Can throw based on chance-of-failure setting."
  [db]
  (m/sp
    (when (zero? (rand-int chance-of-failure))
      (throw (ex-info "Timed out getting boost-records from db" {})))
    (m/? (m/sleep 1000))
    (get @db :boost-records)))

(defn write-total
  "[pure] Task writing total to db, overwrites existing total with given total.
   Can throw based on chance-of-failure setting."
  [db total]
  (m/sp
    (when (zero? (rand-int chance-of-failure))
      (throw (ex-info "Timed out writing total to db" {:total total})))
    (m/? (m/sleep 1000))
    (swap! db assoc :total total)))

(defn write-boost-records
  "[pure] Task writing boost records to db, overwrites existing boost records with given boost records.
   Can throw based on chance-of-failure setting."
  [db boost-records]
  (m/sp
    (when (zero? (rand-int chance-of-failure))
      (throw (ex-info "Timed out writing boost-records to db" {:boost-records boost-records})))
    (m/? (m/sleep 1000))
    (swap! db assoc :boost-records boost-records)))

(def first-or
  "[pure] Task completing with first value produced by flow, or default value if empty."
  (partial m/reduce (comp reduced {})))

;;;; Business Processes

(defn process-bar
  "[pure] Task completing with the result of applying the bar business process on given input, or nil if cancelled.
The structure is mostly the same as in the sequential solution, except we got rid of the loops. Instead, we leverage
backtracking to implement retry logic : when a block needs to be retried, we prepend it with an amb> expression starting
with nil (first attempt is done right away), followed by an arbitrary number of sleeps, then an error (too much
attempts). Code paths leading to retries are marked with (amb>), indicating an absence of result. In this way we define
a flow producing the results of each successive attempt, of which we can extract the first item."
  [input]
  (first-or nil
    (m/ap
      (let [env (:env config)
            db (if (prod? env) prod-db test-db)]
        (try
          (if (valid-bar-input? input)
            (let [credit (apply-credit (:credit input) env)
                  boost (if (pos? credit)
                          (apply-bonus-over-credit credit (:bonus input) env)
                          (apply-generous-bonus-over-credit credit (:generous-bonus input) env))
                  first-name (boost->first-name boost env)]
              (m/amb> nil
                (do (m/? (m/sleep (m/amb> 10 100 1000)))
                    (println "Retrying to query last name after failure."))
                (throw (ex-info "All attempts to query last name failed." {})))
              (let [last-name (try (m/? (query-get-last-name db first-name))
                                   (catch Exception _ (m/amb>)))]
                (if (boost? last-name)
                  (do (m/amb> nil
                        (do (m/? (m/sleep (m/amb> 10 100 1000 1250 1500 2000)))
                            (println "Retrying to update boost-records and total after failure."))
                        (throw (ex-info "All attempts to update boost-records and total failed." {})))
                      (try
                        (let [boost-records (m/? (query-get-boost-records db))
                              total (m/? (query-get-total db))
                              new-boost-record {:first-name first-name
                                                :last-name  last-name
                                                :boost      boost}
                              new-boost-records (conj boost-records new-boost-record)
                              new-total (+ total boost)]
                          (m/? (write-boost-records db new-boost-records))
                          (try (m/? (write-total db new-total))
                               (catch Exception e
                                 (m/amb> nil
                                   (do (m/? (m/sleep (m/amb> 10 100 200)))
                                       (println "Retrying to rollback boost records after updating total, after failing to do so."))
                                   (throw (ex-info "Failed to rollback boost records after updating total, out-of-sync boost record is: " {})))
                                 (try (m/? (write-boost-records db boost-records))
                                      (catch Exception _ (m/amb>)))
                                 (throw e)))
                          (println "Process bar boosted.")
                          {:result :boosted})
                        (catch Exception _ (m/amb>))))
                  (do (println "Process bar did not boost.")
                      {:result :not-boosted}))))
            (do (println "Invalid input passed to bar.")
                {:result :invalid-input :msg "All values of bar input must be numbers"}))
          (catch Cancelled _ (m/amb>))
          (catch Exception e
            (do (println (str "Process bar failed unexpectedly with error: " e))
                {:result :error})))))))

;;;; REPL

(comment
  ;; Run our process to see it go in :prod
  (println
    (m/? (process-bar {:credit         0
                       :bonus          1
                       :generous-bonus 2})))
  ;; Print the db to see if it had the effect we intended to it.
  (println
    (if (prod? (:env config))
      @prod-db
      @test-db))

  ;; Run it with a wrong input
  (println
    (m/? (process-bar {:credit         "0"
                       :bonus          1
                       :generous-bonus 2})))

  ;; Run it again and see what happens to the DB
  (println
    (m/? (process-bar {:credit         2
                       :bonus          12
                       :generous-bonus 23})))
  ;; Print the db to see if it had the effect we intended to it.
  (println
    (if (prod? (:env config))
      @prod-db
      @test-db))

  ;; Change to :test env
  (def config {:env :test})

  ;; Run our process to see it go in :test
  (println
    (m/? (process-bar {:credit         0
                       :bonus          1
                       :generous-bonus 2})))
  ;; Print the db to see if it had the effect we intended to it.
  (println
    (if (prod? (:env config))
      @prod-db
      @test-db)))

Reference : https://gist.github.com/didibus/ab6e15c83ef961e0b7171a2fa2fe925d
Missionary : https://github.com/leonoel/missionary

I notice a number of functions documented with as "[pure]..." are doing IO, such as swap! or deref

They're not doing IO. sp returns a description of the effect, nothing happens until ? is called.

On that question, what would be returned if called normally? Say: (query-get-boost-records prod-db)? Does it return some internal to missionary object? Which is later used by the m/? function, or does it return some data description that could relatively be used to visualize things? i.e., if you wanted to write a test for it?

It returns a task, which is a function taking a success callback and a failure callback and returning a cancellation function. It is not dependent on missionary internals, and it is as opaque and testable as a function can be.

They're not doing IO. sp returns a description of the effect, nothing happens until ? is called.

I want to make sure I understand the reasoning behind that. For example:

(def f (m/sp (spit "/tmp/f.txt" "foo")))

Isn't f an impure function and why? To me it seems like f or the sequential process that it encapsulates/represents is ultimately impure when executed with ?.

f is impure, of course. A function can return an impure function and still be pure.

f is impure, of course. A function can return an impure function and still be pure.

I see, so for example query-get-total itself is pure because all it does when called is return another function, but the function that it returns is impure.

I do think the benefits of that definition of "pure" are maybe a little less? A pure function that returns something impure still seems more difficult and more error prone to use then one that is pure and return a pure thing. Same for a pure function which takes an impure input, that's more challenging to use properly then one that takes pure input.

Does missionary helps you manage the impure aspects? Does it mean with missionary I no longer need to try and maximize how much logic I put in pure functions that return pure things? But can instead sprinkle side-effect wherever?

I'm not sure I see as well how you'd test such a function? Since the task is opaque, how can I assert that it created the correct task?

I do think the benefits of that definition of "pure" are maybe a little less? A pure function that returns something impure still seems more difficult and more error prone to use then one that is pure and return a pure thing. Same for a pure function which takes an impure input, that's more challenging to use properly then one that takes pure input.

That's my understanding too. The impression I got so far is that managing side effects is not one of missionary's goals.

I do think the benefits of that definition of "pure" are maybe a little less? A pure function that returns something impure still seems more difficult and more error prone to use then one that is pure and return a pure thing. Same for a pure function which takes an impure input, that's more challenging to use properly then one that takes pure input.

TBH I don't think this definition of purity is even remotely controversial. Haskellers do functional composition of impure functions all the time, and AFAIK they are rather concerned about purity.

I'm not sure I see as well how you'd test such a function? Since the task is opaque, how can I assert that it created the correct task?

I think the concern you're pointing out is explicit data vs opaque objects, you want to be able to inspect the value. I don't deny the benefits of this approach, however if you push it to the end you will end up throwing away the idea of compilers in favor of interpreters, and this will badly impact performance. You could argue that functions are bad because they're opaque, and you might want to replace them with quote + eval, and you would get inspection, serialization, full control over the evaluation strategy etc. All these benefits are gained at the cost of performance, so it's always a tradeoff. Also keep in mind at some point you need to interop with the host, which is opaque.

Does missionary helps you manage the impure aspects?

The impression I got so far is that managing side effects is not one of missionary's goals.

It's pretty clear that effect management doesn't have the same meaning for you than for me. Could you help me understanding better your expectations from an effect system ? Even would we end up disagreeing, I'd like to be more explicit about goals and non-goals.

Does it mean with missionary I no longer need to try and maximize how much logic I put in pure functions that return pure things? But can instead sprinkle side-effect wherever?

That's a good question, actually. It would be interesting to know why it was a goal in the first place.

That's a good question, actually. It would be interesting to know why it was a goal in the first place.

I can only speak for me, but I find it's about coupling and reuse. A function with pure logic, pure input and pure output is decoupled from everything else, and so it is most simply reused. It also has the least gotchas to using it, there are no outside knowledge needed.

I would even go one further, and say the input shouldn't just be pure, but it should be modeled in terms of the function, that gives you even less coupling, as you're not even coupled to an external structure definition.

It is also easy to test, since you can test it like a black box, assert the mapping of output to input is as you want, and assert that the properties of the mapping you want are maintained over the input range. And since it's decoupled completely from its environment, once it has been tested and demonstrated correct, you just know it works.

I think this is best summarized simply as these type of functions are inherently more modular. I like this quote:

Modular design brings with
it great productivity improvements. First of all, small modules can be coded
quickly and easily. Secondly, general purpose modules can be re-used, leading to
faster development of subsequent programs. Thirdly, the modules of a program
can be tested independently, helping to reduce the time spent debugging.

From: http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html

Now I really like what I've seen of missionary for describing flow charts that model processes, especially if you need concurrency I think it does a very interesting job on that, and the supervision it brings makes it nice to handle error scenarios and edge cases from effectful behavior. So I think it's a great tool to help manage effects in a system, especially a concurrent one.

What I'm not sure is if that mitigates the benefits of the pure+ functions I describe. Or if one should still strive to move as much logic to pure+ function, and then missionary can help with the parts that are unavoidable related to gluing together effects and pure+ functions, or if somehow it invalidates the need for pure+ functions in the first place.

TBH I don't think this definition of purity is even remotely controversial. Haskellers do functional composition of impure functions all the time, and AFAIK they are rather concerned about purity.

I've actually always had this concern with Haskell, and even with some Clojure patterns I've seen, where coupling is still introduced between the functions of the program, but it is hidden behind the veneer of purity. If all my functions pass around state only so it reaches some other functions down the line, or if I've got every function using an IO monad, I feel you've still increased coupling, limited reuse, and have hurt your ability to reason about each function independently of others or the environment they are in, even if each function is technically pure.

I think the concern you're pointing out is explicit data vs opaque objects, you want to be able to inspect the value. I don't deny the benefits of this approach, however if you push it to the end you will end up throwing away the idea of compilers in favor of interpreters, and this will badly impact performance

That's fair, I might be curious though if a hybrid model couldn't be achieved. Could I call the sp wrapped function in a unit test where it would return an interpreted version that I can inspect for correctness. But in production it would return a Task as normal?

It's pretty clear that effect management doesn't have the same meaning for you than for me. Could you help me understanding better your expectations from an effect system ? Even would we end up disagreeing, I'd like to be more explicit about goals and non-goals.

I apologize for the confusion. I was trying to understand missionary's scope because I heard someone saying somewhere that missionary helps with making otherwise impure functions into pure functions, or something along these lines, and that kind of intrigued me.