stepwise_v2 interface doodle

stepwise2.clj

;; defining a Step Functions state machine

(def pizza-making-state-machine
  (sfn/->> request
           (sfn/parallel (make-dough)
                         (make-sauce)
                         (sfn/map {:iterate-over :ingredients} prepare-ingredients))
           (put-ingredients-on-dough)
           (bake)
           (sfn/wait 2 :minutes)
           (sfn/choice (comp not is-pizza-acceptable?)
                       ;; branch off to this fn if condition is true
                       (sfn/fail))
           (serve)))

;; Apply the defined state machine on AWS
(sfn/ensure-state-machine client :pauls-pizza-making-machine pizza-making-state-machine)


;; defining workers as clojure fns

(defn make-dough
  ;; SFN error handling configuration as metadata
  {:retry [{:error-equals     [:sfn.timeout]
            :interval-seconds 60
            :max-attempts     3}]}
  [coll]
  coll)

(defn put-ingredients-on-dough [coll] coll)

(defn bake [coll] coll)

(defn make-sauce [coll] coll)

;; This is a special predicate fn that gets applied with Choice.
;; There's really not much that can be done here but I don't want
;; to fallback to SFN syntax.
(sfn/def-choice-predicate-fn is-pizza-acceptable? [m]
  ;; syntax to be determined
  true)

;; choose where the workers run

(sfn/run-here client
              {make-dough               {:concurrency 2}
               put-ingredients-on-dough {:concurrency 1}
               bake                     {:concurrency 1}})


(sfn/run-on-lambda client
                   {make-sauce {:timeout 40
                                :memory-size 512
                                :max-concurrency 50}})

Thoughts on some internals

Between Steps Data Format

SFN relies on JSON format to pass between states. It would be nice to keep it readable
and for special flow that inspects the payload like Choice. Perhaps Transit format is workable for Choice and sort of human-readable enough?

Large Payload

SFN has a 256kb payload limit. We could offload to S3 as we do in Stepwise v1 already.
But how to make that auto-magical without obfuscation? i.e. automatically offload the
whole payload or only the values so we can still see the key names in Execution logs?