mike-thompson-day8/Booting.md

## Booting.md

      
    Raw
  

              Booting.md
            
          
This Gist is out of date. It has been superseded by a real repo:

https://github.com/Day8/re-frame-async-flow-fx

Coordinating App Startup

The problem

When an App boots, it performs a set of tasks to initialise itself.
Invariably, there are dependencies between these tasks, like task1 has to run before task2.
Because of these dependencies, something has to coordinate how tasks are run.  Within the clojure community,
a library like Stuart Sierra's Component or mount
is often turned to in these moments, but we won't be
doing that here. We'll be using an approach which is more re-frame friendly.
Easy

If the tasks are all synchronous, then the
coordination can be done in code.
Each task is a function, and  we satisfy the task dependencies by correctly
ordering how they are called.
In a re-frame context, we'd have this:
(register-handler
  :initialise
  (fn [db]
    (-> {} 
        task1-fn 
        task2-fn
        task3-fn)))

and in our main function we'd (dispatch [:initialise])
Harder

But, of course, it is never that easy because some of the tasks will be asynchronous.
A booting app will invariably have to
coordinate asynchronous tasks like  "open a websocket", "establish a database connections",
"load from LocalStore",
"GET configuration from an S3 bucket" and "querying the database for the user profile".
Coordinating asynchronous tasks means finding ways to represent and manage time,
and time is a programming menace.
In Greek mythology,  Cronus was the much feared Titan of Time, believed to
bring cruelty and tempestuous disorder, which surely makes him the patron saint of asynchronous programming.
Solutions like promises and futures attempt to make time disappear and allow you to program
with the illusion of synchronous computation. But time has a tendency to act like a liquid under
pressure, finding the cracks and leaking through the abstractions.
Something like CSP (core.async) is
more of an event oriented treatment. Less pretending. But... unfortunately more complicated.
core.async builds a little state machine for you, under the covers, so that you can
build your own state machine on
top of that again via deft use of go loops, channels, gets and puts. Both layers try to model/control time.
In our solution, we'll be using a re-frame variation which hides (most of) the
state machine complexity.
Harder Again

There'll also be failures and errors!
Nothing messes up tight, elegant code quite like error
handling.  Did the Ancient Greeks have a terrifying Titan for the unhappy
path too? They should have.
When one of the asynchronous startup tasks fails, we
must be able to stop the normal boot sequence and put the application
in a satisfactory failed state, explaining to the user
what went wrong (Eg: "No Internet connection" or "Couldn't load user portfolio").
Just Plain Hard

And then there's the familiar pull of efficiency.
We want our app to boot in the shortest possible amount of time. So any asynchronous
tasks which could be done in parallel, should be done in parallel.
So the boot process is seldom linear, one task after an another. Instead, it involves

dependencies like:  when task task1 has finished, we can start task2, task3 and task4 in
parallel.  But task5 can't start until both task2 and task3 has completed successfully.
And task6 can start when task3 is done, but we really don't care if it finishes
properly - it is non essential to a working app.
So, we need to coordinate asynchronous timelines, with complex dependencies, while handling failures.

Not easy, but that's why they pay us the big bucks.
Also, As Data

Because we program in Clojure, we spend time in hammocks watching Rich Hickey videos and
meditating on essential truths like "data is the ultimate in late binding".
So our solution should involve "programming with data" and be, at once, all synonyms of easy.
The  Solution

re-frame has events. That's how we roll.
A re-frame application can't step forward in time, unless an event happens; unless something
does a dispatch.  Events will be the organising principle in our solution exactly
because events are an organising principle  in re-frame itself.
Tasks and Events

Our solution assumes the following about tasks...
If we take an X-ray of an async task, we'll see this event skeleton:

an event is used to start the task
if the task succeeds, then an event is dispatched
if the task fails, then an event is dispatched

So that's three events: one to start and two ways to finish.
Of course, re-frame will route all three of these events to a registered handler. The actual WORK of
starting the task, or handling the errors, will be done in the event handler that you write.
But, here, none of that menial labour concerns us. We care only about the COORDINATION of
tasks. We care only that task2 is started when task1 finishes successfully, and we don't need
to know what task1 or task2 actually do.
To distill that: we care only that the dispatch to start task2 is fired correctly when we
have seen an event saying that task1 finished successfully.
When-E1-Then-E2

Read that last paragraph again.  It further distills to: when event E1 happens then dispatch event E2.  Or, simply, When-E1-Then-E2.
When-E1-Then-E2 is the simple case, with more complicated variations like:

when both events E1 and E2 have happened, then dispatch E3
when either events E1 or E2 happens, then dispatch E3
when event E1 happens, then dispatch both E2 and E3

We call these "rules".
Rules As Data

Collectively, a set of these When-E1-then-E2 rules can describe the entire boot sequence of an app.
So, here's how we might capture an app's boot sequence in data:
[{:when :seen-all-of :events #{:success-db-connect}   :dispatch (list [:do-query-user] [:do-query-site-prefs])}
 {:when :seen-all-of :events #{:success-user-query :success-site-prefs-query}   :dispatch (list [:success-boot] :done)}
 {:when :seen-any-of :events #{:fail-user-query :fail-site-prefs-query :fail-db-connect} :dispatch  (list [:fail-boot] :done)}
 {:when :seen-all-of :events #{:success-user-query}   :dispatch [:do-intercom]}]

That's a vector of 4 maps, where each represents a single rule. Try to read each
line as if it was an English sentence and something like this should emerge: when we have seen all of events E1 and E2, then dispatch this other event
The structure of each rule (map) is:
{:when     X      ;; one of:   :seen-all-of  or :seen-any-off
 :events   Y      ;; a set of one or more event ids
 :dispatch Z}     ;; either a single vector (to dispatch) or a list of vectors (to dispatch). :done is special

We can't issue any database queries until we have a database connection, so the 1st rule (above) says:

When :success-db-connect is dispatched, presumably signalling that we have a database connection...
then (dispatch [:query-user])  and (dispatch [:query-site-prefs])

If both database queries succeed, then we have successfully booted, and the boot process is over. So, the 2nd rule says:

When both success events have been seen (they may arrive in any order),
then (dispatch [:success-queries]) and cleanup because the boot process is :done.

If any task fails, then the boot fails, and the app can't start. So go into a failure mode. And the boot process has done. So the 3rd rules says:

If any one of the various tasks fail...
then (dispatch [:fail-boot]) and cleanup because the boot process is :done.

When we have user data (from the user-query), we can start the intercom process. So the 4th rules days:

When :success-user-query is dispatched
then  (dispatch [:do-intercom])

Further Notes:


The 4th rule starts "intercom" once we have completed the user query. But notice that
nowhere do we wait for a :success-intercom.  We want this process started,
but it is not essential for the app's function, so we don't wait for it to complete.


The coordination processes never actively participates in handling any events. Event handlers
themselves do all that work. They know how to handle success or failure - what state to record so
that twirly things are shown to users, or not. What messages are shown. Etc.


A naming convention for events is adopted. Each task can have 3 associated events which
are named as follows: :do-* is for starting tasks. Task completion is either :success-*
or :fail-*


A dispatch of :done means the boot process is completed.  Clean up the coordinator.
It will have some state somewhere. So get rid of that.  And it will be "sniffing events"


There's nothing in here about the teardown process at the end of the application.  We're only
helping the startup process.


To start the boot process  (dispatch [:do-connect-db])


A word on Retries XXXX


4 Step Solution

There's 4 steps to creating an async boot process for your app.  But steps 2, 3 and 4 are
very simple boilerplate. (XXX How can that even be removed?)
You really only need to do step 1.
Here a quick look at the overall strategy:

You write a "boot-spec" which describes in data, the required boot sequence (in terms of events)
You create a single event handler which will "coordinate" the necessary async events (using the "boot-spec" you created in step 1).
you will create the single event handler using a function that I supply.  You can treat this event handler as a black box.
This event handler will "sniff" events. Events will be "forwarded" to it, so it knows what's going on, so it can do its coordination job (centrally).
This event handler knows (via your boot-spec) that when it sees E2 it should dispatch E3. Etc.
At some point, the boot has happened (failed or succeeded) and the coordinator is decommissions (which is to say, events are no longer forwarded to to event handler).

Anyway, it is a four step process. But, remember, only step 1 is a challenge.  The other steps are pretty paint-by-numbers.
Step 1.  Define the Boot Spec

Above, I gave just the :rules part of the spec.  Here it is in full (two extra bits)
(def boot-spec
 {:db-path  [:place :to :store :state :within :db]     ;; the coordinator needs to keep some state. Where?
  :first-dispatch [:do-db-connection]                  ;; how does the process get kicked off?
  :rules [{:when :seen-all-of :events #{:success-db-connect}   :dispatch (list [:do-query-user] [:do-query-site-prefs])}
         {:when :seen-all-of :events #{:success-user-query :success-site-prefs-query}   :dispatch (list [:success-boot] :done)}
         {:when :seen-any-of :events #{:fail-user-query :fail-site-prefs-query :fail-db-connect} :dispatch  (list [:fail-boot] :done)}
         {:when :seen-all-of :events #{:success-user-query}   :dispatch [:do-intercom]}])

Step 2.  Register an event handler which does all the coordinating work

(register-event-fx
   :boot-async
   (make-bootstrap-handler boot-spec))

The function make-bootstrap-handler is provided as step 4.  We call it, providing a spec and it returns a handler
which does the work.
Any events mentioned in the
This event handler will be forwarded to when any of the events in the "boot-spec"
Step 3. Kick off the process

The initialisation process involves first doing the synchronous (easy) stuff, and then kicking off the async tasks
(register-event-fx
  :initialise
  (fn [context]
    {:db (-> {} task1-fn task2-fn)       ;;  do whatever synchronous work needs to be done
     :dispatch [:boot-async :start]}))   ;; dispatch to the handler created in step 2 

We assume that the synchronous part will set state that causes the GUI to render "Loading ..."  or something.
In your main function, you'll be doing a (dispatch [:initialise]).  That's not included as a separate step.
Step 4. Include This Code  (Library?)

Below is some code which you'll have to include in your application.
It provides make-bootstrap-handler.
The following assumes the existence of certain v0.8.0 features (eg the ability to forward events, aka "sniff events").
Note, in theory you don't actually have to understand ANY of this function. It does the grunt work,
but you don't have to understand it.  It implements your "boot-spec".

WARNING THIS CODE DOES NOT WORK YET  -- IN FACT I KNOW IT IS CURRENTLY BROKEN
I wouldn't even try reading it yet. And anyway, this function will be a black box to anyone using it.

(defn make-bootstrap-handler
  [{:as boot-spec :keys [db-path rules first-dispatch]}]
  (let [all-events  (apply set/union (map :events rules))   ;; all of the events refered to in the spec
        initial-state  {:seen-events   #{}
                        :started-tasks #{}}]
    ;; check that boot-spec is valid.

    ;; return an event handler
    ;; This event handler will receive one of three events
    ;;   (dispatch [:id :steup])
    ;;   (dispatch [:id :done])
    ;;   (dispatch [:id [:in :here :goes :a :forwarded :event vector])
    ;;
    (fn [context [event-id forwarded-event]]
      (let [context-path (cons :db db-path)]
        (cond = forwarded-event

              ;; Setup the boot coordinator:
              ;;  1. Initialise the state  (seen-events and started-tasks)
              ;;  2. dispatch the first event, to kick start
              ;;  3. arrange for the events to get forwarded to the handler
              :setup (-> context
                         (assoc-in context-path initial-state)
                         (assoc :dispatch first-dispatch)
                         (assoc :event-forwarder {:register event-id
                                                  :events   all-events
                                                  :to       event-id}))
              ;; Teardown the boot coordinator:
              ;;  1. remove of state
              ;;  2. stop the events forwarder
              :done (-> context
                        (dissoc context-path)
                        (assoc :event-forwarder {:unregister event-id}))

              ;; Process a forwarded event
              ;; An event has been forwarded to this handler. Figure out what it means.
              ;;  1. work out of this new event means we need to dispatch another
              ;;  2. remember the new state
              (let [_   (assert (vector? forwarded-event))
                    forwarded-event-id     (first forwarded-event)
                    {:keys [seen-events started-tasks]} (get-in context context-path)
                    new-seen-events    (conj seen-events forwarded-event-id)
                    ready-tasks    (startable-tasks rules new-seen-events started-tasks)
                    ready-task-ids (->> ready-tasks (map :dispatch) set)
                    new-started-tasks (set/union started-tasks ready-task-ids)]
                (-> context
                    (assoc :dispatch (map #(conj (:dispatch %) forwarded-event) ready-tasks))
                    (assoc-in context-path {:seen-events   seen-events
                                            :started-tasks new-started-tasks}))))))))