Shake notes

Each key gets intern'd into an Id


Key :-> Id

Id :-> Status

data Status
    = Ready Result -- ^ I have a value
    | Error SomeException -- ^ I have been run and raised an error
    | Loaded Result -- ^ Loaded from the database
    | Waiting Pending (Maybe Result) -- ^ Currently checking if I am valid or building
    | Missing -- ^ I am only here because I got into the Intern table
      deriving Show

newtype Pending = Pending (IORef (IO ()))
    -- you must run this action when you finish, while holding DB lock
    -- after you have set the result to Error or Ready

data Result = Result
    {result :: Value -- ^ the result associated with the Key
    ,built :: {-# UNPACK #-} !Step -- ^ when it was actually run
    ,changed :: {-# UNPACK #-} !Step -- ^ the step for deciding if it's valid
    ,depends :: [[Id]] -- ^ dependencies (don't run them early)
    ,execution :: {-# UNPACK #-} !Float -- ^ how long it took when it was last run (seconds)
    ,traces :: [Trace] -- ^ a trace of the expensive operations (start/end in seconds since beginning of run)
    } deriving Show

-- | Return either an exception (crash), or (how much time you spent waiting, the value)
build :: Pool -> Database -> Ops -> Stack -> [Key] -> Capture (Either SomeException (Seconds,Depends,[Value]))


first, look up the set of keys, and transform them into Id

for each Id
    If Ready or Error, continue immediately
    If Waiting, wait for them
    If Missing, start building them
    If Loaded, move them to Waiting



data Status = Ready | Error | Loaded |  Waiting | Missing

Given ks:

is <- mapM lookupId ks
ss <- mapM lookupStatus is

if any isError ss then
    continue Error
if all isReady ss then
    continue Ready
for all waiting, spawn them into waiting
for any that are missing or loaded, spawn them into a waiting
wait for all the waitings, if any error becomes 


waitFor :: Id -> (Status -> IO ()) -> IO ()



build :: ks -> (Either Exception [Value] -> IO ()) -> IO ()
build ks continue = do
    is <- lookupId ks
    waitFor is continue

waitFor :: [Id] -> (True -> Either Exception [Result] -> IO ()) -> IO ()
waitFor is continue = do
    ss <- mapM lookupStatus is
    if any isError ss then
        continue True Error
    if all isReady ss then
        continue True Ready
    let rest = non-error/ready
    forM rest $ \i -> spawn rest $ \c ->
        if all done then addPool $ continue ...
        if any error then addPool $ continue 
        otherwise wait

spawn :: Id -> (Status -> IO ()) -> IO ()
spawn i = do
    s <- lookupStatus i
    if error or ready then call s
    if waiting then add to wait q
    if missing then turn into waiting and build
    if loaded then waitFor all the children, then continue

    for all waiting, spawn them into waiting
    for any that are missing or loaded, spawn them into a waiting
    wait for all the waitings, if any error becomes 



Transition rule system!

Waiting => 

Loaded => Waiting

Missing => Waiting

Error => Left


all Ready => Right

to build [Key], first map to [Id]

to build [Id]
    if all are Result, return Right
    if any are Error, return Left
    if any are Loaded/Missing turn them into Waiting
    as Waiting ones finish, observe their Result or Error

Missing => Waiting is just run quickly

Loaded => do
    Ask about the list of list of children
    if Error, rebuild
    if Right, continue

in the new system, Loaded CAN ONLY TRANSITION TO Waiting, NEVER anything else

if any waiting becomes 


-- either result me quickly to a Result or Error, or instead give me a way to produce a Waiting value
reduce :: Id -> IO (Either ResultError (IO Waiting))



collapse :: [Id] -> Either SomeException [Result]
    if any error then error
    if all result then results
    otherwise turn them all into waiting

waiting :: Id -> Waiting
waiting Waiting = done
waiting Missing = spawn
waiting Loaded =
    collapse each of the depends in turn
    then run the main function


Pool should have two distinct piles
    start something new
    
----------------------------------------------------------------------


Lint is just an entire run but in a different type of mode?
Any change that requires writing to the database should go boom.

Assumptions are pushed into the rules



AssumeDirty rebuilds 

----------------------------------------------------------------------


imagine all Loaded and Missing became Waiting automatically


ensure :: [Id] -> Either SomeException [Result]
ensure is = do
    
    if all ready then
        Right
    else if any error then
        Error
    else
        ws <- waiting vs
        apply predicate

WHAT OPTIMISATIONS ARE IN PLAY:

* If everything is already Ready, skip rereading it
* If one is error then raise the error immediately
* If one has changed start rebuilding immediately
* Sticking to the same thread if possible


Free monad and interpreter?

Threading through waiting and spawning to Pool is fiddly


MainThread Monad? (just newtype over Identity)