up-forward-toward-%ford-0

up-forward-toward-%ford-0.md

up 1: forward %ford, toward 0k

up: 1
authors: [~rovnys-ricfer(ted@tlon.io), ~pittyp-datfun(anton@tlon.io)]

overview

%ford is urbit's build system. It's one of the seven kernel modules (vanes) in the kernel (arvo). In this document we propose a new interface for %ford intended to address the following issues with the system:

• horns (%ford runes) are arcane, implicit, and the direction of control flow varies from rune to rune.
• The only non-trivial renderer, tree, is split up into eight small files due to limitations of the runes and rendering system.
• The %ford request and result card types have a clay-specific complication (the $tabl subtype of silk and gage).
• Instead of %ford sending notifications to the subscribers on a build when the build is invalidated (which occurs when one or more of the build's dependencies change), it sends the client a hash of the dependencies in its response to the build, to which the client can subscribe to listen for changes. Because a client might subscribe to changes on a hash at any time, %ford must hold onto the mapping from hash to dependencies forever. In practice, this is a space leak.
• %ford doesn't know the difference between a reactive build and a one-off build. This means we need to register dependencies even if a client only needed a build tied to a particular %clay revision and did not want to receive reactive updates.
• %ford performs unnecessary work when a build depends on intermediate steps whose outputs haven't changed since the previous build. For example, if a sidebar on a webpage depends on the titles of all sibling pages but not their contents, if one of those siblings changes but its title does not, then the webpage should not be rebuilt.
• %ford has no cache replacement algorithm. Therefore, old builds are kept indefinitely and are not evicted from cache based on frequency or recency of access. There is currently a band-aid deployed that wipes the %ford cache entirely on a fixed timer. This cache wiping causes latency spikes that are sometimes so severe that they become errors ... the 504 errors that everyone in the urbit community is familiar with. These 504s were the impetus for examining %ford critically, which has led us here.

specification

%ford 0 (kelvin version 0) shall have the following API, defined in zuse.hoon, the kernel standard library:

::
::::    %ford
  ::
++  hood                                                ::  assembly plan
  $:  zus/@ud                                           ::  zuse kelvin
      sur/(list hoof)                                   ::  structures
      lib/(list hoof)                                   ::  libraries
      fan/(list horn)                                   ::  resources
      src/(list hoop)                                   ::  program
  ==                                                    ::
++  hoof  (pair term (unit (pair case ship)))           ::  resource reference
++  hoop                                                ::  source in hood
  $%  {$& p/twig}                                       ::  direct twig
      {$| p/beam}                                       ::  resource location   
  ==                                                    ::
++  hops   {pre/(unit tyke) pof/(unit {p/@ud q/tyke})}  ::  XX late-bound path
++  horn                                                ::  resource tree 
  $%  {$fssg p/twig}                                    ::  /~  twig by hand 
      {$fsbc p/twig}                                    ::  /$  argument 
      {$fszp q/mark}                                    ::  /!mark/ dyn resource
      {$fszy q/mark}                                    ::  /mark/ resource
    ::
      {$fscn p/horn}                                    ::  /%  propagate args
      {$fscb p/horn}                                    ::  /_  homo map
      {$fsts p/term q/horn}                             ::  /=  apply face
      {$fshx p/term q/horn}                             ::  /#  named sub-build
      {$fssm p/twig q/horn}                             ::  /;  operate on
      {$fscl p/hops q/horn}                             ::  /:  relative to
      {$fskt p/twig q/horn}                             ::  /^  cast
      {$fspm p/(list mark) q/horn}                      ::  /&  translates
    ::
      {$fsdt p/(list horn)}                             ::  /.  list
      {$fsbr p/(list horn)}                             ::  /|  options
      {$fscm p/(list (pair path horn))}                 ::  /,  switch by path
  ==                                                    ::
++  milk  (trel ship desk silk)                         ::  sourced silk
++  silk                                                ::  construction layer
  $^  {p/silk q/silk}                                   ::  cons
  $%  {$$ p/cage}                                       ::  literal
      {$alts p/(list silk)}                             ::  options
      {$bake p/mark q/coin r/beam}                      ::  local synthesis
      {$call p/silk q/silk}                             ::  slam
      {$cast p/mark q/silk}                             ::  translate
      {$core p/beam}                                    ::  build program
      {$diff p/silk q/silk}                             ::  diff
      {$dude p/(trap tank) q/silk}                      ::  error wrap
      {$file p/beam}                                    ::  from clay
      {$flag p/(set $@(@uvH beam)) q/silk}              ::  add dependencies
      {$join p/mark q/silk r/silk}                      ::  merge
      {$mash p/mark q/milk r/milk}                      ::  annotate
      {$mute p/silk q/(list (pair wing silk))}          ::  mutant
      {$posh p/silk q/silk}                             ::  patch
      {$plan p/beam q/coin r/hood}                      ::  structured assembly
      {$ride p/twig q/silk}                             ::  silk thru twig
      {$vale p/mark q/*}                                ::  validate
      {$volt p/(cask *)}                                ::  unsafe add type
  ==                                                    ::
::::
++  bilk  (pair beak silk)                              ::  sourced request
++  gage                                                ::  autocons cage+tang
  $^  (pair gage gage)
  (each cage tang)
++  gift-ford                                           ::  out result <-$
          $%  {$made p/gage}                            ::  computed result
              {$mass p/mass}                            ::  memory usage
              {$news p/(set term)}                      ::  $pact update
          ==                                            ::
++  kiss-ford                                           ::  in request ->$
          $%  {$exec p/(unit bilk)}                     ::  make/kill
              {$pact p/(unit bilk)}                     ::  (make&subscibe)/kill
              {$wegh $~}                                ::  report memory
              {$wipe $~}                                ::  clear cache
          ==                                            ::

TODO: cache replacement ++kiss?

The major differences here are in the ++gift and ++kiss interfaces. Also, the ++gage type has been simplified and the $tabl silk has been removed.

rationale

The new ++gift and ++kiss interfaces implement two workflows:

• "immutable," "nonreactive," or "one-off" builds:
  • %ford receives an $exec card containing a request to build a ++silk, so it begins the build. It does not register this build's dependencies to be tracked, and does not subscribe to %clay or any other reactive data source for updates.
  • When the build is complete, %ford responds with a $made gift containing the completed build.
• "mutable," "reactive," or "subscription" builds:
  • %ford receives an $exec card containing a request to build a ++silk, so it begins the build. It registers this build's dependencies to be tracked, stores them in its permanent state, and subscribes to %clay and possibly other reactive data sources.
  • When the build is complete, %ford responds with a $made gift containing the completed build.
  • Each time the build's dependencies update, the reactive data source for the build (usually a %clay file or directory but possibly a %gall app or something else) will respond to %ford notifying it of the update. %ford will then "greedily" re-run the build using the new dependencies, then send out a $news gift to the client that requested the build. If the build had any named sub-builds, the names of the updated sub-builds will be in the $news card as a set of terms. The client, upon receiving this card, may request a new build from %ford (usually at the new %clay revision that triggered the update), and that build will already be cached, except in the unlikely event that %ford was asked to wipe its cache to free memory in between sending the $news card and the client requesting the new build.

integration plan