Skip to content

Instantly share code, notes, and snippets.

@belisarius222
Last active March 17, 2020 22:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save belisarius222/dc2aae8230a22a84d389c19cb613a1d2 to your computer and use it in GitHub Desktop.
Save belisarius222/dc2aae8230a22a84d389c19cb613a1d2 to your computer and use it in GitHub Desktop.
Ford Pinto Spec

Ford Pinto - Smaller and Less Explosive

namesake

Overview and Rationale

Ford Turbo, the current incarnation of Ford, has some exciting features. In retrospect some of those features were the result of irrational exuberance. Many have been in Ford since the beginning.

Ford Pinto is an attempt to take a more measured look at which features Ford actually needs and to support only those features. The overall plan involves passing through Ford Pinto into Ford Focus, which will include performance tuning targeted at the major Ford use cases.

What is Ford actually used for? Building files, and doing mark operations, basically. The primary simplification I'm making to Ford in this revision is to separate those two interfaces. Ford runes other than simple imports have been removed, and there is now a strict layering where building a source file does not depend on mark builds.

I've made several other changes to Ford to ease implementation, maintenance, usage, and performance tuning.

Interface Specification

+$  ford
  |%
++  able
  |%
  ::  $task: request to ford
  ::
  ::    %make: submit a build request
  ::     desk: filesystem context
  ::     case: ~ if live, or pinned filesystem version
  ::      fiz: hoon files to build
  ::      maz: marks to build into $dais's
  ::      caz: mark conversion gates to construct
  ::    %drop: cancel build
  ::
  +$  task
    $%  $:  %make
            =desk
            case=(unit case:clay)
            fiz=(set path)
            maz=(set mark)
            caz=(set [a=mark b=mark])
        ==
        [%drop ~]
    ==
  ::  $gift: response from ford
  ::
  ::    %made: build result
  ::     cass: version at which this result was built
  ::      fiz: (re)built hoon files
  ::      maz: (re)built mark $dais's
  ::      caz: (re)built mark conversion gates
  ::
  +$  gift
    $%  $:  %made
            =cass:clay
            fiz=(map path (each vase tang))
            maz=(map mark (each dais tang))
            caz=(map [a=mark b=mark] (each $-(vase vase) tang))
    ==  ==
  --
  ::  $dais: processed mark core
  ::
  +$  dais
    $_  ^|
    |_  sam=vase
    ++  bunt  sam
    ++  diff  |~(new=_sam [form=form diff=*vase])
    ++  form  *mark
    ++  join  |~([a=vase b=vase] *(unit [form=_form diff=vase]))
    ++  mash  |~([a=[ship desk diff=vase] b=[ship desk diff=vase]] diff=*vase)
    ++  pact  |~(diff=vase sam)
    ++  vale  |~(noun sam)
    ++  volt  |~(noun sam)
    --
  ::  $pile: preprocessed hoon source file
  ::
  +$  pile
    $:  =mont
        gen=hoon
    ==
  ::  $mont: imports declaration
  ::
  ::    /-  imports from /sur
  ::    /+  imports from /lib
  ::    /=  imports from some other path, wrapping face around result
  ::
  +$  mont
    $:  sur=(list taut)
        lib=(list taut)
        raw=(list [=face =path])
    ==
  ::  $taut: /lib or /sur file import
  ::
  +$  taut  [face=(unit term) pax=term]
  --

Interface Notes

Ford's interface has been segregated into %make builds that build a Hoon source file, %cast builds that produce gates for casting from one mark to another, and %mark builds that produce $dais data structures representing compiled mark cores.

Note that now, instead of performing mark operations itself, Ford now only produces a conversion gate or $dais data structure for a mark, and the client will perform the operations itself using that result. One pleasant result of this change is that there are no longer any vases as inputs to Ford builds.

Ford runes have been removed due to popular demand and in an attempt to achieve better layering; now a Hoon file can import a /sur file with /-, a /lib file with /+, and an arbitrary Hoon file with /=. We might add in some Ford runes later as we need them; it won't break the model.

A build is scoped to a desk, within the current ship. Resource dependencies can only be expressed within the current beak. Blocking .^ will no longer be supported. Some other mechanism will need to be written to fetch remote Clay hashes -- it could be moved into :hood, for example. I think Dojo should get special syntax for it so an async Clay scry can be used as a Dojo source.

Errors are not referentially transparent, just like in old (pre-Turbo) Ford. If a build doesn't succeed, Ford does not specify whether the build might succeed if run again. No clients ever made use of that information from Ford Turbo, and it's not clear how they would.

Since a %make build asks for a set of files to be built, the response is a map from filepath to result, where the result is (each vase tang). The first result of a live %make build contains all results, and then each subsequent result contains only files and results where the results differ from the previous iteration. This pattern, where the live result map only contains items that have changed since the previous iteration of the build, is shared by the %mark and %cast builds.

A %mark build result is a map from mark name to (each dais tang). A $dais is a data structure representing a processed mark coret that provides enough information to easily bunt the mark, validate a noun to this mark, or perform a version-control operation: +diff, +join, +mash, or +pact.

A %cast build result is a map from a pair of marks to a conversion gate that takes in a vase of a datum to convert and outputs a vase of the result. An alternative design would produce a vase of a statically-typed gate, but we think this will be more convenient for clients.

Every Ford build must be able to complete in a single Arvo event. Ford can send moves to Clay repeatedly, but it can't block Clay's commit progress. No remote resources, resources from the future, resources from other vanes, or other zany features are supported.

Another way to express this constraint is to say that if Ford and Clay were part of the same vane (and maybe they should be, eventually), a live Ford build must be able to execute synchronously during one activation of that vane.

Ford Turbo tried pretty hard to be a function of the whole scry namespace, at least internally. This never really panned out, and I don't think it's actually desirable. I think we'll all be better off admitting that Ford should only depend on Clay.

Previous Fords have had to do a fair amount of asynchronous work and caching in order to canonicalize Clay paths in /ren, /mar, /lib, and /sur to replace every fas with a hep. This is too specific and introduces unnecessary complexity.

Ford Pinto will scry Clay for the canonicalized version of a /lib, /sur, or /mar path; Clay will assert on insertion that a new /lib or /sur file's canonical path does not conflict with that of an existing file.

Error Handling

Execution of a build could be deferred into its own event for error containment and provenance. If a once build produces a deterministic error, Ford Pinto captures the error trace and sends it back to the requester as a build failure. If the once build crashes nondeterministically, when Ford is woken up with the error notice, it will produce a failure for that build, containing the error report.

For atomicity, Clay probably should not defer Ford builds. Userspace builds might want to, though, so that the agent that requested the build knows it was Ford that crashed, not its own internals. Gall could potentially intercept Ford requests from agents and defer them to guarantee this property.

Internals

If Ford Pinto weighs in at more than 2.000 lines of code, there might be something wrong.

The internals use monadic code for the internal build logic to increase code density and help prevent bugs. Hopefully this does not increase Ford's compile time to an unacceptable level... it seems fine so far.

Ditching renderers and other Ford runes turns Ford from a monadic build system to an applicative build system, which should aid in analyzing and caching builds.

@belisarius222
Copy link
Author

It might be better to allow a case:clay in the %once request. It's more explicitly referentially transparent, and there's no reason Ford shouldn't be able to request older Clay files and build them. This would actually let you request future resources, but Ford would just block until that resource exists. Builds at any revision other than the current one are likely to be very slow, but I have a feeling looking up older data will always be slower than looking up current data, since we'll optimize the common path of current data.

@tacryt-socryp
Copy link

tacryt-socryp commented Mar 6, 2020

This is everything I've ever wanted but didn't know how to ask for 😭😲🤩 I love everything about it. I'm so happy

@liam-fitzgerald
Copy link

The language server currently relies on the %plan schematic. Is there any way to duplicate this functionality with ford pinto?

@belisarius222
Copy link
Author

Hmm, what does the language server use it for? Pinto could probably expose something similar.

@liam-fitzgerald
Copy link

https://github.com/urbit/urbit/blob/a21901494992ff4dafe5329f342a13cf69ab9840/pkg/arvo/app/language-server.hoon#L290

Basically to get a subject with all the dependencies in it. We already parse the the scaffold separately, and because the prelude is a lot simpler we could always just =>(dep1, dep2...).

@sigilante
Copy link

sigilante commented Mar 7, 2020

What happens to horns and twigs? Do they persist internally or are you recomposing with new data structures?

I mean, I guess twigs stick around, just not at the Ford rune level. I've been working on grokking the current byzantine system of runes with /~, /#, etc.—so the simplicity of Pinto is nice, just trying to figure out what happens to the old functionality.

@belisarius222
Copy link
Author

@liam-fitzgerald:
Ok, I'll keep this in mind going forward. Not entirely sure yet what the best way is to express that in Pinto. It's true that parsing out the /-s and /+s should be pretty simple. I could also have Ford return the set of dependencies for a build, which might help, although if the user changes the imports in their editor without saving the file to Clay, the new library won't be in the subject and so the autocomplete hints will be stale. Not sure how bad of an issue that is.

@belisarius222
Copy link
Author

@sigilante:
The current Ford rune system is very expressive, but also difficult to use, and I'd argue it's been used in a broader scope than is ideal. It blurs the line between build-time and run-time in a way that I think has caused performance problems and can make things difficult to reason about.

We mostly don't use the more complex Ford runes these days, except in serving static files directly from Clay to the web. I propose (and I should really add this to an Integration section in this document) that we replace that with a request that can be sent to Eyre to serve a directory of static files, without transformation. With this change, we'll lose the ability to express stateless transformation of static assets to static webpages in Ford runes. @ixv should probably weigh in here on the viability of that approach.

Internally, Ford Pinto uses a data structure currently called a $pile (names subject to change, please nobody jump down my throat about naming just yet) to represent a preprocessed Hoon source file. Here's the latest version of that:
https://github.com/urbit/urbit/blob/b7f891bc423f5e9ddb8c9430f574c9eb7e95137d/pkg/arvo/sys/vane/pinto.hoon#L56

@ixv
Copy link

ixv commented Mar 9, 2020

If I understand you correctly, you're saying you want to be able to import hoon files with ford runes, but not files of any other mark. I think this is a bad idea. Changing the way we serve files from eyre would handle one use case of this, but importing arbitrary files is such a basic utility I would hate to lose it. Ford is a build system with inputs from clay, I think it's unprincipled to restrict that to being the subset of clay files with hoon marks.

To be honest I hate to lose the ability not just to import arbitrary files, but to combine and transform them at build time. Which doesn't necessarily mean I think the current ford rune scheme is sane. I hope we can bring these affordances back in some form some time.

@belisarius222
Copy link
Author

@ixv I'm inclined to agree with you. I'm mostly hoping to do something so minimal in this chunk of work that whatever build-time data import system we build will grow organically and stay more manageable than the current Ford rune system.

Do you know of any current uses of importing non-hoon files in Ford runes? Ideally I'd hold off on adding this feature back into Pinto until we need it at a future date, but if we need it now, I should go ahead and make sure it works. In particular, I think it would mean that a build to produce a mark conversion gate needs to be able to be run live, which isn't possible in the current design ... that might be a flaw in this current design in general, really.

Something I've talked to @joemfb about is separating building and running of Hoon code. One thing we could do is change the API of Ford so you could ask it for a gate that converts one mark to another, as opposed to asking it to perform the conversion. In Ford runes, we would want Ford to both build and run that gate on Clay files, but when sending a move to Ford, maybe we could support a task that asks for a mark conversion gate, and that could be a live build.

The other mark-related builds don't make sense as live builds, which was a large part of my motivation for separating requests for them from other Ford requests. But if they were expressed as requests to build functions that the client would then run on individual datums, they could be live. That might also remove the only times when the client sends a literal cage to Ford, which would allow that kind of build to be safely used from userspace. I'll think about this some more.

@joemfb
Copy link

joemfb commented Mar 12, 2020

:dojo can implement blocking .^ on its own, exactly as %ford did. I'd be inclined to do that for now. Separate syntax can be evaluated ... separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment