Eyre is Arvo's web server and one of Arvo's eight kernel modules (vanes). Eyre is responsible for handling inbound and outbound HTTP requests, parsing mark extensions and query strings in URLs, and sending respective moves based on the requests to the other Arvo vanes, like Clay, Ford and Gall.
This document is for the current iteration of Eyre, which has not yet been designed and implemented into its ideal state.
Let's look at the current data structures of Eyre, starting with its primary interface, +able:eyre, and the supplement data structures the interface relies on. These are all defined in Zuse:
zuse.hoon
:::: ++eyre :: (1e) oldweb
:: ::::
++ eyre ^?
|%
:: ::
:::: ++able:eyre :: (1e1) arvo moves
:: ::::
++ able ^?
|%
+= gift :: out result <-$
$% [%mass p=mass] :: memory usage
[%mack p=(unit tang)] :: message ack
[%sigh p=cage] :: marked http response
[%thou p=httr] :: raw http response
[%thus p=@ud q=(unit hiss)] :: http request+cancel
[%veer p=@ta q=path r=@t] :: drop-through
[%vega p=path q=path] :: drop-through
[%velo p=@t q=@t] :: drop-through
== ::
+= task :: in request ->$
$% [%born ~] :: new unix process
[%crud p=@tas q=(list tank)] :: XX rethink
[%hiss p=(unit user) q=mark r=cage] :: outbound user req
[%init p=@p] :: report install
[%serv p=$@(desk beam)] :: set serving root
[%them p=(unit hiss)] :: outbound request
[%they p=@ud q=httr] :: inbound response
[%chis p=? q=clip r=httq] :: IPC inbound request
[%this p=? q=clip r=httq] :: inbound request
[%thud ~] :: inbound cancel
[%wegh ~] :: report memory
[%went p=sack q=path r=@ud s=coop] :: response confirm
[%west p=sack q=[path *]] :: network request
== ::
-- ::able
::
:::: :: (1e2)
::
++ clip (each @if @is) :: client IP
+= gram :: inter-ship message
$? [[%lon ~] p=hole] :: login request
[[%aut ~] p=hole] :: login reply
[[%hat ~] p=hole q=hart] :: login redirect
[[%get ~] p=@uvH q=[? clip httq]] :: remote request
[[%got ~] p=@uvH q=httr] :: remote response
[[%gib ~] p=@uvH] :: remote cancel
== ::
++ hart {p/? q/(unit @ud) r/host} :: http sec+port+host
++ hiss {p/purl q/moth} :: outbound request
++ host (each (list @t) @if) :: http host
++ httq :: raw http request
$: p/meth :: method
q/@t :: unparsed url
r/(list {p/@t q/@t}) :: headers
s/(unit octs) :: body
== ::
++ httr {p/@ud q/mess r/(unit octs)} :: raw http response
++ math (map @t (list @t)) :: semiparsed headers
++ mess (list {p/@t q/@t}) :: raw http headers
++ meth :: http methods
$? $conn :: CONNECT
$delt :: DELETE
$get :: GET
$head :: HEAD
$opts :: OPTIONS
$post :: POST
$put :: PUT
$trac :: TRACE
== ::
++ moth {p/meth q/math r/(unit octs)} :: http operation
++ octs {p/@ud q/@t} :: octet-stream
++ pork {p/(unit @ta) q/(list @t)} :: fully parsed url
++ purl {p/hart q/pork r/quay} :: parsed url
++ quay (list {p/@t q/@t}) :: parsed url query
-- ::eyre
Let's start with the basics: +tasks and +gifts:
A +task is a +move, or action, that Arvo can send to a vane, telling the vane to perform some computation and produce Arvo a result.
Here are the +tasks that Eyre can perform for Arvo:
+= task :: in request ->$
$% [%born ~] :: new unix process
[%crud p=@tas q=(list tank)] :: XX rethink
[%hiss p=(unit user) q=mark r=cage] :: outbound user req
[%init p=@p] :: report install
[%serv p=$@(desk beam)] :: set serving root
[%them p=(unit hiss)] :: outbound request
[%they p=@ud q=httr] :: inbound response
[%chis p=? q=clip r=httq] :: IPC inbound request
[%this p=? q=clip r=httq] :: inbound request
[%thud ~] :: inbound cancel
[%wegh ~] :: report memory
[%went p=sack q=path r=@ud s=coop] :: response confirm
[%west p=sack q=[path *]] :: network request
== ::
Starting with the basic ones, here are some short explanations for the control flows of the various tasks. See +apex in Eyre for the actual code that handles these tasks:
Eyre only ever receives one %init task in the boot process of the urbit (technically, two, if a galaxy, star, planet, or moon: one for the comet the urbit starts as, and a second for the new initialization post-identity-metamorphoses). This task initializes Eyre, for example setting the default served Clay path (our current %home-desk /web directory).
When the Unix process starts (or restarts), Eyre will receive the duct for communication and set the duct in its state via the %born task.
When Unix makes an HTTP request to an urbit's Vere, that event gets passed through Arvo to Eyre as a %this +task, an inbound request. Depending on the request, Eyre might get tasked with performing one of several different kinds of computations.
The %thud task will cancel an inbound %this if Eyre receives it before completing the initial %this request.
Sometimes a user requests for Eyre to send an outbound HTTP request (like perhaps through a Gall app to send an email notification via sending an outbound HTTP request to the Gmail API). The +task that Eyre ends up receiving from Arvo is a %hiss, which gets converted to a %them upon inferring that the outbound request is meant to go to Unix. The response that Eyre receives from Unix comes in the form of the %they task, which gets wrapped into a type-checked response to the requester (usually Gall, but it could be any of the vanes). More on this in the +gift section below.
The %west task is assigned to Eyre through Arvo when Arvo receives an Ames request from another ship for a local resource. Or, phrased differently, when one ship wants to make a network request for a remote ship's resource, they can send an HTTP request over Ames for the remote ship's Eyre to receive, handle, and produce a response to get sent back over Ames. We use this for, among other things, single-sign-on like functionality for urbit.org and Fora, as well as remote-request proxying for ship.urbit.org via ~marzod currently (soon, this will get replaced by secure, direct TLS connections/terminations directly to ship IP's). Nonetheless, the payload of a %west task contains a +gram, a subtype that Eyre does pattern-matching on to answer different kinds of requests. More on this in the +gift section below.
The %serv tasks simply sets the Clay +beam for Eyre to serve by default. One common user-facing way to set this is to use the |serve Hood generator.
The %chis task is an old hack by @pcmonk to get the now-deprecated urb.py tool to work from Unix.
If Eyre receives an error from Arvo (Unix), it receives it as a %crud task, which sends the error to Dill to print to the terminal.
The %wegh task is used in Vere for debugging to tell Eyre to "weigh itself", e.g. calculate the approximate amount of memory it's using in the loom.
%went tasks are not currently used (unimplemented Ames API detail).
A +gift is a vane's response to a +task requested by Arvo. The vane "gifts" Arvo with the result of the requested computation in the original task.
Eyre's possible +gifts to Arvo are:
+= gift :: out result <-$
$% [%mass p=mass] :: memory usage
[%mack p=(unit tang)] :: message ack
[%sigh p=cage] :: marked http response
[%thou p=httr] :: raw http response
[%thus p=@ud q=(unit hiss)] :: http request+cancel
[%veer p=@ta q=path r=@t] :: drop-through
[%vega p=path q=path] :: drop-through
[%velo p=@t q=@t] :: drop-through
== ::
When Eyre is tasked with a %this, it asynchronously responds with a %thou gift. The request may never resolve -- for example, Eyre may receive an HTTP request telling it to poke a Gall app that doesn't and won't ever exist -- in which case, it never responds. But, asynchronicity aside, a %this task that resolves will always correspond with a %thou gift.
The problem with outbound HTTP requests (%hiss/%them tasks) is that Arvo/Vere only speak gifts. So Eyre has to wrap the outbound request -- which is really more in the spirit of a task, in terms of Eyre's API classification -- inside of a gift. This is what the %thus gift is for. Eyre gifts Arvo with the %them task wrapped inside of the %thus gift, so that Arvo (Vere) can carry out the task in the outside world -- like reaching out to the Gmail API. When Vere receives a response, it will pass it up to Arvo and "task" Eyre with a %they tasks, which is really an inverted gift wrapped inside of the task. The %they task is then converted into a %thou gift, whereby, before in the original %hiss, Eyre took the q.hiss mark and stuck it in the wire to save for this stage of the process, where the type validation occurs. The +httr (HTTP response) in the newly formed %thou gift can now get mark-validated/converted into a formal +cage. Finally, Eyre can respond with the final gift, a %sigh to the requester -- in the above example, this would be Gall, who originally asked for the %hiss outbound request to Gmail. In this case, the %sigh contains the response data that Gall wants.
The %mack gift is part of the interface for one urbit's Eyre to talk to another urbit's Eyre over Ames. Currently a %mack is produced as part of the %west task-handling control-flow for requesting resources over the Urbit network via "Ames-ified" HTTP. Currently, Eyre simply (too simply) produces a null %mack gift to positively acknowledge to Ames that it properly received the %west task. More properly (to be fixed in the future), Eyre should ensure that there is no handling error first before giving out the positive ack, but this is generally an Arvo-wide issue yet to be solved. Nonetheless, the requesting Ames in the above request ultimately gets the response data in a new remote-urbit's Eyre %west task send to remote-urbit's Ames, which is sent over the wire to the local-urbit's Ames and handled from there. The pattern-matching for handling whether the %west tasks are requests or responses is determined by the subtype of the +gram in the %west task payload, handling for which is within the +apex core of Eyre. See the control flow section below for more detail on that.
The %veer, %vega, and %velo gifts are all used for reloading Zuse and the vanes upon updates.
The %mass gift is Eyre's response to the %wegh task, providing an approximation of how many system resources the vane is using.
Next, the core Eyre data structures in Eyre itself:
eyre.hoon
:: :: %eyre, http servant
!? 164
::::
!:
|= pit/vase
=, eyre
=, wired
=, unity
=, format
=, mimes:html
=, html
=> =~
|% :: interfaces
++ move {p/duct q/(wind note gift:able)} :: local move
++ note :: out request $->
$% $: $a :: to %ames
$% {$want p/sock q/{path *}} ::
== == ::
$: $b :: to %behn
$% {$wait p/@da} ::
{$rest p/@da} ::
== == ::
$: $d :: to %dill
$% {$flog p/{$crud p/@tas q/(list tank)}} ::
== == ::
$: $e :: to self
$% {$thud $~} :: proxied death
{$this p/? q/clip r/httq} :: proxied request
{$meta vase} :: type check
== == ::
$: $f :: to %ford
$% {$exec p/@p q/(unit {beak silk:ford})} ::
{$wasp p/@p q/@uvH r/?} ::
== == ::
$: $g :: to %gall
$% {$deal p/sock q/cush:gall} :: full transmission
== == == ::
++ sign :: in result $<-
$? $: $a :: by %ames
$% {$woot p/ship q/coop} :: acknowledgment
{$went ship cape:ames} :: XX ignore
== == ::
$: $b :: by %behn
$% {$wake $~} :: timer activate
== == ::
$: $g :: by %gall
$% {$unto p/cuft:gall} :: within agent
== == ::
$: $e :: by self
$% {$thou p/httr} :: response for proxy
== == ::
$: $f :: by %ford
$% {$made p/@uvH q/gage:ford} ::
{$news p/@uvH} ::
== == ::
$: @tas :: by any
$% {$crud p/@tas q/(list tank)} ::
== == == ::
++ ixor @t :: oryx hash
++ whir $@ $~ :: wire subset
$% {$ac p/hole q/whir} :: cookied
{$at p/hole q/whir} :: authenticated
{$ay p/knot q/knot $~} :: /ay/@p/@uvH :: remote duct
{$ha p/path} :: /ha/[beak] :: GET request
{$he p/whir} :: HEAD request
{$hi p/knot q/mark $~} :: outbound HTTP
{$se p/whir-se q/{user (list @t)}} :: outbound to domain
{$si $~} :: response done
{$of p/ixor q/$@($~ whir-of)} :: associated view
{$ow p/ixor $~} :: dying view
{$on $~} :: dependency
== ::
++ whir-of {p/@taxp q/term r/?($mess $lens) s/wire} :: path in dock
++ whir-se ?($core vi-arm) :: build/call
++ vi-arm
$? $filter-request :: ++out mod request
$filter-response :: ++res use result
$receive-auth-response :: ++bak auth response
$receive-auth-query-string :: ++in handle code
$out
$res
$bak
$in
== ::
-- ::
|% :: models
++ bolo :: eyre state
$: $0 :: version
gub/@t :: random identity
hov/(unit ship) :: master for remote
top/beam :: ford serve prefix
ged/duct :: client interface
ded/(set duct) :: killed requests
lyv/(map duct live) :: living requests
pox/(map @uvH duct) :: proxied sessions
ask/{p/@ud q/(map @ud {p/duct q/hiss})} :: outgoing by number
kes/(map duct @ud) :: outgoing by duct
ney/@uvI :: rolling entropy
dop/(map host ship) :: host aliasing
liz/(jug @uvH (each duct ixor)) :: ford depsets
wup/(map hole cyst) :: secure sessions
sop/(map hole {ship ?}) :: foreign sess names
wix/(map ixor stem) :: open views
sec/(map {user (list @t)} driv) :: security drivers
== ::
::
++ driv :: driver state
%+ pair (unit $@($~ vase)) :: main core
{liv/? req/(qeu (trel duct mark vase:hiss))} :: waiting requests
::
++ live :: in flight
$% {$exec p/whir} :: ford build
{$wasp p/(list @uvH)} :: ford deps
{$xeno p/ship} :: proxied request
{$poll p/ixor} :: session state
==
++ cyst :: client session
$: ced/cred :: credential
{him/ship aut/(set ship)} :: authenticated
cug/(list @t) :: unacked cookies
lax/@da :: last used
way/(map ship {purl duct}) :: waiting auth
vew/(set oryx) :: open views XX expire
== ::
::
++ stem :: client view
$: him/ship :: static identity
ude/(unit {p/duct q/?}) :: stream long-poll?
era/@da :: next wake
die/@da :: collection date
sus/(set {dock $json wire path}) :: subscriptions
eve/{p/@u q/(map @u even)} :: queued events
med/(qeu duct) :: waiting /~/to/
==
++ even :: client event
$% {$news p/@uvH}
{$quit p/{dock path}}
{$rush p/{dock path} q/json}
==
::
++ perk :: parsed request
$% {$auth p/perk-auth}
{$away $~}
{$oath p/knot q/(list @t)}
{$bugs p/?($as $to) $~}
{$beam p/beam}
{$deps p/?($put $delt) q/@uvH}
{$mess p/dock q/mark r/wire s/json}
{$poll p/{i/@uvH t/(list @uvH)}}
{$spur p/spur}
{$subs p/?($put $delt) q/{dock $json wire path}}
{$view p/ixor q/{$~ u/@ud}}
==
::
++ perk-auth :: parsed auth
$% {$at p/pork} :: inject auth
{$del p/(unit ship)}
{$get him/ship rem/pork}
{$js $~}
{$json $~}
{$try him/ship paz/(unit cord)}
{$xen ses/hole rem/pork}
==
::
++ pest :: result
$@ $~
$% {$$ p/httr} :: direct response
{$red $~} :: parent redirect
{$bake p/whir q/mark r/coin s/beam} :: ford request
{$js p/@t} :: script
{$json p/json} :: data
{$html p/manx} :: successful page
{$htme p/manx} :: authentication fail
==
-- ::
Agenda
- How cross-ship wrapping currently works -- the full pipeline
- How Eyre works
Cross-ship Notes
- Eyre receives a $this
- The $this gets parsed and goes to +apex:handle
- From +apex:handle, it goes to +process
- From +process it goes to +parse and goes from a +purl into a +perk
- +process-parsed turns the +perk into either a +pest, or it produces an updated +handle core (the main Eyre state)
- :1456 - +pest - the card that Eyre responds to Arvo with as the HTTP response. Within the local Eyre, producing a +pest is stateless.
- What a +perk turns into
- $htme is “HTML error”
- $bake is a Ford request
- For cross-ship wrapping, the perks we care about are $beam (full Clay path) and $spur (local |served path). On the xship-wrap branch, these both use the wrapping renderer pipeline.
- The $beam/$spur perk case produces a %bake pest for Ford, which is comprised of:
- :wir - There’s some information put in the wire here -- see +whir
- :ext - File extension (mark, used for Ford conversion)
- :arg - Authentication data that goes to /$; also has the query string in it
- :bem - The beam to build
- Once we have the +pest, call +resolve on it
- +resolves takes a (pair (list cookies) pest) as its sample and produces some effects, based on the stem
- Eyre receives a $west:
- Within the $west is a $get-inner
- The $get-inner produces a Ford %bake, which responds with a %made.
- The %made gets wrapped in a wire and packaged into a +gram to get send back over Ames within another $west, which contains the response $got-inner
- With the $got-inner message, we make a Ford build, validating the result is the data we want, then running the %urb-wrap renderer on the inner, thus adding on the outer, and then we cast this all to an %urb and make sure it type-checks to the valid complete web build we want
- When we have our validated data, we call +exec-live to start the live subscription that the browser session is requesting. It can be terminated while it’s still in an interruptible state, in the process of serving. Once the client has the full static page, the :lyv state in Eyre is canceled, and +exec-live stops.
Eyre/Zuse notes
- +ye is the per-event core
- +handle:ye is the main per-HTTP-request core within +ye
- +gram is an Eyre message sent over Ames, in the %west task of Eyre’s Zuse interface. Local Eyre tells local Ames %want (Ames task). Local Ames tells remote Ames turns a local %want into a remote %west into remote Eyre. Remote Eyre gets a +gram.
- +note and +sign are the inter-vane / inter-Arvo move types
- +vi-arm are the interface arms for security drivers
- +bolo is the Eyre state
- +driv in :sec is the state per security driver
- +cyst in :wup is the state per client session
- Named by +hole which is a @p name generated from entropy
- +stem in :wix is the open view per session, which maintains a live connection to Gall (e.g. web Talk UI)
- :ude q long-polling hasn’t been implemented
- :era - long-polling
- :die - connection death
- :eve -
- One auth control flow:
- Vere gets an HTTP request, talks to Arvo, creates a duct, and sends the event to Eyre
- Eyre, having received the typed HTTP request, looks up the “view” state for the ixor (hash) in the request. If no stem exists for the ixor, create a new session, meaning the eve map is null
- The :ude and :eve map can’t both be non-null -- a non-null :ude means that we have an open longpoll still to subscribe to
- In short, we respond on ducts with the queued events
- See Eyre commentary/documentation, even if it’s old
- General point: we flop :mow in our vane state because we always tack on new moves in our per-event core to the head of the list
- General: Gifts are the responses to tasks, which come in. Signs are the responses from other vanes to notes, which go out. All through Arvo. Tasks we can get from Vere. Gifts we can give to Vere. Signs and Notes can only come from / go to other vanes.
- %sighs are for HTTP requests coming from Arvo; %thous are for HTTP requests coming from Vere.
- =, ‘s at the top of the Eyre file pull Zuse cores into the subject
- +ix is the per-view core. +ya is the per-session core. +vi is the auth engine.
- +new-ya creates a new +ya core to operate on per new session, created through the logic in +handle. The new session is created with +new-cyst.
- Anton wants to collapse +cred down into just (unit ship)
Actions
- Cross-ship:
- Create a new +pest for cross-ship requests, to separate categorically from a request on a local ship (which shouldn’t go over Ames)
- Eyre:
- Document:
- State components in +take of Eyre
- Document:
- +ya core: client sessions
- +for-client:
- If you don't have any cookies, create a new +ya. If your cookies are bad, create a new +ya. Otherwise, initialize a +ya core with your session, and acknolwedge your valid cookie (setting it to null).
- +new-cyst:
- Create a new session with the cookies of the HTTP request, setting the last-used time to now.
- The point of this is that when you interact with the session, you get a cookie, which gets a name, and has associated with that a name in Eyre, so that the client and server can remember each other.
- +stat-json is auth.json:
- The first thing it does is create an oryx (CSRF token) for the tab. The process is basically the same as above: if the client and server both remember the oryx being shared, then they can load a shared saved session. This gets stored in
vew.bolo.eyre-state. - This sets up the possibility for longpolling, because now the tab has an identifier, which means the urbit can have an ordered list of things it wants to tell the browser (i.e. ducts).
- The first thing it does is create an oryx (CSRF token) for the tab. The process is basically the same as above: if the client and server both remember the oryx being shared, then they can load a shared saved session. This gets stored in
- +logon:
- The client receives the auth.json from the above process. By posting the auth.json to the proper Eyre endpoint, this ends up calling
+logon, if the password is correct.
- The client receives the auth.json from the above process. By posting the auth.json to the proper Eyre endpoint, this ends up calling
- +logoff
- This just removes the session information from the Eyre state.
- +foreign-auth and +foreign-hat are both for single-sign-on.
- +for-client:
- +ix core: view core, per-tab
[ixor stem]ixoris the hash of the oryx for the view.stemis the view state:- One ship per view
- Duct for long-polling
- Time to send next heartbeat (longpoll)
- Time to cancel longpoll subscription
- Gall subscriptions
- Queued client events (see +even)
- +poll:
- Refresh the die timer
- If we're polling for the event-number of the next queued event, then wait a little bit, and we'll here back (unless the longpoll is cancelled before then).
- If the event-number is greater than that of the next queued event, then that's an error -- crash.
- Otherwise, we're in the clear -- give the event, e.g. call
+give-even.
- Most of these other arms are straightforward.
- +vi core: auth engine for security drivers
- State:
- Local alias for a username of a service
- Web domain
- Driver state (
+driv)
- The state per driver gets input by the data in the
%hisstask- The (unit user) goes in as
usr - The mark goes in as the mark of the queued request (the mark we want to run the validated vase from the cage through)
- The vase from the cage of the
%hiss, which itself is a%hiss(designated by Anton's specialvase:hissDSL-like syntax, which Curtis despises)
- The (unit user) goes in as
+baleis kind of like the security-driver+bowl:gall.- "This is wet-gate vase land." -Anton
- The first
%hissis put in the queue - Then
+pumpcalls the build.- If there's no core, we need to build one.
+build:- Call
+executeon the core from the%hissvase, constructed as a+silk:ford
- Call
- If the pump is blocked, it's blocked.
- If there's nothing on the event queue, return.
- For an empty security driver, pass through the
%them. - Otherwise, call the security driver
+filter-requeston the next request.
- If there's no core, we need to build one.
+sec-moveis the data structure for moves security drivers can send.$ac(/~/ac/...) corresponds to the landing URL that sends the OAuth consent confirmation data to the security driver, and calls the+receive-auth-query-stringarm.- Anton made security drivers a part of Ford so that they could use
.^.
- State:
Anton hack