UP Fleet Tests

up-fleet-tests.md

Title: Urbit Fleet Tests Author: Ted Blackman ~rovnys-ricfer ted@tlon.io Created: ~2017.12.14

Overview

After a month of labor, Tlon gave birth to a new continuity era, which came out kicking and screaming, covered in amniotic fluid. A slew of issues rapidly appeared in the wild. We'd tested the functionality of the new features, but we hadn't stress-tested them. Some bugs, like herpes, only manifest when the system is under stress.

We need a way to perform automated testing on a fleet of Urbits. We'll want to crank up the load to arbitrary severity while retaining the ability to measure results -- crucially, we want to improve our sleep quality by testing loads much worse than the live network.

The usual guidelines of test frameworks still apply. We should be able to start a test quickly, and shut it down reliably. Each test needs to be independent of other tests. And no test may ever touch the production network.

We plan to construct this system inside Urbit, replacing the somewhat anemic urbit/fleet scripts. We'll write a google cloud platform API connector for spawning and reaping ships, and we'll build a set of apps and generators that orchestrate the tests, using ~zod as conductor.

Specification

These tests are federated along the celestial hierarchy. Like a cardiovascular system, ~zod pumps efferent requests through stars and planets to its descendant extremities, and afferent responses converge back up through ancestors to ~zod. Communication for the purposes of test orchestration is expressed with single-level recursion, i.e. a parent talks to its child but not to its grandchildren, and a child talks to its parent but not to its grandparents.

The main actions required by each ship are:

• Spawn children
• Tell parent that spawning has succeeded
• Reap children
• Tell parent that reaping has succeeded
• Tell child to start test phase
• Collect responses from children and use them to determine success or failure
• Respond to parent indicating success of test phase

~zod, the root galaxy, has additional responsibilities:

• Send an email to report test results
• Self-destruct when the test is over

Test Run Control Flow [outdated; needs to be rewritten using single-level recursion]

Each test run shall be conducted as follows:

• Spawn conductor
  • On any machine with gcloud credentials, run a script that creates a new gcloud instance and runs a script on that. Parameters of the test should be specified on the command line when running this script.
  • The script on the newly created instance does the following:
    • Fetch Urbit and Arvo at the specified versions.
      • TODO how to specify the versions
    • Build the urbit executable.
    • Start ~zod in a screen session.
    • Using urb.py, tell ~zod to run its :fleet-tests app.
    • Using urb.py, tell ~zod to poke its :fleet-tests app by running the +start <args>|fleet-tests generator, passing through arguments from the remote command-line script.
• Spawn children
  • The first thing :fleet-tests will do on ~zod is set up gcloud authorization based on the credentials passed in on the command line as test parameters.
  • After setting up gcloud, :fleet-tests will spawn a fixed set of stars and planets (with the number of stars and planets also override-able as test parameters). Given a certain number of stars to spawn, it will start with the lowest star address ~marzod and increment from there as necessary. For planets, it will create an approximately equal (up to a max difference of one, like dealing cards) number of planets under each star using the same method, until the specified number of planets has been created.
  • Child spawning sequence: When +fleet-tests spawns a child, it calculates the ticket for that ship then tells gcloud to start an instance and run a script on that instance. That script (which is probably the same script as the script that started ~zod, called with different parameters) will:
    • Download and build the specified versions of urbit and arvo.
    • Start the specified instance in a screen session with command-line flags such that it becomes the right ship and connects to the right testnet.
    • Using urb.py, start the :fleet-tests app, which on startup will report to ~zod that it's ready to receive commands.
• Run tests
  • Send commands to children
    • At each stage of the test, the conductor's test app sends a command to its children telling them to start that stage of the test. A "stage" could be something like 'urbit-meta-send-10', which would cause each child to send 10 messages to urbit-meta. Once all the messages have been kicked off, the child would wait until the total number of messages in its urbit-meta channel meets the expected number from all ships. Then it would report back to ~zod saying the stage had succeeded from its perspective. ~zod will wait up to a timeout for all its children to report success. If not all of them have responded before the timeout expires, ~zod will report test failure. If the test succeeds, ~zod will tell its children to execute the final stage of the test.
  • Children execute commands and respond
    • When triggered by a message from ~zod to start a new test stage, the fleet-tests app on each child will send moves to whatever apps and vanes it needs to interact with (talk and hall for the urbit-meta example) to tell it to perform the actions for that stage. Then the app will subscribe to the necessary endpoints within those apps and vanes so it will be notified of state changes in such a way that when the test stage's success conditions have been met, the app will send a message back to ~zod to inform it that the stage is done.
  • Collect responses
    • ~zod's generator will
  • Send results to email address passed in as a command line argument.
    • TODO how to specify email address
  • Reap children
• Reap children
  • ~zod will send commands to the gcloud API connector to shut down each of its child instances.
  • Once this has completed, ~zod will send a command to the gcloud API connector to self-destruct its instance.

Modules

The following modules will need to be built to accomplish the above test flow:

• gcloud connector in sec/com
• test app (~zod)
• test app (children)
• conductor generator (~zod)

Rationale

TODO

Integration Plan

TODO

Would it be reasonable to have the system independent of a particular connector?
I could imagine a lot of people would like to use different system, eg.
urbits in separate docker containers (ground work for Urbit Harbour), or even just in separate folders on some unix box and so on.