A proposal for dependable, scalable Urbit hosting systems.
Demand for Urbit hosting is at an all-time high, but current solutions have difficulty satisfying that demand. While designing our own hosting platform, learning from the Tlon and UrbitHost experiences with Kubernetes, and consulting with core Urbit infrastructure engineers, we realized the dire need and feasibility for a better approach to hosting: the Urbit maximalist approach.
With our proposed improvements to Urbit's dependability and scalability, Urbit will be poised to break through the current hosting limits and realize its original ambition: to become the new internet.
Seguro makes Urbit dependable by automatically replicating its event log using open-source solutions from the best of the distributed systems software world.
Armada makes Urbit scalable with a new Arvo vane enabling an Urbit ship to host many other virtualized Urbit ships within itself.
The Urbit runtime was designed to host just a single Urbit instance running in a Unix process, with a single file volume attached for its event log and checkpoints. For individuals running only one or a handful of ships, this architecture is satisfactory. For providing Urbit to the world as a new, decentralized, peer-to-peer network of personal servers, the current implementation has inarguably succeeded.
However, for a quickly maturing platform which needs to scale to meet enormous demand, current Urbit technology is not suitable. Its low dependability and scalability continue to prevent the Urbit community from building many critical services and applications, including but certainly not limited to:
- Urbit hosting at massive scale
- IoT
- Machine learning and parallel algorithms
- High availability applications
For example, large-scale Urbit hosting (an essential milestone for Urbit's overall success and also for ours) built using the current best practices is overly complex, unmanageable, and expensive. Every ship existing as its own process with its own persistent volume becomes difficult to manage and prone to failure when attempting to host at scale. The Kubernetes-based architecture in use by Tlon and UrbitHost orchestrates Docker containers on Google Cloud Platform and DigitalOcean, respectively. This approach is viable at scales from dozens to hundreds of ships, but in practice it is brittle and requires high technical support overhead if attempting to scale up by an order of magnitude or more.
Kubernetes was originally designed by Google specifically for scaling their Google Search application. The Google Search architecture involves many identical commodity servers in a highly parallel structure to process millions of search requests. Kubernetes was tailored to this use case: it is optimized for scaling horizontally, running many servers which are identical replicas of each other. However, Urbit hosting is very different, it requires running many unique servers with their own persistent volumes. Kubernetes is just not ideal for this usage pattern, it fails to leverage most of Kubernetes' strengths but retains all of its complexity and fragility.
While the Kubernetes approach is certainly tenable for smaller deployments, and progress for using it to deploy stateful applications like Urbit has improved over the years, it is the incorrect long-term approach since it does not bet on Urbit as a general-purpose computing platform. Instead, the Kubernetes approach suffers from:
- Legacy Unix tooling
- Heavy consumption of expensive cloud resources
- Overly complex design and implementation
- High barrier to entry
- High cost of technical support
- Obsolescence in a Martian world
- Relegation of Urbit to being JAUP (Just Another Unix Program)
Both Tlon and UrbitHost seemed to have found this approach laden with diffulties and are seeking alternatives.
Our research has led us to conclude that, for any business with a sound model, the Kubernetes approach to hosting is DOA.
From a business perspective, we think it's clear why an Urbit-native approach to hosting is necessary. From a technical perspective, we delve deeper and ask:
- Which non-functional requirements does Urbit hosting need to succeed?
- How does the Kubernetes approach attempt to fulfill them?
- Why can't Urbit itself fulfill those requirements better?
Firstly is dependability, defined by Wikipedia as
a measure of a system's availability, reliability, maintainability... durability, safety and security.
After all, Urbit is your digital home! It needs high dependability. In hosting, that dependability starts with the orchestration technology and must extend downwards to the thousands of Urbit ships it is responsible for managing.
The Kubernetes approach employed by both Tlon and UrbitHost attempts to provide dependability to hosting services through the implementation of a custom Kubernetes Operator.
Operators implement and automate common Day-1 (installation, configuration, etc) and Day-2 (re-configuration, update, backup, failover, restore, etc.) activities in a piece of software running inside your Kubernetes cluster, by integrating natively with Kubernetes concepts and APIs. We call this a Kubernetes-native application. With Operators you can stop treating an application as a collection of primitives like Pods, Deployments, Services or ConfigMaps, but instead as a single object that only exposes the knobs that make sense for the application.
The Kubernetes approach to Urbit hosting uses the Operator SDK to give Kubernetes a primitive understanding of an Urbit ship: its dependencies, how to boot it, its interfaces, and how to react in case of failures or strange behavior. The implementation of the operator (a notoriously difficult software engineering task) is responsible for tightly mapping a set of Urbit ships to a Kubernetes cluster. It is the most critical component in the Kubernetes approach.
In practice, however, current hosting service offerings are fraught with dependability problems, often resulting in 503 Unavailable and other HTTP errors being sent to users. Some of these problems stem from a lack of features in Urbit itself:
- Architectural deficiencies of
vere - Delayed implementation of
%khan - Insufficient, non-standard interfaces for health, error states
Other problems arise from Kubernetes:
- Poor fit to the baked-in Kubernetes assumptions
- Container and volume management overhead
- Overall complexity
Our second requirement, which is no less important, is scalability. Scalability is defined as
the property of a system to handle a growing amount of work by adding resources to the system.
Aside from the Kubernetes cluster itself (whose configuration and management costs should not be underestimated), each ship running with the Kubernetes approach requires:
- A
StatefulSetwith a pod to run the Urbit binary - A persistent volume for storing its event log
- Network resources (
Services, etc.) for UDP and HTTP communications - Optional security resources to restrict access to unauthorized users
With the dependability issues described above and the high cost of increasing Kubernetes clusters, scalability is significantly impeded. After all, Kubernetes is legacy Unix cluster computing technology. It uses bubble gum and paperclips to provide faux isolation and encapsulation atop the Unix platform via Docker containers. Urbit, on the other hand, is a purely functional, general-purpose computer. Shouldn't we be building a cluster computing technology that uses the Urbit computer as its building block rather than a Docker container?
We propose to solve these problems with two separate but related projects:
For improved dependability, Seguro is an Urbit binary extension for replication of events and automatic failover.
For improved scalability, Armada is an extension and improvement of Aqua which enables an Urbit ship to host many virtualized Urbit ships within itself.
Seguro replicates a ship's event log, providing resiliency and redundancy to a currently fault-intolerant Urbit program. When running Seguro, a hardware failure will result in some minimal amount of network downtime but crucially should not cause data loss and therefore should never require a breach in order to gracefully recover.
Seguro will be based on a dqlite and Raft architecture to provide data replication, high availability and failover for the Urbit platform. A Seguro cluster will consist of an elected master and two or more replication slaves. Urbit network UDP events will be received by an IP-level load balancer which will then replicate them to all the members of the cluster. Cluster members will all process these events and record them in their local event log but only the Seguro master will actually emit their side effects back to the network.
If the Seguro master crashes or is otherwise not available to the cluster a Raft election will take place and a new elected master will be promoted and pick up where the system left off. Seguro will be independent from Armada in cases where only high availability is desired, such as hosts of popular groups or any other situation that demands better availability than a standalone Urbit ship can provide. It will unavoidably add some complexity to the intial setup and support of the Urbit ship but, being entirely optional, that complexity ought to only be only a factor for those who choose to use it.
Some effort towards this goal has already been performed and adaption of dqlite/Raft for Urbit was conceived and researched as far back as 2013. The concept needs some responsible organization to champion it and provide resources and management so it can be finished, but the core architecture is sound and, given sufficient development resources, Seguro ought to be a reasonably achievable improvement at this point in Urbit's development.
Challenges to Seguro are in managing the event loop and recording events. An implementation of optimistic run-ahead logic and batching of event logs writes will be necessary for performance, but this should be safe to implement given the added reliability of a clustered architecture. Ideally we may also want aggressive event log truncation to reduce the storage hardware requirements but this may not be necessary for a successful initial launch.
Armada is a project which will allow virtualized Urbit ships to be run within another Urbit ship. Armada will dispatch incoming events to the proper virtual ship and transmit any side effects back out to the network.
Armada will be a natural evolution of the current Aqua Gall app but with improved performance suitable for production use. Aqua is a fleet testing and development tool which runs Arvo instances in Urbit userspace and pretends to be Vere. Aqua loads a pill to boot a fleet of virtual ships and manages them from within the parent Arvo instance. Aqua ships are jetted just like the parent and therefore Aqua ships do not run any slower than their parent. The only significant overhead is in the dispatch of events from the parent to its children.
However, Aqua is strictly a development tool. It is performant enough for testing and development uses but not quite suitable for actual production use. One potential solution to the Aqua performance problem may be turning it from a Gall app to a Vane but more research needs to be done to answer that question conclusively.
Similar to Aqua, the Armada host will receive all the events for its children ships and will dispatch them to the appropriate children within itself for processing, passing the side effects back out to the network accordingly. The Armada host will also include an Urbit-native interface written in Hoon for managing its children and launching new ones from keyfiles or existing piers. The host will also be able to communicate to its children via %khan or remote scry and will centralize management of those children in a sensible user interface.
The advantages of the Armada approach as compared to the traditional Kubernetes method are many. Virtualization on Urbit is effectively free, with no Docker container overhead beyond dispatching events to the children and routing their side effects. The complexity of the Kubernetes control plane and associated resource management moves into Urbit proper, and hosting of ships for star owners (or even moons for planet owners) becomes accessible with reduced external dependencies. This is a much simpler architecture with fewer moving parts and ought to incur significantly reduced support overhead.
Seguro and Armada are not wholly novel inventions. They are both natural evolutions of concepts that have been discussed for years and in some cases partially implemented. As such the development risk of these two projects is much lower than might be naively assumed. The ideas behind Seguro were conceived years ago with some experimental code already checked into github, and Armada is a logical extension of the already functional Aqua application.
By using Seguro and Armada, high volume Urbit hosting should become much simpler to implement and manage. Moving away from the singleton model of one container per Urbit instance will help Urbit to be much more scalable for the nascent Urbit hosting industry.
Hosting at scale is of course the main motivation behind these two projects, but Seguro and Armada should also provide many benefits beyond this as well. Seguro will offer significantly improved reliability to all Urbit users who desire greater dependability for their important ships. Armada will likewise be useful for much more beyond hosting at scale, as any applications such as machine learning which rely on massively parallel systems will be able to leverage Armada to manage their fleets.
Seguro and Armada are logical evolutions of existing Urbit concepts and are feasible upgrades to the core platform. They simply need attention, focus, and a robust push to bring them to fruition, and they’re absolutely necessary for the near-future stages of Urbit evolution.
If we want the world on Urbit, we need both the dependability and scalability Seguro and Armada can provide.
Seguro and Armada are based on time-tested ideas that have been floating around the Urbit space for years and based on our consultations with many of the experienced core developers these goals ought to be realizable. One clear risk is that we will absolutely require New Mars or some other solution to the 2GB loom limit, especially for Armada to be viable. However this ought to not be a significant problem by the time Seguro and Armada are reaching their beta phases, since this specific improvement is also a requirement for projects like Uqbar which are closer to release. We should be able to assume this issue will have been solved one way or another by the time Seguro and Armada become usable.
An Armada-specific risk is that strict event ordering means events cannot be processed in parallel. This poses a difficulty for scaling Armada, since all events will be delivered to the parent ship and will need to be dispatched by a single thread in the current architecture. However, there are potentially some ways of addressing this risk. Moving to scries, which the content distribution project is doing, and making those scries run in parallel, which New Mars is aiming toward, could be one solution. More research will clearly need to be done on this issue but we are hopeful it can be solved or worked around.
Another major risk is regarding performance, we will need a certain baseline level of performance for Seguro to be used without impacting Urbit’s speed as perceived by the end user, and for Armada to scale to the level we envision. Likewise this is also a requirement for Uqbar and other projects, and our understanding is that Urbit still has a significant amount of low-hanging fruit where performance increases can be achieved.
| Date | Milestone |
|---|---|
| February 2022 | Research and investigation |
| March 2022 | Begin development |
| April 2022 | On-board new developer hires |
| June 2022 | Alpha builds |
| September 2022 | Beta testing |
| January 2023 | Version 1.0 release |
Pax Dickinson ~master-forwex pax@wexpert.systems
Matthew LeVan ~mastyr-bottec matt@wexpert.systems