Currently only docker logs are streamable to the command line, and we wish to support this for WASM jobs as well.
Docker achieves these streaming logs by allowing the execution process to write to a ring buffer which will write to a log-driver based on configuration. By default, this driver is a JSON file and so docker will write log entries in the following format as jsonl:
{
"log": "\u001b[32mThe Project Gutenberg EBook of The Cosmic Computer, by Henry Beam Piper\n",
"stream": "stdout",
"time": "2023-03-17T11:37:08.764363458Z"
}
When reading the logs which are expected to be a totally ordered set, docker will process the options provided, and read entries from the log file. For options such as since and until this is likely rather inefficient as it would require reading the entire line and parsing the JSON, just to determine whether the message matches the query (time, matching stream etc).
We can follow a similar approach, of pipes being used to write the wasm output to a concurrent buffer, from where another process will write it to the log. Although it would be more efficient to use a binary encoding, it might introduce more complexity when other attributes are added to the structure (e.g. execution id or K8s' pod id). For now we will follow docker's approach with jsonlines.
New requests for the logs will consist of converting all of the JSON lines into messages for transmission to the client. For requests to follow this will also be true, but will then want to be notified of all new messages as the mediating process writes them to disk.
┌───────────────┐
│ │
│ Wasm Instance │
│ │
└───┬─────────┬─┘
Stdout│ │Stderr
│ │
┌────▼─────────▼────┐
│Multi Reader Logger│
└─────────┬─────────┘
┌──────────────────────┐ │ Read ┌──────────────────┐
│ Non-following reader │ ┌───────▼────────┐◄───────────┤ Following reader │
│ ├────►│Logfile mediator│ │ │
└──────────────────────┘Read └──────┬─────────┴───────────►└──────────────────┘
│ New entries
│
┌───▼───┐
│Logfile│
└───────┘