ES Module WASI Integration Concerns

esm-wasi-integration.md

Current PR: nodejs/node#27850

Initial Discussion of Concerns: nodejs/node#27850 (comment), WebAssembly/esm-integration#30

Summary

Node.js is looking to release support for WASI module imports through --experimental-wasi-modules.

There are three main components to this integration:

1. Support for a special wasi_unstable builtin module with an implementation in Node.js core.
2. A default permissions model for WASI in Node.js
3. Special linking semantics for WASI which are based on custom initialization code in the Web Assembly ES module integration in Node.js.

(1) and (2) I completely understand are experimental and will be worked out as things move forward.

My concern only relates to the semantics of (3) here, and specifically that we are putting together semantics which despite being unstable, may stick or become ingrained in the ecosystem due to the excitement for WASI.

These linking semantics are different from the standard ES module integration semantics, and are an important part of the Wasm / ESM ecosystem integration. As such my request is that we should at least be basing them on a clear and shared design document for how this integration behaves.

PR Linking Semantics

A WASI module is being detected as any Wasm module that imports "wasi_unstable", and in such cases special semantics are being added to the Web Assembly ESM integration in Node.js.

The code below is taken directly from the PR (https://github.com/nodejs/node/pull/27850/files#diff-ab9c666b48467c7030bd93aac6d06eb2R171), where the usesWasi exactly represent the new linking semantics:

let wasi;
if (usesWasi) {
  wasi = new WASI(WASI.defaultConfig);
  reflect.imports.wasi_unstable = wasi.exports;
}

const { exports } = new WebAssembly.Instance(compiled, reflect.imports);
for (const expt of Object.keys(exports))
  reflect.exports[expt].set(exports[expt]);

if (usesWasi) {
  wasi.setMemory(exports.memory);
  exports._start();
}

where the WASI.defaultConfig is taken to be the following object:

WASI.defaultConfig = {
  // https://github.com/WebAssembly/WASI/issues/5
  preopenDirectories: { '.': '.' },
  args: process.argv,
  env: process.env,
};

The linking semantics for WASI in this PR thus take the following steps:

1. When a Wasm module is detected as WASI (by having an import to wasi_unstable), the WASI builtin module is instantated specifically for the WASI module instance, with argv and env for this instance bound to the outer Node.js process argv and env.
2. The memory is assumed to be exported from the WASI module, and injected into this builtin instance.
3. The _start function export of the WASI module is then executed.

Commands and Reactors

There is a WASI issue tracking the Command and Reactor models for WASI in WebAssembly/WASI#13.

This issue describes two major semantics for WASI modules:

1. Commands, which build off from the standard binary main (int argc, char **argv) { return exitCode; } process execution model.
2. Reactors, which more closely resemble how we already use libraries within the lifecycle of a JS application.

We might consider Reactors to become the dominant integration of JS and Wasm in due course.

But in the mean time, if we look at code currently being run in WASI such as on https://wapm.io, what we see is that most WASI modules right now follow the Command model.

The Limitations of Command Imports

A Command module follows a process model - a synchronous execution from start to end, after which the module is likely no longer usable. There could be variations of execution that allow post-completion usage like Emscripten does, but this is still unclear.

Supporting the execution of Command modules on import in the Node.js ES Module Integration leads to a number of limitations:

Argument Passing

Any arguments the user might want to pass to the command need to be set on process.argv of the Node.js application at import time.

This means, in many cases, a custom VM usage or custom Node.js child process would need to be created just to pass the correct arguments.

In this model, it is not clear what the benefit is of supporting Commands through the WASI ESM Integration.

Exit Code

The exit code of the WASI process module is not being respected at all. This may cause issues for debugging scenarios.

Stdio

stdout and stdin are piped to and from the parent process for the duration of the execution of the WASI module. There is no clear way to determine when this piping process starts or ends or to specifically capture these inputs or outputs.

Execution Only on Import

Execution only happens at import time, meaning that we cannot rerun the command at all during the Node.js runtime without creating a child process. If the WASI Command is a long-running process that won't exit without standard input, then the import and entire module loading process may stall indefinitely.

In addition the execution start time is not easily hookable as it is implicitly run through the module system.

Environment

Environment variable customizations are not easily possible with this model, as the outer environment is assumed. This limits encapsulation features.

Steps to Remove the Block

1. Best Case: Proper Command Model

I would ideally like to see a Command model for WASI that can be used flexibly in ESM integration workflows without needing custom Node.js processes to properly pass arguments, so that we are presenting an integration that is useful for users and that does not encourage hacky workflows that build upon unstable semantics.

As an example of the sort of model possible see http://npmjs.org/package/wasi_unstable which demonstrates an experimental approach here.

The key point being that we need a process model in the execution environment for commands, instead of having them run implicitly through the module system imports.

2. Alternative: Disable Command Imports

This block is specifically about the Command scenarios, and it is certainly appreciated that that should not block other types of WASI development.

If there were a way to specifically disable the _start() execution for WASI modules, and then shift to using any new *start* function or similar only for Reactor models in WASI tooling, that would reduce the deviation from the standard Wasm / ESM integration model, while also avoiding the Command module pitfalls mentioned.

3. Compromise: Referenceable Design Document

If we are to land this experimental PR for WASI in Node.js, the minimal requirement should at least be a documentation note that can point to a work-in-progress design document for the WASI / ES Module integration, so that users who are interested in this can track the development of this work, instead of just assuming that the current semantics will resemble the final semantics.

Further clarifications on any of the above are welcome, and I am available for discussion anytime.