Anonymous inline modules

anonymous-inline-modules.md

Note: this document is subsumed by the module blocks proposal

Anonymous inline modules

Anonymous inline modules are syntax for the contents of a module, which can then be imported.

let inlineModule = module {
  export let y = 1;
};
let moduleExports = await import(inlineModule);
assert(moduleExports.y === 1);

assert(await import(inlineModule) === moduleExports);  // cached in the module map

Importing an anonymous inline module needs to be async, as anonymous inline modules may import other modules, which are fetched from the network.

let inlineModule = module {
  export * from "https://foo.com/script.mjs";
};

Anonymous inline modules are only imported through dynamic import(), and not through import statements, as there is no way to address them as a specifier string.

Syntax details

PrimaryExpression :  InlineModuleExpression

InlineModuleExpression : module [no LineTerminator here] { Module }

As module is not a keyword in JavaScript, no newline is permitted between module and {. Probably this will be an easy bug to catch in practice, as accessing the variable module will usually be a ReferenceError.

Realm interaction

As anonymous inline modules behave like module specifiers, they are independent of the Realm where they exist, and they cannot close over any lexically scoped variable outside of the module--they just close over the Realm in which they're imported.

For example, in conjunction with the Realms proposal, anonymous inline modules could permit syntactically local code to be executed in the context of the module:

let module = module {
  export o = Object;
};

let m = await import(module);
assert(m.o === Object);

let r1 = new Realm();
let m1 = await r1.import(module);
assert(m1.o === r1.o);
assert(m1.o !== Object);

assert(m.o !== m1.o);

Use with workers

It should be possible to run a module Worker with anonymous inline modules, and to postMessage an inline module to a worker:

let workerCode = module {
  onmessage = function({data}) {
    let mod = await import(data);
    postMessage(mod.fn());
  }
};

let worker = new Worker(workerCode, {type: "module"});
worker.onmessage = ({data}) => alert(data);
worker.postMessage(module { export function fn() { return "hello!" } });

Maybe it would be possible to store an inline module in IndexedDB as well, but this is more debatable, as persistent code could be a security risk.

Integration with CSP

Content Security Policy (CSP) has two knobs which are relevant to anonymous inline modules

• Turning off eval, which also turns off other APIs which parse JavaScript. eval is disabled by default.
• Restricting the set of URLs allowed for sources, which also disables importing data URLs. By default, the set is unlimited.

Modules already allow the no-eval condition to be met: As modules are retrieved with fetch, they are not considered from eval, whether through new Worker() or Realm.prototype.import. Anonymous inline modules follow this: as they are parsed in syntax with the surrounding JavaScript code, they cannot be a vector for injection attacks, and they are not blocked by this condition.

The source list restriction is then applied to modules. The semantics of anonymous inline modules are basically equivalent to data: URLs, with the distinction that they would always be considered in the sources list (since it's part of a resource that was already loaded as script).

Optimization potential

The hope would be that anonymous inline modules are just as optimizable as normal modules that are imported multiple times. For example, one hope would be that, in some engines, bytecode for an inline module only needs to be generated once, even as it's imported multiple times in different Realms. However, type feedback and JIT-optimized code should probably be maintained separately for each Realm where the inline module is imported, or one module's use would pollute another.

Support in tools

Anonymous inline modules could be transpiled to either data URLs, or to a module in a separate file. Either transformation preserves semantics.

Named modules and bundling

This proposal only allows anonymous module definitions. We could permit a form like module x { } which would define a local variable (much like class declarations), but this proposal omits it to avoid the risk that it be misinterpreted as defining a specifier that can be imported as a string form.

Anonymous inline modules proposal has nothing to do with bundling; it's really just about running modules in Realms or Workers. To bundle multiple modules together into one file, you'd want some way to give specifiers the inline modules, such that they can be imported by other modules. On the other hand, specifiers are not needed for the Realm and Worker use cases. This inline modules proposal does not provide modules with specifiers; a complementary "named inline modules" proposal could do so. Note that there are significant privacy issues to solve with bundling to permit ad blockers; see concerns from Brave.

From https://stackoverflow.com/q/35533107.

Now, if you really do not want direct feedback here that might challenge your own view, that you accept for yourself, then just say that.

In the field, without Internet access, which occurs not infrequently depending on the environment, file: protocol is certainly useful.

This proposal would not change the acceptance of the file protocol. If it currently does not work, this proposal would not enable it.

What are the semantics if the expression is evaluated multiple times; does it yield the same module like template strings would do, or a separate equivalent modulo like a function literal would? I'd hope for the former, so that:

function getModule() {
  return module { export default function() { console.log('hello'); } };
}
console.assert(getModule() === getModule());

so that in turn if you sent this to another realm, as in the worker example, it would be recognized as the same module on the other side (and not needlessly bloat the code memory).

@jeremyroman template strings have a per-realm cache, not a cross-realm one.

I meant a per-realm cache, but I'm assuming that the HTML structured serialize algorithm, or an analogous one in another environment, could leverage that to preserve identity across realms.

but I'm assuming that the HTML structured serialize algorithm, or an analogous one in another environment, could leverage that to preserve identity across realms.

Currently that is impossible with Ecmascript (JavaScript) modules. Tried to import a ReadableStream and fetch() into AudioWorkletGlobalScope. The code is run with the execution context of the code which calls the module. At Chromium or Chrome it is possible to use Transferable Streams to post the readable side of the transform stream to a different context, or use SharedArrayBuffer write and read from the same memory, with some limitations relevant to architecture.

@jeremyroman This is a good question. Given that anonymous inlined modules don't close over anything, it would make sense to me if the same value was returned. That leads to some questions about details about how these modules are identified (as a primitive, or an object), that could affect this question about same-Realm vs cross-Realm consistency. This gist remains vague on that detailed question.

@ljhard

Many things are broken on the file: protocol in browsers anyways; it's not really useful to ever use or test anything on it in my experience.

One example of the usefulness of testing at file: protocol: Capturing monitor device at Nightly and streaming the audio to Chromium which does not support capture of monitor devices at Linux https://gist.github.com/guest271314/04a539c00926e15905b86d05138c113c.

Does that not work properly if you use npx http-server or the equivalent to serve it on http?

Using a local server will more than likely work. Here, prefer to first test at file: protocol using only the capabilities shipped with the browser, to prove or disprove if a given requirement is possible without using a server. With a local server CORS requests must be handled. To get the data from the request at an arbitrary web page that might have content security policy settings an extension would need to be used, where at Chromium it is not possible to transfer, for example, a Unit8Array with extension code postMessage(), the TypedArray needs to be converted to JSON. Using a server in this case will add more complexity, though will eliminate the edge case of copying and pasting before the command that launches Nightly before Chromium completes, and before Chromium window gains focus to get the data copied to the clipboard. In this case file: protocol is very useful for the original requirement and proof-of-concept of writing and reading data across "agent clusters", which should be possible directly since reading and writing to clipboard across applications is possible.

This discussion about file: is completely off topic from the topic of this gist. The gist does not fix file:. There are legitimate security issues around file:.

Asked about file protocol for testing purposes. https: is far more vulnerable re "security issues" than file: protocol.

There are legitimate security issues around file:.

What evidence do you have to substantiate that claim? w3c/webappsec-secure-contexts#66

@littledan A case where this will be useful is initiating an AudioWorklet instance without having to make a network request WebAudio/web-audio-api-v2#109.

@littledan @ljharb It should be possible to load an Ecmascript module without making a network request. For example, when trying to construct an AudioWorklet at console at github.com CSP errors will be thrown. In this case we already have the code. The only non-essential restriction is AudioContext.audioWorklet.addModule() which makes a fetch request where we do not need to make a fetch request. I worked around this on Chromium using DevTools Local Overrides. However, it should be possible to run AudioWorklet at console on GitHub without making a network request where we already have the code inline.

I am banned from WHATWG/HTML and evidenbtly TC39 as well WebAudio/web-audio-api-v2#109 (comment)

This should be talked about at the EcmaScript and Worklet level before considering using it in Web Audio, this is not really a Web Audio API issue.

so I have no way other than this gist to communicate this matter to you folks.

I sent this proposal to es-discuss

Inline module without network request

Web Audio API uses Ecmascript / WHATWG/HTML modules for AudioContext.audioWorklet.addModule(). That means that a network fetch request must be made to construct an AudioWorklet node.

When we already have the code we do not need to make a network fetch request however the design of addModule() demands that a request be made to construct the AudioWorklet node.

If we are a console on GitHub or any other site that has CSP restrictions we cannot construct an AudioWorklet for the purposes of processing and outputting audio even though we can make a quic-transport protocol request.

This proposal is for a means to load a Ecmascript / WHATWG/HTML module without making a network request where we already have the code inline.

In practice the code can be as simple as

AudioContext.audioWorklet.addModule(code, {inline: true})

where {inline: true} configures the module loader to interpret code as raw JavaScript code instead of a URL, negating the need to make a network request.

References:

Alternatives for module loading of AudioWorklet #109 WebAudio/web-audio-api-v2#109

• littledan/anonymous-inline-modules.md
  https://gist.github.com/littledan/f7c1d1abf0e51ad4b526a8eadb2da43b