Anonymous inline modules

anonymous-inline-modules.md

Note: this document is subsumed by the module blocks proposal

Anonymous inline modules

Anonymous inline modules are syntax for the contents of a module, which can then be imported.

let inlineModule = module {
  export let y = 1;
};
let moduleExports = await import(inlineModule);
assert(moduleExports.y === 1);

assert(await import(inlineModule) === moduleExports);  // cached in the module map

Importing an anonymous inline module needs to be async, as anonymous inline modules may import other modules, which are fetched from the network.

let inlineModule = module {
  export * from "https://foo.com/script.mjs";
};

Anonymous inline modules are only imported through dynamic import(), and not through import statements, as there is no way to address them as a specifier string.

Syntax details

PrimaryExpression :  InlineModuleExpression

InlineModuleExpression : module [no LineTerminator here] { Module }

As module is not a keyword in JavaScript, no newline is permitted between module and {. Probably this will be an easy bug to catch in practice, as accessing the variable module will usually be a ReferenceError.

Realm interaction

As anonymous inline modules behave like module specifiers, they are independent of the Realm where they exist, and they cannot close over any lexically scoped variable outside of the module--they just close over the Realm in which they're imported.

For example, in conjunction with the Realms proposal, anonymous inline modules could permit syntactically local code to be executed in the context of the module:

let module = module {
  export o = Object;
};

let m = await import(module);
assert(m.o === Object);

let r1 = new Realm();
let m1 = await r1.import(module);
assert(m1.o === r1.o);
assert(m1.o !== Object);

assert(m.o !== m1.o);

Use with workers

It should be possible to run a module Worker with anonymous inline modules, and to postMessage an inline module to a worker:

let workerCode = module {
  onmessage = function({data}) {
    let mod = await import(data);
    postMessage(mod.fn());
  }
};

let worker = new Worker(workerCode, {type: "module"});
worker.onmessage = ({data}) => alert(data);
worker.postMessage(module { export function fn() { return "hello!" } });

Maybe it would be possible to store an inline module in IndexedDB as well, but this is more debatable, as persistent code could be a security risk.

Integration with CSP

Content Security Policy (CSP) has two knobs which are relevant to anonymous inline modules

• Turning off eval, which also turns off other APIs which parse JavaScript. eval is disabled by default.
• Restricting the set of URLs allowed for sources, which also disables importing data URLs. By default, the set is unlimited.

Modules already allow the no-eval condition to be met: As modules are retrieved with fetch, they are not considered from eval, whether through new Worker() or Realm.prototype.import. Anonymous inline modules follow this: as they are parsed in syntax with the surrounding JavaScript code, they cannot be a vector for injection attacks, and they are not blocked by this condition.

The source list restriction is then applied to modules. The semantics of anonymous inline modules are basically equivalent to data: URLs, with the distinction that they would always be considered in the sources list (since it's part of a resource that was already loaded as script).

Optimization potential

The hope would be that anonymous inline modules are just as optimizable as normal modules that are imported multiple times. For example, one hope would be that, in some engines, bytecode for an inline module only needs to be generated once, even as it's imported multiple times in different Realms. However, type feedback and JIT-optimized code should probably be maintained separately for each Realm where the inline module is imported, or one module's use would pollute another.

Support in tools

Anonymous inline modules could be transpiled to either data URLs, or to a module in a separate file. Either transformation preserves semantics.

Named modules and bundling

This proposal only allows anonymous module definitions. We could permit a form like module x { } which would define a local variable (much like class declarations), but this proposal omits it to avoid the risk that it be misinterpreted as defining a specifier that can be imported as a string form.

Anonymous inline modules proposal has nothing to do with bundling; it's really just about running modules in Realms or Workers. To bundle multiple modules together into one file, you'd want some way to give specifiers the inline modules, such that they can be imported by other modules. On the other hand, specifiers are not needed for the Realm and Worker use cases. This inline modules proposal does not provide modules with specifiers; a complementary "named inline modules" proposal could do so. Note that there are significant privacy issues to solve with bundling to permit ad blockers; see concerns from Brave.

How does this compare/contrast with importing a data URI, which somewhat (and less ergonomically) seems to provide this capability already?

Blob URL's can also be imported. data URI's are particularly useful for testing import and export at file: protocol https://plnkr.co/edit/RaCIg9?preview. Both can be used for Worker or Worklet code

          class AudioWorkletProcessor {}
          class AudioWorkletNativeFileStream extends AudioWorkletProcessor {
            constructor(options) {
              super(options);
          }
          // register processor in AudioWorkletGlobalScope
          function registerProcessor(name, processorCtor) {
            return `${processorCtor};\nregisterProcessor('${name}', ${processorCtor.name});`;
          }
          const worklet = URL.createObjectURL(
            new Blob(
              [
                registerProcessor(
                  'audio-worklet-native-file-stream',
                  AudioWorkletNativeFileStream
                ),
              ],
              { type: 'text/javascript' }
            )
          );

What would be interesting is importing and exporting across contexts. And importing and exporting raw binary data.

To both of these comments: A major goal of this proposal is to have good ergonomics used directly. I see an important part of ergonomics to be, to avoid having to put quotes or template literal backticks around code, as would be needed for data URLs or the Blob API.

@ljharb See #integration-with-csp -- the difference is that it is included in CSP's sources list.

@guest271314 Great, looks like another valid way to transpile this feature.

@littledan Is this proposal origin agnostic? Can Inline Modules be run at file: protocol without any errors or exceptions being thrown?

@guest271314 This proposal is not based on protocols, so I don't understand the question.

If we cannot explicitly use the proposal at file: protocol we will again need to use data URI, BlobURL and XMLHttpRequest() with appropriate flags and preferences set to run Inline modules at file: protocol. Given file: protocol is not an in-secure JavaScript (Ecmascript) modules should be explicitly defined to be protocol agnostic. Will this proposal achieve the same result at file: protocol as at https: protocol?

If a network request is made at

let workerCode = module {
  onmessage = function({data}) {
    let mod = await import(data);
    postMessage(mod.fn());
  }
};

let worker = new Worker(workerCode, {type: "module"});

and the protocol is file: the result should be the same as the protocol where the request is made is https:.

One note about the code example at this proposal is would avoid using Object in code, given if a user (accidentally) does

Object = 1

unexpected results can ensue https://stackoverflow.com/questions/40456491/why-is-there-not-a-built-in-method-in-javascript-to-check-if-an-object-is-a-plai#comment68436319_40457385.

Dynamic import within an inline module has module specifiers interpreted just like in the surrounding context, so on the web, all the same set of protocols would be supported.

The current issue is Chromium 86 and Firefox 79 both throw errors for dynamic imports at file: protocol by default

test.js

const test = {a: 123};
export {test};

test.html

<!DOCTYPE html>
<html>
  <head>
   <meta charset="utf-8">
   <title>Dynamic imports test at file: protocol</title>
  </head>
  <body>
    <script>
      import('./test.js').then(console.log, console.error);
    </script>
  </body>
</html>

Firefox 79 without privacy.file_unique_origin preference set to false at about:config

TypeError: error loading dynamically imported module
    <anonymous> file:///path/to/test.html:9

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at file:///path/to/test.js. (Reason: CORS request not http). [[Learn More]](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSRequestNotHttp)

Chromium 86 (dev channel)

Failed to load module script: The server responded with a non-JavaScript, CSS MIME type of "". Strict MIME type checking is enforced for module scripts per HTML spec.
TypeError: Failed to fetch dynamically imported module: file:///path/to/test.js

which makes it practically impossible to test either dynamic imports or import at file: protocol at Chromium (note we only get provisional headers)

Provisional headers are shown
Origin: file://
Referer
User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4221.3 Safari/537.36

or at Firefox without being aware of the preference(s) to set, and, or, using data URI, Blob URL and XMLHttpRequest() (fetch() is also restricted from making requests for resources at file: protocol).

Whatever you can do in your sphere about that would be helpful for the purpose of testing JavaScript locally, without need for server or a specific protocol.

Many things are broken on the file: protocol in browsers anyways; it's not really useful to ever use or test anything on it in my experience.

@ljhard Well, experience here teaches the opposite lesson. Testing at file: protocol has been very useful here. There are many things broken on every protocol. That is no excuse to not test on those protocols. Is WebAssembly.Memory.prototype.grow() broken or JavaScript (Ecmascript) TypedArray, ArrayBuffer and SharedArrayBuffer broken, etc.? Depends on the perspective. file: protocol is far more "secure" than https: protocol, as there is not even the oppostunity for a browser implementation to, for example, record the users' biometric data and send that data to an undiclosed remote server w3c/webappsec-secure-contexts#66.

If an API is broken, then deprecate it or fix it. In this specific case testing import, import() without the need of a server is prima facie useful.

@ljhard Consider example restricted requirement

I have a requirement where i have to save Form data on iPad, This HTML file will have basic information and form for data collection.

I have to design one page template for iPad, with Photo Gallery, Video Gallery and some text related to project.. More like a presentation. This much is possible and we can keep all the file on iPad and user can access then even if they are not connected to internet.

where the entire procedure must be accomplished without Internet access. While some OS's might be shipped with Apache and PHP, etc. to implement a basic server, neither a server nor Internet access is necessary to achieve the requirement. It should be possible to use import and import() in such a case.

From https://stackoverflow.com/q/35533107.

Now, if you really do not want direct feedback here that might challenge your own view, that you accept for yourself, then just say that.

In the field, without Internet access, which occurs not infrequently depending on the environment, file: protocol is certainly useful.

This proposal would not change the acceptance of the file protocol. If it currently does not work, this proposal would not enable it.

What are the semantics if the expression is evaluated multiple times; does it yield the same module like template strings would do, or a separate equivalent modulo like a function literal would? I'd hope for the former, so that:

function getModule() {
  return module { export default function() { console.log('hello'); } };
}
console.assert(getModule() === getModule());

so that in turn if you sent this to another realm, as in the worker example, it would be recognized as the same module on the other side (and not needlessly bloat the code memory).

@jeremyroman template strings have a per-realm cache, not a cross-realm one.

I meant a per-realm cache, but I'm assuming that the HTML structured serialize algorithm, or an analogous one in another environment, could leverage that to preserve identity across realms.

but I'm assuming that the HTML structured serialize algorithm, or an analogous one in another environment, could leverage that to preserve identity across realms.

Currently that is impossible with Ecmascript (JavaScript) modules. Tried to import a ReadableStream and fetch() into AudioWorkletGlobalScope. The code is run with the execution context of the code which calls the module. At Chromium or Chrome it is possible to use Transferable Streams to post the readable side of the transform stream to a different context, or use SharedArrayBuffer write and read from the same memory, with some limitations relevant to architecture.

@jeremyroman This is a good question. Given that anonymous inlined modules don't close over anything, it would make sense to me if the same value was returned. That leads to some questions about details about how these modules are identified (as a primitive, or an object), that could affect this question about same-Realm vs cross-Realm consistency. This gist remains vague on that detailed question.

@ljhard

Many things are broken on the file: protocol in browsers anyways; it's not really useful to ever use or test anything on it in my experience.

One example of the usefulness of testing at file: protocol: Capturing monitor device at Nightly and streaming the audio to Chromium which does not support capture of monitor devices at Linux https://gist.github.com/guest271314/04a539c00926e15905b86d05138c113c.

Does that not work properly if you use npx http-server or the equivalent to serve it on http?

Using a local server will more than likely work. Here, prefer to first test at file: protocol using only the capabilities shipped with the browser, to prove or disprove if a given requirement is possible without using a server. With a local server CORS requests must be handled. To get the data from the request at an arbitrary web page that might have content security policy settings an extension would need to be used, where at Chromium it is not possible to transfer, for example, a Unit8Array with extension code postMessage(), the TypedArray needs to be converted to JSON. Using a server in this case will add more complexity, though will eliminate the edge case of copying and pasting before the command that launches Nightly before Chromium completes, and before Chromium window gains focus to get the data copied to the clipboard. In this case file: protocol is very useful for the original requirement and proof-of-concept of writing and reading data across "agent clusters", which should be possible directly since reading and writing to clipboard across applications is possible.

This discussion about file: is completely off topic from the topic of this gist. The gist does not fix file:. There are legitimate security issues around file:.

Asked about file protocol for testing purposes. https: is far more vulnerable re "security issues" than file: protocol.

There are legitimate security issues around file:.

What evidence do you have to substantiate that claim? w3c/webappsec-secure-contexts#66

@littledan A case where this will be useful is initiating an AudioWorklet instance without having to make a network request WebAudio/web-audio-api-v2#109.

@littledan @ljharb It should be possible to load an Ecmascript module without making a network request. For example, when trying to construct an AudioWorklet at console at github.com CSP errors will be thrown. In this case we already have the code. The only non-essential restriction is AudioContext.audioWorklet.addModule() which makes a fetch request where we do not need to make a fetch request. I worked around this on Chromium using DevTools Local Overrides. However, it should be possible to run AudioWorklet at console on GitHub without making a network request where we already have the code inline.

I am banned from WHATWG/HTML and evidenbtly TC39 as well WebAudio/web-audio-api-v2#109 (comment)

This should be talked about at the EcmaScript and Worklet level before considering using it in Web Audio, this is not really a Web Audio API issue.

so I have no way other than this gist to communicate this matter to you folks.

I sent this proposal to es-discuss

Inline module without network request

Web Audio API uses Ecmascript / WHATWG/HTML modules for AudioContext.audioWorklet.addModule(). That means that a network fetch request must be made to construct an AudioWorklet node.

When we already have the code we do not need to make a network fetch request however the design of addModule() demands that a request be made to construct the AudioWorklet node.

If we are a console on GitHub or any other site that has CSP restrictions we cannot construct an AudioWorklet for the purposes of processing and outputting audio even though we can make a quic-transport protocol request.

This proposal is for a means to load a Ecmascript / WHATWG/HTML module without making a network request where we already have the code inline.

In practice the code can be as simple as

AudioContext.audioWorklet.addModule(code, {inline: true})

where {inline: true} configures the module loader to interpret code as raw JavaScript code instead of a URL, negating the need to make a network request.

References:

Alternatives for module loading of AudioWorklet #109 WebAudio/web-audio-api-v2#109

• littledan/anonymous-inline-modules.md
  https://gist.github.com/littledan/f7c1d1abf0e51ad4b526a8eadb2da43b