ceejbot/esm_in_node_proposal.md

## esm_in_node_proposal.md

      
    Raw
  

              esm_in_node_proposal.md
            
          
    ESM modules in node: npm edition

The proposal you’re about to read is not just a proposal. We have a working implementation of almost everything we discussed here.
We encourage you to checkout and build our branch: our fork, with the relevant branch selected.  Building and using the implementation will give you a better understanding of what using it as a developer is like.
Our implementation ended up differing from the proposal on some minor points. As our last action item before making a PR, we’re writing documentation on what we did. While I loathe pointing to tests in lieu of documentation, they will be helpful until we complete writing docs: the unit tests.
This repo also contains a bundled version of npm that has a new command, asset.  You can read the documentation for and goals of that command in the the repo where we're developing it: GitHub - iarna/assetize: proof-of-concept. The overview is that it turns a node ES module, with node-style dependencies, into a version of the module (with deps) that can be loaded into a browser as-is. It does so by rewriting import paths at install time.
If you have a counter-proposal, I urge you to do what we did and implement it. This will make clear to you what’s feasible given V8 and language constraints, and what might be nice to have but is not feasible. A working implementation is a better thing to discuss than a wishlist.
And now back to the proposal…

JavaScript in browsers has a new module system: ES modules. JavaScript in node.js has had a module system for years: CommonJS modules. JavaScript developers have nearly universally adopted node as the platform of choice for their tooling. Web application frameworks all use command-line tools written in node, pulling shared code from the npm registry, to build applications that run on the browser.  Open-source code they find on npm is freely used in these applications. Browser app developers now have the following expectations:

Code they write for their tooling uses the same language as their browser applications.
Code they discover in npm's module registry is re-usable in the browser.

CommonJS modules do not work in the browser, but bundling tools like browserify and webpack have allowed CommonJS modules to be built into browser-ready code. Early implementations of ES modules rely on the CommonJS module system under the hood, as Babel transpiles ESM syntax into CommonJS syntax.
However, with the arrival of browser support for ES modules, the expectations have changed. Browser developers now expect node to support ESM natively. It does not yet do so.
The node project's proposal for integrating ES modules currently does not satisfy any of its audiences. Browser-focused developers do not like the .mjs file extension proposal. Tooling developers do not have the hooks they need to provide code coverage tools on par with the ones existing for CommonJS, or transpilers like CoffeeScript. Node-focused developers don't want a new module system in the first place.
We propose that npm provide some useful opinions about how to resolve the impasse, in the form of a design, a working implementation of that design, and a pull request to the node project.
The goals of our design are:

offer the affordances developers need for dependency injection, transpilation, and test instrumentation
satisfy the ergonomics expectations of node-focused developers
satisfy the language expectations of browser-focused developers

Our implementation will include real-world examples demonstrating the success of core use cases.
Principles

These are the principles that drive all of our design decisions:


Existing CommonJS code must continue to work unchanged.


ES modules that don't have platform-specific dependencies must work unchanged in both the browser and node, with the exception that import paths might need to be changed.


The parse goal of a module must be known by its consumer in advance. The module producer should write files that end in .js and advertise the module's parse goal in its documentation.


The remainder of this document examines the consequences of these principles on how ES modules would work in node.
Node global objects

The following node global objects are still defined for ES modules, because they are also defined for browser execution environments:

console, setTimeout, clearTimeout, setImmediate, setInterval

To get the other node pre-defined globals and module locals, you must import them. import * from 'nodejs' provides the following:

process
global
Buffer
require
__dirname
__filename
isMain new!

isMain is true if the current module is a file run directly from node.
We also propose adding loaderhooks as a way of specifying hooks at runtime. An example:
import {loaderHooks} from 'nodejs';
loaderHooks.set({
    instantiate () { /* in pace requiescat */ },
    resolve () { /* sorry fortunato */ }
})
loaderHooks.transpilers.push(async function (url, source) {
	// return a promise for a string
})
Parse goals

Authors do not need to specify parse goals via means like file extensions. Main files are assumed to have the module goal falling back to CommonJS if the runtime encounters require() or any module locals. To avoid the performance hit of the extra parse step, you can invoke node using a flag that invokes common js parsing by default: --mode=legacy. For example, at the top of a script you would use #!/bin/env node --mode=legacy. (The opposing command-line option, --mode=esm, will also short-circuit any attempt to guess the parse goal.)
Additionally, node's existing shebang-stripping parsing logic will be extended to check for --mode=legacy in the shebang of the main module, which will trigger an immediate switch to commonjs mode.
#! /usr/bin/env node --mode=legacy

// We are now in CJS mode without needing a parse step.
const fs = require('fs');
This parse step is only necessary for the initial entry point. All detection of the module goals after that initial entry is driven by the method of consumption: import means the module is parsed ESM, and require that it’s parsed as CJS.
Detecting main

CJS modules currently test if they are main by checking if require.main === module evaluates to true. This does not change. CJS modules can also check module.parent to see if they're the root level module. This should continue to work, but we reserve the right to make module.parent point to a truthy object in cases where a CJS module is being consumed from ESM.
ES modules can test if they are main by examining isMain as imported from 'nodejs'.
#! /usr/bin/env node

import assert from 'assert';
import {isMain} from 'nodejs';

assert(isMain === true);
Resolution

CJS and ESM have identical package resolution logic in node. That is: import * as foo from 'foo' and const foo = require('foo') must resolve to the same file. (That file is likely to be parseable in only one of those goals.) This resolution algorithm continues to do everything it does today (consult package.json for main, look for index.js in a directory, and so on), with the following change.
import './foo' attempts to load code from the exact same locations, with the same priority, as require('./foo').  However:

The resolved file location is read and parsed as ESM, not CJS.
This means that .json and .node will fail, but lifecycle hooks MAY be added to make them work.
The process for extending the ESM loader logic uses lifecycle hooks, not require.extensions.

The node resolver/path resolution mechanism should be implemented as a separate module that can be used by other code; see below for a discussion of hooks that can override this.
ESM/CJS Interoperability in Node.js

Cross-consumption of modules works if authors use the correct consumption method.

CJS code can use import('esm').then()
ESM code can use import {require} from 'nodejs'; require('cjs')

Consumption of ES modules using CommonJS syntax might fail, and vice versa. Specifically:

import 'cjs' fails with an error, because require, exports, and module locals such as __filename are not defined. node intercepts ReferenceError: <module local is not defined and decorates it with with a hint.
require('esm') fails with an error, because import and export are not valid keywords in the script goal. node intercepts SyntaxError and decorates it with an error message suggesting that the programmer might need to import() the module.
Some cross-consumed modules might not fail. These are likely either to be polyfill libraries or trivial cases. Non-trivial cases can test for this === null to see if they're executing in a CJS context or not.

ESM Interoperability between Node.js and Browsers

If browsers, in the future, add  a way to turn import * from 'foo' into a url, then the server responding to requests for that url merely needs to return the expected file, which static servers can do. For example, foo/index.js might be returned when a browser requests foo/, much like how index.html is used.
Until that time, build steps can transform import statements into relative paths to fully-resolved files. More advanced build scripts can reduce the "many request" problems, and HTTP2 can potentially help as well.
The purpose of this proposal is to reduce the friction involved in cross-platform JS applications, especially at development time.  We expect that build steps will always be beneficial in preparing apps for production.
node core modules

All node core modules are exported as both ES and CJS modules. The module object exported is identical.
import {require} from 'nodejs';
import * as fs_ns from 'fs';
import fs from "fs"
let fs_req = require('fs');

fs === fs_ns; // false
fs === fs_ns.default; // true
fs === fs_req; // true
fs_req === fs_ns; // false
The following works for all exports from node core modules:
import {readFile} from 'fs';
Lifecycle hooks

The ES module system shall provide three lifecycle hooks for overriding:

resolver
instantiator
transpiler

The resolver stage is responsible for resolving user-provided import strings into full URLs and their expected format. The default resolver implements the Node module resolution algorithm. For example: it resolves the "./foo.js" from import foo from "./foo.js" into "file:///Users/basil/foo.js" (with a format of js), or "browserify" to "file:///Users/basil/node_modules/browserify". This stage is already present and hookable in Node.js' ESM implementation, and will remain present. Custom hooks allow users to redefine how resolution works for applications. Configuration of these hooks can be done at runtime, and the hooks are applied to the next module graph load. To specify ahead of your entry point, use -r.
The instantiation phase is responsible for turning a file URL into a list of exported names and an evaluate function. The evaluate function is called with an object that allows the exported names to be modified.
function instantiate (url) {
  return {
    exports: ['forThe', 'loveOf', 'godMontresor'],
    evaluate (exports) {
      exports.forThe = 'these names have to be known in advance'
      exports.loveOf = 'due to the way that'
      exports.godMontresor = 'esm is linked.'
    }
  }
}
Instantiation hooks can be defined at runtime alongside resolution hooks. They are already present in the current Node.js ESM implementation and will remain present. They are considered an advanced user feature: they allow modules to be created that are not backed directly by a single file on disk.
Transpilation hooks are suitable for the more common task of source-to-source transpilation. They are not be fired for "dynamic" format modules, only for "js" modules.
Transpilation hooks receive a file URL and source text. They are expected to return a String or Promise for a String. Transpilation hooks fire in order, with the result of each hook passed to the subsequent hook. At the end of this process, the String returned is expected to contain valid ESM source text.
Transpilation hooks can be configured at runtime.
You can configure hooks by providing a module indentifier string to node using the --require CLI option. That module identifier must be loadable by the standard node module resolver.
Some examples:
node --require /path/to/typescript-require foo.ts
node --require /path/to/babel-require foo.js
Wrapup

Our evaluation of the spec and the current node implementation lead us to believe that this proposal is feasible. We also feel it addresses the developer concerns we're aware of. We point to an ESM future while maintaining as many of the ergonomics of the current node.js module system as we can. We believe browser-focused developers and node developers will find it workable.
Misc

vm.Module and vm.runInModule should both be exposed eventually.
Relevant links

ECMAScript® 2018 Language Specification: Modules
Chris's walk of the current node experimental implementation
standard-things/esm