Skip to content

Instantly share code, notes, and snippets.

@ceejbot
Last active June 20, 2024 10:45
Show Gist options
  • Save ceejbot/b49f8789b2ab6b09548ccb72813a1054 to your computer and use it in GitHub Desktop.
Save ceejbot/b49f8789b2ab6b09548ccb72813a1054 to your computer and use it in GitHub Desktop.
npm's proposal for supporting ES modules in node

ESM modules in node: npm edition

The proposal you’re about to read is not just a proposal. We have a working implementation of almost everything we discussed here. We encourage you to checkout and build our branch: our fork, with the relevant branch selected. Building and using the implementation will give you a better understanding of what using it as a developer is like.

Our implementation ended up differing from the proposal on some minor points. As our last action item before making a PR, we’re writing documentation on what we did. While I loathe pointing to tests in lieu of documentation, they will be helpful until we complete writing docs: the unit tests.

This repo also contains a bundled version of npm that has a new command, asset. You can read the documentation for and goals of that command in the the repo where we're developing it: GitHub - iarna/assetize: proof-of-concept. The overview is that it turns a node ES module, with node-style dependencies, into a version of the module (with deps) that can be loaded into a browser as-is. It does so by rewriting import paths at install time.

If you have a counter-proposal, I urge you to do what we did and implement it. This will make clear to you what’s feasible given V8 and language constraints, and what might be nice to have but is not feasible. A working implementation is a better thing to discuss than a wishlist.

And now back to the proposal…

JavaScript in browsers has a new module system: ES modules. JavaScript in node.js has had a module system for years: CommonJS modules. JavaScript developers have nearly universally adopted node as the platform of choice for their tooling. Web application frameworks all use command-line tools written in node, pulling shared code from the npm registry, to build applications that run on the browser. Open-source code they find on npm is freely used in these applications. Browser app developers now have the following expectations:

  • Code they write for their tooling uses the same language as their browser applications.
  • Code they discover in npm's module registry is re-usable in the browser.

CommonJS modules do not work in the browser, but bundling tools like browserify and webpack have allowed CommonJS modules to be built into browser-ready code. Early implementations of ES modules rely on the CommonJS module system under the hood, as Babel transpiles ESM syntax into CommonJS syntax.

However, with the arrival of browser support for ES modules, the expectations have changed. Browser developers now expect node to support ESM natively. It does not yet do so.

The node project's proposal for integrating ES modules currently does not satisfy any of its audiences. Browser-focused developers do not like the .mjs file extension proposal. Tooling developers do not have the hooks they need to provide code coverage tools on par with the ones existing for CommonJS, or transpilers like CoffeeScript. Node-focused developers don't want a new module system in the first place.

We propose that npm provide some useful opinions about how to resolve the impasse, in the form of a design, a working implementation of that design, and a pull request to the node project.

The goals of our design are:

  • offer the affordances developers need for dependency injection, transpilation, and test instrumentation
  • satisfy the ergonomics expectations of node-focused developers
  • satisfy the language expectations of browser-focused developers

Our implementation will include real-world examples demonstrating the success of core use cases.

Principles

These are the principles that drive all of our design decisions:

  • Existing CommonJS code must continue to work unchanged.

  • ES modules that don't have platform-specific dependencies must work unchanged in both the browser and node, with the exception that import paths might need to be changed.

  • The parse goal of a module must be known by its consumer in advance. The module producer should write files that end in .js and advertise the module's parse goal in its documentation.

The remainder of this document examines the consequences of these principles on how ES modules would work in node.

Node global objects

The following node global objects are still defined for ES modules, because they are also defined for browser execution environments:

  • console, setTimeout, clearTimeout, setImmediate, setInterval

To get the other node pre-defined globals and module locals, you must import them. import * from 'nodejs' provides the following:

  • process
  • global
  • Buffer
  • require
  • __dirname
  • __filename
  • isMain new!

isMain is true if the current module is a file run directly from node.

We also propose adding loaderhooks as a way of specifying hooks at runtime. An example:

import {loaderHooks} from 'nodejs';
loaderHooks.set({
    instantiate () { /* in pace requiescat */ },
    resolve () { /* sorry fortunato */ }
})
loaderHooks.transpilers.push(async function (url, source) {
	// return a promise for a string
})

Parse goals

Authors do not need to specify parse goals via means like file extensions. Main files are assumed to have the module goal falling back to CommonJS if the runtime encounters require() or any module locals. To avoid the performance hit of the extra parse step, you can invoke node using a flag that invokes common js parsing by default: --mode=legacy. For example, at the top of a script you would use #!/bin/env node --mode=legacy. (The opposing command-line option, --mode=esm, will also short-circuit any attempt to guess the parse goal.)

Additionally, node's existing shebang-stripping parsing logic will be extended to check for --mode=legacy in the shebang of the main module, which will trigger an immediate switch to commonjs mode.

#! /usr/bin/env node --mode=legacy

// We are now in CJS mode without needing a parse step.
const fs = require('fs');

This parse step is only necessary for the initial entry point. All detection of the module goals after that initial entry is driven by the method of consumption: import means the module is parsed ESM, and require that it’s parsed as CJS.

Detecting main

CJS modules currently test if they are main by checking if require.main === module evaluates to true. This does not change. CJS modules can also check module.parent to see if they're the root level module. This should continue to work, but we reserve the right to make module.parent point to a truthy object in cases where a CJS module is being consumed from ESM.

ES modules can test if they are main by examining isMain as imported from 'nodejs'.

#! /usr/bin/env node

import assert from 'assert';
import {isMain} from 'nodejs';

assert(isMain === true);

Resolution

CJS and ESM have identical package resolution logic in node. That is: import * as foo from 'foo' and const foo = require('foo') must resolve to the same file. (That file is likely to be parseable in only one of those goals.) This resolution algorithm continues to do everything it does today (consult package.json for main, look for index.js in a directory, and so on), with the following change.

import './foo' attempts to load code from the exact same locations, with the same priority, as require('./foo'). However:

  1. The resolved file location is read and parsed as ESM, not CJS.
  2. This means that .json and .node will fail, but lifecycle hooks MAY be added to make them work.
  3. The process for extending the ESM loader logic uses lifecycle hooks, not require.extensions.

The node resolver/path resolution mechanism should be implemented as a separate module that can be used by other code; see below for a discussion of hooks that can override this.

ESM/CJS Interoperability in Node.js

Cross-consumption of modules works if authors use the correct consumption method.

  • CJS code can use import('esm').then()
  • ESM code can use import {require} from 'nodejs'; require('cjs')

Consumption of ES modules using CommonJS syntax might fail, and vice versa. Specifically:

  • import 'cjs' fails with an error, because require, exports, and module locals such as __filename are not defined. node intercepts ReferenceError: <module local is not defined and decorates it with with a hint.
  • require('esm') fails with an error, because import and export are not valid keywords in the script goal. node intercepts SyntaxError and decorates it with an error message suggesting that the programmer might need to import() the module.
  • Some cross-consumed modules might not fail. These are likely either to be polyfill libraries or trivial cases. Non-trivial cases can test for this === null to see if they're executing in a CJS context or not.

ESM Interoperability between Node.js and Browsers

If browsers, in the future, add a way to turn import * from 'foo' into a url, then the server responding to requests for that url merely needs to return the expected file, which static servers can do. For example, foo/index.js might be returned when a browser requests foo/, much like how index.html is used.

Until that time, build steps can transform import statements into relative paths to fully-resolved files. More advanced build scripts can reduce the "many request" problems, and HTTP2 can potentially help as well.

The purpose of this proposal is to reduce the friction involved in cross-platform JS applications, especially at development time. We expect that build steps will always be beneficial in preparing apps for production.

node core modules

All node core modules are exported as both ES and CJS modules. The module object exported is identical.

import {require} from 'nodejs';
import * as fs_ns from 'fs';
import fs from "fs"
let fs_req = require('fs');

fs === fs_ns; // false
fs === fs_ns.default; // true
fs === fs_req; // true
fs_req === fs_ns; // false

The following works for all exports from node core modules:

import {readFile} from 'fs';

Lifecycle hooks

The ES module system shall provide three lifecycle hooks for overriding:

  • resolver
  • instantiator
  • transpiler

The resolver stage is responsible for resolving user-provided import strings into full URLs and their expected format. The default resolver implements the Node module resolution algorithm. For example: it resolves the "./foo.js" from import foo from "./foo.js" into "file:///Users/basil/foo.js" (with a format of js), or "browserify" to "file:///Users/basil/node_modules/browserify". This stage is already present and hookable in Node.js' ESM implementation, and will remain present. Custom hooks allow users to redefine how resolution works for applications. Configuration of these hooks can be done at runtime, and the hooks are applied to the next module graph load. To specify ahead of your entry point, use -r.

The instantiation phase is responsible for turning a file URL into a list of exported names and an evaluate function. The evaluate function is called with an object that allows the exported names to be modified.

function instantiate (url) {
  return {
    exports: ['forThe', 'loveOf', 'godMontresor'],
    evaluate (exports) {
      exports.forThe = 'these names have to be known in advance'
      exports.loveOf = 'due to the way that'
      exports.godMontresor = 'esm is linked.'
    }
  }
}

Instantiation hooks can be defined at runtime alongside resolution hooks. They are already present in the current Node.js ESM implementation and will remain present. They are considered an advanced user feature: they allow modules to be created that are not backed directly by a single file on disk.

Transpilation hooks are suitable for the more common task of source-to-source transpilation. They are not be fired for "dynamic" format modules, only for "js" modules.

Transpilation hooks receive a file URL and source text. They are expected to return a String or Promise for a String. Transpilation hooks fire in order, with the result of each hook passed to the subsequent hook. At the end of this process, the String returned is expected to contain valid ESM source text.

Transpilation hooks can be configured at runtime.

You can configure hooks by providing a module indentifier string to node using the --require CLI option. That module identifier must be loadable by the standard node module resolver.

Some examples:

node --require /path/to/typescript-require foo.ts
node --require /path/to/babel-require foo.js

Wrapup

Our evaluation of the spec and the current node implementation lead us to believe that this proposal is feasible. We also feel it addresses the developer concerns we're aware of. We point to an ESM future while maintaining as many of the ergonomics of the current node.js module system as we can. We believe browser-focused developers and node developers will find it workable.

Misc

vm.Module and vm.runInModule should both be exposed eventually.

Relevant links

ECMAScript® 2018 Language Specification: Modules

Chris's walk of the current node experimental implementation

standard-things/esm

@sompylasar
Copy link

I'm sorry if I missed it in the comments, but how is this article relevant? https://medium.com/@giltayar/native-es-modules-in-nodejs-status-and-future-directions-part-i-ee5ea3001f71 — what's more up to date?

@kapv89
Copy link

kapv89 commented Jan 19, 2018

Please, please, please support this export syntax:

export * as foo from './foo'

@focusaurus
Copy link

+1 to @jmeas for --module=cjs and --module=esm. The term legacy is vague, time-relative, and has negative connotation.

@Pauan
Copy link

Pauan commented Jan 24, 2018

@isaacs

As it happens, throwing or polyfilling based on import {require} from 'nodejs' is a lot easier to do statically than finding all instances of import.meta.require. (Eg: const s = 'equire'; const r = import[{toString: () => 'meta'}][unescape('%72')+s]; r('lodash'))

That is incorrect, using import.meta.require is equally as easy / hard to statically analyze as import {require} from 'nodejs', there is absolutely zero difference.

Here is an example to prove my point:

const s = 'equire'; import({toString: () => 'nodejs'}).then((x) => { const r = x[unescape('%72')+s]; return r('lodash'); });

Here is another example:

import * as x from "nodejs";
const s = 'equire'; const r = x[unescape('%72')+s]; r('lodash');

Regardless of whether Node uses import.meta.require or import { require } from 'nodejs', in either case the solution is the same:

  • Transpilers will throw (or warn) on dynamic access to a module namespace object (e.g. x[some_dynamic_expression] in the above example)
  • Transpilers will throw (or warn) on dynamic import(some_dynamic_expression)
  • Transpilers will throw (or warn) on dynamic import.meta[some_dynamic_expression]

P.S. import.meta is special magical syntax, therefore using import[some_dynamic_expression] is not valid syntax, so I assume you meant import.meta[some_dynamic_expression]

@Pauan
Copy link

Pauan commented Jan 24, 2018

Pros of using import.meta.require:

  • More consistent with the browser.

  • import.meta was literally and specifically designed for this situation, it is the Correct Spec-approved Solution(tm).

  • Less confusing for users, because it is simply a module-local object, with straight-forward semantics.

  • Easy for CommonJS users to understand, because import.meta.__dirname is quite similar to __dirname. You simply need to add import.meta. to the front and everything Just Works(tm).

  • Easier to transpile, because there is only a single syntax that the transpiler needs to worry about, and import.meta was designed to be easy to transpile (the transpiler only needs to add an ordinary object to the top of each module and rewrite import.meta to point to that object).

Cons of using import.meta.require:

  • Not currently implemented in Node.

Pros of using import { require } from 'nodejs':

  • Currently implemented in Node.

Cons of using import { require } from 'nodejs':

  • Transpilers need to worry about transpiling both import { require } from 'nodejs' and import('nodejs'), and these two syntaxes probably have different transpilation strategies, which increases the complexity of the transpiler.

  • Much more complex to (correctly) transpile, because it is not a normal object, it's a module namespace, and in addition it's a module namespace which is specific to each module that imports it. The transpiler can't cache it in the same way that it can cache other module namespaces.

  • More confusing for users, because they are importing things which are specific to the current module, but users generally don't expect import ... from ... to work that way.

@devsnek
Copy link

devsnek commented Feb 2, 2018

import.meta landed in node for those keeping score at home

@mpcref
Copy link

mpcref commented Feb 7, 2018

Thanks for taking the time to write this proposal and for implementing it so those that shall not be named here can stop bringing up strawman arguments against it. Nobody in their right mind should want the current experimental mjs file extension-based implementation that has no regard for interoperability outside NodeJS and doesn't solve any of the real world problems but instead introduces a bunch of new ones. It's bad enough already that NodeJS has determined the faith of javascript development in a negative way for so long by using a synchronous module system (which negates elegant solutions for using the same modules in a browser context, resulting in cumbersome workflows) and also effectively killed CommonJS as a standard due to its opinionated implementation of it. Let's hope history doesn't repeat itself or else NodeJS will await the same faith as Internet Explorer.
Sorry for ranting. :)

@championswimmer
Copy link

This proposal makes me happy as a developer who writes significant amount of both nodejs and browser side code.
.mjs files are cruel and evil.

I really hope we end up with this solution and not .mjs

@tamlyn
Copy link

tamlyn commented Mar 13, 2018

I have this nightmare sometimes where a Bad Thing is about to happen but there's a way that it can be avoided. I try to warn people about the Bad Thing but they keep telling me that everything is alright. I get more and more frantic and then I wake up.

I hope we get something like this implementation instead of mjs.

@tolmasky the situation you describe is exactly the same with the current experimental mjs implementation in Node. In both instances it can be mitigated by publishing both CJS and ESM version sources in the same package. The only difference I see is this proposal requires the consuming developer to select which version to import rather than auto selecting based on the file extension. /cc @TimothyGu

@nickpelone
Copy link

Please god let this be what is implemented. .mjs is nothing short of asinine.

@championswimmer
Copy link

Comments like this do not add any value, but I'll nevertheless add it, just to show numbers for the side that doesn't want '.mjs'

I honestly will leave all interest in JS if .mjs becomes a reality. Let this get implemented. Pretty please.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment