Skip to content

Instantly share code, notes, and snippets.

@ceejbot
Last active June 20, 2024 10:45
Show Gist options
  • Save ceejbot/b49f8789b2ab6b09548ccb72813a1054 to your computer and use it in GitHub Desktop.
Save ceejbot/b49f8789b2ab6b09548ccb72813a1054 to your computer and use it in GitHub Desktop.
npm's proposal for supporting ES modules in node

ESM modules in node: npm edition

The proposal you’re about to read is not just a proposal. We have a working implementation of almost everything we discussed here. We encourage you to checkout and build our branch: our fork, with the relevant branch selected. Building and using the implementation will give you a better understanding of what using it as a developer is like.

Our implementation ended up differing from the proposal on some minor points. As our last action item before making a PR, we’re writing documentation on what we did. While I loathe pointing to tests in lieu of documentation, they will be helpful until we complete writing docs: the unit tests.

This repo also contains a bundled version of npm that has a new command, asset. You can read the documentation for and goals of that command in the the repo where we're developing it: GitHub - iarna/assetize: proof-of-concept. The overview is that it turns a node ES module, with node-style dependencies, into a version of the module (with deps) that can be loaded into a browser as-is. It does so by rewriting import paths at install time.

If you have a counter-proposal, I urge you to do what we did and implement it. This will make clear to you what’s feasible given V8 and language constraints, and what might be nice to have but is not feasible. A working implementation is a better thing to discuss than a wishlist.

And now back to the proposal…

JavaScript in browsers has a new module system: ES modules. JavaScript in node.js has had a module system for years: CommonJS modules. JavaScript developers have nearly universally adopted node as the platform of choice for their tooling. Web application frameworks all use command-line tools written in node, pulling shared code from the npm registry, to build applications that run on the browser. Open-source code they find on npm is freely used in these applications. Browser app developers now have the following expectations:

  • Code they write for their tooling uses the same language as their browser applications.
  • Code they discover in npm's module registry is re-usable in the browser.

CommonJS modules do not work in the browser, but bundling tools like browserify and webpack have allowed CommonJS modules to be built into browser-ready code. Early implementations of ES modules rely on the CommonJS module system under the hood, as Babel transpiles ESM syntax into CommonJS syntax.

However, with the arrival of browser support for ES modules, the expectations have changed. Browser developers now expect node to support ESM natively. It does not yet do so.

The node project's proposal for integrating ES modules currently does not satisfy any of its audiences. Browser-focused developers do not like the .mjs file extension proposal. Tooling developers do not have the hooks they need to provide code coverage tools on par with the ones existing for CommonJS, or transpilers like CoffeeScript. Node-focused developers don't want a new module system in the first place.

We propose that npm provide some useful opinions about how to resolve the impasse, in the form of a design, a working implementation of that design, and a pull request to the node project.

The goals of our design are:

  • offer the affordances developers need for dependency injection, transpilation, and test instrumentation
  • satisfy the ergonomics expectations of node-focused developers
  • satisfy the language expectations of browser-focused developers

Our implementation will include real-world examples demonstrating the success of core use cases.

Principles

These are the principles that drive all of our design decisions:

  • Existing CommonJS code must continue to work unchanged.

  • ES modules that don't have platform-specific dependencies must work unchanged in both the browser and node, with the exception that import paths might need to be changed.

  • The parse goal of a module must be known by its consumer in advance. The module producer should write files that end in .js and advertise the module's parse goal in its documentation.

The remainder of this document examines the consequences of these principles on how ES modules would work in node.

Node global objects

The following node global objects are still defined for ES modules, because they are also defined for browser execution environments:

  • console, setTimeout, clearTimeout, setImmediate, setInterval

To get the other node pre-defined globals and module locals, you must import them. import * from 'nodejs' provides the following:

  • process
  • global
  • Buffer
  • require
  • __dirname
  • __filename
  • isMain new!

isMain is true if the current module is a file run directly from node.

We also propose adding loaderhooks as a way of specifying hooks at runtime. An example:

import {loaderHooks} from 'nodejs';
loaderHooks.set({
    instantiate () { /* in pace requiescat */ },
    resolve () { /* sorry fortunato */ }
})
loaderHooks.transpilers.push(async function (url, source) {
	// return a promise for a string
})

Parse goals

Authors do not need to specify parse goals via means like file extensions. Main files are assumed to have the module goal falling back to CommonJS if the runtime encounters require() or any module locals. To avoid the performance hit of the extra parse step, you can invoke node using a flag that invokes common js parsing by default: --mode=legacy. For example, at the top of a script you would use #!/bin/env node --mode=legacy. (The opposing command-line option, --mode=esm, will also short-circuit any attempt to guess the parse goal.)

Additionally, node's existing shebang-stripping parsing logic will be extended to check for --mode=legacy in the shebang of the main module, which will trigger an immediate switch to commonjs mode.

#! /usr/bin/env node --mode=legacy

// We are now in CJS mode without needing a parse step.
const fs = require('fs');

This parse step is only necessary for the initial entry point. All detection of the module goals after that initial entry is driven by the method of consumption: import means the module is parsed ESM, and require that it’s parsed as CJS.

Detecting main

CJS modules currently test if they are main by checking if require.main === module evaluates to true. This does not change. CJS modules can also check module.parent to see if they're the root level module. This should continue to work, but we reserve the right to make module.parent point to a truthy object in cases where a CJS module is being consumed from ESM.

ES modules can test if they are main by examining isMain as imported from 'nodejs'.

#! /usr/bin/env node

import assert from 'assert';
import {isMain} from 'nodejs';

assert(isMain === true);

Resolution

CJS and ESM have identical package resolution logic in node. That is: import * as foo from 'foo' and const foo = require('foo') must resolve to the same file. (That file is likely to be parseable in only one of those goals.) This resolution algorithm continues to do everything it does today (consult package.json for main, look for index.js in a directory, and so on), with the following change.

import './foo' attempts to load code from the exact same locations, with the same priority, as require('./foo'). However:

  1. The resolved file location is read and parsed as ESM, not CJS.
  2. This means that .json and .node will fail, but lifecycle hooks MAY be added to make them work.
  3. The process for extending the ESM loader logic uses lifecycle hooks, not require.extensions.

The node resolver/path resolution mechanism should be implemented as a separate module that can be used by other code; see below for a discussion of hooks that can override this.

ESM/CJS Interoperability in Node.js

Cross-consumption of modules works if authors use the correct consumption method.

  • CJS code can use import('esm').then()
  • ESM code can use import {require} from 'nodejs'; require('cjs')

Consumption of ES modules using CommonJS syntax might fail, and vice versa. Specifically:

  • import 'cjs' fails with an error, because require, exports, and module locals such as __filename are not defined. node intercepts ReferenceError: <module local is not defined and decorates it with with a hint.
  • require('esm') fails with an error, because import and export are not valid keywords in the script goal. node intercepts SyntaxError and decorates it with an error message suggesting that the programmer might need to import() the module.
  • Some cross-consumed modules might not fail. These are likely either to be polyfill libraries or trivial cases. Non-trivial cases can test for this === null to see if they're executing in a CJS context or not.

ESM Interoperability between Node.js and Browsers

If browsers, in the future, add a way to turn import * from 'foo' into a url, then the server responding to requests for that url merely needs to return the expected file, which static servers can do. For example, foo/index.js might be returned when a browser requests foo/, much like how index.html is used.

Until that time, build steps can transform import statements into relative paths to fully-resolved files. More advanced build scripts can reduce the "many request" problems, and HTTP2 can potentially help as well.

The purpose of this proposal is to reduce the friction involved in cross-platform JS applications, especially at development time. We expect that build steps will always be beneficial in preparing apps for production.

node core modules

All node core modules are exported as both ES and CJS modules. The module object exported is identical.

import {require} from 'nodejs';
import * as fs_ns from 'fs';
import fs from "fs"
let fs_req = require('fs');

fs === fs_ns; // false
fs === fs_ns.default; // true
fs === fs_req; // true
fs_req === fs_ns; // false

The following works for all exports from node core modules:

import {readFile} from 'fs';

Lifecycle hooks

The ES module system shall provide three lifecycle hooks for overriding:

  • resolver
  • instantiator
  • transpiler

The resolver stage is responsible for resolving user-provided import strings into full URLs and their expected format. The default resolver implements the Node module resolution algorithm. For example: it resolves the "./foo.js" from import foo from "./foo.js" into "file:///Users/basil/foo.js" (with a format of js), or "browserify" to "file:///Users/basil/node_modules/browserify". This stage is already present and hookable in Node.js' ESM implementation, and will remain present. Custom hooks allow users to redefine how resolution works for applications. Configuration of these hooks can be done at runtime, and the hooks are applied to the next module graph load. To specify ahead of your entry point, use -r.

The instantiation phase is responsible for turning a file URL into a list of exported names and an evaluate function. The evaluate function is called with an object that allows the exported names to be modified.

function instantiate (url) {
  return {
    exports: ['forThe', 'loveOf', 'godMontresor'],
    evaluate (exports) {
      exports.forThe = 'these names have to be known in advance'
      exports.loveOf = 'due to the way that'
      exports.godMontresor = 'esm is linked.'
    }
  }
}

Instantiation hooks can be defined at runtime alongside resolution hooks. They are already present in the current Node.js ESM implementation and will remain present. They are considered an advanced user feature: they allow modules to be created that are not backed directly by a single file on disk.

Transpilation hooks are suitable for the more common task of source-to-source transpilation. They are not be fired for "dynamic" format modules, only for "js" modules.

Transpilation hooks receive a file URL and source text. They are expected to return a String or Promise for a String. Transpilation hooks fire in order, with the result of each hook passed to the subsequent hook. At the end of this process, the String returned is expected to contain valid ESM source text.

Transpilation hooks can be configured at runtime.

You can configure hooks by providing a module indentifier string to node using the --require CLI option. That module identifier must be loadable by the standard node module resolver.

Some examples:

node --require /path/to/typescript-require foo.ts
node --require /path/to/babel-require foo.js

Wrapup

Our evaluation of the spec and the current node implementation lead us to believe that this proposal is feasible. We also feel it addresses the developer concerns we're aware of. We point to an ESM future while maintaining as many of the ergonomics of the current node.js module system as we can. We believe browser-focused developers and node developers will find it workable.

Misc

vm.Module and vm.runInModule should both be exposed eventually.

Relevant links

ECMAScript® 2018 Language Specification: Modules

Chris's walk of the current node experimental implementation

standard-things/esm

@jdalton
Copy link

jdalton commented Jan 13, 2018

@domenic

The import {readFile} = require('fs') is a typo.
Based on the working implementation it's import {readFile} from "fs"

@allenwb

The

import {require} from 'nodejs';
import * from 'fs' as fs;
fs === require('fs'); // true

is typo'd too. Based on the working implementation it's

import {require} from 'nodejs';
import * as fs_ns from 'fs';
import fs from "fs"
let fs_req = require('fs');

fs === fs_ns; // false
fs === fs_ns.default; // true
fs === fs_req; // true
fs_req === fs_ns; // false

as you would expect.

@ceejbot
Copy link
Author

ceejbot commented Jan 13, 2018

Thanks, JDD. This proposal led the working implementation by a bit, and I did not update it after the rubber met the road of working code. A documentation pass is badly needed. I'll revise now to get rid of some attractive nuisance typos.

@ericblade
Copy link

@Jamesernator

This isn't true in browser land, in order to use a module in the browser it has to be served with a text/javascript (or equivalent) MIME type, it's only the combination of import + MIME type that determines the file as a Javascript module source. In future it's possible that something like WebAssembly might be import-able in which case the MIME type would be different.

I'm pretty sure that browsers can operate upon local files, even with modules, therefore it wouldn't make sense to have a requirement for a server providing a MIME type. I don't know exactly where the specs are located, but I'm really close to absolutely positive that the "module-ness" of a ESM module is determined by it fitting one of two things: loaded via script tag with type="module" (which should, imo, be a separate attribute from 'type' because we already use 'type' for 'text/javascript', and type=module is not the same information as type=text/javascript ..) or imported from ES6 import statement.

In Node, it seems that the module-ness of an item is determined by both being true: .mjs file, and it exports something. (attempting to import a module that doesn't export in 9.3.0 at least resulted in an error when I tried it yesterday)

All that said, I don't like the file extension requirements, particularly as people are now starting to use node as a general scripting language from the shell prompt, it's a bad idea as a whole, IMO. I think as a community, we can come up with a better way, but it might be good to do it when the existing implementation is a little more complete, but definitely don't wait until it's accepted outside of the --experimental-modules switch, that might be too late to get changes. Reason for that being, it's a lot easier to demonstrate criticisms of it on a functioning setup, than it is to rehash it all out in theory, after it's already gone through months of hashing out.

However, I don't have more than a few minutes experience with messing with this yet, so I have to limit my criticisms to this: The way we're going in Node right now, requiring a specific file extension, isn't good. There are worse things that can be done, but this isn't good. This proposal does attempt to fix that, but I suspect that there are a lot of hidden gotchas (as other people have described above) that this doesn't address. My idea posted above probably also has a lot of hidden gotchas. Lots of hidden gotchas were likely discussed over the last several months, and I think it might be a good idea for some code smiths to hammer out some ideas, and try them out, in the near future, now that we have module support, and we can play with it.

I also disagree with adding built-in support to the primary Javascript engine that is in use in the world to directly support transpilation. My first thought is that could probably be added via some sort of native plugin, but I'm not at all familiar with the Node code base to know that for sure. I think that is highly useful for transpiler authors, but I feel that it would lead to very bad habits overall.

@TimothyGu
Copy link

PSA: Node.js will soon support import { readFile } from 'fs';! See nodejs/node#18131 for details.

After this PR, what @allenwb proposed is going to fully work as expected in Node.js.


@ericblade

I'm pretty sure that browsers can operate upon local files, even with modules,

How exactly local files are fetched in browsers is currently unfortunately under-defined; however, if you look at the surrounding steps in that link for other protocols these steps will probably emulate a normal HTTP response, i.e. it will contain looking up the MIME type from a file extension-to-MIME type database. Which is what browsers currently do.

therefore it wouldn't make sense to have a requirement for a server providing a MIME type.

The HTML spec does indeed have this requirement. See https://html.spec.whatwg.org/multipage/webappapis.html#fetch-a-single-module-script, step 8:

  1. (Paraphrasing here) The fetch fails when one of the following is true:

    • response's type is "error"
    • response's status is not an ok status
    • The result of extracting a MIME type from response's header list (ignoring parameters) is not a JavaScript MIME type

@jamesplease
Copy link

jamesplease commented Jan 14, 2018

Overall, I like this a lot. A bunch of good points have been mentioned in the comments. One thought I had is that --module=cjs and --module=esm seem like more expressive flags than terms like “mode” and “legacy,” so I prefer them for that reason.

It also allows for this to be released in a non-breaking way by making cjs the default. Then, at a later date, the flags could be swapped.

@ceejbot
Copy link
Author

ceejbot commented Jan 14, 2018

The names of flags and their values is a bikeshed that I am happy to leave painted any way the node project prefers; I merely propose their existence.

For the commenters with ideas about how things might work differently, I suggest doing what we did, forking either our branch or node's upstream, and implementing your idea to see how it works. You'll learn what's feasible and what's not, and get some direct experience with using the apis and design you're considering. I believe strongly right now that working code expressing meaningful choices among the options is the best way to move forward. We found it easier to do than we'd feared. The foundational work in node & v8 is solid.

@MarkTiedemann
Copy link

I'd prefer from 'node' over from 'nodejs'. :)

@bathos
Copy link

bathos commented Jan 14, 2018

Re:

It’d be great to have:

import.meta.isMain
import.meta.url

and its response,

We ended up avoiding import.meta because we didn't need it, and we liked the clear context-setting of requiring things from "nodejs"-- this makes it clear that you're doing something that isn't going to work in the browser context that will need shimming. But I think it could work just fine either way.

I would also prefer seeing import.meta for these. Somebody who knows more should correct me if I’m wrong, but I’m pretty sure that:

a. the properties of import.meta are environment-specific by definition
b. the import.meta object of a module is mutable to permit polyfilling for cases just like this, so it’s not necessarilly the case that import.meta.isMain would have to be unavailable outside of node.

In contrast, it seems importing bindings from the same module and getting different bindings back per importee is something that isn’t possible in the browser at all — I think it’s less, not more, shimmable, since unlike node, the browser is unlikely to ever afford "meta control" over module loading.


Edit: on reflection, I guess in both cases it’s about equally shimmable — both require a build step of some sort — but the simplicity of just prepending the line import.meta.someNodeThing = ... without changing any of the original code seems preferable.

@isaacs
Copy link

isaacs commented Jan 14, 2018

My 2¢ on import.meta.require/__dirname/etc vs import {require, __dirname, etc.} from "nodejs"

These fields are, in commonjs land, "module-local free variables". They're not globals, they're module-specific, and they're set automatically. It seemed reasonable to me that import.meta.require was the way to go for that reason. @ceejbot and @isntitvacant changed my mind.

Modules loaded in commonjs are a singleton per resolved path. There is no guarantee in node that require('lodash') always resolves to the same thing. In fact, the whole point of the nested node_modules resolution algorithm is to allow package managers to serve different versions of named dependencies to different importing modules. This is no different than if every node module had a file at ./node_modules/nodejs that implemented these fields.

The fact that these vars are module-local gives import.meta.require a sort of intuitive sense. It is also closer to what browsers (plan to?) do for platform-specific bits.

In practice we can expect that people writing code for both Node.js and the browser will have to transpile, even if they are using native es modules in both platforms. The most common reason for this is paths. import {test} from 'tap' is not valid on the browser because 'tap' isn't a valid import url.

However, they will not have to transpile code from es modules into commonjs, or from commonjs into something else. The only work for such a transpiler is handling import urls. (Principle 2 of 3)

Anyone using es modules in both node and the browser will need to either polyfill require or prevent its use. As it happens, throwing or polyfilling based on import {require} from 'nodejs' is a lot easier to do statically than finding all instances of import.meta.require. (Eg: const s = 'equire'; const r = import[{toString: () => 'meta'}][unescape('%72')+s]; r('lodash'))

Thus, Principle 2 was easier to meet using import {require} from 'nodejs' than import.meta.require. A transpiler could either implement require() or statically and deterministically crash, rather than ship a broken program. If it turns out that import.meta.require actually is better for shipping to browsers, then the import-path-munging transpilers can turn it into that easily enough. If it isn't better, than we've lost nothing.

Also, a variable import-module-specific version of import {require} from 'nodejs' was already implemented as part of the --experimental-module-system feature.

Running code and practical considerations should overrule symmetrical esthetics.

It seems to me that import {require} from 'nodejs' is more in line with the principles of this proposal than import.meta.require. Objectors to that approach should perhaps take issue with the principles instead. No point debating tactics without a clearly articulated shared strategic goal.

@TimothyGu
Copy link

TimothyGu commented Jan 14, 2018

@isaacs Thank you for your explanation. The choice behind import { require } from 'nodejs' is a lot more understandable now, and sounds more easily implementable because of this bit:

This is no different than if every node module had a file at ./node_modules/nodejs that implemented these fields.

I am now neutral on this versus import.meta.require – though having it available to ES modules at all is a different debate (my opinion is no, because there would be no need if import supports CJS, but can be convinced otherwise).

Regarding import.meta, I do want it to continue to be used, especially for cases that browsers implement like import.meta.url, even though it's not "necessary" for Node.js scripts.

Regarding __dirname and __filename, I am ambivalent on having them available at all. Having them available to Node.js' ES modules will certainly simplify some file operations, but may encourage increased dependence on Node.js-specific features rather than primitives (like import.meta.url) that are sure to be implemented in both Node.js and browsers. I can easily imagine modules that rely on no other Node.js features than __filename use __filename instead of import.meta.url, thus hindering its usage in browsers (see Principle 2 of 3).


Objectors to that approach should perhaps take issue with the principles instead.

Though the "that approach" in this sentence refers specifically to the debate of import { require } from 'nodejs'; vs. import.meta.require, I think "taking issues with the principles" can help organize my objections to this proposal.

Principle 1

Existing CommonJS code must continue to work unchanged.

I fully agree with this principle. The current ESM implementation in Node.js fulfills this principle.

However, the proposal as it currently stands (rev. fea6cb08fa8920604c8c9193f510a7a47d856cc4) does not fulfill this principle, because of the intrinsic deficiencies the parse goal detection algorithm (relevant parts reproduced below) proposed has.

Authors do not need to specify parse goals via means like file extensions. Main files are assumed to have the module goal falling back to CommonJS if the runtime encounters require() or any module locals.

...

Additionally, node's existing shebang-stripping parsing logic will be extended to check for --mode=legacy in the shebang of the main module, which will trigger an immediate switch to commonjs mode.

  1. There are no reliable way of detecting require() without running the code. And even so, require() is not a reliable way to detect things. A few scenarios follow:

    1. "I intend this file to be ESM, but I accidentally put a require() in the file; now Node.js outputs an error message about my import statement rather than the require(), causing confusion."
    2. "I intend this file to be ESM, but I extended the global object with a require() method of my own."
  2. There are syntactic ambiguities between the Module and Script parse goals. Some of these ambiguities can cause issues. One of them is raised above by @wavebeem.

As https://github.com/bmeck/UnambiguousJavaScriptGrammar shows, a revised version of this algorithm is probably "fine" for most cases. But as a platform-level product, Node.js should not aim for "fine for most cases" but "rock-solid for all cases".

Principle 2

ES modules that don't have platform-specific dependencies must work unchanged in both the browser and node, with the exception that import paths might need to be changed.

I fully agree with this principle. The current ESM implementation in Node.js also fulfills this principle.

See my concerns above about __dirname and __filename however.

Principle 3

The parse goal of a module must be known by its consumer in advance. The module producer should write files that end in .js and advertise the module's parse goal in its documentation.

I don't necessarily disagree with this principle, though it feels like the means to another principle, than a principle in and of itself. Concretely, I as a consumer don't see why I need to know the parse goal of a module; rather this principle sounds like the means to another unstated principle "All JavaScript files must use .js only".

Whether the principle is means or ends, I believe the following competing principle is of more importance than this:

A large codebase must be allowed to gradually transition from CJS to ESM.

The current ESM implementation in Node.js does not fulfill the first principle but does the second, with this proposal exactly opposite (see @tolmasky's comment).

I am of course open to a proposal that allows for both, but so far a solution has not surfaced without breaking one of the other principles.

@JimPanic
Copy link

Yes, please! This proposal seems reasonable! (More reasonable than a separate extension and all that Jazz.)

@isaacs
Copy link

isaacs commented Jan 14, 2018

@TimothyGu

(I'm not sure what "rev fea6cb08fa8920604c8c9193f510a7a47d856cc4" refers to, can you provide a link?)

There are no reliable way of detecting require() without running the code. And even so, require() is not a reliable way to detect things. A few scenarios follow:

While I think your concern here is the right place to poke at, I'm struggling to find a case where it'll behave badly.

"I intend this file to be ESM, but I accidentally put a require() in the file; now Node.js outputs an error message about my import statement rather than the require(), causing confusion."

That's incorrect. If there's an import or export statement, then it forces ESM mode. Your assumption is reasonable from the gist contents itself, but the code is not ambiguous, and it changes some of your conclusions meaningfully, I believe.

$ cat x.js
require('fs')
import {readFile} from 'fs' // <-- triggers ESM mode

$ ./node x.js
ReferenceError: require is not defined
    at file:///Users/isaacs/dev/js/node-master/x.js:1:1
    at ModuleJob.run (internal/loader/ModuleJob.js:104:16)
    at <anonymous>

(Whether it still "causes confusion" or not is debatable.)

If you are intending ESM and incorrectly call require, then you'll either have import statements or not. If you do, you'll get a run-time error (as one would expect from any other undefined function).

So the case remaining is where you intend ESM mode, have no import or export statements, and call require(). But... I think the right answer here is to run the code in CJS mode, because the module is otherwise trivial, and doing commonjs type things.

"I intend this file to be ESM, but I extended the global object with a require() method of my own."

The only way to do this and still have the parse mode be ambiguous is if you do node -r ./define-global-require.js ./x.js. (That is, if x.js is intended to be ESM, but not the main module, then the module loading it will either be using import or require() and thus the parse mode is unambiguous.)

If that module has any import/export statements, then it's unambiguously ESM. So, the remaining case is a module that should be ESM, where a global require is defined in a -r argument module, and the main module has no import or export statements. I think this is a suitably narrow edge case to be handled with documentation, and implies such a trivial module as to be mostly irrelevant. And, the --mode=esm flag is a suitable workaround.

(I'm using require here as a stand-in for the whole set of module locals.)

A large codebase must be allowed to gradually transition from CJS to ESM.

This can be done gradually by using import() in CJS code and import {require} from 'nodejs' in ESM code. The challenge is:

  1. import() returns a promise. So, previously synchronously loaded deps become async.
  2. When the main module changes, it changes the way that consumers have to load the module.

Either of these may require a major version bump. But I don't believe that's a profound blocker. I also suspect that most codebases will switch over rather quickly, as we've seen in some major projects adopting Promises, ES classes, and other language features.

It's counterintuitive, but in a community that is still growing so quickly, the future is much larger than the past. We'll outgrow and abandon most legacy code rather than port it. Major version bumps (and, when that fails, completely new projects/forks to replace old ones) are a very normal process for adopting new features.

@ceejbot
Copy link
Author

ceejbot commented Jan 14, 2018

@isaacs says:

It's counterintuitive, but in a community that is still growing so quickly, the future is much larger than the past. We'll outgrow and abandon most legacy code rather than port it.

This was one of the major drivers of the tradeoffs in this proposal. They are tradeoffs, and there are other workable solutions that make different tradeoffs. Be clear about what your goals are first, then make tradeoffs in service of those goals. Then you can, as we did, figure out if they're feasible in a working implementation.

We chose to treat the current situation as transitional, and make tradeoffs that will be more pleasant once the community is 90% ESM instead of 90% CJS (and falling). (I don't know what the current split of code on the registry is. That would be interesting to know.)

@TimothyGu
Copy link

@isaacs

(I'm not sure what "rev fea6cb08fa8920604c8c9193f510a7a47d856cc4" refers to, can you provide a link?)

I was referring to the revision of the Gist: https://gist.github.com/ceejbot/b49f8789b2ab6b09548ccb72813a1054/fea6cb08fa8920604c8c9193f510a7a47d856cc4

(It's also the commit hash of the Gist if you clone it as a Git repo.)

If there's an import or export statement, then it forces ESM mode.

This is the missing piece that I couldn't find in the initial proposal. Thank you for clarifying, it makes more sense now. (I suspect this could probably be added to the proposal itself.)

I have more specific thoughts about how the npm fork currently implements this scanning, specifically that by using a JavaScript parser such as Acorn it disallows newer features of JavaScript that V8 implements but Acorn doesn't (such like object spread) in CJS. But that may well be intentional.

There remains the question of how the following is treated:

eval('var a = 0');
console.log(a);

this is parsable both CJS and ESM, and by the tiebreakers in this proposal it would be ESM. Yet it can only run under CJS mode because of strict-mode semantics of eval(). (The same issue also applies to eval('delete x'); etc.) (Maybe @chrisdickinson or @ceejbot may be answer this better?)


@isaacs

It's counterintuitive, but in a community that is still growing so quickly, the future is much larger than the past. We'll outgrow and abandon most legacy code rather than port it.

@ceejbot

We chose to treat the current situation as transitional, and make tradeoffs that will be more pleasant once the community is 90% ESM instead of 90% CJS (and falling).

This is indeed the part of the proposal I had (and still have) the most trouble wrapping my head around. The scenario described by @tolmasky is too undesirable, in my opinion. Though I have to acknowledge you (the authors of this proposal) are much more experienced than me in the realm of JavaScript.


I do very much appreciate the time all of you took to answer my questions. I would also like a chance to chat if that is at all possible.

@domenic
Copy link

domenic commented Jan 14, 2018

It's true that it's technically possible to make import { require } from "nodejs" work. But it's frustrating to see that, this object which we specifically created in response to the Node.js community's requests for a place for per-module metadata and functionality, is not being used as such.

In particular, it muddles the purpose of the module system and of import statements, if sometimes import statements are used as a way of getting information about the current module into scope, and sometimes are used as a way of importing functionality and data from a separate package. Previously, "bare" import statements (no ./ or ../) always referred to other packages. Now, they have two competing use cases.

What's worse, there are now two ways to get per-module metadata: one via import { x } from "nodejs", for npm's proposed set of properties, and one from import.meta, for Node's set of properties (such as import.meta.url, and potentially others). What is a user to think? Why does import.meta.url work, but when they want to go to get a filename, import.meta.filename does not? Why does import { isMain } from "nodejs" work, but when they want to go and get a URL, import { url } from "nodejs" does not?

I'd urge you to consider the teachability of this direction, as well as its impact on the wider ecosystem of module users. Dismissing these concerns as "symmetrical esthetics" is, IMO, wrong. They directly bear on the principles of "satisfying the language expectations of browser-focused developers", and even of "satisfying the ergonomics expectations of node-focused developers" given the mismatch between import.meta.url and the npm-proposed properties.

The cited benefits are that it's harder to obfuscate code and prevent transpilation from working. But I don't think accounting for such obfuscated code should weigh very highly.

@teazean
Copy link

teazean commented Jan 15, 2018

Great

@isaacs
Copy link

isaacs commented Jan 15, 2018

@domenic The confusion between some node-specific things being on import.meta, and others on the “nodejs” virtual module, is relevant, imo.

However, I disagree that optimizing for build tooling isn’t appropriate. Most code on npm today (and in the foreseeable future) is intended for consumption by browsers, and passes through a build step. It seems reasonable to me to consider this use case carefully. Even though this proposal is “a node thing for node programs”, and shouldn’t be limited excessively by browser implementations, one common use for node programs is being compiled for browsers.

It could be reasonable to say that node-only properties (like require) are on a nodejs virtual module, and properties shared between the browser and node (like url) will be on the meta object; even if they have node-specific values in node programs.

@domenic
Copy link

domenic commented Jan 16, 2018

I didn't say anything against optimizing for build tooling. I simply said that you shouldn't optimize for obfuscated programs.

@isaacs
Copy link

isaacs commented Jan 16, 2018

@domenic Ah, yes, I was making an assumption. My bad. Yes, obfuscation seems less relevant than transpiling. One might argue that some "transpiling" is obfuscatory, or split hairs and say "obfuscation" is just a special kind of "transpiling", but I think we can agree those arguments strain good faith. :)

@sompylasar
Copy link

I'm sorry if I missed it in the comments, but how is this article relevant? https://medium.com/@giltayar/native-es-modules-in-nodejs-status-and-future-directions-part-i-ee5ea3001f71 — what's more up to date?

@kapv89
Copy link

kapv89 commented Jan 19, 2018

Please, please, please support this export syntax:

export * as foo from './foo'

@focusaurus
Copy link

+1 to @jmeas for --module=cjs and --module=esm. The term legacy is vague, time-relative, and has negative connotation.

@Pauan
Copy link

Pauan commented Jan 24, 2018

@isaacs

As it happens, throwing or polyfilling based on import {require} from 'nodejs' is a lot easier to do statically than finding all instances of import.meta.require. (Eg: const s = 'equire'; const r = import[{toString: () => 'meta'}][unescape('%72')+s]; r('lodash'))

That is incorrect, using import.meta.require is equally as easy / hard to statically analyze as import {require} from 'nodejs', there is absolutely zero difference.

Here is an example to prove my point:

const s = 'equire'; import({toString: () => 'nodejs'}).then((x) => { const r = x[unescape('%72')+s]; return r('lodash'); });

Here is another example:

import * as x from "nodejs";
const s = 'equire'; const r = x[unescape('%72')+s]; r('lodash');

Regardless of whether Node uses import.meta.require or import { require } from 'nodejs', in either case the solution is the same:

  • Transpilers will throw (or warn) on dynamic access to a module namespace object (e.g. x[some_dynamic_expression] in the above example)
  • Transpilers will throw (or warn) on dynamic import(some_dynamic_expression)
  • Transpilers will throw (or warn) on dynamic import.meta[some_dynamic_expression]

P.S. import.meta is special magical syntax, therefore using import[some_dynamic_expression] is not valid syntax, so I assume you meant import.meta[some_dynamic_expression]

@Pauan
Copy link

Pauan commented Jan 24, 2018

Pros of using import.meta.require:

  • More consistent with the browser.

  • import.meta was literally and specifically designed for this situation, it is the Correct Spec-approved Solution(tm).

  • Less confusing for users, because it is simply a module-local object, with straight-forward semantics.

  • Easy for CommonJS users to understand, because import.meta.__dirname is quite similar to __dirname. You simply need to add import.meta. to the front and everything Just Works(tm).

  • Easier to transpile, because there is only a single syntax that the transpiler needs to worry about, and import.meta was designed to be easy to transpile (the transpiler only needs to add an ordinary object to the top of each module and rewrite import.meta to point to that object).

Cons of using import.meta.require:

  • Not currently implemented in Node.

Pros of using import { require } from 'nodejs':

  • Currently implemented in Node.

Cons of using import { require } from 'nodejs':

  • Transpilers need to worry about transpiling both import { require } from 'nodejs' and import('nodejs'), and these two syntaxes probably have different transpilation strategies, which increases the complexity of the transpiler.

  • Much more complex to (correctly) transpile, because it is not a normal object, it's a module namespace, and in addition it's a module namespace which is specific to each module that imports it. The transpiler can't cache it in the same way that it can cache other module namespaces.

  • More confusing for users, because they are importing things which are specific to the current module, but users generally don't expect import ... from ... to work that way.

@devsnek
Copy link

devsnek commented Feb 2, 2018

import.meta landed in node for those keeping score at home

@mpcref
Copy link

mpcref commented Feb 7, 2018

Thanks for taking the time to write this proposal and for implementing it so those that shall not be named here can stop bringing up strawman arguments against it. Nobody in their right mind should want the current experimental mjs file extension-based implementation that has no regard for interoperability outside NodeJS and doesn't solve any of the real world problems but instead introduces a bunch of new ones. It's bad enough already that NodeJS has determined the faith of javascript development in a negative way for so long by using a synchronous module system (which negates elegant solutions for using the same modules in a browser context, resulting in cumbersome workflows) and also effectively killed CommonJS as a standard due to its opinionated implementation of it. Let's hope history doesn't repeat itself or else NodeJS will await the same faith as Internet Explorer.
Sorry for ranting. :)

@championswimmer
Copy link

This proposal makes me happy as a developer who writes significant amount of both nodejs and browser side code.
.mjs files are cruel and evil.

I really hope we end up with this solution and not .mjs

@tamlyn
Copy link

tamlyn commented Mar 13, 2018

I have this nightmare sometimes where a Bad Thing is about to happen but there's a way that it can be avoided. I try to warn people about the Bad Thing but they keep telling me that everything is alright. I get more and more frantic and then I wake up.

I hope we get something like this implementation instead of mjs.

@tolmasky the situation you describe is exactly the same with the current experimental mjs implementation in Node. In both instances it can be mitigated by publishing both CJS and ESM version sources in the same package. The only difference I see is this proposal requires the consuming developer to select which version to import rather than auto selecting based on the file extension. /cc @TimothyGu

@nickpelone
Copy link

Please god let this be what is implemented. .mjs is nothing short of asinine.

@championswimmer
Copy link

Comments like this do not add any value, but I'll nevertheless add it, just to show numbers for the side that doesn't want '.mjs'

I honestly will leave all interest in JS if .mjs becomes a reality. Let this get implemented. Pretty please.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment