Create a gist now

Instantly share code, notes, and snippets.

Embed
What would you like to do?
npm's proposal for supporting ES modules in node

ESM modules in node: npm edition

The proposal you’re about to read is not just a proposal. We have a working implementation of almost everything we discussed here. We encourage you to checkout and build our branch: our fork, with the relevant branch selected. Building and using the implementation will give you a better understanding of what using it as a developer is like.

Our implementation ended up differing from the proposal on some minor points. As our last action item before making a PR, we’re writing documentation on what we did. While I loathe pointing to tests in lieu of documentation, they will be helpful until we complete writing docs: the unit tests.

This repo also contains a bundled version of npm that has a new command, asset. You can read the documentation for and goals of that command in the the repo where we're developing it: GitHub - iarna/assetize: proof-of-concept. The overview is that it turns a node ES module, with node-style dependencies, into a version of the module (with deps) that can be loaded into a browser as-is. It does so by rewriting import paths at install time.

If you have a counter-proposal, I urge you to do what we did and implement it. This will make clear to you what’s feasible given V8 and language constraints, and what might be nice to have but is not feasible. A working implementation is a better thing to discuss than a wishlist.

And now back to the proposal…

JavaScript in browsers has a new module system: ES modules. JavaScript in node.js has had a module system for years: CommonJS modules. JavaScript developers have nearly universally adopted node as the platform of choice for their tooling. Web application frameworks all use command-line tools written in node, pulling shared code from the npm registry, to build applications that run on the browser. Open-source code they find on npm is freely used in these applications. Browser app developers now have the following expectations:

  • Code they write for their tooling uses the same language as their browser applications.
  • Code they discover in npm's module registry is re-usable in the browser.

CommonJS modules do not work in the browser, but bundling tools like browserify and webpack have allowed CommonJS modules to be built into browser-ready code. Early implementations of ES modules rely on the CommonJS module system under the hood, as Babel transpiles ESM syntax into CommonJS syntax.

However, with the arrival of browser support for ES modules, the expectations have changed. Browser developers now expect node to support ESM natively. It does not yet do so.

The node project's proposal for integrating ES modules currently does not satisfy any of its audiences. Browser-focused developers do not like the .mjs file extension proposal. Tooling developers do not have the hooks they need to provide code coverage tools on par with the ones existing for CommonJS, or transpilers like CoffeeScript. Node-focused developers don't want a new module system in the first place.

We propose that npm provide some useful opinions about how to resolve the impasse, in the form of a design, a working implementation of that design, and a pull request to the node project.

The goals of our design are:

  • offer the affordances developers need for dependency injection, transpilation, and test instrumentation
  • satisfy the ergonomics expectations of node-focused developers
  • satisfy the language expectations of browser-focused developers

Our implementation will include real-world examples demonstrating the success of core use cases.

Principles

These are the principles that drive all of our design decisions:

  • Existing CommonJS code must continue to work unchanged.

  • ES modules that don't have platform-specific dependencies must work unchanged in both the browser and node, with the exception that import paths might need to be changed.

  • The parse goal of a module must be known by its consumer in advance. The module producer should write files that end in .js and advertise the module's parse goal in its documentation.

The remainder of this document examines the consequences of these principles on how ES modules would work in node.

Node global objects

The following node global objects are still defined for ES modules, because they are also defined for browser execution environments:

  • console, setTimeout, clearTimeout, setImmediate, setInterval

To get the other node pre-defined globals and module locals, you must import them. import * from 'nodejs' provides the following:

  • process
  • global
  • Buffer
  • require
  • __dirname
  • __filename
  • isMain new!

isMain is true if the current module is a file run directly from node.

We also propose adding loaderhooks as a way of specifying hooks at runtime. An example:

import {loaderHooks} from 'nodejs';
loaderHooks.set({
    instantiate () { /* in pace requiescat */ },
    resolve () { /* sorry fortunato */ }
})
loaderHooks.transpilers.push(async function (url, source) {
	// return a promise for a string
})

Parse goals

Authors do not need to specify parse goals via means like file extensions. Main files are assumed to have the module goal falling back to CommonJS if the runtime encounters require() or any module locals. To avoid the performance hit of the extra parse step, you can invoke node using a flag that invokes common js parsing by default: --mode=legacy. For example, at the top of a script you would use #!/bin/env node --mode=legacy. (The opposing command-line option, --mode=esm, will also short-circuit any attempt to guess the parse goal.)

Additionally, node's existing shebang-stripping parsing logic will be extended to check for --mode=legacy in the shebang of the main module, which will trigger an immediate switch to commonjs mode.

#! /usr/bin/env node --mode=legacy

// We are now in CJS mode without needing a parse step.
const fs = require('fs');

This parse step is only necessary for the initial entry point. All detection of the module goals after that initial entry is driven by the method of consumption: import means the module is parsed ESM, and require that it’s parsed as CJS.

Detecting main

CJS modules currently test if they are main by checking if require.main === module evaluates to true. This does not change. CJS modules can also check module.parent to see if they're the root level module. This should continue to work, but we reserve the right to make module.parent point to a truthy object in cases where a CJS module is being consumed from ESM.

ES modules can test if they are main by examining isMain as imported from 'nodejs'.

#! /usr/bin/env node

import assert from 'assert';
import {isMain} from 'nodejs';

assert(isMain === true);

Resolution

CJS and ESM have identical package resolution logic in node. That is: import * as foo from 'foo' and const foo = require('foo') must resolve to the same file. (That file is likely to be parseable in only one of those goals.) This resolution algorithm continues to do everything it does today (consult package.json for main, look for index.js in a directory, and so on), with the following change.

import './foo' attempts to load code from the exact same locations, with the same priority, as require('./foo'). However:

  1. The resolved file location is read and parsed as ESM, not CJS.
  2. This means that .json and .node will fail, but lifecycle hooks MAY be added to make them work.
  3. The process for extending the ESM loader logic uses lifecycle hooks, not require.extensions.

The node resolver/path resolution mechanism should be implemented as a separate module that can be used by other code; see below for a discussion of hooks that can override this.

ESM/CJS Interoperability in Node.js

Cross-consumption of modules works if authors use the correct consumption method.

  • CJS code can use import('esm').then()
  • ESM code can use import {require} from 'nodejs'; require('cjs')

Consumption of ES modules using CommonJS syntax might fail, and vice versa. Specifically:

  • import 'cjs' fails with an error, because require, exports, and module locals such as __filename are not defined. node intercepts ReferenceError: <module local is not defined and decorates it with with a hint.
  • require('esm') fails with an error, because import and export are not valid keywords in the script goal. node intercepts SyntaxError and decorates it with an error message suggesting that the programmer might need to import() the module.
  • Some cross-consumed modules might not fail. These are likely either to be polyfill libraries or trivial cases. Non-trivial cases can test for this === null to see if they're executing in a CJS context or not.

ESM Interoperability between Node.js and Browsers

If browsers, in the future, add a way to turn import * from 'foo' into a url, then the server responding to requests for that url merely needs to return the expected file, which static servers can do. For example, foo/index.js might be returned when a browser requests foo/, much like how index.html is used.

Until that time, build steps can transform import statements into relative paths to fully-resolved files. More advanced build scripts can reduce the "many request" problems, and HTTP2 can potentially help as well.

The purpose of this proposal is to reduce the friction involved in cross-platform JS applications, especially at development time. We expect that build steps will always be beneficial in preparing apps for production.

node core modules

All node core modules are exported as both ES and CJS modules. The module object exported is identical.

import {require} from 'nodejs';
import * as fs_ns from 'fs';
import fs from "fs"
let fs_req = require('fs');

fs === fs_ns; // false
fs === fs_ns.default; // true
fs === fs_req; // true
fs_req === fs_ns; // false

The following works for all exports from node core modules:

import {readFile} from 'fs';

Lifecycle hooks

The ES module system shall provide three lifecycle hooks for overriding:

  • resolver
  • instantiator
  • transpiler

The resolver stage is responsible for resolving user-provided import strings into full URLs and their expected format. The default resolver implements the Node module resolution algorithm. For example: it resolves the "./foo.js" from import foo from "./foo.js" into "file:///Users/basil/foo.js" (with a format of js), or "browserify" to "file:///Users/basil/node_modules/browserify". This stage is already present and hookable in Node.js' ESM implementation, and will remain present. Custom hooks allow users to redefine how resolution works for applications. Configuration of these hooks can be done at runtime, and the hooks are applied to the next module graph load. To specify ahead of your entry point, use -r.

The instantiation phase is responsible for turning a file URL into a list of exported names and an evaluate function. The evaluate function is called with an object that allows the exported names to be modified.

function instantiate (url) {
  return {
    exports: ['forThe', 'loveOf', 'godMontresor'],
    evaluate (exports) {
      exports.forThe = 'these names have to be known in advance'
      exports.loveOf = 'due to the way that'
      exports.godMontresor = 'esm is linked.'
    }
  }
}

Instantiation hooks can be defined at runtime alongside resolution hooks. They are already present in the current Node.js ESM implementation and will remain present. They are considered an advanced user feature: they allow modules to be created that are not backed directly by a single file on disk.

Transpilation hooks are suitable for the more common task of source-to-source transpilation. They are not be fired for "dynamic" format modules, only for "js" modules.

Transpilation hooks receive a file URL and source text. They are expected to return a String or Promise for a String. Transpilation hooks fire in order, with the result of each hook passed to the subsequent hook. At the end of this process, the String returned is expected to contain valid ESM source text.

Transpilation hooks can be configured at runtime.

You can configure hooks by providing a module indentifier string to node using the --require CLI option. That module identifier must be loadable by the standard node module resolver.

Some examples:

node --require /path/to/typescript-require foo.ts
node --require /path/to/babel-require foo.js

Wrapup

Our evaluation of the spec and the current node implementation lead us to believe that this proposal is feasible. We also feel it addresses the developer concerns we're aware of. We point to an ESM future while maintaining as many of the ergonomics of the current node.js module system as we can. We believe browser-focused developers and node developers will find it workable.

Misc

vm.Module and vm.runInModule should both be exposed eventually.

Relevant links

ECMAScript® 2018 Language Specification: Modules

Chris's walk of the current node experimental implementation

standard-things/esm

@appsforartists

This comment has been minimized.

Show comment
Hide comment
@appsforartists

appsforartists Jan 13, 2018

I believe you mean

import * as fs from 'fs';

You've got the as and from clauses backwards.

citation

appsforartists commented Jan 13, 2018

I believe you mean

import * as fs from 'fs';

You've got the as and from clauses backwards.

citation

@rauschma

This comment has been minimized.

Show comment
Hide comment
@rauschma

rauschma Jan 13, 2018

It’d be great to have:

  • import.meta.isMain
  • import.meta.url

It’d be great to have:

  • import.meta.isMain
  • import.meta.url
@ceejbot

This comment has been minimized.

Show comment
Hide comment
@ceejbot

ceejbot Jan 13, 2018

Thanks for the comments! I just pushed up a cleanup pass. This escaped a bit before I was ready for it :D

We ended up avoiding import.meta because we didn't need it, and we liked the clear context-setting of requiring things from "nodejs"-- this makes it clear that you're doing something that isn't going to work in the browser context that will need shimming. But I think it could work just fine either way.

Owner

ceejbot commented Jan 13, 2018

Thanks for the comments! I just pushed up a cleanup pass. This escaped a bit before I was ready for it :D

We ended up avoiding import.meta because we didn't need it, and we liked the clear context-setting of requiring things from "nodejs"-- this makes it clear that you're doing something that isn't going to work in the browser context that will need shimming. But I think it could work just fine either way.

@jquense

This comment has been minimized.

Show comment
Hide comment
@jquense

jquense Jan 13, 2018

I love this so much.

jquense commented Jan 13, 2018

I love this so much.

@kriskowal

This comment has been minimized.

Show comment
Hide comment
@kriskowal

kriskowal Jan 13, 2018

Is it viable to avoid capturing the configuration for the successive load of the module graph using global state (loaderHooks)?

Although it is reasonable to assume that applications will expand the frontier of their module graph in serial stages, it is certainly possible for concurrent expansions to race to upgrade the loader hooks, resulting in nondeterminism. Would it be reasonable for each expansion of the module graph to create a child graph? Would it be reasonable to reify an object that represents a module graph (a System), where disjoint graphs might be configured differently, not necessarily share all module instances, and coexist in the same application?

Is it viable to avoid capturing the configuration for the successive load of the module graph using global state (loaderHooks)?

Although it is reasonable to assume that applications will expand the frontier of their module graph in serial stages, it is certainly possible for concurrent expansions to race to upgrade the loader hooks, resulting in nondeterminism. Would it be reasonable for each expansion of the module graph to create a child graph? Would it be reasonable to reify an object that represents a module graph (a System), where disjoint graphs might be configured differently, not necessarily share all module instances, and coexist in the same application?

@kriskowal

This comment has been minimized.

Show comment
Hide comment
@kriskowal

kriskowal Jan 13, 2018

Also, I like this proposal. I believe Allen Wirfs-Brock shared the germ of this idea. I find it compelling.

Also, I like this proposal. I believe Allen Wirfs-Brock shared the germ of this idea. I find it compelling.

@appsforartists

This comment has been minimized.

Show comment
Hide comment
@appsforartists

appsforartists Jan 13, 2018

@ceejbot - looks like your from/as clauses are still backwards.

@ceejbot - looks like your from/as clauses are still backwards.

@rauschma

This comment has been minimized.

Show comment
Hide comment
@rauschma

rauschma Jan 13, 2018

@ceejbot That’s a good rationale for isMain! Good luck – cat’s out of the bag.

@ceejbot That’s a good rationale for isMain! Good luck – cat’s out of the bag.

@Kovensky

This comment has been minimized.

Show comment
Hide comment
@Kovensky

Kovensky Jan 13, 2018

w.r.t. your proposal for importing the predefined globals: import * from anything is never going to be acceptable because it becomes a module-scope with statement.

I am also not sure how acceptable it is, in the spec, to have importing the same module having different results for different importers. Module namespace objects are supposed to be cached per realm once imported.

This does look promising, however, but the frustrating part is the lack of a migration path or for progressive enhancement. This would essentially force all libraries to only publish themselves as commonjs in order to still work on all the node versions that don't implement this, and so none of the benefits of this proposal would ever apply to anything other than first party code.

You also need to keep in mind that any require would only ever execute after all imports are done.

Kovensky commented Jan 13, 2018

w.r.t. your proposal for importing the predefined globals: import * from anything is never going to be acceptable because it becomes a module-scope with statement.

I am also not sure how acceptable it is, in the spec, to have importing the same module having different results for different importers. Module namespace objects are supposed to be cached per realm once imported.

This does look promising, however, but the frustrating part is the lack of a migration path or for progressive enhancement. This would essentially force all libraries to only publish themselves as commonjs in order to still work on all the node versions that don't implement this, and so none of the benefits of this proposal would ever apply to anything other than first party code.

You also need to keep in mind that any require would only ever execute after all imports are done.

@wavebeem

This comment has been minimized.

Show comment
Hide comment
@wavebeem

wavebeem Jan 13, 2018

This looks cool, but there's one thing I was curious about.

Don't ESModules have strict mode semantics enabled by default, whereas normal Node.js CommonJS modules do not?

If so, doesn't this cause an issue with detection where a program like:

delete x;

Would be detected as an ESModule, run in strict mode, then fail (where it would silently succeed in non-strict mode). This is admittedly a contrived file, but it seems like it would be a slight break of backwards compatibility?

wavebeem commented Jan 13, 2018

This looks cool, but there's one thing I was curious about.

Don't ESModules have strict mode semantics enabled by default, whereas normal Node.js CommonJS modules do not?

If so, doesn't this cause an issue with detection where a program like:

delete x;

Would be detected as an ESModule, run in strict mode, then fail (where it would silently succeed in non-strict mode). This is admittedly a contrived file, but it seems like it would be a slight break of backwards compatibility?

@Jamesernator

This comment has been minimized.

Show comment
Hide comment
@Jamesernator

Jamesernator Jan 13, 2018

The purpose of this proposal is to reduce the friction involved in cross-platform JS applications

I don't entirely understand this when most of the proposal defines things that only work in nodejs for example import { ... } from "nodejs", loader hooks, etc and removes capabilities that might eventually be supported in browsers (e.g. import config from "./config.json" might become possible in future by serving the right MIME type).

I've been using .mjs with @std/esm for some internal tools that have some code that's shared between both the browser and node and most interoperability issues come from these two issues:

  • There's no way to fork logic depending on host to load resources without forcing everything to become asynchronous
  • There's no way to generate an esm module from commonjs so existing libraries can't be used in a compatible way

The former point is entirely solvable if top-level await in modules is ever supported. For example:

// isWord.mjs

async function loadWords() {
    // Ideally there'll be something similar to import.meta.host === 'node' / import.meta.host === 'browser'
    // in future instead of just detecting global objects
    if (typeof window === 'object') {
        const wordsUrl = new URL(wordsUrl, import.meta.url)
        const response = await fetch(wordsUrl)
        const text = await response.text()
        return text.trim().split('\n')
    } else {
        // In Node we need to import the URL object
        const URL = (await import('url')).URL
        const wordsUrl = new URL(import.meta.url, wordsUrl)
        const fs = await import('fs')
        const text = fs.readFileSync(wordsUrl, 'utf8')
        return text.trim().split('\n')
    }
}

const words = await loadWords()

export default function isWord(string) {
    return words.includes(string)
}

The second point is considerably more difficult, but it could just be that node/npm provides a way to convert a commonjs module into an esm one (obviously this is extremely non-trivial) that can be imported e.g.

npm install --convert underscore.string
import string from "node_modules/underscore.string.mjs"

Jamesernator commented Jan 13, 2018

The purpose of this proposal is to reduce the friction involved in cross-platform JS applications

I don't entirely understand this when most of the proposal defines things that only work in nodejs for example import { ... } from "nodejs", loader hooks, etc and removes capabilities that might eventually be supported in browsers (e.g. import config from "./config.json" might become possible in future by serving the right MIME type).

I've been using .mjs with @std/esm for some internal tools that have some code that's shared between both the browser and node and most interoperability issues come from these two issues:

  • There's no way to fork logic depending on host to load resources without forcing everything to become asynchronous
  • There's no way to generate an esm module from commonjs so existing libraries can't be used in a compatible way

The former point is entirely solvable if top-level await in modules is ever supported. For example:

// isWord.mjs

async function loadWords() {
    // Ideally there'll be something similar to import.meta.host === 'node' / import.meta.host === 'browser'
    // in future instead of just detecting global objects
    if (typeof window === 'object') {
        const wordsUrl = new URL(wordsUrl, import.meta.url)
        const response = await fetch(wordsUrl)
        const text = await response.text()
        return text.trim().split('\n')
    } else {
        // In Node we need to import the URL object
        const URL = (await import('url')).URL
        const wordsUrl = new URL(import.meta.url, wordsUrl)
        const fs = await import('fs')
        const text = fs.readFileSync(wordsUrl, 'utf8')
        return text.trim().split('\n')
    }
}

const words = await loadWords()

export default function isWord(string) {
    return words.includes(string)
}

The second point is considerably more difficult, but it could just be that node/npm provides a way to convert a commonjs module into an esm one (obviously this is extremely non-trivial) that can be imported e.g.

npm install --convert underscore.string
import string from "node_modules/underscore.string.mjs"
@TimothyGu

This comment has been minimized.

Show comment
Hide comment
@TimothyGu

TimothyGu Jan 13, 2018

The topic of module loading has been discussed ad nauseam by members of the Node.js community, in conjunction with TC39. Certain "musts" that have been established in those conversations, such as importing CJS through ESM's import system, are discarded by this proposal. It is quite tiring to see further rehashing of the same conversation. (Ideas like CLI flags, and the parse goal detection algorithm proposed by this Gist have all been raised and subsequently rejected during the conversation in Node.js.)

Politics aside, I have the following technical comments.


import {require} from 'nodejs';
import * as fs from 'fs'; // order of as/from changed by me to make it valid syntax
fs === require('fs'); // true

Assuming the require() function in ESM always returns the same object as it will in CJS, I don't think this can ever work for modules like events, whose main CJS export is a function. A module namespace object is always uncallable and has null as prototype.

On the other hand, one can make require('fs') semantics in ESM differ from those in CJS, to return module namespace objects rather than the normal CJS module.exports object. This works, but probably contradicts a lot of users' assumption that it works the same way as CJS.


To get the other node pre-defined globals and module locals, you must import them. import * from 'nodejs' provides the following

  • process
  • global
  • Buffer
  • require
  • __dirname
  • __filename
  • isMain new!

Things like process, global, Buffer are okay. However, require, __dirname, __filename, and isMain are different for every module. There are only a few ways to do that, none of which is super palatable in my opinion:

  1. Make 'nodejs' a special import whose exported bindings are different for every module. This departs from the usual ESM semantics that "when a module with a URL is loaded, it is loaded forever", implemented in browsers and Node.js (with very good reasons!).
  2. Make 'nodejs' module frozen, but require() etc. check their caller to determine the origin of the require(). Not only is this a departure from traditional CJS, it will only work for functions (not values like __dirname, __filename, and isMain), and it is very unlikely to be implementable in a satisfactory way. (E.g., eval(), new Function(), vm.Script)

The idea of a special import was discussed by TC39 (see presentation) but rejected for good reasons. import.meta was the solution that prevailed, and the one browsers (and JS engines) have already implemented.

TimothyGu commented Jan 13, 2018

The topic of module loading has been discussed ad nauseam by members of the Node.js community, in conjunction with TC39. Certain "musts" that have been established in those conversations, such as importing CJS through ESM's import system, are discarded by this proposal. It is quite tiring to see further rehashing of the same conversation. (Ideas like CLI flags, and the parse goal detection algorithm proposed by this Gist have all been raised and subsequently rejected during the conversation in Node.js.)

Politics aside, I have the following technical comments.


import {require} from 'nodejs';
import * as fs from 'fs'; // order of as/from changed by me to make it valid syntax
fs === require('fs'); // true

Assuming the require() function in ESM always returns the same object as it will in CJS, I don't think this can ever work for modules like events, whose main CJS export is a function. A module namespace object is always uncallable and has null as prototype.

On the other hand, one can make require('fs') semantics in ESM differ from those in CJS, to return module namespace objects rather than the normal CJS module.exports object. This works, but probably contradicts a lot of users' assumption that it works the same way as CJS.


To get the other node pre-defined globals and module locals, you must import them. import * from 'nodejs' provides the following

  • process
  • global
  • Buffer
  • require
  • __dirname
  • __filename
  • isMain new!

Things like process, global, Buffer are okay. However, require, __dirname, __filename, and isMain are different for every module. There are only a few ways to do that, none of which is super palatable in my opinion:

  1. Make 'nodejs' a special import whose exported bindings are different for every module. This departs from the usual ESM semantics that "when a module with a URL is loaded, it is loaded forever", implemented in browsers and Node.js (with very good reasons!).
  2. Make 'nodejs' module frozen, but require() etc. check their caller to determine the origin of the require(). Not only is this a departure from traditional CJS, it will only work for functions (not values like __dirname, __filename, and isMain), and it is very unlikely to be implementable in a satisfactory way. (E.g., eval(), new Function(), vm.Script)

The idea of a special import was discussed by TC39 (see presentation) but rejected for good reasons. import.meta was the solution that prevailed, and the one browsers (and JS engines) have already implemented.

@azz

This comment has been minimized.

Show comment
Hide comment
@azz

azz Jan 13, 2018

The following works for all exports from node core modules:

import {readFile} = require('fs');

Do you mean?

import {readFile} from 'fs';

@TimothyGu r.e. import.meta, is the idea that ESM to CJS would look like this?

const cjs = import.meta.require('cjs');

azz commented Jan 13, 2018

The following works for all exports from node core modules:

import {readFile} = require('fs');

Do you mean?

import {readFile} from 'fs';

@TimothyGu r.e. import.meta, is the idea that ESM to CJS would look like this?

const cjs = import.meta.require('cjs');
@TimothyGu

This comment has been minimized.

Show comment
Hide comment
@TimothyGu

TimothyGu Jan 13, 2018

@azz Yes, that is the current plan in Node.js.

@azz Yes, that is the current plan in Node.js.

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Jan 13, 2018

import { __dirname, filename, require, isMain } from "nodejs";

Please don't do these! Having a single module, "nodejs", whose contents are dependent on who imports it, is very confusing. It's technically possible, but is not something we'll ever support on the web, and muddles the messaging about modules.

Instead, using the import.meta object, which was explicitly designed for this sort of per-module metadata.

import './foo' attempts to load code from the exact same locations, with the same priority, as require('./foo').

This is disappointing. I think relative imports (not bare specifiers) should behave the same in Node as in browsers. In particular, automatically adding filename extensions is a very bad idea, and will decrease interopability.

import {readFile} = require('fs');

This isn't valid syntax; not sure what it's supposed to be saying.

domenic commented Jan 13, 2018

import { __dirname, filename, require, isMain } from "nodejs";

Please don't do these! Having a single module, "nodejs", whose contents are dependent on who imports it, is very confusing. It's technically possible, but is not something we'll ever support on the web, and muddles the messaging about modules.

Instead, using the import.meta object, which was explicitly designed for this sort of per-module metadata.

import './foo' attempts to load code from the exact same locations, with the same priority, as require('./foo').

This is disappointing. I think relative imports (not bare specifiers) should behave the same in Node as in browsers. In particular, automatically adding filename extensions is a very bad idea, and will decrease interopability.

import {readFile} = require('fs');

This isn't valid syntax; not sure what it's supposed to be saying.

@Kovensky

This comment has been minimized.

Show comment
Hide comment
@Kovensky

Kovensky Jan 13, 2018

It actually happens to be valid typescript syntax, but I think that was not the intended meaning 😆

In typescript, it's the same as const {readFile} = require('fs') but makes typescript parse the require call as being a commonjs require instead of an arbitrary function named require.

It actually happens to be valid typescript syntax, but I think that was not the intended meaning 😆

In typescript, it's the same as const {readFile} = require('fs') but makes typescript parse the require call as being a commonjs require instead of an arbitrary function named require.

@tolmasky

This comment has been minimized.

Show comment
Hide comment
@tolmasky

tolmasky Jan 13, 2018

Apologies if I've misunderstood the proposal, just trying to make sure I understand. If my understanding is correct, then any "import" in any file basically "poisons" each requiring file up, potentially infecting the entire application, more or less leading to the necessity of having all-require or all-import applications. For example:

  1. You have an application that currently uses CJS require. I imagine this is the common case today as import is not officially supported in node yet.
  2. You npm install some-package, where some-package uses imports (ESM module).
  3. You attempt to require("some-package") in a file ("myfile.js") and it fails as described above since it uses import and is thus not a candidate for requiring.
  4. No problem, you just call import("some-package") instead. So far so good.
  5. ... except, import() is asynchronous and returns a promise. So unlike the other requires in "myfile.js", uses of "some-package" must exist in a corresponding then callback (or after an await of course). This probably means that you'll have to put the entire non-importing contents of the file in said then callback since everything that depends on it must come after the load.
  6. ... but now every other file that had already required "myfile.js" must now also be modified to compensate for the fact that "myfile.js" now also loads asynchronously. In other words, the promise-ness of import("some-package") percolates up the requirer chain. This is not necessarily surprising, as this is the effect async/await also often has (or turning any deeply nested function into an asynchronous one).
  7. The alternative of course is to convert your entire app to use imports, or at the very least to be rooted at an ESM main file -- this way, the import chain down to the leaf nodes can remain synchronous. This is really quite similar (percolating imports up instead of .thens). You now must however meticulously manage which files must be imported and which must be required. As modules slowly update (some may never of course), every file remains broken into the "import blocks" and "require blocks". To some extent, instead of it being the responsibility of the author to label their module as ESM through the use of .mjs or .js, it is now the job of the user to label it as such in code through the way it is imported.

Is this interpretation accurate?

tolmasky commented Jan 13, 2018

Apologies if I've misunderstood the proposal, just trying to make sure I understand. If my understanding is correct, then any "import" in any file basically "poisons" each requiring file up, potentially infecting the entire application, more or less leading to the necessity of having all-require or all-import applications. For example:

  1. You have an application that currently uses CJS require. I imagine this is the common case today as import is not officially supported in node yet.
  2. You npm install some-package, where some-package uses imports (ESM module).
  3. You attempt to require("some-package") in a file ("myfile.js") and it fails as described above since it uses import and is thus not a candidate for requiring.
  4. No problem, you just call import("some-package") instead. So far so good.
  5. ... except, import() is asynchronous and returns a promise. So unlike the other requires in "myfile.js", uses of "some-package" must exist in a corresponding then callback (or after an await of course). This probably means that you'll have to put the entire non-importing contents of the file in said then callback since everything that depends on it must come after the load.
  6. ... but now every other file that had already required "myfile.js" must now also be modified to compensate for the fact that "myfile.js" now also loads asynchronously. In other words, the promise-ness of import("some-package") percolates up the requirer chain. This is not necessarily surprising, as this is the effect async/await also often has (or turning any deeply nested function into an asynchronous one).
  7. The alternative of course is to convert your entire app to use imports, or at the very least to be rooted at an ESM main file -- this way, the import chain down to the leaf nodes can remain synchronous. This is really quite similar (percolating imports up instead of .thens). You now must however meticulously manage which files must be imported and which must be required. As modules slowly update (some may never of course), every file remains broken into the "import blocks" and "require blocks". To some extent, instead of it being the responsibility of the author to label their module as ESM through the use of .mjs or .js, it is now the job of the user to label it as such in code through the way it is imported.

Is this interpretation accurate?

@CYBAI

This comment has been minimized.

Show comment
Hide comment
@CYBAI

CYBAI Jan 13, 2018

Consumption of ES modules using CommonJS syntax might fail, and vice versa. Specifically:

  • import 'cjs' fails with an error, because require, exports, and module locals such as __filename are not defined. node intercepts ReferenceError: <module local is not defined and decorates it with with a hint.

I think there's a typo in the end of the first point of ESM/CJS Interoperability in Node.js.

- and decorates it with with a hint
+ and decorates it with a hint

CYBAI commented Jan 13, 2018

Consumption of ES modules using CommonJS syntax might fail, and vice versa. Specifically:

  • import 'cjs' fails with an error, because require, exports, and module locals such as __filename are not defined. node intercepts ReferenceError: <module local is not defined and decorates it with with a hint.

I think there's a typo in the end of the first point of ESM/CJS Interoperability in Node.js.

- and decorates it with with a hint
+ and decorates it with a hint
@jeremiahlee

This comment has been minimized.

Show comment
Hide comment
@jeremiahlee

jeremiahlee Jan 13, 2018

This makes me happy as a full stack Node.js user!

This makes me happy as a full stack Node.js user!

@ericblade

This comment has been minimized.

Show comment
Hide comment
@ericblade

ericblade Jan 13, 2018

I have a strong feeling that most all of this has been discussed, perhaps even beaten to death, over the last several months of implementation of --experimental-modules. While I do hope that there is some room for improvement in experimental-modules (i would very much like to see large quantities of platform-agnostic ESM stuff..), I also highly suspect that there's a lot of hidden gotchas in a plan such as this, that people are bringing up here.

The very first problem that I see, isn't it an error to attempt to load a module that can't be resolved? Wouldn't that lead to basically all code written for node failing in browser, since you'd need to import from nodejs ?

I could be totally wrong on that, I don't seem to have a browser handy that supports imports natively yet.

Something that does seem like it could be improved upon, though, is that an ESM module should be determined to be an ESM module by the code importing it. This is also true in browser land. It does seem to me, and keep in mind, there may also be tons of problems to this approach, that attempting to import a CJS module via import, could treat module.exports as the default export, unless module.exports is an object, and then use it's keys as the exports. That does seem like it might be a workable way to resolve the issue with the filename being required to be different, which is something that i also dislike. The current method in node for mixing up ESM and CJS code seems like a mess -- but as I understand it, any other options that people had come up with were fraught with peril.

As a total aside, bringing this up because now I know where a bunch of people are who are interested in this :-) I tried using import.meta in 9.3 and 9.4 today, and it just complained that import was a SyntaxError :( and the code was working as a ESM module, 100% for sure. Maybe I missed something in how that is supposed to work.

I have a strong feeling that most all of this has been discussed, perhaps even beaten to death, over the last several months of implementation of --experimental-modules. While I do hope that there is some room for improvement in experimental-modules (i would very much like to see large quantities of platform-agnostic ESM stuff..), I also highly suspect that there's a lot of hidden gotchas in a plan such as this, that people are bringing up here.

The very first problem that I see, isn't it an error to attempt to load a module that can't be resolved? Wouldn't that lead to basically all code written for node failing in browser, since you'd need to import from nodejs ?

I could be totally wrong on that, I don't seem to have a browser handy that supports imports natively yet.

Something that does seem like it could be improved upon, though, is that an ESM module should be determined to be an ESM module by the code importing it. This is also true in browser land. It does seem to me, and keep in mind, there may also be tons of problems to this approach, that attempting to import a CJS module via import, could treat module.exports as the default export, unless module.exports is an object, and then use it's keys as the exports. That does seem like it might be a workable way to resolve the issue with the filename being required to be different, which is something that i also dislike. The current method in node for mixing up ESM and CJS code seems like a mess -- but as I understand it, any other options that people had come up with were fraught with peril.

As a total aside, bringing this up because now I know where a bunch of people are who are interested in this :-) I tried using import.meta in 9.3 and 9.4 today, and it just complained that import was a SyntaxError :( and the code was working as a ESM module, 100% for sure. Maybe I missed something in how that is supposed to work.

@TimothyGu

This comment has been minimized.

Show comment
Hide comment
@TimothyGu

TimothyGu Jan 13, 2018

@ericblade import.meta still depends on some unmerged V8 changes. It should be supported soon!

@ericblade import.meta still depends on some unmerged V8 changes. It should be supported soon!

@Jamesernator

This comment has been minimized.

Show comment
Hide comment
@Jamesernator

Jamesernator Jan 13, 2018

... an ESM module should be determined to be an ESM module by the code importing it. This is also true in browser land.

@ericblade This isn't true in browser land, in order to use a module in the browser it has to be served with a text/javascript (or equivalent) MIME type, it's only the combination of import + MIME type that determines the file as a Javascript module source. In future it's possible that something like WebAssembly might be import-able in which case the MIME type would be different.

... an ESM module should be determined to be an ESM module by the code importing it. This is also true in browser land.

@ericblade This isn't true in browser land, in order to use a module in the browser it has to be served with a text/javascript (or equivalent) MIME type, it's only the combination of import + MIME type that determines the file as a Javascript module source. In future it's possible that something like WebAssembly might be import-able in which case the MIME type would be different.

@liuyanghejerry

This comment has been minimized.

Show comment
Hide comment
@liuyanghejerry

liuyanghejerry Jan 13, 2018

It doesn't mention but seems like loaderHooks only works in ESM mode, since it's imported from "nodejs"?

It doesn't mention but seems like loaderHooks only works in ESM mode, since it's imported from "nodejs"?

@allenwb

This comment has been minimized.

Show comment
Hide comment
@allenwb

allenwb Jan 13, 2018

Overall, I think this is very nice. Well done!

My major concern is WRT the equivalences you stated in the section Node Core Modules. I don't see how the value returned from require('fs') could possibly meet both the ES requirements for a Module Namespace exotic object and a node programmers expectation for what require returns. However, I think this is easily fixed. I suspect you are just mis-conflating the ESM default export and module namespace objects.

The equivalences that I would expect to see are:

import {require} from 'nodejs';
import * as fs_ns from 'fs';
import fs from 'fs';
fs === fs_ns; //false, a ESM module's default export is not the same as its module namespace object
fs === fs_ns.default; //true,  as per the ES spec
let fs_req = require('fs');
fs === fs_req; // true, the ESM default export for 'fs' should be the same value returned by require
fs_req === fs_ns; //false, require should return the default export, not the namespace object

import {readfile} from('fs');
fs_req.readfile === readfile; //true
fs_req.readfile === fs.readfile; //true
fs_req.readfile === fs_ns.readfile; //true
fs_req.readfile === fs_ns.default.readfile; //true
//assuming that nobody is mutating the fs_req/fs object between these references.
//fs_ns, as an object namespace exotic object, consists of an immutable set of live bindings 

allenwb commented Jan 13, 2018

Overall, I think this is very nice. Well done!

My major concern is WRT the equivalences you stated in the section Node Core Modules. I don't see how the value returned from require('fs') could possibly meet both the ES requirements for a Module Namespace exotic object and a node programmers expectation for what require returns. However, I think this is easily fixed. I suspect you are just mis-conflating the ESM default export and module namespace objects.

The equivalences that I would expect to see are:

import {require} from 'nodejs';
import * as fs_ns from 'fs';
import fs from 'fs';
fs === fs_ns; //false, a ESM module's default export is not the same as its module namespace object
fs === fs_ns.default; //true,  as per the ES spec
let fs_req = require('fs');
fs === fs_req; // true, the ESM default export for 'fs' should be the same value returned by require
fs_req === fs_ns; //false, require should return the default export, not the namespace object

import {readfile} from('fs');
fs_req.readfile === readfile; //true
fs_req.readfile === fs.readfile; //true
fs_req.readfile === fs_ns.readfile; //true
fs_req.readfile === fs_ns.default.readfile; //true
//assuming that nobody is mutating the fs_req/fs object between these references.
//fs_ns, as an object namespace exotic object, consists of an immutable set of live bindings 
@Vanuan

This comment has been minimized.

Show comment
Hide comment
@Vanuan

Vanuan Jan 13, 2018

It is quite tiring to see further rehashing of the same conversation

The problem is that in this conversation Nodejs developers never asked JavaScript authors. And now there's a huge backlash of these Michael Jackson scripts.

Vanuan commented Jan 13, 2018

It is quite tiring to see further rehashing of the same conversation

The problem is that in this conversation Nodejs developers never asked JavaScript authors. And now there's a huge backlash of these Michael Jackson scripts.

@jdalton

This comment has been minimized.

Show comment
Hide comment
@jdalton

jdalton Jan 13, 2018

@domenic

The import {readFile} = require('fs') is a typo.
Based on the working implementation it's import {readFile} from "fs"

@allenwb

The

import {require} from 'nodejs';
import * from 'fs' as fs;
fs === require('fs'); // true

is typo'd too. Based on the working implementation it's

import {require} from 'nodejs';
import * as fs_ns from 'fs';
import fs from "fs"
let fs_req = require('fs');

fs === fs_ns; // false
fs === fs_ns.default; // true
fs === fs_req; // true
fs_req === fs_ns; // false

as you would expect.

jdalton commented Jan 13, 2018

@domenic

The import {readFile} = require('fs') is a typo.
Based on the working implementation it's import {readFile} from "fs"

@allenwb

The

import {require} from 'nodejs';
import * from 'fs' as fs;
fs === require('fs'); // true

is typo'd too. Based on the working implementation it's

import {require} from 'nodejs';
import * as fs_ns from 'fs';
import fs from "fs"
let fs_req = require('fs');

fs === fs_ns; // false
fs === fs_ns.default; // true
fs === fs_req; // true
fs_req === fs_ns; // false

as you would expect.

@ceejbot

This comment has been minimized.

Show comment
Hide comment
@ceejbot

ceejbot Jan 13, 2018

Thanks, JDD. This proposal led the working implementation by a bit, and I did not update it after the rubber met the road of working code. A documentation pass is badly needed. I'll revise now to get rid of some attractive nuisance typos.

Owner

ceejbot commented Jan 13, 2018

Thanks, JDD. This proposal led the working implementation by a bit, and I did not update it after the rubber met the road of working code. A documentation pass is badly needed. I'll revise now to get rid of some attractive nuisance typos.

@ericblade

This comment has been minimized.

Show comment
Hide comment
@ericblade

ericblade Jan 13, 2018

@Jamesernator

This isn't true in browser land, in order to use a module in the browser it has to be served with a text/javascript (or equivalent) MIME type, it's only the combination of import + MIME type that determines the file as a Javascript module source. In future it's possible that something like WebAssembly might be import-able in which case the MIME type would be different.

I'm pretty sure that browsers can operate upon local files, even with modules, therefore it wouldn't make sense to have a requirement for a server providing a MIME type. I don't know exactly where the specs are located, but I'm really close to absolutely positive that the "module-ness" of a ESM module is determined by it fitting one of two things: loaded via script tag with type="module" (which should, imo, be a separate attribute from 'type' because we already use 'type' for 'text/javascript', and type=module is not the same information as type=text/javascript ..) or imported from ES6 import statement.

In Node, it seems that the module-ness of an item is determined by both being true: .mjs file, and it exports something. (attempting to import a module that doesn't export in 9.3.0 at least resulted in an error when I tried it yesterday)

All that said, I don't like the file extension requirements, particularly as people are now starting to use node as a general scripting language from the shell prompt, it's a bad idea as a whole, IMO. I think as a community, we can come up with a better way, but it might be good to do it when the existing implementation is a little more complete, but definitely don't wait until it's accepted outside of the --experimental-modules switch, that might be too late to get changes. Reason for that being, it's a lot easier to demonstrate criticisms of it on a functioning setup, than it is to rehash it all out in theory, after it's already gone through months of hashing out.

However, I don't have more than a few minutes experience with messing with this yet, so I have to limit my criticisms to this: The way we're going in Node right now, requiring a specific file extension, isn't good. There are worse things that can be done, but this isn't good. This proposal does attempt to fix that, but I suspect that there are a lot of hidden gotchas (as other people have described above) that this doesn't address. My idea posted above probably also has a lot of hidden gotchas. Lots of hidden gotchas were likely discussed over the last several months, and I think it might be a good idea for some code smiths to hammer out some ideas, and try them out, in the near future, now that we have module support, and we can play with it.

I also disagree with adding built-in support to the primary Javascript engine that is in use in the world to directly support transpilation. My first thought is that could probably be added via some sort of native plugin, but I'm not at all familiar with the Node code base to know that for sure. I think that is highly useful for transpiler authors, but I feel that it would lead to very bad habits overall.

@Jamesernator

This isn't true in browser land, in order to use a module in the browser it has to be served with a text/javascript (or equivalent) MIME type, it's only the combination of import + MIME type that determines the file as a Javascript module source. In future it's possible that something like WebAssembly might be import-able in which case the MIME type would be different.

I'm pretty sure that browsers can operate upon local files, even with modules, therefore it wouldn't make sense to have a requirement for a server providing a MIME type. I don't know exactly where the specs are located, but I'm really close to absolutely positive that the "module-ness" of a ESM module is determined by it fitting one of two things: loaded via script tag with type="module" (which should, imo, be a separate attribute from 'type' because we already use 'type' for 'text/javascript', and type=module is not the same information as type=text/javascript ..) or imported from ES6 import statement.

In Node, it seems that the module-ness of an item is determined by both being true: .mjs file, and it exports something. (attempting to import a module that doesn't export in 9.3.0 at least resulted in an error when I tried it yesterday)

All that said, I don't like the file extension requirements, particularly as people are now starting to use node as a general scripting language from the shell prompt, it's a bad idea as a whole, IMO. I think as a community, we can come up with a better way, but it might be good to do it when the existing implementation is a little more complete, but definitely don't wait until it's accepted outside of the --experimental-modules switch, that might be too late to get changes. Reason for that being, it's a lot easier to demonstrate criticisms of it on a functioning setup, than it is to rehash it all out in theory, after it's already gone through months of hashing out.

However, I don't have more than a few minutes experience with messing with this yet, so I have to limit my criticisms to this: The way we're going in Node right now, requiring a specific file extension, isn't good. There are worse things that can be done, but this isn't good. This proposal does attempt to fix that, but I suspect that there are a lot of hidden gotchas (as other people have described above) that this doesn't address. My idea posted above probably also has a lot of hidden gotchas. Lots of hidden gotchas were likely discussed over the last several months, and I think it might be a good idea for some code smiths to hammer out some ideas, and try them out, in the near future, now that we have module support, and we can play with it.

I also disagree with adding built-in support to the primary Javascript engine that is in use in the world to directly support transpilation. My first thought is that could probably be added via some sort of native plugin, but I'm not at all familiar with the Node code base to know that for sure. I think that is highly useful for transpiler authors, but I feel that it would lead to very bad habits overall.

@TimothyGu

This comment has been minimized.

Show comment
Hide comment
@TimothyGu

TimothyGu Jan 13, 2018

PSA: Node.js will soon support import { readFile } from 'fs';! See nodejs/node#18131 for details.

After this PR, what @allenwb proposed is going to fully work as expected in Node.js.


@ericblade

I'm pretty sure that browsers can operate upon local files, even with modules,

How exactly local files are fetched in browsers is currently unfortunately under-defined; however, if you look at the surrounding steps in that link for other protocols these steps will probably emulate a normal HTTP response, i.e. it will contain looking up the MIME type from a file extension-to-MIME type database. Which is what browsers currently do.

therefore it wouldn't make sense to have a requirement for a server providing a MIME type.

The HTML spec does indeed have this requirement. See https://html.spec.whatwg.org/multipage/webappapis.html#fetch-a-single-module-script, step 8:

  1. (Paraphrasing here) The fetch fails when one of the following is true:

    • response's type is "error"
    • response's status is not an ok status
    • The result of extracting a MIME type from response's header list (ignoring parameters) is not a JavaScript MIME type

PSA: Node.js will soon support import { readFile } from 'fs';! See nodejs/node#18131 for details.

After this PR, what @allenwb proposed is going to fully work as expected in Node.js.


@ericblade

I'm pretty sure that browsers can operate upon local files, even with modules,

How exactly local files are fetched in browsers is currently unfortunately under-defined; however, if you look at the surrounding steps in that link for other protocols these steps will probably emulate a normal HTTP response, i.e. it will contain looking up the MIME type from a file extension-to-MIME type database. Which is what browsers currently do.

therefore it wouldn't make sense to have a requirement for a server providing a MIME type.

The HTML spec does indeed have this requirement. See https://html.spec.whatwg.org/multipage/webappapis.html#fetch-a-single-module-script, step 8:

  1. (Paraphrasing here) The fetch fails when one of the following is true:

    • response's type is "error"
    • response's status is not an ok status
    • The result of extracting a MIME type from response's header list (ignoring parameters) is not a JavaScript MIME type
@jamesplease

This comment has been minimized.

Show comment
Hide comment
@jamesplease

jamesplease Jan 14, 2018

Overall, I like this a lot. A bunch of good points have been mentioned in the comments. One thought I had is that --module=cjs and --module=esm seem like more expressive flags than terms like “mode” and “legacy,” so I prefer them for that reason.

It also allows for this to be released in a non-breaking way by making cjs the default. Then, at a later date, the flags could be swapped.

jamesplease commented Jan 14, 2018

Overall, I like this a lot. A bunch of good points have been mentioned in the comments. One thought I had is that --module=cjs and --module=esm seem like more expressive flags than terms like “mode” and “legacy,” so I prefer them for that reason.

It also allows for this to be released in a non-breaking way by making cjs the default. Then, at a later date, the flags could be swapped.

@ceejbot

This comment has been minimized.

Show comment
Hide comment
@ceejbot

ceejbot Jan 14, 2018

The names of flags and their values is a bikeshed that I am happy to leave painted any way the node project prefers; I merely propose their existence.

For the commenters with ideas about how things might work differently, I suggest doing what we did, forking either our branch or node's upstream, and implementing your idea to see how it works. You'll learn what's feasible and what's not, and get some direct experience with using the apis and design you're considering. I believe strongly right now that working code expressing meaningful choices among the options is the best way to move forward. We found it easier to do than we'd feared. The foundational work in node & v8 is solid.

Owner

ceejbot commented Jan 14, 2018

The names of flags and their values is a bikeshed that I am happy to leave painted any way the node project prefers; I merely propose their existence.

For the commenters with ideas about how things might work differently, I suggest doing what we did, forking either our branch or node's upstream, and implementing your idea to see how it works. You'll learn what's feasible and what's not, and get some direct experience with using the apis and design you're considering. I believe strongly right now that working code expressing meaningful choices among the options is the best way to move forward. We found it easier to do than we'd feared. The foundational work in node & v8 is solid.

@MarkTiedemann

This comment has been minimized.

Show comment
Hide comment
@MarkTiedemann

MarkTiedemann Jan 14, 2018

I'd prefer from 'node' over from 'nodejs'. :)

I'd prefer from 'node' over from 'nodejs'. :)

@bathos

This comment has been minimized.

Show comment
Hide comment
@bathos

bathos Jan 14, 2018

Re:

It’d be great to have:

import.meta.isMain
import.meta.url

and its response,

We ended up avoiding import.meta because we didn't need it, and we liked the clear context-setting of requiring things from "nodejs"-- this makes it clear that you're doing something that isn't going to work in the browser context that will need shimming. But I think it could work just fine either way.

I would also prefer seeing import.meta for these. Somebody who knows more should correct me if I’m wrong, but I’m pretty sure that:

a. the properties of import.meta are environment-specific by definition
b. the import.meta object of a module is mutable to permit polyfilling for cases just like this, so it’s not necessarilly the case that import.meta.isMain would have to be unavailable outside of node.

In contrast, it seems importing bindings from the same module and getting different bindings back per importee is something that isn’t possible in the browser at all — I think it’s less, not more, shimmable, since unlike node, the browser is unlikely to ever afford "meta control" over module loading.


Edit: on reflection, I guess in both cases it’s about equally shimmable — both require a build step of some sort — but the simplicity of just prepending the line import.meta.someNodeThing = ... without changing any of the original code seems preferable.

bathos commented Jan 14, 2018

Re:

It’d be great to have:

import.meta.isMain
import.meta.url

and its response,

We ended up avoiding import.meta because we didn't need it, and we liked the clear context-setting of requiring things from "nodejs"-- this makes it clear that you're doing something that isn't going to work in the browser context that will need shimming. But I think it could work just fine either way.

I would also prefer seeing import.meta for these. Somebody who knows more should correct me if I’m wrong, but I’m pretty sure that:

a. the properties of import.meta are environment-specific by definition
b. the import.meta object of a module is mutable to permit polyfilling for cases just like this, so it’s not necessarilly the case that import.meta.isMain would have to be unavailable outside of node.

In contrast, it seems importing bindings from the same module and getting different bindings back per importee is something that isn’t possible in the browser at all — I think it’s less, not more, shimmable, since unlike node, the browser is unlikely to ever afford "meta control" over module loading.


Edit: on reflection, I guess in both cases it’s about equally shimmable — both require a build step of some sort — but the simplicity of just prepending the line import.meta.someNodeThing = ... without changing any of the original code seems preferable.

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Jan 14, 2018

My 2¢ on import.meta.require/__dirname/etc vs import {require, __dirname, etc.} from "nodejs"

These fields are, in commonjs land, "module-local free variables". They're not globals, they're module-specific, and they're set automatically. It seemed reasonable to me that import.meta.require was the way to go for that reason. @ceejbot and @isntitvacant changed my mind.

Modules loaded in commonjs are a singleton per resolved path. There is no guarantee in node that require('lodash') always resolves to the same thing. In fact, the whole point of the nested node_modules resolution algorithm is to allow package managers to serve different versions of named dependencies to different importing modules. This is no different than if every node module had a file at ./node_modules/nodejs that implemented these fields.

The fact that these vars are module-local gives import.meta.require a sort of intuitive sense. It is also closer to what browsers (plan to?) do for platform-specific bits.

In practice we can expect that people writing code for both Node.js and the browser will have to transpile, even if they are using native es modules in both platforms. The most common reason for this is paths. import {test} from 'tap' is not valid on the browser because 'tap' isn't a valid import url.

However, they will not have to transpile code from es modules into commonjs, or from commonjs into something else. The only work for such a transpiler is handling import urls. (Principle 2 of 3)

Anyone using es modules in both node and the browser will need to either polyfill require or prevent its use. As it happens, throwing or polyfilling based on import {require} from 'nodejs' is a lot easier to do statically than finding all instances of import.meta.require. (Eg: const s = 'equire'; const r = import[{toString: () => 'meta'}][unescape('%72')+s]; r('lodash'))

Thus, Principle 2 was easier to meet using import {require} from 'nodejs' than import.meta.require. A transpiler could either implement require() or statically and deterministically crash, rather than ship a broken program. If it turns out that import.meta.require actually is better for shipping to browsers, then the import-path-munging transpilers can turn it into that easily enough. If it isn't better, than we've lost nothing.

Also, a variable import-module-specific version of import {require} from 'nodejs' was already implemented as part of the --experimental-module-system feature.

Running code and practical considerations should overrule symmetrical esthetics.

It seems to me that import {require} from 'nodejs' is more in line with the principles of this proposal than import.meta.require. Objectors to that approach should perhaps take issue with the principles instead. No point debating tactics without a clearly articulated shared strategic goal.

isaacs commented Jan 14, 2018

My 2¢ on import.meta.require/__dirname/etc vs import {require, __dirname, etc.} from "nodejs"

These fields are, in commonjs land, "module-local free variables". They're not globals, they're module-specific, and they're set automatically. It seemed reasonable to me that import.meta.require was the way to go for that reason. @ceejbot and @isntitvacant changed my mind.

Modules loaded in commonjs are a singleton per resolved path. There is no guarantee in node that require('lodash') always resolves to the same thing. In fact, the whole point of the nested node_modules resolution algorithm is to allow package managers to serve different versions of named dependencies to different importing modules. This is no different than if every node module had a file at ./node_modules/nodejs that implemented these fields.

The fact that these vars are module-local gives import.meta.require a sort of intuitive sense. It is also closer to what browsers (plan to?) do for platform-specific bits.

In practice we can expect that people writing code for both Node.js and the browser will have to transpile, even if they are using native es modules in both platforms. The most common reason for this is paths. import {test} from 'tap' is not valid on the browser because 'tap' isn't a valid import url.

However, they will not have to transpile code from es modules into commonjs, or from commonjs into something else. The only work for such a transpiler is handling import urls. (Principle 2 of 3)

Anyone using es modules in both node and the browser will need to either polyfill require or prevent its use. As it happens, throwing or polyfilling based on import {require} from 'nodejs' is a lot easier to do statically than finding all instances of import.meta.require. (Eg: const s = 'equire'; const r = import[{toString: () => 'meta'}][unescape('%72')+s]; r('lodash'))

Thus, Principle 2 was easier to meet using import {require} from 'nodejs' than import.meta.require. A transpiler could either implement require() or statically and deterministically crash, rather than ship a broken program. If it turns out that import.meta.require actually is better for shipping to browsers, then the import-path-munging transpilers can turn it into that easily enough. If it isn't better, than we've lost nothing.

Also, a variable import-module-specific version of import {require} from 'nodejs' was already implemented as part of the --experimental-module-system feature.

Running code and practical considerations should overrule symmetrical esthetics.

It seems to me that import {require} from 'nodejs' is more in line with the principles of this proposal than import.meta.require. Objectors to that approach should perhaps take issue with the principles instead. No point debating tactics without a clearly articulated shared strategic goal.

@TimothyGu

This comment has been minimized.

Show comment
Hide comment
@TimothyGu

TimothyGu Jan 14, 2018

@isaacs Thank you for your explanation. The choice behind import { require } from 'nodejs' is a lot more understandable now, and sounds more easily implementable because of this bit:

This is no different than if every node module had a file at ./node_modules/nodejs that implemented these fields.

I am now neutral on this versus import.meta.require – though having it available to ES modules at all is a different debate (my opinion is no, because there would be no need if import supports CJS, but can be convinced otherwise).

Regarding import.meta, I do want it to continue to be used, especially for cases that browsers implement like import.meta.url, even though it's not "necessary" for Node.js scripts.

Regarding __dirname and __filename, I am ambivalent on having them available at all. Having them available to Node.js' ES modules will certainly simplify some file operations, but may encourage increased dependence on Node.js-specific features rather than primitives (like import.meta.url) that are sure to be implemented in both Node.js and browsers. I can easily imagine modules that rely on no other Node.js features than __filename use __filename instead of import.meta.url, thus hindering its usage in browsers (see Principle 2 of 3).


Objectors to that approach should perhaps take issue with the principles instead.

Though the "that approach" in this sentence refers specifically to the debate of import { require } from 'nodejs'; vs. import.meta.require, I think "taking issues with the principles" can help organize my objections to this proposal.

Principle 1

Existing CommonJS code must continue to work unchanged.

I fully agree with this principle. The current ESM implementation in Node.js fulfills this principle.

However, the proposal as it currently stands (rev. fea6cb08fa8920604c8c9193f510a7a47d856cc4) does not fulfill this principle, because of the intrinsic deficiencies the parse goal detection algorithm (relevant parts reproduced below) proposed has.

Authors do not need to specify parse goals via means like file extensions. Main files are assumed to have the module goal falling back to CommonJS if the runtime encounters require() or any module locals.

...

Additionally, node's existing shebang-stripping parsing logic will be extended to check for --mode=legacy in the shebang of the main module, which will trigger an immediate switch to commonjs mode.

  1. There are no reliable way of detecting require() without running the code. And even so, require() is not a reliable way to detect things. A few scenarios follow:

    1. "I intend this file to be ESM, but I accidentally put a require() in the file; now Node.js outputs an error message about my import statement rather than the require(), causing confusion."
    2. "I intend this file to be ESM, but I extended the global object with a require() method of my own."
  2. There are syntactic ambiguities between the Module and Script parse goals. Some of these ambiguities can cause issues. One of them is raised above by @wavebeem.

As https://github.com/bmeck/UnambiguousJavaScriptGrammar shows, a revised version of this algorithm is probably "fine" for most cases. But as a platform-level product, Node.js should not aim for "fine for most cases" but "rock-solid for all cases".

Principle 2

ES modules that don't have platform-specific dependencies must work unchanged in both the browser and node, with the exception that import paths might need to be changed.

I fully agree with this principle. The current ESM implementation in Node.js also fulfills this principle.

See my concerns above about __dirname and __filename however.

Principle 3

The parse goal of a module must be known by its consumer in advance. The module producer should write files that end in .js and advertise the module's parse goal in its documentation.

I don't necessarily disagree with this principle, though it feels like the means to another principle, than a principle in and of itself. Concretely, I as a consumer don't see why I need to know the parse goal of a module; rather this principle sounds like the means to another unstated principle "All JavaScript files must use .js only".

Whether the principle is means or ends, I believe the following competing principle is of more importance than this:

A large codebase must be allowed to gradually transition from CJS to ESM.

The current ESM implementation in Node.js does not fulfill the first principle but does the second, with this proposal exactly opposite (see @tolmasky's comment).

I am of course open to a proposal that allows for both, but so far a solution has not surfaced without breaking one of the other principles.

TimothyGu commented Jan 14, 2018

@isaacs Thank you for your explanation. The choice behind import { require } from 'nodejs' is a lot more understandable now, and sounds more easily implementable because of this bit:

This is no different than if every node module had a file at ./node_modules/nodejs that implemented these fields.

I am now neutral on this versus import.meta.require – though having it available to ES modules at all is a different debate (my opinion is no, because there would be no need if import supports CJS, but can be convinced otherwise).

Regarding import.meta, I do want it to continue to be used, especially for cases that browsers implement like import.meta.url, even though it's not "necessary" for Node.js scripts.

Regarding __dirname and __filename, I am ambivalent on having them available at all. Having them available to Node.js' ES modules will certainly simplify some file operations, but may encourage increased dependence on Node.js-specific features rather than primitives (like import.meta.url) that are sure to be implemented in both Node.js and browsers. I can easily imagine modules that rely on no other Node.js features than __filename use __filename instead of import.meta.url, thus hindering its usage in browsers (see Principle 2 of 3).


Objectors to that approach should perhaps take issue with the principles instead.

Though the "that approach" in this sentence refers specifically to the debate of import { require } from 'nodejs'; vs. import.meta.require, I think "taking issues with the principles" can help organize my objections to this proposal.

Principle 1

Existing CommonJS code must continue to work unchanged.

I fully agree with this principle. The current ESM implementation in Node.js fulfills this principle.

However, the proposal as it currently stands (rev. fea6cb08fa8920604c8c9193f510a7a47d856cc4) does not fulfill this principle, because of the intrinsic deficiencies the parse goal detection algorithm (relevant parts reproduced below) proposed has.

Authors do not need to specify parse goals via means like file extensions. Main files are assumed to have the module goal falling back to CommonJS if the runtime encounters require() or any module locals.

...

Additionally, node's existing shebang-stripping parsing logic will be extended to check for --mode=legacy in the shebang of the main module, which will trigger an immediate switch to commonjs mode.

  1. There are no reliable way of detecting require() without running the code. And even so, require() is not a reliable way to detect things. A few scenarios follow:

    1. "I intend this file to be ESM, but I accidentally put a require() in the file; now Node.js outputs an error message about my import statement rather than the require(), causing confusion."
    2. "I intend this file to be ESM, but I extended the global object with a require() method of my own."
  2. There are syntactic ambiguities between the Module and Script parse goals. Some of these ambiguities can cause issues. One of them is raised above by @wavebeem.

As https://github.com/bmeck/UnambiguousJavaScriptGrammar shows, a revised version of this algorithm is probably "fine" for most cases. But as a platform-level product, Node.js should not aim for "fine for most cases" but "rock-solid for all cases".

Principle 2

ES modules that don't have platform-specific dependencies must work unchanged in both the browser and node, with the exception that import paths might need to be changed.

I fully agree with this principle. The current ESM implementation in Node.js also fulfills this principle.

See my concerns above about __dirname and __filename however.

Principle 3

The parse goal of a module must be known by its consumer in advance. The module producer should write files that end in .js and advertise the module's parse goal in its documentation.

I don't necessarily disagree with this principle, though it feels like the means to another principle, than a principle in and of itself. Concretely, I as a consumer don't see why I need to know the parse goal of a module; rather this principle sounds like the means to another unstated principle "All JavaScript files must use .js only".

Whether the principle is means or ends, I believe the following competing principle is of more importance than this:

A large codebase must be allowed to gradually transition from CJS to ESM.

The current ESM implementation in Node.js does not fulfill the first principle but does the second, with this proposal exactly opposite (see @tolmasky's comment).

I am of course open to a proposal that allows for both, but so far a solution has not surfaced without breaking one of the other principles.

@JimPanic

This comment has been minimized.

Show comment
Hide comment
@JimPanic

JimPanic Jan 14, 2018

Yes, please! This proposal seems reasonable! (More reasonable than a separate extension and all that Jazz.)

Yes, please! This proposal seems reasonable! (More reasonable than a separate extension and all that Jazz.)

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Jan 14, 2018

@TimothyGu

(I'm not sure what "rev fea6cb08fa8920604c8c9193f510a7a47d856cc4" refers to, can you provide a link?)

There are no reliable way of detecting require() without running the code. And even so, require() is not a reliable way to detect things. A few scenarios follow:

While I think your concern here is the right place to poke at, I'm struggling to find a case where it'll behave badly.

"I intend this file to be ESM, but I accidentally put a require() in the file; now Node.js outputs an error message about my import statement rather than the require(), causing confusion."

That's incorrect. If there's an import or export statement, then it forces ESM mode. Your assumption is reasonable from the gist contents itself, but the code is not ambiguous, and it changes some of your conclusions meaningfully, I believe.

$ cat x.js
require('fs')
import {readFile} from 'fs' // <-- triggers ESM mode

$ ./node x.js
ReferenceError: require is not defined
    at file:///Users/isaacs/dev/js/node-master/x.js:1:1
    at ModuleJob.run (internal/loader/ModuleJob.js:104:16)
    at <anonymous>

(Whether it still "causes confusion" or not is debatable.)

If you are intending ESM and incorrectly call require, then you'll either have import statements or not. If you do, you'll get a run-time error (as one would expect from any other undefined function).

So the case remaining is where you intend ESM mode, have no import or export statements, and call require(). But... I think the right answer here is to run the code in CJS mode, because the module is otherwise trivial, and doing commonjs type things.

"I intend this file to be ESM, but I extended the global object with a require() method of my own."

The only way to do this and still have the parse mode be ambiguous is if you do node -r ./define-global-require.js ./x.js. (That is, if x.js is intended to be ESM, but not the main module, then the module loading it will either be using import or require() and thus the parse mode is unambiguous.)

If that module has any import/export statements, then it's unambiguously ESM. So, the remaining case is a module that should be ESM, where a global require is defined in a -r argument module, and the main module has no import or export statements. I think this is a suitably narrow edge case to be handled with documentation, and implies such a trivial module as to be mostly irrelevant. And, the --mode=esm flag is a suitable workaround.

(I'm using require here as a stand-in for the whole set of module locals.)

A large codebase must be allowed to gradually transition from CJS to ESM.

This can be done gradually by using import() in CJS code and import {require} from 'nodejs' in ESM code. The challenge is:

  1. import() returns a promise. So, previously synchronously loaded deps become async.
  2. When the main module changes, it changes the way that consumers have to load the module.

Either of these may require a major version bump. But I don't believe that's a profound blocker. I also suspect that most codebases will switch over rather quickly, as we've seen in some major projects adopting Promises, ES classes, and other language features.

It's counterintuitive, but in a community that is still growing so quickly, the future is much larger than the past. We'll outgrow and abandon most legacy code rather than port it. Major version bumps (and, when that fails, completely new projects/forks to replace old ones) are a very normal process for adopting new features.

isaacs commented Jan 14, 2018

@TimothyGu

(I'm not sure what "rev fea6cb08fa8920604c8c9193f510a7a47d856cc4" refers to, can you provide a link?)

There are no reliable way of detecting require() without running the code. And even so, require() is not a reliable way to detect things. A few scenarios follow:

While I think your concern here is the right place to poke at, I'm struggling to find a case where it'll behave badly.

"I intend this file to be ESM, but I accidentally put a require() in the file; now Node.js outputs an error message about my import statement rather than the require(), causing confusion."

That's incorrect. If there's an import or export statement, then it forces ESM mode. Your assumption is reasonable from the gist contents itself, but the code is not ambiguous, and it changes some of your conclusions meaningfully, I believe.

$ cat x.js
require('fs')
import {readFile} from 'fs' // <-- triggers ESM mode

$ ./node x.js
ReferenceError: require is not defined
    at file:///Users/isaacs/dev/js/node-master/x.js:1:1
    at ModuleJob.run (internal/loader/ModuleJob.js:104:16)
    at <anonymous>

(Whether it still "causes confusion" or not is debatable.)

If you are intending ESM and incorrectly call require, then you'll either have import statements or not. If you do, you'll get a run-time error (as one would expect from any other undefined function).

So the case remaining is where you intend ESM mode, have no import or export statements, and call require(). But... I think the right answer here is to run the code in CJS mode, because the module is otherwise trivial, and doing commonjs type things.

"I intend this file to be ESM, but I extended the global object with a require() method of my own."

The only way to do this and still have the parse mode be ambiguous is if you do node -r ./define-global-require.js ./x.js. (That is, if x.js is intended to be ESM, but not the main module, then the module loading it will either be using import or require() and thus the parse mode is unambiguous.)

If that module has any import/export statements, then it's unambiguously ESM. So, the remaining case is a module that should be ESM, where a global require is defined in a -r argument module, and the main module has no import or export statements. I think this is a suitably narrow edge case to be handled with documentation, and implies such a trivial module as to be mostly irrelevant. And, the --mode=esm flag is a suitable workaround.

(I'm using require here as a stand-in for the whole set of module locals.)

A large codebase must be allowed to gradually transition from CJS to ESM.

This can be done gradually by using import() in CJS code and import {require} from 'nodejs' in ESM code. The challenge is:

  1. import() returns a promise. So, previously synchronously loaded deps become async.
  2. When the main module changes, it changes the way that consumers have to load the module.

Either of these may require a major version bump. But I don't believe that's a profound blocker. I also suspect that most codebases will switch over rather quickly, as we've seen in some major projects adopting Promises, ES classes, and other language features.

It's counterintuitive, but in a community that is still growing so quickly, the future is much larger than the past. We'll outgrow and abandon most legacy code rather than port it. Major version bumps (and, when that fails, completely new projects/forks to replace old ones) are a very normal process for adopting new features.

@ceejbot

This comment has been minimized.

Show comment
Hide comment
@ceejbot

ceejbot Jan 14, 2018

@isaacs says:

It's counterintuitive, but in a community that is still growing so quickly, the future is much larger than the past. We'll outgrow and abandon most legacy code rather than port it.

This was one of the major drivers of the tradeoffs in this proposal. They are tradeoffs, and there are other workable solutions that make different tradeoffs. Be clear about what your goals are first, then make tradeoffs in service of those goals. Then you can, as we did, figure out if they're feasible in a working implementation.

We chose to treat the current situation as transitional, and make tradeoffs that will be more pleasant once the community is 90% ESM instead of 90% CJS (and falling). (I don't know what the current split of code on the registry is. That would be interesting to know.)

Owner

ceejbot commented Jan 14, 2018

@isaacs says:

It's counterintuitive, but in a community that is still growing so quickly, the future is much larger than the past. We'll outgrow and abandon most legacy code rather than port it.

This was one of the major drivers of the tradeoffs in this proposal. They are tradeoffs, and there are other workable solutions that make different tradeoffs. Be clear about what your goals are first, then make tradeoffs in service of those goals. Then you can, as we did, figure out if they're feasible in a working implementation.

We chose to treat the current situation as transitional, and make tradeoffs that will be more pleasant once the community is 90% ESM instead of 90% CJS (and falling). (I don't know what the current split of code on the registry is. That would be interesting to know.)

@TimothyGu

This comment has been minimized.

Show comment
Hide comment
@TimothyGu

TimothyGu Jan 14, 2018

@isaacs

(I'm not sure what "rev fea6cb08fa8920604c8c9193f510a7a47d856cc4" refers to, can you provide a link?)

I was referring to the revision of the Gist: https://gist.github.com/ceejbot/b49f8789b2ab6b09548ccb72813a1054/fea6cb08fa8920604c8c9193f510a7a47d856cc4

(It's also the commit hash of the Gist if you clone it as a Git repo.)

If there's an import or export statement, then it forces ESM mode.

This is the missing piece that I couldn't find in the initial proposal. Thank you for clarifying, it makes more sense now. (I suspect this could probably be added to the proposal itself.)

I have more specific thoughts about how the npm fork currently implements this scanning, specifically that by using a JavaScript parser such as Acorn it disallows newer features of JavaScript that V8 implements but Acorn doesn't (such like object spread) in CJS. But that may well be intentional.

There remains the question of how the following is treated:

eval('var a = 0');
console.log(a);

this is parsable both CJS and ESM, and by the tiebreakers in this proposal it would be ESM. Yet it can only run under CJS mode because of strict-mode semantics of eval(). (The same issue also applies to eval('delete x'); etc.) (Maybe @chrisdickinson or @ceejbot may be answer this better?)


@isaacs

It's counterintuitive, but in a community that is still growing so quickly, the future is much larger than the past. We'll outgrow and abandon most legacy code rather than port it.

@ceejbot

We chose to treat the current situation as transitional, and make tradeoffs that will be more pleasant once the community is 90% ESM instead of 90% CJS (and falling).

This is indeed the part of the proposal I had (and still have) the most trouble wrapping my head around. The scenario described by @tolmasky is too undesirable, in my opinion. Though I have to acknowledge you (the authors of this proposal) are much more experienced than me in the realm of JavaScript.


I do very much appreciate the time all of you took to answer my questions. I would also like a chance to chat if that is at all possible.

@isaacs

(I'm not sure what "rev fea6cb08fa8920604c8c9193f510a7a47d856cc4" refers to, can you provide a link?)

I was referring to the revision of the Gist: https://gist.github.com/ceejbot/b49f8789b2ab6b09548ccb72813a1054/fea6cb08fa8920604c8c9193f510a7a47d856cc4

(It's also the commit hash of the Gist if you clone it as a Git repo.)

If there's an import or export statement, then it forces ESM mode.

This is the missing piece that I couldn't find in the initial proposal. Thank you for clarifying, it makes more sense now. (I suspect this could probably be added to the proposal itself.)

I have more specific thoughts about how the npm fork currently implements this scanning, specifically that by using a JavaScript parser such as Acorn it disallows newer features of JavaScript that V8 implements but Acorn doesn't (such like object spread) in CJS. But that may well be intentional.

There remains the question of how the following is treated:

eval('var a = 0');
console.log(a);

this is parsable both CJS and ESM, and by the tiebreakers in this proposal it would be ESM. Yet it can only run under CJS mode because of strict-mode semantics of eval(). (The same issue also applies to eval('delete x'); etc.) (Maybe @chrisdickinson or @ceejbot may be answer this better?)


@isaacs

It's counterintuitive, but in a community that is still growing so quickly, the future is much larger than the past. We'll outgrow and abandon most legacy code rather than port it.

@ceejbot

We chose to treat the current situation as transitional, and make tradeoffs that will be more pleasant once the community is 90% ESM instead of 90% CJS (and falling).

This is indeed the part of the proposal I had (and still have) the most trouble wrapping my head around. The scenario described by @tolmasky is too undesirable, in my opinion. Though I have to acknowledge you (the authors of this proposal) are much more experienced than me in the realm of JavaScript.


I do very much appreciate the time all of you took to answer my questions. I would also like a chance to chat if that is at all possible.

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Jan 14, 2018

It's true that it's technically possible to make import { require } from "nodejs" work. But it's frustrating to see that, this object which we specifically created in response to the Node.js community's requests for a place for per-module metadata and functionality, is not being used as such.

In particular, it muddles the purpose of the module system and of import statements, if sometimes import statements are used as a way of getting information about the current module into scope, and sometimes are used as a way of importing functionality and data from a separate package. Previously, "bare" import statements (no ./ or ../) always referred to other packages. Now, they have two competing use cases.

What's worse, there are now two ways to get per-module metadata: one via import { x } from "nodejs", for npm's proposed set of properties, and one from import.meta, for Node's set of properties (such as import.meta.url, and potentially others). What is a user to think? Why does import.meta.url work, but when they want to go to get a filename, import.meta.filename does not? Why does import { isMain } from "nodejs" work, but when they want to go and get a URL, import { url } from "nodejs" does not?

I'd urge you to consider the teachability of this direction, as well as its impact on the wider ecosystem of module users. Dismissing these concerns as "symmetrical esthetics" is, IMO, wrong. They directly bear on the principles of "satisfying the language expectations of browser-focused developers", and even of "satisfying the ergonomics expectations of node-focused developers" given the mismatch between import.meta.url and the npm-proposed properties.

The cited benefits are that it's harder to obfuscate code and prevent transpilation from working. But I don't think accounting for such obfuscated code should weigh very highly.

domenic commented Jan 14, 2018

It's true that it's technically possible to make import { require } from "nodejs" work. But it's frustrating to see that, this object which we specifically created in response to the Node.js community's requests for a place for per-module metadata and functionality, is not being used as such.

In particular, it muddles the purpose of the module system and of import statements, if sometimes import statements are used as a way of getting information about the current module into scope, and sometimes are used as a way of importing functionality and data from a separate package. Previously, "bare" import statements (no ./ or ../) always referred to other packages. Now, they have two competing use cases.

What's worse, there are now two ways to get per-module metadata: one via import { x } from "nodejs", for npm's proposed set of properties, and one from import.meta, for Node's set of properties (such as import.meta.url, and potentially others). What is a user to think? Why does import.meta.url work, but when they want to go to get a filename, import.meta.filename does not? Why does import { isMain } from "nodejs" work, but when they want to go and get a URL, import { url } from "nodejs" does not?

I'd urge you to consider the teachability of this direction, as well as its impact on the wider ecosystem of module users. Dismissing these concerns as "symmetrical esthetics" is, IMO, wrong. They directly bear on the principles of "satisfying the language expectations of browser-focused developers", and even of "satisfying the ergonomics expectations of node-focused developers" given the mismatch between import.meta.url and the npm-proposed properties.

The cited benefits are that it's harder to obfuscate code and prevent transpilation from working. But I don't think accounting for such obfuscated code should weigh very highly.

@teazean

This comment has been minimized.

Show comment
Hide comment

teazean commented Jan 15, 2018

Great

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Jan 15, 2018

@domenic The confusion between some node-specific things being on import.meta, and others on the “nodejs” virtual module, is relevant, imo.

However, I disagree that optimizing for build tooling isn’t appropriate. Most code on npm today (and in the foreseeable future) is intended for consumption by browsers, and passes through a build step. It seems reasonable to me to consider this use case carefully. Even though this proposal is “a node thing for node programs”, and shouldn’t be limited excessively by browser implementations, one common use for node programs is being compiled for browsers.

It could be reasonable to say that node-only properties (like require) are on a nodejs virtual module, and properties shared between the browser and node (like url) will be on the meta object; even if they have node-specific values in node programs.

isaacs commented Jan 15, 2018

@domenic The confusion between some node-specific things being on import.meta, and others on the “nodejs” virtual module, is relevant, imo.

However, I disagree that optimizing for build tooling isn’t appropriate. Most code on npm today (and in the foreseeable future) is intended for consumption by browsers, and passes through a build step. It seems reasonable to me to consider this use case carefully. Even though this proposal is “a node thing for node programs”, and shouldn’t be limited excessively by browser implementations, one common use for node programs is being compiled for browsers.

It could be reasonable to say that node-only properties (like require) are on a nodejs virtual module, and properties shared between the browser and node (like url) will be on the meta object; even if they have node-specific values in node programs.

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Jan 16, 2018

I didn't say anything against optimizing for build tooling. I simply said that you shouldn't optimize for obfuscated programs.

domenic commented Jan 16, 2018

I didn't say anything against optimizing for build tooling. I simply said that you shouldn't optimize for obfuscated programs.

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Jan 16, 2018

@domenic Ah, yes, I was making an assumption. My bad. Yes, obfuscation seems less relevant than transpiling. One might argue that some "transpiling" is obfuscatory, or split hairs and say "obfuscation" is just a special kind of "transpiling", but I think we can agree those arguments strain good faith. :)

isaacs commented Jan 16, 2018

@domenic Ah, yes, I was making an assumption. My bad. Yes, obfuscation seems less relevant than transpiling. One might argue that some "transpiling" is obfuscatory, or split hairs and say "obfuscation" is just a special kind of "transpiling", but I think we can agree those arguments strain good faith. :)

@sompylasar

This comment has been minimized.

Show comment
Hide comment
@sompylasar

sompylasar Jan 18, 2018

I'm sorry if I missed it in the comments, but how is this article relevant? https://medium.com/@giltayar/native-es-modules-in-nodejs-status-and-future-directions-part-i-ee5ea3001f71 — what's more up to date?

I'm sorry if I missed it in the comments, but how is this article relevant? https://medium.com/@giltayar/native-es-modules-in-nodejs-status-and-future-directions-part-i-ee5ea3001f71 — what's more up to date?

@kapv89

This comment has been minimized.

Show comment
Hide comment
@kapv89

kapv89 Jan 19, 2018

Please, please, please support this export syntax:

export * as foo from './foo'

kapv89 commented Jan 19, 2018

Please, please, please support this export syntax:

export * as foo from './foo'
@focusaurus

This comment has been minimized.

Show comment
Hide comment
@focusaurus

focusaurus Jan 20, 2018

+1 to @jmeas for --module=cjs and --module=esm. The term legacy is vague, time-relative, and has negative connotation.

+1 to @jmeas for --module=cjs and --module=esm. The term legacy is vague, time-relative, and has negative connotation.

@Pauan

This comment has been minimized.

Show comment
Hide comment
@Pauan

Pauan Jan 24, 2018

@isaacs

As it happens, throwing or polyfilling based on import {require} from 'nodejs' is a lot easier to do statically than finding all instances of import.meta.require. (Eg: const s = 'equire'; const r = import[{toString: () => 'meta'}][unescape('%72')+s]; r('lodash'))

That is incorrect, using import.meta.require is equally as easy / hard to statically analyze as import {require} from 'nodejs', there is absolutely zero difference.

Here is an example to prove my point:

const s = 'equire'; import({toString: () => 'nodejs'}).then((x) => { const r = x[unescape('%72')+s]; return r('lodash'); });

Here is another example:

import * as x from "nodejs";
const s = 'equire'; const r = x[unescape('%72')+s]; r('lodash');

Regardless of whether Node uses import.meta.require or import { require } from 'nodejs', in either case the solution is the same:

  • Transpilers will throw (or warn) on dynamic access to a module namespace object (e.g. x[some_dynamic_expression] in the above example)
  • Transpilers will throw (or warn) on dynamic import(some_dynamic_expression)
  • Transpilers will throw (or warn) on dynamic import.meta[some_dynamic_expression]

P.S. import.meta is special magical syntax, therefore using import[some_dynamic_expression] is not valid syntax, so I assume you meant import.meta[some_dynamic_expression]

Pauan commented Jan 24, 2018

@isaacs

As it happens, throwing or polyfilling based on import {require} from 'nodejs' is a lot easier to do statically than finding all instances of import.meta.require. (Eg: const s = 'equire'; const r = import[{toString: () => 'meta'}][unescape('%72')+s]; r('lodash'))

That is incorrect, using import.meta.require is equally as easy / hard to statically analyze as import {require} from 'nodejs', there is absolutely zero difference.

Here is an example to prove my point:

const s = 'equire'; import({toString: () => 'nodejs'}).then((x) => { const r = x[unescape('%72')+s]; return r('lodash'); });

Here is another example:

import * as x from "nodejs";
const s = 'equire'; const r = x[unescape('%72')+s]; r('lodash');

Regardless of whether Node uses import.meta.require or import { require } from 'nodejs', in either case the solution is the same:

  • Transpilers will throw (or warn) on dynamic access to a module namespace object (e.g. x[some_dynamic_expression] in the above example)
  • Transpilers will throw (or warn) on dynamic import(some_dynamic_expression)
  • Transpilers will throw (or warn) on dynamic import.meta[some_dynamic_expression]

P.S. import.meta is special magical syntax, therefore using import[some_dynamic_expression] is not valid syntax, so I assume you meant import.meta[some_dynamic_expression]

@Pauan

This comment has been minimized.

Show comment
Hide comment
@Pauan

Pauan Jan 24, 2018

Pros of using import.meta.require:

  • More consistent with the browser.

  • import.meta was literally and specifically designed for this situation, it is the Correct Spec-approved Solution(tm).

  • Less confusing for users, because it is simply a module-local object, with straight-forward semantics.

  • Easy for CommonJS users to understand, because import.meta.__dirname is quite similar to __dirname. You simply need to add import.meta. to the front and everything Just Works(tm).

  • Easier to transpile, because there is only a single syntax that the transpiler needs to worry about, and import.meta was designed to be easy to transpile (the transpiler only needs to add an ordinary object to the top of each module and rewrite import.meta to point to that object).

Cons of using import.meta.require:

  • Not currently implemented in Node.

Pros of using import { require } from 'nodejs':

  • Currently implemented in Node.

Cons of using import { require } from 'nodejs':

  • Transpilers need to worry about transpiling both import { require } from 'nodejs' and import('nodejs'), and these two syntaxes probably have different transpilation strategies, which increases the complexity of the transpiler.

  • Much more complex to (correctly) transpile, because it is not a normal object, it's a module namespace, and in addition it's a module namespace which is specific to each module that imports it. The transpiler can't cache it in the same way that it can cache other module namespaces.

  • More confusing for users, because they are importing things which are specific to the current module, but users generally don't expect import ... from ... to work that way.

Pauan commented Jan 24, 2018

Pros of using import.meta.require:

  • More consistent with the browser.

  • import.meta was literally and specifically designed for this situation, it is the Correct Spec-approved Solution(tm).

  • Less confusing for users, because it is simply a module-local object, with straight-forward semantics.

  • Easy for CommonJS users to understand, because import.meta.__dirname is quite similar to __dirname. You simply need to add import.meta. to the front and everything Just Works(tm).

  • Easier to transpile, because there is only a single syntax that the transpiler needs to worry about, and import.meta was designed to be easy to transpile (the transpiler only needs to add an ordinary object to the top of each module and rewrite import.meta to point to that object).

Cons of using import.meta.require:

  • Not currently implemented in Node.

Pros of using import { require } from 'nodejs':

  • Currently implemented in Node.

Cons of using import { require } from 'nodejs':

  • Transpilers need to worry about transpiling both import { require } from 'nodejs' and import('nodejs'), and these two syntaxes probably have different transpilation strategies, which increases the complexity of the transpiler.

  • Much more complex to (correctly) transpile, because it is not a normal object, it's a module namespace, and in addition it's a module namespace which is specific to each module that imports it. The transpiler can't cache it in the same way that it can cache other module namespaces.

  • More confusing for users, because they are importing things which are specific to the current module, but users generally don't expect import ... from ... to work that way.

@devsnek

This comment has been minimized.

Show comment
Hide comment
@devsnek

devsnek Feb 2, 2018

import.meta landed in node for those keeping score at home

devsnek commented Feb 2, 2018

import.meta landed in node for those keeping score at home

@cref

This comment has been minimized.

Show comment
Hide comment
@cref

cref Feb 7, 2018

Thanks for taking the time to write this proposal and for implementing it so those that shall not be named here can stop bringing up strawman arguments against it. Nobody in their right mind should want the current experimental mjs file extension-based implementation that has no regard for interoperability outside NodeJS and doesn't solve any of the real world problems but instead introduces a bunch of new ones. It's bad enough already that NodeJS has determined the faith of javascript development in a negative way for so long by using a synchronous module system (which negates elegant solutions for using the same modules in a browser context, resulting in cumbersome workflows) and also effectively killed CommonJS as a standard due to its opinionated implementation of it. Let's hope history doesn't repeat itself or else NodeJS will await the same faith as Internet Explorer.
Sorry for ranting. :)

cref commented Feb 7, 2018

Thanks for taking the time to write this proposal and for implementing it so those that shall not be named here can stop bringing up strawman arguments against it. Nobody in their right mind should want the current experimental mjs file extension-based implementation that has no regard for interoperability outside NodeJS and doesn't solve any of the real world problems but instead introduces a bunch of new ones. It's bad enough already that NodeJS has determined the faith of javascript development in a negative way for so long by using a synchronous module system (which negates elegant solutions for using the same modules in a browser context, resulting in cumbersome workflows) and also effectively killed CommonJS as a standard due to its opinionated implementation of it. Let's hope history doesn't repeat itself or else NodeJS will await the same faith as Internet Explorer.
Sorry for ranting. :)

@championswimmer

This comment has been minimized.

Show comment
Hide comment
@championswimmer

championswimmer Feb 22, 2018

This proposal makes me happy as a developer who writes significant amount of both nodejs and browser side code.
.mjs files are cruel and evil.

I really hope we end up with this solution and not .mjs

This proposal makes me happy as a developer who writes significant amount of both nodejs and browser side code.
.mjs files are cruel and evil.

I really hope we end up with this solution and not .mjs

@tamlyn

This comment has been minimized.

Show comment
Hide comment
@tamlyn

tamlyn Mar 13, 2018

I have this nightmare sometimes where a Bad Thing is about to happen but there's a way that it can be avoided. I try to warn people about the Bad Thing but they keep telling me that everything is alright. I get more and more frantic and then I wake up.

I hope we get something like this implementation instead of mjs.

@tolmasky the situation you describe is exactly the same with the current experimental mjs implementation in Node. In both instances it can be mitigated by publishing both CJS and ESM version sources in the same package. The only difference I see is this proposal requires the consuming developer to select which version to import rather than auto selecting based on the file extension. /cc @TimothyGu

tamlyn commented Mar 13, 2018

I have this nightmare sometimes where a Bad Thing is about to happen but there's a way that it can be avoided. I try to warn people about the Bad Thing but they keep telling me that everything is alright. I get more and more frantic and then I wake up.

I hope we get something like this implementation instead of mjs.

@tolmasky the situation you describe is exactly the same with the current experimental mjs implementation in Node. In both instances it can be mitigated by publishing both CJS and ESM version sources in the same package. The only difference I see is this proposal requires the consuming developer to select which version to import rather than auto selecting based on the file extension. /cc @TimothyGu

@nickpelone

This comment has been minimized.

Show comment
Hide comment
@nickpelone

nickpelone Apr 5, 2018

Please god let this be what is implemented. .mjs is nothing short of asinine.

Please god let this be what is implemented. .mjs is nothing short of asinine.

@championswimmer

This comment has been minimized.

Show comment
Hide comment
@championswimmer

championswimmer Apr 26, 2018

Comments like this do not add any value, but I'll nevertheless add it, just to show numbers for the side that doesn't want '.mjs'

I honestly will leave all interest in JS if .mjs becomes a reality. Let this get implemented. Pretty please.

Comments like this do not add any value, but I'll nevertheless add it, just to show numbers for the side that doesn't want '.mjs'

I honestly will leave all interest in JS if .mjs becomes a reality. Let this get implemented. Pretty please.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment