Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
ES Modules are terrible, actually

ES Modules are terrible, actually

This post was adapted from an earlier Twitter thread.

It's incredible how many collective developer hours have been wasted on pushing through the turd that is ES Modules (often mistakenly called "ES6 Modules"). Causing a big ecosystem divide and massive tooling support issues, for... well, no reason, really. There are no actual advantages to it. At all.

It looks shiny and new and some libraries use it in their documentation without any explanation, so people assume that it's the new thing that must be used. And then I end up having to explain to them why, unlike CommonJS, it doesn't actually work everywhere yet, and may never do so. For example, you can't import ESM modules from a CommonJS file! (Update: I've released a module that works around this issue.)

And then there's Rollup, which apparently requires ESM to be used, at least to get things like treeshaking. Which then makes people believe that treeshaking is not possible with CommonJS modules. Well, it is - Rollup just chose not to support it.

And then there's Babel, which tried to transpile import/export to require/module.exports, sidestepping the ongoing effort of standardizing the module semantics for ESM, causing broken imports and require("foo").default nonsense and spec design issues all over the place.

And then people go "but you can use ESM in browsers without a build step!", apparently not realizing that that is an utterly useless feature because loading a full dependency tree over the network would be unreasonably and unavoidably slow - you'd need as many roundtrips as there are levels of depth in your dependency tree - and so you need some kind of build step anyway, eliminating this entire supposed benefit.

And then people go "well you can statically analyze it better!", apparently not realizing that ESM doesn't actually change any of the JS semantics other than the import/export syntax, and that the import/export statements are equally analyzable as top-level require/module.exports.

"But in CommonJS you can use those elsewhere too, and that breaks static analyzers!", I hear you say. Well, yes, absolutely. But that is inherent in dynamic imports, which by the way, ESM also supports with its dynamic import() syntax. So it doesn't solve that either! Any static analyzer still needs to deal with the case of dynamic imports somehow - it's just rearranging deck chairs on the Titanic.

And then, people go "but now we at least have a standard module system!", apparently not realizing that CommonJS was literally that, the result of an attempt to standardize the various competing module systems in JS. Which, against all odds, actually succeeded!

... and then promptly got destroyed by ESM, which reintroduced a split and all sorts of incompatibility in the ecosystem, rather than just importing some updated variant of CommonJS into the language specification, which would have sidestepped almost all of these issues.

And while the initial CommonJS standardization effort succeeded due to none of the competing module systems being in particularly widespread use yet, CommonJS is so ubiquitous in Javascript-land nowadays that it will never fully go away. Which means that runtimes will forever have to keep supporting two module systems, and developers will forever be paying the cost of the interoperability issues between them.

But it's the future!

Is it really? The vast majority of people who believe they're currently using ESM, aren't even actually doing so - they're feeding their entire codebase through Babel, which deftly converts all of those snazzy import and export statements back into CommonJS syntax. Which works. So what's the point of the new module system again, if it all works with CommonJS anyway?

And it gets worse; import and export are designed as special-cased statements. Aside from the obvious problem of needing to learn a special syntax (which doesn't quite work like object destructuring) instead of reusing core language concepts, this is also a downgrade from CommonJS' require, which is a first-class expression due to just being a function call.

That might sound irrelevant on the face of it, but it has very real consequences. For example, the following pattern is simply not possible with ESM:

const someInitializedModule = require("module-name")(someOptions);

Or how about this one? Also no longer possible:

const app = express();
// ...
app.use("/users", require("./routers/users"));

Having language features available as a first-class expression is one of the most desirable properties in language design; yet for some completely unclear reason, ESM proponents decided to remove that property. There's just no way anymore to directly combine an import statement with some other JS syntax, whether or not the module path is statically specified.

The only way around this is with await import, which would break the supposed static analyzer benefits, only work in async contexts, and even then require weird hacks with parentheses to make it work correctly.

It also means that you now need to make a choice: do you want to be able to use ESM-only dependencies, or do you want to have access to patterns like the above that help you keep your codebase maintainable? ESM or maintainability, your choice!

So, congratulations, ESM proponents. You've destroyed a successful userland specification, wasted many (hundreds of?) thousands of hours of collective developer time, many hours of my own personal unpaid time trying to support people with the fallout, and created ecosystem fragmentation that will never go away, in exchange for... fuck all.

This is a disaster, and the only remaining way I see to fix it is to stop trying to make ESM happen, and deprecate it in favour of some variant of CommonJS modules being absorbed into the spec. It's not too late yet; but at some point it will be.

@privatenumber

This comment has been minimized.

Copy link

@privatenumber privatenumber commented Sep 1, 2021

RE: sindresorhus/meta#15 (reply in thread)

Some feedback

  • Spend the extra 5 min publishing this as a blog instead of a gist. Even something like Medium or dev.to would make a big difference in making it more approachable. To me, a gist gives the impression that it's scrappy or that it's a "note".

  • I would change the title. Right off the bat, it sounds like a rant/complaining and I am deterred from reading it. Try to rephrase it in a more informative/relatable way. eg. "Why ES Modules will hurt you and the ecosystem"

  • I would also take the snark and complaining out. It leaves a bad taste in the reader's mouth. Try to equip the reader with knowledge of workaround and leave them with hope that there are next steps that they can take to support you and your cause.

  • Add more structure to your post and optimize for skimmability; not because people will actually skim, but so people can get an idea of your post and be drawn into reading it. More headings, more bullet-points, more snippets. The wall of text is daunting, especially for those that are busy and are just trying to take a 5 min break.

  • Currently, your points are kind of all over like you're talking. I think it will make more sense to follow something like:

    • Context
      • What is ESM? (And what problem is it designed to solve?)
      • What is CJS?
      • Summary of the recent changes to the ecosystem
    • Your problem statement: "ESM will hurt the ecosystem"
      • Your arguments debunking the misconceptions of ESM
      • Your arguments for the long term impact of ESM
    • Your proposal and how others can help
  • I would then share the blog post on Hacker News, Reddit, and Twitter.

Feedback to the content

  • FYI, Webpack 5 has CommonJS tree-shaking built in. Maybe better to reference this than some plugin.

  • Webpack/Babel builds that are building to CJS/AMD/UMD is considered outdated IMO. Webpack is working on ESM outputs. Next gen bundlers like Rollup, Vite, Esbuild, Snowpack don't compile to CJS (by default).

  • The only real argument I see is not being able to inline require(), which isn't even a big deal to me as a full-time JS developer. The rest all seem to be misconceptions on ESM (which I didn't have, BTW).

  • require(variable) also breaks static analysis. await import(variable) has a slight advantage of import being reserved, so it can be analyzed with more certainty. In the sense that require can be any random function, I would say that import() is the true first-class expression.


Please take this feedback at face-value and either incorporate them or just disregard them. I'm not interested in having a discussion. Good luck!

BTW, I'm very impressed by how much you're able to write at a high quality in such a short period of time.

And FWIW I am still pro-ESM because of the standardization it will bring. You mentioned before that with ESM, maintainers will have to maintain multiple module outputs, but that's what I currently do: UMD & ESM. In the future, I'm glad I can remove UMD.

@fou33

This comment has been minimized.

Copy link

@fou33 fou33 commented Sep 24, 2021

RE: sindresorhus/meta#15 (reply in thread)
UNBELIEVEBLE !!!

Some feedback

  • Spend the extra 5 min publishing this as a blog instead of a gist. Even something like Medium or dev.to would make a big difference in making it more approachable. To me, a gist gives the impression that it's scrappy or that it's a "note".

  • I would change the title. Right off the bat, it sounds like a rant/complaining and I am deterred from reading it. Try to rephrase it in a more informative/relatable way. eg. "Why ES Modules will hurt you and the ecosystem"

  • I would also take the snark and complaining out. It leaves a bad taste in the reader's mouth. Try to equip the reader with knowledge of workaround and leave them with hope that there are next steps that they can take to support you and your cause.

  • Add more structure to your post and optimize for skimmability; not because people will actually skim, but so people can get an idea of your post and be drawn into reading it. More headings, more bullet-points, more snippets. The wall of text is daunting, especially for those that are busy and are just trying to take a 5 min break.

  • Currently, your points are kind of all over like you're talking. I think it will make more sense to follow something like:

    • Context

      • What is ESM? (And what problem is it designed to solve?)
      • What is CJS?
      • Summary of the recent changes to the ecosystem
    • Your problem statement: "ESM will hurt the ecosystem"

      • Your arguments debunking the misconceptions of ESM
      • Your arguments for the long term impact of ESM
    • Your proposal and how others can help

  • I would then share the blog post on Hacker News, Reddit, and Twitter.

Feedback to the content

  • FYI, Webpack 5 has CommonJS tree-shaking built in. Maybe better to reference this than some plugin.
  • Webpack/Babel builds that are building to CJS/AMD/UMD is considered outdated IMO. Webpack is working on ESM outputs. Next gen bundlers like Rollup, Vite, Esbuild, Snowpack don't compile to CJS (by default).
  • The only real argument I see is not being able to inline require(), which isn't even a big deal to me as a full-time JS developer. The rest all seem to be misconceptions on ESM (which I didn't have, BTW).
  • require(variable) also breaks static analysis. await import(variable) has a slight advantage of import being reserved, so it can be analyzed with more certainty. In the sense that require can be any random function, I would say that import() is the true first-class expression.

Please take this feedback at face-value and either incorporate them or just disregard them. I'm not interested in having a discussion. Good luck!

BTW, I'm very impressed by how much you're able to write at a high quality in such a short period of time.

And FWIW I am still pro-ESM because of the standardization it will bring. You mentioned before that with ESM, maintainers will have to maintain multiple module outputs, but that's what I currently do: UMD

@mk-pmb

This comment has been minimized.

Copy link

@mk-pmb mk-pmb commented Oct 17, 2021

@fou33 Please edit your comment to fix the quoting and quote only the parts relevant to your addition. I needed a diff tool to identify your contributions.

@mk-pmb

This comment has been minimized.

Copy link

@mk-pmb mk-pmb commented Oct 17, 2021

As for structure feedback, it would have helped me read (by reduceing my mental load) if you start with a debunk section where I can quickly find the headline for what my personal motivation is for using ESM, so I can asap read why I'm mistaken.

Which I'm not convinced I am.
I often work in scenarios where the round trip times for use in a browser are much less a hassle than setting up the tool chain for bundling at the correct time. When loading from LAN, I can just pre-bundle the deep dependencies, and my main project's dependencies are shallow enough. At least that was my experience back in the days when CujoJS still worked. My hope was that shiny new ESM might once again bring back that bliss of "just works". I can still optimize download times later when the app does what I want.

I fully agree with your language design complaints though. I wish we could somehow upgrade CJS to the async nature of the web, and find a way to make it usable for quickly sketched experiments even on webspace that has very limited editing capability (e.g. a text file editor in some CMS file manager) and does not support CGI or fancy proxy setup. Just like in the good old days with CujoJS.

Every now and then I spend another weekend on trying to make it work with SystemJS and babel in the browser, neither an elegant solution nor have I had any significant success yet.

@OmgImAlexis

This comment has been minimized.

Copy link

@OmgImAlexis OmgImAlexis commented Nov 5, 2021

TLDR: I don't like new things. 👍

@jimmywarting

This comment has been minimized.

Copy link

@jimmywarting jimmywarting commented Nov 5, 2021

here is one thing, browser needed a modular system and require didn't fit well into. you say that one should use bundler/minifiers. it's a valid case but not always desirable. Some wish wish to use buildless tools and not being forced to learn NodeJS or being stuck with it as a must have in your tool belt. code splitting is also highly preferable. and using npm is not for everyone.
require works grate for NodeJS alone where it has access to the filesystem but the problem with it is the resolver, and it's sync nature. Browser (Deno, and nodes new http-loader) isn't built for handling it synchronous and blocking things, that would be bad. the other reason is that it can't figure out remotely what file you needed when you write require('./foo') it dose not know if you ment ./foo, ./foo.js or ./foo/index.js, same thing with require('../')

TLDR: i also think it sucks to be devided into cjs and esm, but i prefer the new change.

@Aeolun

This comment has been minimized.

Copy link

@Aeolun Aeolun commented Nov 7, 2021

It leaves a bad taste in the reader's mouth.

Not really. It reflected my thoughts perfectly.

I like the syntax of ESM better, but I’ve never understood why what the difference was from a practical perspective.

Anything we’re doing with ESM could have been done with CommonJS (or by marginally modifying CommonJS).

@kesor

This comment has been minimized.

Copy link

@kesor kesor commented Nov 7, 2021

A couple more broken silly things that ES modules bring with it:

  • Node.js has a way to clear cached files loaded with require but the cache used by its ES module implementation is completely hidden for some reason and has no way to access it. So doing "hot reload" during development like for example mocha and other tools use, is basically impossible. To be fair, it is possible (maybe) by loading the same module multiple times into memory and giving it a different name each time ... which is a hack so ugly it makes me want to throw up.
  • ES modules are more or less working in browser, but in most cases you would have to use the importmap feature, which is unsupported and hidden behind a feature flag in a small selection of modern browsers. So basically you can't really do import "vue" but rather have to specify full URLs for each of your imports in EACH of the files where you might want to import it. Turning short statements into full-url statements and duplicated all over the place.
@sebakerckhof

This comment has been minimized.

Copy link

@sebakerckhof sebakerckhof commented Nov 7, 2021

It's not the fact imports are top-level only that makes it statically analyzable (since the analyzer could just hoist nested imports, potentially after tree-shaking). It's the fact they can't be variable names like commonjs allows (e.g. require(someVariable)).

In fact there's an es proposal to allow nested imports by benjamin newman: https://github.com/benjamn/reify/blob/main/PROPOSAL.md . And you can use it via the reify compiler or with babel plugins. Granted, the proposal doesn't seem to get a lot of traction.

@princefishthrower

This comment has been minimized.

Copy link

@princefishthrower princefishthrower commented Nov 7, 2021

It gets better - with TypeScript and ESM you have to include the .js extension for your imports, even when they aren't really .js files until being compiled... TypeScript has chosen to follow in the steps of Rollup to not support automatically appending these during the compile stage - they claim that's "the developer's responsibility". Love ESM ❤️ 😂

@feeela

This comment has been minimized.

Copy link

@feeela feeela commented Nov 7, 2021

  • Node.js has a way to clear cached files loaded with require but the cache used by its ES module implementation is completely hidden for some reason and has no way to access it. So doing "hot reload" during development like for example mocha and other tools use, is basically impossible. To be fair, it is possible (maybe) by loading the same module multiple times into memory and giving it a different name each time ... which is a hack so ugly it makes me want to throw up.

Why do you use a cache on your development environment at all? You could always set the cache expiration to 0 during development. This is a server setting and the caching behaviour should be defined through the environment, not through one of many scripting languages, that may be used in a browser.

@shellscape

This comment has been minimized.

Copy link

@shellscape shellscape commented Nov 7, 2021

And then there's Rollup, which apparently requires ESM to be used, at least to get things like treeshaking. Which then makes people believe that treeshaking is not possible with CommonJS modules. Well, it is - Rollup just chose not to support it.

Ignorance is showing here. Rollup was created to be predecated on ESM. There's no "apparently" to it. CommonJS adapters for it came later. The original author liked ESM and built something around it. Tossing it in the mix as something "bad" because of that isn't fair. You wouldn't wear blue clothes, and then complain about blue, if you didn't like blue.

@churchofthought

This comment has been minimized.

Copy link

@churchofthought churchofthought commented Nov 7, 2021

It gets better - with TypeScript and ESM you have to include the .js extension for your imports, even when they aren't really .js files until being compiled... TypeScript has chosen to follow in the steps of Rollup to not support automatically appending these during the compile stage - they claim that's "the developer's responsibility". Love ESM ❤️ 😂

@princefishthrower I had the same issue which drove me fucking crazy. And it's insane the Typescript folks close all tickets regarding this issue with TSC.

What worked for me is to keep my extensionless imports and use the Node flag --experimental-specifier-resolution=node

Let me know if this helps you at all.

@mk-pmb

This comment has been minimized.

Copy link

@mk-pmb mk-pmb commented Nov 7, 2021

@feeela

set the cache expiration to 0 during development. This is a server setting

Are you talking about HTTP caching? That would be unrelated to an ESM module cache afaik.

@tomasklaen

This comment has been minimized.

Copy link

@tomasklaen tomasklaen commented Nov 7, 2021

Node.js has a way to clear cached files loaded with require but the cache used by its ES module implementation is completely hidden

Yes, the inability to clear cache and reload modules during runtime is a huge issue with ES modules that crippled my app until I figured out a workaround. I've created an import() abstraction that loads modules from a symlinked node_modules directory, and every time I need to reload them I rename the symlink. This workaround makes me sick, and has an issue of introducing memory leaks, since the old modules are not cleared.

Another place where the lack of ESM support is a big issue atm is electron, which still doesn't support loading ESM node modules via import() in renderer process. And due to ESM push, there is a lot of npm modules that are ESM only now, such as everything new and recently touched by sindresorhus. I can't use any of these modules in my electron app as there is just no mechanism to import them. (I need to load modules dynamically during runtime, they are not part of the packaged app).

This whole environment of TS(X)<>JS(X)<>commonjs<>ESM<>node<>electron<>browsers and all of the bundlers and packagers is an insane mess.

@mk-pmb

This comment has been minimized.

Copy link

@mk-pmb mk-pmb commented Nov 7, 2021

@tomasklaen afail the ESM module cache works by full URL, so you can probably keep the files in position and just change the ?query or #hash part of the URL.

@kesor

This comment has been minimized.

Copy link

@kesor kesor commented Nov 15, 2021

Yes, the inability to clear cache and reload modules during runtime is a huge issue with ES modules that crippled my app until I figured out a workaround. I've created an import() abstraction that loads modules from a symlinked node_modules directory, and every time I need to reload them I rename the symlink. This workaround makes me sick, and has an issue of introducing memory leaks, since the old modules are not cleared.

Exactly this. Solving stupid standards using ugly hacks ... this shouldn't even be a thing.

@srcspider

This comment has been minimized.

Copy link

@srcspider srcspider commented Nov 21, 2021

ES Modules, in short:

What we got:

JS file imports.... for youtubers and snippets only! 💩 ...and browsers that dont' exist, ever, woohoo
Buy-in now and get bonus buzzword too! like "static analysis", "security" and "modern" to add to your clickbait titles

What we should have probably gotten:

// "from" before "import" so autocomplete can ACTUALLY help people
from "node:fs" import fs;
from "url:https://something/v1.2.3/hello.js" import hello;
from "lodash" import _;
from "lodash" import { padEnd: pad_end };
from "../../../../../../../place/whatever/unreadable" import unreadableImport;
from "/tools/some/place/in/project/thingy" import { default: thingy, thingyDb };

const import1 = await import('url:https://something/v1.2.3/hello.js');
const import2 = importSync(`/app/routes/${something}`); // throws unsupported error if used in browsers, etc
// when consumers import this function it's seen as writeFileSync(file, data, options)
export secure function writeFileSync(sec, file, data, options) {
    const callerModule = sec.sig.module;
    const callerFilePath = sec.sig.moduleSourcePath;
    const contextEnvironment = sec.environment;

    const contextSecurityMode = sec.mode; 
          // for nodejs translates to package.json / security / mode (insecure, basic, zerotrust)

    const context = sec.context; 
          // for nodejs translates to package.json / security / whitelist / "node:fs"

    if (mode != 'insecure' && ! canWriteToFilesystem(context, callerModule)) {
         throw new SecurityError("writeFileSync is not allowed in to be used in the context of ${callerModule} (file: ${callerFilePath})");
    }

    // blah blah
}

Module Resolution Changes:

  • package.json is read as a json6 file now
  • package.yaml is also read if package.json is not found
  • const ... = require(...) is treated as language construct that translates to: from "module" import moduleObject;
    • the pattern require(module)(conf) is also automatically understood
    • var, let, const patterns are automatically understood; other strange patterns may not be understood
    • nested destruction in require style imports will throw error (only one level of destruction is permitted)
    • any ambiguous usage of require will throw error
    • use of variables in require is not permitted and will throw syntax error (switch to either import or importSync)
  • semicolon at the end of import is mandatory (it's required for any systems that have import / from syntax already)
  • all internal modules require use of "node:" prefix
  • any url can be loaded using "url:" prefix
  • any module starting with a slash will translate to whatever you have set in the new package.json / moduleRoot
    property; by default "./src"

Security Changes:

  • many interactive modes have been added to ensure that security settings can be done easily and painlessly
  • by default node now runs in one of several security modes
    • insecure: no restrictions
      • even in insecure mode, modules are still loaded as "read-only"
    • (default) basic mode: file reads ok, all http blocked, all writes blocked, url imports allowed
    • zerotrust: everything blocked and even static url imports need to be whitelisted too
  • in all modes that block, you can whitelist modules
    • modules in "node:" prefix use "whitelist" field in package.json
      • depending on module multiple settings may be available including url whitelisting and other authorization and authentication
        options as you need or the modules are able to provide
    • modules in 3rd parties have all the capabilities that "node:" modules have in terms of security
    • when a module comes with a whitelist, you can choose to trust the modules whitelist (this is explicitly choice, not implicit)
      • trusting a module requires version to be trusted (interactive mode is recommended)
      • in zero-trust mode only "node:" and import (ie. dynamic imports) rules are respected when you choose to "trust" a module's whitelist (you have to individually trust every other sub-module it uses)
      • you can choose to ignore a modules whitelist (ie. not trust them) and only trust settings you specify
      • for legacy modules that do not have their own whitelist you have to trust them individually
  • audit authorities can now be specified
    • when specified all modules (down to their individual versions) automatically must be verified by at least one audit authority
      • submodules to a module are treated completely independent of their parent, each sub-dependency is another check
    • you can specify individual modules yourself while this mode is active (ie. act as an audit authority)
      • when doing so if a module has dependencies you have to check and specify them as well (including every single version)
  • contents of npm packages are now easily visible in npm registry
    • in addition to file visibility metadata regarding usage of node: modules, url: modules etc is also visible
@JasonHenriksen

This comment has been minimized.

Copy link

@JasonHenriksen JasonHenriksen commented Dec 2, 2021

I throught I was reasonably competent in TypeScript. But trying to write a new module that uses some other off the shelf module I simply cannot get past all of the import problems. I've been three days as trying to simply do an import!

I am seriously considering dropping TypeScript and JavaScript both over this ridiculous mess.

  • I don't want to specify file name extensions, it's not the early 90s any more.
  • I don't want to have to care about ESM vs CJS. I shouldn't even have to know that they hell they are in order to build something that works.
  • I don't want to know "oh this well work if you add this magic word to that package.json, but oh, your jest tests wont pick it up and you'll have no idea why."

How can I advocate for TypeScript or JavaScript when I can't even do an import simply!?!

@jimmywarting

This comment has been minimized.

Copy link

@jimmywarting jimmywarting commented Dec 2, 2021

I don't want to specify file name extensions, it's not the early 90s any more.

This might be alright for cjs and node's eco-system where it has access to the filesystem and can lookup the extension and also scan for index.js files.

But when you take browser or Deno's modular system with import statements and introduce it into NodeJS and tries to resolve a path remotely then you can't automatically lookup files like indexes and extension less files. Browser, Deno or NodeJS new http-module loader. It can't guess what you meant by import('./foo') if it should be ./foo/index.js or ./foo.js. It would require that servers where able to resolve paths the same way NodeJS handles CommonJS

it was a bad design decision that was done way back that have come back and bit us now when browser have gotten a own modular system.

I don't want to have to care about ESM vs CJS. I shouldn't even have to know that they hell they are in order to build something that works.

Then use ESM and {"type": "module"} NodeJS can load cjs but not the other way around (unless you don't use dynamic import) due to the async nature of how ES modules are built (eg with top level await).

your jest tests wont pick it up

Just don't use Jest, change testing library.

How can I advocate for TypeScript

I have forcefully been required to work with TypeScript 2y @ work in my total 12y working with JavaScript... And TS was nothing but painful with very little gain. spend lots of wastful time compiling and don't run anywhere without being compiled in the first place. I have never advocated TypeScript and I wouldn't recommend it to anybody... TypeScript is not JavaScript... it's a own language by Microsoft built on top of JavaScript... TypeScript is not a first class programming language... JavaScript is, that why TypeScript will always be one step behind, and you might not need typescript to have a type safety language

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment