Skip to content

Instantly share code, notes, and snippets.

@joepie91
Last active December 19, 2024 00:04
Show Gist options
  • Save joepie91/bca2fda868c1e8b2c2caf76af7dfcad3 to your computer and use it in GitHub Desktop.
Save joepie91/bca2fda868c1e8b2c2caf76af7dfcad3 to your computer and use it in GitHub Desktop.
ES Modules are terrible, actually

ES Modules are terrible, actually

This post was adapted from an earlier Twitter thread.

It's incredible how many collective developer hours have been wasted on pushing through the turd that is ES Modules (often mistakenly called "ES6 Modules"). Causing a big ecosystem divide and massive tooling support issues, for... well, no reason, really. There are no actual advantages to it. At all.

It looks shiny and new and some libraries use it in their documentation without any explanation, so people assume that it's the new thing that must be used. And then I end up having to explain to them why, unlike CommonJS, it doesn't actually work everywhere yet, and may never do so. For example, you can't import ESM modules from a CommonJS file! (Update: I've released a module that works around this issue.)

And then there's Rollup, which apparently requires ESM to be used, at least to get things like treeshaking. Which then makes people believe that treeshaking is not possible with CommonJS modules. Well, it is - Rollup just chose not to support it.

And then there's Babel, which tried to transpile import/export to require/module.exports, sidestepping the ongoing effort of standardizing the module semantics for ESM, causing broken imports and require("foo").default nonsense and spec design issues all over the place.

And then people go "but you can use ESM in browsers without a build step!", apparently not realizing that that is an utterly useless feature because loading a full dependency tree over the network would be unreasonably and unavoidably slow - you'd need as many roundtrips as there are levels of depth in your dependency tree - and so you need some kind of build step anyway, eliminating this entire supposed benefit.

And then people go "well you can statically analyze it better!", apparently not realizing that ESM doesn't actually change any of the JS semantics other than the import/export syntax, and that the import/export statements are equally analyzable as top-level require/module.exports.

"But in CommonJS you can use those elsewhere too, and that breaks static analyzers!", I hear you say. Well, yes, absolutely. But that is inherent in dynamic imports, which by the way, ESM also supports with its dynamic import() syntax. So it doesn't solve that either! Any static analyzer still needs to deal with the case of dynamic imports somehow - it's just rearranging deck chairs on the Titanic.

And then, people go "but now we at least have a standard module system!", apparently not realizing that CommonJS was literally that, the result of an attempt to standardize the various competing module systems in JS. Which, against all odds, actually succeeded!

... and then promptly got destroyed by ESM, which reintroduced a split and all sorts of incompatibility in the ecosystem, rather than just importing some updated variant of CommonJS into the language specification, which would have sidestepped almost all of these issues.

And while the initial CommonJS standardization effort succeeded due to none of the competing module systems being in particularly widespread use yet, CommonJS is so ubiquitous in Javascript-land nowadays that it will never fully go away. Which means that runtimes will forever have to keep supporting two module systems, and developers will forever be paying the cost of the interoperability issues between them.

But it's the future!

Is it really? The vast majority of people who believe they're currently using ESM, aren't even actually doing so - they're feeding their entire codebase through Babel, which deftly converts all of those snazzy import and export statements back into CommonJS syntax. Which works. So what's the point of the new module system again, if it all works with CommonJS anyway?

And it gets worse; import and export are designed as special-cased statements. Aside from the obvious problem of needing to learn a special syntax (which doesn't quite work like object destructuring) instead of reusing core language concepts, this is also a downgrade from CommonJS' require, which is a first-class expression due to just being a function call.

That might sound irrelevant on the face of it, but it has very real consequences. For example, the following pattern is simply not possible with ESM:

const someInitializedModule = require("module-name")(someOptions);

Or how about this one? Also no longer possible:

const app = express();
// ...
app.use("/users", require("./routers/users"));

Having language features available as a first-class expression is one of the most desirable properties in language design; yet for some completely unclear reason, ESM proponents decided to remove that property. There's just no way anymore to directly combine an import statement with some other JS syntax, whether or not the module path is statically specified.

The only way around this is with await import, which would break the supposed static analyzer benefits, only work in async contexts, and even then require weird hacks with parentheses to make it work correctly.

It also means that you now need to make a choice: do you want to be able to use ESM-only dependencies, or do you want to have access to patterns like the above that help you keep your codebase maintainable? ESM or maintainability, your choice!

So, congratulations, ESM proponents. You've destroyed a successful userland specification, wasted many (hundreds of?) thousands of hours of collective developer time, many hours of my own personal unpaid time trying to support people with the fallout, and created ecosystem fragmentation that will never go away, in exchange for... fuck all.

This is a disaster, and the only remaining way I see to fix it is to stop trying to make ESM happen, and deprecate it in favour of some variant of CommonJS modules being absorbed into the spec. It's not too late yet; but at some point it will be.

@divmgl
Copy link

divmgl commented Jan 18, 2024

@camj256 I don't understand your apparent frustration. I really don't understand an idea of bitterness. The last time I check Node.js doesn't run the world, technology is not static, time keeps on ticking. Ever try esbuild? There's a --format=cjs option...

This is disingenuous. ESBuild is a community built tool. Most of the complaints here are re: the underlying system that exists in Node.js without any additional tooling. There's no reason why the transition from CommonJS to ES Modules had to be so painful. I'd like to think that this was one of the many catalysts for Bun's existence.

I actually landed on an approach to solving a majority of these ES Modules issues with Node.js, TypeScript and ES Modules and that's to just use ESBuild, as you recommended. I had to build my own scripts to basically take the packages in my monorepo and convert them to valid ESBuild packages (this includes server-side packages).

Thankfully there's now an excellent tool that uses ESBuild under the hood to solve a lot of these pain points: https://github.com/privatenumber/tsx I'm super grateful that a tool like this exists (and works basically flawlessly out of the box) but all of this could have been avoided with a better migration path.

@guest271314
Copy link

@divmgl

This is disingenuous.

No. That's my genuine opinion.

Most of the complaints here are re: the underlying system that exists in Node.js without any additional tooling.

There's nothing you can do about what Node.js maintainers decide to do in the nodejs main branch. You can fork node and do whatever you want.

Yes, I have read the problem statement.

Programmers solve problems.

@guest271314
Copy link

@Rush From [Native Messaging] Python 2 was sunset January 1st, 2020 #489

https://www.python.org/doc/sunset-python-2/

Sunsetting Python 2

We are volunteers who make and take care of the Python programming language. We have decided that January 1, 2020, was the day that we sunset Python 2. That means that we will not improve it anymore after that day, even if someone finds a security problem in it. You should upgrade to Python 3 as soon as you can.

I advised the GoogleChrome/chrome-extensions-samples folks their code was broken here GoogleChrome/chrome-extensions-samples#805 and the MDN Web Docs folks their code was broken here mdn/webextensions-examples#509.

GoogleChrome folks had banned me by then. Then closed the issue. They never fixed their broken Python 2 code https://github.com/GoogleChrome/chrome-extensions-samples/blob/main/_archive/mv2/api/nativeMessaging/host/native-messaging-example-host that doesn't work on Linux anyway because there in general is no /usr/bin/python, there is a /usr/bin/python3.

MDN Web Docs also wound up mergin a working Node.js Native Messaging host example mdn/content#27228. GoogleChrome samples is still stuck in 2013.

@mk-pmb
Copy link

mk-pmb commented Jan 19, 2024

Just my default rant about JS "classes" (click to open)

@camj256

I understand wanting to write things as classes

Understanding is ok, I just hope you don't agree. Rather, teach your folks why the class-less object-centric approach is (can be proven to be) mightier than class-centric approach. Hijacking the term "Object-oriented Programming" for the latter was a genious but evil marketing move from false prophets trying to lead all mankind astray.
Last time I checked, ECMAScript still doesn't have classes, and that's a good thing. Recent editions added some ways to abuse the class keyword and a terminology of "class-like" somethings. It's a tradition for JS to introduce footguns. Firing them is entirely optional, and best left to fools.

@camj256
Copy link

camj256 commented Jan 19, 2024

@mk-pmb Yeah exactly. However, there are enough people in this community that don't agree anymore. ...and thats a problem. (clearly, as demonstrated by this thread) It's making things sloppy and less compatible. Yeah, the language is only hurting itself trying to make changes like this at this point.

Our codebase is easy to read, maintain, and is lightning fast for how large it is. For how well its designed actually I don't really want to hack together builds either.

@leonsilicon
Copy link

leonsilicon commented Apr 26, 2024

Update: Node.js (v22, and hopefully v20) now supports requiring (most) ESM code from CommonJS (nodejs/node#51977 (comment); @joyeecheung wrote an insightful post about require(esm) worth reading), so it turns out that the ESM spec wasn't actually inherently incompatible with CommonJS, but rather just due to how Node.js implemented it. So this gist (and all the incompatibility-related criticism towards the ESM spec itself) can now be considered outdated :)

@dboreham
Copy link

Ended up here after a hair-pulling fest on one of our projects.
So...JS people have managed to re-invent Python3.
In about 10 years everything will be fine and all will be forgotten.

@Rush
Copy link

Rush commented May 17, 2024

Since we're ranting, does anyone else have issues with node-gyp? Why on earth is it still using Python? Recently, I had to lock my Python version to 3.8.19 because my project won't build with newer versions. This is all due to older dependencies that rely on an outdated version of node-gyp. It's incredibly frustrating!

@JamesAlp
Copy link

Update: Node.js (v22, and hopefully v20) now supports requiring (most) ESM code from CommonJS (nodejs/node#51977 (comment); @joyeecheung wrote an insightful post about require(esm) worth reading), so it turns out that the ESM spec wasn't actually inherently incompatible with CommonJS, but rather just due to how Node.js implemented it. So this gist (and all the incompatibility-related criticism towards the ESM spec itself) can now be considered outdated :)

"Most" is definitely correct lol. For example if you wish to use Chalk, you must install 4.1.2 or earlier in your CJS project. Trying to convert your NestJS server to ESM instead of just keeping it CJS introduces so very many problems that just should not be problems. All the different imports now needing to use file extensions but only on some imports, needing to modify your tsconfig and package files with specific settings, a host of other changes and, surprise, sometimes the development build just cannot locate certain files at runtime and everything breaks! But other times when you start it up again it'll just work for that import but fail on another. I'm sure there is a solution to this... but for me the ultimate solution is to just skip all of this nonsense, stick with CJS and just install an older version of a package if I really want it.

@leonsilicon
Copy link

leonsilicon commented Jun 14, 2024

"Most" is definitely correct lol.

Most meaning ESM code that doesn't use TLA (top-level await), which is the vast majority (chalk does not use TLA)

Also that seems like a NestJS problem, not an ESM one (i.e. it's NestJS's responsibility to support ESM - it doesn't seem fair to criticize the ESM spec itself because NestJS doesn't support it well)

@guest271314
Copy link

@ClickClickDerk
Copy link

IDK this might be old but i learned js5 like 12 years ago and took a break the past like 5 years just coding little projects for myself. I'm here today cause i dont understand modules and find them to be dumb. you can break up you js code to files that pages depend on it. I also taught myself to code so i went through one hell of a learning curve. i learned html, css, then js (client-side) before ever looking at how a server side language works and i remember because js is a loosely type language it made learning php or python more challanging and vice versa for the guys that knew strict server side programming didnt understand js to well. looking at it now es6 looks like the server side guys are turning the loosely typed language into be more readable as they understand how server side works. looking at all these js files and how AI is programming im like not every file has to be 1-2kb or less and have a ton of modules. i wrote free movies site back in the day must of had over 200,000 lines of js and it would load a bit slow on a phone but a desktop or laptop no problem. its 10 years later and everything is 100x faster. break up your files, lazy load, use ajax and you code is blazing fast today that way but now im hitting walls left and right with modules. if babel is really just turning es6 into older js syntax then that doesnt make sense. i look at babel as owning everyones code and going to strong arm people into making js and babel the standard. eww javascript is a beautiful language and es6 is ruining it. imo lol

@cmawhorter
Copy link

here i stand sit today in hour three of trying to get a complicated esm project packaged. this is -- not exaggerating -- the 10th time i've been here in the last several years with various projects.

this time around, i've tried doing everything right: it's a fresh project, i use esm deps AFAIK (sub-deps. who knows), i'm building it in node22 with a type: module in package.json. the only caveat to is that i'm using typescript, but i've followed all those guides people point to that exclaim esm is easy and it's time to use it.

from my experience, esm only works in the lab.

It's incredible how many collective developer hours have been wasted on pushing through the turd that is ES Modules

though i don't agree esm is a turd, OP is absolutely correct about that opening line. i remember filling out a dev survey about the possibility of esm long ago and begging in my responses to not do exactly what has been done.

so here we all are. and here i am. again. considering abandoning esm and using cjs because it just fucking works and who has the time for all this bullshit.

i'll see you all again in six months.

@guest271314
Copy link

@cmawhorter

i'm building it in node22 with a type: module in package.json. the only caveat to is that i'm using typescript.

That's one way of doing things.

Another way is to just use node --experimental-default-type=module --experimental-network-imports, static import and dynamic import() without any package.json file at all.

Yet another way is to use Deno, which support TypeScript and Import Maps out of the box. Again, no package.json involved, unless you want package.json involved.

Good luck!

@cmawhorter
Copy link

cmawhorter commented Jul 25, 2024 via email

@dboreham
Copy link

The last comments above I think illustrate a larger problem which is that tooling is "fixed" for the node.js case (cli process running on Linux) but the same tooling is also used to build browser-hosted applications. In my experience the tooling isn't fixed for those use cases.

@jensbodal
Copy link

I wouldn't call using experimental flags a "fix", just progress towards the goal. Fact is it's 4 years after this post was created and it's not easy at all to setup ESM compatible packages in a ubiquitous consistent way. At least for the frontend and my uses cases I can just use Vite for the backend. Unfortunately the Vite+NestJS options haven't worked out so far for us so still stuck on CJS for our NestJs+Express/Fastify backends.

@Rush
Copy link

Rush commented Jul 25, 2024

here i stand sit today in hour three of trying to get a complicated esm project packaged. this is -- not exaggerating -- the 10th time i've been here in the last several years with various projects.

this time around, i've tried doing everything right: it's a fresh project, i use esm deps AFAIK (sub-deps. who knows), i'm building it in node22 with a type: module in package.json. the only caveat to is that i'm using typescript, but i've followed all those guides people point to that exclaim esm is easy and it's time to use it.

from my experience, esm only works in the lab.

It's incredible how many collective developer hours have been wasted on pushing through the turd that is ES Modules

though i don't agree esm is a turd, OP is absolutely correct about that opening line. i remember filling out a dev survey about the possibility of esm long ago and begging in my responses to not do exactly what has been done.

so here we all are. and here i am. again. considering abandoning esm and using cjs because it just fucking works and who has the time for all this bullshit.

i'll see you all again in six months.

To me, the worst part of all of this is that tens of thousands of developers will take all of this for granted. Thinking that all of that has to be hard and one needs PhD in tooling. Ridiculous.

@mk-pmb
Copy link

mk-pmb commented Jul 25, 2024

Everyone complaining please provide links to your projects so that people interested in improving our situation have lots of test material.

@csvan
Copy link

csvan commented Jul 25, 2024

Everyone complaining please provide links to your projects so that people interested in improving our situation have lots of test material.

I would wager the vast majority of people "complaining" here - including me - work in corporate environments with closed sources.

@guest271314
Copy link

I'm not tied to Node.js exclusively.

If there is a Node.js-specific package I'm interested in, I immediately bundle to Ecmascript Modules with bun build or deno bundle, or deno_emit. No more CommonJS and require(). That's it.

No need for esbuild or webpack. Just use multiple JavaScript runtimes that do what Node.js can't or doesn't do, then use the bundeld code in Node.js world. If you must only use Node.js.

There are dozens of JavaScript engines and runtimes circa 2024. Why would any JavaScript programmer, developer, or hacker only use one?

@JamesAlp
Copy link

here i stand sit today in hour three of trying to get a complicated esm project packaged. this is -- not exaggerating -- the 10th time i've been here in the last several years with various projects.
this time around, i've tried doing everything right: it's a fresh project, i use esm deps AFAIK (sub-deps. who knows), i'm building it in node22 with a type: module in package.json. the only caveat to is that i'm using typescript, but i've followed all those guides people point to that exclaim esm is easy and it's time to use it.
from my experience, esm only works in the lab.

It's incredible how many collective developer hours have been wasted on pushing through the turd that is ES Modules

though i don't agree esm is a turd, OP is absolutely correct about that opening line. i remember filling out a dev survey about the possibility of esm long ago and begging in my responses to not do exactly what has been done.
so here we all are. and here i am. again. considering abandoning esm and using cjs because it just fucking works and who has the time for all this bullshit.
i'll see you all again in six months.

To me, the worst part of all of this is that tens of thousands of developers will take all of this for granted. Thinking that all of that has to be hard and one needs PhD in tooling. Ridiculous.

I'm not sure if that was meant to be a dig or what, but regardless I'd be interested to hear what your tooling is that makes using ESM easier if you're willing to share. I fully moved back to just CJS because I didn't really want to keep getting a headache trying to figure it out.

@mk-pmb
Copy link

mk-pmb commented Jul 26, 2024

I would wager the vast majority of people "complaining" here - including me - work in corporate environments with closed sources.

It may still be possible to make a minimum failing example with very similar structure. Even if management doesn't consider it worth work time, someone may opt to do it in their leisure time just for the hopes of a less annoying job experience soon. If management fears that even a minimum failing example would be too much disclosure, the hopes of a more efficient future workflow may convince them.

@JamesAlp
Copy link

I would wager the vast majority of people "complaining" here - including me - work in corporate environments with closed sources.

It may still be possible to make a minimum failing example with very similar structure. Even if management doesn't consider it worth work time, someone may opt to do it in their leisure time just for the hopes of a less annoying job experience soon. If management fears that even a minimum failing example would be too much disclosure, the hopes of a more efficient future workflow may convince them.

This is a pretty naïve way of thinking. Upper management at a company almost never care about the particular way a technology works, they just care if it works or not. Those managers aren't thinking of how they can improve the technology they don't even know the name of when they go to work every day, they're thinking "how do I get these numbers to go up?" and anything else better be very directly related to that topic or they don't care at all.

@guest271314
Copy link

Wow. Sounds like a hostile work environment. Stifling. No thank, you.

On the other hand, professional programmers should know how to make things work, on their own. So it's not matter of how to do something. It's solely a matter of what people that cut the checks want.

I read above about confusion re the former. And resignation as to the latter.

You have to want to understand how something works in order to make that process work.

@mk-pmb
Copy link

mk-pmb commented Jul 27, 2024

Yeah. If a company's programmers' efficiency and/or morale suffers from ESM incompatibilities, and management impedes the release of a minimum failing example project, their problem is no longer about ESM.

@JamesAlp
Copy link

That's just the thing though. Does a company suffer from lack of esm support? I would wager most don't bother with esm.

@iambumblehead
Copy link

Common Lisp, Perl and Python were similarly marred with "bad decisions" after becoming popular, what a coincidence!

@rubengmurray
Copy link

chai-as-promised(@8.0.0) another one that creates dreaded compatibility errors now. There really should be some railguards in-place that an installation of ESM dependency is prevented in a non-ESM project. Such a waste of time.

@guest271314
Copy link

There really should be some railguards in-place that an installation of ESM dependency is prevented in a non-ESM project.

Read the source code?

https://www.npmjs.com/package/chai-as-promised?activeTab=code at /lib/chai-as-promised.js

import * as checkErrorDefault from 'check-error';

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment