Skip to content

Instantly share code, notes, and snippets.

@joepie91
Last active July 12, 2024 23:04
Show Gist options
  • Save joepie91/bca2fda868c1e8b2c2caf76af7dfcad3 to your computer and use it in GitHub Desktop.
Save joepie91/bca2fda868c1e8b2c2caf76af7dfcad3 to your computer and use it in GitHub Desktop.
ES Modules are terrible, actually

ES Modules are terrible, actually

This post was adapted from an earlier Twitter thread.

It's incredible how many collective developer hours have been wasted on pushing through the turd that is ES Modules (often mistakenly called "ES6 Modules"). Causing a big ecosystem divide and massive tooling support issues, for... well, no reason, really. There are no actual advantages to it. At all.

It looks shiny and new and some libraries use it in their documentation without any explanation, so people assume that it's the new thing that must be used. And then I end up having to explain to them why, unlike CommonJS, it doesn't actually work everywhere yet, and may never do so. For example, you can't import ESM modules from a CommonJS file! (Update: I've released a module that works around this issue.)

And then there's Rollup, which apparently requires ESM to be used, at least to get things like treeshaking. Which then makes people believe that treeshaking is not possible with CommonJS modules. Well, it is - Rollup just chose not to support it.

And then there's Babel, which tried to transpile import/export to require/module.exports, sidestepping the ongoing effort of standardizing the module semantics for ESM, causing broken imports and require("foo").default nonsense and spec design issues all over the place.

And then people go "but you can use ESM in browsers without a build step!", apparently not realizing that that is an utterly useless feature because loading a full dependency tree over the network would be unreasonably and unavoidably slow - you'd need as many roundtrips as there are levels of depth in your dependency tree - and so you need some kind of build step anyway, eliminating this entire supposed benefit.

And then people go "well you can statically analyze it better!", apparently not realizing that ESM doesn't actually change any of the JS semantics other than the import/export syntax, and that the import/export statements are equally analyzable as top-level require/module.exports.

"But in CommonJS you can use those elsewhere too, and that breaks static analyzers!", I hear you say. Well, yes, absolutely. But that is inherent in dynamic imports, which by the way, ESM also supports with its dynamic import() syntax. So it doesn't solve that either! Any static analyzer still needs to deal with the case of dynamic imports somehow - it's just rearranging deck chairs on the Titanic.

And then, people go "but now we at least have a standard module system!", apparently not realizing that CommonJS was literally that, the result of an attempt to standardize the various competing module systems in JS. Which, against all odds, actually succeeded!

... and then promptly got destroyed by ESM, which reintroduced a split and all sorts of incompatibility in the ecosystem, rather than just importing some updated variant of CommonJS into the language specification, which would have sidestepped almost all of these issues.

And while the initial CommonJS standardization effort succeeded due to none of the competing module systems being in particularly widespread use yet, CommonJS is so ubiquitous in Javascript-land nowadays that it will never fully go away. Which means that runtimes will forever have to keep supporting two module systems, and developers will forever be paying the cost of the interoperability issues between them.

But it's the future!

Is it really? The vast majority of people who believe they're currently using ESM, aren't even actually doing so - they're feeding their entire codebase through Babel, which deftly converts all of those snazzy import and export statements back into CommonJS syntax. Which works. So what's the point of the new module system again, if it all works with CommonJS anyway?

And it gets worse; import and export are designed as special-cased statements. Aside from the obvious problem of needing to learn a special syntax (which doesn't quite work like object destructuring) instead of reusing core language concepts, this is also a downgrade from CommonJS' require, which is a first-class expression due to just being a function call.

That might sound irrelevant on the face of it, but it has very real consequences. For example, the following pattern is simply not possible with ESM:

const someInitializedModule = require("module-name")(someOptions);

Or how about this one? Also no longer possible:

const app = express();
// ...
app.use("/users", require("./routers/users"));

Having language features available as a first-class expression is one of the most desirable properties in language design; yet for some completely unclear reason, ESM proponents decided to remove that property. There's just no way anymore to directly combine an import statement with some other JS syntax, whether or not the module path is statically specified.

The only way around this is with await import, which would break the supposed static analyzer benefits, only work in async contexts, and even then require weird hacks with parentheses to make it work correctly.

It also means that you now need to make a choice: do you want to be able to use ESM-only dependencies, or do you want to have access to patterns like the above that help you keep your codebase maintainable? ESM or maintainability, your choice!

So, congratulations, ESM proponents. You've destroyed a successful userland specification, wasted many (hundreds of?) thousands of hours of collective developer time, many hours of my own personal unpaid time trying to support people with the fallout, and created ecosystem fragmentation that will never go away, in exchange for... fuck all.

This is a disaster, and the only remaining way I see to fix it is to stop trying to make ESM happen, and deprecate it in favour of some variant of CommonJS modules being absorbed into the spec. It's not too late yet; but at some point it will be.

@Rush
Copy link

Rush commented Jan 17, 2024

I’ve already probably lost more hours to ESM than the code I work on would benefit from moving it. I could have been writing a feature, or making an optimisation, but instead I was wasting time on incompatibility issues. Truly awful.

relate to this so much. :) I don't understand why ESM is such a pain in the butt and why some module maintainers decided to shove it down our throats. I'm thinking of newer versions of node-fetch, for example, which cannot be directly used from a CJS workspace.

@guest271314
Copy link

@Rush You can use WHATWG fetch() in node. You don't need a library. I think the why is ECMA-262 specified Ecmascript Modules. esbuild can compile to CommonJS. bun can compile for the browser or to Ecmascript Modules. esm.sh does a decent job of exporting Node.js modules as Ecmascript Modules.

@Rush
Copy link

Rush commented Jan 18, 2024

@Rush You can use WHATWG fetch() in node. You don't need a library. I think the why is ECMA-262 specified Ecmascript Modules. esbuild can compile to CommonJS. bun can compile for the browser or to Ecmascript Modules. esm.sh does a decent job of exporting Node.js modules as Ecmascript Modules.

Thank you. Indeed there are a number of workarounds that can make things work, granted one spends a ton of hours on it.

@guest271314
Copy link

A modicum of effort. I managed to run the same code in Node.js, Deno, Bun. Next is running the same code in the browser, with node:fs replaced with WICG File System Access. No CommonJs, though essentially the same scenario, where different JavaScript/TypeScript runtimes might do things differently. The original code forked was written for Node.js, from my understanding that Node.js implemented Ed25519, where from what I gather node:crypto cannot be polyfilled. I substituted Web Cryptography API for Node.js crypto https://github.com/guest271314/wbn-sign-webcrypto and https://github.com/guest271314/webbundle for the ability to run the same common code in multiple JavaScript runtimes https://github.com/guest271314/telnet-client.

@Rush
Copy link

Rush commented Jan 18, 2024

I believe it varies. Managing libraries is generally less complex compared to overseeing multi-year projects that involve thousands of files, intricate business logic, and sophisticated build and deployment setups. Kudos to you for ensuring such remarkable flexibility in your own projects. By allowing multiple runtimes you clearly value & respect your users.

@guest271314
Copy link

By allowing multiple runtimes you clearly value & respect your users.

I did it for my own interests, not for users. I essentially had zero experience using Web Cryptography API, and didn't really have an interest in Signed Web Bundles or Isolated Web Apps. My interest was in Direct Sockets TCPSocket in the browser which Chromium authors decided to gate behind SWBN and IWA. I experiment with multiple JavaScript runtimes anyway. I'm just saying whether its 1 file or 1 million files, I don't think it matters, choosing sides and committing to one or both re CommonJS and EcmaScript Modules.

FYI Bun figured out a way to use require() and Ecmascript Modules in the same file.

@Rush
Copy link

Rush commented Jan 18, 2024

Yeah, I do believe Node could have picked a less "efficient" way but one that brings more compatibility. Now we're in the Python2/Python3 situation. It took years to resolve, if ever.

@guest271314
Copy link

Maybe, maybe not. I'm not tethered to Node.js. Try Bun. Try Deno. Try txiki.js. Test them all. The Bun folks have demonstrated quite a bit in a short span of time, particularly catering to people vested in Node.js. bun install, the package manager is built in to the single executable JavaScript runtime, gets rid of npm. bun build ./index.js --compile --outfile=bun_standalone just works on the first try https://gist.github.com/guest271314/9b1adad3db3deba64e118f844a77bad6. Node.js is still using CommonJS. Great for folks who use CommonJS, I suppose. In the mean time multiple other JavaScript engines, runtimes, interpreters https://gist.github.com/guest271314/bd292fc33e1b30dede0643a283fadc6a are doing their own thing irrespective of what happens in Node.js world.

@camj256
Copy link

camj256 commented Jan 18, 2024

I'm considering this a war within our language. As others have said forcing this in some repos has caused me to just not touch them. Sure we can move some things to work with others but at this point we just bring everything in house if we can. I built my codebase before es6 even existed, so having these modules really isn't fun for us unfortunately and we're quite a bit bigger than just 'fixing it'. I've all but left Node for other languages at this point unless its serving up websites. I have no problem with es6, but the esm modules are plain stupidity being forced on us. I'd rather have 2 different package managers if we are headed down this path. I'm tempted to start building out something like that myself to push this rift further apart out of spite.

@guest271314
Copy link

@Rush Python 2 has been "deprecated" for a while. I filed an issue for Google extensions to update their samples to Python 3 and fix their broken code. The Python 2 code is still there. No Python 3 code the last time I checked. MDN Web Docs merged the PR I filed.

@camj256
Copy link

camj256 commented Jan 18, 2024

Maybe, maybe not. I'm not tethered to Node.js. Try Bun. Try Deno. Try txiki.js. Test them all. The Bun folks have demonstrated quite a bit in a short span of time, particularly catering to people vested in Node.js. bun install, the package manager is built in to the single executable JavaScript runtime, gets rid of npm. bun build ./index.js --compile --outfile=bun_standalone just works on the first try https://gist.github.com/guest271314/9b1adad3db3deba64e118f844a77bad6. Node.js is still using CommonJS. Great for folks who use CommonJS, I suppose. In the mean time multiple other JavaScript engines, runtimes, interpreters https://gist.github.com/guest271314/bd292fc33e1b30dede0643a283fadc6a are doing their own thing irrespective of what happens in Node.js world.

I'm tethered to node. See my previous comment. Think that can help? We are rather large with thousands of files and require an extremely efficient stack.

@guest271314
Copy link

I'm considering this a war within our language.

Not at all. Not re JavaScript as specified by ECMA-262. CommonJS was not adopted as the module loader. Ecmascript Modules are. It's an internal Node.js maintainer/user issue.

I built my codebase before es6 even existed

Technology is not static. Moore's Law teaches us that.

I've all but left Node for other languages

Node is not a programming language.

I'm tempted to start building out something like that myself to push this rift further apart out of spite.

Do what you do.

@camj256
Copy link

camj256 commented Jan 18, 2024

Do we think there is a reason mongo 5 and 6 barely got support and 7 is EoL within a year? and 4.4.27 end next month for other reasons? Aside from them going public. I have endless amounts of S to talk.

@guest271314
Copy link

I'm tethered to node.

Not sure what to say.

I bought in to the Palm Pilot IPO. How many people are rolling around with Palm Pilots today? Even Blackberry's?

We are rather large with thousands of files and require an extremely efficient stack.

Well, if you are stuck in CommonJS world, not sure what to say. Except try using Bun, which caters heavily to Node.js API's and philosophies, and figured out how to run CommonJS and Ecmascript Modules in the same file - before Node.js maintainers.

@guest271314
Copy link

I'm not sure what you are talking about. All I know is I am going to use and exploit any and all JavaScript engines and runtimes to achieve my aims, by any means necessary.

esbuild should be able to compile all of your code to CommonJS. Or compile all of your code to Ecmascript Modules. Or use Bun and run them both.

For some Node.js API's there's nothing anybody can do except move on, e.g., crypto, which cannot be polyfilled as far as I know.

@camj256
Copy link

camj256 commented Jan 18, 2024

I'm tethered to node.

Not sure what to say.

I bought in to the Palm Pilot IPO. How many people are rolling around with Palm Pilots today? Even Blackberry's?

We are rather large with thousands of files and require an extremely efficient stack.

Well, if you are stuck in CommonJS world, not sure what to say. Except try using Bun, which caters heavily to Node.js API's and philosophies, and figured out how to run CommonJS and Ecmascript Modules in the same file - before Node.js maintainers.

We're probably around the same age and experience then. I could die on my bed fighting you on this, but I won't. I actually appreciate your responses.

You probably have me by a few years, but not much. I appreciate your response. I'm bitter because it was super easy for my team to work in for a long period of time. Yes, I know every team has to adapt to the times. It still feels like a split, especially lately. I have a huge codebase built in 2013-2014 on what worked amazing, and still does. Naturally I asked my team to code in the same voice the code was written in. Its getting harder and some of these libraries are pretty good these days. I know we can make them work still but we avoid them if possible for consistency and readability for newer team members. Its definitely not aging well...

I'm starting to feel like that's a losing battle; and that's fine, but it sucks. I understand wanting to write things as classes, and if I could, I'd rebuild it all in a different voice today. This is quite a change though IMO and integrating some of these modules isn't what I expected, or hoped for when designing my codebase 10+ years ago. (yeah, I know, even pre es6) I'll bitch till we definitely lose this argument and have to redesign our entire codebase and my CFO and CEO will certainly love that conversation as well. Gotta keep up with the times. In the mean time we're moving a lot of pipelines to python. Specifically anything not serving up web pages.

For what its worth I did learn a few things in the last few comments. So thank you, I'll research some of that further. I've had my head in other places, no excuses there.

At the moment though, I still F'ing hate the way things are headed. lol :)

@guest271314
Copy link

@camj256 I don't understand your apparent frustration. I really don't understand an idea of bitterness. The last time I check Node.js doesn't run the world, technology is not static, time keeps on ticking. Ever try esbuild? There's a --format=cjs option...

@Rush
Copy link

Rush commented Jan 18, 2024

In the mean time we're moving a lot of pipelines to python

Ouch!! That's one hell in your face argument. :-)

@Rush Python 2 has been "deprecated" for a while.

Just because things are called "deprecated" does not mean they do not cause issues :) I think Python2/Python3 divide is mostly over but I remember wasting countless hours on this.

@divmgl
Copy link

divmgl commented Jan 18, 2024

@camj256 I don't understand your apparent frustration. I really don't understand an idea of bitterness. The last time I check Node.js doesn't run the world, technology is not static, time keeps on ticking. Ever try esbuild? There's a --format=cjs option...

This is disingenuous. ESBuild is a community built tool. Most of the complaints here are re: the underlying system that exists in Node.js without any additional tooling. There's no reason why the transition from CommonJS to ES Modules had to be so painful. I'd like to think that this was one of the many catalysts for Bun's existence.

I actually landed on an approach to solving a majority of these ES Modules issues with Node.js, TypeScript and ES Modules and that's to just use ESBuild, as you recommended. I had to build my own scripts to basically take the packages in my monorepo and convert them to valid ESBuild packages (this includes server-side packages).

Thankfully there's now an excellent tool that uses ESBuild under the hood to solve a lot of these pain points: https://github.com/privatenumber/tsx I'm super grateful that a tool like this exists (and works basically flawlessly out of the box) but all of this could have been avoided with a better migration path.

@guest271314
Copy link

@divmgl

This is disingenuous.

No. That's my genuine opinion.

Most of the complaints here are re: the underlying system that exists in Node.js without any additional tooling.

There's nothing you can do about what Node.js maintainers decide to do in the nodejs main branch. You can fork node and do whatever you want.

Yes, I have read the problem statement.

Programmers solve problems.

@guest271314
Copy link

@Rush From [Native Messaging] Python 2 was sunset January 1st, 2020 #489

https://www.python.org/doc/sunset-python-2/

Sunsetting Python 2

We are volunteers who make and take care of the Python programming language. We have decided that January 1, 2020, was the day that we sunset Python 2. That means that we will not improve it anymore after that day, even if someone finds a security problem in it. You should upgrade to Python 3 as soon as you can.

I advised the GoogleChrome/chrome-extensions-samples folks their code was broken here GoogleChrome/chrome-extensions-samples#805 and the MDN Web Docs folks their code was broken here mdn/webextensions-examples#509.

GoogleChrome folks had banned me by then. Then closed the issue. They never fixed their broken Python 2 code https://github.com/GoogleChrome/chrome-extensions-samples/blob/main/_archive/mv2/api/nativeMessaging/host/native-messaging-example-host that doesn't work on Linux anyway because there in general is no /usr/bin/python, there is a /usr/bin/python3.

MDN Web Docs also wound up mergin a working Node.js Native Messaging host example mdn/content#27228. GoogleChrome samples is still stuck in 2013.

@mk-pmb
Copy link

mk-pmb commented Jan 19, 2024

Just my default rant about JS "classes" (click to open)

@camj256

I understand wanting to write things as classes

Understanding is ok, I just hope you don't agree. Rather, teach your folks why the class-less object-centric approach is (can be proven to be) mightier than class-centric approach. Hijacking the term "Object-oriented Programming" for the latter was a genious but evil marketing move from false prophets trying to lead all mankind astray.
Last time I checked, ECMAScript still doesn't have classes, and that's a good thing. Recent editions added some ways to abuse the class keyword and a terminology of "class-like" somethings. It's a tradition for JS to introduce footguns. Firing them is entirely optional, and best left to fools.

@camj256
Copy link

camj256 commented Jan 19, 2024

@mk-pmb Yeah exactly. However, there are enough people in this community that don't agree anymore. ...and thats a problem. (clearly, as demonstrated by this thread) It's making things sloppy and less compatible. Yeah, the language is only hurting itself trying to make changes like this at this point.

Our codebase is easy to read, maintain, and is lightning fast for how large it is. For how well its designed actually I don't really want to hack together builds either.

@leonsilicon
Copy link

leonsilicon commented Apr 26, 2024

Update: Node.js (v22, and hopefully v20) now supports requiring (most) ESM code from CommonJS (nodejs/node#51977 (comment); @joyeecheung wrote an insightful post about require(esm) worth reading), so it turns out that the ESM spec wasn't actually inherently incompatible with CommonJS, but rather just due to how Node.js implemented it. So this gist (and all the incompatibility-related criticism towards the ESM spec itself) can now be considered outdated :)

@dboreham
Copy link

Ended up here after a hair-pulling fest on one of our projects.
So...JS people have managed to re-invent Python3.
In about 10 years everything will be fine and all will be forgotten.

@Rush
Copy link

Rush commented May 17, 2024

Since we're ranting, does anyone else have issues with node-gyp? Why on earth is it still using Python? Recently, I had to lock my Python version to 3.8.19 because my project won't build with newer versions. This is all due to older dependencies that rely on an outdated version of node-gyp. It's incredibly frustrating!

@JamesAlp
Copy link

Update: Node.js (v22, and hopefully v20) now supports requiring (most) ESM code from CommonJS (nodejs/node#51977 (comment); @joyeecheung wrote an insightful post about require(esm) worth reading), so it turns out that the ESM spec wasn't actually inherently incompatible with CommonJS, but rather just due to how Node.js implemented it. So this gist (and all the incompatibility-related criticism towards the ESM spec itself) can now be considered outdated :)

"Most" is definitely correct lol. For example if you wish to use Chalk, you must install 4.1.2 or earlier in your CJS project. Trying to convert your NestJS server to ESM instead of just keeping it CJS introduces so very many problems that just should not be problems. All the different imports now needing to use file extensions but only on some imports, needing to modify your tsconfig and package files with specific settings, a host of other changes and, surprise, sometimes the development build just cannot locate certain files at runtime and everything breaks! But other times when you start it up again it'll just work for that import but fail on another. I'm sure there is a solution to this... but for me the ultimate solution is to just skip all of this nonsense, stick with CJS and just install an older version of a package if I really want it.

@leonsilicon
Copy link

leonsilicon commented Jun 14, 2024

"Most" is definitely correct lol.

Most meaning ESM code that doesn't use TLA (top-level await), which is the vast majority (chalk does not use TLA)

Also that seems like a NestJS problem, not an ESM one (i.e. it's NestJS's responsibility to support ESM - it doesn't seem fair to criticize the ESM spec itself because NestJS doesn't support it well)

@guest271314
Copy link

@ClickClickDerk
Copy link

IDK this might be old but i learned js5 like 12 years ago and took a break the past like 5 years just coding little projects for myself. I'm here today cause i dont understand modules and find them to be dumb. you can break up you js code to files that pages depend on it. I also taught myself to code so i went through one hell of a learning curve. i learned html, css, then js (client-side) before ever looking at how a server side language works and i remember because js is a loosely type language it made learning php or python more challanging and vice versa for the guys that knew strict server side programming didnt understand js to well. looking at it now es6 looks like the server side guys are turning the loosely typed language into be more readable as they understand how server side works. looking at all these js files and how AI is programming im like not every file has to be 1-2kb or less and have a ton of modules. i wrote free movies site back in the day must of had over 200,000 lines of js and it would load a bit slow on a phone but a desktop or laptop no problem. its 10 years later and everything is 100x faster. break up your files, lazy load, use ajax and you code is blazing fast today that way but now im hitting walls left and right with modules. if babel is really just turning es6 into older js syntax then that doesnt make sense. i look at babel as owning everyones code and going to strong arm people into making js and babel the standard. eww javascript is a beautiful language and es6 is ruining it. imo lol

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment