Skip to content

Instantly share code, notes, and snippets.

@joepie91
Last active April 14, 2024 04:25
Show Gist options
  • Star 191 You must be signed in to star a gist
  • Fork 4 You must be signed in to fork a gist
  • Save joepie91/bca2fda868c1e8b2c2caf76af7dfcad3 to your computer and use it in GitHub Desktop.
Save joepie91/bca2fda868c1e8b2c2caf76af7dfcad3 to your computer and use it in GitHub Desktop.
ES Modules are terrible, actually

ES Modules are terrible, actually

This post was adapted from an earlier Twitter thread.

It's incredible how many collective developer hours have been wasted on pushing through the turd that is ES Modules (often mistakenly called "ES6 Modules"). Causing a big ecosystem divide and massive tooling support issues, for... well, no reason, really. There are no actual advantages to it. At all.

It looks shiny and new and some libraries use it in their documentation without any explanation, so people assume that it's the new thing that must be used. And then I end up having to explain to them why, unlike CommonJS, it doesn't actually work everywhere yet, and may never do so. For example, you can't import ESM modules from a CommonJS file! (Update: I've released a module that works around this issue.)

And then there's Rollup, which apparently requires ESM to be used, at least to get things like treeshaking. Which then makes people believe that treeshaking is not possible with CommonJS modules. Well, it is - Rollup just chose not to support it.

And then there's Babel, which tried to transpile import/export to require/module.exports, sidestepping the ongoing effort of standardizing the module semantics for ESM, causing broken imports and require("foo").default nonsense and spec design issues all over the place.

And then people go "but you can use ESM in browsers without a build step!", apparently not realizing that that is an utterly useless feature because loading a full dependency tree over the network would be unreasonably and unavoidably slow - you'd need as many roundtrips as there are levels of depth in your dependency tree - and so you need some kind of build step anyway, eliminating this entire supposed benefit.

And then people go "well you can statically analyze it better!", apparently not realizing that ESM doesn't actually change any of the JS semantics other than the import/export syntax, and that the import/export statements are equally analyzable as top-level require/module.exports.

"But in CommonJS you can use those elsewhere too, and that breaks static analyzers!", I hear you say. Well, yes, absolutely. But that is inherent in dynamic imports, which by the way, ESM also supports with its dynamic import() syntax. So it doesn't solve that either! Any static analyzer still needs to deal with the case of dynamic imports somehow - it's just rearranging deck chairs on the Titanic.

And then, people go "but now we at least have a standard module system!", apparently not realizing that CommonJS was literally that, the result of an attempt to standardize the various competing module systems in JS. Which, against all odds, actually succeeded!

... and then promptly got destroyed by ESM, which reintroduced a split and all sorts of incompatibility in the ecosystem, rather than just importing some updated variant of CommonJS into the language specification, which would have sidestepped almost all of these issues.

And while the initial CommonJS standardization effort succeeded due to none of the competing module systems being in particularly widespread use yet, CommonJS is so ubiquitous in Javascript-land nowadays that it will never fully go away. Which means that runtimes will forever have to keep supporting two module systems, and developers will forever be paying the cost of the interoperability issues between them.

But it's the future!

Is it really? The vast majority of people who believe they're currently using ESM, aren't even actually doing so - they're feeding their entire codebase through Babel, which deftly converts all of those snazzy import and export statements back into CommonJS syntax. Which works. So what's the point of the new module system again, if it all works with CommonJS anyway?

And it gets worse; import and export are designed as special-cased statements. Aside from the obvious problem of needing to learn a special syntax (which doesn't quite work like object destructuring) instead of reusing core language concepts, this is also a downgrade from CommonJS' require, which is a first-class expression due to just being a function call.

That might sound irrelevant on the face of it, but it has very real consequences. For example, the following pattern is simply not possible with ESM:

const someInitializedModule = require("module-name")(someOptions);

Or how about this one? Also no longer possible:

const app = express();
// ...
app.use("/users", require("./routers/users"));

Having language features available as a first-class expression is one of the most desirable properties in language design; yet for some completely unclear reason, ESM proponents decided to remove that property. There's just no way anymore to directly combine an import statement with some other JS syntax, whether or not the module path is statically specified.

The only way around this is with await import, which would break the supposed static analyzer benefits, only work in async contexts, and even then require weird hacks with parentheses to make it work correctly.

It also means that you now need to make a choice: do you want to be able to use ESM-only dependencies, or do you want to have access to patterns like the above that help you keep your codebase maintainable? ESM or maintainability, your choice!

So, congratulations, ESM proponents. You've destroyed a successful userland specification, wasted many (hundreds of?) thousands of hours of collective developer time, many hours of my own personal unpaid time trying to support people with the fallout, and created ecosystem fragmentation that will never go away, in exchange for... fuck all.

This is a disaster, and the only remaining way I see to fix it is to stop trying to make ESM happen, and deprecate it in favour of some variant of CommonJS modules being absorbed into the spec. It's not too late yet; but at some point it will be.

@mk-pmb
Copy link

mk-pmb commented Nov 21, 2023

Hahaha, just 4 days lost? I'm quite envious of you. :D
Anyway, the dynamic import should actually have worked. You have to await the results and use them, in the unlikely case that you missed that detail. Also sometimes you need to unpack the default export.

As a package author you can declare multiple exports for your package. See this PR where I helped them expose package.json.
Everyone please always expose package.json in your ESM modules!

Also shameless plug: If you encounter such ESM problems again, try the crutch I made for myself: nodemjs. Maybe it works for you.

@jimmywarting
Copy link

jimmywarting commented Nov 21, 2023

if you are so sure about using 'import' from common js then send some code samples - maybe it will help someone else.

Sure @cybafelo, i can give you 3 example (from easy to more adv)

I created a basic hello world example that uses express (cjs) and node-fetch (esm-only)

you can read more about the solutions over here at this gist of how you can use both cjs and esm together

@guest271314
Copy link

@ehaynes99

I think I'm getting the gist of what you are talking about here.

There was a general push by the JavaScript stakeholders in Node.js, a package management system was created, delivered. And then the rug was pulled from under consumers and a switch was made that is incompatible with the original push.

I tried a dose of my own medicine with a little-bitty repository on GitHub. Errors thrown using jsDelivr's ESM URL talking about this or that export doesn't have a default export. I don't think esbuild or rollup cleanly convert from CJS to ESM. Seems like there should have been at least a couple ways tried and tested to do that, both ways, before shipping ESM. OTOH, CJS via npm is completely dependent on node. There ain't no standalone npm eecutable like every other sane package management system, e.g., aptitude.

@Aeolun
Copy link

Aeolun commented Dec 6, 2023

Every few years I come back to this same Gist... It's just absurd how long this absurd migration has continued.

@camj256
Copy link

camj256 commented Dec 25, 2023

To this day couldn't agree more. Been coding in Node since 2014 full time. Its still a mess IMO. Try and get people to code in the original structure of how whatever app we're working on was built but its becoming more of a PITA every year.

@michaeljnash
Copy link

michaeljnash commented Dec 28, 2023

You have some points wrong/mixed up about load times and dependency trees affecting load times to the extent you think they do.

ES Modules, when used in modern browsers, are designed to take advantage of certain features like HTTP/2 and the browser's ability to asynchronously load resources. HTTP/2, compared to its predecessor HTTP/1.1, allows for more efficient multiplexing, header compression, and server push, among other features, which can significantly improve loading times for multiple resources, including ES Modules.

When considering dependency trees and loading times, ES Modules do support asynchronous loading, allowing browsers to fetch modules in parallel and mitigate the impact of long dependency trees. This asynchronous loading behavior helps in improving the overall performance, especially when dealing with complex applications that have a significant number of modules or dependencies.

@guest271314
Copy link

I'm curious what you folks think about deno throwing module not found error for dynamic import() with raw string specifier denoland/deno#20945, https://gist.github.com/guest271314/4637bb1288321256d7c14c72ebc81137

@Rush
Copy link

Rush commented Jan 17, 2024

I’ve already probably lost more hours to ESM than the code I work on would benefit from moving it. I could have been writing a feature, or making an optimisation, but instead I was wasting time on incompatibility issues. Truly awful.

relate to this so much. :) I don't understand why ESM is such a pain in the butt and why some module maintainers decided to shove it down our throats. I'm thinking of newer versions of node-fetch, for example, which cannot be directly used from a CJS workspace.

@guest271314
Copy link

@Rush You can use WHATWG fetch() in node. You don't need a library. I think the why is ECMA-262 specified Ecmascript Modules. esbuild can compile to CommonJS. bun can compile for the browser or to Ecmascript Modules. esm.sh does a decent job of exporting Node.js modules as Ecmascript Modules.

@Rush
Copy link

Rush commented Jan 18, 2024

@Rush You can use WHATWG fetch() in node. You don't need a library. I think the why is ECMA-262 specified Ecmascript Modules. esbuild can compile to CommonJS. bun can compile for the browser or to Ecmascript Modules. esm.sh does a decent job of exporting Node.js modules as Ecmascript Modules.

Thank you. Indeed there are a number of workarounds that can make things work, granted one spends a ton of hours on it.

@guest271314
Copy link

A modicum of effort. I managed to run the same code in Node.js, Deno, Bun. Next is running the same code in the browser, with node:fs replaced with WICG File System Access. No CommonJs, though essentially the same scenario, where different JavaScript/TypeScript runtimes might do things differently. The original code forked was written for Node.js, from my understanding that Node.js implemented Ed25519, where from what I gather node:crypto cannot be polyfilled. I substituted Web Cryptography API for Node.js crypto https://github.com/guest271314/wbn-sign-webcrypto and https://github.com/guest271314/webbundle for the ability to run the same common code in multiple JavaScript runtimes https://github.com/guest271314/telnet-client.

@Rush
Copy link

Rush commented Jan 18, 2024

I believe it varies. Managing libraries is generally less complex compared to overseeing multi-year projects that involve thousands of files, intricate business logic, and sophisticated build and deployment setups. Kudos to you for ensuring such remarkable flexibility in your own projects. By allowing multiple runtimes you clearly value & respect your users.

@guest271314
Copy link

By allowing multiple runtimes you clearly value & respect your users.

I did it for my own interests, not for users. I essentially had zero experience using Web Cryptography API, and didn't really have an interest in Signed Web Bundles or Isolated Web Apps. My interest was in Direct Sockets TCPSocket in the browser which Chromium authors decided to gate behind SWBN and IWA. I experiment with multiple JavaScript runtimes anyway. I'm just saying whether its 1 file or 1 million files, I don't think it matters, choosing sides and committing to one or both re CommonJS and EcmaScript Modules.

FYI Bun figured out a way to use require() and Ecmascript Modules in the same file.

@Rush
Copy link

Rush commented Jan 18, 2024

Yeah, I do believe Node could have picked a less "efficient" way but one that brings more compatibility. Now we're in the Python2/Python3 situation. It took years to resolve, if ever.

@guest271314
Copy link

Maybe, maybe not. I'm not tethered to Node.js. Try Bun. Try Deno. Try txiki.js. Test them all. The Bun folks have demonstrated quite a bit in a short span of time, particularly catering to people vested in Node.js. bun install, the package manager is built in to the single executable JavaScript runtime, gets rid of npm. bun build ./index.js --compile --outfile=bun_standalone just works on the first try https://gist.github.com/guest271314/9b1adad3db3deba64e118f844a77bad6. Node.js is still using CommonJS. Great for folks who use CommonJS, I suppose. In the mean time multiple other JavaScript engines, runtimes, interpreters https://gist.github.com/guest271314/bd292fc33e1b30dede0643a283fadc6a are doing their own thing irrespective of what happens in Node.js world.

@camj256
Copy link

camj256 commented Jan 18, 2024

I'm considering this a war within our language. As others have said forcing this in some repos has caused me to just not touch them. Sure we can move some things to work with others but at this point we just bring everything in house if we can. I built my codebase before es6 even existed, so having these modules really isn't fun for us unfortunately and we're quite a bit bigger than just 'fixing it'. I've all but left Node for other languages at this point unless its serving up websites. I have no problem with es6, but the esm modules are plain stupidity being forced on us. I'd rather have 2 different package managers if we are headed down this path. I'm tempted to start building out something like that myself to push this rift further apart out of spite.

@guest271314
Copy link

@Rush Python 2 has been "deprecated" for a while. I filed an issue for Google extensions to update their samples to Python 3 and fix their broken code. The Python 2 code is still there. No Python 3 code the last time I checked. MDN Web Docs merged the PR I filed.

@camj256
Copy link

camj256 commented Jan 18, 2024

Maybe, maybe not. I'm not tethered to Node.js. Try Bun. Try Deno. Try txiki.js. Test them all. The Bun folks have demonstrated quite a bit in a short span of time, particularly catering to people vested in Node.js. bun install, the package manager is built in to the single executable JavaScript runtime, gets rid of npm. bun build ./index.js --compile --outfile=bun_standalone just works on the first try https://gist.github.com/guest271314/9b1adad3db3deba64e118f844a77bad6. Node.js is still using CommonJS. Great for folks who use CommonJS, I suppose. In the mean time multiple other JavaScript engines, runtimes, interpreters https://gist.github.com/guest271314/bd292fc33e1b30dede0643a283fadc6a are doing their own thing irrespective of what happens in Node.js world.

I'm tethered to node. See my previous comment. Think that can help? We are rather large with thousands of files and require an extremely efficient stack.

@guest271314
Copy link

I'm considering this a war within our language.

Not at all. Not re JavaScript as specified by ECMA-262. CommonJS was not adopted as the module loader. Ecmascript Modules are. It's an internal Node.js maintainer/user issue.

I built my codebase before es6 even existed

Technology is not static. Moore's Law teaches us that.

I've all but left Node for other languages

Node is not a programming language.

I'm tempted to start building out something like that myself to push this rift further apart out of spite.

Do what you do.

@camj256
Copy link

camj256 commented Jan 18, 2024

Do we think there is a reason mongo 5 and 6 barely got support and 7 is EoL within a year? and 4.4.27 end next month for other reasons? Aside from them going public. I have endless amounts of S to talk.

@guest271314
Copy link

I'm tethered to node.

Not sure what to say.

I bought in to the Palm Pilot IPO. How many people are rolling around with Palm Pilots today? Even Blackberry's?

We are rather large with thousands of files and require an extremely efficient stack.

Well, if you are stuck in CommonJS world, not sure what to say. Except try using Bun, which caters heavily to Node.js API's and philosophies, and figured out how to run CommonJS and Ecmascript Modules in the same file - before Node.js maintainers.

@guest271314
Copy link

I'm not sure what you are talking about. All I know is I am going to use and exploit any and all JavaScript engines and runtimes to achieve my aims, by any means necessary.

esbuild should be able to compile all of your code to CommonJS. Or compile all of your code to Ecmascript Modules. Or use Bun and run them both.

For some Node.js API's there's nothing anybody can do except move on, e.g., crypto, which cannot be polyfilled as far as I know.

@camj256
Copy link

camj256 commented Jan 18, 2024

I'm tethered to node.

Not sure what to say.

I bought in to the Palm Pilot IPO. How many people are rolling around with Palm Pilots today? Even Blackberry's?

We are rather large with thousands of files and require an extremely efficient stack.

Well, if you are stuck in CommonJS world, not sure what to say. Except try using Bun, which caters heavily to Node.js API's and philosophies, and figured out how to run CommonJS and Ecmascript Modules in the same file - before Node.js maintainers.

We're probably around the same age and experience then. I could die on my bed fighting you on this, but I won't. I actually appreciate your responses.

You probably have me by a few years, but not much. I appreciate your response. I'm bitter because it was super easy for my team to work in for a long period of time. Yes, I know every team has to adapt to the times. It still feels like a split, especially lately. I have a huge codebase built in 2013-2014 on what worked amazing, and still does. Naturally I asked my team to code in the same voice the code was written in. Its getting harder and some of these libraries are pretty good these days. I know we can make them work still but we avoid them if possible for consistency and readability for newer team members. Its definitely not aging well...

I'm starting to feel like that's a losing battle; and that's fine, but it sucks. I understand wanting to write things as classes, and if I could, I'd rebuild it all in a different voice today. This is quite a change though IMO and integrating some of these modules isn't what I expected, or hoped for when designing my codebase 10+ years ago. (yeah, I know, even pre es6) I'll bitch till we definitely lose this argument and have to redesign our entire codebase and my CFO and CEO will certainly love that conversation as well. Gotta keep up with the times. In the mean time we're moving a lot of pipelines to python. Specifically anything not serving up web pages.

For what its worth I did learn a few things in the last few comments. So thank you, I'll research some of that further. I've had my head in other places, no excuses there.

At the moment though, I still F'ing hate the way things are headed. lol :)

@guest271314
Copy link

@camj256 I don't understand your apparent frustration. I really don't understand an idea of bitterness. The last time I check Node.js doesn't run the world, technology is not static, time keeps on ticking. Ever try esbuild? There's a --format=cjs option...

@Rush
Copy link

Rush commented Jan 18, 2024

In the mean time we're moving a lot of pipelines to python

Ouch!! That's one hell in your face argument. :-)

@Rush Python 2 has been "deprecated" for a while.

Just because things are called "deprecated" does not mean they do not cause issues :) I think Python2/Python3 divide is mostly over but I remember wasting countless hours on this.

@divmgl
Copy link

divmgl commented Jan 18, 2024

@camj256 I don't understand your apparent frustration. I really don't understand an idea of bitterness. The last time I check Node.js doesn't run the world, technology is not static, time keeps on ticking. Ever try esbuild? There's a --format=cjs option...

This is disingenuous. ESBuild is a community built tool. Most of the complaints here are re: the underlying system that exists in Node.js without any additional tooling. There's no reason why the transition from CommonJS to ES Modules had to be so painful. I'd like to think that this was one of the many catalysts for Bun's existence.

I actually landed on an approach to solving a majority of these ES Modules issues with Node.js, TypeScript and ES Modules and that's to just use ESBuild, as you recommended. I had to build my own scripts to basically take the packages in my monorepo and convert them to valid ESBuild packages (this includes server-side packages).

Thankfully there's now an excellent tool that uses ESBuild under the hood to solve a lot of these pain points: https://github.com/privatenumber/tsx I'm super grateful that a tool like this exists (and works basically flawlessly out of the box) but all of this could have been avoided with a better migration path.

@guest271314
Copy link

@divmgl

This is disingenuous.

No. That's my genuine opinion.

Most of the complaints here are re: the underlying system that exists in Node.js without any additional tooling.

There's nothing you can do about what Node.js maintainers decide to do in the nodejs main branch. You can fork node and do whatever you want.

Yes, I have read the problem statement.

Programmers solve problems.

@guest271314
Copy link

@Rush From [Native Messaging] Python 2 was sunset January 1st, 2020 #489

https://www.python.org/doc/sunset-python-2/

Sunsetting Python 2

We are volunteers who make and take care of the Python programming language. We have decided that January 1, 2020, was the day that we sunset Python 2. That means that we will not improve it anymore after that day, even if someone finds a security problem in it. You should upgrade to Python 3 as soon as you can.

I advised the GoogleChrome/chrome-extensions-samples folks their code was broken here GoogleChrome/chrome-extensions-samples#805 and the MDN Web Docs folks their code was broken here mdn/webextensions-examples#509.

GoogleChrome folks had banned me by then. Then closed the issue. They never fixed their broken Python 2 code https://github.com/GoogleChrome/chrome-extensions-samples/blob/main/_archive/mv2/api/nativeMessaging/host/native-messaging-example-host that doesn't work on Linux anyway because there in general is no /usr/bin/python, there is a /usr/bin/python3.

MDN Web Docs also wound up mergin a working Node.js Native Messaging host example mdn/content#27228. GoogleChrome samples is still stuck in 2013.

@mk-pmb
Copy link

mk-pmb commented Jan 19, 2024

Just my default rant about JS "classes" (click to open)

@camj256

I understand wanting to write things as classes

Understanding is ok, I just hope you don't agree. Rather, teach your folks why the class-less object-centric approach is (can be proven to be) mightier than class-centric approach. Hijacking the term "Object-oriented Programming" for the latter was a genious but evil marketing move from false prophets trying to lead all mankind astray.
Last time I checked, ECMAScript still doesn't have classes, and that's a good thing. Recent editions added some ways to abuse the class keyword and a terminology of "class-like" somethings. It's a tradition for JS to introduce footguns. Firing them is entirely optional, and best left to fools.

@camj256
Copy link

camj256 commented Jan 19, 2024

@mk-pmb Yeah exactly. However, there are enough people in this community that don't agree anymore. ...and thats a problem. (clearly, as demonstrated by this thread) It's making things sloppy and less compatible. Yeah, the language is only hurting itself trying to make changes like this at this point.

Our codebase is easy to read, maintain, and is lightning fast for how large it is. For how well its designed actually I don't really want to hack together builds either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment