Skip to content

Instantly share code, notes, and snippets.

@joepie91
Last active March 25, 2024 12:17
Show Gist options
  • Star 190 You must be signed in to star a gist
  • Fork 4 You must be signed in to fork a gist
  • Save joepie91/bca2fda868c1e8b2c2caf76af7dfcad3 to your computer and use it in GitHub Desktop.
Save joepie91/bca2fda868c1e8b2c2caf76af7dfcad3 to your computer and use it in GitHub Desktop.
ES Modules are terrible, actually

ES Modules are terrible, actually

This post was adapted from an earlier Twitter thread.

It's incredible how many collective developer hours have been wasted on pushing through the turd that is ES Modules (often mistakenly called "ES6 Modules"). Causing a big ecosystem divide and massive tooling support issues, for... well, no reason, really. There are no actual advantages to it. At all.

It looks shiny and new and some libraries use it in their documentation without any explanation, so people assume that it's the new thing that must be used. And then I end up having to explain to them why, unlike CommonJS, it doesn't actually work everywhere yet, and may never do so. For example, you can't import ESM modules from a CommonJS file! (Update: I've released a module that works around this issue.)

And then there's Rollup, which apparently requires ESM to be used, at least to get things like treeshaking. Which then makes people believe that treeshaking is not possible with CommonJS modules. Well, it is - Rollup just chose not to support it.

And then there's Babel, which tried to transpile import/export to require/module.exports, sidestepping the ongoing effort of standardizing the module semantics for ESM, causing broken imports and require("foo").default nonsense and spec design issues all over the place.

And then people go "but you can use ESM in browsers without a build step!", apparently not realizing that that is an utterly useless feature because loading a full dependency tree over the network would be unreasonably and unavoidably slow - you'd need as many roundtrips as there are levels of depth in your dependency tree - and so you need some kind of build step anyway, eliminating this entire supposed benefit.

And then people go "well you can statically analyze it better!", apparently not realizing that ESM doesn't actually change any of the JS semantics other than the import/export syntax, and that the import/export statements are equally analyzable as top-level require/module.exports.

"But in CommonJS you can use those elsewhere too, and that breaks static analyzers!", I hear you say. Well, yes, absolutely. But that is inherent in dynamic imports, which by the way, ESM also supports with its dynamic import() syntax. So it doesn't solve that either! Any static analyzer still needs to deal with the case of dynamic imports somehow - it's just rearranging deck chairs on the Titanic.

And then, people go "but now we at least have a standard module system!", apparently not realizing that CommonJS was literally that, the result of an attempt to standardize the various competing module systems in JS. Which, against all odds, actually succeeded!

... and then promptly got destroyed by ESM, which reintroduced a split and all sorts of incompatibility in the ecosystem, rather than just importing some updated variant of CommonJS into the language specification, which would have sidestepped almost all of these issues.

And while the initial CommonJS standardization effort succeeded due to none of the competing module systems being in particularly widespread use yet, CommonJS is so ubiquitous in Javascript-land nowadays that it will never fully go away. Which means that runtimes will forever have to keep supporting two module systems, and developers will forever be paying the cost of the interoperability issues between them.

But it's the future!

Is it really? The vast majority of people who believe they're currently using ESM, aren't even actually doing so - they're feeding their entire codebase through Babel, which deftly converts all of those snazzy import and export statements back into CommonJS syntax. Which works. So what's the point of the new module system again, if it all works with CommonJS anyway?

And it gets worse; import and export are designed as special-cased statements. Aside from the obvious problem of needing to learn a special syntax (which doesn't quite work like object destructuring) instead of reusing core language concepts, this is also a downgrade from CommonJS' require, which is a first-class expression due to just being a function call.

That might sound irrelevant on the face of it, but it has very real consequences. For example, the following pattern is simply not possible with ESM:

const someInitializedModule = require("module-name")(someOptions);

Or how about this one? Also no longer possible:

const app = express();
// ...
app.use("/users", require("./routers/users"));

Having language features available as a first-class expression is one of the most desirable properties in language design; yet for some completely unclear reason, ESM proponents decided to remove that property. There's just no way anymore to directly combine an import statement with some other JS syntax, whether or not the module path is statically specified.

The only way around this is with await import, which would break the supposed static analyzer benefits, only work in async contexts, and even then require weird hacks with parentheses to make it work correctly.

It also means that you now need to make a choice: do you want to be able to use ESM-only dependencies, or do you want to have access to patterns like the above that help you keep your codebase maintainable? ESM or maintainability, your choice!

So, congratulations, ESM proponents. You've destroyed a successful userland specification, wasted many (hundreds of?) thousands of hours of collective developer time, many hours of my own personal unpaid time trying to support people with the fallout, and created ecosystem fragmentation that will never go away, in exchange for... fuck all.

This is a disaster, and the only remaining way I see to fix it is to stop trying to make ESM happen, and deprecate it in favour of some variant of CommonJS modules being absorbed into the spec. It's not too late yet; but at some point it will be.

@joepie91
Copy link
Author

joepie91 commented May 24, 2023

Like I said:

It's only true in the absolute most technical pedantic sense, and completely false in any way that anybody actually cares about.

Pipelining doesn't help. Concurrent connections don't help. If you have a dependency graph of 4 levels, and you are loading modules directly over the network, that means that you don't know what the next level of dependencies is going to be until you've parsed the previous level - in other words, there's a serial dependency chain. That means that there are 4 serial network roundtrips to be made.

This is impossible to optimize without some kind of build step - the browser simply cannot act on information that it does not already have. Every solution for this that involves precomputing the dependency graph (bundling, generating a manifest, whatever), will be functionally equivalent to bundling.

This is why modules can never work directly in a browser (at least, over the internet). It has nothing to do with which module system you use - it's a fundamental limitation of physics, of how networks work. ESM cannot break the laws of physics either.

@leonsilicon
Copy link

leonsilicon commented May 24, 2023

@joepie91

The browser and the internet are two separate things. The primary goal of ESM is to not to run module-based JavaScript on an internet-facing production website, but rather to support running module-based JavaScript in a browser environment (which supports running JavaScript without an internet connection, similar to how Node.js is an environment for running your JavaScript). The browser is becoming an increasingly integrated part of a developer's workflow/toolkit, especially with projects like StackBlitz and technologies like WebContainers on the rise.

Requiring an expensive bundler step in order to run developers' JavaScript code on a browser is a pretty significant limitation that hinders the ability for these tools to provide an optimal developer experience for JavaScript developers. We saw Vite leverage ESM support in browsers to provide a much more responsive and painless developer experience compared to webpack, and it's a huge part of what makes innovations like StackBlitz (a fully web-based IDE) possible.

The huge benefit of ESM is ultimately about developer experience, not about production website performance. In development, latency/multiple network calls are not an issue because everything is local, but CPU/computational power is. The problem of running JavaScript on a production website was already mostly solved with bundlers like webpack, but there was still a lot to be desired from running JavaScript in development, which is where ESM really shines.

@joepie91
Copy link
Author

The browser and the internet are two separate things. The primary goal of ESM is to not to run module-based JavaScript on an internet-facing production website, but rather to support running module-based JavaScript in a browser environment (which supports running JavaScript without an internet connection, similar to how Node.js is an environment for running your JavaScript). The browser is becoming an increasingly integrated part of a developer's workflow/toolkit, especially with projects like StackBlitz and technologies like WebContainers on the rise.

This is historical revisionism. ESM was absolutely originally pitched as a way to use modules on public-facing websites without a bundler. That the goalposts are now being moved now that that turns out non-viable doesn't change that.

Requiring an expensive bundler step in order to run developers' JavaScript code on a browser is a pretty significant limitation that hinders the ability for these tools to provide an optimal developer experience for JavaScript developers.

Bundlers have provided sub-100ms reloads for a decade already. This was a non-issue to begin with.

We saw Vite leverage ESM support in browsers to provide a much more responsive and painless developer experience compared to webpack,

Yes, compared to Webpack, which famously is very difficult to configure correctly, resulting in most people running it in absurd configurations that take forever to produce a new (dev) build. Have you ever tried running a 'clean' configuration that only does the bundling part? Because I have, and you'll get full builds under 500ms, and partial builds under 50ms.

Somehow, Browserify never had this issue to begin with - perhaps because it was actually viable to configure by hand, rather than copy-pasting mystery configuration snippets and boilerplate repositories from all around the web.

Again: bundle times in development are an imaginary issue, and always have been. ESM doesn't solve anything here that wasn't already solved; but we sure are collectively paying the price for its existence now, with years of module compatibility misery.

@leonsilicon
Copy link

leonsilicon commented May 24, 2023

A clean configuration of a bundler is not equivalent to bundling in practice. I wouldn’t be so quick to dismiss long bundle times in practice as an imaginary issue given the extensive work and effort from the community and large companies into developing new solutions to this common issue of long bundling times, from Vite to Turbopack.

Browserify was created when JavaScript projects were much, much simpler and when projects were much less large and complex than they are today. Have you ever tried running a “clean configuration” of Webpack on the types of codebases that the folks at Vercel had to support, who have concluded that spending a significant amount of effort and time rewriting Webpack in a more efficient language was the best approach to solve this “imaginary” issue of long bundling times?

@joepie91
Copy link
Author

A clean configuration of a bundler is not equivalent to bundling in practice.

Yes, it absolutely is. "Bundling" is the actual task that 'native module loading' is meant to replace - it doesn't replace transpiling, minification, etc. (which is where the vast majority of the real-world "bundling" time goes, and which ESM doesn't solve). The bundling itself - the part that ESM replaces - is also extremely fast, because depending on implementation it's either a string concat or an AST subtree merge, and not much else.

This is precisely why the whole "native module loading for performance" doesn't make sense.

Browserify was created when JavaScript projects were much, much simpler and when projects were much less large and complex than they are today. Have you ever tried running a “clean configuration” of Webpack on the types of codebases that the folks at Vercel had to support, who have concluded that spending a significant amount of effort and time rewriting Webpack in a more efficient language was the best approach to solve this “imaginary” issue of long bundling times?

Yes. I have also analyzed the 'benchmarks' of tons of supposedly-faster bundlers. The common factor among them is that they either compare unequal configurations (bundling-only config of their bundler vs. random boilerplate repo for Webpack), or that the part that they've sped up isn't actually the bundling but rather eg. the transpilation - which would have been entirely possible without replacing the bundler too.

Likewise, I've profiled many real-world bundler setups. The bottleneck was never the actual bundling process itself, but always one or more particularly slow plugins (which, again, implement functionality that ESM doesn't).

Nothing has fundamentally changed about bundling since Browserify. It's still the same mechanism, emulating the same module system, with the same requirements (keep in mind that things like bundle splitting also already existed for Browserify, and are not a new requirement).

@leonsilicon
Copy link

If the part that’s sped up by tools like turbopack isn’t bundling, then what is it? What is the exact explanation or so-called slow plugin that causes a significant discrepancy between the benchmarks of webpack and turbopack? Transpilation is already sped up by swc, minification by terser, so I can’t really see what other possible “operation” that was sped up by the Rust rewrite of webpack other than bundling (but would definitely be interested to know about!)

@jimmywarting
Copy link

jimmywarting commented May 25, 2023

I'm honestly not that sold on that bundeling everything is the best thing for you.
I believe code splitting and lazy import is the way to go.
Only deliver things when you need them and don't ship everything at once.
All the javascript on a website is often the bottleneck now days as it's what takes up most time. A image that's 2x larger in size can take faster to load then a single equally sized javascript bundle without any dependencies (cuz it was bundled).

Ofc you may not want to depend solo on just only on 100% modular system (with whatever system you decide to go with, esm, cjs, umd, require.js or heck even importScripts)
you do not exactly want to load React as ESM in it uncompiled / non-bundle state and doing 4+ roundtrip to fetch all their 100+ files. that would be bad i agree. cuz they have no lazy import or code splitting what so ever... everything is just import x from y in top of all the files. so it would have to go and fetch everything, even the things it dose not currently need.

But i don't think you exactly need to import the checkout/payment script on the start page of a eCommerce where it is only needed on the last checkout page before leaving the site. that would be somewhere where bundling everything would be bad idea.

But if you are building a PWA that should be able to work offline wherever you are, then bundelling is a grate option for you.


This is why modules can never work directly in a browser (at least, over the internet). It has nothing to do with which module system you use - it's a fundamental limitation of physics, of how networks work. ESM cannot break the laws of physics either.

you make it sound like es modules are bad when you are talking about something completely different.
you talk about bundling vs not bundling, resolving dependencies chain, using build steps etc etc.

it's what the ESM have allowed us to archive to do without a build tool which at the end is what you are complaining about. and that have nothing to do with cjs or es modules. So i would rephrase the statement from "Once again: ESM does not "work in the browser" any more than CommonJS does." cuz ESM is something that do work in browsers and everywhere else where as CJS dose not work everywhere. you can't use require(path) inside of a browser. CJS only works in NodeJS.
If you want to tell that ESM is bad performance wise. then just lead with that and say so.
Not that it dose "not work" cuz it can, You can have a super simple site with very few small javascript files without much dependencies.
you can even have ESM that is bundled. you maybe want to resolve a relative path to some other asset that isn't javascript relative to import.meta.url cuz maybe you wish to resolve the path to some image, css or wasm file that you depend on as well then it's even more nice to use that rather then using absolut paths.

I for once prefer build-less setup that don't require me to install a bunch of code that someone else have built by having access to my machine and bloating down my computer with lots of files. I like it when i can just use the built in simple php server that comes with my mac and just do php -S localhost:8080 without having to set up esbuild, webpack, typescript, ts-node or whatever when i'm just testing things out.

cuz for library development ESM is really nice to work with (locally), but for application development then build tools are more likely needed.

@mk-pmb
Copy link

mk-pmb commented May 28, 2023

I've had a lot of use cases where I was happy to trade a full minute of load time in someone's browser for being able to just upload files to the cheap static webserver in their NAS. And be able to edit a single file on their NAS over a slow connection, where uploading a bundle would have taken annoying amounts of time. Or be able to advise them on the phone what line to change how. A bundling step drastically increases the server requirements.

Also in some tiny modules I have more UMD boilerplate than acutal code, just to make the same file work in node.js and the browser.

@jimmywarting Your simple static webserver argument could be even stronger if you use python's, becasue python ships with a lot of GNU/Linux Live DVDs. Thinking about convincing school teachers to set up a JS learning environment for their students.

@Rizean
Copy link

Rizean commented May 29, 2023

Whatever time ESM might save is completely lost when you try to do anything remotely complex. ESM + Nodejs + TS + Jest in a complex network application is a nightmare. I feel like we have spent more time working around ESM issues than we have working on the APP. Mocks are nearly impossible.

@jimmywarting
Copy link

@Rizean then use fewer build tools.

  • Replace TS with jsdoc like svelte@4 did https://www.youtube.com/watch?v=MJHO6FSioPI
  • Replace Jest with NodeJS own builtin test runner instead of dealing with complex experimental VM that Jest is using
    • NodeJS comes with it's own test runner and have had assertion built in since forever.
    • NodeJS even have built in mocking

fewer build tools / steps / compiling / transpiling == king

@StevePavlin
Copy link

StevePavlin commented May 29, 2023

My advice is if you're writing a greenfield application right now to avoid third party libraries/transpilers as much as possible to keep the bundling complexity down. As OP mentioned, there is still a bundling step required to have large enough files to be compressed. The current browser spec is very inefficient since it will make HTTP calls for each file, even with HTTP2 muxing this is unreasonable for production.

My pipeline right now is just CommonJS + Webpack to replace CSS/require statements, or react if I choose to use it.
Most of the modules are Pure CPU like a redux application, therefore they are deterministically testable using vanilla Node.js CJS.

The fact that the node team thought it was a good idea to force this standard on the backend, removing features we know well such as __dirname is beyond me.

The core problem here is that the npm ecosystem has leaked to the frontend, and people are trying to force isomorphic modules when they should actually be separate (due to backend only modules like fs, net, etc.).

WebAssembly (specifically the WASI spec/sandbox) is the closest attempt I've seen to fix this. Fingers crossed for widespread adoption.

@jimmywarting
Copy link

jimmywarting commented May 30, 2023

The fact that the node team thought it was a good idea to force this standard on the backend, removing features we know well such as __dirname is beyond me.

thing the reasoning for removing it was that it never fitted well into the enviorment of browsers where it have no access to any filesystem, hence why import.meta.url was born.


Something that's beyond me is how they could invent a cjs system where they tough index.js files where "cute" as ryan dal puts it.
That it was a good idea and that loading files without extension was also a good idea too was a stupid misstake and it turned out to be a footgun in the end.

it ended being the "Not how browsers work" ideologi. remote http resolver have no access to the file system so they can't guess what you meant to import.
the hole require.resolve algoritm requires stat'ing thousands of small files in the src + node_modules dir cuz it always as to resolve require('./foo') to look for a folder with a index file, a json file or if there is any ./foo.js file.

I made a benchmark of having 5000 cjs files that require() empty modules.exports = {} with and without explicit path (that included the full ./foo/index.js path. in a variety of different folders with and without index.js files

The result was thet it booted a few ms / sec quicker if you used explicit paths instead.
NodeJS didn't had to stat every directory and having to guess what you meant to import.

i made this test after i read the hole: cost of many small modules


if NodeJS runtime was born after ESM had been introduced into browser then the ecosystem would have looked a hole lot different right now.
To bad NodeJS created CJS before ESM was born. or that ESM wasn't invented earlier, then we would not have been in this situation now otherwise where we have 2 competing modular systems

@mk-pmb
Copy link

mk-pmb commented May 30, 2023

The amount of ueseless stat()ing is a void issue because you can easily avoid it in your project with an auto-fixing linter, and you can easily mitigate it in 3rd party libs by bundling your dependencies separately from your app. (If the separation is cumbersome, that's a misfeature of your bundler.)

It's very easy to melt lots of small modules into a huge one. It's often very hard to do it the opposite way. Seems like an obvious choice for me. We're lucky that we had the opportunity to establish a small module ecosystem. There's a reason why Node.js became so popular, and why lots of other programming languages started imitating Node's module flexibility.
The performance issues are something that machines should be able to solve automatically. I assume you meant this with

Update (21 May 2018): Later versions […] added […] features […] which address most of the concerns raised in this blog post.

@jimmywarting
Copy link

"The amount of useless stat()ing bundlers and linters have to make to fix them is just as bad",
they should have been fixed already in the first place. a linter should not have to dive into the node_modules to fix them

but that wasn't the only point i was trying to make by saying that stat()ing makes things slower.
it was that remote http resolver can't for the life of it figure out what file it need to request when you don't have access to the file system and stat() things like when if you are building a experimental http loader that could require cjs files remotely that could require npm packages without having to install them directly to have some form of lazy installer.

i'm saying that this hole stat()ing is just pointless and unnecessary.
what good dose it actually bring?
so what if you have to type a few extra keys. omitting it is just a very lazy part on us self

it's just puts more unnecessary work on resolving paths the NodeJS style and making it more complex then what it really needs to be.
someone who is building the next feature esbuild / webpack / rollup might not want to have to deal with the same issues NodeJS have had to deal with. if something isn't running things the NodeJS style and sees require('./util') then it wouldn't know what to do with it cuz ./util dose not actually exist...

and how often do you compile something server side like a backend really?

the amount of work NodeJS have to do to figure out a cjs path is insane compared to the simple logic of strict ESM paths where it could just open() a destination instead of looking at what kind of files there are in a specific directory.

@mk-pmb
Copy link

mk-pmb commented Jun 3, 2023

it's just puts more unnecessary work on resolving paths

I belong to the school of thought that in ideal scenario, a code file should only contain intent. Only stuff that the programmer has decided and means to express. Any implementation details that can be automatically determined by a build step, should be left to the build step. Such derived information would only be noise, and distract the human reader from the actual intent expressed.

Of course, an ideal programming language would allow for fallback mechanisms to degrade gracefully in less-than-ideal scenarios. In this case, it means that I can optionally use a linter to stain my code files with path info, trading a bit of noise for increased compatibility. But that's a fallback option, not as default.

the amount of work NodeJS have to do to figure out

… can easily be reduced by smart caching of resolver results. We could even store them in a standardized way to help browsers optimistically fetch lots of stuff early.

@StoneCypher
Copy link

StoneCypher commented Jun 14, 2023

oh look, someone's complaining about es modules without reading the ten year discussion that led to how they work

and then all his complaints are either something about a tool, or that randos make claims

and it's a disaster because (wait for it) await import

like. really? dude this is a relatively successful thing for such a big change

there are real disasters out there. this isn't one of them

@StoneCypher
Copy link

StoneCypher commented Jun 14, 2023

Once again: ESM does not "work in the browser" any more than CommonJS does. It is completely impractical from a performance perspective to load ESM libraries directly without a bundling step of some sort, for fundamental network latency reasons.

This is the absolute weirdest claim. Like maybe you didn't know that most large websites make multiple hits to their backend to fetch script, over nonsense like code splitting.

tHiS cAnT bE dOnE fOr FuNdAmEnTaL nEtWoRk LaTeNcY rEaSoNs appears to be a rephrasing of I think this is a bad strategy

And that's cool, and all - hell, I even agree with you in this case - but lots of people don't, and are actively using the system in the way you seem to believe isn't feasible

In the meantime, representing bundling as a fundamental tool, but also suggesting CommonJS was the best way, when CommonJS modules generally should not be bundled

It seems like most of this discussion boils down to "ES6 modules are bad because I wouldn't have made them that way"

@smolinari
Copy link

smolinari commented Jul 6, 2023

My 2 cents is, what is ranted about above is mostly about the cost of standardization. And, I'm sad the OP can't see how having one standard is the main selling point of ESM and hugely valuable for the community to move forward at a better, faster pace. I want to work with JavaScript the same way everywhere I work with it. That's the win!

One can fight the new way, or one can join it and support the overall goal of that one huge advantage of standardization, so it CAN take on its full value. Nothing written above really speaks against using ESM or it being the full on standard for all JavaScript nor is it pointed out how the world of NodeJS won't be able to move forward with it (other than there will be more people, who won't change too). All that is pointed out, again, is why it is painful to change.

developers will forever be paying the cost of the interoperability issues between them

Only if developers take on this stance and don't change, then yes. But, if enough developers change, we'll be alright and at some point the momentum will be so good, all the change pain will be worth all the effort... the effort noted as "hundreds of thousands hours of wasted time". Please don't make that situation worse. Make it better, cause we aren't turning back. Accept this change is happening. Understand the huge value for the community of one standard way and move on.

Edit: I ran into this just 10 minutes after writing this comment....a proper example of how it should be done.

image

Scott

@jimmywarting
Copy link

developers will forever be paying the cost of the interoperability issues between them

Only if developers take on this stance and don't change

I really think npm, yarn and others needs to default to generate "type": "module" today. the more and more new packages keep pumping out cjs then the more wasteful hours will be put in overall.

@mikkimichaelis
Copy link

mikkimichaelis commented Jul 10, 2023

I read the document and 100% agree. I don't have time to read all the comments, sorry, I've already wasted too much time dealing with converting commonjs to ESM. Why did I start this? Because some package was ESM only and I thought "oh, ESM must be the future" and a little research proved this to be true.

Way too much time was spent updating imports and trying to figure out tsconfig.json and package.json settings - for what? I'm writing a hybrid app so I don't really care at all about 'tree shaking' - it's a downloaded app - and node backend apps that don't even use tree shaking.

So the only benefit was the ability to use that one stupid ESM only nom package....I've totally forgotten what package that was at this point, I've been digging through code so long I've no idea what even started this ESM conversion process - but when I find it it's GONE.

@leonsilicon
Copy link

a change that requires a non-trivial migration process != the change is inherently bad

nobody is forcing you to migrate to esm; you're free to use the commonjs version of that package and remain on commonjs for as long as you want. if you don't see much benefit with migrating to ESM other than using the newest version of one specific package, i don't think you should be spending time migrating, just like how you're not required to always migrate to the newest version of (good) javascript frameworks if you don't see much benefit

@divmgl
Copy link

divmgl commented Jul 10, 2023 via email

@leonsilicon
Copy link

This is not a good argument. Some packages are dropping CommonJS support
and their newer ESM versions have upgrades that aren't available in the
CommonJS versions.

say you have a project that uses vanilla javascript, and there's a package that you want to use that only supports react. if you start migrating your project to use react just to use this react package, it isn't fair to blame react for being bad because it's a "migration nightmare"

on the other hand, the lack of ESM support from node.js is a valid criticism (and I'd generally agree with you on that), but the fact that commonjs -> esm is non-trivial migration process isn't a valid criticism for why ESM is inherently bad

@divmgl
Copy link

divmgl commented Jul 11, 2023

A majority of the complaints here are ESM with Node.js. Frontend ESM has been handled for years by transpilers (like Babel). The remaining challenge with ESM is still the Node.js implementation.

@bentorkington
Copy link

say you have a project that uses vanilla javascript, and there's a package that you want to use that only supports react. if you start migrating your project to use react just to use this react package, it isn't fair to blame react for being bad because it's a "migration nightmare"

But people aren't replacing CJS modules with React-only modules, so this nonexistent comparison makes no sense.

They are replacing CJS modules with ESM however, so it could well be said consumers of these modules are somewhat being forced.

@joepie91
Copy link
Author

joepie91 commented Jul 14, 2023

I'm sad the OP can't see how having one standard is the main selling point of ESM

We already had one standard. It was called CommonJS. The whole problem is that the introduction of ESM means that we now have two.

Only if developers take on this stance and don't change, then yes.

You do realize that there's a huge ecosystem of existing modules that will never get updated, right?

And both of these points are mentioned in the original post, by the way. Did you even read it?

@leonsilicon
Copy link

leonsilicon commented Jul 14, 2023

But people aren't replacing CJS modules with React-only modules, so this nonexistent comparison makes no sense.

You're missing the point of the comparison; the point I meant to get across with that comparison was that an optional change resulting in a non-trivial migration process isn't a valid consideration when assessing the inherent utility of the new change. In other words, just because the migration process from CJS to ESM isn't trivial, it says nothing about the inherent utility of ESM (i.e. whether it's inherently good or bad).

They are replacing CJS modules with ESM however, so it could well be said consumers of these modules are somewhat being forced.

”replace” is a misleading term used here, implying that old CJS packages become unusable when ESM was specifically designed to preserve compatibility with CJS. In the same way libraries often release breaking changes alongside new features, it's a stretch to say that they're "replacing" the old version of the package and forcing you to upgrade.

@leonsilicon
Copy link

leonsilicon commented Jul 14, 2023

We already had one standard. It was called CommonJS. The whole problem is that the introduction of ESM means that we now have two.

With this logic you could also argue that we already had a standard called ES5, and so the problem with the introduction of ES6 means we now have two.

I think it's more appropriate to think of ESM as an updated "standard" (that is, if you consider CommonJS a standard, which I don't think it was), with one of the key benefits being native support for a wider range of JavaScript execution environments (since the old CJS "standard" wasn't compatible with browsers).

@alex-kinokon
Copy link

With this logic you could also argue that we already had a standard called ES5, and so the problem with the introduction of ES6 means we now have two.

ES2015 extends ES5. It’s a strict superset of ES5. Everything you can do in ES5, you can still do it in later ECMAScript versions. ES modules, on the other hand, are reimplementing CommonJS features that already exist.

if you consider CommonJS a standard, which I don't think it was

CommonJS was standardized 6 years before ES2015. I don’t know why people keep calling it a non-standard when the specs is just one google away.

@leonsilicon
Copy link

leonsilicon commented Jul 20, 2023

ES modules, on the other hand, are reimplementing CommonJS features that already exist.

And they can import CommonJS modules, so therefore everything you can do in CommonJS you can do with ESM using import cjsPackage from "./file.cjs"

CommonJS was standardized 6 years before ES2015.

This isn’t part of the actual ECMAScript standard (due to legitimate technical limitations with supporting other JS environments), which is what most people mean when they say CommonJS isn’t standard

@jimmywarting
Copy link

jimmywarting commented Jul 20, 2023

if you consider CommonJS a standard, which I don't think it was

CommonJS was standardized 6 years before ES2015. I don’t know why people keep calling it a non-standard when the specs is just one google away.

I don't view CommonJS as a "standard" either. In the early days, when executing JavaScript in NodeJS beta, the only practical way was to bundle everything into a single file and then run it. To address this limitation, they came up with the idea of creating a function that would read the content of a file and "eval" it.

Essentially, it's just a function wrapper that could have been devised by anyone. The following is an example of how it works:

(function(exports, require, module, __filename, __dirname) {
  // Module code actually lives in here
})

Thus, you could say they do something like this:

function require(path) {
  // 1. Read the file
  const code = readFile(path)
  // 2. Construct a new function
  const cjsFunction = new Function('exports', 'require', 'module', '__filename', '__dirname', code)
  cjsFunction(...args)
  // Perform actions that change `module.exports` to create a module and cache it as a singleton
}

this is just an over simplification.

The approach taken by CommonJS wasn't groundbreaking; anyone could have built a module system like this one even before NodeJS was invented, utilizing some form of synchronous XHR. However, before NodeJS, there was at least a way to load things using script tags, which NodeJS couldn't utilize due to being a JavaScript runtime without a DOM.

I wouldn't categorize anything that Deno, Bun, or NodeJS does as creating a "standard"; they each solve problems in different ways. Their entire I/O systems are vastly different, and they are not obligated to adhere to any web specification standard. However, they are gradually starting to implement more standards because they recognize that creating something specific to their environment is not conducive to cross-compatible solutions and makes it harder to run code on other platforms.

Every other server side engine and also the web adhere to a web specification, which is whatwg, TC39. 👈 this is what i consider a "standard".

This is just my own personal opinion.

i wouldn't be surprised if servers side engine later brings in the hole https://github.com/whatwg/fs and start saying that node:fs is now consider deprecated. cuz using file system access is the new way to go to solve things in browsers and it may become the new mainstream solution for server side solution as well.

@Turbine1991
Copy link

Turbine1991 commented Jul 23, 2023

The main issue is the import/es modules syntax is quite overengineered and contains syntax one wouldn't typically associate with JavaScript. It's also extremely overengineered, trickier to type, harder to interpret at times and less consistent at times.

Who the hell has time for all this:

export let name1, name2/*, … */; // also var
export const name1 = 1, name2 = 2/*, … */; // also var, let
export function functionName() { /* … */ }
export class ClassName { /* … */ }
export function* generatorFunctionName() { /* … */ }
export const { name1, name2: bar } = o;
export const [ name1, name2 ] = array;

// Export list
export { name1, /* …, */ nameN };
export { variable1 as name1, variable2 as name2, /* …, */ nameN };
export { variable1 as "string name" };
export { name1 as default /*, … */ };

// Default exports
export default expression;
export default function functionName() { /* … */ }
export default class ClassName { /* … */ }
export default function* generatorFunctionName() { /* … */ }
export default function () { /* … */ }
export default class { /* … */ }
export default function* () { /* … */ }

// Aggregating modules
export * from "module-name";
export * as name1 from "module-name";
export { name1, /* …, */ nameN } from "module-name";
export { import1 as name1, import2 as name2, /* …, */ nameN } from "module-name";
export { default, /* …, */ } from "module-name";
export { default as name1 } from "module-name";

And then we have import.

import defaultExport from "module-name";
import * as name from "module-name";
import { export1 } from "module-name";
import { export1 as alias1 } from "module-name";
import { default as alias } from "module-name";
import { export1, export2 } from "module-name";
import { export1, export2 as alias2, /* … */ } from "module-name";
import { "string name" as alias } from "module-name";
import defaultExport, { export1, /* … */ } from "module-name";
import defaultExport, * as name from "module-name";
import "module-name";

@leonsilicon
Copy link

leonsilicon commented Jul 23, 2023

contains syntax one wouldn't typically associate with JavaScript.

All the ESM examples you gave seem to build on top of a lot of existing JS syntax and semantics (strings, destructuring, comma-based separation), with the addition of only two new keywords. Seems pretty JavaScript-y to me 🤷‍♂️

the import/es modules syntax is quite overengineered

Definitely subjective, but ESM syntax seems pretty simple to me and typically leads to much cleaner code in my experience. Also, ESM is only “overengineered” because building an ergonomic import/export API in JavaScript with minimal surprise factor is inherently going to involve more cases than just one way to import and export things; I can give you a dozen ways to export a single variable in CommonJS:

module.exports = myVar;
module.exports = { myVar };
module.exports.myVar = myVar;
exports.myVar = myVar;
exports["myVar"] = myVar;
var m = "myVar"; exports[m] = myVar;
Object.defineProperty(exports, myVar, { value: myVar })
var e = exports; e.myVar = myVar;
eval('module.exports = myVar')
Object.assign(exports, { myVar })
function fn(e) { e.myVar = myVar } fn(module.exports);
globalThis["module"].exports = myVar;

trickier to type

TypeScript works a lot better with import/export syntax than CommonJS (in fact, they even have the very straightforward import type syntax that builds really ergonomically on import/export syntax)

harder to interpret at times and less consistent at times.

Actually, one of the pitfalls I’ve had to deal with in CommonJS is how many different ways there are to mutate module.exports, compared to the relatively easy static analysis of ESM (see (rollup/plugins#1398 (comment)). I’d argue that ESM is much easier to statically analyze because it lacks dynamic keyword exports. It’s also much cleaner to read because you can place the export “marker” at the variable declaration site, whereas with CommonJS I’d frequently end up with code like:

const myVar = /*
  hundreds of lines of code
*/

exports.myVar = myVar

which makes it much less clear that myVar is exported than export const myVar and makes it harder to trace code and use tooling like go-to-definition functionality.

@smolinari
Copy link

because building an ergonomic import/export API in JavaScript with minimal surprise factor is inherently going to involve more cases than just one way to import and export things

Yep. I was going to say the examples look like flexibility to me and not "over-engineering".

Scott

@ehaynes99
Copy link

ES2015 extends ES5. It’s a strict superset of ES5. Everything you can do in ES5, you can still do it in later ECMAScript versions. ES modules, on the other hand, are reimplementing CommonJS features that already exist.
And they can import CommonJS modules, so therefore everything you can do in CommonJS you can do with ESM using import cjsPackage from "./file.cjs"

Not true. You can't require. You can run ES5 code in any interpreter without changes. That's what "superset" means, and history has shown over and over that it's the better way to evolve. Providing an alternate mechanism that requires rewriting the code is not the same at all. The larger the superset, the larger the chance that there is some small mistake, but intent matters, and ESM's intent was to nuke the ecosystem and be "revolutionary".

They succeeded.

As a comparison, arrow functions are better than the function keyword in every way. They have no binding of this issues. They don't have broken hoisting that allows referencing values before initialization. Every value they reference is statically analyzable. They're better. I wish everyone would just stop using function. However, when they were introduced, they didn't just remove the function keyword, because that would cause a python2 fiasco. Entire codebases would have required rewrites. In short, it would break things, and that breakage would mean poor adoption.

Whether you agree about arrow functions is not the point (and I in no way mean to sidetrack this thread on that); it's merely an example of why forcing incompatible changes across an ecosystem are harmful, even if you're ever-so-certain that the new way is better.

Whether commonjs was an "official" standard or not, it was a de facto standard. For the first time in 15 years, the ecosystem had overwhelmingly settled on a module system, because there was finally a sorely-needed package manager. For 5+ years, you had a state of:

  • a project is abandoned, and uses something else
  • a project is actively maintained and exports only one type of module, commonjs
  • a project is actively maintained and exports multiple types of modules, but one of them is commonjs

You can point out its flaws, but regardless, the new, "better" thing should have tipped a hat and been compatible. It wasn't, and here we are 8 years later with horrible adoption rates.

The most prolific advocate of ESM started forcing this a few years ago. His projects mostly look like this:

p-waterfall
Version              Downloads (Last 7 Days)          Published
3.0.0 (ESM)          105                              2 years ago
2.1.1 (commonjs)     1,079,014                        3 years ago

TypeScript works a lot better with import/export syntax than CommonJS (in fact, they even have the very straightforward import type syntax that builds really ergonomically on import/export syntax)

TypeScript quite happily converts all of those to commonjs by default. The syntax isn't the issue. It would be trivial to add support for import/export syntax without changing the underlying module architecture. But they did. They made asynchronous loading a requirement, and it was completely unnecessary.

It could have been that:

  • you add import/export syntax, with parsers that complain if you use a variable rather than a static path
  • you leave the how of loading it up to the runtime

Browsers could load it asynchronously. Node could load it synchronously. It would have taken a couple of days to implement in node, rather than years. It doesn't matter in the end; you can't use dependencies within a file until those dependencies are loaded. ESM unnecessarily pushed it into the spec. It was based on this pipe dream that we could all just start shipping code without transpiling or bundling, and your files could just import other files via tens of thousands of calls over the network. However:

  • we all still compile anyway, whether for TypeScript or JSX or css modules or environment variable substitution or tree shaking or minification or...
  • that would perform terribly anyway without HTTP/2, whose adoption has been about as successful as IPv6 or ESM

This would preclude top-level await, which would break... absolutely nothing. Side effects on load are a terrible pattern anyway; you should be able to preemptively load modules. There should be only one file that does that, which is the entry point. The rest should export a function that the importing file can call. If you have async effects as a result of import 'some-file.js', then you have no error handling for the file rejecting its load anyway. The language has no syntax for "Promise that can't reject", so why would you even want to take away the caller's ability to deal with it and proceed?

At the end of the day, most of the decade-long debate has been about the syntax. The syntax is the easy part. In fact, I would guess that at least half of those who are advocates of ESM based solely on the syntax aren't even aware that their code is being transpiled to commonjs anyway. Unless you depend on top-level await, you would never know the difference.

And finally, some common misconceptions:

  • ESM are statically analyzable - all code is statically analyzable. An AST parser does exactly that. You can iterate through a whole file and find all of the import/require in either case.
  • But ESM requires static imports/exports - no, you can await import(someVariable) just as you can require(someVariable). Both have to deal with it being something you can't statically analyze, and likely both should have linting rules that say "do you realize you're about to break a bunch of things?"
  • Static analyzability is a feature - Well, sure, if you're using a bundler or compiler. It serves no purpose at all in a browser that's going to load dependencies over the network.
  • It will lead to downloading too much on the web - no, we have code splitting for that. In fact, we have code splitting even with ESM. I'd love for someone to find a single example of a production site that's using ESM to solve this.

@leonsilicon
Copy link

Not true. You can't require. You can run ES5 code in any interpreter without changes. That's what "superset" means, and history has shown over and over that it's the better way to evolve. Providing an alternate mechanism that requires rewriting the code is not the same at all. The larger the superset, the larger the chance that there is some small mistake, but intent matters, and ESM's intent was to nuke the ecosystem and be "revolutionary".

If running code in any interpreter without changes is a goal we aim to achieve, CommonJS fails that, as you can't run CommonJS code in any interpreter (i.e. the browser) without changes.

In addition, you can "require" in ESM:

import { createRequire } from 'node:module'
const require = createRequire(import.meta.url)

ESM's intent was not to nuke the ecosystem; it was to provide a standard that allows you to use JavaScript modules in any interpreter, which CommonJS could not achieve due to inherent limitations with its specification that browsers were unable to adopt. CommonJS was always specific to Node.js.

Browsers could load it asynchronously.

And browsers/any CommonJS environment can do that with ESM packages using the await import() syntax.

ESM are statically analyzable - all code is statically analyzable. An AST parser does exactly that. You can iterate through a whole file and find all of the import/require in either case.

ESM imports/exports are statically analyzable. CommonJS imports/exports are not. That's why issues like rollup/plugins#1398 (comment) exist.

But ESM requires static imports/exports - no, you can await import(someVariable) just as you can require(someVariable). Both have to deal with it being something you can't statically analyze, and likely both should have linting rules that say "do you realize you're about to break a bunch of things?"

Yes, but you can't do await export(someVariable) like you can do in CommonJS, so you can statically analyze any top-level import statement because you can guarantee that the exports from an ESM package are only from top-level exports (i.e. statically analyzable).

@jensbodal
Copy link

Is there a way yet to get typescript to not require file extensions when converting to ESM? The only way I’m aware of is to use vite.

@privatenumber
Copy link

privatenumber commented Aug 11, 2023

@jensbodal

This seems like a strange place to ask... but yes, there are plenty of ways using bundlers.

Under the hood, Vite uses Rollup, so you can use that. Or esbuild. Or Webpack.

For running, you can also use tsx.

In TypeScript (for compiling or type checking), I believe you just need to change compilerOptions.moduleResolution to a value other than NodeNext or Node16 (e.g. bundler).

@jensbodal
Copy link

I mean the entire reason why I’m following this post is looking up “why es modules are terrible” when trying to convert a typescript package to esm and becoming extremely annoyed that I needed to start adding .js extensions to typescript file imports in order to support esm.

Changing the module resolution didn’t use to work but maybe now with the new bundler option?

microsoft/TypeScript#49083

@ehaynes99
Copy link

And browsers/any CommonJS environment can do that with ESM packages using the await import() syntax.

You're missing my point. That's in the source code, not in the interpreter code. What I mean is that:

  • node could:
    • parse a source file
    • find all of the things that are imported
    • read them synchronously from disk <-- only difference
    • inject the imported variables
    • execute the result
  • a browser could:
    • parse a source file
    • find all of the things that are imported
    • read them via network calls to the server <-- only difference
    • inject the imported variables
    • execute the result

ESM didn't have to insist that modules be loaded asynchronously. They could have simply created a spec for the injection process and let the runtimes use the most appropriate mechanism. The ONLY reason for the hard requirement is top-level await.

It's not even a good goal. The language gives no way to express "a promise that cannot reject".

// some file
import { db } from 'path/to/database.js'

// path/to/database.js

export const db = await connectToDatabase({ username: 'not a real user', /...

ESM imports/exports are statically analyzable. CommonJS imports/exports are not. That's why issues like rollup/plugins#1398 (comment) exist.

Again, you're only arguing syntax. I prefer the ESM syntax. I write TypeScript in node for a living these days. I haven't used commonjs syntax in years. I haven't use esm modules in years either. It's trivial to convert the syntaxes for the 99% of cases where they work exactly the same, which TypeScript does just fine. The syntax parsing is not the issue. The incompatible "features" are:

  • ESM top level await - I don't even see that as a feature, and yet it is the reason that ESM destroyed interoperability
  • commonjs arbitrary exports:
module.exports[Math.random().toString()] = 'hello'

If we had only broken the commonjs packages that did something stupid like that, adoption would have taken a few months. It would have been a reasonable compromise. But here we are 8 years later with no end in sight. ESM is nowhere close to 50% adoption, and ESM advocates (again, mostly over the syntax rather than the implementation) refuse to admit that it's not working.

As far as the specific issue with ansi-styles, you literally bring up a case by the one guy I named as the only pushing this. It's an obsession for him. A new religion. I posted one example, but feel free to look through the "versions" tab of the rest of his libs. 90% of them look the same: esm version: 1, commonjs version: 100,000. But that aside, that's not even a case of code that isn't statically analyzable. An AST parser that reads the whole file can see that statically defined keys are added to the exports object. It's no different from:

export let stuff

export const init = () => {
    if (Math.random() < 0.5) {
    stuff = 'some stuff'
  }
}

Let's be realistic here. The "browser devs" got their shorts in a wad that the "server devs" were pushing patterns that didn't work well for the browser (even though they really liked npm). Instead of finding a compromise, they decided it was time for retribution. Let's scorch the Earth and start anew! Make node listen to US for a change!

I've been both of those devs. I spent multiple years of my life in both camps. I don't have a side anymore. My sister hates my brother, and I want them to get along. If both give up on an infinitesimally small portion of their actual, real world use cases, we can end the war.

@leonsilicon
Copy link

leonsilicon commented Aug 12, 2023

read them via network calls to the server <-- only difference

This is infeasible in practice because browsers can’t practically make blocking calls to fetch dependencies (since require is synchronous at the syntax level). Synchronous module loading is an inherent limitation of the CommonJS module system and that made it impossible for browsers to support it.

In addition, just because you can statically analyze require calls does not mean you can load them asynchronously in advance. Here’s an example:

const file = require('./file.js');

let res = {}

if (file.foo) {
  res.foo = require('./foo.js')
}

if (file.bar) {
  res.bar = require('./bar.js')
}

if (file.baz) {
  res.baz = require('./baz.js')
}

module.exports = res

These files must be loaded synchronously in order; the browser can’t just download all three files in advance because they might not necessarily be used. While the browser is resolving and downloading these files, no other code can execute because require has to synchronously resolve to a value at the call site (meaning that your webpage is completely unresponsive while it fetches this module from the network). This problem is not present in ESM because module loading is asynchronous, making it possible for other code to execute while waiting for the browser to resolve and download a certain module.

In addition, if you want an example of how actual ESM (not ESM syntax transpiled to CommonJS) has improved real-world developer ergonomics, take a look at Vite.

@ehaynes99
Copy link

the browser can’t just download all three files in advance because they might not necessarily be used.

Why not? It wouldn't be that unreasonable of a solution. It's what ESM syntax would make you do anyway...

import file from './file.js'
import foo from './foo.js'
import bar from './bar.js'
import baz from './baz.js'

const res = {}

if (file.foo) {
  res.foo = foo
}

if (file.bar) {
  res.bar = bar
}

if (file.baz) {
  res.baz = baz
}

export default res

@leonsilicon
Copy link

leonsilicon commented Aug 12, 2023

Why not? It wouldn't be that unreasonable of a solution. It's what ESM syntax would make you do anyway...

It's unreasonable because fetching synchronously from a browser is a blocking operation (especially if those assets may not end up being used); the webpage would be frozen while the browser fetches those assets. There is no way to make the "require" function operate asynchronously in a robust way because it would violate JavaScript's fundamental principle that only one synchronous function is executed at a time:

let myVar = 'bar';
function foo() {
  myVar = 'baz';

  // The browser has no choice but to hang/be unresponsive while it
  // waits for this require call to finish executing; you can't fetch 
  // this asset in advance because there is no way to tell if the `foo`
  // function is going to be called at runtime, and fetching 
  // optimistically is not a scalable solution because it leads
  // to wasted bandwidth (this is the equivalent of a `fetch` request;
  // you can't expect the browser to make that operation synchronous by
  // statically extracting the URLs through AST parsing to fetch those
  // resources in advance).
  try { require('./file.js') } catch {}

  // JavaScript guarantees that this will log "baz" no matter what;
  // you can't have another function (e.g. an event handler) change `myVar`
  // before the require call finishes because `require` is a synchronous
  // operation and JavaScript guarantees that only one synchronous
  // function runs at a time. Therefore, the entire webpage must become
  // unresponsive while JavaScript waits for the `require` call to finish. 
  console.log(myVar);
}

There is no way to avoid a synchronous HTTP request for dynamic "require" operations that cause the webpage to become unresponsive while the browser fetches the assets (this is why synchronous HTTP requests are deprecated, see https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Synchronous_and_Asynchronous_Requests#synchronous_request). It is a necessary requirement for module loading to be an asynchronous operation for browsers to support it, and asynchronous module loading is fundamentally incompatible with CommonJS's synchronous require function, which is why browsers cannot implement CommonJS as it is.

If JavaScript wants a standardized module loading system that works in all JavaScript environments (which we all agree is a worthy goal to pursue), it needs to be asynchronous; CommonJS did not fit that requirement and thus could never be that standard.

In addition, the correct ESM equivalent is:

import file from './file.js'

const res = {}

if (file.foo) {
  res.foo = await import('./foo.js')
}

if (file.bar) {
  res.bar = await import('./bar.js')
}

if (file.baz) {
  res.baz = await import('./baz.js')
}

export default res

In this case, the browser only fetches the dependencies as necessary, and since it's done asynchronously, other JavaScript code can run while the browser waits for those resources, so the webpage doesn't freeze and lead to a terrible UX for the end user.

@fulldecent
Copy link

I think the main remaining issue with ES modules is that the browser can't include them using sub-resource integrity (SRI).

@ehaynes99
Copy link

It's unreasonable because fetching synchronously from a browser is a blocking operation

I'm not suggesting that the browser fetch it synchronously. When you do:

import { something } from 'some-file.js'

something()

That it's asynchronous is invisible to you. You don't have to await something. The browser asynchronously loads the file, parses it, finds all of the import statements, then recursively loads all of those the same way. Only then does it begin executing the file synchronously. It could do the same thing for the require statements. It would be functionally like the ESM version I pasted above, not yours with top-level await, but it would be a reasonable compromise. The implication there would be that you can't do lazy loading with require, but it would still work. The explicit await import could still exist for the purpose of lazy loading, and code targeting browsers would be free to use it that way.

Let's be realistic. The code that needs compatibility are libraries. They're not going to be doing much in the way of lazy loading. That would be application code, which could explicitly use async (rather than top-level) quite easily.

This would be done in the native code of the browser, but here's a fully working example in plain JS. All of the files it's loading are in the example-sources directory in the root of the repo. The brunt of it is:

// module-loader.js

const { tokenize } = require('esprima')
const fetch = require('node-fetch-commonjs')

const BASE_URL =
  'https://raw.githubusercontent.com/ehaynes99/async-require-example/master/example-sources/'

const getRequires = (content) => {
  const tokens = tokenize(content)
  const iterator = tokens[Symbol.iterator]()
  const requires = []

  let token = iterator.next()
  while (!token.done) {
    if (token.value.type === 'Identifier' && token.value.value === 'require') {
      iterator.next() // skip '('
      token = iterator.next()
      if (token.value.type !== 'String') {
        throw new TypeError(`Could not load file. Non-static require: ${token.value.type} ${token.value}`)
      }
      requires.push(token.value.value.slice(1, -1))
    }
    token = iterator.next()
  }
  return requires
}

const loadFile = async (relPath) => {
  const url = new URL(relPath, BASE_URL).toString()
  const response = await fetch(url)
  const source = await response.text()
  const requireCache = {}
  await Promise.all(
    getRequires(source).map(async (file) => {
      requireCache[file] = await loadFile(file)
    }),
  )
  let module = {}
  with({ module, require: (file) => requireCache[file]}) {
    eval(source)
  }
  return module.exports
}

module.exports = {
  loadFile,
}

output:

res { foo: 'stuff from another-file.js', baz: 'bazzzzzzzzz!' }

@leonsilicon
Copy link

leonsilicon commented Aug 13, 2023

Here’s an example that doesn’t use top-level await that browsers can’t support:

function register(cb) {
  // save cb somewhere to be called later
}

// common strategy for lazy loading resources
register(() => require('./foo.js'))
register(() => require('./bar.js'))
register(() => require('./baz.js'))

In addition, AST parsing only works if you enforce a strict set of limitations on what you can do with the require function, which is already incompatible with a significant amount of existing CommonJS packages. One is that you must make require a reserved word, since it is legal to currently name a variable require (and thus make it incompatible with using AST parsing to determine its uses ahead of time). This would violate JavaScript’s promise of backwards compatibility (since there already exists real world code using functions named require), so it is not a feasible solution. The function would need to be renamed to be absorbed into the ECMAScript spec, which would in turn force all Node.js modules to be rewritten anyways (since you would not be allowed to alias this renamed function as require since that’s incompatible with using AST parsing).

On the other hand, the dynamic import doesn’t rely on AST parsing to determine its uses ahead of time (because it’s asynchronous so it doesn’t need to be).

Ultimately it boils down to whether it’s a good idea to attempt to statically analyze fetch calls using AST parsing to attempt to asynchronously load the resources in advance (under a specific set of conditions) to avoid the problems of synchronous HTTP requests, which it doesn’t even fully solve. Intuition tells me that this "solution" is too flaky, unrobust, and borderline hijacks the semantics of a JavaScript function, making it unsuitable as a standard that is absorbed into the spec and become forever part of the JavaScript language.

@guest271314
Copy link

This is incredible!

The only folks in JavaScript world who have issues with Ecmascript Modules are the folks tethered to node executable alone.

CommonJS and Ecmascript Modules are incompatible. Choose one of the other.

If you write source code for JavaScript runtimes other than Node.js Ecmascript Modules are the standard, not CommonJS. You might get some traction with CommonJS usage in Bun, which tries to emulate Node.js, however you will not find CommonJS to be the standard in Deno, QuickJS, txiki.js, et al.

This topic arises almost excusively dues to developers having a legacy preference for Node.js style of writing JavaScript. Which is fine. Stick with Node.js and the default CommonJS loader.

@ehaynes99
Copy link

AST tokenizing was just a simple means to make an example. The JS engines have far more sophisticated parsers. They're generating native machine code, interning string literals, allocating memory, and nowadays, literally rearranging the code to optimize it, both during initial parsing and at runtime. They're doing plenty of determining of uses ahead of time. It's been a long time since any of the engines were actually interpreting source in real time.

since it is legal to currently name a variable require (and thus make it incompatible with using AST parsing to determine its uses ahead of time)

Nonsense. My tinker toy example just flattened the tree and naively searched for any use of the identifier by name without even considering scope. The language allows shadowing, and you can reason about it with AST by traversing it as a tree. If the global require were shadowed in some scope, it could just be ignored. Only references to the global require would need to be evaluated. In a real implementation with machine code, it's not even ambiguous. They're distinct memory addresses. Actual variable names don't exist at that point. You can't say "give me the variable name(s) of this value", because they're completely elided.

Beyond that, commonjs modules already have to treat require and module as reserved. There wouldn't be any existing examples of this "problem".

const require = () => {
      ^
SyntaxError: Identifier 'require' has already been declared

Regardless, my point was merely that it's not as insurmountable a problem as people make it out to be. ECMA could have made a commonjs standard where non-literal names are disallowed. Even dynamic binding could be supported as long the names are static. E.g. even with AST, you could statically determine the names:

module.exports.init = () => {
  Object.assign(module.exports, {
    first: 'one',
    second: 'two'
  })
}

is the same as:

export let first
export let second

export const init = () => {
  first = 'one'
  second = 'two'
}

But fine, let's say that adoption of the new standard were actually a bigger priority than ideology

Earlier you said:

ESM's intent was not to nuke the ecosystem; it was to provide a standard that allows you to use JavaScript modules in any interpreter, which CommonJS could not achieve due to inherent limitations with its specification that browsers were unable to adopt.

Well, it failed due to inherent limitations with its specification that node was unable to adopt. They made a concious decision to make async loading of modules part of the spec rather than an implementation detail, knowing full well that it wouldn't work in the interpreter that made up half of the ecosystem. Top-level await is the ONLY reason it had to be so, and it's completely unnecessary. It's the tiniest bit of syntactic convenience that's easily -- and better -- solved by exporting a function. As I said before, you guarantee that errors are just thrown off into the ether when you don't await a promise, and you can't await a static import. If they had left that "feature" out of it, the sync/async loading could be left up to the interpreter. It would have been feasible that the "standard" would actually be adopted.

But that's not what happened. They said "screw you" to the "other side", and now the "standard" isn't standard, nor does it look like it ever will be. ESM is not "the future", no matter how many years people keep saying it. It's one of two competing, incompatible "standards". Sooner or later, we're all going to have to go back to the design goal of ESM and actually adhere to it. Revise the spec without that "feature" and the whole thing could be settled. Node could load ESM synchronously, browsers could load them asynchronously. Other than cases of dynamic export names in existing commonjs code -- which I posit are extremely minimal in actual practice, and a rather hacky abuse when they actually do occur -- almost all of adoption could be an eslint autofix.

ESM won't "win the war" without changes. Period. Deny it for another decade if you want, or be part of the solution.

@leonsilicon
Copy link

AST parsing is not a robust solution.

Object['a' + 'ssign'] = () => { /* noop */ }
module.exports.init = () => {
  Object.assign(module.exports, {
    first: 'one',
    second: 'two'
  })
}

@leonsilicon
Copy link

leonsilicon commented Aug 14, 2023

It’s more about being self-aware that you likely don’t know and haven’t thought through/considered nearly as much about designing a robust JavaScript module loading standard to be so stubbornly confident that you supposedly are smarter and can easily come up with a better approach than what some of the smartest JavaScript engineers in the world have spent years designing and deciding on and instead attribute their decision as maliciously (or stupidly) wanting to destroy the JS ecosystem.

@guest271314
Copy link

and you can't await a static import

The best we get?

<script>
  onerror = (e) => console.dir(e);
</script>

@guest271314
Copy link

are smarter and can easily come up with a better approach than what some of the smartest JavaScript engineers in the world have spent years designing and deciding

That is possible.

Just because an individual or group of individuals are smart doesn't mean they make the correct decision. Nor that the decision they make cannot be improved upon.

Good enough ain't good enough.

@leonsilicon
Copy link

leonsilicon commented Aug 14, 2023

Not saying it isn’t possible; I’m commenting more so about the way one approaches the problem; suggesting that these smart JavaScript engineers deliberately wanted to destroy the JS ecosystem and that they were too stupid to think of a 30-line AST parsing solution isn’t very realistic/productive.

@guest271314
Copy link

People get comfortable in their domains. Shun constructive feedback. Ban dissent. Do the bidding of their masters. Want to be masters of others.

suggesting that these people deliberately wanted to destroy the JS ecosystem and that they were too stupid to think of a 30-line AST parsing solution isn’t very realistic/productive.

Happening right now in a different venue.

[wei] Ensure Origin Trial enables full feature.

Has happened before where "smart" people were involved

Effect of Temperature and O-Ring Gland Finish on SealingAbility of Viton V747-75.

The Challenger Disaster: Deadly Engineering Mistakes

The temperature on that day was about -7 °C. This was the 10th flight of the Space Shuttle Challenger.

And other events in more recent history where "smart" people made mistakes, didn't listen to scrutiny or feedback, got rid of dissent, threw money fiat currency at a design based on reasons other than the design.

@ehaynes99
Copy link

Again, AST parsing is primitive compared to what a real interpreter can do. Nevertheless:

const objectAssignBroken = (script) => {
  const parsed = esprima.parseScript(script).body
  const iterator = parsed[Symbol.iterator]()
  
  let token = iterator.next()
  while (!token.done) {
    if (token.value.type === 'ExpressionStatement') {
      const { type, operator } = token.value.expression
      if (type === 'AssignmentExpression' && operator === '=') {
        if (token.value.expression.type === 'AssignmentExpression') {
          const { left, right } = token.value.expression
          if (left.object.type === 'Identifier' && left.object.name === 'Object') {
            const { property } = left
            if (property.type === 'BinaryExpression' && property.operator === '+') {
              if (property.left.type === 'Literal' && property.right.type === 'Literal') {
                const propertyName = property.left.value + property.right.value
                if (propertyName === 'assign') {
                  return true
                }
              }
            }
          }
        }
      }
    }
    token = iterator.next()
  }
  return false
}

Catches your example.

I'm not confident that I'm smarter or can easily come up with a better approach. I'm confident that, retroactively, we can see that their clever design didn't work out the way that they intended. Really smart people are really inclined to ignore those kinds of things after spending years designing and deciding on things when that happens.

Brendan Eich is far smarter than me, and yet... https://github.com/denysdovhan/wtfjs

James Gosling is likely far smarter than me. He was also likely the most destructive influence in the history of computer science and completely intractable, no matter how much evidence stacks up that Java style OOP was missing dynamic programming capabilities hackily provided by a reflection api (which has metasticized into reflect-metadata and "experimental decorators").

Really smart people came up with a broken standard. It's not arguable. It'll never be the standard as it is now.

If you have to resort to ad-hominem attacks about my intelligence, you're out of substantive things to say. See you in ten years when this still isn't resolved.

@smolinari
Copy link

And I say again, the hugely overarching main advantage to having a standard between the browser and node worlds, which makes practically all arguments about what is right or wrong mute...is

....having....a....single....standard.

Even if the chosen standard is flawed somewhat, having the one standard will always be better than having "competing standards". With one standard, everyone can concentrate on making that one standard better. With competing standards, everyone will continue to argue about which one is right or better and progress will always be hampered because of the wasteful discussions, as it was in the past, and as this thread continues to demonstrate.

I just love the idea of JavaScript being....finally....truly isomorphic. It will be the first ever language to do it and while doing it, be relatively "simple" to use too. ESM is a great step in the direction of that simplification and it will keep JavaScript in use for many years to come. Why? Because building applications with JavaScript will be the fastest and relatively easiest way compared to using any other backend language. I'm personally counting on it. 😁

Scott

@nabilfreeman
Copy link

I respectfully disagree... Browsers, servers and embedded applications are extremely different with very little overlap in use cases.

Yes, they share a common language which is already fantastic, and means developers can easily move between roles which were previously very different from each other.

Why is further standardisation required which hampers one platform at the expense of another?

I understand your idealism about creating the "ultimate language" that transcends everything. I had similar thoughts when I first picked up Node.js a decade ago.

But the discussions in this Gist are the perfect example of why this kind of idealism immediately becomes impractical as soon as it hits the real world.

The complaints here are real practical issues with the implementation of ES Modules which is holding back and discouraging adoption. It's because this standard was designed and shipped in a theoretical space without thinking about real world usage.

I think that is a very valid criticism and doesn't come from a place of orthodoxy.

The thing that many people are disappointed about is that JS was already making huge strides in cross-platform compatibility without significant breaking changes like this, and those efforts and achievements have been disregarded. It's not like the ecosystem has been static for 10 years and this is a necessary shake-up.

@guest271314
Copy link

Even if the chosen standard is flawed somewhat, having the one standard will always be better than having "competing standards".

The "standards" are competing.

I, too, would appreciate to be able to write the same code for use in Node.js, Deno, txiki.js, QuickJS, Bun. However, in practice none of those JavaScript runtimes implement I/O the same. Deno and Node.js implement fetch() observably differently; Bun's implementation of ReadableStream is not standard. Deno's and Node.js's executable bytes increases daily or weekly. Browsers are not different re compatibility in multiple Web API's; e.g., an audio MediaStreamTrack doesn't produce silence on Chrome per Media Capture and Streams specification; Mozilla expressed negative views about WICG File System Access API, even though File API, Drag and Drop, Clipboard API's, and HTML <a> and <form> elements and open() and achieve the same I/O.

Use the appropriate tool for the requirement. QuickJS for compiling a JavaScript runtime to WASM and embedding; Node.js for legacy stability, and packages if you depend on packages, or other reasons you would use node; Deno for Streams Standard, Fetch Standard, and Web API implementations.

We can utilize dynamic import(). Or CommonJS if you want. People are still using require() in the wild. Or roll your own and use that.

@leonsilicon
Copy link

leonsilicon commented Aug 14, 2023

The problem is, there was no standard in the browser for a module loading system. The browser would've had to implement its own standard because asynchronous module loading was a requirement for any sort of module loading support in a browser; that would've resulted in a separate standard on its own.

We all agree having one standard is better than having multiple separate ones. CommonJS could not be that standard for browsers. Inventing a new, separate standard that only works on the browser and one that only works for Node.js seems to be a worse compromise than agreeing on one standard (ESM) for both environments (and all future JavaScript environments), even if it takes longer for adoption in the short term.

@jimmywarting
Copy link

jimmywarting commented Aug 14, 2023

you could statically determine the names:

module.exports.init = () => {
  Object.assign(module.exports, {
    first: 'one',
    second: 'two'
  })
}

is the same as:

export let first
export let second

export const init = () => {
  first = 'one'
  second = 'two'
}

for most usecases cases it probably is... but In ESM, imports are live read-only views on exported-values. As a result, when you do import a from "somemodule";, you cannot assign to a no matter how you declare a in the module. So in a way ESM is safer than CJS, imagine two packages loading the same ESM module and one tries to monkey patch somemodule. that would not be such a good thing

However, since imported variables are live views, they do change according to the "raw" exported variable in exports. Consider the following code (borrowed from the reference article below):

    //------ lib.js ------
    export let counter = 3;
    export function incCounter() {
        counter++;
    }
    
    //------ main1.js ------
    import { counter, incCounter } from './lib.js';
    
    // The imported value `counter` is live
    console.log(counter); // 3
    incCounter();
    console.log(counter); // 4
    
    // The imported value can’t be changed
    counter++; // TypeError

As you can see, the difference really lies in lib.js, not main1.js.


To summarize:

  • You cannot assign to import-ed variables, no matter how you declare the corresponding variables in the module.
  • The traditional let-vs-const semantics applies to the declared variable in the module.
    • If the variable is declared const, it cannot be reassigned or rebound in anywhere.
    • If the variable is declared let, it can only be reassigned in the module (but not the user). If it is changed, the import-ed variable changes accordingly.

Reference:
http://exploringjs.com/es6/ch_modules.html#leanpub-auto-in-es6-imports-are-live-read-only-views-on-exported-values

@leonsilicon
Copy link

leonsilicon commented Aug 16, 2023

The most prolific advocate of ESM started forcing this a few years ago. His projects mostly look like this:

p-waterfall
Version Downloads (Last 7 Days) Published
3.0.0 (ESM) 105 2 years ago
2.1.1 (commonjs) 1,079,014 3 years ago

cherrypicked af

https://www.npmjs.com/package/is-port-reachable?activeTab=versions

@rubengmurray
Copy link

Funny this should be mentioned...

https://gist.github.com/joepie91/bca2fda868c1e8b2c2caf76af7dfcad3?permalink_comment_id=4662054#gistcomment-4662054

I ran into an ESM issue with is-port-reachable that cost me.

Ended up lifting the code directly from the npm package itself out into a .ts file in my own test suite to cut out the frustration.

Then: installing chalk@5 gives this wonderful error message:

✖ ERROR: TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /nn/tests/integration/setup/index.ts
    at new NodeError (node:internal/errors:400:5)
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:79:11)
    at defaultGetFormat (node:internal/modules/esm/get_format:121:38)
    at defaultLoad (node:internal/modules/esm/load:81:20)
    at nextLoad (node:internal/modules/esm/loader:163:28)
    at ESMLoader.load (node:internal/modules/esm/loader:605:26)
    at ESMLoader.moduleProvider (node:internal/modules/esm/loader:457:22)
    at new ModuleJob (node:internal/modules/esm/module_job:64:26)
    at ESMLoader.#createModuleJob (node:internal/modules/esm/loader:480:17)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:434:34) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'

Nothing in there about what's causing the issue... just some incidental mention of ESM.

After banging a few of our heads against the wall in the team we finally discovered that installing chalk@4 got things working 😭

It all feels completely unnecessarily vague and unproductive. Chalk 4 works just fine. It's just a coloured log outputter. Really awful developer experience.

🕐 🕥 😩

@nabilfreeman
Copy link

The key is to keep on CommonJS versions as there are still very few if any ESM updates to major libraries that have new functionality.

I use npx npm-check-updates -u to auto upgrade all packages, then inspect the package.json diff do a quick scan of the new node_modules to look for ES modules and exclude them from the upgrade.

I wish there was an argument npm-check-updates could accept like target or something where you could specify commonjs. That would be a good PR actually but I guess no way for it to know until it's actually downloaded and installed all the new versions.

@leonsilicon
Copy link

leonsilicon commented Aug 23, 2023

i will say that the tooling for esm is really underwhelming; ive had to basically implement my own tooling like https://github.com/Tunnel-Labs/loader to properly support ESM with typescript (since ts-node has a ton of issues with ESM)

for me the extra effort’s been worth it since having a TypeScript+ESM monorepo provides a much nicer DX especially when it comes to circular imports (ive found that in practice ESM handles circular import cycles much better than commonjs) but i totally understand the frustration for people who don’t give a shit about this ESM vs CommonJS stuff and just want to write JavaScript and TypeScript lol

@jimmywarting
Copy link

jimmywarting commented Aug 23, 2023

I get the feeling that the most frustration from this thread seems to come from ppl using TypeScript and not so much from vanilla javascript users

@guest271314
Copy link

@jimmywarting I think the majority of the people who have issues with ESM as Node.js users, because node is set to use CommonJS by default.

Not an issue when using deno, qjs, tjs, orbun which use Ecmascript Modules by default.

In 2023 we can use esbuild or even deno_emit to produce a ESM version of a package. I usually just do that by hand if necessary.

@EhudKirsh
Copy link

ES Modules are crap!
Finally, someone talking my language and down to earth.

ES Modules need to be deprecated like they essentially deprecated 'require'. 'require' just works! I like simplicity. This 'import' BS doesn't work for me, even with this 'await' nonsense.

I don't want stupid features, I just want code to simply work!

@sheerlox
Copy link

sheerlox commented Nov 6, 2023

I suggest all upset and uninformed people of this thread (and author) read the very insightful comments under Sindre's "Pure ESM package" gist, especially this one, but more generally: go learn some new thing and stop hating ❤️

thanks for the entertainment though

@guest271314
Copy link

Node.js finally got around to providing a means to use Ecmascript Modules without naming the file with an .mjs extension when no package.json file exists on the file system:

#!/usr/bin/env -S ./node --experimental-default-type=module

If people want to use CommonJS they can still do that.

@4skinSkywalker
Copy link

4skinSkywalker commented Nov 11, 2023

I like ES Modules syntax (except the need of the extension in import x from './index.js';) but I find it frustrating to ALWAYS have to deal with incompatibility issue that I have to fix in various ways in order to run projects that depend on multiple 3rd party's libraries.

Whoever pushed toward this standardization analized the matter through purely technical lenses. The purely technical standpoint took into the account the "20% faster" and "40% lighter", but it didn't take into the account the hundreds of thousands (if not more) of hours of brain energy lost because of Error [ERR_REQUIRE_ESM]: require() of ES Module -> SyntaxError: Cannot use import statement outside a module. -> "type": "module" (in your package.json) -> TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" -> [...and so on and so on...].

@guest271314
Copy link

@4skinSkywalker That's only n issue in Node.js world. Not an issue when using Deno, Bun, QuickJS, txiki.js, SpiderMonkey.

@nabilfreeman
Copy link

@guest271314 yes, of course not a problem, because the technologies you listed are the industry standard. 🙄

@4skinSkywalker
Copy link

@guest271314 good luck convincing banks and assurance companys to switch from Node.js to whatever of the above.

@guest271314
Copy link

@4skinSkywalker In the case of Deno it's the same V8 JavaScript engine. In the case of QuickJS we're taking about the author of FFMpeg, TCC, et al. Fastly and others use SpiderMonkey. Bun actually puts effort in to listening to developers and fixing bugs. txiki.js is QuickJS with libuv and WASM3.

I didn't know the job was to convince banks and assurance companies to use the JavaScript runtime. Banks and insurance know how to fuck shit up whatever programming language is used. See synthetic derivatives and Goldman Sachs shorting Lehman Brothers because they are all greedy bastards. Nothing to do with any programming language.

@mk-pmb
Copy link

mk-pmb commented Nov 12, 2023

@4skinSkywalker Wait what, Node.js can run COBOL nowadays? 😮

@guest271314
Copy link

@4skinSkywalker The last time I checked jsDelivr, unpkg and other CDN's provide Ecmascript Module versions of Node.js packages. Using Deno and Bun or even browserify we can bundle dependencies into a single script, e.g., https://gist.github.com/guest271314/252b64b3f06593434fae209b3dc4303f.

@4skinSkywalker
Copy link

4skinSkywalker commented Nov 12, 2023

What I wanted to say by referring to banks and assurance companys is that there are certain customers that are rather inflexible and won't allow the new shiny thing in their repertoire of technologies, you can tell by the fact that the banking industry is still working with COBOL code. Good luck telling them to switch their Node.js-based intranet dashboards to use Bun instead.

@guest271314
Copy link

Why would banks be using Node.js at all?

Deno and Node.js use the same JavaScript engine: V8.

I'm not talking about the arbitrary decisions corporations make. I'm talking about the technical fact that you can convert any Node.js package or dependency to Ecmascript Modules - instead of using CommonJS loader system, that is mainly a Node.js thing.

@guest271314
Copy link

Good luck telling them to switch their Node.js-based intranet dashboards to use Bun instead.

Municipalities and governments tend to have contracts with Microsoft to supply software. That doesn't make Windows and Internet Explorer superior to Red Hat Linux or Debian or Ubuntu and Firefox.

You are talking about marketing, not technology.

@nabilfreeman
Copy link

We don’t even need to be talking about large/legacy orgs - experienced startups do not want to mess around with the bleeding edge and will instead go for proven technologies with LTS support

@rubengmurray
Copy link

@4skinSkywalker That's only n issue in Node.js world. Not an issue when using Deno, Bun, QuickJS, txiki.js, SpiderMonkey.

You mean ONLY an issue for 99% of users rather than 1%? Ah yes, not a problem at all.

@guest271314
Copy link

You mean ONLY an issue for 99% of users rather than 1%? Ah yes, not a problem at all.

It's not an issue at all, really.

Just use jsDelivr to get Ecmascript Module versions of your dependencies if you don't know how to do that yourself using esbuild, et al.

People can't on the one hand claim to be so entrenched in Node.js world that they have all these dependencies, then claim they don't have the competency to convert their code to use Ecmascript Modules.

If people want to use CommonJS, they can. That's the default module loader in Node.js world.

However, the world outside of Node.js doesn't revolve around or wait for Node.js maintainers to do stuff. That is, the very individual who created Node.js created Deno. That kind of tells us something about the state of Node.js.

I don't know how 99% of people can amass all of these dependencies, then not know how to convert those dependencies to Ecmascript Modules when there's plenty of tools to do that?

@guest271314
Copy link

Re

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" -> [...and so on and so on...].

here you go tsm: CDN<TS => JS>.

Node.js doesn't own JavaScript, Oracle does. Node.js doesn't specifiy JavaScript, TC39 does in ECMA-262.

@4skinSkywalker
Copy link

It's not an issue at all, really.

Yes it is, and that's because this situation is creating difficulties, wasted time and brain energy for the vast majority of js consumers. Before this standardization everything was an npm install away, now it's way less plug-and-play.

they have all these dependencies, then claim they don't have the competency to convert their code to use Ecmascript Modules.

I don't think you have realized the huge transaction cost people (and company) would have to pay. If you look at the problem at large it's not about competency but economical expense.

However, the world outside of Node.js doesn't revolve around or wait for Node.js maintainers to do stuff.

OK, but the Node.js world is huge, infact the Node Package Manager is the largest software registry in existence.

I don't know how 99% of people can amass all of these dependencies

Why do you even care?

then not know how to convert those dependencies to Ecmascript Modules when there's plenty of tools to do that?

Why should they? Is there a 10x gain in doing that?

@guest271314
Copy link

Yes it is, and that's because this situation is creating difficulties, wasted time and brain energy for the vast majority of js consumers. Before this standardization everything was an npm install away, now it's way less plug-and-play.

No, it's not.

Pick a side.

This is not 2009. People are compiling QuickJS to WASM and running that JavaScript runtime in a WASI environment. QuickJS is ECMA-262 compliant and after strip is less than 1 MB. node Nightly release from yesterday is 97.6 MB; deno executable is 126.6 MB; bun executable is 87.0 MB. Clearly developers using JavaScript for an embedded microcontroller or as a serverless javaScript runtime in a WASI environment are not choosing node - or deno.

I don't think you have realized the huge transaction cost people (and company) would have to pay. If you look at the problem at large it's not about competency but economical expense.

I don't get it.

CommonJS is over. That's a Node.js legacy thing. If you want to maintain module loader system, you are making the economical decision to do when when the rest of JavaScript world is migrating to or already has migrated to Ecmascript Modules. Though Bun does still use CommonJS (require()) and have published articles on the matter; though does not implement all of Node.js's internal modules; e.g., node:fs/promises.

If you want CommonJS to not be over in your organization, you are going to have to pay for that, with time and energy, which is what economics is.

What you are not going to do is convince everybody outside of your organization to undo adoption of Ecmascript Modules. You can try. That's gonna cost you, too. Do an experiment to see which approach costs you the most.

The solution is bundling your dependedncies to Ecmascript Modules.

Unless you philosophically or technically protest adoption of Ecmascript Modules.

Then it's back to the 2d sentence: Pick a side.

Both sides will cost you.

OK, but the Node.js world is huge, infact the Node Package Manager is the largest software registry in existence.

"NPM" does not mean "Node Package Manager" https://www.npmjs.com/package/npm?activeTab=readme

Contrary to popular belief, npm is not in fact an acronym for "Node Package Manager"

and GitHub owns NPM, not Node.js.

When you fetch Node.js nightly archive, which I do every day or so, you will be warned that the npm in the archive is not the lastest version, and there are no plans to change that - because Node.js organization does not own or control npm (NPM) - GitHub does, [BUG] npm in Node.js nightly release consistently prints warning messages #6820.

Why do you even care?

I don't, really. I test and experiment using multiple JavaScript engines and runtimes. When you do that you'll immediately notice Node.js is different than the rest. Ecmascript Modules are supported out of the box. No "package.js", no third-party package script. You generally just get the executable. Packages are a different realm from the JavaScript runtime itself.

i just marvel at the X, Y problem. Pick a side. Be done with the matter.

Why should they? Is there a 10x gain in doing that?

To solve the crises developers and organizations have created for themselves by relying and depending solely on Node.js philosophy and third-party packages, clearly omitting to hedge against technologies outside of Node.js influence usurping Node.js initial monopoly of non-browser JavaScript runtimes.

So you gain by setlling the matter. Litigation costs. You are litigating into oblivion.

@guest271314
Copy link

@4skinSkywalker If you think the CommonJS/Ecmascript Module schism is a mess try using the same I/O (stdin, stdout, stderr) JavaScript code to run in a Node.js, Deno, QuickJS, and Bun environment Common I/O (stdin/stdout/stderr) module specification #47.

@ehaynes99
Copy link

I suggest all upset and uninformed people of this thread (and author) read the very insightful comments under Sindre's "Pure ESM package" gist, especially this one, but more generally: go learn some new thing and stop hating ❤️

There's nothing insightful there. It's the same tired rhetoric repeated throughout this thread. This part is particularly amusing:

once they support ESM properly we'll be unblocked from a mad scramble to ESM over the next year or so. In 2 years, we might have an ecosystem of maintained packages that are almost entirely pure ESM.

That post was 23 months ago.

Sindre's original post there kept getting longer and longer, as he kept having to explain more and more things that break during the switch.

But no need to care about my opionion. Since Sindre has chosen to be the champion for ESM, and people often reference his choice (and since I just noticed someone accuse me of cherry picking above) I thought I would get some real data. So I threw together this little script: https://github.com/ehaynes99/sindres-quest. It:

  • downloads the list of all projects returned by the npmjs registry api's search endpoint with author:sindresorhus
  • downloads the extended metadata for each project from the registry api's package endpoint. This includes the tarball path of each version of each package
  • downloads the tarball, extracts the package.json, and checks whether type === 'module'
  • downloads the list of download counts for the last week, which is the same data as shows on the "versions" tab on a package's page on https://npmjs.com
  • sums up the total of downloads, partitioned by ESM/commonjs, and prints out the totals

Results

As of Nov 12, 2023, the results are as follows:

  • number of packages: 857
  • total number of versions: 8,609
  • total downloads: 6,167,976,795
  • total downloads of ESM versions: 439,771,975
  • total downloads of CJS versions: 5,728,204,820
  • percentage of downloads that are ESM versions: 7.13%

Make of it what you will, but we're not even close after roughly 100 months since ES2015 was released.

@leonsilicon
Copy link

leonsilicon commented Nov 13, 2023

Most developers are so dependant on build tooling they don't actually know how to write a CJS module; millions of developers have their source as ESM. To them, CJS is an implementation detail of Babel, Next.js and Node.js. Most of the installs on npm flow from a few popular libraries/frameworks that suck in an ungodly amount of dependencies on install. If one or two switch to pure ESM and update their dependencies to newer pure ESM versions, overnight millions of CJS installations evaporate.

from https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c?permalink_comment_id=3992076#gistcomment-3992076

and yes, choosing a package with a 105 to 1,079,014 download ratio (which is 0.01% of downloads compared to your script's 7.13%) as an accurate representation of the CJS/ESM download counts seems pretty cherrypicked to me

fwiw i just took a look at p-waterfall's dependents and the reason why there's a million installs for version 2.1.0 is because it's depended on by lerna, supporting the reason for the skewed download counts i referenced above:

image

@guest271314
Copy link

but we're not even close after roughly 100 months since ES2015 was released.

That's just in Node.js world, correct?

It seems to me that the Node.js folks can simply convert the entirety of NPM packages to Ecmascript Modules and be done with the schism.

That appears to be the only actionable path forward.

People outside of Node.js world are not clamouring for CommonJS. They just want to convert the CommonJS third-party package they depend on to Ecmascript Modules in a reliable process.

@guest271314
Copy link

Does nobody trust jsDelivr to provide their packages as Ecmascript Modules?

I have a difficult time understanding what the primary issue is. It reads like a philosophical debate rather than attempts and failures at solving an articulated problem: Legacy third-party code that uses CommonJS written for a Node.js environment where there are multiple JavaScript runtimes besides Node.js that are not tethered to CommonJS.

Help me understand how seasoned Node.js developers cannot solve this on their own.

@ehaynes99
Copy link

ehaynes99 commented Nov 13, 2023

fwiw i just took a look at p-waterfall's dependents and the reason why there's a million installs for version 2.1.0 is because it's depended on by lerna

And? That's an integral part of a tremendous number of projects. Even if it's refactored to ESM by it's new NX owners, they'll push far too many breaking changes for it to be worth it for fully functional projects to update. We're a decade out on that, guaranteed.

Even without lerna, it's a lib that could be as useful in node as it is in the browser, and thus should not be ESM only. Either way, if you think I went out of my way to pick that one, then feel free, but it just happened to be the most recent of my collisions with his libraries. I have exceptions in dependabot for chalk, ora, globby to name a few, as well as overrides in multiple package files, and for downstream fallout like inquirer as well (actually, still my belief: SBoudrias/Inquirer.js#1159 (comment)).

But again, that's why I just went ahead and generated the real data. You can't really argue with the 7%. It's abysmal.

It seems to me that the Node.js folks can simply convert the entirety of NPM packages to Ecmascript Modules and be done with the schism.

Then you are FAR too inexperienced in the industry to have a meaningful opinion. "Just rewrite all of your code" is what they said for python2 -> python3. "It'll be quick", they said. While we finally saw some linux distros drop it after 13 years, it's still not completely solved. It's the death knell of entire companies. You're literally asking an entire ecosystem with software that works to dedicate hundreds of billions of dollars for rewrites just because of your quasi-religious belief that in a "standard".

I've spent most of my career refactoring, TS for the last 3 years, half a dozen others for the 14 years prior. Strict TS, linting, test coverage, monorepos... the list goes on. It takes months, if not years. But I also spent years consulting for Fortune <20-1000>, and there's not a chance in hell that any of them are going to pay a second time for a product that already works. Go ahead and insert some anti-corporate nonsense here like the post cited above. You'll help prove my point.

If you work on small projects -- especially those with "new framework-itis" -- it might seem like a small thing to just "convert the entirety" because some other dude on the Internet said "trust me bro, it's the future".

Help me understand how seasoned Node.js developers cannot solve this on their own.

Well, just read the list.

Otherwise, I've already. stated. them. before

But you're missing the point. nodejs cannot be ESM only until EVERY SINGLE NODE APP IS ESM ONLY. They have a contract of backwards compatibility, as every software platform should. ECMA decided to fix half of the problem and break the other half that already worked. Thus, we'll never get past this without revisions of that broken spec.

You can read my comments above. I'm not suggesting that commonjs is better. I don't think that there is even much contention that ESM syntax is better. But the ESM spec enforced runtime constraints that were -- and still are -- incompatible with half of the ecosystem. It's not going to just work itself out. Post a date when it'll be complete here, and I'll cite it in a post years from now like above when it doesn't happen. ESM needs to change, or it will eventually overtake the python conversion as "the worst decision smart people ever made" in the software industry.

I have a difficult time understanding what the primary issue is.

Thank you. Yes, that's the primary issue.

@guest271314
Copy link

Then you are FAR too inexperienced in the industry to have a meaningful opinion. "Just rewrite all of your code" is what they said for python2 -> python3.

Yes, you have to rewrite code. We've known that since at least Moore's Law. You have 18 months of monopoly, if you're lucky and nobody rearranges your gear and markets your product to a different target demographic than you think is important.

It is certainly possible to convert the entire NPM third-party package system to Ecmascript Modules. jsDelivr already does that for you.

Technology is not static. Nobody is waiting around for what Node.js management is going to do.

Google hasn't updated it's code from Python 2 either. And is publishing broken code after being notified about said code GoogleChrome/chrome-extensions-samples#805. There's also some "whitelisted" and "blacklisted" language still in current Chromium source code that Google Safe Browsing folks claim is too much work to get rid of, even being aware of Chromium source code deliberately getting rid of that garbage. It's called laziness.

"the worst decision smart people ever made" in the software industry.

Alleged smart people make horrible decisions all of the time.

N.A.S.A. launched the space shuttle when they knew the O-rings they were relying on expanded and contracted based on temperature. It was cold. N.A.S.A. launched anyway. And so forth.

You have a solution that is achievable. The third-party packages hosted by GitHub-owned NPM are finite. Convert them all to Ecmascript Module version.

Politicking into oblivion has not changed anything. And Node.js is moving towards Ecamascript Modules, not away from Ecamascript Modules with new "experimental" flags.

Corporations come and go. E.I.C., the first "modern corporation" has it's own private army. Yale University is named after the sacked pres. of E.I.C., Elihu Yale. Then they got too big for their britches.

@guest271314
Copy link

They have a contract of backwards compatibility,

Wait a minute. Who are you talking about?

Are you talking about a contract between third-party software developers and the corporations who buy their gear?

You can't be talking about NPM, now owned by GitHub claiming that CommonJS would be the only module loader in all of the third-party software hosted by NPM?

The node executable happily loads CommonJS as the default loader. You have to deliberately use Ecamascript Modules. Otherwise you've got your default CommonJS loader.

@guest271314
Copy link

@titanism

// dynamically import ESM module
let esmModule;
import('some-esm-module').then((obj) => {
  esmModule = obj;
});

I was going to suggest you should be able to do

const {resolve, reject, promise:esmModule} = Promise.withResolvers();
// dynamically import ESM module
import('some-esm-module').then(resolve);

However, node v22.0.0-nightly202311135e250bd726 has not implemented Promise.withResolvers(). bun version 1.0.11 and deno 1.38.1 have.

@leonsilicon
Copy link

But the ESM spec enforced runtime constraints that were -- and still are -- incompatible with half of the ecosystem.

No it didn’t; take a look at Bun.js which implemented support for cross-loading of CommonJS and ESM. The incompatibility stems from Node.js’s implementation, not the spec.

@guest271314
Copy link

@leondreamed This can't be about the executable itself supporting Ecmascript Modules or not. I downloaded the nightly archive and got rid of everything except the node executable for a while. Ecmascript Module support without a package.json file on the machine required .mjs extension to use import. In node 22 these options are available

  --experimental-default-type=...
                              set module system to use by default
  --experimental-detect-module
                              when ambiguous modules fail to evaluate
                              because they contain ES module syntax,
                              try again to evaluate them as ES
                              modules
  --experimental-import-meta-resolve
                              experimental ES Module
                              import.meta.resolve() parentURL support
  --loader, --experimental-loader=...
                              use the specified module as a custom
                              loader

The "ecosystem" are third-party software authors - this ain't about the node executable itself. node executable support CommonJS require() out of the box - without any package.json file or node_modules folder present on the system. So why are people mad at Node.js and TC39?

@ehaynes99
Copy link

Wait a minute. Who are you talking about?

Are you talking about a contract between third-party software developers and the corporations who buy their gear?

You can't be talking about NPM, now owned by GitHub claiming that CommonJS would be the only module loader in all of the third-party software hosted by NPM?

I'm talking about the nodejs project. They're not going to remove require, so it will never be ESM only.

Alleged smart people make horrible decisions all of the time.

That's been my argument all along. Allegedly smart people created the ESM spec, and yet almost a decade later, we're still dealing with the consequences of the design.

And Node.js is moving towards Ecamascript Modules, not away from Ecamascript Modules with new "experimental" flags.

Yes, movING and EXPERIMENTAL, which means "it's definitely not time to convert libraries targeting node yet". It's still marked experimental in next year's v22.

But the ESM spec enforced runtime constraints that were -- and still are -- incompatible with half of the ecosystem.

No it didn’t; take a look at Bun.js which implemented support for cross-loading of CommonJS and ESM. The incompatibility stems from Node.js’s implementation, not the spec.

node's implementation existed long before they even started dreaming about ESM, so yes, they absolutely created a spec that was incompatible. They assumed it would be easy for node to "just rewrite everything". They were wrong. Hell, it took years for even the browsers to support it, so the vast majority of frontend apps are still bundled to this day. Practically no one actually loading imports over the wire, which was the whole motivation for making async loading part of the spec to begin with.

And of course bun can implement the spec, because bun was created yesterday. But now you're making the rewrite even bigger, changing the entire platform, which then ripples throughout the whole codebase.

Have you ever done a large rewrite? They always take way longer than anyone anticipates. Things break. You interrupt your business and spend tons of developer hours. Sometimes, it's worthwhile if the result is easier to maintain, safer, or more performant. But in converting an existing application to ESM, you would get... NOTHING. Best case scenario, you end up with an application that works exactly the same as it did before. If you want to do it for personal projects, knock yourself out, but for a business, spending a bucket of cash on an endeavor with large amounts of risk and zero reward would be foolish.

Then there's the nature of the refactor itself. You can't iterate on this. For a plain JS node application, you would have to make all of the changes in one giant commit. Even if you're using a transpiler or TypeScript for module conversion, you might be able to iterate on updating all of the individual files, but you still have a big checklist of stuff to change when you finally flip the type flag. Oh, and you have to rework all of your jest tests to use unstable_mockModule, so you can't even rely on those mid-refactor. All while tracking changes in an actively developed project. With multiple projects, you have to do them in topological order of dependencies. All of the apps have to be converted before any of the libraries, and if libraries depend on each other, then you have to iterate dependents -> dependencies all the way through them all.

And for all that, you end up with a codebase littered with "unstable" this and "experimental" that without ANY functional improvement at all.

@rubengmurray
Copy link

Here here, Eric.

I’ll never see the beauty of top level await in existing services, but for the absolute hell i’d have to go through to get there, I don’t care.

I’ve already probably lost more hours to ESM than the code I work on would benefit from moving it. I could have been writing a feature, or making an optimisation, but instead I was wasting time on incompatibility issues. Truly awful.

@guest271314
Copy link

I'm talking about the nodejs project. They're not going to remove require, so it will never be ESM only.

Right. You are talking about what you see as a Node.js specific issue. Not a JavaScript issue.

The JavaScript programming language does not revolve around Node.js. There are multiple JavaScript engines and runtimes that are not dependent on what Node.js does.

Allegedly smart people created the ESM spec, and yet almost a decade later, we're still dealing with the consequences of the design.

ECMA-262 specifies the syntax and semantics of the JavaScript programming language - not Node.js.

Have you ever done a large rewrite?

Yes. I usually start with the understanding there are multiple JavaScript engines, runtimes, and environments.

I wrote five (5) different JavaScript Native Messaging hosts. ECMA-262 does not specify I/O. So no two (2) JavaScript engines or runtimes implement reading stdin, writing to stdout or reading stderr the same. One for Node.js, one for Deno, one for Bun, onefor txiki.js, one for QuickJS https://github.com/guest271314/NativeMessagingHosts.

In the Web API world there are V8, SpiderMonkey, JavaScriptCore. It is not uncommon to write HTML, CSS, and use Web API's that target each environment. There is no hegemony in JavaScript world. Node.js folks don't run JavaScript.

What you are describing is not novel. In the Web API world there are multiple stackeholders. W3C, WHATWG, WICG. Chromium could easily be updated multiple times per day. Trying keeping up with that.

Chromium's implementation of MediaStreamTrack of kind is not compliant with the controlling W3C Media Capture and Streams specification; the track does not produce silence. That means all downstream browsers are broken on that API, too; Brave, Edge, etc.

Firefox still has not implemented W3C Media Capture From Element captureStream(); nor navigator.permissions.request().

Chromium authors do whatever they want. a WICG Drafts have been implemented and shipped. Chromium authors ain;t sitting around waitin on Mozilla folks to agree with them. The topic might come up in compat discussions, but that ain't stopping Chrome from shipping Isolated Web Apps, TCPSocket() in the browser per Direct Sockets. WebTransport took a while to be implemented in Firefox, so did Transferabl Streams. Look the folks who champion that Isolated Web Apps, they are still using CommonJS https://github.com/GoogleChromeLabs/telnet-client.

Node.js is just part of the JavaScript world. Not the JavaScript world. Why and how do you think Deno came about?

Anyway, you have choices.

There's really nothing you can do except hack yourself out of the mental and philosophical hole you have dug for yourself. As the late U.S. Sec'y of Defense Donald Rumsfeld once remarked, the first thing you do when you are in a hole is top digging. You are still digging in the same hole instead of backfilling and building on top of that leveled earth.

@ehaynes99
Copy link

ehaynes99 commented Nov 15, 2023

To be clear, you're asking everyone in the world who has a working application who doesn't give two shits about the success of ESM to just suck it up, spend untold amounts of developer hours (let's ballpark it at $150 each), take on a bunch of risk, gain absolutely nothing from the effort, and finish with a result that only benefits the "true believers" in this "standard". And somehow I'm the one in the philosophical hole?

I have no "side" here. I think they both have problems, neither is going away, and we'll never settle on one of these particular 2 choices. There is ZERO advantage to ESM on the server side. None. It's marginally worse, TBH, but let's just say "after a bunch of work, it's the same".

FWIW, all 50 of the projects I help manage are in a consistent enough state that I could quite likely script the whole conversion and be done with it. But it took me a year to get them there in between "real work", with help for the last 4-6 months. So let's pose a hypothetical: One day, someone is writing a little console app for use by other developers, and thinks it would be useful to add a bit of color to some output. They find the colors library, but then realize that the author went insane, and settle on the de-facto alternative, chalk.

But wait, it's ESM only! Should they:

  • use chalk@^4
  • convince the entire company that it's time to move the whole thing to deno... or is it bun... or whatever flavor of the week node alternative with less than 1% market share popped up in the last 30 days.

Get real. It's just not the reality for a ton of companies/projects. They have customers. They have a reputation. They have mouths to feed.

You're also ex post facto blaming things that existing long before ECMA started dreaming about their glorious new spec. At the time:

  • node was 100% of the server side JS
  • the Node Package Manager was the only viable package manager
  • practically all of the software developed for web at the time was developed in a node environment and bundled to cjs too (hell, most of it still is)

So yes, when defining a new spec for ALL modules, they absolutely owed a hat tip to the only module system that had ever stabilized the carnage of CDNs and global everything.

It's not as if there is some constant form of pushback against everything ECMA does. The other features they've developed are adopted rapidly in node, browsers, TypeScript, and all of those other environments. Any code written in the last few years is almost certainly using async/await rather the Promise api directly, but they didn't make you choose between one or the other. They usually get it right. Here they did not.

The rest of your post... I don't know what any of that has to do with this. Yes, browsers are historically inconsistent. Some of them decide some feature you want aren't that important. It's not some fundamental language feature that's supposed to be "the standard".

The "true believers" have been saying for 8 years now that "it ain't broke, so don't fix it". I've wasted way too much time on this thread, but a rhetorical question... How long will it take before you realize that just maybe, they didn't shit gold on this one? We're already pretty close to the decade mark, so that can't be it. Is it 15? 20? 100?

@ehaynes99
Copy link

Here here, Eric.

I’ll never see the beauty of top level await in existing services, but for the absolute hell i’d have to go through to get there, I don’t care.

I don't even see it as a feature. Top level await is a side effect on import, which is a horrible pattern. Modules should never "do stuff" when you load them, only provide values, some of whom can be functions that "do stuff".

// some-file.js
await http.get('not-a-real-address') // explodes
// whatever.js
import 'some-file.js' // nothing I can do, because the `Promise` is a "secret". I don't `await`

vs a perfectly reasonable way to load things

export const doStuff = async () => {
  await http.get('not-a-real-address') // explodes
}
// whatever.js
import { doStuff } from 'some-file.js'

const whatever = async () => {
  try {
    await doStuff()
  } catch (error) {
    console.log('I don\'t want my app to die!!!!!!!!!!!!!!!!!')
  }
}

Async import could still work just fine with await import('some-file.js'), and MUCH more importantly, node could just rewrite require to implement it in terms of import. Everything could have shifted to ESM by early 2016.

@guest271314
Copy link

FWIW, all 50 of the projects I help manage are in a consistent enough state that I could quite likely script the whole conversion and be done with it.

That's your answer.

@ehaynes99
Copy link

FWIW, all 50 of the projects I help manage are in a consistent enough state that I could quite likely script the whole conversion and be done with it.

That's your answer.

I don't need an answer to your problem.

@guest271314
Copy link

I don't even see it as a feature. Top level await is a side effect on import, which is a horrible pattern. Modules should never "do stuff" when you load them, only provide values,

That's rather opinionated. Those are just your personal preferences.

How does the module output the value to you without doing stuff?

// whatever.js
import 'some-file.js' // nothing I can do, because the `Promise` is a "secret". I don't `await`

You absolutely can do something.

Intercept and modify the request with a ServiceWorker.

Serve JSON using import assertions, then you get values that you can do stuff to afterwards.

vs a perfectly reasonable way to load things

Then do that in your own code?

I don't need an answer to your problem.

I thought you were trying to massage a problem statement out of your personal preferences and failure to convert your own code to the module loader you persoanally prefer.

I don't have a problem with Ecmascript Modules. If a repository uses CommonsJS, I use the loader they use uness I'm targeting a specific environment.

If you think JavaScript should do this or that you can certainly roll your own JavaScript runtime from scratch to do whatever you write out, or fork an existing JavaScript engine or run time and write it out - implement the detail that you prefer, for yourself.

@leonsilicon
Copy link

leonsilicon commented Nov 18, 2023

Honestly, I think it boils down to this:

  • Browsers needed a JavaScript module loading standard
  • The module loading standard adopted by browsers cannot be synchronous (and thus couldn't be CommonJS)
  • ESM was that standard designed for browsers
    • Top-level await is important for browsers because bandwidth matters, and browsers should be able to dynamically import and export modules across the network on the top level (for example, a production vs development top-level export based on the window.location.href value, just like how CommonJS can dynamically export different values based on the process.env.NODE_ENV value)
  • A single JavaScript module standard for all runtimes is better than a separate one for browsers and server-side JS like Node.js, so ESM was developed to be that unified standard that was embedded into the JavaScript spec
  • Node.js decided to try and move the ecosystem exclusively towards ESM, which involved preventing CJS users from importing ESM that led to this "require(esm)" problem

ESM itself isn't the problem, because browsers needed a module system that was optimized for browsers. There was inevitably going to be two separate module systems (an asynchronous one and a synchronous one). The problem largely stems from Node.js deciding to move the ecosystem exclusively towards the asynchronous module system for the benefit of a single universal module system (an alternative approach such as the one by Bun.js is to provide first-class support both module systems, and only error when they are used in incompatible ways, such as the synchronous module loading system trying to synchronously import a file using the asynchronous module loading system that uses top-level await).

@guest271314
Copy link

I really don't see a problem statement here. This is not 2009. Node.js is not the only JavaScript runtime that does not target the browser.

Circa 2023 there are more JavaScript runtimes that do not target the browser than those that do.

We have options. You can Roll your own JavaScript runtime and do whatever you want, like CloudFlare's Workerd, WasmEdge's QuickJS server implementation, Wasmer's WinterJS, dune, VM Labs WASM Workers Server, Fastly that uses SpiderMonkey, etc.

Just look at one way Node.js documents reading stdin. An event listener .on("readable") where in the handler there is a while loop. Now, how is the result supposed to be returned from that? I think it is nostalgic to think of Node.js as the standard for non-browser JavaScript. However, if developers are not constantly testing and running multiple JavaScript engines and runtimes, I can see how people can become myopic.

@ehaynes99
Copy link

The module loading standard adopted by browsers cannot be synchronous

We've already covered this. Again, ESM doesn't even expose the fact that it's async to you at all. The behavior is exactly the same.

// I'm about to be blocked, waiting for an arbitrary hierarchy of files to be loaded
import { someThing } from './some-module.js'
// flow of execution does not move to here until the magic promise is fulfilled

It's a distinction without a difference.

Top-level await is important for browsers because bandwidth matters, and browsers should be able to dynamically import and export modules across the network on the top level (for example, a production vs development top-level export based on the window.location.href value

Top level await is JUST SYNTAX. It doesn't provide any functionality that you couldn't have without it. Applications have been doing code splitting for years without it, or before await existed at all. You can't even use an import statement for that, only the import function.

const appModule = window.location.href.startsWith('http://localhost')
  ?'./development-application.js' 
  : './production-application.js'

// without top level await
import(appModule)
  .then((app) => app.start())
  .catch(console.error)

// with top level await
try {
  const app = await import(appModule)
  app.start()
} catch (error) {
  console.error(error)
}

A single JavaScript module standard for all runtimes is better than a separate one for browsers and server-side JS like Node.js, so ESM was developed to be that unified standard that was embedded into the JavaScript spec

Couldn't agree more. It would have been great for them to make a spec like that.

I thought you were trying to massage a problem statement out of your personal preferences and failure to convert your own code to the module loader you persoanally prefer.

Sorry, you thought wrong. I'm not trying to solve some problem. My personal preferences don't matter, other than that I sure would prefer if we actually had a standard, rather than just pretending we do. I'm pointing out real reasons why this didn't go as planned, and how you shouldn't hold your breath for it work out any time soon. I'm trying to point out how tiny of a change to the spec would actually be required to allow it to become the standard, rather than one of two disparate systems that's making glacially slow gains.

You've been continually pointing out differences in implementations. You are correct: ECMA didn't define the standard for interacting with stdio. They allowed flexibility in how the implementations interacted with the underlying operating system. THAT'S WHAT WE NEED HERE. Imports are just a mechanism of loading files. Whether they're loaded from disk, over a network, or if it sends a letter in the mail and waits for someone to mail the result back. No matter how the content is provided, when the interpreter runs in to an import statement, it stops, asks the implementation to load the content of the uri as a string, interprets it, assigns the result to variables, and proceeds with processing the file.

You wouldn't even have to give up any functionality. Absolutely nothing about how you build software, how it behaves, or what you can accomplish would change. It's one dinky piece of syntax that's just obfuscating a Promise when the module could just export the Promise, or better yet, a function that returns a Promise. You're literally just taking a function body and dropping it into the root scope of the module instead, and replacing its call signature with an import statement. That's it. That's all you would have to give up so that we could actually have a standard. Node could rewrite require to use ESM under the hood. It would be trivial. The divide would finally be over.

@leonsilicon
Copy link

leonsilicon commented Nov 19, 2023

We've already covered this. Again, ESM doesn't even expose the fact that it's async to you at all. The behavior is exactly the same.

await import(…) does

// without top level await
import(appModule)
.then((app) => app.start())
.catch(console.error)

This is not the equivalent of:

// react/index.js
if (process.env.NODE_ENV === 'production') {
  module.exports = require('./react.production.js')
} else {
  module.exports = require('./react.development.js')
}

@divmgl
Copy link

divmgl commented Nov 19, 2023

Anyone who is arguing about the syntax of ESM are unable to see the forest for the trees. The issue is Node's handling of the ESM migration. Bun proves that it's definitely possible to have both APIs and still have a smooth migration between the two.

Node is a mess now. Add in the complexity from TypeScript and monorepos and you're looking at hours wasted, especially if you've inherited a legacy CommonJS codebase. Were it not for the amazing community packages (right now really enjoying using tsx, tRPC and Prisma) I probably wouldn't use Node. While Bun is still super immature, it is a sight to behold. It's so freaking fast and starting new projects in it feels amazing. It feels like what Node was during the CommonJS era. Also Bun's package manager feels 100x faster than npm and the runtime includes a testing library (no more Jest vs Vitest debates).

I'm very grateful for Node and the Node Foundation but I also think it's important to be critical when something isn't working out.

@guest271314
Copy link

My personal preferences don't matter, other than that I sure would prefer if we actually had a standard, rather than just pretending we do.

Then you are talking about your personal preferences.

Node.js is not the only JavaScript runtime. The Node.js folks are aware of the schism and have taken some steps to make all stakeholders happy, https://nodejs.org/en/blog/announcements/v21-release-announce#esm---experimental-default-type-flag-to-flip-module-defaults.

We are also exploring using detection of ES module syntax as a way of Node.js knowing when to interpret files as ES modules. Our goal is to eventually find a way to support ES module syntax by default with minimal breaking changes.

@guest271314
Copy link

The Node.js folks even took into account the case of no package.json file, a Node.js-specific artifact

  • Files ending in .js or with no extension, if there is no package.json file present in the same folder or any parent folder.

You can always just fetch the raw text of the script then do whatever you want thereafter with the script text, e.g., pass a Data URL or Blob URL of the script to import(). That get's rid of your non-observable EcmaScript Module Promise behaviour.

@guest271314
Copy link

Also Bun's package manager feels 100x faster than npm

Well, Node.js doesn't own or control npm, GitHub does.

@ehaynes99
Copy link

ehaynes99 commented Nov 19, 2023

await import(…) does

That's literally the next thing I said. The import function exposes the Promise. import statements hide it. That hiding means there was no need for the spec to enforce anything whatsoever about how it's provided by an implementation, because the flow of control around import statements literally is synchronous in nature.

This is not the equivalent of:

// react/index.js
if (process.env.NODE_ENV === 'production') {
  module.exports = require('./react.production.js')
} else {
  module.exports = require('./react.development.js')
}

And? ESM doesn't have a direct equivalent. It didn't need an exact equivalent, only a functional equivalent. Dynamic imports in ESM require the use the import function, not import statements. The import function returns a Promise. Why should re-exporting it change that? The equivalent for ESM could have been:

const appModule = window.location.href.startsWith('http://localhost')
  ? './development-application.js'
  : './production-application.js'

const appModulePromise: Promise<AppModule> = import(appModule)

export default appModulePromise

It was a design decision, not a technical limitation, that required obfuscating that Promise. The design could have made that come out the other side as exactly what it is: a Promise

import appModulePromise from './app-module.js'

appModulePromise.then(// ...

Then you are talking about your personal preferences.

🤦

Node.js is not the only JavaScript runtime.

So you keep saying. It doesn't address the problems with ESM adoption at all, but you still keep saying it. Companies would be astronomically stupid to rewrite existing applications in Deno, Bun, or whatever runtime-of-the-week is cool right now "just cuz". We're not just talking about 9 year old apps. We're talking about practically all server side JS to this day. Ever hear Deno talk about market share? You would if they actually had any. Even most of the web software is still developed in a node environment. Jest is overwhelmingly the most popular test framework. No matter the test framework, mocking with ESM is still much harder, and just compounded by things that flatten out their function bodies into top-level await. Yet apparently, converting all of that is only solution you're willing to entertain.

Fact of the matter is, none of you have come up with a technical limitation that prohibited ESM from being more flexible, only things that you can't do with the spec as designed. None of you have ever even considered the possibility of a change in ESM to solve the problems it created. It's not "something node did wrong" to fail to implement ESM 5 years before ESM existed. It's something ECMA did wrong to release a spec that was incompatible with the only relevant module system in existence beforehand. There are no technical reasons that it couldn't have been done. They should have allowed for a mechanism that commonjs interpreters could be rewritten to adapt to ESM, not expect all of the commonjs code to be rewritten in ESM.

But I give up. I've provided numerous examples of how these "impossible" things could have been done without limiting the functionality at all. You'll apparently never admit that it's anything less than perfect the way it is, though, so keep sitting around hoping it will someday be the standard.

export const solution = async () => {
  let years = 0
  while (years++ < Infinity) {
    console.log('JS does not depend on node')
    await setTimeout(TWO_SECONDS)
    console.log('node is the only thing standing in the way of ESM being the standard')
    await setTimeout(ONE_YEAR)
  }
}

@guest271314
Copy link

So you keep saying. It doesn't address the problems with ESM adoption at all, but you still keep saying it. Companies would be astronomically stupid to rewrite existing applications in Deno, Bun, or whatever runtime-of-the-week is cool right now "just cuz". Yet apparently, that's the only solution you're willing to entertain.

I don't think you understand my stake. I don't care about companies or corporations. My motiviation is not accumulation of dead slave mastas posin' on yo dolla. I'm looking at this case without any preference for any JavaScript runtime or engine as a JavaScript programming language hacker .

That is, how do I get around restrictions, specified or implemented, to achieve the arbitrary goal I set, or meet the requirement somebody else set, by any means.

Somebody brought up my use of new Function() to get variables defined using const in an arrow function, and paraphrasing, said the use of new Function()() was eval-like.

I immediately thought of your description of import statements. That is, that is a box, the module is executed. You don't know what's being executed. And there is a live two-way binding, so that side-effect is persistent.

I would approach achieving the requirement, in spite of any specification or implemenation choices by doing something like

const scriptText = await (await fetch("./exports.js")).text();
const mod = await import(URL.createObjectURL(new Blob([scriptText], {type:"text/javascript"})));
mod.fn();

now you have transparency.

@leonsilicon
Copy link

leonsilicon commented Nov 20, 2023

And? ESM doesn't have a direct equivalent.

ya it does

let React;
if (process.env.NODE_ENV === 'development') {
  React = await import('./react.development.js')
} else {
  React = await import('./react.production.js')
}

export default React;

Why should re-exporting it change that?

because downstream consumers of a library shouldn’t have to (await React).useEffect

I've provided numerous examples of how these "impossible" things could have been done without limiting the functionality

you might need to update your definition of “functionality” if you think that forcing top-level dynamic exports to be promises and downstream consumers to have to await your package export to use it (which is what you’d do without top-level await) is “not limiting functionality”

@cybafelo
Copy link

cybafelo commented Nov 21, 2023

Ok - So I just spent the last three days trying to convert my entire library to ESM - because of some library that I would like to use is ESM and cannot be used by CommonJS.

I have now modified thousands of lines of import / export statements - i cannot commit because i cannot get it stable. I cannot import my server side library without it going into an infinite loop with no indication of where the error is happening - at least with CommonJS it would spit out an error and say - you have an infinite import loop here -> but no, ESM just hangs.

I cannot control how the library is loaded - before my tests starts my entire library is loaded directly from the index.js file in the package.json folder, no matter how i configure it i simply cannot just import classes without it triggering a full load. then, finally - and this is the worst - i cannot be sure at any point that any thing exists without calling 'await import()'.

Literally EVERYTHING about ESM SUCKS - and you are right - after converting - i have gained nothing - except then i could theoretically use THREE.js OrbitControls - because they were so stupid to convert to ESM (and they are brilliant)

Yes - and forget importing native modules without some fancy trick of turning the import into a require statement - since ESM simply chokes on native modules - what a MESS

@jimmywarting
Copy link

@cybafelo you know that you can still import ESM packages using async import() while still running stuff as commonjs?
you did not have to change everything just b/c THREE.js OrbitControls did decide to use ESM.

@cybafelo
Copy link

@cybafelo you know that you can still import ESM packages using async import() while still running stuff as commonjs? you did not have to change everything just b/c THREE.js OrbitControls did decide to use ESM.

No i did not know that - and i would like to keep my code 'future compatible' - according to references online ESM is the next step after commonjs - but so was typescript - which turned out to be a microsoft initiative to make itself more powerful, and in a word, 'evil'. the world is so polluted with new 'bad ideas' that really its impossible to know who to believe - if you are so sure about using 'import' from common js then send some code samples - maybe it will help someone else. I was not able to import it with my best abilities - hence the migration to ESM. Which is now - after three days and insane brain drain - almost working.

@cybafelo
Copy link

cybafelo commented Nov 21, 2023

Also - now - after converting and realizing that I had made some mistakes myself which lead to a tremendous amount of frustration - I do feel confident that ESM could work. What bothers me the most is that I'm not able to selectively decide what gets executed using import. I give an example - i have two index files,

package.json
index.js
src/index.js

Now - when I import one of the files (using common js) - I know which one is going to execute.

When I want to import only 'src/index.js' from anywhere (well, technically, from mocha test script) using ESM - the whole damn index.js is executed - BEFORE EVEN STARTING TO EVALUATE MY TEST CLASS!!!

(and yes - I have looked at and tested the 'exports' option in package.json until all i see is 'exports')

I have put my class declarations into 'src/index.js', and 'index.js' loads my classes from 'src/index.js' but does additional things like start systems etc. VERY frustrating and i cannot seem to work around it, because there are times that I simply do not want to start my systems - i am just interested in my class definitions.

Either it's a mocha thing or an ESM thing - but i don't have the time or will power to understand the module loader code, or mocha, for that matter. The point is - if you want to replace something like commonjs with an equivalent - it should be - an equivalent. All I see is, an attempt at an equivalent - a 'promise' (no pun intended) - to be an equivalent, and in all honesty - not an equivalent.

Because if it was - I would not have to rewrite ALL of my test code which are only interested in src/index.js, and absolutely cannot afford to have index.js execute. - so let's leave this for day 4 - also - imagine every developer wastes 4 days in his life converting to ESM? you'd have lost a century pretty quickly (i'll let someone else calculate that)

@mk-pmb
Copy link

mk-pmb commented Nov 21, 2023

Hahaha, just 4 days lost? I'm quite envious of you. :D
Anyway, the dynamic import should actually have worked. You have to await the results and use them, in the unlikely case that you missed that detail. Also sometimes you need to unpack the default export.

As a package author you can declare multiple exports for your package. See this PR where I helped them expose package.json.
Everyone please always expose package.json in your ESM modules!

Also shameless plug: If you encounter such ESM problems again, try the crutch I made for myself: nodemjs. Maybe it works for you.

@jimmywarting
Copy link

jimmywarting commented Nov 21, 2023

if you are so sure about using 'import' from common js then send some code samples - maybe it will help someone else.

Sure @cybafelo, i can give you 3 example (from easy to more adv)

I created a basic hello world example that uses express (cjs) and node-fetch (esm-only)

you can read more about the solutions over here at this gist of how you can use both cjs and esm together

@guest271314
Copy link

@ehaynes99

I think I'm getting the gist of what you are talking about here.

There was a general push by the JavaScript stakeholders in Node.js, a package management system was created, delivered. And then the rug was pulled from under consumers and a switch was made that is incompatible with the original push.

I tried a dose of my own medicine with a little-bitty repository on GitHub. Errors thrown using jsDelivr's ESM URL talking about this or that export doesn't have a default export. I don't think esbuild or rollup cleanly convert from CJS to ESM. Seems like there should have been at least a couple ways tried and tested to do that, both ways, before shipping ESM. OTOH, CJS via npm is completely dependent on node. There ain't no standalone npm eecutable like every other sane package management system, e.g., aptitude.

@Aeolun
Copy link

Aeolun commented Dec 6, 2023

Every few years I come back to this same Gist... It's just absurd how long this absurd migration has continued.

@camj256
Copy link

camj256 commented Dec 25, 2023

To this day couldn't agree more. Been coding in Node since 2014 full time. Its still a mess IMO. Try and get people to code in the original structure of how whatever app we're working on was built but its becoming more of a PITA every year.

@michaeljnash
Copy link

michaeljnash commented Dec 28, 2023

You have some points wrong/mixed up about load times and dependency trees affecting load times to the extent you think they do.

ES Modules, when used in modern browsers, are designed to take advantage of certain features like HTTP/2 and the browser's ability to asynchronously load resources. HTTP/2, compared to its predecessor HTTP/1.1, allows for more efficient multiplexing, header compression, and server push, among other features, which can significantly improve loading times for multiple resources, including ES Modules.

When considering dependency trees and loading times, ES Modules do support asynchronous loading, allowing browsers to fetch modules in parallel and mitigate the impact of long dependency trees. This asynchronous loading behavior helps in improving the overall performance, especially when dealing with complex applications that have a significant number of modules or dependencies.

@guest271314
Copy link

I'm curious what you folks think about deno throwing module not found error for dynamic import() with raw string specifier denoland/deno#20945, https://gist.github.com/guest271314/4637bb1288321256d7c14c72ebc81137

@Rush
Copy link

Rush commented Jan 17, 2024

I’ve already probably lost more hours to ESM than the code I work on would benefit from moving it. I could have been writing a feature, or making an optimisation, but instead I was wasting time on incompatibility issues. Truly awful.

relate to this so much. :) I don't understand why ESM is such a pain in the butt and why some module maintainers decided to shove it down our throats. I'm thinking of newer versions of node-fetch, for example, which cannot be directly used from a CJS workspace.

@guest271314
Copy link

@Rush You can use WHATWG fetch() in node. You don't need a library. I think the why is ECMA-262 specified Ecmascript Modules. esbuild can compile to CommonJS. bun can compile for the browser or to Ecmascript Modules. esm.sh does a decent job of exporting Node.js modules as Ecmascript Modules.

@Rush
Copy link

Rush commented Jan 18, 2024

@Rush You can use WHATWG fetch() in node. You don't need a library. I think the why is ECMA-262 specified Ecmascript Modules. esbuild can compile to CommonJS. bun can compile for the browser or to Ecmascript Modules. esm.sh does a decent job of exporting Node.js modules as Ecmascript Modules.

Thank you. Indeed there are a number of workarounds that can make things work, granted one spends a ton of hours on it.

@guest271314
Copy link

A modicum of effort. I managed to run the same code in Node.js, Deno, Bun. Next is running the same code in the browser, with node:fs replaced with WICG File System Access. No CommonJs, though essentially the same scenario, where different JavaScript/TypeScript runtimes might do things differently. The original code forked was written for Node.js, from my understanding that Node.js implemented Ed25519, where from what I gather node:crypto cannot be polyfilled. I substituted Web Cryptography API for Node.js crypto https://github.com/guest271314/wbn-sign-webcrypto and https://github.com/guest271314/webbundle for the ability to run the same common code in multiple JavaScript runtimes https://github.com/guest271314/telnet-client.

@Rush
Copy link

Rush commented Jan 18, 2024

I believe it varies. Managing libraries is generally less complex compared to overseeing multi-year projects that involve thousands of files, intricate business logic, and sophisticated build and deployment setups. Kudos to you for ensuring such remarkable flexibility in your own projects. By allowing multiple runtimes you clearly value & respect your users.

@guest271314
Copy link

By allowing multiple runtimes you clearly value & respect your users.

I did it for my own interests, not for users. I essentially had zero experience using Web Cryptography API, and didn't really have an interest in Signed Web Bundles or Isolated Web Apps. My interest was in Direct Sockets TCPSocket in the browser which Chromium authors decided to gate behind SWBN and IWA. I experiment with multiple JavaScript runtimes anyway. I'm just saying whether its 1 file or 1 million files, I don't think it matters, choosing sides and committing to one or both re CommonJS and EcmaScript Modules.

FYI Bun figured out a way to use require() and Ecmascript Modules in the same file.

@Rush
Copy link

Rush commented Jan 18, 2024

Yeah, I do believe Node could have picked a less "efficient" way but one that brings more compatibility. Now we're in the Python2/Python3 situation. It took years to resolve, if ever.

@guest271314
Copy link

Maybe, maybe not. I'm not tethered to Node.js. Try Bun. Try Deno. Try txiki.js. Test them all. The Bun folks have demonstrated quite a bit in a short span of time, particularly catering to people vested in Node.js. bun install, the package manager is built in to the single executable JavaScript runtime, gets rid of npm. bun build ./index.js --compile --outfile=bun_standalone just works on the first try https://gist.github.com/guest271314/9b1adad3db3deba64e118f844a77bad6. Node.js is still using CommonJS. Great for folks who use CommonJS, I suppose. In the mean time multiple other JavaScript engines, runtimes, interpreters https://gist.github.com/guest271314/bd292fc33e1b30dede0643a283fadc6a are doing their own thing irrespective of what happens in Node.js world.

@camj256
Copy link

camj256 commented Jan 18, 2024

I'm considering this a war within our language. As others have said forcing this in some repos has caused me to just not touch them. Sure we can move some things to work with others but at this point we just bring everything in house if we can. I built my codebase before es6 even existed, so having these modules really isn't fun for us unfortunately and we're quite a bit bigger than just 'fixing it'. I've all but left Node for other languages at this point unless its serving up websites. I have no problem with es6, but the esm modules are plain stupidity being forced on us. I'd rather have 2 different package managers if we are headed down this path. I'm tempted to start building out something like that myself to push this rift further apart out of spite.

@guest271314
Copy link

@Rush Python 2 has been "deprecated" for a while. I filed an issue for Google extensions to update their samples to Python 3 and fix their broken code. The Python 2 code is still there. No Python 3 code the last time I checked. MDN Web Docs merged the PR I filed.

@camj256
Copy link

camj256 commented Jan 18, 2024

Maybe, maybe not. I'm not tethered to Node.js. Try Bun. Try Deno. Try txiki.js. Test them all. The Bun folks have demonstrated quite a bit in a short span of time, particularly catering to people vested in Node.js. bun install, the package manager is built in to the single executable JavaScript runtime, gets rid of npm. bun build ./index.js --compile --outfile=bun_standalone just works on the first try https://gist.github.com/guest271314/9b1adad3db3deba64e118f844a77bad6. Node.js is still using CommonJS. Great for folks who use CommonJS, I suppose. In the mean time multiple other JavaScript engines, runtimes, interpreters https://gist.github.com/guest271314/bd292fc33e1b30dede0643a283fadc6a are doing their own thing irrespective of what happens in Node.js world.

I'm tethered to node. See my previous comment. Think that can help? We are rather large with thousands of files and require an extremely efficient stack.

@guest271314
Copy link

I'm considering this a war within our language.

Not at all. Not re JavaScript as specified by ECMA-262. CommonJS was not adopted as the module loader. Ecmascript Modules are. It's an internal Node.js maintainer/user issue.

I built my codebase before es6 even existed

Technology is not static. Moore's Law teaches us that.

I've all but left Node for other languages

Node is not a programming language.

I'm tempted to start building out something like that myself to push this rift further apart out of spite.

Do what you do.

@camj256
Copy link

camj256 commented Jan 18, 2024

Do we think there is a reason mongo 5 and 6 barely got support and 7 is EoL within a year? and 4.4.27 end next month for other reasons? Aside from them going public. I have endless amounts of S to talk.

@guest271314
Copy link

I'm tethered to node.

Not sure what to say.

I bought in to the Palm Pilot IPO. How many people are rolling around with Palm Pilots today? Even Blackberry's?

We are rather large with thousands of files and require an extremely efficient stack.

Well, if you are stuck in CommonJS world, not sure what to say. Except try using Bun, which caters heavily to Node.js API's and philosophies, and figured out how to run CommonJS and Ecmascript Modules in the same file - before Node.js maintainers.

@guest271314
Copy link

I'm not sure what you are talking about. All I know is I am going to use and exploit any and all JavaScript engines and runtimes to achieve my aims, by any means necessary.

esbuild should be able to compile all of your code to CommonJS. Or compile all of your code to Ecmascript Modules. Or use Bun and run them both.

For some Node.js API's there's nothing anybody can do except move on, e.g., crypto, which cannot be polyfilled as far as I know.

@camj256
Copy link

camj256 commented Jan 18, 2024

I'm tethered to node.

Not sure what to say.

I bought in to the Palm Pilot IPO. How many people are rolling around with Palm Pilots today? Even Blackberry's?

We are rather large with thousands of files and require an extremely efficient stack.

Well, if you are stuck in CommonJS world, not sure what to say. Except try using Bun, which caters heavily to Node.js API's and philosophies, and figured out how to run CommonJS and Ecmascript Modules in the same file - before Node.js maintainers.

We're probably around the same age and experience then. I could die on my bed fighting you on this, but I won't. I actually appreciate your responses.

You probably have me by a few years, but not much. I appreciate your response. I'm bitter because it was super easy for my team to work in for a long period of time. Yes, I know every team has to adapt to the times. It still feels like a split, especially lately. I have a huge codebase built in 2013-2014 on what worked amazing, and still does. Naturally I asked my team to code in the same voice the code was written in. Its getting harder and some of these libraries are pretty good these days. I know we can make them work still but we avoid them if possible for consistency and readability for newer team members. Its definitely not aging well...

I'm starting to feel like that's a losing battle; and that's fine, but it sucks. I understand wanting to write things as classes, and if I could, I'd rebuild it all in a different voice today. This is quite a change though IMO and integrating some of these modules isn't what I expected, or hoped for when designing my codebase 10+ years ago. (yeah, I know, even pre es6) I'll bitch till we definitely lose this argument and have to redesign our entire codebase and my CFO and CEO will certainly love that conversation as well. Gotta keep up with the times. In the mean time we're moving a lot of pipelines to python. Specifically anything not serving up web pages.

For what its worth I did learn a few things in the last few comments. So thank you, I'll research some of that further. I've had my head in other places, no excuses there.

At the moment though, I still F'ing hate the way things are headed. lol :)

@guest271314
Copy link

@camj256 I don't understand your apparent frustration. I really don't understand an idea of bitterness. The last time I check Node.js doesn't run the world, technology is not static, time keeps on ticking. Ever try esbuild? There's a --format=cjs option...

@Rush
Copy link

Rush commented Jan 18, 2024

In the mean time we're moving a lot of pipelines to python

Ouch!! That's one hell in your face argument. :-)

@Rush Python 2 has been "deprecated" for a while.

Just because things are called "deprecated" does not mean they do not cause issues :) I think Python2/Python3 divide is mostly over but I remember wasting countless hours on this.

@divmgl
Copy link

divmgl commented Jan 18, 2024

@camj256 I don't understand your apparent frustration. I really don't understand an idea of bitterness. The last time I check Node.js doesn't run the world, technology is not static, time keeps on ticking. Ever try esbuild? There's a --format=cjs option...

This is disingenuous. ESBuild is a community built tool. Most of the complaints here are re: the underlying system that exists in Node.js without any additional tooling. There's no reason why the transition from CommonJS to ES Modules had to be so painful. I'd like to think that this was one of the many catalysts for Bun's existence.

I actually landed on an approach to solving a majority of these ES Modules issues with Node.js, TypeScript and ES Modules and that's to just use ESBuild, as you recommended. I had to build my own scripts to basically take the packages in my monorepo and convert them to valid ESBuild packages (this includes server-side packages).

Thankfully there's now an excellent tool that uses ESBuild under the hood to solve a lot of these pain points: https://github.com/privatenumber/tsx I'm super grateful that a tool like this exists (and works basically flawlessly out of the box) but all of this could have been avoided with a better migration path.

@guest271314
Copy link

@divmgl

This is disingenuous.

No. That's my genuine opinion.

Most of the complaints here are re: the underlying system that exists in Node.js without any additional tooling.

There's nothing you can do about what Node.js maintainers decide to do in the nodejs main branch. You can fork node and do whatever you want.

Yes, I have read the problem statement.

Programmers solve problems.

@guest271314
Copy link

@Rush From [Native Messaging] Python 2 was sunset January 1st, 2020 #489

https://www.python.org/doc/sunset-python-2/

Sunsetting Python 2

We are volunteers who make and take care of the Python programming language. We have decided that January 1, 2020, was the day that we sunset Python 2. That means that we will not improve it anymore after that day, even if someone finds a security problem in it. You should upgrade to Python 3 as soon as you can.

I advised the GoogleChrome/chrome-extensions-samples folks their code was broken here GoogleChrome/chrome-extensions-samples#805 and the MDN Web Docs folks their code was broken here mdn/webextensions-examples#509.

GoogleChrome folks had banned me by then. Then closed the issue. They never fixed their broken Python 2 code https://github.com/GoogleChrome/chrome-extensions-samples/blob/main/_archive/mv2/api/nativeMessaging/host/native-messaging-example-host that doesn't work on Linux anyway because there in general is no /usr/bin/python, there is a /usr/bin/python3.

MDN Web Docs also wound up mergin a working Node.js Native Messaging host example mdn/content#27228. GoogleChrome samples is still stuck in 2013.

@mk-pmb
Copy link

mk-pmb commented Jan 19, 2024

Just my default rant about JS "classes" (click to open)

@camj256

I understand wanting to write things as classes

Understanding is ok, I just hope you don't agree. Rather, teach your folks why the class-less object-centric approach is (can be proven to be) mightier than class-centric approach. Hijacking the term "Object-oriented Programming" for the latter was a genious but evil marketing move from false prophets trying to lead all mankind astray.
Last time I checked, ECMAScript still doesn't have classes, and that's a good thing. Recent editions added some ways to abuse the class keyword and a terminology of "class-like" somethings. It's a tradition for JS to introduce footguns. Firing them is entirely optional, and best left to fools.

@camj256
Copy link

camj256 commented Jan 19, 2024

@mk-pmb Yeah exactly. However, there are enough people in this community that don't agree anymore. ...and thats a problem. (clearly, as demonstrated by this thread) It's making things sloppy and less compatible. Yeah, the language is only hurting itself trying to make changes like this at this point.

Our codebase is easy to read, maintain, and is lightning fast for how large it is. For how well its designed actually I don't really want to hack together builds either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment