Skip to content

Instantly share code, notes, and snippets.

@joepie91
Last active June 15, 2024 04:54
Show Gist options
  • Save joepie91/bca2fda868c1e8b2c2caf76af7dfcad3 to your computer and use it in GitHub Desktop.
Save joepie91/bca2fda868c1e8b2c2caf76af7dfcad3 to your computer and use it in GitHub Desktop.
ES Modules are terrible, actually

ES Modules are terrible, actually

This post was adapted from an earlier Twitter thread.

It's incredible how many collective developer hours have been wasted on pushing through the turd that is ES Modules (often mistakenly called "ES6 Modules"). Causing a big ecosystem divide and massive tooling support issues, for... well, no reason, really. There are no actual advantages to it. At all.

It looks shiny and new and some libraries use it in their documentation without any explanation, so people assume that it's the new thing that must be used. And then I end up having to explain to them why, unlike CommonJS, it doesn't actually work everywhere yet, and may never do so. For example, you can't import ESM modules from a CommonJS file! (Update: I've released a module that works around this issue.)

And then there's Rollup, which apparently requires ESM to be used, at least to get things like treeshaking. Which then makes people believe that treeshaking is not possible with CommonJS modules. Well, it is - Rollup just chose not to support it.

And then there's Babel, which tried to transpile import/export to require/module.exports, sidestepping the ongoing effort of standardizing the module semantics for ESM, causing broken imports and require("foo").default nonsense and spec design issues all over the place.

And then people go "but you can use ESM in browsers without a build step!", apparently not realizing that that is an utterly useless feature because loading a full dependency tree over the network would be unreasonably and unavoidably slow - you'd need as many roundtrips as there are levels of depth in your dependency tree - and so you need some kind of build step anyway, eliminating this entire supposed benefit.

And then people go "well you can statically analyze it better!", apparently not realizing that ESM doesn't actually change any of the JS semantics other than the import/export syntax, and that the import/export statements are equally analyzable as top-level require/module.exports.

"But in CommonJS you can use those elsewhere too, and that breaks static analyzers!", I hear you say. Well, yes, absolutely. But that is inherent in dynamic imports, which by the way, ESM also supports with its dynamic import() syntax. So it doesn't solve that either! Any static analyzer still needs to deal with the case of dynamic imports somehow - it's just rearranging deck chairs on the Titanic.

And then, people go "but now we at least have a standard module system!", apparently not realizing that CommonJS was literally that, the result of an attempt to standardize the various competing module systems in JS. Which, against all odds, actually succeeded!

... and then promptly got destroyed by ESM, which reintroduced a split and all sorts of incompatibility in the ecosystem, rather than just importing some updated variant of CommonJS into the language specification, which would have sidestepped almost all of these issues.

And while the initial CommonJS standardization effort succeeded due to none of the competing module systems being in particularly widespread use yet, CommonJS is so ubiquitous in Javascript-land nowadays that it will never fully go away. Which means that runtimes will forever have to keep supporting two module systems, and developers will forever be paying the cost of the interoperability issues between them.

But it's the future!

Is it really? The vast majority of people who believe they're currently using ESM, aren't even actually doing so - they're feeding their entire codebase through Babel, which deftly converts all of those snazzy import and export statements back into CommonJS syntax. Which works. So what's the point of the new module system again, if it all works with CommonJS anyway?

And it gets worse; import and export are designed as special-cased statements. Aside from the obvious problem of needing to learn a special syntax (which doesn't quite work like object destructuring) instead of reusing core language concepts, this is also a downgrade from CommonJS' require, which is a first-class expression due to just being a function call.

That might sound irrelevant on the face of it, but it has very real consequences. For example, the following pattern is simply not possible with ESM:

const someInitializedModule = require("module-name")(someOptions);

Or how about this one? Also no longer possible:

const app = express();
// ...
app.use("/users", require("./routers/users"));

Having language features available as a first-class expression is one of the most desirable properties in language design; yet for some completely unclear reason, ESM proponents decided to remove that property. There's just no way anymore to directly combine an import statement with some other JS syntax, whether or not the module path is statically specified.

The only way around this is with await import, which would break the supposed static analyzer benefits, only work in async contexts, and even then require weird hacks with parentheses to make it work correctly.

It also means that you now need to make a choice: do you want to be able to use ESM-only dependencies, or do you want to have access to patterns like the above that help you keep your codebase maintainable? ESM or maintainability, your choice!

So, congratulations, ESM proponents. You've destroyed a successful userland specification, wasted many (hundreds of?) thousands of hours of collective developer time, many hours of my own personal unpaid time trying to support people with the fallout, and created ecosystem fragmentation that will never go away, in exchange for... fuck all.

This is a disaster, and the only remaining way I see to fix it is to stop trying to make ESM happen, and deprecate it in favour of some variant of CommonJS modules being absorbed into the spec. It's not too late yet; but at some point it will be.

@4skinSkywalker
Copy link

4skinSkywalker commented Nov 12, 2023

What I wanted to say by referring to banks and assurance companys is that there are certain customers that are rather inflexible and won't allow the new shiny thing in their repertoire of technologies, you can tell by the fact that the banking industry is still working with COBOL code. Good luck telling them to switch their Node.js-based intranet dashboards to use Bun instead.

@guest271314
Copy link

Why would banks be using Node.js at all?

Deno and Node.js use the same JavaScript engine: V8.

I'm not talking about the arbitrary decisions corporations make. I'm talking about the technical fact that you can convert any Node.js package or dependency to Ecmascript Modules - instead of using CommonJS loader system, that is mainly a Node.js thing.

@guest271314
Copy link

Good luck telling them to switch their Node.js-based intranet dashboards to use Bun instead.

Municipalities and governments tend to have contracts with Microsoft to supply software. That doesn't make Windows and Internet Explorer superior to Red Hat Linux or Debian or Ubuntu and Firefox.

You are talking about marketing, not technology.

@nabilfreeman
Copy link

We don’t even need to be talking about large/legacy orgs - experienced startups do not want to mess around with the bleeding edge and will instead go for proven technologies with LTS support

@rubengmurray
Copy link

@4skinSkywalker That's only n issue in Node.js world. Not an issue when using Deno, Bun, QuickJS, txiki.js, SpiderMonkey.

You mean ONLY an issue for 99% of users rather than 1%? Ah yes, not a problem at all.

@guest271314
Copy link

You mean ONLY an issue for 99% of users rather than 1%? Ah yes, not a problem at all.

It's not an issue at all, really.

Just use jsDelivr to get Ecmascript Module versions of your dependencies if you don't know how to do that yourself using esbuild, et al.

People can't on the one hand claim to be so entrenched in Node.js world that they have all these dependencies, then claim they don't have the competency to convert their code to use Ecmascript Modules.

If people want to use CommonJS, they can. That's the default module loader in Node.js world.

However, the world outside of Node.js doesn't revolve around or wait for Node.js maintainers to do stuff. That is, the very individual who created Node.js created Deno. That kind of tells us something about the state of Node.js.

I don't know how 99% of people can amass all of these dependencies, then not know how to convert those dependencies to Ecmascript Modules when there's plenty of tools to do that?

@guest271314
Copy link

Re

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" -> [...and so on and so on...].

here you go tsm: CDN<TS => JS>.

Node.js doesn't own JavaScript, Oracle does. Node.js doesn't specifiy JavaScript, TC39 does in ECMA-262.

@4skinSkywalker
Copy link

It's not an issue at all, really.

Yes it is, and that's because this situation is creating difficulties, wasted time and brain energy for the vast majority of js consumers. Before this standardization everything was an npm install away, now it's way less plug-and-play.

they have all these dependencies, then claim they don't have the competency to convert their code to use Ecmascript Modules.

I don't think you have realized the huge transaction cost people (and company) would have to pay. If you look at the problem at large it's not about competency but economical expense.

However, the world outside of Node.js doesn't revolve around or wait for Node.js maintainers to do stuff.

OK, but the Node.js world is huge, infact the Node Package Manager is the largest software registry in existence.

I don't know how 99% of people can amass all of these dependencies

Why do you even care?

then not know how to convert those dependencies to Ecmascript Modules when there's plenty of tools to do that?

Why should they? Is there a 10x gain in doing that?

@guest271314
Copy link

Yes it is, and that's because this situation is creating difficulties, wasted time and brain energy for the vast majority of js consumers. Before this standardization everything was an npm install away, now it's way less plug-and-play.

No, it's not.

Pick a side.

This is not 2009. People are compiling QuickJS to WASM and running that JavaScript runtime in a WASI environment. QuickJS is ECMA-262 compliant and after strip is less than 1 MB. node Nightly release from yesterday is 97.6 MB; deno executable is 126.6 MB; bun executable is 87.0 MB. Clearly developers using JavaScript for an embedded microcontroller or as a serverless javaScript runtime in a WASI environment are not choosing node - or deno.

I don't think you have realized the huge transaction cost people (and company) would have to pay. If you look at the problem at large it's not about competency but economical expense.

I don't get it.

CommonJS is over. That's a Node.js legacy thing. If you want to maintain module loader system, you are making the economical decision to do when when the rest of JavaScript world is migrating to or already has migrated to Ecmascript Modules. Though Bun does still use CommonJS (require()) and have published articles on the matter; though does not implement all of Node.js's internal modules; e.g., node:fs/promises.

If you want CommonJS to not be over in your organization, you are going to have to pay for that, with time and energy, which is what economics is.

What you are not going to do is convince everybody outside of your organization to undo adoption of Ecmascript Modules. You can try. That's gonna cost you, too. Do an experiment to see which approach costs you the most.

The solution is bundling your dependedncies to Ecmascript Modules.

Unless you philosophically or technically protest adoption of Ecmascript Modules.

Then it's back to the 2d sentence: Pick a side.

Both sides will cost you.

OK, but the Node.js world is huge, infact the Node Package Manager is the largest software registry in existence.

"NPM" does not mean "Node Package Manager" https://www.npmjs.com/package/npm?activeTab=readme

Contrary to popular belief, npm is not in fact an acronym for "Node Package Manager"

and GitHub owns NPM, not Node.js.

When you fetch Node.js nightly archive, which I do every day or so, you will be warned that the npm in the archive is not the lastest version, and there are no plans to change that - because Node.js organization does not own or control npm (NPM) - GitHub does, [BUG] npm in Node.js nightly release consistently prints warning messages #6820.

Why do you even care?

I don't, really. I test and experiment using multiple JavaScript engines and runtimes. When you do that you'll immediately notice Node.js is different than the rest. Ecmascript Modules are supported out of the box. No "package.js", no third-party package script. You generally just get the executable. Packages are a different realm from the JavaScript runtime itself.

i just marvel at the X, Y problem. Pick a side. Be done with the matter.

Why should they? Is there a 10x gain in doing that?

To solve the crises developers and organizations have created for themselves by relying and depending solely on Node.js philosophy and third-party packages, clearly omitting to hedge against technologies outside of Node.js influence usurping Node.js initial monopoly of non-browser JavaScript runtimes.

So you gain by setlling the matter. Litigation costs. You are litigating into oblivion.

@guest271314
Copy link

@4skinSkywalker If you think the CommonJS/Ecmascript Module schism is a mess try using the same I/O (stdin, stdout, stderr) JavaScript code to run in a Node.js, Deno, QuickJS, and Bun environment Common I/O (stdin/stdout/stderr) module specification #47.

@ehaynes99
Copy link

I suggest all upset and uninformed people of this thread (and author) read the very insightful comments under Sindre's "Pure ESM package" gist, especially this one, but more generally: go learn some new thing and stop hating ❤️

There's nothing insightful there. It's the same tired rhetoric repeated throughout this thread. This part is particularly amusing:

once they support ESM properly we'll be unblocked from a mad scramble to ESM over the next year or so. In 2 years, we might have an ecosystem of maintained packages that are almost entirely pure ESM.

That post was 23 months ago.

Sindre's original post there kept getting longer and longer, as he kept having to explain more and more things that break during the switch.

But no need to care about my opionion. Since Sindre has chosen to be the champion for ESM, and people often reference his choice (and since I just noticed someone accuse me of cherry picking above) I thought I would get some real data. So I threw together this little script: https://github.com/ehaynes99/sindres-quest. It:

  • downloads the list of all projects returned by the npmjs registry api's search endpoint with author:sindresorhus
  • downloads the extended metadata for each project from the registry api's package endpoint. This includes the tarball path of each version of each package
  • downloads the tarball, extracts the package.json, and checks whether type === 'module'
  • downloads the list of download counts for the last week, which is the same data as shows on the "versions" tab on a package's page on https://npmjs.com
  • sums up the total of downloads, partitioned by ESM/commonjs, and prints out the totals

Results

As of Nov 12, 2023, the results are as follows:

  • number of packages: 857
  • total number of versions: 8,609
  • total downloads: 6,167,976,795
  • total downloads of ESM versions: 439,771,975
  • total downloads of CJS versions: 5,728,204,820
  • percentage of downloads that are ESM versions: 7.13%

Make of it what you will, but we're not even close after roughly 100 months since ES2015 was released.

@leonsilicon
Copy link

leonsilicon commented Nov 13, 2023

Most developers are so dependant on build tooling they don't actually know how to write a CJS module; millions of developers have their source as ESM. To them, CJS is an implementation detail of Babel, Next.js and Node.js. Most of the installs on npm flow from a few popular libraries/frameworks that suck in an ungodly amount of dependencies on install. If one or two switch to pure ESM and update their dependencies to newer pure ESM versions, overnight millions of CJS installations evaporate.

from https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c?permalink_comment_id=3992076#gistcomment-3992076

and yes, choosing a package with a 105 to 1,079,014 download ratio (which is 0.01% of downloads compared to your script's 7.13%) as an accurate representation of the CJS/ESM download counts seems pretty cherrypicked to me

fwiw i just took a look at p-waterfall's dependents and the reason why there's a million installs for version 2.1.0 is because it's depended on by lerna, supporting the reason for the skewed download counts i referenced above:

image

@guest271314
Copy link

but we're not even close after roughly 100 months since ES2015 was released.

That's just in Node.js world, correct?

It seems to me that the Node.js folks can simply convert the entirety of NPM packages to Ecmascript Modules and be done with the schism.

That appears to be the only actionable path forward.

People outside of Node.js world are not clamouring for CommonJS. They just want to convert the CommonJS third-party package they depend on to Ecmascript Modules in a reliable process.

@guest271314
Copy link

Does nobody trust jsDelivr to provide their packages as Ecmascript Modules?

I have a difficult time understanding what the primary issue is. It reads like a philosophical debate rather than attempts and failures at solving an articulated problem: Legacy third-party code that uses CommonJS written for a Node.js environment where there are multiple JavaScript runtimes besides Node.js that are not tethered to CommonJS.

Help me understand how seasoned Node.js developers cannot solve this on their own.

@ehaynes99
Copy link

ehaynes99 commented Nov 13, 2023

fwiw i just took a look at p-waterfall's dependents and the reason why there's a million installs for version 2.1.0 is because it's depended on by lerna

And? That's an integral part of a tremendous number of projects. Even if it's refactored to ESM by it's new NX owners, they'll push far too many breaking changes for it to be worth it for fully functional projects to update. We're a decade out on that, guaranteed.

Even without lerna, it's a lib that could be as useful in node as it is in the browser, and thus should not be ESM only. Either way, if you think I went out of my way to pick that one, then feel free, but it just happened to be the most recent of my collisions with his libraries. I have exceptions in dependabot for chalk, ora, globby to name a few, as well as overrides in multiple package files, and for downstream fallout like inquirer as well (actually, still my belief: SBoudrias/Inquirer.js#1159 (comment)).

But again, that's why I just went ahead and generated the real data. You can't really argue with the 7%. It's abysmal.

It seems to me that the Node.js folks can simply convert the entirety of NPM packages to Ecmascript Modules and be done with the schism.

Then you are FAR too inexperienced in the industry to have a meaningful opinion. "Just rewrite all of your code" is what they said for python2 -> python3. "It'll be quick", they said. While we finally saw some linux distros drop it after 13 years, it's still not completely solved. It's the death knell of entire companies. You're literally asking an entire ecosystem with software that works to dedicate hundreds of billions of dollars for rewrites just because of your quasi-religious belief that in a "standard".

I've spent most of my career refactoring, TS for the last 3 years, half a dozen others for the 14 years prior. Strict TS, linting, test coverage, monorepos... the list goes on. It takes months, if not years. But I also spent years consulting for Fortune <20-1000>, and there's not a chance in hell that any of them are going to pay a second time for a product that already works. Go ahead and insert some anti-corporate nonsense here like the post cited above. You'll help prove my point.

If you work on small projects -- especially those with "new framework-itis" -- it might seem like a small thing to just "convert the entirety" because some other dude on the Internet said "trust me bro, it's the future".

Help me understand how seasoned Node.js developers cannot solve this on their own.

Well, just read the list.

Otherwise, I've already. stated. them. before

But you're missing the point. nodejs cannot be ESM only until EVERY SINGLE NODE APP IS ESM ONLY. They have a contract of backwards compatibility, as every software platform should. ECMA decided to fix half of the problem and break the other half that already worked. Thus, we'll never get past this without revisions of that broken spec.

You can read my comments above. I'm not suggesting that commonjs is better. I don't think that there is even much contention that ESM syntax is better. But the ESM spec enforced runtime constraints that were -- and still are -- incompatible with half of the ecosystem. It's not going to just work itself out. Post a date when it'll be complete here, and I'll cite it in a post years from now like above when it doesn't happen. ESM needs to change, or it will eventually overtake the python conversion as "the worst decision smart people ever made" in the software industry.

I have a difficult time understanding what the primary issue is.

Thank you. Yes, that's the primary issue.

@guest271314
Copy link

Then you are FAR too inexperienced in the industry to have a meaningful opinion. "Just rewrite all of your code" is what they said for python2 -> python3.

Yes, you have to rewrite code. We've known that since at least Moore's Law. You have 18 months of monopoly, if you're lucky and nobody rearranges your gear and markets your product to a different target demographic than you think is important.

It is certainly possible to convert the entire NPM third-party package system to Ecmascript Modules. jsDelivr already does that for you.

Technology is not static. Nobody is waiting around for what Node.js management is going to do.

Google hasn't updated it's code from Python 2 either. And is publishing broken code after being notified about said code GoogleChrome/chrome-extensions-samples#805. There's also some "whitelisted" and "blacklisted" language still in current Chromium source code that Google Safe Browsing folks claim is too much work to get rid of, even being aware of Chromium source code deliberately getting rid of that garbage. It's called laziness.

"the worst decision smart people ever made" in the software industry.

Alleged smart people make horrible decisions all of the time.

N.A.S.A. launched the space shuttle when they knew the O-rings they were relying on expanded and contracted based on temperature. It was cold. N.A.S.A. launched anyway. And so forth.

You have a solution that is achievable. The third-party packages hosted by GitHub-owned NPM are finite. Convert them all to Ecmascript Module version.

Politicking into oblivion has not changed anything. And Node.js is moving towards Ecamascript Modules, not away from Ecamascript Modules with new "experimental" flags.

Corporations come and go. E.I.C., the first "modern corporation" has it's own private army. Yale University is named after the sacked pres. of E.I.C., Elihu Yale. Then they got too big for their britches.

@guest271314
Copy link

They have a contract of backwards compatibility,

Wait a minute. Who are you talking about?

Are you talking about a contract between third-party software developers and the corporations who buy their gear?

You can't be talking about NPM, now owned by GitHub claiming that CommonJS would be the only module loader in all of the third-party software hosted by NPM?

The node executable happily loads CommonJS as the default loader. You have to deliberately use Ecamascript Modules. Otherwise you've got your default CommonJS loader.

@guest271314
Copy link

@titanism

// dynamically import ESM module
let esmModule;
import('some-esm-module').then((obj) => {
  esmModule = obj;
});

I was going to suggest you should be able to do

const {resolve, reject, promise:esmModule} = Promise.withResolvers();
// dynamically import ESM module
import('some-esm-module').then(resolve);

However, node v22.0.0-nightly202311135e250bd726 has not implemented Promise.withResolvers(). bun version 1.0.11 and deno 1.38.1 have.

@leonsilicon
Copy link

But the ESM spec enforced runtime constraints that were -- and still are -- incompatible with half of the ecosystem.

No it didn’t; take a look at Bun.js which implemented support for cross-loading of CommonJS and ESM. The incompatibility stems from Node.js’s implementation, not the spec.

@guest271314
Copy link

@leondreamed This can't be about the executable itself supporting Ecmascript Modules or not. I downloaded the nightly archive and got rid of everything except the node executable for a while. Ecmascript Module support without a package.json file on the machine required .mjs extension to use import. In node 22 these options are available

  --experimental-default-type=...
                              set module system to use by default
  --experimental-detect-module
                              when ambiguous modules fail to evaluate
                              because they contain ES module syntax,
                              try again to evaluate them as ES
                              modules
  --experimental-import-meta-resolve
                              experimental ES Module
                              import.meta.resolve() parentURL support
  --loader, --experimental-loader=...
                              use the specified module as a custom
                              loader

The "ecosystem" are third-party software authors - this ain't about the node executable itself. node executable support CommonJS require() out of the box - without any package.json file or node_modules folder present on the system. So why are people mad at Node.js and TC39?

@ehaynes99
Copy link

Wait a minute. Who are you talking about?

Are you talking about a contract between third-party software developers and the corporations who buy their gear?

You can't be talking about NPM, now owned by GitHub claiming that CommonJS would be the only module loader in all of the third-party software hosted by NPM?

I'm talking about the nodejs project. They're not going to remove require, so it will never be ESM only.

Alleged smart people make horrible decisions all of the time.

That's been my argument all along. Allegedly smart people created the ESM spec, and yet almost a decade later, we're still dealing with the consequences of the design.

And Node.js is moving towards Ecamascript Modules, not away from Ecamascript Modules with new "experimental" flags.

Yes, movING and EXPERIMENTAL, which means "it's definitely not time to convert libraries targeting node yet". It's still marked experimental in next year's v22.

But the ESM spec enforced runtime constraints that were -- and still are -- incompatible with half of the ecosystem.

No it didn’t; take a look at Bun.js which implemented support for cross-loading of CommonJS and ESM. The incompatibility stems from Node.js’s implementation, not the spec.

node's implementation existed long before they even started dreaming about ESM, so yes, they absolutely created a spec that was incompatible. They assumed it would be easy for node to "just rewrite everything". They were wrong. Hell, it took years for even the browsers to support it, so the vast majority of frontend apps are still bundled to this day. Practically no one actually loading imports over the wire, which was the whole motivation for making async loading part of the spec to begin with.

And of course bun can implement the spec, because bun was created yesterday. But now you're making the rewrite even bigger, changing the entire platform, which then ripples throughout the whole codebase.

Have you ever done a large rewrite? They always take way longer than anyone anticipates. Things break. You interrupt your business and spend tons of developer hours. Sometimes, it's worthwhile if the result is easier to maintain, safer, or more performant. But in converting an existing application to ESM, you would get... NOTHING. Best case scenario, you end up with an application that works exactly the same as it did before. If you want to do it for personal projects, knock yourself out, but for a business, spending a bucket of cash on an endeavor with large amounts of risk and zero reward would be foolish.

Then there's the nature of the refactor itself. You can't iterate on this. For a plain JS node application, you would have to make all of the changes in one giant commit. Even if you're using a transpiler or TypeScript for module conversion, you might be able to iterate on updating all of the individual files, but you still have a big checklist of stuff to change when you finally flip the type flag. Oh, and you have to rework all of your jest tests to use unstable_mockModule, so you can't even rely on those mid-refactor. All while tracking changes in an actively developed project. With multiple projects, you have to do them in topological order of dependencies. All of the apps have to be converted before any of the libraries, and if libraries depend on each other, then you have to iterate dependents -> dependencies all the way through them all.

And for all that, you end up with a codebase littered with "unstable" this and "experimental" that without ANY functional improvement at all.

@rubengmurray
Copy link

Here here, Eric.

I’ll never see the beauty of top level await in existing services, but for the absolute hell i’d have to go through to get there, I don’t care.

I’ve already probably lost more hours to ESM than the code I work on would benefit from moving it. I could have been writing a feature, or making an optimisation, but instead I was wasting time on incompatibility issues. Truly awful.

@guest271314
Copy link

I'm talking about the nodejs project. They're not going to remove require, so it will never be ESM only.

Right. You are talking about what you see as a Node.js specific issue. Not a JavaScript issue.

The JavaScript programming language does not revolve around Node.js. There are multiple JavaScript engines and runtimes that are not dependent on what Node.js does.

Allegedly smart people created the ESM spec, and yet almost a decade later, we're still dealing with the consequences of the design.

ECMA-262 specifies the syntax and semantics of the JavaScript programming language - not Node.js.

Have you ever done a large rewrite?

Yes. I usually start with the understanding there are multiple JavaScript engines, runtimes, and environments.

I wrote five (5) different JavaScript Native Messaging hosts. ECMA-262 does not specify I/O. So no two (2) JavaScript engines or runtimes implement reading stdin, writing to stdout or reading stderr the same. One for Node.js, one for Deno, one for Bun, onefor txiki.js, one for QuickJS https://github.com/guest271314/NativeMessagingHosts.

In the Web API world there are V8, SpiderMonkey, JavaScriptCore. It is not uncommon to write HTML, CSS, and use Web API's that target each environment. There is no hegemony in JavaScript world. Node.js folks don't run JavaScript.

What you are describing is not novel. In the Web API world there are multiple stackeholders. W3C, WHATWG, WICG. Chromium could easily be updated multiple times per day. Trying keeping up with that.

Chromium's implementation of MediaStreamTrack of kind is not compliant with the controlling W3C Media Capture and Streams specification; the track does not produce silence. That means all downstream browsers are broken on that API, too; Brave, Edge, etc.

Firefox still has not implemented W3C Media Capture From Element captureStream(); nor navigator.permissions.request().

Chromium authors do whatever they want. a WICG Drafts have been implemented and shipped. Chromium authors ain;t sitting around waitin on Mozilla folks to agree with them. The topic might come up in compat discussions, but that ain't stopping Chrome from shipping Isolated Web Apps, TCPSocket() in the browser per Direct Sockets. WebTransport took a while to be implemented in Firefox, so did Transferabl Streams. Look the folks who champion that Isolated Web Apps, they are still using CommonJS https://github.com/GoogleChromeLabs/telnet-client.

Node.js is just part of the JavaScript world. Not the JavaScript world. Why and how do you think Deno came about?

Anyway, you have choices.

There's really nothing you can do except hack yourself out of the mental and philosophical hole you have dug for yourself. As the late U.S. Sec'y of Defense Donald Rumsfeld once remarked, the first thing you do when you are in a hole is top digging. You are still digging in the same hole instead of backfilling and building on top of that leveled earth.

@ehaynes99
Copy link

ehaynes99 commented Nov 15, 2023

To be clear, you're asking everyone in the world who has a working application who doesn't give two shits about the success of ESM to just suck it up, spend untold amounts of developer hours (let's ballpark it at $150 each), take on a bunch of risk, gain absolutely nothing from the effort, and finish with a result that only benefits the "true believers" in this "standard". And somehow I'm the one in the philosophical hole?

I have no "side" here. I think they both have problems, neither is going away, and we'll never settle on one of these particular 2 choices. There is ZERO advantage to ESM on the server side. None. It's marginally worse, TBH, but let's just say "after a bunch of work, it's the same".

FWIW, all 50 of the projects I help manage are in a consistent enough state that I could quite likely script the whole conversion and be done with it. But it took me a year to get them there in between "real work", with help for the last 4-6 months. So let's pose a hypothetical: One day, someone is writing a little console app for use by other developers, and thinks it would be useful to add a bit of color to some output. They find the colors library, but then realize that the author went insane, and settle on the de-facto alternative, chalk.

But wait, it's ESM only! Should they:

  • use chalk@^4
  • convince the entire company that it's time to move the whole thing to deno... or is it bun... or whatever flavor of the week node alternative with less than 1% market share popped up in the last 30 days.

Get real. It's just not the reality for a ton of companies/projects. They have customers. They have a reputation. They have mouths to feed.

You're also ex post facto blaming things that existing long before ECMA started dreaming about their glorious new spec. At the time:

  • node was 100% of the server side JS
  • the Node Package Manager was the only viable package manager
  • practically all of the software developed for web at the time was developed in a node environment and bundled to cjs too (hell, most of it still is)

So yes, when defining a new spec for ALL modules, they absolutely owed a hat tip to the only module system that had ever stabilized the carnage of CDNs and global everything.

It's not as if there is some constant form of pushback against everything ECMA does. The other features they've developed are adopted rapidly in node, browsers, TypeScript, and all of those other environments. Any code written in the last few years is almost certainly using async/await rather the Promise api directly, but they didn't make you choose between one or the other. They usually get it right. Here they did not.

The rest of your post... I don't know what any of that has to do with this. Yes, browsers are historically inconsistent. Some of them decide some feature you want aren't that important. It's not some fundamental language feature that's supposed to be "the standard".

The "true believers" have been saying for 8 years now that "it ain't broke, so don't fix it". I've wasted way too much time on this thread, but a rhetorical question... How long will it take before you realize that just maybe, they didn't shit gold on this one? We're already pretty close to the decade mark, so that can't be it. Is it 15? 20? 100?

@ehaynes99
Copy link

Here here, Eric.

I’ll never see the beauty of top level await in existing services, but for the absolute hell i’d have to go through to get there, I don’t care.

I don't even see it as a feature. Top level await is a side effect on import, which is a horrible pattern. Modules should never "do stuff" when you load them, only provide values, some of whom can be functions that "do stuff".

// some-file.js
await http.get('not-a-real-address') // explodes
// whatever.js
import 'some-file.js' // nothing I can do, because the `Promise` is a "secret". I don't `await`

vs a perfectly reasonable way to load things

export const doStuff = async () => {
  await http.get('not-a-real-address') // explodes
}
// whatever.js
import { doStuff } from 'some-file.js'

const whatever = async () => {
  try {
    await doStuff()
  } catch (error) {
    console.log('I don\'t want my app to die!!!!!!!!!!!!!!!!!')
  }
}

Async import could still work just fine with await import('some-file.js'), and MUCH more importantly, node could just rewrite require to implement it in terms of import. Everything could have shifted to ESM by early 2016.

@guest271314
Copy link

FWIW, all 50 of the projects I help manage are in a consistent enough state that I could quite likely script the whole conversion and be done with it.

That's your answer.

@ehaynes99
Copy link

FWIW, all 50 of the projects I help manage are in a consistent enough state that I could quite likely script the whole conversion and be done with it.

That's your answer.

I don't need an answer to your problem.

@guest271314
Copy link

I don't even see it as a feature. Top level await is a side effect on import, which is a horrible pattern. Modules should never "do stuff" when you load them, only provide values,

That's rather opinionated. Those are just your personal preferences.

How does the module output the value to you without doing stuff?

// whatever.js
import 'some-file.js' // nothing I can do, because the `Promise` is a "secret". I don't `await`

You absolutely can do something.

Intercept and modify the request with a ServiceWorker.

Serve JSON using import assertions, then you get values that you can do stuff to afterwards.

vs a perfectly reasonable way to load things

Then do that in your own code?

I don't need an answer to your problem.

I thought you were trying to massage a problem statement out of your personal preferences and failure to convert your own code to the module loader you persoanally prefer.

I don't have a problem with Ecmascript Modules. If a repository uses CommonsJS, I use the loader they use uness I'm targeting a specific environment.

If you think JavaScript should do this or that you can certainly roll your own JavaScript runtime from scratch to do whatever you write out, or fork an existing JavaScript engine or run time and write it out - implement the detail that you prefer, for yourself.

@leonsilicon
Copy link

leonsilicon commented Nov 18, 2023

Honestly, I think it boils down to this:

  • Browsers needed a JavaScript module loading standard
  • The module loading standard adopted by browsers cannot be synchronous (and thus couldn't be CommonJS)
  • ESM was that standard designed for browsers
    • Top-level await is important for browsers because bandwidth matters, and browsers should be able to dynamically import and export modules across the network on the top level (for example, a production vs development top-level export based on the window.location.href value, just like how CommonJS can dynamically export different values based on the process.env.NODE_ENV value)
  • A single JavaScript module standard for all runtimes is better than a separate one for browsers and server-side JS like Node.js, so ESM was developed to be that unified standard that was embedded into the JavaScript spec
  • Node.js decided to try and move the ecosystem exclusively towards ESM, which involved preventing CJS users from importing ESM that led to this "require(esm)" problem

ESM itself isn't the problem, because browsers needed a module system that was optimized for browsers. There was inevitably going to be two separate module systems (an asynchronous one and a synchronous one). The problem largely stems from Node.js deciding to move the ecosystem exclusively towards the asynchronous module system for the benefit of a single universal module system (an alternative approach such as the one by Bun.js is to provide first-class support both module systems, and only error when they are used in incompatible ways, such as the synchronous module loading system trying to synchronously import a file using the asynchronous module loading system that uses top-level await).

@guest271314
Copy link

I really don't see a problem statement here. This is not 2009. Node.js is not the only JavaScript runtime that does not target the browser.

Circa 2023 there are more JavaScript runtimes that do not target the browser than those that do.

We have options. You can Roll your own JavaScript runtime and do whatever you want, like CloudFlare's Workerd, WasmEdge's QuickJS server implementation, Wasmer's WinterJS, dune, VM Labs WASM Workers Server, Fastly that uses SpiderMonkey, etc.

Just look at one way Node.js documents reading stdin. An event listener .on("readable") where in the handler there is a while loop. Now, how is the result supposed to be returned from that? I think it is nostalgic to think of Node.js as the standard for non-browser JavaScript. However, if developers are not constantly testing and running multiple JavaScript engines and runtimes, I can see how people can become myopic.

@ehaynes99
Copy link

The module loading standard adopted by browsers cannot be synchronous

We've already covered this. Again, ESM doesn't even expose the fact that it's async to you at all. The behavior is exactly the same.

// I'm about to be blocked, waiting for an arbitrary hierarchy of files to be loaded
import { someThing } from './some-module.js'
// flow of execution does not move to here until the magic promise is fulfilled

It's a distinction without a difference.

Top-level await is important for browsers because bandwidth matters, and browsers should be able to dynamically import and export modules across the network on the top level (for example, a production vs development top-level export based on the window.location.href value

Top level await is JUST SYNTAX. It doesn't provide any functionality that you couldn't have without it. Applications have been doing code splitting for years without it, or before await existed at all. You can't even use an import statement for that, only the import function.

const appModule = window.location.href.startsWith('http://localhost')
  ?'./development-application.js' 
  : './production-application.js'

// without top level await
import(appModule)
  .then((app) => app.start())
  .catch(console.error)

// with top level await
try {
  const app = await import(appModule)
  app.start()
} catch (error) {
  console.error(error)
}

A single JavaScript module standard for all runtimes is better than a separate one for browsers and server-side JS like Node.js, so ESM was developed to be that unified standard that was embedded into the JavaScript spec

Couldn't agree more. It would have been great for them to make a spec like that.

I thought you were trying to massage a problem statement out of your personal preferences and failure to convert your own code to the module loader you persoanally prefer.

Sorry, you thought wrong. I'm not trying to solve some problem. My personal preferences don't matter, other than that I sure would prefer if we actually had a standard, rather than just pretending we do. I'm pointing out real reasons why this didn't go as planned, and how you shouldn't hold your breath for it work out any time soon. I'm trying to point out how tiny of a change to the spec would actually be required to allow it to become the standard, rather than one of two disparate systems that's making glacially slow gains.

You've been continually pointing out differences in implementations. You are correct: ECMA didn't define the standard for interacting with stdio. They allowed flexibility in how the implementations interacted with the underlying operating system. THAT'S WHAT WE NEED HERE. Imports are just a mechanism of loading files. Whether they're loaded from disk, over a network, or if it sends a letter in the mail and waits for someone to mail the result back. No matter how the content is provided, when the interpreter runs in to an import statement, it stops, asks the implementation to load the content of the uri as a string, interprets it, assigns the result to variables, and proceeds with processing the file.

You wouldn't even have to give up any functionality. Absolutely nothing about how you build software, how it behaves, or what you can accomplish would change. It's one dinky piece of syntax that's just obfuscating a Promise when the module could just export the Promise, or better yet, a function that returns a Promise. You're literally just taking a function body and dropping it into the root scope of the module instead, and replacing its call signature with an import statement. That's it. That's all you would have to give up so that we could actually have a standard. Node could rewrite require to use ESM under the hood. It would be trivial. The divide would finally be over.

@leonsilicon
Copy link

leonsilicon commented Nov 19, 2023

We've already covered this. Again, ESM doesn't even expose the fact that it's async to you at all. The behavior is exactly the same.

await import(…) does

// without top level await
import(appModule)
.then((app) => app.start())
.catch(console.error)

This is not the equivalent of:

// react/index.js
if (process.env.NODE_ENV === 'production') {
  module.exports = require('./react.production.js')
} else {
  module.exports = require('./react.development.js')
}

@divmgl
Copy link

divmgl commented Nov 19, 2023

Anyone who is arguing about the syntax of ESM are unable to see the forest for the trees. The issue is Node's handling of the ESM migration. Bun proves that it's definitely possible to have both APIs and still have a smooth migration between the two.

Node is a mess now. Add in the complexity from TypeScript and monorepos and you're looking at hours wasted, especially if you've inherited a legacy CommonJS codebase. Were it not for the amazing community packages (right now really enjoying using tsx, tRPC and Prisma) I probably wouldn't use Node. While Bun is still super immature, it is a sight to behold. It's so freaking fast and starting new projects in it feels amazing. It feels like what Node was during the CommonJS era. Also Bun's package manager feels 100x faster than npm and the runtime includes a testing library (no more Jest vs Vitest debates).

I'm very grateful for Node and the Node Foundation but I also think it's important to be critical when something isn't working out.

@guest271314
Copy link

My personal preferences don't matter, other than that I sure would prefer if we actually had a standard, rather than just pretending we do.

Then you are talking about your personal preferences.

Node.js is not the only JavaScript runtime. The Node.js folks are aware of the schism and have taken some steps to make all stakeholders happy, https://nodejs.org/en/blog/announcements/v21-release-announce#esm---experimental-default-type-flag-to-flip-module-defaults.

We are also exploring using detection of ES module syntax as a way of Node.js knowing when to interpret files as ES modules. Our goal is to eventually find a way to support ES module syntax by default with minimal breaking changes.

@guest271314
Copy link

The Node.js folks even took into account the case of no package.json file, a Node.js-specific artifact

  • Files ending in .js or with no extension, if there is no package.json file present in the same folder or any parent folder.

You can always just fetch the raw text of the script then do whatever you want thereafter with the script text, e.g., pass a Data URL or Blob URL of the script to import(). That get's rid of your non-observable EcmaScript Module Promise behaviour.

@guest271314
Copy link

Also Bun's package manager feels 100x faster than npm

Well, Node.js doesn't own or control npm, GitHub does.

@ehaynes99
Copy link

ehaynes99 commented Nov 19, 2023

await import(…) does

That's literally the next thing I said. The import function exposes the Promise. import statements hide it. That hiding means there was no need for the spec to enforce anything whatsoever about how it's provided by an implementation, because the flow of control around import statements literally is synchronous in nature.

This is not the equivalent of:

// react/index.js
if (process.env.NODE_ENV === 'production') {
  module.exports = require('./react.production.js')
} else {
  module.exports = require('./react.development.js')
}

And? ESM doesn't have a direct equivalent. It didn't need an exact equivalent, only a functional equivalent. Dynamic imports in ESM require the use the import function, not import statements. The import function returns a Promise. Why should re-exporting it change that? The equivalent for ESM could have been:

const appModule = window.location.href.startsWith('http://localhost')
  ? './development-application.js'
  : './production-application.js'

const appModulePromise: Promise<AppModule> = import(appModule)

export default appModulePromise

It was a design decision, not a technical limitation, that required obfuscating that Promise. The design could have made that come out the other side as exactly what it is: a Promise

import appModulePromise from './app-module.js'

appModulePromise.then(// ...

Then you are talking about your personal preferences.

🤦

Node.js is not the only JavaScript runtime.

So you keep saying. It doesn't address the problems with ESM adoption at all, but you still keep saying it. Companies would be astronomically stupid to rewrite existing applications in Deno, Bun, or whatever runtime-of-the-week is cool right now "just cuz". We're not just talking about 9 year old apps. We're talking about practically all server side JS to this day. Ever hear Deno talk about market share? You would if they actually had any. Even most of the web software is still developed in a node environment. Jest is overwhelmingly the most popular test framework. No matter the test framework, mocking with ESM is still much harder, and just compounded by things that flatten out their function bodies into top-level await. Yet apparently, converting all of that is only solution you're willing to entertain.

Fact of the matter is, none of you have come up with a technical limitation that prohibited ESM from being more flexible, only things that you can't do with the spec as designed. None of you have ever even considered the possibility of a change in ESM to solve the problems it created. It's not "something node did wrong" to fail to implement ESM 5 years before ESM existed. It's something ECMA did wrong to release a spec that was incompatible with the only relevant module system in existence beforehand. There are no technical reasons that it couldn't have been done. They should have allowed for a mechanism that commonjs interpreters could be rewritten to adapt to ESM, not expect all of the commonjs code to be rewritten in ESM.

But I give up. I've provided numerous examples of how these "impossible" things could have been done without limiting the functionality at all. You'll apparently never admit that it's anything less than perfect the way it is, though, so keep sitting around hoping it will someday be the standard.

export const solution = async () => {
  let years = 0
  while (years++ < Infinity) {
    console.log('JS does not depend on node')
    await setTimeout(TWO_SECONDS)
    console.log('node is the only thing standing in the way of ESM being the standard')
    await setTimeout(ONE_YEAR)
  }
}

@guest271314
Copy link

So you keep saying. It doesn't address the problems with ESM adoption at all, but you still keep saying it. Companies would be astronomically stupid to rewrite existing applications in Deno, Bun, or whatever runtime-of-the-week is cool right now "just cuz". Yet apparently, that's the only solution you're willing to entertain.

I don't think you understand my stake. I don't care about companies or corporations. My motiviation is not accumulation of dead slave mastas posin' on yo dolla. I'm looking at this case without any preference for any JavaScript runtime or engine as a JavaScript programming language hacker .

That is, how do I get around restrictions, specified or implemented, to achieve the arbitrary goal I set, or meet the requirement somebody else set, by any means.

Somebody brought up my use of new Function() to get variables defined using const in an arrow function, and paraphrasing, said the use of new Function()() was eval-like.

I immediately thought of your description of import statements. That is, that is a box, the module is executed. You don't know what's being executed. And there is a live two-way binding, so that side-effect is persistent.

I would approach achieving the requirement, in spite of any specification or implemenation choices by doing something like

const scriptText = await (await fetch("./exports.js")).text();
const mod = await import(URL.createObjectURL(new Blob([scriptText], {type:"text/javascript"})));
mod.fn();

now you have transparency.

@leonsilicon
Copy link

leonsilicon commented Nov 20, 2023

And? ESM doesn't have a direct equivalent.

ya it does

let React;
if (process.env.NODE_ENV === 'development') {
  React = await import('./react.development.js')
} else {
  React = await import('./react.production.js')
}

export default React;

Why should re-exporting it change that?

because downstream consumers of a library shouldn’t have to (await React).useEffect

I've provided numerous examples of how these "impossible" things could have been done without limiting the functionality

you might need to update your definition of “functionality” if you think that forcing top-level dynamic exports to be promises and downstream consumers to have to await your package export to use it (which is what you’d do without top-level await) is “not limiting functionality”

@cybafelo
Copy link

cybafelo commented Nov 21, 2023

Ok - So I just spent the last three days trying to convert my entire library to ESM - because of some library that I would like to use is ESM and cannot be used by CommonJS.

I have now modified thousands of lines of import / export statements - i cannot commit because i cannot get it stable. I cannot import my server side library without it going into an infinite loop with no indication of where the error is happening - at least with CommonJS it would spit out an error and say - you have an infinite import loop here -> but no, ESM just hangs.

I cannot control how the library is loaded - before my tests starts my entire library is loaded directly from the index.js file in the package.json folder, no matter how i configure it i simply cannot just import classes without it triggering a full load. then, finally - and this is the worst - i cannot be sure at any point that any thing exists without calling 'await import()'.

Literally EVERYTHING about ESM SUCKS - and you are right - after converting - i have gained nothing - except then i could theoretically use THREE.js OrbitControls - because they were so stupid to convert to ESM (and they are brilliant)

Yes - and forget importing native modules without some fancy trick of turning the import into a require statement - since ESM simply chokes on native modules - what a MESS

@jimmywarting
Copy link

@cybafelo you know that you can still import ESM packages using async import() while still running stuff as commonjs?
you did not have to change everything just b/c THREE.js OrbitControls did decide to use ESM.

@cybafelo
Copy link

@cybafelo you know that you can still import ESM packages using async import() while still running stuff as commonjs? you did not have to change everything just b/c THREE.js OrbitControls did decide to use ESM.

No i did not know that - and i would like to keep my code 'future compatible' - according to references online ESM is the next step after commonjs - but so was typescript - which turned out to be a microsoft initiative to make itself more powerful, and in a word, 'evil'. the world is so polluted with new 'bad ideas' that really its impossible to know who to believe - if you are so sure about using 'import' from common js then send some code samples - maybe it will help someone else. I was not able to import it with my best abilities - hence the migration to ESM. Which is now - after three days and insane brain drain - almost working.

@cybafelo
Copy link

cybafelo commented Nov 21, 2023

Also - now - after converting and realizing that I had made some mistakes myself which lead to a tremendous amount of frustration - I do feel confident that ESM could work. What bothers me the most is that I'm not able to selectively decide what gets executed using import. I give an example - i have two index files,

package.json
index.js
src/index.js

Now - when I import one of the files (using common js) - I know which one is going to execute.

When I want to import only 'src/index.js' from anywhere (well, technically, from mocha test script) using ESM - the whole damn index.js is executed - BEFORE EVEN STARTING TO EVALUATE MY TEST CLASS!!!

(and yes - I have looked at and tested the 'exports' option in package.json until all i see is 'exports')

I have put my class declarations into 'src/index.js', and 'index.js' loads my classes from 'src/index.js' but does additional things like start systems etc. VERY frustrating and i cannot seem to work around it, because there are times that I simply do not want to start my systems - i am just interested in my class definitions.

Either it's a mocha thing or an ESM thing - but i don't have the time or will power to understand the module loader code, or mocha, for that matter. The point is - if you want to replace something like commonjs with an equivalent - it should be - an equivalent. All I see is, an attempt at an equivalent - a 'promise' (no pun intended) - to be an equivalent, and in all honesty - not an equivalent.

Because if it was - I would not have to rewrite ALL of my test code which are only interested in src/index.js, and absolutely cannot afford to have index.js execute. - so let's leave this for day 4 - also - imagine every developer wastes 4 days in his life converting to ESM? you'd have lost a century pretty quickly (i'll let someone else calculate that)

@mk-pmb
Copy link

mk-pmb commented Nov 21, 2023

Hahaha, just 4 days lost? I'm quite envious of you. :D
Anyway, the dynamic import should actually have worked. You have to await the results and use them, in the unlikely case that you missed that detail. Also sometimes you need to unpack the default export.

As a package author you can declare multiple exports for your package. See this PR where I helped them expose package.json.
Everyone please always expose package.json in your ESM modules!

Also shameless plug: If you encounter such ESM problems again, try the crutch I made for myself: nodemjs. Maybe it works for you.

@jimmywarting
Copy link

jimmywarting commented Nov 21, 2023

if you are so sure about using 'import' from common js then send some code samples - maybe it will help someone else.

Sure @cybafelo, i can give you 3 example (from easy to more adv)

I created a basic hello world example that uses express (cjs) and node-fetch (esm-only)

you can read more about the solutions over here at this gist of how you can use both cjs and esm together

@guest271314
Copy link

@ehaynes99

I think I'm getting the gist of what you are talking about here.

There was a general push by the JavaScript stakeholders in Node.js, a package management system was created, delivered. And then the rug was pulled from under consumers and a switch was made that is incompatible with the original push.

I tried a dose of my own medicine with a little-bitty repository on GitHub. Errors thrown using jsDelivr's ESM URL talking about this or that export doesn't have a default export. I don't think esbuild or rollup cleanly convert from CJS to ESM. Seems like there should have been at least a couple ways tried and tested to do that, both ways, before shipping ESM. OTOH, CJS via npm is completely dependent on node. There ain't no standalone npm eecutable like every other sane package management system, e.g., aptitude.

@Aeolun
Copy link

Aeolun commented Dec 6, 2023

Every few years I come back to this same Gist... It's just absurd how long this absurd migration has continued.

@camj256
Copy link

camj256 commented Dec 25, 2023

To this day couldn't agree more. Been coding in Node since 2014 full time. Its still a mess IMO. Try and get people to code in the original structure of how whatever app we're working on was built but its becoming more of a PITA every year.

@michaeljnash
Copy link

michaeljnash commented Dec 28, 2023

You have some points wrong/mixed up about load times and dependency trees affecting load times to the extent you think they do.

ES Modules, when used in modern browsers, are designed to take advantage of certain features like HTTP/2 and the browser's ability to asynchronously load resources. HTTP/2, compared to its predecessor HTTP/1.1, allows for more efficient multiplexing, header compression, and server push, among other features, which can significantly improve loading times for multiple resources, including ES Modules.

When considering dependency trees and loading times, ES Modules do support asynchronous loading, allowing browsers to fetch modules in parallel and mitigate the impact of long dependency trees. This asynchronous loading behavior helps in improving the overall performance, especially when dealing with complex applications that have a significant number of modules or dependencies.

@guest271314
Copy link

I'm curious what you folks think about deno throwing module not found error for dynamic import() with raw string specifier denoland/deno#20945, https://gist.github.com/guest271314/4637bb1288321256d7c14c72ebc81137

@Rush
Copy link

Rush commented Jan 17, 2024

I’ve already probably lost more hours to ESM than the code I work on would benefit from moving it. I could have been writing a feature, or making an optimisation, but instead I was wasting time on incompatibility issues. Truly awful.

relate to this so much. :) I don't understand why ESM is such a pain in the butt and why some module maintainers decided to shove it down our throats. I'm thinking of newer versions of node-fetch, for example, which cannot be directly used from a CJS workspace.

@guest271314
Copy link

@Rush You can use WHATWG fetch() in node. You don't need a library. I think the why is ECMA-262 specified Ecmascript Modules. esbuild can compile to CommonJS. bun can compile for the browser or to Ecmascript Modules. esm.sh does a decent job of exporting Node.js modules as Ecmascript Modules.

@Rush
Copy link

Rush commented Jan 18, 2024

@Rush You can use WHATWG fetch() in node. You don't need a library. I think the why is ECMA-262 specified Ecmascript Modules. esbuild can compile to CommonJS. bun can compile for the browser or to Ecmascript Modules. esm.sh does a decent job of exporting Node.js modules as Ecmascript Modules.

Thank you. Indeed there are a number of workarounds that can make things work, granted one spends a ton of hours on it.

@guest271314
Copy link

A modicum of effort. I managed to run the same code in Node.js, Deno, Bun. Next is running the same code in the browser, with node:fs replaced with WICG File System Access. No CommonJs, though essentially the same scenario, where different JavaScript/TypeScript runtimes might do things differently. The original code forked was written for Node.js, from my understanding that Node.js implemented Ed25519, where from what I gather node:crypto cannot be polyfilled. I substituted Web Cryptography API for Node.js crypto https://github.com/guest271314/wbn-sign-webcrypto and https://github.com/guest271314/webbundle for the ability to run the same common code in multiple JavaScript runtimes https://github.com/guest271314/telnet-client.

@Rush
Copy link

Rush commented Jan 18, 2024

I believe it varies. Managing libraries is generally less complex compared to overseeing multi-year projects that involve thousands of files, intricate business logic, and sophisticated build and deployment setups. Kudos to you for ensuring such remarkable flexibility in your own projects. By allowing multiple runtimes you clearly value & respect your users.

@guest271314
Copy link

By allowing multiple runtimes you clearly value & respect your users.

I did it for my own interests, not for users. I essentially had zero experience using Web Cryptography API, and didn't really have an interest in Signed Web Bundles or Isolated Web Apps. My interest was in Direct Sockets TCPSocket in the browser which Chromium authors decided to gate behind SWBN and IWA. I experiment with multiple JavaScript runtimes anyway. I'm just saying whether its 1 file or 1 million files, I don't think it matters, choosing sides and committing to one or both re CommonJS and EcmaScript Modules.

FYI Bun figured out a way to use require() and Ecmascript Modules in the same file.

@Rush
Copy link

Rush commented Jan 18, 2024

Yeah, I do believe Node could have picked a less "efficient" way but one that brings more compatibility. Now we're in the Python2/Python3 situation. It took years to resolve, if ever.

@guest271314
Copy link

Maybe, maybe not. I'm not tethered to Node.js. Try Bun. Try Deno. Try txiki.js. Test them all. The Bun folks have demonstrated quite a bit in a short span of time, particularly catering to people vested in Node.js. bun install, the package manager is built in to the single executable JavaScript runtime, gets rid of npm. bun build ./index.js --compile --outfile=bun_standalone just works on the first try https://gist.github.com/guest271314/9b1adad3db3deba64e118f844a77bad6. Node.js is still using CommonJS. Great for folks who use CommonJS, I suppose. In the mean time multiple other JavaScript engines, runtimes, interpreters https://gist.github.com/guest271314/bd292fc33e1b30dede0643a283fadc6a are doing their own thing irrespective of what happens in Node.js world.

@camj256
Copy link

camj256 commented Jan 18, 2024

I'm considering this a war within our language. As others have said forcing this in some repos has caused me to just not touch them. Sure we can move some things to work with others but at this point we just bring everything in house if we can. I built my codebase before es6 even existed, so having these modules really isn't fun for us unfortunately and we're quite a bit bigger than just 'fixing it'. I've all but left Node for other languages at this point unless its serving up websites. I have no problem with es6, but the esm modules are plain stupidity being forced on us. I'd rather have 2 different package managers if we are headed down this path. I'm tempted to start building out something like that myself to push this rift further apart out of spite.

@guest271314
Copy link

@Rush Python 2 has been "deprecated" for a while. I filed an issue for Google extensions to update their samples to Python 3 and fix their broken code. The Python 2 code is still there. No Python 3 code the last time I checked. MDN Web Docs merged the PR I filed.

@camj256
Copy link

camj256 commented Jan 18, 2024

Maybe, maybe not. I'm not tethered to Node.js. Try Bun. Try Deno. Try txiki.js. Test them all. The Bun folks have demonstrated quite a bit in a short span of time, particularly catering to people vested in Node.js. bun install, the package manager is built in to the single executable JavaScript runtime, gets rid of npm. bun build ./index.js --compile --outfile=bun_standalone just works on the first try https://gist.github.com/guest271314/9b1adad3db3deba64e118f844a77bad6. Node.js is still using CommonJS. Great for folks who use CommonJS, I suppose. In the mean time multiple other JavaScript engines, runtimes, interpreters https://gist.github.com/guest271314/bd292fc33e1b30dede0643a283fadc6a are doing their own thing irrespective of what happens in Node.js world.

I'm tethered to node. See my previous comment. Think that can help? We are rather large with thousands of files and require an extremely efficient stack.

@guest271314
Copy link

I'm considering this a war within our language.

Not at all. Not re JavaScript as specified by ECMA-262. CommonJS was not adopted as the module loader. Ecmascript Modules are. It's an internal Node.js maintainer/user issue.

I built my codebase before es6 even existed

Technology is not static. Moore's Law teaches us that.

I've all but left Node for other languages

Node is not a programming language.

I'm tempted to start building out something like that myself to push this rift further apart out of spite.

Do what you do.

@camj256
Copy link

camj256 commented Jan 18, 2024

Do we think there is a reason mongo 5 and 6 barely got support and 7 is EoL within a year? and 4.4.27 end next month for other reasons? Aside from them going public. I have endless amounts of S to talk.

@guest271314
Copy link

I'm tethered to node.

Not sure what to say.

I bought in to the Palm Pilot IPO. How many people are rolling around with Palm Pilots today? Even Blackberry's?

We are rather large with thousands of files and require an extremely efficient stack.

Well, if you are stuck in CommonJS world, not sure what to say. Except try using Bun, which caters heavily to Node.js API's and philosophies, and figured out how to run CommonJS and Ecmascript Modules in the same file - before Node.js maintainers.

@guest271314
Copy link

I'm not sure what you are talking about. All I know is I am going to use and exploit any and all JavaScript engines and runtimes to achieve my aims, by any means necessary.

esbuild should be able to compile all of your code to CommonJS. Or compile all of your code to Ecmascript Modules. Or use Bun and run them both.

For some Node.js API's there's nothing anybody can do except move on, e.g., crypto, which cannot be polyfilled as far as I know.

@camj256
Copy link

camj256 commented Jan 18, 2024

I'm tethered to node.

Not sure what to say.

I bought in to the Palm Pilot IPO. How many people are rolling around with Palm Pilots today? Even Blackberry's?

We are rather large with thousands of files and require an extremely efficient stack.

Well, if you are stuck in CommonJS world, not sure what to say. Except try using Bun, which caters heavily to Node.js API's and philosophies, and figured out how to run CommonJS and Ecmascript Modules in the same file - before Node.js maintainers.

We're probably around the same age and experience then. I could die on my bed fighting you on this, but I won't. I actually appreciate your responses.

You probably have me by a few years, but not much. I appreciate your response. I'm bitter because it was super easy for my team to work in for a long period of time. Yes, I know every team has to adapt to the times. It still feels like a split, especially lately. I have a huge codebase built in 2013-2014 on what worked amazing, and still does. Naturally I asked my team to code in the same voice the code was written in. Its getting harder and some of these libraries are pretty good these days. I know we can make them work still but we avoid them if possible for consistency and readability for newer team members. Its definitely not aging well...

I'm starting to feel like that's a losing battle; and that's fine, but it sucks. I understand wanting to write things as classes, and if I could, I'd rebuild it all in a different voice today. This is quite a change though IMO and integrating some of these modules isn't what I expected, or hoped for when designing my codebase 10+ years ago. (yeah, I know, even pre es6) I'll bitch till we definitely lose this argument and have to redesign our entire codebase and my CFO and CEO will certainly love that conversation as well. Gotta keep up with the times. In the mean time we're moving a lot of pipelines to python. Specifically anything not serving up web pages.

For what its worth I did learn a few things in the last few comments. So thank you, I'll research some of that further. I've had my head in other places, no excuses there.

At the moment though, I still F'ing hate the way things are headed. lol :)

@guest271314
Copy link

@camj256 I don't understand your apparent frustration. I really don't understand an idea of bitterness. The last time I check Node.js doesn't run the world, technology is not static, time keeps on ticking. Ever try esbuild? There's a --format=cjs option...

@Rush
Copy link

Rush commented Jan 18, 2024

In the mean time we're moving a lot of pipelines to python

Ouch!! That's one hell in your face argument. :-)

@Rush Python 2 has been "deprecated" for a while.

Just because things are called "deprecated" does not mean they do not cause issues :) I think Python2/Python3 divide is mostly over but I remember wasting countless hours on this.

@divmgl
Copy link

divmgl commented Jan 18, 2024

@camj256 I don't understand your apparent frustration. I really don't understand an idea of bitterness. The last time I check Node.js doesn't run the world, technology is not static, time keeps on ticking. Ever try esbuild? There's a --format=cjs option...

This is disingenuous. ESBuild is a community built tool. Most of the complaints here are re: the underlying system that exists in Node.js without any additional tooling. There's no reason why the transition from CommonJS to ES Modules had to be so painful. I'd like to think that this was one of the many catalysts for Bun's existence.

I actually landed on an approach to solving a majority of these ES Modules issues with Node.js, TypeScript and ES Modules and that's to just use ESBuild, as you recommended. I had to build my own scripts to basically take the packages in my monorepo and convert them to valid ESBuild packages (this includes server-side packages).

Thankfully there's now an excellent tool that uses ESBuild under the hood to solve a lot of these pain points: https://github.com/privatenumber/tsx I'm super grateful that a tool like this exists (and works basically flawlessly out of the box) but all of this could have been avoided with a better migration path.

@guest271314
Copy link

@divmgl

This is disingenuous.

No. That's my genuine opinion.

Most of the complaints here are re: the underlying system that exists in Node.js without any additional tooling.

There's nothing you can do about what Node.js maintainers decide to do in the nodejs main branch. You can fork node and do whatever you want.

Yes, I have read the problem statement.

Programmers solve problems.

@guest271314
Copy link

@Rush From [Native Messaging] Python 2 was sunset January 1st, 2020 #489

https://www.python.org/doc/sunset-python-2/

Sunsetting Python 2

We are volunteers who make and take care of the Python programming language. We have decided that January 1, 2020, was the day that we sunset Python 2. That means that we will not improve it anymore after that day, even if someone finds a security problem in it. You should upgrade to Python 3 as soon as you can.

I advised the GoogleChrome/chrome-extensions-samples folks their code was broken here GoogleChrome/chrome-extensions-samples#805 and the MDN Web Docs folks their code was broken here mdn/webextensions-examples#509.

GoogleChrome folks had banned me by then. Then closed the issue. They never fixed their broken Python 2 code https://github.com/GoogleChrome/chrome-extensions-samples/blob/main/_archive/mv2/api/nativeMessaging/host/native-messaging-example-host that doesn't work on Linux anyway because there in general is no /usr/bin/python, there is a /usr/bin/python3.

MDN Web Docs also wound up mergin a working Node.js Native Messaging host example mdn/content#27228. GoogleChrome samples is still stuck in 2013.

@mk-pmb
Copy link

mk-pmb commented Jan 19, 2024

Just my default rant about JS "classes" (click to open)

@camj256

I understand wanting to write things as classes

Understanding is ok, I just hope you don't agree. Rather, teach your folks why the class-less object-centric approach is (can be proven to be) mightier than class-centric approach. Hijacking the term "Object-oriented Programming" for the latter was a genious but evil marketing move from false prophets trying to lead all mankind astray.
Last time I checked, ECMAScript still doesn't have classes, and that's a good thing. Recent editions added some ways to abuse the class keyword and a terminology of "class-like" somethings. It's a tradition for JS to introduce footguns. Firing them is entirely optional, and best left to fools.

@camj256
Copy link

camj256 commented Jan 19, 2024

@mk-pmb Yeah exactly. However, there are enough people in this community that don't agree anymore. ...and thats a problem. (clearly, as demonstrated by this thread) It's making things sloppy and less compatible. Yeah, the language is only hurting itself trying to make changes like this at this point.

Our codebase is easy to read, maintain, and is lightning fast for how large it is. For how well its designed actually I don't really want to hack together builds either.

@leonsilicon
Copy link

leonsilicon commented Apr 26, 2024

Update: Node.js (v22, and hopefully v20) now supports requiring (most) ESM code from CommonJS (nodejs/node#51977 (comment); @joyeecheung wrote an insightful post about require(esm) worth reading), so it turns out that the ESM spec wasn't actually inherently incompatible with CommonJS, but rather just due to how Node.js implemented it. So this gist (and all the incompatibility-related criticism towards the ESM spec itself) can now be considered outdated :)

@dboreham
Copy link

Ended up here after a hair-pulling fest on one of our projects.
So...JS people have managed to re-invent Python3.
In about 10 years everything will be fine and all will be forgotten.

@Rush
Copy link

Rush commented May 17, 2024

Since we're ranting, does anyone else have issues with node-gyp? Why on earth is it still using Python? Recently, I had to lock my Python version to 3.8.19 because my project won't build with newer versions. This is all due to older dependencies that rely on an outdated version of node-gyp. It's incredibly frustrating!

@JamesAlp
Copy link

Update: Node.js (v22, and hopefully v20) now supports requiring (most) ESM code from CommonJS (nodejs/node#51977 (comment); @joyeecheung wrote an insightful post about require(esm) worth reading), so it turns out that the ESM spec wasn't actually inherently incompatible with CommonJS, but rather just due to how Node.js implemented it. So this gist (and all the incompatibility-related criticism towards the ESM spec itself) can now be considered outdated :)

"Most" is definitely correct lol. For example if you wish to use Chalk, you must install 4.1.2 or earlier in your CJS project. Trying to convert your NestJS server to ESM instead of just keeping it CJS introduces so very many problems that just should not be problems. All the different imports now needing to use file extensions but only on some imports, needing to modify your tsconfig and package files with specific settings, a host of other changes and, surprise, sometimes the development build just cannot locate certain files at runtime and everything breaks! But other times when you start it up again it'll just work for that import but fail on another. I'm sure there is a solution to this... but for me the ultimate solution is to just skip all of this nonsense, stick with CJS and just install an older version of a package if I really want it.

@leonsilicon
Copy link

leonsilicon commented Jun 14, 2024

"Most" is definitely correct lol.

Most meaning ESM code that doesn't use TLA (top-level await), which is the vast majority (chalk does not use TLA)

Also that seems like a NestJS problem, not an ESM one (i.e. it's NestJS's responsibility to support ESM - it doesn't seem fair to criticize the ESM spec itself because NestJS doesn't support it well)

@guest271314
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment