weswigham/new-node.md Secret

## new-node.md

      
    Raw
  

              new-node.md
            
          
    As many of you have probably noticed, node's default module resolution algorithm, and the modules that algorithm supports, have changed a lot over the node 12 to node 15 timeframe. A lot of those changes are not directly respected by typescript, and so, today, using TS with a modern node installation may involve various workaround or hacks to get your node project to typecheck (overriding dependency types with the paths compiler option, structuring your project to avoid using newer resolution features TS doesn't understand). The current situation is suboptimal, and we've gotten here by taking a wait-and-see approach with these changes, because we didn't want to rush into supporting an experimental resolver. At this point, the new node resolution rules are unflagged, in the wild, and in use - as such, it now falls to us to update our tooling to understand node's view of the module system. There are a few broad categories of changes we need to make and, ideally, make in concert with one another so we continue to have a cohesive story for using TS to develop packages for node. I group those changes into analysis, emit, and typescript. This document is intended as a sort of broad-strokes list of things-we-should-do. You could derive seperate proposals or a todo list from this, however I think having this (quite daunting, really) list of changes all in one place, with related changes grouped together, has value in showing the cohesion the features have when considered together. Consider it a primer on the endeavor we've implicitly taken on.
First,
Analysis

These changes are what I would consider the bare minimum changes we need to make include updating our JS and TS analysis to match the runtime behavior of node (under a new moduleResolution setting). node's new commonjs resolver is significantly more complicated than the resolver we currently have implemented for moduleResolution: node - it also does much more, and allows much more complex scenarios to be described for the resolver. These new features include:

Implicit support for the .mjs and .cjs file extensions (with those providing meaning to the runtime with regards to the format of the javascript file in question)
package.json fields which alter resolution within a package, including:

"type": "module" - This changes how we should interpret .js files within the scope of the package
exports - This changes what package paths are visible to package consumers
imports - This changes what package paths are usable within the package itself
Package self-names - This allows a package to use its' own name to refer to itself
Policies (these are still under development) - These allow a package to override or remap a dependency's dependencies. Following these remappings could prove important for correct analysis and declaration lookup, since they're runtime remappings and not install-time remappings performed by npm
Throwing on the require of an esm-format-only module


The new esm resolver, which, while similar to the cjs loader, has different rules

These include respecting the package.json entries supported by the cjs resolver, but also
Resolving file:// and data:// URLs - the first of which we sort-of support already in non-node resolution, but the second of which we have no plan for. data:// url support is probably less important initially, but will likely grow in importance over time as data:// URLs are (ab)used for cachebusting and quick-and-dirty remappings for policies. The work to support them is probably greater than most else here in this list, as for proper language server support we'd need to support synthetic views on files that don't actually exist.
A complete removal of index and extension lookup (except when pulling in cjs formatted packages)
Pulling named exports out of cjs packages syntactically in a way that's similar to what we already do but not quite the same


A new system of hooks for modifying resolution at runtime - I think it's safe to mostly ignore this, at least to start. These get into the realm of needing real compiler extensions to be reasonably interpretable.

Supporting these twinned resolvers requires recognizing that in a given tree of JS, multiple module formats may interplay and each resolve against one another differently. This is substantially different from our current simplified cjs-only or esm-only approach. All of these, taken together, form a new moduleResolution: node12. (The name can be bikeshed, but node12 is the oldest version all these changes have been backported to, after being developed in the 14-15 timeframe, mostly.)
Next,
Emit

Alongside understanding the input JS structure, we also probably need to support emitting JS that in compatible with modern node. That would imply:

Emitting import() expressions as-is in cjs output (as node supports dynamic import in cjs and relies on it to import esm-format code in cjs files)
Supporting emitting both cjs-format and esm-format modules within a single package (as a static esm wrapper for a cjs core is one of the only ways to safely use singletons in a dual-mode package world) - and by this, I don't mean emitting the same file twice, but rather having different input files correspond to differing output formats. In this way, if part of your input is under a type: module scope, and part is not, we respect that when emitting .ts files as .js files. That's, I think, a key invariant we should keep - a .ts input file always produces a .js output file. Specifically, we'll always produce whatever .js format is appropriate for the file in question (node esm under a type: module scope, node cjs otherwise). Checking and resolution of these .ts files should match this. This would make our emit and typechecking of .ts files dependent on the settings in the containing package.json (in addition to the project tsconfig).
Supporting emitting .cjs and .mjs files directly in cases when type: module scopes are inflexible. In these cases, having a non-ts (or extended) extension I think makes the most sense, so this doesn't require tedious postprocessing or configuration. This is less critical than the type: module happy path, but the extensions exist to handle edge cases where that won't work, and I think we should probably handle those cases, too. To support this, we could support new TS extensions like .mts and .cts that contain the same format information the .cjs and .mjs extensions have. It's just as inelegant as .cjs and .mjs and naturally leads to needing to support .d.mts and .d.cts extensions for declarations as well. Supporting emit like this is less critical than analysis for .cjs and .mjs files - but the same declaration format questions arise for declarations for those js files (And .d.cjs would just be weird, right?).

Emit is comparably simpler for us to support, but the second bullet point potentially implies a fairly substantial change to how we transform module files. And the third is something we've resisted greatly in the past while discussing the direction the new node resolver was moving in. All of these, I would think, would be supported under a new module: node compiler option, which also implies moduleResolution: node12.
Lastly,
TypeScript

On top of these analysis and emit semantics, we need to layer TS behavior in a sensible way, as I've already alluded to.
The first and most pressing point is declaration files. Alongside all of these new resolution locations we check, we need to be able to locate a corresponding .d.ts file, and that .d.ts file needs to encode just as much format information as the .js (or .ts or .tsx or .cjs or .mjs) file it potentially came from. .js files (and, under this proposal, by extension .ts(x)?) are nicer for us, since their format information is contained in the containing package.json. As I mentioned above, it would make sense to support the other acceptable JS extensions, .cjs and .mjs through new TS extensions, like .cts and .mts from which the declaration variant would naturally follow. That is, I think, what we should largely prefer - trying to ignore the format differences or encode them in a differing manner would be both unexpected and unintuitive. To avoid unneeded growth in extension support, both of these should probably be interpreted as jsx compatible source, like .tsx. This would bring our set of recognized extensions to:

.js
.mjs
.cjs
.ts
.tsx
.cts
.mts
.d.ts
.d.mts
.d.cts
.json

which is an additional 6 beyond the 5 we currently recognize (.js, .ts, .tsx, .d.ts, .json). We could limit support for this new extensions to moduleResolution: node12 mode. These new exensions would need to affect wildcard loading for include and exclude - namely .mts, .mjs, and .d.mts should all have the same priority relationship .ts, .js, and .d.ts have today, and should be loaded alongside (rather than instead of) the existing match for include lookups (and likewise for the .cjs extension trio).
Secondly, we need to support listing types in some way in conditional exports, eg, if we have exports:
{
    "exports": {
        "/": {
            "import": "./esm/index.js",
            "require": "./cjs/index.js"
        }
    }
}
Looking up ./esm/index.d.ts and ./cjs/index.d.ts is well and good to start, but it's likely TS users may want to have their declaration output in a seperate tree, so we should look to be supporting something akin to
{
    "exports": {
        "/": {
            "import+types": "./ts/esm/index.d.ts",
            "require+types": "./ts/cjs/index.d.ts",
            "import": "./esm/index.js",
            "require": "./cjs/index.js",
        }
    }
}
by defining our own composite condition. This isn't immediately needed for the "happy path" scenario of using declaration files that emit alongside input files, but to properly support declarationDir in modern node packages, we'll have to support a pattern like this. This gets really awkward combined with typesVersions - a package.json field we already interpret to provide similar-ish functionality. Compositing the two (with backwards compatability in mind) may end up looking something like
{
    "exports": {
        "/": {
            "import+types>4.4": "./ts/esm/index.d.ts",
            "require+types>4.4": "./ts/cjs/index.d.ts",
            "import+types": "./ts4.4/esm/index.d.ts",
            "require+types": "./ts4.4/cjs/index.d.ts",
            "import": "./esm/index.js",
            "require": "./cjs/index.js",
        }
    },
    "typesVersions": {
        "<=4.3": "./tsold/mixed/handwritten.d.ts"
    }
}
TL;DR

We have quite a bit of work to look at. A lot of it only makes sense when taken as a whole, so trying to adopt minimal proposals and incremental changes, while it helps break up the tasks, reduces how integrated the resulting solution feels. We really just want a single new moduleResolution and module setting to all-up support all new node features. More piecewise introduction of support would fragment configuration and make new node package setup much harder. To accomplish this, we need to pick out the features from the above we absolutely will support, and decide how to best get them reviewed and merged together, as doing all of this at once would surely produce a PR too unwieldly to review.