no-probe-resolution.md

TypeScript type resolution without probing

Supporting expressing resolution entirely in code

Problem

TypeScript’s module resolution for type definitions currently heavily relies on probing. Some examples of this are how node_modules/ folders are probed for @types/ packages when a bare specifier is imported that does not provide it’s own types, or how importing a file with a .js extension will resolve types to a sibling file with a .d.ts extension instead.

Probing is problematic for us at Deno, because we are unable to perform any kind of probing when importing files using https:// specifiers. This is because it is neither side-effect free to perform probing on https:// (ie it is observable), and it is incredibly slow because of network round trip times. Non Deno TypeScript users have also reported similar issues with probing due to reliance on network file systems (microsoft/TypeScript#11979). Additionally, probing has a performance impact for all users regardless of disk speed, because work needs to be performed that would not have to be if explicit resolution was used instead of probing.

While existing Node resolvers rely heavily on probing, Deno’s resolution behaviour, like browsers, does not rely on probing. Our TypeScript behaviour reflects this in multiple ways:

• Deno does not probe for sibling .d.ts files
• Deno does not auto-discover @types/ for packages imported with npm: specifiers
• Deno does not allow importing .ts files using .js extensions

This is a problem for Deno, because TypeScript does not currently provide any alternatives to probing for multiple of the behaviours outlined.

To work around this in Deno, we have two custom extensions to TypeScript in Deno that enable type resolution without probing:

• // @deno-types="...." to override the types of an import, at an import site
  • This is useful to specify the explicit @types/ package for a given runtime import without having to probe
• An overloaded /// <reference types="./foo.d.ts" /> that enables specifying the type declarations to use for a module, at that modules’ source declaration site.
  • If we could do this again, we would not have overloaded TypeScript semantics.
  • This is useful for specifying the .d.ts file to use for a .js source, especially when you have manually written a .js + .d.ts pair

We opened an issue about this many years ago, but we have not had traction on this so far: microsoft/TypeScript#33437.

Finally, with the new JSR project, we’d like to be able to generate .js and .d.ts pairs that will resolve consistently, regardless of the current position of these files (in node_modules/ or not), regardless of the moduleResolution used, and regardless of any external factors.

Proposed solution to override probing

We are proposing to upstream three separate new features into TypeScript that would eliminate the requirement on probing that TypeScript currently has. Additionally it would allow us to deprecate the existing Deno specific override comments proposed above.

We propose that these features are enabled in all moduleResolution modes, particularly so that they can be used in modules published and consumed via node_modules/ folders where the code author does not control the consumers moduleResolution mode.

We are very happy to put in the work to implement and write tests for this within TypeScript.

Add a // @ts-types="<specifier>" comment, enabling users to explicitly choose the type definition used for an import

This comment can be placed immediately above any import / export statement, or dynamic import() expression, following similar semantics to // @ts-ignore. Exact details about how dynamic imports are targeted and where a comment is valid are still TBD.

When specified above a import statement or dynamic import, TypeScript will use the specifier in this comment instead of the specifier in the import statement or dynamic import during type checking. During emit of JavaScript code this comment is ignored. The specifier in the import statement or dynamic import is not rewritten on emit - it is emitted as is.

During declaration emit, imports / exports annotated with this comment will be rewritten to use the specifier in the comment. Dynamic imports are never emitted into .d.ts file, so there is no risk of having to rewrite a dynamic specifier. If a dynamic import needs to turn into a type import expression, this expression will use the specifier from the // @ts-types comment.

The comment is not valid on import type or export type statements.

If no comment is specified, the current behaviour is preserved.

Example TS input:

// @ts-types="./index.d.ts"
export * from "./dist/index.js";

// @ts-types="./foo.d.ts"
const foo = await import("./dist/foo.js");

// @ts-types="./bar.d.ts"
import { Bar, bar } from "./dist/bar.js";
export const barred: Bar = bar();

JavaScript emit:

export * from "./dist/index.js";

export const foo = await import("./dist/foo.js");

import { bar } from "./dist/bar.js";
export const barred = bar();

TypeScript declaration emit:

export * from "./index.d.ts";

export declare const foo: import("./foo.d.ts");

import { Bar } from "./dist/bar.js";
export declare const barred: Bar;

Add a // @ts-placeholder-replace-file-types="<specifier>" comment, which can be placed at the top of the file

Name very much open to bikeshedding.

This comment can be placed at the top of a file, and is valid in the same positions as @jsx pragma comments. Only one comment per file is allowed.

When specified at the top of a .js file, TypeScript will use the specifier in this comment as the type definitions for this file, instead of the inferred type definitions that are based on the file’s source code.

Example JS input:

// @ts-placeholder-replace-file-types="./index.d.ts"

export function bar() {
	console.log("bar");
}

JavaScript emit:

// @ts-placeholder-replace-file-types="./index.d.ts"

export function bar() {
	console.log("bar");
}

No TypeScript declaration emit, because input is JS.

In order to maintain backwards compatibility, and not regress performance for probing resolvers, the comment is only respected if the (probing) resolver found the .js and would attempt to generate type declarations from the JS code using inference. Specifically this means that probing resolvers will prefer type declarations found through probing. For example if a index.js file has a // @ts-placeholder-replace-file-types="./other-index.d.ts", but a index.d.ts file also exists as a sibling of this file, and a third file foo.ts contains import import "./index.js", and a probing resolver such as moduleResolution: nodenext or moduleResolution: bundler is used, the types imported from foo.ts will still be ./index.d.ts, because the probing behaviour found this file before finding the JS source code.

Add a jsxImportSourceTypes option to compilerOptions, and a // @jsxImportSourceTypes ... pragma

This option is required because otherwise without @types/ probing, you can not easially specify types for react.

When specified, TypeScript will not use at the jsxImportSource compiler option or pragma during type checking. During emit of JavaScript code, TypeScript will still emit the specifier in jsxImportSource to import the JSX factory. During emit of declaration files, TypeScript will emit the specifier in the jsxImportSourceTypes if it needs to reference JSX related types. If no jsxImportSourceTypes option / pragma is specified, the current behaviour is preserved (jsxImportSource used for both runtime and type emit / type checking).

Example input:

/* @jsxRuntime automatic */
/* @jsxImportSource react */
/* @jsxImportSourceTypes @types/react */
export default <div>Hello World!</div>

During type checking, JSX types are loaded from @types/react

JavaScript emit:

import { jsx as _jsx } from "react/jsx-runtime";
/* @jsxRuntime automatic */
/* @jsxImportSource react */
/* @jsxImportSourceTypes @types/react */
export default _jsx("div", { children: "Hello World!" });

TypeScript declaration emit:

declare const _default: import("@types/react/jsx-runtime").JSX.Element;
export default _default;

FAQ

Is this “explicit module resolution”?

This does not propose adding a moduleResolution: "explicit" or moduleResolution: "minimal" from microsoft/TypeScript#50152 yet. This adds all the features that are required to make this useful in the future.

We think that moduleResolution: "bundler" + a linter can handle many of the same cases as an explicit module resolution mode.

We are not opposed to an explicit module resolution mode, but do not think this anywhere near as high of a priority as the features proposed here.

If TypeScript were to add this resolution mode, we think that together with the three features proposed here, one could express any module resolution configuration TypeScript has right now using the minimal resolution mode + explicit resolution comments. This would make an ideal compile target, and we’d ensure that code emitted by JSR would work in this configuration.

These are mostly my own thoughts/questions/responses, with a bit from the team mixed in via our design meeting discussion.

High-level question

• Deno’s custom extensions to TypeScript resolution have been working for Deno so far. What are the JSR-specific problems or motivations for this proposal, and what workarounds are currently in place?

// @ts-types="<specifier>" (in TS files)

we’d like to be able to generate .js and .d.ts pairs that will resolve consistently, regardless of the current position of these files (in node_modules/ or not), regardless of the moduleResolution used, and regardless of any external factors.

we think that together with the three features proposed here, one could express any module resolution configuration TypeScript has right now using the minimal resolution mode + explicit resolution comments. This would make an ideal compile target

It’s a little unclear what all the implications of these two comments are, but there’s something important to keep in mind. Declaration files must be representative of the JavaScript files they describe, and the behavior of the TypeScript compiler is driven by what it understands those JavaScript files to contain (based on the declaration files) and the host system it understands to be running the code or resolving modules (based on compilerOptions). This has two immediate implications that might be relevant for JSR's transformers and these proposals:

1. It’s not safe for any system to modify declaration files without making an equivalent change to JavaScript files, or vice versa. For example, you couldn’t replace a module specifier "./foo" in a .d.ts file with "./foo.js" without also changing the corresponding import in the .js file, because that would make the declaration file misrepresent the JavaScript file and falsely add/remove checker errors. Similarly, bundling JavaScript files together necessitates bundling their declaration files together with a similar structure to be perfectly safe and accurate.

2. Since different (runtime/bundler) module resolvers work differently, it’s generally a non-goal for TypeScript to “bake in” the result of module resolution into declaration files. For example, it wouldn’t make sense to generate from (using your proposed syntax):

   // @ts-types="../internal/foo.d.ts"
   import { Foo } from "#internal/foo";

   a declaration file like:

   import { Foo } from "../internal/foo.d.ts";

   because different module resolvers might set different conditions and resolve to a different JavaScript file, necessitating TypeScript to resolve to a different declaration file. This kind of “baking in” is technically never safe, though it might be reasonable to make an exception for fully-specified relative paths:

   // @ts-types="./foo.d.mts"
   import { Foo } from "./foo.mjs";

   But this seems to remove much of the value of the feature.

Here’s an alternate strawman proposal: when transforming files being published to JSR, replace all internally-resolving module specifiers with their fully-specified, relative output-extension-containing path (e.g. replace "./foo" or "./foo.ts" with "./foo.js"). Then, when generating declaration files, annotate any import that has a JS file extension but no corresponding declaration file with a // @ts-untyped annotation. Then propose that TypeScript and Deno recognize this annotation as a signal to look at the JavaScript file itself for type information, while unannotated imports resolve with declaration file extension substitution:

// foo.d.ts
export * from "./bar.js"; // loads ./bar.d.ts
// @ts-untyped
export * from "./baz.js"; // loads ./baz.js

For internal imports, this would allow both Deno and TypeScript to resolve everything consistently without any probing. What scenarios would this not cover?

@ts-placeholder-replace-file-types (in JS files)

• There were concerns about how this would affect existing (probing) moduleResolution modes; however, this is addressed in the document:

  In order to maintain backwards compatibility, and not regress performance for probing resolvers, the comment is only respected if the (probing) resolver found the .js and would attempt to generate type declarations from the JS code using inference. Specifically this means that probing resolvers will prefer type declarations found through probing. For example if a index.js file has a // @ts-placeholder-replace-file-types="./other-index.d.ts", but a index.d.ts file also exists as a sibling of this file, and a third file foo.ts contains import import "./index.js", and a probing resolver such as moduleResolution: nodenext or moduleResolution: bundler is used, the types imported from foo.ts will still be ./index.d.ts, because the probing behaviour found this file before finding the JS source code.

• What scenarios does Deno/JSR need this for?
• Why is the inconsistency with existing moduleResolution behaviors acceptable? My understanding was that the desired outcome was to allow published code to be resolvable without probing, and that tsc -p on that code (downloaded to disk) should result in the same module graph as deno check.

// @jsxImportSourceTypes

This feels a little sketchy to me for a slightly different reason from the ones mentioned earlier, since node_modules/@types lookups are already a TypeScript-specific resolution feature. What happens if someone uses this comment to force TypeScript to look up @types/react, but then a future version of react, present in the end-user’s tree, starts shipping its own types? (Others on the team expressed concerns with this that I’m not sure are represented by this question alone.)

Thanks for looking at this.

Deno’s custom extensions to TypeScript resolution have been working for Deno so far. What are the JSR-specific problems or motivations for this proposal, and what workarounds are currently in place?

Yes, in general this has worked relatively well for Deno. Issues arise at the boundary between "TypeScript in tsc" and "TypeScript in Deno". Namely, it is very difficult to write code right now that resolves identically for both source and declarations in both Deno and TypeScript. This means that when writing source code to publish to JSR, you can either be targeting "Deno's TypeScript" or "tsc's TypeScript", but not both at the same time. This means that if you are writing code to work both in Deno and in Node, but are targeting "Deno's TypeScript", one can not easially test that code in Node without complex source transforms. This is especially bothersome when using the in-editor completions. Code written for "Deno's TypeScript" needs to use Deno's LSP, and code written for "tsc's TypeScript" needs to use tsserver. If TSC's behaviour were to be a superset of Deno's behaviour, this becomes less of a problem.

We work around this right now in JSR by doing two relatively complex import specifier transforms. One during publishing, and one during npm tarball emit. The first transform during publish takes in TS source code authored for either Deno, or tsc with the bundler resolution mode (roughly), and generates TS source code that explicitly encodes resolution behaviour for both source and definition resolutions. For example, if the original source code uses bundler resolution (which probes for sibling definition files), and a user imports a .js file with a sibling .d.ts file, we will annotate the .js import with a // @deno-types="./name.d.ts" comment. This transform also happens when you are using the bundler resolution mode and have an import for "express". In this case, we will annotate the import with a // @deno-types="npm:@types/express@..." comment.

We then perform a second more complex transform during npm tarball emit where we walk the TS source module graph starting at all of the exports, emit JS and DTS files, and in these rewrite all specifiers to their resolved source or definition forms as per the explicit annotations in the TS source code. For example when you have the following input files:

# main.js
/// <reference types="./types/main.d.ts" />

export { A } from "./a.js";
export const foo = 1;

# types/main.d.ts
export { A } from "../a.js";
export const foo: number;

# a.js
/// <reference types="./a.d.ts" />

export class A {
  foo = 1;
}

# a.d.ts
export class A {
  foo: number;
}

# jsr.json
{
  "name": "@scope/foo",
  "version": "0.0.1",
  "exports": {
    ".": "./main.js"
  }
}

The following output is generated:

== /a.d.ts ==
export class A {
  foo: number;
}

== /a.js ==


export class A {
  foo = 1;
}

== /types/main.d.ts ==
export { A } from "../a.d.ts";
export const foo: number;

== /main.js ==


export { A } from "./a.js";
export const foo = 1;

== /package.json ==
{
  "name": "@jsr/scope__foo",
  "version": "0.0.1",
  "homepage": "http://jsr.test/@scope/foo",
  "type": "module",
  "dependencies": {},
  "exports": {
    ".": {
      "types": "./types/main.d.ts",
      "default": "./main.js"
    }
  },
  "_jsr_revision": 0
}

Both of these transforms are relatively complex, because they can not occur without knowing the resolution state of the entire program (ie files can not be emitted independently). If TS offered a target that allowed us to emit source and definition files that themselves have the expressivity of the resolution in the input files, we would not have to perform this stateful rewriting of specifiers during the NPM tarball emit, and could reduce to it a much more reasonable "every .ts extension is rewritten to .js, and all .js files have a comment at the top of the file to indicate the relevant .d.ts file.

RE: // @ts-types="" (in TS files)

We are in agreement that the .js and .d.ts transforms have to be in sync. Neither JSR's transform or Deno ever transform js source independently of d.ts and vice versa.

For internal imports, this would allow both Deno and TypeScript to resolve everything consistently without any probing. What scenarios would this not cover?

If I am understanding correctly, in this scenario, Deno would still need to probe for sibling .d.ts files.

I missed one of the critical use cases in the original explainer: when you want to specify that express's types are actually located at @types/express, because without probing, this can not be done automatically because there is no dependency manifest (it would need to be probed for). As such, you need to annotate express imports (in TS source) with an explicit annotation that types can be found in @types/express. This scenario would not be covered by the @ts-untyped.

Since different (runtime/bundler) module resolvers work differently, it’s generally a non-goal for TypeScript to “bake in” the result of module resolution into declaration files.

I don't understand this. In the described scenarios you are not generally baking in module resolution behaviour. Your are just baking in behaviour in cases where the user is explicitly wants to ignore the resolution behaviour of the source specifier and wants to override instead. I am not suggesting you bake in this resolution in all cases (ie when .js is in source, you emit .d.ts into definition etc). This baking in only happens when the user explicitly wants to specify their own types.

For example, when importing express in TS source, with a // @ts-types="@types/express" annotation, the JS file could contain express, while the definition file could contain either import ... from "@types/express" or /* @ts-types="@types/express" */ import ... from "express";. In what cases would this break something?

RE: @ts-placeholder-replace-file-types (in JS files)

What scenarios does Deno/JSR need this for?

When a user explicitly writes a JS file (for example code generated) and wants to provide a definition file for this JS file. Because Deno does not probe for sibling definition files, this link between source and definition must be explicit. A common scenario we see is code generated js files from WASM build tools, with separate code generated d.ts files.

Why is the inconsistency with existing moduleResolution behaviors acceptable?

Because we were unable to come up with another solution that would not regress TS performance significantly. In the limit, we'd like to propose a moduleResolution: "explicit" mode that performs no probing which would not exhibit this behaviour in a local project. For now we'd "fix" the situation by adding a rule during publishing that .d.ts files next to .js files are not allowed if there is no // @ts-placeholder-replace-file-types comment in the JS file, pointing to that .d.ts file.

It's not great - there may be better solutions here. Do you have any ideas?

RE: @jsxImportSourceTypes

The concerns here also apply to @ts-types then I imagine? This is no different to having an imaginary @ts-types comment for jsxImportSource.

If React starts shipping it's own types, the types in the explicit comments are preferred. I don't think this is any different from:

import { foo as foo_ } from "untyped-library";
// untyped-library is missing types for `foo 
const foo: "string" = foo_;