Strawperson proposal on interfacing multiple tools

README.md

Attempts to solve the problem of multiple tools working together, which I talk about here.

Everything in this document is debatable.

The way I see it, there are multiple parts to the solution:

1. A file which multiple tools can read, which allows them to understand what's going on in a specific project, syntax-wise. This is only at the project level.
2. An interface transpilers can implement, placed at a certain file location (or specified by some package.json field)

compilerrc.json

This file can be placed in an ES project, to define what tools are required to run this project, and what output optimisers it requires.

Other instances of this file, when found in other folders, should have no effect.

{
  "languageExtensions: {
    "\\.tsx?$": {
      "tool": "typescript",
      "options": {
        "foo": "bar"
      },
      "except": "./node_modules/"
    },
    "\\.css?$": {
      "tool": "postcss",
      "options": {
        "foo": "bar"
      },
      "except": "./node_modules/"
    },
    "./docs/.+\\.md$": {
      "tool": "mdx",
      "options": {...}
    }
  },
  "toolchain": {
    "./src/.+\\.ts$": [
      { "tool": "some-macros", options: {...} },
      "./global-definer.js"
    ]
  },
  "optimization": {
    "./node_modules/.+\\.min\\.js": [],
    "./(src|docs)/.+": [
      "./some-pre-terser-step.js",
      {
        "tool": "terser",
        "options": { "onlySafeFor": "thisProject" }
      }
    ],
    ".+": ["terser"]
  }
}

As you can see, this makes it possible to assign certain tools to certain paths at certain phases.

• languageExtensions is for the first part of the toolchain. It's reserved for added syntax. Their output must be standard ES.
• toolchain is where source-to-source transforms live.
• optimization is for the output. It's not a part of the toolchain, because it happens after.

It's also important to mention what's left out in this syntax:

• The bundler is left out, because it's just one of the tools which consumes this file
• The linter is also left out, because it also has to read this file to understand how modules are resolved (such that you can get warnings about unused exports, or imports which are the wrong type, for example).
• Typecheckers, much like linters, would not output code, but instead check the code. A second part of the typechecker would be responsible for transforming the code into standards-compliant ES.

It might be useful to allow for conditions in some way, so that production code can behave differently, and non-production code may include helpers.

toolchain link interface

This is the interface for one step of the toolchain. A transpiler, such as babel, typescript, MDX, SVGR, or even PostCSS can implement this.

It's centered around a function which takes code and returns code. The input may or may not be ES, but the output must be ES.

// node_modules/some-lib/transpilerrc.jsm

export default async ({
  input: { code, sourceMap, contentType },
  outputContentType,
  resolve
}) => {
  const { code, sourceMap, contentType } = input
  
  assert(contentType === 'application/javascript')
  assert(outputTypes.includes(outputContentType))
  
  const outputESTree = outputContentType === 'text/x-estree'
  
  const transpiled = doThisToolsThing({ code, sourceMap, getCodeExternalToThisModule: resolve, outputESTree })
  
  if (outputESTree) {
    return { code: transpiled, sourceMap: null }
  }
  
  return { code: transpiled.source, sourceMap: transpiled.sourceMap }
}

export const inputTypes = ['application/javascript']
export const outputTypes = ['application/javascript', 'text/x-estree']  // I don't actually know the content type for ESTree

Key places:

• The resolve function is an asynchronous function that asks the caller to find a module. It takes the same string you would pass to import(). If returning this module would cause a cyclical dependency, a special value is returned so this case is correctly handled (maybe null?)
• The default export is the star of the show. It receives code, accompanied with a source map, and returns code, accompanied with a source map.
• inputTypes / outputTypes - These specify what this function accepts, and what it returns. The caller needs to know these beforehand. If a certain project supports only a certain very unique kind of AST, it should also support application/javascript so that every tool doesn't have to evolve to interpret every single AST.

Possible extensions to this protocol:

• A mutable object argument holding the state of the current compilation, so that one link in the toolchain may leave information for others.
• Requiring the ouptut of the function to be JSON serializable. This makes it possible to use multiple Workers to bundle a project together.