TypeScript Compiler API

README.md

Requirements/Goals:

• Plugins must be loaded via tsc.
• Plugins must be loaded in language service.
• Plugins must be loaded explicitly (i.e. from tsconfig.json), and not automatically from node_modules.
• Plugins may need custom configuration.
• Plugins may need an activation event to support initialization.
• Plugins may need a deactivation event to support cleanup.
• Minifier plugins must to be able to perform a pre-emit walk of program and return custom transformations for emit.

Open Questions:

• How should plugin configuration be applied?
  • Should we follow Babel's model for configuration?

    "plugins": ["plugin-1", ["plugin-2", { }]]
    

  • Or should we use a simple hash map?

    "plugins": { "plugin-1": true, "plugin-2": { } }
    

• How should we enforce a minimum typescript-api version for plugins?
• How much context do we need about a plugin to report useful diagnostics (i.e. % time spent in plugin, etc.)?
• Should we be maintain a plugin blacklist for known problematic plugins (poor performance, unstable, etc.)?
• Should we follow a naming scheme similar to Babel plugins (i.e. typescript-plugin-x or ts-plugin-x)?

api.d.ts

// actually will be just the .d.ts from compiler, plus the following declarations:
export interface CompilerPluginContext {
    readonly compilerHost: CompilerHost;
    readonly compilerOptions: CompilerOptions;
    readonly options: MapLike<any>; // plugin config options
}

export type CompilerPluginHookResult<T> = PromiseLike<T> | T;

export interface CompilerPluginModule {
    activate?(context: CompilerPluginContext, args: CompilerPluginActivationArgs): 
        CompilerPluginHookResult<CompilerPluginActivationResult | void>;
    
    preParse?(context: CompilerPluginContext, args: CompilerPluginPreParseArgs): 
        CompilerPluginHookResult<CompilerPluginPreParseResult | void>;
    
    preEmit?(context: CompilerPluginContext, args: CompilerPluginPreEmitArgs): 
        CompilerPluginHookResult<CompilerPluginPreEmitResult | void>;

    deactivate?(context: CompilerPluginContext): CompilerPluginHookResult<void>;
}

export interface CompilerPluginResult {
    diagnostics?: ReadonlyArray<Diagnostic>;
}

export interface CompilerPluginActivationArgs {
}

/**
 * An optional result that can be returned from the `CompilerPluginModule.activate` hook.
 */
export interface CompilerPluginActivationResult extends CompilerPluginResult {
}

export interface CompilerPluginPreParseArgs {
    readonly rootNames: readonly string[];
    readonly projectReferences: readonly ProjectReference[] | undefined;
}

export interface CompilerPluginPreParseResult extends CompilerPluginResult {
    rootNames?: readonly string[];
    projectReferences?: readonly ProjectReference[];
    preprocessors?: readonly TransformerFactory<SourceFile>[];
}

export interface CompilerPluginPreEmitArgs {
    /** The current `Program`. */
    readonly program: BaseProgram;
    /** The `SourceFile` that is about to be emitted, or `undefined` when emitting all outputs. */
    readonly targetSourceFile: SourceFile | undefined;
    /** A `CancellationToken` that can be used to abort an operation when running in the language service. */
    readonly cancellationToken: CancellationToken | undefined;
}

/**
 * An optional result that can be returned from the `CompilerPluginModule.preEmit` hook.
 */
export interface CompilerPluginPreEmitResult extends CompilerPluginResult {
    customTransformers?: CustomTransformers;
}

export interface CompilerPluginDeactivationResult extends CompilerPluginResult {
}
    
// NOTE: All factory methods originally on `ts` have been moved to `NodeFactory`, which is accessible
// from the `factory` property of a transformation context. Existing factory methods exist for
// backwards compatibility when using the full TypeScript Language Service or TS Server API, but
// will not be accessible for Compiler Plugins.
export interface NodeFactory {
    createNodeArray<T extends Node>(elements?: readonly T[], hasTrailingComma?: boolean): NodeArray<T>;
    createNumericLiteral(value: string | number, numericLiteralFlags?: TokenFlags): NumericLiteral;
    ...
}
    
export interface TransformationContext {
    // The `NodeFactory` you should use to create nodes.
    // NOTE: Preprocessors (returned from the `preParse` hook) and Emit Transformers (returned
    // from the `preEmit` hook) will be provided *different* `NodeFactory` instances and should not be conflated.
    readonly factory: NodeFactory;
    ...
}

export interface BaseProgram {
    // essentially the same as `Program`, except the `emit` method is not provided.
    ...
}

my-plugin.index.ts

// NOTE: Import from "typescript" as normal, which is redirected to the active compiler.
import * as ts from "typescript";

// triggered when the extension is loaded
// - opportunity to inspect/mutate compiler options
// - opportunity to hook compiler host functionality
// - opportunity to return user-defined diagnostics
// - may be asynchronous (i.e. return a Promise)
export function activate(context: ts.CompilerPluginContext) {
  // return { diagnostics };
}

// triggered immediately before parse begins
// - opportunity to customize the root names or project references
// - opportunity to inject custom preprocessors
// - opportunity to return user-defined diagnostics
// - may be asynchronous (i.e. return a Promise)
// NOTE: Custom preprocessors should only be used to augment the AST with additional type information
//       and should not be used to change runtime semantics. This restriction is not currently enforced,
//       but may be in the future.
export function preParse(context: ts.CompilerPluginContext, 
                         { rootNames, projectReferences }: ts.CompilerPluginPreParseArgs) {
  // return { diagnostics, rootNames, projectReferences, preprocessors };
}

// triggered immediately before emit begins
// - opportunity to inject custom emit transformers
// - opportunity to return user-defined diagnostics
// - may be asynchronous (i.e. return a Promise)
export function preEmit(context: ts.CompilerPluginContext, 
                        { program, targetSourceFile, cancellationToken }: ts.CompilerPluginPreEmitArgs) {
  // return { diagnostics, customTransformers };
}

// triggered when the extension is unloaded (i.e. in between compilations in watch mode)
// - opportunity to perform cleanup
// - may be asynchronous (i.e. return a Promise)
export function deactivate(context: ts.CompilerPluginContext) {
  // return;
}

my-plugin.package.json

{
  "name": "typescript-plugin-my-plugin",
  "version": "1.0.0",
  "main": "index.js",
  "typescriptPlugin": {
    "activationEvents": ["preEmit"],
    "pluginDependencies": ["other-plugin"],
  },
  "dependencies": {
    "typescript-api": "^3.4.0",
    "other-plugin": "^1.0.0"
  }
}

project.tsconfig.json

{
  "compilerOptions": {
  },
  "plugins": {
    "my-plugin": true,
    "other-plugin": { "options": "here" }
  }
}

When should plugins be loaded/unloaded? Is there a compiler-wide "initialization" phase this effectively hooks into? Or is activation done on demand/lazily, just prior to the first real hook that needs to be run? Under what scenarios can a plugin be unloaded?

Plugins must be loaded only from tsconfig.json.

It seems reasonable to me to allow a language server host to inject plugins at will (given that it could always edit the tsconfigs before allowing the compiler to read them anyway). A blessed entrypoint for this would be good, I think.

Minifier plugins must to be able to perform a pre-emit walk of program and return custom transformations for emit.

As it is, I don't see how that'd work with this API? You can hook into preEmit, but how do you add transforms for the emit step? Will we add an API onto Program to edit the compiler options? Another member on the CompilerPluginPreEmitResult?

Extension: I don't think plugins should be viewed in isolation - namely I think a one package = one plugin loaded model is a little constraining. Look at the "presets" ecosystem around babel - they're effectively plugins that just collect other plugins into a group. It's not strictly necessary, but I think exposing something like

declare namespace ts {
    export function register(name: string, plugin: CompilerPlugin): void;
}

to allow a single plugin to, during its activate hook, register other plugins could be useful in that regard.

Ah! On the subject of names: CompilerHost provides a trace method for debugging - I imagine plugin authors will probably use it and find it useful for tracking plugin status alongside other compiler traces. It would be nice if the CompilerHost we pass to a given plugin wraps the trace method to prefix the output with the plugin's name (so plugin output is easier to identify in log files).

Also worth noting: The CompilerHost in a language service has a very limited lifetime. The host we pass to a plugin should probably be a proxy to a real host, which after the normal synchronous execution of the plugin expires, we poison the methods of and remove the reference to the underlying "real" host, to prevent any methods getting saved off and (potentially accidentally) used asynchronously and potentially poisoning or retaining old programs.

When should plugins be loaded/unloaded? Is there a compiler-wide "initialization" phase this effectively hooks into? Or is activation done on demand/lazily, just prior to the first real hook that needs to be run? Under what scenarios can a plugin be unloaded?

My initial thoughts are that activate and deactivate would run whenever we created or destroyed a compiler host in a context in which plugins could be run. For a "watch mode"-like scenario, that would mean we would deactivate and then reactivate plugins in between each triggered compilation.

Plugins must be loaded only from tsconfig.json.

It seems reasonable to me to allow a language server host to inject plugins at will (given that it could always edit the tsconfigs before allowing the compiler to read them anyway). A blessed entrypoint for this would be good, I think.

That's not the goal of that statement. Rather, the goal is to state that plugins are not loaded automatically from node_modules, and instead need to be loaded explicitly.

Minifier plugins must to be able to perform a pre-emit walk of program and return custom transformations for emit.

As it is, I don't see how that'd work with this API? You can hook into preEmit, but how do you add transforms for the emit step? Will we add an API onto Program to edit the compiler options? Another member on the CompilerPluginPreEmitResult?

That was an oversight, I've added a customTransforms property to CompilerPluginPreEmitResult as well as a note that we will need to specify other options as necessary.

Extension: I don't think plugins should be viewed in isolation - namely I think a one package = one plugin loaded model is a little constraining. Look at the "presets" ecosystem around babel - they're effectively plugins that just collect other plugins into a group. It's not strictly necessary, but I think exposing something like

declare namespace ts {
   export function register(name: string, plugin: CompilerPlugin): void;
}

to allow a single plugin to, during its activate hook, register other plugins could be useful in that regard.

I'd generally prefer to be more declarative, specifying something akin to VS Code's activation events in the plugin's package.json, and possibly plugin dependencies (which would need to be added to both the "dependencies" map as well as the plugin's configuration in package.json).

For reference, tsickle's emit function is defined here (emitWithTsickle):
https://github.com/angular/tsickle/blob/master/src/tsickle.ts#L92

This would work in the context of this proposal I think, except two bits:

• tsickle produces an additional ${name}.externs.js file (containing Closure Compiler "externs" definitions for any ambient / declare APIs defined in .ts files). The name of the file is given as a custom option in tsconfig.json.
• EmitTransfomers has a beforeTsickle set of transforms - I think these were previously used by Angular's compiler, but might be no longer needed in the newer Angular version ("ngIvy").

The writeFileDelegate work does a textual .d.ts transformation by appending some definitions, but I believe we can replace that with a proper .d.ts transformer, now that that's supported.

This is the plugin API we use when running tsc under Bazel/Blaze:
https://github.com/bazelbuild/rules_typescript/blob/master/internal/tsc_wrapped/plugin_api.ts

we have an implementation of the Angular compiler in terms of this API. It can add new rootFiles to the Program (we have some "synthetic" outputs like "*.ngfactory.js"). we might be able to drop that requirement in the future though, once we roll out the new Angular compiler.

For the new Angular compiler, it looks like this would mostly work except:

• We make use of the CompilerOptions, which don't seem to be available in the proposed API. For example, we call resolveModuleName which requires the CompilerOptions and a CompilerHost.

I believe the TransformationContext of a transform gives access to the CompilerOptions, but this is too late in the lifecycle for our compiler.

• For incremental compilation, plugins need to be able to pass information about a previous compilation through to the next one. This looks like it might be possible by saving a reference to the Program and checking when it changes, but an explicit lifecycle for this would be much cleaner.

Currently we know whether a SourceFile has changed by checking referential equality between the same file in the old and new Programs. Will this method still work?

• For incremental compilation, Angular semantics can be non-local. An edit to file module.ts might require that we re-emit component.ts, even if component.ts hasn't changed on disk itself, because the transform output will have changed. Would a plugin be able to do this by calling Program.emit? I could see issues if that caused preEmit to be reentrant, or if it skipped over the plugin chain entirely.

@alxhub I think you can program.getCompilerOptions().

I've prototyped putting tsickle on this. This all looks fine, though I have two questions:

With this API, it's unclear to me when a plugin can emit diagnostics into the array that preEmit returns. We could walk the AST once to produce diagnostics eagerly in the preEmit hook, and then a second time in the returned transformers without producing diagnostics (but carefully still avoiding the error conditions). That seems expensive though, and would also require somewhat awkward code that implements the same walking and error detection logic twice.

Another question is how a plugin would be notified of the actual emit happening for a (1) an individual source file and (2) the compilation unit as a whole. We need to emit extra outputs, some per file, some per compilation unit (i.e. once for a run of tsc over a tsconfig.json). With the API as is, I can see how we can write files through the compiler host, but I wouldn't know when is the right moment to do so. On preEmit seems wrong, within some transformer seems worse :-)

I've updated my compilerPluginModel branch with the latest changes for the compiler plugin prototype. Notable changes:

• Added a preParse hook to allow plugins to change root names and project references, and add "preprocessor" transforms (intended to be used to inject type annotations and casts into existing source code).
• All hooks can now be asynchronous (i.e. return a Promise).
• Accessing the TypeScript API from a plugin is achieved using import * as ts from "typescript" syntax.
• All factory functions normally used to create/update ASTs have been moved to a special NodeFactory object. You can access a NodeFactory from within a preprocessor/transformer by accessing the factory property of the provided TransformationContext:

  import * as ts from "typescript";
  function myTransformer(context: ts.TransformationContext) {
    // old: createIdentifier("foo")
    // new:
    const { factory } = context;
    factory.createIdentifier("foo")
  }

• The parser now also uses a NodeFactory to create AST nodes.
• Exceptions thrown by a plugin are wrapped in user-code diagnostics.