DavidWells/build-plugin-output-spec.md

## build-plugin-output-spec.md

      
    Raw
  

              build-plugin-output-spec.md
            
          
    Build Plugin Outputs Spec

Plugin outputs are a way for values to be passed between plugins.
These outputs will allow for more advances use cases of plugins that depend on values from each other.
For example, a mongodb-atlas-plugin might pass a connection string back as an output that a custom-rest-api-function-plugin might use to automatically scaffold a serverless function with said connection string.
To achieve a feature like this, Netlify Build requires additional configuration from the plugin author so outputs and build order can be correctly executed.
Considerations

Plugin outputs must work in these 2 scenarios:

Plugins defined in netlify config file
Plugins being programmatically invoked ("orchestrator plugins")

Below is a proposal on how these two can work.
1. Plugins defined in netlify config file

Here we use 2 plugins. pluginOne has required inputs and pluginTwo's output values are used in pluginOne.


// Has required config property
module.exports = function pluginOne(pluginConfig) {
  return {
    // Name of plugin
    name: 'plugin-one',
    // Config needed for plugin
    config: {
      biz: {
        type: 'string',
        required: true,
      }
    },
    // Lifecycle functions to run
    onBuild: ({ pluginConfig }) => {
      console.log(pluginConfig.biz)
    },
  }
}
// Exposes 1 output during onPreBuild phase
module.exports = function pluginTwo(pluginConfig) {
  return {
    // Name of plugin
    name: 'plugin-two',
    // Outputs returned for DAG resolution
    outputs: {
      foo: {
        type: 'string',
        // `when` is important for DAG resolution
        when: 'onPreBuild'
      }
    },
    // Lifecycle functions to run
    onPreBuild: (pluginAPI) => {
      // Output value `foo` is returned from onPreBuild phase
      return {
        foo: 'hello'
      }
    },
  }
}


# Netlify Config
build:
  publish: build
  functions: functions

plugins:
  - package: plugin-one
    config:
      biz: ${pluginTwo.outputs.foo}
      # ^ output referenced from `id`
      # given to plugin-two
  - package: plugin-two
    id: pluginTwo


As an aside, the configuration could also be ordered like so:
# Netlify Config
build:
  publish: build
  functions: functions

plugins:
  - package: plugin-two
    id: pluginTwo
  - package: plugin-one
    config:
      biz: ${pluginTwo.outputs.foo}


What happens on build:
Inputs & outputs dependancies are ordered up front via a DAG and cycles throw an error.

Required inputs validated in plugin-one. Output syntax is recognized, the when value is read & lifecycle is checked to verify ordering works.
Order works, so onPreBuild functionality from plugin-two runs
Then output foo from plugin-two is returned. ${pluginTwo.outputs.foo} then fully resolved to hello.
onBuild functionality from plugin-one runs with it's biz config value set to hello
Then build ends


2. Plugins used programmatically

Here we use 3 plugins. The plugin-one & plugin-two are programmatically used in a third plugin called orchestratorPlugin. Because they are programmatically used, the order of lifecycle methods in plugin-one & plugin-two do not matter. Effectively they are used as normal NPM modules.


Programmatic usage from orchestrator plugin. Because these are used programmatically and are not defined in the Netlify config file, they can be called in any order the user wishes.
const pluginOne = require('plugin-one')
const pluginTwo = require('plugin-two')

module.exports = function orchestratorPlugin(config) {
  return {
    // Name of plugin
    name: 'orchestrator-plugin',
    // Config needed for plugin
    config: {
      optOne: {
        type: 'string',
        required: true,
      },
      optTwo: {
        type: 'string',
        required: true,
      }
    },
    // Lifecycle functions to run
    onInit: async ({ pluginConfig }) => {
      // initialize plugins with config
      const one = pluginOne({ biz: pluginConfig.optOne  })
      const two = pluginTwo({ zaz: pluginConfig.optTwo })

      const [outputFromOne, outputFromTwo] = await Promise.all([
        one.onPreBuild(pluginAPI),
        two.onBuild(pluginAPI),
      ])
      // Do custom stuff with outputFromOne / outputFromTwo
    },
  }
}
module.exports = function pluginOne(pluginConfig) {
  return {
    // Name of plugin
    name: 'plugin-one',
    // Config needed for plugin
    config: {
      biz: {
        type: 'string',
        required: true,
      }
    },
    // Outputs returned for DAG resolution
    outputs: {
      wow: {
        type: 'string',
        when: 'onBuild'
      }
    },
    onPreBuild: ({ pluginConfig }) => {
      console.log(pluginConfig.biz)
      return {
        wow: 'nice'
      }
    },
  }
}
// Has required config property
module.exports = function pluginTwo(pluginConfig) {
  return {
    // Name of plugin
    name: 'plugin-two',
    // Config needed for plugin
    config: {
      zaz: {
        type: 'string',
        required: true,
      }
    },
    // outputs returned for DAG resolution
    outputs: {
      wow: {
        type: 'string',
        when: 'onBuild'
      }
    },
    onBuild: ({ pluginConfig }) => {
      console.log(pluginConfig.zaz)
    },
  }
}


build:
  publish: build

plugins:
  - package: orchestrator-plugin
    config:
      optOne: hello
      optTwo: goodbye


What happens on build:
Because they are programmatically used, the order of lifecycle methods in plugin-one & plugin-two do not matter.

Required inputs validated from read config file
orchestrator-plugin loads & onInit functionality runs with config set to optOne: hello & optTwo: goodbye
plugin-one & plugin-two are initialized with config
Then plugin-one.onPreBuild & plugin-two.onBuild methods are called
Then output from plugin-one.onPreBuild & plugin-two.onBuild are referenced in the code of orchestrator-plugin.
Then the build ends

Implementation Proposal

The way values are resolved and lifecycle events are ordered depend on the inputs & outputs of each event listener.

There are probably multiple ways to implement something like this. It would be important to note how mature tools like terraform & cloudformation use DAG as their mechanism for resolving & ordering operations.

I'm proposing we use a DAG algorithm to resolve the order in which plugin lifecycle methods fire.
The DAG will tell us the order in which the build steps should happen. It will also be able to detect cycles and throw an error if plugin outputs don't exist yet where they are being used as plugin config (inputs).
The DAG implementation really only effects plugins that are defined in the Netlify configuration file vs the plugins that are called programmatically (because they are just function calls & resolve themselves in user code)
Proposed Resolution flow:

Related issues


older outputs example netlify/build#494
older syntax discussion netlify/build#396
orchestrator plugins netlify/build#603

Feedback and comments welcome