Grunt Pipeline

grunt-pipeline.md

The idea behind grunt pipeline is that instead of hardcoding the inputs and outputs, a pipeline configuration will figure out which files are out of date, and only recompile the ones that are needed.

Let's look at this from a high-level, assuming input files like this:

app/
  router.coffee
  config.coffee
  app.js
  models/
    post.js
    comment.js
  controllers/
    posts.js
    comments.js
  routes/
    posts.js
    comments.js

Assume that we want to transpile the CoffeeScript files into JS, then transpile the JS files via ES6 module transpiler, and then concatenate all of the remaining JS files into a single app.js.

First Compile

Note: In practice, we wouldn't use tmp-X. Instead, we would namespace by a SHA of whatever configuration inputs were used to build the pipeline, which could include bower/npm versions of packages, the SHA any .json configuration, etc.

We will have to emulate the environment that multitasks are expecting, but it should be easy.

The best implementation strategy for the steps below is to build a full tree of all inputs and outputs before doing any work, and then pruning branches that don't need to be followed (see "Second Compile" below).

Tree for the Above Example

▼ <output>/app.js (contrib-concat)
  ▼ tmp-2/router.js (es6-module-transpiler)
    ▼ tmp-1/router.js (coffee)
      - <input>/router.coffee
  ▼ tmp-2/config.js (es6-module-transpiler)
    ▼ tmp-1/config.js (coffee)
      - <input>/config.coffee
  ▼ tmp-2/models/post.js (es6-module-transpiler)
    - <input>/models/post.js
  ▼ tmp-2/models/comment.js (es6-module-transpiler)
    - <input>/models/comment.js
  ▼ tmp-2/controllers/posts.js (es6-module-transpiler)
    - <input>/controllers/posts.js
  ▼ tmp-2/controllers/comments.js (es6-module-transpiler)
    - <input>/controllers/comments.js
  ▼ tmp-2/routes/posts.js (es6-module-transpiler)
    - <input>/routers/posts.js
  ▼ tmp-2/routes/comments.js (es6-module-transpiler)
    - <input>/routes/comments.js

Pruning Steps

Recursively walk the tree.

At every level, make a list of the direct children, and compare their actual SHAs with the SHAs in the manifest. If the SHAs match, prune the current node.

Step 1

Create a manifest, which will be flushed to manifest.json when we're done.

In each of the following steps, calculate the SHA of the input files and stick them in the manifest.

Step 2

Input files: **/*.coffee Output files: tmp-1/**/*.js Multi-Task: grunt-coffee

Step 3

Input files: {INPUT,tmp-1}/**/*.js Output files: tmp-2/**/*.js Multi-Task: grunt-es6-module-transpiler

Step 4

Input files: tmp-2/**/*.js Output files: OUTPUT/app.js Multi-task: grunt-contrib-concat

Second Compile

Step 1

Load the manifest.

Steps 2 through 4

For each of the above steps, remove input files whose SHAs haven't changed.

Again, the easiest implementation strategy would be to build up a tree of all inputs and outputs before doing anything else, and then pruning branches whose SHAs haven't changed.

Instead of creating a whole new environment for running multitasks, why not just dynamically create an alternate config target for them to run that has the subset of files you care about, then run just that target, normally?

1. user runs pipeline:foo task:target that has ["coffee:bar", "transpile:bar", "concat:bar"] defined.
2. the src patterns for coffee:bar, transpile:bar, concat:bar are expanded, analyzed, and all non-modified files are filtered out.
3. a __bar target is dynamically written into the coffee config using grunt.config.set("coffee.__bar", {...}) that contains all the options from coffee.bar but a rewritten files list.
4. step 3 is repeated for transpile and concat.
5. tasks ["coffee:__bar", "transpile:__bar", "concat:__bar"] are run.
6. the __bar targets are deleted.

Also, if these tasks are to be run in a sub-shell, a __Gruntfile.js could be written out with just the necessary config and the tasks could be run from that.