Broccoli challenges

ember-cli-build.js

/* jshint node: true */

const EmberApp = require('ember-cli/lib/broccoli/ember-app');
const generateWhitelabelIndexes = require('./generate-whitelabel-indexes');

module.exports = function(defaults) {
  const app = new EmberApp(defaults, {
    // ...all sorts of config
  });

  let appTree = app.toTree(trees)
  return generateWhitelabelIndexes(appTree);
};

foray.md

Broccoli / Ember CLI challenges

Today David Baker of the Ember.js Learning Core Team helped me devise a solution for a Broccoli use case and encouraged me to share some of my learnings and stumbling blocks.

Use Case

As part of some of our effort to improve our ability to smoke test our various whitelabeled Cordova/Ember apps, I needed a way to generate a separate index.html for each of our whitelabels.

Here's the relevant bits of our Ember app folder structure:

app/
  index.html
branding/
  storemart/
    config.json
  buynlarge/
    config.json
  ...

After an ember build or ember serve, I needed the output folder to look like this:

dist/
  assets/
    ...js and css, etc
  storemart-index.html
  buynlarge-index.html

In short, the requirements were:

1. For each **/config.json in the top level branding folder, I needed to generate an properly named index.html file with that config.json embedded inside of it.
2. This processing step should take place after broccoli-asset-rev has fingerprinted all the included assets.

The Challenge

1. It's easiest to write a Broccoli plugin when you're dealing with a 1-to-1 input to output file mapping; in this case, depending on how you look at it, it's an M:N mapping depending on how many config.json there are in the branding/ folder, or on a per-config-file basis, it's a 2-1 mapping: [index.html, branding/BRAND/config.json] => BRAND-index.html
2. I was vaguely aware of post-processing hooks that could be implemented in addons, but a) I suspected there'd be issues with non-guaranteed order of postprocessing when I needed my plugin to run after the broccoli-asset-rev postprocessor and b) I already had a sizeable learning curve on my plate with figuring out how to write the broccoli plugin that I didn't want to mess with in-repo addons.

The Solution

I ended up using Broccoli Multifilter which was explicitly made to help write Broccoli plugins with M:N dependencies (such as Sass plugin which needs to account for Sass files specifying their own internal imports and dependencies).

I passed the finalized (and asset-fingerprinted) Ember app tree obtained via app.toTree(trees) to the new processing step, which:

1. Creates a broccoli funnel for config.json files in the top level branding/ folder
2. Passes the app tree and the config.json funnel to a customer Multifilter plugin
3. The Multifilter plugin determines the current globbed array of config.json files from the funnel input node, and passes it to buildAndCache.
4. For each entry you pass as the first arg to buildAndCache, the Multifilter will invoke a callback giving you an opportunity to a) take files from the input folders and synchronously write them to output folders, and b) declare a list of dependent input files to watch for changes, so that if any of them change, this callback will be invoked again, giving you the opportunity to process the changed files and declare any new dependencies. This simple API provides a lot more power than I needed for my simple use case, but it allowed me to declare the 2:1 mapping I needed.

I'm happy with the solution we arrived at, though certainly curious as to the alternate approaches I could have taken.

Stumbling blocks

Even though it didn't end up being a lot of code in the end, it took me almost a full day to figure out. I figured I'd share some of the roadblocks and points of confusion that I encountered.

Get-in-get-out nature of writing Broccoli plugins

I rarely have to write Broccoli plugins. When I do, I struggle for a day, get something working, and generally speaking all the lessons learned fall out of my brain because I have no reason to revisit that code for the next decade.

This isn't really a criticism, and I'm not sure I know of what the alternative is; Ember's a high level abstraction that runs atop the low-level Broccoli abstraction, and by framework design, it should be rare when you have to dip into Broccoli, but those handful of times that you do have to dig into writing Broccoli plugins (that are just slightly off the beaten path), it's a real pain.

I'm guessing a lot of other frameworks require you to get your hands dirty with webpack config, which on the one hand might steepen the learning curve make it harder to standardize around framework conventions, but on the other hand, you're less helpless to build what you need to build when you need to go off the beaten path.

In-repo Addon

It's possible that I could have had a simpler time getting things working if I'd defined an in-repo addon with the postprocessTree hook (I still would have needed to figure out how to model the M:N depenendency with Multifilter), but that's a LOT to take on.

Generally speaking, it feels weird to have to escalate to a heavy abstraction such as addons just to implement a build tool hook; Rails has a nice pattern where most of the callbacks they expose (for routing, validation, etc) can be implemented with inline lambdas, but when things get too messy you can define your hooks and any other complicated logic in a class (e.g. custom validators or routing constraints).

Knowing that I might have to reach for an in-repo addon greatly inflated my sense of dread when tackling this problem.

Side note: I've felt this dread before when tackling complicated use cases with ember-cli-deploy; generally speaking, I feel like it's a good rule that most things that can be achieved with an (in-repo) addon should be achievable with a blob of code in ember-cli-build.js.

Which kind of Broccoli plugin do I need?

When you're pretty certain you need to write a Broccoli plugin, it's not clear where to start. Some quick googling and you might run into the following as potential starting points:

1. https://github.com/broccolijs/broccoli-plugin
2. https://github.com/broccolijs/broccoli-filter
3. https://github.com/stefanpenner/broccoli-persistent-filter
4. https://github.com/ember-cli/broccoli-caching-writer
5. https://github.com/broccolijs/broccoli-multifilter

You basically get the gist pretty quickly that broccoli-plugin is too low level for most needs, but when you're a Broccoli noob, it's not clear what your needs are or what a plugin needs to do to fulfill the Broccoli contract. I think you need to have some understanding of the following:

1. How are dependencies modeled in your plugin? 1-to-1 input file to output file?
2. How can plugin output be cached to prevent needless rebuilds?

For me, I couldn't quite place my needs, and it felt very daunting that even to experiment with one of these plugins, I would have felt like I needed to correctly nail the MD5 cache hashing strategy/API for it to even work. Again, this could just be par for the course, I'm just sharing what's the challenging about this stuff.

Array of inputNodes vs single merged inputNode?

Some broccoli plugins in the ecosystem accept a single input node as an argument, and others expect an array of input nodes. This is still very challenging for me to wrap my brain around: is there a conceptual difference between an array of input nodes, vs a single input node that is the result of a merge of that array of nodes? Are there some things you can do with one that you can't do with the other?

In my solution, I pass two separate nodes to my Multifilter plugin: the processed app tree node and the funnel of whitelabel config.jsons; but I think I could have also written this as a Multifilter plugin that accepts a single already-merged node and produces the same output?

I'm not exactly sure of my source of confusion; in the end I think I solved the problem without fully understanding it.

How to mutate a tree?

The plugin I wrote takes two nodes (the processed app tree and a funnel of whitelabel configs) and produces a folder containing only BRAND-index.htmls, but if that's all I returned, then the only contents of my dist/ folder would be those html files, and none of the JS assets. When I realized that I couldn't just return the result of the Multifilter node, I had that sense of dread again of "oh god I've made a huge mistake" and figured I'd fundamentally misunderstood Broccoli once again.

After some random Googling and Githubbing I found a pattern that I could use to apply my changes to the original app output tree, which seems obvious enough, but still strikes me as odd and unintuitive:

  return new Merge([appTree, whitelabelHtml], {
    overwrite: true
  });

In other words, if you're trying to mutate the tree, you have to express the mutation as a Broccoli plugin, and apply the mutation by merging it back into the tree with overwrite: true. I get it, it makes sense. But it was very hard to find what I was looking for and the learning curve and mental model didn't nudge me in the direction to thinking this is what Broccoli actually wants me to do. I also wonder if there's a performance hit involved in modeling it this way, but I don't know the Broccoli internals enough to comment, nor have I done any measuring, so maybe you should just strike this perf musing from your mental record.

Funnel feels heavy

This is probably an ignorant observation, but it seems a common use case that Broccoli plugins need to share an API for selecting files that they operate on. In my case, I used a Funnel to select config.json files from a separate branding/ directory, but instead of just selecting the relevant files for use in the next Multifilter plugin, or some reason, I had to think about what destDir they should be written to. This is weird for me to wrap my head around: why does such a utility plugin need to write to a folder at all? Can't it just be an in-memory mapping or virtual file or something that a downstream plugin can consume? It just made me think "hmm maybe I'm using the wrong plugin here or misunderstanding Broccoli".

Conclusion

It was awesome seeing my solution work in the end (especially awesome seeing how adding/deleting config.json files would add/delete the BRAND-index.html files), but it was a long and unpleasant path getting there. Again, this could just be the nature of build tooling, or I might just suck at this stuff, or both. I just figured I'd share in case there are some insights as to how to better document the Broccoli mental model and its ecosystem of plugins.

Maybe it'd be nice if someone wrote a cookbook or a website like http://goshdarnblocksyntax.com/ to help guide you to what you actually need to build.

It could be a procession of questions, like:

1. Have you checked to see whether it's already built? (EmberObserver link)
2. Is it a 1-to-1 transformation from input file to output file?

• Subquestion: do you care about caching/efficieny and are you comfortable generating hashes?
  • Yes: use ABC
  • No: use XYZ

3. For M:N dependencies, try broccoli-multifilter

Anyway, please let me know if you've got any insights to how else I could have modeled this, or whether you've encountered the same struggles.

generate-whitelabel-indexes.js

let Multifilter = require("broccoli-multifilter");
const fs = require('fs');
const path = require('path');
const glob = require('glob').sync;
const Merge = require('broccoli-merge-trees');
var Funnel = require('broccoli-funnel');

class GenerateWhitelabelIndex extends Multifilter {
  build() {
    let root = this.inputPaths[1];
    let configJsons = glob(`${root}/whitelabels/**/config.json`).sort().map(match => {
      return path.relative(root, match);
    })

    const indexPath = path.join(this.inputPaths[0], "index.html");

    return this.buildAndCache(
      configJsons,
      (inputFile, outputDirectory) => {
        let brandName = inputFile.split('/')[1];

        let configPath = path.join(root, inputFile);
        let configString = fs.readFileSync(configPath).toString();

        let contents = fs.readFileSync(indexPath).toString();

        contents = contents.replace(/FPR_BRANDING_CONFIG.*/, `FPR_BRANDING_CONFIG = ${configString};`)
                           .replace(/FPR_BUILD_DATA.*/, `FPR_BUILD_DATA = ${configString};`);

        let fullOutputPath = path.join(outputDirectory, `${brandName}-index.html`);
        fs.writeFileSync(fullOutputPath, contents);

        return {
          dependencies: [ configPath, indexPath ]
        };
      }
    );
  }
}

module.exports = function(appTree) {
  let whiteLabelConfigs = new Funnel('branding', {
    include: ['**/config.json'],
    destDir: "whitelabels",
  });

  let whitelabelHtml = new GenerateWhitelabelIndex([appTree, whiteLabelConfigs], {
    annotation: "Generate Whitelabel Indexes",
  })

  return new Merge([appTree, whitelabelHtml], {
    overwrite: true
  });
}

index.html

<!DOCTYPE html>
<html class="">
  <head>
    <!-- other meta stuff -->
    
    {{content-for 'head'}}

    <link rel="stylesheet" href="assets/vendor.css">
    <link rel="stylesheet" href="assets/customer-ember.css">
    <link rel="manifest" href="manifest.json">
  </head>

  <body>
    <script>
      window.FPR_BRANDING_CONFIG = { replace: "me" };
    </script>

    <script src="assets/vendor.js"></script>
    <script src="assets/customer-ember.js"></script>
  </body>
</html>

thanks this is great