Instantly share code, notes, and snippets.

Embed
What would you like to do?
Thoughts on how to write faster broccoli plugins
  1. Ensure your build method returns a promise, and any file system or network operations are done in an asynchronous fashion when possible.
  2. Do not share memory between instances of your broccoli plugin. This prevents it from being used in parallel. All data should be in local or instance variables or in the caches or the tmp directory.
  3. Cache the output of your plugin, if the input is the same and your plugin is n:1 from inputs to output, make sure you aren't rerunning your code.
  4. Note there are two caching modes, build and rebuild. Build is when you are starting from persistent caches only, while rebuild is when files that you are watching have changed. Both cases need to be optimized for.
  5. Caching is very important. Learn all about hashes and cache keys here:
  1. Please remember there is no need for your hash functions to be cryptographically secure. Choose fast hashing functions, like trivial, CRC32, MD5.
  2. If you are thinking of extending from:
  • broccoli-writer, consider broccoli-caching-writer
  • broccoli-filter, consider broccoli-persistent-filter
  1. Do not just use a directory name as the input tree to a broccoli plugin, use UnwatchedDir or WatchedDir from broccoli-source.
  2. Need to recursively traverse all files in a node?
  • Walk and create a list first, then process second, do not walk and process as you go.
  • Consider using node-walk or node-walk-sync instead of fs.walk or glob.sync.
  1. Update to the latest versions of any code you are using, especially other broccoli plugins.
  2. A single poorly performing broccoli plugin can greatly slow down an entire build. Make sure your plugin isn't the weak link in the speedy pipeline.
  3. Check out heimdall visualizer to see where your build is slowing down: https://github.com/rwjblue/heimdalljs-visualizer https://rwjblue.github.io/heimdalljs-visualizer/ (instructions?)
@oligriffiths

This comment has been minimized.

oligriffiths commented Jun 29, 2018

  1. ensure your build method returns a promise, and any file system operations are done in an asynchronous fashion. Some things will have to be sync, for example, shelling out to a CLI process for example, so saying everything must be async is a little misleading.
  2. good!
  3. agreed, links to how to do this would be great. We need to improve this API, and I actually think making plugins opt-out (default is opted in) would be a good API change.
  4. nice
  5. nice
  6. perhaps explain the difference between the caching modes (build vs rebuild)
  7. perhaps an example might be useful here, when would someone do this?
  8. this is cool, something that broccoli should really provide OOTB
  9. yup
  10. perhaps some links to how to enable the heimdall visualizer output https://github.com/rwjblue/heimdalljs-visualizer https://rwjblue.github.io/heimdalljs-visualizer/ would be good here

This is a good list, thanks for writing up.

(Edited to match numbers up - Gaurav)

@Gaurav0

This comment has been minimized.

Owner

Gaurav0 commented Jun 29, 2018

Thank you @oligriffiths, will edit.

@nightire

This comment has been minimized.

nightire commented Jun 30, 2018

  1. I remember broccoli-unwatched-tree and broccoli-watched-tree are deprecated? Is https://github.com/broccolijs/broccoli-source a better choice?
@Gaurav0

This comment has been minimized.

Owner

Gaurav0 commented Jun 30, 2018

@nightire. Thanks. Updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment