#Sometimes You Need to Build a Larger Thing First
If you wish to make an apple pie from scratch, you must first invent the universe. -- Carl Sagan
In the beginning, I didn't expect Shadow DOM to be all that complex. The concept seemed straightforward enough, and when I hacked initial chunk of flesh out of the XBL2 spec, there wasn't all that much. Boy, was I wrong.
...
In an ideal Extensible Web Manifesto scenario, we study the problem, determine overall solution, and carefully carve out the primitives until there aren't any left. Perhaps I should note that this is almost never possible when adding brand-new features to a platform.
In reality, the problem space is fuzzy and over-constrained, and determining the shape of overall solution is non-trivial. Because of how deeply Shadow DOM was reaching into core DOM, it was impossible to fully evaluate without runnable code. A good way to put it is this: had we started with a polyfill, the gaps in polyfill fidelity would have been so large that we could've easily gotten lost in them.
In situations like these, the only realistic path forward is to start writing code and feeding it to web developers, sometimes by force.
As one dramatic example, we didn't realize that distributions were even necessary until people started building actual things with our implementation. It was a sobering moment, stages of grief and all that.
So we kept using developer feedback as a searchlight, and refined our approximation until it fit the solution space. After three and some years of these iterations, we have a full implementation that fills in the solution space nearly completely, with specs and a test suite.
Throughout the process, I kept the layering mantra in the back of my mind. What are the smaller primitives that Shadow DOM is built on? Are they useful by themselves? Are they consistent with how Web platform works or maybe even already used in other places?
Despite these questions not being fully answered, we pressed on. We needed to press on, because while the platonic ideals are great, the power in user's hands is better than a beautiful system that will potentially exist in some idyllic future.
As a matter of fact, we Extensible Web Manifesto folks need to accept this as a good, useful pattern: build something that works and commit to the overall shape of the solution, but understand that it's made of smaller primitives. Get that solution into developers' hands, and then improve and add power to it by exposing those smaller primitives. Use developers feedback to prioritize what primitive pops out next.
The main requirement of using this pattern is needing to continuously check your work and exercising the crap out of the currently-hidden primitives: do they explain something else in the platform? If not, are they consistent with the rest of the platform?
This is the “start-high, dig-down” approach that you sometimes need to employ when new primitives won’t necessarily show direct value to users. E.g., would CSP be useful without the header version? Probably not. Now the challenge is to describe what it is and how it works. Both are valuable, but providing value to users is Job #1. Explaining the platform is about providing more value to more users. Filling in the picture, as it were.
It’s important that we don’t allow ourselves to be fooled into thinking that there’s immaculate conception for web platform features, no matter our good intentions. The intent to layer, to provide an explanation, etc. doesn’t mean you can just provide primitives or just provide high-level stuff... and birthing something new that has both totally explained is too hard most of the time. We need to be impatient both for power and for explanations, but we should have a bias towards power first. If we live to fight another day, we have all the time we’ll need to explain it.
The good news is that this approach equally applies to even seemingly smallest primitives: you never know when an atomic thing turns to be a couple or more of different even smaller things. Continuous primitive archeology FTW.