The following is my thinking on how we will get a good balance between sharing configuration and having understandable configuration.
Unfortunately, I think the current plan of having the provisioner compare bids accross clouds is too aggressive for the current time. The time and money units that different clouds make available aren't easily comparable (and in gcp's case, it is hard to get the information in an understandable way in general). We can still allow bidding within a provider, but each workertype will need to specify which provider is responsible for creating its workers.
This has side benefits of simplifying the provisioning loop and the provider interfaces considerably. Each provider will still need to provide a standardised way to list/terminate/etc instances but that should be doable.
The current rules-based system for generating worker configuration is proving difficult to implement efficiently and might be difficult to reason about in practice. Let's choose a simpler system.
Each provider will have a set of schemas that define what a valid worker configuration is. It will also have a set of schemas that define a valid form of shared config will look like. An example of this would be awsRegionsMapping in aws-reference.yml below.
This will be all hardcoded into the aws provider itself. At docs generation time, it will publish this reference in the same way that we do for api/event/log references. This way it will be available in the UI to validate inputs as well as showing up in the documentation. I believe with this system, we can validate that all configurations are correct at the time they are created/edited rather than waiting for a loop of the provisioner to tell us what went wrong.
The nice part about these shared configs is that it would be easy for whatever is generating new AMI to update the configuration and know that all workertypes will pick up their new config.