- Yes to "tiny" cookbook layouts
- Yes to removing implicit filespecificity
- Yes to user-defined, explicit filespecificity
- Yes to modular, modifiable code that can support dialects
- No to including any dialects in core or supporting them beyond best-effort Open Source Community courtesy.
Chef's biggest strength is that it is an internally reprogrammable system. You can read Martin Fowler's piece on the topic here: http://martinfowler.com/bliki/InternalReprogrammability.html but the summary is that you don't have to ask permission to add new features to Chef or change existing ones. Chef provides a nice DSL that allows you to succinctly express how you want your machines to be configured, but your recipes are just code that looks exactly the same to the Ruby VM as Chef's internal code.
The flip side of this power is a lot of responsibility: Chef performs a fundamental task in your organization, and if it stops working, say, because the code you used to extend it no longer works with the latest version, you're immediately searching for workarounds (at best).
This leads me to two conclusions: clearly, it's beneficial if Chef's internal code and your extensions can come to an agreement on how to work together. This means that having defined hooks for adding new functionality, swapping out components, etc. is a good thing.
Secondly, it means that anyone working in Chef is never going to be on the path to full mastery if they avoid working in Ruby; any "dialect" is going to be a second class citizen, no matter what (one could hypothetically write a full interface to the Ruby VM via C, but this seems insane).
In an article (somewhat) inspired by the Fowler article above, Neal Ford discusses different kinds of tools as being either "contextual" or "composable:" http://nealford.com/memeagora/2013/01/22/why_everyone_eventually_hates_maven.html A particularly memorable line from that article: "Maven is a classic contextual tool: it is opinionated, rigid, generic, and dogmatic, which is exactly what is needed at the beginning of a project."
The purpose of Chef cookbooks having a proscribed layout is the same. Every (non-absolute-beginner) Chef user knows where to look for things, because they're always in the same place. When you create a new cookbook, you don't spend any time thinking about how to organize it, because there's only one option. This is a good thing.
On the flip side, however, the structure is optimized for a certain size and complexity of cookbook. If anyone used merb back in the day, one neat thing about it was that the directory structure was completely customizable (for experts, anyway) and it supported "small," "medium," and "large" application layouts out of the box. The cool thing about this was that each one had its own convention for where things should go, so you retained the ability for everyone to understand what parts of a project were in what files.
In summary, it's pretty sensible for there to be different sizes of cookbook, provided they're generally standardized (even if only by community convention and not code convention) and it's not too difficult to migrate from one size to another as a cookbook becomes more complicated.
I'm skeptical that dialects will do much to make Chef easier to learn overall. I rarely see Ruby syntax being a barrier to learning except among those who have a pre-formed bias against it before learning Chef.
I can certainly imagine that in some organizations, there will be some class of user who has no desire or need to exceed a certain threshold of mastery, and this user will have ready access to experts for help troubleshooting or for assistance with more ambitious tasks. In this environment I can see dialects being a perfectly pragmatic way for people to get shit done.
Outside of a closed environment like that, however, my suspicion is that the problems will vastly outweigh the benefits. For example, in the "bad old days" before Omnibus packages, everyone brought their own Ruby when installing Chef, and before they could even get Chef to print a version number had to deal with and begin to understand various methods of installing Ruby, Rubygems, why their distribution of choice had modified those things to work in surprising and non-default ways, etc. One mitigating factor was that everyone else using Chef had a reasonable chance of having encountered the same issues or knowing how to troubleshoot them. If we fragment the community, the "helpfulness density" will decrease. For example, how many Chef users will immediately know what to do if my npm installation is broken?
That said, I'm opposed to telling you that you can't do this, and in any case, technically, I can't stop you from monkey patching your way around anything I might try to stop you. So I think the best option is to come to a fruitful compromise where functionality like dialects can be added to Chef via stable APIs, and anyone interested can experiment with it or run it in production to their heart's content, but it's not a feature in core.