This proposal encompasses several changes to the Chef file loader system and API to both simplify Chef for new users as well as enabling the development of Chef DSL extensions outside the core codebase.
Currently for attribute and recipe files, only <folder>/*.rb is loaded. This can be trivial extended to load all files in the relevant folders, allowing for non-Ruby dialects discussed below. As long as the core Ruby dialect only processes .rb files, this would be backwards compatible, though case should be taken to ignore hidden files and other things not named appropriately to be used as recipes/attributes.
Additionally this can be extended to load bare attributes.* and recipe.* files, to cover the common use case of only having a single attributes file or recipe. This is by far the default case for attributes files, and still relatively common for recipes, and simplifies things for these cases while not adding noticeable complexity to the current folder-based scheme, or likely affecting backwards compatibility (though it is possible that previously valid cookbooks that contained these files for no reason would become a syntax error). An error can be raised during loading if both the bare file and folder for a given type are present. This check can also be added to the existing knife-based cookbook checker, and external tools such as food critic. These bare files would be mapped internally to <cookbook name>::default, which is the usual name for single files currently.
One of the most complex things to explain to new users of Chef is the files/default and templates/default folder prefix. While the feature of overriding files on a per-host and per-platform basis can be useful, I suggest an alternate implementation. The #source attribute to cookbook_file and others can be augmented to support passing an array of paths, in which case the first path found to exist in the cookbook is used. This would allow fully reimplementing the current behavior like so:
cookbook_file '/etc/name' do
source [
"host-#{node['fqdn']}/name",
"#{node['platform']}-#{node['platform_version']}/name",
"#{node['platform']}/name",
"default/name"]
...
endWhile definitely somewhat verbose, it is important to remember that it is by far the exception that this behavior is used, and even less often that every piece of that lookup is desired. This new layout would also allow many other forms of overrides, such as per-role or per-environment. Due to all existing cookbooks requiring an update, this would be a backwards-incompatible change and would have to wait for Chef 12.
At least initially we can add the Array argument to source with no compatibility issues, however that means to use the simplified layout a user would have to write source ['name'] which is definitely subpar. We can do that in a point release and simply announce it as a pending deprecation for Chef 12, at which point a string argument will be interpreted as a simple path by default. This definitely gives people time to update their cookbooks, but it would be updating to an intermediary syntax and then likely changing it again after it becomes the default, which is again unfortunate. A middle ground would be to extend the default lookup path to include the bare name as the final option, so no current code will break, but users can transition directly to the new mode if they want. There is a risk of certain pathological paths simply not working with the new syntax, but these should be quite rare (ex. source 'default/name where both templates/default/default/name and templates/default/name exist) if we even see them at all. Any user hitting this kind of collision could temporarily use source ['default/name'] to make the meaning explicit. At some future point, likely Chef 12, we would simply remove the default lookup path so a string argument would just be a direct path lookup within the templates folder.
Currently attributes and recipe files use a Ruby DSL to declare node attributes and resources. While it has been one of the biggest advantages of Chef, it certainly has downsides. Not everyone wants to learn Ruby for simple tasks, and some things can become unnecessarily complex. Attribute files, for example, are general purely declarative, nested data structures which encourage a lot of repetition when adding deeply nested attributes. Adding an API to register new input dialects for attributes and recipes would allow for more flexibility and potentially make Chef more attractive to a currently underserved set of users. Libraries, resources, and providers are all more intimately tied to Ruby, and so would not be passed through the dialect system. This fits well with the goal being new users and simple uses cases however, as all three of those are relatively advanced features. New dialects could be made available as cookbooks containing a library file which calls something like Chef::Dialect.register_recipe(MyDialectClass, 'ext') and similar for attribute dialects. This would allow for the reuse of existing cookbook sharing, management, and deployment mechanisms for dialects.
Many non-Ruby developers as scared away from Chef under the assumption that they either can't learn enough Ruby to use Chef effectively, or simply do not want to. Adding other languages to Chef allows users to get started with Chef easily and quickly, only learning what they need to accomplish their immediate goal. As long as non-Ruby dialects are kept relatively similar to the Ruby dialect, users can slowly edge their way in to more advanced features as they find them needed, instead of a large upfront learning curve.
YAML provides a simple, human-readable format for nested data structures. Before someone jumps in with a note about how YAML has proven to be very difficult to armor against accidental Ruby code execution, I would remind that the current format is in fact remote code execution in a nutshell. While only the simplest of recipes could be expressed in YAML, many basic attribute files would be vastly simpler in YAML than Ruby code.
Once we have a mapping for YAML as mentioned above, adding JSON support as well would be trivial and possibly useful for some automated generation and tooling.
The proof-of-concept chef-funnel project implements what would be recipe dialects for Javascript and Python. While I would argue these should remain outside of Chef core, they would be much easier to maintain and more flexible in the long run if this API is added to Chef.
So for a new user, just getting started with Chef, maybe one day we could have their first cookbook look like this:
first/
attributes.yml
recipe.rb
templates/
config.erb
With the improvements to file and template loading, the biggest downside is breaking backwards compat in a way that will require every single cookbook (within a small margin of error) to change and will not be cross compatible on different versions of Chef. There are definitely ways to work around this, like enabling the new behavior on a per-cookbook basis in metadata.rb, and then allowing for some length of time with the old behavior as simply deprecated but present. Regardless it would have to wait for a major revision such as Chef 12.
With the dialect loading, the obvious downside is a risk of fragmentation in the cookbook community. While we can certainly create a standard that all Opscode cookbooks will only use the Ruby dialect, and we can present that to the community as a best practice for shared or reusable/library cookbooks. There will definitely be some people that share cookbooks written in non-core dialects, however as the dialect cookbooks can just be listed as dependencies in the metadata, existing tools like Berkshelf and Librarian will at least ensure that they can run. There will still be a persistent risk of raising the difficulty threshold for contributing to the community, and this will have to be monitored carefully.
Non-Ruby dialects could also lead to user frustration. This could be due to low-quality dialects leading to feelings of being a "second-class citizen" within the Chef ecosystem. It could also result from difficulties when hitting the barrier where you are forced into using Ruby, such as when making LWRPs. While this is definitely an advanced feature as mentioned before, it is still a part of the trajectory of many Chef users as their configuration becomes more complex over time. Users could also get frustrated if there is a new feature of Chef that is not yet exposed in their favored dialect, and they have to migrate that recipe to Ruby in order to use the feature. This is somewhat ameliorated by the similarity of other dialects (at least of the two existing ones) to the underlying Ruby code, so learning a non-Ruby dialect is not entirely "wasted", and much of the knowledge transfers across dialects.
One possible future direction would be adding dialects for templates as well. Mustache would be one option, but making it similarly pluggable might be interesting to explore at a later date. In a similar vein, adding dialect support for cookbook metadata.rb files might be a nice touch, and relatively low-effort.
I 😻 links. Everywhere.