Last active
October 8, 2021 13:17
-
-
Save waylan/703743f0d5caa6651e1036e310fe1dac to your computer and use it in GitHub Desktop.
Proof of concept for yaml inherit tag. Unfortunately, defining the tag is ugly. This was based on the code in [this comment](https://gist.github.com/joshbode/569627ced3076931b02f#gistcomment-2309157).
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Unforunately, the tag can't be a standalone item by itelf. | |
# After all, the following document is invalid: | |
# foo | |
# a: bar | |
# Therefore the tag needs to fit in with the structure of the document. | |
# I have used a null key and then discard it in the loader | |
# so that it is not in the final output. | |
null: !inherit parent.yml | |
b: baz | |
c: bar |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import yaml | |
import os.path | |
def merge(a, b): | |
""" Merge two documents. TODO: implement deep merge""" | |
a.update(b) | |
return a | |
class InheritLoaderMeta(type): | |
def __new__(metacls, __name__, __bases__, __dict__): | |
"""Add include constructer to class.""" | |
# register the inherit constructor on the class | |
cls = super().__new__(metacls, __name__, __bases__, __dict__) | |
cls.add_constructor('!inherit', cls.construct_inherit) | |
return cls | |
class InheritLoader(yaml.Loader, metaclass=InheritLoaderMeta): | |
"""YAML Loader with `!inherit` constructor.""" | |
def __init__(self, stream): | |
"""Initialise Loader.""" | |
try: | |
self._root = os.path.split(stream.name)[0] | |
except AttributeError: | |
self._root = os.path.curdir | |
self._parent_data = None | |
super().__init__(stream) | |
def construct_inherit(self, node): | |
"""load parent file referenced at node.""" | |
filename = os.path.abspath(os.path.join( | |
self._root, self.construct_scalar(node) | |
)) | |
extension = os.path.splitext(filename)[1].lstrip('.') | |
with open(filename, 'r') as f: | |
if extension in ('yaml', 'yml'): | |
self._parent_data = yaml.load(f, InheritLoader) | |
else: | |
self._parent_data = ''.join(f.readlines()) | |
return '' | |
def construct_document(self, node): | |
""" Merge parent with document.""" | |
doc = super().construct_document(node) | |
parent = self._parent_data | |
if parent is not None: | |
# Reset parent data for a multi-document file | |
self._parent_data = None | |
if None in doc: | |
# remove empty None key | |
doc.pop(None) | |
return merge(parent, doc) | |
return doc | |
if __name__ == '__main__': | |
with open('example.yml', 'r') as f: | |
data = yaml.load(f, Loader=InheritLoader) | |
print(data) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
a: foo | |
b: bar |
Hey waylan,
for a project I also ended up writing a yaml parser like yours that supports inheritance. For the declaration of the base file I chose this:
!extend : base.yml
So the value after the constructor is simply None, and then the parser knows it has to evaluate what's been assigned to this key.
Although one tends to forget the ":" in between, which leads to confusing error messages, it's quite clean and readable. And the wording "extend" is taken directly from jinja2, which I hoped would be intuitive for users that know it.
Nice to see I'm not the only one missing a de-facto tool for DRY and extendable config files. 👍
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It is possible to define a tag at the root of the document. However it has to fit into the document structure. And a document comprised of key-value pairs cannot have a random string which is neither a key nor a value of a key. For example, this is not a valid document:
Therefore, neither is this:
That's why I went with:
Then before doing the merge, I remove the empty
null
key.Interestingly, in Example 2.24 of the spec, there is a tag defined on the document start token which presumably defines the type of the object to be returned (in the example, the type would be a
shape
and the items in the document define the properties of the shape).However I couldn't find a way to define the parent in this case. The tag
shape
defined the type, but there is no value attached directly to the tag. In fact, this is invalid:--- !inhereit parent.yml
. Andinherit
without defining the document to inherit from is not useful.Similarly, I couldn't work out how the tags on the
-
items worked in that example. When I tried it I just got syntax errors.I also looked at anchors, but they must be nested in a key mapping as part of the value, so that isn't useful.
Another option I explored was a directive. A directive goes on the first line (starts with a
%
) and its purpose is to inform the parser how to process the document. Seems ideal for this purpose. However, the spec clearly states that only the directives defined in the spec are allowed and there is no provision for private directives. I suppose we could ignore that, but the parser doesn't provide a public API for defining your own directives, so it would be a hack.