Skip to content

Instantly share code, notes, and snippets.

@waylan
Last active October 8, 2021 13:17
Show Gist options
  • Save waylan/703743f0d5caa6651e1036e310fe1dac to your computer and use it in GitHub Desktop.
Save waylan/703743f0d5caa6651e1036e310fe1dac to your computer and use it in GitHub Desktop.
Proof of concept for yaml inherit tag. Unfortunately, defining the tag is ugly. This was based on the code in [this comment](https://gist.github.com/joshbode/569627ced3076931b02f#gistcomment-2309157).
# Unforunately, the tag can't be a standalone item by itelf.
# After all, the following document is invalid:
# foo
# a: bar
# Therefore the tag needs to fit in with the structure of the document.
# I have used a null key and then discard it in the loader
# so that it is not in the final output.
null: !inherit parent.yml
b: baz
c: bar
import yaml
import os.path
def merge(a, b):
""" Merge two documents. TODO: implement deep merge"""
a.update(b)
return a
class InheritLoaderMeta(type):
def __new__(metacls, __name__, __bases__, __dict__):
"""Add include constructer to class."""
# register the inherit constructor on the class
cls = super().__new__(metacls, __name__, __bases__, __dict__)
cls.add_constructor('!inherit', cls.construct_inherit)
return cls
class InheritLoader(yaml.Loader, metaclass=InheritLoaderMeta):
"""YAML Loader with `!inherit` constructor."""
def __init__(self, stream):
"""Initialise Loader."""
try:
self._root = os.path.split(stream.name)[0]
except AttributeError:
self._root = os.path.curdir
self._parent_data = None
super().__init__(stream)
def construct_inherit(self, node):
"""load parent file referenced at node."""
filename = os.path.abspath(os.path.join(
self._root, self.construct_scalar(node)
))
extension = os.path.splitext(filename)[1].lstrip('.')
with open(filename, 'r') as f:
if extension in ('yaml', 'yml'):
self._parent_data = yaml.load(f, InheritLoader)
else:
self._parent_data = ''.join(f.readlines())
return ''
def construct_document(self, node):
""" Merge parent with document."""
doc = super().construct_document(node)
parent = self._parent_data
if parent is not None:
# Reset parent data for a multi-document file
self._parent_data = None
if None in doc:
# remove empty None key
doc.pop(None)
return merge(parent, doc)
return doc
if __name__ == '__main__':
with open('example.yml', 'r') as f:
data = yaml.load(f, Loader=InheritLoader)
print(data)
a: foo
b: bar
@waylan
Copy link
Author

waylan commented Oct 30, 2020

It is possible to define a tag at the root of the document. However it has to fit into the document structure. And a document comprised of key-value pairs cannot have a random string which is neither a key nor a value of a key. For example, this is not a valid document:

foo
a: bar

Therefore, neither is this:

!inherit parent.yaml
a: bar

That's why I went with:

null: !inherit parent.yaml
a: bar

Then before doing the merge, I remove the empty null key.

Interestingly, in Example 2.24 of the spec, there is a tag defined on the document start token which presumably defines the type of the object to be returned (in the example, the type would be a shape and the items in the document define the properties of the shape).

--- !shape
- !circle
  center: &ORIGIN {x: 73, y: 129}
  radius: 7
- !line
  start: *ORIGIN
  finish: { x: 89, y: 102 }
- !label
  start: *ORIGIN
  color: 0xFFEEBB
  text: Pretty vector drawing.

However I couldn't find a way to define the parent in this case. The tag shape defined the type, but there is no value attached directly to the tag. In fact, this is invalid: --- !inhereit parent.yml. And inherit without defining the document to inherit from is not useful.

Similarly, I couldn't work out how the tags on the - items worked in that example. When I tried it I just got syntax errors.

I also looked at anchors, but they must be nested in a key mapping as part of the value, so that isn't useful.

Another option I explored was a directive. A directive goes on the first line (starts with a %) and its purpose is to inform the parser how to process the document. Seems ideal for this purpose. However, the spec clearly states that only the directives defined in the spec are allowed and there is no provision for private directives. I suppose we could ignore that, but the parser doesn't provide a public API for defining your own directives, so it would be a hack.

@J3ronimo
Copy link

J3ronimo commented Oct 8, 2021

Hey waylan,

for a project I also ended up writing a yaml parser like yours that supports inheritance. For the declaration of the base file I chose this:

!extend : base.yml

So the value after the constructor is simply None, and then the parser knows it has to evaluate what's been assigned to this key.
Although one tends to forget the ":" in between, which leads to confusing error messages, it's quite clean and readable. And the wording "extend" is taken directly from jinja2, which I hoped would be intuitive for users that know it.

Nice to see I'm not the only one missing a de-facto tool for DRY and extendable config files. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment