OTIO has a schema upgrading system, which allows newer versions of OTIO to read files from older versions of the library. There is no way to create files that are compliant with older versions of the library, however.
The goal is to build a system into OTIO that allows interoperability both forwards and backwards in version numbers.
Some of the challenges to be solved:
- There are some missing features in the existing system, such as awareness of schema name changes and unit test deficiencies that need to be addressed
- The "Future information" problem - either older versions of OTIO need to be taught how to downgrade schemas, or future versions of OTIO need to know what to downgrade to be compliant with a given version of OTIO. This includes situations where schema names need to be changed. OTIO currently versions all the leaf schemas independently, so some mechanism needs to exist that allows a system to downgrade to the correct collection of versions associated with a given release of OTIO.
- User interface - if old versions could do it automatically, as with
upgrading, that would be great, but I suspect a new tool or flag will need to
be added that allows
write_to_file()to handle downgrading. - This system should ideally also be extended to handle plugin schemas (such as those used internally at Pixar). The upgrade system is set up to work for them, so a way to downgrade those as well should be created.
- Does OTIO allow schema changes that break compatability? (vs. never changing it, protobuf style accretion, or backwards compatability built into code/library directly)
- Do we keep knowledge about version translation inside of OTIO or do we extern that to a plugin/service/shell script external to the main library that can be more frequently updated? (really: do we restrict schema version transformation to be OTIO-blind)
- Does OTIO define schema in code or via a contraption (USD, XML/XSLT, JSON Schema, etc)?
My answers to the questions:
YES. we allow compatability breaking changes because the data is a user interface and usually serialized in a human readable (UTF-8) form. Protobuf style means we need extra glue that knows how to promote values that differ between versions and also means that the raw data is more confusing (do I look in the Clip.1 information or the Clip.2?). Furthermore, we know that we want to change timing model things in the near future, and this would make that work significantly more complicated.
Do we embed the knowledge of schema translation IN OTIO or extern it? Do OTIO Schema changes require OTIO to correctly execute?
My suspiscion is that transformation requires OTIO to correctly evaluate, especially as we start to introduce stronger concepts of reference space transformation/change of basis and stronger notions of discrete vs continuous math. Furthermore, externing the knowledge of schema translation adds complexity to introducing new schemas and schema versions to OTIO. For example, adding schemas and upgrade functions currently resides all in one spot in typeRegistry.cpp, if the library for doing translation was external, code would have to be added there as well.
Additionally, making it external means complexity in communicating with that utility. I think we should pursue an internal solution if possible.
When we initially were building out OTIO we looked at the USD contraption for doing this (where USD schemas are defined in USD snippets and then stitched together with C++ to form the C++ implementation at build time), but we lacked the resources (and percieved need) for doing this. This still seems like a lot of work for not that much gain, given how often the core schemas are evolving. Right now it might be a pragmatic recomendation, but I don't feel like given other priorities its worth either introducing a contraption or porting the core schemas to one.
- Allow modern OTIO software to write out old OTIO schema (controlled by a configuration option), store information about what old OTIO Schema is in future OTIO version code (not data)
- Have an external system that can coerce a file between versions
- Allow old OTIO software to read modern OTIO schema (via some runtime plugin, service, or some other contraption).
- Modify our strategy for schema changes to ensure that revisions can still be read by old software (within some range of versions).
- Never change the schema.
- Protobuf style: append new schema to old schema, so if you serialize a clip.2 you get a clip.1 and a clip.2
- refactor unit tests about version promotion into its own harness
- Break existing test into two test methods
- add a test for schema name changes, ie: FirstSchema.1 → SecondSchema.2 → SecondSchema.3
- update algorithm in updater loop to check for schema name changes, and add cycle detection (can only visit each schema name once)
- add unit test with cycle in it to catch exception
Version manifest is a mapping of a name to a dictionary that maps schema names to version numbers. They associate schema versions to releases of OTIO. For example:
{
"latest": {
},
"0.15.0": {
"Clip": 1,
"Timeline": 2,
"etc...": null,
},
"0.14.0": {
"Clip": 1,
"Timeline": 1,
"etc...": null,
},
}{
"pixar": {
"2022.7" : {
"InternalMetadata": 4,
"OtherSchema": 3,
},
"2022.6" : {
"InternalMetadata": 3,
"OtherSchema": 3,
}
}
}otio.adapters.write_to_file(
my_otio,
"somefile.otio",
version_manifest: {"pixar": "2022.6", "OTIO_CORE": "0.14.0"}
)For the core OTIO repository, we add a build step that generates that manifest by looking at the type registry and adds a "latest" field, which is the default target, to the dictionary.
We can back populate this by running the python script to generate these versions on previous releases.
The downgrade system takes either a name of one of these keys, or a manually constructed one.
For third party plugins, the script could be run by invoking python and generate a third party manifest maybe? I'm not sure about this.
ISSUE: how to do this for third party plugins?
- introduce
register_downgrade_functionto pair withregister_upgrade_function, similar signature - add algorithm for walking across the downgrade functions, given a
dictionary mapping of target versions, ie:
{ "Clip": 1, "MediaReference":None}→ None = not represented, version = downgrade to that version - add a discovery system for finding these version manifests by configuration names (Manifest system?)
- add metadata for the targeted otio version, if the downgrading system was invoked
- Figure out how to handle third party plugin versioning through this system (Or don't?)
- Build some tooling to display the version differences between two manifests
-
otiostatvalidation for current schema and target schema
The upgrader/downgrader code could migrate into a sidecar package, that knows how to do the upgrading and downgrading. I think there are problems with this:
- interacting with plugins
- you might need to know OTIO things in order to upgrade/downgrade datatypes
- when adding new schema, you would also need to go to the sidecar package and describe upgrade/downgrade mechanics, rather than do it all in one place
All of the below is in typeRegistry.cpp.
Renames:
56: register_type_from_existing_type("Filler", 1, "Gap", nullptr);
69: register_type_from_existing_type("SerializeableCollection", 1, "SerializableCollection", nullptr);
75: register_type_from_existing_type("Sequence", 1, "Track", nullptr);Upgrades:
register_upgrade_function(Marker::Schema::name, 2,
[](AnyDictionary* d) {
(*d)["marked_range"] = (*d)["range"];
d->erase("range");
});
register_upgrade_function(Clip::Schema::name, 2, [](AnyDictionary* d) {
auto media_ref = (*d)["media_reference"];
// The default ctor of Clip used to set media_reference to
// MissingReference. To preserve the same behaviour, if we don't have a
// valid MediaReference, do it here too.
if (media_ref.type() != typeid(SerializableObject::Retainer<>))
{
media_ref = SerializableObject::Retainer<>(new MissingReference);
}
(*d)["media_references"] =
AnyDictionary{ { Clip::default_media_key, media_ref } };
(*d)["active_media_reference_key"] =
std::string(Clip::default_media_key);
d->erase("media_reference");
});