Skip to content

Instantly share code, notes, and snippets.

Last active Jan 15, 2019
What would you like to do?

There are several problems with the current patches to implement S3-style versioning

Listings require that we read the entire archive container

  • S3 listings must be ORDERED BY name ASC, timestamp DESC
  • versioned_write's current naming convention ('%03x%s/%s' % (len(name), name, timestamp) imposes an index more like ORDERED BY LENGTH(name) ASC, name ASC, timestamp ASC


  • Use multi-character delimiters to at least get us to ORDERED BY name ASC, timestamp ASC by doing something more like name + delimiter + timestamp where delimiter is some randomly-generated GUID
  • Maybe also start using 10000000000 - timestamp so we get timestamp DESC?
  • Downsides:
    • Gives us still more ways that versioned_writes can behave radically differently depending upon (mutable, runtime!) config. X-Versions-Location vs X-History-Location starts to feel minor by comparison.
    • 10000000000 - timestamp further complicates our Y2286 problem 😛
    • No good way to migrate from one format to the other
  • Questions:
    • How should users specify delimiter? X-History-Location: archive-container/<delimiter>? X-History-Location: archive-container/?delimiter=<delimiter>? X-Versions-Delimiter: <delimiter>? Some other thing?
    • Can we document it sensibly? Looking at as it stands, this new data layout seems like it would be difficult to explain.
    • How much should we worry about users changing delimiters, or trying to turn on delimiter versioning while there's still data from non-delimiter versioning?

When versioning is suspended, all previous (non-null) versions need to be frozen


  • Implement versioning with symlinks. When versioning is enabled, all new data gets written directly to the archive container, and a system symlink (probably with a special content-type, a la the delete marker content type) gets written in the primary container

When an archived version gets restored then archived again, the timestamp baked into its name changes


  • Similar to an existing patch, store some X-Object-Sysmeta-Versioned-Writes-Timestamp header – but do it on initial PUT rather than archive/restore
  • Could act as a flag to identify null version

Need to be able to identify null version for version listings


  • If we have delimiter support, shove another flag in the name, probably at the end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment