Skip to content

Instantly share code, notes, and snippets.

@mtth
Last active June 12, 2016 02:44
Show Gist options
  • Save mtth/527318445e5b52bfd491c0483ff5f9d3 to your computer and use it in GitHub Desktop.
Save mtth/527318445e5b52bfd491c0483ff5f9d3 to your computer and use it in GitHub Desktop.
Avro evolution aliases enhancement proposition

Let's assume that we have a Kafka producer emitting the following values:

union {
  record Vehicle {
    long id;
  },
  record Car {
    long id;
    boolean selfDriving;
  }
}

At a later point in time, a new vehicle becomes supported by the system and must be added to the schema:

union {
  record Vehicle {
    long id;
  },
  record Car {
    long id;
    boolean selfDriving;
  },
  @aliases(["Vehicle"]) // Ignored when on the producer's schema.
  record Bus {
    long id;
    int capacity;
  }
}

We would like to be able to deploy the change to the producer without having to migrate all the consumers: existing consumers would treat each Bus as a Vehicle until they upgrade.

However we can't do so under the current evolution rules since the alias is ignored (it would work if we added the alias to each consumer's schema but this isn't practical since it would also require a global migration). Note also that we can't preemptively add aliases on the consumers since the names of the records aren't known beforehand.

Allowing the consumers to use the producer's aliases would fix this. If we make sure that writer aliases are used last (for example only falling back to them if neither the names nor the consumers' aliases match), this doesn't change any of the current allowed evolution rules and expands them to support additional cases (without introducing any new syntax).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment