- When the leader gets a new configuration, it assigns a monotonically increasing version number to it and stores it in a k8s annotation (similar to how the leader election algorithm uses annotations). The version # will be for router-internal use only.
- When the leader gets a new configuration, it increments the version, stores it in the annotation, then sends the version along with the new configuration to all other router replicas. Each replica only responds when it has finished updating both its config and version number. If any replica doesn't respond within a reasonable timeout, the leader terminates it.
- When a non-leader comes online with a version #, it compares its local known value with that in the annotation. If they don't match, it requests the most up to date version number and config from the leader
- When a non-leader comes online with no version, it requests the most up to date version and config from the leader.
- If a non-leader gets one or more configurations pushed to it while it is receiving the version # and config from the leader, it applies the most up to date version number of all configs that it gets
The current implementation of the router polls the Kubernetes API for changes in the routable services. Let's say the controller adds a new routable service. After it does so, it queries the leader (see below for how it does this) to ensure that all replicas have received this new routable service. The controller doesn't finish the deploy until the leader says that all replicas have the new routable service.
The k8s leader election protocol provides for all replicas to know who the leader is. If a request goes to a non-leader, it should just forward it to a leader. This behavior allows the controller (or any other entity) to just query the router service, and wherever the request lands, it will make it to the leader.
An alternative implementation for this is that the leader can write its updated status to an annotation on the router's service.