Internal update protocol
- When the leader gets a new configuration, it assigns a monotonically increasing version number to it and stores it in a k8s annotation (similar to how the leader election algorithm uses annotations). The version # will be for router-internal use only.
- When the leader gets a new configuration, it increments the version, stores it in the annotation, then sends the version along with the new configuration to all other router replicas. Each replica only responds when it has finished updating both its config and version number. If any replica doesn't respond within a reasonable timeout, the leader terminates it.
- When a non-leader comes online with a version #, it compares its local known value with that in the annotation. If they don't match, it requests the most up to date version number and config from the leader
- When a non-leader comes online with no version, it requests the most up to date version and config from the leader.
- If a non-leader gets one or more configurat