Skip to content

Instantly share code, notes, and snippets.

@srnagar
Last active August 30, 2019 21:12
Show Gist options
  • Save srnagar/714b0064668b4d1d3520fcc52e5879c6 to your computer and use it in GitHub Desktop.
Save srnagar/714b0064668b4d1d3520fcc52e5879c6 to your computer and use it in GitHub Desktop.
Proposal to workaround the need for offset provider

Alternate solution to offset provider

In track 1 library, Event Processor Host (EPH) used an offset provider to bootstrap an application for the first time. When an EPH starts up and claims ownership of a partition, it needs to determine the position (sequence number/offset) to start processing events from. If a checkpoint is available, the EPH will start processing the events from the checkpointed sequence number. If it's not available (i.e the application has never used EPH before or the storage is new/migrated and all checkpoints are lost), then EPH will fallback to using user provided implementation of an offset provider that has the ability to return a sequence number to start processing from for a given partition id.

Given that the offset provider is used occasionally and increases the complexity of EPH interface and implementation, in Track 2 library, the offset provider can be deprecated and Event Processor (EPH equivalent in Track 2) can be simplified. If/when applications want to provide a custom checkpoint for new Event Processor, they can follow the steps below to achieve the purpose offset provider served in Track 1.

New Event Processor application or new storage used by PartitionManager.

  1. Add a new interface method to the PartitionManager - overridePartitionCheckpoint
  2. This interface method will take the following parameters:
    • Event Hub namespace
    • Event Hub name
    • Consumer group name
    • Map of partition id and EventPosition

Implementation of this interface method will create/update the storage with PartitionOwnership entries for all the partitions in the map with the following details:

  • Event Hub name

  • Consumer group name

  • Partition Id

  • Owner id - empty string or null (load balancer will have to treat this as unowned partition)

  • Sequence number - The sequence number corresponding to the EventPosition in the map

  • Offset - The offset corresponding to the EventPosition in the map

  • ETag - * This method will force update the ownership record. ETag is not required to match.

  • Last modified time - current timestamp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment