Note1: types are java compatible
Note2: all strings are in UTF format
- short length
- byte [length]
- int magic: 0x4960672d (0x60 0x49 0x2d 0x67)
- int version: 1|2|3
- long checkpointId
- int numMasterStates
numMasterStates times- int magic: 0xc96b1696 (0x6b 0xc9 0x96 0x16)
- int bytesToRead
- int version
- String name
- byte[bytesToRead - sizeof(version) - sizeof(name)] data
- int numTaskStates
numTaskStates times- long jobVertexIdLow
- long jobVertexIdHigh
- int parallelism
- int maxParallelism
- StreamStateHandle coordinatorState
- int numSubTaskStates
numSubTaskStates times- int storedSubtaskIndex: if negative then finished
- int hasManagedOperatorState
hasManagedOperatorState != 0
- OperatorStateHandle managedOperatorState
- int hasRawOperatorState
hasRawOperatorState != 0
- OperatorStateHandle rawOperatorState
- KeyedStateHandle managedKeyedState
- KeyedStateHandle rawKeyedState
- int type
type == 0: NULL_HANDLE
- no data
type == 1: BYTE_STREAM_STATE_HANDLE
- String handleName
- long size
- byte[size] data
type == 2: FILE_STREAM_STATE_HANDLE
- long size
- String path: UTF format
type == 3: KEY_GROUPS_HANDLE
- int startKeyGroup
- int numKeyGroups
- long[numKeyGroups] offsets
- StreamStateHandle
type == 6: RELATIVE_STREAM_STATE_HANDLE
- String relativePath
- long size
- int type
type == 0: NULL_HANDLE
- no data
type == 4: PARTITIONABLE_OPERATOR_STATE_HANDLE
- int count
count times
- String key
- int mode: SPLIT_DISTRIBUTE|UNION|BROADCAST
- int numOffsets
- long[numOffsets] offsets
- StreamStateHandle
- int type
type == 0: NULL_HANDLE
- no data
Too volatile to document it.
If coordinatorState is not null then Kafka stores its state in the following format:
- int coordinatorSerdeVersion
- int enumSerializerVersion
- int serializedEnumCheckpointSize
- KafkaSourceEnumState state
- int numPartitions
numPartitions times- String topic
- int partition
- String topic
- int partition
- long offset
- long stoppingOffset