Skip to content

Instantly share code, notes, and snippets.

@chuhlomin
Last active December 11, 2017 20:06
Show Gist options
  • Save chuhlomin/cff16cc09c9feedb5078ba1d8d65ba03 to your computer and use it in GitHub Desktop.
Save chuhlomin/cff16cc09c9feedb5078ba1d8d65ba03 to your computer and use it in GitHub Desktop.
Kafka Message Binary Format
XXXX XXXX = 8 bits
1. XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX - baseOffset
XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX
2. XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX - batchLength
3. XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX - partitionLeaderEpoch
4. XXXX XXXX - magic (current magic value is 2)
5. XXXX XXXX XXXX XXXX - attributes
Compression
000 - no comporession
001 - gzip
010 - snappy
011 - lz4
Timestamp
0 - create time
1 - log append time
X - isTransactional
X - isControlBatch
XX XXXX XXXX - unused
6. XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX - lastOffsetDelta
7. XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX - firstTimestamp
XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX
8. XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX - maxTimestamp
XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX
9. XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX - producerId
XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX
10. XXXX XXXX XXXX XXXX XXXX XXXX XXXX XXXX - producerEpoch
11. ... - baseSequence
12. ... - records
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment