Skip to content

Instantly share code, notes, and snippets.

@rmoff

rmoff/connect-offsets.md

Last active May 1, 2020 18:43
Show Gist options
  • Star 4 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rmoff/063e878f4ff0c0f8c09e9d8d15262c10 to your computer and use it in GitHub Desktop.
Save rmoff/063e878f4ff0c0f8c09e9d8d15262c10 to your computer and use it in GitHub Desktop.
Download ZIP
How to view Kafka Connect offsets
Raw
connect-offsets.md 
$ ./bin/kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --from-beginning \
  --property print.key=true \
  --topic connect-offsets
["jdbc_source_mysql_foobar_01",{"table":"foobar"}]      {"timestamp_nanos":0,"timestamp":1501796305000}
["jdbc_source_mysql_smt_foobar_02",{"table":"wibble"}]	{"timestamp_nanos":0,"timestamp":1501800342000}
["jdbc_source_mysql_smt_foobar_02",{"table":"wibble"}]	{"timestamp_nanos":0,"timestamp":1501800707000}

Not all connectors will use this. For example, the HDFS connector uses HDFS itself to track offsets as part of the exactly-once delivery mechanics.

The key is the connector name and additional elements (e.g. for the JDBC connector the table). The value is the offset being tracked.

This is only for Kafka Connect in Distributed mode. When run standalone a local file is used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment