Kafkacat notes

kafkcat_hints.md

Note: Assuming you add --bootstrap-server to all of them

Select few keys from stream and convert it to CSV using jq

> kcat -C -t <TOPIC> -e -o -beginning  |   jq -r '. | {id: .id, name: .name, tombstone: .tombstone} | [.[]] | @csv' > prod_companies.csv

Reset the consumer to beginning of topic (Make sure consumer is not stable at that point)

> kafka-consumer-groups.sh --reset-offsets --to-earliest --group <CONSUMER_GROUP> --topic <TOPIC> --to-earliest --execute

Get the size of the topic in human readable format

> kafka-log-dirs.sh --describe --topic-list <TOPIC> | grep '^{' | jq '[ ..|.size? | numbers ] | add' | numfmt --to iec --format '%8.4f'

Filter kafka messages (from the beginning of offset) based on macthing criteria

>  kcat -C -t <TOPIC> -C -e -o -beginning  | jq -r '. | select(.<KEY_TO_MATCH> == "<VALUE_TO_MATCH>")'