kafka_note

kafkacat.md

• list messages trong 1 partition

kafkacat -b 10.5.69.6:9092 -t imap-2 -p 1

• Metadata listing:

root@trangnth-kafka:/opt/confluent# kafkacat -L -b 10.5.6.6:9092 -t imap-2 
Metadata for imap-2 (from broker 1: 10.5.6.6:9092/1):
 3 brokers:
  broker 2 at 10.5.6.1:9092
  broker 3 at 10.5.6.19:9092
  broker 1 at 10.5.6.6:9092
 1 topics:
  topic "imap-2" with 3 partitions:
    partition 0, leader 1, replicas: 1,2,3, isrs: 1,2,3
    partition 1, leader 2, replicas: 2,3,1, isrs: 2,3,1
    partition 2, leader 3, replicas: 3,1,2, isrs: 3,1,2

• mode consumer, count 2 messages

kafkacat -C -b 10.5.69.6:9092 -t imap-2 -p 1 -c 2

• Print message keys prefixing the message

kafkacat -C -b 10.5.69.6:9092 -t imap-2 -p 1 -c 2 -K ","
kafkacat -b 10.5.69.6:9092 -C -t imap-2 -o beginning -c 2 -K :

• Format message output

root@trangnth-kafka:/opt/confluent# kafkacat -C -b 10.5.69.6:9092 -t imap-2 -p 1 -c 2 -K "," -f 'Topic: %t [%p] - offset %o - key: %k -> %s\n'
Topic: imap-2 [1] - offset 0 - key: trangnth2@com -> {"event_timestamp":"1622518580","user":"trangnth2@com","mailbox":"INBOX","event":"FlagsSet","uidvalidity":1617243887,"uids":[6603],"flags":["\\Seen"]}
Topic: imap-2 [1] - offset 1 - key: trangnth2@com -> {"event_timestamp":"1622518580","user":"trangnth2@com","mailbox":"INBOX","event":"MessageRead","uidvalidity":1617243887,"uids":[6603]}

kt.md

• install

go get -u github.com/fgeller/kt

• help

root@trangnth-flink:~# kt -h
kt is a tool for Kafka.

Usage:

	kt command [arguments]

The commands are:

	consume    consume messages.
	produce    produce messages.
	topic      topic information.
	group      consumer group information and modification

Use "kt [command] -help" for for information about the command.

More at https://github.com/fgeller/kt

• list topic, parttion

kt topic -brokers 10.5.6.6:9092
kt topic -brokers 10.5.6.6:9092 -filter imap-2 -partitions

• list all message in partition 1

kt consume -brokers 10.5.6.6:9092 -topic imap-2 -offsets 1

• list message offset 0->2 in partition 1

root@trangnth-flink:~# kt consume -brokers 10.5.6.6:9092 -topic imap-2 -offsets 1=0:2
{
  "partition": 1,
  "offset": 0,
  "key": "trangnth2@com",
  "value": "{\"event_timestamp\":\"1622518580\",\"user\":\"trangnth2@com\",\"mailbox\":\"INBOX\",\"event\":\"FlagsSet\",\"uidvalidity\":1617243887,\"uids\":[6603],\"flags\":[\"\\\\Seen\"]}",
  "timestamp": "2021-06-01T10:36:20.986+07:00"
}
{
  "partition": 1,
  "offset": 1,
  "key": "trangnth2@com",
  "value": "{\"event_timestamp\":\"1622518580\",\"user\":\"trangnth2@com\",\"mailbox\":\"INBOX\",\"event\":\"MessageRead\",\"uidvalidity\":1617243887,\"uids\":[6603]}",
  "timestamp": "2021-06-01T10:36:20.986+07:00"
}
{
  "partition": 1,
  "offset": 2,
  "key": "trangnth2@com",
  "value": "{\"event_timestamp\":\"1622518581\",\"user\":\"trangnth2@com\",\"mailbox\":\"INBOX\",\"event\":\"FlagsSet\",\"uidvalidity\":1617243887,\"uids\":[6601],\"flags\":[\"\\\\Seen\"]}",
  "timestamp": "2021-06-01T10:36:21.207+07:00"
}

• Follow a topic, starting relative to newest offset

kt consume -brokers 10.5.6.6:9092 -topic imap-2 -offsets all=newest-1: