Skip to content

Instantly share code, notes, and snippets.

@seruman

seruman/gist:6f9a6718e29760af41683a4b45bd5ef2

Created March 3, 2021 12:56
Show Gist options
  • Save seruman/6f9a6718e29760af41683a4b45bd5ef2 to your computer and use it in GitHub Desktop.
Save seruman/6f9a6718e29760af41683a4b45bd5ef2 to your computer and use it in GitHub Desktop.
Download ZIP
kafka-lag-exporter topic re-create logs
Raw
gistfile1.txt
2021-03-03 12:49:19,839 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-03-03 12:49:19,994 INFO c.l.k.KafkaClusterManager$ akka://kafka-lag-exporter/user - Starting Kafka Lag Exporter with configuration:
Poll interval: 30 seconds
Lookup table size: 60
Metrics whitelist: [.*]
Admin client consumer group id: kafkalagexporter
Kafka client timeout: 30 seconds
Prometheus:
port: 8000
Statically defined Clusters:
Cluster name: local
Cluster Kafka bootstrap brokers: kafka:9092
Consumer group whitelist: [.*]
Topic whitelist: [.*]
Topic blacklist: []
Watchers:
Strimzi: false
2021-03-03 12:49:20,142 INFO c.l.k.KafkaClusterManager$ akka://kafka-lag-exporter/user - Cluster Added:
Cluster name: local
Cluster Kafka bootstrap brokers: kafka:9092
Consumer group whitelist: [.*]
Topic whitelist: [.*]
Topic blacklist: []
2021-03-03 12:49:20,156 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Spawned ConsumerGroupCollector for cluster: local
2021-03-03 12:49:20,218 INFO o.a.k.c.consumer.ConsumerConfig akka://kafka-lag-exporter/user/consumer-group-collector-local - ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [kafka:9092]
check.crcs = true
client.dns.lookup = default
client.id =
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = kafkalagexporter
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 1000
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.2
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
2021-03-03 12:49:20,511 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-local - Kafka version: 2.5.0
2021-03-03 12:49:20,512 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-local - Kafka commitId: 66563e712b0b9f84
2021-03-03 12:49:20,513 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-local - Kafka startTimeMs: 1614775760509
2021-03-03 12:49:20,527 INFO o.a.k.c.admin.AdminClientConfig akka://kafka-lag-exporter/user/consumer-group-collector-local - AdminClientConfig values:
bootstrap.servers = [kafka:9092]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 0
retry.backoff.ms = 1000
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.2
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
2021-03-03 12:49:20,581 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-local - Kafka version: 2.5.0
2021-03-03 12:49:20,582 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-local - Kafka commitId: 66563e712b0b9f84
2021-03-03 12:49:20,583 INFO o.a.kafka.common.utils.AppInfoParser akka://kafka-lag-exporter/user/consumer-group-collector-local - Kafka startTimeMs: 1614775760581
2021-03-03 12:49:20,599 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Collecting offsets
2021-03-03 12:49:20,626 WARN o.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.29.0.4:9092) could not be established. Broker may not be available.
2021-03-03 12:49:22,140 DEBUG c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Received Offsets Snapshot:
Timestamp: 1614775762117
Groups:
Earliest Offsets:
Topic Partition Earliest
Latest Offsets:
Topic Partition Offset
Last Group Offsets:
Group Topic Partition Offset
2021-03-03 12:49:22,147 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Updating lookup tables
2021-03-03 12:49:22,154 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Reporting offsets
2021-03-03 12:49:22,170 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Clearing evicted metrics
2021-03-03 12:49:22,179 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Polling in 30 seconds
2021-03-03 12:49:22,182 DEBUG c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Received Meta data:
MetaData(1540)
2021-03-03 12:49:52,164 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Collecting offsets
2021-03-03 12:49:52,267 INFO org.apache.kafka.clients.Metadata - [Consumer clientId=consumer-kafkalagexporter-1, groupId=kafkalagexporter] Cluster ID: yK1Qb6F6Sj-fjIGuy_Yd8w
2021-03-03 12:49:52,298 DEBUG c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Received Offsets Snapshot:
Timestamp: 1614775792230
Groups: my-group
Earliest Offsets:
Topic Partition Earliest
my-topic 0 0
Latest Offsets:
Topic Partition Offset
my-topic 0 100
Last Group Offsets:
Group Topic Partition Offset
my-group my-topic 0 100
2021-03-03 12:49:52,303 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Updating lookup tables
2021-03-03 12:49:52,334 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Reporting offsets
2021-03-03 12:49:52,361 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Clearing evicted metrics
2021-03-03 12:49:52,362 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Polling in 30 seconds
2021-03-03 12:49:52,363 DEBUG c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Received Meta data:
MetaData(129)
2021-03-03 12:50:22,353 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Collecting offsets
2021-03-03 12:50:22,392 DEBUG c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Received Offsets Snapshot:
Timestamp: 1614775822380
Groups: my-group
Earliest Offsets:
Topic Partition Earliest
my-topic 0 0
Latest Offsets:
Topic Partition Offset
my-topic 0 100
Last Group Offsets:
Group Topic Partition Offset
my-group my-topic 0 100
2021-03-03 12:50:22,395 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Updating lookup tables
2021-03-03 12:50:22,396 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Reporting offsets
2021-03-03 12:50:22,399 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Clearing evicted metrics
2021-03-03 12:50:22,400 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Polling in 30 seconds
2021-03-03 12:50:22,401 DEBUG c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Received Meta data:
MetaData(24)
2021-03-03 12:50:52,379 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Collecting offsets
2021-03-03 12:50:52,400 DEBUG c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Received Offsets Snapshot:
Timestamp: 1614775852391
Groups: my-group
Earliest Offsets:
Topic Partition Earliest
my-topic 0 0
Latest Offsets:
Topic Partition Offset
my-topic 0 20
Last Group Offsets:
Group Topic Partition Offset
my-group my-topic 0 20
2021-03-03 12:50:52,403 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Updating lookup tables
2021-03-03 12:50:52,404 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Reporting offsets
2021-03-03 12:50:52,414 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Clearing evicted metrics
2021-03-03 12:50:52,425 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Polling in 30 seconds
2021-03-03 12:50:52,426 DEBUG c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Received Meta data:
MetaData(20)
2021-03-03 12:51:11,620 INFO c.l.k.KafkaClusterManager$ akka://kafka-lag-exporter/user - Attempting graceful shutdown
2021-03-03 12:51:11,642 INFO c.l.k.MetricsReporter$ akka://kafka-lag-exporter/user/prometheus-lag-reporter - Gracefully stopped Prometheus metrics endpoint HTTP server
2021-03-03 12:51:11,654 INFO c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-local - Gracefully stopped polling and Kafka client for cluster: local
2021-03-03 12:51:11,698 WARN akka.actor.CoordinatedShutdown CoordinatedShutdown(akka://kafka-lag-exporter) - Could not addJvmShutdownHook, due to: Shutdown in progress
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment