Skip to content

Instantly share code, notes, and snippets.

@brettrann
Last active October 12, 2017 04:31
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save brettrann/ce52343692696a45d5b9f4df723bcd14 to your computer and use it in GitHub Desktop.
Save brettrann/ce52343692696a45d5b9f4df723bcd14 to your computer and use it in GitHub Desktop.
log clean thread dying after compacting 100GB+ consumer offset
[2017-10-12 01:47:01,013] INFO Cleaner 0: Beginning cleaning of log __consumer_offsets-36. (kafka.log.LogCleaner)
Last message '[2017-10-12 01:47:01' repeated 1 times, suppressed by syslog-ng on kafka9
[2017-10-12 01:47:01,013] INFO Cleaner 0: Building offset map for __consumer_offsets-36... (kafka.log.LogCleaner)
Last message '[2017-10-12 01:47:01' repeated 1 times, suppressed by syslog-ng on kafka9
[2017-10-12 01:47:01,029] INFO Cleaner 0: Building offset map for log __consumer_offsets-36 for 11 segments in offset range [0, 2694523870). (kafka.log.LogCleaner)
Last message '[2017-10-12 01:47:01' repeated 1 times, suppressed by syslog-ng on kafka9
[2017-10-12 01:47:08,396] INFO Cleaner 0: Offset map for log __consumer_offsets-36 complete. (kafka.log.LogCleaner)
Last message '[2017-10-12 01:47:08' repeated 1 times, suppressed by syslog-ng on kafka9
[2017-10-12 01:47:08,397] INFO Cleaner 0: Cleaning log __consumer_offsets-36 (cleaning prior to Thu Oct 12 00:53:07 UTC 2017, discarding tombstones prior to Thu Jan 01 00:00:00 UTC 1970)... (kafka.log.LogCleaner)
Last message '[2017-10-12 01:47:08' repeated 1 times, suppressed by syslog-ng on kafka9
[2017-10-12 01:47:08,397] INFO Cleaner 0: Cleaning segment 0 in log __consumer_offsets-36 (largest timestamp Wed Oct 11 05:31:59 UTC 2017) into 0, retaining deletes. (kafka.log.LogCleaner)
Last message '[2017-10-12 01:47:08' repeated 1 times, suppressed by syslog-ng on kafka9
[2017-10-12 01:47:08,402] ERROR [kafka-log-cleaner-thread-0]: Error due to (kafka.log.LogCleaner)
java.lang.IllegalArgumentException: requirement failed: largest offset in message set can not be safely converted to relative offset.
at scala.Predef$.require(Predef.scala:224)
at kafka.log.LogSegment.append(LogSegment.scala:121)
at kafka.log.Cleaner.cleanInto(LogCleaner.scala:547)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:443)
at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:385)
at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:384)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.log.Cleaner.doClean(LogCleaner.scala:384)
at kafka.log.Cleaner.clean(LogCleaner.scala:361)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:256)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:236)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
[2017-10-12 01:47:08,402] ERROR [kafka-log-cleaner-thread-0]: Error due to (kafka.log.LogCleaner)
java.lang.IllegalArgumentException: requirement failed: largest offset in message set can not be safely converted to relative offset.
at scala.Predef$.require(Predef.scala:224)
at kafka.log.LogSegment.append(LogSegment.scala:121)
at kafka.log.Cleaner.cleanInto(LogCleaner.scala:547)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:443)
at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:385)
at kafka.log.Cleaner$$anonfun$doClean$4.apply(LogCleaner.scala:384)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.log.Cleaner.doClean(LogCleaner.scala:384)
at kafka.log.Cleaner.clean(LogCleaner.scala:361)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:256)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:236)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
[2017-10-12 01:47:08,406] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment