Skip to content

Instantly share code, notes, and snippets.

@jasonk000
Created December 3, 2021 19:31
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jasonk000/40c11dce3faed44d3a89c27e0227b982 to your computer and use it in GitHub Desktop.
Save jasonk000/40c11dce3faed44d3a89c27e0227b982 to your computer and use it in GitHub Desktop.
druid task shutdown process
2021-12-03T18:34:37,787 INFO [KafkaSupervisor-xxxx] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [%s] because: [No task in the corresponding pending completion taskGroup[%d] succeeded before completion timeout elapsed]; index_kafka_xxxx_c51e10928edd560_moiikjbd; [Ljava.lang.Object;@47355796
2021-12-03T18:34:37,790 INFO [KafkaSupervisor-xxxx] org.apache.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to worker: x.x.x.x:7103, status 200 OK, response: {"task":"index_kafka_xxxx_c51e10928edd560_moiikjbd"}
2021-12-03T18:34:37,790 INFO [KafkaSupervisor-xxxx] org.apache.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_xxxx_c51e10928edd560_moiikjbd] from activeTasks
2021-12-03T18:34:37,791 INFO [KafkaSupervisor-xxxx] org.apache.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_xxxx_c51e10928edd560_moiikjbd] from TaskLock[TimeChunkLock{type=EXCLUSIVE, groupId='index_kafka_xxxx', dataSource='xxxx', interval=2021-12-03T18:00:00.000Z/2021-12-03T18:15:00.000Z, version='2021-12-03T18:00:00.348Z', priority=75, revoked=false}]
2021-12-03T18:34:37,791 INFO [KafkaSupervisor-xxxx] org.apache.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_xxxx_c51e10928edd560_moiikjbd] from TaskLock[TimeChunkLock{type=EXCLUSIVE, groupId='index_kafka_xxxx', dataSource='xxxx', interval=2021-12-03T18:30:00.000Z/2021-12-03T18:45:00.000Z, version='2021-12-03T18:30:00.300Z', priority=75, revoked=false}]
2021-12-03T18:34:37,791 INFO [KafkaSupervisor-xxxx] org.apache.druid.indexing.overlord.HeapMemoryTaskStorage - Updating task index_kafka_xxxx_c51e10928edd560_moiikjbd to status: TaskStatus{id=index_kafka_xxxx_c51e10928edd560_moiikjbd, status=FAILED, duration=-1, errorMsg=null}
2021-12-03T18:34:37,792 INFO [KafkaSupervisor-xxxx] org.apache.druid.indexing.overlord.TaskQueue - Task done: AbstractTask{id='index_kafka_xxxx_c51e10928edd560_moiikjbd', groupId='index_kafka_xxxx', taskResource=TaskResource{availabilityGroup='index_kafka_xxxx_c51e10928edd560', requiredCapacity=1}, dataSource='xxxx', context={checkpoints={"0":{"51":100423947778,"132":100423522850}}, druid.indexer.fork.property.druid.processing.numThreads=3, druid.indexer.runner.javaOpts=-Xms1G -Xmx1G -XX:+PreserveFramePointer -XX:MaxDirectMemorySize=3G -XX:ActiveProcessorCount=4 , IS_INCREMENTAL_HANDOFF_SUPPORTED=true, forceTimeChunkLock=true, druid.indexer.fork.property.druid.server.http.numThreads=8}}
2021-12-03T18:34:37,995 INFO [ForkJoinPool.commonPool-worker-38] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_kafka_xxxx_c51e10928edd560_moiikjbd] because: [task is not in knownTaskIds[[Ljava.lang.Object;@7d9c3915]]
2021-12-03T18:34:38,000 INFO [ForkJoinPool.commonPool-worker-38] org.apache.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to worker: x.x.x.x:7103, status 200 OK, response: {"task":"index_kafka_xxxx_c51e10928edd560_moiikjbd"}
2021-12-03T18:34:38,567 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[x.x.x.x:7103] wrote FAILED status for task [index_kafka_xxxx_c51e10928edd560_moiikjbd] on [TaskLocation{host='x.x.x.x', port=8137, tlsPort=-1}]
2021-12-03T18:34:38,567 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[x.x.x.x:7103] completed task[index_kafka_xxxx_c51e10928edd560_moiikjbd] with status[FAILED]
2021-12-03T18:34:38,567 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_kafka_xxxx_c51e10928edd560_moiikjbd
2021-12-03T18:34:38,567 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_kafka_xxxx_c51e10928edd560_moiikjbd] because: [notified status change from task]
2021-12-03T18:34:38,567 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_kafka_xxxx_c51e10928edd560_moiikjbd] on worker[x.x.x.x:7103]
2021-12-03T18:34:38,568 WARN [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Unknown task completed: index_kafka_xxxx_c51e10928edd560_moiikjbd
2021-12-03T18:34:38,568 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Task FAILED: AbstractTask{id='index_kafka_xxxx_c51e10928edd560_moiikjbd', groupId='index_kafka_xxxx', taskResource=TaskResource{availabilityGroup='index_kafka_xxxx_c51e10928edd560', requiredCapacity=1}, dataSource='xxxx', context={checkpoints={"0":{"51":100423947778,"132":100423522850}}, druid.indexer.fork.property.druid.processing.numThreads=3, druid.indexer.runner.javaOpts=-Xms1G -Xmx1G -XX:+PreserveFramePointer -XX:MaxDirectMemorySize=3G -XX:ActiveProcessorCount=4 , IS_INCREMENTAL_HANDOFF_SUPPORTED=true, forceTimeChunkLock=true, druid.indexer.fork.property.druid.server.http.numThreads=8}} (72969 run duration)
2021-12-03T18:34:38,568 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Task[index_kafka_xxxx_c51e10928edd560_moiikjbd] went bye bye.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment