Skip to content

Instantly share code, notes, and snippets.

@cresny

cresny/savepoint-fail.log

Created June 20, 2017 21:48
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save cresny/87c997b558064a7ec5e8021ae7456653 to your computer and use it in GitHub Desktop.
Save cresny/87c997b558064a7ec5e8021ae7456653 to your computer and use it in GitHub Desktop.
Download ZIP
restart after two external checkpoints, then fail on first subsequent checkpoint
Raw
savepoint-fail.log
2017-06-20 21:39:04.768 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Persisting periodic checkpoints externally at s3://mm-flink-checkpoints//cliff.
2017-06-20 21:39:07.727 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:07.732 [ip-10-150-96-53] INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Attempting to load RocksDB native library and store it under '/media/flink/tmp0'
2017-06-20 21:39:07.733 [ip-10-150-96-53] DEBUG org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Attempting to create RocksDB native library folder /media/flink/tmp0/rocksdb-lib-d91dd1136bd29e4ab106d848dbc56d4a
2017-06-20 21:39:07.752 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:07.753 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:07.805 [ip-10-150-96-53] INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Successfully loaded RocksDB native library
2017-06-20 21:39:07.815 [ip-10-150-96-53] INFO o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Initializing RocksDB keyed state backend from snapshot.
2017-06-20 21:39:07.816 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Restoring snapshot from state handles: null.
2017-06-20 21:39:07.834 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:07.869 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:07.876 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:08.006 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:08.075 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:08.125 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:08.170 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:09.319 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:40:04.780 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 1 @ 1497994804776
2017-06-20 21:40:04.815 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 1 from task fa7cd512e7d6a0d285645847d4d24937 of job c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:40:12.625 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, synchronous part) in thread Thread[Async calls on Source: S3 File Watcher (1/1),5,Flink Task Threads] took 36 ms.
2017-06-20 21:40:12.643 [ip-10-150-96-53] INFO com.mediamath.reporting.combiner.Combiner - start combiner snapshot for id 0
2017-06-20 21:40:12.675 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 1 from task 36afeeb89bb5f7058c1d39a38ac3f559 of job c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:40:12.726 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, synchronous part) in thread Thread[S3 File Reader -> Parser -> Timestamps/Watermarks -> Combiner (1/1),5,Flink Task Threads] took 57 ms.
2017-06-20 21:40:13.386 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 1 from task 22fa65098f40ad2dca6c90c4f4720437 of job c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:40:13.359 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, asynchronous part) in thread Thread[pool-10-thread-1,5,Flink Task Threads] took 722 ms.
2017-06-20 21:40:16.140 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, synchronous part) in thread Thread[S3 File Reader -> Parser -> Timestamps/Watermarks -> Combiner (1/1),5,Flink Task Threads] took 3405 ms.
2017-06-20 21:40:16.467 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, asynchronous part) in thread Thread[pool-9-thread-1,5,Flink Task Threads] took 316 ms.
2017-06-20 21:40:18.731 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, asynchronous part) in thread Thread[pool-9-thread-1,5,Flink Task Threads] took 2264 ms.
2017-06-20 21:40:18.750 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 1 from task 7302a4d606f288d7670a842e7351ab8b of job c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:40:19.182 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/55a19c83-0de5-432a-8e31-4ddf76dfc7be [5592632 bytes]
2017-06-20 21:40:19.197 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - mapping 000011.sst -> File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/55a19c83-0de5-432a-8e31-4ddf76dfc7be [5592632 bytes]
2017-06-20 21:40:19.212 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/aa747ed8-11f9-4973-b41a-f026a0121224'}
2017-06-20 21:40:19.852 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/7f4bad77-0ddc-4978-9657-37455f51c140 [8289 bytes]
2017-06-20 21:40:19.853 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/39600c3a-cf24-4fe4-9857-814f6c1f567c'}
2017-06-20 21:40:19.879 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 1 from task ac6e5334725fb026597ee9e5ec134e6a of job c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:40:19.879 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 3c249f46784e7918941b8254a455626e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:40:19.880 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 2507a43fb0c9c91dd4f7e60bcfef8d4e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:40:19.880 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: bb0199e122f4970209706820e4def95b, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 28626796)
2017-06-20 21:40:19.880 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: d1c48e48788f3367ffb96f7809bcfb9d, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 12589273)
2017-06-20 21:40:19.880 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registeredStates.put(WindowOperator_8_0-KeyGroupRange{startKeyGroup=0, endKeyGroup=127}-000011.sst, org.apache.flink.runtime.state.SharedStateRegistry$SharedStateEntry@25b23aa4);
2017-06-20 21:40:19.881 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 44aee4921a91b9a2911691cc10171a24, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 2188313)
2017-06-20 21:40:19.881 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: cc6f3a804a6c2964c83f3008635bbb25, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 3862)
2017-06-20 21:40:19.881 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 70d8f5f0f7af92550fed76e829eeea46, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:40:20.848 [ip-10-150-96-228] DEBUG o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serialize 1
2017-06-20 21:40:20.861 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/96664dd2-4a93-46ce-8c5c-802773047088 [28626796 bytes]
2017-06-20 21:40:20.861 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/a89341ab-ea37-41ac-878b-7bdd00505aaa [23438 bytes]
2017-06-20 21:40:20.862 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/55a19c83-0de5-432a-8e31-4ddf76dfc7be [5592632 bytes]
2017-06-20 21:40:20.862 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/aa747ed8-11f9-4973-b41a-f026a0121224'}
2017-06-20 21:40:20.862 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/7f4bad77-0ddc-4978-9657-37455f51c140 [8289 bytes]
2017-06-20 21:40:20.862 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/39600c3a-cf24-4fe4-9857-814f6c1f567c'}
2017-06-20 21:40:20.863 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/aa7627e4-8a6b-4843-81e4-0de41e5fb07f [6964543 bytes]
2017-06-20 21:40:20.868 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/f08927a1-4545-4deb-8daf-cebe6cfbb398 [2188313 bytes]
2017-06-20 21:40:20.868 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/27bdb18f-00a6-42a2-a22f-1fcb4c399161 [3862 bytes]
2017-06-20 21:40:21.722 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 1 (43408244 bytes in 16945 ms).
2017-06-20 21:40:21.722 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint state: OperatorState(operatorID: 3c249f46784e7918941b8254a455626e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0), OperatorState(operatorID: 2507a43fb0c9c91dd4f7e60bcfef8d4e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0), OperatorState(operatorID: bb0199e122f4970209706820e4def95b, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 28626796), OperatorState(operatorID: d1c48e48788f3367ffb96f7809bcfb9d, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 12589273), OperatorState(operatorID: 44aee4921a91b9a2911691cc10171a24, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 2188313), OperatorState(operatorID: cc6f3a804a6c2964c83f3008635bbb25, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 3862), OperatorState(operatorID: 70d8f5f0f7af92550fed76e829eeea46, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:41:09.722 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 2 @ 1497994869722
2017-06-20 21:41:09.735 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 2 from task fa7cd512e7d6a0d285645847d4d24937 of job c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:41:14.339 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, synchronous part) in thread Thread[Async calls on Source: S3 File Watcher (1/1),5,Flink Task Threads] took 0 ms.
2017-06-20 21:41:14.382 [ip-10-150-96-53] INFO com.mediamath.reporting.combiner.Combiner - start combiner snapshot for id 0
2017-06-20 21:41:14.390 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 2 from task 36afeeb89bb5f7058c1d39a38ac3f559 of job c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:41:14.417 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, synchronous part) in thread Thread[S3 File Reader -> Parser -> Timestamps/Watermarks -> Combiner (1/1),5,Flink Task Threads] took 30 ms.
2017-06-20 21:41:15.010 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, asynchronous part) in thread Thread[pool-10-thread-2,5,Flink Task Threads] took 656 ms.
2017-06-20 21:41:15.017 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 2 from task 22fa65098f40ad2dca6c90c4f4720437 of job c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:41:16.273 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/304ba12f-42b7-42b6-b1dc-b7b3e69b4fb7 [5987139 bytes]
2017-06-20 21:41:16.273 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - mapping 000012.sst -> File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/304ba12f-42b7-42b6-b1dc-b7b3e69b4fb7 [5987139 bytes]
2017-06-20 21:41:16.273 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - 000011.sst exists already, mapping it -> PlaceholderStreamStateHandle
2017-06-20 21:41:16.274 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/1062b8b6-1877-4a3d-9fea-52ee64c3cd58'}
2017-06-20 21:41:16.712 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/8f625ca1-b79a-43cf-a528-14de13bca247 [8289 bytes]
2017-06-20 21:41:16.712 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/0061918f-2a7c-45fc-8cb6-47f4bbd057d1'}
2017-06-20 21:41:16.729 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 2 from task ac6e5334725fb026597ee9e5ec134e6a of job c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:41:18.326 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, synchronous part) in thread Thread[S3 File Reader -> Parser -> Timestamps/Watermarks -> Combiner (1/1),5,Flink Task Threads] took 3895 ms.
2017-06-20 21:41:18.812 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, asynchronous part) in thread Thread[pool-9-thread-1,5,Flink Task Threads] took 485 ms.
2017-06-20 21:41:21.981 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020, asynchronous part) in thread Thread[pool-9-thread-1,5,Flink Task Threads] took 3168 ms.
2017-06-20 21:41:21.999 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 2 from task 7302a4d606f288d7670a842e7351ab8b of job c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:41:22.000 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 3c249f46784e7918941b8254a455626e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:41:22.000 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 2507a43fb0c9c91dd4f7e60bcfef8d4e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:41:22.000 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: bb0199e122f4970209706820e4def95b, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 61147562)
2017-06-20 21:41:22.000 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: d1c48e48788f3367ffb96f7809bcfb9d, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 16656374)
2017-06-20 21:41:22.001 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/55a19c83-0de5-432a-8e31-4ddf76dfc7be [5592632 bytes] is apparent duplicate -> scheduledStateDeletion = state;
2017-06-20 21:41:22.001 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - org.apache.flink.runtime.state.SharedStateRegistry$SharedStateEntry@25b23aa4 reference count = 2
2017-06-20 21:41:22.001 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - org.apache.flink.runtime.state.PlaceholderStreamStateHandle@468c112a is Placeholder, not deleting
2017-06-20 21:41:22.001 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registeredStates.put(WindowOperator_8_0-KeyGroupRange{startKeyGroup=0, endKeyGroup=127}-000012.sst, org.apache.flink.runtime.state.SharedStateRegistry$SharedStateEntry@42b39d23);
2017-06-20 21:41:22.001 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 44aee4921a91b9a2911691cc10171a24, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 2935159)
2017-06-20 21:41:22.001 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: cc6f3a804a6c2964c83f3008635bbb25, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 3862)
2017-06-20 21:41:22.001 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 70d8f5f0f7af92550fed76e829eeea46, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:41:22.040 [ip-10-150-96-228] DEBUG o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serialize 2
2017-06-20 21:41:22.063 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/df7c3856-e87d-4872-a7c1-5598624999e4 [61147562 bytes]
2017-06-20 21:41:22.064 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/066eb9ec-4cb5-473f-8f3a-237b6435b6fa [23438 bytes]
2017-06-20 21:41:22.064 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/55a19c83-0de5-432a-8e31-4ddf76dfc7be [5592632 bytes]
2017-06-20 21:41:22.064 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/304ba12f-42b7-42b6-b1dc-b7b3e69b4fb7 [5987139 bytes]
2017-06-20 21:41:22.064 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/1062b8b6-1877-4a3d-9fea-52ee64c3cd58'}
2017-06-20 21:41:22.064 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/8f625ca1-b79a-43cf-a528-14de13bca247 [8289 bytes]
2017-06-20 21:41:22.065 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/0061918f-2a7c-45fc-8cb6-47f4bbd057d1'}
2017-06-20 21:41:22.065 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/33ca9b02-db01-47fa-a216-55034d2b3eab [10636944 bytes]
2017-06-20 21:41:22.072 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/73f80124-215c-44af-93f3-bfb7323bd5ba [2935159 bytes]
2017-06-20 21:41:22.072 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/bd62e544-f4c6-4f42-a030-b961cf67ac7b [3862 bytes]
2017-06-20 21:41:22.379 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 2 (80742957 bytes in 12657 ms).
2017-06-20 21:41:22.379 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint state: OperatorState(operatorID: 3c249f46784e7918941b8254a455626e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0), OperatorState(operatorID: 2507a43fb0c9c91dd4f7e60bcfef8d4e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0), OperatorState(operatorID: bb0199e122f4970209706820e4def95b, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 61147562), OperatorState(operatorID: d1c48e48788f3367ffb96f7809bcfb9d, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 16656374), OperatorState(operatorID: 44aee4921a91b9a2911691cc10171a24, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 2935159), OperatorState(operatorID: cc6f3a804a6c2964c83f3008635bbb25, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 3862), OperatorState(operatorID: 70d8f5f0f7af92550fed76e829eeea46, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:41:33.890 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - deleting instance base path /media/flink/tmp1/flink-io-f75e134b-721b-4129-87e6-1122b036eae8/job-c5e1089f1647880d96ed2d46b8409020_op-WindowOperator_8_0_uuid-388671d8-02ec-4380-90cd-1f20eb2ea903
2017-06-20 21:41:38.525 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping checkpoint coordinator for job c5e1089f1647880d96ed2d46b8409020
2017-06-20 21:41:38.526 [ip-10-150-96-228] INFO o.a.f.runtime.checkpoint.StandaloneCompletedCheckpointStore - Shutting down
2017-06-20 21:41:38.527 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CompletedCheckpoint - Persistent checkpoint with ID 1 at 's3://mm-flink-checkpoints/cliff/checkpoint_metadata-70f3b599d065' not discarded.
2017-06-20 21:41:38.527 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CompletedCheckpoint - Persistent checkpoint with ID 2 at 's3://mm-flink-checkpoints/cliff/checkpoint_metadata-793e9e00002c' not discarded.
2017-06-20 21:41:58.547 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Persisting periodic checkpoints externally at s3://mm-flink-checkpoints//cliff.
2017-06-20 21:41:58.548 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Starting job from savepoint s3://mm-flink-checkpoints/cliff/checkpoint_metadata-793e9e00002c ()
2017-06-20 21:41:58.549 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.savepoint.SavepointStore - Loading savepoint from s3://mm-flink-checkpoints/cliff/checkpoint_metadata-793e9e00002c
2017-06-20 21:41:58.795 [ip-10-150-96-228] DEBUG o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - deserialize 2:
[OperatorState(operatorID: 3c249f46784e7918941b8254a455626e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0), OperatorState(operatorID: 2507a43fb0c9c91dd4f7e60bcfef8d4e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0), OperatorState(operatorID: bb0199e122f4970209706820e4def95b, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 61147562), OperatorState(operatorID: d1c48e48788f3367ffb96f7809bcfb9d, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 22249006), OperatorState(operatorID: 44aee4921a91b9a2911691cc10171a24, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 2935159), OperatorState(operatorID: cc6f3a804a6c2964c83f3008635bbb25, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 3862), OperatorState(operatorID: 70d8f5f0f7af92550fed76e829eeea46, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)]
[]
2017-06-20 21:41:59.382 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Reset the checkpoint ID to 3.
2017-06-20 21:41:59.383 [ip-10-150-96-228] INFO o.a.f.runtime.checkpoint.StandaloneCompletedCheckpointStore - org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore@396765a6 recover: SharedStateRegistry org.apache.flink.runtime.state.SharedStateRegistry@724fadb2
2017-06-20 21:41:59.383 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 2 @ 0 for 522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:41:59.386 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore
2017-06-20 21:41:59.848 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:41:59.851 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:41:59.866 [ip-10-150-96-53] INFO o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Initializing RocksDB keyed state backend from snapshot.
2017-06-20 21:41:59.866 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Restoring snapshot from state handles: [org.apache.flink.runtime.state.IncrementalKeyedStateHandle@9661fc97].
2017-06-20 21:41:59.874 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - restoring instance org.apache.flink.runtime.state.IncrementalKeyedStateHandle@9661fc97, hasExtrayKeys: false
2017-06-20 21:41:59.874 [ip-10-150-96-53] TRACE o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - reading 000011.sst -> File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-1/55a19c83-0de5-432a-8e31-4ddf76dfc7be [5592632 bytes]
2017-06-20 21:41:59.898 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:41:59.911 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:41:59.988 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:42:00.079 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:42:00.095 [ip-10-150-96-53] TRACE o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - reading 000012.sst -> File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/304ba12f-42b7-42b6-b1dc-b7b3e69b4fb7 [5987139 bytes]
2017-06-20 21:42:00.159 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:42:00.277 [ip-10-150-96-53] TRACE o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - reading MANIFEST-000006 -> ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/1062b8b6-1877-4a3d-9fea-52ee64c3cd58'}
2017-06-20 21:42:00.289 [ip-10-150-96-53] TRACE o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - reading OPTIONS-000010 -> File State: s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/8f625ca1-b79a-43cf-a528-14de13bca247 [8289 bytes]
2017-06-20 21:42:00.347 [ip-10-150-96-53] TRACE o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - reading CURRENT -> ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020/chk-2/0061918f-2a7c-45fc-8cb6-47f4bbd057d1'}
2017-06-20 21:42:00.471 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:42:00.949 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:42:01.086 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:42:01.121 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:42:59.386 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 3 @ 1497994979386
2017-06-20 21:42:59.400 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 3 from task 0afbc13f841687c6f5e3d3a072f7a0be of job 522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:43:00.337 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34, synchronous part) in thread Thread[Async calls on Source: S3 File Watcher (1/1),5,Flink Task Threads] took 2 ms.
2017-06-20 21:43:00.414 [ip-10-150-96-53] INFO com.mediamath.reporting.combiner.Combiner - start combiner snapshot for id 0
2017-06-20 21:43:00.421 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 3 from task 09d6dfef7c36fd4d8e54586afe58d644 of job 522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:43:00.444 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34, synchronous part) in thread Thread[S3 File Reader -> Parser -> Timestamps/Watermarks -> Combiner (1/1),5,Flink Task Threads] took 21 ms.
2017-06-20 21:43:00.824 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34, asynchronous part) in thread Thread[pool-11-thread-1,5,Flink Task Threads] took 485 ms.
2017-06-20 21:43:00.832 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 3 from task 12f76c24754e69ee348ec52276801e6c of job 522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:43:02.831 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle File State: s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34/chk-3/e37f9a87-2a5b-485b-a81e-247f82e2f0a7 [5472914 bytes]
2017-06-20 21:43:02.832 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - mapping 000017.sst -> File State: s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34/chk-3/e37f9a87-2a5b-485b-a81e-247f82e2f0a7 [5472914 bytes]
2017-06-20 21:43:02.832 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - 000012.sst exists already, mapping it -> PlaceholderStreamStateHandle
2017-06-20 21:43:02.832 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - 000011.sst exists already, mapping it -> PlaceholderStreamStateHandle
2017-06-20 21:43:02.832 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34/chk-3/c22939d1-43e9-49bb-bdd2-876d510b84a2'}
2017-06-20 21:43:03.066 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle File State: s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34/chk-3/e56659bb-3cb6-4f92-9133-3505eaf9dc43 [8290 bytes]
2017-06-20 21:43:03.095 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 3 from task 14357479966b6f0c53112292b6e3dda7 of job 522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:43:03.085 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Got StreamStateHandle from stream.closeAndGetHandle ByteStreamStateHandle{handleName='s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34/chk-3/9c100d38-6752-4f03-a548-54a495407b04'}
2017-06-20 21:43:05.445 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34, synchronous part) in thread Thread[S3 File Reader -> Parser -> Timestamps/Watermarks -> Combiner (1/1),5,Flink Task Threads] took 4990 ms.
2017-06-20 21:43:05.831 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34, asynchronous part) in thread Thread[pool-13-thread-1,5,Flink Task Threads] took 370 ms.
2017-06-20 21:43:09.181 [ip-10-150-96-53] INFO org.apache.flink.runtime.state.DefaultOperatorStateBackend - DefaultOperatorStateBackend snapshot (File Stream Factory @ s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34, asynchronous part) in thread Thread[pool-13-thread-1,5,Flink Task Threads] took 3349 ms.
2017-06-20 21:43:09.214 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received acknowledge message for checkpoint 3 from task d53a108c1f38a035211d1d24b610b383 of job 522565e4a5c6a822f3047af4ea894f34.
2017-06-20 21:43:09.216 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 3c249f46784e7918941b8254a455626e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:43:09.218 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 2507a43fb0c9c91dd4f7e60bcfef8d4e, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:43:09.218 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: bb0199e122f4970209706820e4def95b, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 87601693)
2017-06-20 21:43:09.218 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: d1c48e48788f3367ffb96f7809bcfb9d, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 18373364)
2017-06-20 21:43:09.218 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registeredStates.put(WindowOperator_26_0-KeyGroupRange{startKeyGroup=0, endKeyGroup=127}-000011.sst, org.apache.flink.runtime.state.SharedStateRegistry$SharedStateEntry@302fe16d);
2017-06-20 21:43:09.218 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registeredStates.put(WindowOperator_26_0-KeyGroupRange{startKeyGroup=0, endKeyGroup=127}-000017.sst, org.apache.flink.runtime.state.SharedStateRegistry$SharedStateEntry@341063b0);
2017-06-20 21:43:09.218 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registeredStates.put(WindowOperator_26_0-KeyGroupRange{startKeyGroup=0, endKeyGroup=127}-000012.sst, org.apache.flink.runtime.state.SharedStateRegistry$SharedStateEntry@7112eb56);
2017-06-20 21:43:09.218 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 44aee4921a91b9a2911691cc10171a24, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 1987538)
2017-06-20 21:43:09.219 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: cc6f3a804a6c2964c83f3008635bbb25, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 3862)
2017-06-20 21:43:09.220 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - registering OperatorState(operatorID: 70d8f5f0f7af92550fed76e829eeea46, parallelism: 1, maxParallelism: 128, sub task states: 1, total size (bytes): 0)
2017-06-20 21:43:09.249 [ip-10-150-96-228] DEBUG o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serialize 3
2017-06-20 21:43:09.262 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34/chk-3/d8038bfb-e98f-4dc2-96f1-90e9a6d897d6 [87601693 bytes]
2017-06-20 21:43:09.262 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing File State: s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34/chk-3/1fabc1fc-1099-4e24-a36b-c53151a107ec [23438 bytes]
2017-06-20 21:43:09.262 [ip-10-150-96-228] TRACE o.a.flink.runtime.checkpoint.savepoint.SavepointV2Serializer - serializing org.apache.flink.runtime.state.PlaceholderStreamStateHandle@4758cef9
2017-06-20 21:43:10.271 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - WindowOperator_26_0-KeyGroupRange{startKeyGroup=0, endKeyGroup=127}-000011.sst ReferenceCount() <= 0 -> registeredStates.remove(org.apache.flink.runtime.state.SharedStateRegistry$SharedStateEntry@302fe16d)
2017-06-20 21:43:10.272 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - org.apache.flink.runtime.state.PlaceholderStreamStateHandle@4758cef9 is Placeholder, not deleting
2017-06-20 21:43:10.272 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - WindowOperator_26_0-KeyGroupRange{startKeyGroup=0, endKeyGroup=127}-000017.sst ReferenceCount() <= 0 -> registeredStates.remove(org.apache.flink.runtime.state.SharedStateRegistry$SharedStateEntry@341063b0)
2017-06-20 21:43:10.272 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - !isPlaceholder(File State: s3://mm-prod-stats-pulse-streaming/test/state/522565e4a5c6a822f3047af4ea894f34/chk-3/e37f9a87-2a5b-485b-a81e-247f82e2f0a7 [5472914 bytes]) -> asyncDisposalExecutor.execute(..)
2017-06-20 21:43:10.273 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - WindowOperator_26_0-KeyGroupRange{startKeyGroup=0, endKeyGroup=127}-000012.sst ReferenceCount() <= 0 -> registeredStates.remove(org.apache.flink.runtime.state.SharedStateRegistry$SharedStateEntry@7112eb56)
2017-06-20 21:43:10.273 [ip-10-150-96-228] DEBUG org.apache.flink.runtime.state.SharedStateRegistry - org.apache.flink.runtime.state.PlaceholderStreamStateHandle@713107a2 is Placeholder, not deleting
2017-06-20 21:43:52.497 [ip-10-150-96-53] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - deleting instance base path /media/flink/tmp0/flink-io-1c3e03f6-bdcf-46d8-9719-76bd6f4d374c/job-522565e4a5c6a822f3047af4ea894f34_op-WindowOperator_26_0_uuid-964ab18a-5b3e-4cc4-a258-dae5ea139b0f
2017-06-20 21:43:55.759 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping checkpoint coordinator for job 522565e4a5c6a822f3047af4ea894f34
2017-06-20 21:43:55.760 [ip-10-150-96-228] INFO o.a.f.runtime.checkpoint.StandaloneCompletedCheckpointStore - Shutting down
2017-06-20 21:43:55.760 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CompletedCheckpoint - Persistent checkpoint with ID 2 at 's3://mm-flink-checkpoints/cliff/checkpoint_metadata-793e9e00002c' not discarded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment