yihua/gist:c391601e859c8f828dc2678330dce793 Secret

## gistfile1.txt
22/02/07 23:47:18 INFO DAGScheduler: Submitting ResultStage 88 (MapPartitionsRDD[197] at map at BaseSparkCommitActionExecutor.java:286), which has no missing parents
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_75 stored as values in memory (estimated size 432.2 KiB, free 984.5 MiB)
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_75_piece0 stored as bytes in memory (estimated size 151.7 KiB, free 984.3 MiB)
22/02/07 23:47:18 INFO BlockManagerInfo: Added broadcast_75_piece0 in memory on 192.168.1.37:53037 (size: 151.7 KiB, free: 986.8 MiB)
22/02/07 23:47:18 INFO SparkContext: Created broadcast 75 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 88 (MapPartitionsRDD[197] at map at BaseSparkCommitActionExecutor.java:286) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:18 INFO TaskSchedulerImpl: Adding task set 88.0 with 1 tasks resource profile 0
22/02/07 23:47:18 INFO TaskSetManager: Starting task 0.0 in stage 88.0 (TID 733) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO Executor: Running task 0.0 in stage 88.0 (TID 733)
22/02/07 23:47:18 INFO ShuffleBlockFetcherIterator: Getting 1 (1051.0 B) non-empty blocks including 1 (1051.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:18 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220207233505725 for file files-0000
22/02/07 23:47:18 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:18 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:18 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO AbstractTableFileSystemView: Took 1 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:18 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:18 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:18 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=27, NumFileGroups=1, FileGroupsCreationTime=5, StoreTimeTaken=0
22/02/07 23:47:18 INFO DirectWriteMarkers: Creating Marker Path=/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207233505725/files/files-0000_0-88-733_00000000000000.hfile.marker.APPEND
22/02/07 23:47:18 INFO DirectWriteMarkers: [direct] Created marker file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207233505725/files/files-0000_0-88-733_00000000000000.hfile.marker.APPEND in 28 ms
22/02/07 23:47:18 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer
22/02/07 23:47:18 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.25_0-67-645
22/02/07 23:47:18 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.26_0-88-733', fileLen=-1}
22/02/07 23:47:18 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:47:18 INFO CodecPool: Got brand-new compressor [.gz]
22/02/07 23:47:18 INFO CodecPool: Got brand-new compressor [.gz]
22/02/07 23:47:18 INFO HoodieAppendHandle: AppendHandle for partitionPath files filePath files/.files-0000_00000000000000.log.26_0-88-733, took 67 ms.
22/02/07 23:47:18 INFO MemoryStore: Block rdd_196_0 stored as values in memory (estimated size 1497.0 B, free 984.3 MiB)
22/02/07 23:47:18 INFO BlockManagerInfo: Added rdd_196_0 in memory on 192.168.1.37:53037 (size: 1497.0 B, free: 986.8 MiB)
22/02/07 23:47:18 INFO Executor: Finished task 0.0 in stage 88.0 (TID 733). 2651 bytes result sent to driver
22/02/07 23:47:18 INFO TaskSetManager: Finished task 0.0 in stage 88.0 (TID 733) in 95 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:18 INFO TaskSchedulerImpl: Removed TaskSet 88.0, whose tasks have all completed, from pool
22/02/07 23:47:18 INFO DAGScheduler: ResultStage 88 (collect at BaseSparkCommitActionExecutor.java:286) finished in 0.164 s
22/02/07 23:47:18 INFO DAGScheduler: Job 54 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:18 INFO TaskSchedulerImpl: Killing all running tasks in stage 88: Stage finished
22/02/07 23:47:18 INFO DAGScheduler: Job 54 finished: collect at BaseSparkCommitActionExecutor.java:286, took 0.243793 s
22/02/07 23:47:18 INFO CommitUtils: Creating  metadata for UPSERT_PREPPED numWriteStats:1numReplaceFileIds:0
22/02/07 23:47:18 INFO SparkContext: Starting job: collect at BaseSparkCommitActionExecutor.java:294
22/02/07 23:47:18 INFO DAGScheduler: Got job 55 (collect at BaseSparkCommitActionExecutor.java:294) with 1 output partitions
22/02/07 23:47:18 INFO DAGScheduler: Final stage: ResultStage 90 (collect at BaseSparkCommitActionExecutor.java:294)
22/02/07 23:47:18 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 89)
22/02/07 23:47:18 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:18 INFO DAGScheduler: Submitting ResultStage 90 (MapPartitionsRDD[198] at map at BaseSparkCommitActionExecutor.java:294), which has no missing parents
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_76 stored as values in memory (estimated size 432.2 KiB, free 983.9 MiB)
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_76_piece0 stored as bytes in memory (estimated size 151.7 KiB, free 983.8 MiB)
22/02/07 23:47:18 INFO BlockManagerInfo: Added broadcast_76_piece0 in memory on 192.168.1.37:53037 (size: 151.7 KiB, free: 986.7 MiB)
22/02/07 23:47:18 INFO SparkContext: Created broadcast 76 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 90 (MapPartitionsRDD[198] at map at BaseSparkCommitActionExecutor.java:294) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:18 INFO TaskSchedulerImpl: Adding task set 90.0 with 1 tasks resource profile 0
22/02/07 23:47:18 INFO TaskSetManager: Starting task 0.0 in stage 90.0 (TID 734) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO Executor: Running task 0.0 in stage 90.0 (TID 734)
22/02/07 23:47:18 INFO BlockManager: Found block rdd_196_0 locally
22/02/07 23:47:18 INFO Executor: Finished task 0.0 in stage 90.0 (TID 734). 2264 bytes result sent to driver
22/02/07 23:47:18 INFO TaskSetManager: Finished task 0.0 in stage 90.0 (TID 734) in 19 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:18 INFO TaskSchedulerImpl: Removed TaskSet 90.0, whose tasks have all completed, from pool
22/02/07 23:47:18 INFO DAGScheduler: ResultStage 90 (collect at BaseSparkCommitActionExecutor.java:294) finished in 0.090 s
22/02/07 23:47:18 INFO DAGScheduler: Job 55 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:18 INFO TaskSchedulerImpl: Killing all running tasks in stage 90: Stage finished
22/02/07 23:47:18 INFO DAGScheduler: Job 55 finished: collect at BaseSparkCommitActionExecutor.java:294, took 0.091642 s
22/02/07 23:47:18 INFO BaseSparkCommitActionExecutor: Committing 20220207233505725, action Type deltacommit, operation Type UPSERT_PREPPED
22/02/07 23:47:18 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/07 23:47:18 INFO DAGScheduler: Got job 56 (collect at HoodieSparkEngineContext.java:134) with 1 output partitions
22/02/07 23:47:18 INFO DAGScheduler: Final stage: ResultStage 91 (collect at HoodieSparkEngineContext.java:134)
22/02/07 23:47:18 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:18 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:18 INFO DAGScheduler: Submitting ResultStage 91 (MapPartitionsRDD[200] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_77 stored as values in memory (estimated size 99.2 KiB, free 983.7 MiB)
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_77_piece0 stored as bytes in memory (estimated size 35.1 KiB, free 983.6 MiB)
22/02/07 23:47:18 INFO BlockManagerInfo: Added broadcast_77_piece0 in memory on 192.168.1.37:53037 (size: 35.1 KiB, free: 986.6 MiB)
22/02/07 23:47:18 INFO SparkContext: Created broadcast 77 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 91 (MapPartitionsRDD[200] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:18 INFO TaskSchedulerImpl: Adding task set 91.0 with 1 tasks resource profile 0
22/02/07 23:47:18 INFO TaskSetManager: Starting task 0.0 in stage 91.0 (TID 735) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4456 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO Executor: Running task 0.0 in stage 91.0 (TID 735)
22/02/07 23:47:18 INFO Executor: Finished task 0.0 in stage 91.0 (TID 735). 796 bytes result sent to driver
22/02/07 23:47:18 INFO TaskSetManager: Finished task 0.0 in stage 91.0 (TID 735) in 18 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:18 INFO TaskSchedulerImpl: Removed TaskSet 91.0, whose tasks have all completed, from pool
22/02/07 23:47:18 INFO DAGScheduler: ResultStage 91 (collect at HoodieSparkEngineContext.java:134) finished in 0.036 s
22/02/07 23:47:18 INFO DAGScheduler: Job 56 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:18 INFO TaskSchedulerImpl: Killing all running tasks in stage 91: Stage finished
22/02/07 23:47:18 INFO DAGScheduler: Job 56 finished: collect at HoodieSparkEngineContext.java:134, took 0.037406 s
22/02/07 23:47:18 INFO HoodieActiveTimeline: Marking instant complete [==>20220207233505725__deltacommit__INFLIGHT]
22/02/07 23:47:18 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233505725.deltacommit.inflight
22/02/07 23:47:18 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233505725.deltacommit
22/02/07 23:47:18 INFO HoodieActiveTimeline: Completed [==>20220207233505725__deltacommit__INFLIGHT]
22/02/07 23:47:18 INFO BaseSparkCommitActionExecutor: Committed 20220207233505725
22/02/07 23:47:18 INFO SparkContext: Starting job: collectAsMap at HoodieSparkEngineContext.java:148
22/02/07 23:47:18 INFO DAGScheduler: Got job 57 (collectAsMap at HoodieSparkEngineContext.java:148) with 1 output partitions
22/02/07 23:47:18 INFO DAGScheduler: Final stage: ResultStage 92 (collectAsMap at HoodieSparkEngineContext.java:148)
22/02/07 23:47:18 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:18 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:18 INFO DAGScheduler: Submitting ResultStage 92 (MapPartitionsRDD[202] at mapToPair at HoodieSparkEngineContext.java:145), which has no missing parents
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_78 stored as values in memory (estimated size 99.4 KiB, free 983.5 MiB)
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_78_piece0 stored as bytes in memory (estimated size 35.1 KiB, free 983.5 MiB)
22/02/07 23:47:18 INFO BlockManagerInfo: Added broadcast_78_piece0 in memory on 192.168.1.37:53037 (size: 35.1 KiB, free: 986.6 MiB)
22/02/07 23:47:18 INFO SparkContext: Created broadcast 78 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 92 (MapPartitionsRDD[202] at mapToPair at HoodieSparkEngineContext.java:145) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:18 INFO TaskSchedulerImpl: Adding task set 92.0 with 1 tasks resource profile 0
22/02/07 23:47:18 INFO TaskSetManager: Starting task 0.0 in stage 92.0 (TID 736) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4456 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO Executor: Running task 0.0 in stage 92.0 (TID 736)
22/02/07 23:47:18 INFO Executor: Finished task 0.0 in stage 92.0 (TID 736). 883 bytes result sent to driver
22/02/07 23:47:18 INFO TaskSetManager: Finished task 0.0 in stage 92.0 (TID 736) in 7 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:18 INFO TaskSchedulerImpl: Removed TaskSet 92.0, whose tasks have all completed, from pool
22/02/07 23:47:18 INFO DAGScheduler: ResultStage 92 (collectAsMap at HoodieSparkEngineContext.java:148) finished in 0.026 s
22/02/07 23:47:18 INFO DAGScheduler: Job 57 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:18 INFO TaskSchedulerImpl: Killing all running tasks in stage 92: Stage finished
22/02/07 23:47:18 INFO DAGScheduler: Job 57 finished: collectAsMap at HoodieSparkEngineContext.java:148, took 0.026577 s
22/02/07 23:47:18 INFO FSUtils: Removed directory at /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207233505725
22/02/07 23:47:18 INFO HoodieHeartbeatClient: Stopping heartbeat for instant 20220207233505725
22/02/07 23:47:18 INFO HoodieHeartbeatClient: Stopped heartbeat for instant 20220207233505725
22/02/07 23:47:18 INFO HeartbeatUtils: Deleted the heartbeat for instant 20220207233505725
22/02/07 23:47:18 INFO HoodieHeartbeatClient: Deleted heartbeat file for instant 20220207233505725
22/02/07 23:47:18 INFO SparkContext: Starting job: collect at SparkHoodieBackedTableMetadataWriter.java:154
22/02/07 23:47:18 INFO DAGScheduler: Got job 58 (collect at SparkHoodieBackedTableMetadataWriter.java:154) with 1 output partitions
22/02/07 23:47:18 INFO DAGScheduler: Final stage: ResultStage 94 (collect at SparkHoodieBackedTableMetadataWriter.java:154)
22/02/07 23:47:18 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 93)
22/02/07 23:47:18 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:18 INFO DAGScheduler: Submitting ResultStage 94 (MapPartitionsRDD[196] at flatMap at BaseSparkCommitActionExecutor.java:181), which has no missing parents
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_79 stored as values in memory (estimated size 431.8 KiB, free 983.1 MiB)
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_79_piece0 stored as bytes in memory (estimated size 151.6 KiB, free 982.9 MiB)
22/02/07 23:47:18 INFO BlockManagerInfo: Added broadcast_79_piece0 in memory on 192.168.1.37:53037 (size: 151.6 KiB, free: 986.4 MiB)
22/02/07 23:47:18 INFO SparkContext: Created broadcast 79 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 94 (MapPartitionsRDD[196] at flatMap at BaseSparkCommitActionExecutor.java:181) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:18 INFO TaskSchedulerImpl: Adding task set 94.0 with 1 tasks resource profile 0
22/02/07 23:47:18 INFO TaskSetManager: Starting task 0.0 in stage 94.0 (TID 737) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO Executor: Running task 0.0 in stage 94.0 (TID 737)
22/02/07 23:47:18 INFO BlockManager: Found block rdd_196_0 locally
22/02/07 23:47:18 INFO Executor: Finished task 0.0 in stage 94.0 (TID 737). 2344 bytes result sent to driver
22/02/07 23:47:18 INFO TaskSetManager: Finished task 0.0 in stage 94.0 (TID 737) in 19 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:18 INFO TaskSchedulerImpl: Removed TaskSet 94.0, whose tasks have all completed, from pool
22/02/07 23:47:18 INFO DAGScheduler: ResultStage 94 (collect at SparkHoodieBackedTableMetadataWriter.java:154) finished in 0.086 s
22/02/07 23:47:18 INFO DAGScheduler: Job 58 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:18 INFO TaskSchedulerImpl: Killing all running tasks in stage 94: Stage finished
22/02/07 23:47:18 INFO DAGScheduler: Job 58 finished: collect at SparkHoodieBackedTableMetadataWriter.java:154, took 0.087892 s
22/02/07 23:47:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:18 INFO SparkRDDWriteClient: Committing Compaction 20220207233505725. Finished with result HoodieCommitMetadata{partitionToWriteStats={2022/1/31=[HoodieWriteStat{fileId='356c259b-ec4a-42d4-b893-9fa3daf45451-0', path='2022/1/31/356c259b-ec4a-42d4-b893-9fa3daf45451-0_0-29-110_20220207233505725.parquet', prevCommit='20220207232657914', numWrites=290546, numDeletes=0, numUpdateWrites=34154, totalWriteBytes=105378409, totalWriteErrors=0, tempPath='null', partitionPath='2022/1/31', totalLogRecords=34154, totalLogFilesCompacted=1, totalLogSizeCompacted=50596866, totalUpdatedRecordsCompacted=34154, totalLogBlocks=1, totalCorruptLogBlock=0, totalRollbackBlocks=0}], 2022/1/30=[HoodieWriteStat{fileId='09f346a5-4fa1-4ae3-93e5-5a23bec43dcf-0', path='2022/1/30/09f346a5-4fa1-4ae3-93e5-5a23bec43dcf-0_9-29-199_20220207233505725.parquet', prevCommit='20220207232657914', numWrites=289380, numDeletes=0, numUpdateWrites=33398, totalWriteBytes=104962704, totalWriteErrors=0, tempPath='null', partitionPath='2022/1/30', totalLogRecords=33398, totalLogFilesCompacted=1, totalLogSizeCompacted=49489132, totalUpdatedRecordsCompacted=33398, totalLogBlocks=1, totalCorruptLogBlock=0, totalRollbackBlocks=0}], 2022/1/28=[HoodieWriteStat{fileId='e766accf-2d87-4fb2-a256-ff45e728094a-0', path='2022/1/28/e766accf-2d87-4fb2-a256-ff45e728094a-0_8-29-198_20220207233505725.parquet', prevCommit='20220207232657914', numWrites=290534, numDeletes=0, numUpdateWrites=33601, totalWriteBytes=105382961, totalWriteErrors=0, tempPath='null', partitionPath='2022/1/28', totalLogRecords=33601, totalLogFilesCompacted=1, totalLogSizeCompacted=49790590, totalUpdatedRecordsCompacted=33601, totalLogBlocks=1, totalCorruptLogBlock=0, totalRollbackBlocks=0}], 2022/1/27=[HoodieWriteStat{fileId='9d74eab4-a6ea-4d22-9f9d-4578f3fb0fad-0', path='2022/1/27/9d74eab4-a6ea-4d22-9f9d-4578f3fb0fad-0_2-29-112_20220207233505725.parquet', prevCommit='20220207232657914', numWrites=291718, numDeletes=0, numUpdateWrites=33978, totalWriteBytes=105817505, totalWriteErrors=0, tempPath='null', partitionPath='2022/1/27', totalLogRecords=33978, totalLogFilesCompacted=1, totalLogSizeCompacted=50382044, totalUpdatedRecordsCompacted=33978, totalLogBlocks=1, totalCorruptLogBlock=0, totalRollbackBlocks=0}], 2022/2/2=[HoodieWriteStat{fileId='102d0248-cf64-42a2-8be8-dbc5c7a693d1-0', path='2022/2/2/102d0248-cf64-42a2-8be8-dbc5c7a693d1-0_5-29-155_20220207233505725.parquet', prevCommit='20220207232657914', numWrites=290378, numDeletes=0, numUpdateWrites=33666, totalWriteBytes=105329124, totalWriteErrors=0, tempPath='null', partitionPath='2022/2/2', totalLogRecords=33666, totalLogFilesCompacted=1, totalLogSizeCompacted=49851666, totalUpdatedRecordsCompacted=33666, totalLogBlocks=1, totalCorruptLogBlock=0, totalRollbackBlocks=0}], 2022/1/29=[HoodieWriteStat{fileId='5cabcfe9-978a-4c55-bc96-130ccab4f081-0', path='2022/1/29/5cabcfe9-978a-4c55-bc96-130ccab4f081-0_6-29-196_20220207233505725.parquet', prevCommit='20220207232657914', numWrites=288472, numDeletes=0, numUpdateWrites=33574, totalWriteBytes=104648309, totalWriteErrors=0, tempPath='null', partitionPath='2022/1/29', totalLogRecords=33574, totalLogFilesCompacted=1, totalLogSizeCompacted=49816430, totalUpdatedRecordsCompacted=33574, totalLogBlocks=1, totalCorruptLogBlock=0, totalRollbackBlocks=0}], 2022/1/24=[HoodieWriteStat{fileId='dac159fb-2bef-4f60-902f-ed6360241988-0', path='2022/1/24/dac159fb-2bef-4f60-902f-ed6360241988-0_3-29-153_20220207233505725.parquet', prevCommit='20220207232657914', numWrites=290726, numDeletes=0, numUpdateWrites=33886, totalWriteBytes=105464758, totalWriteErrors=0, tempPath='null', partitionPath='2022/1/24', totalLogRecords=33886, totalLogFilesCompacted=1, totalLogSizeCompacted=50279533, totalUpdatedRecordsCompacted=33886, totalLogBlocks=1, totalCorruptLogBlock=0, totalRollbackBlocks=0}], 2022/2/1=[HoodieWriteStat{fileId='aca43bde-dc01-41e1-be66-74f78bb58ed8-0', path='2022/2/1/aca43bde-dc01-41e1-be66-74f78bb58ed8-0_7-29-197_20220207233505725.parquet', prevCommit='20220207232657914', numWrites=290139, numDeletes=0, numUpdateWrites=33610, totalWriteBytes=105249814, totalWriteErrors=0, tempPath='null', partitionPath='2022/2/1', totalLogRecords=33610, totalLogFilesCompacted=1, totalLogSizeCompacted=49802370, totalUpdatedRecordsCompacted=33610, totalLogBlocks=1, totalCorruptLogBlock=0, totalRollbackBlocks=0}], 2022/1/26=[HoodieWriteStat{fileId='df3a0019-62aa-4cea-b4c1-82439c339b6f-0', path='2022/1/26/df3a0019-62aa-4cea-b4c1-82439c339b6f-0_4-29-154_20220207233505725.parquet', prevCommit='20220207232657914', numWrites=293471, numDeletes=0, numUpdateWrites=33635, totalWriteBytes=106465405, totalWriteErrors=0, tempPath='null', partitionPath='2022/1/26', totalLogRecords=33635, totalLogFilesCompacted=1, totalLogSizeCompacted=49907522, totalUpdatedRecordsCompacted=33635, totalLogBlocks=1, totalCorruptLogBlock=0, totalRollbackBlocks=0}], 2022/1/25=[HoodieWriteStat{fileId='8fd44984-ac08-4808-b595-da40ff0d8ad6-0', path='2022/1/25/8fd44984-ac08-4808-b595-da40ff0d8ad6-0_1-29-111_20220207233505725.parquet', prevCommit='20220207232657914', numWrites=291547, numDeletes=0, numUpdateWrites=34000, totalWriteBytes=105710738, totalWriteErrors=0, tempPath='null', partitionPath='2022/1/25', totalLogRecords=34000, totalLogFilesCompacted=1, totalLogSizeCompacted=50447414, totalUpdatedRecordsCompacted=34000, totalLogBlocks=1, totalCorruptLogBlock=0, totalRollbackBlocks=0}]}, compacted=true, extraMetadata={schema={"type":"record","name":"hoodie_source","namespace":"hoodie.source","fields":[{"name":"key","type":["null","string"],"default":null},{"name":"partition","type":["null","string"],"default":null},{"name":"ts","type":["null","long"],"default":null},{"name":"textField","type":["null","string"],"default":null},{"name":"decimalField","type":["null","float"],"default":null},{"name":"longField","type":["null","long"],"default":null},{"name":"arrayField","type":["null",{"type":"array","items":["int","null"]}],"default":null},{"name":"mapField","type":["null",{"type":"map","values":["int","null"]}],"default":null},{"name":"round","type":["null","int"],"default":null}]}}, operationType=UNKNOWN}
22/02/07 23:47:18 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207233505725.compaction.inflight
22/02/07 23:47:18 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207233505725.commit
22/02/07 23:47:18 INFO TransactionManager: Transaction ending with transaction owner Option{val=[==>20220207233505725__compaction__INFLIGHT]}
22/02/07 23:47:18 INFO InProcessLockProvider: Thread async_compact_thread RELEASING in-process lock.
22/02/07 23:47:18 INFO InProcessLockProvider: Thread pool-26-thread-1 ACQUIRED in-process lock.
22/02/07 23:47:18 INFO TransactionManager: Transaction started for Optional.empty with latest completed transaction instant Optional.empty
22/02/07 23:47:18 INFO InProcessLockProvider: Thread async_compact_thread RELEASED in-process lock.
22/02/07 23:47:18 INFO TransactionManager: Transaction ended with transaction owner Option{val=[==>20220207233505725__compaction__INFLIGHT]}
22/02/07 23:47:18 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53112/v1/hoodie/marker/dir/delete?markerdirpath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220207233505725)
22/02/07 23:47:18 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:18 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:47:18 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:18 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:18 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:18 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:18 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:18 INFO HoodieTableMetadataUtil: Loading latest file slices for metadata table partition files
22/02/07 23:47:18 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:18 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:18 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:18 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=28, NumFileGroups=1, FileGroupsCreationTime=6, StoreTimeTaken=0
22/02/07 23:47:18 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:18 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:47:18 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:18 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:47:18 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:18 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:18 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO HoodieTableMetadataUtil: Updating at 20220207234319263 from Clean. #partitions_updated=10, #files_deleted=20
22/02/07 23:47:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:18 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:18 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:18 INFO HoodieTableMetadataUtil: Loading latest file slices for metadata table partition files
22/02/07 23:47:18 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:18 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:18 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:18 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=28, NumFileGroups=1, FileGroupsCreationTime=5, StoreTimeTaken=0
22/02/07 23:47:18 INFO BaseHoodieClient: Embedded Timeline Server is disabled. Not starting timeline service
22/02/07 23:47:18 INFO SparkContext: Starting job: collectAsMap at HoodieSparkEngineContext.java:148
22/02/07 23:47:18 INFO DAGScheduler: Got job 59 (collectAsMap at HoodieSparkEngineContext.java:148) with 8 output partitions
22/02/07 23:47:18 INFO DAGScheduler: Final stage: ResultStage 95 (collectAsMap at HoodieSparkEngineContext.java:148)
22/02/07 23:47:18 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:18 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:18 INFO DAGScheduler: Submitting ResultStage 95 (MapPartitionsRDD[204] at mapToPair at HoodieSparkEngineContext.java:145), which has no missing parents
22/02/07 23:47:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:18 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:18 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_80 stored as values in memory (estimated size 99.4 KiB, free 982.8 MiB)
22/02/07 23:47:18 INFO MemoryStore: Block broadcast_80_piece0 stored as bytes in memory (estimated size 35.1 KiB, free 982.8 MiB)
22/02/07 23:47:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:18 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:18 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:18 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:18 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:18 INFO BlockManagerInfo: Added broadcast_80_piece0 in memory on 192.168.1.37:53037 (size: 35.1 KiB, free: 986.4 MiB)
22/02/07 23:47:18 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:18 INFO SparkContext: Created broadcast 80 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:18 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 95 (MapPartitionsRDD[204] at mapToPair at HoodieSparkEngineContext.java:145) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
22/02/07 23:47:18 INFO TaskSchedulerImpl: Adding task set 95.0 with 8 tasks resource profile 0
22/02/07 23:47:18 INFO TaskSetManager: Starting task 0.0 in stage 95.0 (TID 738) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4443 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO TaskSetManager: Starting task 1.0 in stage 95.0 (TID 739) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4443 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO Executor: Running task 1.0 in stage 95.0 (TID 739)
22/02/07 23:47:18 INFO Executor: Running task 0.0 in stage 95.0 (TID 738)
22/02/07 23:47:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:18 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:18 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:18 INFO Executor: Finished task 1.0 in stage 95.0 (TID 739). 913 bytes result sent to driver
22/02/07 23:47:18 INFO Executor: Finished task 0.0 in stage 95.0 (TID 738). 913 bytes result sent to driver
22/02/07 23:47:18 INFO TaskSetManager: Starting task 2.0 in stage 95.0 (TID 740) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4443 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO Executor: Running task 2.0 in stage 95.0 (TID 740)
22/02/07 23:47:18 INFO TaskSetManager: Starting task 3.0 in stage 95.0 (TID 741) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4442 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO Executor: Running task 3.0 in stage 95.0 (TID 741)
22/02/07 23:47:18 INFO TaskSetManager: Finished task 0.0 in stage 95.0 (TID 738) in 11 ms on 192.168.1.37 (executor driver) (1/8)
22/02/07 23:47:18 INFO TaskSetManager: Finished task 1.0 in stage 95.0 (TID 739) in 10 ms on 192.168.1.37 (executor driver) (2/8)
22/02/07 23:47:18 INFO AsyncCleanerService: Async auto cleaning is not enabled. Not running cleaner now
22/02/07 23:47:18 INFO Executor: Finished task 2.0 in stage 95.0 (TID 740). 870 bytes result sent to driver
22/02/07 23:47:18 INFO TaskSetManager: Starting task 4.0 in stage 95.0 (TID 742) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4442 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO TaskSetManager: Finished task 2.0 in stage 95.0 (TID 740) in 7 ms on 192.168.1.37 (executor driver) (3/8)
22/02/07 23:47:18 INFO Executor: Running task 4.0 in stage 95.0 (TID 742)
22/02/07 23:47:18 INFO Executor: Finished task 3.0 in stage 95.0 (TID 741). 912 bytes result sent to driver
22/02/07 23:47:18 INFO TaskSetManager: Starting task 5.0 in stage 95.0 (TID 743) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4446 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO TaskSetManager: Finished task 3.0 in stage 95.0 (TID 741) in 9 ms on 192.168.1.37 (executor driver) (4/8)
22/02/07 23:47:18 INFO Executor: Running task 5.0 in stage 95.0 (TID 743)
22/02/07 23:47:18 INFO Executor: Finished task 4.0 in stage 95.0 (TID 742). 912 bytes result sent to driver
22/02/07 23:47:18 INFO TaskSetManager: Starting task 6.0 in stage 95.0 (TID 744) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4442 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO TaskSetManager: Finished task 4.0 in stage 95.0 (TID 742) in 8 ms on 192.168.1.37 (executor driver) (5/8)
22/02/07 23:47:18 INFO Executor: Running task 6.0 in stage 95.0 (TID 744)
22/02/07 23:47:18 INFO Executor: Finished task 5.0 in stage 95.0 (TID 743). 916 bytes result sent to driver
22/02/07 23:47:18 INFO TaskSetManager: Starting task 7.0 in stage 95.0 (TID 745) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4442 bytes) taskResourceAssignments Map()
22/02/07 23:47:18 INFO Executor: Running task 7.0 in stage 95.0 (TID 745)
22/02/07 23:47:18 INFO TaskSetManager: Finished task 5.0 in stage 95.0 (TID 743) in 7 ms on 192.168.1.37 (executor driver) (6/8)
22/02/07 23:47:18 INFO SparkContext: Starting job: countByKey at BaseSparkCommitActionExecutor.java:196
22/02/07 23:47:18 INFO DAGScheduler: Registering RDD 210 (countByKey at BaseSparkCommitActionExecutor.java:196) as input to shuffle 28
22/02/07 23:47:18 INFO DAGScheduler: Got job 60 (countByKey at BaseSparkCommitActionExecutor.java:196) with 1 output partitions
22/02/07 23:47:18 INFO DAGScheduler: Final stage: ResultStage 97 (countByKey at BaseSparkCommitActionExecutor.java:196)
22/02/07 23:47:18 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 96)
22/02/07 23:47:18 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 96)
22/02/07 23:47:18 INFO DAGScheduler: Submitting ShuffleMapStage 96 (MapPartitionsRDD[210] at countByKey at BaseSparkCommitActionExecutor.java:196), which has no missing parents
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_81 stored as values in memory (estimated size 13.2 KiB, free 982.8 MiB)
22/02/07 23:47:19 INFO Executor: Finished task 6.0 in stage 95.0 (TID 744). 869 bytes result sent to driver
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_81_piece0 stored as bytes in memory (estimated size 5.6 KiB, free 982.8 MiB)
22/02/07 23:47:19 INFO TaskSetManager: Finished task 6.0 in stage 95.0 (TID 744) in 8 ms on 192.168.1.37 (executor driver) (7/8)
22/02/07 23:47:19 INFO BlockManagerInfo: Added broadcast_81_piece0 in memory on 192.168.1.37:53037 (size: 5.6 KiB, free: 986.4 MiB)
22/02/07 23:47:19 INFO SparkContext: Created broadcast 81 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:19 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 96 (MapPartitionsRDD[210] at countByKey at BaseSparkCommitActionExecutor.java:196) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:19 INFO TaskSchedulerImpl: Adding task set 96.0 with 1 tasks resource profile 0
22/02/07 23:47:19 INFO TaskSetManager: Starting task 0.0 in stage 96.0 (TID 746) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 6514 bytes) taskResourceAssignments Map()
22/02/07 23:47:19 INFO Executor: Running task 0.0 in stage 96.0 (TID 746)
22/02/07 23:47:19 INFO Executor: Finished task 7.0 in stage 95.0 (TID 745). 912 bytes result sent to driver
22/02/07 23:47:19 INFO TaskSetManager: Finished task 7.0 in stage 95.0 (TID 745) in 10 ms on 192.168.1.37 (executor driver) (8/8)
22/02/07 23:47:19 INFO MemoryStore: Block rdd_208_0 stored as values in memory (estimated size 3.8 KiB, free 982.8 MiB)
22/02/07 23:47:19 INFO TaskSchedulerImpl: Removed TaskSet 95.0, whose tasks have all completed, from pool
22/02/07 23:47:19 INFO BlockManagerInfo: Added rdd_208_0 in memory on 192.168.1.37:53037 (size: 3.8 KiB, free: 986.4 MiB)
22/02/07 23:47:19 INFO DAGScheduler: ResultStage 95 (collectAsMap at HoodieSparkEngineContext.java:148) finished in 0.055 s
22/02/07 23:47:19 INFO DAGScheduler: Job 59 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:19 INFO TaskSchedulerImpl: Killing all running tasks in stage 95: Stage finished
22/02/07 23:47:19 INFO DAGScheduler: Job 59 finished: collectAsMap at HoodieSparkEngineContext.java:148, took 0.056810 s
22/02/07 23:47:19 INFO FSUtils: Removed directory at /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220207233505725
22/02/07 23:47:19 INFO SparkRDDWriteClient: Compacted successfully on commit 20220207233505725
22/02/07 23:47:19 INFO AsyncCompactService: Finished Compaction for instant [==>20220207233505725__compaction__INFLIGHT]
22/02/07 23:47:19 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:47:19 INFO Executor: Finished task 0.0 in stage 96.0 (TID 746). 1086 bytes result sent to driver
22/02/07 23:47:19 INFO TaskSetManager: Finished task 0.0 in stage 96.0 (TID 746) in 6 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:19 INFO TaskSchedulerImpl: Removed TaskSet 96.0, whose tasks have all completed, from pool
22/02/07 23:47:19 INFO DAGScheduler: ShuffleMapStage 96 (countByKey at BaseSparkCommitActionExecutor.java:196) finished in 0.009 s
22/02/07 23:47:19 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:19 INFO DAGScheduler: running: Set(ResultStage 73)
22/02/07 23:47:19 INFO DAGScheduler: waiting: Set(ResultStage 97)
22/02/07 23:47:19 INFO DAGScheduler: failed: Set()
22/02/07 23:47:19 INFO DAGScheduler: Submitting ResultStage 97 (ShuffledRDD[211] at countByKey at BaseSparkCommitActionExecutor.java:196), which has no missing parents
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_82 stored as values in memory (estimated size 5.6 KiB, free 982.8 MiB)
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_82_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 982.8 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Added broadcast_82_piece0 in memory on 192.168.1.37:53037 (size: 3.2 KiB, free: 986.4 MiB)
22/02/07 23:47:19 INFO SparkContext: Created broadcast 82 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 97 (ShuffledRDD[211] at countByKey at BaseSparkCommitActionExecutor.java:196) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:19 INFO TaskSchedulerImpl: Adding task set 97.0 with 1 tasks resource profile 0
22/02/07 23:47:19 INFO TaskSetManager: Starting task 0.0 in stage 97.0 (TID 747) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:19 INFO Executor: Running task 0.0 in stage 97.0 (TID 747)
22/02/07 23:47:19 INFO ShuffleBlockFetcherIterator: Getting 1 (142.0 B) non-empty blocks including 1 (142.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:19 INFO Executor: Finished task 0.0 in stage 97.0 (TID 747). 1312 bytes result sent to driver
22/02/07 23:47:19 INFO TaskSetManager: Finished task 0.0 in stage 97.0 (TID 747) in 4 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:19 INFO TaskSchedulerImpl: Removed TaskSet 97.0, whose tasks have all completed, from pool
22/02/07 23:47:19 INFO DAGScheduler: ResultStage 97 (countByKey at BaseSparkCommitActionExecutor.java:196) finished in 0.006 s
22/02/07 23:47:19 INFO DAGScheduler: Job 60 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:19 INFO TaskSchedulerImpl: Killing all running tasks in stage 97: Stage finished
22/02/07 23:47:19 INFO DAGScheduler: Job 60 finished: countByKey at BaseSparkCommitActionExecutor.java:196, took 0.017995 s
22/02/07 23:47:19 INFO BaseSparkCommitActionExecutor: Workload profile :WorkloadProfile {globalStat=WorkloadStat {numInserts=0, numUpdates=10}, partitionStat={files=WorkloadStat {numInserts=0, numUpdates=10}}, operationType=UPSERT_PREPPED}
22/02/07 23:47:19 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234319263.deltacommit.requested
22/02/07 23:47:19 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234319263.deltacommit.inflight
22/02/07 23:47:19 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:19 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:19 INFO SparkContext: Starting job: collect at SparkRejectUpdateStrategy.java:52
22/02/07 23:47:19 INFO DAGScheduler: Registering RDD 214 (distinct at SparkRejectUpdateStrategy.java:52) as input to shuffle 29
22/02/07 23:47:19 INFO DAGScheduler: Got job 61 (collect at SparkRejectUpdateStrategy.java:52) with 1 output partitions
22/02/07 23:47:19 INFO DAGScheduler: Final stage: ResultStage 99 (collect at SparkRejectUpdateStrategy.java:52)
22/02/07 23:47:19 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 98)
22/02/07 23:47:19 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 98)
22/02/07 23:47:19 INFO DAGScheduler: Submitting ShuffleMapStage 98 (MapPartitionsRDD[214] at distinct at SparkRejectUpdateStrategy.java:52), which has no missing parents
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_83 stored as values in memory (estimated size 13.2 KiB, free 982.8 MiB)
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_83_piece0 stored as bytes in memory (estimated size 5.5 KiB, free 982.8 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Added broadcast_83_piece0 in memory on 192.168.1.37:53037 (size: 5.5 KiB, free: 986.4 MiB)
22/02/07 23:47:19 INFO SparkContext: Created broadcast 83 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:19 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 98 (MapPartitionsRDD[214] at distinct at SparkRejectUpdateStrategy.java:52) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:19 INFO TaskSchedulerImpl: Adding task set 98.0 with 1 tasks resource profile 0
22/02/07 23:47:19 INFO TaskSetManager: Starting task 0.0 in stage 98.0 (TID 748) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 6514 bytes) taskResourceAssignments Map()
22/02/07 23:47:19 INFO Executor: Running task 0.0 in stage 98.0 (TID 748)
22/02/07 23:47:19 INFO BlockManager: Found block rdd_208_0 locally
22/02/07 23:47:19 INFO Executor: Finished task 0.0 in stage 98.0 (TID 748). 1129 bytes result sent to driver
22/02/07 23:47:19 INFO TaskSetManager: Finished task 0.0 in stage 98.0 (TID 748) in 4 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:19 INFO TaskSchedulerImpl: Removed TaskSet 98.0, whose tasks have all completed, from pool
22/02/07 23:47:19 INFO DAGScheduler: ShuffleMapStage 98 (distinct at SparkRejectUpdateStrategy.java:52) finished in 0.006 s
22/02/07 23:47:19 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:19 INFO DAGScheduler: running: Set(ResultStage 73)
22/02/07 23:47:19 INFO DAGScheduler: waiting: Set(ResultStage 99)
22/02/07 23:47:19 INFO DAGScheduler: failed: Set()
22/02/07 23:47:19 INFO DAGScheduler: Submitting ResultStage 99 (MapPartitionsRDD[216] at distinct at SparkRejectUpdateStrategy.java:52), which has no missing parents
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_84 stored as values in memory (estimated size 6.5 KiB, free 982.8 MiB)
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_84_piece0 stored as bytes in memory (estimated size 3.5 KiB, free 982.7 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Added broadcast_84_piece0 in memory on 192.168.1.37:53037 (size: 3.5 KiB, free: 986.4 MiB)
22/02/07 23:47:19 INFO SparkContext: Created broadcast 84 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 99 (MapPartitionsRDD[216] at distinct at SparkRejectUpdateStrategy.java:52) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:19 INFO TaskSchedulerImpl: Adding task set 99.0 with 1 tasks resource profile 0
22/02/07 23:47:19 INFO TaskSetManager: Starting task 0.0 in stage 99.0 (TID 749) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:19 INFO Executor: Running task 0.0 in stage 99.0 (TID 749)
22/02/07 23:47:19 INFO ShuffleBlockFetcherIterator: Getting 1 (117.0 B) non-empty blocks including 1 (117.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:19 INFO Executor: Finished task 0.0 in stage 99.0 (TID 749). 1249 bytes result sent to driver
22/02/07 23:47:19 INFO TaskSetManager: Finished task 0.0 in stage 99.0 (TID 749) in 4 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:19 INFO TaskSchedulerImpl: Removed TaskSet 99.0, whose tasks have all completed, from pool
22/02/07 23:47:19 INFO DAGScheduler: ResultStage 99 (collect at SparkRejectUpdateStrategy.java:52) finished in 0.006 s
22/02/07 23:47:19 INFO DAGScheduler: Job 61 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:19 INFO TaskSchedulerImpl: Killing all running tasks in stage 99: Stage finished
22/02/07 23:47:19 INFO DAGScheduler: Job 61 finished: collect at SparkRejectUpdateStrategy.java:52, took 0.014181 s
22/02/07 23:47:19 INFO UpsertPartitioner: AvgRecordSize => 1024
22/02/07 23:47:19 INFO BlockManager: Removing RDD 178
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_77_piece0 on 192.168.1.37:53037 in memory (size: 35.1 KiB, free: 986.4 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_75_piece0 on 192.168.1.37:53037 in memory (size: 151.7 KiB, free: 986.6 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_84_piece0 on 192.168.1.37:53037 in memory (size: 3.5 KiB, free: 986.6 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_81_piece0 on 192.168.1.37:53037 in memory (size: 5.6 KiB, free: 986.6 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_73_piece0 on 192.168.1.37:53037 in memory (size: 111.7 KiB, free: 986.7 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_82_piece0 on 192.168.1.37:53037 in memory (size: 3.2 KiB, free: 986.7 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_74_piece0 on 192.168.1.37:53037 in memory (size: 116.6 KiB, free: 986.8 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_76_piece0 on 192.168.1.37:53037 in memory (size: 151.7 KiB, free: 987.0 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_80_piece0 on 192.168.1.37:53037 in memory (size: 35.1 KiB, free: 987.0 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_83_piece0 on 192.168.1.37:53037 in memory (size: 5.5 KiB, free: 987.0 MiB)
22/02/07 23:47:19 INFO BlockManager: Removing RDD 196
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_79_piece0 on 192.168.1.37:53037 in memory (size: 151.6 KiB, free: 987.2 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Removed broadcast_78_piece0 on 192.168.1.37:53037 in memory (size: 35.1 KiB, free: 987.2 MiB)
22/02/07 23:47:19 INFO SparkContext: Starting job: collectAsMap at UpsertPartitioner.java:248
22/02/07 23:47:19 INFO DAGScheduler: Got job 62 (collectAsMap at UpsertPartitioner.java:248) with 1 output partitions
22/02/07 23:47:19 INFO DAGScheduler: Final stage: ResultStage 100 (collectAsMap at UpsertPartitioner.java:248)
22/02/07 23:47:19 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:19 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:19 INFO DAGScheduler: Submitting ResultStage 100 (MapPartitionsRDD[218] at mapToPair at UpsertPartitioner.java:247), which has no missing parents
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_85 stored as values in memory (estimated size 319.5 KiB, free 985.5 MiB)
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_85_piece0 stored as bytes in memory (estimated size 111.7 KiB, free 985.3 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Added broadcast_85_piece0 in memory on 192.168.1.37:53037 (size: 111.7 KiB, free: 987.1 MiB)
22/02/07 23:47:19 INFO SparkContext: Created broadcast 85 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 100 (MapPartitionsRDD[218] at mapToPair at UpsertPartitioner.java:247) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:19 INFO TaskSchedulerImpl: Adding task set 100.0 with 1 tasks resource profile 0
22/02/07 23:47:19 INFO TaskSetManager: Starting task 0.0 in stage 100.0 (TID 750) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4338 bytes) taskResourceAssignments Map()
22/02/07 23:47:19 INFO Executor: Running task 0.0 in stage 100.0 (TID 750)
22/02/07 23:47:19 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:19 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:19 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:19 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:19 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:19 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:19 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=28, NumFileGroups=1, FileGroupsCreationTime=6, StoreTimeTaken=0
22/02/07 23:47:19 INFO Executor: Finished task 0.0 in stage 100.0 (TID 750). 829 bytes result sent to driver
22/02/07 23:47:19 INFO TaskSetManager: Finished task 0.0 in stage 100.0 (TID 750) in 27 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:19 INFO TaskSchedulerImpl: Removed TaskSet 100.0, whose tasks have all completed, from pool
22/02/07 23:47:19 INFO DAGScheduler: ResultStage 100 (collectAsMap at UpsertPartitioner.java:248) finished in 0.078 s
22/02/07 23:47:19 INFO DAGScheduler: Job 62 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:19 INFO TaskSchedulerImpl: Killing all running tasks in stage 100: Stage finished
22/02/07 23:47:19 INFO DAGScheduler: Job 62 finished: collectAsMap at UpsertPartitioner.java:248, took 0.079156 s
22/02/07 23:47:19 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:19 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:19 INFO UpsertPartitioner: Total Buckets :1, buckets info => {0=BucketInfo {bucketType=UPDATE, fileIdPrefix=files-0000, partitionPath=files}},
Partition to insert buckets => {},
UpdateLocations mapped to buckets =>{files-0000=0}
22/02/07 23:47:19 INFO BaseSparkCommitActionExecutor: no validators configured.
22/02/07 23:47:19 INFO BaseCommitActionExecutor: Auto commit enabled: Committing 20220207234319263
22/02/07 23:47:19 INFO SparkContext: Starting job: collect at BaseSparkCommitActionExecutor.java:286
22/02/07 23:47:19 INFO DAGScheduler: Registering RDD 219 (mapToPair at BaseSparkCommitActionExecutor.java:233) as input to shuffle 30
22/02/07 23:47:19 INFO DAGScheduler: Got job 63 (collect at BaseSparkCommitActionExecutor.java:286) with 1 output partitions
22/02/07 23:47:19 INFO DAGScheduler: Final stage: ResultStage 102 (collect at BaseSparkCommitActionExecutor.java:286)
22/02/07 23:47:19 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 101)
22/02/07 23:47:19 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 101)
22/02/07 23:47:19 INFO DAGScheduler: Submitting ShuffleMapStage 101 (MapPartitionsRDD[219] at mapToPair at BaseSparkCommitActionExecutor.java:233), which has no missing parents
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_86 stored as values in memory (estimated size 327.4 KiB, free 985.0 MiB)
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_86_piece0 stored as bytes in memory (estimated size 116.7 KiB, free 984.9 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Added broadcast_86_piece0 in memory on 192.168.1.37:53037 (size: 116.7 KiB, free: 987.0 MiB)
22/02/07 23:47:19 INFO SparkContext: Created broadcast 86 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:19 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 101 (MapPartitionsRDD[219] at mapToPair at BaseSparkCommitActionExecutor.java:233) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:19 INFO TaskSchedulerImpl: Adding task set 101.0 with 1 tasks resource profile 0
22/02/07 23:47:19 INFO TaskSetManager: Starting task 0.0 in stage 101.0 (TID 751) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 6514 bytes) taskResourceAssignments Map()
22/02/07 23:47:19 INFO Executor: Running task 0.0 in stage 101.0 (TID 751)
22/02/07 23:47:19 INFO BlockManager: Found block rdd_208_0 locally
22/02/07 23:47:19 INFO Executor: Finished task 0.0 in stage 101.0 (TID 751). 1043 bytes result sent to driver
22/02/07 23:47:19 INFO TaskSetManager: Finished task 0.0 in stage 101.0 (TID 751) in 20 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:19 INFO TaskSchedulerImpl: Removed TaskSet 101.0, whose tasks have all completed, from pool
22/02/07 23:47:19 INFO DAGScheduler: ShuffleMapStage 101 (mapToPair at BaseSparkCommitActionExecutor.java:233) finished in 0.072 s
22/02/07 23:47:19 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:19 INFO DAGScheduler: running: Set(ResultStage 73)
22/02/07 23:47:19 INFO DAGScheduler: waiting: Set(ResultStage 102)
22/02/07 23:47:19 INFO DAGScheduler: failed: Set()
22/02/07 23:47:19 INFO DAGScheduler: Submitting ResultStage 102 (MapPartitionsRDD[224] at map at BaseSparkCommitActionExecutor.java:286), which has no missing parents
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_87 stored as values in memory (estimated size 432.4 KiB, free 984.5 MiB)
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_87_piece0 stored as bytes in memory (estimated size 151.6 KiB, free 984.3 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Added broadcast_87_piece0 in memory on 192.168.1.37:53037 (size: 151.6 KiB, free: 986.8 MiB)
22/02/07 23:47:19 INFO SparkContext: Created broadcast 87 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 102 (MapPartitionsRDD[224] at map at BaseSparkCommitActionExecutor.java:286) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:19 INFO TaskSchedulerImpl: Adding task set 102.0 with 1 tasks resource profile 0
22/02/07 23:47:19 INFO TaskSetManager: Starting task 0.0 in stage 102.0 (TID 752) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:19 INFO Executor: Running task 0.0 in stage 102.0 (TID 752)
22/02/07 23:47:19 INFO ShuffleBlockFetcherIterator: Getting 1 (1156.0 B) non-empty blocks including 1 (1156.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:19 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220207234319263 for file files-0000
22/02/07 23:47:19 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:19 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:19 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:19 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:19 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:19 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:19 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=28, NumFileGroups=1, FileGroupsCreationTime=5, StoreTimeTaken=0
22/02/07 23:47:19 INFO DirectWriteMarkers: Creating Marker Path=/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207234319263/files/files-0000_0-102-752_00000000000000.hfile.marker.APPEND
22/02/07 23:47:19 INFO DirectWriteMarkers: [direct] Created marker file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207234319263/files/files-0000_0-102-752_00000000000000.hfile.marker.APPEND in 16 ms
22/02/07 23:47:19 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer
22/02/07 23:47:19 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.26_0-88-733
22/02/07 23:47:19 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.27_0-102-752', fileLen=-1}
22/02/07 23:47:19 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:47:19 INFO CodecPool: Got brand-new compressor [.gz]
22/02/07 23:47:19 INFO CodecPool: Got brand-new compressor [.gz]
22/02/07 23:47:19 INFO HoodieAppendHandle: AppendHandle for partitionPath files filePath files/.files-0000_00000000000000.log.27_0-102-752, took 48 ms.
22/02/07 23:47:19 INFO MemoryStore: Block rdd_223_0 stored as values in memory (estimated size 1542.0 B, free 984.3 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Added rdd_223_0 in memory on 192.168.1.37:53037 (size: 1542.0 B, free: 986.8 MiB)
22/02/07 23:47:19 INFO Executor: Finished task 0.0 in stage 102.0 (TID 752). 2696 bytes result sent to driver
22/02/07 23:47:19 INFO TaskSetManager: Finished task 0.0 in stage 102.0 (TID 752) in 72 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:19 INFO TaskSchedulerImpl: Removed TaskSet 102.0, whose tasks have all completed, from pool
22/02/07 23:47:19 INFO DAGScheduler: ResultStage 102 (collect at BaseSparkCommitActionExecutor.java:286) finished in 0.139 s
22/02/07 23:47:19 INFO DAGScheduler: Job 63 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:19 INFO TaskSchedulerImpl: Killing all running tasks in stage 102: Stage finished
22/02/07 23:47:19 INFO DAGScheduler: Job 63 finished: collect at BaseSparkCommitActionExecutor.java:286, took 0.213943 s
22/02/07 23:47:19 INFO CommitUtils: Creating  metadata for UPSERT_PREPPED numWriteStats:1numReplaceFileIds:0
22/02/07 23:47:19 INFO SparkContext: Starting job: collect at BaseSparkCommitActionExecutor.java:294
22/02/07 23:47:19 INFO DAGScheduler: Got job 64 (collect at BaseSparkCommitActionExecutor.java:294) with 1 output partitions
22/02/07 23:47:19 INFO DAGScheduler: Final stage: ResultStage 104 (collect at BaseSparkCommitActionExecutor.java:294)
22/02/07 23:47:19 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 103)
22/02/07 23:47:19 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:19 INFO DAGScheduler: Submitting ResultStage 104 (MapPartitionsRDD[225] at map at BaseSparkCommitActionExecutor.java:294), which has no missing parents
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_88 stored as values in memory (estimated size 432.4 KiB, free 983.9 MiB)
22/02/07 23:47:19 INFO MemoryStore: Block broadcast_88_piece0 stored as bytes in memory (estimated size 151.6 KiB, free 983.8 MiB)
22/02/07 23:47:19 INFO BlockManagerInfo: Added broadcast_88_piece0 in memory on 192.168.1.37:53037 (size: 151.6 KiB, free: 986.7 MiB)
22/02/07 23:47:19 INFO SparkContext: Created broadcast 88 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 104 (MapPartitionsRDD[225] at map at BaseSparkCommitActionExecutor.java:294) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:19 INFO TaskSchedulerImpl: Adding task set 104.0 with 1 tasks resource profile 0
22/02/07 23:47:19 INFO TaskSetManager: Starting task 0.0 in stage 104.0 (TID 753) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:19 INFO Executor: Running task 0.0 in stage 104.0 (TID 753)
22/02/07 23:47:19 INFO BlockManager: Found block rdd_223_0 locally
22/02/07 23:47:19 INFO Executor: Finished task 0.0 in stage 104.0 (TID 753). 2309 bytes result sent to driver
22/02/07 23:47:19 INFO TaskSetManager: Finished task 0.0 in stage 104.0 (TID 753) in 19 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:19 INFO TaskSchedulerImpl: Removed TaskSet 104.0, whose tasks have all completed, from pool
22/02/07 23:47:19 INFO DAGScheduler: ResultStage 104 (collect at BaseSparkCommitActionExecutor.java:294) finished in 0.084 s
22/02/07 23:47:19 INFO DAGScheduler: Job 64 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:19 INFO TaskSchedulerImpl: Killing all running tasks in stage 104: Stage finished
22/02/07 23:47:19 INFO DAGScheduler: Job 64 finished: collect at BaseSparkCommitActionExecutor.java:294, took 0.085288 s
22/02/07 23:47:19 INFO BaseSparkCommitActionExecutor: Committing 20220207234319263, action Type deltacommit, operation Type UPSERT_PREPPED
22/02/07 23:47:20 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/07 23:47:20 INFO DAGScheduler: Got job 65 (collect at HoodieSparkEngineContext.java:134) with 1 output partitions
22/02/07 23:47:20 INFO DAGScheduler: Final stage: ResultStage 105 (collect at HoodieSparkEngineContext.java:134)
22/02/07 23:47:20 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:20 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:20 INFO DAGScheduler: Submitting ResultStage 105 (MapPartitionsRDD[227] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_89 stored as values in memory (estimated size 99.2 KiB, free 983.7 MiB)
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_89_piece0 stored as bytes in memory (estimated size 35.1 KiB, free 983.6 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Added broadcast_89_piece0 in memory on 192.168.1.37:53037 (size: 35.1 KiB, free: 986.6 MiB)
22/02/07 23:47:20 INFO SparkContext: Created broadcast 89 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 105 (MapPartitionsRDD[227] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:20 INFO TaskSchedulerImpl: Adding task set 105.0 with 1 tasks resource profile 0
22/02/07 23:47:20 INFO TaskSetManager: Starting task 0.0 in stage 105.0 (TID 754) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4456 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 0.0 in stage 105.0 (TID 754)
22/02/07 23:47:20 INFO Executor: Finished task 0.0 in stage 105.0 (TID 754). 796 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Finished task 0.0 in stage 105.0 (TID 754) in 13 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:20 INFO TaskSchedulerImpl: Removed TaskSet 105.0, whose tasks have all completed, from pool
22/02/07 23:47:20 INFO DAGScheduler: ResultStage 105 (collect at HoodieSparkEngineContext.java:134) finished in 0.030 s
22/02/07 23:47:20 INFO DAGScheduler: Job 65 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:20 INFO TaskSchedulerImpl: Killing all running tasks in stage 105: Stage finished
22/02/07 23:47:20 INFO DAGScheduler: Job 65 finished: collect at HoodieSparkEngineContext.java:134, took 0.030579 s
22/02/07 23:47:20 INFO HoodieActiveTimeline: Marking instant complete [==>20220207234319263__deltacommit__INFLIGHT]
22/02/07 23:47:20 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234319263.deltacommit.inflight
22/02/07 23:47:20 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234319263.deltacommit
22/02/07 23:47:20 INFO HoodieActiveTimeline: Completed [==>20220207234319263__deltacommit__INFLIGHT]
22/02/07 23:47:20 INFO BaseSparkCommitActionExecutor: Committed 20220207234319263
22/02/07 23:47:20 INFO SparkContext: Starting job: collectAsMap at HoodieSparkEngineContext.java:148
22/02/07 23:47:20 INFO DAGScheduler: Got job 66 (collectAsMap at HoodieSparkEngineContext.java:148) with 1 output partitions
22/02/07 23:47:20 INFO DAGScheduler: Final stage: ResultStage 106 (collectAsMap at HoodieSparkEngineContext.java:148)
22/02/07 23:47:20 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:20 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:20 INFO DAGScheduler: Submitting ResultStage 106 (MapPartitionsRDD[229] at mapToPair at HoodieSparkEngineContext.java:145), which has no missing parents
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_90 stored as values in memory (estimated size 99.4 KiB, free 983.5 MiB)
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_90_piece0 stored as bytes in memory (estimated size 35.1 KiB, free 983.5 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Added broadcast_90_piece0 in memory on 192.168.1.37:53037 (size: 35.1 KiB, free: 986.6 MiB)
22/02/07 23:47:20 INFO SparkContext: Created broadcast 90 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 106 (MapPartitionsRDD[229] at mapToPair at HoodieSparkEngineContext.java:145) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:20 INFO TaskSchedulerImpl: Adding task set 106.0 with 1 tasks resource profile 0
22/02/07 23:47:20 INFO TaskSetManager: Starting task 0.0 in stage 106.0 (TID 755) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4456 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 0.0 in stage 106.0 (TID 755)
22/02/07 23:47:20 INFO Executor: Finished task 0.0 in stage 106.0 (TID 755). 883 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Finished task 0.0 in stage 106.0 (TID 755) in 6 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:20 INFO TaskSchedulerImpl: Removed TaskSet 106.0, whose tasks have all completed, from pool
22/02/07 23:47:20 INFO DAGScheduler: ResultStage 106 (collectAsMap at HoodieSparkEngineContext.java:148) finished in 0.023 s
22/02/07 23:47:20 INFO DAGScheduler: Job 66 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:20 INFO TaskSchedulerImpl: Killing all running tasks in stage 106: Stage finished
22/02/07 23:47:20 INFO DAGScheduler: Job 66 finished: collectAsMap at HoodieSparkEngineContext.java:148, took 0.024868 s
22/02/07 23:47:20 INFO FSUtils: Removed directory at /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207234319263
22/02/07 23:47:20 INFO SparkContext: Starting job: collect at SparkHoodieBackedTableMetadataWriter.java:154
22/02/07 23:47:20 INFO DAGScheduler: Got job 67 (collect at SparkHoodieBackedTableMetadataWriter.java:154) with 1 output partitions
22/02/07 23:47:20 INFO DAGScheduler: Final stage: ResultStage 108 (collect at SparkHoodieBackedTableMetadataWriter.java:154)
22/02/07 23:47:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 107)
22/02/07 23:47:20 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:20 INFO DAGScheduler: Submitting ResultStage 108 (MapPartitionsRDD[223] at flatMap at BaseSparkCommitActionExecutor.java:181), which has no missing parents
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_91 stored as values in memory (estimated size 432.0 KiB, free 983.1 MiB)
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_91_piece0 stored as bytes in memory (estimated size 151.6 KiB, free 982.9 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Added broadcast_91_piece0 in memory on 192.168.1.37:53037 (size: 151.6 KiB, free: 986.4 MiB)
22/02/07 23:47:20 INFO SparkContext: Created broadcast 91 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 108 (MapPartitionsRDD[223] at flatMap at BaseSparkCommitActionExecutor.java:181) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:20 INFO TaskSchedulerImpl: Adding task set 108.0 with 1 tasks resource profile 0
22/02/07 23:47:20 INFO TaskSetManager: Starting task 0.0 in stage 108.0 (TID 756) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 0.0 in stage 108.0 (TID 756)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_223_0 locally
22/02/07 23:47:20 INFO Executor: Finished task 0.0 in stage 108.0 (TID 756). 2389 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Finished task 0.0 in stage 108.0 (TID 756) in 19 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:20 INFO TaskSchedulerImpl: Removed TaskSet 108.0, whose tasks have all completed, from pool
22/02/07 23:47:20 INFO DAGScheduler: ResultStage 108 (collect at SparkHoodieBackedTableMetadataWriter.java:154) finished in 0.083 s
22/02/07 23:47:20 INFO DAGScheduler: Job 67 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:20 INFO TaskSchedulerImpl: Killing all running tasks in stage 108: Stage finished
22/02/07 23:47:20 INFO DAGScheduler: Job 67 finished: collect at SparkHoodieBackedTableMetadataWriter.java:154, took 0.083618 s
22/02/07 23:47:20 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:20 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234123810.clean.inflight
22/02/07 23:47:20 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/1/25 fileID ad0f8936-7fd4-4fab-a973-c83346480fcd-0, took 7656 ms.
22/02/07 23:47:20 INFO MemoryStore: Block rdd_166_18 stored as values in memory (estimated size 395.0 B, free 982.9 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Added rdd_166_18 in memory on 192.168.1.37:53037 (size: 395.0 B, free: 986.4 MiB)
22/02/07 23:47:20 INFO Executor: Finished task 18.0 in stage 73.0 (TID 704). 1649 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Finished task 18.0 in stage 73.0 (TID 704) in 7696 ms on 192.168.1.37 (executor driver) (20/20)
22/02/07 23:47:20 INFO TaskSchedulerImpl: Removed TaskSet 73.0, whose tasks have all completed, from pool
22/02/07 23:47:20 INFO DAGScheduler: ResultStage 73 (sum at DeltaSync.java:561) finished in 59.731 s
22/02/07 23:47:20 INFO DAGScheduler: Job 45 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:20 INFO TaskSchedulerImpl: Killing all running tasks in stage 73: Stage finished
22/02/07 23:47:20 INFO DAGScheduler: Job 45 finished: sum at DeltaSync.java:561, took 63.054197 s
22/02/07 23:47:20 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234123810.clean
22/02/07 23:47:20 INFO CleanActionExecutor: Marked clean started on 20220207234123810 as complete
22/02/07 23:47:20 INFO TransactionManager: Transaction ending with transaction owner Optional.empty
22/02/07 23:47:20 INFO InProcessLockProvider: Thread pool-26-thread-1 RELEASING in-process lock.
22/02/07 23:47:20 INFO InProcessLockProvider: Thread pool-26-thread-1 RELEASED in-process lock.
22/02/07 23:47:20 INFO TransactionManager: Transaction ended with transaction owner Optional.empty
22/02/07 23:47:20 INFO CleanActionExecutor: Finishing previously unfinished cleaner instant=[==>20220207234228524__clean__REQUESTED]
22/02/07 23:47:20 INFO SparkContext: Starting job: sum at DeltaSync.java:562
22/02/07 23:47:20 INFO DAGScheduler: Got job 68 (sum at DeltaSync.java:562) with 20 output partitions
22/02/07 23:47:20 INFO DAGScheduler: Final stage: ResultStage 114 (sum at DeltaSync.java:562)
22/02/07 23:47:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 113)
22/02/07 23:47:20 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234228524.clean.requested
22/02/07 23:47:20 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:20 INFO DAGScheduler: Submitting ResultStage 114 (MapPartitionsRDD[230] at mapToDouble at DeltaSync.java:562), which has no missing parents
22/02/07 23:47:20 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234228524.clean.inflight
22/02/07 23:47:20 INFO CleanActionExecutor: Using cleanerParallelism: 10
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_92 stored as values in memory (estimated size 667.6 KiB, free 982.3 MiB)
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_92_piece0 stored as bytes in memory (estimated size 237.2 KiB, free 982.1 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Added broadcast_92_piece0 in memory on 192.168.1.37:53037 (size: 237.2 KiB, free: 986.2 MiB)
22/02/07 23:47:20 INFO SparkContext: Created broadcast 92 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:20 INFO DAGScheduler: Submitting 20 missing tasks from ResultStage 114 (MapPartitionsRDD[230] at mapToDouble at DeltaSync.java:562) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/07 23:47:20 INFO TaskSchedulerImpl: Adding task set 114.0 with 20 tasks resource profile 0
22/02/07 23:47:20 INFO TaskSetManager: Starting task 0.0 in stage 114.0 (TID 757) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Starting task 1.0 in stage 114.0 (TID 758) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Starting task 2.0 in stage 114.0 (TID 759) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 0.0 in stage 114.0 (TID 757)
22/02/07 23:47:20 INFO Executor: Running task 1.0 in stage 114.0 (TID 758)
22/02/07 23:47:20 INFO Executor: Running task 2.0 in stage 114.0 (TID 759)
22/02/07 23:47:20 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:122
22/02/07 23:47:20 INFO DAGScheduler: Registering RDD 233 (mapPartitionsToPair at HoodieSparkEngineContext.java:116) as input to shuffle 31
22/02/07 23:47:20 INFO DAGScheduler: Got job 69 (collect at HoodieSparkEngineContext.java:122) with 10 output partitions
22/02/07 23:47:20 INFO DAGScheduler: Final stage: ResultStage 116 (collect at HoodieSparkEngineContext.java:122)
22/02/07 23:47:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 115)
22/02/07 23:47:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 115)
22/02/07 23:47:20 INFO DAGScheduler: Submitting ShuffleMapStage 115 (MapPartitionsRDD[233] at mapPartitionsToPair at HoodieSparkEngineContext.java:116), which has no missing parents
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_2 locally
22/02/07 23:47:20 INFO Executor: Finished task 2.0 in stage 114.0 (TID 759). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_1 locally
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_0 locally
22/02/07 23:47:20 INFO TaskSetManager: Starting task 3.0 in stage 114.0 (TID 760) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 2.0 in stage 114.0 (TID 759) in 31 ms on 192.168.1.37 (executor driver) (1/20)
22/02/07 23:47:20 INFO Executor: Running task 3.0 in stage 114.0 (TID 760)
22/02/07 23:47:20 INFO Executor: Finished task 1.0 in stage 114.0 (TID 758). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO Executor: Finished task 0.0 in stage 114.0 (TID 757). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 4.0 in stage 114.0 (TID 761) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Starting task 5.0 in stage 114.0 (TID 762) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 4.0 in stage 114.0 (TID 761)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 0.0 in stage 114.0 (TID 757) in 31 ms on 192.168.1.37 (executor driver) (2/20)
22/02/07 23:47:20 INFO Executor: Running task 5.0 in stage 114.0 (TID 762)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 1.0 in stage 114.0 (TID 758) in 31 ms on 192.168.1.37 (executor driver) (3/20)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_3 locally
22/02/07 23:47:20 INFO Executor: Finished task 3.0 in stage 114.0 (TID 760). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 6.0 in stage 114.0 (TID 763) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 6.0 in stage 114.0 (TID 763)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 3.0 in stage 114.0 (TID 760) in 28 ms on 192.168.1.37 (executor driver) (4/20)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_4 locally
22/02/07 23:47:20 INFO Executor: Finished task 4.0 in stage 114.0 (TID 761). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 7.0 in stage 114.0 (TID 764) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 4.0 in stage 114.0 (TID 761) in 29 ms on 192.168.1.37 (executor driver) (5/20)
22/02/07 23:47:20 INFO Executor: Running task 7.0 in stage 114.0 (TID 764)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_5 locally
22/02/07 23:47:20 INFO Executor: Finished task 5.0 in stage 114.0 (TID 762). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 8.0 in stage 114.0 (TID 765) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 5.0 in stage 114.0 (TID 762) in 30 ms on 192.168.1.37 (executor driver) (6/20)
22/02/07 23:47:20 INFO Executor: Running task 8.0 in stage 114.0 (TID 765)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_6 locally
22/02/07 23:47:20 INFO Executor: Finished task 6.0 in stage 114.0 (TID 763). 1391 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 9.0 in stage 114.0 (TID 766) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 6.0 in stage 114.0 (TID 763) in 87 ms on 192.168.1.37 (executor driver) (7/20)
22/02/07 23:47:20 INFO Executor: Running task 9.0 in stage 114.0 (TID 766)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_7 locally
22/02/07 23:47:20 INFO Executor: Finished task 7.0 in stage 114.0 (TID 764). 1391 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 10.0 in stage 114.0 (TID 767) (192.168.1.37, executor driver, partition 10, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 10.0 in stage 114.0 (TID 767)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 7.0 in stage 114.0 (TID 764) in 87 ms on 192.168.1.37 (executor driver) (8/20)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_8 locally
22/02/07 23:47:20 INFO Executor: Finished task 8.0 in stage 114.0 (TID 765). 1391 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 11.0 in stage 114.0 (TID 768) (192.168.1.37, executor driver, partition 11, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 8.0 in stage 114.0 (TID 765) in 86 ms on 192.168.1.37 (executor driver) (9/20)
22/02/07 23:47:20 INFO Executor: Running task 11.0 in stage 114.0 (TID 768)
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_93 stored as values in memory (estimated size 612.1 KiB, free 981.5 MiB)
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_93_piece0 stored as bytes in memory (estimated size 214.0 KiB, free 981.2 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Added broadcast_93_piece0 in memory on 192.168.1.37:53037 (size: 214.0 KiB, free: 986.0 MiB)
22/02/07 23:47:20 INFO SparkContext: Created broadcast 93 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:20 INFO DAGScheduler: Submitting 10 missing tasks from ShuffleMapStage 115 (MapPartitionsRDD[233] at mapPartitionsToPair at HoodieSparkEngineContext.java:116) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/07 23:47:20 INFO TaskSchedulerImpl: Adding task set 115.0 with 10 tasks resource profile 0
22/02/07 23:47:20 INFO BlockManagerInfo: Removed broadcast_90_piece0 on 192.168.1.37:53037 in memory (size: 35.1 KiB, free: 986.0 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Removed broadcast_91_piece0 on 192.168.1.37:53037 in memory (size: 151.6 KiB, free: 986.2 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Removed broadcast_89_piece0 on 192.168.1.37:53037 in memory (size: 35.1 KiB, free: 986.2 MiB)
22/02/07 23:47:20 INFO BlockManager: Removing RDD 208
22/02/07 23:47:20 INFO BlockManagerInfo: Removed broadcast_85_piece0 on 192.168.1.37:53037 in memory (size: 111.7 KiB, free: 986.3 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Removed broadcast_87_piece0 on 192.168.1.37:53037 in memory (size: 151.6 KiB, free: 986.5 MiB)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_11 locally
22/02/07 23:47:20 INFO Executor: Finished task 11.0 in stage 114.0 (TID 768). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 12.0 in stage 114.0 (TID 769) (192.168.1.37, executor driver, partition 12, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 12.0 in stage 114.0 (TID 769)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 11.0 in stage 114.0 (TID 768) in 37 ms on 192.168.1.37 (executor driver) (10/20)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_10 locally
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_9 locally
22/02/07 23:47:20 INFO Executor: Finished task 10.0 in stage 114.0 (TID 767). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO Executor: Finished task 9.0 in stage 114.0 (TID 766). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 13.0 in stage 114.0 (TID 770) (192.168.1.37, executor driver, partition 13, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 13.0 in stage 114.0 (TID 770)
22/02/07 23:47:20 INFO TaskSetManager: Starting task 14.0 in stage 114.0 (TID 771) (192.168.1.37, executor driver, partition 14, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO BlockManager: Removing RDD 223
22/02/07 23:47:20 INFO Executor: Running task 14.0 in stage 114.0 (TID 771)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 10.0 in stage 114.0 (TID 767) in 38 ms on 192.168.1.37 (executor driver) (11/20)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 9.0 in stage 114.0 (TID 766) in 40 ms on 192.168.1.37 (executor driver) (12/20)
22/02/07 23:47:20 INFO BlockManagerInfo: Removed broadcast_88_piece0 on 192.168.1.37:53037 in memory (size: 151.6 KiB, free: 986.6 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Removed broadcast_86_piece0 on 192.168.1.37:53037 in memory (size: 116.7 KiB, free: 986.7 MiB)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_13 locally
22/02/07 23:47:20 INFO Executor: Finished task 13.0 in stage 114.0 (TID 770). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 15.0 in stage 114.0 (TID 772) (192.168.1.37, executor driver, partition 15, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 15.0 in stage 114.0 (TID 772)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 13.0 in stage 114.0 (TID 770) in 29 ms on 192.168.1.37 (executor driver) (13/20)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_12 locally
22/02/07 23:47:20 INFO Executor: Finished task 12.0 in stage 114.0 (TID 769). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 16.0 in stage 114.0 (TID 773) (192.168.1.37, executor driver, partition 16, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 12.0 in stage 114.0 (TID 769) in 31 ms on 192.168.1.37 (executor driver) (14/20)
22/02/07 23:47:20 INFO Executor: Running task 16.0 in stage 114.0 (TID 773)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_14 locally
22/02/07 23:47:20 INFO Executor: Finished task 14.0 in stage 114.0 (TID 771). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 17.0 in stage 114.0 (TID 774) (192.168.1.37, executor driver, partition 17, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 14.0 in stage 114.0 (TID 771) in 31 ms on 192.168.1.37 (executor driver) (15/20)
22/02/07 23:47:20 INFO Executor: Running task 17.0 in stage 114.0 (TID 774)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_15 locally
22/02/07 23:47:20 INFO Executor: Finished task 15.0 in stage 114.0 (TID 772). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 18.0 in stage 114.0 (TID 775) (192.168.1.37, executor driver, partition 18, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 15.0 in stage 114.0 (TID 772) in 29 ms on 192.168.1.37 (executor driver) (16/20)
22/02/07 23:47:20 INFO Executor: Running task 18.0 in stage 114.0 (TID 775)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_16 locally
22/02/07 23:47:20 INFO Executor: Finished task 16.0 in stage 114.0 (TID 773). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 19.0 in stage 114.0 (TID 776) (192.168.1.37, executor driver, partition 19, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 16.0 in stage 114.0 (TID 773) in 29 ms on 192.168.1.37 (executor driver) (17/20)
22/02/07 23:47:20 INFO Executor: Running task 19.0 in stage 114.0 (TID 776)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_17 locally
22/02/07 23:47:20 INFO Executor: Finished task 17.0 in stage 114.0 (TID 774). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 0.0 in stage 115.0 (TID 777) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 17.0 in stage 114.0 (TID 774) in 29 ms on 192.168.1.37 (executor driver) (18/20)
22/02/07 23:47:20 INFO Executor: Running task 0.0 in stage 115.0 (TID 777)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_18 locally
22/02/07 23:47:20 INFO Executor: Finished task 18.0 in stage 114.0 (TID 775). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 1.0 in stage 115.0 (TID 778) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 18.0 in stage 114.0 (TID 775) in 29 ms on 192.168.1.37 (executor driver) (19/20)
22/02/07 23:47:20 INFO Executor: Running task 1.0 in stage 115.0 (TID 778)
22/02/07 23:47:20 INFO BlockManager: Found block rdd_166_19 locally
22/02/07 23:47:20 INFO Executor: Finished task 19.0 in stage 114.0 (TID 776). 1348 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 2.0 in stage 115.0 (TID 779) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 19.0 in stage 114.0 (TID 776) in 29 ms on 192.168.1.37 (executor driver) (20/20)
22/02/07 23:47:20 INFO Executor: Running task 2.0 in stage 115.0 (TID 779)
22/02/07 23:47:20 INFO TaskSchedulerImpl: Removed TaskSet 114.0, whose tasks have all completed, from pool
22/02/07 23:47:20 INFO DAGScheduler: ResultStage 114 (sum at DeltaSync.java:562) finished in 0.412 s
22/02/07 23:47:20 INFO DAGScheduler: Job 68 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:20 INFO TaskSchedulerImpl: Killing all running tasks in stage 114: Stage finished
22/02/07 23:47:20 INFO DAGScheduler: Job 68 finished: sum at DeltaSync.java:562, took 0.414468 s
22/02/07 23:47:20 INFO Executor: Finished task 0.0 in stage 115.0 (TID 777). 1095 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 3.0 in stage 115.0 (TID 780) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 0.0 in stage 115.0 (TID 777) in 30 ms on 192.168.1.37 (executor driver) (1/10)
22/02/07 23:47:20 INFO Executor: Running task 3.0 in stage 115.0 (TID 780)
22/02/07 23:47:20 INFO Executor: Finished task 1.0 in stage 115.0 (TID 778). 1095 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 4.0 in stage 115.0 (TID 781) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4739 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 1.0 in stage 115.0 (TID 778) in 32 ms on 192.168.1.37 (executor driver) (2/10)
22/02/07 23:47:20 INFO Executor: Running task 4.0 in stage 115.0 (TID 781)
22/02/07 23:47:20 INFO Executor: Finished task 2.0 in stage 115.0 (TID 779). 1095 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 5.0 in stage 115.0 (TID 782) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 2.0 in stage 115.0 (TID 779) in 31 ms on 192.168.1.37 (executor driver) (3/10)
22/02/07 23:47:20 INFO Executor: Running task 5.0 in stage 115.0 (TID 782)
22/02/07 23:47:20 INFO Executor: Finished task 3.0 in stage 115.0 (TID 780). 1095 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 6.0 in stage 115.0 (TID 783) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 6.0 in stage 115.0 (TID 783)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 3.0 in stage 115.0 (TID 780) in 31 ms on 192.168.1.37 (executor driver) (4/10)
22/02/07 23:47:20 INFO Executor: Finished task 5.0 in stage 115.0 (TID 782). 1095 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 7.0 in stage 115.0 (TID 784) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4739 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 5.0 in stage 115.0 (TID 782) in 29 ms on 192.168.1.37 (executor driver) (5/10)
22/02/07 23:47:20 INFO Executor: Running task 7.0 in stage 115.0 (TID 784)
22/02/07 23:47:20 INFO Executor: Finished task 6.0 in stage 115.0 (TID 783). 1095 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 8.0 in stage 115.0 (TID 785) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 8.0 in stage 115.0 (TID 785)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 6.0 in stage 115.0 (TID 783) in 29 ms on 192.168.1.37 (executor driver) (6/10)
22/02/07 23:47:20 INFO Executor: Finished task 4.0 in stage 115.0 (TID 781). 1095 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 9.0 in stage 115.0 (TID 786) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 4.0 in stage 115.0 (TID 781) in 33 ms on 192.168.1.37 (executor driver) (7/10)
22/02/07 23:47:20 INFO Executor: Running task 9.0 in stage 115.0 (TID 786)
22/02/07 23:47:20 INFO Executor: Finished task 7.0 in stage 115.0 (TID 784). 1095 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Finished task 7.0 in stage 115.0 (TID 784) in 30 ms on 192.168.1.37 (executor driver) (8/10)
22/02/07 23:47:20 INFO Executor: Finished task 8.0 in stage 115.0 (TID 785). 1095 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Finished task 8.0 in stage 115.0 (TID 785) in 30 ms on 192.168.1.37 (executor driver) (9/10)
22/02/07 23:47:20 INFO Executor: Finished task 9.0 in stage 115.0 (TID 786). 1095 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Finished task 9.0 in stage 115.0 (TID 786) in 29 ms on 192.168.1.37 (executor driver) (10/10)
22/02/07 23:47:20 INFO TaskSchedulerImpl: Removed TaskSet 115.0, whose tasks have all completed, from pool
22/02/07 23:47:20 INFO DAGScheduler: ShuffleMapStage 115 (mapPartitionsToPair at HoodieSparkEngineContext.java:116) finished in 0.353 s
22/02/07 23:47:20 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:20 INFO DAGScheduler: running: Set()
22/02/07 23:47:20 INFO DAGScheduler: waiting: Set(ResultStage 116)
22/02/07 23:47:20 INFO DAGScheduler: failed: Set()
22/02/07 23:47:20 INFO DAGScheduler: Submitting ResultStage 116 (MapPartitionsRDD[235] at map at HoodieSparkEngineContext.java:121), which has no missing parents
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_94 stored as values in memory (estimated size 7.7 KiB, free 984.1 MiB)
22/02/07 23:47:20 INFO MemoryStore: Block broadcast_94_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 984.1 MiB)
22/02/07 23:47:20 INFO BlockManagerInfo: Added broadcast_94_piece0 in memory on 192.168.1.37:53037 (size: 4.0 KiB, free: 986.7 MiB)
22/02/07 23:47:20 INFO SparkContext: Created broadcast 94 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:20 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 116 (MapPartitionsRDD[235] at map at HoodieSparkEngineContext.java:121) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/07 23:47:20 INFO TaskSchedulerImpl: Adding task set 116.0 with 10 tasks resource profile 0
22/02/07 23:47:20 INFO TaskSetManager: Starting task 1.0 in stage 116.0 (TID 787) (192.168.1.37, executor driver, partition 1, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Starting task 2.0 in stage 116.0 (TID 788) (192.168.1.37, executor driver, partition 2, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Starting task 3.0 in stage 116.0 (TID 789) (192.168.1.37, executor driver, partition 3, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 3.0 in stage 116.0 (TID 789)
22/02/07 23:47:20 INFO Executor: Running task 2.0 in stage 116.0 (TID 788)
22/02/07 23:47:20 INFO Executor: Running task 1.0 in stage 116.0 (TID 787)
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:20 INFO Executor: Finished task 2.0 in stage 116.0 (TID 788). 1673 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 4.0 in stage 116.0 (TID 790) (192.168.1.37, executor driver, partition 4, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 4.0 in stage 116.0 (TID 790)
22/02/07 23:47:20 INFO Executor: Finished task 1.0 in stage 116.0 (TID 787). 1673 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 5.0 in stage 116.0 (TID 791) (192.168.1.37, executor driver, partition 5, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 1.0 in stage 116.0 (TID 787) in 5 ms on 192.168.1.37 (executor driver) (1/10)
22/02/07 23:47:20 INFO Executor: Running task 5.0 in stage 116.0 (TID 791)
22/02/07 23:47:20 INFO Executor: Finished task 3.0 in stage 116.0 (TID 789). 1673 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 6.0 in stage 116.0 (TID 792) (192.168.1.37, executor driver, partition 6, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 2.0 in stage 116.0 (TID 788) in 5 ms on 192.168.1.37 (executor driver) (2/10)
22/02/07 23:47:20 INFO Executor: Running task 6.0 in stage 116.0 (TID 792)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 3.0 in stage 116.0 (TID 789) in 5 ms on 192.168.1.37 (executor driver) (3/10)
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Getting 2 (552.0 B) non-empty blocks including 2 (552.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Getting 2 (552.0 B) non-empty blocks including 2 (552.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:20 INFO Executor: Finished task 4.0 in stage 116.0 (TID 790). 1673 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 8.0 in stage 116.0 (TID 793) (192.168.1.37, executor driver, partition 8, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 8.0 in stage 116.0 (TID 793)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 4.0 in stage 116.0 (TID 790) in 4 ms on 192.168.1.37 (executor driver) (4/10)
22/02/07 23:47:20 INFO Executor: Finished task 5.0 in stage 116.0 (TID 791). 2034 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 9.0 in stage 116.0 (TID 794) (192.168.1.37, executor driver, partition 9, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 9.0 in stage 116.0 (TID 794)
22/02/07 23:47:20 INFO Executor: Finished task 6.0 in stage 116.0 (TID 792). 2034 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Finished task 5.0 in stage 116.0 (TID 791) in 4 ms on 192.168.1.37 (executor driver) (5/10)
22/02/07 23:47:20 INFO TaskSetManager: Starting task 0.0 in stage 116.0 (TID 795) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO Executor: Running task 0.0 in stage 116.0 (TID 795)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 6.0 in stage 116.0 (TID 792) in 4 ms on 192.168.1.37 (executor driver) (6/10)
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:20 INFO Executor: Finished task 0.0 in stage 116.0 (TID 795). 1140 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Starting task 7.0 in stage 116.0 (TID 796) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:20 INFO TaskSetManager: Finished task 0.0 in stage 116.0 (TID 795) in 1 ms on 192.168.1.37 (executor driver) (7/10)
22/02/07 23:47:20 INFO Executor: Running task 7.0 in stage 116.0 (TID 796)
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:20 INFO Executor: Finished task 8.0 in stage 116.0 (TID 793). 1673 bytes result sent to driver
22/02/07 23:47:20 INFO Executor: Finished task 7.0 in stage 116.0 (TID 796). 1140 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Finished task 8.0 in stage 116.0 (TID 793) in 3 ms on 192.168.1.37 (executor driver) (8/10)
22/02/07 23:47:20 INFO TaskSetManager: Finished task 7.0 in stage 116.0 (TID 796) in 1 ms on 192.168.1.37 (executor driver) (9/10)
22/02/07 23:47:20 INFO Executor: Finished task 9.0 in stage 116.0 (TID 794). 1673 bytes result sent to driver
22/02/07 23:47:20 INFO TaskSetManager: Finished task 9.0 in stage 116.0 (TID 794) in 4 ms on 192.168.1.37 (executor driver) (10/10)
22/02/07 23:47:20 INFO TaskSchedulerImpl: Removed TaskSet 116.0, whose tasks have all completed, from pool
22/02/07 23:47:20 INFO DAGScheduler: ResultStage 116 (collect at HoodieSparkEngineContext.java:122) finished in 0.014 s
22/02/07 23:47:20 INFO DAGScheduler: Job 69 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:20 INFO TaskSchedulerImpl: Killing all running tasks in stage 116: Stage finished
22/02/07 23:47:20 INFO DAGScheduler: Job 69 finished: collect at HoodieSparkEngineContext.java:122, took 0.369359 s
22/02/07 23:47:20 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:47:20 INFO TransactionManager: Transaction starting for Optional.empty with latest completed transaction instant Optional.empty
22/02/07 23:47:20 INFO InProcessLockProvider: Thread pool-26-thread-1 ACQUIRING in-process lock.
22/02/07 23:47:20 INFO InProcessLockProvider: Thread pool-26-thread-1 ACQUIRED in-process lock.
22/02/07 23:47:20 INFO TransactionManager: Transaction started for Optional.empty with latest completed transaction instant Optional.empty
22/02/07 23:47:20 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:20 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:47:20 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:20 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:20 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:20 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:20 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:20 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:20 INFO HoodieTableMetadataUtil: Loading latest file slices for metadata table partition files
22/02/07 23:47:20 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:20 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:20 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=29, NumFileGroups=1, FileGroupsCreationTime=6, StoreTimeTaken=0
22/02/07 23:47:20 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:20 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:20 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:20 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:20 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:47:20 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:20 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:47:20 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:20 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:20 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:20 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:20 INFO HoodieTableMetadataUtil: Updating at 20220207234319263 from Clean. #partitions_updated=10, #files_deleted=20
22/02/07 23:47:20 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:20 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:20 INFO HoodieTableMetadataUtil: Loading latest file slices for metadata table partition files
22/02/07 23:47:20 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:20 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:20 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=29, NumFileGroups=1, FileGroupsCreationTime=5, StoreTimeTaken=0
22/02/07 23:47:20 INFO BaseHoodieClient: Embedded Timeline Server is disabled. Not starting timeline service
22/02/07 23:47:20 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:20 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:20 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:20 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:20 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:20 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:20 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:20 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:20 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:20 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:20 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:20 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:20 INFO SparkContext: Starting job: collect at SparkRDDWriteClient.java:126
22/02/07 23:47:20 INFO DAGScheduler: Got job 70 (collect at SparkRDDWriteClient.java:126) with 20 output partitions
22/02/07 23:47:20 INFO DAGScheduler: Final stage: ResultStage 122 (collect at SparkRDDWriteClient.java:126)
22/02/07 23:47:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 121)
22/02/07 23:47:20 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:20 INFO DAGScheduler: Submitting ResultStage 122 (MapPartitionsRDD[236] at map at SparkRDDWriteClient.java:126), which has no missing parents
22/02/07 23:47:20 INFO AsyncCleanerService: Async auto cleaning is not enabled. Not running cleaner now
22/02/07 23:47:20 INFO SparkContext: Starting job: countByKey at BaseSparkCommitActionExecutor.java:196
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_95 stored as values in memory (estimated size 667.4 KiB, free 983.4 MiB)
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_95_piece0 stored as bytes in memory (estimated size 237.1 KiB, free 983.2 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Added broadcast_95_piece0 in memory on 192.168.1.37:53037 (size: 237.1 KiB, free: 986.5 MiB)
22/02/07 23:47:21 INFO SparkContext: Created broadcast 95 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:21 INFO DAGScheduler: Submitting 20 missing tasks from ResultStage 122 (MapPartitionsRDD[236] at map at SparkRDDWriteClient.java:126) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/07 23:47:21 INFO TaskSchedulerImpl: Adding task set 122.0 with 20 tasks resource profile 0
22/02/07 23:47:21 INFO TaskSetManager: Starting task 0.0 in stage 122.0 (TID 797) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO TaskSetManager: Starting task 1.0 in stage 122.0 (TID 798) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO TaskSetManager: Starting task 2.0 in stage 122.0 (TID 799) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO DAGScheduler: Registering RDD 242 (countByKey at BaseSparkCommitActionExecutor.java:196) as input to shuffle 32
22/02/07 23:47:21 INFO Executor: Running task 0.0 in stage 122.0 (TID 797)
22/02/07 23:47:21 INFO DAGScheduler: Got job 71 (countByKey at BaseSparkCommitActionExecutor.java:196) with 1 output partitions
22/02/07 23:47:21 INFO DAGScheduler: Final stage: ResultStage 124 (countByKey at BaseSparkCommitActionExecutor.java:196)
22/02/07 23:47:21 INFO Executor: Running task 2.0 in stage 122.0 (TID 799)
22/02/07 23:47:21 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 123)
22/02/07 23:47:21 INFO Executor: Running task 1.0 in stage 122.0 (TID 798)
22/02/07 23:47:21 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 123)
22/02/07 23:47:21 INFO DAGScheduler: Submitting ShuffleMapStage 123 (MapPartitionsRDD[242] at countByKey at BaseSparkCommitActionExecutor.java:196), which has no missing parents
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_96 stored as values in memory (estimated size 13.3 KiB, free 983.2 MiB)
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_96_piece0 stored as bytes in memory (estimated size 5.6 KiB, free 983.2 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Added broadcast_96_piece0 in memory on 192.168.1.37:53037 (size: 5.6 KiB, free: 986.5 MiB)
22/02/07 23:47:21 INFO SparkContext: Created broadcast 96 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:21 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 123 (MapPartitionsRDD[242] at countByKey at BaseSparkCommitActionExecutor.java:196) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:21 INFO TaskSchedulerImpl: Adding task set 123.0 with 1 tasks resource profile 0
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_0 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_2 locally
22/02/07 23:47:21 INFO Executor: Finished task 0.0 in stage 122.0 (TID 797). 1743 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Finished task 2.0 in stage 122.0 (TID 799). 1639 bytes result sent to driver
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_1 locally
22/02/07 23:47:21 INFO TaskSetManager: Starting task 3.0 in stage 122.0 (TID 800) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Finished task 1.0 in stage 122.0 (TID 798). 1639 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Running task 3.0 in stage 122.0 (TID 800)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 4.0 in stage 122.0 (TID 801) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO TaskSetManager: Finished task 0.0 in stage 122.0 (TID 797) in 31 ms on 192.168.1.37 (executor driver) (1/20)
22/02/07 23:47:21 INFO Executor: Running task 4.0 in stage 122.0 (TID 801)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 2.0 in stage 122.0 (TID 799) in 31 ms on 192.168.1.37 (executor driver) (2/20)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 5.0 in stage 122.0 (TID 802) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO TaskSetManager: Finished task 1.0 in stage 122.0 (TID 798) in 31 ms on 192.168.1.37 (executor driver) (3/20)
22/02/07 23:47:21 INFO Executor: Running task 5.0 in stage 122.0 (TID 802)
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_3 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_5 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_4 locally
22/02/07 23:47:21 INFO Executor: Finished task 4.0 in stage 122.0 (TID 801). 1639 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Finished task 5.0 in stage 122.0 (TID 802). 1743 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Finished task 3.0 in stage 122.0 (TID 800). 1743 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Starting task 6.0 in stage 122.0 (TID 803) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 6.0 in stage 122.0 (TID 803)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 7.0 in stage 122.0 (TID 804) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 7.0 in stage 122.0 (TID 804)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 8.0 in stage 122.0 (TID 805) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 8.0 in stage 122.0 (TID 805)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 4.0 in stage 122.0 (TID 801) in 29 ms on 192.168.1.37 (executor driver) (4/20)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 3.0 in stage 122.0 (TID 800) in 29 ms on 192.168.1.37 (executor driver) (5/20)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 5.0 in stage 122.0 (TID 802) in 29 ms on 192.168.1.37 (executor driver) (6/20)
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_8 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_7 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_6 locally
22/02/07 23:47:21 INFO Executor: Finished task 8.0 in stage 122.0 (TID 805). 1784 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Finished task 7.0 in stage 122.0 (TID 804). 1787 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Finished task 6.0 in stage 122.0 (TID 803). 1682 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Starting task 9.0 in stage 122.0 (TID 806) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 9.0 in stage 122.0 (TID 806)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 10.0 in stage 122.0 (TID 807) (192.168.1.37, executor driver, partition 10, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO TaskSetManager: Finished task 8.0 in stage 122.0 (TID 805) in 109 ms on 192.168.1.37 (executor driver) (7/20)
22/02/07 23:47:21 INFO Executor: Running task 10.0 in stage 122.0 (TID 807)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 11.0 in stage 122.0 (TID 808) (192.168.1.37, executor driver, partition 11, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO TaskSetManager: Finished task 7.0 in stage 122.0 (TID 804) in 109 ms on 192.168.1.37 (executor driver) (8/20)
22/02/07 23:47:21 INFO Executor: Running task 11.0 in stage 122.0 (TID 808)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 6.0 in stage 122.0 (TID 803) in 109 ms on 192.168.1.37 (executor driver) (9/20)
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_9 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_10 locally
22/02/07 23:47:21 INFO Executor: Finished task 9.0 in stage 122.0 (TID 806). 1637 bytes result sent to driver
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_11 locally
22/02/07 23:47:21 INFO TaskSetManager: Starting task 12.0 in stage 122.0 (TID 809) (192.168.1.37, executor driver, partition 12, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Finished task 10.0 in stage 122.0 (TID 807). 1745 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Finished task 9.0 in stage 122.0 (TID 806) in 32 ms on 192.168.1.37 (executor driver) (10/20)
22/02/07 23:47:21 INFO Executor: Running task 12.0 in stage 122.0 (TID 809)
22/02/07 23:47:21 INFO Executor: Finished task 11.0 in stage 122.0 (TID 808). 1640 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Starting task 13.0 in stage 122.0 (TID 810) (192.168.1.37, executor driver, partition 13, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO TaskSetManager: Finished task 10.0 in stage 122.0 (TID 807) in 32 ms on 192.168.1.37 (executor driver) (11/20)
22/02/07 23:47:21 INFO Executor: Running task 13.0 in stage 122.0 (TID 810)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 14.0 in stage 122.0 (TID 811) (192.168.1.37, executor driver, partition 14, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 14.0 in stage 122.0 (TID 811)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 11.0 in stage 122.0 (TID 808) in 32 ms on 192.168.1.37 (executor driver) (12/20)
22/02/07 23:47:21 INFO BlockManagerInfo: Removed broadcast_92_piece0 on 192.168.1.37:53037 in memory (size: 237.2 KiB, free: 986.7 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Removed broadcast_94_piece0 on 192.168.1.37:53037 in memory (size: 4.0 KiB, free: 986.7 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Removed broadcast_93_piece0 on 192.168.1.37:53037 in memory (size: 214.0 KiB, free: 987.0 MiB)
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_12 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_14 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_13 locally
22/02/07 23:47:21 INFO Executor: Finished task 12.0 in stage 122.0 (TID 809). 1745 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Finished task 13.0 in stage 122.0 (TID 810). 1640 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Finished task 14.0 in stage 122.0 (TID 811). 1638 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Starting task 15.0 in stage 122.0 (TID 812) (192.168.1.37, executor driver, partition 15, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 15.0 in stage 122.0 (TID 812)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 16.0 in stage 122.0 (TID 813) (192.168.1.37, executor driver, partition 16, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 16.0 in stage 122.0 (TID 813)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 17.0 in stage 122.0 (TID 814) (192.168.1.37, executor driver, partition 17, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 17.0 in stage 122.0 (TID 814)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 12.0 in stage 122.0 (TID 809) in 32 ms on 192.168.1.37 (executor driver) (13/20)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 13.0 in stage 122.0 (TID 810) in 32 ms on 192.168.1.37 (executor driver) (14/20)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 14.0 in stage 122.0 (TID 811) in 32 ms on 192.168.1.37 (executor driver) (15/20)
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_17 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_16 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_15 locally
22/02/07 23:47:21 INFO Executor: Finished task 17.0 in stage 122.0 (TID 814). 1745 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Finished task 16.0 in stage 122.0 (TID 813). 1639 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Finished task 15.0 in stage 122.0 (TID 812). 1743 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Starting task 18.0 in stage 122.0 (TID 815) (192.168.1.37, executor driver, partition 18, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 18.0 in stage 122.0 (TID 815)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 19.0 in stage 122.0 (TID 816) (192.168.1.37, executor driver, partition 19, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 19.0 in stage 122.0 (TID 816)
22/02/07 23:47:21 INFO TaskSetManager: Starting task 0.0 in stage 123.0 (TID 817) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 6514 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 0.0 in stage 123.0 (TID 817)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 17.0 in stage 122.0 (TID 814) in 28 ms on 192.168.1.37 (executor driver) (16/20)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 16.0 in stage 122.0 (TID 813) in 30 ms on 192.168.1.37 (executor driver) (17/20)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 15.0 in stage 122.0 (TID 812) in 30 ms on 192.168.1.37 (executor driver) (18/20)
22/02/07 23:47:21 INFO MemoryStore: Block rdd_240_0 stored as values in memory (estimated size 3.8 KiB, free 984.9 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Added rdd_240_0 in memory on 192.168.1.37:53037 (size: 3.8 KiB, free: 986.9 MiB)
22/02/07 23:47:21 INFO Executor: Finished task 0.0 in stage 123.0 (TID 817). 1043 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Finished task 0.0 in stage 123.0 (TID 817) in 5 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:21 INFO TaskSchedulerImpl: Removed TaskSet 123.0, whose tasks have all completed, from pool
22/02/07 23:47:21 INFO DAGScheduler: ShuffleMapStage 123 (countByKey at BaseSparkCommitActionExecutor.java:196) finished in 0.265 s
22/02/07 23:47:21 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:21 INFO DAGScheduler: running: Set(ResultStage 122)
22/02/07 23:47:21 INFO DAGScheduler: waiting: Set(ResultStage 124)
22/02/07 23:47:21 INFO DAGScheduler: failed: Set()
22/02/07 23:47:21 INFO DAGScheduler: Submitting ResultStage 124 (ShuffledRDD[243] at countByKey at BaseSparkCommitActionExecutor.java:196), which has no missing parents
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_97 stored as values in memory (estimated size 5.6 KiB, free 984.9 MiB)
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_97_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 984.9 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Added broadcast_97_piece0 in memory on 192.168.1.37:53037 (size: 3.2 KiB, free: 986.9 MiB)
22/02/07 23:47:21 INFO SparkContext: Created broadcast 97 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:21 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 124 (ShuffledRDD[243] at countByKey at BaseSparkCommitActionExecutor.java:196) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:21 INFO TaskSchedulerImpl: Adding task set 124.0 with 1 tasks resource profile 0
22/02/07 23:47:21 INFO TaskSetManager: Starting task 0.0 in stage 124.0 (TID 818) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 0.0 in stage 124.0 (TID 818)
22/02/07 23:47:21 INFO ShuffleBlockFetcherIterator: Getting 1 (142.0 B) non-empty blocks including 1 (142.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:21 INFO Executor: Finished task 0.0 in stage 124.0 (TID 818). 1312 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Finished task 0.0 in stage 124.0 (TID 818) in 4 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:21 INFO TaskSchedulerImpl: Removed TaskSet 124.0, whose tasks have all completed, from pool
22/02/07 23:47:21 INFO DAGScheduler: ResultStage 124 (countByKey at BaseSparkCommitActionExecutor.java:196) finished in 0.006 s
22/02/07 23:47:21 INFO DAGScheduler: Job 71 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:21 INFO TaskSchedulerImpl: Killing all running tasks in stage 124: Stage finished
22/02/07 23:47:21 INFO DAGScheduler: Job 71 finished: countByKey at BaseSparkCommitActionExecutor.java:196, took 0.355558 s
22/02/07 23:47:21 INFO BaseSparkCommitActionExecutor: Workload profile :WorkloadProfile {globalStat=WorkloadStat {numInserts=0, numUpdates=10}, partitionStat={files=WorkloadStat {numInserts=0, numUpdates=10}}, operationType=UPSERT_PREPPED}
22/02/07 23:47:21 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234319263.deltacommit.requested
22/02/07 23:47:21 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234319263.deltacommit.inflight
22/02/07 23:47:21 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:21 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_19 locally
22/02/07 23:47:21 INFO BlockManager: Found block rdd_166_18 locally
22/02/07 23:47:21 INFO Executor: Finished task 19.0 in stage 122.0 (TID 816). 1745 bytes result sent to driver
22/02/07 23:47:21 INFO Executor: Finished task 18.0 in stage 122.0 (TID 815). 1639 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Finished task 19.0 in stage 122.0 (TID 816) in 30 ms on 192.168.1.37 (executor driver) (19/20)
22/02/07 23:47:21 INFO TaskSetManager: Finished task 18.0 in stage 122.0 (TID 815) in 30 ms on 192.168.1.37 (executor driver) (20/20)
22/02/07 23:47:21 INFO TaskSchedulerImpl: Removed TaskSet 122.0, whose tasks have all completed, from pool
22/02/07 23:47:21 INFO DAGScheduler: ResultStage 122 (collect at SparkRDDWriteClient.java:126) finished in 0.386 s
22/02/07 23:47:21 INFO DAGScheduler: Job 70 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:21 INFO TaskSchedulerImpl: Killing all running tasks in stage 122: Stage finished
22/02/07 23:47:21 INFO DAGScheduler: Job 70 finished: collect at SparkRDDWriteClient.java:126, took 0.387860 s
22/02/07 23:47:21 INFO BaseHoodieWriteClient: Committing 20220207234318886 action deltacommit
22/02/07 23:47:21 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:21 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:47:21 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:21 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:21 INFO SparkContext: Starting job: collect at SparkRejectUpdateStrategy.java:52
22/02/07 23:47:21 INFO DAGScheduler: Registering RDD 246 (distinct at SparkRejectUpdateStrategy.java:52) as input to shuffle 33
22/02/07 23:47:21 INFO DAGScheduler: Got job 72 (collect at SparkRejectUpdateStrategy.java:52) with 1 output partitions
22/02/07 23:47:21 INFO DAGScheduler: Final stage: ResultStage 126 (collect at SparkRejectUpdateStrategy.java:52)
22/02/07 23:47:21 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 125)
22/02/07 23:47:21 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 125)
22/02/07 23:47:21 INFO DAGScheduler: Submitting ShuffleMapStage 125 (MapPartitionsRDD[246] at distinct at SparkRejectUpdateStrategy.java:52), which has no missing parents
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_98 stored as values in memory (estimated size 13.4 KiB, free 984.8 MiB)
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_98_piece0 stored as bytes in memory (estimated size 5.5 KiB, free 984.8 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Added broadcast_98_piece0 in memory on 192.168.1.37:53037 (size: 5.5 KiB, free: 986.9 MiB)
22/02/07 23:47:21 INFO SparkContext: Created broadcast 98 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:21 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 125 (MapPartitionsRDD[246] at distinct at SparkRejectUpdateStrategy.java:52) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:21 INFO TaskSchedulerImpl: Adding task set 125.0 with 1 tasks resource profile 0
22/02/07 23:47:21 INFO TaskSetManager: Starting task 0.0 in stage 125.0 (TID 819) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 6514 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 0.0 in stage 125.0 (TID 819)
22/02/07 23:47:21 INFO BlockManager: Found block rdd_240_0 locally
22/02/07 23:47:21 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:47:21 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:21 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:47:21 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:21 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:21 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:21 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:21 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/07 23:47:21 INFO FileSystemViewManager: Creating remote first table view
22/02/07 23:47:21 INFO CommitUtils: Creating  metadata for UPSERT numWriteStats:20numReplaceFileIds:0
22/02/07 23:47:21 INFO Executor: Finished task 0.0 in stage 125.0 (TID 819). 1129 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Finished task 0.0 in stage 125.0 (TID 819) in 4 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:21 INFO TaskSchedulerImpl: Removed TaskSet 125.0, whose tasks have all completed, from pool
22/02/07 23:47:21 INFO TransactionManager: Transaction starting for Option{val=[==>20220207234318886__deltacommit__INFLIGHT]} with latest completed transaction instant Option{val=[20220207233322808__deltacommit__COMPLETED]}
22/02/07 23:47:21 INFO InProcessLockProvider: Thread pool-22-thread-1 ACQUIRING in-process lock.
22/02/07 23:47:21 INFO DAGScheduler: ShuffleMapStage 125 (distinct at SparkRejectUpdateStrategy.java:52) finished in 0.006 s
22/02/07 23:47:21 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:21 INFO DAGScheduler: running: Set()
22/02/07 23:47:21 INFO DAGScheduler: waiting: Set(ResultStage 126)
22/02/07 23:47:21 INFO DAGScheduler: failed: Set()
22/02/07 23:47:21 INFO DAGScheduler: Submitting ResultStage 126 (MapPartitionsRDD[248] at distinct at SparkRejectUpdateStrategy.java:52), which has no missing parents
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_99 stored as values in memory (estimated size 6.5 KiB, free 984.8 MiB)
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_99_piece0 stored as bytes in memory (estimated size 3.5 KiB, free 984.8 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Added broadcast_99_piece0 in memory on 192.168.1.37:53037 (size: 3.5 KiB, free: 986.9 MiB)
22/02/07 23:47:21 INFO SparkContext: Created broadcast 99 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:21 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 126 (MapPartitionsRDD[248] at distinct at SparkRejectUpdateStrategy.java:52) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:21 INFO TaskSchedulerImpl: Adding task set 126.0 with 1 tasks resource profile 0
22/02/07 23:47:21 INFO TaskSetManager: Starting task 0.0 in stage 126.0 (TID 820) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 0.0 in stage 126.0 (TID 820)
22/02/07 23:47:21 INFO ShuffleBlockFetcherIterator: Getting 1 (117.0 B) non-empty blocks including 1 (117.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:21 INFO Executor: Finished task 0.0 in stage 126.0 (TID 820). 1249 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Finished task 0.0 in stage 126.0 (TID 820) in 4 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:21 INFO TaskSchedulerImpl: Removed TaskSet 126.0, whose tasks have all completed, from pool
22/02/07 23:47:21 INFO DAGScheduler: ResultStage 126 (collect at SparkRejectUpdateStrategy.java:52) finished in 0.006 s
22/02/07 23:47:21 INFO DAGScheduler: Job 72 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:21 INFO TaskSchedulerImpl: Killing all running tasks in stage 126: Stage finished
22/02/07 23:47:21 INFO DAGScheduler: Job 72 finished: collect at SparkRejectUpdateStrategy.java:52, took 0.014405 s
22/02/07 23:47:21 INFO UpsertPartitioner: AvgRecordSize => 1024
22/02/07 23:47:21 INFO SparkContext: Starting job: collectAsMap at UpsertPartitioner.java:248
22/02/07 23:47:21 INFO DAGScheduler: Got job 73 (collectAsMap at UpsertPartitioner.java:248) with 1 output partitions
22/02/07 23:47:21 INFO DAGScheduler: Final stage: ResultStage 127 (collectAsMap at UpsertPartitioner.java:248)
22/02/07 23:47:21 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:21 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:21 INFO DAGScheduler: Submitting ResultStage 127 (MapPartitionsRDD[250] at mapToPair at UpsertPartitioner.java:247), which has no missing parents
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_100 stored as values in memory (estimated size 319.5 KiB, free 984.5 MiB)
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_100_piece0 stored as bytes in memory (estimated size 111.7 KiB, free 984.4 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Added broadcast_100_piece0 in memory on 192.168.1.37:53037 (size: 111.7 KiB, free: 986.8 MiB)
22/02/07 23:47:21 INFO SparkContext: Created broadcast 100 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:21 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 127 (MapPartitionsRDD[250] at mapToPair at UpsertPartitioner.java:247) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:21 INFO TaskSchedulerImpl: Adding task set 127.0 with 1 tasks resource profile 0
22/02/07 23:47:21 INFO TaskSetManager: Starting task 0.0 in stage 127.0 (TID 821) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4338 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 0.0 in stage 127.0 (TID 821)
22/02/07 23:47:21 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:21 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:21 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:21 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:21 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:21 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:21 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=29, NumFileGroups=1, FileGroupsCreationTime=5, StoreTimeTaken=0
22/02/07 23:47:21 INFO Executor: Finished task 0.0 in stage 127.0 (TID 821). 829 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Finished task 0.0 in stage 127.0 (TID 821) in 22 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:21 INFO TaskSchedulerImpl: Removed TaskSet 127.0, whose tasks have all completed, from pool
22/02/07 23:47:21 INFO DAGScheduler: ResultStage 127 (collectAsMap at UpsertPartitioner.java:248) finished in 0.069 s
22/02/07 23:47:21 INFO DAGScheduler: Job 73 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:21 INFO TaskSchedulerImpl: Killing all running tasks in stage 127: Stage finished
22/02/07 23:47:21 INFO DAGScheduler: Job 73 finished: collectAsMap at UpsertPartitioner.java:248, took 0.069673 s
22/02/07 23:47:21 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:21 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:21 INFO UpsertPartitioner: Total Buckets :1, buckets info => {0=BucketInfo {bucketType=UPDATE, fileIdPrefix=files-0000, partitionPath=files}},
Partition to insert buckets => {},
UpdateLocations mapped to buckets =>{files-0000=0}
22/02/07 23:47:21 INFO BaseSparkCommitActionExecutor: no validators configured.
22/02/07 23:47:21 INFO BaseCommitActionExecutor: Auto commit enabled: Committing 20220207234319263
22/02/07 23:47:21 INFO SparkContext: Starting job: collect at BaseSparkCommitActionExecutor.java:286
22/02/07 23:47:21 INFO DAGScheduler: Registering RDD 251 (mapToPair at BaseSparkCommitActionExecutor.java:233) as input to shuffle 34
22/02/07 23:47:21 INFO DAGScheduler: Got job 74 (collect at BaseSparkCommitActionExecutor.java:286) with 1 output partitions
22/02/07 23:47:21 INFO DAGScheduler: Final stage: ResultStage 129 (collect at BaseSparkCommitActionExecutor.java:286)
22/02/07 23:47:21 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 128)
22/02/07 23:47:21 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 128)
22/02/07 23:47:21 INFO DAGScheduler: Submitting ShuffleMapStage 128 (MapPartitionsRDD[251] at mapToPair at BaseSparkCommitActionExecutor.java:233), which has no missing parents
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_101 stored as values in memory (estimated size 327.5 KiB, free 984.1 MiB)
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_101_piece0 stored as bytes in memory (estimated size 116.9 KiB, free 984.0 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Added broadcast_101_piece0 in memory on 192.168.1.37:53037 (size: 116.9 KiB, free: 986.7 MiB)
22/02/07 23:47:21 INFO SparkContext: Created broadcast 101 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:21 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 128 (MapPartitionsRDD[251] at mapToPair at BaseSparkCommitActionExecutor.java:233) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:21 INFO TaskSchedulerImpl: Adding task set 128.0 with 1 tasks resource profile 0
22/02/07 23:47:21 INFO TaskSetManager: Starting task 0.0 in stage 128.0 (TID 822) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 6514 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 0.0 in stage 128.0 (TID 822)
22/02/07 23:47:21 INFO BlockManager: Found block rdd_240_0 locally
22/02/07 23:47:21 INFO Executor: Finished task 0.0 in stage 128.0 (TID 822). 1043 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Finished task 0.0 in stage 128.0 (TID 822) in 17 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:21 INFO TaskSchedulerImpl: Removed TaskSet 128.0, whose tasks have all completed, from pool
22/02/07 23:47:21 INFO DAGScheduler: ShuffleMapStage 128 (mapToPair at BaseSparkCommitActionExecutor.java:233) finished in 0.064 s
22/02/07 23:47:21 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:21 INFO DAGScheduler: running: Set()
22/02/07 23:47:21 INFO DAGScheduler: waiting: Set(ResultStage 129)
22/02/07 23:47:21 INFO DAGScheduler: failed: Set()
22/02/07 23:47:21 INFO DAGScheduler: Submitting ResultStage 129 (MapPartitionsRDD[256] at map at BaseSparkCommitActionExecutor.java:286), which has no missing parents
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_102 stored as values in memory (estimated size 432.5 KiB, free 983.5 MiB)
22/02/07 23:47:21 INFO MemoryStore: Block broadcast_102_piece0 stored as bytes in memory (estimated size 151.7 KiB, free 983.4 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Added broadcast_102_piece0 in memory on 192.168.1.37:53037 (size: 151.7 KiB, free: 986.6 MiB)
22/02/07 23:47:21 INFO SparkContext: Created broadcast 102 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:21 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 129 (MapPartitionsRDD[256] at map at BaseSparkCommitActionExecutor.java:286) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:21 INFO TaskSchedulerImpl: Adding task set 129.0 with 1 tasks resource profile 0
22/02/07 23:47:21 INFO TaskSetManager: Starting task 0.0 in stage 129.0 (TID 823) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:21 INFO Executor: Running task 0.0 in stage 129.0 (TID 823)
22/02/07 23:47:21 INFO ShuffleBlockFetcherIterator: Getting 1 (1156.0 B) non-empty blocks including 1 (1156.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:21 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220207234319263 for file files-0000
22/02/07 23:47:21 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:21 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:21 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:21 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:21 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:21 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:21 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=29, NumFileGroups=1, FileGroupsCreationTime=5, StoreTimeTaken=0
22/02/07 23:47:21 INFO DirectWriteMarkers: Creating Marker Path=/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207234319263/files/files-0000_0-129-823_00000000000000.hfile.marker.APPEND
22/02/07 23:47:21 INFO DirectWriteMarkers: [direct] Created marker file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207234319263/files/files-0000_0-129-823_00000000000000.hfile.marker.APPEND in 15 ms
22/02/07 23:47:21 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer
22/02/07 23:47:21 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.27_0-102-752
22/02/07 23:47:21 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.28_0-129-823', fileLen=-1}
22/02/07 23:47:21 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:47:21 INFO CodecPool: Got brand-new compressor [.gz]
22/02/07 23:47:21 INFO CodecPool: Got brand-new compressor [.gz]
22/02/07 23:47:21 INFO HoodieAppendHandle: AppendHandle for partitionPath files filePath files/.files-0000_00000000000000.log.28_0-129-823, took 46 ms.
22/02/07 23:47:21 INFO MemoryStore: Block rdd_255_0 stored as values in memory (estimated size 1587.0 B, free 983.4 MiB)
22/02/07 23:47:21 INFO BlockManagerInfo: Added rdd_255_0 in memory on 192.168.1.37:53037 (size: 1587.0 B, free: 986.6 MiB)
22/02/07 23:47:21 INFO Executor: Finished task 0.0 in stage 129.0 (TID 823). 2741 bytes result sent to driver
22/02/07 23:47:21 INFO TaskSetManager: Finished task 0.0 in stage 129.0 (TID 823) in 70 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:21 INFO TaskSchedulerImpl: Removed TaskSet 129.0, whose tasks have all completed, from pool
22/02/07 23:47:21 INFO DAGScheduler: ResultStage 129 (collect at BaseSparkCommitActionExecutor.java:286) finished in 0.135 s
22/02/07 23:47:21 INFO DAGScheduler: Job 74 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:21 INFO TaskSchedulerImpl: Killing all running tasks in stage 129: Stage finished
22/02/07 23:47:21 INFO DAGScheduler: Job 74 finished: collect at BaseSparkCommitActionExecutor.java:286, took 0.200107 s
22/02/07 23:47:21 INFO CommitUtils: Creating  metadata for UPSERT_PREPPED numWriteStats:1numReplaceFileIds:0
22/02/07 23:47:22 INFO SparkContext: Starting job: collect at BaseSparkCommitActionExecutor.java:294
22/02/07 23:47:22 INFO DAGScheduler: Got job 75 (collect at BaseSparkCommitActionExecutor.java:294) with 1 output partitions
22/02/07 23:47:22 INFO DAGScheduler: Final stage: ResultStage 131 (collect at BaseSparkCommitActionExecutor.java:294)
22/02/07 23:47:22 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 130)
22/02/07 23:47:22 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:22 INFO DAGScheduler: Submitting ResultStage 131 (MapPartitionsRDD[257] at map at BaseSparkCommitActionExecutor.java:294), which has no missing parents
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_103 stored as values in memory (estimated size 432.5 KiB, free 983.0 MiB)
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_103_piece0 stored as bytes in memory (estimated size 151.7 KiB, free 982.8 MiB)
22/02/07 23:47:22 INFO BlockManagerInfo: Added broadcast_103_piece0 in memory on 192.168.1.37:53037 (size: 151.7 KiB, free: 986.4 MiB)
22/02/07 23:47:22 INFO SparkContext: Created broadcast 103 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 131 (MapPartitionsRDD[257] at map at BaseSparkCommitActionExecutor.java:294) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:22 INFO TaskSchedulerImpl: Adding task set 131.0 with 1 tasks resource profile 0
22/02/07 23:47:22 INFO TaskSetManager: Starting task 0.0 in stage 131.0 (TID 824) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:22 INFO Executor: Running task 0.0 in stage 131.0 (TID 824)
22/02/07 23:47:22 INFO BlockManager: Found block rdd_255_0 locally
22/02/07 23:47:22 INFO Executor: Finished task 0.0 in stage 131.0 (TID 824). 2354 bytes result sent to driver
22/02/07 23:47:22 INFO TaskSetManager: Finished task 0.0 in stage 131.0 (TID 824) in 43 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:22 INFO TaskSchedulerImpl: Removed TaskSet 131.0, whose tasks have all completed, from pool
22/02/07 23:47:22 INFO DAGScheduler: ResultStage 131 (collect at BaseSparkCommitActionExecutor.java:294) finished in 0.118 s
22/02/07 23:47:22 INFO DAGScheduler: Job 75 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:22 INFO TaskSchedulerImpl: Killing all running tasks in stage 131: Stage finished
22/02/07 23:47:22 INFO DAGScheduler: Job 75 finished: collect at BaseSparkCommitActionExecutor.java:294, took 0.118664 s
22/02/07 23:47:22 INFO BaseSparkCommitActionExecutor: Committing 20220207234319263, action Type deltacommit, operation Type UPSERT_PREPPED
22/02/07 23:47:22 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/07 23:47:22 INFO DAGScheduler: Got job 76 (collect at HoodieSparkEngineContext.java:134) with 1 output partitions
22/02/07 23:47:22 INFO DAGScheduler: Final stage: ResultStage 132 (collect at HoodieSparkEngineContext.java:134)
22/02/07 23:47:22 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:22 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:22 INFO DAGScheduler: Submitting ResultStage 132 (MapPartitionsRDD[259] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_104 stored as values in memory (estimated size 99.2 KiB, free 982.7 MiB)
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_104_piece0 stored as bytes in memory (estimated size 35.1 KiB, free 982.7 MiB)
22/02/07 23:47:22 INFO BlockManagerInfo: Added broadcast_104_piece0 in memory on 192.168.1.37:53037 (size: 35.1 KiB, free: 986.4 MiB)
22/02/07 23:47:22 INFO SparkContext: Created broadcast 104 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 132 (MapPartitionsRDD[259] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:22 INFO TaskSchedulerImpl: Adding task set 132.0 with 1 tasks resource profile 0
22/02/07 23:47:22 INFO TaskSetManager: Starting task 0.0 in stage 132.0 (TID 825) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4456 bytes) taskResourceAssignments Map()
22/02/07 23:47:22 INFO Executor: Running task 0.0 in stage 132.0 (TID 825)
22/02/07 23:47:22 INFO Executor: Finished task 0.0 in stage 132.0 (TID 825). 796 bytes result sent to driver
22/02/07 23:47:22 INFO TaskSetManager: Finished task 0.0 in stage 132.0 (TID 825) in 14 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:22 INFO TaskSchedulerImpl: Removed TaskSet 132.0, whose tasks have all completed, from pool
22/02/07 23:47:22 INFO DAGScheduler: ResultStage 132 (collect at HoodieSparkEngineContext.java:134) finished in 0.032 s
22/02/07 23:47:22 INFO DAGScheduler: Job 76 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:22 INFO TaskSchedulerImpl: Killing all running tasks in stage 132: Stage finished
22/02/07 23:47:22 INFO DAGScheduler: Job 76 finished: collect at HoodieSparkEngineContext.java:134, took 0.032826 s
22/02/07 23:47:22 INFO HoodieActiveTimeline: Marking instant complete [==>20220207234319263__deltacommit__INFLIGHT]
22/02/07 23:47:22 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234319263.deltacommit.inflight
22/02/07 23:47:22 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234319263.deltacommit
22/02/07 23:47:22 INFO HoodieActiveTimeline: Completed [==>20220207234319263__deltacommit__INFLIGHT]
22/02/07 23:47:22 INFO BaseSparkCommitActionExecutor: Committed 20220207234319263
22/02/07 23:47:22 INFO SparkContext: Starting job: collectAsMap at HoodieSparkEngineContext.java:148
22/02/07 23:47:22 INFO DAGScheduler: Got job 77 (collectAsMap at HoodieSparkEngineContext.java:148) with 1 output partitions
22/02/07 23:47:22 INFO DAGScheduler: Final stage: ResultStage 133 (collectAsMap at HoodieSparkEngineContext.java:148)
22/02/07 23:47:22 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:22 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:22 INFO DAGScheduler: Submitting ResultStage 133 (MapPartitionsRDD[261] at mapToPair at HoodieSparkEngineContext.java:145), which has no missing parents
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_105 stored as values in memory (estimated size 99.4 KiB, free 982.6 MiB)
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_105_piece0 stored as bytes in memory (estimated size 35.1 KiB, free 982.6 MiB)
22/02/07 23:47:22 INFO BlockManagerInfo: Added broadcast_105_piece0 in memory on 192.168.1.37:53037 (size: 35.1 KiB, free: 986.3 MiB)
22/02/07 23:47:22 INFO SparkContext: Created broadcast 105 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 133 (MapPartitionsRDD[261] at mapToPair at HoodieSparkEngineContext.java:145) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:22 INFO TaskSchedulerImpl: Adding task set 133.0 with 1 tasks resource profile 0
22/02/07 23:47:22 INFO TaskSetManager: Starting task 0.0 in stage 133.0 (TID 826) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4456 bytes) taskResourceAssignments Map()
22/02/07 23:47:22 INFO Executor: Running task 0.0 in stage 133.0 (TID 826)
22/02/07 23:47:22 INFO Executor: Finished task 0.0 in stage 133.0 (TID 826). 883 bytes result sent to driver
22/02/07 23:47:22 INFO TaskSetManager: Finished task 0.0 in stage 133.0 (TID 826) in 6 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:22 INFO TaskSchedulerImpl: Removed TaskSet 133.0, whose tasks have all completed, from pool
22/02/07 23:47:22 INFO DAGScheduler: ResultStage 133 (collectAsMap at HoodieSparkEngineContext.java:148) finished in 0.024 s
22/02/07 23:47:22 INFO DAGScheduler: Job 77 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:22 INFO TaskSchedulerImpl: Killing all running tasks in stage 133: Stage finished
22/02/07 23:47:22 INFO DAGScheduler: Job 77 finished: collectAsMap at HoodieSparkEngineContext.java:148, took 0.024743 s
22/02/07 23:47:22 INFO FSUtils: Removed directory at /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207234319263
22/02/07 23:47:22 INFO SparkContext: Starting job: collect at SparkHoodieBackedTableMetadataWriter.java:154
22/02/07 23:47:22 INFO DAGScheduler: Got job 78 (collect at SparkHoodieBackedTableMetadataWriter.java:154) with 1 output partitions
22/02/07 23:47:22 INFO DAGScheduler: Final stage: ResultStage 135 (collect at SparkHoodieBackedTableMetadataWriter.java:154)
22/02/07 23:47:22 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 134)
22/02/07 23:47:22 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:22 INFO DAGScheduler: Submitting ResultStage 135 (MapPartitionsRDD[255] at flatMap at BaseSparkCommitActionExecutor.java:181), which has no missing parents
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_106 stored as values in memory (estimated size 432.1 KiB, free 982.1 MiB)
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_106_piece0 stored as bytes in memory (estimated size 151.6 KiB, free 982.0 MiB)
22/02/07 23:47:22 INFO BlockManagerInfo: Added broadcast_106_piece0 in memory on 192.168.1.37:53037 (size: 151.6 KiB, free: 986.2 MiB)
22/02/07 23:47:22 INFO SparkContext: Created broadcast 106 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 135 (MapPartitionsRDD[255] at flatMap at BaseSparkCommitActionExecutor.java:181) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:22 INFO TaskSchedulerImpl: Adding task set 135.0 with 1 tasks resource profile 0
22/02/07 23:47:22 INFO TaskSetManager: Starting task 0.0 in stage 135.0 (TID 827) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:22 INFO Executor: Running task 0.0 in stage 135.0 (TID 827)
22/02/07 23:47:22 INFO BlockManager: Found block rdd_255_0 locally
22/02/07 23:47:22 INFO Executor: Finished task 0.0 in stage 135.0 (TID 827). 2434 bytes result sent to driver
22/02/07 23:47:22 INFO TaskSetManager: Finished task 0.0 in stage 135.0 (TID 827) in 20 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:22 INFO TaskSchedulerImpl: Removed TaskSet 135.0, whose tasks have all completed, from pool
22/02/07 23:47:22 INFO DAGScheduler: ResultStage 135 (collect at SparkHoodieBackedTableMetadataWriter.java:154) finished in 0.084 s
22/02/07 23:47:22 INFO DAGScheduler: Job 78 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:22 INFO TaskSchedulerImpl: Killing all running tasks in stage 135: Stage finished
22/02/07 23:47:22 INFO DAGScheduler: Job 78 finished: collect at SparkHoodieBackedTableMetadataWriter.java:154, took 0.085832 s
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:22 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234228524.clean.inflight
22/02/07 23:47:22 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234228524.clean
22/02/07 23:47:22 INFO CleanActionExecutor: Marked clean started on 20220207234228524 as complete
22/02/07 23:47:22 INFO TransactionManager: Transaction ending with transaction owner Optional.empty
22/02/07 23:47:22 INFO InProcessLockProvider: Thread pool-26-thread-1 RELEASING in-process lock.
22/02/07 23:47:22 INFO InProcessLockProvider: Thread pool-26-thread-1 RELEASED in-process lock.
22/02/07 23:47:22 INFO TransactionManager: Transaction ended with transaction owner Optional.empty
22/02/07 23:47:22 INFO InProcessLockProvider: Thread pool-22-thread-1 ACQUIRED in-process lock.
22/02/07 23:47:22 INFO TransactionManager: Transaction started for Option{val=[==>20220207234318886__deltacommit__INFLIGHT]} with latest completed transaction instant Option{val=[20220207233322808__deltacommit__COMPLETED]}
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO CleanActionExecutor: Finishing previously unfinished cleaner instant=[==>20220207234319263__clean__REQUESTED]
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:22 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234319263.clean.requested
22/02/07 23:47:22 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234319263.clean.inflight
22/02/07 23:47:22 INFO CleanActionExecutor: Using cleanerParallelism: 10
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/07 23:47:22 INFO FileSystemViewManager: Creating remote first table view
22/02/07 23:47:22 INFO SimpleConcurrentFileWritesConflictResolutionStrategy: Found conflicting writes between first operation = {actionType=deltacommit, instantTime=20220207234318886, actionState=INFLIGHT'}, second operation = {actionType=commit, instantTime=20220207233505725, actionState=COMPLETED'} , intersecting file ids [dac159fb-2bef-4f60-902f-ed6360241988-0, e766accf-2d87-4fb2-a256-ff45e728094a-0, 9d74eab4-a6ea-4d22-9f9d-4578f3fb0fad-0, 8fd44984-ac08-4808-b595-da40ff0d8ad6-0, aca43bde-dc01-41e1-be66-74f78bb58ed8-0, df3a0019-62aa-4cea-b4c1-82439c339b6f-0, 356c259b-ec4a-42d4-b893-9fa3daf45451-0, 5cabcfe9-978a-4c55-bc96-130ccab4f081-0, 09f346a5-4fa1-4ae3-93e5-5a23bec43dcf-0, 102d0248-cf64-42a2-8be8-dbc5c7a693d1-0]
22/02/07 23:47:22 INFO TransactionUtils: Conflict encountered between current instant = {actionType=deltacommit, instantTime=20220207234318886, actionState=INFLIGHT'} and instant = {actionType=commit, instantTime=20220207233505725, actionState=COMPLETED'}, attempting to resolve it...
22/02/07 23:47:22 INFO TransactionUtils: Successfully resolved conflicts, if any
22/02/07 23:47:22 INFO BaseHoodieWriteClient: Committing 20220207234318886 action deltacommit
22/02/07 23:47:22 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53112/v1/hoodie/marker/dir/exists?markerdirpath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220207234318886)
22/02/07 23:47:22 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53112/v1/hoodie/marker/create-and-merge?markerdirpath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220207234318886)
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:22 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:22 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:22 INFO HoodieTableMetadataUtil: Loading latest file slices for metadata table partition files
22/02/07 23:47:22 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:22 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:22 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:22 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=30, NumFileGroups=1, FileGroupsCreationTime=6, StoreTimeTaken=0
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableMetadataUtil: Updating at 20220207234318886 from Commit/UPSERT. #partitions_updated=11
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:22 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:22 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:22 INFO HoodieTableMetadataUtil: Loading latest file slices for metadata table partition files
22/02/07 23:47:22 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:22 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:22 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:22 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:122
22/02/07 23:47:22 INFO DAGScheduler: Registering RDD 265 (mapPartitionsToPair at HoodieSparkEngineContext.java:116) as input to shuffle 35
22/02/07 23:47:22 INFO DAGScheduler: Got job 79 (collect at HoodieSparkEngineContext.java:122) with 10 output partitions
22/02/07 23:47:22 INFO DAGScheduler: Final stage: ResultStage 137 (collect at HoodieSparkEngineContext.java:122)
22/02/07 23:47:22 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 136)
22/02/07 23:47:22 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 136)
22/02/07 23:47:22 INFO DAGScheduler: Submitting ShuffleMapStage 136 (MapPartitionsRDD[265] at mapPartitionsToPair at HoodieSparkEngineContext.java:116), which has no missing parents
22/02/07 23:47:22 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=30, NumFileGroups=1, FileGroupsCreationTime=6, StoreTimeTaken=0
22/02/07 23:47:22 INFO BaseHoodieClient: Embedded Timeline Server is disabled. Not starting timeline service
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:22 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:22 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:47:22 INFO HoodieBackedTableMetadataWriter: Cannot compact metadata table as there are 6 inflight instants before latest deltacommit 20220207234332930: [[==>20220207230354778__rollback__INFLIGHT], [==>20220207234123811__rollback__INFLIGHT], [==>20220207234155839__rollback__INFLIGHT], [==>20220207234228129__deltacommit__REQUESTED], [==>20220207234228525__rollback__INFLIGHT], [==>20220207234247693__rollback__INFLIGHT]]
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:22 INFO BaseHoodieWriteClient: Generate a new instant time: 20220207234318886 action: deltacommit
22/02/07 23:47:22 INFO HoodieHeartbeatClient: Received request to start heartbeat for instant time 20220207234318886
22/02/07 23:47:22 INFO HoodieActiveTimeline: Creating a new instant [==>20220207234318886__deltacommit__REQUESTED]
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:22 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:22 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:22 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:22 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:22 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:22 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:22 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:22 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:22 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:22 INFO AsyncCleanerService: Async auto cleaning is not enabled. Not running cleaner now
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_107 stored as values in memory (estimated size 612.1 KiB, free 981.4 MiB)
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_107_piece0 stored as bytes in memory (estimated size 214.0 KiB, free 981.2 MiB)
22/02/07 23:47:22 INFO BlockManagerInfo: Added broadcast_107_piece0 in memory on 192.168.1.37:53037 (size: 214.0 KiB, free: 986.0 MiB)
22/02/07 23:47:22 INFO SparkContext: Created broadcast 107 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:22 INFO DAGScheduler: Submitting 10 missing tasks from ShuffleMapStage 136 (MapPartitionsRDD[265] at mapPartitionsToPair at HoodieSparkEngineContext.java:116) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/07 23:47:22 INFO TaskSchedulerImpl: Adding task set 136.0 with 10 tasks resource profile 0
22/02/07 23:47:22 INFO TaskSetManager: Starting task 0.0 in stage 136.0 (TID 828) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:22 INFO TaskSetManager: Starting task 1.0 in stage 136.0 (TID 829) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:22 INFO TaskSetManager: Starting task 2.0 in stage 136.0 (TID 830) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:22 INFO Executor: Running task 0.0 in stage 136.0 (TID 828)
22/02/07 23:47:22 INFO Executor: Running task 1.0 in stage 136.0 (TID 829)
22/02/07 23:47:22 INFO Executor: Running task 2.0 in stage 136.0 (TID 830)
22/02/07 23:47:22 INFO SparkContext: Starting job: countByKey at BaseSparkCommitActionExecutor.java:196
22/02/07 23:47:22 INFO DAGScheduler: Registering RDD 271 (countByKey at BaseSparkCommitActionExecutor.java:196) as input to shuffle 36
22/02/07 23:47:22 INFO DAGScheduler: Got job 80 (countByKey at BaseSparkCommitActionExecutor.java:196) with 1 output partitions
22/02/07 23:47:22 INFO DAGScheduler: Final stage: ResultStage 139 (countByKey at BaseSparkCommitActionExecutor.java:196)
22/02/07 23:47:22 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 138)
22/02/07 23:47:22 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 138)
22/02/07 23:47:22 INFO DAGScheduler: Submitting ShuffleMapStage 138 (MapPartitionsRDD[271] at countByKey at BaseSparkCommitActionExecutor.java:196), which has no missing parents
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_108 stored as values in memory (estimated size 13.5 KiB, free 981.2 MiB)
22/02/07 23:47:22 INFO MemoryStore: Block broadcast_108_piece0 stored as bytes in memory (estimated size 5.6 KiB, free 981.2 MiB)
22/02/07 23:47:22 INFO BlockManagerInfo: Added broadcast_108_piece0 in memory on 192.168.1.37:53037 (size: 5.6 KiB, free: 986.0 MiB)
22/02/07 23:47:22 INFO SparkContext: Created broadcast 108 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:22 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 138 (MapPartitionsRDD[271] at countByKey at BaseSparkCommitActionExecutor.java:196) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:22 INFO TaskSchedulerImpl: Adding task set 138.0 with 1 tasks resource profile 0
22/02/07 23:47:23 INFO Executor: Finished task 1.0 in stage 136.0 (TID 829). 1138 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 3.0 in stage 136.0 (TID 831) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 1.0 in stage 136.0 (TID 829) in 173 ms on 192.168.1.37 (executor driver) (1/10)
22/02/07 23:47:23 INFO Executor: Running task 3.0 in stage 136.0 (TID 831)
22/02/07 23:47:23 INFO Executor: Finished task 0.0 in stage 136.0 (TID 828). 1138 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 4.0 in stage 136.0 (TID 832) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4739 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 0.0 in stage 136.0 (TID 828) in 174 ms on 192.168.1.37 (executor driver) (2/10)
22/02/07 23:47:23 INFO Executor: Running task 4.0 in stage 136.0 (TID 832)
22/02/07 23:47:23 INFO Executor: Finished task 2.0 in stage 136.0 (TID 830). 1138 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 5.0 in stage 136.0 (TID 833) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 2.0 in stage 136.0 (TID 830) in 175 ms on 192.168.1.37 (executor driver) (3/10)
22/02/07 23:47:23 INFO Executor: Running task 5.0 in stage 136.0 (TID 833)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_106_piece0 on 192.168.1.37:53037 in memory (size: 151.6 KiB, free: 986.1 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_102_piece0 on 192.168.1.37:53037 in memory (size: 151.7 KiB, free: 986.3 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_104_piece0 on 192.168.1.37:53037 in memory (size: 35.1 KiB, free: 986.3 MiB)
22/02/07 23:47:23 INFO BlockManager: Removing RDD 255
22/02/07 23:47:23 INFO BlockManager: Removing RDD 240
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_97_piece0 on 192.168.1.37:53037 in memory (size: 3.2 KiB, free: 986.3 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_101_piece0 on 192.168.1.37:53037 in memory (size: 116.9 KiB, free: 986.4 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_103_piece0 on 192.168.1.37:53037 in memory (size: 151.7 KiB, free: 986.6 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_98_piece0 on 192.168.1.37:53037 in memory (size: 5.5 KiB, free: 986.6 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_99_piece0 on 192.168.1.37:53037 in memory (size: 3.5 KiB, free: 986.6 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_96_piece0 on 192.168.1.37:53037 in memory (size: 5.6 KiB, free: 986.6 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_100_piece0 on 192.168.1.37:53037 in memory (size: 111.7 KiB, free: 986.7 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_105_piece0 on 192.168.1.37:53037 in memory (size: 35.1 KiB, free: 986.7 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Removed broadcast_95_piece0 on 192.168.1.37:53037 in memory (size: 237.1 KiB, free: 987.0 MiB)
22/02/07 23:47:23 INFO Executor: Finished task 3.0 in stage 136.0 (TID 831). 1095 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 6.0 in stage 136.0 (TID 834) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 3.0 in stage 136.0 (TID 831) in 46 ms on 192.168.1.37 (executor driver) (4/10)
22/02/07 23:47:23 INFO Executor: Running task 6.0 in stage 136.0 (TID 834)
22/02/07 23:47:23 INFO Executor: Finished task 4.0 in stage 136.0 (TID 832). 1095 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 7.0 in stage 136.0 (TID 835) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4739 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 4.0 in stage 136.0 (TID 832) in 45 ms on 192.168.1.37 (executor driver) (5/10)
22/02/07 23:47:23 INFO Executor: Running task 7.0 in stage 136.0 (TID 835)
22/02/07 23:47:23 INFO Executor: Finished task 5.0 in stage 136.0 (TID 833). 1095 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 8.0 in stage 136.0 (TID 836) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 5.0 in stage 136.0 (TID 833) in 48 ms on 192.168.1.37 (executor driver) (6/10)
22/02/07 23:47:23 INFO Executor: Running task 8.0 in stage 136.0 (TID 836)
22/02/07 23:47:23 INFO Executor: Finished task 6.0 in stage 136.0 (TID 834). 1095 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 9.0 in stage 136.0 (TID 837) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4742 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 6.0 in stage 136.0 (TID 834) in 37 ms on 192.168.1.37 (executor driver) (7/10)
22/02/07 23:47:23 INFO Executor: Running task 9.0 in stage 136.0 (TID 837)
22/02/07 23:47:23 INFO Executor: Finished task 7.0 in stage 136.0 (TID 835). 1095 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 0.0 in stage 138.0 (TID 838) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 6663 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 7.0 in stage 136.0 (TID 835) in 37 ms on 192.168.1.37 (executor driver) (8/10)
22/02/07 23:47:23 INFO Executor: Running task 0.0 in stage 138.0 (TID 838)
22/02/07 23:47:23 INFO MemoryStore: Block rdd_269_0 stored as values in memory (estimated size 4.2 KiB, free 984.9 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Added rdd_269_0 in memory on 192.168.1.37:53037 (size: 4.2 KiB, free: 987.0 MiB)
22/02/07 23:47:23 INFO Executor: Finished task 8.0 in stage 136.0 (TID 836). 1095 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 8.0 in stage 136.0 (TID 836) in 35 ms on 192.168.1.37 (executor driver) (9/10)
22/02/07 23:47:23 INFO Executor: Finished task 0.0 in stage 138.0 (TID 838). 1086 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 0.0 in stage 138.0 (TID 838) in 7 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:23 INFO TaskSchedulerImpl: Removed TaskSet 138.0, whose tasks have all completed, from pool
22/02/07 23:47:23 INFO DAGScheduler: ShuffleMapStage 138 (countByKey at BaseSparkCommitActionExecutor.java:196) finished in 0.253 s
22/02/07 23:47:23 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:23 INFO DAGScheduler: running: Set(ShuffleMapStage 136)
22/02/07 23:47:23 INFO DAGScheduler: waiting: Set(ResultStage 139, ResultStage 137)
22/02/07 23:47:23 INFO DAGScheduler: failed: Set()
22/02/07 23:47:23 INFO DAGScheduler: Submitting ResultStage 139 (ShuffledRDD[272] at countByKey at BaseSparkCommitActionExecutor.java:196), which has no missing parents
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_109 stored as values in memory (estimated size 5.6 KiB, free 984.9 MiB)
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_109_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 984.9 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Added broadcast_109_piece0 in memory on 192.168.1.37:53037 (size: 3.2 KiB, free: 987.0 MiB)
22/02/07 23:47:23 INFO SparkContext: Created broadcast 109 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:23 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 139 (ShuffledRDD[272] at countByKey at BaseSparkCommitActionExecutor.java:196) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:23 INFO TaskSchedulerImpl: Adding task set 139.0 with 1 tasks resource profile 0
22/02/07 23:47:23 INFO TaskSetManager: Starting task 0.0 in stage 139.0 (TID 839) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO Executor: Running task 0.0 in stage 139.0 (TID 839)
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 1 (142.0 B) non-empty blocks including 1 (142.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO Executor: Finished task 0.0 in stage 139.0 (TID 839). 1355 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 0.0 in stage 139.0 (TID 839) in 5 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:23 INFO TaskSchedulerImpl: Removed TaskSet 139.0, whose tasks have all completed, from pool
22/02/07 23:47:23 INFO DAGScheduler: ResultStage 139 (countByKey at BaseSparkCommitActionExecutor.java:196) finished in 0.007 s
22/02/07 23:47:23 INFO DAGScheduler: Job 80 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:23 INFO TaskSchedulerImpl: Killing all running tasks in stage 139: Stage finished
22/02/07 23:47:23 INFO DAGScheduler: Job 80 finished: countByKey at BaseSparkCommitActionExecutor.java:196, took 0.262741 s
22/02/07 23:47:23 INFO BaseSparkCommitActionExecutor: Workload profile :WorkloadProfile {globalStat=WorkloadStat {numInserts=0, numUpdates=11}, partitionStat={files=WorkloadStat {numInserts=0, numUpdates=11}}, operationType=UPSERT_PREPPED}
22/02/07 23:47:23 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234318886.deltacommit.requested
22/02/07 23:47:23 INFO FileIOUtils: Created a new file in meta path: /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234318886.deltacommit.inflight
22/02/07 23:47:23 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234318886.deltacommit.inflight
22/02/07 23:47:23 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:23 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:23 INFO Executor: Finished task 9.0 in stage 136.0 (TID 837). 1095 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 9.0 in stage 136.0 (TID 837) in 44 ms on 192.168.1.37 (executor driver) (10/10)
22/02/07 23:47:23 INFO TaskSchedulerImpl: Removed TaskSet 136.0, whose tasks have all completed, from pool
22/02/07 23:47:23 INFO DAGScheduler: ShuffleMapStage 136 (mapPartitionsToPair at HoodieSparkEngineContext.java:116) finished in 0.413 s
22/02/07 23:47:23 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:23 INFO DAGScheduler: running: Set()
22/02/07 23:47:23 INFO DAGScheduler: waiting: Set(ResultStage 137)
22/02/07 23:47:23 INFO DAGScheduler: failed: Set()
22/02/07 23:47:23 INFO DAGScheduler: Submitting ResultStage 137 (MapPartitionsRDD[267] at map at HoodieSparkEngineContext.java:121), which has no missing parents
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_110 stored as values in memory (estimated size 7.7 KiB, free 984.9 MiB)
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_110_piece0 stored as bytes in memory (estimated size 4.0 KiB, free 984.9 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Added broadcast_110_piece0 in memory on 192.168.1.37:53037 (size: 4.0 KiB, free: 987.0 MiB)
22/02/07 23:47:23 INFO SparkContext: Created broadcast 110 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:23 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 137 (MapPartitionsRDD[267] at map at HoodieSparkEngineContext.java:121) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/07 23:47:23 INFO TaskSchedulerImpl: Adding task set 137.0 with 10 tasks resource profile 0
22/02/07 23:47:23 INFO TaskSetManager: Starting task 1.0 in stage 137.0 (TID 840) (192.168.1.37, executor driver, partition 1, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Starting task 2.0 in stage 137.0 (TID 841) (192.168.1.37, executor driver, partition 2, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Starting task 3.0 in stage 137.0 (TID 842) (192.168.1.37, executor driver, partition 3, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO Executor: Running task 2.0 in stage 137.0 (TID 841)
22/02/07 23:47:23 INFO Executor: Running task 3.0 in stage 137.0 (TID 842)
22/02/07 23:47:23 INFO Executor: Running task 1.0 in stage 137.0 (TID 840)
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO Executor: Finished task 3.0 in stage 137.0 (TID 842). 1673 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 4.0 in stage 137.0 (TID 843) (192.168.1.37, executor driver, partition 4, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 3.0 in stage 137.0 (TID 842) in 5 ms on 192.168.1.37 (executor driver) (1/10)
22/02/07 23:47:23 INFO Executor: Finished task 2.0 in stage 137.0 (TID 841). 1673 bytes result sent to driver
22/02/07 23:47:23 INFO Executor: Running task 4.0 in stage 137.0 (TID 843)
22/02/07 23:47:23 INFO Executor: Finished task 1.0 in stage 137.0 (TID 840). 1673 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 5.0 in stage 137.0 (TID 844) (192.168.1.37, executor driver, partition 5, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 2.0 in stage 137.0 (TID 841) in 5 ms on 192.168.1.37 (executor driver) (2/10)
22/02/07 23:47:23 INFO Executor: Running task 5.0 in stage 137.0 (TID 844)
22/02/07 23:47:23 INFO TaskSetManager: Starting task 6.0 in stage 137.0 (TID 845) (192.168.1.37, executor driver, partition 6, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO Executor: Running task 6.0 in stage 137.0 (TID 845)
22/02/07 23:47:23 INFO TaskSetManager: Finished task 1.0 in stage 137.0 (TID 840) in 6 ms on 192.168.1.37 (executor driver) (3/10)
22/02/07 23:47:23 INFO SparkContext: Starting job: collect at SparkRejectUpdateStrategy.java:52
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 2 (552.0 B) non-empty blocks including 2 (552.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO DAGScheduler: Registering RDD 275 (distinct at SparkRejectUpdateStrategy.java:52) as input to shuffle 37
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 2 (552.0 B) non-empty blocks including 2 (552.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO DAGScheduler: Got job 81 (collect at SparkRejectUpdateStrategy.java:52) with 1 output partitions
22/02/07 23:47:23 INFO DAGScheduler: Final stage: ResultStage 141 (collect at SparkRejectUpdateStrategy.java:52)
22/02/07 23:47:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 140)
22/02/07 23:47:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 140)
22/02/07 23:47:23 INFO DAGScheduler: Submitting ShuffleMapStage 140 (MapPartitionsRDD[275] at distinct at SparkRejectUpdateStrategy.java:52), which has no missing parents
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_111 stored as values in memory (estimated size 13.5 KiB, free 984.9 MiB)
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_111_piece0 stored as bytes in memory (estimated size 5.5 KiB, free 984.9 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Added broadcast_111_piece0 in memory on 192.168.1.37:53037 (size: 5.5 KiB, free: 987.0 MiB)
22/02/07 23:47:23 INFO SparkContext: Created broadcast 111 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:23 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 140 (MapPartitionsRDD[275] at distinct at SparkRejectUpdateStrategy.java:52) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:23 INFO TaskSchedulerImpl: Adding task set 140.0 with 1 tasks resource profile 0
22/02/07 23:47:23 INFO Executor: Finished task 6.0 in stage 137.0 (TID 845). 2034 bytes result sent to driver
22/02/07 23:47:23 INFO Executor: Finished task 4.0 in stage 137.0 (TID 843). 1673 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 8.0 in stage 137.0 (TID 846) (192.168.1.37, executor driver, partition 8, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO Executor: Running task 8.0 in stage 137.0 (TID 846)
22/02/07 23:47:23 INFO TaskSetManager: Starting task 9.0 in stage 137.0 (TID 847) (192.168.1.37, executor driver, partition 9, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 6.0 in stage 137.0 (TID 845) in 5 ms on 192.168.1.37 (executor driver) (4/10)
22/02/07 23:47:23 INFO Executor: Running task 9.0 in stage 137.0 (TID 847)
22/02/07 23:47:23 INFO TaskSetManager: Finished task 4.0 in stage 137.0 (TID 843) in 5 ms on 192.168.1.37 (executor driver) (5/10)
22/02/07 23:47:23 INFO Executor: Finished task 5.0 in stage 137.0 (TID 844). 2034 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 0.0 in stage 137.0 (TID 848) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 5.0 in stage 137.0 (TID 844) in 6 ms on 192.168.1.37 (executor driver) (6/10)
22/02/07 23:47:23 INFO Executor: Running task 0.0 in stage 137.0 (TID 848)
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 1 (276.0 B) non-empty blocks including 1 (276.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO Executor: Finished task 0.0 in stage 137.0 (TID 848). 1140 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 7.0 in stage 137.0 (TID 849) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 0.0 in stage 137.0 (TID 848) in 1 ms on 192.168.1.37 (executor driver) (7/10)
22/02/07 23:47:23 INFO Executor: Running task 7.0 in stage 137.0 (TID 849)
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO Executor: Finished task 7.0 in stage 137.0 (TID 849). 1140 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Starting task 0.0 in stage 140.0 (TID 850) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 6663 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO TaskSetManager: Finished task 7.0 in stage 137.0 (TID 849) in 2 ms on 192.168.1.37 (executor driver) (8/10)
22/02/07 23:47:23 INFO Executor: Running task 0.0 in stage 140.0 (TID 850)
22/02/07 23:47:23 INFO Executor: Finished task 8.0 in stage 137.0 (TID 846). 1673 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 8.0 in stage 137.0 (TID 846) in 5 ms on 192.168.1.37 (executor driver) (9/10)
22/02/07 23:47:23 INFO Executor: Finished task 9.0 in stage 137.0 (TID 847). 1673 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 9.0 in stage 137.0 (TID 847) in 5 ms on 192.168.1.37 (executor driver) (10/10)
22/02/07 23:47:23 INFO TaskSchedulerImpl: Removed TaskSet 137.0, whose tasks have all completed, from pool
22/02/07 23:47:23 INFO DAGScheduler: ResultStage 137 (collect at HoodieSparkEngineContext.java:122) finished in 0.017 s
22/02/07 23:47:23 INFO DAGScheduler: Job 79 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:23 INFO TaskSchedulerImpl: Killing all running tasks in stage 137: Stage finished
22/02/07 23:47:23 INFO DAGScheduler: Job 79 finished: collect at HoodieSparkEngineContext.java:122, took 0.432248 s
22/02/07 23:47:23 INFO BlockManager: Found block rdd_269_0 locally
22/02/07 23:47:23 INFO Executor: Finished task 0.0 in stage 140.0 (TID 850). 1172 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 0.0 in stage 140.0 (TID 850) in 5 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:23 INFO TaskSchedulerImpl: Removed TaskSet 140.0, whose tasks have all completed, from pool
22/02/07 23:47:23 INFO DAGScheduler: ShuffleMapStage 140 (distinct at SparkRejectUpdateStrategy.java:52) finished in 0.012 s
22/02/07 23:47:23 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:23 INFO DAGScheduler: running: Set()
22/02/07 23:47:23 INFO DAGScheduler: waiting: Set(ResultStage 141)
22/02/07 23:47:23 INFO DAGScheduler: failed: Set()
22/02/07 23:47:23 INFO DAGScheduler: Submitting ResultStage 141 (MapPartitionsRDD[277] at distinct at SparkRejectUpdateStrategy.java:52), which has no missing parents
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_112 stored as values in memory (estimated size 6.5 KiB, free 984.9 MiB)
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_112_piece0 stored as bytes in memory (estimated size 3.5 KiB, free 984.9 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Added broadcast_112_piece0 in memory on 192.168.1.37:53037 (size: 3.5 KiB, free: 987.0 MiB)
22/02/07 23:47:23 INFO SparkContext: Created broadcast 112 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:23 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 141 (MapPartitionsRDD[277] at distinct at SparkRejectUpdateStrategy.java:52) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:23 INFO TaskSchedulerImpl: Adding task set 141.0 with 1 tasks resource profile 0
22/02/07 23:47:23 INFO TaskSetManager: Starting task 0.0 in stage 141.0 (TID 851) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO Executor: Running task 0.0 in stage 141.0 (TID 851)
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 1 (117.0 B) non-empty blocks including 1 (117.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO Executor: Finished task 0.0 in stage 141.0 (TID 851). 1249 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 0.0 in stage 141.0 (TID 851) in 4 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:23 INFO TaskSchedulerImpl: Removed TaskSet 141.0, whose tasks have all completed, from pool
22/02/07 23:47:23 INFO DAGScheduler: ResultStage 141 (collect at SparkRejectUpdateStrategy.java:52) finished in 0.006 s
22/02/07 23:47:23 INFO DAGScheduler: Job 81 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:23 INFO TaskSchedulerImpl: Killing all running tasks in stage 141: Stage finished
22/02/07 23:47:23 INFO DAGScheduler: Job 81 finished: collect at SparkRejectUpdateStrategy.java:52, took 0.019203 s
22/02/07 23:47:23 INFO UpsertPartitioner: AvgRecordSize => 1024
22/02/07 23:47:23 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:47:23 INFO TransactionManager: Transaction starting for Optional.empty with latest completed transaction instant Optional.empty
22/02/07 23:47:23 INFO InProcessLockProvider: Thread pool-26-thread-1 ACQUIRING in-process lock.
22/02/07 23:47:23 INFO SparkContext: Starting job: collectAsMap at UpsertPartitioner.java:248
22/02/07 23:47:23 INFO DAGScheduler: Got job 82 (collectAsMap at UpsertPartitioner.java:248) with 1 output partitions
22/02/07 23:47:23 INFO DAGScheduler: Final stage: ResultStage 142 (collectAsMap at UpsertPartitioner.java:248)
22/02/07 23:47:23 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:23 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:23 INFO DAGScheduler: Submitting ResultStage 142 (MapPartitionsRDD[279] at mapToPair at UpsertPartitioner.java:247), which has no missing parents
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_113 stored as values in memory (estimated size 319.6 KiB, free 984.6 MiB)
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_113_piece0 stored as bytes in memory (estimated size 111.6 KiB, free 984.5 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Added broadcast_113_piece0 in memory on 192.168.1.37:53037 (size: 111.6 KiB, free: 986.8 MiB)
22/02/07 23:47:23 INFO SparkContext: Created broadcast 113 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:23 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 142 (MapPartitionsRDD[279] at mapToPair at UpsertPartitioner.java:247) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:23 INFO TaskSchedulerImpl: Adding task set 142.0 with 1 tasks resource profile 0
22/02/07 23:47:23 INFO TaskSetManager: Starting task 0.0 in stage 142.0 (TID 852) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4338 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO Executor: Running task 0.0 in stage 142.0 (TID 852)
22/02/07 23:47:23 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:23 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:23 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:23 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:23 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:23 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:23 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=30, NumFileGroups=1, FileGroupsCreationTime=7, StoreTimeTaken=0
22/02/07 23:47:23 INFO Executor: Finished task 0.0 in stage 142.0 (TID 852). 829 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 0.0 in stage 142.0 (TID 852) in 29 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:23 INFO TaskSchedulerImpl: Removed TaskSet 142.0, whose tasks have all completed, from pool
22/02/07 23:47:23 INFO DAGScheduler: ResultStage 142 (collectAsMap at UpsertPartitioner.java:248) finished in 0.085 s
22/02/07 23:47:23 INFO DAGScheduler: Job 82 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:23 INFO TaskSchedulerImpl: Killing all running tasks in stage 142: Stage finished
22/02/07 23:47:23 INFO DAGScheduler: Job 82 finished: collectAsMap at UpsertPartitioner.java:248, took 0.086061 s
22/02/07 23:47:23 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:23 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:23 INFO UpsertPartitioner: Total Buckets :1, buckets info => {0=BucketInfo {bucketType=UPDATE, fileIdPrefix=files-0000, partitionPath=files}},
Partition to insert buckets => {},
UpdateLocations mapped to buckets =>{files-0000=0}
22/02/07 23:47:23 INFO BaseSparkCommitActionExecutor: no validators configured.
22/02/07 23:47:23 INFO BaseCommitActionExecutor: Auto commit enabled: Committing 20220207234318886
22/02/07 23:47:23 INFO SparkContext: Starting job: collect at BaseSparkCommitActionExecutor.java:286
22/02/07 23:47:23 INFO DAGScheduler: Registering RDD 280 (mapToPair at BaseSparkCommitActionExecutor.java:233) as input to shuffle 38
22/02/07 23:47:23 INFO DAGScheduler: Got job 83 (collect at BaseSparkCommitActionExecutor.java:286) with 1 output partitions
22/02/07 23:47:23 INFO DAGScheduler: Final stage: ResultStage 144 (collect at BaseSparkCommitActionExecutor.java:286)
22/02/07 23:47:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 143)
22/02/07 23:47:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 143)
22/02/07 23:47:23 INFO DAGScheduler: Submitting ShuffleMapStage 143 (MapPartitionsRDD[280] at mapToPair at BaseSparkCommitActionExecutor.java:233), which has no missing parents
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_114 stored as values in memory (estimated size 327.7 KiB, free 984.1 MiB)
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_114_piece0 stored as bytes in memory (estimated size 117.1 KiB, free 984.0 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Added broadcast_114_piece0 in memory on 192.168.1.37:53037 (size: 117.1 KiB, free: 986.7 MiB)
22/02/07 23:47:23 INFO SparkContext: Created broadcast 114 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:23 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 143 (MapPartitionsRDD[280] at mapToPair at BaseSparkCommitActionExecutor.java:233) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:23 INFO TaskSchedulerImpl: Adding task set 143.0 with 1 tasks resource profile 0
22/02/07 23:47:23 INFO TaskSetManager: Starting task 0.0 in stage 143.0 (TID 853) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 6663 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO Executor: Running task 0.0 in stage 143.0 (TID 853)
22/02/07 23:47:23 INFO BlockManager: Found block rdd_269_0 locally
22/02/07 23:47:23 INFO Executor: Finished task 0.0 in stage 143.0 (TID 853). 1043 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 0.0 in stage 143.0 (TID 853) in 18 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:23 INFO TaskSchedulerImpl: Removed TaskSet 143.0, whose tasks have all completed, from pool
22/02/07 23:47:23 INFO DAGScheduler: ShuffleMapStage 143 (mapToPair at BaseSparkCommitActionExecutor.java:233) finished in 0.087 s
22/02/07 23:47:23 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:23 INFO DAGScheduler: running: Set()
22/02/07 23:47:23 INFO DAGScheduler: waiting: Set(ResultStage 144)
22/02/07 23:47:23 INFO DAGScheduler: failed: Set()
22/02/07 23:47:23 INFO DAGScheduler: Submitting ResultStage 144 (MapPartitionsRDD[285] at map at BaseSparkCommitActionExecutor.java:286), which has no missing parents
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_115 stored as values in memory (estimated size 432.7 KiB, free 983.6 MiB)
22/02/07 23:47:23 INFO MemoryStore: Block broadcast_115_piece0 stored as bytes in memory (estimated size 151.7 KiB, free 983.5 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Added broadcast_115_piece0 in memory on 192.168.1.37:53037 (size: 151.7 KiB, free: 986.6 MiB)
22/02/07 23:47:23 INFO SparkContext: Created broadcast 115 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:23 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 144 (MapPartitionsRDD[285] at map at BaseSparkCommitActionExecutor.java:286) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:23 INFO TaskSchedulerImpl: Adding task set 144.0 with 1 tasks resource profile 0
22/02/07 23:47:23 INFO TaskSetManager: Starting task 0.0 in stage 144.0 (TID 854) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:23 INFO Executor: Running task 0.0 in stage 144.0 (TID 854)
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Getting 1 (1692.0 B) non-empty blocks including 1 (1692.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:23 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220207234318886 for file files-0000
22/02/07 23:47:23 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:23 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:23 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:23 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:23 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:23 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:23 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=30, NumFileGroups=1, FileGroupsCreationTime=13, StoreTimeTaken=0
22/02/07 23:47:23 INFO DirectWriteMarkers: Creating Marker Path=/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207234318886/files/files-0000_0-144-854_00000000000000.hfile.marker.APPEND
22/02/07 23:47:23 INFO DirectWriteMarkers: [direct] Created marker file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207234318886/files/files-0000_0-144-854_00000000000000.hfile.marker.APPEND in 56 ms
22/02/07 23:47:23 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer
22/02/07 23:47:23 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.28_0-129-823
22/02/07 23:47:23 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.29_0-144-854', fileLen=-1}
22/02/07 23:47:23 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:47:23 INFO CodecPool: Got brand-new compressor [.gz]
22/02/07 23:47:23 INFO CodecPool: Got brand-new compressor [.gz]
22/02/07 23:47:23 INFO HoodieAppendHandle: AppendHandle for partitionPath files filePath files/.files-0000_00000000000000.log.29_0-144-854, took 107 ms.
22/02/07 23:47:23 INFO MemoryStore: Block rdd_284_0 stored as values in memory (estimated size 1635.0 B, free 983.5 MiB)
22/02/07 23:47:23 INFO BlockManagerInfo: Added rdd_284_0 in memory on 192.168.1.37:53037 (size: 1635.0 B, free: 986.6 MiB)
22/02/07 23:47:23 INFO Executor: Finished task 0.0 in stage 144.0 (TID 854). 2789 bytes result sent to driver
22/02/07 23:47:23 INFO TaskSetManager: Finished task 0.0 in stage 144.0 (TID 854) in 148 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:23 INFO TaskSchedulerImpl: Removed TaskSet 144.0, whose tasks have all completed, from pool
22/02/07 23:47:23 INFO DAGScheduler: ResultStage 144 (collect at BaseSparkCommitActionExecutor.java:286) finished in 0.219 s
22/02/07 23:47:23 INFO DAGScheduler: Job 83 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:23 INFO TaskSchedulerImpl: Killing all running tasks in stage 144: Stage finished
22/02/07 23:47:23 INFO DAGScheduler: Job 83 finished: collect at BaseSparkCommitActionExecutor.java:286, took 0.310207 s
22/02/07 23:47:23 INFO CommitUtils: Creating  metadata for UPSERT_PREPPED numWriteStats:1numReplaceFileIds:0
22/02/07 23:47:24 INFO SparkContext: Starting job: collect at BaseSparkCommitActionExecutor.java:294
22/02/07 23:47:24 INFO DAGScheduler: Got job 84 (collect at BaseSparkCommitActionExecutor.java:294) with 1 output partitions
22/02/07 23:47:24 INFO DAGScheduler: Final stage: ResultStage 146 (collect at BaseSparkCommitActionExecutor.java:294)
22/02/07 23:47:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 145)
22/02/07 23:47:24 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:24 INFO DAGScheduler: Submitting ResultStage 146 (MapPartitionsRDD[286] at map at BaseSparkCommitActionExecutor.java:294), which has no missing parents
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_116 stored as values in memory (estimated size 432.7 KiB, free 983.0 MiB)
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_116_piece0 stored as bytes in memory (estimated size 151.7 KiB, free 982.9 MiB)
22/02/07 23:47:24 INFO BlockManagerInfo: Added broadcast_116_piece0 in memory on 192.168.1.37:53037 (size: 151.7 KiB, free: 986.4 MiB)
22/02/07 23:47:24 INFO SparkContext: Created broadcast 116 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:24 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 146 (MapPartitionsRDD[286] at map at BaseSparkCommitActionExecutor.java:294) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:24 INFO TaskSchedulerImpl: Adding task set 146.0 with 1 tasks resource profile 0
22/02/07 23:47:24 INFO TaskSetManager: Starting task 0.0 in stage 146.0 (TID 855) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:24 INFO Executor: Running task 0.0 in stage 146.0 (TID 855)
22/02/07 23:47:24 INFO BlockManager: Found block rdd_284_0 locally
22/02/07 23:47:24 INFO Executor: Finished task 0.0 in stage 146.0 (TID 855). 2402 bytes result sent to driver
22/02/07 23:47:24 INFO TaskSetManager: Finished task 0.0 in stage 146.0 (TID 855) in 19 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:24 INFO TaskSchedulerImpl: Removed TaskSet 146.0, whose tasks have all completed, from pool
22/02/07 23:47:24 INFO DAGScheduler: ResultStage 146 (collect at BaseSparkCommitActionExecutor.java:294) finished in 0.084 s
22/02/07 23:47:24 INFO DAGScheduler: Job 84 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:24 INFO TaskSchedulerImpl: Killing all running tasks in stage 146: Stage finished
22/02/07 23:47:24 INFO DAGScheduler: Job 84 finished: collect at BaseSparkCommitActionExecutor.java:294, took 0.085445 s
22/02/07 23:47:24 INFO BaseSparkCommitActionExecutor: Committing 20220207234318886, action Type deltacommit, operation Type UPSERT_PREPPED
22/02/07 23:47:24 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/07 23:47:24 INFO DAGScheduler: Got job 85 (collect at HoodieSparkEngineContext.java:134) with 1 output partitions
22/02/07 23:47:24 INFO DAGScheduler: Final stage: ResultStage 147 (collect at HoodieSparkEngineContext.java:134)
22/02/07 23:47:24 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:24 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:24 INFO DAGScheduler: Submitting ResultStage 147 (MapPartitionsRDD[288] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_117 stored as values in memory (estimated size 99.2 KiB, free 982.8 MiB)
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_117_piece0 stored as bytes in memory (estimated size 35.1 KiB, free 982.8 MiB)
22/02/07 23:47:24 INFO BlockManagerInfo: Added broadcast_117_piece0 in memory on 192.168.1.37:53037 (size: 35.1 KiB, free: 986.4 MiB)
22/02/07 23:47:24 INFO SparkContext: Created broadcast 117 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:24 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 147 (MapPartitionsRDD[288] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:24 INFO TaskSchedulerImpl: Adding task set 147.0 with 1 tasks resource profile 0
22/02/07 23:47:24 INFO TaskSetManager: Starting task 0.0 in stage 147.0 (TID 856) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4456 bytes) taskResourceAssignments Map()
22/02/07 23:47:24 INFO Executor: Running task 0.0 in stage 147.0 (TID 856)
22/02/07 23:47:24 INFO Executor: Finished task 0.0 in stage 147.0 (TID 856). 796 bytes result sent to driver
22/02/07 23:47:24 INFO TaskSetManager: Finished task 0.0 in stage 147.0 (TID 856) in 13 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:24 INFO TaskSchedulerImpl: Removed TaskSet 147.0, whose tasks have all completed, from pool
22/02/07 23:47:24 INFO DAGScheduler: ResultStage 147 (collect at HoodieSparkEngineContext.java:134) finished in 0.031 s
22/02/07 23:47:24 INFO DAGScheduler: Job 85 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:24 INFO TaskSchedulerImpl: Killing all running tasks in stage 147: Stage finished
22/02/07 23:47:24 INFO DAGScheduler: Job 85 finished: collect at HoodieSparkEngineContext.java:134, took 0.031981 s
22/02/07 23:47:24 INFO HoodieActiveTimeline: Marking instant complete [==>20220207234318886__deltacommit__INFLIGHT]
22/02/07 23:47:24 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234318886.deltacommit.inflight
22/02/07 23:47:24 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234318886.deltacommit
22/02/07 23:47:24 INFO HoodieActiveTimeline: Completed [==>20220207234318886__deltacommit__INFLIGHT]
22/02/07 23:47:24 INFO BaseSparkCommitActionExecutor: Committed 20220207234318886
22/02/07 23:47:24 INFO SparkContext: Starting job: collectAsMap at HoodieSparkEngineContext.java:148
22/02/07 23:47:24 INFO DAGScheduler: Got job 86 (collectAsMap at HoodieSparkEngineContext.java:148) with 1 output partitions
22/02/07 23:47:24 INFO DAGScheduler: Final stage: ResultStage 148 (collectAsMap at HoodieSparkEngineContext.java:148)
22/02/07 23:47:24 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:24 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:24 INFO DAGScheduler: Submitting ResultStage 148 (MapPartitionsRDD[290] at mapToPair at HoodieSparkEngineContext.java:145), which has no missing parents
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_118 stored as values in memory (estimated size 99.4 KiB, free 982.7 MiB)
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_118_piece0 stored as bytes in memory (estimated size 35.1 KiB, free 982.6 MiB)
22/02/07 23:47:24 INFO BlockManagerInfo: Added broadcast_118_piece0 in memory on 192.168.1.37:53037 (size: 35.1 KiB, free: 986.4 MiB)
22/02/07 23:47:24 INFO SparkContext: Created broadcast 118 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:24 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 148 (MapPartitionsRDD[290] at mapToPair at HoodieSparkEngineContext.java:145) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:24 INFO TaskSchedulerImpl: Adding task set 148.0 with 1 tasks resource profile 0
22/02/07 23:47:24 INFO TaskSetManager: Starting task 0.0 in stage 148.0 (TID 857) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4456 bytes) taskResourceAssignments Map()
22/02/07 23:47:24 INFO Executor: Running task 0.0 in stage 148.0 (TID 857)
22/02/07 23:47:24 INFO Executor: Finished task 0.0 in stage 148.0 (TID 857). 883 bytes result sent to driver
22/02/07 23:47:24 INFO TaskSetManager: Finished task 0.0 in stage 148.0 (TID 857) in 5 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:24 INFO TaskSchedulerImpl: Removed TaskSet 148.0, whose tasks have all completed, from pool
22/02/07 23:47:24 INFO DAGScheduler: ResultStage 148 (collectAsMap at HoodieSparkEngineContext.java:148) finished in 0.023 s
22/02/07 23:47:24 INFO DAGScheduler: Job 86 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:24 INFO TaskSchedulerImpl: Killing all running tasks in stage 148: Stage finished
22/02/07 23:47:24 INFO DAGScheduler: Job 86 finished: collectAsMap at HoodieSparkEngineContext.java:148, took 0.024730 s
22/02/07 23:47:24 INFO FSUtils: Removed directory at /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/.temp/20220207234318886
22/02/07 23:47:24 INFO HoodieHeartbeatClient: Stopping heartbeat for instant 20220207234318886
22/02/07 23:47:24 INFO HoodieHeartbeatClient: Stopped heartbeat for instant 20220207234318886
22/02/07 23:47:24 INFO HeartbeatUtils: Deleted the heartbeat for instant 20220207234318886
22/02/07 23:47:24 INFO HoodieHeartbeatClient: Deleted heartbeat file for instant 20220207234318886
22/02/07 23:47:24 INFO SparkContext: Starting job: collect at SparkHoodieBackedTableMetadataWriter.java:154
22/02/07 23:47:24 INFO DAGScheduler: Got job 87 (collect at SparkHoodieBackedTableMetadataWriter.java:154) with 1 output partitions
22/02/07 23:47:24 INFO DAGScheduler: Final stage: ResultStage 150 (collect at SparkHoodieBackedTableMetadataWriter.java:154)
22/02/07 23:47:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 149)
22/02/07 23:47:24 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:24 INFO DAGScheduler: Submitting ResultStage 150 (MapPartitionsRDD[284] at flatMap at BaseSparkCommitActionExecutor.java:181), which has no missing parents
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_119 stored as values in memory (estimated size 432.3 KiB, free 982.2 MiB)
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_119_piece0 stored as bytes in memory (estimated size 151.7 KiB, free 982.1 MiB)
22/02/07 23:47:24 INFO BlockManagerInfo: Added broadcast_119_piece0 in memory on 192.168.1.37:53037 (size: 151.7 KiB, free: 986.2 MiB)
22/02/07 23:47:24 INFO SparkContext: Created broadcast 119 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:24 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 150 (MapPartitionsRDD[284] at flatMap at BaseSparkCommitActionExecutor.java:181) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:24 INFO TaskSchedulerImpl: Adding task set 150.0 with 1 tasks resource profile 0
22/02/07 23:47:24 INFO TaskSetManager: Starting task 0.0 in stage 150.0 (TID 858) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:24 INFO Executor: Running task 0.0 in stage 150.0 (TID 858)
22/02/07 23:47:24 INFO BlockManager: Found block rdd_284_0 locally
22/02/07 23:47:24 INFO Executor: Finished task 0.0 in stage 150.0 (TID 858). 2482 bytes result sent to driver
22/02/07 23:47:24 INFO TaskSetManager: Finished task 0.0 in stage 150.0 (TID 858) in 19 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:24 INFO TaskSchedulerImpl: Removed TaskSet 150.0, whose tasks have all completed, from pool
22/02/07 23:47:24 INFO DAGScheduler: ResultStage 150 (collect at SparkHoodieBackedTableMetadataWriter.java:154) finished in 0.081 s
22/02/07 23:47:24 INFO DAGScheduler: Job 87 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:24 INFO TaskSchedulerImpl: Killing all running tasks in stage 150: Stage finished
22/02/07 23:47:24 INFO DAGScheduler: Job 87 finished: collect at SparkHoodieBackedTableMetadataWriter.java:154, took 0.082850 s
22/02/07 23:47:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:24 INFO BaseHoodieWriteClient: Scheduling cleaning at instant time :20220207234318886002
22/02/07 23:47:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:24 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:24 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:24 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:24 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:24 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:24 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:24 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:24 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:100
22/02/07 23:47:24 INFO DAGScheduler: Got job 88 (collect at HoodieSparkEngineContext.java:100) with 1 output partitions
22/02/07 23:47:24 INFO DAGScheduler: Final stage: ResultStage 151 (collect at HoodieSparkEngineContext.java:100)
22/02/07 23:47:24 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:24 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:24 INFO DAGScheduler: Submitting ResultStage 151 (MapPartitionsRDD[292] at map at HoodieSparkEngineContext.java:100), which has no missing parents
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_120 stored as values in memory (estimated size 99.3 KiB, free 982.0 MiB)
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_120_piece0 stored as bytes in memory (estimated size 35.4 KiB, free 981.9 MiB)
22/02/07 23:47:24 INFO BlockManagerInfo: Added broadcast_120_piece0 in memory on 192.168.1.37:53037 (size: 35.4 KiB, free: 986.2 MiB)
22/02/07 23:47:24 INFO SparkContext: Created broadcast 120 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:24 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 151 (MapPartitionsRDD[292] at map at HoodieSparkEngineContext.java:100) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:24 INFO TaskSchedulerImpl: Adding task set 151.0 with 1 tasks resource profile 0
22/02/07 23:47:24 INFO TaskSetManager: Starting task 0.0 in stage 151.0 (TID 859) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4440 bytes) taskResourceAssignments Map()
22/02/07 23:47:24 INFO Executor: Running task 0.0 in stage 151.0 (TID 859)
22/02/07 23:47:24 INFO Executor: Finished task 0.0 in stage 151.0 (TID 859). 1239 bytes result sent to driver
22/02/07 23:47:24 INFO TaskSetManager: Finished task 0.0 in stage 151.0 (TID 859) in 19 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:24 INFO TaskSchedulerImpl: Removed TaskSet 151.0, whose tasks have all completed, from pool
22/02/07 23:47:24 INFO DAGScheduler: ResultStage 151 (collect at HoodieSparkEngineContext.java:100) finished in 0.038 s
22/02/07 23:47:24 INFO DAGScheduler: Job 88 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:24 INFO TaskSchedulerImpl: Killing all running tasks in stage 151: Stage finished
22/02/07 23:47:24 INFO DAGScheduler: Job 88 finished: collect at HoodieSparkEngineContext.java:100, took 0.038960 s
22/02/07 23:47:24 INFO CleanPlanner: Total Partitions to clean : 1, with policy KEEP_LATEST_COMMITS
22/02/07 23:47:24 INFO CleanPlanner: Using cleanerParallelism: 1
22/02/07 23:47:24 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:100
22/02/07 23:47:24 INFO DAGScheduler: Got job 89 (collect at HoodieSparkEngineContext.java:100) with 1 output partitions
22/02/07 23:47:24 INFO DAGScheduler: Final stage: ResultStage 152 (collect at HoodieSparkEngineContext.java:100)
22/02/07 23:47:24 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:24 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:24 INFO DAGScheduler: Submitting ResultStage 152 (MapPartitionsRDD[294] at map at HoodieSparkEngineContext.java:100), which has no missing parents
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_121 stored as values in memory (estimated size 322.8 KiB, free 981.6 MiB)
22/02/07 23:47:24 INFO MemoryStore: Block broadcast_121_piece0 stored as bytes in memory (estimated size 113.1 KiB, free 981.5 MiB)
22/02/07 23:47:24 INFO BlockManagerInfo: Added broadcast_121_piece0 in memory on 192.168.1.37:53037 (size: 113.1 KiB, free: 986.1 MiB)
22/02/07 23:47:24 INFO SparkContext: Created broadcast 121 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:24 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 152 (MapPartitionsRDD[294] at map at HoodieSparkEngineContext.java:100) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:24 INFO TaskSchedulerImpl: Adding task set 152.0 with 1 tasks resource profile 0
22/02/07 23:47:24 INFO TaskSetManager: Starting task 0.0 in stage 152.0 (TID 860) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4338 bytes) taskResourceAssignments Map()
22/02/07 23:47:24 INFO Executor: Running task 0.0 in stage 152.0 (TID 860)
22/02/07 23:47:24 INFO CleanPlanner: Cleaning files, retaining latest 3 commits.
22/02/07 23:47:24 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:24 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=31, NumFileGroups=1, FileGroupsCreationTime=8, StoreTimeTaken=0
22/02/07 23:47:24 INFO CleanPlanner: 0 patterns used to delete in partition path:files
22/02/07 23:47:24 INFO Executor: Finished task 0.0 in stage 152.0 (TID 860). 881 bytes result sent to driver
22/02/07 23:47:24 INFO TaskSetManager: Finished task 0.0 in stage 152.0 (TID 860) in 36 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:24 INFO TaskSchedulerImpl: Removed TaskSet 152.0, whose tasks have all completed, from pool
22/02/07 23:47:24 INFO DAGScheduler: ResultStage 152 (collect at HoodieSparkEngineContext.java:100) finished in 0.093 s
22/02/07 23:47:24 INFO DAGScheduler: Job 89 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:24 INFO TaskSchedulerImpl: Killing all running tasks in stage 152: Stage finished
22/02/07 23:47:24 INFO DAGScheduler: Job 89 finished: collect at HoodieSparkEngineContext.java:100, took 0.093479 s
22/02/07 23:47:24 INFO BaseHoodieWriteClient: Cleaner started
22/02/07 23:47:24 INFO BaseHoodieWriteClient: Cleaned failed attempts if any
22/02/07 23:47:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:24 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:24 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:24 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:24 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:24 INFO HoodieHeartbeatClient: Heartbeat not found in internal map, falling back to reading from DFS
22/02/07 23:47:24 WARN HoodieHeartbeatClient: Heartbeat expired, currentTime = 1644306444961, last heartbeat = Heartbeat{instantTime='20220207234247693', isHeartbeatStarted=false, isHeartbeatStopped=false, lastHeartbeatTime=1644306168000, numHeartbeats=0, timer=java.util.Timer@552f38f}, heartbeat interval = 60000
22/02/07 23:47:24 INFO HoodieHeartbeatClient: Heartbeat not found in internal map, falling back to reading from DFS
22/02/07 23:47:24 WARN HoodieHeartbeatClient: Heartbeat expired, currentTime = 1644306444962, last heartbeat = Heartbeat{instantTime='20220207234155839', isHeartbeatStarted=false, isHeartbeatStopped=false, lastHeartbeatTime=1644306116000, numHeartbeats=0, timer=java.util.Timer@3c1a13f5}, heartbeat interval = 60000
22/02/07 23:47:24 INFO HoodieHeartbeatClient: Heartbeat not found in internal map, falling back to reading from DFS
22/02/07 23:47:24 WARN HoodieHeartbeatClient: Heartbeat expired, currentTime = 1644306444962, last heartbeat = Heartbeat{instantTime='20220207233705644', isHeartbeatStarted=false, isHeartbeatStopped=false, lastHeartbeatTime=1644305874000, numHeartbeats=0, timer=java.util.Timer@30ba0cd9}, heartbeat interval = 60000
22/02/07 23:47:24 INFO BaseHoodieWriteClient: Begin rollback of instant 20220207234247693
22/02/07 23:47:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:24 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:24 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:24 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:24 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:24 INFO BaseHoodieWriteClient: Scheduling Rollback at instant time :20220207234724963
22/02/07 23:47:25 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:100
22/02/07 23:47:25 INFO DAGScheduler: Got job 90 (collect at HoodieSparkEngineContext.java:100) with 1 output partitions
22/02/07 23:47:25 INFO DAGScheduler: Final stage: ResultStage 153 (collect at HoodieSparkEngineContext.java:100)
22/02/07 23:47:25 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:25 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:25 INFO DAGScheduler: Submitting ResultStage 153 (MapPartitionsRDD[296] at map at HoodieSparkEngineContext.java:100), which has no missing parents
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_122 stored as values in memory (estimated size 99.3 KiB, free 981.4 MiB)
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_122_piece0 stored as bytes in memory (estimated size 35.4 KiB, free 981.4 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Added broadcast_122_piece0 in memory on 192.168.1.37:53037 (size: 35.4 KiB, free: 986.0 MiB)
22/02/07 23:47:25 INFO SparkContext: Created broadcast 122 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:25 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 153 (MapPartitionsRDD[296] at map at HoodieSparkEngineContext.java:100) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:25 INFO TaskSchedulerImpl: Adding task set 153.0 with 1 tasks resource profile 0
22/02/07 23:47:25 INFO TaskSetManager: Starting task 0.0 in stage 153.0 (TID 861) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4440 bytes) taskResourceAssignments Map()
22/02/07 23:47:25 INFO Executor: Running task 0.0 in stage 153.0 (TID 861)
22/02/07 23:47:25 INFO Executor: Finished task 0.0 in stage 153.0 (TID 861). 1239 bytes result sent to driver
22/02/07 23:47:25 INFO TaskSetManager: Finished task 0.0 in stage 153.0 (TID 861) in 11 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:25 INFO TaskSchedulerImpl: Removed TaskSet 153.0, whose tasks have all completed, from pool
22/02/07 23:47:25 INFO DAGScheduler: ResultStage 153 (collect at HoodieSparkEngineContext.java:100) finished in 0.034 s
22/02/07 23:47:25 INFO DAGScheduler: Job 90 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:25 INFO TaskSchedulerImpl: Killing all running tasks in stage 153: Stage finished
22/02/07 23:47:25 INFO DAGScheduler: Job 90 finished: collect at HoodieSparkEngineContext.java:100, took 0.034994 s
22/02/07 23:47:25 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/07 23:47:25 INFO DAGScheduler: Got job 91 (collect at HoodieSparkEngineContext.java:134) with 1 output partitions
22/02/07 23:47:25 INFO DAGScheduler: Final stage: ResultStage 154 (collect at HoodieSparkEngineContext.java:134)
22/02/07 23:47:25 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:25 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:25 INFO DAGScheduler: Submitting ResultStage 154 (MapPartitionsRDD[298] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_123 stored as values in memory (estimated size 318.4 KiB, free 981.1 MiB)
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_123_piece0 stored as bytes in memory (estimated size 111.1 KiB, free 981.0 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Added broadcast_123_piece0 in memory on 192.168.1.37:53037 (size: 111.1 KiB, free: 985.9 MiB)
22/02/07 23:47:25 INFO SparkContext: Created broadcast 123 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:25 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 154 (MapPartitionsRDD[298] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:25 INFO TaskSchedulerImpl: Adding task set 154.0 with 1 tasks resource profile 0
22/02/07 23:47:25 INFO TaskSetManager: Starting task 0.0 in stage 154.0 (TID 862) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4338 bytes) taskResourceAssignments Map()
22/02/07 23:47:25 INFO Executor: Running task 0.0 in stage 154.0 (TID 862)
22/02/07 23:47:25 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__deltacommit__COMPLETED]}
22/02/07 23:47:25 INFO Executor: Finished task 0.0 in stage 154.0 (TID 862). 878 bytes result sent to driver
22/02/07 23:47:25 INFO TaskSetManager: Finished task 0.0 in stage 154.0 (TID 862) in 31 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:25 INFO TaskSchedulerImpl: Removed TaskSet 154.0, whose tasks have all completed, from pool
22/02/07 23:47:25 INFO DAGScheduler: ResultStage 154 (collect at HoodieSparkEngineContext.java:134) finished in 0.085 s
22/02/07 23:47:25 INFO DAGScheduler: Job 91 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:25 INFO TaskSchedulerImpl: Killing all running tasks in stage 154: Stage finished
22/02/07 23:47:25 INFO DAGScheduler: Job 91 finished: collect at HoodieSparkEngineContext.java:134, took 0.085601 s
22/02/07 23:47:25 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:100
22/02/07 23:47:25 INFO DAGScheduler: Got job 92 (collect at HoodieSparkEngineContext.java:100) with 1 output partitions
22/02/07 23:47:25 INFO DAGScheduler: Final stage: ResultStage 155 (collect at HoodieSparkEngineContext.java:100)
22/02/07 23:47:25 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:25 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:25 INFO DAGScheduler: Submitting ResultStage 155 (MapPartitionsRDD[300] at map at HoodieSparkEngineContext.java:100), which has no missing parents
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_124 stored as values in memory (estimated size 128.4 KiB, free 980.8 MiB)
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_124_piece0 stored as bytes in memory (estimated size 45.8 KiB, free 980.8 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Added broadcast_124_piece0 in memory on 192.168.1.37:53037 (size: 45.8 KiB, free: 985.9 MiB)
22/02/07 23:47:25 INFO SparkContext: Created broadcast 124 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:25 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 155 (MapPartitionsRDD[300] at map at HoodieSparkEngineContext.java:100) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:25 INFO TaskSchedulerImpl: Adding task set 155.0 with 1 tasks resource profile 0
22/02/07 23:47:25 INFO TaskSetManager: Starting task 0.0 in stage 155.0 (TID 863) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4413 bytes) taskResourceAssignments Map()
22/02/07 23:47:25 INFO Executor: Running task 0.0 in stage 155.0 (TID 863)
22/02/07 23:47:25 INFO Executor: Finished task 0.0 in stage 155.0 (TID 863). 912 bytes result sent to driver
22/02/07 23:47:25 INFO TaskSetManager: Finished task 0.0 in stage 155.0 (TID 863) in 15 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:25 INFO TaskSchedulerImpl: Removed TaskSet 155.0, whose tasks have all completed, from pool
22/02/07 23:47:25 INFO DAGScheduler: ResultStage 155 (collect at HoodieSparkEngineContext.java:100) finished in 0.032 s
22/02/07 23:47:25 INFO DAGScheduler: Job 92 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:25 INFO TaskSchedulerImpl: Killing all running tasks in stage 155: Stage finished
22/02/07 23:47:25 INFO DAGScheduler: Job 92 finished: collect at HoodieSparkEngineContext.java:100, took 0.032461 s
22/02/07 23:47:25 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220207234724963__rollback__REQUESTED]}
22/02/07 23:47:25 INFO BaseRollbackPlanActionExecutor: Requesting Rollback with instant time [==>20220207234724963__rollback__REQUESTED]
22/02/07 23:47:25 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220207234724963__rollback__REQUESTED]}
22/02/07 23:47:25 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234724963.rollback.requested
22/02/07 23:47:25 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234724963.rollback.inflight
22/02/07 23:47:25 INFO MergeOnReadRollbackActionExecutor: Rolling back instant [==>20220207234247693__deltacommit__INFLIGHT]
22/02/07 23:47:25 INFO MergeOnReadRollbackActionExecutor: Unpublished [==>20220207234247693__deltacommit__INFLIGHT]
22/02/07 23:47:25 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/07 23:47:25 INFO DAGScheduler: Got job 93 (collect at HoodieSparkEngineContext.java:134) with 1 output partitions
22/02/07 23:47:25 INFO DAGScheduler: Final stage: ResultStage 156 (collect at HoodieSparkEngineContext.java:134)
22/02/07 23:47:25 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:25 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:25 INFO DAGScheduler: Submitting ResultStage 156 (MapPartitionsRDD[302] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_125 stored as values in memory (estimated size 128.7 KiB, free 980.7 MiB)
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_125_piece0 stored as bytes in memory (estimated size 46.0 KiB, free 980.6 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Added broadcast_125_piece0 in memory on 192.168.1.37:53037 (size: 46.0 KiB, free: 985.8 MiB)
22/02/07 23:47:25 INFO SparkContext: Created broadcast 125 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:25 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 156 (MapPartitionsRDD[302] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:25 INFO TaskSchedulerImpl: Adding task set 156.0 with 1 tasks resource profile 0
22/02/07 23:47:25 INFO TaskSetManager: Starting task 0.0 in stage 156.0 (TID 864) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4460 bytes) taskResourceAssignments Map()
22/02/07 23:47:25 INFO Executor: Running task 0.0 in stage 156.0 (TID 864)
22/02/07 23:47:25 INFO Executor: Finished task 0.0 in stage 156.0 (TID 864). 930 bytes result sent to driver
22/02/07 23:47:25 INFO TaskSetManager: Finished task 0.0 in stage 156.0 (TID 864) in 7 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:25 INFO TaskSchedulerImpl: Removed TaskSet 156.0, whose tasks have all completed, from pool
22/02/07 23:47:25 INFO DAGScheduler: ResultStage 156 (collect at HoodieSparkEngineContext.java:134) finished in 0.025 s
22/02/07 23:47:25 INFO DAGScheduler: Job 93 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:25 INFO TaskSchedulerImpl: Killing all running tasks in stage 156: Stage finished
22/02/07 23:47:25 INFO DAGScheduler: Job 93 finished: collect at HoodieSparkEngineContext.java:134, took 0.025762 s
22/02/07 23:47:25 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:129
22/02/07 23:47:25 INFO DAGScheduler: Registering RDD 304 (mapToPair at HoodieSparkEngineContext.java:128) as input to shuffle 39
22/02/07 23:47:25 INFO DAGScheduler: Got job 94 (collect at HoodieSparkEngineContext.java:129) with 1 output partitions
22/02/07 23:47:25 INFO DAGScheduler: Final stage: ResultStage 158 (collect at HoodieSparkEngineContext.java:129)
22/02/07 23:47:25 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 157)
22/02/07 23:47:25 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 157)
22/02/07 23:47:25 INFO DAGScheduler: Submitting ShuffleMapStage 157 (MapPartitionsRDD[304] at mapToPair at HoodieSparkEngineContext.java:128), which has no missing parents
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_126 stored as values in memory (estimated size 6.8 KiB, free 980.6 MiB)
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_126_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 980.6 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Added broadcast_126_piece0 in memory on 192.168.1.37:53037 (size: 3.8 KiB, free: 985.8 MiB)
22/02/07 23:47:25 INFO SparkContext: Created broadcast 126 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:25 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 157 (MapPartitionsRDD[304] at mapToPair at HoodieSparkEngineContext.java:128) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:25 INFO TaskSchedulerImpl: Adding task set 157.0 with 1 tasks resource profile 0
22/02/07 23:47:25 INFO TaskSetManager: Starting task 0.0 in stage 157.0 (TID 865) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4497 bytes) taskResourceAssignments Map()
22/02/07 23:47:25 INFO Executor: Running task 0.0 in stage 157.0 (TID 865)
22/02/07 23:47:25 INFO Executor: Finished task 0.0 in stage 157.0 (TID 865). 1043 bytes result sent to driver
22/02/07 23:47:25 INFO TaskSetManager: Finished task 0.0 in stage 157.0 (TID 865) in 5 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:25 INFO TaskSchedulerImpl: Removed TaskSet 157.0, whose tasks have all completed, from pool
22/02/07 23:47:25 INFO DAGScheduler: ShuffleMapStage 157 (mapToPair at HoodieSparkEngineContext.java:128) finished in 0.007 s
22/02/07 23:47:25 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:25 INFO DAGScheduler: running: Set()
22/02/07 23:47:25 INFO DAGScheduler: waiting: Set(ResultStage 158)
22/02/07 23:47:25 INFO DAGScheduler: failed: Set()
22/02/07 23:47:25 INFO DAGScheduler: Submitting ResultStage 158 (MapPartitionsRDD[306] at map at HoodieSparkEngineContext.java:129), which has no missing parents
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_127 stored as values in memory (estimated size 7.3 KiB, free 980.6 MiB)
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_127_piece0 stored as bytes in memory (estimated size 3.9 KiB, free 980.6 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Added broadcast_127_piece0 in memory on 192.168.1.37:53037 (size: 3.9 KiB, free: 985.8 MiB)
22/02/07 23:47:25 INFO SparkContext: Created broadcast 127 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:25 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 158 (MapPartitionsRDD[306] at map at HoodieSparkEngineContext.java:129) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:25 INFO TaskSchedulerImpl: Adding task set 158.0 with 1 tasks resource profile 0
22/02/07 23:47:25 INFO TaskSetManager: Starting task 0.0 in stage 158.0 (TID 866) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:25 INFO Executor: Running task 0.0 in stage 158.0 (TID 866)
22/02/07 23:47:25 INFO ShuffleBlockFetcherIterator: Getting 1 (156.0 B) non-empty blocks including 1 (156.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:25 INFO Executor: Finished task 0.0 in stage 158.0 (TID 866). 1303 bytes result sent to driver
22/02/07 23:47:25 INFO TaskSetManager: Finished task 0.0 in stage 158.0 (TID 866) in 84 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:25 INFO TaskSchedulerImpl: Removed TaskSet 158.0, whose tasks have all completed, from pool
22/02/07 23:47:25 INFO DAGScheduler: ResultStage 158 (collect at HoodieSparkEngineContext.java:129) finished in 0.087 s
22/02/07 23:47:25 INFO DAGScheduler: Job 94 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:25 INFO TaskSchedulerImpl: Killing all running tasks in stage 158: Stage finished
22/02/07 23:47:25 INFO DAGScheduler: Job 94 finished: collect at HoodieSparkEngineContext.java:129, took 0.096025 s
22/02/07 23:47:25 INFO BaseRollbackActionExecutor: Deleting instant=[==>20220207234247693__deltacommit__INFLIGHT]
22/02/07 23:47:25 INFO HoodieActiveTimeline: Deleting instant [==>20220207234247693__deltacommit__INFLIGHT]
22/02/07 23:47:25 INFO HoodieActiveTimeline: Removed instant [==>20220207234247693__deltacommit__INFLIGHT]
22/02/07 23:47:25 INFO HoodieActiveTimeline: Deleting instant [==>20220207234247693__deltacommit__REQUESTED]
22/02/07 23:47:25 INFO HoodieActiveTimeline: Removed instant [==>20220207234247693__deltacommit__REQUESTED]
22/02/07 23:47:25 INFO BaseRollbackActionExecutor: Deleted pending commit [==>20220207234247693__deltacommit__REQUESTED]
22/02/07 23:47:25 INFO MergeOnReadRollbackActionExecutor: Time(in ms) taken to finish rollback 185
22/02/07 23:47:25 INFO BaseRollbackActionExecutor: Rolled back inflight instant 20220207234247693
22/02/07 23:47:25 INFO BaseRollbackActionExecutor: Index rolled back for commits [==>20220207234247693__deltacommit__INFLIGHT]
22/02/07 23:47:25 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234724963.rollback.inflight
22/02/07 23:47:25 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234724963.rollback
22/02/07 23:47:25 INFO BaseRollbackActionExecutor: Rollback of Commits [20220207234247693] is complete
22/02/07 23:47:25 INFO HeartbeatUtils: Deleted the heartbeat for instant 20220207234247693
22/02/07 23:47:25 INFO BaseHoodieWriteClient: Begin rollback of instant 20220207234155839
22/02/07 23:47:25 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:25 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:25 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:25 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:25 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234724963__rollback__COMPLETED]}
22/02/07 23:47:25 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:25 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:25 INFO BaseHoodieWriteClient: Scheduling Rollback at instant time :20220207234725651
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_110_piece0 on 192.168.1.37:53037 in memory (size: 4.0 KiB, free: 985.8 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_119_piece0 on 192.168.1.37:53037 in memory (size: 151.7 KiB, free: 986.0 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_107_piece0 on 192.168.1.37:53037 in memory (size: 214.0 KiB, free: 986.2 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_116_piece0 on 192.168.1.37:53037 in memory (size: 151.7 KiB, free: 986.3 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_117_piece0 on 192.168.1.37:53037 in memory (size: 35.1 KiB, free: 986.4 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_121_piece0 on 192.168.1.37:53037 in memory (size: 113.1 KiB, free: 986.5 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_113_piece0 on 192.168.1.37:53037 in memory (size: 111.6 KiB, free: 986.6 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_111_piece0 on 192.168.1.37:53037 in memory (size: 5.5 KiB, free: 986.6 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_115_piece0 on 192.168.1.37:53037 in memory (size: 151.7 KiB, free: 986.7 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_118_piece0 on 192.168.1.37:53037 in memory (size: 35.1 KiB, free: 986.8 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_108_piece0 on 192.168.1.37:53037 in memory (size: 5.6 KiB, free: 986.8 MiB)
22/02/07 23:47:25 INFO BlockManager: Removing RDD 284
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_126_piece0 on 192.168.1.37:53037 in memory (size: 3.8 KiB, free: 986.8 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_122_piece0 on 192.168.1.37:53037 in memory (size: 35.4 KiB, free: 986.8 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_125_piece0 on 192.168.1.37:53037 in memory (size: 46.0 KiB, free: 986.9 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_114_piece0 on 192.168.1.37:53037 in memory (size: 117.1 KiB, free: 987.0 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_112_piece0 on 192.168.1.37:53037 in memory (size: 3.5 KiB, free: 987.0 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_109_piece0 on 192.168.1.37:53037 in memory (size: 3.2 KiB, free: 987.0 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_120_piece0 on 192.168.1.37:53037 in memory (size: 35.4 KiB, free: 987.0 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_123_piece0 on 192.168.1.37:53037 in memory (size: 111.1 KiB, free: 987.1 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Removed broadcast_124_piece0 on 192.168.1.37:53037 in memory (size: 45.8 KiB, free: 987.2 MiB)
22/02/07 23:47:25 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:100
22/02/07 23:47:25 INFO DAGScheduler: Got job 95 (collect at HoodieSparkEngineContext.java:100) with 1 output partitions
22/02/07 23:47:25 INFO DAGScheduler: Final stage: ResultStage 159 (collect at HoodieSparkEngineContext.java:100)
22/02/07 23:47:25 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:25 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:25 INFO DAGScheduler: Submitting ResultStage 159 (MapPartitionsRDD[308] at map at HoodieSparkEngineContext.java:100), which has no missing parents
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_128 stored as values in memory (estimated size 99.3 KiB, free 985.7 MiB)
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_128_piece0 stored as bytes in memory (estimated size 35.4 KiB, free 985.6 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Added broadcast_128_piece0 in memory on 192.168.1.37:53037 (size: 35.4 KiB, free: 987.1 MiB)
22/02/07 23:47:25 INFO SparkContext: Created broadcast 128 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:25 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 159 (MapPartitionsRDD[308] at map at HoodieSparkEngineContext.java:100) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:25 INFO TaskSchedulerImpl: Adding task set 159.0 with 1 tasks resource profile 0
22/02/07 23:47:25 INFO TaskSetManager: Starting task 0.0 in stage 159.0 (TID 867) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4440 bytes) taskResourceAssignments Map()
22/02/07 23:47:25 INFO Executor: Running task 0.0 in stage 159.0 (TID 867)
22/02/07 23:47:25 INFO Executor: Finished task 0.0 in stage 159.0 (TID 867). 1239 bytes result sent to driver
22/02/07 23:47:25 INFO TaskSetManager: Finished task 0.0 in stage 159.0 (TID 867) in 6 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:25 INFO TaskSchedulerImpl: Removed TaskSet 159.0, whose tasks have all completed, from pool
22/02/07 23:47:25 INFO DAGScheduler: ResultStage 159 (collect at HoodieSparkEngineContext.java:100) finished in 0.024 s
22/02/07 23:47:25 INFO DAGScheduler: Job 95 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:25 INFO TaskSchedulerImpl: Killing all running tasks in stage 159: Stage finished
22/02/07 23:47:25 INFO DAGScheduler: Job 95 finished: collect at HoodieSparkEngineContext.java:100, took 0.025290 s
22/02/07 23:47:25 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/07 23:47:25 INFO DAGScheduler: Got job 96 (collect at HoodieSparkEngineContext.java:134) with 1 output partitions
22/02/07 23:47:25 INFO DAGScheduler: Final stage: ResultStage 160 (collect at HoodieSparkEngineContext.java:134)
22/02/07 23:47:25 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:25 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:25 INFO DAGScheduler: Submitting ResultStage 160 (MapPartitionsRDD[310] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_129 stored as values in memory (estimated size 318.4 KiB, free 985.3 MiB)
22/02/07 23:47:25 INFO MemoryStore: Block broadcast_129_piece0 stored as bytes in memory (estimated size 111.2 KiB, free 985.2 MiB)
22/02/07 23:47:25 INFO BlockManagerInfo: Added broadcast_129_piece0 in memory on 192.168.1.37:53037 (size: 111.2 KiB, free: 987.0 MiB)
22/02/07 23:47:25 INFO SparkContext: Created broadcast 129 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:25 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 160 (MapPartitionsRDD[310] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:25 INFO TaskSchedulerImpl: Adding task set 160.0 with 1 tasks resource profile 0
22/02/07 23:47:25 INFO TaskSetManager: Starting task 0.0 in stage 160.0 (TID 868) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4338 bytes) taskResourceAssignments Map()
22/02/07 23:47:25 INFO Executor: Running task 0.0 in stage 160.0 (TID 868)
22/02/07 23:47:26 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234724963__rollback__COMPLETED]}
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 160.0 (TID 868). 878 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 160.0 (TID 868) in 23 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 160.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ResultStage 160 (collect at HoodieSparkEngineContext.java:134) finished in 0.077 s
22/02/07 23:47:26 INFO DAGScheduler: Job 96 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 160: Stage finished
22/02/07 23:47:26 INFO DAGScheduler: Job 96 finished: collect at HoodieSparkEngineContext.java:134, took 0.078157 s
22/02/07 23:47:26 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:100
22/02/07 23:47:26 INFO DAGScheduler: Got job 97 (collect at HoodieSparkEngineContext.java:100) with 1 output partitions
22/02/07 23:47:26 INFO DAGScheduler: Final stage: ResultStage 161 (collect at HoodieSparkEngineContext.java:100)
22/02/07 23:47:26 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:26 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:26 INFO DAGScheduler: Submitting ResultStage 161 (MapPartitionsRDD[312] at map at HoodieSparkEngineContext.java:100), which has no missing parents
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_130 stored as values in memory (estimated size 128.4 KiB, free 985.1 MiB)
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_130_piece0 stored as bytes in memory (estimated size 45.8 KiB, free 985.0 MiB)
22/02/07 23:47:26 INFO BlockManagerInfo: Added broadcast_130_piece0 in memory on 192.168.1.37:53037 (size: 45.8 KiB, free: 987.0 MiB)
22/02/07 23:47:26 INFO SparkContext: Created broadcast 130 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 161 (MapPartitionsRDD[312] at map at HoodieSparkEngineContext.java:100) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:26 INFO TaskSchedulerImpl: Adding task set 161.0 with 1 tasks resource profile 0
22/02/07 23:47:26 INFO TaskSetManager: Starting task 0.0 in stage 161.0 (TID 869) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4413 bytes) taskResourceAssignments Map()
22/02/07 23:47:26 INFO Executor: Running task 0.0 in stage 161.0 (TID 869)
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 161.0 (TID 869). 912 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 161.0 (TID 869) in 10 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 161.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ResultStage 161 (collect at HoodieSparkEngineContext.java:100) finished in 0.028 s
22/02/07 23:47:26 INFO DAGScheduler: Job 97 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 161: Stage finished
22/02/07 23:47:26 INFO DAGScheduler: Job 97 finished: collect at HoodieSparkEngineContext.java:100, took 0.028385 s
22/02/07 23:47:26 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220207234725651__rollback__REQUESTED]}
22/02/07 23:47:26 INFO BaseRollbackPlanActionExecutor: Requesting Rollback with instant time [==>20220207234725651__rollback__REQUESTED]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220207234725651__rollback__REQUESTED]}
22/02/07 23:47:26 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234725651.rollback.requested
22/02/07 23:47:26 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234725651.rollback.inflight
22/02/07 23:47:26 INFO MergeOnReadRollbackActionExecutor: Rolling back instant [==>20220207234155839__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO MergeOnReadRollbackActionExecutor: Unpublished [==>20220207234155839__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/07 23:47:26 INFO DAGScheduler: Got job 98 (collect at HoodieSparkEngineContext.java:134) with 1 output partitions
22/02/07 23:47:26 INFO DAGScheduler: Final stage: ResultStage 162 (collect at HoodieSparkEngineContext.java:134)
22/02/07 23:47:26 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:26 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:26 INFO DAGScheduler: Submitting ResultStage 162 (MapPartitionsRDD[314] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_131 stored as values in memory (estimated size 128.7 KiB, free 984.9 MiB)
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_131_piece0 stored as bytes in memory (estimated size 46.0 KiB, free 984.9 MiB)
22/02/07 23:47:26 INFO BlockManagerInfo: Added broadcast_131_piece0 in memory on 192.168.1.37:53037 (size: 46.0 KiB, free: 986.9 MiB)
22/02/07 23:47:26 INFO SparkContext: Created broadcast 131 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 162 (MapPartitionsRDD[314] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:26 INFO TaskSchedulerImpl: Adding task set 162.0 with 1 tasks resource profile 0
22/02/07 23:47:26 INFO TaskSetManager: Starting task 0.0 in stage 162.0 (TID 870) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4460 bytes) taskResourceAssignments Map()
22/02/07 23:47:26 INFO Executor: Running task 0.0 in stage 162.0 (TID 870)
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 162.0 (TID 870). 930 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 162.0 (TID 870) in 6 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 162.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ResultStage 162 (collect at HoodieSparkEngineContext.java:134) finished in 0.025 s
22/02/07 23:47:26 INFO DAGScheduler: Job 98 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 162: Stage finished
22/02/07 23:47:26 INFO DAGScheduler: Job 98 finished: collect at HoodieSparkEngineContext.java:134, took 0.025178 s
22/02/07 23:47:26 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:129
22/02/07 23:47:26 INFO DAGScheduler: Registering RDD 316 (mapToPair at HoodieSparkEngineContext.java:128) as input to shuffle 40
22/02/07 23:47:26 INFO DAGScheduler: Got job 99 (collect at HoodieSparkEngineContext.java:129) with 1 output partitions
22/02/07 23:47:26 INFO DAGScheduler: Final stage: ResultStage 164 (collect at HoodieSparkEngineContext.java:129)
22/02/07 23:47:26 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 163)
22/02/07 23:47:26 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 163)
22/02/07 23:47:26 INFO DAGScheduler: Submitting ShuffleMapStage 163 (MapPartitionsRDD[316] at mapToPair at HoodieSparkEngineContext.java:128), which has no missing parents
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_132 stored as values in memory (estimated size 6.8 KiB, free 984.9 MiB)
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_132_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 984.9 MiB)
22/02/07 23:47:26 INFO BlockManagerInfo: Added broadcast_132_piece0 in memory on 192.168.1.37:53037 (size: 3.8 KiB, free: 986.9 MiB)
22/02/07 23:47:26 INFO SparkContext: Created broadcast 132 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:26 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 163 (MapPartitionsRDD[316] at mapToPair at HoodieSparkEngineContext.java:128) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:26 INFO TaskSchedulerImpl: Adding task set 163.0 with 1 tasks resource profile 0
22/02/07 23:47:26 INFO TaskSetManager: Starting task 0.0 in stage 163.0 (TID 871) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4497 bytes) taskResourceAssignments Map()
22/02/07 23:47:26 INFO Executor: Running task 0.0 in stage 163.0 (TID 871)
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 163.0 (TID 871). 1043 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 163.0 (TID 871) in 4 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 163.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ShuffleMapStage 163 (mapToPair at HoodieSparkEngineContext.java:128) finished in 0.006 s
22/02/07 23:47:26 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:26 INFO DAGScheduler: running: Set()
22/02/07 23:47:26 INFO DAGScheduler: waiting: Set(ResultStage 164)
22/02/07 23:47:26 INFO DAGScheduler: failed: Set()
22/02/07 23:47:26 INFO DAGScheduler: Submitting ResultStage 164 (MapPartitionsRDD[318] at map at HoodieSparkEngineContext.java:129), which has no missing parents
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_133 stored as values in memory (estimated size 7.3 KiB, free 984.8 MiB)
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_133_piece0 stored as bytes in memory (estimated size 3.9 KiB, free 984.8 MiB)
22/02/07 23:47:26 INFO BlockManagerInfo: Added broadcast_133_piece0 in memory on 192.168.1.37:53037 (size: 3.9 KiB, free: 986.9 MiB)
22/02/07 23:47:26 INFO SparkContext: Created broadcast 133 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 164 (MapPartitionsRDD[318] at map at HoodieSparkEngineContext.java:129) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:26 INFO TaskSchedulerImpl: Adding task set 164.0 with 1 tasks resource profile 0
22/02/07 23:47:26 INFO TaskSetManager: Starting task 0.0 in stage 164.0 (TID 872) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:26 INFO Executor: Running task 0.0 in stage 164.0 (TID 872)
22/02/07 23:47:26 INFO ShuffleBlockFetcherIterator: Getting 1 (156.0 B) non-empty blocks including 1 (156.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 164.0 (TID 872). 1303 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 164.0 (TID 872) in 4 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 164.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ResultStage 164 (collect at HoodieSparkEngineContext.java:129) finished in 0.006 s
22/02/07 23:47:26 INFO DAGScheduler: Job 99 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 164: Stage finished
22/02/07 23:47:26 INFO DAGScheduler: Job 99 finished: collect at HoodieSparkEngineContext.java:129, took 0.014314 s
22/02/07 23:47:26 INFO BaseRollbackActionExecutor: Deleting instant=[==>20220207234155839__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Deleting instant [==>20220207234155839__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Removed instant [==>20220207234155839__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Deleting instant [==>20220207234155839__deltacommit__REQUESTED]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Removed instant [==>20220207234155839__deltacommit__REQUESTED]
22/02/07 23:47:26 INFO BaseRollbackActionExecutor: Deleted pending commit [==>20220207234155839__deltacommit__REQUESTED]
22/02/07 23:47:26 INFO MergeOnReadRollbackActionExecutor: Time(in ms) taken to finish rollback 102
22/02/07 23:47:26 INFO BaseRollbackActionExecutor: Rolled back inflight instant 20220207234155839
22/02/07 23:47:26 INFO BaseRollbackActionExecutor: Index rolled back for commits [==>20220207234155839__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234725651.rollback.inflight
22/02/07 23:47:26 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234725651.rollback
22/02/07 23:47:26 INFO BaseRollbackActionExecutor: Rollback of Commits [20220207234155839] is complete
22/02/07 23:47:26 INFO HeartbeatUtils: Deleted the heartbeat for instant 20220207234155839
22/02/07 23:47:26 INFO BaseHoodieWriteClient: Begin rollback of instant 20220207233705644
22/02/07 23:47:26 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:26 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:26 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:26 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:26 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234725651__rollback__COMPLETED]}
22/02/07 23:47:26 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:26 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:26 INFO BaseHoodieWriteClient: Scheduling Rollback at instant time :20220207234726235
22/02/07 23:47:26 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:100
22/02/07 23:47:26 INFO DAGScheduler: Got job 100 (collect at HoodieSparkEngineContext.java:100) with 1 output partitions
22/02/07 23:47:26 INFO DAGScheduler: Final stage: ResultStage 165 (collect at HoodieSparkEngineContext.java:100)
22/02/07 23:47:26 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:26 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:26 INFO DAGScheduler: Submitting ResultStage 165 (MapPartitionsRDD[320] at map at HoodieSparkEngineContext.java:100), which has no missing parents
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_134 stored as values in memory (estimated size 99.3 KiB, free 984.7 MiB)
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_134_piece0 stored as bytes in memory (estimated size 35.4 KiB, free 984.7 MiB)
22/02/07 23:47:26 INFO BlockManagerInfo: Added broadcast_134_piece0 in memory on 192.168.1.37:53037 (size: 35.4 KiB, free: 986.9 MiB)
22/02/07 23:47:26 INFO SparkContext: Created broadcast 134 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 165 (MapPartitionsRDD[320] at map at HoodieSparkEngineContext.java:100) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:26 INFO TaskSchedulerImpl: Adding task set 165.0 with 1 tasks resource profile 0
22/02/07 23:47:26 INFO TaskSetManager: Starting task 0.0 in stage 165.0 (TID 873) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4440 bytes) taskResourceAssignments Map()
22/02/07 23:47:26 INFO Executor: Running task 0.0 in stage 165.0 (TID 873)
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 165.0 (TID 873). 1239 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 165.0 (TID 873) in 6 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 165.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ResultStage 165 (collect at HoodieSparkEngineContext.java:100) finished in 0.024 s
22/02/07 23:47:26 INFO DAGScheduler: Job 100 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 165: Stage finished
22/02/07 23:47:26 INFO DAGScheduler: Job 100 finished: collect at HoodieSparkEngineContext.java:100, took 0.026249 s
22/02/07 23:47:26 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/07 23:47:26 INFO DAGScheduler: Got job 101 (collect at HoodieSparkEngineContext.java:134) with 1 output partitions
22/02/07 23:47:26 INFO DAGScheduler: Final stage: ResultStage 166 (collect at HoodieSparkEngineContext.java:134)
22/02/07 23:47:26 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:26 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:26 INFO DAGScheduler: Submitting ResultStage 166 (MapPartitionsRDD[322] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_135 stored as values in memory (estimated size 318.4 KiB, free 984.4 MiB)
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_135_piece0 stored as bytes in memory (estimated size 111.1 KiB, free 984.3 MiB)
22/02/07 23:47:26 INFO BlockManagerInfo: Added broadcast_135_piece0 in memory on 192.168.1.37:53037 (size: 111.1 KiB, free: 986.8 MiB)
22/02/07 23:47:26 INFO SparkContext: Created broadcast 135 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 166 (MapPartitionsRDD[322] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:26 INFO TaskSchedulerImpl: Adding task set 166.0 with 1 tasks resource profile 0
22/02/07 23:47:26 INFO TaskSetManager: Starting task 0.0 in stage 166.0 (TID 874) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4338 bytes) taskResourceAssignments Map()
22/02/07 23:47:26 INFO Executor: Running task 0.0 in stage 166.0 (TID 874)
22/02/07 23:47:26 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234725651__rollback__COMPLETED]}
22/02/07 23:47:26 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:26 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:26 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:26 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:47:26 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:47:26 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:47:26 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=31, NumFileGroups=1, FileGroupsCreationTime=9, StoreTimeTaken=0
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 166.0 (TID 874). 998 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 166.0 (TID 874) in 44 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 166.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ResultStage 166 (collect at HoodieSparkEngineContext.java:134) finished in 0.098 s
22/02/07 23:47:26 INFO DAGScheduler: Job 101 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 166: Stage finished
22/02/07 23:47:26 INFO DAGScheduler: Job 101 finished: collect at HoodieSparkEngineContext.java:134, took 0.099007 s
22/02/07 23:47:26 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:100
22/02/07 23:47:26 INFO DAGScheduler: Got job 102 (collect at HoodieSparkEngineContext.java:100) with 1 output partitions
22/02/07 23:47:26 INFO DAGScheduler: Final stage: ResultStage 167 (collect at HoodieSparkEngineContext.java:100)
22/02/07 23:47:26 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:26 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:26 INFO DAGScheduler: Submitting ResultStage 167 (MapPartitionsRDD[324] at map at HoodieSparkEngineContext.java:100), which has no missing parents
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_136 stored as values in memory (estimated size 128.4 KiB, free 984.2 MiB)
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_136_piece0 stored as bytes in memory (estimated size 45.8 KiB, free 984.1 MiB)
22/02/07 23:47:26 INFO BlockManagerInfo: Added broadcast_136_piece0 in memory on 192.168.1.37:53037 (size: 45.8 KiB, free: 986.8 MiB)
22/02/07 23:47:26 INFO SparkContext: Created broadcast 136 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 167 (MapPartitionsRDD[324] at map at HoodieSparkEngineContext.java:100) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:26 INFO TaskSchedulerImpl: Adding task set 167.0 with 1 tasks resource profile 0
22/02/07 23:47:26 INFO TaskSetManager: Starting task 0.0 in stage 167.0 (TID 875) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4533 bytes) taskResourceAssignments Map()
22/02/07 23:47:26 INFO Executor: Running task 0.0 in stage 167.0 (TID 875)
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 167.0 (TID 875). 1100 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 167.0 (TID 875) in 12 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 167.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ResultStage 167 (collect at HoodieSparkEngineContext.java:100) finished in 0.031 s
22/02/07 23:47:26 INFO DAGScheduler: Job 102 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 167: Stage finished
22/02/07 23:47:26 INFO DAGScheduler: Job 102 finished: collect at HoodieSparkEngineContext.java:100, took 0.032009 s
22/02/07 23:47:26 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220207234726235__rollback__REQUESTED]}
22/02/07 23:47:26 INFO BaseRollbackPlanActionExecutor: Requesting Rollback with instant time [==>20220207234726235__rollback__REQUESTED]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220207234726235__rollback__REQUESTED]}
22/02/07 23:47:26 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234726235.rollback.requested
22/02/07 23:47:26 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234726235.rollback.inflight
22/02/07 23:47:26 INFO MergeOnReadRollbackActionExecutor: Rolling back instant [==>20220207233705644__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO MergeOnReadRollbackActionExecutor: Unpublished [==>20220207233705644__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/07 23:47:26 INFO DAGScheduler: Got job 103 (collect at HoodieSparkEngineContext.java:134) with 1 output partitions
22/02/07 23:47:26 INFO DAGScheduler: Final stage: ResultStage 168 (collect at HoodieSparkEngineContext.java:134)
22/02/07 23:47:26 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:26 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:26 INFO DAGScheduler: Submitting ResultStage 168 (MapPartitionsRDD[326] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_137 stored as values in memory (estimated size 128.7 KiB, free 984.0 MiB)
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_137_piece0 stored as bytes in memory (estimated size 46.0 KiB, free 983.9 MiB)
22/02/07 23:47:26 INFO BlockManagerInfo: Added broadcast_137_piece0 in memory on 192.168.1.37:53037 (size: 46.0 KiB, free: 986.7 MiB)
22/02/07 23:47:26 INFO SparkContext: Created broadcast 137 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 168 (MapPartitionsRDD[326] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:26 INFO TaskSchedulerImpl: Adding task set 168.0 with 1 tasks resource profile 0
22/02/07 23:47:26 INFO TaskSetManager: Starting task 0.0 in stage 168.0 (TID 876) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4590 bytes) taskResourceAssignments Map()
22/02/07 23:47:26 INFO Executor: Running task 0.0 in stage 168.0 (TID 876)
22/02/07 23:47:26 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer
22/02/07 23:47:26 INFO HoodieLogFormat$WriterBuilder: Computing the next log version for files-0000 in /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files
22/02/07 23:47:26 INFO HoodieLogFormat$WriterBuilder: Computed the next log version for files-0000 in /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files as 29 with write-token 0-144-854
22/02/07 23:47:26 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.29_0-144-854
22/02/07 23:47:26 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.30_1-0-1', fileLen=-1}
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 168.0 (TID 876). 1382 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 168.0 (TID 876) in 35 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 168.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ResultStage 168 (collect at HoodieSparkEngineContext.java:134) finished in 0.059 s
22/02/07 23:47:26 INFO DAGScheduler: Job 103 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 168: Stage finished
22/02/07 23:47:26 INFO DAGScheduler: Job 103 finished: collect at HoodieSparkEngineContext.java:134, took 0.059969 s
22/02/07 23:47:26 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:129
22/02/07 23:47:26 INFO DAGScheduler: Registering RDD 328 (mapToPair at HoodieSparkEngineContext.java:128) as input to shuffle 41
22/02/07 23:47:26 INFO DAGScheduler: Got job 104 (collect at HoodieSparkEngineContext.java:129) with 1 output partitions
22/02/07 23:47:26 INFO DAGScheduler: Final stage: ResultStage 170 (collect at HoodieSparkEngineContext.java:129)
22/02/07 23:47:26 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 169)
22/02/07 23:47:26 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 169)
22/02/07 23:47:26 INFO DAGScheduler: Submitting ShuffleMapStage 169 (MapPartitionsRDD[328] at mapToPair at HoodieSparkEngineContext.java:128), which has no missing parents
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_138 stored as values in memory (estimated size 6.8 KiB, free 983.9 MiB)
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_138_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 983.9 MiB)
22/02/07 23:47:26 INFO BlockManagerInfo: Added broadcast_138_piece0 in memory on 192.168.1.37:53037 (size: 3.8 KiB, free: 986.7 MiB)
22/02/07 23:47:26 INFO SparkContext: Created broadcast 138 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:26 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 169 (MapPartitionsRDD[328] at mapToPair at HoodieSparkEngineContext.java:128) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:26 INFO TaskSchedulerImpl: Adding task set 169.0 with 1 tasks resource profile 0
22/02/07 23:47:26 INFO TaskSetManager: Starting task 0.0 in stage 169.0 (TID 877) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4906 bytes) taskResourceAssignments Map()
22/02/07 23:47:26 INFO Executor: Running task 0.0 in stage 169.0 (TID 877)
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 169.0 (TID 877). 1086 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 169.0 (TID 877) in 8 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 169.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ShuffleMapStage 169 (mapToPair at HoodieSparkEngineContext.java:128) finished in 0.011 s
22/02/07 23:47:26 INFO DAGScheduler: looking for newly runnable stages
22/02/07 23:47:26 INFO DAGScheduler: running: Set()
22/02/07 23:47:26 INFO DAGScheduler: waiting: Set(ResultStage 170)
22/02/07 23:47:26 INFO DAGScheduler: failed: Set()
22/02/07 23:47:26 INFO DAGScheduler: Submitting ResultStage 170 (MapPartitionsRDD[330] at map at HoodieSparkEngineContext.java:129), which has no missing parents
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_139 stored as values in memory (estimated size 7.3 KiB, free 983.9 MiB)
22/02/07 23:47:26 INFO MemoryStore: Block broadcast_139_piece0 stored as bytes in memory (estimated size 3.9 KiB, free 983.9 MiB)
22/02/07 23:47:26 INFO BlockManagerInfo: Added broadcast_139_piece0 in memory on 192.168.1.37:53037 (size: 3.9 KiB, free: 986.7 MiB)
22/02/07 23:47:26 INFO SparkContext: Created broadcast 139 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:26 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 170 (MapPartitionsRDD[330] at map at HoodieSparkEngineContext.java:129) (first 15 tasks are for partitions Vector(0))
22/02/07 23:47:26 INFO TaskSchedulerImpl: Adding task set 170.0 with 1 tasks resource profile 0
22/02/07 23:47:26 INFO TaskSetManager: Starting task 0.0 in stage 170.0 (TID 878) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/07 23:47:26 INFO Executor: Running task 0.0 in stage 170.0 (TID 878)
22/02/07 23:47:26 INFO ShuffleBlockFetcherIterator: Getting 1 (405.0 B) non-empty blocks including 1 (405.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/07 23:47:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/07 23:47:26 INFO Executor: Finished task 0.0 in stage 170.0 (TID 878). 1633 bytes result sent to driver
22/02/07 23:47:26 INFO TaskSetManager: Finished task 0.0 in stage 170.0 (TID 878) in 6 ms on 192.168.1.37 (executor driver) (1/1)
22/02/07 23:47:26 INFO TaskSchedulerImpl: Removed TaskSet 170.0, whose tasks have all completed, from pool
22/02/07 23:47:26 INFO DAGScheduler: ResultStage 170 (collect at HoodieSparkEngineContext.java:129) finished in 0.008 s
22/02/07 23:47:26 INFO DAGScheduler: Job 104 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:26 INFO TaskSchedulerImpl: Killing all running tasks in stage 170: Stage finished
22/02/07 23:47:26 INFO DAGScheduler: Job 104 finished: collect at HoodieSparkEngineContext.java:129, took 0.021499 s
22/02/07 23:47:26 INFO BaseRollbackActionExecutor: Deleting instant=[==>20220207233705644__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Deleting instant [==>20220207233705644__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Removed instant [==>20220207233705644__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Deleting instant [==>20220207233705644__deltacommit__REQUESTED]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Removed instant [==>20220207233705644__deltacommit__REQUESTED]
22/02/07 23:47:26 INFO BaseRollbackActionExecutor: Deleted pending commit [==>20220207233705644__deltacommit__REQUESTED]
22/02/07 23:47:26 INFO MergeOnReadRollbackActionExecutor: Time(in ms) taken to finish rollback 170
22/02/07 23:47:26 INFO BaseRollbackActionExecutor: Rolled back inflight instant 20220207233705644
22/02/07 23:47:26 INFO BaseRollbackActionExecutor: Index rolled back for commits [==>20220207233705644__deltacommit__INFLIGHT]
22/02/07 23:47:26 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234726235.rollback.inflight
22/02/07 23:47:26 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207234726235.rollback
22/02/07 23:47:26 INFO BaseRollbackActionExecutor: Rollback of Commits [20220207233705644] is complete
22/02/07 23:47:26 INFO HeartbeatUtils: Deleted the heartbeat for instant 20220207233705644
22/02/07 23:47:26 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:26 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:26 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:26 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:26 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234726235__rollback__COMPLETED]}
22/02/07 23:47:26 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:26 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:26 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:26 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:47:26 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:26 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:47:26 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234726235__rollback__COMPLETED]}
22/02/07 23:47:26 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/07 23:47:26 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/07 23:47:26 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234726235__rollback__COMPLETED]}
22/02/07 23:47:26 INFO HoodieLogFormat$WriterBuilder: Building HoodieLogFormat Writer
22/02/07 23:47:26 INFO HoodieLogFormat$WriterBuilder: Computing the next log version for commits in /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/archived
22/02/07 23:47:26 INFO HoodieLogFormat$WriterBuilder: Computed the next log version for commits in /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/archived as 2 with write-token 1-0-1
22/02/07 23:47:26 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/archived/.commits_.archive.2_1-0-1
22/02/07 23:47:26 INFO HoodieTimelineArchiveLog: Archiving instants [[==>20220207232143180__deltacommit__REQUESTED], [==>20220207232143180__deltacommit__INFLIGHT], [20220207232143180__deltacommit__COMPLETED], [==>20220207232657914__deltacommit__REQUESTED], [==>20220207232657914__deltacommit__INFLIGHT], [20220207232657914__deltacommit__COMPLETED], [==>20220207232658076__deltacommit__REQUESTED], [==>20220207232658076__deltacommit__INFLIGHT], [20220207232658076__deltacommit__COMPLETED], [==>20220207232723131__deltacommit__REQUESTED], [==>20220207232723131__deltacommit__INFLIGHT], [20220207232723131__deltacommit__COMPLETED], [==>20220207233159957__deltacommit__REQUESTED], [==>20220207233159957__deltacommit__INFLIGHT], [20220207233159957__deltacommit__COMPLETED], [==>20220207233322808__deltacommit__REQUESTED], [==>20220207233322808__deltacommit__INFLIGHT], [20220207233322808__deltacommit__COMPLETED], [==>20220207233323715__deltacommit__REQUESTED], [==>20220207233323715__deltacommit__INFLIGHT], [20220207233323715__deltacommit__COMPLETED], [==>20220207233342239__deltacommit__REQUESTED], [==>20220207233342239__deltacommit__INFLIGHT], [20220207233342239__deltacommit__COMPLETED]]
22/02/07 23:47:26 INFO HoodieTimelineArchiveLog: Wrapper schema {"type":"record","name":"HoodieArchivedMetaEntry","namespace":"org.apache.hudi.avro.model","fields":[{"name":"hoodieCommitMetadata","type":["null",{"type":"record","name":"HoodieCommitMetadata","fields":[{"name":"partitionToWriteStats","type":["null",{"type":"map","values":{"type":"array","items":{"type":"record","name":"HoodieWriteStat","fields":[{"name":"fileId","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"path","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"prevCommit","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"numWrites","type":["null","long"],"default":null},{"name":"numDeletes","type":["null","long"],"default":null},{"name":"numUpdateWrites","type":["null","long"],"default":null},{"name":"totalWriteBytes","type":["null","long"],"default":null},{"name":"totalWriteErrors","type":["null","long"],"default":null},{"name":"partitionPath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"totalLogRecords","type":["null","long"],"default":null},{"name":"totalLogFiles","type":["null","long"],"default":null},{"name":"totalUpdatedRecordsCompacted","type":["null","long"],"default":null},{"name":"numInserts","type":["null","long"],"default":null},{"name":"totalLogBlocks","type":["null","long"],"default":null},{"name":"totalCorruptLogBlock","type":["null","long"],"default":null},{"name":"totalRollbackBlocks","type":["null","long"],"default":null},{"name":"fileSizeInBytes","type":["null","long"],"default":null}]}},"avro.java.string":"String"}],"default":null},{"name":"extraMetadata","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String","default":null}],"default":null},{"name":"version","type":["int","null"],"default":1},{"name":"operationType","type":["null",{"type":"string","avro.java.string":"String"}],"default":null}]}],"default":null},{"name":"hoodieCleanMetadata","type":["null",{"type":"record","name":"HoodieCleanMetadata","fields":[{"name":"startCleanTime","type":{"type":"string","avro.java.string":"String"}},{"name":"timeTakenInMillis","type":"long"},{"name":"totalFilesDeleted","type":"int"},{"name":"earliestCommitToRetain","type":{"type":"string","avro.java.string":"String"}},{"name":"partitionMetadata","type":{"type":"map","values":{"type":"record","name":"HoodieCleanPartitionMetadata","fields":[{"name":"partitionPath","type":{"type":"string","avro.java.string":"String"}},{"name":"policy","type":{"type":"string","avro.java.string":"String"}},{"name":"deletePathPatterns","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}},{"name":"successDeleteFiles","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}},{"name":"failedDeleteFiles","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}}]},"avro.java.string":"String"}},{"name":"version","type":["int","null"],"default":1},{"name":"bootstrapPartitionMetadata","type":["null",{"type":"map","values":"HoodieCleanPartitionMetadata","avro.java.string":"String","default":null}],"default":null}]}],"default":null},{"name":"hoodieCompactionMetadata","type":["null",{"type":"record","name":"HoodieCompactionMetadata","fields":[{"name":"partitionToCompactionWriteStats","type":["null",{"type":"map","values":{"type":"array","items":{"type":"record","name":"HoodieCompactionWriteStat","fields":[{"name":"partitionPath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"totalLogRecords","type":["null","long"],"default":null},{"name":"totalLogFiles","type":["null","long"],"default":null},{"name":"totalUpdatedRecordsCompacted","type":["null","long"],"default":null},{"name":"hoodieWriteStat","type":["null","HoodieWriteStat"],"default":null}]}},"avro.java.string":"String"}]}]}],"default":null},{"name":"hoodieRollbackMetadata","type":["null",{"type":"record","name":"HoodieRollbackMetadata","fields":[{"name":"startRollbackTime","type":{"type":"string","avro.java.string":"String"}},{"name":"timeTakenInMillis","type":"long"},{"name":"totalFilesDeleted","type":"int"},{"name":"commitsRollback","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}},{"name":"partitionMetadata","type":{"type":"map","values":{"type":"record","name":"HoodieRollbackPartitionMetadata","fields":[{"name":"partitionPath","type":{"type":"string","avro.java.string":"String"}},{"name":"successDeleteFiles","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}},{"name":"failedDeleteFiles","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}},{"name":"rollbackLogFiles","type":["null",{"type":"map","values":"long","avro.java.string":"String"}],"default":null}]},"avro.java.string":"String"}},{"name":"version","type":["int","null"],"default":1},{"name":"instantsRollback","type":{"type":"array","items":{"type":"record","name":"HoodieInstantInfo","fields":[{"name":"commitTime","type":{"type":"string","avro.java.string":"String"}},{"name":"action","type":{"type":"string","avro.java.string":"String"}}]},"default":[]},"default":[]}]}],"default":null},{"name":"hoodieSavePointMetadata","type":["null",{"type":"record","name":"HoodieSavepointMetadata","fields":[{"name":"savepointedBy","type":{"type":"string","avro.java.string":"String"}},{"name":"savepointedAt","type":"long"},{"name":"comments","type":{"type":"string","avro.java.string":"String"}},{"name":"partitionMetadata","type":{"type":"map","values":{"type":"record","name":"HoodieSavepointPartitionMetadata","fields":[{"name":"partitionPath","type":{"type":"string","avro.java.string":"String"}},{"name":"savepointDataFile","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}}]},"avro.java.string":"String"}},{"name":"version","type":["int","null"],"default":1}]}],"default":null},{"name":"commitTime","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"actionType","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1},{"name":"hoodieCompactionPlan","type":["null",{"type":"record","name":"HoodieCompactionPlan","fields":[{"name":"operations","type":["null",{"type":"array","items":{"type":"record","name":"HoodieCompactionOperation","fields":[{"name":"baseInstantTime","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"deltaFilePaths","type":["null",{"type":"array","items":{"type":"string","avro.java.string":"String"}}],"default":null},{"name":"dataFilePath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"fileId","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"partitionPath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"metrics","type":["null",{"type":"map","values":"double","avro.java.string":"String"}],"default":null},{"name":"bootstrapFilePath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null}]}}],"default":null},{"name":"extraMetadata","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1}]}],"default":null},{"name":"hoodieCleanerPlan","type":["null",{"type":"record","name":"HoodieCleanerPlan","fields":[{"name":"earliestInstantToRetain","type":["null",{"type":"record","name":"HoodieActionInstant","fields":[{"name":"timestamp","type":{"type":"string","avro.java.string":"String"}},{"name":"action","type":{"type":"string","avro.java.string":"String"}},{"name":"state","type":{"type":"string","avro.java.string":"String"}}]}],"default":null},{"name":"policy","type":{"type":"string","avro.java.string":"String"}},{"name":"filesToBeDeletedPerPartition","type":["null",{"type":"map","values":{"type":"array","items":{"type":"string","avro.java.string":"String"}},"avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1},{"name":"filePathsToBeDeletedPerPartition","type":["null",{"type":"map","values":{"type":"array","items":{"type":"record","name":"HoodieCleanFileInfo","fields":[{"name":"filePath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"isBootstrapBaseFile","type":["null","boolean"],"default":null}]}},"avro.java.string":"String"}],"doc":"This field replaces the field filesToBeDeletedPerPartition","default":null}]}],"default":null},{"name":"actionState","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"hoodieReplaceCommitMetadata","type":["null",{"type":"record","name":"HoodieReplaceCommitMetadata","fields":[{"name":"partitionToWriteStats","type":["null",{"type":"map","values":{"type":"array","items":"HoodieWriteStat"},"avro.java.string":"String"}],"default":null},{"name":"extraMetadata","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1},{"name":"operationType","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"partitionToReplaceFileIds","type":["null",{"type":"map","values":{"type":"array","items":{"type":"string","avro.java.string":"String"}},"avro.java.string":"String"}],"default":null}]}],"default":null},{"name":"hoodieRequestedReplaceMetadata","type":["null",{"type":"record","name":"HoodieRequestedReplaceMetadata","fields":[{"name":"operationType","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"clusteringPlan","type":["null",{"type":"record","name":"HoodieClusteringPlan","fields":[{"name":"inputGroups","type":["null",{"type":"array","items":{"type":"record","name":"HoodieClusteringGroup","fields":[{"name":"slices","type":["null",{"type":"array","items":{"type":"record","name":"HoodieSliceInfo","fields":[{"name":"dataFilePath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"deltaFilePaths","type":["null",{"type":"array","items":{"type":"string","avro.java.string":"String"}}],"default":null},{"name":"fileId","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"partitionPath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"bootstrapFilePath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1}]}}],"default":null},{"name":"metrics","type":["null",{"type":"map","values":"double","avro.java.string":"String"}],"default":null},{"name":"numOutputFileGroups","type":["int","null"],"default":1},{"name":"version","type":["int","null"],"default":1}]}}],"default":null},{"name":"strategy","type":["null",{"type":"record","name":"HoodieClusteringStrategy","fields":[{"name":"strategyClassName","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"strategyParams","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1}]}],"default":null},{"name":"extraMetadata","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1},{"name":"preserveHoodieMetadata","type":["null","boolean"],"default":null}]}],"default":null},{"name":"extraMetadata","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1}]}],"default":null},{"name":"HoodieInflightReplaceMetadata","type":["null","HoodieCommitMetadata"],"default":null}]}
22/02/07 23:47:26 INFO HoodieLogFormatWriter: Append not supported.. Rolling over to HoodieLogFile{pathStr='/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/archived/.commits_.archive.3_1-0-1', fileLen=-1}
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Deleting archived instants [[==>20220207232143180__deltacommit__REQUESTED], [==>20220207232143180__deltacommit__INFLIGHT], [20220207232143180__deltacommit__COMPLETED], [==>20220207232657914__deltacommit__REQUESTED], [==>20220207232657914__deltacommit__INFLIGHT], [20220207232657914__deltacommit__COMPLETED], [==>20220207232658076__deltacommit__REQUESTED], [==>20220207232658076__deltacommit__INFLIGHT], [20220207232658076__deltacommit__COMPLETED], [==>20220207232723131__deltacommit__REQUESTED], [==>20220207232723131__deltacommit__INFLIGHT], [20220207232723131__deltacommit__COMPLETED], [==>20220207233159957__deltacommit__REQUESTED], [==>20220207233159957__deltacommit__INFLIGHT], [20220207233159957__deltacommit__COMPLETED], [==>20220207233322808__deltacommit__REQUESTED], [==>20220207233322808__deltacommit__INFLIGHT], [20220207233322808__deltacommit__COMPLETED], [==>20220207233323715__deltacommit__REQUESTED], [==>20220207233323715__deltacommit__INFLIGHT], [20220207233323715__deltacommit__COMPLETED], [==>20220207233342239__deltacommit__REQUESTED], [==>20220207233342239__deltacommit__INFLIGHT], [20220207233342239__deltacommit__COMPLETED]]
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Deleting instants [[==>20220207232143180__deltacommit__REQUESTED], [==>20220207232143180__deltacommit__INFLIGHT], [20220207232143180__deltacommit__COMPLETED], [==>20220207232657914__deltacommit__REQUESTED], [==>20220207232657914__deltacommit__INFLIGHT], [20220207232657914__deltacommit__COMPLETED], [==>20220207232658076__deltacommit__REQUESTED], [==>20220207232658076__deltacommit__INFLIGHT], [20220207232658076__deltacommit__COMPLETED], [==>20220207232723131__deltacommit__REQUESTED], [==>20220207232723131__deltacommit__INFLIGHT], [20220207232723131__deltacommit__COMPLETED], [==>20220207233159957__deltacommit__REQUESTED], [==>20220207233159957__deltacommit__INFLIGHT], [20220207233159957__deltacommit__COMPLETED], [==>20220207233322808__deltacommit__REQUESTED], [==>20220207233322808__deltacommit__INFLIGHT], [20220207233322808__deltacommit__COMPLETED], [==>20220207233323715__deltacommit__REQUESTED], [==>20220207233323715__deltacommit__INFLIGHT], [20220207233323715__deltacommit__COMPLETED], [==>20220207233342239__deltacommit__REQUESTED], [==>20220207233342239__deltacommit__INFLIGHT], [20220207233342239__deltacommit__COMPLETED]]
22/02/07 23:47:27 INFO SparkContext: Starting job: collectAsMap at HoodieSparkEngineContext.java:148
22/02/07 23:47:27 INFO DAGScheduler: Got job 105 (collectAsMap at HoodieSparkEngineContext.java:148) with 24 output partitions
22/02/07 23:47:27 INFO DAGScheduler: Final stage: ResultStage 171 (collectAsMap at HoodieSparkEngineContext.java:148)
22/02/07 23:47:27 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:27 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:27 INFO DAGScheduler: Submitting ResultStage 171 (MapPartitionsRDD[332] at mapToPair at HoodieSparkEngineContext.java:145), which has no missing parents
22/02/07 23:47:27 INFO MemoryStore: Block broadcast_140 stored as values in memory (estimated size 211.0 KiB, free 983.7 MiB)
22/02/07 23:47:27 INFO MemoryStore: Block broadcast_140_piece0 stored as bytes in memory (estimated size 75.7 KiB, free 983.6 MiB)
22/02/07 23:47:27 INFO BlockManagerInfo: Added broadcast_140_piece0 in memory on 192.168.1.37:53037 (size: 75.7 KiB, free: 986.6 MiB)
22/02/07 23:47:27 INFO SparkContext: Created broadcast 140 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:27 INFO DAGScheduler: Submitting 24 missing tasks from ResultStage 171 (MapPartitionsRDD[332] at mapToPair at HoodieSparkEngineContext.java:145) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/07 23:47:27 INFO TaskSchedulerImpl: Adding task set 171.0 with 24 tasks resource profile 0
22/02/07 23:47:27 INFO TaskSetManager: Starting task 0.0 in stage 171.0 (TID 879) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4461 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Starting task 1.0 in stage 171.0 (TID 880) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4460 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Starting task 2.0 in stage 171.0 (TID 881) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4451 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO Executor: Running task 2.0 in stage 171.0 (TID 881)
22/02/07 23:47:27 INFO Executor: Running task 0.0 in stage 171.0 (TID 879)
22/02/07 23:47:27 INFO Executor: Running task 1.0 in stage 171.0 (TID 880)
22/02/07 23:47:27 INFO Executor: Finished task 0.0 in stage 171.0 (TID 879). 931 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 1.0 in stage 171.0 (TID 880). 930 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 3.0 in stage 171.0 (TID 882) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4461 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO Executor: Finished task 2.0 in stage 171.0 (TID 881). 921 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Finished task 0.0 in stage 171.0 (TID 879) in 19 ms on 192.168.1.37 (executor driver) (1/24)
22/02/07 23:47:27 INFO Executor: Running task 3.0 in stage 171.0 (TID 882)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 4.0 in stage 171.0 (TID 883) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4460 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO Executor: Running task 4.0 in stage 171.0 (TID 883)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 5.0 in stage 171.0 (TID 884) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4451 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 1.0 in stage 171.0 (TID 880) in 20 ms on 192.168.1.37 (executor driver) (2/24)
22/02/07 23:47:27 INFO Executor: Running task 5.0 in stage 171.0 (TID 884)
22/02/07 23:47:27 INFO TaskSetManager: Finished task 2.0 in stage 171.0 (TID 881) in 20 ms on 192.168.1.37 (executor driver) (3/24)
22/02/07 23:47:27 INFO Executor: Finished task 5.0 in stage 171.0 (TID 884). 921 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 3.0 in stage 171.0 (TID 882). 931 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 4.0 in stage 171.0 (TID 883). 930 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 6.0 in stage 171.0 (TID 885) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4461 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 5.0 in stage 171.0 (TID 884) in 18 ms on 192.168.1.37 (executor driver) (4/24)
22/02/07 23:47:27 INFO Executor: Running task 6.0 in stage 171.0 (TID 885)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 7.0 in stage 171.0 (TID 886) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4460 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 3.0 in stage 171.0 (TID 882) in 20 ms on 192.168.1.37 (executor driver) (5/24)
22/02/07 23:47:27 INFO Executor: Running task 7.0 in stage 171.0 (TID 886)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 8.0 in stage 171.0 (TID 887) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4451 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 4.0 in stage 171.0 (TID 883) in 20 ms on 192.168.1.37 (executor driver) (6/24)
22/02/07 23:47:27 INFO Executor: Running task 8.0 in stage 171.0 (TID 887)
22/02/07 23:47:27 INFO Executor: Finished task 7.0 in stage 171.0 (TID 886). 930 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 6.0 in stage 171.0 (TID 885). 931 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 9.0 in stage 171.0 (TID 888) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4461 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 7.0 in stage 171.0 (TID 886) in 17 ms on 192.168.1.37 (executor driver) (7/24)
22/02/07 23:47:27 INFO Executor: Running task 9.0 in stage 171.0 (TID 888)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 10.0 in stage 171.0 (TID 889) (192.168.1.37, executor driver, partition 10, PROCESS_LOCAL, 4460 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 6.0 in stage 171.0 (TID 885) in 17 ms on 192.168.1.37 (executor driver) (8/24)
22/02/07 23:47:27 INFO Executor: Running task 10.0 in stage 171.0 (TID 889)
22/02/07 23:47:27 INFO Executor: Finished task 8.0 in stage 171.0 (TID 887). 921 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 11.0 in stage 171.0 (TID 890) (192.168.1.37, executor driver, partition 11, PROCESS_LOCAL, 4451 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 8.0 in stage 171.0 (TID 887) in 17 ms on 192.168.1.37 (executor driver) (9/24)
22/02/07 23:47:27 INFO Executor: Running task 11.0 in stage 171.0 (TID 890)
22/02/07 23:47:27 INFO Executor: Finished task 9.0 in stage 171.0 (TID 888). 931 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 10.0 in stage 171.0 (TID 889). 930 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 11.0 in stage 171.0 (TID 890). 921 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 12.0 in stage 171.0 (TID 891) (192.168.1.37, executor driver, partition 12, PROCESS_LOCAL, 4461 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 9.0 in stage 171.0 (TID 888) in 18 ms on 192.168.1.37 (executor driver) (10/24)
22/02/07 23:47:27 INFO Executor: Running task 12.0 in stage 171.0 (TID 891)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 13.0 in stage 171.0 (TID 892) (192.168.1.37, executor driver, partition 13, PROCESS_LOCAL, 4460 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 10.0 in stage 171.0 (TID 889) in 17 ms on 192.168.1.37 (executor driver) (11/24)
22/02/07 23:47:27 INFO Executor: Running task 13.0 in stage 171.0 (TID 892)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 14.0 in stage 171.0 (TID 893) (192.168.1.37, executor driver, partition 14, PROCESS_LOCAL, 4451 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 11.0 in stage 171.0 (TID 890) in 18 ms on 192.168.1.37 (executor driver) (12/24)
22/02/07 23:47:27 INFO Executor: Running task 14.0 in stage 171.0 (TID 893)
22/02/07 23:47:27 INFO Executor: Finished task 14.0 in stage 171.0 (TID 893). 921 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 12.0 in stage 171.0 (TID 891). 931 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 13.0 in stage 171.0 (TID 892). 930 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 15.0 in stage 171.0 (TID 894) (192.168.1.37, executor driver, partition 15, PROCESS_LOCAL, 4461 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 14.0 in stage 171.0 (TID 893) in 17 ms on 192.168.1.37 (executor driver) (13/24)
22/02/07 23:47:27 INFO Executor: Running task 15.0 in stage 171.0 (TID 894)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 16.0 in stage 171.0 (TID 895) (192.168.1.37, executor driver, partition 16, PROCESS_LOCAL, 4460 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 12.0 in stage 171.0 (TID 891) in 18 ms on 192.168.1.37 (executor driver) (14/24)
22/02/07 23:47:27 INFO Executor: Running task 16.0 in stage 171.0 (TID 895)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 17.0 in stage 171.0 (TID 896) (192.168.1.37, executor driver, partition 17, PROCESS_LOCAL, 4451 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 13.0 in stage 171.0 (TID 892) in 18 ms on 192.168.1.37 (executor driver) (15/24)
22/02/07 23:47:27 INFO Executor: Running task 17.0 in stage 171.0 (TID 896)
22/02/07 23:47:27 INFO Executor: Finished task 15.0 in stage 171.0 (TID 894). 931 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 18.0 in stage 171.0 (TID 897) (192.168.1.37, executor driver, partition 18, PROCESS_LOCAL, 4461 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 15.0 in stage 171.0 (TID 894) in 14 ms on 192.168.1.37 (executor driver) (16/24)
22/02/07 23:47:27 INFO Executor: Running task 18.0 in stage 171.0 (TID 897)
22/02/07 23:47:27 INFO Executor: Finished task 16.0 in stage 171.0 (TID 895). 930 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 19.0 in stage 171.0 (TID 898) (192.168.1.37, executor driver, partition 19, PROCESS_LOCAL, 4460 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO Executor: Finished task 17.0 in stage 171.0 (TID 896). 921 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Finished task 16.0 in stage 171.0 (TID 895) in 15 ms on 192.168.1.37 (executor driver) (17/24)
22/02/07 23:47:27 INFO Executor: Running task 19.0 in stage 171.0 (TID 898)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 20.0 in stage 171.0 (TID 899) (192.168.1.37, executor driver, partition 20, PROCESS_LOCAL, 4451 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO Executor: Running task 20.0 in stage 171.0 (TID 899)
22/02/07 23:47:27 INFO TaskSetManager: Finished task 17.0 in stage 171.0 (TID 896) in 15 ms on 192.168.1.37 (executor driver) (18/24)
22/02/07 23:47:27 INFO Executor: Finished task 19.0 in stage 171.0 (TID 898). 930 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 21.0 in stage 171.0 (TID 900) (192.168.1.37, executor driver, partition 21, PROCESS_LOCAL, 4461 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 19.0 in stage 171.0 (TID 898) in 15 ms on 192.168.1.37 (executor driver) (19/24)
22/02/07 23:47:27 INFO Executor: Running task 21.0 in stage 171.0 (TID 900)
22/02/07 23:47:27 INFO Executor: Finished task 18.0 in stage 171.0 (TID 897). 931 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 20.0 in stage 171.0 (TID 899). 921 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 22.0 in stage 171.0 (TID 901) (192.168.1.37, executor driver, partition 22, PROCESS_LOCAL, 4460 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 18.0 in stage 171.0 (TID 897) in 20 ms on 192.168.1.37 (executor driver) (20/24)
22/02/07 23:47:27 INFO Executor: Running task 22.0 in stage 171.0 (TID 901)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 23.0 in stage 171.0 (TID 902) (192.168.1.37, executor driver, partition 23, PROCESS_LOCAL, 4451 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 20.0 in stage 171.0 (TID 899) in 17 ms on 192.168.1.37 (executor driver) (21/24)
22/02/07 23:47:27 INFO Executor: Running task 23.0 in stage 171.0 (TID 902)
22/02/07 23:47:27 INFO Executor: Finished task 21.0 in stage 171.0 (TID 900). 931 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Finished task 21.0 in stage 171.0 (TID 900) in 18 ms on 192.168.1.37 (executor driver) (22/24)
22/02/07 23:47:27 INFO Executor: Finished task 23.0 in stage 171.0 (TID 902). 921 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Finished task 23.0 in stage 171.0 (TID 902) in 16 ms on 192.168.1.37 (executor driver) (23/24)
22/02/07 23:47:27 INFO Executor: Finished task 22.0 in stage 171.0 (TID 901). 930 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Finished task 22.0 in stage 171.0 (TID 901) in 17 ms on 192.168.1.37 (executor driver) (24/24)
22/02/07 23:47:27 INFO TaskSchedulerImpl: Removed TaskSet 171.0, whose tasks have all completed, from pool
22/02/07 23:47:27 INFO DAGScheduler: ResultStage 171 (collectAsMap at HoodieSparkEngineContext.java:148) finished in 0.188 s
22/02/07 23:47:27 INFO DAGScheduler: Job 105 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:27 INFO TaskSchedulerImpl: Killing all running tasks in stage 171: Stage finished
22/02/07 23:47:27 INFO DAGScheduler: Job 105 finished: collectAsMap at HoodieSparkEngineContext.java:148, took 0.189512 s
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233159957.deltacommit.requested : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233159957.deltacommit.inflight : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233322808.deltacommit.inflight : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233323715.deltacommit : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233342239.deltacommit.inflight : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233323715.deltacommit.inflight : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232143180.deltacommit : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233342239.deltacommit : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232143180.deltacommit.inflight : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232143180.deltacommit.requested : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233342239.deltacommit.requested : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233322808.deltacommit.requested : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233159957.deltacommit : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232657914.deltacommit.requested : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232723131.deltacommit : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232657914.deltacommit : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232723131.deltacommit.requested : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232658076.deltacommit : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232723131.deltacommit.inflight : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233323715.deltacommit.requested : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232658076.deltacommit.requested : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232658076.deltacommit.inflight : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207233322808.deltacommit : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Archived and deleted instant file /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/20220207232657914.deltacommit.inflight : true
22/02/07 23:47:27 INFO HoodieTimelineArchiveLog: Latest Committed Instant=Option{val=[20220207233342239__deltacommit__COMPLETED]}
22/02/07 23:47:27 INFO HoodieActiveTimeline: Marking instant complete [==>20220207234318886__deltacommit__INFLIGHT]
22/02/07 23:47:27 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234318886.deltacommit.inflight
22/02/07 23:47:27 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234318886.deltacommit
22/02/07 23:47:27 INFO HoodieActiveTimeline: Completed [==>20220207234318886__deltacommit__INFLIGHT]
22/02/07 23:47:27 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53112/v1/hoodie/marker/dir/delete?markerdirpath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220207234318886)
22/02/07 23:47:27 INFO SparkContext: Starting job: collectAsMap at HoodieSparkEngineContext.java:148
22/02/07 23:47:27 INFO DAGScheduler: Got job 106 (collectAsMap at HoodieSparkEngineContext.java:148) with 13 output partitions
22/02/07 23:47:27 INFO DAGScheduler: Final stage: ResultStage 172 (collectAsMap at HoodieSparkEngineContext.java:148)
22/02/07 23:47:27 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:47:27 INFO DAGScheduler: Missing parents: List()
22/02/07 23:47:27 INFO DAGScheduler: Submitting ResultStage 172 (MapPartitionsRDD[334] at mapToPair at HoodieSparkEngineContext.java:145), which has no missing parents
22/02/07 23:47:27 INFO MemoryStore: Block broadcast_141 stored as values in memory (estimated size 99.4 KiB, free 983.6 MiB)
22/02/07 23:47:27 INFO MemoryStore: Block broadcast_141_piece0 stored as bytes in memory (estimated size 35.1 KiB, free 983.5 MiB)
22/02/07 23:47:27 INFO BlockManagerInfo: Added broadcast_141_piece0 in memory on 192.168.1.37:53037 (size: 35.1 KiB, free: 986.6 MiB)
22/02/07 23:47:27 INFO SparkContext: Created broadcast 141 from broadcast at DAGScheduler.scala:1427
22/02/07 23:47:27 INFO DAGScheduler: Submitting 13 missing tasks from ResultStage 172 (MapPartitionsRDD[334] at mapToPair at HoodieSparkEngineContext.java:145) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12))
22/02/07 23:47:27 INFO TaskSchedulerImpl: Adding task set 172.0 with 13 tasks resource profile 0
22/02/07 23:47:27 INFO TaskSetManager: Starting task 0.0 in stage 172.0 (TID 903) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4443 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Starting task 1.0 in stage 172.0 (TID 904) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4443 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Starting task 2.0 in stage 172.0 (TID 905) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4443 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO Executor: Running task 2.0 in stage 172.0 (TID 905)
22/02/07 23:47:27 INFO Executor: Running task 0.0 in stage 172.0 (TID 903)
22/02/07 23:47:27 INFO Executor: Running task 1.0 in stage 172.0 (TID 904)
22/02/07 23:47:27 INFO Executor: Finished task 2.0 in stage 172.0 (TID 905). 870 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 1.0 in stage 172.0 (TID 904). 870 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 3.0 in stage 172.0 (TID 906) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4443 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 2.0 in stage 172.0 (TID 905) in 6 ms on 192.168.1.37 (executor driver) (1/13)
22/02/07 23:47:27 INFO Executor: Running task 3.0 in stage 172.0 (TID 906)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 4.0 in stage 172.0 (TID 907) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4442 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 1.0 in stage 172.0 (TID 904) in 6 ms on 192.168.1.37 (executor driver) (2/13)
22/02/07 23:47:27 INFO Executor: Running task 4.0 in stage 172.0 (TID 907)
22/02/07 23:47:27 INFO Executor: Finished task 0.0 in stage 172.0 (TID 903). 913 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 5.0 in stage 172.0 (TID 908) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4443 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 0.0 in stage 172.0 (TID 903) in 12 ms on 192.168.1.37 (executor driver) (3/13)
22/02/07 23:47:27 INFO Executor: Running task 5.0 in stage 172.0 (TID 908)
22/02/07 23:47:27 INFO Executor: Finished task 4.0 in stage 172.0 (TID 907). 869 bytes result sent to driver
22/02/07 23:47:27 INFO Executor: Finished task 3.0 in stage 172.0 (TID 906). 870 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 6.0 in stage 172.0 (TID 909) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4443 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO Executor: Running task 6.0 in stage 172.0 (TID 909)
22/02/07 23:47:27 INFO TaskSetManager: Starting task 7.0 in stage 172.0 (TID 910) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4443 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO Executor: Running task 7.0 in stage 172.0 (TID 910)
22/02/07 23:47:27 INFO TaskSetManager: Finished task 4.0 in stage 172.0 (TID 907) in 8 ms on 192.168.1.37 (executor driver) (4/13)
22/02/07 23:47:27 INFO TaskSetManager: Finished task 3.0 in stage 172.0 (TID 906) in 8 ms on 192.168.1.37 (executor driver) (5/13)
22/02/07 23:47:27 INFO Executor: Finished task 5.0 in stage 172.0 (TID 908). 870 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 8.0 in stage 172.0 (TID 911) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4442 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 5.0 in stage 172.0 (TID 908) in 7 ms on 192.168.1.37 (executor driver) (6/13)
22/02/07 23:47:27 INFO Executor: Running task 8.0 in stage 172.0 (TID 911)
22/02/07 23:47:27 INFO Executor: Finished task 6.0 in stage 172.0 (TID 909). 870 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 9.0 in stage 172.0 (TID 912) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4442 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 6.0 in stage 172.0 (TID 909) in 6 ms on 192.168.1.37 (executor driver) (7/13)
22/02/07 23:47:27 INFO Executor: Running task 9.0 in stage 172.0 (TID 912)
22/02/07 23:47:27 INFO Executor: Finished task 7.0 in stage 172.0 (TID 910). 870 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 10.0 in stage 172.0 (TID 913) (192.168.1.37, executor driver, partition 10, PROCESS_LOCAL, 4442 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO Executor: Running task 10.0 in stage 172.0 (TID 913)
22/02/07 23:47:27 INFO TaskSetManager: Finished task 7.0 in stage 172.0 (TID 910) in 7 ms on 192.168.1.37 (executor driver) (8/13)
22/02/07 23:47:27 INFO Executor: Finished task 8.0 in stage 172.0 (TID 911). 869 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 11.0 in stage 172.0 (TID 914) (192.168.1.37, executor driver, partition 11, PROCESS_LOCAL, 4446 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO TaskSetManager: Finished task 8.0 in stage 172.0 (TID 911) in 7 ms on 192.168.1.37 (executor driver) (9/13)
22/02/07 23:47:27 INFO Executor: Running task 11.0 in stage 172.0 (TID 914)
22/02/07 23:47:27 INFO Executor: Finished task 9.0 in stage 172.0 (TID 912). 869 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Starting task 12.0 in stage 172.0 (TID 915) (192.168.1.37, executor driver, partition 12, PROCESS_LOCAL, 4442 bytes) taskResourceAssignments Map()
22/02/07 23:47:27 INFO Executor: Running task 12.0 in stage 172.0 (TID 915)
22/02/07 23:47:27 INFO TaskSetManager: Finished task 9.0 in stage 172.0 (TID 912) in 7 ms on 192.168.1.37 (executor driver) (10/13)
22/02/07 23:47:27 INFO Executor: Finished task 10.0 in stage 172.0 (TID 913). 869 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Finished task 10.0 in stage 172.0 (TID 913) in 7 ms on 192.168.1.37 (executor driver) (11/13)
22/02/07 23:47:27 INFO Executor: Finished task 11.0 in stage 172.0 (TID 914). 873 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Finished task 11.0 in stage 172.0 (TID 914) in 6 ms on 192.168.1.37 (executor driver) (12/13)
22/02/07 23:47:27 INFO Executor: Finished task 12.0 in stage 172.0 (TID 915). 869 bytes result sent to driver
22/02/07 23:47:27 INFO TaskSetManager: Finished task 12.0 in stage 172.0 (TID 915) in 7 ms on 192.168.1.37 (executor driver) (13/13)
22/02/07 23:47:27 INFO TaskSchedulerImpl: Removed TaskSet 172.0, whose tasks have all completed, from pool
22/02/07 23:47:27 INFO DAGScheduler: ResultStage 172 (collectAsMap at HoodieSparkEngineContext.java:148) finished in 0.052 s
22/02/07 23:47:27 INFO DAGScheduler: Job 106 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:47:27 INFO TaskSchedulerImpl: Killing all running tasks in stage 172: Stage finished
22/02/07 23:47:27 INFO DAGScheduler: Job 106 finished: collectAsMap at HoodieSparkEngineContext.java:148, took 0.052793 s
22/02/07 23:47:27 INFO FSUtils: Removed directory at /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220207234318886
22/02/07 23:47:27 INFO BaseHoodieWriteClient: Cleaner has been spawned already. Waiting for it to finish
22/02/07 23:47:27 INFO AsyncCleanerService: Waiting for async cleaner to finish
22/02/07 23:47:29 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:47:39 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:47:49 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:47:59 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:48:09 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:48:19 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:48:23 INFO InProcessLockProvider: Thread pool-26-thread-1 FAILED_TO_ACQUIRE in-process lock.
22/02/07 23:48:23 INFO LockManager: Retrying to acquire lock...
22/02/07 23:48:29 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:48:31 INFO LruBlockCache: totalSize=1.49 MB, freeSize=1.42 GB, max=1.42 GB, blockCount=0, accesses=16, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0
22/02/07 23:48:33 INFO TransactionManager: Transaction ending with transaction owner Optional.empty
22/02/07 23:48:33 INFO InProcessLockProvider: Thread pool-26-thread-1 RELEASING in-process lock.
22/02/07 23:48:33 WARN CleanActionExecutor: Failed to perform previous clean operation, instant: [==>20220207234319263__clean__REQUESTED]
org.apache.hudi.exception.HoodieLockException: Thread pool-26-thread-1 FAILED_TO_RELEASE in-process lock.
	at org.apache.hudi.client.transaction.lock.InProcessLockProvider.unlock(InProcessLockProvider.java:97)
	at org.apache.hudi.client.transaction.lock.LockManager.unlock(LockManager.java:88)
	at org.apache.hudi.client.transaction.TransactionManager.endTransaction(TransactionManager.java:80)
	at org.apache.hudi.table.action.clean.CleanActionExecutor.runClean(CleanActionExecutor.java:226)
	at org.apache.hudi.table.action.clean.CleanActionExecutor.runPendingClean(CleanActionExecutor.java:177)
	at org.apache.hudi.table.action.clean.CleanActionExecutor.lambda$execute$6(CleanActionExecutor.java:244)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.apache.hudi.table.action.clean.CleanActionExecutor.execute(CleanActionExecutor.java:238)
	at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.clean(HoodieSparkCopyOnWriteTable.java:336)
	at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:737)
	at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:703)
	at org.apache.hudi.client.AsyncCleanerService.lambda$startService$0(AsyncCleanerService.java:51)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalMonitorStateException
	at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131)
	at org.apache.hudi.client.transaction.lock.InProcessLockProvider.unlock(InProcessLockProvider.java:95)
	... 15 more
22/02/07 23:48:33 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:48:33 INFO BaseHoodieWriteClient: Scheduling cleaning at instant time :20220207234319308
22/02/07 23:48:33 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:48:33 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:48:33 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:48:33 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:48:33 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:48:33 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:48:33 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/07 23:48:33 INFO FileSystemViewManager: Creating remote first table view
22/02/07 23:48:33 INFO FileSystemViewManager: Creating remote view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53112, Timeout=300
22/02/07 23:48:33 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:33 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:33 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:48:33 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/refresh/?basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:33 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:48:33 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:33 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:33 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:48:33 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/compactions/pending/?basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:33 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:48:33 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO FileSystemViewManager: Creating InMemory based view for basePath /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:48:33 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:48:33 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:33 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:48:33 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:48:33 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:48:33 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:33 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:33 INFO RequestHandler: Syncing view as client passed last known instant 20220207234332930 as last known instant but server has the following last instant on timeline :Option{val=[20220207234318886__deltacommit__COMPLETED]}
22/02/07 23:48:33 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:48:33 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:33 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:33 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:48:33 INFO CleanPlanner: Incremental Cleaning mode is enabled. Looking up partition-paths that have since changed since last cleaned at 20220207231514670. New Instant to retain : Option{val=[20220207232657914__deltacommit__COMPLETED]}
22/02/07 23:48:33 INFO CleanPlanner: Total Partitions to clean : 10, with policy KEEP_LATEST_COMMITS
22/02/07 23:48:33 INFO CleanPlanner: Using cleanerParallelism: 10
22/02/07 23:48:34 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:100
22/02/07 23:48:34 INFO DAGScheduler: Got job 107 (collect at HoodieSparkEngineContext.java:100) with 10 output partitions
22/02/07 23:48:34 INFO DAGScheduler: Final stage: ResultStage 173 (collect at HoodieSparkEngineContext.java:100)
22/02/07 23:48:34 INFO DAGScheduler: Parents of final stage: List()
22/02/07 23:48:34 INFO DAGScheduler: Missing parents: List()
22/02/07 23:48:34 INFO DAGScheduler: Submitting ResultStage 173 (MapPartitionsRDD[336] at map at HoodieSparkEngineContext.java:100), which has no missing parents
22/02/07 23:48:34 INFO MemoryStore: Block broadcast_142 stored as values in memory (estimated size 542.9 KiB, free 983.0 MiB)
22/02/07 23:48:34 INFO MemoryStore: Block broadcast_142_piece0 stored as bytes in memory (estimated size 190.8 KiB, free 982.8 MiB)
22/02/07 23:48:34 INFO BlockManagerInfo: Added broadcast_142_piece0 in memory on 192.168.1.37:53037 (size: 190.8 KiB, free: 986.4 MiB)
22/02/07 23:48:34 INFO SparkContext: Created broadcast 142 from broadcast at DAGScheduler.scala:1427
22/02/07 23:48:34 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 173 (MapPartitionsRDD[336] at map at HoodieSparkEngineContext.java:100) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/07 23:48:34 INFO TaskSchedulerImpl: Adding task set 173.0 with 10 tasks resource profile 0
22/02/07 23:48:34 INFO TaskSetManager: Starting task 0.0 in stage 173.0 (TID 916) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/07 23:48:34 INFO TaskSetManager: Starting task 1.0 in stage 173.0 (TID 917) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/07 23:48:34 INFO TaskSetManager: Starting task 2.0 in stage 173.0 (TID 918) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/07 23:48:34 INFO Executor: Running task 2.0 in stage 173.0 (TID 918)
22/02/07 23:48:34 INFO Executor: Running task 1.0 in stage 173.0 (TID 917)
22/02/07 23:48:34 INFO Executor: Running task 0.0 in stage 173.0 (TID 916)
22/02/07 23:48:36 INFO CleanPlanner: Cleaning 2022/1/28, retaining latest 4 commits.
22/02/07 23:48:36 INFO CleanPlanner: Cleaning 2022/1/30, retaining latest 4 commits.
22/02/07 23:48:36 INFO CleanPlanner: Cleaning 2022/1/31, retaining latest 4 commits.
22/02/07 23:48:36 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F1%2F31&maxinstant=20220207232657914&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:36 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F1%2F28&maxinstant=20220207232657914&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:36 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F1%2F30&maxinstant=20220207232657914&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:36 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/31)
22/02/07 23:48:36 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/30)
22/02/07 23:48:36 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/28)
22/02/07 23:48:36 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/07 23:48:36 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/07 23:48:36 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/07 23:48:36 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234726235__rollback__COMPLETED]}
22/02/07 23:48:36 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:36 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:36 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:36 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:36 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:36 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:36 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:48:36 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:48:36 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_129_piece0 on 192.168.1.37:53037 in memory (size: 111.2 KiB, free: 986.5 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_133_piece0 on 192.168.1.37:53037 in memory (size: 3.9 KiB, free: 986.5 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_140_piece0 on 192.168.1.37:53037 in memory (size: 75.7 KiB, free: 986.6 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_127_piece0 on 192.168.1.37:53037 in memory (size: 3.9 KiB, free: 986.6 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_137_piece0 on 192.168.1.37:53037 in memory (size: 46.0 KiB, free: 986.6 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_130_piece0 on 192.168.1.37:53037 in memory (size: 45.8 KiB, free: 986.7 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_134_piece0 on 192.168.1.37:53037 in memory (size: 35.4 KiB, free: 986.7 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_138_piece0 on 192.168.1.37:53037 in memory (size: 3.8 KiB, free: 986.7 MiB)
22/02/07 23:48:36 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=32, NumFileGroups=1, FileGroupsCreationTime=19, StoreTimeTaken=0
22/02/07 23:48:36 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=32, NumFileGroups=1, FileGroupsCreationTime=20, StoreTimeTaken=0
22/02/07 23:48:36 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=32, NumFileGroups=1, FileGroupsCreationTime=19, StoreTimeTaken=0
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_141_piece0 on 192.168.1.37:53037 in memory (size: 35.1 KiB, free: 986.8 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_136_piece0 on 192.168.1.37:53037 in memory (size: 45.8 KiB, free: 986.8 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_132_piece0 on 192.168.1.37:53037 in memory (size: 3.8 KiB, free: 986.8 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_135_piece0 on 192.168.1.37:53037 in memory (size: 111.1 KiB, free: 986.9 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_131_piece0 on 192.168.1.37:53037 in memory (size: 46.0 KiB, free: 987.0 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_139_piece0 on 192.168.1.37:53037 in memory (size: 3.9 KiB, free: 987.0 MiB)
22/02/07 23:48:36 INFO BlockManagerInfo: Removed broadcast_128_piece0 on 192.168.1.37:53037 in memory (size: 35.4 KiB, free: 987.0 MiB)
22/02/07 23:48:36 INFO BlockManager: Removing RDD 269
22/02/07 23:48:36 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:48:36 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:48:36 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:48:36 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234726235__rollback__COMPLETED]}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-0-0', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a delete block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-0-0
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-10-10', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-10-10', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-10-10 at instant 00000000000000
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.2_0-62-477', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.2_0-62-477', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.2_0-62-477 at instant 20220207225442605
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 2
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:36 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO ExternalSpillableMap: Estimated Payload size => 376
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.3_0-118-1011', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.3_0-118-1011', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.3_0-118-1011 at instant 20220207225650146
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:36 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.4_0-174-1547', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.4_0-174-1547', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.4_0-174-1547 at instant 20220207225808195
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:36 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.5_0-63-588', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.5_0-63-588', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.5_0-63-588 at instant 20220207230738729
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:36 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.6_1-0-1', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.6_1-0-1', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a command block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.6_1-0-1
22/02/07 23:48:36 WARN AbstractHoodieLogRecordReader: TargetInstantTime 20220207230441831 invalid or extra rollback command block in file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.6_1-0-1
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of applied rollback blocks 0
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.7_1-0-1', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.7_1-0-1', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a command block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.7_1-0-1
22/02/07 23:48:36 WARN AbstractHoodieLogRecordReader: TargetInstantTime 20220207230113074 invalid or extra rollback command block in file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.7_1-0-1
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of applied rollback blocks 0
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.8_0-59-581', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.8_0-59-581', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.8_0-59-581 at instant 20220207231027283
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:36 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.9_0-87-664', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.9_0-87-664', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.9_0-87-664 at instant 20220207231249456
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:36 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.10_0-138-1198', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.10_0-138-1198', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.10_0-138-1198 at instant 20220207231249333
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:36 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.11_0-198-1756', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.11_0-198-1756', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.11_0-198-1756 at instant 20220207231514670
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:36 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:36 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.12_0-224-1824', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.12_0-224-1824', fileLen=-1}
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.12_0-224-1824 at instant 20220207231826373
22/02/07 23:48:36 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.13_0-273-2345', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.13_0-273-2345', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.13_0-273-2345 at instant 20220207231826271
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.14_0-48-524', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.14_0-48-524', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.14_0-48-524 at instant 20220207232143180
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.15_0-77-619', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.15_0-77-619', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.15_0-77-619 at instant 20220207232723131
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.16_0-94-665', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.16_0-94-665', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.16_0-94-665 at instant 20220207232658076
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.17_0-143-1216', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.17_0-143-1216', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.17_0-143-1216 at instant 20220207232657914
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.18_0-15-44', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.18_0-15-44', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.18_0-15-44 at instant 20220207233109945
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.19_0-14-34', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.19_0-14-34', fileLen=-1}
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.20_0-56-582', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.20_0-56-582', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.20_0-56-582 at instant 20220207233322808
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.21_1-0-1', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.21_1-0-1', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a command block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.21_1-0-1
22/02/07 23:48:37 WARN AbstractHoodieLogRecordReader: TargetInstantTime 20220207233109945 invalid or extra rollback command block in file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.21_1-0-1
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of applied rollback blocks 0
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.22_1-0-1', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.22_1-0-1', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a command block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.22_1-0-1
22/02/07 23:48:37 WARN AbstractHoodieLogRecordReader: TargetInstantTime 20220207233014827 invalid or extra rollback command block in file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.22_1-0-1
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of applied rollback blocks 0
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.23_0-27-204', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.23_0-27-204', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.23_0-27-204 at instant 20220207233828801
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.24_0-31-208', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.24_0-31-208', fileLen=-1}
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.25_0-67-645', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.25_0-67-645', fileLen=-1}
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.26_0-88-733', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.26_0-88-733', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.26_0-88-733 at instant 20220207233505725
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.27_0-102-752', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.27_0-102-752', fileLen=-1}
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.28_0-129-823', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.28_0-129-823', fileLen=-1}
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.29_0-144-854', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.29_0-144-854', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a data block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.29_0-144-854 at instant 20220207234318886
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieLogFormatReader: Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.30_1-0-1', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.30_1-0-1', fileLen=-1}
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Reading a command block from file file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.30_1-0-1
22/02/07 23:48:37 WARN AbstractHoodieLogRecordReader: TargetInstantTime 20220207233705644 invalid or extra rollback command block in file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_00000000000000.log.30_1-0-1
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of applied rollback blocks 0
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Merging the final data blocks
22/02/07 23:48:37 INFO AbstractHoodieLogRecordReader: Number of remaining logblocks to merge 1
22/02/07 23:48:37 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/07 23:48:37 INFO HoodieMergedLogRecordScanner: Number of log files scanned => 31
22/02/07 23:48:37 INFO HoodieMergedLogRecordScanner: MaxMemoryInBytes allowed for compaction => 1073741824
22/02/07 23:48:37 INFO HoodieMergedLogRecordScanner: Number of entries in MemoryBasedMap in ExternalSpillableMap => 11
22/02/07 23:48:37 INFO HoodieMergedLogRecordScanner: Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 4136
22/02/07 23:48:37 INFO HoodieMergedLogRecordScanner: Number of entries in BitCaskDiskMap in ExternalSpillableMap => 0
22/02/07 23:48:37 INFO HoodieMergedLogRecordScanner: Size of file spilled to disk => 0
22/02/07 23:48:37 INFO HoodieBackedTableMetadata: Opened 31 metadata log files (dataset instant=20220207234332930, metadata instant=20220207234726235) in 689 ms
22/02/07 23:48:37 WARN BaseTableMetadata: Metadata record for 2022/1/30 encountered some files to be deleted which was not added before. Ignoring the spurious deletes as the `_hoodie.metadata.ignore.spurious.deletes` config is set to true
22/02/07 23:48:37 WARN BaseTableMetadata: Metadata record for 2022/1/28 encountered some files to be deleted which was not added before. Ignoring the spurious deletes as the `_hoodie.metadata.ignore.spurious.deletes` config is set to true
22/02/07 23:48:37 WARN BaseTableMetadata: Metadata record for 2022/1/31 encountered some files to be deleted which was not added before. Ignoring the spurious deletes as the `_hoodie.metadata.ignore.spurious.deletes` config is set to true
22/02/07 23:48:37 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/30, #files=10
22/02/07 23:48:37 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/28, #files=10
22/02/07 23:48:37 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/31, #files=10
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=10, NumFileGroups=2, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=10, NumFileGroups=2, FileGroupsCreationTime=2, StoreTimeTaken=0
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=10, NumFileGroups=2, FileGroupsCreationTime=2, StoreTimeTaken=0
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F1%2F31&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F1%2F28&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F1%2F30&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO CleanPlanner: 2 patterns used to delete in partition path:2022/1/30
22/02/07 23:48:37 INFO CleanPlanner: 2 patterns used to delete in partition path:2022/1/28
22/02/07 23:48:37 INFO CleanPlanner: 2 patterns used to delete in partition path:2022/1/31
22/02/07 23:48:37 INFO Executor: Finished task 0.0 in stage 173.0 (TID 916). 1281 bytes result sent to driver
22/02/07 23:48:37 INFO Executor: Finished task 1.0 in stage 173.0 (TID 917). 1281 bytes result sent to driver
22/02/07 23:48:37 INFO Executor: Finished task 2.0 in stage 173.0 (TID 918). 1281 bytes result sent to driver
22/02/07 23:48:37 INFO TaskSetManager: Starting task 3.0 in stage 173.0 (TID 919) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/07 23:48:37 INFO TaskSetManager: Starting task 4.0 in stage 173.0 (TID 920) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4341 bytes) taskResourceAssignments Map()
22/02/07 23:48:37 INFO Executor: Running task 3.0 in stage 173.0 (TID 919)
22/02/07 23:48:37 INFO TaskSetManager: Finished task 0.0 in stage 173.0 (TID 916) in 2518 ms on 192.168.1.37 (executor driver) (1/10)
22/02/07 23:48:37 INFO Executor: Running task 4.0 in stage 173.0 (TID 920)
22/02/07 23:48:37 INFO TaskSetManager: Starting task 5.0 in stage 173.0 (TID 921) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/07 23:48:37 INFO TaskSetManager: Finished task 1.0 in stage 173.0 (TID 917) in 2517 ms on 192.168.1.37 (executor driver) (2/10)
22/02/07 23:48:37 INFO Executor: Running task 5.0 in stage 173.0 (TID 921)
22/02/07 23:48:37 INFO TaskSetManager: Finished task 2.0 in stage 173.0 (TID 918) in 2518 ms on 192.168.1.37 (executor driver) (3/10)
22/02/07 23:48:37 INFO CleanPlanner: Cleaning 2022/1/29, retaining latest 4 commits.
22/02/07 23:48:37 INFO CleanPlanner: Cleaning 2022/1/27, retaining latest 4 commits.
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F1%2F29&maxinstant=20220207232657914&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F1%2F27&maxinstant=20220207232657914&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/27)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/29)
22/02/07 23:48:37 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/07 23:48:37 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:48:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:48:37 INFO CleanPlanner: Cleaning 2022/2/2, retaining latest 4 commits.
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F2%2F2&maxinstant=20220207232657914&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (2022/2/2)
22/02/07 23:48:37 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=32, NumFileGroups=1, FileGroupsCreationTime=8, StoreTimeTaken=0
22/02/07 23:48:37 WARN BaseTableMetadata: Metadata record for 2022/1/29 encountered some files to be deleted which was not added before. Ignoring the spurious deletes as the `_hoodie.metadata.ignore.spurious.deletes` config is set to true
22/02/07 23:48:37 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/29, #files=10
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=32, NumFileGroups=1, FileGroupsCreationTime=9, StoreTimeTaken=0
22/02/07 23:48:37 WARN BaseTableMetadata: Metadata record for 2022/1/27 encountered some files to be deleted which was not added before. Ignoring the spurious deletes as the `_hoodie.metadata.ignore.spurious.deletes` config is set to true
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=10, NumFileGroups=2, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/07 23:48:37 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/27, #files=10
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=32, NumFileGroups=1, FileGroupsCreationTime=6, StoreTimeTaken=0
22/02/07 23:48:37 WARN BaseTableMetadata: Metadata record for 2022/2/2 encountered some files to be deleted which was not added before. Ignoring the spurious deletes as the `_hoodie.metadata.ignore.spurious.deletes` config is set to true
22/02/07 23:48:37 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/2/2, #files=10
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=10, NumFileGroups=2, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=10, NumFileGroups=2, FileGroupsCreationTime=0, StoreTimeTaken=0
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F1%2F29&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F2%2F2&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F1%2F27&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO CleanPlanner: 2 patterns used to delete in partition path:2022/2/2
22/02/07 23:48:37 INFO Executor: Finished task 4.0 in stage 173.0 (TID 920). 1235 bytes result sent to driver
22/02/07 23:48:37 INFO CleanPlanner: 2 patterns used to delete in partition path:2022/1/27
22/02/07 23:48:37 INFO CleanPlanner: 2 patterns used to delete in partition path:2022/1/29
22/02/07 23:48:37 INFO TaskSetManager: Starting task 6.0 in stage 173.0 (TID 922) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/07 23:48:37 INFO Executor: Finished task 3.0 in stage 173.0 (TID 919). 1238 bytes result sent to driver
22/02/07 23:48:37 INFO Executor: Running task 6.0 in stage 173.0 (TID 922)
22/02/07 23:48:37 INFO Executor: Finished task 5.0 in stage 173.0 (TID 921). 1238 bytes result sent to driver
22/02/07 23:48:37 INFO TaskSetManager: Starting task 7.0 in stage 173.0 (TID 923) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4341 bytes) taskResourceAssignments Map()
22/02/07 23:48:37 INFO Executor: Running task 7.0 in stage 173.0 (TID 923)
22/02/07 23:48:37 INFO TaskSetManager: Finished task 4.0 in stage 173.0 (TID 920) in 65 ms on 192.168.1.37 (executor driver) (4/10)
22/02/07 23:48:37 INFO TaskSetManager: Finished task 3.0 in stage 173.0 (TID 919) in 66 ms on 192.168.1.37 (executor driver) (5/10)
22/02/07 23:48:37 INFO TaskSetManager: Starting task 8.0 in stage 173.0 (TID 924) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/07 23:48:37 INFO TaskSetManager: Finished task 5.0 in stage 173.0 (TID 921) in 67 ms on 192.168.1.37 (executor driver) (6/10)
22/02/07 23:48:37 INFO Executor: Running task 8.0 in stage 173.0 (TID 924)
22/02/07 23:48:37 INFO CleanPlanner: Cleaning 2022/1/24, retaining latest 4 commits.
22/02/07 23:48:37 INFO CleanPlanner: Cleaning 2022/2/1, retaining latest 4 commits.
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F1%2F24&maxinstant=20220207232657914&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F2%2F1&maxinstant=20220207232657914&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO CleanPlanner: Cleaning 2022/1/26, retaining latest 4 commits.
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F1%2F26&maxinstant=20220207232657914&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (2022/2/1)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/24)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/26)
22/02/07 23:48:37 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/07 23:48:37 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/07 23:48:37 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=32, NumFileGroups=1, FileGroupsCreationTime=9, StoreTimeTaken=0
22/02/07 23:48:37 WARN BaseTableMetadata: Metadata record for 2022/1/26 encountered some files to be deleted which was not added before. Ignoring the spurious deletes as the `_hoodie.metadata.ignore.spurious.deletes` config is set to true
22/02/07 23:48:37 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/26, #files=10
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=32, NumFileGroups=1, FileGroupsCreationTime=9, StoreTimeTaken=0
22/02/07 23:48:37 WARN BaseTableMetadata: Metadata record for 2022/2/1 encountered some files to be deleted which was not added before. Ignoring the spurious deletes as the `_hoodie.metadata.ignore.spurious.deletes` config is set to true
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=10, NumFileGroups=2, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/07 23:48:37 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/2/1, #files=10
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=32, NumFileGroups=1, FileGroupsCreationTime=11, StoreTimeTaken=0
22/02/07 23:48:37 WARN BaseTableMetadata: Metadata record for 2022/1/24 encountered some files to be deleted which was not added before. Ignoring the spurious deletes as the `_hoodie.metadata.ignore.spurious.deletes` config is set to true
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=10, NumFileGroups=2, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/07 23:48:37 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/24, #files=10
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=10, NumFileGroups=2, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F1%2F26&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F2%2F1&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F1%2F24&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO CleanPlanner: 2 patterns used to delete in partition path:2022/1/24
22/02/07 23:48:37 INFO Executor: Finished task 6.0 in stage 173.0 (TID 922). 1238 bytes result sent to driver
22/02/07 23:48:37 INFO CleanPlanner: 2 patterns used to delete in partition path:2022/1/26
22/02/07 23:48:37 INFO Executor: Finished task 8.0 in stage 173.0 (TID 924). 1238 bytes result sent to driver
22/02/07 23:48:37 INFO TaskSetManager: Starting task 9.0 in stage 173.0 (TID 925) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/07 23:48:37 INFO Executor: Running task 9.0 in stage 173.0 (TID 925)
22/02/07 23:48:37 INFO TaskSetManager: Finished task 6.0 in stage 173.0 (TID 922) in 64 ms on 192.168.1.37 (executor driver) (7/10)
22/02/07 23:48:37 INFO TaskSetManager: Finished task 8.0 in stage 173.0 (TID 924) in 63 ms on 192.168.1.37 (executor driver) (8/10)
22/02/07 23:48:37 INFO CleanPlanner: 2 patterns used to delete in partition path:2022/2/1
22/02/07 23:48:37 INFO Executor: Finished task 7.0 in stage 173.0 (TID 923). 1235 bytes result sent to driver
22/02/07 23:48:37 INFO TaskSetManager: Finished task 7.0 in stage 173.0 (TID 923) in 67 ms on 192.168.1.37 (executor driver) (9/10)
22/02/07 23:48:37 INFO CleanPlanner: Cleaning 2022/1/25, retaining latest 4 commits.
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F1%2F25&maxinstant=20220207232657914&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/25)
22/02/07 23:48:37 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
22/02/07 23:48:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/07 23:48:37 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=32, NumFileGroups=1, FileGroupsCreationTime=10, StoreTimeTaken=0
22/02/07 23:48:37 WARN BaseTableMetadata: Metadata record for 2022/1/25 encountered some files to be deleted which was not added before. Ignoring the spurious deletes as the `_hoodie.metadata.ignore.spurious.deletes` config is set to true
22/02/07 23:48:37 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/25, #files=10
22/02/07 23:48:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=10, NumFileGroups=2, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/07 23:48:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53112/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F1%2F25&basepath=%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220207234332930&timelinehash=1b646b783f0c4ec737d0b5179065dbdf0d353e634f61b8bf280461a51c5ac166)
22/02/07 23:48:37 INFO CleanPlanner: 2 patterns used to delete in partition path:2022/1/25
22/02/07 23:48:37 INFO Executor: Finished task 9.0 in stage 173.0 (TID 925). 1238 bytes result sent to driver
22/02/07 23:48:37 INFO TaskSetManager: Finished task 9.0 in stage 173.0 (TID 925) in 69 ms on 192.168.1.37 (executor driver) (10/10)
22/02/07 23:48:37 INFO TaskSchedulerImpl: Removed TaskSet 173.0, whose tasks have all completed, from pool
22/02/07 23:48:37 INFO DAGScheduler: ResultStage 173 (collect at HoodieSparkEngineContext.java:100) finished in 2.998 s
22/02/07 23:48:37 INFO DAGScheduler: Job 107 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/07 23:48:37 INFO TaskSchedulerImpl: Killing all running tasks in stage 173: Stage finished
22/02/07 23:48:37 INFO DAGScheduler: Job 107 finished: collect at HoodieSparkEngineContext.java:100, took 3.011902 s
22/02/07 23:48:37 INFO CleanPlanner: Requesting Cleaning with instant time [==>20220207234319308__clean__REQUESTED]
22/02/07 23:48:37 INFO BaseHoodieWriteClient: Cleaner started
22/02/07 23:48:37 INFO BaseHoodieWriteClient: Cleaned failed attempts if any
22/02/07 23:48:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:37 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:48:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:37 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:48:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:37 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:48:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:48:37 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:48:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:48:37 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/07 23:48:37 INFO FileSystemViewManager: Creating remote first table view
22/02/07 23:48:37 INFO HoodieHeartbeatClient: Heartbeat not found in internal map, falling back to reading from DFS
22/02/07 23:48:37 WARN HoodieHeartbeatClient: Heartbeat expired, currentTime = 1644306517588, last heartbeat = Heartbeat{instantTime='20220207234228129', isHeartbeatStarted=false, isHeartbeatStopped=false, lastHeartbeatTime=1644306148000, numHeartbeats=0, timer=java.util.Timer@3f0d0d92}, heartbeat interval = 60000
22/02/07 23:48:37 INFO BaseHoodieWriteClient: Begin rollback of instant 20220207234228129
22/02/07 23:48:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:37 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:48:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:37 INFO HoodieTableMetaClient: Loading Active commit timeline for /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220207234332930__rollback__COMPLETED]}
22/02/07 23:48:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:37 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/07 23:48:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/07 23:48:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:48:37 INFO HoodieTableConfig: Loading table properties from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/07 23:48:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/07 23:48:37 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/07 23:48:37 INFO FileSystemViewManager: Creating remote first table view
22/02/07 23:48:37 INFO BaseHoodieWriteClient: Scheduling Rollback at instant time :20220207234837604
22/02/07 23:48:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220207234837604__rollback__REQUESTED]}
22/02/07 23:48:37 INFO BaseRollbackPlanActionExecutor: Requesting Rollback with instant time [==>20220207234837604__rollback__REQUESTED]
22/02/07 23:48:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220207234837604__rollback__REQUESTED]}
22/02/07 23:48:37 INFO HoodieActiveTimeline: Checking for file exists ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234837604.rollback.requested
22/02/07 23:48:37 INFO HoodieActiveTimeline: Create new file for toInstant ?/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220207234837604.rollback.inflight
22/02/07 23:48:37 INFO MergeOnReadRollbackActionExecutor: Rolling back instant [==>20220207234228129__deltacommit__REQUESTED]
22/02/07 23:48:37 INFO BaseRollbackActionExecutor: Deleting instant=[==>20220207234228129__deltacommit__REQUESTED]
22/02/07 23:48:37 INFO HoodieActiveTimeline: Deleting instant [==>20220207234228129__deltacommit__REQUESTED]
22/02/07 23:48:37 INFO HoodieActiveTimeline: Removed instant [==>20220207234228129__deltacommit__REQUESTED]
22/02/07 23:48:37 INFO BaseRollbackActionExecutor: Deleted pending commit [==>20220207234228129__deltacommit__REQUESTED]
22/02/07 23:48:37 INFO MergeOnReadRollbackActionExecutor: Time(in ms) taken to finish rollback 2
22/02/07 23:48:37 INFO BaseRollbackActionExecutor: Rolled back inflight instant 20220207234228129
22/02/07 23:48:37 INFO BaseRollbackActionExecutor: Index rolled back for commits [==>20220207234228129__deltacommit__REQUESTED]
22/02/07 23:48:37 INFO TransactionManager: Transaction starting for Optional.empty with latest completed transaction instant Optional.empty
22/02/07 23:48:37 INFO LockManager: LockProvider org.apache.hudi.client.transaction.lock.InProcessLockProvider
22/02/07 23:48:37 INFO InProcessLockProvider: Thread pool-26-thread-1 ACQUIRING in-process lock.
22/02/07 23:48:39 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:48:49 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:48:59 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:49:09 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:49:19 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:49:29 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:49:37 INFO InProcessLockProvider: Thread pool-26-thread-1 FAILED_TO_ACQUIRE in-process lock.
22/02/07 23:49:37 INFO LockManager: Retrying to acquire lock...
22/02/07 23:49:39 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/07 23:49:47 INFO TransactionManager: Transaction ending with transaction owner Optional.empty
22/02/07 23:49:47 INFO InProcessLockProvider: Thread pool-26-thread-1 RELEASING in-process lock.
22/02/07 23:49:47 ERROR HoodieAsyncService: Service shutdown with error
java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4 commits 20220207234228129
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:89)
	at org.apache.hudi.client.AsyncCleanerService.waitForCompletion(AsyncCleanerService.java:71)
	at org.apache.hudi.client.BaseHoodieWriteClient.autoCleanOnCommit(BaseHoodieWriteClient.java:523)
	at org.apache.hudi.client.BaseHoodieWriteClient.postCommit(BaseHoodieWriteClient.java:462)
	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:200)
	at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:127)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:578)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:323)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:643)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4 commits 20220207234228129
	at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:666)
	at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:971)
	at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:954)
	at org.apache.hudi.client.BaseHoodieWriteClient.lambda$clean$33796fd2$1(BaseHoodieWriteClient.java:736)
	at org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:135)
	at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:735)
	at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:703)
	at org.apache.hudi.client.AsyncCleanerService.lambda$startService$0(AsyncCleanerService.java:51)
	... 4 more
Caused by: org.apache.hudi.exception.HoodieLockException: Thread pool-26-thread-1 FAILED_TO_RELEASE in-process lock.
	at org.apache.hudi.client.transaction.lock.InProcessLockProvider.unlock(InProcessLockProvider.java:97)
	at org.apache.hudi.client.transaction.lock.LockManager.unlock(LockManager.java:88)
	at org.apache.hudi.client.transaction.TransactionManager.endTransaction(TransactionManager.java:80)
	at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.finishRollback(BaseRollbackActionExecutor.java:252)
	at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:122)
	at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:144)
	at org.apache.hudi.table.HoodieSparkMergeOnReadTable.rollback(HoodieSparkMergeOnReadTable.java:170)
	at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:651)
	... 11 more
Caused by: java.lang.IllegalMonitorStateException
	at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131)
	at org.apache.hudi.client.transaction.lock.InProcessLockProvider.unlock(InProcessLockProvider.java:95)
	... 18 more
22/02/07 23:49:47 INFO HoodieHeartbeatClient: Stopping heartbeat for instant 20220207234318886
22/02/07 23:49:47 INFO HoodieHeartbeatClient: Stopped heartbeat for instant 20220207234318886
22/02/07 23:49:47 INFO HeartbeatUtils: Deleted the heartbeat for instant 20220207234318886
22/02/07 23:49:47 INFO HoodieHeartbeatClient: Deleted heartbeat file for instant 20220207234318886
22/02/07 23:49:47 INFO TransactionManager: Transaction ending with transaction owner Option{val=[==>20220207234318886__deltacommit__INFLIGHT]}
22/02/07 23:49:47 INFO InProcessLockProvider: Thread pool-22-thread-1 RELEASING in-process lock.
22/02/07 23:49:47 INFO InProcessLockProvider: Thread pool-22-thread-1 RELEASED in-process lock.
22/02/07 23:49:47 INFO TransactionManager: Transaction ended with transaction owner Option{val=[==>20220207234318886__deltacommit__INFLIGHT]}
22/02/07 23:49:47 ERROR HoodieDeltaStreamer: Shutting down delta-sync due to exception
org.apache.hudi.exception.HoodieException: Error waiting for async cleaning to finish
	at org.apache.hudi.client.AsyncCleanerService.waitForCompletion(AsyncCleanerService.java:73)
	at org.apache.hudi.client.BaseHoodieWriteClient.autoCleanOnCommit(BaseHoodieWriteClient.java:523)
	at org.apache.hudi.client.BaseHoodieWriteClient.postCommit(BaseHoodieWriteClient.java:462)
	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:200)
	at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:127)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:578)
	at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:323)
	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:643)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4 commits 20220207234228129
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at org.apache.hudi.async.HoodieAsyncService.waitForShutdown(HoodieAsyncService.java:89)
	at org.apache.hudi.client.AsyncCleanerService.waitForCompletion(AsyncCleanerService.java:71)
	... 11 more
Caused by: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback /Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4 commits 20220207234228129
	at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:666)
	at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:971)
	at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:954)
	at org.apache.hudi.client.BaseHoodieWriteClient.lambda$clean$33796fd2$1(BaseHoodieWriteClient.java:736)
	at org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:135)
	at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:735)
	at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:703)
	at org.apache.hudi.client.AsyncCleanerService.lambda$startService$0(AsyncCleanerService.java:51)
	... 4 more
Caused by: org.apache.hudi.exception.HoodieLockException: Thread pool-26-thread-1 FAILED_TO_RELEASE in-process lock.
	at org.apache.hudi.client.transaction.lock.InProcessLockProvider.unlock(InProcessLockProvider.java:97)
	at org.apache.hudi.client.transaction.lock.LockManager.unlock(LockManager.java:88)
	at org.apache.hudi.client.transaction.TransactionManager.endTransaction(TransactionManager.java:80)
	at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.finishRollback(BaseRollbackActionExecutor.java:252)
	at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:122)
	at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:144)
	at org.apache.hudi.table.HoodieSparkMergeOnReadTable.rollback(HoodieSparkMergeOnReadTable.java:170)
	at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:651)
	... 11 more
Caused by: java.lang.IllegalMonitorStateException
	at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131)
	at org.apache.hudi.client.transaction.lock.InProcessLockProvider.unlock(InProcessLockProvider.java:95)
	... 18 more
22/02/07 23:49:47 INFO HoodieDeltaStreamer: Delta Sync shutdown. Error ?true
22/02/07 23:49:47 WARN HoodieDeltaStreamer: Gracefully shutting down compactor
22/02/07 23:49:49 INFO AsyncCompactService: Compactor shutting down properly!!