Skip to content

Instantly share code, notes, and snippets.

@yihua

yihua/deltastreamer Secret

Created February 9, 2022 01:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yihua/e00a1caddacbdc570b5b757049750f39 to your computer and use it in GitHub Desktop.
Save yihua/e00a1caddacbdc570b5b757049750f39 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
deltastreamer
Start deltastreamer run 2 ...
:: loading settings :: url = jar:file:/Users/ethan/Work/lib/spark-3.2.0-bin-hadoop3.2/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /Users/ethan/.ivy2/cache
The jars for the packages stored in: /Users/ethan/.ivy2/jars
org.apache.spark#spark-avro_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-7a11fc06-8202-47c3-86e8-57e720ace6e1;1.0
confs: [default]
found org.apache.spark#spark-avro_2.12;3.2.0 in spark-list
found org.tukaani#xz;1.8 in local-m2-cache
found org.spark-project.spark#unused;1.0.0 in local-m2-cache
:: resolution report :: resolve 146ms :: artifacts dl 5ms
:: modules in use:
org.apache.spark#spark-avro_2.12;3.2.0 from spark-list in [default]
org.spark-project.spark#unused;1.0.0 from local-m2-cache in [default]
org.tukaani#xz;1.8 from local-m2-cache in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 3 | 0 | 0 | 0 || 3 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-7a11fc06-8202-47c3-86e8-57e720ace6e1
confs: [default]
0 artifacts copied, 3 already retrieved (0kB/4ms)
22/02/08 16:51:30 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (org.apache.hudi.utilities.deltastreamer.SchedulerConfGenerator).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/02/08 16:51:30 INFO SparkContext: Running Spark version 3.2.0
22/02/08 16:51:30 INFO ResourceUtils: ==============================================================
22/02/08 16:51:30 INFO ResourceUtils: No custom resources configured for spark.driver.
22/02/08 16:51:30 INFO ResourceUtils: ==============================================================
22/02/08 16:51:30 INFO SparkContext: Submitted application: delta-streamer-metadata_test_ds_mor_continuous_table_4
22/02/08 16:51:30 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/02/08 16:51:30 INFO ResourceProfile: Limiting resource is cpu
22/02/08 16:51:30 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/02/08 16:51:30 INFO SecurityManager: Changing view acls to: ethan
22/02/08 16:51:30 INFO SecurityManager: Changing modify acls to: ethan
22/02/08 16:51:30 INFO SecurityManager: Changing view acls groups to:
22/02/08 16:51:30 INFO SecurityManager: Changing modify acls groups to:
22/02/08 16:51:30 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ethan); groups with view permissions: Set(); users with modify permissions: Set(ethan); groups with modify permissions: Set()
22/02/08 16:51:30 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
22/02/08 16:51:30 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
22/02/08 16:51:30 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
22/02/08 16:51:31 INFO Utils: Successfully started service 'sparkDriver' on port 53041.
22/02/08 16:51:31 INFO SparkEnv: Registering MapOutputTracker
22/02/08 16:51:31 INFO SparkEnv: Registering BlockManagerMaster
22/02/08 16:51:31 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/02/08 16:51:31 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/02/08 16:51:31 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
22/02/08 16:51:31 INFO DiskBlockManager: Created local directory at /private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/blockmgr-8f9a77ac-936c-485e-bb8f-8d94bcda78c3
22/02/08 16:51:31 INFO MemoryStore: MemoryStore started with capacity 2004.6 MiB
22/02/08 16:51:31 INFO SparkEnv: Registering OutputCommitCoordinator
22/02/08 16:51:31 INFO Utils: Successfully started service 'SparkUI' on port 6680.
22/02/08 16:51:31 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.37:6680
22/02/08 16:51:31 INFO SparkContext: Added JAR file:///Users/ethan/Work/repo/hudi-benchmarks/target/hudi-benchmarks-0.1-SNAPSHOT.jar at spark://192.168.1.37:53041/jars/hudi-benchmarks-0.1-SNAPSHOT.jar with timestamp 1644367890739
22/02/08 16:51:31 INFO SparkContext: Added JAR file:///Users/ethan/.ivy2/jars/org.apache.spark_spark-avro_2.12-3.2.0.jar at spark://192.168.1.37:53041/jars/org.apache.spark_spark-avro_2.12-3.2.0.jar with timestamp 1644367890739
22/02/08 16:51:31 INFO SparkContext: Added JAR file:///Users/ethan/.ivy2/jars/org.tukaani_xz-1.8.jar at spark://192.168.1.37:53041/jars/org.tukaani_xz-1.8.jar with timestamp 1644367890739
22/02/08 16:51:31 INFO SparkContext: Added JAR file:///Users/ethan/.ivy2/jars/org.spark-project.spark_unused-1.0.0.jar at spark://192.168.1.37:53041/jars/org.spark-project.spark_unused-1.0.0.jar with timestamp 1644367890739
22/02/08 16:51:31 INFO SparkContext: Added JAR file:/Users/ethan/Work/lib/hudi-utilities-bundle_2.12-0.11.0-SNAPSHOT-error-inj.jar at spark://192.168.1.37:53041/jars/hudi-utilities-bundle_2.12-0.11.0-SNAPSHOT-error-inj.jar with timestamp 1644367890739
22/02/08 16:51:31 INFO Executor: Starting executor ID driver on host 192.168.1.37
22/02/08 16:51:31 INFO Executor: Fetching spark://192.168.1.37:53041/jars/org.spark-project.spark_unused-1.0.0.jar with timestamp 1644367890739
22/02/08 16:51:31 INFO TransportClientFactory: Successfully created connection to /192.168.1.37:53041 after 23 ms (0 ms spent in bootstraps)
22/02/08 16:51:31 INFO Utils: Fetching spark://192.168.1.37:53041/jars/org.spark-project.spark_unused-1.0.0.jar to /private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/spark-b250a14c-7f58-499b-b277-6d128d9f93d4/userFiles-aa57dfe8-8e68-4020-aefa-37963a7f08f8/fetchFileTemp5821164490728343261.tmp
22/02/08 16:51:31 INFO Executor: Adding file:/private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/spark-b250a14c-7f58-499b-b277-6d128d9f93d4/userFiles-aa57dfe8-8e68-4020-aefa-37963a7f08f8/org.spark-project.spark_unused-1.0.0.jar to class loader
22/02/08 16:51:31 INFO Executor: Fetching spark://192.168.1.37:53041/jars/hudi-utilities-bundle_2.12-0.11.0-SNAPSHOT-error-inj.jar with timestamp 1644367890739
22/02/08 16:51:31 INFO Utils: Fetching spark://192.168.1.37:53041/jars/hudi-utilities-bundle_2.12-0.11.0-SNAPSHOT-error-inj.jar to /private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/spark-b250a14c-7f58-499b-b277-6d128d9f93d4/userFiles-aa57dfe8-8e68-4020-aefa-37963a7f08f8/fetchFileTemp8071790277708104551.tmp
22/02/08 16:51:31 INFO Executor: Adding file:/private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/spark-b250a14c-7f58-499b-b277-6d128d9f93d4/userFiles-aa57dfe8-8e68-4020-aefa-37963a7f08f8/hudi-utilities-bundle_2.12-0.11.0-SNAPSHOT-error-inj.jar to class loader
22/02/08 16:51:31 INFO Executor: Fetching spark://192.168.1.37:53041/jars/org.tukaani_xz-1.8.jar with timestamp 1644367890739
22/02/08 16:51:31 INFO Utils: Fetching spark://192.168.1.37:53041/jars/org.tukaani_xz-1.8.jar to /private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/spark-b250a14c-7f58-499b-b277-6d128d9f93d4/userFiles-aa57dfe8-8e68-4020-aefa-37963a7f08f8/fetchFileTemp5877655294202730543.tmp
22/02/08 16:51:31 INFO Executor: Adding file:/private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/spark-b250a14c-7f58-499b-b277-6d128d9f93d4/userFiles-aa57dfe8-8e68-4020-aefa-37963a7f08f8/org.tukaani_xz-1.8.jar to class loader
22/02/08 16:51:31 INFO Executor: Fetching spark://192.168.1.37:53041/jars/hudi-benchmarks-0.1-SNAPSHOT.jar with timestamp 1644367890739
22/02/08 16:51:31 INFO Utils: Fetching spark://192.168.1.37:53041/jars/hudi-benchmarks-0.1-SNAPSHOT.jar to /private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/spark-b250a14c-7f58-499b-b277-6d128d9f93d4/userFiles-aa57dfe8-8e68-4020-aefa-37963a7f08f8/fetchFileTemp1685532935449261139.tmp
22/02/08 16:51:31 INFO Executor: Adding file:/private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/spark-b250a14c-7f58-499b-b277-6d128d9f93d4/userFiles-aa57dfe8-8e68-4020-aefa-37963a7f08f8/hudi-benchmarks-0.1-SNAPSHOT.jar to class loader
22/02/08 16:51:31 INFO Executor: Fetching spark://192.168.1.37:53041/jars/org.apache.spark_spark-avro_2.12-3.2.0.jar with timestamp 1644367890739
22/02/08 16:51:31 INFO Utils: Fetching spark://192.168.1.37:53041/jars/org.apache.spark_spark-avro_2.12-3.2.0.jar to /private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/spark-b250a14c-7f58-499b-b277-6d128d9f93d4/userFiles-aa57dfe8-8e68-4020-aefa-37963a7f08f8/fetchFileTemp7161140638313476747.tmp
22/02/08 16:51:31 INFO Executor: Adding file:/private/var/folders/60/wk8qzx310fd32b2dp7mhzvdc0000gn/T/spark-b250a14c-7f58-499b-b277-6d128d9f93d4/userFiles-aa57dfe8-8e68-4020-aefa-37963a7f08f8/org.apache.spark_spark-avro_2.12-3.2.0.jar to class loader
22/02/08 16:51:31 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53051.
22/02/08 16:51:31 INFO NettyBlockTransferService: Server created on 192.168.1.37:53051
22/02/08 16:51:31 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/02/08 16:51:31 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.1.37, 53051, None)
22/02/08 16:51:31 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.37:53051 with 2004.6 MiB RAM, BlockManagerId(driver, 192.168.1.37, 53051, None)
22/02/08 16:51:31 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.1.37, 53051, None)
22/02/08 16:51:31 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.1.37, 53051, None)
22/02/08 16:51:31 INFO SingleEventLogFileWriter: Logging events to file:/Users/ethan/Work/data/hudi/spark-logs/local-1644367891423.inprogress
22/02/08 16:51:32 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
22/02/08 16:51:32 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
22/02/08 16:51:32 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect.
22/02/08 16:51:32 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:32 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:32 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:32 INFO HoodieDeltaStreamer: Creating delta streamer with configs:
benchmark.input.source.path: file:/Users/ethan/Work/data/hudi/benchmark_sample_upserts2
hoodie.bulkinsert.shuffle.parallelism: 40
hoodie.clean.async: true
hoodie.cleaner.commits.retained: 4
hoodie.cleaner.policy.failed.writes: LAZY
hoodie.clustering.async.enabled: false
hoodie.clustering.async.max.commits: 6
hoodie.compact.inline.max.delta.commits: 3
hoodie.datasource.write.partitionpath.field: partition
hoodie.datasource.write.reconcile.schema: false
hoodie.datasource.write.recordkey.field: key
hoodie.delete.shuffle.parallelism: 40
hoodie.deltastreamer.schemaprovider.source.schema.file: file:/Users/ethan/Work/scripts/benchmark_schema.avsc
hoodie.deltastreamer.schemaprovider.target.schema.file: file:/Users/ethan/Work/scripts/benchmark_schema.avsc
hoodie.deltastreamer.source.dfs.root: file:/Users/ethan/Work/data/hudi/benchmark_sample_upserts2
hoodie.insert.shuffle.parallelism: 40
hoodie.keep.max.commits: 7
hoodie.keep.min.commits: 5
hoodie.metadata.compact.max.delta.commits: 5
hoodie.metadata.keep.max.commits: 12
hoodie.metadata.keep.min.commits: 8
hoodie.upsert.shuffle.parallelism: 40
hoodie.write.concurrency.mode: optimistic_concurrency_control
hoodie.write.lock.provider: org.apache.hudi.client.transaction.lock.InProcessLockProvider
22/02/08 16:51:32 INFO HoodieSparkKeyGeneratorFactory: The value of hoodie.datasource.write.keygenerator.type is empty, use SIMPLE
22/02/08 16:51:32 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:32 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:32 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:32 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830__rollback__COMPLETED]}
22/02/08 16:51:32 INFO HoodieDeltaStreamer: Setting Spark Pool name for delta-sync to hoodiedeltasync
22/02/08 16:51:32 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:32 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:32 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:32 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830__rollback__COMPLETED]}
22/02/08 16:51:32 INFO DeltaSync: Checkpoint to resume from : Option{val=4}
22/02/08 16:51:32 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
22/02/08 16:51:32 INFO SharedState: Warehouse path is 'file:/Users/ethan/Work/scripts/spark-warehouse'.
22/02/08 16:51:33 INFO InMemoryFileIndex: It took 25 ms to list leaf files for 5 paths.
22/02/08 16:51:33 INFO SparkContext: Starting job: parquet at BenchmarkDataSource.scala:31
22/02/08 16:51:33 INFO DAGScheduler: Got job 0 (parquet at BenchmarkDataSource.scala:31) with 1 output partitions
22/02/08 16:51:33 INFO DAGScheduler: Final stage: ResultStage 0 (parquet at BenchmarkDataSource.scala:31)
22/02/08 16:51:33 INFO DAGScheduler: Parents of final stage: List()
22/02/08 16:51:33 INFO DAGScheduler: Missing parents: List()
22/02/08 16:51:33 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at parquet at BenchmarkDataSource.scala:31), which has no missing parents
22/02/08 16:51:33 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 102.5 KiB, free 2004.5 MiB)
22/02/08 16:51:33 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 36.8 KiB, free 2004.5 MiB)
22/02/08 16:51:33 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.1.37:53051 (size: 36.8 KiB, free: 2004.6 MiB)
22/02/08 16:51:33 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1427
22/02/08 16:51:33 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at parquet at BenchmarkDataSource.scala:31) (first 15 tasks are for partitions Vector(0))
22/02/08 16:51:33 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0
22/02/08 16:51:33 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4470 bytes) taskResourceAssignments Map()
22/02/08 16:51:33 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
22/02/08 16:51:34 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1532 bytes result sent to driver
22/02/08 16:51:34 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 498 ms on 192.168.1.37 (executor driver) (1/1)
22/02/08 16:51:34 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
22/02/08 16:51:34 INFO DAGScheduler: ResultStage 0 (parquet at BenchmarkDataSource.scala:31) finished in 0.864 s
22/02/08 16:51:34 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:51:34 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
22/02/08 16:51:34 INFO DAGScheduler: Job 0 finished: parquet at BenchmarkDataSource.scala:31, took 0.910414 s
22/02/08 16:51:34 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 192.168.1.37:53051 in memory (size: 36.8 KiB, free: 2004.6 MiB)
22/02/08 16:51:35 INFO FileSourceStrategy: Pushed Filters:
22/02/08 16:51:35 INFO FileSourceStrategy: Post-Scan Filters:
22/02/08 16:51:35 INFO FileSourceStrategy: Output Data Schema: struct<key: string, partition: string, ts: bigint, textField: string, decimalField: float ... 7 more fields>
22/02/08 16:51:35 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 354.3 KiB, free 2004.3 MiB)
22/02/08 16:51:35 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 35.3 KiB, free 2004.2 MiB)
22/02/08 16:51:35 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.1.37:53051 (size: 35.3 KiB, free: 2004.6 MiB)
22/02/08 16:51:35 INFO SparkContext: Created broadcast 1 from toRdd at HoodieSparkUtils.scala:158
22/02/08 16:51:35 INFO FileSourceScanExec: Planning scan with bin packing, max size: 131086737 bytes, open cost is considered as scanning 4194304 bytes.
22/02/08 16:51:36 INFO SparkContext: Starting job: isEmpty at DeltaSync.java:472
22/02/08 16:51:36 INFO DAGScheduler: Got job 1 (isEmpty at DeltaSync.java:472) with 1 output partitions
22/02/08 16:51:36 INFO DAGScheduler: Final stage: ResultStage 1 (isEmpty at DeltaSync.java:472)
22/02/08 16:51:36 INFO DAGScheduler: Parents of final stage: List()
22/02/08 16:51:36 INFO DAGScheduler: Missing parents: List()
22/02/08 16:51:36 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[6] at mapPartitions at HoodieSparkUtils.scala:159), which has no missing parents
22/02/08 16:51:36 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 44.9 KiB, free 2004.2 MiB)
22/02/08 16:51:36 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 16.7 KiB, free 2004.2 MiB)
22/02/08 16:51:36 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.1.37:53051 (size: 16.7 KiB, free: 2004.5 MiB)
22/02/08 16:51:36 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1427
22/02/08 16:51:36 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[6] at mapPartitions at HoodieSparkUtils.scala:159) (first 15 tasks are for partitions Vector(0))
22/02/08 16:51:36 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks resource profile 0
22/02/08 16:51:36 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4955 bytes) taskResourceAssignments Map()
22/02/08 16:51:36 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
22/02/08 16:51:36 INFO CodeGenerator: Code generated in 177.08003 ms
22/02/08 16:51:36 INFO FileScanRDD: Reading File path: file:///Users/ethan/Work/data/hudi/benchmark_sample_upserts2/4/part-00002-c9573f99-c902-40be-8b98-0ea430c905c2-c000.snappy.parquet, range: 0-74459252, partition values: [empty row]
22/02/08 16:51:36 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 139975 records.
22/02/08 16:51:36 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:51:36 INFO CodecPool: Got brand-new decompressor [.snappy]
22/02/08 16:51:36 INFO InternalParquetRecordReader: block read in memory in 118 ms. row count = 139975
22/02/08 16:51:36 INFO CodeGenerator: Code generated in 38.907045 ms
22/02/08 16:51:36 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 3179 bytes result sent to driver
22/02/08 16:51:36 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 831 ms on 192.168.1.37 (executor driver) (1/1)
22/02/08 16:51:36 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
22/02/08 16:51:36 INFO DAGScheduler: ResultStage 1 (isEmpty at DeltaSync.java:472) finished in 0.838 s
22/02/08 16:51:36 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:51:36 INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished
22/02/08 16:51:36 INFO DAGScheduler: Job 1 finished: isEmpty at DeltaSync.java:472, took 0.843092 s
22/02/08 16:51:36 INFO DeltaSync: Setting up new Hoodie Write Client
22/02/08 16:51:36 INFO EmbeddedTimelineService: Starting Timeline service !!
22/02/08 16:51:36 INFO EmbeddedTimelineService: Overriding hostIp to (192.168.1.37) found in spark-conf. It was null
22/02/08 16:51:36 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/08 16:51:36 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/08 16:51:36 INFO log: Logging initialized @7767ms to org.apache.hudi.org.eclipse.jetty.util.log.Slf4jLog
22/02/08 16:51:37 INFO Javalin:
__ __ _
/ /____ _ _ __ ____ _ / /(_)____
__ / // __ `/| | / // __ `// // // __ \
/ /_/ // /_/ / | |/ // /_/ // // // / / /
\____/ \__,_/ |___/ \__,_//_//_//_/ /_/
https://javalin.io/documentation
22/02/08 16:51:37 INFO Javalin: Starting Javalin ...
22/02/08 16:51:37 INFO Javalin: Listening on http://localhost:53121/
22/02/08 16:51:37 INFO Javalin: Javalin started in 211ms \o/
22/02/08 16:51:37 INFO TimelineService: Starting Timeline server on port :53121
22/02/08 16:51:37 INFO EmbeddedTimelineService: Started embedded timeline server at 192.168.1.37:53121
22/02/08 16:51:37 INFO BaseHoodieClient: Timeline Server already running. Not restarting the service
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830__rollback__COMPLETED]}
22/02/08 16:51:37 INFO AsyncCompactService: Setting pool name for compaction to hoodiecompact
22/02/08 16:51:37 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:51:37 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.1.37:53051 in memory (size: 16.7 KiB, free: 2004.6 MiB)
22/02/08 16:51:37 INFO SparkContext: Starting job: isEmpty at DeltaSync.java:565
22/02/08 16:51:37 INFO DAGScheduler: Got job 2 (isEmpty at DeltaSync.java:565) with 1 output partitions
22/02/08 16:51:37 INFO DAGScheduler: Final stage: ResultStage 2 (isEmpty at DeltaSync.java:565)
22/02/08 16:51:37 INFO DAGScheduler: Parents of final stage: List()
22/02/08 16:51:37 INFO DAGScheduler: Missing parents: List()
22/02/08 16:51:37 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[7] at map at DeltaSync.java:479), which has no missing parents
22/02/08 16:51:37 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 50.4 KiB, free 2004.2 MiB)
22/02/08 16:51:37 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 19.7 KiB, free 2004.2 MiB)
22/02/08 16:51:37 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.1.37:53051 (size: 19.7 KiB, free: 2004.5 MiB)
22/02/08 16:51:37 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1427
22/02/08 16:51:37 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[7] at map at DeltaSync.java:479) (first 15 tasks are for partitions Vector(0))
22/02/08 16:51:37 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks resource profile 0
22/02/08 16:51:37 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4955 bytes) taskResourceAssignments Map()
22/02/08 16:51:37 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
22/02/08 16:51:37 INFO FileScanRDD: Reading File path: file:///Users/ethan/Work/data/hudi/benchmark_sample_upserts2/4/part-00002-c9573f99-c902-40be-8b98-0ea430c905c2-c000.snappy.parquet, range: 0-74459252, partition values: [empty row]
22/02/08 16:51:37 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 139975 records.
22/02/08 16:51:37 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:51:37 INFO InternalParquetRecordReader: block read in memory in 45 ms. row count = 139975
22/02/08 16:51:37 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 2909 bytes result sent to driver
22/02/08 16:51:37 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 102 ms on 192.168.1.37 (executor driver) (1/1)
22/02/08 16:51:37 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
22/02/08 16:51:37 INFO DAGScheduler: ResultStage 2 (isEmpty at DeltaSync.java:565) finished in 0.109 s
22/02/08 16:51:37 INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:51:37 INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished
22/02/08 16:51:37 INFO DAGScheduler: Job 2 finished: isEmpty at DeltaSync.java:565, took 0.116906 s
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830__rollback__COMPLETED]}
22/02/08 16:51:37 INFO BaseHoodieWriteClient: Generate a new instant time: 20220208165137552 action: deltacommit
22/02/08 16:51:37 INFO HoodieHeartbeatClient: Received request to start heartbeat for instant time 20220208165137552
22/02/08 16:51:37 INFO HoodieActiveTimeline: Creating a new instant [==>20220208165137552__deltacommit__REQUESTED]
22/02/08 16:51:37 INFO DeltaSync: Starting commit : 20220208165137552
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:37 INFO TransactionManager: Transaction starting without a transaction owner
22/02/08 16:51:37 INFO LockManager: LockProvider org.apache.hudi.client.transaction.lock.InProcessLockProvider
22/02/08 16:51:37 INFO InProcessLockProvider: Thread pool-22-thread-1 ACQUIRING in-process lock.
22/02/08 16:51:37 INFO InProcessLockProvider: Thread pool-22-thread-1 ACQUIRED in-process lock.
22/02/08 16:51:37 INFO TransactionManager: Transaction started without a transaction owner
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:37 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:51:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830001__commit__COMPLETED]}
22/02/08 16:51:37 INFO AbstractTableFileSystemView: Took 2 ms to read 0 instants, 0 replaced file groups
22/02/08 16:51:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:51:37 INFO HoodieTableMetadataUtil: Loading latest file slices for metadata table partition files
22/02/08 16:51:37 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:51:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:51:37 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:51:37 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=7, StoreTimeTaken=1
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:37 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:51:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830001__commit__COMPLETED]}
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:37 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:51:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:37 INFO TransactionManager: Transaction ending without a transaction owner
22/02/08 16:51:37 INFO InProcessLockProvider: Thread pool-22-thread-1 RELEASING in-process lock.
22/02/08 16:51:37 INFO InProcessLockProvider: Thread pool-22-thread-1 RELEASED in-process lock.
22/02/08 16:51:37 INFO TransactionManager: Transaction ended without a transaction owner
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:37 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:51:37 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:37 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:51:37 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:51:37 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:51:37 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:37 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:51:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/refresh/?basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:51:37 INFO BlockManagerInfo: Removed broadcast_3_piece0 on 192.168.1.37:53051 in memory (size: 19.7 KiB, free: 2004.6 MiB)
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:37 INFO AbstractTableFileSystemView: Took 1 ms to read 0 instants, 0 replaced file groups
22/02/08 16:51:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:37 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/refresh/?basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:37 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:51:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:51:37 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:38 INFO AsyncCleanerService: Auto cleaning is enabled. Running cleaner async to write operation at instant time 20220208165138006
22/02/08 16:51:38 INFO BaseHoodieWriteClient: Scheduling cleaning at instant time :20220208165138006
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:38 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:51:38 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:51:38 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:51:38 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:51:38 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:51:38 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:38 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/refresh/?basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:51:38 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:38 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:51:38 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:51:38 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:38 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/compactions/pending/?basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:38 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:51:38 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:51:38 INFO RequestHandler: Syncing view as client passed last known instant 20220208164926830 as last known instant but server has the following last instant on timeline :Option{val=[20220208164721920__deltacommit__COMPLETED]}
22/02/08 16:51:38 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:38 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:51:38 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:51:38 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:38 INFO CleanPlanner: No earliest commit to retain. No need to scan partitions !!
22/02/08 16:51:38 INFO CleanPlanner: Nothing to clean here. It is already clean
22/02/08 16:51:38 INFO BaseHoodieWriteClient: Cleaner started
22/02/08 16:51:38 INFO BaseHoodieWriteClient: Cleaned failed attempts if any
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:38 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:51:38 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:51:38 INFO HoodieHeartbeatClient: Heartbeat not found in internal map, falling back to reading from DFS
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:51:38 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:38 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:51:38 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:51:38 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:51:38 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:51:38 INFO SparkContext: Starting job: collect at HoodieJavaRDD.java:140
22/02/08 16:51:38 INFO DAGScheduler: Registering RDD 8 (mapToPair at SparkWriteHelper.java:64) as input to shuffle 1
22/02/08 16:51:38 INFO DAGScheduler: Registering RDD 14 (distinct at HoodieJavaRDD.java:130) as input to shuffle 0
22/02/08 16:51:38 INFO DAGScheduler: Got job 3 (collect at HoodieJavaRDD.java:140) with 40 output partitions
22/02/08 16:51:38 INFO DAGScheduler: Final stage: ResultStage 5 (collect at HoodieJavaRDD.java:140)
22/02/08 16:51:38 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
22/02/08 16:51:38 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 4)
22/02/08 16:51:38 INFO DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[8] at mapToPair at SparkWriteHelper.java:64), which has no missing parents
22/02/08 16:51:38 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 52.9 KiB, free 2004.2 MiB)
22/02/08 16:51:38 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 21.1 KiB, free 2004.1 MiB)
22/02/08 16:51:38 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.1.37:53051 (size: 21.1 KiB, free: 2004.5 MiB)
22/02/08 16:51:38 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1427
22/02/08 16:51:38 INFO DAGScheduler: Submitting 5 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[8] at mapToPair at SparkWriteHelper.java:64) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4))
22/02/08 16:51:38 INFO TaskSchedulerImpl: Adding task set 3.0 with 5 tasks resource profile 0
22/02/08 16:51:38 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4944 bytes) taskResourceAssignments Map()
22/02/08 16:51:38 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 4) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4944 bytes) taskResourceAssignments Map()
22/02/08 16:51:38 INFO TaskSetManager: Starting task 2.0 in stage 3.0 (TID 5) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4944 bytes) taskResourceAssignments Map()
22/02/08 16:51:38 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
22/02/08 16:51:38 INFO Executor: Running task 1.0 in stage 3.0 (TID 4)
22/02/08 16:51:38 INFO Executor: Running task 2.0 in stage 3.0 (TID 5)
22/02/08 16:51:38 INFO FileScanRDD: Reading File path: file:///Users/ethan/Work/data/hudi/benchmark_sample_upserts2/4/part-00002-c9573f99-c902-40be-8b98-0ea430c905c2-c000.snappy.parquet, range: 0-74459252, partition values: [empty row]
22/02/08 16:51:38 INFO FileScanRDD: Reading File path: file:///Users/ethan/Work/data/hudi/benchmark_sample_upserts2/4/part-00004-c9573f99-c902-40be-8b98-0ea430c905c2-c000.snappy.parquet, range: 0-74457624, partition values: [empty row]
22/02/08 16:51:38 INFO FileScanRDD: Reading File path: file:///Users/ethan/Work/data/hudi/benchmark_sample_upserts2/4/part-00000-c9573f99-c902-40be-8b98-0ea430c905c2-c000.snappy.parquet, range: 0-74457748, partition values: [empty row]
22/02/08 16:51:38 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 139975 records.
22/02/08 16:51:38 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 139971 records.
22/02/08 16:51:38 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 139972 records.
22/02/08 16:51:38 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:51:38 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:51:38 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:51:38 INFO InternalParquetRecordReader: block read in memory in 220 ms. row count = 139975
22/02/08 16:51:38 INFO CodecPool: Got brand-new decompressor [.snappy]
22/02/08 16:51:38 INFO InternalParquetRecordReader: block read in memory in 234 ms. row count = 139972
22/02/08 16:51:38 INFO CodecPool: Got brand-new decompressor [.snappy]
22/02/08 16:51:38 INFO InternalParquetRecordReader: block read in memory in 237 ms. row count = 139971
22/02/08 16:51:47 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:51:49 INFO Executor: Finished task 2.0 in stage 3.0 (TID 5). 1756 bytes result sent to driver
22/02/08 16:51:49 INFO TaskSetManager: Starting task 3.0 in stage 3.0 (TID 6) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4944 bytes) taskResourceAssignments Map()
22/02/08 16:51:49 INFO Executor: Running task 3.0 in stage 3.0 (TID 6)
22/02/08 16:51:49 INFO TaskSetManager: Finished task 2.0 in stage 3.0 (TID 5) in 10902 ms on 192.168.1.37 (executor driver) (1/5)
22/02/08 16:51:49 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1799 bytes result sent to driver
22/02/08 16:51:49 INFO TaskSetManager: Starting task 4.0 in stage 3.0 (TID 7) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4944 bytes) taskResourceAssignments Map()
22/02/08 16:51:49 INFO Executor: Finished task 1.0 in stage 3.0 (TID 4). 1799 bytes result sent to driver
22/02/08 16:51:49 INFO Executor: Running task 4.0 in stage 3.0 (TID 7)
22/02/08 16:51:49 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 10906 ms on 192.168.1.37 (executor driver) (2/5)
22/02/08 16:51:49 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 4) in 10908 ms on 192.168.1.37 (executor driver) (3/5)
22/02/08 16:51:49 INFO FileScanRDD: Reading File path: file:///Users/ethan/Work/data/hudi/benchmark_sample_upserts2/4/part-00003-c9573f99-c902-40be-8b98-0ea430c905c2-c000.snappy.parquet, range: 0-74457326, partition values: [empty row]
22/02/08 16:51:49 INFO FileScanRDD: Reading File path: file:///Users/ethan/Work/data/hudi/benchmark_sample_upserts2/4/part-00001-c9573f99-c902-40be-8b98-0ea430c905c2-c000.snappy.parquet, range: 0-74456741, partition values: [empty row]
22/02/08 16:51:49 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 139974 records.
22/02/08 16:51:49 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 139975 records.
22/02/08 16:51:49 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:51:49 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:51:49 INFO InternalParquetRecordReader: block read in memory in 166 ms. row count = 139974
22/02/08 16:51:49 INFO InternalParquetRecordReader: block read in memory in 165 ms. row count = 139975
22/02/08 16:51:57 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:51:58 INFO Executor: Finished task 4.0 in stage 3.0 (TID 7). 1756 bytes result sent to driver
22/02/08 16:51:58 INFO TaskSetManager: Finished task 4.0 in stage 3.0 (TID 7) in 9729 ms on 192.168.1.37 (executor driver) (4/5)
22/02/08 16:51:58 INFO Executor: Finished task 3.0 in stage 3.0 (TID 6). 1756 bytes result sent to driver
22/02/08 16:51:58 INFO TaskSetManager: Finished task 3.0 in stage 3.0 (TID 6) in 9740 ms on 192.168.1.37 (executor driver) (5/5)
22/02/08 16:51:58 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
22/02/08 16:51:58 INFO DAGScheduler: ShuffleMapStage 3 (mapToPair at SparkWriteHelper.java:64) finished in 20.651 s
22/02/08 16:51:58 INFO DAGScheduler: looking for newly runnable stages
22/02/08 16:51:58 INFO DAGScheduler: running: Set()
22/02/08 16:51:58 INFO DAGScheduler: waiting: Set(ResultStage 5, ShuffleMapStage 4)
22/02/08 16:51:58 INFO DAGScheduler: failed: Set()
22/02/08 16:51:58 INFO DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[14] at distinct at HoodieJavaRDD.java:130), which has no missing parents
22/02/08 16:51:58 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 9.2 KiB, free 2004.1 MiB)
22/02/08 16:51:58 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 2004.1 MiB)
22/02/08 16:51:58 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.1.37:53051 (size: 4.7 KiB, free: 2004.5 MiB)
22/02/08 16:51:58 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1427
22/02/08 16:51:58 INFO DAGScheduler: Submitting 40 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[14] at distinct at HoodieJavaRDD.java:130) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/08 16:51:58 INFO TaskSchedulerImpl: Adding task set 4.0 with 40 tasks resource profile 0
22/02/08 16:51:58 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 8) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:58 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID 9) (192.168.1.37, executor driver, partition 1, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:58 INFO TaskSetManager: Starting task 2.0 in stage 4.0 (TID 10) (192.168.1.37, executor driver, partition 2, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:58 INFO Executor: Running task 0.0 in stage 4.0 (TID 8)
22/02/08 16:51:58 INFO Executor: Running task 1.0 in stage 4.0 (TID 9)
22/02/08 16:51:58 INFO Executor: Running task 2.0 in stage 4.0 (TID 10)
22/02/08 16:51:58 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:58 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:58 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 9 ms
22/02/08 16:51:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 8 ms
22/02/08 16:51:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 8 ms
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_0 stored as values in memory (estimated size 25.2 MiB, free 1917.6 MiB)
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_1 stored as values in memory (estimated size 25.5 MiB, free 1917.6 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_0 in memory on 192.168.1.37:53051 (size: 25.2 MiB, free: 1979.3 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_1 in memory on 192.168.1.37:53051 (size: 25.5 MiB, free: 1953.8 MiB)
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_2 stored as values in memory (estimated size 25.5 MiB, free 1927.9 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_2 in memory on 192.168.1.37:53051 (size: 25.5 MiB, free: 1928.3 MiB)
22/02/08 16:51:59 INFO Executor: Finished task 1.0 in stage 4.0 (TID 9). 1468 bytes result sent to driver
22/02/08 16:51:59 INFO TaskSetManager: Starting task 3.0 in stage 4.0 (TID 11) (192.168.1.37, executor driver, partition 3, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Running task 3.0 in stage 4.0 (TID 11)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID 9) in 457 ms on 192.168.1.37 (executor driver) (1/40)
22/02/08 16:51:59 INFO Executor: Finished task 2.0 in stage 4.0 (TID 10). 1468 bytes result sent to driver
22/02/08 16:51:59 INFO TaskSetManager: Starting task 4.0 in stage 4.0 (TID 12) (192.168.1.37, executor driver, partition 4, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Running task 4.0 in stage 4.0 (TID 12)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 2.0 in stage 4.0 (TID 10) in 460 ms on 192.168.1.37 (executor driver) (2/40)
22/02/08 16:51:59 INFO Executor: Finished task 0.0 in stage 4.0 (TID 8). 1468 bytes result sent to driver
22/02/08 16:51:59 INFO TaskSetManager: Starting task 5.0 in stage 4.0 (TID 13) (192.168.1.37, executor driver, partition 5, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO Executor: Running task 5.0 in stage 4.0 (TID 13)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 8) in 462 ms on 192.168.1.37 (executor driver) (3/40)
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_5 stored as values in memory (estimated size 25.1 MiB, free 1824.8 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_5 in memory on 192.168.1.37:53051 (size: 25.1 MiB, free: 1903.3 MiB)
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_4 stored as values in memory (estimated size 25.0 MiB, free 1837.9 MiB)
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_3 stored as values in memory (estimated size 25.1 MiB, free 1852.8 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_4 in memory on 192.168.1.37:53051 (size: 25.0 MiB, free: 1878.2 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_3 in memory on 192.168.1.37:53051 (size: 25.1 MiB, free: 1853.2 MiB)
22/02/08 16:51:59 INFO Executor: Finished task 5.0 in stage 4.0 (TID 13). 1425 bytes result sent to driver
22/02/08 16:51:59 INFO TaskSetManager: Starting task 6.0 in stage 4.0 (TID 14) (192.168.1.37, executor driver, partition 6, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Finished task 4.0 in stage 4.0 (TID 12). 1425 bytes result sent to driver
22/02/08 16:51:59 INFO Executor: Running task 6.0 in stage 4.0 (TID 14)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 5.0 in stage 4.0 (TID 13) in 162 ms on 192.168.1.37 (executor driver) (4/40)
22/02/08 16:51:59 INFO TaskSetManager: Starting task 7.0 in stage 4.0 (TID 15) (192.168.1.37, executor driver, partition 7, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Finished task 3.0 in stage 4.0 (TID 11). 1425 bytes result sent to driver
22/02/08 16:51:59 INFO Executor: Running task 7.0 in stage 4.0 (TID 15)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 4.0 in stage 4.0 (TID 12) in 165 ms on 192.168.1.37 (executor driver) (5/40)
22/02/08 16:51:59 INFO TaskSetManager: Starting task 8.0 in stage 4.0 (TID 16) (192.168.1.37, executor driver, partition 8, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Running task 8.0 in stage 4.0 (TID 16)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 3.0 in stage 4.0 (TID 11) in 168 ms on 192.168.1.37 (executor driver) (6/40)
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_6 stored as values in memory (estimated size 25.0 MiB, free 1754.8 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_6 in memory on 192.168.1.37:53051 (size: 25.0 MiB, free: 1828.2 MiB)
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_7 stored as values in memory (estimated size 25.0 MiB, free 1766.9 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_7 in memory on 192.168.1.37:53051 (size: 25.0 MiB, free: 1803.2 MiB)
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_8 stored as values in memory (estimated size 25.5 MiB, free 1777.3 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_8 in memory on 192.168.1.37:53051 (size: 25.5 MiB, free: 1777.7 MiB)
22/02/08 16:51:59 INFO Executor: Finished task 6.0 in stage 4.0 (TID 14). 1425 bytes result sent to driver
22/02/08 16:51:59 INFO TaskSetManager: Starting task 9.0 in stage 4.0 (TID 17) (192.168.1.37, executor driver, partition 9, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Running task 9.0 in stage 4.0 (TID 17)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 6.0 in stage 4.0 (TID 14) in 150 ms on 192.168.1.37 (executor driver) (7/40)
22/02/08 16:51:59 INFO Executor: Finished task 7.0 in stage 4.0 (TID 15). 1425 bytes result sent to driver
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO TaskSetManager: Starting task 10.0 in stage 4.0 (TID 18) (192.168.1.37, executor driver, partition 10, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Running task 10.0 in stage 4.0 (TID 18)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 7.0 in stage 4.0 (TID 15) in 152 ms on 192.168.1.37 (executor driver) (8/40)
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO Executor: Finished task 8.0 in stage 4.0 (TID 16). 1425 bytes result sent to driver
22/02/08 16:51:59 INFO TaskSetManager: Starting task 11.0 in stage 4.0 (TID 19) (192.168.1.37, executor driver, partition 11, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Running task 11.0 in stage 4.0 (TID 19)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 8.0 in stage 4.0 (TID 16) in 157 ms on 192.168.1.37 (executor driver) (9/40)
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_9 stored as values in memory (estimated size 25.1 MiB, free 1675.6 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_9 in memory on 192.168.1.37:53051 (size: 25.1 MiB, free: 1752.6 MiB)
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_10 stored as values in memory (estimated size 24.9 MiB, free 1688.1 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_10 in memory on 192.168.1.37:53051 (size: 24.9 MiB, free: 1727.7 MiB)
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_11 stored as values in memory (estimated size 25.5 MiB, free 1701.8 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_11 in memory on 192.168.1.37:53051 (size: 25.5 MiB, free: 1702.2 MiB)
22/02/08 16:51:59 INFO Executor: Finished task 10.0 in stage 4.0 (TID 18). 1468 bytes result sent to driver
22/02/08 16:51:59 INFO TaskSetManager: Starting task 12.0 in stage 4.0 (TID 20) (192.168.1.37, executor driver, partition 12, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Finished task 9.0 in stage 4.0 (TID 17). 1468 bytes result sent to driver
22/02/08 16:51:59 INFO Executor: Running task 12.0 in stage 4.0 (TID 20)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 10.0 in stage 4.0 (TID 18) in 220 ms on 192.168.1.37 (executor driver) (10/40)
22/02/08 16:51:59 INFO TaskSetManager: Starting task 13.0 in stage 4.0 (TID 21) (192.168.1.37, executor driver, partition 13, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Running task 13.0 in stage 4.0 (TID 21)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 9.0 in stage 4.0 (TID 17) in 225 ms on 192.168.1.37 (executor driver) (11/40)
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO Executor: Finished task 11.0 in stage 4.0 (TID 19). 1468 bytes result sent to driver
22/02/08 16:51:59 INFO TaskSetManager: Starting task 14.0 in stage 4.0 (TID 22) (192.168.1.37, executor driver, partition 14, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:51:59 INFO Executor: Running task 14.0 in stage 4.0 (TID 22)
22/02/08 16:51:59 INFO TaskSetManager: Finished task 11.0 in stage 4.0 (TID 19) in 225 ms on 192.168.1.37 (executor driver) (12/40)
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:51:59 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_12 stored as values in memory (estimated size 25.5 MiB, free 1595.2 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_12 in memory on 192.168.1.37:53051 (size: 25.5 MiB, free: 1676.8 MiB)
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_13 stored as values in memory (estimated size 25.3 MiB, free 1613.3 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_13 in memory on 192.168.1.37:53051 (size: 25.3 MiB, free: 1651.4 MiB)
22/02/08 16:51:59 INFO MemoryStore: Block rdd_10_14 stored as values in memory (estimated size 25.4 MiB, free 1625.6 MiB)
22/02/08 16:51:59 INFO BlockManagerInfo: Added rdd_10_14 in memory on 192.168.1.37:53051 (size: 25.4 MiB, free: 1626.0 MiB)
22/02/08 16:52:00 INFO Executor: Finished task 12.0 in stage 4.0 (TID 20). 1468 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 15.0 in stage 4.0 (TID 23) (192.168.1.37, executor driver, partition 15, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Running task 15.0 in stage 4.0 (TID 23)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 12.0 in stage 4.0 (TID 20) in 215 ms on 192.168.1.37 (executor driver) (13/40)
22/02/08 16:52:00 INFO Executor: Finished task 13.0 in stage 4.0 (TID 21). 1468 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 16.0 in stage 4.0 (TID 24) (192.168.1.37, executor driver, partition 16, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO TaskSetManager: Finished task 13.0 in stage 4.0 (TID 21) in 216 ms on 192.168.1.37 (executor driver) (14/40)
22/02/08 16:52:00 INFO Executor: Running task 16.0 in stage 4.0 (TID 24)
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.1 MiB) non-empty blocks including 5 (11.1 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO Executor: Finished task 14.0 in stage 4.0 (TID 22). 1468 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 17.0 in stage 4.0 (TID 25) (192.168.1.37, executor driver, partition 17, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Running task 17.0 in stage 4.0 (TID 25)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 14.0 in stage 4.0 (TID 22) in 220 ms on 192.168.1.37 (executor driver) (15/40)
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_15 stored as values in memory (estimated size 25.2 MiB, free 1522.2 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_15 in memory on 192.168.1.37:53051 (size: 25.2 MiB, free: 1600.8 MiB)
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_16 stored as values in memory (estimated size 25.4 MiB, free 1536.8 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_16 in memory on 192.168.1.37:53051 (size: 25.4 MiB, free: 1575.4 MiB)
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_17 stored as values in memory (estimated size 25.3 MiB, free 1549.6 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_17 in memory on 192.168.1.37:53051 (size: 25.3 MiB, free: 1550.1 MiB)
22/02/08 16:52:00 INFO Executor: Finished task 15.0 in stage 4.0 (TID 23). 1425 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 18.0 in stage 4.0 (TID 26) (192.168.1.37, executor driver, partition 18, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Finished task 16.0 in stage 4.0 (TID 24). 1425 bytes result sent to driver
22/02/08 16:52:00 INFO Executor: Running task 18.0 in stage 4.0 (TID 26)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 15.0 in stage 4.0 (TID 23) in 170 ms on 192.168.1.37 (executor driver) (16/40)
22/02/08 16:52:00 INFO TaskSetManager: Starting task 19.0 in stage 4.0 (TID 27) (192.168.1.37, executor driver, partition 19, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO TaskSetManager: Finished task 16.0 in stage 4.0 (TID 24) in 170 ms on 192.168.1.37 (executor driver) (17/40)
22/02/08 16:52:00 INFO Executor: Running task 19.0 in stage 4.0 (TID 27)
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO Executor: Finished task 17.0 in stage 4.0 (TID 25). 1425 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 20.0 in stage 4.0 (TID 28) (192.168.1.37, executor driver, partition 20, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Running task 20.0 in stage 4.0 (TID 28)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 17.0 in stage 4.0 (TID 25) in 159 ms on 192.168.1.37 (executor driver) (18/40)
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO BlockManagerInfo: Removed broadcast_4_piece0 on 192.168.1.37:53051 in memory (size: 21.1 KiB, free: 1550.1 MiB)
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_19 stored as values in memory (estimated size 25.1 MiB, free 1463.1 MiB)
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_18 stored as values in memory (estimated size 25.4 MiB, free 1463.1 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_19 in memory on 192.168.1.37:53051 (size: 25.1 MiB, free: 1525.0 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_18 in memory on 192.168.1.37:53051 (size: 25.4 MiB, free: 1499.5 MiB)
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_20 stored as values in memory (estimated size 25.4 MiB, free 1473.7 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_20 in memory on 192.168.1.37:53051 (size: 25.4 MiB, free: 1474.1 MiB)
22/02/08 16:52:00 INFO Executor: Finished task 18.0 in stage 4.0 (TID 26). 1425 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 21.0 in stage 4.0 (TID 29) (192.168.1.37, executor driver, partition 21, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Running task 21.0 in stage 4.0 (TID 29)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 18.0 in stage 4.0 (TID 26) in 500 ms on 192.168.1.37 (executor driver) (19/40)
22/02/08 16:52:00 INFO Executor: Finished task 19.0 in stage 4.0 (TID 27). 1468 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 22.0 in stage 4.0 (TID 30) (192.168.1.37, executor driver, partition 22, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Running task 22.0 in stage 4.0 (TID 30)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 19.0 in stage 4.0 (TID 27) in 501 ms on 192.168.1.37 (executor driver) (20/40)
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO Executor: Finished task 20.0 in stage 4.0 (TID 28). 1468 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 23.0 in stage 4.0 (TID 31) (192.168.1.37, executor driver, partition 23, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Running task 23.0 in stage 4.0 (TID 31)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 20.0 in stage 4.0 (TID 28) in 500 ms on 192.168.1.37 (executor driver) (21/40)
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_21 stored as values in memory (estimated size 25.4 MiB, free 1384.4 MiB)
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_22 stored as values in memory (estimated size 25.3 MiB, free 1384.4 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_21 in memory on 192.168.1.37:53051 (size: 25.4 MiB, free: 1448.7 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_22 in memory on 192.168.1.37:53051 (size: 25.3 MiB, free: 1423.3 MiB)
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_23 stored as values in memory (estimated size 25.5 MiB, free 1397.5 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_23 in memory on 192.168.1.37:53051 (size: 25.5 MiB, free: 1397.8 MiB)
22/02/08 16:52:00 INFO Executor: Finished task 22.0 in stage 4.0 (TID 30). 1425 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 24.0 in stage 4.0 (TID 32) (192.168.1.37, executor driver, partition 24, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Finished task 21.0 in stage 4.0 (TID 29). 1425 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Finished task 22.0 in stage 4.0 (TID 30) in 146 ms on 192.168.1.37 (executor driver) (22/40)
22/02/08 16:52:00 INFO Executor: Running task 24.0 in stage 4.0 (TID 32)
22/02/08 16:52:00 INFO TaskSetManager: Starting task 25.0 in stage 4.0 (TID 33) (192.168.1.37, executor driver, partition 25, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Running task 25.0 in stage 4.0 (TID 33)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 21.0 in stage 4.0 (TID 29) in 147 ms on 192.168.1.37 (executor driver) (23/40)
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO Executor: Finished task 23.0 in stage 4.0 (TID 31). 1425 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 26.0 in stage 4.0 (TID 34) (192.168.1.37, executor driver, partition 26, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Running task 26.0 in stage 4.0 (TID 34)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 23.0 in stage 4.0 (TID 31) in 145 ms on 192.168.1.37 (executor driver) (24/40)
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_24 stored as values in memory (estimated size 25.3 MiB, free 1294.2 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_24 in memory on 192.168.1.37:53051 (size: 25.3 MiB, free: 1372.5 MiB)
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_25 stored as values in memory (estimated size 25.3 MiB, free 1309.9 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_25 in memory on 192.168.1.37:53051 (size: 25.3 MiB, free: 1347.1 MiB)
22/02/08 16:52:00 INFO MemoryStore: Block rdd_10_26 stored as values in memory (estimated size 25.4 MiB, free 1321.4 MiB)
22/02/08 16:52:00 INFO BlockManagerInfo: Added rdd_10_26 in memory on 192.168.1.37:53051 (size: 25.4 MiB, free: 1321.7 MiB)
22/02/08 16:52:00 INFO Executor: Finished task 24.0 in stage 4.0 (TID 32). 1425 bytes result sent to driver
22/02/08 16:52:00 INFO TaskSetManager: Starting task 27.0 in stage 4.0 (TID 35) (192.168.1.37, executor driver, partition 27, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Finished task 25.0 in stage 4.0 (TID 33). 1425 bytes result sent to driver
22/02/08 16:52:00 INFO Executor: Running task 27.0 in stage 4.0 (TID 35)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 24.0 in stage 4.0 (TID 32) in 180 ms on 192.168.1.37 (executor driver) (25/40)
22/02/08 16:52:00 INFO TaskSetManager: Starting task 28.0 in stage 4.0 (TID 36) (192.168.1.37, executor driver, partition 28, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:00 INFO Executor: Running task 28.0 in stage 4.0 (TID 36)
22/02/08 16:52:00 INFO TaskSetManager: Finished task 25.0 in stage 4.0 (TID 33) in 179 ms on 192.168.1.37 (executor driver) (26/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.1 MiB) non-empty blocks including 5 (11.1 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Finished task 26.0 in stage 4.0 (TID 34). 1425 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 29.0 in stage 4.0 (TID 37) (192.168.1.37, executor driver, partition 29, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 29.0 in stage 4.0 (TID 37)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 26.0 in stage 4.0 (TID 34) in 180 ms on 192.168.1.37 (executor driver) (27/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_28 stored as values in memory (estimated size 24.8 MiB, free 1214.9 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_28 in memory on 192.168.1.37:53051 (size: 24.8 MiB, free: 1296.9 MiB)
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_27 stored as values in memory (estimated size 25.5 MiB, free 1229.1 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_27 in memory on 192.168.1.37:53051 (size: 25.5 MiB, free: 1271.4 MiB)
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_29 stored as values in memory (estimated size 25.5 MiB, free 1245.6 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_29 in memory on 192.168.1.37:53051 (size: 25.5 MiB, free: 1245.9 MiB)
22/02/08 16:52:01 INFO Executor: Finished task 28.0 in stage 4.0 (TID 36). 1425 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 30.0 in stage 4.0 (TID 38) (192.168.1.37, executor driver, partition 30, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 30.0 in stage 4.0 (TID 38)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 28.0 in stage 4.0 (TID 36) in 203 ms on 192.168.1.37 (executor driver) (28/40)
22/02/08 16:52:01 INFO Executor: Finished task 27.0 in stage 4.0 (TID 35). 1425 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 31.0 in stage 4.0 (TID 39) (192.168.1.37, executor driver, partition 31, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 31.0 in stage 4.0 (TID 39)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 27.0 in stage 4.0 (TID 35) in 207 ms on 192.168.1.37 (executor driver) (29/40)
22/02/08 16:52:01 INFO Executor: Finished task 29.0 in stage 4.0 (TID 37). 1425 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 32.0 in stage 4.0 (TID 40) (192.168.1.37, executor driver, partition 32, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 32.0 in stage 4.0 (TID 40)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 29.0 in stage 4.0 (TID 37) in 205 ms on 192.168.1.37 (executor driver) (30/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.1 MiB) non-empty blocks including 5 (11.1 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_31 stored as values in memory (estimated size 25.1 MiB, free 1144.5 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_31 in memory on 192.168.1.37:53051 (size: 25.1 MiB, free: 1220.8 MiB)
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_30 stored as values in memory (estimated size 25.3 MiB, free 1157.2 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_30 in memory on 192.168.1.37:53051 (size: 25.3 MiB, free: 1195.6 MiB)
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_32 stored as values in memory (estimated size 25.3 MiB, free 1169.9 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_32 in memory on 192.168.1.37:53051 (size: 25.3 MiB, free: 1170.3 MiB)
22/02/08 16:52:01 INFO Executor: Finished task 31.0 in stage 4.0 (TID 39). 1425 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 33.0 in stage 4.0 (TID 41) (192.168.1.37, executor driver, partition 33, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 33.0 in stage 4.0 (TID 41)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 31.0 in stage 4.0 (TID 39) in 148 ms on 192.168.1.37 (executor driver) (31/40)
22/02/08 16:52:01 INFO Executor: Finished task 30.0 in stage 4.0 (TID 38). 1425 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 34.0 in stage 4.0 (TID 42) (192.168.1.37, executor driver, partition 34, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO Executor: Finished task 32.0 in stage 4.0 (TID 40). 1382 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 30.0 in stage 4.0 (TID 38) in 152 ms on 192.168.1.37 (executor driver) (32/40)
22/02/08 16:52:01 INFO Executor: Running task 34.0 in stage 4.0 (TID 42)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 35.0 in stage 4.0 (TID 43) (192.168.1.37, executor driver, partition 35, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 32.0 in stage 4.0 (TID 40) in 149 ms on 192.168.1.37 (executor driver) (33/40)
22/02/08 16:52:01 INFO Executor: Running task 35.0 in stage 4.0 (TID 43)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_33 stored as values in memory (estimated size 25.5 MiB, free 1057.0 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_33 in memory on 192.168.1.37:53051 (size: 25.5 MiB, free: 1144.8 MiB)
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_35 stored as values in memory (estimated size 25.2 MiB, free 1078.0 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_35 in memory on 192.168.1.37:53051 (size: 25.2 MiB, free: 1119.6 MiB)
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_34 stored as values in memory (estimated size 25.4 MiB, free 1093.8 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_34 in memory on 192.168.1.37:53051 (size: 25.4 MiB, free: 1094.2 MiB)
22/02/08 16:52:01 INFO Executor: Finished task 33.0 in stage 4.0 (TID 41). 1468 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 36.0 in stage 4.0 (TID 44) (192.168.1.37, executor driver, partition 36, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 33.0 in stage 4.0 (TID 41) in 202 ms on 192.168.1.37 (executor driver) (34/40)
22/02/08 16:52:01 INFO Executor: Running task 36.0 in stage 4.0 (TID 44)
22/02/08 16:52:01 INFO Executor: Finished task 35.0 in stage 4.0 (TID 43). 1425 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 37.0 in stage 4.0 (TID 45) (192.168.1.37, executor driver, partition 37, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 37.0 in stage 4.0 (TID 45)
22/02/08 16:52:01 INFO Executor: Finished task 34.0 in stage 4.0 (TID 42). 1468 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 35.0 in stage 4.0 (TID 43) in 199 ms on 192.168.1.37 (executor driver) (35/40)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 38.0 in stage 4.0 (TID 46) (192.168.1.37, executor driver, partition 38, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 38.0 in stage 4.0 (TID 46)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 34.0 in stage 4.0 (TID 42) in 200 ms on 192.168.1.37 (executor driver) (36/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.1 MiB) non-empty blocks including 5 (11.1 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 18 ms
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_36 stored as values in memory (estimated size 24.9 MiB, free 990.3 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_36 in memory on 192.168.1.37:53051 (size: 24.9 MiB, free: 1069.3 MiB)
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_37 stored as values in memory (estimated size 25.3 MiB, free 1004.7 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_37 in memory on 192.168.1.37:53051 (size: 25.3 MiB, free: 1044.0 MiB)
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_38 stored as values in memory (estimated size 25.3 MiB, free 1018.3 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_38 in memory on 192.168.1.37:53051 (size: 25.3 MiB, free: 1018.7 MiB)
22/02/08 16:52:01 INFO Executor: Finished task 36.0 in stage 4.0 (TID 44). 1425 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 39.0 in stage 4.0 (TID 47) (192.168.1.37, executor driver, partition 39, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 39.0 in stage 4.0 (TID 47)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 36.0 in stage 4.0 (TID 44) in 186 ms on 192.168.1.37 (executor driver) (37/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Finished task 37.0 in stage 4.0 (TID 45). 1468 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 37.0 in stage 4.0 (TID 45) in 189 ms on 192.168.1.37 (executor driver) (38/40)
22/02/08 16:52:01 INFO Executor: Finished task 38.0 in stage 4.0 (TID 46). 1425 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 38.0 in stage 4.0 (TID 46) in 210 ms on 192.168.1.37 (executor driver) (39/40)
22/02/08 16:52:01 INFO MemoryStore: Block rdd_10_39 stored as values in memory (estimated size 25.5 MiB, free 992.8 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added rdd_10_39 in memory on 192.168.1.37:53051 (size: 25.5 MiB, free: 993.2 MiB)
22/02/08 16:52:01 INFO Executor: Finished task 39.0 in stage 4.0 (TID 47). 1425 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 39.0 in stage 4.0 (TID 47) in 149 ms on 192.168.1.37 (executor driver) (40/40)
22/02/08 16:52:01 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
22/02/08 16:52:01 INFO DAGScheduler: ShuffleMapStage 4 (distinct at HoodieJavaRDD.java:130) finished in 3.101 s
22/02/08 16:52:01 INFO DAGScheduler: looking for newly runnable stages
22/02/08 16:52:01 INFO DAGScheduler: running: Set()
22/02/08 16:52:01 INFO DAGScheduler: waiting: Set(ResultStage 5)
22/02/08 16:52:01 INFO DAGScheduler: failed: Set()
22/02/08 16:52:01 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[16] at distinct at HoodieJavaRDD.java:130), which has no missing parents
22/02/08 16:52:01 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 6.5 KiB, free 992.8 MiB)
22/02/08 16:52:01 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 3.5 KiB, free 992.8 MiB)
22/02/08 16:52:01 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.1.37:53051 (size: 3.5 KiB, free: 993.1 MiB)
22/02/08 16:52:01 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:01 INFO DAGScheduler: Submitting 40 missing tasks from ResultStage 5 (MapPartitionsRDD[16] at distinct at HoodieJavaRDD.java:130) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/08 16:52:01 INFO TaskSchedulerImpl: Adding task set 5.0 with 40 tasks resource profile 0
22/02/08 16:52:01 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 48) (192.168.1.37, executor driver, partition 1, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Starting task 2.0 in stage 5.0 (TID 49) (192.168.1.37, executor driver, partition 2, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Starting task 3.0 in stage 5.0 (TID 50) (192.168.1.37, executor driver, partition 3, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 1.0 in stage 5.0 (TID 48)
22/02/08 16:52:01 INFO Executor: Running task 2.0 in stage 5.0 (TID 49)
22/02/08 16:52:01 INFO Executor: Running task 3.0 in stage 5.0 (TID 50)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 40 (2.1 KiB) non-empty blocks including 40 (2.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 40 (2.1 KiB) non-empty blocks including 40 (2.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 40 (2.1 KiB) non-empty blocks including 40 (2.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Finished task 2.0 in stage 5.0 (TID 49). 1237 bytes result sent to driver
22/02/08 16:52:01 INFO Executor: Finished task 3.0 in stage 5.0 (TID 50). 1237 bytes result sent to driver
22/02/08 16:52:01 INFO Executor: Finished task 1.0 in stage 5.0 (TID 48). 1237 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 4.0 in stage 5.0 (TID 51) (192.168.1.37, executor driver, partition 4, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 2.0 in stage 5.0 (TID 49) in 14 ms on 192.168.1.37 (executor driver) (1/40)
22/02/08 16:52:01 INFO Executor: Running task 4.0 in stage 5.0 (TID 51)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 5.0 in stage 5.0 (TID 52) (192.168.1.37, executor driver, partition 5, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 3.0 in stage 5.0 (TID 50) in 14 ms on 192.168.1.37 (executor driver) (2/40)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 6.0 in stage 5.0 (TID 53) (192.168.1.37, executor driver, partition 6, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 6.0 in stage 5.0 (TID 53)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 48) in 15 ms on 192.168.1.37 (executor driver) (3/40)
22/02/08 16:52:01 INFO Executor: Running task 5.0 in stage 5.0 (TID 52)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 40 (2.1 KiB) non-empty blocks including 40 (2.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 40 (2.1 KiB) non-empty blocks including 40 (2.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 40 (2.1 KiB) non-empty blocks including 40 (2.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
22/02/08 16:52:01 INFO Executor: Finished task 4.0 in stage 5.0 (TID 51). 1237 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 28.0 in stage 5.0 (TID 54) (192.168.1.37, executor driver, partition 28, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Finished task 6.0 in stage 5.0 (TID 53). 1194 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 4.0 in stage 5.0 (TID 51) in 13 ms on 192.168.1.37 (executor driver) (4/40)
22/02/08 16:52:01 INFO Executor: Running task 28.0 in stage 5.0 (TID 54)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 29.0 in stage 5.0 (TID 55) (192.168.1.37, executor driver, partition 29, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Finished task 5.0 in stage 5.0 (TID 52). 1237 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 6.0 in stage 5.0 (TID 53) in 11 ms on 192.168.1.37 (executor driver) (5/40)
22/02/08 16:52:01 INFO Executor: Running task 29.0 in stage 5.0 (TID 55)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 35.0 in stage 5.0 (TID 56) (192.168.1.37, executor driver, partition 35, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 35.0 in stage 5.0 (TID 56)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 5.0 in stage 5.0 (TID 52) in 13 ms on 192.168.1.37 (executor driver) (6/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 40 (2.1 KiB) non-empty blocks including 40 (2.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 40 (2.1 KiB) non-empty blocks including 40 (2.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 40 (2.1 KiB) non-empty blocks including 40 (2.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Finished task 29.0 in stage 5.0 (TID 55). 1237 bytes result sent to driver
22/02/08 16:52:01 INFO Executor: Finished task 28.0 in stage 5.0 (TID 54). 1237 bytes result sent to driver
22/02/08 16:52:01 INFO Executor: Finished task 35.0 in stage 5.0 (TID 56). 1193 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 36.0 in stage 5.0 (TID 57) (192.168.1.37, executor driver, partition 36, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 29.0 in stage 5.0 (TID 55) in 9 ms on 192.168.1.37 (executor driver) (7/40)
22/02/08 16:52:01 INFO Executor: Running task 36.0 in stage 5.0 (TID 57)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 58) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 0.0 in stage 5.0 (TID 58)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 28.0 in stage 5.0 (TID 54) in 11 ms on 192.168.1.37 (executor driver) (8/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 40 (2.1 KiB) non-empty blocks including 40 (2.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO TaskSetManager: Starting task 7.0 in stage 5.0 (TID 59) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Finished task 35.0 in stage 5.0 (TID 56) in 10 ms on 192.168.1.37 (executor driver) (9/40)
22/02/08 16:52:01 INFO Executor: Running task 7.0 in stage 5.0 (TID 59)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Finished task 0.0 in stage 5.0 (TID 58). 1140 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO TaskSetManager: Starting task 8.0 in stage 5.0 (TID 60) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Running task 8.0 in stage 5.0 (TID 60)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 58) in 3 ms on 192.168.1.37 (executor driver) (10/40)
22/02/08 16:52:01 INFO Executor: Finished task 7.0 in stage 5.0 (TID 59). 1183 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 9.0 in stage 5.0 (TID 61) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 7.0 in stage 5.0 (TID 59) in 3 ms on 192.168.1.37 (executor driver) (11/40)
22/02/08 16:52:01 INFO Executor: Running task 9.0 in stage 5.0 (TID 61)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Finished task 8.0 in stage 5.0 (TID 60). 1226 bytes result sent to driver
22/02/08 16:52:01 INFO Executor: Finished task 9.0 in stage 5.0 (TID 61). 1183 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 10.0 in stage 5.0 (TID 62) (192.168.1.37, executor driver, partition 10, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 8.0 in stage 5.0 (TID 60) in 4 ms on 192.168.1.37 (executor driver) (12/40)
22/02/08 16:52:01 INFO Executor: Running task 10.0 in stage 5.0 (TID 62)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 11.0 in stage 5.0 (TID 63) (192.168.1.37, executor driver, partition 11, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 9.0 in stage 5.0 (TID 61) in 4 ms on 192.168.1.37 (executor driver) (13/40)
22/02/08 16:52:01 INFO Executor: Running task 11.0 in stage 5.0 (TID 63)
22/02/08 16:52:01 INFO Executor: Finished task 36.0 in stage 5.0 (TID 57). 1193 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO TaskSetManager: Starting task 12.0 in stage 5.0 (TID 64) (192.168.1.37, executor driver, partition 12, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Running task 12.0 in stage 5.0 (TID 64)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 36.0 in stage 5.0 (TID 57) in 11 ms on 192.168.1.37 (executor driver) (14/40)
22/02/08 16:52:01 INFO Executor: Finished task 10.0 in stage 5.0 (TID 62). 1226 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 13.0 in stage 5.0 (TID 65) (192.168.1.37, executor driver, partition 13, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 10.0 in stage 5.0 (TID 62) in 4 ms on 192.168.1.37 (executor driver) (15/40)
22/02/08 16:52:01 INFO Executor: Running task 13.0 in stage 5.0 (TID 65)
22/02/08 16:52:01 INFO Executor: Finished task 11.0 in stage 5.0 (TID 63). 1226 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 14.0 in stage 5.0 (TID 66) (192.168.1.37, executor driver, partition 14, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 14.0 in stage 5.0 (TID 66)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 11.0 in stage 5.0 (TID 63) in 4 ms on 192.168.1.37 (executor driver) (16/40)
22/02/08 16:52:01 INFO Executor: Finished task 12.0 in stage 5.0 (TID 64). 1183 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 15.0 in stage 5.0 (TID 67) (192.168.1.37, executor driver, partition 15, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 12.0 in stage 5.0 (TID 64) in 4 ms on 192.168.1.37 (executor driver) (17/40)
22/02/08 16:52:01 INFO Executor: Running task 15.0 in stage 5.0 (TID 67)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Finished task 13.0 in stage 5.0 (TID 65). 1226 bytes result sent to driver
22/02/08 16:52:01 INFO Executor: Finished task 14.0 in stage 5.0 (TID 66). 1140 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 16.0 in stage 5.0 (TID 68) (192.168.1.37, executor driver, partition 16, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 16.0 in stage 5.0 (TID 68)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 13.0 in stage 5.0 (TID 65) in 4 ms on 192.168.1.37 (executor driver) (18/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO TaskSetManager: Starting task 17.0 in stage 5.0 (TID 69) (192.168.1.37, executor driver, partition 17, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Finished task 14.0 in stage 5.0 (TID 66) in 4 ms on 192.168.1.37 (executor driver) (19/40)
22/02/08 16:52:01 INFO Executor: Running task 17.0 in stage 5.0 (TID 69)
22/02/08 16:52:01 INFO Executor: Finished task 15.0 in stage 5.0 (TID 67). 1226 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 18.0 in stage 5.0 (TID 70) (192.168.1.37, executor driver, partition 18, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 18.0 in stage 5.0 (TID 70)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 15.0 in stage 5.0 (TID 67) in 4 ms on 192.168.1.37 (executor driver) (20/40)
22/02/08 16:52:01 INFO Executor: Finished task 16.0 in stage 5.0 (TID 68). 1226 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 19.0 in stage 5.0 (TID 71) (192.168.1.37, executor driver, partition 19, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Finished task 17.0 in stage 5.0 (TID 69). 1226 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 16.0 in stage 5.0 (TID 68) in 3 ms on 192.168.1.37 (executor driver) (21/40)
22/02/08 16:52:01 INFO Executor: Running task 19.0 in stage 5.0 (TID 71)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 20.0 in stage 5.0 (TID 72) (192.168.1.37, executor driver, partition 20, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Running task 20.0 in stage 5.0 (TID 72)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 17.0 in stage 5.0 (TID 69) in 3 ms on 192.168.1.37 (executor driver) (22/40)
22/02/08 16:52:01 INFO Executor: Finished task 18.0 in stage 5.0 (TID 70). 1183 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 21.0 in stage 5.0 (TID 73) (192.168.1.37, executor driver, partition 21, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 18.0 in stage 5.0 (TID 70) in 3 ms on 192.168.1.37 (executor driver) (23/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO Executor: Running task 21.0 in stage 5.0 (TID 73)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Finished task 19.0 in stage 5.0 (TID 71). 1183 bytes result sent to driver
22/02/08 16:52:01 INFO Executor: Finished task 20.0 in stage 5.0 (TID 72). 1140 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 22.0 in stage 5.0 (TID 74) (192.168.1.37, executor driver, partition 22, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 22.0 in stage 5.0 (TID 74)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Finished task 19.0 in stage 5.0 (TID 71) in 4 ms on 192.168.1.37 (executor driver) (24/40)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 23.0 in stage 5.0 (TID 75) (192.168.1.37, executor driver, partition 23, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Finished task 21.0 in stage 5.0 (TID 73). 1140 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 20.0 in stage 5.0 (TID 72) in 4 ms on 192.168.1.37 (executor driver) (25/40)
22/02/08 16:52:01 INFO Executor: Running task 23.0 in stage 5.0 (TID 75)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 24.0 in stage 5.0 (TID 76) (192.168.1.37, executor driver, partition 24, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO Executor: Running task 24.0 in stage 5.0 (TID 76)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 21.0 in stage 5.0 (TID 73) in 3 ms on 192.168.1.37 (executor driver) (26/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Finished task 22.0 in stage 5.0 (TID 74). 1226 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 25.0 in stage 5.0 (TID 77) (192.168.1.37, executor driver, partition 25, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Finished task 23.0 in stage 5.0 (TID 75). 1140 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Running task 25.0 in stage 5.0 (TID 77)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 22.0 in stage 5.0 (TID 74) in 4 ms on 192.168.1.37 (executor driver) (27/40)
22/02/08 16:52:01 INFO Executor: Finished task 24.0 in stage 5.0 (TID 76). 1140 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Starting task 26.0 in stage 5.0 (TID 78) (192.168.1.37, executor driver, partition 26, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 23.0 in stage 5.0 (TID 75) in 4 ms on 192.168.1.37 (executor driver) (28/40)
22/02/08 16:52:01 INFO Executor: Running task 26.0 in stage 5.0 (TID 78)
22/02/08 16:52:01 INFO TaskSetManager: Starting task 27.0 in stage 5.0 (TID 79) (192.168.1.37, executor driver, partition 27, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Running task 27.0 in stage 5.0 (TID 79)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 24.0 in stage 5.0 (TID 76) in 4 ms on 192.168.1.37 (executor driver) (29/40)
22/02/08 16:52:01 INFO Executor: Finished task 25.0 in stage 5.0 (TID 77). 1183 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 30.0 in stage 5.0 (TID 80) (192.168.1.37, executor driver, partition 30, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Running task 30.0 in stage 5.0 (TID 80)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 25.0 in stage 5.0 (TID 77) in 4 ms on 192.168.1.37 (executor driver) (30/40)
22/02/08 16:52:01 INFO Executor: Finished task 26.0 in stage 5.0 (TID 78). 1183 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 31.0 in stage 5.0 (TID 81) (192.168.1.37, executor driver, partition 31, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Finished task 27.0 in stage 5.0 (TID 79). 1140 bytes result sent to driver
22/02/08 16:52:01 INFO Executor: Running task 31.0 in stage 5.0 (TID 81)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 26.0 in stage 5.0 (TID 78) in 4 ms on 192.168.1.37 (executor driver) (31/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO TaskSetManager: Starting task 32.0 in stage 5.0 (TID 82) (192.168.1.37, executor driver, partition 32, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Finished task 27.0 in stage 5.0 (TID 79) in 4 ms on 192.168.1.37 (executor driver) (32/40)
22/02/08 16:52:01 INFO Executor: Running task 32.0 in stage 5.0 (TID 82)
22/02/08 16:52:01 INFO Executor: Finished task 30.0 in stage 5.0 (TID 80). 1183 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 33.0 in stage 5.0 (TID 83) (192.168.1.37, executor driver, partition 33, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO TaskSetManager: Finished task 30.0 in stage 5.0 (TID 80) in 3 ms on 192.168.1.37 (executor driver) (33/40)
22/02/08 16:52:01 INFO Executor: Running task 33.0 in stage 5.0 (TID 83)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO Executor: Finished task 31.0 in stage 5.0 (TID 81). 1140 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 34.0 in stage 5.0 (TID 84) (192.168.1.37, executor driver, partition 34, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Finished task 32.0 in stage 5.0 (TID 82). 1140 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 31.0 in stage 5.0 (TID 81) in 3 ms on 192.168.1.37 (executor driver) (34/40)
22/02/08 16:52:01 INFO Executor: Running task 34.0 in stage 5.0 (TID 84)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO TaskSetManager: Starting task 37.0 in stage 5.0 (TID 85) (192.168.1.37, executor driver, partition 37, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Finished task 32.0 in stage 5.0 (TID 82) in 4 ms on 192.168.1.37 (executor driver) (35/40)
22/02/08 16:52:01 INFO Executor: Running task 37.0 in stage 5.0 (TID 85)
22/02/08 16:52:01 INFO Executor: Finished task 33.0 in stage 5.0 (TID 83). 1183 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO TaskSetManager: Starting task 38.0 in stage 5.0 (TID 86) (192.168.1.37, executor driver, partition 38, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Running task 38.0 in stage 5.0 (TID 86)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 33.0 in stage 5.0 (TID 83) in 3 ms on 192.168.1.37 (executor driver) (36/40)
22/02/08 16:52:01 INFO Executor: Finished task 34.0 in stage 5.0 (TID 84). 1226 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Starting task 39.0 in stage 5.0 (TID 87) (192.168.1.37, executor driver, partition 39, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:01 INFO Executor: Finished task 37.0 in stage 5.0 (TID 85). 1183 bytes result sent to driver
22/02/08 16:52:01 INFO Executor: Running task 39.0 in stage 5.0 (TID 87)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 34.0 in stage 5.0 (TID 84) in 3 ms on 192.168.1.37 (executor driver) (37/40)
22/02/08 16:52:01 INFO TaskSetManager: Finished task 37.0 in stage 5.0 (TID 85) in 4 ms on 192.168.1.37 (executor driver) (38/40)
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO Executor: Finished task 38.0 in stage 5.0 (TID 86). 1226 bytes result sent to driver
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:01 INFO TaskSetManager: Finished task 38.0 in stage 5.0 (TID 86) in 3 ms on 192.168.1.37 (executor driver) (39/40)
22/02/08 16:52:01 INFO Executor: Finished task 39.0 in stage 5.0 (TID 87). 1140 bytes result sent to driver
22/02/08 16:52:01 INFO TaskSetManager: Finished task 39.0 in stage 5.0 (TID 87) in 3 ms on 192.168.1.37 (executor driver) (40/40)
22/02/08 16:52:01 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
22/02/08 16:52:01 INFO DAGScheduler: ResultStage 5 (collect at HoodieJavaRDD.java:140) finished in 0.076 s
22/02/08 16:52:01 INFO DAGScheduler: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:52:01 INFO TaskSchedulerImpl: Killing all running tasks in stage 5: Stage finished
22/02/08 16:52:01 INFO DAGScheduler: Job 3 finished: collect at HoodieJavaRDD.java:140, took 23.864777 s
22/02/08 16:52:02 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:134
22/02/08 16:52:02 INFO DAGScheduler: Got job 4 (collect at HoodieSparkEngineContext.java:134) with 10 output partitions
22/02/08 16:52:02 INFO DAGScheduler: Final stage: ResultStage 6 (collect at HoodieSparkEngineContext.java:134)
22/02/08 16:52:02 INFO DAGScheduler: Parents of final stage: List()
22/02/08 16:52:02 INFO DAGScheduler: Missing parents: List()
22/02/08 16:52:02 INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[18] at flatMap at HoodieSparkEngineContext.java:134), which has no missing parents
22/02/08 16:52:02 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 510.9 KiB, free 992.3 MiB)
22/02/08 16:52:02 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 178.4 KiB, free 992.1 MiB)
22/02/08 16:52:02 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.1.37:53051 (size: 178.4 KiB, free: 993.0 MiB)
22/02/08 16:52:02 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:02 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 6 (MapPartitionsRDD[18] at flatMap at HoodieSparkEngineContext.java:134) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/08 16:52:02 INFO TaskSchedulerImpl: Adding task set 6.0 with 10 tasks resource profile 0
22/02/08 16:52:02 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 88) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO TaskSetManager: Starting task 1.0 in stage 6.0 (TID 89) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO TaskSetManager: Starting task 2.0 in stage 6.0 (TID 90) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO Executor: Running task 0.0 in stage 6.0 (TID 88)
22/02/08 16:52:02 INFO Executor: Running task 2.0 in stage 6.0 (TID 90)
22/02/08 16:52:02 INFO Executor: Running task 1.0 in stage 6.0 (TID 89)
22/02/08 16:52:02 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:02 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:02 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:02 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:02 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:02 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022%2F1%2F26&maxinstant=20220208164721920&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:02 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022%2F1%2F25&maxinstant=20220208164721920&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:02 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022%2F1%2F24&maxinstant=20220208164721920&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/24)
22/02/08 16:52:02 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/25)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/26)
22/02/08 16:52:02 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/08 16:52:02 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/08 16:52:02 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830001__commit__COMPLETED]}
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=2, StoreTimeTaken=0
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=2, StoreTimeTaken=0
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=2, StoreTimeTaken=0
22/02/08 16:52:02 INFO CacheConfig: Allocating LruBlockCache size=1.42 GB, blockSize=64 KB
22/02/08 16:52:02 INFO CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
22/02/08 16:52:02 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:02 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:02 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:02 INFO HoodieBackedTableMetadata: Opened metadata base file from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/files-0000_0-93-815_20220208164926830001.hfile at instant 20220208164926830001 in 185 ms
22/02/08 16:52:02 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:52:02 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:52:02 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:52:02 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830001__commit__COMPLETED]}
22/02/08 16:52:02 INFO AbstractHoodieLogRecordReader: Scanning log file HoodieLogFile{pathStr='file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/.files-0000_20220208164926830001.log.1_0-103-825', fileLen=-1}
22/02/08 16:52:02 INFO HoodieMergedLogRecordScanner: Number of log files scanned => 1
22/02/08 16:52:02 INFO HoodieMergedLogRecordScanner: MaxMemoryInBytes allowed for compaction => 1073741824
22/02/08 16:52:02 INFO HoodieMergedLogRecordScanner: Number of entries in MemoryBasedMap in ExternalSpillableMap => 0
22/02/08 16:52:02 INFO HoodieMergedLogRecordScanner: Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 0
22/02/08 16:52:02 INFO HoodieMergedLogRecordScanner: Number of entries in BitCaskDiskMap in ExternalSpillableMap => 0
22/02/08 16:52:02 INFO HoodieMergedLogRecordScanner: Size of file spilled to disk => 0
22/02/08 16:52:02 INFO HoodieBackedTableMetadata: Opened 1 metadata log files (dataset instant=20220208164926830, metadata instant=20220208164926830001) in 48 ms
22/02/08 16:52:02 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:02 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/26, #files=4
22/02/08 16:52:02 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/24, #files=4
22/02/08 16:52:02 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/25, #files=4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO Executor: Finished task 1.0 in stage 6.0 (TID 89). 1113 bytes result sent to driver
22/02/08 16:52:02 INFO Executor: Finished task 0.0 in stage 6.0 (TID 88). 1070 bytes result sent to driver
22/02/08 16:52:02 INFO Executor: Finished task 2.0 in stage 6.0 (TID 90). 1113 bytes result sent to driver
22/02/08 16:52:02 INFO TaskSetManager: Starting task 3.0 in stage 6.0 (TID 91) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO Executor: Running task 3.0 in stage 6.0 (TID 91)
22/02/08 16:52:02 INFO TaskSetManager: Starting task 4.0 in stage 6.0 (TID 92) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO Executor: Running task 4.0 in stage 6.0 (TID 92)
22/02/08 16:52:02 INFO TaskSetManager: Starting task 5.0 in stage 6.0 (TID 93) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO Executor: Running task 5.0 in stage 6.0 (TID 93)
22/02/08 16:52:02 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 88) in 365 ms on 192.168.1.37 (executor driver) (1/10)
22/02/08 16:52:02 INFO TaskSetManager: Finished task 2.0 in stage 6.0 (TID 90) in 364 ms on 192.168.1.37 (executor driver) (2/10)
22/02/08 16:52:02 INFO TaskSetManager: Finished task 1.0 in stage 6.0 (TID 89) in 364 ms on 192.168.1.37 (executor driver) (3/10)
22/02/08 16:52:02 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:02 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022%2F1%2F27&maxinstant=20220208164721920&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:02 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:02 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:02 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022%2F1%2F28&maxinstant=20220208164721920&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/27)
22/02/08 16:52:02 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/08 16:52:02 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022%2F1%2F29&maxinstant=20220208164721920&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/28)
22/02/08 16:52:02 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/29)
22/02/08 16:52:02 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=0, StoreTimeTaken=0
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/27, #files=4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=2, StoreTimeTaken=0
22/02/08 16:52:02 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/28, #files=4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/29, #files=4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=0, StoreTimeTaken=0
22/02/08 16:52:02 INFO Executor: Finished task 5.0 in stage 6.0 (TID 93). 1070 bytes result sent to driver
22/02/08 16:52:02 INFO Executor: Finished task 4.0 in stage 6.0 (TID 92). 1070 bytes result sent to driver
22/02/08 16:52:02 INFO Executor: Finished task 3.0 in stage 6.0 (TID 91). 1070 bytes result sent to driver
22/02/08 16:52:02 INFO TaskSetManager: Starting task 6.0 in stage 6.0 (TID 94) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO Executor: Running task 6.0 in stage 6.0 (TID 94)
22/02/08 16:52:02 INFO TaskSetManager: Starting task 7.0 in stage 6.0 (TID 95) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO Executor: Running task 7.0 in stage 6.0 (TID 95)
22/02/08 16:52:02 INFO TaskSetManager: Finished task 5.0 in stage 6.0 (TID 93) in 40 ms on 192.168.1.37 (executor driver) (4/10)
22/02/08 16:52:02 INFO TaskSetManager: Finished task 4.0 in stage 6.0 (TID 92) in 40 ms on 192.168.1.37 (executor driver) (5/10)
22/02/08 16:52:02 INFO TaskSetManager: Starting task 8.0 in stage 6.0 (TID 96) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4341 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO Executor: Running task 8.0 in stage 6.0 (TID 96)
22/02/08 16:52:02 INFO TaskSetManager: Finished task 3.0 in stage 6.0 (TID 91) in 45 ms on 192.168.1.37 (executor driver) (6/10)
22/02/08 16:52:02 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:02 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:02 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:02 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:02 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:02 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022%2F2%2F1&maxinstant=20220208164721920&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:02 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022%2F1%2F31&maxinstant=20220208164721920&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:02 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022%2F1%2F30&maxinstant=20220208164721920&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/31)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (2022/2/1)
22/02/08 16:52:02 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (2022/1/30)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/08 16:52:02 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/2/1, #files=4
22/02/08 16:52:02 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/31, #files=4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=0, StoreTimeTaken=0
22/02/08 16:52:02 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/1/30, #files=4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=0, StoreTimeTaken=0
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO Executor: Finished task 8.0 in stage 6.0 (TID 96). 1068 bytes result sent to driver
22/02/08 16:52:02 INFO Executor: Finished task 7.0 in stage 6.0 (TID 95). 1070 bytes result sent to driver
22/02/08 16:52:02 INFO Executor: Finished task 6.0 in stage 6.0 (TID 94). 1070 bytes result sent to driver
22/02/08 16:52:02 INFO TaskSetManager: Starting task 9.0 in stage 6.0 (TID 97) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4341 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO Executor: Running task 9.0 in stage 6.0 (TID 97)
22/02/08 16:52:02 INFO TaskSetManager: Finished task 8.0 in stage 6.0 (TID 96) in 34 ms on 192.168.1.37 (executor driver) (7/10)
22/02/08 16:52:02 INFO TaskSetManager: Finished task 7.0 in stage 6.0 (TID 95) in 34 ms on 192.168.1.37 (executor driver) (8/10)
22/02/08 16:52:02 INFO TaskSetManager: Finished task 6.0 in stage 6.0 (TID 94) in 35 ms on 192.168.1.37 (executor driver) (9/10)
22/02/08 16:52:02 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:02 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:02 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafiles/beforeoron/latest/?partition=2022%2F2%2F2&maxinstant=20220208164721920&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (2022/2/2)
22/02/08 16:52:02 INFO HoodieTableMetadataUtil: Loading latest merged file slices for metadata table partition files
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:02 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:02 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:52:02 INFO BaseTableMetadata: Listed file in partition from metadata: partition=2022/2/2, #files=4
22/02/08 16:52:02 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=4, NumFileGroups=1, FileGroupsCreationTime=0, StoreTimeTaken=0
22/02/08 16:52:02 INFO Executor: Finished task 9.0 in stage 6.0 (TID 97). 1068 bytes result sent to driver
22/02/08 16:52:02 INFO TaskSetManager: Finished task 9.0 in stage 6.0 (TID 97) in 30 ms on 192.168.1.37 (executor driver) (10/10)
22/02/08 16:52:02 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool
22/02/08 16:52:02 INFO DAGScheduler: ResultStage 6 (collect at HoodieSparkEngineContext.java:134) finished in 0.535 s
22/02/08 16:52:02 INFO DAGScheduler: Job 4 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:52:02 INFO TaskSchedulerImpl: Killing all running tasks in stage 6: Stage finished
22/02/08 16:52:02 INFO DAGScheduler: Job 4 finished: collect at HoodieSparkEngineContext.java:134, took 0.537168 s
22/02/08 16:52:02 INFO MapPartitionsRDD: Removing RDD 10 from persistence list
22/02/08 16:52:02 INFO BlockManager: Removing RDD 10
22/02/08 16:52:02 INFO SparkContext: Starting job: countByKey at BaseSparkCommitActionExecutor.java:196
22/02/08 16:52:02 INFO DAGScheduler: Registering RDD 21 (mapToPair at HoodieJavaRDD.java:122) as input to shuffle 3
22/02/08 16:52:02 INFO DAGScheduler: Registering RDD 11 (mapToPair at HoodieJavaRDD.java:122) as input to shuffle 2
22/02/08 16:52:02 INFO DAGScheduler: Registering RDD 29 (countByKey at BaseSparkCommitActionExecutor.java:196) as input to shuffle 4
22/02/08 16:52:02 INFO DAGScheduler: Got job 5 (countByKey at BaseSparkCommitActionExecutor.java:196) with 40 output partitions
22/02/08 16:52:02 INFO DAGScheduler: Final stage: ResultStage 11 (countByKey at BaseSparkCommitActionExecutor.java:196)
22/02/08 16:52:02 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 10)
22/02/08 16:52:02 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 10)
22/02/08 16:52:02 INFO DAGScheduler: Submitting ShuffleMapStage 8 (MapPartitionsRDD[21] at mapToPair at HoodieJavaRDD.java:122), which has no missing parents
22/02/08 16:52:02 INFO BlockManagerInfo: Removed broadcast_6_piece0 on 192.168.1.37:53051 in memory (size: 3.5 KiB, free: 2004.4 MiB)
22/02/08 16:52:02 INFO BlockManagerInfo: Removed broadcast_7_piece0 on 192.168.1.37:53051 in memory (size: 178.4 KiB, free: 2004.6 MiB)
22/02/08 16:52:02 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 513.0 KiB, free 2003.7 MiB)
22/02/08 16:52:02 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 179.4 KiB, free 2003.5 MiB)
22/02/08 16:52:02 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.1.37:53051 (size: 179.4 KiB, free: 2004.4 MiB)
22/02/08 16:52:02 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:02 INFO DAGScheduler: Submitting 10 missing tasks from ShuffleMapStage 8 (MapPartitionsRDD[21] at mapToPair at HoodieJavaRDD.java:122) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/08 16:52:02 INFO TaskSchedulerImpl: Adding task set 8.0 with 10 tasks resource profile 0
22/02/08 16:52:02 INFO DAGScheduler: Submitting ShuffleMapStage 9 (MapPartitionsRDD[11] at mapToPair at HoodieJavaRDD.java:122), which has no missing parents
22/02/08 16:52:02 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 98) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4594 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO TaskSetManager: Starting task 1.0 in stage 8.0 (TID 99) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4594 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO TaskSetManager: Starting task 2.0 in stage 8.0 (TID 100) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4594 bytes) taskResourceAssignments Map()
22/02/08 16:52:02 INFO Executor: Running task 0.0 in stage 8.0 (TID 98)
22/02/08 16:52:02 INFO Executor: Running task 1.0 in stage 8.0 (TID 99)
22/02/08 16:52:02 INFO Executor: Running task 2.0 in stage 8.0 (TID 100)
22/02/08 16:52:02 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 8.1 KiB, free 2003.5 MiB)
22/02/08 16:52:02 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 4.3 KiB, free 2003.5 MiB)
22/02/08 16:52:02 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on 192.168.1.37:53051 (size: 4.3 KiB, free: 2004.4 MiB)
22/02/08 16:52:02 INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:02 INFO DAGScheduler: Submitting 40 missing tasks from ShuffleMapStage 9 (MapPartitionsRDD[11] at mapToPair at HoodieJavaRDD.java:122) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/08 16:52:02 INFO TaskSchedulerImpl: Adding task set 9.0 with 40 tasks resource profile 0
22/02/08 16:52:03 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 157700 records.
22/02/08 16:52:03 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 155565 records.
22/02/08 16:52:03 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 154930 records.
22/02/08 16:52:03 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:03 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:03 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:03 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:03 INFO InternalParquetRecordReader: block read in memory in 4 ms. row count = 154930
22/02/08 16:52:03 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:03 INFO InternalParquetRecordReader: block read in memory in 5 ms. row count = 155565
22/02/08 16:52:03 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:03 INFO InternalParquetRecordReader: block read in memory in 5 ms. row count = 157700
22/02/08 16:52:05 INFO Executor: Finished task 0.0 in stage 8.0 (TID 98). 1039 bytes result sent to driver
22/02/08 16:52:05 INFO TaskSetManager: Starting task 3.0 in stage 8.0 (TID 101) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4594 bytes) taskResourceAssignments Map()
22/02/08 16:52:05 INFO Executor: Running task 3.0 in stage 8.0 (TID 101)
22/02/08 16:52:05 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 98) in 2119 ms on 192.168.1.37 (executor driver) (1/10)
22/02/08 16:52:05 INFO Executor: Finished task 1.0 in stage 8.0 (TID 99). 1039 bytes result sent to driver
22/02/08 16:52:05 INFO TaskSetManager: Starting task 4.0 in stage 8.0 (TID 102) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4594 bytes) taskResourceAssignments Map()
22/02/08 16:52:05 INFO Executor: Running task 4.0 in stage 8.0 (TID 102)
22/02/08 16:52:05 INFO TaskSetManager: Finished task 1.0 in stage 8.0 (TID 99) in 2124 ms on 192.168.1.37 (executor driver) (2/10)
22/02/08 16:52:05 INFO Executor: Finished task 2.0 in stage 8.0 (TID 100). 1039 bytes result sent to driver
22/02/08 16:52:05 INFO TaskSetManager: Starting task 5.0 in stage 8.0 (TID 103) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4594 bytes) taskResourceAssignments Map()
22/02/08 16:52:05 INFO Executor: Running task 5.0 in stage 8.0 (TID 103)
22/02/08 16:52:05 INFO TaskSetManager: Finished task 2.0 in stage 8.0 (TID 100) in 2150 ms on 192.168.1.37 (executor driver) (3/10)
22/02/08 16:52:05 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 155885 records.
22/02/08 16:52:05 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:05 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 154700 records.
22/02/08 16:52:05 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:05 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:05 INFO InternalParquetRecordReader: block read in memory in 6 ms. row count = 155885
22/02/08 16:52:05 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:05 INFO InternalParquetRecordReader: block read in memory in 6 ms. row count = 154700
22/02/08 16:52:05 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 153140 records.
22/02/08 16:52:05 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:05 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:05 INFO InternalParquetRecordReader: block read in memory in 5 ms. row count = 153140
22/02/08 16:52:06 INFO BlockManagerInfo: Removed broadcast_5_piece0 on 192.168.1.37:53051 in memory (size: 4.7 KiB, free: 2004.4 MiB)
22/02/08 16:52:07 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:52:08 INFO Executor: Finished task 4.0 in stage 8.0 (TID 102). 1039 bytes result sent to driver
22/02/08 16:52:08 INFO TaskSetManager: Starting task 6.0 in stage 8.0 (TID 104) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4594 bytes) taskResourceAssignments Map()
22/02/08 16:52:08 INFO Executor: Running task 6.0 in stage 8.0 (TID 104)
22/02/08 16:52:08 INFO TaskSetManager: Finished task 4.0 in stage 8.0 (TID 102) in 3734 ms on 192.168.1.37 (executor driver) (4/10)
22/02/08 16:52:08 INFO Executor: Finished task 3.0 in stage 8.0 (TID 101). 1039 bytes result sent to driver
22/02/08 16:52:08 INFO TaskSetManager: Starting task 7.0 in stage 8.0 (TID 105) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4594 bytes) taskResourceAssignments Map()
22/02/08 16:52:08 INFO TaskSetManager: Finished task 3.0 in stage 8.0 (TID 101) in 3743 ms on 192.168.1.37 (executor driver) (5/10)
22/02/08 16:52:08 INFO Executor: Running task 7.0 in stage 8.0 (TID 105)
22/02/08 16:52:08 INFO Executor: Finished task 5.0 in stage 8.0 (TID 103). 1039 bytes result sent to driver
22/02/08 16:52:08 INFO TaskSetManager: Starting task 8.0 in stage 8.0 (TID 106) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4592 bytes) taskResourceAssignments Map()
22/02/08 16:52:08 INFO Executor: Running task 8.0 in stage 8.0 (TID 106)
22/02/08 16:52:08 INFO TaskSetManager: Finished task 5.0 in stage 8.0 (TID 103) in 3713 ms on 192.168.1.37 (executor driver) (6/10)
22/02/08 16:52:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 154900 records.
22/02/08 16:52:08 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 153820 records.
22/02/08 16:52:08 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:08 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 154480 records.
22/02/08 16:52:08 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:08 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:08 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:08 INFO InternalParquetRecordReader: block read in memory in 23 ms. row count = 154900
22/02/08 16:52:08 INFO InternalParquetRecordReader: block read in memory in 23 ms. row count = 153820
22/02/08 16:52:08 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:08 INFO InternalParquetRecordReader: block read in memory in 23 ms. row count = 154480
22/02/08 16:52:11 INFO Executor: Finished task 6.0 in stage 8.0 (TID 104). 1039 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 9.0 in stage 8.0 (TID 107) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4592 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO Executor: Running task 9.0 in stage 8.0 (TID 107)
22/02/08 16:52:11 INFO TaskSetManager: Finished task 6.0 in stage 8.0 (TID 104) in 2211 ms on 192.168.1.37 (executor driver) (7/10)
22/02/08 16:52:11 INFO Executor: Finished task 8.0 in stage 8.0 (TID 106). 1039 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 108) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO Executor: Running task 0.0 in stage 9.0 (TID 108)
22/02/08 16:52:11 INFO TaskSetManager: Finished task 8.0 in stage 8.0 (TID 106) in 2213 ms on 192.168.1.37 (executor driver) (8/10)
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:11 INFO Executor: Finished task 7.0 in stage 8.0 (TID 105). 1039 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 1.0 in stage 9.0 (TID 109) (192.168.1.37, executor driver, partition 1, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO TaskSetManager: Finished task 7.0 in stage 8.0 (TID 105) in 2227 ms on 192.168.1.37 (executor driver) (9/10)
22/02/08 16:52:11 INFO Executor: Running task 1.0 in stage 9.0 (TID 109)
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:11 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 154900 records.
22/02/08 16:52:11 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:11 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:11 INFO InternalParquetRecordReader: block read in memory in 5 ms. row count = 154900
22/02/08 16:52:11 INFO Executor: Finished task 0.0 in stage 9.0 (TID 108). 1383 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 2.0 in stage 9.0 (TID 110) (192.168.1.37, executor driver, partition 2, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 108) in 225 ms on 192.168.1.37 (executor driver) (1/40)
22/02/08 16:52:11 INFO Executor: Running task 2.0 in stage 9.0 (TID 110)
22/02/08 16:52:11 INFO Executor: Finished task 1.0 in stage 9.0 (TID 109). 1426 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 3.0 in stage 9.0 (TID 111) (192.168.1.37, executor driver, partition 3, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO TaskSetManager: Finished task 1.0 in stage 9.0 (TID 109) in 213 ms on 192.168.1.37 (executor driver) (2/40)
22/02/08 16:52:11 INFO Executor: Running task 3.0 in stage 9.0 (TID 111)
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:11 INFO Executor: Finished task 3.0 in stage 9.0 (TID 111). 1426 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 4.0 in stage 9.0 (TID 112) (192.168.1.37, executor driver, partition 4, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO Executor: Running task 4.0 in stage 9.0 (TID 112)
22/02/08 16:52:11 INFO TaskSetManager: Finished task 3.0 in stage 9.0 (TID 111) in 194 ms on 192.168.1.37 (executor driver) (3/40)
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:11 INFO Executor: Finished task 2.0 in stage 9.0 (TID 110). 1426 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 5.0 in stage 9.0 (TID 113) (192.168.1.37, executor driver, partition 5, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO Executor: Running task 5.0 in stage 9.0 (TID 113)
22/02/08 16:52:11 INFO TaskSetManager: Finished task 2.0 in stage 9.0 (TID 110) in 200 ms on 192.168.1.37 (executor driver) (4/40)
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:11 INFO Executor: Finished task 4.0 in stage 9.0 (TID 112). 1383 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 6.0 in stage 9.0 (TID 114) (192.168.1.37, executor driver, partition 6, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO Executor: Running task 6.0 in stage 9.0 (TID 114)
22/02/08 16:52:11 INFO TaskSetManager: Finished task 4.0 in stage 9.0 (TID 112) in 185 ms on 192.168.1.37 (executor driver) (5/40)
22/02/08 16:52:11 INFO Executor: Finished task 5.0 in stage 9.0 (TID 113). 1383 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 7.0 in stage 9.0 (TID 115) (192.168.1.37, executor driver, partition 7, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO Executor: Running task 7.0 in stage 9.0 (TID 115)
22/02/08 16:52:11 INFO TaskSetManager: Finished task 5.0 in stage 9.0 (TID 113) in 181 ms on 192.168.1.37 (executor driver) (6/40)
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:11 INFO Executor: Finished task 6.0 in stage 9.0 (TID 114). 1383 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 8.0 in stage 9.0 (TID 116) (192.168.1.37, executor driver, partition 8, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO Executor: Running task 8.0 in stage 9.0 (TID 116)
22/02/08 16:52:11 INFO TaskSetManager: Finished task 6.0 in stage 9.0 (TID 114) in 188 ms on 192.168.1.37 (executor driver) (7/40)
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:11 INFO Executor: Finished task 7.0 in stage 9.0 (TID 115). 1383 bytes result sent to driver
22/02/08 16:52:11 INFO TaskSetManager: Starting task 9.0 in stage 9.0 (TID 117) (192.168.1.37, executor driver, partition 9, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:11 INFO TaskSetManager: Finished task 7.0 in stage 9.0 (TID 115) in 190 ms on 192.168.1.37 (executor driver) (8/40)
22/02/08 16:52:11 INFO Executor: Running task 9.0 in stage 9.0 (TID 117)
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:11 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:12 INFO Executor: Finished task 9.0 in stage 9.0 (TID 117). 1426 bytes result sent to driver
22/02/08 16:52:12 INFO TaskSetManager: Starting task 10.0 in stage 9.0 (TID 118) (192.168.1.37, executor driver, partition 10, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:12 INFO TaskSetManager: Finished task 9.0 in stage 9.0 (TID 117) in 203 ms on 192.168.1.37 (executor driver) (9/40)
22/02/08 16:52:12 INFO Executor: Running task 10.0 in stage 9.0 (TID 118)
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:12 INFO Executor: Finished task 8.0 in stage 9.0 (TID 116). 1426 bytes result sent to driver
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:12 INFO TaskSetManager: Starting task 11.0 in stage 9.0 (TID 119) (192.168.1.37, executor driver, partition 11, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:12 INFO TaskSetManager: Finished task 8.0 in stage 9.0 (TID 116) in 209 ms on 192.168.1.37 (executor driver) (10/40)
22/02/08 16:52:12 INFO Executor: Running task 11.0 in stage 9.0 (TID 119)
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:12 INFO Executor: Finished task 10.0 in stage 9.0 (TID 118). 1426 bytes result sent to driver
22/02/08 16:52:12 INFO TaskSetManager: Starting task 12.0 in stage 9.0 (TID 120) (192.168.1.37, executor driver, partition 12, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:12 INFO Executor: Running task 12.0 in stage 9.0 (TID 120)
22/02/08 16:52:12 INFO TaskSetManager: Finished task 10.0 in stage 9.0 (TID 118) in 192 ms on 192.168.1.37 (executor driver) (11/40)
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:12 INFO Executor: Finished task 11.0 in stage 9.0 (TID 119). 1383 bytes result sent to driver
22/02/08 16:52:12 INFO TaskSetManager: Starting task 13.0 in stage 9.0 (TID 121) (192.168.1.37, executor driver, partition 13, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:12 INFO Executor: Running task 13.0 in stage 9.0 (TID 121)
22/02/08 16:52:12 INFO TaskSetManager: Finished task 11.0 in stage 9.0 (TID 119) in 193 ms on 192.168.1.37 (executor driver) (12/40)
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:12 INFO Executor: Finished task 12.0 in stage 9.0 (TID 120). 1383 bytes result sent to driver
22/02/08 16:52:12 INFO TaskSetManager: Starting task 14.0 in stage 9.0 (TID 122) (192.168.1.37, executor driver, partition 14, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:12 INFO Executor: Running task 14.0 in stage 9.0 (TID 122)
22/02/08 16:52:12 INFO TaskSetManager: Finished task 12.0 in stage 9.0 (TID 120) in 184 ms on 192.168.1.37 (executor driver) (13/40)
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:12 INFO Executor: Finished task 13.0 in stage 9.0 (TID 121). 1383 bytes result sent to driver
22/02/08 16:52:12 INFO TaskSetManager: Starting task 15.0 in stage 9.0 (TID 123) (192.168.1.37, executor driver, partition 15, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:12 INFO Executor: Running task 15.0 in stage 9.0 (TID 123)
22/02/08 16:52:12 INFO TaskSetManager: Finished task 13.0 in stage 9.0 (TID 121) in 185 ms on 192.168.1.37 (executor driver) (14/40)
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Getting 5 (11.1 MiB) non-empty blocks including 5 (11.1 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:12 INFO Executor: Finished task 14.0 in stage 9.0 (TID 122). 1426 bytes result sent to driver
22/02/08 16:52:12 INFO TaskSetManager: Starting task 16.0 in stage 9.0 (TID 124) (192.168.1.37, executor driver, partition 16, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:12 INFO Executor: Running task 16.0 in stage 9.0 (TID 124)
22/02/08 16:52:12 INFO TaskSetManager: Finished task 14.0 in stage 9.0 (TID 122) in 209 ms on 192.168.1.37 (executor driver) (15/40)
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:12 INFO Executor: Finished task 15.0 in stage 9.0 (TID 123). 1426 bytes result sent to driver
22/02/08 16:52:12 INFO TaskSetManager: Starting task 17.0 in stage 9.0 (TID 125) (192.168.1.37, executor driver, partition 17, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:12 INFO TaskSetManager: Finished task 15.0 in stage 9.0 (TID 123) in 206 ms on 192.168.1.37 (executor driver) (16/40)
22/02/08 16:52:12 INFO Executor: Running task 17.0 in stage 9.0 (TID 125)
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:12 INFO Executor: Finished task 16.0 in stage 9.0 (TID 124). 1383 bytes result sent to driver
22/02/08 16:52:12 INFO TaskSetManager: Starting task 18.0 in stage 9.0 (TID 126) (192.168.1.37, executor driver, partition 18, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:12 INFO TaskSetManager: Finished task 16.0 in stage 9.0 (TID 124) in 186 ms on 192.168.1.37 (executor driver) (17/40)
22/02/08 16:52:12 INFO Executor: Running task 18.0 in stage 9.0 (TID 126)
22/02/08 16:52:12 INFO Executor: Finished task 17.0 in stage 9.0 (TID 125). 1383 bytes result sent to driver
22/02/08 16:52:12 INFO TaskSetManager: Starting task 19.0 in stage 9.0 (TID 127) (192.168.1.37, executor driver, partition 19, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:12 INFO TaskSetManager: Finished task 17.0 in stage 9.0 (TID 125) in 185 ms on 192.168.1.37 (executor driver) (18/40)
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:12 INFO Executor: Running task 19.0 in stage 9.0 (TID 127)
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 18.0 in stage 9.0 (TID 126). 1383 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 20.0 in stage 9.0 (TID 128) (192.168.1.37, executor driver, partition 20, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 20.0 in stage 9.0 (TID 128)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 18.0 in stage 9.0 (TID 126) in 184 ms on 192.168.1.37 (executor driver) (19/40)
22/02/08 16:52:13 INFO Executor: Finished task 19.0 in stage 9.0 (TID 127). 1426 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 21.0 in stage 9.0 (TID 129) (192.168.1.37, executor driver, partition 21, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 21.0 in stage 9.0 (TID 129)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 19.0 in stage 9.0 (TID 127) in 184 ms on 192.168.1.37 (executor driver) (20/40)
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 9.0 in stage 8.0 (TID 107). 1039 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 22.0 in stage 9.0 (TID 130) (192.168.1.37, executor driver, partition 22, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 22.0 in stage 9.0 (TID 130)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 9.0 in stage 8.0 (TID 107) in 2023 ms on 192.168.1.37 (executor driver) (10/10)
22/02/08 16:52:13 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool
22/02/08 16:52:13 INFO DAGScheduler: ShuffleMapStage 8 (mapToPair at HoodieJavaRDD.java:122) finished in 10.169 s
22/02/08 16:52:13 INFO DAGScheduler: looking for newly runnable stages
22/02/08 16:52:13 INFO DAGScheduler: running: Set(ShuffleMapStage 9)
22/02/08 16:52:13 INFO DAGScheduler: waiting: Set(ShuffleMapStage 10, ResultStage 11)
22/02/08 16:52:13 INFO DAGScheduler: failed: Set()
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 20.0 in stage 9.0 (TID 128). 1383 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 23.0 in stage 9.0 (TID 131) (192.168.1.37, executor driver, partition 23, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 23.0 in stage 9.0 (TID 131)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 20.0 in stage 9.0 (TID 128) in 188 ms on 192.168.1.37 (executor driver) (21/40)
22/02/08 16:52:13 INFO Executor: Finished task 21.0 in stage 9.0 (TID 129). 1383 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 24.0 in stage 9.0 (TID 132) (192.168.1.37, executor driver, partition 24, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 24.0 in stage 9.0 (TID 132)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 21.0 in stage 9.0 (TID 129) in 188 ms on 192.168.1.37 (executor driver) (22/40)
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 22.0 in stage 9.0 (TID 130). 1383 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 25.0 in stage 9.0 (TID 133) (192.168.1.37, executor driver, partition 25, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO TaskSetManager: Finished task 22.0 in stage 9.0 (TID 130) in 182 ms on 192.168.1.37 (executor driver) (23/40)
22/02/08 16:52:13 INFO Executor: Running task 25.0 in stage 9.0 (TID 133)
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 24.0 in stage 9.0 (TID 132). 1426 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 26.0 in stage 9.0 (TID 134) (192.168.1.37, executor driver, partition 26, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 26.0 in stage 9.0 (TID 134)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 24.0 in stage 9.0 (TID 132) in 209 ms on 192.168.1.37 (executor driver) (24/40)
22/02/08 16:52:13 INFO Executor: Finished task 23.0 in stage 9.0 (TID 131). 1426 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 27.0 in stage 9.0 (TID 135) (192.168.1.37, executor driver, partition 27, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 27.0 in stage 9.0 (TID 135)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 23.0 in stage 9.0 (TID 131) in 212 ms on 192.168.1.37 (executor driver) (25/40)
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 25.0 in stage 9.0 (TID 133). 1426 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 28.0 in stage 9.0 (TID 136) (192.168.1.37, executor driver, partition 28, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 28.0 in stage 9.0 (TID 136)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 25.0 in stage 9.0 (TID 133) in 221 ms on 192.168.1.37 (executor driver) (26/40)
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.1 MiB) non-empty blocks including 5 (11.1 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 26.0 in stage 9.0 (TID 134). 1426 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 29.0 in stage 9.0 (TID 137) (192.168.1.37, executor driver, partition 29, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 29.0 in stage 9.0 (TID 137)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 26.0 in stage 9.0 (TID 134) in 217 ms on 192.168.1.37 (executor driver) (27/40)
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 27.0 in stage 9.0 (TID 135). 1383 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 30.0 in stage 9.0 (TID 138) (192.168.1.37, executor driver, partition 30, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 30.0 in stage 9.0 (TID 138)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 27.0 in stage 9.0 (TID 135) in 219 ms on 192.168.1.37 (executor driver) (28/40)
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 28.0 in stage 9.0 (TID 136). 1383 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 31.0 in stage 9.0 (TID 139) (192.168.1.37, executor driver, partition 31, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 31.0 in stage 9.0 (TID 139)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 28.0 in stage 9.0 (TID 136) in 200 ms on 192.168.1.37 (executor driver) (29/40)
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 29.0 in stage 9.0 (TID 137). 1383 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 32.0 in stage 9.0 (TID 140) (192.168.1.37, executor driver, partition 32, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 32.0 in stage 9.0 (TID 140)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 29.0 in stage 9.0 (TID 137) in 198 ms on 192.168.1.37 (executor driver) (30/40)
22/02/08 16:52:13 INFO Executor: Finished task 30.0 in stage 9.0 (TID 138). 1383 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 33.0 in stage 9.0 (TID 141) (192.168.1.37, executor driver, partition 33, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 33.0 in stage 9.0 (TID 141)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 30.0 in stage 9.0 (TID 138) in 195 ms on 192.168.1.37 (executor driver) (31/40)
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.1 MiB) non-empty blocks including 5 (11.1 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:13 INFO Executor: Finished task 31.0 in stage 9.0 (TID 139). 1383 bytes result sent to driver
22/02/08 16:52:13 INFO TaskSetManager: Starting task 34.0 in stage 9.0 (TID 142) (192.168.1.37, executor driver, partition 34, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:13 INFO Executor: Running task 34.0 in stage 9.0 (TID 142)
22/02/08 16:52:13 INFO TaskSetManager: Finished task 31.0 in stage 9.0 (TID 139) in 194 ms on 192.168.1.37 (executor driver) (32/40)
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO Executor: Finished task 33.0 in stage 9.0 (TID 141). 1426 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Starting task 35.0 in stage 9.0 (TID 143) (192.168.1.37, executor driver, partition 35, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO Executor: Running task 35.0 in stage 9.0 (TID 143)
22/02/08 16:52:14 INFO TaskSetManager: Finished task 33.0 in stage 9.0 (TID 141) in 195 ms on 192.168.1.37 (executor driver) (33/40)
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO Executor: Finished task 32.0 in stage 9.0 (TID 140). 1426 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Starting task 36.0 in stage 9.0 (TID 144) (192.168.1.37, executor driver, partition 36, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO Executor: Running task 36.0 in stage 9.0 (TID 144)
22/02/08 16:52:14 INFO TaskSetManager: Finished task 32.0 in stage 9.0 (TID 140) in 198 ms on 192.168.1.37 (executor driver) (34/40)
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 5 (11.1 MiB) non-empty blocks including 5 (11.1 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO Executor: Finished task 34.0 in stage 9.0 (TID 142). 1383 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Starting task 37.0 in stage 9.0 (TID 145) (192.168.1.37, executor driver, partition 37, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO TaskSetManager: Finished task 34.0 in stage 9.0 (TID 142) in 188 ms on 192.168.1.37 (executor driver) (35/40)
22/02/08 16:52:14 INFO Executor: Running task 37.0 in stage 9.0 (TID 145)
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO Executor: Finished task 35.0 in stage 9.0 (TID 143). 1383 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Starting task 38.0 in stage 9.0 (TID 146) (192.168.1.37, executor driver, partition 38, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO Executor: Running task 38.0 in stage 9.0 (TID 146)
22/02/08 16:52:14 INFO TaskSetManager: Finished task 35.0 in stage 9.0 (TID 143) in 183 ms on 192.168.1.37 (executor driver) (36/40)
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO Executor: Finished task 36.0 in stage 9.0 (TID 144). 1383 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Starting task 39.0 in stage 9.0 (TID 147) (192.168.1.37, executor driver, partition 39, NODE_LOCAL, 4260 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO Executor: Running task 39.0 in stage 9.0 (TID 147)
22/02/08 16:52:14 INFO TaskSetManager: Finished task 36.0 in stage 9.0 (TID 144) in 184 ms on 192.168.1.37 (executor driver) (37/40)
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 5 (11.3 MiB) non-empty blocks including 5 (11.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO Executor: Finished task 37.0 in stage 9.0 (TID 145). 1383 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Finished task 37.0 in stage 9.0 (TID 145) in 182 ms on 192.168.1.37 (executor driver) (38/40)
22/02/08 16:52:14 INFO Executor: Finished task 38.0 in stage 9.0 (TID 146). 1383 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Finished task 38.0 in stage 9.0 (TID 146) in 186 ms on 192.168.1.37 (executor driver) (39/40)
22/02/08 16:52:14 INFO Executor: Finished task 39.0 in stage 9.0 (TID 147). 1383 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Finished task 39.0 in stage 9.0 (TID 147) in 185 ms on 192.168.1.37 (executor driver) (40/40)
22/02/08 16:52:14 INFO TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool
22/02/08 16:52:14 INFO DAGScheduler: ShuffleMapStage 9 (mapToPair at HoodieJavaRDD.java:122) finished in 11.398 s
22/02/08 16:52:14 INFO DAGScheduler: looking for newly runnable stages
22/02/08 16:52:14 INFO DAGScheduler: running: Set()
22/02/08 16:52:14 INFO DAGScheduler: waiting: Set(ShuffleMapStage 10, ResultStage 11)
22/02/08 16:52:14 INFO DAGScheduler: failed: Set()
22/02/08 16:52:14 INFO DAGScheduler: Submitting ShuffleMapStage 10 (MapPartitionsRDD[29] at countByKey at BaseSparkCommitActionExecutor.java:196), which has no missing parents
22/02/08 16:52:14 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 9.6 KiB, free 2003.5 MiB)
22/02/08 16:52:14 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 5.1 KiB, free 2003.5 MiB)
22/02/08 16:52:14 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on 192.168.1.37:53051 (size: 5.1 KiB, free: 2004.4 MiB)
22/02/08 16:52:14 INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:14 INFO DAGScheduler: Submitting 40 missing tasks from ShuffleMapStage 10 (MapPartitionsRDD[29] at countByKey at BaseSparkCommitActionExecutor.java:196) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/08 16:52:14 INFO TaskSchedulerImpl: Adding task set 10.0 with 40 tasks resource profile 0
22/02/08 16:52:14 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 148) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO TaskSetManager: Starting task 1.0 in stage 10.0 (TID 149) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO TaskSetManager: Starting task 2.0 in stage 10.0 (TID 150) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO Executor: Running task 0.0 in stage 10.0 (TID 148)
22/02/08 16:52:14 INFO Executor: Running task 1.0 in stage 10.0 (TID 149)
22/02/08 16:52:14 INFO Executor: Running task 2.0 in stage 10.0 (TID 150)
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 10 (1531.0 KiB) non-empty blocks including 10 (1531.0 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO MemoryStore: Block rdd_27_0 stored as values in memory (estimated size 26.1 MiB, free 1810.4 MiB)
22/02/08 16:52:14 INFO BlockManagerInfo: Added rdd_27_0 in memory on 192.168.1.37:53051 (size: 26.1 MiB, free: 1978.2 MiB)
22/02/08 16:52:14 INFO MemoryStore: Block rdd_27_2 stored as values in memory (estimated size 26.4 MiB, free 1870.3 MiB)
22/02/08 16:52:14 INFO BlockManagerInfo: Added rdd_27_2 in memory on 192.168.1.37:53051 (size: 26.4 MiB, free: 1951.9 MiB)
22/02/08 16:52:14 INFO MemoryStore: Block rdd_27_1 stored as values in memory (estimated size 26.4 MiB, free 1924.6 MiB)
22/02/08 16:52:14 INFO BlockManagerInfo: Added rdd_27_1 in memory on 192.168.1.37:53051 (size: 26.4 MiB, free: 1925.5 MiB)
22/02/08 16:52:14 INFO Executor: Finished task 2.0 in stage 10.0 (TID 150). 1469 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Starting task 3.0 in stage 10.0 (TID 151) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO TaskSetManager: Finished task 2.0 in stage 10.0 (TID 150) in 351 ms on 192.168.1.37 (executor driver) (1/40)
22/02/08 16:52:14 INFO Executor: Running task 3.0 in stage 10.0 (TID 151)
22/02/08 16:52:14 INFO Executor: Finished task 1.0 in stage 10.0 (TID 149). 1469 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Starting task 4.0 in stage 10.0 (TID 152) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO Executor: Running task 4.0 in stage 10.0 (TID 152)
22/02/08 16:52:14 INFO TaskSetManager: Finished task 1.0 in stage 10.0 (TID 149) in 352 ms on 192.168.1.37 (executor driver) (2/40)
22/02/08 16:52:14 INFO Executor: Finished task 0.0 in stage 10.0 (TID 148). 1469 bytes result sent to driver
22/02/08 16:52:14 INFO TaskSetManager: Starting task 5.0 in stage 10.0 (TID 153) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:14 INFO Executor: Running task 5.0 in stage 10.0 (TID 153)
22/02/08 16:52:14 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 148) in 353 ms on 192.168.1.37 (executor driver) (3/40)
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 10 (1472.7 KiB) non-empty blocks including 10 (1472.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:14 INFO MemoryStore: Block rdd_27_4 stored as values in memory (estimated size 25.9 MiB, free 1758.7 MiB)
22/02/08 16:52:14 INFO BlockManagerInfo: Added rdd_27_4 in memory on 192.168.1.37:53051 (size: 25.9 MiB, free: 1899.5 MiB)
22/02/08 16:52:14 INFO MemoryStore: Block rdd_27_5 stored as values in memory (estimated size 26.0 MiB, free 1778.2 MiB)
22/02/08 16:52:14 INFO BlockManagerInfo: Added rdd_27_5 in memory on 192.168.1.37:53051 (size: 26.0 MiB, free: 1873.6 MiB)
22/02/08 16:52:14 INFO MemoryStore: Block rdd_27_3 stored as values in memory (estimated size 26.0 MiB, free 1846.7 MiB)
22/02/08 16:52:14 INFO BlockManagerInfo: Added rdd_27_3 in memory on 192.168.1.37:53051 (size: 26.0 MiB, free: 1847.6 MiB)
22/02/08 16:52:15 INFO Executor: Finished task 5.0 in stage 10.0 (TID 153). 1469 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 6.0 in stage 10.0 (TID 154) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO Executor: Running task 6.0 in stage 10.0 (TID 154)
22/02/08 16:52:15 INFO TaskSetManager: Finished task 5.0 in stage 10.0 (TID 153) in 291 ms on 192.168.1.37 (executor driver) (4/40)
22/02/08 16:52:15 INFO Executor: Finished task 4.0 in stage 10.0 (TID 152). 1426 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 7.0 in stage 10.0 (TID 155) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO Executor: Running task 7.0 in stage 10.0 (TID 155)
22/02/08 16:52:15 INFO TaskSetManager: Finished task 4.0 in stage 10.0 (TID 152) in 293 ms on 192.168.1.37 (executor driver) (5/40)
22/02/08 16:52:15 INFO Executor: Finished task 3.0 in stage 10.0 (TID 151). 1426 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 8.0 in stage 10.0 (TID 156) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO Executor: Running task 8.0 in stage 10.0 (TID 156)
22/02/08 16:52:15 INFO TaskSetManager: Finished task 3.0 in stage 10.0 (TID 151) in 295 ms on 192.168.1.37 (executor driver) (6/40)
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO MemoryStore: Block rdd_27_6 stored as values in memory (estimated size 25.9 MiB, free 1691.5 MiB)
22/02/08 16:52:15 INFO BlockManagerInfo: Added rdd_27_6 in memory on 192.168.1.37:53051 (size: 25.9 MiB, free: 1821.7 MiB)
22/02/08 16:52:15 INFO MemoryStore: Block rdd_27_7 stored as values in memory (estimated size 25.9 MiB, free 1749.4 MiB)
22/02/08 16:52:15 INFO BlockManagerInfo: Added rdd_27_7 in memory on 192.168.1.37:53051 (size: 25.9 MiB, free: 1795.8 MiB)
22/02/08 16:52:15 INFO MemoryStore: Block rdd_27_8 stored as values in memory (estimated size 26.4 MiB, free 1768.6 MiB)
22/02/08 16:52:15 INFO BlockManagerInfo: Added rdd_27_8 in memory on 192.168.1.37:53051 (size: 26.4 MiB, free: 1769.5 MiB)
22/02/08 16:52:15 INFO Executor: Finished task 6.0 in stage 10.0 (TID 154). 1426 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 9.0 in stage 10.0 (TID 157) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO Executor: Running task 9.0 in stage 10.0 (TID 157)
22/02/08 16:52:15 INFO TaskSetManager: Finished task 6.0 in stage 10.0 (TID 154) in 282 ms on 192.168.1.37 (executor driver) (7/40)
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO Executor: Finished task 7.0 in stage 10.0 (TID 155). 1426 bytes result sent to driver
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1516.4 KiB) non-empty blocks including 10 (1516.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO TaskSetManager: Starting task 10.0 in stage 10.0 (TID 158) (192.168.1.37, executor driver, partition 10, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO TaskSetManager: Finished task 7.0 in stage 10.0 (TID 155) in 283 ms on 192.168.1.37 (executor driver) (8/40)
22/02/08 16:52:15 INFO Executor: Running task 10.0 in stage 10.0 (TID 158)
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO Executor: Finished task 8.0 in stage 10.0 (TID 156). 1426 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 11.0 in stage 10.0 (TID 159) (192.168.1.37, executor driver, partition 11, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO Executor: Running task 11.0 in stage 10.0 (TID 159)
22/02/08 16:52:15 INFO TaskSetManager: Finished task 8.0 in stage 10.0 (TID 156) in 289 ms on 192.168.1.37 (executor driver) (9/40)
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1531.0 KiB) non-empty blocks including 10 (1531.0 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO MemoryStore: Block rdd_27_10 stored as values in memory (estimated size 25.8 MiB, free 1656.1 MiB)
22/02/08 16:52:15 INFO MemoryStore: Block rdd_27_9 stored as values in memory (estimated size 25.9 MiB, free 1630.1 MiB)
22/02/08 16:52:15 INFO BlockManagerInfo: Added rdd_27_10 in memory on 192.168.1.37:53051 (size: 25.8 MiB, free: 1743.6 MiB)
22/02/08 16:52:15 INFO BlockManagerInfo: Added rdd_27_9 in memory on 192.168.1.37:53051 (size: 25.9 MiB, free: 1717.7 MiB)
22/02/08 16:52:15 INFO MemoryStore: Block rdd_27_11 stored as values in memory (estimated size 26.4 MiB, free 1690.4 MiB)
22/02/08 16:52:15 INFO BlockManagerInfo: Added rdd_27_11 in memory on 192.168.1.37:53051 (size: 26.4 MiB, free: 1691.3 MiB)
22/02/08 16:52:15 INFO Executor: Finished task 10.0 in stage 10.0 (TID 158). 1469 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 12.0 in stage 10.0 (TID 160) (192.168.1.37, executor driver, partition 12, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO TaskSetManager: Finished task 10.0 in stage 10.0 (TID 158) in 359 ms on 192.168.1.37 (executor driver) (10/40)
22/02/08 16:52:15 INFO Executor: Running task 12.0 in stage 10.0 (TID 160)
22/02/08 16:52:15 INFO Executor: Finished task 9.0 in stage 10.0 (TID 157). 1469 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 13.0 in stage 10.0 (TID 161) (192.168.1.37, executor driver, partition 13, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO Executor: Running task 13.0 in stage 10.0 (TID 161)
22/02/08 16:52:15 INFO TaskSetManager: Finished task 9.0 in stage 10.0 (TID 157) in 363 ms on 192.168.1.37 (executor driver) (11/40)
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1472.7 KiB) non-empty blocks including 10 (1472.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO Executor: Finished task 11.0 in stage 10.0 (TID 159). 1469 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 14.0 in stage 10.0 (TID 162) (192.168.1.37, executor driver, partition 14, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO Executor: Running task 14.0 in stage 10.0 (TID 162)
22/02/08 16:52:15 INFO TaskSetManager: Finished task 11.0 in stage 10.0 (TID 159) in 362 ms on 192.168.1.37 (executor driver) (12/40)
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO MemoryStore: Block rdd_27_12 stored as values in memory (estimated size 26.4 MiB, free 1493.0 MiB)
22/02/08 16:52:15 INFO BlockManagerInfo: Added rdd_27_12 in memory on 192.168.1.37:53051 (size: 26.4 MiB, free: 1664.9 MiB)
22/02/08 16:52:15 INFO MemoryStore: Block rdd_27_13 stored as values in memory (estimated size 26.2 MiB, free 1548.1 MiB)
22/02/08 16:52:15 INFO BlockManagerInfo: Added rdd_27_13 in memory on 192.168.1.37:53051 (size: 26.2 MiB, free: 1638.7 MiB)
22/02/08 16:52:15 INFO MemoryStore: Block rdd_27_14 stored as values in memory (estimated size 26.3 MiB, free 1611.5 MiB)
22/02/08 16:52:15 INFO BlockManagerInfo: Added rdd_27_14 in memory on 192.168.1.37:53051 (size: 26.3 MiB, free: 1612.4 MiB)
22/02/08 16:52:15 INFO Executor: Finished task 12.0 in stage 10.0 (TID 160). 1469 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 15.0 in stage 10.0 (TID 163) (192.168.1.37, executor driver, partition 15, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO TaskSetManager: Finished task 12.0 in stage 10.0 (TID 160) in 306 ms on 192.168.1.37 (executor driver) (13/40)
22/02/08 16:52:15 INFO Executor: Running task 15.0 in stage 10.0 (TID 163)
22/02/08 16:52:15 INFO Executor: Finished task 13.0 in stage 10.0 (TID 161). 1426 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 16.0 in stage 10.0 (TID 164) (192.168.1.37, executor driver, partition 16, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:15 INFO Executor: Running task 16.0 in stage 10.0 (TID 164)
22/02/08 16:52:15 INFO TaskSetManager: Finished task 13.0 in stage 10.0 (TID 161) in 306 ms on 192.168.1.37 (executor driver) (14/40)
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1472.7 KiB) non-empty blocks including 10 (1472.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:15 INFO Executor: Finished task 14.0 in stage 10.0 (TID 162). 1426 bytes result sent to driver
22/02/08 16:52:15 INFO TaskSetManager: Starting task 17.0 in stage 10.0 (TID 165) (192.168.1.37, executor driver, partition 17, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:16 INFO Executor: Running task 17.0 in stage 10.0 (TID 165)
22/02/08 16:52:16 INFO TaskSetManager: Finished task 14.0 in stage 10.0 (TID 162) in 308 ms on 192.168.1.37 (executor driver) (15/40)
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 10 (1516.4 KiB) non-empty blocks including 10 (1516.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO MemoryStore: Block rdd_27_15 stored as values in memory (estimated size 26.1 MiB, free 1451.4 MiB)
22/02/08 16:52:16 INFO BlockManagerInfo: Added rdd_27_15 in memory on 192.168.1.37:53051 (size: 26.1 MiB, free: 1586.3 MiB)
22/02/08 16:52:16 INFO MemoryStore: Block rdd_27_16 stored as values in memory (estimated size 26.3 MiB, free 1517.9 MiB)
22/02/08 16:52:16 INFO BlockManagerInfo: Added rdd_27_16 in memory on 192.168.1.37:53051 (size: 26.3 MiB, free: 1560.0 MiB)
22/02/08 16:52:16 INFO MemoryStore: Block rdd_27_17 stored as values in memory (estimated size 26.2 MiB, free 1532.9 MiB)
22/02/08 16:52:16 INFO BlockManagerInfo: Added rdd_27_17 in memory on 192.168.1.37:53051 (size: 26.2 MiB, free: 1533.7 MiB)
22/02/08 16:52:16 INFO Executor: Finished task 15.0 in stage 10.0 (TID 163). 1426 bytes result sent to driver
22/02/08 16:52:16 INFO TaskSetManager: Starting task 18.0 in stage 10.0 (TID 166) (192.168.1.37, executor driver, partition 18, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:16 INFO Executor: Running task 18.0 in stage 10.0 (TID 166)
22/02/08 16:52:16 INFO TaskSetManager: Finished task 15.0 in stage 10.0 (TID 163) in 284 ms on 192.168.1.37 (executor driver) (16/40)
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO Executor: Finished task 16.0 in stage 10.0 (TID 164). 1426 bytes result sent to driver
22/02/08 16:52:16 INFO TaskSetManager: Starting task 19.0 in stage 10.0 (TID 167) (192.168.1.37, executor driver, partition 19, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:16 INFO Executor: Running task 19.0 in stage 10.0 (TID 167)
22/02/08 16:52:16 INFO TaskSetManager: Finished task 16.0 in stage 10.0 (TID 164) in 287 ms on 192.168.1.37 (executor driver) (17/40)
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO Executor: Finished task 17.0 in stage 10.0 (TID 165). 1426 bytes result sent to driver
22/02/08 16:52:16 INFO TaskSetManager: Starting task 20.0 in stage 10.0 (TID 168) (192.168.1.37, executor driver, partition 20, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:16 INFO Executor: Running task 20.0 in stage 10.0 (TID 168)
22/02/08 16:52:16 INFO TaskSetManager: Finished task 17.0 in stage 10.0 (TID 165) in 281 ms on 192.168.1.37 (executor driver) (18/40)
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO MemoryStore: Block rdd_27_18 stored as values in memory (estimated size 26.3 MiB, free 1377.3 MiB)
22/02/08 16:52:16 INFO BlockManagerInfo: Added rdd_27_18 in memory on 192.168.1.37:53051 (size: 26.3 MiB, free: 1507.4 MiB)
22/02/08 16:52:16 INFO MemoryStore: Block rdd_27_19 stored as values in memory (estimated size 26.0 MiB, free 1438.4 MiB)
22/02/08 16:52:16 INFO BlockManagerInfo: Added rdd_27_19 in memory on 192.168.1.37:53051 (size: 26.0 MiB, free: 1481.4 MiB)
22/02/08 16:52:16 INFO MemoryStore: Block rdd_27_20 stored as values in memory (estimated size 26.3 MiB, free 1454.2 MiB)
22/02/08 16:52:16 INFO BlockManagerInfo: Added rdd_27_20 in memory on 192.168.1.37:53051 (size: 26.3 MiB, free: 1455.1 MiB)
22/02/08 16:52:16 INFO Executor: Finished task 18.0 in stage 10.0 (TID 166). 1426 bytes result sent to driver
22/02/08 16:52:16 INFO TaskSetManager: Starting task 21.0 in stage 10.0 (TID 169) (192.168.1.37, executor driver, partition 21, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:16 INFO Executor: Running task 21.0 in stage 10.0 (TID 169)
22/02/08 16:52:16 INFO TaskSetManager: Finished task 18.0 in stage 10.0 (TID 166) in 331 ms on 192.168.1.37 (executor driver) (19/40)
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO Executor: Finished task 19.0 in stage 10.0 (TID 167). 1426 bytes result sent to driver
22/02/08 16:52:16 INFO TaskSetManager: Starting task 22.0 in stage 10.0 (TID 170) (192.168.1.37, executor driver, partition 22, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:16 INFO Executor: Running task 22.0 in stage 10.0 (TID 170)
22/02/08 16:52:16 INFO TaskSetManager: Finished task 19.0 in stage 10.0 (TID 167) in 332 ms on 192.168.1.37 (executor driver) (20/40)
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO Executor: Finished task 20.0 in stage 10.0 (TID 168). 1426 bytes result sent to driver
22/02/08 16:52:16 INFO TaskSetManager: Starting task 23.0 in stage 10.0 (TID 171) (192.168.1.37, executor driver, partition 23, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:16 INFO Executor: Running task 23.0 in stage 10.0 (TID 171)
22/02/08 16:52:16 INFO TaskSetManager: Finished task 20.0 in stage 10.0 (TID 168) in 332 ms on 192.168.1.37 (executor driver) (21/40)
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Getting 10 (1531.0 KiB) non-empty blocks including 10 (1531.0 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:16 INFO BlockManagerInfo: Removed broadcast_8_piece0 on 192.168.1.37:53051 in memory (size: 179.4 KiB, free: 1455.2 MiB)
22/02/08 16:52:16 INFO BlockManagerInfo: Removed broadcast_9_piece0 on 192.168.1.37:53051 in memory (size: 4.3 KiB, free: 1455.2 MiB)
22/02/08 16:52:17 INFO MemoryStore: Block rdd_27_21 stored as values in memory (estimated size 26.3 MiB, free 1254.0 MiB)
22/02/08 16:52:17 INFO BlockManagerInfo: Added rdd_27_21 in memory on 192.168.1.37:53051 (size: 26.3 MiB, free: 1428.9 MiB)
22/02/08 16:52:17 INFO MemoryStore: Block rdd_27_22 stored as values in memory (estimated size 26.3 MiB, free 1314.7 MiB)
22/02/08 16:52:17 INFO BlockManagerInfo: Added rdd_27_22 in memory on 192.168.1.37:53051 (size: 26.3 MiB, free: 1402.6 MiB)
22/02/08 16:52:17 INFO MemoryStore: Block rdd_27_23 stored as values in memory (estimated size 26.4 MiB, free 1375.9 MiB)
22/02/08 16:52:17 INFO BlockManagerInfo: Added rdd_27_23 in memory on 192.168.1.37:53051 (size: 26.4 MiB, free: 1376.2 MiB)
22/02/08 16:52:17 INFO Executor: Finished task 21.0 in stage 10.0 (TID 169). 1426 bytes result sent to driver
22/02/08 16:52:17 INFO TaskSetManager: Starting task 24.0 in stage 10.0 (TID 172) (192.168.1.37, executor driver, partition 24, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:17 INFO Executor: Running task 24.0 in stage 10.0 (TID 172)
22/02/08 16:52:17 INFO TaskSetManager: Finished task 21.0 in stage 10.0 (TID 169) in 554 ms on 192.168.1.37 (executor driver) (22/40)
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 10 (1516.4 KiB) non-empty blocks including 10 (1516.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO Executor: Finished task 22.0 in stage 10.0 (TID 170). 1426 bytes result sent to driver
22/02/08 16:52:17 INFO TaskSetManager: Starting task 25.0 in stage 10.0 (TID 173) (192.168.1.37, executor driver, partition 25, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:17 INFO TaskSetManager: Finished task 22.0 in stage 10.0 (TID 170) in 557 ms on 192.168.1.37 (executor driver) (23/40)
22/02/08 16:52:17 INFO Executor: Running task 25.0 in stage 10.0 (TID 173)
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO Executor: Finished task 23.0 in stage 10.0 (TID 171). 1426 bytes result sent to driver
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO TaskSetManager: Starting task 26.0 in stage 10.0 (TID 174) (192.168.1.37, executor driver, partition 26, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:17 INFO Executor: Running task 26.0 in stage 10.0 (TID 174)
22/02/08 16:52:17 INFO TaskSetManager: Finished task 23.0 in stage 10.0 (TID 171) in 558 ms on 192.168.1.37 (executor driver) (24/40)
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO MemoryStore: Block rdd_27_24 stored as values in memory (estimated size 26.2 MiB, free 1164.9 MiB)
22/02/08 16:52:17 INFO BlockManagerInfo: Added rdd_27_24 in memory on 192.168.1.37:53051 (size: 26.2 MiB, free: 1350.0 MiB)
22/02/08 16:52:17 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:52:17 INFO MemoryStore: Block rdd_27_26 stored as values in memory (estimated size 26.3 MiB, free 1297.1 MiB)
22/02/08 16:52:17 INFO MemoryStore: Block rdd_27_25 stored as values in memory (estimated size 26.2 MiB, free 1297.1 MiB)
22/02/08 16:52:17 INFO BlockManagerInfo: Added rdd_27_26 in memory on 192.168.1.37:53051 (size: 26.3 MiB, free: 1323.7 MiB)
22/02/08 16:52:17 INFO BlockManagerInfo: Added rdd_27_25 in memory on 192.168.1.37:53051 (size: 26.2 MiB, free: 1297.4 MiB)
22/02/08 16:52:17 INFO Executor: Finished task 24.0 in stage 10.0 (TID 172). 1469 bytes result sent to driver
22/02/08 16:52:17 INFO TaskSetManager: Starting task 27.0 in stage 10.0 (TID 175) (192.168.1.37, executor driver, partition 27, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:17 INFO Executor: Running task 27.0 in stage 10.0 (TID 175)
22/02/08 16:52:17 INFO TaskSetManager: Finished task 24.0 in stage 10.0 (TID 172) in 341 ms on 192.168.1.37 (executor driver) (25/40)
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 10 (1472.7 KiB) non-empty blocks including 10 (1472.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO Executor: Finished task 26.0 in stage 10.0 (TID 174). 1469 bytes result sent to driver
22/02/08 16:52:17 INFO TaskSetManager: Starting task 28.0 in stage 10.0 (TID 176) (192.168.1.37, executor driver, partition 28, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:17 INFO Executor: Running task 28.0 in stage 10.0 (TID 176)
22/02/08 16:52:17 INFO TaskSetManager: Finished task 26.0 in stage 10.0 (TID 174) in 339 ms on 192.168.1.37 (executor driver) (26/40)
22/02/08 16:52:17 INFO Executor: Finished task 25.0 in stage 10.0 (TID 173). 1469 bytes result sent to driver
22/02/08 16:52:17 INFO TaskSetManager: Starting task 29.0 in stage 10.0 (TID 177) (192.168.1.37, executor driver, partition 29, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:17 INFO TaskSetManager: Finished task 25.0 in stage 10.0 (TID 173) in 343 ms on 192.168.1.37 (executor driver) (27/40)
22/02/08 16:52:17 INFO Executor: Running task 29.0 in stage 10.0 (TID 177)
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 10 (1472.7 KiB) non-empty blocks including 10 (1472.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO MemoryStore: Block rdd_27_27 stored as values in memory (estimated size 26.4 MiB, free 1165.3 MiB)
22/02/08 16:52:17 INFO BlockManagerInfo: Added rdd_27_27 in memory on 192.168.1.37:53051 (size: 26.4 MiB, free: 1271.0 MiB)
22/02/08 16:52:17 INFO MemoryStore: Block rdd_27_28 stored as values in memory (estimated size 25.7 MiB, free 1202.9 MiB)
22/02/08 16:52:17 INFO BlockManagerInfo: Added rdd_27_28 in memory on 192.168.1.37:53051 (size: 25.7 MiB, free: 1245.3 MiB)
22/02/08 16:52:17 INFO MemoryStore: Block rdd_27_29 stored as values in memory (estimated size 26.4 MiB, free 1218.6 MiB)
22/02/08 16:52:17 INFO BlockManagerInfo: Added rdd_27_29 in memory on 192.168.1.37:53051 (size: 26.4 MiB, free: 1218.9 MiB)
22/02/08 16:52:17 INFO Executor: Finished task 27.0 in stage 10.0 (TID 175). 1426 bytes result sent to driver
22/02/08 16:52:17 INFO TaskSetManager: Starting task 30.0 in stage 10.0 (TID 178) (192.168.1.37, executor driver, partition 30, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:17 INFO Executor: Running task 30.0 in stage 10.0 (TID 178)
22/02/08 16:52:17 INFO TaskSetManager: Finished task 27.0 in stage 10.0 (TID 175) in 320 ms on 192.168.1.37 (executor driver) (28/40)
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO Executor: Finished task 28.0 in stage 10.0 (TID 176). 1426 bytes result sent to driver
22/02/08 16:52:17 INFO TaskSetManager: Starting task 31.0 in stage 10.0 (TID 179) (192.168.1.37, executor driver, partition 31, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:17 INFO Executor: Running task 31.0 in stage 10.0 (TID 179)
22/02/08 16:52:17 INFO TaskSetManager: Finished task 28.0 in stage 10.0 (TID 176) in 316 ms on 192.168.1.37 (executor driver) (29/40)
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO Executor: Finished task 29.0 in stage 10.0 (TID 177). 1426 bytes result sent to driver
22/02/08 16:52:17 INFO TaskSetManager: Starting task 32.0 in stage 10.0 (TID 180) (192.168.1.37, executor driver, partition 32, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:17 INFO Executor: Running task 32.0 in stage 10.0 (TID 180)
22/02/08 16:52:17 INFO TaskSetManager: Finished task 29.0 in stage 10.0 (TID 177) in 318 ms on 192.168.1.37 (executor driver) (30/40)
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Getting 10 (1516.4 KiB) non-empty blocks including 10 (1516.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:17 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO MemoryStore: Block rdd_27_30 stored as values in memory (estimated size 26.2 MiB, free 994.3 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added rdd_27_30 in memory on 192.168.1.37:53051 (size: 26.2 MiB, free: 1192.8 MiB)
22/02/08 16:52:18 INFO MemoryStore: Block rdd_27_31 stored as values in memory (estimated size 26.0 MiB, free 1078.8 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added rdd_27_31 in memory on 192.168.1.37:53051 (size: 26.0 MiB, free: 1166.8 MiB)
22/02/08 16:52:18 INFO MemoryStore: Block rdd_27_32 stored as values in memory (estimated size 26.2 MiB, free 1140.2 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added rdd_27_32 in memory on 192.168.1.37:53051 (size: 26.2 MiB, free: 1140.6 MiB)
22/02/08 16:52:18 INFO Executor: Finished task 30.0 in stage 10.0 (TID 178). 1426 bytes result sent to driver
22/02/08 16:52:18 INFO TaskSetManager: Starting task 33.0 in stage 10.0 (TID 181) (192.168.1.37, executor driver, partition 33, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:18 INFO TaskSetManager: Finished task 30.0 in stage 10.0 (TID 178) in 315 ms on 192.168.1.37 (executor driver) (31/40)
22/02/08 16:52:18 INFO Executor: Running task 33.0 in stage 10.0 (TID 181)
22/02/08 16:52:18 INFO Executor: Finished task 31.0 in stage 10.0 (TID 179). 1426 bytes result sent to driver
22/02/08 16:52:18 INFO TaskSetManager: Starting task 34.0 in stage 10.0 (TID 182) (192.168.1.37, executor driver, partition 34, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:18 INFO Executor: Running task 34.0 in stage 10.0 (TID 182)
22/02/08 16:52:18 INFO TaskSetManager: Finished task 31.0 in stage 10.0 (TID 179) in 310 ms on 192.168.1.37 (executor driver) (32/40)
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO Executor: Finished task 32.0 in stage 10.0 (TID 180). 1426 bytes result sent to driver
22/02/08 16:52:18 INFO TaskSetManager: Starting task 35.0 in stage 10.0 (TID 183) (192.168.1.37, executor driver, partition 35, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:18 INFO Executor: Running task 35.0 in stage 10.0 (TID 183)
22/02/08 16:52:18 INFO TaskSetManager: Finished task 32.0 in stage 10.0 (TID 180) in 312 ms on 192.168.1.37 (executor driver) (33/40)
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO MemoryStore: Block rdd_27_34 stored as values in memory (estimated size 26.3 MiB, free 945.1 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added rdd_27_34 in memory on 192.168.1.37:53051 (size: 26.3 MiB, free: 1114.3 MiB)
22/02/08 16:52:18 INFO MemoryStore: Block rdd_27_33 stored as values in memory (estimated size 26.4 MiB, free 999.0 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added rdd_27_33 in memory on 192.168.1.37:53051 (size: 26.4 MiB, free: 1087.9 MiB)
22/02/08 16:52:18 INFO MemoryStore: Block rdd_27_35 stored as values in memory (estimated size 26.1 MiB, free 1061.4 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added rdd_27_35 in memory on 192.168.1.37:53051 (size: 26.1 MiB, free: 1061.8 MiB)
22/02/08 16:52:18 INFO Executor: Finished task 34.0 in stage 10.0 (TID 182). 1426 bytes result sent to driver
22/02/08 16:52:18 INFO TaskSetManager: Starting task 36.0 in stage 10.0 (TID 184) (192.168.1.37, executor driver, partition 36, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:18 INFO Executor: Running task 36.0 in stage 10.0 (TID 184)
22/02/08 16:52:18 INFO TaskSetManager: Finished task 34.0 in stage 10.0 (TID 182) in 322 ms on 192.168.1.37 (executor driver) (34/40)
22/02/08 16:52:18 INFO Executor: Finished task 33.0 in stage 10.0 (TID 181). 1426 bytes result sent to driver
22/02/08 16:52:18 INFO TaskSetManager: Starting task 37.0 in stage 10.0 (TID 185) (192.168.1.37, executor driver, partition 37, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:18 INFO Executor: Running task 37.0 in stage 10.0 (TID 185)
22/02/08 16:52:18 INFO TaskSetManager: Finished task 33.0 in stage 10.0 (TID 181) in 323 ms on 192.168.1.37 (executor driver) (35/40)
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO Executor: Finished task 35.0 in stage 10.0 (TID 183). 1426 bytes result sent to driver
22/02/08 16:52:18 INFO TaskSetManager: Starting task 38.0 in stage 10.0 (TID 186) (192.168.1.37, executor driver, partition 38, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:18 INFO Executor: Running task 38.0 in stage 10.0 (TID 186)
22/02/08 16:52:18 INFO TaskSetManager: Finished task 35.0 in stage 10.0 (TID 183) in 320 ms on 192.168.1.37 (executor driver) (36/40)
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 10 (1487.2 KiB) non-empty blocks including 10 (1487.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO MemoryStore: Block rdd_27_36 stored as values in memory (estimated size 25.8 MiB, free 910.0 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added rdd_27_36 in memory on 192.168.1.37:53051 (size: 25.8 MiB, free: 1036.0 MiB)
22/02/08 16:52:18 INFO MemoryStore: Block rdd_27_37 stored as values in memory (estimated size 26.2 MiB, free 927.5 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added rdd_27_37 in memory on 192.168.1.37:53051 (size: 26.2 MiB, free: 1009.7 MiB)
22/02/08 16:52:18 INFO MemoryStore: Block rdd_27_38 stored as values in memory (estimated size 26.2 MiB, free 983.2 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added rdd_27_38 in memory on 192.168.1.37:53051 (size: 26.2 MiB, free: 983.5 MiB)
22/02/08 16:52:18 INFO Executor: Finished task 36.0 in stage 10.0 (TID 184). 1426 bytes result sent to driver
22/02/08 16:52:18 INFO TaskSetManager: Starting task 39.0 in stage 10.0 (TID 187) (192.168.1.37, executor driver, partition 39, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:18 INFO TaskSetManager: Finished task 36.0 in stage 10.0 (TID 184) in 309 ms on 192.168.1.37 (executor driver) (37/40)
22/02/08 16:52:18 INFO Executor: Running task 39.0 in stage 10.0 (TID 187)
22/02/08 16:52:18 INFO Executor: Finished task 37.0 in stage 10.0 (TID 185). 1426 bytes result sent to driver
22/02/08 16:52:18 INFO TaskSetManager: Finished task 37.0 in stage 10.0 (TID 185) in 309 ms on 192.168.1.37 (executor driver) (38/40)
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 1 (11.4 MiB) non-empty blocks including 1 (11.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO Executor: Finished task 38.0 in stage 10.0 (TID 186). 1426 bytes result sent to driver
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Getting 10 (1501.8 KiB) non-empty blocks including 10 (1501.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:18 INFO TaskSetManager: Finished task 38.0 in stage 10.0 (TID 186) in 308 ms on 192.168.1.37 (executor driver) (39/40)
22/02/08 16:52:18 INFO MemoryStore: Block rdd_27_39 stored as values in memory (estimated size 26.4 MiB, free 956.8 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added rdd_27_39 in memory on 192.168.1.37:53051 (size: 26.4 MiB, free: 957.1 MiB)
22/02/08 16:52:18 INFO Executor: Finished task 39.0 in stage 10.0 (TID 187). 1383 bytes result sent to driver
22/02/08 16:52:18 INFO TaskSetManager: Finished task 39.0 in stage 10.0 (TID 187) in 233 ms on 192.168.1.37 (executor driver) (40/40)
22/02/08 16:52:18 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
22/02/08 16:52:18 INFO DAGScheduler: ShuffleMapStage 10 (countByKey at BaseSparkCommitActionExecutor.java:196) finished in 4.605 s
22/02/08 16:52:18 INFO DAGScheduler: looking for newly runnable stages
22/02/08 16:52:18 INFO DAGScheduler: running: Set()
22/02/08 16:52:18 INFO DAGScheduler: waiting: Set(ResultStage 11)
22/02/08 16:52:18 INFO DAGScheduler: failed: Set()
22/02/08 16:52:18 INFO DAGScheduler: Submitting ResultStage 11 (ShuffledRDD[30] at countByKey at BaseSparkCommitActionExecutor.java:196), which has no missing parents
22/02/08 16:52:18 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 5.6 KiB, free 956.8 MiB)
22/02/08 16:52:18 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 956.8 MiB)
22/02/08 16:52:18 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on 192.168.1.37:53051 (size: 3.2 KiB, free: 957.1 MiB)
22/02/08 16:52:19 INFO SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:19 INFO DAGScheduler: Submitting 40 missing tasks from ResultStage 11 (ShuffledRDD[30] at countByKey at BaseSparkCommitActionExecutor.java:196) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/08 16:52:19 INFO TaskSchedulerImpl: Adding task set 11.0 with 40 tasks resource profile 0
22/02/08 16:52:19 INFO TaskSetManager: Starting task 8.0 in stage 11.0 (TID 188) (192.168.1.37, executor driver, partition 8, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Starting task 9.0 in stage 11.0 (TID 189) (192.168.1.37, executor driver, partition 9, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Starting task 12.0 in stage 11.0 (TID 190) (192.168.1.37, executor driver, partition 12, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 8.0 in stage 11.0 (TID 188)
22/02/08 16:52:19 INFO Executor: Running task 12.0 in stage 11.0 (TID 190)
22/02/08 16:52:19 INFO Executor: Running task 9.0 in stage 11.0 (TID 189)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (7.4 KiB) non-empty blocks including 40 (7.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (7.4 KiB) non-empty blocks including 40 (7.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (4.6 KiB) non-empty blocks including 40 (4.6 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 8.0 in stage 11.0 (TID 188). 1348 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 9.0 in stage 11.0 (TID 189). 1349 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 12.0 in stage 11.0 (TID 190). 1263 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 14.0 in stage 11.0 (TID 191) (192.168.1.37, executor driver, partition 14, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 14.0 in stage 11.0 (TID 191)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 17.0 in stage 11.0 (TID 192) (192.168.1.37, executor driver, partition 17, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 8.0 in stage 11.0 (TID 188) in 10 ms on 192.168.1.37 (executor driver) (1/40)
22/02/08 16:52:19 INFO Executor: Running task 17.0 in stage 11.0 (TID 192)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 9.0 in stage 11.0 (TID 189) in 10 ms on 192.168.1.37 (executor driver) (2/40)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 18.0 in stage 11.0 (TID 193) (192.168.1.37, executor driver, partition 18, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 18.0 in stage 11.0 (TID 193)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 12.0 in stage 11.0 (TID 190) in 10 ms on 192.168.1.37 (executor driver) (3/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (3.8 KiB) non-empty blocks including 40 (3.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (3.8 KiB) non-empty blocks including 40 (3.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (4.6 KiB) non-empty blocks including 40 (4.6 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 18.0 in stage 11.0 (TID 193). 1264 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 17.0 in stage 11.0 (TID 192). 1241 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 14.0 in stage 11.0 (TID 191). 1241 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 20.0 in stage 11.0 (TID 194) (192.168.1.37, executor driver, partition 20, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 20.0 in stage 11.0 (TID 194)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 21.0 in stage 11.0 (TID 195) (192.168.1.37, executor driver, partition 21, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 21.0 in stage 11.0 (TID 195)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 24.0 in stage 11.0 (TID 196) (192.168.1.37, executor driver, partition 24, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 24.0 in stage 11.0 (TID 196)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 18.0 in stage 11.0 (TID 193) in 8 ms on 192.168.1.37 (executor driver) (4/40)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 17.0 in stage 11.0 (TID 192) in 9 ms on 192.168.1.37 (executor driver) (5/40)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 14.0 in stage 11.0 (TID 191) in 9 ms on 192.168.1.37 (executor driver) (6/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (7.4 KiB) non-empty blocks including 40 (7.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (3.8 KiB) non-empty blocks including 40 (3.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (7.4 KiB) non-empty blocks including 40 (7.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 24.0 in stage 11.0 (TID 196). 1349 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 21.0 in stage 11.0 (TID 195). 1241 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 30.0 in stage 11.0 (TID 197) (192.168.1.37, executor driver, partition 30, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Finished task 20.0 in stage 11.0 (TID 194). 1349 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Running task 30.0 in stage 11.0 (TID 197)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 31.0 in stage 11.0 (TID 198) (192.168.1.37, executor driver, partition 31, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 31.0 in stage 11.0 (TID 198)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 24.0 in stage 11.0 (TID 196) in 8 ms on 192.168.1.37 (executor driver) (7/40)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 21.0 in stage 11.0 (TID 195) in 8 ms on 192.168.1.37 (executor driver) (8/40)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 33.0 in stage 11.0 (TID 199) (192.168.1.37, executor driver, partition 33, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 20.0 in stage 11.0 (TID 194) in 9 ms on 192.168.1.37 (executor driver) (9/40)
22/02/08 16:52:19 INFO Executor: Running task 33.0 in stage 11.0 (TID 199)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (3.8 KiB) non-empty blocks including 40 (3.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (3.8 KiB) non-empty blocks including 40 (3.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (8.1 KiB) non-empty blocks including 40 (8.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 30.0 in stage 11.0 (TID 197). 1241 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 35.0 in stage 11.0 (TID 200) (192.168.1.37, executor driver, partition 35, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 35.0 in stage 11.0 (TID 200)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 30.0 in stage 11.0 (TID 197) in 8 ms on 192.168.1.37 (executor driver) (10/40)
22/02/08 16:52:19 INFO Executor: Finished task 33.0 in stage 11.0 (TID 199). 1371 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 31.0 in stage 11.0 (TID 198). 1241 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 38.0 in stage 11.0 (TID 201) (192.168.1.37, executor driver, partition 38, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 38.0 in stage 11.0 (TID 201)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 33.0 in stage 11.0 (TID 199) in 7 ms on 192.168.1.37 (executor driver) (11/40)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 31.0 in stage 11.0 (TID 198) in 8 ms on 192.168.1.37 (executor driver) (12/40)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 39.0 in stage 11.0 (TID 202) (192.168.1.37, executor driver, partition 39, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (11.9 KiB) non-empty blocks including 40 (11.9 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO Executor: Running task 39.0 in stage 11.0 (TID 202)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (7.4 KiB) non-empty blocks including 40 (7.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (7.4 KiB) non-empty blocks including 40 (7.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 39.0 in stage 11.0 (TID 202). 1349 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 35.0 in stage 11.0 (TID 200). 1515 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 38.0 in stage 11.0 (TID 201). 1349 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 0.0 in stage 11.0 (TID 203) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 39.0 in stage 11.0 (TID 202) in 7 ms on 192.168.1.37 (executor driver) (13/40)
22/02/08 16:52:19 INFO Executor: Running task 0.0 in stage 11.0 (TID 203)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 1.0 in stage 11.0 (TID 204) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 1.0 in stage 11.0 (TID 204)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 35.0 in stage 11.0 (TID 200) in 9 ms on 192.168.1.37 (executor driver) (14/40)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 2.0 in stage 11.0 (TID 205) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 38.0 in stage 11.0 (TID 201) in 8 ms on 192.168.1.37 (executor driver) (15/40)
22/02/08 16:52:19 INFO Executor: Running task 2.0 in stage 11.0 (TID 205)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 0.0 in stage 11.0 (TID 203). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 1.0 in stage 11.0 (TID 204). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 3.0 in stage 11.0 (TID 206) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Finished task 0.0 in stage 11.0 (TID 203) in 3 ms on 192.168.1.37 (executor driver) (16/40)
22/02/08 16:52:19 INFO Executor: Running task 3.0 in stage 11.0 (TID 206)
22/02/08 16:52:19 INFO Executor: Finished task 2.0 in stage 11.0 (TID 205). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 4.0 in stage 11.0 (TID 207) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 1.0 in stage 11.0 (TID 204) in 2 ms on 192.168.1.37 (executor driver) (17/40)
22/02/08 16:52:19 INFO Executor: Running task 4.0 in stage 11.0 (TID 207)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 5.0 in stage 11.0 (TID 208) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 2.0 in stage 11.0 (TID 205) in 3 ms on 192.168.1.37 (executor driver) (18/40)
22/02/08 16:52:19 INFO Executor: Running task 5.0 in stage 11.0 (TID 208)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO Executor: Finished task 3.0 in stage 11.0 (TID 206). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Starting task 6.0 in stage 11.0 (TID 209) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Finished task 4.0 in stage 11.0 (TID 207). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Finished task 3.0 in stage 11.0 (TID 206) in 2 ms on 192.168.1.37 (executor driver) (19/40)
22/02/08 16:52:19 INFO Executor: Running task 6.0 in stage 11.0 (TID 209)
22/02/08 16:52:19 INFO Executor: Finished task 5.0 in stage 11.0 (TID 208). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 7.0 in stage 11.0 (TID 210) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 7.0 in stage 11.0 (TID 210)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 4.0 in stage 11.0 (TID 207) in 3 ms on 192.168.1.37 (executor driver) (20/40)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 10.0 in stage 11.0 (TID 211) (192.168.1.37, executor driver, partition 10, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 5.0 in stage 11.0 (TID 208) in 2 ms on 192.168.1.37 (executor driver) (21/40)
22/02/08 16:52:19 INFO Executor: Running task 10.0 in stage 11.0 (TID 211)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 6.0 in stage 11.0 (TID 209). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 7.0 in stage 11.0 (TID 210). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 11.0 in stage 11.0 (TID 212) (192.168.1.37, executor driver, partition 11, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Running task 11.0 in stage 11.0 (TID 212)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 6.0 in stage 11.0 (TID 209) in 3 ms on 192.168.1.37 (executor driver) (22/40)
22/02/08 16:52:19 INFO Executor: Finished task 10.0 in stage 11.0 (TID 211). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 13.0 in stage 11.0 (TID 213) (192.168.1.37, executor driver, partition 13, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 7.0 in stage 11.0 (TID 210) in 2 ms on 192.168.1.37 (executor driver) (23/40)
22/02/08 16:52:19 INFO Executor: Running task 13.0 in stage 11.0 (TID 213)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 15.0 in stage 11.0 (TID 214) (192.168.1.37, executor driver, partition 15, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 10.0 in stage 11.0 (TID 211) in 3 ms on 192.168.1.37 (executor driver) (24/40)
22/02/08 16:52:19 INFO Executor: Running task 15.0 in stage 11.0 (TID 214)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 11.0 in stage 11.0 (TID 212). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 13.0 in stage 11.0 (TID 213). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 16.0 in stage 11.0 (TID 215) (192.168.1.37, executor driver, partition 16, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Finished task 15.0 in stage 11.0 (TID 214). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Finished task 11.0 in stage 11.0 (TID 212) in 3 ms on 192.168.1.37 (executor driver) (25/40)
22/02/08 16:52:19 INFO Executor: Running task 16.0 in stage 11.0 (TID 215)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 19.0 in stage 11.0 (TID 216) (192.168.1.37, executor driver, partition 19, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 13.0 in stage 11.0 (TID 213) in 3 ms on 192.168.1.37 (executor driver) (26/40)
22/02/08 16:52:19 INFO Executor: Running task 19.0 in stage 11.0 (TID 216)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 22.0 in stage 11.0 (TID 217) (192.168.1.37, executor driver, partition 22, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO TaskSetManager: Finished task 15.0 in stage 11.0 (TID 214) in 3 ms on 192.168.1.37 (executor driver) (27/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Running task 22.0 in stage 11.0 (TID 217)
22/02/08 16:52:19 INFO Executor: Finished task 16.0 in stage 11.0 (TID 215). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Starting task 23.0 in stage 11.0 (TID 218) (192.168.1.37, executor driver, partition 23, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Finished task 19.0 in stage 11.0 (TID 216). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Running task 23.0 in stage 11.0 (TID 218)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 16.0 in stage 11.0 (TID 215) in 2 ms on 192.168.1.37 (executor driver) (28/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Starting task 25.0 in stage 11.0 (TID 219) (192.168.1.37, executor driver, partition 25, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 19.0 in stage 11.0 (TID 216) in 2 ms on 192.168.1.37 (executor driver) (29/40)
22/02/08 16:52:19 INFO Executor: Running task 25.0 in stage 11.0 (TID 219)
22/02/08 16:52:19 INFO Executor: Finished task 22.0 in stage 11.0 (TID 217). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO TaskSetManager: Starting task 26.0 in stage 11.0 (TID 220) (192.168.1.37, executor driver, partition 26, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Running task 26.0 in stage 11.0 (TID 220)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 22.0 in stage 11.0 (TID 217) in 2 ms on 192.168.1.37 (executor driver) (30/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 23.0 in stage 11.0 (TID 218). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 25.0 in stage 11.0 (TID 219). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 27.0 in stage 11.0 (TID 221) (192.168.1.37, executor driver, partition 27, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 23.0 in stage 11.0 (TID 218) in 2 ms on 192.168.1.37 (executor driver) (31/40)
22/02/08 16:52:19 INFO Executor: Running task 27.0 in stage 11.0 (TID 221)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO TaskSetManager: Starting task 28.0 in stage 11.0 (TID 222) (192.168.1.37, executor driver, partition 28, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Running task 28.0 in stage 11.0 (TID 222)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 25.0 in stage 11.0 (TID 219) in 3 ms on 192.168.1.37 (executor driver) (32/40)
22/02/08 16:52:19 INFO Executor: Finished task 26.0 in stage 11.0 (TID 220). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 29.0 in stage 11.0 (TID 223) (192.168.1.37, executor driver, partition 29, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Finished task 26.0 in stage 11.0 (TID 220) in 2 ms on 192.168.1.37 (executor driver) (33/40)
22/02/08 16:52:19 INFO Executor: Running task 29.0 in stage 11.0 (TID 223)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 27.0 in stage 11.0 (TID 221). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 28.0 in stage 11.0 (TID 222). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 32.0 in stage 11.0 (TID 224) (192.168.1.37, executor driver, partition 32, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 32.0 in stage 11.0 (TID 224)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 27.0 in stage 11.0 (TID 221) in 2 ms on 192.168.1.37 (executor driver) (34/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Starting task 34.0 in stage 11.0 (TID 225) (192.168.1.37, executor driver, partition 34, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 28.0 in stage 11.0 (TID 222) in 3 ms on 192.168.1.37 (executor driver) (35/40)
22/02/08 16:52:19 INFO Executor: Running task 34.0 in stage 11.0 (TID 225)
22/02/08 16:52:19 INFO Executor: Finished task 29.0 in stage 11.0 (TID 223). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 36.0 in stage 11.0 (TID 226) (192.168.1.37, executor driver, partition 36, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 36.0 in stage 11.0 (TID 226)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 29.0 in stage 11.0 (TID 223) in 2 ms on 192.168.1.37 (executor driver) (36/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 32.0 in stage 11.0 (TID 224). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 34.0 in stage 11.0 (TID 225). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 37.0 in stage 11.0 (TID 227) (192.168.1.37, executor driver, partition 37, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 32.0 in stage 11.0 (TID 224) in 2 ms on 192.168.1.37 (executor driver) (37/40)
22/02/08 16:52:19 INFO Executor: Running task 37.0 in stage 11.0 (TID 227)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Finished task 34.0 in stage 11.0 (TID 225) in 3 ms on 192.168.1.37 (executor driver) (38/40)
22/02/08 16:52:19 INFO Executor: Finished task 36.0 in stage 11.0 (TID 226). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Finished task 36.0 in stage 11.0 (TID 226) in 2 ms on 192.168.1.37 (executor driver) (39/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 37.0 in stage 11.0 (TID 227). 1141 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Finished task 37.0 in stage 11.0 (TID 227) in 2 ms on 192.168.1.37 (executor driver) (40/40)
22/02/08 16:52:19 INFO TaskSchedulerImpl: Removed TaskSet 11.0, whose tasks have all completed, from pool
22/02/08 16:52:19 INFO DAGScheduler: ResultStage 11 (countByKey at BaseSparkCommitActionExecutor.java:196) finished in 0.063 s
22/02/08 16:52:19 INFO DAGScheduler: Job 5 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:52:19 INFO TaskSchedulerImpl: Killing all running tasks in stage 11: Stage finished
22/02/08 16:52:19 INFO DAGScheduler: Job 5 finished: countByKey at BaseSparkCommitActionExecutor.java:196, took 16.155694 s
22/02/08 16:52:19 INFO BaseSparkCommitActionExecutor: Workload profile :WorkloadProfile {globalStat=WorkloadStat {numInserts=350005, numUpdates=349862}, partitionStat={2022/1/31=WorkloadStat {numInserts=34955, numUpdates=34965}, 2022/1/30=WorkloadStat {numInserts=34690, numUpdates=34779}, 2022/1/28=WorkloadStat {numInserts=34990, numUpdates=34810}, 2022/1/27=WorkloadStat {numInserts=35245, numUpdates=35281}, 2022/2/2=WorkloadStat {numInserts=34880, numUpdates=35122}, 2022/1/29=WorkloadStat {numInserts=34490, numUpdates=34456}, 2022/1/24=WorkloadStat {numInserts=35065, numUpdates=35051}, 2022/2/1=WorkloadStat {numInserts=34885, numUpdates=34882}, 2022/1/26=WorkloadStat {numInserts=35565, numUpdates=35307}, 2022/1/25=WorkloadStat {numInserts=35240, numUpdates=35209}}, operationType=UPSERT}
22/02/08 16:52:19 INFO HoodieActiveTimeline: Checking for file exists ?file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220208165137552.deltacommit.requested
22/02/08 16:52:19 INFO HoodieActiveTimeline: Create new file for toInstant ?file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/20220208165137552.deltacommit.inflight
22/02/08 16:52:19 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:19 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:19 INFO SparkContext: Starting job: collect at SparkRejectUpdateStrategy.java:52
22/02/08 16:52:19 INFO DAGScheduler: Registering RDD 33 (distinct at SparkRejectUpdateStrategy.java:52) as input to shuffle 5
22/02/08 16:52:19 INFO DAGScheduler: Got job 6 (collect at SparkRejectUpdateStrategy.java:52) with 40 output partitions
22/02/08 16:52:19 INFO DAGScheduler: Final stage: ResultStage 16 (collect at SparkRejectUpdateStrategy.java:52)
22/02/08 16:52:19 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 15)
22/02/08 16:52:19 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 15)
22/02/08 16:52:19 INFO DAGScheduler: Submitting ShuffleMapStage 15 (MapPartitionsRDD[33] at distinct at SparkRejectUpdateStrategy.java:52), which has no missing parents
22/02/08 16:52:19 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 9.6 KiB, free 956.7 MiB)
22/02/08 16:52:19 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 4.9 KiB, free 956.7 MiB)
22/02/08 16:52:19 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on 192.168.1.37:53051 (size: 4.9 KiB, free: 957.1 MiB)
22/02/08 16:52:19 INFO SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:19 INFO DAGScheduler: Submitting 40 missing tasks from ShuffleMapStage 15 (MapPartitionsRDD[33] at distinct at SparkRejectUpdateStrategy.java:52) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/08 16:52:19 INFO TaskSchedulerImpl: Adding task set 15.0 with 40 tasks resource profile 0
22/02/08 16:52:19 INFO TaskSetManager: Starting task 0.0 in stage 15.0 (TID 228) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Starting task 1.0 in stage 15.0 (TID 229) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Starting task 2.0 in stage 15.0 (TID 230) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 0.0 in stage 15.0 (TID 228)
22/02/08 16:52:19 INFO Executor: Running task 1.0 in stage 15.0 (TID 229)
22/02/08 16:52:19 INFO Executor: Running task 2.0 in stage 15.0 (TID 230)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_1 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_2 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_0 locally
22/02/08 16:52:19 INFO BlockManagerInfo: Removed broadcast_11_piece0 on 192.168.1.37:53051 in memory (size: 3.2 KiB, free: 957.1 MiB)
22/02/08 16:52:19 INFO Executor: Finished task 0.0 in stage 15.0 (TID 228). 1254 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 3.0 in stage 15.0 (TID 231) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 3.0 in stage 15.0 (TID 231)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 0.0 in stage 15.0 (TID 228) in 74 ms on 192.168.1.37 (executor driver) (1/40)
22/02/08 16:52:19 INFO Executor: Finished task 1.0 in stage 15.0 (TID 229). 1254 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 4.0 in stage 15.0 (TID 232) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 4.0 in stage 15.0 (TID 232)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 1.0 in stage 15.0 (TID 229) in 75 ms on 192.168.1.37 (executor driver) (2/40)
22/02/08 16:52:19 INFO Executor: Finished task 2.0 in stage 15.0 (TID 230). 1254 bytes result sent to driver
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_3 locally
22/02/08 16:52:19 INFO TaskSetManager: Starting task 5.0 in stage 15.0 (TID 233) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 5.0 in stage 15.0 (TID 233)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 2.0 in stage 15.0 (TID 230) in 75 ms on 192.168.1.37 (executor driver) (3/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_4 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_5 locally
22/02/08 16:52:19 INFO Executor: Finished task 4.0 in stage 15.0 (TID 232). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 6.0 in stage 15.0 (TID 234) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 6.0 in stage 15.0 (TID 234)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 4.0 in stage 15.0 (TID 232) in 27 ms on 192.168.1.37 (executor driver) (4/40)
22/02/08 16:52:19 INFO Executor: Finished task 5.0 in stage 15.0 (TID 233). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 7.0 in stage 15.0 (TID 235) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 7.0 in stage 15.0 (TID 235)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 5.0 in stage 15.0 (TID 233) in 28 ms on 192.168.1.37 (executor driver) (5/40)
22/02/08 16:52:19 INFO Executor: Finished task 3.0 in stage 15.0 (TID 231). 1211 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 8.0 in stage 15.0 (TID 236) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_6 locally
22/02/08 16:52:19 INFO Executor: Running task 8.0 in stage 15.0 (TID 236)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 3.0 in stage 15.0 (TID 231) in 30 ms on 192.168.1.37 (executor driver) (6/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_7 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_8 locally
22/02/08 16:52:19 INFO Executor: Finished task 7.0 in stage 15.0 (TID 235). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 9.0 in stage 15.0 (TID 237) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 9.0 in stage 15.0 (TID 237)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 7.0 in stage 15.0 (TID 235) in 27 ms on 192.168.1.37 (executor driver) (7/40)
22/02/08 16:52:19 INFO Executor: Finished task 6.0 in stage 15.0 (TID 234). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 10.0 in stage 15.0 (TID 238) (192.168.1.37, executor driver, partition 10, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 10.0 in stage 15.0 (TID 238)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 6.0 in stage 15.0 (TID 234) in 29 ms on 192.168.1.37 (executor driver) (8/40)
22/02/08 16:52:19 INFO Executor: Finished task 8.0 in stage 15.0 (TID 236). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_9 locally
22/02/08 16:52:19 INFO TaskSetManager: Starting task 11.0 in stage 15.0 (TID 239) (192.168.1.37, executor driver, partition 11, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 11.0 in stage 15.0 (TID 239)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 8.0 in stage 15.0 (TID 236) in 29 ms on 192.168.1.37 (executor driver) (9/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_10 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_11 locally
22/02/08 16:52:19 INFO Executor: Finished task 9.0 in stage 15.0 (TID 237). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 12.0 in stage 15.0 (TID 240) (192.168.1.37, executor driver, partition 12, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 12.0 in stage 15.0 (TID 240)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 9.0 in stage 15.0 (TID 237) in 28 ms on 192.168.1.37 (executor driver) (10/40)
22/02/08 16:52:19 INFO Executor: Finished task 11.0 in stage 15.0 (TID 239). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 13.0 in stage 15.0 (TID 241) (192.168.1.37, executor driver, partition 13, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 13.0 in stage 15.0 (TID 241)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 11.0 in stage 15.0 (TID 239) in 28 ms on 192.168.1.37 (executor driver) (11/40)
22/02/08 16:52:19 INFO Executor: Finished task 10.0 in stage 15.0 (TID 238). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_12 locally
22/02/08 16:52:19 INFO TaskSetManager: Starting task 14.0 in stage 15.0 (TID 242) (192.168.1.37, executor driver, partition 14, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 14.0 in stage 15.0 (TID 242)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 10.0 in stage 15.0 (TID 238) in 29 ms on 192.168.1.37 (executor driver) (12/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_13 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_14 locally
22/02/08 16:52:19 INFO Executor: Finished task 12.0 in stage 15.0 (TID 240). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 15.0 in stage 15.0 (TID 243) (192.168.1.37, executor driver, partition 15, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 12.0 in stage 15.0 (TID 240) in 29 ms on 192.168.1.37 (executor driver) (13/40)
22/02/08 16:52:19 INFO Executor: Running task 15.0 in stage 15.0 (TID 243)
22/02/08 16:52:19 INFO Executor: Finished task 14.0 in stage 15.0 (TID 242). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 16.0 in stage 15.0 (TID 244) (192.168.1.37, executor driver, partition 16, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 16.0 in stage 15.0 (TID 244)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 14.0 in stage 15.0 (TID 242) in 28 ms on 192.168.1.37 (executor driver) (14/40)
22/02/08 16:52:19 INFO Executor: Finished task 13.0 in stage 15.0 (TID 241). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 17.0 in stage 15.0 (TID 245) (192.168.1.37, executor driver, partition 17, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 13.0 in stage 15.0 (TID 241) in 29 ms on 192.168.1.37 (executor driver) (15/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_15 locally
22/02/08 16:52:19 INFO Executor: Running task 17.0 in stage 15.0 (TID 245)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_16 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_17 locally
22/02/08 16:52:19 INFO Executor: Finished task 16.0 in stage 15.0 (TID 244). 1211 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 15.0 in stage 15.0 (TID 243). 1254 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 17.0 in stage 15.0 (TID 245). 1211 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 18.0 in stage 15.0 (TID 246) (192.168.1.37, executor driver, partition 18, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 18.0 in stage 15.0 (TID 246)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 16.0 in stage 15.0 (TID 244) in 66 ms on 192.168.1.37 (executor driver) (16/40)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 19.0 in stage 15.0 (TID 247) (192.168.1.37, executor driver, partition 19, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 15.0 in stage 15.0 (TID 243) in 67 ms on 192.168.1.37 (executor driver) (17/40)
22/02/08 16:52:19 INFO Executor: Running task 19.0 in stage 15.0 (TID 247)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 20.0 in stage 15.0 (TID 248) (192.168.1.37, executor driver, partition 20, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 20.0 in stage 15.0 (TID 248)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 17.0 in stage 15.0 (TID 245) in 66 ms on 192.168.1.37 (executor driver) (18/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_18 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_19 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_20 locally
22/02/08 16:52:19 INFO Executor: Finished task 18.0 in stage 15.0 (TID 246). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 21.0 in stage 15.0 (TID 249) (192.168.1.37, executor driver, partition 21, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 21.0 in stage 15.0 (TID 249)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 18.0 in stage 15.0 (TID 246) in 28 ms on 192.168.1.37 (executor driver) (19/40)
22/02/08 16:52:19 INFO Executor: Finished task 20.0 in stage 15.0 (TID 248). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 22.0 in stage 15.0 (TID 250) (192.168.1.37, executor driver, partition 22, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 22.0 in stage 15.0 (TID 250)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 20.0 in stage 15.0 (TID 248) in 29 ms on 192.168.1.37 (executor driver) (20/40)
22/02/08 16:52:19 INFO Executor: Finished task 19.0 in stage 15.0 (TID 247). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_21 locally
22/02/08 16:52:19 INFO TaskSetManager: Starting task 23.0 in stage 15.0 (TID 251) (192.168.1.37, executor driver, partition 23, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 19.0 in stage 15.0 (TID 247) in 30 ms on 192.168.1.37 (executor driver) (21/40)
22/02/08 16:52:19 INFO Executor: Running task 23.0 in stage 15.0 (TID 251)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_22 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_23 locally
22/02/08 16:52:19 INFO Executor: Finished task 22.0 in stage 15.0 (TID 250). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 24.0 in stage 15.0 (TID 252) (192.168.1.37, executor driver, partition 24, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 22.0 in stage 15.0 (TID 250) in 28 ms on 192.168.1.37 (executor driver) (22/40)
22/02/08 16:52:19 INFO Executor: Running task 24.0 in stage 15.0 (TID 252)
22/02/08 16:52:19 INFO Executor: Finished task 23.0 in stage 15.0 (TID 251). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 25.0 in stage 15.0 (TID 253) (192.168.1.37, executor driver, partition 25, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 25.0 in stage 15.0 (TID 253)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 23.0 in stage 15.0 (TID 251) in 28 ms on 192.168.1.37 (executor driver) (23/40)
22/02/08 16:52:19 INFO Executor: Finished task 21.0 in stage 15.0 (TID 249). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_24 locally
22/02/08 16:52:19 INFO TaskSetManager: Starting task 26.0 in stage 15.0 (TID 254) (192.168.1.37, executor driver, partition 26, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 26.0 in stage 15.0 (TID 254)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 21.0 in stage 15.0 (TID 249) in 31 ms on 192.168.1.37 (executor driver) (24/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_25 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_26 locally
22/02/08 16:52:19 INFO Executor: Finished task 24.0 in stage 15.0 (TID 252). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 27.0 in stage 15.0 (TID 255) (192.168.1.37, executor driver, partition 27, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 24.0 in stage 15.0 (TID 252) in 29 ms on 192.168.1.37 (executor driver) (25/40)
22/02/08 16:52:19 INFO Executor: Running task 27.0 in stage 15.0 (TID 255)
22/02/08 16:52:19 INFO Executor: Finished task 25.0 in stage 15.0 (TID 253). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 28.0 in stage 15.0 (TID 256) (192.168.1.37, executor driver, partition 28, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 28.0 in stage 15.0 (TID 256)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 25.0 in stage 15.0 (TID 253) in 28 ms on 192.168.1.37 (executor driver) (26/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_27 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_28 locally
22/02/08 16:52:19 INFO Executor: Finished task 26.0 in stage 15.0 (TID 254). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 29.0 in stage 15.0 (TID 257) (192.168.1.37, executor driver, partition 29, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 29.0 in stage 15.0 (TID 257)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 26.0 in stage 15.0 (TID 254) in 30 ms on 192.168.1.37 (executor driver) (27/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_29 locally
22/02/08 16:52:19 INFO Executor: Finished task 28.0 in stage 15.0 (TID 256). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 30.0 in stage 15.0 (TID 258) (192.168.1.37, executor driver, partition 30, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 30.0 in stage 15.0 (TID 258)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 28.0 in stage 15.0 (TID 256) in 28 ms on 192.168.1.37 (executor driver) (28/40)
22/02/08 16:52:19 INFO Executor: Finished task 27.0 in stage 15.0 (TID 255). 1211 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 31.0 in stage 15.0 (TID 259) (192.168.1.37, executor driver, partition 31, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 27.0 in stage 15.0 (TID 255) in 29 ms on 192.168.1.37 (executor driver) (29/40)
22/02/08 16:52:19 INFO Executor: Running task 31.0 in stage 15.0 (TID 259)
22/02/08 16:52:19 INFO Executor: Finished task 29.0 in stage 15.0 (TID 257). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 32.0 in stage 15.0 (TID 260) (192.168.1.37, executor driver, partition 32, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_31 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_30 locally
22/02/08 16:52:19 INFO Executor: Running task 32.0 in stage 15.0 (TID 260)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 29.0 in stage 15.0 (TID 257) in 28 ms on 192.168.1.37 (executor driver) (30/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_32 locally
22/02/08 16:52:19 INFO Executor: Finished task 31.0 in stage 15.0 (TID 259). 1211 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 33.0 in stage 15.0 (TID 261) (192.168.1.37, executor driver, partition 33, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 33.0 in stage 15.0 (TID 261)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 31.0 in stage 15.0 (TID 259) in 33 ms on 192.168.1.37 (executor driver) (31/40)
22/02/08 16:52:19 INFO Executor: Finished task 32.0 in stage 15.0 (TID 260). 1211 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 34.0 in stage 15.0 (TID 262) (192.168.1.37, executor driver, partition 34, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 34.0 in stage 15.0 (TID 262)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 32.0 in stage 15.0 (TID 260) in 32 ms on 192.168.1.37 (executor driver) (32/40)
22/02/08 16:52:19 INFO Executor: Finished task 30.0 in stage 15.0 (TID 258). 1211 bytes result sent to driver
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_33 locally
22/02/08 16:52:19 INFO TaskSetManager: Starting task 35.0 in stage 15.0 (TID 263) (192.168.1.37, executor driver, partition 35, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 35.0 in stage 15.0 (TID 263)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 30.0 in stage 15.0 (TID 258) in 35 ms on 192.168.1.37 (executor driver) (33/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_34 locally
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_35 locally
22/02/08 16:52:19 INFO Executor: Finished task 34.0 in stage 15.0 (TID 262). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 36.0 in stage 15.0 (TID 264) (192.168.1.37, executor driver, partition 36, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 36.0 in stage 15.0 (TID 264)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 34.0 in stage 15.0 (TID 262) in 29 ms on 192.168.1.37 (executor driver) (34/40)
22/02/08 16:52:19 INFO Executor: Finished task 33.0 in stage 15.0 (TID 261). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 37.0 in stage 15.0 (TID 265) (192.168.1.37, executor driver, partition 37, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 37.0 in stage 15.0 (TID 265)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 33.0 in stage 15.0 (TID 261) in 30 ms on 192.168.1.37 (executor driver) (35/40)
22/02/08 16:52:19 INFO Executor: Finished task 35.0 in stage 15.0 (TID 263). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_36 locally
22/02/08 16:52:19 INFO TaskSetManager: Starting task 38.0 in stage 15.0 (TID 266) (192.168.1.37, executor driver, partition 38, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 38.0 in stage 15.0 (TID 266)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_37 locally
22/02/08 16:52:19 INFO TaskSetManager: Finished task 35.0 in stage 15.0 (TID 263) in 30 ms on 192.168.1.37 (executor driver) (36/40)
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_38 locally
22/02/08 16:52:19 INFO Executor: Finished task 36.0 in stage 15.0 (TID 264). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 39.0 in stage 15.0 (TID 267) (192.168.1.37, executor driver, partition 39, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 36.0 in stage 15.0 (TID 264) in 29 ms on 192.168.1.37 (executor driver) (37/40)
22/02/08 16:52:19 INFO Executor: Running task 39.0 in stage 15.0 (TID 267)
22/02/08 16:52:19 INFO Executor: Finished task 37.0 in stage 15.0 (TID 265). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Finished task 37.0 in stage 15.0 (TID 265) in 29 ms on 192.168.1.37 (executor driver) (38/40)
22/02/08 16:52:19 INFO Executor: Finished task 38.0 in stage 15.0 (TID 266). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO BlockManager: Found block rdd_27_39 locally
22/02/08 16:52:19 INFO TaskSetManager: Finished task 38.0 in stage 15.0 (TID 266) in 29 ms on 192.168.1.37 (executor driver) (39/40)
22/02/08 16:52:19 INFO Executor: Finished task 39.0 in stage 15.0 (TID 267). 1168 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Finished task 39.0 in stage 15.0 (TID 267) in 30 ms on 192.168.1.37 (executor driver) (40/40)
22/02/08 16:52:19 INFO TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool
22/02/08 16:52:19 INFO DAGScheduler: ShuffleMapStage 15 (distinct at SparkRejectUpdateStrategy.java:52) finished in 0.494 s
22/02/08 16:52:19 INFO DAGScheduler: looking for newly runnable stages
22/02/08 16:52:19 INFO DAGScheduler: running: Set()
22/02/08 16:52:19 INFO DAGScheduler: waiting: Set(ResultStage 16)
22/02/08 16:52:19 INFO DAGScheduler: failed: Set()
22/02/08 16:52:19 INFO DAGScheduler: Submitting ResultStage 16 (MapPartitionsRDD[35] at distinct at SparkRejectUpdateStrategy.java:52), which has no missing parents
22/02/08 16:52:19 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 6.5 KiB, free 956.7 MiB)
22/02/08 16:52:19 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 3.5 KiB, free 956.7 MiB)
22/02/08 16:52:19 INFO BlockManagerInfo: Added broadcast_13_piece0 in memory on 192.168.1.37:53051 (size: 3.5 KiB, free: 957.1 MiB)
22/02/08 16:52:19 INFO SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:19 INFO DAGScheduler: Submitting 40 missing tasks from ResultStage 16 (MapPartitionsRDD[35] at distinct at SparkRejectUpdateStrategy.java:52) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/08 16:52:19 INFO TaskSchedulerImpl: Adding task set 16.0 with 40 tasks resource profile 0
22/02/08 16:52:19 INFO TaskSetManager: Starting task 0.0 in stage 16.0 (TID 268) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Starting task 5.0 in stage 16.0 (TID 269) (192.168.1.37, executor driver, partition 5, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Starting task 8.0 in stage 16.0 (TID 270) (192.168.1.37, executor driver, partition 8, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 5.0 in stage 16.0 (TID 269)
22/02/08 16:52:19 INFO Executor: Running task 8.0 in stage 16.0 (TID 270)
22/02/08 16:52:19 INFO Executor: Running task 0.0 in stage 16.0 (TID 268)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (5.5 KiB) non-empty blocks including 40 (5.5 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (5.5 KiB) non-empty blocks including 40 (5.5 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (5.5 KiB) non-empty blocks including 40 (5.5 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 0.0 in stage 16.0 (TID 268). 1323 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 8.0 in stage 16.0 (TID 270). 1323 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 5.0 in stage 16.0 (TID 269). 1281 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 13.0 in stage 16.0 (TID 271) (192.168.1.37, executor driver, partition 13, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 13.0 in stage 16.0 (TID 271)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 14.0 in stage 16.0 (TID 272) (192.168.1.37, executor driver, partition 14, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 14.0 in stage 16.0 (TID 272)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 17.0 in stage 16.0 (TID 273) (192.168.1.37, executor driver, partition 17, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 17.0 in stage 16.0 (TID 273)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 8.0 in stage 16.0 (TID 270) in 11 ms on 192.168.1.37 (executor driver) (1/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (5.5 KiB) non-empty blocks including 40 (5.5 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (5.5 KiB) non-empty blocks including 40 (5.5 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (5.5 KiB) non-empty blocks including 40 (5.5 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Finished task 5.0 in stage 16.0 (TID 269) in 12 ms on 192.168.1.37 (executor driver) (2/40)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 0.0 in stage 16.0 (TID 268) in 12 ms on 192.168.1.37 (executor driver) (3/40)
22/02/08 16:52:19 INFO Executor: Finished task 13.0 in stage 16.0 (TID 271). 1281 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 20.0 in stage 16.0 (TID 274) (192.168.1.37, executor driver, partition 20, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Finished task 17.0 in stage 16.0 (TID 273). 1281 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Running task 20.0 in stage 16.0 (TID 274)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 13.0 in stage 16.0 (TID 271) in 8 ms on 192.168.1.37 (executor driver) (4/40)
22/02/08 16:52:19 INFO Executor: Finished task 14.0 in stage 16.0 (TID 272). 1281 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 21.0 in stage 16.0 (TID 275) (192.168.1.37, executor driver, partition 21, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 17.0 in stage 16.0 (TID 273) in 8 ms on 192.168.1.37 (executor driver) (5/40)
22/02/08 16:52:19 INFO Executor: Running task 21.0 in stage 16.0 (TID 275)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 24.0 in stage 16.0 (TID 276) (192.168.1.37, executor driver, partition 24, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 24.0 in stage 16.0 (TID 276)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (5.5 KiB) non-empty blocks including 40 (5.5 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (8.1 KiB) non-empty blocks including 40 (8.1 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 40 (5.5 KiB) non-empty blocks including 40 (5.5 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Finished task 14.0 in stage 16.0 (TID 272) in 10 ms on 192.168.1.37 (executor driver) (6/40)
22/02/08 16:52:19 INFO Executor: Finished task 24.0 in stage 16.0 (TID 276). 1281 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 1.0 in stage 16.0 (TID 277) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 1.0 in stage 16.0 (TID 277)
22/02/08 16:52:19 INFO Executor: Finished task 20.0 in stage 16.0 (TID 274). 1281 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Finished task 24.0 in stage 16.0 (TID 276) in 8 ms on 192.168.1.37 (executor driver) (7/40)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 2.0 in stage 16.0 (TID 278) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 2.0 in stage 16.0 (TID 278)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 20.0 in stage 16.0 (TID 274) in 10 ms on 192.168.1.37 (executor driver) (8/40)
22/02/08 16:52:19 INFO Executor: Finished task 21.0 in stage 16.0 (TID 275). 1333 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 3.0 in stage 16.0 (TID 279) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 3.0 in stage 16.0 (TID 279)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 1.0 in stage 16.0 (TID 277). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 2.0 in stage 16.0 (TID 278). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 4.0 in stage 16.0 (TID 280) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 4.0 in stage 16.0 (TID 280)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO TaskSetManager: Finished task 1.0 in stage 16.0 (TID 277) in 3 ms on 192.168.1.37 (executor driver) (9/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Starting task 6.0 in stage 16.0 (TID 281) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 21.0 in stage 16.0 (TID 275) in 12 ms on 192.168.1.37 (executor driver) (10/40)
22/02/08 16:52:19 INFO Executor: Running task 6.0 in stage 16.0 (TID 281)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 2.0 in stage 16.0 (TID 278) in 2 ms on 192.168.1.37 (executor driver) (11/40)
22/02/08 16:52:19 INFO Executor: Finished task 3.0 in stage 16.0 (TID 279). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 7.0 in stage 16.0 (TID 282) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 7.0 in stage 16.0 (TID 282)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 3.0 in stage 16.0 (TID 279) in 2 ms on 192.168.1.37 (executor driver) (12/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 4.0 in stage 16.0 (TID 280). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO TaskSetManager: Starting task 9.0 in stage 16.0 (TID 283) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 6.0 in stage 16.0 (TID 281). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Finished task 4.0 in stage 16.0 (TID 280) in 2 ms on 192.168.1.37 (executor driver) (13/40)
22/02/08 16:52:19 INFO Executor: Running task 9.0 in stage 16.0 (TID 283)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 10.0 in stage 16.0 (TID 284) (192.168.1.37, executor driver, partition 10, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Finished task 7.0 in stage 16.0 (TID 282). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Finished task 6.0 in stage 16.0 (TID 281) in 3 ms on 192.168.1.37 (executor driver) (14/40)
22/02/08 16:52:19 INFO Executor: Running task 10.0 in stage 16.0 (TID 284)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 11.0 in stage 16.0 (TID 285) (192.168.1.37, executor driver, partition 11, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 7.0 in stage 16.0 (TID 282) in 2 ms on 192.168.1.37 (executor driver) (15/40)
22/02/08 16:52:19 INFO Executor: Running task 11.0 in stage 16.0 (TID 285)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 9.0 in stage 16.0 (TID 283). 1183 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 10.0 in stage 16.0 (TID 284). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 11.0 in stage 16.0 (TID 285). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 12.0 in stage 16.0 (TID 286) (192.168.1.37, executor driver, partition 12, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 9.0 in stage 16.0 (TID 283) in 3 ms on 192.168.1.37 (executor driver) (16/40)
22/02/08 16:52:19 INFO Executor: Running task 12.0 in stage 16.0 (TID 286)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 15.0 in stage 16.0 (TID 287) (192.168.1.37, executor driver, partition 15, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 10.0 in stage 16.0 (TID 284) in 3 ms on 192.168.1.37 (executor driver) (17/40)
22/02/08 16:52:19 INFO Executor: Running task 15.0 in stage 16.0 (TID 287)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 16.0 in stage 16.0 (TID 288) (192.168.1.37, executor driver, partition 16, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 11.0 in stage 16.0 (TID 285) in 2 ms on 192.168.1.37 (executor driver) (18/40)
22/02/08 16:52:19 INFO Executor: Running task 16.0 in stage 16.0 (TID 288)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 12.0 in stage 16.0 (TID 286). 1183 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 16.0 in stage 16.0 (TID 288). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 15.0 in stage 16.0 (TID 287). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 18.0 in stage 16.0 (TID 289) (192.168.1.37, executor driver, partition 18, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 12.0 in stage 16.0 (TID 286) in 3 ms on 192.168.1.37 (executor driver) (19/40)
22/02/08 16:52:19 INFO Executor: Running task 18.0 in stage 16.0 (TID 289)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 19.0 in stage 16.0 (TID 290) (192.168.1.37, executor driver, partition 19, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 16.0 in stage 16.0 (TID 288) in 3 ms on 192.168.1.37 (executor driver) (20/40)
22/02/08 16:52:19 INFO Executor: Running task 19.0 in stage 16.0 (TID 290)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 22.0 in stage 16.0 (TID 291) (192.168.1.37, executor driver, partition 22, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 15.0 in stage 16.0 (TID 287) in 3 ms on 192.168.1.37 (executor driver) (21/40)
22/02/08 16:52:19 INFO Executor: Running task 22.0 in stage 16.0 (TID 291)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 18.0 in stage 16.0 (TID 289). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 22.0 in stage 16.0 (TID 291). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 19.0 in stage 16.0 (TID 290). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 23.0 in stage 16.0 (TID 292) (192.168.1.37, executor driver, partition 23, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 23.0 in stage 16.0 (TID 292)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 18.0 in stage 16.0 (TID 289) in 3 ms on 192.168.1.37 (executor driver) (22/40)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 25.0 in stage 16.0 (TID 293) (192.168.1.37, executor driver, partition 25, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 22.0 in stage 16.0 (TID 291) in 2 ms on 192.168.1.37 (executor driver) (23/40)
22/02/08 16:52:19 INFO Executor: Running task 25.0 in stage 16.0 (TID 293)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 26.0 in stage 16.0 (TID 294) (192.168.1.37, executor driver, partition 26, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 26.0 in stage 16.0 (TID 294)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 19.0 in stage 16.0 (TID 290) in 3 ms on 192.168.1.37 (executor driver) (24/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO Executor: Finished task 23.0 in stage 16.0 (TID 292). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Starting task 27.0 in stage 16.0 (TID 295) (192.168.1.37, executor driver, partition 27, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Finished task 23.0 in stage 16.0 (TID 292) in 2 ms on 192.168.1.37 (executor driver) (25/40)
22/02/08 16:52:19 INFO Executor: Finished task 25.0 in stage 16.0 (TID 293). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Running task 27.0 in stage 16.0 (TID 295)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 28.0 in stage 16.0 (TID 296) (192.168.1.37, executor driver, partition 28, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Finished task 26.0 in stage 16.0 (TID 294). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Running task 28.0 in stage 16.0 (TID 296)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 29.0 in stage 16.0 (TID 297) (192.168.1.37, executor driver, partition 29, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 29.0 in stage 16.0 (TID 297)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 25.0 in stage 16.0 (TID 293) in 2 ms on 192.168.1.37 (executor driver) (26/40)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 26.0 in stage 16.0 (TID 294) in 2 ms on 192.168.1.37 (executor driver) (27/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 27.0 in stage 16.0 (TID 295). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO TaskSetManager: Starting task 30.0 in stage 16.0 (TID 298) (192.168.1.37, executor driver, partition 30, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Finished task 28.0 in stage 16.0 (TID 296). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Finished task 27.0 in stage 16.0 (TID 295) in 2 ms on 192.168.1.37 (executor driver) (28/40)
22/02/08 16:52:19 INFO Executor: Running task 30.0 in stage 16.0 (TID 298)
22/02/08 16:52:19 INFO Executor: Finished task 29.0 in stage 16.0 (TID 297). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 31.0 in stage 16.0 (TID 299) (192.168.1.37, executor driver, partition 31, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 28.0 in stage 16.0 (TID 296) in 2 ms on 192.168.1.37 (executor driver) (29/40)
22/02/08 16:52:19 INFO Executor: Running task 31.0 in stage 16.0 (TID 299)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 32.0 in stage 16.0 (TID 300) (192.168.1.37, executor driver, partition 32, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 32.0 in stage 16.0 (TID 300)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 29.0 in stage 16.0 (TID 297) in 2 ms on 192.168.1.37 (executor driver) (30/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 30.0 in stage 16.0 (TID 298). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 31.0 in stage 16.0 (TID 299). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 32.0 in stage 16.0 (TID 300). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 33.0 in stage 16.0 (TID 301) (192.168.1.37, executor driver, partition 33, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 30.0 in stage 16.0 (TID 298) in 3 ms on 192.168.1.37 (executor driver) (31/40)
22/02/08 16:52:19 INFO Executor: Running task 33.0 in stage 16.0 (TID 301)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 34.0 in stage 16.0 (TID 302) (192.168.1.37, executor driver, partition 34, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 34.0 in stage 16.0 (TID 302)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 35.0 in stage 16.0 (TID 303) (192.168.1.37, executor driver, partition 35, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 31.0 in stage 16.0 (TID 299) in 3 ms on 192.168.1.37 (executor driver) (32/40)
22/02/08 16:52:19 INFO Executor: Running task 35.0 in stage 16.0 (TID 303)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 32.0 in stage 16.0 (TID 300) in 2 ms on 192.168.1.37 (executor driver) (33/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 33.0 in stage 16.0 (TID 301). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 35.0 in stage 16.0 (TID 303). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 34.0 in stage 16.0 (TID 302). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 36.0 in stage 16.0 (TID 304) (192.168.1.37, executor driver, partition 36, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 33.0 in stage 16.0 (TID 301) in 3 ms on 192.168.1.37 (executor driver) (34/40)
22/02/08 16:52:19 INFO Executor: Running task 36.0 in stage 16.0 (TID 304)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 37.0 in stage 16.0 (TID 305) (192.168.1.37, executor driver, partition 37, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 37.0 in stage 16.0 (TID 305)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 35.0 in stage 16.0 (TID 303) in 2 ms on 192.168.1.37 (executor driver) (35/40)
22/02/08 16:52:19 INFO TaskSetManager: Starting task 38.0 in stage 16.0 (TID 306) (192.168.1.37, executor driver, partition 38, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Finished task 34.0 in stage 16.0 (TID 302) in 3 ms on 192.168.1.37 (executor driver) (36/40)
22/02/08 16:52:19 INFO Executor: Running task 38.0 in stage 16.0 (TID 306)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 37.0 in stage 16.0 (TID 305). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 36.0 in stage 16.0 (TID 304). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO Executor: Finished task 38.0 in stage 16.0 (TID 306). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Starting task 39.0 in stage 16.0 (TID 307) (192.168.1.37, executor driver, partition 39, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 39.0 in stage 16.0 (TID 307)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 37.0 in stage 16.0 (TID 305) in 3 ms on 192.168.1.37 (executor driver) (37/40)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 36.0 in stage 16.0 (TID 304) in 3 ms on 192.168.1.37 (executor driver) (38/40)
22/02/08 16:52:19 INFO TaskSetManager: Finished task 38.0 in stage 16.0 (TID 306) in 2 ms on 192.168.1.37 (executor driver) (39/40)
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:19 INFO Executor: Finished task 39.0 in stage 16.0 (TID 307). 1140 bytes result sent to driver
22/02/08 16:52:19 INFO TaskSetManager: Finished task 39.0 in stage 16.0 (TID 307) in 3 ms on 192.168.1.37 (executor driver) (40/40)
22/02/08 16:52:19 INFO TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool
22/02/08 16:52:19 INFO DAGScheduler: ResultStage 16 (collect at SparkRejectUpdateStrategy.java:52) finished in 0.056 s
22/02/08 16:52:19 INFO DAGScheduler: Job 6 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:52:19 INFO TaskSchedulerImpl: Killing all running tasks in stage 16: Stage finished
22/02/08 16:52:19 INFO DAGScheduler: Job 6 finished: collect at SparkRejectUpdateStrategy.java:52, took 0.554702 s
22/02/08 16:52:19 INFO UpsertPartitioner: AvgRecordSize => 1024
22/02/08 16:52:19 INFO SparkContext: Starting job: collectAsMap at UpsertPartitioner.java:248
22/02/08 16:52:19 INFO DAGScheduler: Got job 7 (collectAsMap at UpsertPartitioner.java:248) with 10 output partitions
22/02/08 16:52:19 INFO DAGScheduler: Final stage: ResultStage 17 (collectAsMap at UpsertPartitioner.java:248)
22/02/08 16:52:19 INFO DAGScheduler: Parents of final stage: List()
22/02/08 16:52:19 INFO DAGScheduler: Missing parents: List()
22/02/08 16:52:19 INFO DAGScheduler: Submitting ResultStage 17 (MapPartitionsRDD[37] at mapToPair at UpsertPartitioner.java:247), which has no missing parents
22/02/08 16:52:19 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 514.0 KiB, free 956.2 MiB)
22/02/08 16:52:19 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 179.4 KiB, free 956.1 MiB)
22/02/08 16:52:19 INFO BlockManagerInfo: Added broadcast_14_piece0 in memory on 192.168.1.37:53051 (size: 179.4 KiB, free: 956.9 MiB)
22/02/08 16:52:19 INFO SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:19 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 17 (MapPartitionsRDD[37] at mapToPair at UpsertPartitioner.java:247) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/08 16:52:19 INFO TaskSchedulerImpl: Adding task set 17.0 with 10 tasks resource profile 0
22/02/08 16:52:19 INFO TaskSetManager: Starting task 0.0 in stage 17.0 (TID 308) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Starting task 1.0 in stage 17.0 (TID 309) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO TaskSetManager: Starting task 2.0 in stage 17.0 (TID 310) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:19 INFO Executor: Running task 0.0 in stage 17.0 (TID 308)
22/02/08 16:52:19 INFO Executor: Running task 1.0 in stage 17.0 (TID 309)
22/02/08 16:52:19 INFO Executor: Running task 2.0 in stage 17.0 (TID 310)
22/02/08 16:52:20 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:20 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:20 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:20 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:20 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:20 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/slices/beforeoron/latest/?partition=2022%2F1%2F31&maxinstant=20220208164721920&includependingcompaction=false&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:20 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/slices/beforeoron/latest/?partition=2022%2F1%2F30&maxinstant=20220208164721920&includependingcompaction=false&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:20 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/slices/beforeoron/latest/?partition=2022%2F1%2F28&maxinstant=20220208164721920&includependingcompaction=false&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:20 INFO Executor: Finished task 2.0 in stage 17.0 (TID 310). 994 bytes result sent to driver
22/02/08 16:52:20 INFO Executor: Finished task 1.0 in stage 17.0 (TID 309). 994 bytes result sent to driver
22/02/08 16:52:20 INFO Executor: Finished task 0.0 in stage 17.0 (TID 308). 994 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 3.0 in stage 17.0 (TID 311) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO Executor: Running task 3.0 in stage 17.0 (TID 311)
22/02/08 16:52:20 INFO TaskSetManager: Starting task 4.0 in stage 17.0 (TID 312) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4341 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO Executor: Running task 4.0 in stage 17.0 (TID 312)
22/02/08 16:52:20 INFO TaskSetManager: Starting task 5.0 in stage 17.0 (TID 313) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO Executor: Running task 5.0 in stage 17.0 (TID 313)
22/02/08 16:52:20 INFO TaskSetManager: Finished task 1.0 in stage 17.0 (TID 309) in 44 ms on 192.168.1.37 (executor driver) (1/10)
22/02/08 16:52:20 INFO TaskSetManager: Finished task 2.0 in stage 17.0 (TID 310) in 44 ms on 192.168.1.37 (executor driver) (2/10)
22/02/08 16:52:20 INFO TaskSetManager: Finished task 0.0 in stage 17.0 (TID 308) in 44 ms on 192.168.1.37 (executor driver) (3/10)
22/02/08 16:52:20 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:20 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:20 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:20 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:20 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:20 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 1 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/slices/beforeoron/latest/?partition=2022%2F2%2F2&maxinstant=20220208164721920&includependingcompaction=false&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:20 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/slices/beforeoron/latest/?partition=2022%2F1%2F29&maxinstant=20220208164721920&includependingcompaction=false&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:20 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/slices/beforeoron/latest/?partition=2022%2F1%2F27&maxinstant=20220208164721920&includependingcompaction=false&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:20 INFO Executor: Finished task 5.0 in stage 17.0 (TID 313). 994 bytes result sent to driver
22/02/08 16:52:20 INFO Executor: Finished task 4.0 in stage 17.0 (TID 312). 993 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 6.0 in stage 17.0 (TID 314) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO Executor: Finished task 3.0 in stage 17.0 (TID 311). 994 bytes result sent to driver
22/02/08 16:52:20 INFO Executor: Running task 6.0 in stage 17.0 (TID 314)
22/02/08 16:52:20 INFO TaskSetManager: Finished task 5.0 in stage 17.0 (TID 313) in 29 ms on 192.168.1.37 (executor driver) (4/10)
22/02/08 16:52:20 INFO TaskSetManager: Starting task 7.0 in stage 17.0 (TID 315) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4341 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Finished task 4.0 in stage 17.0 (TID 312) in 30 ms on 192.168.1.37 (executor driver) (5/10)
22/02/08 16:52:20 INFO Executor: Running task 7.0 in stage 17.0 (TID 315)
22/02/08 16:52:20 INFO TaskSetManager: Starting task 8.0 in stage 17.0 (TID 316) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Finished task 3.0 in stage 17.0 (TID 311) in 30 ms on 192.168.1.37 (executor driver) (6/10)
22/02/08 16:52:20 INFO Executor: Running task 8.0 in stage 17.0 (TID 316)
22/02/08 16:52:20 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:20 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:20 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:20 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:20 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/slices/beforeoron/latest/?partition=2022%2F2%2F1&maxinstant=20220208164721920&includependingcompaction=false&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:20 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/slices/beforeoron/latest/?partition=2022%2F1%2F24&maxinstant=20220208164721920&includependingcompaction=false&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/slices/beforeoron/latest/?partition=2022%2F1%2F26&maxinstant=20220208164721920&includependingcompaction=false&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:20 INFO Executor: Finished task 8.0 in stage 17.0 (TID 316). 994 bytes result sent to driver
22/02/08 16:52:20 INFO Executor: Finished task 7.0 in stage 17.0 (TID 315). 993 bytes result sent to driver
22/02/08 16:52:20 INFO Executor: Finished task 6.0 in stage 17.0 (TID 314). 994 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 9.0 in stage 17.0 (TID 317) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4342 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO Executor: Running task 9.0 in stage 17.0 (TID 317)
22/02/08 16:52:20 INFO TaskSetManager: Finished task 8.0 in stage 17.0 (TID 316) in 32 ms on 192.168.1.37 (executor driver) (7/10)
22/02/08 16:52:20 INFO TaskSetManager: Finished task 7.0 in stage 17.0 (TID 315) in 32 ms on 192.168.1.37 (executor driver) (8/10)
22/02/08 16:52:20 INFO TaskSetManager: Finished task 6.0 in stage 17.0 (TID 314) in 33 ms on 192.168.1.37 (executor driver) (9/10)
22/02/08 16:52:20 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:20 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:20 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/slices/beforeoron/latest/?partition=2022%2F1%2F25&maxinstant=20220208164721920&includependingcompaction=false&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:20 INFO Executor: Finished task 9.0 in stage 17.0 (TID 317). 994 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Finished task 9.0 in stage 17.0 (TID 317) in 26 ms on 192.168.1.37 (executor driver) (10/10)
22/02/08 16:52:20 INFO TaskSchedulerImpl: Removed TaskSet 17.0, whose tasks have all completed, from pool
22/02/08 16:52:20 INFO DAGScheduler: ResultStage 17 (collectAsMap at UpsertPartitioner.java:248) finished in 0.205 s
22/02/08 16:52:20 INFO DAGScheduler: Job 7 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:52:20 INFO TaskSchedulerImpl: Killing all running tasks in stage 17: Stage finished
22/02/08 16:52:20 INFO DAGScheduler: Job 7 finished: collectAsMap at UpsertPartitioner.java:248, took 0.205484 s
22/02/08 16:52:20 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:20 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:20 INFO UpsertPartitioner: For partitionPath : 2022/1/31 Small Files => [SmallFile {location=HoodieRecordLocation {instantTime=20220208164721920, fileId=6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0}, sizeBytes=56575485}]
22/02/08 16:52:20 INFO UpsertPartitioner: Assigning 34955 inserts to existing update bucket 0
22/02/08 16:52:20 INFO UpsertPartitioner: Total insert buckets for partition path 2022/1/31 => [(InsertBucket {bucketNumber=0, weight=1.0},1.0)]
22/02/08 16:52:20 INFO UpsertPartitioner: For partitionPath : 2022/1/30 Small Files => [SmallFile {location=HoodieRecordLocation {instantTime=20220208164721920, fileId=272a5505-e183-4c3f-98a2-5d92f194272c-0}, sizeBytes=56184571}]
22/02/08 16:52:20 INFO UpsertPartitioner: Assigning 34690 inserts to existing update bucket 1
22/02/08 16:52:20 INFO UpsertPartitioner: Total insert buckets for partition path 2022/1/30 => [(InsertBucket {bucketNumber=1, weight=1.0},1.0)]
22/02/08 16:52:20 INFO UpsertPartitioner: For partitionPath : 2022/1/28 Small Files => [SmallFile {location=HoodieRecordLocation {instantTime=20220208164721920, fileId=ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0}, sizeBytes=56498758}]
22/02/08 16:52:20 INFO UpsertPartitioner: Assigning 34990 inserts to existing update bucket 2
22/02/08 16:52:20 INFO UpsertPartitioner: Total insert buckets for partition path 2022/1/28 => [(InsertBucket {bucketNumber=2, weight=1.0},1.0)]
22/02/08 16:52:20 INFO UpsertPartitioner: For partitionPath : 2022/1/27 Small Files => [SmallFile {location=HoodieRecordLocation {instantTime=20220208164721920, fileId=01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0}, sizeBytes=56934604}]
22/02/08 16:52:20 INFO UpsertPartitioner: Assigning 35245 inserts to existing update bucket 3
22/02/08 16:52:20 INFO UpsertPartitioner: Total insert buckets for partition path 2022/1/27 => [(InsertBucket {bucketNumber=3, weight=1.0},1.0)]
22/02/08 16:52:20 INFO UpsertPartitioner: For partitionPath : 2022/2/2 Small Files => [SmallFile {location=HoodieRecordLocation {instantTime=20220208164721920, fileId=bfb4a6e0-cd86-481f-8781-f763c3c88016-0}, sizeBytes=56577939}]
22/02/08 16:52:20 INFO UpsertPartitioner: Assigning 34880 inserts to existing update bucket 4
22/02/08 16:52:20 INFO UpsertPartitioner: Total insert buckets for partition path 2022/2/2 => [(InsertBucket {bucketNumber=4, weight=1.0},1.0)]
22/02/08 16:52:20 INFO UpsertPartitioner: For partitionPath : 2022/1/29 Small Files => [SmallFile {location=HoodieRecordLocation {instantTime=20220208164721920, fileId=cbbad821-34e6-47c7-9f8b-f41da613023b-0}, sizeBytes=55946106}]
22/02/08 16:52:20 INFO UpsertPartitioner: Assigning 34490 inserts to existing update bucket 5
22/02/08 16:52:20 INFO UpsertPartitioner: Total insert buckets for partition path 2022/1/29 => [(InsertBucket {bucketNumber=5, weight=1.0},1.0)]
22/02/08 16:52:20 INFO UpsertPartitioner: For partitionPath : 2022/1/24 Small Files => [SmallFile {location=HoodieRecordLocation {instantTime=20220208164721920, fileId=b2aa5767-67fa-407b-ab8d-0c41e3f61983-0}, sizeBytes=56587361}]
22/02/08 16:52:20 INFO UpsertPartitioner: Assigning 35065 inserts to existing update bucket 6
22/02/08 16:52:20 INFO UpsertPartitioner: Total insert buckets for partition path 2022/1/24 => [(InsertBucket {bucketNumber=6, weight=1.0},1.0)]
22/02/08 16:52:20 INFO UpsertPartitioner: For partitionPath : 2022/2/1 Small Files => [SmallFile {location=HoodieRecordLocation {instantTime=20220208164721920, fileId=bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0}, sizeBytes=56422994}]
22/02/08 16:52:20 INFO UpsertPartitioner: Assigning 34885 inserts to existing update bucket 7
22/02/08 16:52:20 INFO UpsertPartitioner: Total insert buckets for partition path 2022/2/1 => [(InsertBucket {bucketNumber=7, weight=1.0},1.0)]
22/02/08 16:52:20 INFO UpsertPartitioner: For partitionPath : 2022/1/26 Small Files => [SmallFile {location=HoodieRecordLocation {instantTime=20220208164721920, fileId=b7054c05-f9cd-4f0b-95b8-1492803c6aff-0}, sizeBytes=57589297}]
22/02/08 16:52:20 INFO UpsertPartitioner: Assigning 35565 inserts to existing update bucket 8
22/02/08 16:52:20 INFO UpsertPartitioner: Total insert buckets for partition path 2022/1/26 => [(InsertBucket {bucketNumber=8, weight=1.0},1.0)]
22/02/08 16:52:20 INFO UpsertPartitioner: For partitionPath : 2022/1/25 Small Files => [SmallFile {location=HoodieRecordLocation {instantTime=20220208164721920, fileId=0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0}, sizeBytes=56793878}]
22/02/08 16:52:20 INFO UpsertPartitioner: Assigning 35240 inserts to existing update bucket 9
22/02/08 16:52:20 INFO UpsertPartitioner: Total insert buckets for partition path 2022/1/25 => [(InsertBucket {bucketNumber=9, weight=1.0},1.0)]
22/02/08 16:52:20 INFO UpsertPartitioner: Total Buckets :10, buckets info => {0=BucketInfo {bucketType=UPDATE, fileIdPrefix=6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0, partitionPath=2022/1/31}, 1=BucketInfo {bucketType=UPDATE, fileIdPrefix=272a5505-e183-4c3f-98a2-5d92f194272c-0, partitionPath=2022/1/30}, 2=BucketInfo {bucketType=UPDATE, fileIdPrefix=ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0, partitionPath=2022/1/28}, 3=BucketInfo {bucketType=UPDATE, fileIdPrefix=01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0, partitionPath=2022/1/27}, 4=BucketInfo {bucketType=UPDATE, fileIdPrefix=bfb4a6e0-cd86-481f-8781-f763c3c88016-0, partitionPath=2022/2/2}, 5=BucketInfo {bucketType=UPDATE, fileIdPrefix=cbbad821-34e6-47c7-9f8b-f41da613023b-0, partitionPath=2022/1/29}, 6=BucketInfo {bucketType=UPDATE, fileIdPrefix=b2aa5767-67fa-407b-ab8d-0c41e3f61983-0, partitionPath=2022/1/24}, 7=BucketInfo {bucketType=UPDATE, fileIdPrefix=bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0, partitionPath=2022/2/1}, 8=BucketInfo {bucketType=UPDATE, fileIdPrefix=b7054c05-f9cd-4f0b-95b8-1492803c6aff-0, partitionPath=2022/1/26}, 9=BucketInfo {bucketType=UPDATE, fileIdPrefix=0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0, partitionPath=2022/1/25}},
Partition to insert buckets => {2022/1/31=[(InsertBucket {bucketNumber=0, weight=1.0},1.0)], 2022/1/30=[(InsertBucket {bucketNumber=1, weight=1.0},1.0)], 2022/1/28=[(InsertBucket {bucketNumber=2, weight=1.0},1.0)], 2022/1/27=[(InsertBucket {bucketNumber=3, weight=1.0},1.0)], 2022/2/2=[(InsertBucket {bucketNumber=4, weight=1.0},1.0)], 2022/1/29=[(InsertBucket {bucketNumber=5, weight=1.0},1.0)], 2022/1/24=[(InsertBucket {bucketNumber=6, weight=1.0},1.0)], 2022/2/1=[(InsertBucket {bucketNumber=7, weight=1.0},1.0)], 2022/1/26=[(InsertBucket {bucketNumber=8, weight=1.0},1.0)], 2022/1/25=[(InsertBucket {bucketNumber=9, weight=1.0},1.0)]},
UpdateLocations mapped to buckets =>{ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0=2, bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0=7, b7054c05-f9cd-4f0b-95b8-1492803c6aff-0=8, 0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0=9, 01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0=3, 6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0=0, 272a5505-e183-4c3f-98a2-5d92f194272c-0=1, bfb4a6e0-cd86-481f-8781-f763c3c88016-0=4, b2aa5767-67fa-407b-ab8d-0c41e3f61983-0=6, cbbad821-34e6-47c7-9f8b-f41da613023b-0=5}
22/02/08 16:52:20 INFO BaseSparkCommitActionExecutor: no validators configured.
22/02/08 16:52:20 INFO BaseCommitActionExecutor: Auto commit disabled for 20220208165137552
22/02/08 16:52:20 INFO SparkContext: Starting job: sum at DeltaSync.java:592
22/02/08 16:52:20 INFO DAGScheduler: Registering RDD 38 (mapToPair at BaseSparkCommitActionExecutor.java:233) as input to shuffle 6
22/02/08 16:52:20 INFO DAGScheduler: Got job 8 (sum at DeltaSync.java:592) with 10 output partitions
22/02/08 16:52:20 INFO DAGScheduler: Final stage: ResultStage 22 (sum at DeltaSync.java:592)
22/02/08 16:52:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 21)
22/02/08 16:52:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 21)
22/02/08 16:52:20 INFO DAGScheduler: Submitting ShuffleMapStage 21 (MapPartitionsRDD[38] at mapToPair at BaseSparkCommitActionExecutor.java:233), which has no missing parents
22/02/08 16:52:20 INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated size 520.0 KiB, free 955.6 MiB)
22/02/08 16:52:20 INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 185.6 KiB, free 955.4 MiB)
22/02/08 16:52:20 INFO BlockManagerInfo: Added broadcast_15_piece0 in memory on 192.168.1.37:53051 (size: 185.6 KiB, free: 956.8 MiB)
22/02/08 16:52:20 INFO SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:20 INFO DAGScheduler: Submitting 40 missing tasks from ShuffleMapStage 21 (MapPartitionsRDD[38] at mapToPair at BaseSparkCommitActionExecutor.java:233) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/02/08 16:52:20 INFO TaskSchedulerImpl: Adding task set 21.0 with 40 tasks resource profile 0
22/02/08 16:52:20 INFO TaskSetManager: Starting task 0.0 in stage 21.0 (TID 318) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Starting task 1.0 in stage 21.0 (TID 319) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Starting task 2.0 in stage 21.0 (TID 320) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO Executor: Running task 1.0 in stage 21.0 (TID 319)
22/02/08 16:52:20 INFO Executor: Running task 2.0 in stage 21.0 (TID 320)
22/02/08 16:52:20 INFO Executor: Running task 0.0 in stage 21.0 (TID 318)
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_1 locally
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_2 locally
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_0 locally
22/02/08 16:52:20 INFO BlockManagerInfo: Removed broadcast_13_piece0 on 192.168.1.37:53051 in memory (size: 3.5 KiB, free: 956.8 MiB)
22/02/08 16:52:20 INFO BlockManagerInfo: Removed broadcast_14_piece0 on 192.168.1.37:53051 in memory (size: 179.4 KiB, free: 956.9 MiB)
22/02/08 16:52:20 INFO Executor: Finished task 0.0 in stage 21.0 (TID 318). 1095 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 3.0 in stage 21.0 (TID 321) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Finished task 0.0 in stage 21.0 (TID 318) in 287 ms on 192.168.1.37 (executor driver) (1/40)
22/02/08 16:52:20 INFO Executor: Running task 3.0 in stage 21.0 (TID 321)
22/02/08 16:52:20 INFO Executor: Finished task 1.0 in stage 21.0 (TID 319). 1095 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 4.0 in stage 21.0 (TID 322) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Finished task 1.0 in stage 21.0 (TID 319) in 292 ms on 192.168.1.37 (executor driver) (2/40)
22/02/08 16:52:20 INFO Executor: Running task 4.0 in stage 21.0 (TID 322)
22/02/08 16:52:20 INFO Executor: Finished task 2.0 in stage 21.0 (TID 320). 1095 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 5.0 in stage 21.0 (TID 323) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Finished task 2.0 in stage 21.0 (TID 320) in 295 ms on 192.168.1.37 (executor driver) (3/40)
22/02/08 16:52:20 INFO Executor: Running task 5.0 in stage 21.0 (TID 323)
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_3 locally
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_4 locally
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_5 locally
22/02/08 16:52:20 INFO Executor: Finished task 3.0 in stage 21.0 (TID 321). 1052 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 6.0 in stage 21.0 (TID 324) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Finished task 3.0 in stage 21.0 (TID 321) in 188 ms on 192.168.1.37 (executor driver) (4/40)
22/02/08 16:52:20 INFO Executor: Running task 6.0 in stage 21.0 (TID 324)
22/02/08 16:52:20 INFO Executor: Finished task 4.0 in stage 21.0 (TID 322). 1052 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 7.0 in stage 21.0 (TID 325) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Finished task 4.0 in stage 21.0 (TID 322) in 188 ms on 192.168.1.37 (executor driver) (5/40)
22/02/08 16:52:20 INFO Executor: Running task 7.0 in stage 21.0 (TID 325)
22/02/08 16:52:20 INFO Executor: Finished task 5.0 in stage 21.0 (TID 323). 1052 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 8.0 in stage 21.0 (TID 326) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Finished task 5.0 in stage 21.0 (TID 323) in 186 ms on 192.168.1.37 (executor driver) (6/40)
22/02/08 16:52:20 INFO Executor: Running task 8.0 in stage 21.0 (TID 326)
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_6 locally
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_7 locally
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_8 locally
22/02/08 16:52:20 INFO Executor: Finished task 6.0 in stage 21.0 (TID 324). 1052 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 9.0 in stage 21.0 (TID 327) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO Executor: Running task 9.0 in stage 21.0 (TID 327)
22/02/08 16:52:20 INFO TaskSetManager: Finished task 6.0 in stage 21.0 (TID 324) in 177 ms on 192.168.1.37 (executor driver) (7/40)
22/02/08 16:52:20 INFO Executor: Finished task 7.0 in stage 21.0 (TID 325). 1052 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 10.0 in stage 21.0 (TID 328) (192.168.1.37, executor driver, partition 10, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO Executor: Running task 10.0 in stage 21.0 (TID 328)
22/02/08 16:52:20 INFO TaskSetManager: Finished task 7.0 in stage 21.0 (TID 325) in 179 ms on 192.168.1.37 (executor driver) (8/40)
22/02/08 16:52:20 INFO Executor: Finished task 8.0 in stage 21.0 (TID 326). 1052 bytes result sent to driver
22/02/08 16:52:20 INFO TaskSetManager: Starting task 11.0 in stage 21.0 (TID 329) (192.168.1.37, executor driver, partition 11, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:20 INFO TaskSetManager: Finished task 8.0 in stage 21.0 (TID 326) in 181 ms on 192.168.1.37 (executor driver) (9/40)
22/02/08 16:52:20 INFO Executor: Running task 11.0 in stage 21.0 (TID 329)
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_9 locally
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_10 locally
22/02/08 16:52:20 INFO BlockManager: Found block rdd_27_11 locally
22/02/08 16:52:21 INFO BlockManagerInfo: Removed broadcast_12_piece0 on 192.168.1.37:53051 in memory (size: 4.9 KiB, free: 956.9 MiB)
22/02/08 16:52:21 INFO BlockManagerInfo: Removed broadcast_10_piece0 on 192.168.1.37:53051 in memory (size: 5.1 KiB, free: 956.9 MiB)
22/02/08 16:52:21 INFO Executor: Finished task 9.0 in stage 21.0 (TID 327). 1095 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 12.0 in stage 21.0 (TID 330) (192.168.1.37, executor driver, partition 12, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO Executor: Running task 12.0 in stage 21.0 (TID 330)
22/02/08 16:52:21 INFO TaskSetManager: Finished task 9.0 in stage 21.0 (TID 327) in 350 ms on 192.168.1.37 (executor driver) (10/40)
22/02/08 16:52:21 INFO Executor: Finished task 10.0 in stage 21.0 (TID 328). 1095 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 13.0 in stage 21.0 (TID 331) (192.168.1.37, executor driver, partition 13, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO Executor: Running task 13.0 in stage 21.0 (TID 331)
22/02/08 16:52:21 INFO TaskSetManager: Finished task 10.0 in stage 21.0 (TID 328) in 348 ms on 192.168.1.37 (executor driver) (11/40)
22/02/08 16:52:21 INFO Executor: Finished task 11.0 in stage 21.0 (TID 329). 1095 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 14.0 in stage 21.0 (TID 332) (192.168.1.37, executor driver, partition 14, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO Executor: Running task 14.0 in stage 21.0 (TID 332)
22/02/08 16:52:21 INFO TaskSetManager: Finished task 11.0 in stage 21.0 (TID 329) in 352 ms on 192.168.1.37 (executor driver) (12/40)
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_12 locally
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_13 locally
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_14 locally
22/02/08 16:52:21 INFO Executor: Finished task 12.0 in stage 21.0 (TID 330). 1052 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 15.0 in stage 21.0 (TID 333) (192.168.1.37, executor driver, partition 15, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO TaskSetManager: Finished task 12.0 in stage 21.0 (TID 330) in 185 ms on 192.168.1.37 (executor driver) (13/40)
22/02/08 16:52:21 INFO Executor: Running task 15.0 in stage 21.0 (TID 333)
22/02/08 16:52:21 INFO Executor: Finished task 13.0 in stage 21.0 (TID 331). 1052 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 16.0 in stage 21.0 (TID 334) (192.168.1.37, executor driver, partition 16, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO Executor: Running task 16.0 in stage 21.0 (TID 334)
22/02/08 16:52:21 INFO TaskSetManager: Finished task 13.0 in stage 21.0 (TID 331) in 185 ms on 192.168.1.37 (executor driver) (14/40)
22/02/08 16:52:21 INFO Executor: Finished task 14.0 in stage 21.0 (TID 332). 1052 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 17.0 in stage 21.0 (TID 335) (192.168.1.37, executor driver, partition 17, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO TaskSetManager: Finished task 14.0 in stage 21.0 (TID 332) in 184 ms on 192.168.1.37 (executor driver) (15/40)
22/02/08 16:52:21 INFO Executor: Running task 17.0 in stage 21.0 (TID 335)
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_15 locally
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_16 locally
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_17 locally
22/02/08 16:52:21 INFO Executor: Finished task 15.0 in stage 21.0 (TID 333). 1052 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 18.0 in stage 21.0 (TID 336) (192.168.1.37, executor driver, partition 18, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO Executor: Running task 18.0 in stage 21.0 (TID 336)
22/02/08 16:52:21 INFO TaskSetManager: Finished task 15.0 in stage 21.0 (TID 333) in 185 ms on 192.168.1.37 (executor driver) (16/40)
22/02/08 16:52:21 INFO Executor: Finished task 16.0 in stage 21.0 (TID 334). 1052 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 19.0 in stage 21.0 (TID 337) (192.168.1.37, executor driver, partition 19, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO TaskSetManager: Finished task 16.0 in stage 21.0 (TID 334) in 188 ms on 192.168.1.37 (executor driver) (17/40)
22/02/08 16:52:21 INFO Executor: Running task 19.0 in stage 21.0 (TID 337)
22/02/08 16:52:21 INFO Executor: Finished task 17.0 in stage 21.0 (TID 335). 1052 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 20.0 in stage 21.0 (TID 338) (192.168.1.37, executor driver, partition 20, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO TaskSetManager: Finished task 17.0 in stage 21.0 (TID 335) in 187 ms on 192.168.1.37 (executor driver) (18/40)
22/02/08 16:52:21 INFO Executor: Running task 20.0 in stage 21.0 (TID 338)
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_18 locally
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_19 locally
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_20 locally
22/02/08 16:52:21 INFO Executor: Finished task 18.0 in stage 21.0 (TID 336). 1095 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 21.0 in stage 21.0 (TID 339) (192.168.1.37, executor driver, partition 21, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO TaskSetManager: Finished task 18.0 in stage 21.0 (TID 336) in 195 ms on 192.168.1.37 (executor driver) (19/40)
22/02/08 16:52:21 INFO Executor: Running task 21.0 in stage 21.0 (TID 339)
22/02/08 16:52:21 INFO Executor: Finished task 19.0 in stage 21.0 (TID 337). 1095 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 22.0 in stage 21.0 (TID 340) (192.168.1.37, executor driver, partition 22, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO Executor: Running task 22.0 in stage 21.0 (TID 340)
22/02/08 16:52:21 INFO TaskSetManager: Finished task 19.0 in stage 21.0 (TID 337) in 198 ms on 192.168.1.37 (executor driver) (20/40)
22/02/08 16:52:21 INFO Executor: Finished task 20.0 in stage 21.0 (TID 338). 1095 bytes result sent to driver
22/02/08 16:52:21 INFO TaskSetManager: Starting task 23.0 in stage 21.0 (TID 341) (192.168.1.37, executor driver, partition 23, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:21 INFO TaskSetManager: Finished task 20.0 in stage 21.0 (TID 338) in 197 ms on 192.168.1.37 (executor driver) (21/40)
22/02/08 16:52:21 INFO Executor: Running task 23.0 in stage 21.0 (TID 341)
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_21 locally
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_22 locally
22/02/08 16:52:21 INFO BlockManager: Found block rdd_27_23 locally
22/02/08 16:52:22 INFO Executor: Finished task 21.0 in stage 21.0 (TID 339). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 24.0 in stage 21.0 (TID 342) (192.168.1.37, executor driver, partition 24, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO TaskSetManager: Finished task 21.0 in stage 21.0 (TID 339) in 175 ms on 192.168.1.37 (executor driver) (22/40)
22/02/08 16:52:22 INFO Executor: Running task 24.0 in stage 21.0 (TID 342)
22/02/08 16:52:22 INFO Executor: Finished task 22.0 in stage 21.0 (TID 340). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 25.0 in stage 21.0 (TID 343) (192.168.1.37, executor driver, partition 25, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO TaskSetManager: Finished task 22.0 in stage 21.0 (TID 340) in 178 ms on 192.168.1.37 (executor driver) (23/40)
22/02/08 16:52:22 INFO Executor: Running task 25.0 in stage 21.0 (TID 343)
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_24 locally
22/02/08 16:52:22 INFO Executor: Finished task 23.0 in stage 21.0 (TID 341). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 26.0 in stage 21.0 (TID 344) (192.168.1.37, executor driver, partition 26, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 26.0 in stage 21.0 (TID 344)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 23.0 in stage 21.0 (TID 341) in 184 ms on 192.168.1.37 (executor driver) (24/40)
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_25 locally
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_26 locally
22/02/08 16:52:22 INFO Executor: Finished task 24.0 in stage 21.0 (TID 342). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 27.0 in stage 21.0 (TID 345) (192.168.1.37, executor driver, partition 27, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO TaskSetManager: Finished task 24.0 in stage 21.0 (TID 342) in 169 ms on 192.168.1.37 (executor driver) (25/40)
22/02/08 16:52:22 INFO Executor: Running task 27.0 in stage 21.0 (TID 345)
22/02/08 16:52:22 INFO Executor: Finished task 25.0 in stage 21.0 (TID 343). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 28.0 in stage 21.0 (TID 346) (192.168.1.37, executor driver, partition 28, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 28.0 in stage 21.0 (TID 346)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 25.0 in stage 21.0 (TID 343) in 170 ms on 192.168.1.37 (executor driver) (26/40)
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_27 locally
22/02/08 16:52:22 INFO Executor: Finished task 26.0 in stage 21.0 (TID 344). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 29.0 in stage 21.0 (TID 347) (192.168.1.37, executor driver, partition 29, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 29.0 in stage 21.0 (TID 347)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 26.0 in stage 21.0 (TID 344) in 172 ms on 192.168.1.37 (executor driver) (27/40)
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_28 locally
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_29 locally
22/02/08 16:52:22 INFO Executor: Finished task 27.0 in stage 21.0 (TID 345). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 30.0 in stage 21.0 (TID 348) (192.168.1.37, executor driver, partition 30, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 30.0 in stage 21.0 (TID 348)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 27.0 in stage 21.0 (TID 345) in 171 ms on 192.168.1.37 (executor driver) (28/40)
22/02/08 16:52:22 INFO Executor: Finished task 28.0 in stage 21.0 (TID 346). 1095 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 31.0 in stage 21.0 (TID 349) (192.168.1.37, executor driver, partition 31, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 31.0 in stage 21.0 (TID 349)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 28.0 in stage 21.0 (TID 346) in 184 ms on 192.168.1.37 (executor driver) (29/40)
22/02/08 16:52:22 INFO Executor: Finished task 29.0 in stage 21.0 (TID 347). 1095 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 32.0 in stage 21.0 (TID 350) (192.168.1.37, executor driver, partition 32, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 32.0 in stage 21.0 (TID 350)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 29.0 in stage 21.0 (TID 347) in 185 ms on 192.168.1.37 (executor driver) (30/40)
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_30 locally
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_31 locally
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_32 locally
22/02/08 16:52:22 INFO Executor: Finished task 30.0 in stage 21.0 (TID 348). 1095 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 33.0 in stage 21.0 (TID 351) (192.168.1.37, executor driver, partition 33, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 33.0 in stage 21.0 (TID 351)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 30.0 in stage 21.0 (TID 348) in 180 ms on 192.168.1.37 (executor driver) (31/40)
22/02/08 16:52:22 INFO Executor: Finished task 31.0 in stage 21.0 (TID 349). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 34.0 in stage 21.0 (TID 352) (192.168.1.37, executor driver, partition 34, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 34.0 in stage 21.0 (TID 352)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 31.0 in stage 21.0 (TID 349) in 166 ms on 192.168.1.37 (executor driver) (32/40)
22/02/08 16:52:22 INFO Executor: Finished task 32.0 in stage 21.0 (TID 350). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 35.0 in stage 21.0 (TID 353) (192.168.1.37, executor driver, partition 35, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 35.0 in stage 21.0 (TID 353)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 32.0 in stage 21.0 (TID 350) in 162 ms on 192.168.1.37 (executor driver) (33/40)
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_33 locally
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_34 locally
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_35 locally
22/02/08 16:52:22 INFO Executor: Finished task 33.0 in stage 21.0 (TID 351). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 36.0 in stage 21.0 (TID 354) (192.168.1.37, executor driver, partition 36, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 36.0 in stage 21.0 (TID 354)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 33.0 in stage 21.0 (TID 351) in 163 ms on 192.168.1.37 (executor driver) (34/40)
22/02/08 16:52:22 INFO Executor: Finished task 34.0 in stage 21.0 (TID 352). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 37.0 in stage 21.0 (TID 355) (192.168.1.37, executor driver, partition 37, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 37.0 in stage 21.0 (TID 355)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 34.0 in stage 21.0 (TID 352) in 162 ms on 192.168.1.37 (executor driver) (35/40)
22/02/08 16:52:22 INFO Executor: Finished task 35.0 in stage 21.0 (TID 353). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 38.0 in stage 21.0 (TID 356) (192.168.1.37, executor driver, partition 38, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 38.0 in stage 21.0 (TID 356)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 35.0 in stage 21.0 (TID 353) in 160 ms on 192.168.1.37 (executor driver) (36/40)
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_36 locally
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_37 locally
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_38 locally
22/02/08 16:52:22 INFO Executor: Finished task 36.0 in stage 21.0 (TID 354). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Starting task 39.0 in stage 21.0 (TID 357) (192.168.1.37, executor driver, partition 39, PROCESS_LOCAL, 4323 bytes) taskResourceAssignments Map()
22/02/08 16:52:22 INFO Executor: Running task 39.0 in stage 21.0 (TID 357)
22/02/08 16:52:22 INFO TaskSetManager: Finished task 36.0 in stage 21.0 (TID 354) in 170 ms on 192.168.1.37 (executor driver) (37/40)
22/02/08 16:52:22 INFO Executor: Finished task 37.0 in stage 21.0 (TID 355). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Finished task 37.0 in stage 21.0 (TID 355) in 176 ms on 192.168.1.37 (executor driver) (38/40)
22/02/08 16:52:22 INFO BlockManager: Found block rdd_27_39 locally
22/02/08 16:52:22 INFO Executor: Finished task 38.0 in stage 21.0 (TID 356). 1052 bytes result sent to driver
22/02/08 16:52:22 INFO TaskSetManager: Finished task 38.0 in stage 21.0 (TID 356) in 176 ms on 192.168.1.37 (executor driver) (39/40)
22/02/08 16:52:23 INFO Executor: Finished task 39.0 in stage 21.0 (TID 357). 1052 bytes result sent to driver
22/02/08 16:52:23 INFO TaskSetManager: Finished task 39.0 in stage 21.0 (TID 357) in 154 ms on 192.168.1.37 (executor driver) (40/40)
22/02/08 16:52:23 INFO TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool
22/02/08 16:52:23 INFO DAGScheduler: ShuffleMapStage 21 (mapToPair at BaseSparkCommitActionExecutor.java:233) finished in 2.815 s
22/02/08 16:52:23 INFO DAGScheduler: looking for newly runnable stages
22/02/08 16:52:23 INFO DAGScheduler: running: Set()
22/02/08 16:52:23 INFO DAGScheduler: waiting: Set(ResultStage 22)
22/02/08 16:52:23 INFO DAGScheduler: failed: Set()
22/02/08 16:52:23 INFO DAGScheduler: Submitting ResultStage 22 (MapPartitionsRDD[43] at mapToDouble at DeltaSync.java:592), which has no missing parents
22/02/08 16:52:23 INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 663.1 KiB, free 955.4 MiB)
22/02/08 16:52:23 INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 234.2 KiB, free 955.2 MiB)
22/02/08 16:52:23 INFO BlockManagerInfo: Added broadcast_16_piece0 in memory on 192.168.1.37:53051 (size: 234.2 KiB, free: 956.7 MiB)
22/02/08 16:52:23 INFO SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1427
22/02/08 16:52:23 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 22 (MapPartitionsRDD[43] at mapToDouble at DeltaSync.java:592) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/08 16:52:23 INFO TaskSchedulerImpl: Adding task set 22.0 with 10 tasks resource profile 0
22/02/08 16:52:23 INFO TaskSetManager: Starting task 0.0 in stage 22.0 (TID 358) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:23 INFO TaskSetManager: Starting task 1.0 in stage 22.0 (TID 359) (192.168.1.37, executor driver, partition 1, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:23 INFO TaskSetManager: Starting task 2.0 in stage 22.0 (TID 360) (192.168.1.37, executor driver, partition 2, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:23 INFO Executor: Running task 0.0 in stage 22.0 (TID 358)
22/02/08 16:52:23 INFO Executor: Running task 1.0 in stage 22.0 (TID 359)
22/02/08 16:52:23 INFO Executor: Running task 2.0 in stage 22.0 (TID 360)
22/02/08 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 40 (45.4 MiB) non-empty blocks including 40 (45.4 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 40 (45.6 MiB) non-empty blocks including 40 (45.6 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 40 (45.6 MiB) non-empty blocks including 40 (45.6 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:23 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220208165137552 for file 272a5505-e183-4c3f-98a2-5d92f194272c-0
22/02/08 16:52:23 INFO BaseSparkDeltaCommitActionExecutor: Small file corrections for updates for commit 20220208165137552 for file 272a5505-e183-4c3f-98a2-5d92f194272c-0
22/02/08 16:52:23 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220208165137552 for file 6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0
22/02/08 16:52:23 INFO BaseSparkDeltaCommitActionExecutor: Small file corrections for updates for commit 20220208165137552 for file 6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0
22/02/08 16:52:23 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220208165137552 for file ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0
22/02/08 16:52:23 INFO BaseSparkDeltaCommitActionExecutor: Small file corrections for updates for commit 20220208165137552 for file ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0
22/02/08 16:52:23 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:23 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:23 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:23 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:23 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:23 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:23 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:23 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:23 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:23 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:23 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:23 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:23 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafile/latest/partition?partition=2022%2F1%2F30&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&fileid=272a5505-e183-4c3f-98a2-5d92f194272c-0&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:23 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafile/latest/partition?partition=2022%2F1%2F31&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&fileid=6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:23 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:23 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:23 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:23 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:23 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:23 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:23 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafile/latest/partition?partition=2022%2F1%2F28&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&fileid=ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:23 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824
22/02/08 16:52:23 INFO ExternalSpillableMap: Estimated Payload size => 2000
22/02/08 16:52:23 INFO ExternalSpillableMap: New Estimated Payload size => 1845
22/02/08 16:52:23 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824
22/02/08 16:52:23 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824
22/02/08 16:52:23 INFO ExternalSpillableMap: Estimated Payload size => 2000
22/02/08 16:52:23 INFO ExternalSpillableMap: New Estimated Payload size => 1823
22/02/08 16:52:23 INFO ExternalSpillableMap: Estimated Payload size => 2000
22/02/08 16:52:23 INFO ExternalSpillableMap: New Estimated Payload size => 1832
22/02/08 16:52:23 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 69800Total size in bytes of MemoryBasedMap => 127873616Number of entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
22/02/08 16:52:23 INFO HoodieMergeHandle: partitionPath:2022/1/28, fileId to be merged:ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0
22/02/08 16:52:23 INFO HoodieMergeHandle: Merging new data into oldPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/28/ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0_2-22-360_20220208164721920.parquet, as newPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/28/ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0_2-22-360_20220208165137552.parquet
22/02/08 16:52:23 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 69920Total size in bytes of MemoryBasedMap => 127464188Number of entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
22/02/08 16:52:23 INFO HoodieMergeHandle: partitionPath:2022/1/31, fileId to be merged:6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0
22/02/08 16:52:23 INFO HoodieMergeHandle: Merging new data into oldPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/31/6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0_0-22-358_20220208164721920.parquet, as newPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/31/6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0_0-22-358_20220208165137552.parquet
22/02/08 16:52:23 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 69469Total size in bytes of MemoryBasedMap => 128170381Number of entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
22/02/08 16:52:23 INFO HoodieMergeHandle: partitionPath:2022/1/30, fileId to be merged:272a5505-e183-4c3f-98a2-5d92f194272c-0
22/02/08 16:52:23 INFO HoodieMergeHandle: Merging new data into oldPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/30/272a5505-e183-4c3f-98a2-5d92f194272c-0_1-22-359_20220208164721920.parquet, as newPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/30/272a5505-e183-4c3f-98a2-5d92f194272c-0_1-22-359_20220208165137552.parquet
22/02/08 16:52:23 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create?markername=2022%2F1%2F30%2F272a5505-e183-4c3f-98a2-5d92f194272c-0_1-22-359_20220208165137552.parquet.marker.MERGE&markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:52:23 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create?markername=2022%2F1%2F28%2Fddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0_2-22-360_20220208165137552.parquet.marker.MERGE&markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:52:23 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create?markername=2022%2F1%2F31%2F6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0_0-22-358_20220208165137552.parquet.marker.MERGE&markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:52:23 INFO MarkerHandler: Request: create marker file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220208165137552 2022/1/31/6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0_0-22-358_20220208165137552.parquet.marker.MERGE
22/02/08 16:52:23 INFO MarkerHandler: Request: create marker file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220208165137552 2022/1/28/ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0_2-22-360_20220208165137552.parquet.marker.MERGE
22/02/08 16:52:23 INFO MarkerHandler: Request: create marker file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220208165137552 2022/1/30/272a5505-e183-4c3f-98a2-5d92f194272c-0_1-22-359_20220208165137552.parquet.marker.MERGE
22/02/08 16:52:23 INFO TimelineServerBasedWriteMarkers: [timeline-server-based] Created marker file 2022/1/31/6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0_0-22-358_20220208165137552.parquet.marker.MERGE in 88 ms
22/02/08 16:52:23 INFO TimelineServerBasedWriteMarkers: [timeline-server-based] Created marker file 2022/1/28/ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0_2-22-360_20220208165137552.parquet.marker.MERGE in 89 ms
22/02/08 16:52:23 INFO TimelineServerBasedWriteMarkers: [timeline-server-based] Created marker file 2022/1/30/272a5505-e183-4c3f-98a2-5d92f194272c-0_1-22-359_20220208165137552.parquet.marker.MERGE in 89 ms
22/02/08 16:52:24 INFO CodecPool: Got brand-new compressor [.gz]
22/02/08 16:52:24 INFO CodecPool: Got brand-new compressor [.gz]
22/02/08 16:52:24 INFO CodecPool: Got brand-new compressor [.gz]
22/02/08 16:52:24 INFO IteratorBasedQueueProducer: starting to buffer records
22/02/08 16:52:24 INFO IteratorBasedQueueProducer: starting to buffer records
22/02/08 16:52:24 INFO IteratorBasedQueueProducer: starting to buffer records
22/02/08 16:52:24 INFO BoundedInMemoryExecutor: starting consumer thread
22/02/08 16:52:24 INFO BoundedInMemoryExecutor: starting consumer thread
22/02/08 16:52:24 INFO BoundedInMemoryExecutor: starting consumer thread
22/02/08 16:52:24 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 153820 records.
22/02/08 16:52:24 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 154700 records.
22/02/08 16:52:24 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:24 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:24 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 154900 records.
22/02/08 16:52:24 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:24 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:24 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:24 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:24 INFO InternalParquetRecordReader: block read in memory in 70 ms. row count = 153820
22/02/08 16:52:24 INFO InternalParquetRecordReader: block read in memory in 70 ms. row count = 154900
22/02/08 16:52:24 INFO InternalParquetRecordReader: block read in memory in 70 ms. row count = 154700
22/02/08 16:52:25 INFO BlockManagerInfo: Removed broadcast_15_piece0 on 192.168.1.37:53051 in memory (size: 185.6 KiB, free: 956.9 MiB)
22/02/08 16:52:27 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:52:36 INFO IteratorBasedQueueProducer: finished buffering records
22/02/08 16:52:36 INFO BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
22/02/08 16:52:36 INFO IteratorBasedQueueProducer: finished buffering records
22/02/08 16:52:36 INFO BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
22/02/08 16:52:36 INFO IteratorBasedQueueProducer: finished buffering records
22/02/08 16:52:36 INFO BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
22/02/08 16:52:37 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:52:40 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/1/30 fileID 272a5505-e183-4c3f-98a2-5d92f194272c-0, took 17108 ms.
22/02/08 16:52:40 INFO MemoryStore: Block rdd_42_1 stored as values in memory (estimated size 395.0 B, free 955.9 MiB)
22/02/08 16:52:40 INFO BlockManagerInfo: Added rdd_42_1 in memory on 192.168.1.37:53051 (size: 395.0 B, free: 956.9 MiB)
22/02/08 16:52:40 INFO Executor: Finished task 1.0 in stage 22.0 (TID 359). 1649 bytes result sent to driver
22/02/08 16:52:40 INFO TaskSetManager: Starting task 3.0 in stage 22.0 (TID 361) (192.168.1.37, executor driver, partition 3, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:40 INFO Executor: Running task 3.0 in stage 22.0 (TID 361)
22/02/08 16:52:40 INFO TaskSetManager: Finished task 1.0 in stage 22.0 (TID 359) in 17242 ms on 192.168.1.37 (executor driver) (1/10)
22/02/08 16:52:40 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/1/28 fileID ddb09340-3b0f-4919-bb7b-a59d78e1dfdc-0, took 17136 ms.
22/02/08 16:52:40 INFO MemoryStore: Block rdd_42_2 stored as values in memory (estimated size 395.0 B, free 955.9 MiB)
22/02/08 16:52:40 INFO BlockManagerInfo: Added rdd_42_2 in memory on 192.168.1.37:53051 (size: 395.0 B, free: 956.9 MiB)
22/02/08 16:52:40 INFO Executor: Finished task 2.0 in stage 22.0 (TID 360). 1649 bytes result sent to driver
22/02/08 16:52:40 INFO TaskSetManager: Starting task 4.0 in stage 22.0 (TID 362) (192.168.1.37, executor driver, partition 4, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:40 INFO Executor: Running task 4.0 in stage 22.0 (TID 362)
22/02/08 16:52:40 INFO TaskSetManager: Finished task 2.0 in stage 22.0 (TID 360) in 17264 ms on 192.168.1.37 (executor driver) (2/10)
22/02/08 16:52:40 INFO ShuffleBlockFetcherIterator: Getting 40 (46.3 MiB) non-empty blocks including 40 (46.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:40 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/1/31 fileID 6cef380a-a0c4-4a4b-93e4-e8d55343aaf9-0, took 17160 ms.
22/02/08 16:52:40 INFO MemoryStore: Block rdd_42_0 stored as values in memory (estimated size 395.0 B, free 955.9 MiB)
22/02/08 16:52:40 INFO BlockManagerInfo: Added rdd_42_0 in memory on 192.168.1.37:53051 (size: 395.0 B, free: 956.9 MiB)
22/02/08 16:52:40 INFO Executor: Finished task 0.0 in stage 22.0 (TID 358). 1649 bytes result sent to driver
22/02/08 16:52:40 INFO TaskSetManager: Starting task 5.0 in stage 22.0 (TID 363) (192.168.1.37, executor driver, partition 5, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:40 INFO TaskSetManager: Finished task 0.0 in stage 22.0 (TID 358) in 17289 ms on 192.168.1.37 (executor driver) (3/10)
22/02/08 16:52:40 INFO Executor: Running task 5.0 in stage 22.0 (TID 363)
22/02/08 16:52:40 INFO ShuffleBlockFetcherIterator: Getting 40 (45.8 MiB) non-empty blocks including 40 (45.8 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:40 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220208165137552 for file bfb4a6e0-cd86-481f-8781-f763c3c88016-0
22/02/08 16:52:40 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220208165137552 for file 01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0
22/02/08 16:52:40 INFO BaseSparkDeltaCommitActionExecutor: Small file corrections for updates for commit 20220208165137552 for file bfb4a6e0-cd86-481f-8781-f763c3c88016-0
22/02/08 16:52:40 INFO BaseSparkDeltaCommitActionExecutor: Small file corrections for updates for commit 20220208165137552 for file 01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0
22/02/08 16:52:40 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:40 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:40 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:40 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:40 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:40 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:40 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:40 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:40 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:40 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:40 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:40 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:40 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafile/latest/partition?partition=2022%2F1%2F27&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&fileid=01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:40 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafile/latest/partition?partition=2022%2F2%2F2&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&fileid=bfb4a6e0-cd86-481f-8781-f763c3c88016-0&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:40 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824
22/02/08 16:52:40 INFO ExternalSpillableMap: Estimated Payload size => 2208
22/02/08 16:52:40 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824
22/02/08 16:52:40 INFO ExternalSpillableMap: Estimated Payload size => 2000
22/02/08 16:52:40 INFO ExternalSpillableMap: New Estimated Payload size => 1821
22/02/08 16:52:40 INFO ExternalSpillableMap: New Estimated Payload size => 1827
22/02/08 16:52:40 INFO ShuffleBlockFetcherIterator: Getting 40 (44.9 MiB) non-empty blocks including 40 (44.9 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:40 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220208165137552 for file cbbad821-34e6-47c7-9f8b-f41da613023b-0
22/02/08 16:52:40 INFO BaseSparkDeltaCommitActionExecutor: Small file corrections for updates for commit 20220208165137552 for file cbbad821-34e6-47c7-9f8b-f41da613023b-0
22/02/08 16:52:40 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:40 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:40 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:40 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:40 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:40 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:40 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafile/latest/partition?partition=2022%2F1%2F29&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&fileid=cbbad821-34e6-47c7-9f8b-f41da613023b-0&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:40 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824
22/02/08 16:52:40 INFO ExternalSpillableMap: Estimated Payload size => 2224
22/02/08 16:52:40 INFO ExternalSpillableMap: New Estimated Payload size => 1847
22/02/08 16:52:41 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 70526Total size in bytes of MemoryBasedMap => 128851078Number of entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
22/02/08 16:52:41 INFO HoodieMergeHandle: partitionPath:2022/1/27, fileId to be merged:01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0
22/02/08 16:52:41 INFO HoodieMergeHandle: Merging new data into oldPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/27/01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0_3-22-361_20220208164721920.parquet, as newPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/27/01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0_3-22-361_20220208165137552.parquet
22/02/08 16:52:41 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create?markername=2022%2F1%2F27%2F01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0_3-22-361_20220208165137552.parquet.marker.MERGE&markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:52:41 INFO MarkerHandler: Request: create marker file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220208165137552 2022/1/27/01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0_3-22-361_20220208165137552.parquet.marker.MERGE
22/02/08 16:52:41 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 70002Total size in bytes of MemoryBasedMap => 127473646Number of entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
22/02/08 16:52:41 INFO HoodieMergeHandle: partitionPath:2022/2/2, fileId to be merged:bfb4a6e0-cd86-481f-8781-f763c3c88016-0
22/02/08 16:52:41 INFO HoodieMergeHandle: Merging new data into oldPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/2/2/bfb4a6e0-cd86-481f-8781-f763c3c88016-0_4-22-362_20220208164721920.parquet, as newPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/2/2/bfb4a6e0-cd86-481f-8781-f763c3c88016-0_4-22-362_20220208165137552.parquet
22/02/08 16:52:41 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create?markername=2022%2F2%2F2%2Fbfb4a6e0-cd86-481f-8781-f763c3c88016-0_4-22-362_20220208165137552.parquet.marker.MERGE&markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:52:41 INFO MarkerHandler: Request: create marker file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220208165137552 2022/2/2/bfb4a6e0-cd86-481f-8781-f763c3c88016-0_4-22-362_20220208165137552.parquet.marker.MERGE
22/02/08 16:52:41 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 68946Total size in bytes of MemoryBasedMap => 127343354Number of entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
22/02/08 16:52:41 INFO HoodieMergeHandle: partitionPath:2022/1/29, fileId to be merged:cbbad821-34e6-47c7-9f8b-f41da613023b-0
22/02/08 16:52:41 INFO HoodieMergeHandle: Merging new data into oldPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/29/cbbad821-34e6-47c7-9f8b-f41da613023b-0_5-22-363_20220208164721920.parquet, as newPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/29/cbbad821-34e6-47c7-9f8b-f41da613023b-0_5-22-363_20220208165137552.parquet
22/02/08 16:52:41 INFO TimelineServerBasedWriteMarkers: [timeline-server-based] Created marker file 2022/1/27/01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0_3-22-361_20220208165137552.parquet.marker.MERGE in 19 ms
22/02/08 16:52:41 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create?markername=2022%2F1%2F29%2Fcbbad821-34e6-47c7-9f8b-f41da613023b-0_5-22-363_20220208165137552.parquet.marker.MERGE&markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:52:41 INFO MarkerHandler: Request: create marker file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220208165137552 2022/1/29/cbbad821-34e6-47c7-9f8b-f41da613023b-0_5-22-363_20220208165137552.parquet.marker.MERGE
22/02/08 16:52:41 INFO CodecPool: Got brand-new compressor [.gz]
22/02/08 16:52:41 INFO IteratorBasedQueueProducer: starting to buffer records
22/02/08 16:52:41 INFO BoundedInMemoryExecutor: starting consumer thread
22/02/08 16:52:41 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 155885 records.
22/02/08 16:52:41 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:41 INFO TimelineServerBasedWriteMarkers: [timeline-server-based] Created marker file 2022/2/2/bfb4a6e0-cd86-481f-8781-f763c3c88016-0_4-22-362_20220208165137552.parquet.marker.MERGE in 56 ms
22/02/08 16:52:41 INFO TimelineServerBasedWriteMarkers: [timeline-server-based] Created marker file 2022/1/29/cbbad821-34e6-47c7-9f8b-f41da613023b-0_5-22-363_20220208165137552.parquet.marker.MERGE in 45 ms
22/02/08 16:52:41 INFO CodecPool: Got brand-new compressor [.gz]
22/02/08 16:52:41 INFO CodecPool: Got brand-new compressor [.gz]
22/02/08 16:52:41 INFO IteratorBasedQueueProducer: starting to buffer records
22/02/08 16:52:41 INFO IteratorBasedQueueProducer: starting to buffer records
22/02/08 16:52:41 INFO BoundedInMemoryExecutor: starting consumer thread
22/02/08 16:52:41 INFO BoundedInMemoryExecutor: starting consumer thread
22/02/08 16:52:41 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 153140 records.
22/02/08 16:52:41 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:41 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 154900 records.
22/02/08 16:52:41 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:41 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:41 INFO InternalParquetRecordReader: block read in memory in 73 ms. row count = 155885
22/02/08 16:52:41 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:41 INFO InternalParquetRecordReader: block read in memory in 180 ms. row count = 153140
22/02/08 16:52:41 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:41 INFO InternalParquetRecordReader: block read in memory in 184 ms. row count = 154900
22/02/08 16:52:47 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:52:52 INFO IteratorBasedQueueProducer: finished buffering records
22/02/08 16:52:52 INFO BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
22/02/08 16:52:52 INFO IteratorBasedQueueProducer: finished buffering records
22/02/08 16:52:52 INFO BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
22/02/08 16:52:52 INFO IteratorBasedQueueProducer: finished buffering records
22/02/08 16:52:52 INFO BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
22/02/08 16:52:56 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/1/29 fileID cbbad821-34e6-47c7-9f8b-f41da613023b-0, took 15502 ms.
22/02/08 16:52:56 INFO MemoryStore: Block rdd_42_5 stored as values in memory (estimated size 395.0 B, free 955.9 MiB)
22/02/08 16:52:56 INFO BlockManagerInfo: Added rdd_42_5 in memory on 192.168.1.37:53051 (size: 395.0 B, free: 956.9 MiB)
22/02/08 16:52:56 INFO Executor: Finished task 5.0 in stage 22.0 (TID 363). 1649 bytes result sent to driver
22/02/08 16:52:56 INFO TaskSetManager: Starting task 6.0 in stage 22.0 (TID 364) (192.168.1.37, executor driver, partition 6, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:56 INFO TaskSetManager: Finished task 5.0 in stage 22.0 (TID 363) in 15556 ms on 192.168.1.37 (executor driver) (4/10)
22/02/08 16:52:56 INFO Executor: Running task 6.0 in stage 22.0 (TID 364)
22/02/08 16:52:56 INFO ShuffleBlockFetcherIterator: Getting 40 (46.0 MiB) non-empty blocks including 40 (46.0 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:56 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220208165137552 for file b2aa5767-67fa-407b-ab8d-0c41e3f61983-0
22/02/08 16:52:56 INFO BaseSparkDeltaCommitActionExecutor: Small file corrections for updates for commit 20220208165137552 for file b2aa5767-67fa-407b-ab8d-0c41e3f61983-0
22/02/08 16:52:56 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:56 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:56 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:56 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:56 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:56 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:56 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafile/latest/partition?partition=2022%2F1%2F24&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&fileid=b2aa5767-67fa-407b-ab8d-0c41e3f61983-0&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:56 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824
22/02/08 16:52:56 INFO ExternalSpillableMap: Estimated Payload size => 2000
22/02/08 16:52:56 INFO ExternalSpillableMap: New Estimated Payload size => 1816
22/02/08 16:52:56 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/1/27 fileID 01e591d6-6caf-4d9b-bd29-d0e6d2fd9c67-0, took 15687 ms.
22/02/08 16:52:56 INFO MemoryStore: Block rdd_42_3 stored as values in memory (estimated size 395.0 B, free 955.9 MiB)
22/02/08 16:52:56 INFO BlockManagerInfo: Added rdd_42_3 in memory on 192.168.1.37:53051 (size: 395.0 B, free: 956.9 MiB)
22/02/08 16:52:56 INFO Executor: Finished task 3.0 in stage 22.0 (TID 361). 1649 bytes result sent to driver
22/02/08 16:52:56 INFO TaskSetManager: Starting task 7.0 in stage 22.0 (TID 365) (192.168.1.37, executor driver, partition 7, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:56 INFO Executor: Running task 7.0 in stage 22.0 (TID 365)
22/02/08 16:52:56 INFO TaskSetManager: Finished task 3.0 in stage 22.0 (TID 361) in 15775 ms on 192.168.1.37 (executor driver) (5/10)
22/02/08 16:52:56 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/2/2 fileID bfb4a6e0-cd86-481f-8781-f763c3c88016-0, took 15706 ms.
22/02/08 16:52:56 INFO MemoryStore: Block rdd_42_4 stored as values in memory (estimated size 393.0 B, free 955.9 MiB)
22/02/08 16:52:56 INFO BlockManagerInfo: Added rdd_42_4 in memory on 192.168.1.37:53051 (size: 393.0 B, free: 956.9 MiB)
22/02/08 16:52:56 INFO Executor: Finished task 4.0 in stage 22.0 (TID 362). 1649 bytes result sent to driver
22/02/08 16:52:56 INFO TaskSetManager: Starting task 8.0 in stage 22.0 (TID 366) (192.168.1.37, executor driver, partition 8, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:52:56 INFO Executor: Running task 8.0 in stage 22.0 (TID 366)
22/02/08 16:52:56 INFO TaskSetManager: Finished task 4.0 in stage 22.0 (TID 362) in 15760 ms on 192.168.1.37 (executor driver) (6/10)
22/02/08 16:52:56 INFO ShuffleBlockFetcherIterator: Getting 40 (45.6 MiB) non-empty blocks including 40 (45.6 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:56 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220208165137552 for file bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0
22/02/08 16:52:56 INFO BaseSparkDeltaCommitActionExecutor: Small file corrections for updates for commit 20220208165137552 for file bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0
22/02/08 16:52:56 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:56 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:56 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:56 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:56 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:56 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:56 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafile/latest/partition?partition=2022%2F2%2F1&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&fileid=bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:56 INFO ShuffleBlockFetcherIterator: Getting 40 (46.3 MiB) non-empty blocks including 40 (46.3 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:52:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:52:56 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824
22/02/08 16:52:56 INFO ExternalSpillableMap: Estimated Payload size => 2216
22/02/08 16:52:56 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220208165137552 for file b7054c05-f9cd-4f0b-95b8-1492803c6aff-0
22/02/08 16:52:56 INFO BaseSparkDeltaCommitActionExecutor: Small file corrections for updates for commit 20220208165137552 for file b7054c05-f9cd-4f0b-95b8-1492803c6aff-0
22/02/08 16:52:56 INFO ExternalSpillableMap: New Estimated Payload size => 1818
22/02/08 16:52:56 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:52:56 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:52:56 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:52:56 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:52:56 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:52:56 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:52:56 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafile/latest/partition?partition=2022%2F1%2F26&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&fileid=b7054c05-f9cd-4f0b-95b8-1492803c6aff-0&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:52:56 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824
22/02/08 16:52:56 INFO ExternalSpillableMap: Estimated Payload size => 2224
22/02/08 16:52:56 INFO ExternalSpillableMap: New Estimated Payload size => 1836
22/02/08 16:52:56 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 70116Total size in bytes of MemoryBasedMap => 127330704Number of entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
22/02/08 16:52:56 INFO HoodieMergeHandle: partitionPath:2022/1/24, fileId to be merged:b2aa5767-67fa-407b-ab8d-0c41e3f61983-0
22/02/08 16:52:56 INFO HoodieMergeHandle: Merging new data into oldPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/24/b2aa5767-67fa-407b-ab8d-0c41e3f61983-0_6-22-364_20220208164721920.parquet, as newPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/24/b2aa5767-67fa-407b-ab8d-0c41e3f61983-0_6-22-364_20220208165137552.parquet
22/02/08 16:52:56 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create?markername=2022%2F1%2F24%2Fb2aa5767-67fa-407b-ab8d-0c41e3f61983-0_6-22-364_20220208165137552.parquet.marker.MERGE&markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:52:56 INFO MarkerHandler: Request: create marker file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220208165137552 2022/1/24/b2aa5767-67fa-407b-ab8d-0c41e3f61983-0_6-22-364_20220208165137552.parquet.marker.MERGE
22/02/08 16:52:56 INFO TimelineServerBasedWriteMarkers: [timeline-server-based] Created marker file 2022/1/24/b2aa5767-67fa-407b-ab8d-0c41e3f61983-0_6-22-364_20220208165137552.parquet.marker.MERGE in 43 ms
22/02/08 16:52:56 INFO CodecPool: Got brand-new compressor [.gz]
22/02/08 16:52:56 INFO IteratorBasedQueueProducer: starting to buffer records
22/02/08 16:52:56 INFO BoundedInMemoryExecutor: starting consumer thread
22/02/08 16:52:56 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 154930 records.
22/02/08 16:52:56 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:56 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:56 INFO InternalParquetRecordReader: block read in memory in 41 ms. row count = 154930
22/02/08 16:52:56 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 69767Total size in bytes of MemoryBasedMap => 126836462Number of entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
22/02/08 16:52:56 INFO HoodieMergeHandle: partitionPath:2022/2/1, fileId to be merged:bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0
22/02/08 16:52:56 INFO HoodieMergeHandle: Merging new data into oldPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/2/1/bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0_7-22-365_20220208164721920.parquet, as newPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/2/1/bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0_7-22-365_20220208165137552.parquet
22/02/08 16:52:56 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create?markername=2022%2F2%2F1%2Fbf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0_7-22-365_20220208165137552.parquet.marker.MERGE&markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:52:56 INFO MarkerHandler: Request: create marker file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220208165137552 2022/2/1/bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0_7-22-365_20220208165137552.parquet.marker.MERGE
22/02/08 16:52:56 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 70872Total size in bytes of MemoryBasedMap => 130121064Number of entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
22/02/08 16:52:56 INFO HoodieMergeHandle: partitionPath:2022/1/26, fileId to be merged:b7054c05-f9cd-4f0b-95b8-1492803c6aff-0
22/02/08 16:52:56 INFO HoodieMergeHandle: Merging new data into oldPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/26/b7054c05-f9cd-4f0b-95b8-1492803c6aff-0_8-22-366_20220208164721920.parquet, as newPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/26/b7054c05-f9cd-4f0b-95b8-1492803c6aff-0_8-22-366_20220208165137552.parquet
22/02/08 16:52:56 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create?markername=2022%2F1%2F26%2Fb7054c05-f9cd-4f0b-95b8-1492803c6aff-0_8-22-366_20220208165137552.parquet.marker.MERGE&markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:52:56 INFO MarkerHandler: Request: create marker file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220208165137552 2022/1/26/b7054c05-f9cd-4f0b-95b8-1492803c6aff-0_8-22-366_20220208165137552.parquet.marker.MERGE
22/02/08 16:52:56 INFO TimelineServerBasedWriteMarkers: [timeline-server-based] Created marker file 2022/2/1/bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0_7-22-365_20220208165137552.parquet.marker.MERGE in 36 ms
22/02/08 16:52:56 INFO TimelineServerBasedWriteMarkers: [timeline-server-based] Created marker file 2022/1/26/b7054c05-f9cd-4f0b-95b8-1492803c6aff-0_8-22-366_20220208165137552.parquet.marker.MERGE in 24 ms
22/02/08 16:52:56 INFO CodecPool: Got brand-new compressor [.gz]
22/02/08 16:52:56 INFO CodecPool: Got brand-new compressor [.gz]
22/02/08 16:52:56 INFO IteratorBasedQueueProducer: starting to buffer records
22/02/08 16:52:56 INFO IteratorBasedQueueProducer: starting to buffer records
22/02/08 16:52:56 INFO BoundedInMemoryExecutor: starting consumer thread
22/02/08 16:52:56 INFO BoundedInMemoryExecutor: starting consumer thread
22/02/08 16:52:56 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 157700 records.
22/02/08 16:52:56 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:56 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 154480 records.
22/02/08 16:52:56 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:52:56 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:56 INFO InternalParquetRecordReader: block read in memory in 46 ms. row count = 157700
22/02/08 16:52:56 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:52:56 INFO InternalParquetRecordReader: block read in memory in 47 ms. row count = 154480
22/02/08 16:52:57 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:53:06 INFO IteratorBasedQueueProducer: finished buffering records
22/02/08 16:53:06 INFO BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
22/02/08 16:53:06 INFO IteratorBasedQueueProducer: finished buffering records
22/02/08 16:53:06 INFO BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
22/02/08 16:53:06 INFO IteratorBasedQueueProducer: finished buffering records
22/02/08 16:53:06 INFO BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
22/02/08 16:53:07 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:53:09 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/1/24 fileID b2aa5767-67fa-407b-ab8d-0c41e3f61983-0, took 13436 ms.
22/02/08 16:53:09 INFO MemoryStore: Block rdd_42_6 stored as values in memory (estimated size 395.0 B, free 955.9 MiB)
22/02/08 16:53:09 INFO BlockManagerInfo: Added rdd_42_6 in memory on 192.168.1.37:53051 (size: 395.0 B, free: 956.9 MiB)
22/02/08 16:53:09 INFO Executor: Finished task 6.0 in stage 22.0 (TID 364). 1649 bytes result sent to driver
22/02/08 16:53:09 INFO TaskSetManager: Starting task 9.0 in stage 22.0 (TID 367) (192.168.1.37, executor driver, partition 9, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:09 INFO TaskSetManager: Finished task 6.0 in stage 22.0 (TID 364) in 13476 ms on 192.168.1.37 (executor driver) (7/10)
22/02/08 16:53:09 INFO Executor: Running task 9.0 in stage 22.0 (TID 367)
22/02/08 16:53:09 INFO ShuffleBlockFetcherIterator: Getting 40 (46.2 MiB) non-empty blocks including 40 (46.2 MiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:53:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:53:09 INFO BaseSparkDeltaCommitActionExecutor: Merging updates for commit 20220208165137552 for file 0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0
22/02/08 16:53:09 INFO BaseSparkDeltaCommitActionExecutor: Small file corrections for updates for commit 20220208165137552 for file 0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0
22/02/08 16:53:09 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:53:09 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:53:09 INFO FileSystemViewManager: Creating remote view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4. Server=192.168.1.37:53121, Timeout=300
22/02/08 16:53:09 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:09 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:53:09 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:53:09 INFO RemoteHoodieTableFileSystemView: Sending request : (http://192.168.1.37:53121/v1/hoodie/view/datafile/latest/partition?partition=2022%2F1%2F25&basepath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4&fileid=0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0&lastinstantts=20220208164926830&timelinehash=a6e1dd7875fd9d23a4814e7708c21ac81715b97406bc60ab7f723d175f280d66)
22/02/08 16:53:09 INFO HoodieMergeHandle: MaxMemoryPerPartitionMerge => 1073741824
22/02/08 16:53:09 INFO ExternalSpillableMap: Estimated Payload size => 2224
22/02/08 16:53:09 INFO ExternalSpillableMap: New Estimated Payload size => 1845
22/02/08 16:53:09 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/2/1 fileID bf4a2be8-810e-4e4d-a806-0eaef62a0ea9-0, took 13437 ms.
22/02/08 16:53:09 INFO MemoryStore: Block rdd_42_7 stored as values in memory (estimated size 393.0 B, free 955.9 MiB)
22/02/08 16:53:09 INFO BlockManagerInfo: Added rdd_42_7 in memory on 192.168.1.37:53051 (size: 393.0 B, free: 956.9 MiB)
22/02/08 16:53:09 INFO Executor: Finished task 7.0 in stage 22.0 (TID 365). 1649 bytes result sent to driver
22/02/08 16:53:09 INFO TaskSetManager: Finished task 7.0 in stage 22.0 (TID 365) in 13476 ms on 192.168.1.37 (executor driver) (8/10)
22/02/08 16:53:09 INFO HoodieMergeHandle: Number of entries in MemoryBasedMap => 70449Total size in bytes of MemoryBasedMap => 129978481Number of entries in BitCaskDiskMap => 0Size of file spilled to disk => 0
22/02/08 16:53:09 INFO HoodieMergeHandle: partitionPath:2022/1/25, fileId to be merged:0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0
22/02/08 16:53:09 INFO HoodieMergeHandle: Merging new data into oldPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/25/0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0_9-22-367_20220208164721920.parquet, as newPath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/2022/1/25/0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0_9-22-367_20220208165137552.parquet
22/02/08 16:53:09 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create?markername=2022%2F1%2F25%2F0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0_9-22-367_20220208165137552.parquet.marker.MERGE&markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:53:09 INFO MarkerHandler: Request: create marker file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/.temp/20220208165137552 2022/1/25/0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0_9-22-367_20220208165137552.parquet.marker.MERGE
22/02/08 16:53:09 INFO TimelineServerBasedWriteMarkers: [timeline-server-based] Created marker file 2022/1/25/0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0_9-22-367_20220208165137552.parquet.marker.MERGE in 70 ms
22/02/08 16:53:09 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/1/26 fileID b7054c05-f9cd-4f0b-95b8-1492803c6aff-0, took 13659 ms.
22/02/08 16:53:09 INFO MemoryStore: Block rdd_42_8 stored as values in memory (estimated size 395.0 B, free 955.9 MiB)
22/02/08 16:53:09 INFO BlockManagerInfo: Added rdd_42_8 in memory on 192.168.1.37:53051 (size: 395.0 B, free: 956.9 MiB)
22/02/08 16:53:09 INFO Executor: Finished task 8.0 in stage 22.0 (TID 366). 1649 bytes result sent to driver
22/02/08 16:53:09 INFO TaskSetManager: Finished task 8.0 in stage 22.0 (TID 366) in 13697 ms on 192.168.1.37 (executor driver) (9/10)
22/02/08 16:53:09 INFO CodecPool: Got brand-new compressor [.gz]
22/02/08 16:53:10 INFO IteratorBasedQueueProducer: starting to buffer records
22/02/08 16:53:10 INFO BoundedInMemoryExecutor: starting consumer thread
22/02/08 16:53:10 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 155565 records.
22/02/08 16:53:10 INFO InternalParquetRecordReader: at row 0. reading next block
22/02/08 16:53:10 INFO CodecPool: Got brand-new decompressor [.gz]
22/02/08 16:53:10 INFO InternalParquetRecordReader: block read in memory in 41 ms. row count = 155565
22/02/08 16:53:17 INFO HoodieAsyncService: Waiting for next instant upto 10 seconds
22/02/08 16:53:19 INFO IteratorBasedQueueProducer: finished buffering records
22/02/08 16:53:19 INFO BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
22/02/08 16:53:23 INFO HoodieMergeHandle: MergeHandle for partitionPath 2022/1/25 fileID 0ecefe19-2553-49c1-a6f9-1d4e4ff8c413-0, took 13806 ms.
22/02/08 16:53:23 INFO MemoryStore: Block rdd_42_9 stored as values in memory (estimated size 395.0 B, free 955.9 MiB)
22/02/08 16:53:23 INFO BlockManagerInfo: Added rdd_42_9 in memory on 192.168.1.37:53051 (size: 395.0 B, free: 956.9 MiB)
22/02/08 16:53:23 INFO Executor: Finished task 9.0 in stage 22.0 (TID 367). 1649 bytes result sent to driver
22/02/08 16:53:23 INFO TaskSetManager: Finished task 9.0 in stage 22.0 (TID 367) in 13865 ms on 192.168.1.37 (executor driver) (10/10)
22/02/08 16:53:23 INFO TaskSchedulerImpl: Removed TaskSet 22.0, whose tasks have all completed, from pool
22/02/08 16:53:23 INFO DAGScheduler: ResultStage 22 (sum at DeltaSync.java:592) finished in 60.406 s
22/02/08 16:53:23 INFO DAGScheduler: Job 8 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:53:23 INFO TaskSchedulerImpl: Killing all running tasks in stage 22: Stage finished
22/02/08 16:53:23 INFO DAGScheduler: Job 8 finished: sum at DeltaSync.java:592, took 63.228720 s
22/02/08 16:53:23 INFO SparkContext: Starting job: sum at DeltaSync.java:593
22/02/08 16:53:23 INFO DAGScheduler: Got job 9 (sum at DeltaSync.java:593) with 10 output partitions
22/02/08 16:53:23 INFO DAGScheduler: Final stage: ResultStage 27 (sum at DeltaSync.java:593)
22/02/08 16:53:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 26)
22/02/08 16:53:23 INFO DAGScheduler: Missing parents: List()
22/02/08 16:53:23 INFO DAGScheduler: Submitting ResultStage 27 (MapPartitionsRDD[45] at mapToDouble at DeltaSync.java:593), which has no missing parents
22/02/08 16:53:23 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 663.1 KiB, free 955.2 MiB)
22/02/08 16:53:23 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 234.2 KiB, free 955.0 MiB)
22/02/08 16:53:23 INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on 192.168.1.37:53051 (size: 234.2 KiB, free: 956.7 MiB)
22/02/08 16:53:23 INFO SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1427
22/02/08 16:53:23 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 27 (MapPartitionsRDD[45] at mapToDouble at DeltaSync.java:593) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/08 16:53:23 INFO TaskSchedulerImpl: Adding task set 27.0 with 10 tasks resource profile 0
22/02/08 16:53:23 INFO TaskSetManager: Starting task 0.0 in stage 27.0 (TID 368) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:23 INFO TaskSetManager: Starting task 1.0 in stage 27.0 (TID 369) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:23 INFO TaskSetManager: Starting task 2.0 in stage 27.0 (TID 370) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:23 INFO Executor: Running task 1.0 in stage 27.0 (TID 369)
22/02/08 16:53:23 INFO Executor: Running task 0.0 in stage 27.0 (TID 368)
22/02/08 16:53:23 INFO Executor: Running task 2.0 in stage 27.0 (TID 370)
22/02/08 16:53:23 INFO BlockManager: Found block rdd_42_1 locally
22/02/08 16:53:23 INFO BlockManager: Found block rdd_42_0 locally
22/02/08 16:53:23 INFO Executor: Finished task 1.0 in stage 27.0 (TID 369). 1391 bytes result sent to driver
22/02/08 16:53:23 INFO BlockManager: Found block rdd_42_2 locally
22/02/08 16:53:23 INFO Executor: Finished task 0.0 in stage 27.0 (TID 368). 1348 bytes result sent to driver
22/02/08 16:53:23 INFO Executor: Finished task 2.0 in stage 27.0 (TID 370). 1348 bytes result sent to driver
22/02/08 16:53:23 INFO TaskSetManager: Starting task 3.0 in stage 27.0 (TID 371) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:23 INFO Executor: Running task 3.0 in stage 27.0 (TID 371)
22/02/08 16:53:23 INFO TaskSetManager: Starting task 4.0 in stage 27.0 (TID 372) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:23 INFO TaskSetManager: Starting task 5.0 in stage 27.0 (TID 373) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:23 INFO TaskSetManager: Finished task 2.0 in stage 27.0 (TID 370) in 59 ms on 192.168.1.37 (executor driver) (1/10)
22/02/08 16:53:23 INFO Executor: Running task 5.0 in stage 27.0 (TID 373)
22/02/08 16:53:23 INFO Executor: Running task 4.0 in stage 27.0 (TID 372)
22/02/08 16:53:23 INFO TaskSetManager: Finished task 1.0 in stage 27.0 (TID 369) in 59 ms on 192.168.1.37 (executor driver) (2/10)
22/02/08 16:53:23 INFO TaskSetManager: Finished task 0.0 in stage 27.0 (TID 368) in 59 ms on 192.168.1.37 (executor driver) (3/10)
22/02/08 16:53:23 INFO BlockManager: Found block rdd_42_3 locally
22/02/08 16:53:23 INFO Executor: Finished task 3.0 in stage 27.0 (TID 371). 1391 bytes result sent to driver
22/02/08 16:53:23 INFO TaskSetManager: Starting task 6.0 in stage 27.0 (TID 374) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:23 INFO TaskSetManager: Finished task 3.0 in stage 27.0 (TID 371) in 101 ms on 192.168.1.37 (executor driver) (4/10)
22/02/08 16:53:23 INFO Executor: Running task 6.0 in stage 27.0 (TID 374)
22/02/08 16:53:23 INFO BlockManager: Found block rdd_42_5 locally
22/02/08 16:53:23 INFO Executor: Finished task 5.0 in stage 27.0 (TID 373). 1391 bytes result sent to driver
22/02/08 16:53:23 INFO TaskSetManager: Starting task 7.0 in stage 27.0 (TID 375) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:23 INFO TaskSetManager: Finished task 5.0 in stage 27.0 (TID 373) in 103 ms on 192.168.1.37 (executor driver) (5/10)
22/02/08 16:53:23 INFO Executor: Running task 7.0 in stage 27.0 (TID 375)
22/02/08 16:53:23 INFO BlockManager: Found block rdd_42_4 locally
22/02/08 16:53:23 INFO Executor: Finished task 4.0 in stage 27.0 (TID 372). 1391 bytes result sent to driver
22/02/08 16:53:23 INFO TaskSetManager: Starting task 8.0 in stage 27.0 (TID 376) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:23 INFO TaskSetManager: Finished task 4.0 in stage 27.0 (TID 372) in 104 ms on 192.168.1.37 (executor driver) (6/10)
22/02/08 16:53:23 INFO Executor: Running task 8.0 in stage 27.0 (TID 376)
22/02/08 16:53:23 INFO BlockManager: Found block rdd_42_6 locally
22/02/08 16:53:23 INFO Executor: Finished task 6.0 in stage 27.0 (TID 374). 1348 bytes result sent to driver
22/02/08 16:53:23 INFO TaskSetManager: Starting task 9.0 in stage 27.0 (TID 377) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:23 INFO TaskSetManager: Finished task 6.0 in stage 27.0 (TID 374) in 40 ms on 192.168.1.37 (executor driver) (7/10)
22/02/08 16:53:23 INFO Executor: Running task 9.0 in stage 27.0 (TID 377)
22/02/08 16:53:23 INFO BlockManager: Found block rdd_42_7 locally
22/02/08 16:53:23 INFO Executor: Finished task 7.0 in stage 27.0 (TID 375). 1348 bytes result sent to driver
22/02/08 16:53:23 INFO TaskSetManager: Finished task 7.0 in stage 27.0 (TID 375) in 41 ms on 192.168.1.37 (executor driver) (8/10)
22/02/08 16:53:23 INFO BlockManager: Found block rdd_42_8 locally
22/02/08 16:53:23 INFO Executor: Finished task 8.0 in stage 27.0 (TID 376). 1348 bytes result sent to driver
22/02/08 16:53:23 INFO TaskSetManager: Finished task 8.0 in stage 27.0 (TID 376) in 42 ms on 192.168.1.37 (executor driver) (9/10)
22/02/08 16:53:23 INFO BlockManager: Found block rdd_42_9 locally
22/02/08 16:53:23 INFO Executor: Finished task 9.0 in stage 27.0 (TID 377). 1348 bytes result sent to driver
22/02/08 16:53:23 INFO TaskSetManager: Finished task 9.0 in stage 27.0 (TID 377) in 31 ms on 192.168.1.37 (executor driver) (10/10)
22/02/08 16:53:23 INFO TaskSchedulerImpl: Removed TaskSet 27.0, whose tasks have all completed, from pool
22/02/08 16:53:23 INFO DAGScheduler: ResultStage 27 (sum at DeltaSync.java:593) finished in 0.384 s
22/02/08 16:53:23 INFO DAGScheduler: Job 9 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:53:23 INFO TaskSchedulerImpl: Killing all running tasks in stage 27: Stage finished
22/02/08 16:53:23 INFO DAGScheduler: Job 9 finished: sum at DeltaSync.java:593, took 0.387454 s
22/02/08 16:53:24 INFO SparkContext: Starting job: collect at SparkRDDWriteClient.java:126
22/02/08 16:53:24 INFO DAGScheduler: Got job 10 (collect at SparkRDDWriteClient.java:126) with 10 output partitions
22/02/08 16:53:24 INFO DAGScheduler: Final stage: ResultStage 32 (collect at SparkRDDWriteClient.java:126)
22/02/08 16:53:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 31)
22/02/08 16:53:24 INFO DAGScheduler: Missing parents: List()
22/02/08 16:53:24 INFO DAGScheduler: Submitting ResultStage 32 (MapPartitionsRDD[47] at map at SparkRDDWriteClient.java:126), which has no missing parents
22/02/08 16:53:24 INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 663.0 KiB, free 954.4 MiB)
22/02/08 16:53:24 INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 234.1 KiB, free 954.1 MiB)
22/02/08 16:53:24 INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on 192.168.1.37:53051 (size: 234.1 KiB, free: 956.4 MiB)
22/02/08 16:53:24 INFO SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1427
22/02/08 16:53:24 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 32 (MapPartitionsRDD[47] at map at SparkRDDWriteClient.java:126) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
22/02/08 16:53:24 INFO TaskSchedulerImpl: Adding task set 32.0 with 10 tasks resource profile 0
22/02/08 16:53:24 INFO TaskSetManager: Starting task 0.0 in stage 32.0 (TID 378) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO TaskSetManager: Starting task 1.0 in stage 32.0 (TID 379) (192.168.1.37, executor driver, partition 1, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO TaskSetManager: Starting task 2.0 in stage 32.0 (TID 380) (192.168.1.37, executor driver, partition 2, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO Executor: Running task 0.0 in stage 32.0 (TID 378)
22/02/08 16:53:24 INFO Executor: Running task 2.0 in stage 32.0 (TID 380)
22/02/08 16:53:24 INFO Executor: Running task 1.0 in stage 32.0 (TID 379)
22/02/08 16:53:24 INFO BlockManager: Found block rdd_42_2 locally
22/02/08 16:53:24 INFO BlockManager: Found block rdd_42_0 locally
22/02/08 16:53:24 INFO Executor: Finished task 2.0 in stage 32.0 (TID 380). 1639 bytes result sent to driver
22/02/08 16:53:24 INFO Executor: Finished task 0.0 in stage 32.0 (TID 378). 1639 bytes result sent to driver
22/02/08 16:53:24 INFO TaskSetManager: Starting task 3.0 in stage 32.0 (TID 381) (192.168.1.37, executor driver, partition 3, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO Executor: Running task 3.0 in stage 32.0 (TID 381)
22/02/08 16:53:24 INFO TaskSetManager: Starting task 4.0 in stage 32.0 (TID 382) (192.168.1.37, executor driver, partition 4, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO TaskSetManager: Finished task 2.0 in stage 32.0 (TID 380) in 34 ms on 192.168.1.37 (executor driver) (1/10)
22/02/08 16:53:24 INFO Executor: Running task 4.0 in stage 32.0 (TID 382)
22/02/08 16:53:24 INFO TaskSetManager: Finished task 0.0 in stage 32.0 (TID 378) in 34 ms on 192.168.1.37 (executor driver) (2/10)
22/02/08 16:53:24 INFO BlockManager: Found block rdd_42_1 locally
22/02/08 16:53:24 INFO Executor: Finished task 1.0 in stage 32.0 (TID 379). 1639 bytes result sent to driver
22/02/08 16:53:24 INFO TaskSetManager: Starting task 5.0 in stage 32.0 (TID 383) (192.168.1.37, executor driver, partition 5, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO TaskSetManager: Finished task 1.0 in stage 32.0 (TID 379) in 35 ms on 192.168.1.37 (executor driver) (3/10)
22/02/08 16:53:24 INFO Executor: Running task 5.0 in stage 32.0 (TID 383)
22/02/08 16:53:24 INFO BlockManager: Found block rdd_42_4 locally
22/02/08 16:53:24 INFO BlockManager: Found block rdd_42_3 locally
22/02/08 16:53:24 INFO Executor: Finished task 4.0 in stage 32.0 (TID 382). 1637 bytes result sent to driver
22/02/08 16:53:24 INFO TaskSetManager: Starting task 6.0 in stage 32.0 (TID 384) (192.168.1.37, executor driver, partition 6, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO Executor: Finished task 3.0 in stage 32.0 (TID 381). 1639 bytes result sent to driver
22/02/08 16:53:24 INFO TaskSetManager: Finished task 4.0 in stage 32.0 (TID 382) in 31 ms on 192.168.1.37 (executor driver) (4/10)
22/02/08 16:53:24 INFO Executor: Running task 6.0 in stage 32.0 (TID 384)
22/02/08 16:53:24 INFO TaskSetManager: Starting task 7.0 in stage 32.0 (TID 385) (192.168.1.37, executor driver, partition 7, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO TaskSetManager: Finished task 3.0 in stage 32.0 (TID 381) in 33 ms on 192.168.1.37 (executor driver) (5/10)
22/02/08 16:53:24 INFO Executor: Running task 7.0 in stage 32.0 (TID 385)
22/02/08 16:53:24 INFO BlockManager: Found block rdd_42_5 locally
22/02/08 16:53:24 INFO Executor: Finished task 5.0 in stage 32.0 (TID 383). 1639 bytes result sent to driver
22/02/08 16:53:24 INFO TaskSetManager: Starting task 8.0 in stage 32.0 (TID 386) (192.168.1.37, executor driver, partition 8, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO Executor: Running task 8.0 in stage 32.0 (TID 386)
22/02/08 16:53:24 INFO TaskSetManager: Finished task 5.0 in stage 32.0 (TID 383) in 34 ms on 192.168.1.37 (executor driver) (6/10)
22/02/08 16:53:24 INFO BlockManager: Found block rdd_42_7 locally
22/02/08 16:53:24 INFO BlockManager: Found block rdd_42_6 locally
22/02/08 16:53:24 INFO Executor: Finished task 6.0 in stage 32.0 (TID 384). 1639 bytes result sent to driver
22/02/08 16:53:24 INFO Executor: Finished task 7.0 in stage 32.0 (TID 385). 1637 bytes result sent to driver
22/02/08 16:53:24 INFO TaskSetManager: Starting task 9.0 in stage 32.0 (TID 387) (192.168.1.37, executor driver, partition 9, PROCESS_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO Executor: Running task 9.0 in stage 32.0 (TID 387)
22/02/08 16:53:24 INFO TaskSetManager: Finished task 6.0 in stage 32.0 (TID 384) in 41 ms on 192.168.1.37 (executor driver) (7/10)
22/02/08 16:53:24 INFO TaskSetManager: Finished task 7.0 in stage 32.0 (TID 385) in 41 ms on 192.168.1.37 (executor driver) (8/10)
22/02/08 16:53:24 INFO BlockManager: Found block rdd_42_8 locally
22/02/08 16:53:24 INFO Executor: Finished task 8.0 in stage 32.0 (TID 386). 1639 bytes result sent to driver
22/02/08 16:53:24 INFO TaskSetManager: Finished task 8.0 in stage 32.0 (TID 386) in 39 ms on 192.168.1.37 (executor driver) (9/10)
22/02/08 16:53:24 INFO BlockManager: Found block rdd_42_9 locally
22/02/08 16:53:24 INFO Executor: Finished task 9.0 in stage 32.0 (TID 387). 1639 bytes result sent to driver
22/02/08 16:53:24 INFO TaskSetManager: Finished task 9.0 in stage 32.0 (TID 387) in 32 ms on 192.168.1.37 (executor driver) (10/10)
22/02/08 16:53:24 INFO TaskSchedulerImpl: Removed TaskSet 32.0, whose tasks have all completed, from pool
22/02/08 16:53:24 INFO DAGScheduler: ResultStage 32 (collect at SparkRDDWriteClient.java:126) finished in 0.245 s
22/02/08 16:53:24 INFO DAGScheduler: Job 10 is finished. Cancelling potential speculative or zombie tasks for this job
22/02/08 16:53:24 INFO TaskSchedulerImpl: Killing all running tasks in stage 32: Stage finished
22/02/08 16:53:24 INFO DAGScheduler: Job 10 finished: collect at SparkRDDWriteClient.java:126, took 0.247787 s
22/02/08 16:53:24 INFO BaseHoodieWriteClient: Committing 20220208165137552 action deltacommit
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__INFLIGHT]}
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:53:24 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:53:24 INFO CommitUtils: Creating metadata for UPSERT numWriteStats:10numReplaceFileIds:0
22/02/08 16:53:24 INFO TransactionManager: Transaction starting for Option{val=[==>20220208165137552__deltacommit__INFLIGHT]} with latest completed transaction instant Option{val=[20220208164721920__deltacommit__COMPLETED]}
22/02/08 16:53:24 INFO InProcessLockProvider: Thread pool-22-thread-1 ACQUIRING in-process lock.
22/02/08 16:53:24 INFO InProcessLockProvider: Thread pool-22-thread-1 ACQUIRED in-process lock.
22/02/08 16:53:24 INFO TransactionManager: Transaction started for Option{val=[==>20220208165137552__deltacommit__INFLIGHT]} with latest completed transaction instant Option{val=[20220208164721920__deltacommit__COMPLETED]}
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__INFLIGHT]}
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO FileSystemViewManager: Creating View Manager with storage type :REMOTE_FIRST
22/02/08 16:53:24 INFO FileSystemViewManager: Creating remote first table view
22/02/08 16:53:24 INFO TransactionUtils: Successfully resolved conflicts, if any
22/02/08 16:53:24 INFO BaseHoodieWriteClient: Committing 20220208165137552 action deltacommit
22/02/08 16:53:24 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/dir/exists?markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:53:24 INFO TimelineServerBasedWriteMarkers: Sending request : (http://192.168.1.37:53121/v1/hoodie/marker/create-and-merge?markerdirpath=file%3A%2FUsers%2Fethan%2FWork%2Fdata%2Fhudi%2Fmetadata_test_ds_mor_continuous_4%2F.hoodie%2F.temp%2F20220208165137552)
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830001__commit__COMPLETED]}
22/02/08 16:53:24 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:53:24 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:53:24 INFO HoodieTableMetadataUtil: Loading latest file slices for metadata table partition files
22/02/08 16:53:24 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:53:24 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:53:24 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:53:24 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830001__commit__COMPLETED]}
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__INFLIGHT]}
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableMetadataUtil: Updating at 20220208165137552 from Commit/UPSERT. #partitions_updated=11
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830001__commit__COMPLETED]}
22/02/08 16:53:24 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:53:24 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:53:24 INFO HoodieTableMetadataUtil: Loading latest file slices for metadata table partition files
22/02/08 16:53:24 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:53:24 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:53:24 INFO AbstractTableFileSystemView: Building file system view for partition (files)
22/02/08 16:53:24 INFO AbstractTableFileSystemView: addFilesToView: NumFiles=9, NumFileGroups=1, FileGroupsCreationTime=0, StoreTimeTaken=1
22/02/08 16:53:24 INFO BaseHoodieClient: Embedded Timeline Server is disabled. Not starting timeline service
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830001__commit__COMPLETED]}
22/02/08 16:53:24 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/08 16:53:24 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830001__commit__COMPLETED]}
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__INFLIGHT]}
22/02/08 16:53:24 INFO HoodieBackedTableMetadataWriter: Cannot compact metadata table as there are 1 inflight instants before latest deltacommit 20220208164926830: [[==>20220208164926747__deltacommit__INFLIGHT]]
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[20220208164926830001__commit__COMPLETED]}
22/02/08 16:53:24 INFO BaseHoodieWriteClient: Generate a new instant time: 20220208165137552 action: deltacommit
22/02/08 16:53:24 INFO HoodieHeartbeatClient: Received request to start heartbeat for instant time 20220208165137552
22/02/08 16:53:24 INFO HoodieActiveTimeline: Creating a new instant [==>20220208165137552__deltacommit__REQUESTED]
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableConfig: Loading table properties from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/.hoodie/hoodie.properties
22/02/08 16:53:24 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieTableMetaClient: Loading Active commit timeline for file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:53:24 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/02/08 16:53:24 INFO FileSystemViewManager: Creating in-memory based Table View
22/02/08 16:53:24 INFO FileSystemViewManager: Creating InMemory based view for basePath file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata
22/02/08 16:53:24 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:53:24 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:53:24 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220208165137552__deltacommit__REQUESTED]}
22/02/08 16:53:24 INFO AbstractTableFileSystemView: Took 0 ms to read 0 instants, 0 replaced file groups
22/02/08 16:53:24 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/02/08 16:53:24 WARN TableSchemaResolver: Failed to read operation field from avro schema
java.lang.IllegalArgumentException: Unknown file format :file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/files-0000_0-93-815_20220208164926830001.hfile
at org.apache.hudi.common.table.TableSchemaResolver.getTableParquetSchemaFromDataFile(TableSchemaResolver.java:103)
at org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaFromDataFile(TableSchemaResolver.java:119)
at org.apache.hudi.common.table.TableSchemaResolver.hasOperationField(TableSchemaResolver.java:480)
at org.apache.hudi.common.table.TableSchemaResolver.<init>(TableSchemaResolver.java:65)
at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:682)
at org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:698)
at org.apache.hudi.client.SparkRDDWriteClient.upsertPreppedRecords(SparkRDDWriteClient.java:173)
at org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.commit(SparkHoodieBackedTableMetadataWriter.java:154)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:663)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:675)
at org.apache.hudi.client.BaseHoodieWriteClient.lambda$writeTableMetadata$0(BaseHoodieWriteClient.java:273)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at org.apache.hudi.client.BaseHoodieWriteClient.writeTableMetadata(BaseHoodieWriteClient.java:273)
at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:229)
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:199)
at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:127)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:609)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:329)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:652)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
22/02/08 16:53:24 INFO AsyncCleanerService: Async auto cleaning is not enabled. Not running cleaner now
22/02/08 16:53:24 INFO SparkContext: Starting job: countByKey at BaseSparkCommitActionExecutor.java:196
22/02/08 16:53:24 INFO DAGScheduler: Registering RDD 53 (countByKey at BaseSparkCommitActionExecutor.java:196) as input to shuffle 7
22/02/08 16:53:24 INFO DAGScheduler: Got job 11 (countByKey at BaseSparkCommitActionExecutor.java:196) with 1 output partitions
22/02/08 16:53:24 INFO DAGScheduler: Final stage: ResultStage 34 (countByKey at BaseSparkCommitActionExecutor.java:196)
22/02/08 16:53:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 33)
22/02/08 16:53:24 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 33)
22/02/08 16:53:24 INFO DAGScheduler: Submitting ShuffleMapStage 33 (MapPartitionsRDD[53] at countByKey at BaseSparkCommitActionExecutor.java:196), which has no missing parents
22/02/08 16:53:24 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 9.8 KiB, free 954.1 MiB)
22/02/08 16:53:24 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 5.3 KiB, free 954.1 MiB)
22/02/08 16:53:24 INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on 192.168.1.37:53051 (size: 5.3 KiB, free: 956.4 MiB)
22/02/08 16:53:24 INFO SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1427
22/02/08 16:53:24 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 33 (MapPartitionsRDD[53] at countByKey at BaseSparkCommitActionExecutor.java:196) (first 15 tasks are for partitions Vector(0))
22/02/08 16:53:24 INFO TaskSchedulerImpl: Adding task set 33.0 with 1 tasks resource profile 0
22/02/08 16:53:24 INFO TaskSetManager: Starting task 0.0 in stage 33.0 (TID 388) (192.168.1.37, executor driver, partition 0, PROCESS_LOCAL, 5867 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO Executor: Running task 0.0 in stage 33.0 (TID 388)
22/02/08 16:53:24 INFO MemoryStore: Block rdd_51_0 stored as values in memory (estimated size 4.0 KiB, free 954.1 MiB)
22/02/08 16:53:24 INFO BlockManagerInfo: Added rdd_51_0 in memory on 192.168.1.37:53051 (size: 4.0 KiB, free: 956.4 MiB)
22/02/08 16:53:24 INFO Executor: Finished task 0.0 in stage 33.0 (TID 388). 1086 bytes result sent to driver
22/02/08 16:53:24 INFO TaskSetManager: Finished task 0.0 in stage 33.0 (TID 388) in 17 ms on 192.168.1.37 (executor driver) (1/1)
22/02/08 16:53:24 INFO TaskSchedulerImpl: Removed TaskSet 33.0, whose tasks have all completed, from pool
22/02/08 16:53:24 INFO DAGScheduler: ShuffleMapStage 33 (countByKey at BaseSparkCommitActionExecutor.java:196) finished in 0.022 s
22/02/08 16:53:24 INFO DAGScheduler: looking for newly runnable stages
22/02/08 16:53:24 INFO DAGScheduler: running: Set()
22/02/08 16:53:24 INFO DAGScheduler: waiting: Set(ResultStage 34)
22/02/08 16:53:24 INFO DAGScheduler: failed: Set()
22/02/08 16:53:24 INFO DAGScheduler: Submitting ResultStage 34 (ShuffledRDD[54] at countByKey at BaseSparkCommitActionExecutor.java:196), which has no missing parents
22/02/08 16:53:24 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 5.6 KiB, free 954.1 MiB)
22/02/08 16:53:24 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 954.1 MiB)
22/02/08 16:53:24 INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on 192.168.1.37:53051 (size: 3.2 KiB, free: 956.4 MiB)
22/02/08 16:53:24 INFO SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1427
22/02/08 16:53:24 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 34 (ShuffledRDD[54] at countByKey at BaseSparkCommitActionExecutor.java:196) (first 15 tasks are for partitions Vector(0))
22/02/08 16:53:24 INFO TaskSchedulerImpl: Adding task set 34.0 with 1 tasks resource profile 0
22/02/08 16:53:24 INFO TaskSetManager: Starting task 0.0 in stage 34.0 (TID 389) (192.168.1.37, executor driver, partition 0, NODE_LOCAL, 4271 bytes) taskResourceAssignments Map()
22/02/08 16:53:24 INFO Executor: Running task 0.0 in stage 34.0 (TID 389)
22/02/08 16:53:24 INFO ShuffleBlockFetcherIterator: Getting 1 (156.0 B) non-empty blocks including 1 (156.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
22/02/08 16:53:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
22/02/08 16:53:24 INFO Executor: Finished task 0.0 in stage 34.0 (TID 389). 1318 bytes result sent to driver
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment