Skip to content

Instantly share code, notes, and snippets.

@lukemarsden

lukemarsden/gist:54d3b9ec3e60c04fd3a03e13bee5921f

Created August 12, 2022 14:41
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lukemarsden/54d3b9ec3e60c04fd3a03e13bee5921f to your computer and use it in GitHub Desktop.
Save lukemarsden/54d3b9ec3e60c04fd3a03e13bee5921f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
This file has been truncated, but you can view the full file.
22/08/12 15:38:16 WARN Utils: Your hostname, mind resolves to a loopback address: 127.0.1.1; using 10.1.255.235 instead (on interface enp6s0f0)
22/08/12 15:38:16 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
22/08/12 15:38:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/08/12 15:38:17 INFO SparkContext: Running Spark version 3.3.0
22/08/12 15:38:17 INFO ResourceUtils: ==============================================================
22/08/12 15:38:17 INFO ResourceUtils: No custom resources configured for spark.driver.
22/08/12 15:38:17 INFO ResourceUtils: ==============================================================
22/08/12 15:38:17 INFO SparkContext: Submitted application: spark.py
22/08/12 15:38:17 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/08/12 15:38:17 INFO ResourceProfile: Limiting resource is cpu
22/08/12 15:38:17 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/08/12 15:38:17 INFO SecurityManager: Changing view acls to: luke
22/08/12 15:38:17 INFO SecurityManager: Changing modify acls to: luke
22/08/12 15:38:17 INFO SecurityManager: Changing view acls groups to:
22/08/12 15:38:17 INFO SecurityManager: Changing modify acls groups to:
22/08/12 15:38:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(luke); groups with view permissions: Set(); users with modify permissions: Set(luke); groups with modify permissions: Set()
22/08/12 15:38:17 INFO Utils: Successfully started service 'sparkDriver' on port 45957.
22/08/12 15:38:17 INFO SparkEnv: Registering MapOutputTracker
22/08/12 15:38:17 INFO SparkEnv: Registering BlockManagerMaster
22/08/12 15:38:17 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/08/12 15:38:17 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/08/12 15:38:17 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
22/08/12 15:38:17 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-9174e306-0f09-40d9-abef-26535038caf6
22/08/12 15:38:17 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
22/08/12 15:38:17 INFO SparkEnv: Registering OutputCommitCoordinator
22/08/12 15:38:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/08/12 15:38:18 INFO SparkContext: Added JAR file:///home/luke/pp/pachyderm/spark/hadoop-aws-3.3.3.jar at spark://10.1.255.235:45957/jars/hadoop-aws-3.3.3.jar with timestamp 1660315097534
22/08/12 15:38:18 INFO SparkContext: Added JAR file:///home/luke/pp/pachyderm/spark/aws-java-sdk-bundle-1.12.264.jar at spark://10.1.255.235:45957/jars/aws-java-sdk-bundle-1.12.264.jar with timestamp 1660315097534
22/08/12 15:38:18 INFO Executor: Starting executor ID driver on host 10.1.255.235
22/08/12 15:38:18 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
22/08/12 15:38:18 INFO Executor: Fetching spark://10.1.255.235:45957/jars/hadoop-aws-3.3.3.jar with timestamp 1660315097534
22/08/12 15:38:18 INFO TransportClientFactory: Successfully created connection to /10.1.255.235:45957 after 26 ms (0 ms spent in bootstraps)
22/08/12 15:38:18 INFO Utils: Fetching spark://10.1.255.235:45957/jars/hadoop-aws-3.3.3.jar to /tmp/spark-0e285136-17be-4fa8-b05d-f3dfea3d2f9c/userFiles-df24a7ce-4811-4269-9e12-0cb56123479d/fetchFileTemp11122189119415078725.tmp
22/08/12 15:38:18 INFO Executor: Adding file:/tmp/spark-0e285136-17be-4fa8-b05d-f3dfea3d2f9c/userFiles-df24a7ce-4811-4269-9e12-0cb56123479d/hadoop-aws-3.3.3.jar to class loader
22/08/12 15:38:18 INFO Executor: Fetching spark://10.1.255.235:45957/jars/aws-java-sdk-bundle-1.12.264.jar with timestamp 1660315097534
22/08/12 15:38:18 INFO Utils: Fetching spark://10.1.255.235:45957/jars/aws-java-sdk-bundle-1.12.264.jar to /tmp/spark-0e285136-17be-4fa8-b05d-f3dfea3d2f9c/userFiles-df24a7ce-4811-4269-9e12-0cb56123479d/fetchFileTemp15273113245366153136.tmp
22/08/12 15:38:18 INFO Executor: Adding file:/tmp/spark-0e285136-17be-4fa8-b05d-f3dfea3d2f9c/userFiles-df24a7ce-4811-4269-9e12-0cb56123479d/aws-java-sdk-bundle-1.12.264.jar to class loader
22/08/12 15:38:18 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33757.
22/08/12 15:38:18 INFO NettyBlockTransferService: Server created on 10.1.255.235:33757
22/08/12 15:38:18 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/08/12 15:38:18 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.1.255.235, 33757, None)
22/08/12 15:38:18 INFO BlockManagerMasterEndpoint: Registering block manager 10.1.255.235:33757 with 434.4 MiB RAM, BlockManagerId(driver, 10.1.255.235, 33757, None)
22/08/12 15:38:18 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.1.255.235, 33757, None)
22/08/12 15:38:18 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.1.255.235, 33757, None)
[('spark.driver.extraJavaOptions', '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'), ('spark.hadoop.fs.s3a.connection.ssl.enabled', 'false'), ('spark.driver.host', '10.1.255.235'), ('spark.hadoop.fs.s3a.path.style.access', 'true'), ('spark.repl.local.jars', 'file:///home/luke/pp/pachyderm/spark/hadoop-aws-3.3.3.jar,file:///home/luke/pp/pachyderm/spark/aws-java-sdk-bundle-1.12.264.jar'), ('spark.executor.id', 'driver'), ('spark.hadoop.fs.s3a.change.detection.mode', 'none'), ('spark.app.submitTime', '1660315097058'), ('spark.app.initial.jar.urls', 'spark://10.1.255.235:45957/jars/hadoop-aws-3.3.3.jar,spark://10.1.255.235:45957/jars/aws-java-sdk-bundle-1.12.264.jar'), ('spark.jars', 'file:///home/luke/pp/pachyderm/spark/hadoop-aws-3.3.3.jar,file:///home/luke/pp/pachyderm/spark/aws-java-sdk-bundle-1.12.264.jar'), ('spark.hadoop.fs.s3a.impl', 'org.apache.hadoop.fs.s3a.S3AFileSystem'), ('spark.hadoop.fs.s3a.change.detection.version.required', 'false'), ('spark.app.name', 'spark.py'), ('spark.rdd.compress', 'True'), ('spark.hadoop.fs.s3a.endpoint', 'http://localhost:30600'), ('spark.app.startTime', '1660315097534'), ('spark.executor.extraJavaOptions', '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'), ('spark.hadoop.fs.s3a.access.key', 'lemon'), ('spark.serializer.objectStreamReset', '100'), ('spark.master', 'local[*]'), ('spark.submit.pyFiles', ''), ('spark.driver.port', '45957'), ('spark.hadoop.fs.s3a.secret.key', 'lemon'), ('spark.submit.deployMode', 'client'), ('spark.app.id', 'local-1660315098218')]
22/08/12 15:38:19 DEBUG FileSystem: Looking for FS supporting file
22/08/12 15:38:19 DEBUG FileSystem: looking for configuration option fs.file.impl
22/08/12 15:38:19 DEBUG FileSystem: Looking in service filesystems for implementation class
22/08/12 15:38:19 DEBUG FileSystem: FS for file is class org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem
22/08/12 15:38:19 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
22/08/12 15:38:19 INFO SharedState: Warehouse path is 'file:/home/luke/pp/pachyderm/spark/spark-warehouse'.
22/08/12 15:38:19 DEBUG FsUrlStreamHandlerFactory: Creating handler for protocol jar
22/08/12 15:38:19 DEBUG FileSystem: Looking for FS supporting jar
22/08/12 15:38:19 DEBUG FileSystem: looking for configuration option fs.jar.impl
22/08/12 15:38:19 DEBUG FileSystem: Looking in service filesystems for implementation class
22/08/12 15:38:19 DEBUG FsUrlStreamHandlerFactory: Unknown protocol jar, delegating to default implementation
22/08/12 15:38:20 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$pythonToJava$1
22/08/12 15:38:20 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$pythonToJava$1) is now cleaned +++
22/08/12 15:38:20 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$toJavaArray$1
22/08/12 15:38:20 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$toJavaArray$1) is now cleaned +++
22/08/12 15:38:20 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$applySchemaToPythonRDD$1
22/08/12 15:38:20 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$applySchemaToPythonRDD$1) is now cleaned +++
22/08/12 15:38:20 DEBUG CatalystSqlParser: Parsing command: spark_grouping_id
22/08/12 15:38:21 DEBUG WholeStageCodegenExec:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private scala.collection.Iterator rdd_input_0;
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[2];
/* 011 */
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) {
/* 013 */ this.references = references;
/* 014 */ }
/* 015 */
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) {
/* 017 */ partitionIndex = index;
/* 018 */ this.inputs = inputs;
/* 019 */ rdd_input_0 = inputs[0];
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0);
/* 021 */ rdd_mutableStateArray_0[1] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 64);
/* 022 */
/* 023 */ }
/* 024 */
/* 025 */ protected void processNext() throws java.io.IOException {
/* 026 */ while ( rdd_input_0.hasNext()) {
/* 027 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next();
/* 028 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 029 */ // common sub-expressions
/* 030 */
/* 031 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0);
/* 032 */ long rdd_value_0 = rdd_isNull_0 ?
/* 033 */ -1L : (rdd_row_0.getLong(0));
/* 034 */ boolean project_isNull_0 = rdd_isNull_0;
/* 035 */ UTF8String project_value_0 = null;
/* 036 */ if (!rdd_isNull_0) {
/* 037 */ project_value_0 = UTF8String.fromString(String.valueOf(rdd_value_0));
/* 038 */ }
/* 039 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1);
/* 040 */ double rdd_value_1 = rdd_isNull_1 ?
/* 041 */ -1.0 : (rdd_row_0.getDouble(1));
/* 042 */ boolean project_isNull_2 = rdd_isNull_1;
/* 043 */ UTF8String project_value_2 = null;
/* 044 */ if (!rdd_isNull_1) {
/* 045 */ project_value_2 = UTF8String.fromString(String.valueOf(rdd_value_1));
/* 046 */ }
/* 047 */ rdd_mutableStateArray_0[1].reset();
/* 048 */
/* 049 */ rdd_mutableStateArray_0[1].zeroOutNullBytes();
/* 050 */
/* 051 */ if (project_isNull_0) {
/* 052 */ rdd_mutableStateArray_0[1].setNullAt(0);
/* 053 */ } else {
/* 054 */ rdd_mutableStateArray_0[1].write(0, project_value_0);
/* 055 */ }
/* 056 */
/* 057 */ if (project_isNull_2) {
/* 058 */ rdd_mutableStateArray_0[1].setNullAt(1);
/* 059 */ } else {
/* 060 */ rdd_mutableStateArray_0[1].write(1, project_value_2);
/* 061 */ }
/* 062 */ append((rdd_mutableStateArray_0[1].getRow()));
/* 063 */ if (shouldStop()) return;
/* 064 */ }
/* 065 */ }
/* 066 */
/* 067 */ }
22/08/12 15:38:21 DEBUG CodeGenerator:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private scala.collection.Iterator rdd_input_0;
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[2];
/* 011 */
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) {
/* 013 */ this.references = references;
/* 014 */ }
/* 015 */
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) {
/* 017 */ partitionIndex = index;
/* 018 */ this.inputs = inputs;
/* 019 */ rdd_input_0 = inputs[0];
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0);
/* 021 */ rdd_mutableStateArray_0[1] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 64);
/* 022 */
/* 023 */ }
/* 024 */
/* 025 */ protected void processNext() throws java.io.IOException {
/* 026 */ while ( rdd_input_0.hasNext()) {
/* 027 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next();
/* 028 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 029 */ // common sub-expressions
/* 030 */
/* 031 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0);
/* 032 */ long rdd_value_0 = rdd_isNull_0 ?
/* 033 */ -1L : (rdd_row_0.getLong(0));
/* 034 */ boolean project_isNull_0 = rdd_isNull_0;
/* 035 */ UTF8String project_value_0 = null;
/* 036 */ if (!rdd_isNull_0) {
/* 037 */ project_value_0 = UTF8String.fromString(String.valueOf(rdd_value_0));
/* 038 */ }
/* 039 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1);
/* 040 */ double rdd_value_1 = rdd_isNull_1 ?
/* 041 */ -1.0 : (rdd_row_0.getDouble(1));
/* 042 */ boolean project_isNull_2 = rdd_isNull_1;
/* 043 */ UTF8String project_value_2 = null;
/* 044 */ if (!rdd_isNull_1) {
/* 045 */ project_value_2 = UTF8String.fromString(String.valueOf(rdd_value_1));
/* 046 */ }
/* 047 */ rdd_mutableStateArray_0[1].reset();
/* 048 */
/* 049 */ rdd_mutableStateArray_0[1].zeroOutNullBytes();
/* 050 */
/* 051 */ if (project_isNull_0) {
/* 052 */ rdd_mutableStateArray_0[1].setNullAt(0);
/* 053 */ } else {
/* 054 */ rdd_mutableStateArray_0[1].write(0, project_value_0);
/* 055 */ }
/* 056 */
/* 057 */ if (project_isNull_2) {
/* 058 */ rdd_mutableStateArray_0[1].setNullAt(1);
/* 059 */ } else {
/* 060 */ rdd_mutableStateArray_0[1].write(1, project_value_2);
/* 061 */ }
/* 062 */ append((rdd_mutableStateArray_0[1].getRow()));
/* 063 */ if (shouldStop()) return;
/* 064 */ }
/* 065 */ }
/* 066 */
/* 067 */ }
22/08/12 15:38:22 INFO CodeGenerator: Code generated in 160.653731 ms
22/08/12 15:38:22 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$doExecute$4$adapted
22/08/12 15:38:22 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$doExecute$4$adapted) is now cleaned +++
22/08/12 15:38:22 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$executeTake$2
22/08/12 15:38:22 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$executeTake$2) is now cleaned +++
22/08/12 15:38:22 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$runJob$5
22/08/12 15:38:22 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$runJob$5) is now cleaned +++
22/08/12 15:38:22 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0
22/08/12 15:38:22 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 6 took 0.000961 seconds
22/08/12 15:38:22 DEBUG DAGScheduler: Merging stage rdd profiles: Set()
22/08/12 15:38:22 INFO DAGScheduler: Got job 0 (showString at NativeMethodAccessorImpl.java:0) with 1 output partitions
22/08/12 15:38:22 INFO DAGScheduler: Final stage: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0)
22/08/12 15:38:22 INFO DAGScheduler: Parents of final stage: List()
22/08/12 15:38:22 INFO DAGScheduler: Missing parents: List()
22/08/12 15:38:22 DEBUG DAGScheduler: submitStage(ResultStage 0 (name=showString at NativeMethodAccessorImpl.java:0;jobs=0))
22/08/12 15:38:22 DEBUG DAGScheduler: missing: List()
22/08/12 15:38:22 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents
22/08/12 15:38:22 DEBUG DAGScheduler: submitMissingTasks(ResultStage 0)
22/08/12 15:38:22 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 12.4 KiB, free 434.4 MiB)
22/08/12 15:38:22 DEBUG BlockManager: Put block broadcast_0 locally took 31 ms
22/08/12 15:38:22 DEBUG BlockManager: Putting block broadcast_0 without replication took 33 ms
22/08/12 15:38:22 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 6.6 KiB, free 434.4 MiB)
22/08/12 15:38:22 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_0_piece0 for BlockManagerId(driver, 10.1.255.235, 33757, None)
22/08/12 15:38:22 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.1.255.235:33757 (size: 6.6 KiB, free: 434.4 MiB)
22/08/12 15:38:22 DEBUG BlockManagerMaster: Updated info of block broadcast_0_piece0
22/08/12 15:38:22 DEBUG BlockManager: Told master about block broadcast_0_piece0
22/08/12 15:38:22 DEBUG BlockManager: Put block broadcast_0_piece0 locally took 10 ms
22/08/12 15:38:22 DEBUG BlockManager: Putting block broadcast_0_piece0 without replication took 10 ms
22/08/12 15:38:22 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1513
22/08/12 15:38:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
22/08/12 15:38:22 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0
22/08/12 15:38:22 DEBUG TaskSetManager: Epoch for TaskSet 0.0: 0
22/08/12 15:38:22 DEBUG TaskSetManager: Adding pending tasks took 1 ms
22/08/12 15:38:22 DEBUG TaskSetManager: Valid locality levels for TaskSet 0.0: NO_PREF, ANY
22/08/12 15:38:22 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_0.0, runningTasks: 0
22/08/12 15:38:22 DEBUG TaskSetManager: Valid locality levels for TaskSet 0.0: NO_PREF, ANY
22/08/12 15:38:22 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (10.1.255.235, executor driver, partition 0, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:22 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY
22/08/12 15:38:22 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
22/08/12 15:38:22 DEBUG ExecutorMetricsPoller: stageTCMP: (0, 0) -> 1
22/08/12 15:38:22 DEBUG BlockManager: Getting local block broadcast_0
22/08/12 15:38:22 DEBUG BlockManager: Level for block broadcast_0 is StorageLevel(disk, memory, deserialized, 1 replicas)
22/08/12 15:38:22 INFO PythonRunner: Times: total = 418, boot = 361, init = 57, finish = 0
22/08/12 15:38:22 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1789 bytes result sent to driver
22/08/12 15:38:22 DEBUG ExecutorMetricsPoller: stageTCMP: (0, 0) -> 0
22/08/12 15:38:22 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 567 ms on 10.1.255.235 (executor driver) (1/1)
22/08/12 15:38:22 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
22/08/12 15:38:22 INFO PythonAccumulatorV2: Connected to AccumulatorServer at host: 127.0.0.1 port: 51953
22/08/12 15:38:22 INFO DAGScheduler: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0) finished in 0.770 s
22/08/12 15:38:22 DEBUG DAGScheduler: After removal of stage 0, remaining stages = 0
22/08/12 15:38:22 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/12 15:38:22 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
22/08/12 15:38:22 INFO DAGScheduler: Job 0 finished: showString at NativeMethodAccessorImpl.java:0, took 0.820238 s
22/08/12 15:38:22 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$executeTake$2
22/08/12 15:38:22 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$executeTake$2) is now cleaned +++
22/08/12 15:38:22 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$runJob$5
22/08/12 15:38:22 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$runJob$5) is now cleaned +++
22/08/12 15:38:22 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0
22/08/12 15:38:22 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 6 took 0.000145 seconds
22/08/12 15:38:22 DEBUG DAGScheduler: Merging stage rdd profiles: Set()
22/08/12 15:38:22 INFO DAGScheduler: Got job 1 (showString at NativeMethodAccessorImpl.java:0) with 4 output partitions
22/08/12 15:38:22 INFO DAGScheduler: Final stage: ResultStage 1 (showString at NativeMethodAccessorImpl.java:0)
22/08/12 15:38:22 INFO DAGScheduler: Parents of final stage: List()
22/08/12 15:38:22 INFO DAGScheduler: Missing parents: List()
22/08/12 15:38:22 DEBUG DAGScheduler: submitStage(ResultStage 1 (name=showString at NativeMethodAccessorImpl.java:0;jobs=1))
22/08/12 15:38:22 DEBUG DAGScheduler: missing: List()
22/08/12 15:38:22 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents
22/08/12 15:38:22 DEBUG DAGScheduler: submitMissingTasks(ResultStage 1)
22/08/12 15:38:22 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 12.4 KiB, free 434.4 MiB)
22/08/12 15:38:22 DEBUG BlockManager: Put block broadcast_1 locally took 2 ms
22/08/12 15:38:22 DEBUG BlockManager: Putting block broadcast_1 without replication took 2 ms
22/08/12 15:38:22 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.6 KiB, free 434.4 MiB)
22/08/12 15:38:22 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_1_piece0 for BlockManagerId(driver, 10.1.255.235, 33757, None)
22/08/12 15:38:22 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.1.255.235:33757 (size: 6.6 KiB, free: 434.4 MiB)
22/08/12 15:38:22 DEBUG BlockManagerMaster: Updated info of block broadcast_1_piece0
22/08/12 15:38:22 DEBUG BlockManager: Told master about block broadcast_1_piece0
22/08/12 15:38:22 DEBUG BlockManager: Put block broadcast_1_piece0 locally took 2 ms
22/08/12 15:38:22 DEBUG BlockManager: Putting block broadcast_1_piece0 without replication took 3 ms
22/08/12 15:38:22 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1513
22/08/12 15:38:22 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(1, 2, 3, 4))
22/08/12 15:38:22 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks resource profile 0
22/08/12 15:38:22 DEBUG TaskSetManager: Epoch for TaskSet 1.0: 0
22/08/12 15:38:22 DEBUG TaskSetManager: Adding pending tasks took 0 ms
22/08/12 15:38:22 DEBUG TaskSetManager: Valid locality levels for TaskSet 1.0: NO_PREF, ANY
22/08/12 15:38:22 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_1.0, runningTasks: 0
22/08/12 15:38:22 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1) (10.1.255.235, executor driver, partition 1, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:22 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2) (10.1.255.235, executor driver, partition 2, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:22 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3) (10.1.255.235, executor driver, partition 3, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:22 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 4) (10.1.255.235, executor driver, partition 4, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:22 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY
22/08/12 15:38:22 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
22/08/12 15:38:22 INFO Executor: Running task 2.0 in stage 1.0 (TID 3)
22/08/12 15:38:22 INFO Executor: Running task 1.0 in stage 1.0 (TID 2)
22/08/12 15:38:22 INFO Executor: Running task 3.0 in stage 1.0 (TID 4)
22/08/12 15:38:22 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 1
22/08/12 15:38:22 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 2
22/08/12 15:38:22 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 3
22/08/12 15:38:22 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 4
22/08/12 15:38:22 DEBUG BlockManager: Getting local block broadcast_1
22/08/12 15:38:22 DEBUG BlockManager: Level for block broadcast_1 is StorageLevel(disk, memory, deserialized, 1 replicas)
22/08/12 15:38:22 INFO PythonRunner: Times: total = 65, boot = -86, init = 151, finish = 0
22/08/12 15:38:22 INFO Executor: Finished task 2.0 in stage 1.0 (TID 3). 1789 bytes result sent to driver
22/08/12 15:38:22 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 3
22/08/12 15:38:22 INFO PythonRunner: Times: total = 61, boot = 7, init = 54, finish = 0
22/08/12 15:38:22 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 3) in 84 ms on 10.1.255.235 (executor driver) (1/4)
22/08/12 15:38:22 INFO PythonRunner: Times: total = 67, boot = 3, init = 64, finish = 0
22/08/12 15:38:22 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1789 bytes result sent to driver
22/08/12 15:38:22 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 2
22/08/12 15:38:22 INFO Executor: Finished task 1.0 in stage 1.0 (TID 2). 1789 bytes result sent to driver
22/08/12 15:38:22 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 1
22/08/12 15:38:22 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 91 ms on 10.1.255.235 (executor driver) (2/4)
22/08/12 15:38:22 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 92 ms on 10.1.255.235 (executor driver) (3/4)
22/08/12 15:38:23 INFO PythonRunner: Times: total = 85, boot = 4, init = 81, finish = 0
22/08/12 15:38:23 INFO Executor: Finished task 3.0 in stage 1.0 (TID 4). 1789 bytes result sent to driver
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 0
22/08/12 15:38:23 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 4) in 109 ms on 10.1.255.235 (executor driver) (4/4)
22/08/12 15:38:23 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
22/08/12 15:38:23 INFO DAGScheduler: ResultStage 1 (showString at NativeMethodAccessorImpl.java:0) finished in 0.127 s
22/08/12 15:38:23 DEBUG DAGScheduler: After removal of stage 1, remaining stages = 0
22/08/12 15:38:23 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/12 15:38:23 INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished
22/08/12 15:38:23 INFO DAGScheduler: Job 1 finished: showString at NativeMethodAccessorImpl.java:0, took 0.136228 s
22/08/12 15:38:23 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$executeTake$2
22/08/12 15:38:23 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$executeTake$2) is now cleaned +++
22/08/12 15:38:23 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$runJob$5
22/08/12 15:38:23 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$runJob$5) is now cleaned +++
22/08/12 15:38:23 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0
22/08/12 15:38:23 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 6 took 0.000218 seconds
22/08/12 15:38:23 DEBUG DAGScheduler: Merging stage rdd profiles: Set()
22/08/12 15:38:23 INFO DAGScheduler: Got job 2 (showString at NativeMethodAccessorImpl.java:0) with 11 output partitions
22/08/12 15:38:23 INFO DAGScheduler: Final stage: ResultStage 2 (showString at NativeMethodAccessorImpl.java:0)
22/08/12 15:38:23 INFO DAGScheduler: Parents of final stage: List()
22/08/12 15:38:23 INFO DAGScheduler: Missing parents: List()
22/08/12 15:38:23 DEBUG DAGScheduler: submitStage(ResultStage 2 (name=showString at NativeMethodAccessorImpl.java:0;jobs=2))
22/08/12 15:38:23 DEBUG DAGScheduler: missing: List()
22/08/12 15:38:23 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents
22/08/12 15:38:23 DEBUG DAGScheduler: submitMissingTasks(ResultStage 2)
22/08/12 15:38:23 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 12.4 KiB, free 434.4 MiB)
22/08/12 15:38:23 DEBUG BlockManager: Put block broadcast_2 locally took 1 ms
22/08/12 15:38:23 DEBUG BlockManager: Putting block broadcast_2 without replication took 1 ms
22/08/12 15:38:23 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 6.6 KiB, free 434.3 MiB)
22/08/12 15:38:23 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_2_piece0 for BlockManagerId(driver, 10.1.255.235, 33757, None)
22/08/12 15:38:23 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.1.255.235:33757 (size: 6.6 KiB, free: 434.4 MiB)
22/08/12 15:38:23 DEBUG BlockManagerMaster: Updated info of block broadcast_2_piece0
22/08/12 15:38:23 DEBUG BlockManager: Told master about block broadcast_2_piece0
22/08/12 15:38:23 DEBUG BlockManager: Put block broadcast_2_piece0 locally took 1 ms
22/08/12 15:38:23 DEBUG BlockManager: Putting block broadcast_2_piece0 without replication took 1 ms
22/08/12 15:38:23 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1513
22/08/12 15:38:23 INFO DAGScheduler: Submitting 11 missing tasks from ResultStage 2 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15))
22/08/12 15:38:23 INFO TaskSchedulerImpl: Adding task set 2.0 with 11 tasks resource profile 0
22/08/12 15:38:23 DEBUG TaskSetManager: Epoch for TaskSet 2.0: 0
22/08/12 15:38:23 DEBUG TaskSetManager: Adding pending tasks took 0 ms
22/08/12 15:38:23 DEBUG TaskSetManager: Valid locality levels for TaskSet 2.0: NO_PREF, ANY
22/08/12 15:38:23 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_2.0, runningTasks: 0
22/08/12 15:38:23 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 5) (10.1.255.235, executor driver, partition 5, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 6) (10.1.255.235, executor driver, partition 6, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 7) (10.1.255.235, executor driver, partition 7, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 8) (10.1.255.235, executor driver, partition 8, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 INFO TaskSetManager: Starting task 4.0 in stage 2.0 (TID 9) (10.1.255.235, executor driver, partition 9, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 INFO TaskSetManager: Starting task 5.0 in stage 2.0 (TID 10) (10.1.255.235, executor driver, partition 10, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 INFO TaskSetManager: Starting task 6.0 in stage 2.0 (TID 11) (10.1.255.235, executor driver, partition 11, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 INFO TaskSetManager: Starting task 7.0 in stage 2.0 (TID 12) (10.1.255.235, executor driver, partition 12, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 INFO TaskSetManager: Starting task 8.0 in stage 2.0 (TID 13) (10.1.255.235, executor driver, partition 13, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 INFO TaskSetManager: Starting task 9.0 in stage 2.0 (TID 14) (10.1.255.235, executor driver, partition 14, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 INFO TaskSetManager: Starting task 10.0 in stage 2.0 (TID 15) (10.1.255.235, executor driver, partition 15, PROCESS_LOCAL, 4471 bytes) taskResourceAssignments Map()
22/08/12 15:38:23 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY
22/08/12 15:38:23 INFO Executor: Running task 0.0 in stage 2.0 (TID 5)
22/08/12 15:38:23 INFO Executor: Running task 1.0 in stage 2.0 (TID 6)
22/08/12 15:38:23 INFO Executor: Running task 2.0 in stage 2.0 (TID 7)
22/08/12 15:38:23 INFO Executor: Running task 3.0 in stage 2.0 (TID 8)
22/08/12 15:38:23 INFO Executor: Running task 4.0 in stage 2.0 (TID 9)
22/08/12 15:38:23 INFO Executor: Running task 5.0 in stage 2.0 (TID 10)
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 1
22/08/12 15:38:23 INFO Executor: Running task 6.0 in stage 2.0 (TID 11)
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 2
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 5
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 3
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 4
22/08/12 15:38:23 INFO Executor: Running task 7.0 in stage 2.0 (TID 12)
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 6
22/08/12 15:38:23 INFO Executor: Running task 8.0 in stage 2.0 (TID 13)
22/08/12 15:38:23 INFO Executor: Running task 9.0 in stage 2.0 (TID 14)
22/08/12 15:38:23 INFO Executor: Running task 10.0 in stage 2.0 (TID 15)
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 7
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 8
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 9
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 10
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 11
22/08/12 15:38:23 DEBUG BlockManager: Getting local block broadcast_2
22/08/12 15:38:23 DEBUG BlockManager: Level for block broadcast_2 is StorageLevel(disk, memory, deserialized, 1 replicas)
22/08/12 15:38:23 INFO PythonRunner: Times: total = 61, boot = -66, init = 127, finish = 0
22/08/12 15:38:23 INFO Executor: Finished task 4.0 in stage 2.0 (TID 9). 1789 bytes result sent to driver
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 10
22/08/12 15:38:23 INFO TaskSetManager: Finished task 4.0 in stage 2.0 (TID 9) in 86 ms on 10.1.255.235 (executor driver) (1/11)
22/08/12 15:38:23 INFO PythonRunner: Times: total = 78, boot = 7, init = 71, finish = 0
22/08/12 15:38:23 INFO PythonRunner: Times: total = 86, boot = -66, init = 152, finish = 0
22/08/12 15:38:23 INFO Executor: Finished task 0.0 in stage 2.0 (TID 5). 1789 bytes result sent to driver
22/08/12 15:38:23 INFO PythonRunner: Times: total = 83, boot = 12, init = 71, finish = 0
22/08/12 15:38:23 INFO Executor: Finished task 9.0 in stage 2.0 (TID 14). 1789 bytes result sent to driver
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 9
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 8
22/08/12 15:38:23 INFO PythonRunner: Times: total = 89, boot = 19, init = 69, finish = 1
22/08/12 15:38:23 INFO Executor: Finished task 6.0 in stage 2.0 (TID 11). 1789 bytes result sent to driver
22/08/12 15:38:23 INFO PythonRunner: Times: total = 93, boot = -49, init = 142, finish = 0
22/08/12 15:38:23 INFO PythonRunner: Times: total = 95, boot = -71, init = 166, finish = 0
22/08/12 15:38:23 INFO Executor: Finished task 8.0 in stage 2.0 (TID 13). 1789 bytes result sent to driver
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 7
22/08/12 15:38:23 INFO TaskSetManager: Finished task 9.0 in stage 2.0 (TID 14) in 113 ms on 10.1.255.235 (executor driver) (2/11)
22/08/12 15:38:23 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 5) in 118 ms on 10.1.255.235 (executor driver) (3/11)
22/08/12 15:38:23 INFO TaskSetManager: Finished task 8.0 in stage 2.0 (TID 13) in 116 ms on 10.1.255.235 (executor driver) (4/11)
22/08/12 15:38:23 INFO Executor: Finished task 5.0 in stage 2.0 (TID 10). 1789 bytes result sent to driver
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 6
22/08/12 15:38:23 INFO TaskSetManager: Finished task 6.0 in stage 2.0 (TID 11) in 123 ms on 10.1.255.235 (executor driver) (5/11)
22/08/12 15:38:23 INFO PythonRunner: Times: total = 104, boot = 3, init = 101, finish = 0
22/08/12 15:38:23 INFO Executor: Finished task 7.0 in stage 2.0 (TID 12). 1789 bytes result sent to driver
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 5
22/08/12 15:38:23 INFO TaskSetManager: Finished task 5.0 in stage 2.0 (TID 10) in 124 ms on 10.1.255.235 (executor driver) (6/11)
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 4
22/08/12 15:38:23 INFO TaskSetManager: Finished task 7.0 in stage 2.0 (TID 12) in 129 ms on 10.1.255.235 (executor driver) (7/11)
22/08/12 15:38:23 INFO Executor: Finished task 3.0 in stage 2.0 (TID 8). 1789 bytes result sent to driver
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 3
22/08/12 15:38:23 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 8) in 136 ms on 10.1.255.235 (executor driver) (8/11)
22/08/12 15:38:23 INFO PythonRunner: Times: total = 117, boot = 23, init = 94, finish = 0
22/08/12 15:38:23 INFO PythonRunner: Times: total = 121, boot = 6, init = 115, finish = 0
22/08/12 15:38:23 INFO Executor: Finished task 1.0 in stage 2.0 (TID 6). 1789 bytes result sent to driver
22/08/12 15:38:23 INFO Executor: Finished task 2.0 in stage 2.0 (TID 7). 1789 bytes result sent to driver
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 2
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 1
22/08/12 15:38:23 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 6) in 148 ms on 10.1.255.235 (executor driver) (9/11)
22/08/12 15:38:23 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 7) in 149 ms on 10.1.255.235 (executor driver) (10/11)
22/08/12 15:38:23 INFO PythonRunner: Times: total = 146, boot = 23, init = 123, finish = 0
22/08/12 15:38:23 INFO Executor: Finished task 10.0 in stage 2.0 (TID 15). 1830 bytes result sent to driver
22/08/12 15:38:23 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 0
22/08/12 15:38:23 INFO TaskSetManager: Finished task 10.0 in stage 2.0 (TID 15) in 176 ms on 10.1.255.235 (executor driver) (11/11)
22/08/12 15:38:23 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
22/08/12 15:38:23 INFO DAGScheduler: ResultStage 2 (showString at NativeMethodAccessorImpl.java:0) finished in 0.190 s
22/08/12 15:38:23 DEBUG DAGScheduler: After removal of stage 2, remaining stages = 0
22/08/12 15:38:23 INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/12 15:38:23 INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished
22/08/12 15:38:23 INFO DAGScheduler: Job 2 finished: showString at NativeMethodAccessorImpl.java:0, took 0.196476 s
22/08/12 15:38:23 DEBUG GenerateSafeProjection: code for createexternalrow(input[0, string, true].toString, input[1, string, true].toString, StructField(a,StringType,true), StructField(b,StringType,true)):
/* 001 */ public java.lang.Object generate(Object[] references) {
/* 002 */ return new SpecificSafeProjection(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
/* 006 */
/* 007 */ private Object[] references;
/* 008 */ private InternalRow mutableRow;
/* 009 */
/* 010 */
/* 011 */ public SpecificSafeProjection(Object[] references) {
/* 012 */ this.references = references;
/* 013 */ mutableRow = (InternalRow) references[references.length - 1];
/* 014 */
/* 015 */ }
/* 016 */
/* 017 */ public void initialize(int partitionIndex) {
/* 018 */
/* 019 */ }
/* 020 */
/* 021 */ public java.lang.Object apply(java.lang.Object _i) {
/* 022 */ InternalRow i = (InternalRow) _i;
/* 023 */ org.apache.spark.sql.Row value_5 = CreateExternalRow_0(i);
/* 024 */ if (false) {
/* 025 */ mutableRow.setNullAt(0);
/* 026 */ } else {
/* 027 */
/* 028 */ mutableRow.update(0, value_5);
/* 029 */ }
/* 030 */
/* 031 */ return mutableRow;
/* 032 */ }
/* 033 */
/* 034 */
/* 035 */ private org.apache.spark.sql.Row CreateExternalRow_0(InternalRow i) {
/* 036 */ Object[] values_0 = new Object[2];
/* 037 */
/* 038 */ boolean isNull_2 = i.isNullAt(0);
/* 039 */ UTF8String value_2 = isNull_2 ?
/* 040 */ null : (i.getUTF8String(0));
/* 041 */ boolean isNull_1 = true;
/* 042 */ java.lang.String value_1 = null;
/* 043 */ if (!isNull_2) {
/* 044 */ isNull_1 = false;
/* 045 */ if (!isNull_1) {
/* 046 */
/* 047 */ Object funcResult_0 = null;
/* 048 */ funcResult_0 = value_2.toString();
/* 049 */ value_1 = (java.lang.String) funcResult_0;
/* 050 */
/* 051 */ }
/* 052 */ }
/* 053 */ if (isNull_1) {
/* 054 */ values_0[0] = null;
/* 055 */ } else {
/* 056 */ values_0[0] = value_1;
/* 057 */ }
/* 058 */
/* 059 */ boolean isNull_4 = i.isNullAt(1);
/* 060 */ UTF8String value_4 = isNull_4 ?
/* 061 */ null : (i.getUTF8String(1));
/* 062 */ boolean isNull_3 = true;
/* 063 */ java.lang.String value_3 = null;
/* 064 */ if (!isNull_4) {
/* 065 */ isNull_3 = false;
/* 066 */ if (!isNull_3) {
/* 067 */
/* 068 */ Object funcResult_1 = null;
/* 069 */ funcResult_1 = value_4.toString();
/* 070 */ value_3 = (java.lang.String) funcResult_1;
/* 071 */
/* 072 */ }
/* 073 */ }
/* 074 */ if (isNull_3) {
/* 075 */ values_0[1] = null;
/* 076 */ } else {
/* 077 */ values_0[1] = value_3;
/* 078 */ }
/* 079 */
/* 080 */ final org.apache.spark.sql.Row value_0 = new org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema(values_0, ((org.apache.spark.sql.types.StructType) references[0] /* schema */));
/* 081 */
/* 082 */ return value_0;
/* 083 */ }
/* 084 */
/* 085 */ }
22/08/12 15:38:23 DEBUG CodeGenerator:
/* 001 */ public java.lang.Object generate(Object[] references) {
/* 002 */ return new SpecificSafeProjection(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
/* 006 */
/* 007 */ private Object[] references;
/* 008 */ private InternalRow mutableRow;
/* 009 */
/* 010 */
/* 011 */ public SpecificSafeProjection(Object[] references) {
/* 012 */ this.references = references;
/* 013 */ mutableRow = (InternalRow) references[references.length - 1];
/* 014 */
/* 015 */ }
/* 016 */
/* 017 */ public void initialize(int partitionIndex) {
/* 018 */
/* 019 */ }
/* 020 */
/* 021 */ public java.lang.Object apply(java.lang.Object _i) {
/* 022 */ InternalRow i = (InternalRow) _i;
/* 023 */ org.apache.spark.sql.Row value_5 = CreateExternalRow_0(i);
/* 024 */ if (false) {
/* 025 */ mutableRow.setNullAt(0);
/* 026 */ } else {
/* 027 */
/* 028 */ mutableRow.update(0, value_5);
/* 029 */ }
/* 030 */
/* 031 */ return mutableRow;
/* 032 */ }
/* 033 */
/* 034 */
/* 035 */ private org.apache.spark.sql.Row CreateExternalRow_0(InternalRow i) {
/* 036 */ Object[] values_0 = new Object[2];
/* 037 */
/* 038 */ boolean isNull_2 = i.isNullAt(0);
/* 039 */ UTF8String value_2 = isNull_2 ?
/* 040 */ null : (i.getUTF8String(0));
/* 041 */ boolean isNull_1 = true;
/* 042 */ java.lang.String value_1 = null;
/* 043 */ if (!isNull_2) {
/* 044 */ isNull_1 = false;
/* 045 */ if (!isNull_1) {
/* 046 */
/* 047 */ Object funcResult_0 = null;
/* 048 */ funcResult_0 = value_2.toString();
/* 049 */ value_1 = (java.lang.String) funcResult_0;
/* 050 */
/* 051 */ }
/* 052 */ }
/* 053 */ if (isNull_1) {
/* 054 */ values_0[0] = null;
/* 055 */ } else {
/* 056 */ values_0[0] = value_1;
/* 057 */ }
/* 058 */
/* 059 */ boolean isNull_4 = i.isNullAt(1);
/* 060 */ UTF8String value_4 = isNull_4 ?
/* 061 */ null : (i.getUTF8String(1));
/* 062 */ boolean isNull_3 = true;
/* 063 */ java.lang.String value_3 = null;
/* 064 */ if (!isNull_4) {
/* 065 */ isNull_3 = false;
/* 066 */ if (!isNull_3) {
/* 067 */
/* 068 */ Object funcResult_1 = null;
/* 069 */ funcResult_1 = value_4.toString();
/* 070 */ value_3 = (java.lang.String) funcResult_1;
/* 071 */
/* 072 */ }
/* 073 */ }
/* 074 */ if (isNull_3) {
/* 075 */ values_0[1] = null;
/* 076 */ } else {
/* 077 */ values_0[1] = value_3;
/* 078 */ }
/* 079 */
/* 080 */ final org.apache.spark.sql.Row value_0 = new org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema(values_0, ((org.apache.spark.sql.types.StructType) references[0] /* schema */));
/* 081 */
/* 082 */ return value_0;
/* 083 */ }
/* 084 */
/* 085 */ }
22/08/12 15:38:23 INFO CodeGenerator: Code generated in 15.338673 ms
+---+---+
| a| b|
+---+---+
| 1|2.0|
+---+---+
22/08/12 15:38:23 DEBUG FileSystem: Starting: Acquiring creator semaphore for s3a://master.rando2/nonemptyprefix4
22/08/12 15:38:23 DEBUG FileSystem: Acquiring creator semaphore for s3a://master.rando2/nonemptyprefix4: duration 0:00.000s
22/08/12 15:38:23 DEBUG FileSystem: Starting: Creating FS s3a://master.rando2/nonemptyprefix4
22/08/12 15:38:23 DEBUG FileSystem: Looking for FS supporting s3a
22/08/12 15:38:23 DEBUG FileSystem: looking for configuration option fs.s3a.impl
22/08/12 15:38:23 DEBUG Configuration: Reloading 9 existing configurations
22/08/12 15:38:23 DEBUG FileSystem: Filesystem s3a defined in configuration option
22/08/12 15:38:23 DEBUG FileSystem: FS for s3a is class org.apache.hadoop.fs.s3a.S3AFileSystem
22/08/12 15:38:23 DEBUG AbstractService: Service: NoopAuditor entered state INITED
22/08/12 15:38:23 DEBUG AbstractService: Service NoopAuditor is started
22/08/12 15:38:23 DEBUG S3AFileSystem: Initializing S3AFileSystem for master.rando2
22/08/12 15:38:23 DEBUG S3AUtils: Propagating entries under fs.s3a.bucket.master.rando2.
22/08/12 15:38:23 DEBUG S3AUtils: Data is unencrypted
22/08/12 15:38:23 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms
22/08/12 15:38:23 DEBUG MetricsSystemImpl: from system property: null
22/08/12 15:38:23 DEBUG MetricsSystemImpl: from environment variable: null
22/08/12 15:38:23 DEBUG MetricsConfig: Could not locate file hadoop-metrics2-s3a-file-system.properties
org.apache.hadoop.shaded.org.apache.commons.configuration2.ex.ConfigurationException: Could not locate: org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocator@5419d944[fileName=hadoop-metrics2-s3a-file-system.properties,basePath=<null>,sourceURL=,encoding=<null>,fileSystem=<null>,locationStrategy=<null>]
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocatorUtils.locateOrThrow(FileLocatorUtils.java:346)
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:972)
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:702)
at org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:118)
at org.apache.hadoop.metrics2.impl.MetricsConfig.create(MetricsConfig.java:97)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:482)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.getMetricsSystem(S3AInstrumentation.java:251)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.registerAsMetricsSource(S3AInstrumentation.java:264)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.<init>(S3AInstrumentation.java:235)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:466)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461)
at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:558)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:793)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
22/08/12 15:38:23 DEBUG MetricsConfig: Could not locate file hadoop-metrics2.properties
org.apache.hadoop.shaded.org.apache.commons.configuration2.ex.ConfigurationException: Could not locate: org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocator@14a0842[fileName=hadoop-metrics2.properties,basePath=<null>,sourceURL=,encoding=<null>,fileSystem=<null>,locationStrategy=<null>]
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocatorUtils.locateOrThrow(FileLocatorUtils.java:346)
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:972)
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:702)
at org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:118)
at org.apache.hadoop.metrics2.impl.MetricsConfig.create(MetricsConfig.java:97)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:482)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.getMetricsSystem(S3AInstrumentation.java:251)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.registerAsMetricsSource(S3AInstrumentation.java:264)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.<init>(S3AInstrumentation.java:235)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:466)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461)
at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:558)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:793)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
22/08/12 15:38:23 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
22/08/12 15:38:23 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: period
22/08/12 15:38:23 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: periodMillis
22/08/12 15:38:23 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableCounterLong org.apache.hadoop.metrics2.impl.MetricsSystemImpl.droppedPubAll with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Dropped updates by all sinks"})
22/08/12 15:38:23 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableStat org.apache.hadoop.metrics2.impl.MetricsSystemImpl.publishStat with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Publish", "Publishing stats"})
22/08/12 15:38:23 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableStat org.apache.hadoop.metrics2.impl.MetricsSystemImpl.snapshotStat with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Snapshot", "Snapshot stats"})
22/08/12 15:38:23 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: source.source.start_mbeans
22/08/12 15:38:23 DEBUG MetricsConfig: poking parent 'MetricsConfig' for key: source.start_mbeans
22/08/12 15:38:23 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: *.source.start_mbeans
22/08/12 15:38:23 DEBUG MetricsSourceAdapter: Updating attr cache...
22/08/12 15:38:23 DEBUG MetricsSourceAdapter: Done. # tags & metrics=10
22/08/12 15:38:23 DEBUG MetricsSourceAdapter: Updating info cache...
22/08/12 15:38:23 DEBUG MetricsSystemImpl: [javax.management.MBeanAttributeInfo[description=Metrics context, name=tag.Context, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of active metrics sources, name=NumActiveSources, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of all registered metrics sources, name=NumAllSources, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of active metrics sinks, name=NumActiveSinks, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of all registered metrics sinks, name=NumAllSinks, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Dropped updates by all sinks, name=DroppedPubAll, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of ops for publishing stats, name=PublishNumOps, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Average time for publishing stats, name=PublishAvgTime, type=java.lang.Double, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of ops for snapshot stats, name=SnapshotNumOps, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Average time for snapshot stats, name=SnapshotAvgTime, type=java.lang.Double, read-only, descriptor={}]]
22/08/12 15:38:23 DEBUG MetricsSourceAdapter: Done
22/08/12 15:38:23 DEBUG MBeans: Registered Hadoop:service=s3a-file-system,name=MetricsSystem,sub=Stats
22/08/12 15:38:23 DEBUG MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
22/08/12 15:38:23 INFO MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
22/08/12 15:38:23 INFO MetricsSystemImpl: s3a-file-system metrics system started
22/08/12 15:38:23 DEBUG MBeans: Registered Hadoop:service=s3a-file-system,name=MetricsSystem,sub=Control
22/08/12 15:38:23 DEBUG MetricsSystemImpl: S3AMetrics1-master.rando2,
22/08/12 15:38:23 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: source.source.start_mbeans
22/08/12 15:38:23 DEBUG MetricsConfig: poking parent 'MetricsConfig' for key: source.start_mbeans
22/08/12 15:38:23 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: *.source.start_mbeans
22/08/12 15:38:23 DEBUG MetricsSourceAdapter: Updating attr cache...
22/08/12 15:38:23 DEBUG MetricsSourceAdapter: Done. # tags & metrics=187
22/08/12 15:38:23 DEBUG MetricsSourceAdapter: Updating info cache...
22/08/12 15:38:23 DEBUG MetricsSystemImpl: [javax.management.MBeanAttributeInfo[description=Metrics context, name=tag.Context, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=A unique identifier for the instance, name=tag.s3aFileSystemId, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Hostname from the FS URL, name=tag.bucket, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Local hostname, name=tag.Hostname, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of directories created through the object store., name=directories_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of directories deleted through the object store., name=directories_deleted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files copied within the object store., name=files_copied, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of bytes copied within the object store., name=files_copied_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files created through the object store., name=files_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files deleted from the object store., name=files_deleted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files whose delete request was rejected, name=files_delete_rejected, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of fake directory entries created in the object store., name=fake_directories_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of fake directory deletes submitted to object store., name=fake_directories_deleted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Errors caught and ignored, name=ignored_errors, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of createNonRecursive(), name=op_create_non_recursive, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of hflush(), name=op_hflush, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of hsync(), name=op_hsync, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listLocatedStatus(), name=op_list_located_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of open(), name=op_open, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object copy requests, name=object_copy_requests, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Objects deleted in delete requests, name=object_delete_objects, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of requests for object metadata, name=object_metadata_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object put/multipart upload completed count, name=object_put_request_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=number of bytes uploaded, name=object_put_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of S3 Select requests issued, name=object_select_requests, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of times the TCP stream was aborted, name=stream_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Bytes read from an input stream in read() calls, name=stream_read_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes discarded by aborting an input stream, name=stream_read_bytes_discarded_in_abort, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes read and discarded when closing an input stream, name=stream_read_bytes_discarded_in_close, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of times the TCP stream was closed, name=stream_read_closed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total count of times an attempt to close an input stream was made, name=stream_read_close_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of exceptions raised during input stream reads, name=stream_read_exceptions, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of readFully() operations in an input stream, name=stream_read_fully_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total count of times an input stream to object store data was opened, name=stream_read_opened, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of read() operations in an input stream, name=stream_read_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of incomplete read() operations in an input stream, name=stream_read_operations_incomplete, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of version mismatches encountered while reading an input stream, name=stream_read_version_mismatches, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of executed seek operations which went backwards in a stream, name=stream_read_seek_backward_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes moved backwards during seek operations in an input stream, name=stream_read_bytes_backwards_on_seek, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes read and discarded during seek() in an input stream, name=stream_read_seek_bytes_discarded, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes skipped during forward seek operations an input stream, name=stream_read_seek_bytes_skipped, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of executed seek operations which went forward in an input stream, name=stream_read_seek_forward_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of seek operations in an input stream, name=stream_read_seek_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of times the seek policy was dynamically changed in an input stream, name=stream_read_seek_policy_changed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total count of bytes read from an input stream, name=stream_read_total_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of stream write failures reported, name=stream_write_exceptions, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of failures when finalizing a multipart upload, name=stream_write_exceptions_completing_upload, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of block/partition uploads completed, name=stream_write_block_uploads, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of number of block uploads committed, name=stream_write_block_uploads_committed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of number of block uploads aborted, name=stream_write_block_uploads_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of total time taken for uploads to complete, name=stream_write_total_time, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of total data uploaded, name=stream_write_total_data, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes written to output stream (including all not yet uploaded), name=stream_write_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of files to commit created, name=committer_commits_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of files committed, name=committer_commits_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of successful jobs, name=committer_jobs_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of failed jobs, name=committer_jobs_failed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of successful tasks, name=committer_tasks_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of failed tasks, name=committer_tasks_failed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Amount of data committed, name=committer_bytes_committed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes uploaded duing commit operations, name=committer_bytes_uploaded, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of commits failed, name=committer_commits.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of commits aborted, name=committer_commits_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of commits reverted, name=committer_commits_reverted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of files created under 'magic' paths, name=committer_magic_files_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store put one metadata path request, name=s3guard_metadatastore_put_path_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store initialization times, name=s3guard_metadatastore_initialization, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store records deleted, name=s3guard_metadatastore_record_deletes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store records read, name=s3guard_metadatastore_record_reads, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store records written, name=s3guard_metadatastore_record_writes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store retry events, name=s3guard_metadatastore_retry, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store throttled events, name=s3guard_metadatastore_throttled, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store authoritative directories updated from S3, name=s3guard_metadatastore_authoritative_directories_updated, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=requests made of the remote store, name=store_io_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=retried requests made of the remote store, name=store_io_retry, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Requests throttled and retried, name=store_io_throttled, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Uploader Instantiated, name=multipart_instantiated, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Part Put Operation, name=multipart_upload_part_put, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Part Put Bytes, name=multipart_upload_part_put_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Aborted, name=multipart_upload_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Abort Unner Path Invoked, name=multipart_upload_abort_under_path_invoked, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Completed, name=multipart_upload_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Started, name=multipart_upload_started, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Audit access check was rejected, name=audit_access_check_failure, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Audit Span Created, name=audit_span_creation, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Audit failure/rejection, name=audit_failure, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=AWS request made, name=audit_request_execution, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Current number of active put requests, name=object_put_request_active, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=number of bytes queued for upload/being actively uploaded, name=object_put_bytes_pending, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of block/partition uploads active, name=stream_write_block_uploads_active, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Gauge of block/partitions uploads queued to be written, name=stream_write_block_uploads_pending, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Gauge of data queued to be written, name=stream_write_block_uploads_data_pending, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=gauge to indicate if client side encryption is enabled, name=client_side_encryption_enabled, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Executor acquired., name=action_executor_acquired, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Executor acquired., name=action_executor_acquired.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=HEAD request., name=action_http_head_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=HEAD request., name=action_http_head_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=GET request., name=action_http_get_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=GET request., name=action_http_get_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of abort(), name=op_abort, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of abort(), name=op_abort.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of access(), name=op_access, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of access(), name=op_access.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of copyFromLocalFile(), name=op_copy_from_local_file, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of copyFromLocalFile(), name=op_copy_from_local_file.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of create(), name=op_create, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of create(), name=op_create.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of delete(), name=op_delete, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of delete(), name=op_delete.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of exists(), name=op_exists, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of exists(), name=op_exists.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getContentSummary(), name=op_get_content_summary, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getContentSummary(), name=op_get_content_summary.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getDelegationToken(), name=op_get_delegation_token, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getDelegationToken(), name=op_get_delegation_token.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileChecksum(), name=op_get_file_checksum, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileChecksum(), name=op_get_file_checksum.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileStatus(), name=op_get_file_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileStatus(), name=op_get_file_status.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of globStatus(), name=op_glob_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of globStatus(), name=op_glob_status.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isDirectory(), name=op_is_directory, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isDirectory(), name=op_is_directory.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isFile(), name=op_is_file, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isFile(), name=op_is_file.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listFiles(), name=op_list_files, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listFiles(), name=op_list_files.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listStatus(), name=op_list_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listStatus(), name=op_list_status.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of mkdirs(), name=op_mkdirs, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of mkdirs(), name=op_mkdirs.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of rename(), name=op_rename, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of rename(), name=op_rename.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path), name=op_xattr_get_map, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path), name=op_xattr_get_map.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttr(Path, String), name=op_xattr_get_named, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttr(Path, String), name=op_xattr_get_named.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of xattr(), name=op_xattr_get_named_map, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of xattr(), name=op_xattr_get_named_map.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path, List<String> names), name=op_xattr_list, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path, List<String> names), name=op_xattr_list.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object delete requests, name=object_delete_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object delete requests, name=object_delete_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object bulk delete requests, name=object_bulk_delete_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object bulk delete requests, name=object_bulk_delete_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of object listings made, name=object_list_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of object listings made, name=object_list_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of continued object listings made, name=object_continue_list_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of continued object listings made, name=object_continue_list_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload initiated, name=object_multipart_initiated, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload initiated, name=object_multipart_initiated.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload aborted, name=object_multipart_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload aborted, name=object_multipart_aborted.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object put/multipart upload count, name=object_put_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object put/multipart upload count, name=object_put_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total queue duration of all block uploads, name=stream_write_queue_duration, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total queue duration of all block uploads, name=stream_write_queue_duration.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to commit an entire job, name=committer_commit_job, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to commit an entire job, name=committer_commit_job.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to materialize a file in job commit, name=committer_materialize_file, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to materialize a file in job commit, name=committer_materialize_file.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of files uploaded from a local staging path, name=committer_stage_file_upload, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of files uploaded from a local staging path, name=committer_stage_file_upload.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Store Existence Probe, name=store_exists_probe, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Store Existence Probe, name=store_exists_probe.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of delegation tokens issued, name=delegation_tokens_issued, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of delegation tokens issued, name=delegation_tokens_issued.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload List, name=multipart_upload_list, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload List, name=multipart_upload_list.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of ops for s3Guard metadata store put one metadata path latency with 1s interval, name=S3guard_metadatastore_put_path_latencyNumOps, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=50 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency50thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=75 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency75thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=90 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency90thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=95 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency95thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=99 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency99thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of events for s3Guard metadata store throttle rate with 1s interval, name=S3guard_metadatastore_throttle_rateNumEvents, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=50 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate50thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=75 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate75thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=90 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate90thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=95 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate95thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=99 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate99thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of events for rate of S3 request throttling with 1s interval, name=Store_io_throttle_rateNumEvents, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=50 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate50thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=75 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate75thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=90 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate90thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=95 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate95thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=99 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate99thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}]]
22/08/12 15:38:23 DEBUG MetricsSourceAdapter: Done
22/08/12 15:38:23 DEBUG MBeans: Registered Hadoop:service=s3a-file-system,name=S3AMetrics1-master.rando2
22/08/12 15:38:23 DEBUG MetricsSourceAdapter: MBean for source S3AMetrics1-master.rando2 registered.
22/08/12 15:38:23 DEBUG MetricsSystemImpl: Registered source S3AMetrics1-master.rando2
22/08/12 15:38:23 DEBUG S3AFileSystem: Client Side Encryption enabled: false
22/08/12 15:38:23 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms
22/08/12 15:38:23 DEBUG S3GuardExistsRetryPolicy: Retrying on recoverable S3Guard table/S3 inconsistencies 7 times with an initial interval of 2000ms
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.paging.maximum is 5000
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.block.size is 33554432
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.readahead.range is 65536
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.max.total.tasks is 32
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.threads.keepalivetime is 60
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.executor.capacity is 16
22/08/12 15:38:23 DEBUG SignerManager: No custom signers specified
22/08/12 15:38:23 DEBUG AuditIntegration: auditing is disabled
22/08/12 15:38:23 DEBUG AbstractService: Service: NoopAuditManagerS3A entered state INITED
22/08/12 15:38:23 DEBUG CompositeService: NoopAuditManagerS3A: initing services, size=0
22/08/12 15:38:23 DEBUG CompositeService: Adding service NoopAuditor
22/08/12 15:38:23 DEBUG AbstractService: Service: NoopAuditor entered state INITED
22/08/12 15:38:23 DEBUG CompositeService: NoopAuditManagerS3A: starting services, size=1
22/08/12 15:38:23 DEBUG AbstractService: Service NoopAuditor is started
22/08/12 15:38:23 DEBUG AbstractService: Service NoopAuditManagerS3A is started
22/08/12 15:38:23 DEBUG AuditIntegration: Started Audit Manager Service NoopAuditManagerS3A in state NoopAuditManagerS3A: STARTED
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.internal.upload.part.count.limit is 10000
22/08/12 15:38:23 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms
22/08/12 15:38:23 DEBUG S3AUtils: Credential provider class is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
22/08/12 15:38:23 DEBUG S3AUtils: Credential provider class is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
22/08/12 15:38:23 DEBUG S3AUtils: Credential provider class is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
22/08/12 15:38:23 DEBUG S3AUtils: Credential provider class is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
22/08/12 15:38:23 DEBUG S3AUtils: For URI s3a://master.rando2/nonemptyprefix4, using credentials AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@4e765549]
22/08/12 15:38:23 DEBUG S3AFileSystem: Using credential provider AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@4e765549]
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.connection.maximum is 96
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.attempts.maximum is 20
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.connection.establish.timeout is 5000
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.connection.timeout is 200000
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.socket.send.buffer is 8192
22/08/12 15:38:23 DEBUG S3AUtils: Value of fs.s3a.socket.recv.buffer is 8192
22/08/12 15:38:23 DEBUG S3AUtils: Using User-Agent: Hadoop 3.3.2
22/08/12 15:38:23 DEBUG S3AUtils: Data is unencrypted
22/08/12 15:38:23 DEBUG AmazonWebServiceClient: Internal logging successfully configured to commons logger: true
22/08/12 15:38:23 DEBUG AwsSdkMetrics: Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics
22/08/12 15:38:23 DEBUG DefaultS3ClientFactory: Creating endpoint configuration for "http://localhost:30600"
22/08/12 15:38:23 DEBUG DefaultS3ClientFactory: Endpoint URI = http://localhost:30600
22/08/12 15:38:23 DEBUG DefaultS3ClientFactory: Endpoint http://localhost:30600 is not the default; parsing
22/08/12 15:38:23 DEBUG DefaultS3ClientFactory: Region for endpoint http://localhost:30600, URI http://localhost:30600 is determined as null
22/08/12 15:38:23 DEBUG FsUrlStreamHandlerFactory: Creating handler for protocol http
22/08/12 15:38:23 DEBUG FsUrlStreamHandlerFactory: Unknown protocol http, delegating to default implementation
22/08/12 15:38:23 DEBUG CsmConfigurationProviderChain: Unable to load configuration from com.amazonaws.monitoring.EnvironmentVariableCsmConfigurationProvider@62b78077: Unable to load Client Side Monitoring configurations from environment variables!
22/08/12 15:38:23 DEBUG CsmConfigurationProviderChain: Unable to load configuration from com.amazonaws.monitoring.SystemPropertyCsmConfigurationProvider@40c4eed6: Unable to load Client Side Monitoring configurations from system properties variables!
22/08/12 15:38:23 DEBUG PoolingHttpClientConnectionManager: Closing connections idle longer than 60000 MILLISECONDS
22/08/12 15:38:23 DEBUG CsmConfigurationProviderChain: Unable to load configuration from com.amazonaws.monitoring.ProfileCsmConfigurationProvider@1a042e4c: The 'default' profile does not define all the required properties!
22/08/12 15:38:24 DEBUG S3AFileSystem: skipping check for bucket existence
22/08/12 15:38:24 DEBUG S3AFileSystem: Input fadvise policy = normal
22/08/12 15:38:24 DEBUG S3AFileSystem: Change detection policy = ETagChangeDetectionPolicy mode=None
22/08/12 15:38:24 DEBUG S3AFileSystem: Filesystem support for magic committers is enabled
22/08/12 15:38:24 DEBUG S3AUtils: Value of fs.s3a.fast.upload.active.blocks is 4
22/08/12 15:38:24 DEBUG S3AFileSystem: Using S3ABlockOutputStream with buffer = disk; block=67108864; queue limit=4
22/08/12 15:38:24 DEBUG S3Guard: Metastore option source [core-default.xml]
22/08/12 15:38:24 DEBUG S3Guard: Using NullMetadataStore metadata store for s3a filesystem
22/08/12 15:38:24 DEBUG S3AFileSystem: S3Guard is disabled on this bucket: master.rando2
22/08/12 15:38:24 DEBUG DirectoryPolicyImpl: Directory markers will be deleted
22/08/12 15:38:24 DEBUG S3AFileSystem: Directory marker retention policy is DirectoryMarkerRetention{policy='delete'}
22/08/12 15:38:24 DEBUG S3AUtils: Value of fs.s3a.multipart.purge.age is 86400
22/08/12 15:38:24 DEBUG S3AUtils: Value of fs.s3a.bulk.delete.page.size is 250
22/08/12 15:38:24 DEBUG FileSystem: Creating FS s3a://master.rando2/nonemptyprefix4: duration 0:00.753s
22/08/12 15:38:24 DEBUG FileCommitProtocol: Creating committer org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol; job abc4d155-280f-4c55-9237-3ebebea73a2f; output=s3a://master.rando2/nonemptyprefix4; dynamic=false
22/08/12 15:38:24 DEBUG FileCommitProtocol: Using (String, String, Boolean) constructor
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 1
22/08/12 15:38:24 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4 (nonemptyprefix4); needEmptyDirectory=false
22/08/12 15:38:24 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 1
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 1
22/08/12 15:38:24 DEBUG S3AFileSystem: HEAD nonemptyprefix4 with change tracker null
22/08/12 15:38:24 DEBUG Invoker: Starting: create credentials
22/08/12 15:38:24 DEBUG Invoker: create credentials: duration 0:00.002s
22/08/12 15:38:24 DEBUG AWSCredentialProviderList: No credentials from TemporaryAWSCredentialsProvider: org.apache.hadoop.fs.s3a.auth.NoAwsCredentialsException: Session credentials in Hadoop configuration: No AWS Credentials
22/08/12 15:38:24 DEBUG AWSCredentialProviderList: Using credentials from SimpleAWSCredentialsProvider
22/08/12 15:38:24 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4 Headers: (amz-sdk-invocation-id: e0cdbae7-6b00-a098-116f-6578df6a6a24, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4
amz-sdk-invocation-id:e0cdbae7-6b00-a098-116f-6578df6a6a24
amz-sdk-request:attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143824Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143824Z
20220812/us-east-1/s3/aws4_request
904dc861a62c13bf22140cb090d8ad63be51b45d859cc491e568e5f9e8017716"
22/08/12 15:38:24 DEBUG AWS4Signer: Generating a new signing key as the signing key not available in the cache for the date 1660262400000
22/08/12 15:38:24 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:24 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 0 of 96; total allocated: 0 of 96]
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:24 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:24 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59620<->127.0.0.1:30600
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4 HTTP/1.1
22/08/12 15:38:24 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix4 HTTP/1.1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: e0cdbae7-6b00-a098-116f-6578df6a6a24
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-request: attempt=1;max=21
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=3748ec145dc8538ff64f64dc1711c5dcb51490f16a91b2ce7404a9a1a3ecfbcc
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143824Z
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix4 HTTP/1.1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: e0cdbae7-6b00-a098-116f-6578df6a6a24[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: attempt=1;max=21[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=3748ec145dc8538ff64f64dc1711c5dcb51490f16a91b2ce7404a9a1a3ecfbcc[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143824Z[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 5e1841f6-f2de-4a9c-b532-2fa60d7b1152[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 5e1841f6-f2de-4a9c-b532-2fa60d7b1152[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:24 GMT[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Length: 238[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 5e1841f6-f2de-4a9c-b532-2fa60d7b1152
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 5e1841f6-f2de-4a9c-b532-2fa60d7b1152
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Length: 238
22/08/12 15:38:24 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 5e1841f6-f2de-4a9c-b532-2fa60d7b1152; S3 Extended Request ID: 5e1841f6-f2de-4a9c-b532-2fa60d7b1152; Proxy: null), S3 Extended Request ID: 5e1841f6-f2de-4a9c-b532-2fa60d7b1152
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 1
22/08/12 15:38:24 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 5e1841f6-f2de-4a9c-b532-2fa60d7b1152; S3 Extended Request ID: 5e1841f6-f2de-4a9c-b532-2fa60d7b1152; Proxy: null), S3 Extended Request ID: 5e1841f6-f2de-4a9c-b532-2fa60d7b1152], RequestType=[GetObjectMetadataRequest], AWSRequestID=[5e1841f6-f2de-4a9c-b532-2fa60d7b1152], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[126.069], HttpClientSendRequestTime=[2.603], HttpRequestTime=[63.678], ApiCallLatency=[112.918], RequestSigningTime=[31.432], CredentialsRequestTime=[4.018, 0.014], HttpClientReceiveResponseTime=[17.338],
22/08/12 15:38:24 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:24 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 1
22/08/12 15:38:24 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 46201aef-0f32-bcae-06cd-1ac0ac482489, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F
amz-sdk-invocation-id:46201aef-0f32-bcae-06cd-1ac0ac482489
amz-sdk-request:ttl=20220812T144144Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143824Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143824Z
20220812/us-east-1/s3/aws4_request
b64105385a0b5c4942da56fbf484f6e03afa00294a41f4ec1aa6d3f75eba94c2"
22/08/12 15:38:24 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:24 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:24 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 46201aef-0f32-bcae-06cd-1ac0ac482489
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=5c8408895b4e3f2426bbd34466887244ab034a9c618efeeb2e3df6bd421ba2af
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143824Z
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 46201aef-0f32-bcae-06cd-1ac0ac482489[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=5c8408895b4e3f2426bbd34466887244ab034a9c618efeeb2e3df6bd421ba2af[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143824Z[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 7f2599a7-94e9-431f-a0e8-5751fa7af33c[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 7f2599a7-94e9-431f-a0e8-5751fa7af33c[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:24 GMT[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Length: 276[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/</Prefix></ListBucketResult>"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 7f2599a7-94e9-431f-a0e8-5751fa7af33c
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 7f2599a7-94e9-431f-a0e8-5751fa7af33c
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Length: 276
22/08/12 15:38:24 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:24 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:24 DEBUG request: Received successful response: 200, AWS Request ID: 7f2599a7-94e9-431f-a0e8-5751fa7af33c
22/08/12 15:38:24 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:24 DEBUG requestId: AWS Request ID: 7f2599a7-94e9-431f-a0e8-5751fa7af33c
22/08/12 15:38:24 DEBUG requestId: AWS Extended Request ID: 7f2599a7-94e9-431f-a0e8-5751fa7af33c
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 2
22/08/12 15:38:24 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[7f2599a7-94e9-431f-a0e8-5751fa7af33c], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[46.111], ClientExecuteTime=[81.933], HttpClientSendRequestTime=[1.589], HttpRequestTime=[17.367], ApiCallLatency=[81.698], RequestSigningTime=[0.564], CredentialsRequestTime=[0.007, 0.011], HttpClientReceiveResponseTime=[13.82],
22/08/12 15:38:24 DEBUG S3AFileSystem: LIST: duration 0:00.083s
22/08/12 15:38:24 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 1
22/08/12 15:38:24 INFO ParquetFileFormat: Using default output committer for Parquet: org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:24 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246545752692661967718_0000}; taskId=attempt_202208121538246545752692661967718_0000_m_000000_0, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@1436381c}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246545752692661967718_0000}; taskId=attempt_202208121538246545752692661967718_0000_m_000000_0, status=''}
22/08/12 15:38:24 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:24 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:24 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:24 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246545752692661967718_0000}; taskId=attempt_202208121538246545752692661967718_0000_m_000000_0, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@6b6bcb6f}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246545752692661967718_0000}; taskId=attempt_202208121538246545752692661967718_0000_m_000000_0, status=''}
22/08/12 15:38:24 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:24 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:24 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter op_mkdirs by 1 with final value 1
22/08/12 15:38:24 DEBUG MkdirOperation: Making directory: s3a://master.rando2/nonemptyprefix4/_temporary/0
22/08/12 15:38:24 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0 (nonemptyprefix4/_temporary/0); needEmptyDirectory=false
22/08/12 15:38:24 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0
22/08/12 15:38:24 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:24 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 2
22/08/12 15:38:24 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 1fd182ac-e68a-0cb1-7abc-f00cee5d5ddf, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F
amz-sdk-invocation-id:1fd182ac-e68a-0cb1-7abc-f00cee5d5ddf
amz-sdk-request:ttl=20220812T144144Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143824Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143824Z
20220812/us-east-1/s3/aws4_request
f0c7354ea11e8abf78d87b1f3487c2065ed34a3d02b73955438cf0e81198ba9a"
22/08/12 15:38:24 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:24 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:24 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 1fd182ac-e68a-0cb1-7abc-f00cee5d5ddf
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=829067cd1adf1671f9e821a1bdc3d741385ece5de8800851b7e176b0ee9e5d35
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143824Z
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 1fd182ac-e68a-0cb1-7abc-f00cee5d5ddf[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=829067cd1adf1671f9e821a1bdc3d741385ece5de8800851b7e176b0ee9e5d35[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143824Z[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: e3f4cf05-a74e-4364-bfaa-7d8fc56983e9[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: e3f4cf05-a74e-4364-bfaa-7d8fc56983e9[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:24 GMT[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Length: 289[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/</Prefix></ListBucketResult>"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: e3f4cf05-a74e-4364-bfaa-7d8fc56983e9
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: e3f4cf05-a74e-4364-bfaa-7d8fc56983e9
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Length: 289
22/08/12 15:38:24 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:24 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:24 DEBUG request: Received successful response: 200, AWS Request ID: e3f4cf05-a74e-4364-bfaa-7d8fc56983e9
22/08/12 15:38:24 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:24 DEBUG requestId: AWS Request ID: e3f4cf05-a74e-4364-bfaa-7d8fc56983e9
22/08/12 15:38:24 DEBUG requestId: AWS Extended Request ID: e3f4cf05-a74e-4364-bfaa-7d8fc56983e9
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 3
22/08/12 15:38:24 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[e3f4cf05-a74e-4364-bfaa-7d8fc56983e9], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[2.106], ClientExecuteTime=[22.007], HttpClientSendRequestTime=[0.957], HttpRequestTime=[18.234], ApiCallLatency=[21.781], RequestSigningTime=[0.435], CredentialsRequestTime=[0.008, 0.004], HttpClientReceiveResponseTime=[16.358],
22/08/12 15:38:24 DEBUG S3AFileSystem: LIST: duration 0:00.023s
22/08/12 15:38:24 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0
22/08/12 15:38:24 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0 (nonemptyprefix4/_temporary/0); needEmptyDirectory=false
22/08/12 15:38:24 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 2
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 2
22/08/12 15:38:24 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0 with change tracker null
22/08/12 15:38:24 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0 Headers: (amz-sdk-invocation-id: f8d4d022-1fe4-870f-a25d-81d6e83c05b1, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0
amz-sdk-invocation-id:f8d4d022-1fe4-870f-a25d-81d6e83c05b1
amz-sdk-request:ttl=20220812T144144Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143824Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143824Z
20220812/us-east-1/s3/aws4_request
fbf24219f64860542430578d17132ee2bca926169d57faa202f679fa3d816ef7"
22/08/12 15:38:24 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:24 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0 HTTP/1.1
22/08/12 15:38:24 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0 HTTP/1.1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: f8d4d022-1fe4-870f-a25d-81d6e83c05b1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=71fd03e6613cc1421f7ce8e78b479d944e054669d530f23e841a0257354b282f
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143824Z
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0 HTTP/1.1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: f8d4d022-1fe4-870f-a25d-81d6e83c05b1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=71fd03e6613cc1421f7ce8e78b479d944e054669d530f23e841a0257354b282f[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143824Z[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 391fc50c-c596-4c35-a42f-c4aa0c01fefd[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 391fc50c-c596-4c35-a42f-c4aa0c01fefd[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:24 GMT[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Length: 251[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 391fc50c-c596-4c35-a42f-c4aa0c01fefd
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 391fc50c-c596-4c35-a42f-c4aa0c01fefd
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Length: 251
22/08/12 15:38:24 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 391fc50c-c596-4c35-a42f-c4aa0c01fefd; S3 Extended Request ID: 391fc50c-c596-4c35-a42f-c4aa0c01fefd; Proxy: null), S3 Extended Request ID: 391fc50c-c596-4c35-a42f-c4aa0c01fefd
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 4
22/08/12 15:38:24 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 391fc50c-c596-4c35-a42f-c4aa0c01fefd; S3 Extended Request ID: 391fc50c-c596-4c35-a42f-c4aa0c01fefd; Proxy: null), S3 Extended Request ID: 391fc50c-c596-4c35-a42f-c4aa0c01fefd], RequestType=[GetObjectMetadataRequest], AWSRequestID=[391fc50c-c596-4c35-a42f-c4aa0c01fefd], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[18.599], HttpClientSendRequestTime=[0.949], HttpRequestTime=[17.28], ApiCallLatency=[18.363], RequestSigningTime=[0.321], CredentialsRequestTime=[0.007, 0.003], HttpClientReceiveResponseTime=[15.281],
22/08/12 15:38:24 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0
22/08/12 15:38:24 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary (nonemptyprefix4/_temporary); needEmptyDirectory=false
22/08/12 15:38:24 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary
22/08/12 15:38:24 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:24 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 3
22/08/12 15:38:24 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: c5667583-4b6c-611e-a99d-2bb58628cc12, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F
amz-sdk-invocation-id:c5667583-4b6c-611e-a99d-2bb58628cc12
amz-sdk-request:ttl=20220812T144144Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143824Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143824Z
20220812/us-east-1/s3/aws4_request
5a6e60ebecfe3464de650dd5dcc2da1d8711955aa4751b64185f43d5dc51274d"
22/08/12 15:38:24 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:24 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:24 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: c5667583-4b6c-611e-a99d-2bb58628cc12
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=39ecbdf749c5d312dd873f06ef53971ec1f33f7ddcad08d110d33d5edc4dab51
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143824Z
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: c5667583-4b6c-611e-a99d-2bb58628cc12[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=39ecbdf749c5d312dd873f06ef53971ec1f33f7ddcad08d110d33d5edc4dab51[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143824Z[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: e517f7a7-21ca-44f1-b78b-42b5c4388f9d[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: e517f7a7-21ca-44f1-b78b-42b5c4388f9d[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:24 GMT[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Length: 287[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/</Prefix></ListBucketResult>"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: e517f7a7-21ca-44f1-b78b-42b5c4388f9d
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: e517f7a7-21ca-44f1-b78b-42b5c4388f9d
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Length: 287
22/08/12 15:38:24 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:24 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:24 DEBUG request: Received successful response: 200, AWS Request ID: e517f7a7-21ca-44f1-b78b-42b5c4388f9d
22/08/12 15:38:24 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:24 DEBUG requestId: AWS Request ID: e517f7a7-21ca-44f1-b78b-42b5c4388f9d
22/08/12 15:38:24 DEBUG requestId: AWS Extended Request ID: e517f7a7-21ca-44f1-b78b-42b5c4388f9d
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 5
22/08/12 15:38:24 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[e517f7a7-21ca-44f1-b78b-42b5c4388f9d], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[3.018], ClientExecuteTime=[20.579], HttpClientSendRequestTime=[1.066], HttpRequestTime=[16.08], ApiCallLatency=[20.371], RequestSigningTime=[0.399], CredentialsRequestTime=[0.005, 0.003], HttpClientReceiveResponseTime=[14.017],
22/08/12 15:38:24 DEBUG S3AFileSystem: LIST: duration 0:00.021s
22/08/12 15:38:24 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary
22/08/12 15:38:24 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary (nonemptyprefix4/_temporary); needEmptyDirectory=false
22/08/12 15:38:24 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 3
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 3
22/08/12 15:38:24 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary with change tracker null
22/08/12 15:38:24 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary Headers: (amz-sdk-invocation-id: 5a076218-dada-102c-f2bc-75e004b85acf, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary
amz-sdk-invocation-id:5a076218-dada-102c-f2bc-75e004b85acf
amz-sdk-request:ttl=20220812T144144Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143824Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143824Z
20220812/us-east-1/s3/aws4_request
b7f55c19595c26466707c0b529521255e5f3fc0d866d64d9b5bd5fb804b737bb"
22/08/12 15:38:24 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:24 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary HTTP/1.1
22/08/12 15:38:24 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix4/_temporary HTTP/1.1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 5a076218-dada-102c-f2bc-75e004b85acf
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9c7ffe777ef558f9eb3f4476fd96efd3fd4734038120876fcd8e9b6c4525c78d
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143824Z
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix4/_temporary HTTP/1.1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 5a076218-dada-102c-f2bc-75e004b85acf[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9c7ffe777ef558f9eb3f4476fd96efd3fd4734038120876fcd8e9b6c4525c78d[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143824Z[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 073f2363-81ab-4837-accc-34d4573eaf39[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 073f2363-81ab-4837-accc-34d4573eaf39[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:24 GMT[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Length: 249[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 073f2363-81ab-4837-accc-34d4573eaf39
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 073f2363-81ab-4837-accc-34d4573eaf39
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Length: 249
22/08/12 15:38:24 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 073f2363-81ab-4837-accc-34d4573eaf39; S3 Extended Request ID: 073f2363-81ab-4837-accc-34d4573eaf39; Proxy: null), S3 Extended Request ID: 073f2363-81ab-4837-accc-34d4573eaf39
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 6
22/08/12 15:38:24 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 073f2363-81ab-4837-accc-34d4573eaf39; S3 Extended Request ID: 073f2363-81ab-4837-accc-34d4573eaf39; Proxy: null), S3 Extended Request ID: 073f2363-81ab-4837-accc-34d4573eaf39], RequestType=[GetObjectMetadataRequest], AWSRequestID=[073f2363-81ab-4837-accc-34d4573eaf39], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[18.652], HttpClientSendRequestTime=[1.582], HttpRequestTime=[16.583], ApiCallLatency=[18.314], RequestSigningTime=[0.586], CredentialsRequestTime=[0.01, 0.007], HttpClientReceiveResponseTime=[13.548],
22/08/12 15:38:24 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary
22/08/12 15:38:24 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4 (nonemptyprefix4); needEmptyDirectory=false
22/08/12 15:38:24 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4
22/08/12 15:38:24 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:24 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 4
22/08/12 15:38:24 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 71c31567-9fac-c7d4-e34b-56f05a6a6de7, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F
amz-sdk-invocation-id:71c31567-9fac-c7d4-e34b-56f05a6a6de7
amz-sdk-request:ttl=20220812T144144Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143824Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143824Z
20220812/us-east-1/s3/aws4_request
be681043d43c8e293b54e203e1672e1c3e67da930102df9fd3229461ebbffcbe"
22/08/12 15:38:24 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:24 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:24 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 71c31567-9fac-c7d4-e34b-56f05a6a6de7
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=ab9d50e808a05de63ae65365949c8a827099302af4dfe878de33608619482505
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143824Z
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 71c31567-9fac-c7d4-e34b-56f05a6a6de7[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=ab9d50e808a05de63ae65365949c8a827099302af4dfe878de33608619482505[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143824Z[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 7b0a4c5e-b2a2-48b9-ba55-74b67813db6b[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 7b0a4c5e-b2a2-48b9-ba55-74b67813db6b[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:24 GMT[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Length: 276[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/</Prefix></ListBucketResult>"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 7b0a4c5e-b2a2-48b9-ba55-74b67813db6b
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 7b0a4c5e-b2a2-48b9-ba55-74b67813db6b
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Length: 276
22/08/12 15:38:24 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:24 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:24 DEBUG request: Received successful response: 200, AWS Request ID: 7b0a4c5e-b2a2-48b9-ba55-74b67813db6b
22/08/12 15:38:24 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:24 DEBUG requestId: AWS Request ID: 7b0a4c5e-b2a2-48b9-ba55-74b67813db6b
22/08/12 15:38:24 DEBUG requestId: AWS Extended Request ID: 7b0a4c5e-b2a2-48b9-ba55-74b67813db6b
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 7
22/08/12 15:38:24 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[7b0a4c5e-b2a2-48b9-ba55-74b67813db6b], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[2.808], ClientExecuteTime=[21.695], HttpClientSendRequestTime=[1.535], HttpRequestTime=[16.774], ApiCallLatency=[21.394], RequestSigningTime=[0.696], CredentialsRequestTime=[0.009, 0.007], HttpClientReceiveResponseTime=[14.012],
22/08/12 15:38:24 DEBUG S3AFileSystem: LIST: duration 0:00.022s
22/08/12 15:38:24 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4
22/08/12 15:38:24 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4 (nonemptyprefix4); needEmptyDirectory=false
22/08/12 15:38:24 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 4
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 4
22/08/12 15:38:24 DEBUG S3AFileSystem: HEAD nonemptyprefix4 with change tracker null
22/08/12 15:38:24 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4 Headers: (amz-sdk-invocation-id: 20214092-e63c-0993-e612-4de9e8f97f3b, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4
amz-sdk-invocation-id:20214092-e63c-0993-e612-4de9e8f97f3b
amz-sdk-request:ttl=20220812T144144Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143824Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143824Z
20220812/us-east-1/s3/aws4_request
ffc5f43d19ab97e4fdbbb0599cdbaad7304f0a87b5e4d6affe139107b699925c"
22/08/12 15:38:24 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:24 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4 HTTP/1.1
22/08/12 15:38:24 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix4 HTTP/1.1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 20214092-e63c-0993-e612-4de9e8f97f3b
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=92d61d7849758f68afa7d1e9134d29152995f0893ec15cd82fea37d7476083d1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143824Z
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix4 HTTP/1.1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 20214092-e63c-0993-e612-4de9e8f97f3b[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=92d61d7849758f68afa7d1e9134d29152995f0893ec15cd82fea37d7476083d1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143824Z[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 48d655bd-b7c8-4674-8774-27e0890a6f7e[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 48d655bd-b7c8-4674-8774-27e0890a6f7e[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:24 GMT[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Length: 238[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 48d655bd-b7c8-4674-8774-27e0890a6f7e
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 48d655bd-b7c8-4674-8774-27e0890a6f7e
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Length: 238
22/08/12 15:38:24 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 48d655bd-b7c8-4674-8774-27e0890a6f7e; S3 Extended Request ID: 48d655bd-b7c8-4674-8774-27e0890a6f7e; Proxy: null), S3 Extended Request ID: 48d655bd-b7c8-4674-8774-27e0890a6f7e
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 8
22/08/12 15:38:24 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 48d655bd-b7c8-4674-8774-27e0890a6f7e; S3 Extended Request ID: 48d655bd-b7c8-4674-8774-27e0890a6f7e; Proxy: null), S3 Extended Request ID: 48d655bd-b7c8-4674-8774-27e0890a6f7e], RequestType=[GetObjectMetadataRequest], AWSRequestID=[48d655bd-b7c8-4674-8774-27e0890a6f7e], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[18.114], HttpClientSendRequestTime=[0.953], HttpRequestTime=[15.839], ApiCallLatency=[17.795], RequestSigningTime=[0.479], CredentialsRequestTime=[0.009, 0.006], HttpClientReceiveResponseTime=[13.499],
22/08/12 15:38:24 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4
22/08/12 15:38:24 DEBUG Invoker: Starting: PUT 0-byte object
22/08/12 15:38:24 DEBUG S3AFileSystem: PUT 0 bytes to nonemptyprefix4/_temporary/0/
22/08/12 15:38:24 DEBUG S3AFileSystem: PUT start 0 bytes
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request by 1 with final value 1
22/08/12 15:38:24 DEBUG request: Sending Request: PUT http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/ Headers: (amz-sdk-invocation-id: 464599fe-3f9a-199a-60e4-987b034cf981, Content-Length: 0, Content-Type: application/x-directory, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 Canonical Request: '"PUT
/master.rando2/nonemptyprefix4/_temporary/0/
amz-sdk-invocation-id:464599fe-3f9a-199a-60e4-987b034cf981
amz-sdk-request:attempt=1;max=21
amz-sdk-retry:0/0/500
content-length:86
content-type:application/x-directory
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20220812T143824Z
x-amz-decoded-content-length:0
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD"
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143824Z
20220812/us-east-1/s3/aws4_request
4c4d81419bf1e1669654cabd1a0d5ee989979cc1bd22fc14b529fe69c0b699cb"
22/08/12 15:38:24 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:24 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG MainClientExec: Executing request PUT /master.rando2/nonemptyprefix4/_temporary/0/ HTTP/1.1
22/08/12 15:38:24 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> PUT /master.rando2/nonemptyprefix4/_temporary/0/ HTTP/1.1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 464599fe-3f9a-199a-60e4-987b034cf981
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-request: attempt=1;max=21
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=786286e4fa11b529daadbe9c52766036cb049e3e5f804d87500358d225f877d9
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Type: application/x-directory
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143824Z
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-decoded-content-length: 0
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Length: 86
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Expect: 100-continue
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "PUT /master.rando2/nonemptyprefix4/_temporary/0/ HTTP/1.1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 464599fe-3f9a-199a-60e4-987b034cf981[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: attempt=1;max=21[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=786286e4fa11b529daadbe9c52766036cb049e3e5f804d87500358d225f877d9[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Type: application/x-directory[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143824Z[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-decoded-content-length: 0[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Length: 86[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Expect: 100-continue[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 100 Continue[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 100 Continue
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "0;chunk-signature=6cf8f7612e1b096235499b42ddda6c79a5cdb90bde5d1d2d20a84011518ede55[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Etag: "0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8"[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Version-Id: 569370187a1f4a47b8662f7a1a3ea766[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:24 GMT[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Length: 0[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Etag: "0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Version-Id: 569370187a1f4a47b8662f7a1a3ea766
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Length: 0
22/08/12 15:38:24 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG request: Received successful response: 200, AWS Request ID: null
22/08/12 15:38:24 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:24 DEBUG requestId: AWS Request ID: not available
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 9
22/08/12 15:38:24 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[PutObjectRequest], AWSRequestID=[null], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[0.183], ClientExecuteTime=[128.248], HttpClientSendRequestTime=[4.051], HttpRequestTime=[122.933], ApiCallLatency=[127.757], RequestSigningTime=[1.325], CredentialsRequestTime=[0.01, 0.007], HttpClientReceiveResponseTime=[117.565],
22/08/12 15:38:24 DEBUG S3AFileSystem: PUT completed success=true; 0 bytes
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request_completed by 1 with final value 1
22/08/12 15:38:24 DEBUG S3AFileSystem: Finished write to nonemptyprefix4/_temporary/0/, len 0. etag 0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8, version 569370187a1f4a47b8662f7a1a3ea766
22/08/12 15:38:24 DEBUG CallableSupplier: Starting: Waiting for task completion
22/08/12 15:38:24 DEBUG S3AFileSystem: Starting: Deleting 2 keys
22/08/12 15:38:24 DEBUG S3AFileSystem: Initiating delete operation for 2 objects
22/08/12 15:38:24 DEBUG S3AFileSystem: nonemptyprefix4/_temporary/
22/08/12 15:38:24 DEBUG S3AFileSystem: nonemptyprefix4/
22/08/12 15:38:24 DEBUG S3AFileSystem: Starting: DELETE 2 keys
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_bulk_delete_request by 1 with final value 1
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter object_delete_objects by 2 with final value 2
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(80)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 80
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 80
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(60)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 60
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 60
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(70)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 70
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 70
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(37)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 37
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 37
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(11)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 11
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 11
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(1)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 1
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 1
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(38)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 38
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 38
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(79)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 79
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 79
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(12)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 12
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 12
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(7)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 7
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 7
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(43)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 43
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 43
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(45)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 45
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 45
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(15)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 15
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 15
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(82)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 82
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 82
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(67)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 67
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 67
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(53)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 53
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 53
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(50)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 50
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 50
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(64)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 64
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 64
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(44)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 44
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 44
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(34)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 34
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 34
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(23)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 23
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 23
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(83)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 83
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 83
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(75)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 75
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 75
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(86)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 86
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 86
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(58)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 58
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 58
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(14)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 14
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 14
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(77)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 77
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 77
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(30)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 30
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 30
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(78)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 78
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 78
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(68)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 68
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 68
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(20)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 20
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 20
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(52)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 52
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 52
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(28)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 28
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 28
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(16)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 16
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 16
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(18)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 18
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 18
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(25)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 25
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 25
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(65)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 65
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 65
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(47)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 47
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 47
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(35)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 35
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 35
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(55)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 55
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 55
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(46)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 46
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 46
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(21)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 21
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 21
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(72)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 72
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 72
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(84)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 84
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 84
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(51)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 51
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 51
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(71)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 71
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 71
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(85)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 85
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 85
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(62)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 62
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 62
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanBroadcast(0)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning broadcast 0
22/08/12 15:38:24 DEBUG TorrentBroadcast: Unpersisting TorrentBroadcast 0
22/08/12 15:38:24 DEBUG BlockManagerStorageEndpoint: removing broadcast 0
22/08/12 15:38:24 DEBUG BlockManager: Removing broadcast 0
22/08/12 15:38:24 DEBUG BlockManager: Removing block broadcast_0_piece0
22/08/12 15:38:24 DEBUG MemoryStore: Block broadcast_0_piece0 of size 6711 dropped from memory (free 455449818)
22/08/12 15:38:24 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_0_piece0 for BlockManagerId(driver, 10.1.255.235, 33757, None)
22/08/12 15:38:24 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 10.1.255.235:33757 in memory (size: 6.6 KiB, free: 434.4 MiB)
22/08/12 15:38:24 DEBUG BlockManagerMaster: Updated info of block broadcast_0_piece0
22/08/12 15:38:24 DEBUG BlockManager: Told master about block broadcast_0_piece0
22/08/12 15:38:24 DEBUG BlockManager: Removing block broadcast_0
22/08/12 15:38:24 DEBUG MemoryStore: Block broadcast_0 of size 12720 dropped from memory (free 455462538)
22/08/12 15:38:24 DEBUG BlockManagerStorageEndpoint: Done removing broadcast 0, response is 0
22/08/12 15:38:24 DEBUG BlockManagerStorageEndpoint: Sent response: 0 to 10.1.255.235:45957
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned broadcast 0
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(13)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 13
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 13
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(81)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 81
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 81
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(74)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 74
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 74
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(2)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 2
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 2
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(22)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 22
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 22
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(57)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 57
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 57
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(39)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 39
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 39
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(26)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 26
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 26
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(5)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 5
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 5
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(73)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 73
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 73
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanBroadcast(1)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning broadcast 1
22/08/12 15:38:24 DEBUG TorrentBroadcast: Unpersisting TorrentBroadcast 1
22/08/12 15:38:24 DEBUG BlockManagerStorageEndpoint: removing broadcast 1
22/08/12 15:38:24 DEBUG BlockManager: Removing broadcast 1
22/08/12 15:38:24 DEBUG BlockManager: Removing block broadcast_1
22/08/12 15:38:24 DEBUG MemoryStore: Block broadcast_1 of size 12720 dropped from memory (free 455475258)
22/08/12 15:38:24 DEBUG BlockManager: Removing block broadcast_1_piece0
22/08/12 15:38:24 DEBUG MemoryStore: Block broadcast_1_piece0 of size 6717 dropped from memory (free 455481975)
22/08/12 15:38:24 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_1_piece0 for BlockManagerId(driver, 10.1.255.235, 33757, None)
22/08/12 15:38:24 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 10.1.255.235:33757 in memory (size: 6.6 KiB, free: 434.4 MiB)
22/08/12 15:38:24 DEBUG BlockManagerMaster: Updated info of block broadcast_1_piece0
22/08/12 15:38:24 DEBUG BlockManager: Told master about block broadcast_1_piece0
22/08/12 15:38:24 DEBUG BlockManagerStorageEndpoint: Done removing broadcast 1, response is 0
22/08/12 15:38:24 DEBUG BlockManagerStorageEndpoint: Sent response: 0 to 10.1.255.235:45957
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned broadcast 1
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(8)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 8
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 8
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(63)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 63
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 63
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(27)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 27
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 27
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(17)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 17
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 17
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(49)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 49
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 49
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(29)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 29
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 29
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(19)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 19
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 19
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanBroadcast(2)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning broadcast 2
22/08/12 15:38:24 DEBUG TorrentBroadcast: Unpersisting TorrentBroadcast 2
22/08/12 15:38:24 DEBUG BlockManagerStorageEndpoint: removing broadcast 2
22/08/12 15:38:24 DEBUG BlockManager: Removing broadcast 2
22/08/12 15:38:24 DEBUG BlockManager: Removing block broadcast_2_piece0
22/08/12 15:38:24 DEBUG MemoryStore: Block broadcast_2_piece0 of size 6719 dropped from memory (free 455488694)
22/08/12 15:38:24 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_2_piece0 for BlockManagerId(driver, 10.1.255.235, 33757, None)
22/08/12 15:38:24 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 10.1.255.235:33757 in memory (size: 6.6 KiB, free: 434.4 MiB)
22/08/12 15:38:24 DEBUG BlockManagerMaster: Updated info of block broadcast_2_piece0
22/08/12 15:38:24 DEBUG BlockManager: Told master about block broadcast_2_piece0
22/08/12 15:38:24 DEBUG BlockManager: Removing block broadcast_2
22/08/12 15:38:24 DEBUG MemoryStore: Block broadcast_2 of size 12720 dropped from memory (free 455501414)
22/08/12 15:38:24 DEBUG BlockManagerStorageEndpoint: Done removing broadcast 2, response is 0
22/08/12 15:38:24 DEBUG BlockManagerStorageEndpoint: Sent response: 0 to 10.1.255.235:45957
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned broadcast 2
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(9)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 9
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 9
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(42)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 42
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 42
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(87)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 87
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 87
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(32)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 32
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 32
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(48)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 48
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 48
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(66)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 66
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 66
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(54)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 54
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 54
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(61)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 61
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 61
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(33)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 33
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 33
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(56)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 56
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 56
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(41)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 41
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 41
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(6)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 6
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 6
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(3)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 3
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 3
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(24)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 24
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 24
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(69)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 69
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 69
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(76)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 76
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 76
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(40)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 40
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 40
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(31)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 31
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 31
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(4)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 4
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 4
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(36)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 36
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 36
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(10)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 10
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 10
22/08/12 15:38:24 DEBUG ContextCleaner: Got cleaning task CleanAccum(59)
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaning accumulator 59
22/08/12 15:38:24 DEBUG ContextCleaner: Cleaned accumulator 59
22/08/12 15:38:24 DEBUG request: Sending Request: POST http://localhost:30600 /master.rando2/ Parameters: ({"delete":[null]}Headers: (amz-sdk-invocation-id: c5c6c734-9368-6ce5-f969-0983e167267e, Content-Length: 135, Content-MD5: 3SF0VBSRpiKd/7k7YtvVCg==, Content-Type: application/xml, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 Canonical Request: '"POST
/master.rando2/
delete=
amz-sdk-invocation-id:c5c6c734-9368-6ce5-f969-0983e167267e
amz-sdk-request:ttl=20220812T144144Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-length:135
content-md5:3SF0VBSRpiKd/7k7YtvVCg==
content-type:application/xml
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:7e755bc57d27177b508bd2cc05d2b12b5d75af2ae9b934418236fe3af0a572b7
x-amz-date:20220812T143824Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
7e755bc57d27177b508bd2cc05d2b12b5d75af2ae9b934418236fe3af0a572b7"
22/08/12 15:38:24 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143824Z
20220812/us-east-1/s3/aws4_request
0f2a04e9bbc0d83371aa50dbf8d3530eeb10bf74f4017fa5945459f03e234c93"
22/08/12 15:38:24 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:24 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:24 DEBUG MainClientExec: Executing request POST /master.rando2/?delete HTTP/1.1
22/08/12 15:38:24 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> POST /master.rando2/?delete HTTP/1.1
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: c5c6c734-9368-6ce5-f969-0983e167267e
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=d93545f8f41f10d6bb5da272c0f0e99828a88fad39a5238a4b408fa17b32d019
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-MD5: 3SF0VBSRpiKd/7k7YtvVCg==
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Type: application/xml
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: 7e755bc57d27177b508bd2cc05d2b12b5d75af2ae9b934418236fe3af0a572b7
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143824Z
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Content-Length: 135
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "POST /master.rando2/?delete HTTP/1.1[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: c5c6c734-9368-6ce5-f969-0983e167267e[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144144Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=d93545f8f41f10d6bb5da272c0f0e99828a88fad39a5238a4b408fa17b32d019[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-MD5: 3SF0VBSRpiKd/7k7YtvVCg==[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: 7e755bc57d27177b508bd2cc05d2b12b5d75af2ae9b934418236fe3af0a572b7[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143824Z[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Content-Length: 135[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 >> "<Delete><Quiet>true</Quiet><Object><Key>nonemptyprefix4/_temporary/</Key></Object><Object><Key>nonemptyprefix4/</Key></Object></Delete>"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: c8a7af9c-e7a9-4c70-bede-5d939480673d[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: c8a7af9c-e7a9-4c70-bede-5d939480673d[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:24 GMT[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "Content-Length: 116[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:24 DEBUG wire: http-outgoing-0 << "<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>"
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: c8a7af9c-e7a9-4c70-bede-5d939480673d
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: c8a7af9c-e7a9-4c70-bede-5d939480673d
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG headers: http-outgoing-0 << Content-Length: 116
22/08/12 15:38:24 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:24 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:24 GMT
22/08/12 15:38:24 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$DeleteObjectsHandler
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:24 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:24 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:24 DEBUG request: Received successful response: 200, AWS Request ID: c8a7af9c-e7a9-4c70-bede-5d939480673d
22/08/12 15:38:24 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:24 DEBUG requestId: AWS Request ID: c8a7af9c-e7a9-4c70-bede-5d939480673d
22/08/12 15:38:24 DEBUG requestId: AWS Extended Request ID: c8a7af9c-e7a9-4c70-bede-5d939480673d
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 10
22/08/12 15:38:24 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[DeleteObjectsRequest], AWSRequestID=[c8a7af9c-e7a9-4c70-bede-5d939480673d], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[2.864], ClientExecuteTime=[85.191], HttpClientSendRequestTime=[0.644], HttpRequestTime=[79.356], ApiCallLatency=[84.921], RequestSigningTime=[1.517], CredentialsRequestTime=[0.008, 0.004], HttpClientReceiveResponseTime=[77.762],
22/08/12 15:38:24 DEBUG S3AFileSystem: DELETE 2 keys: duration 0:00.201s
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter fake_directories_deleted by 2 with final value 2
22/08/12 15:38:24 DEBUG S3AFileSystem: Deleting 2 keys: duration 0:00.203s
22/08/12 15:38:24 DEBUG CallableSupplier: Waiting for task completion: duration 0:00.204s
22/08/12 15:38:24 DEBUG Invoker: PUT 0-byte object : duration 0:00.341s
22/08/12 15:38:24 DEBUG Progress: PUT nonemptyprefix4/_temporary/0: 0 bytes
22/08/12 15:38:24 DEBUG IOStatisticsStoreImpl: Incrementing counter directories_created by 1 with final value 1
22/08/12 15:38:24 DEBUG WholeStageCodegenExec:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private scala.collection.Iterator rdd_input_0;
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1];
/* 011 */
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) {
/* 013 */ this.references = references;
/* 014 */ }
/* 015 */
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) {
/* 017 */ partitionIndex = index;
/* 018 */ this.inputs = inputs;
/* 019 */ rdd_input_0 = inputs[0];
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0);
/* 021 */
/* 022 */ }
/* 023 */
/* 024 */ protected void processNext() throws java.io.IOException {
/* 025 */ while ( rdd_input_0.hasNext()) {
/* 026 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next();
/* 027 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 028 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0);
/* 029 */ long rdd_value_0 = rdd_isNull_0 ?
/* 030 */ -1L : (rdd_row_0.getLong(0));
/* 031 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1);
/* 032 */ double rdd_value_1 = rdd_isNull_1 ?
/* 033 */ -1.0 : (rdd_row_0.getDouble(1));
/* 034 */ rdd_mutableStateArray_0[0].reset();
/* 035 */
/* 036 */ rdd_mutableStateArray_0[0].zeroOutNullBytes();
/* 037 */
/* 038 */ if (rdd_isNull_0) {
/* 039 */ rdd_mutableStateArray_0[0].setNullAt(0);
/* 040 */ } else {
/* 041 */ rdd_mutableStateArray_0[0].write(0, rdd_value_0);
/* 042 */ }
/* 043 */
/* 044 */ if (rdd_isNull_1) {
/* 045 */ rdd_mutableStateArray_0[0].setNullAt(1);
/* 046 */ } else {
/* 047 */ rdd_mutableStateArray_0[0].write(1, rdd_value_1);
/* 048 */ }
/* 049 */ append((rdd_mutableStateArray_0[0].getRow()));
/* 050 */ if (shouldStop()) return;
/* 051 */ }
/* 052 */ }
/* 053 */
/* 054 */ }
22/08/12 15:38:24 DEBUG CodeGenerator:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private scala.collection.Iterator rdd_input_0;
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1];
/* 011 */
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) {
/* 013 */ this.references = references;
/* 014 */ }
/* 015 */
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) {
/* 017 */ partitionIndex = index;
/* 018 */ this.inputs = inputs;
/* 019 */ rdd_input_0 = inputs[0];
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0);
/* 021 */
/* 022 */ }
/* 023 */
/* 024 */ protected void processNext() throws java.io.IOException {
/* 025 */ while ( rdd_input_0.hasNext()) {
/* 026 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next();
/* 027 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 028 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0);
/* 029 */ long rdd_value_0 = rdd_isNull_0 ?
/* 030 */ -1L : (rdd_row_0.getLong(0));
/* 031 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1);
/* 032 */ double rdd_value_1 = rdd_isNull_1 ?
/* 033 */ -1.0 : (rdd_row_0.getDouble(1));
/* 034 */ rdd_mutableStateArray_0[0].reset();
/* 035 */
/* 036 */ rdd_mutableStateArray_0[0].zeroOutNullBytes();
/* 037 */
/* 038 */ if (rdd_isNull_0) {
/* 039 */ rdd_mutableStateArray_0[0].setNullAt(0);
/* 040 */ } else {
/* 041 */ rdd_mutableStateArray_0[0].write(0, rdd_value_0);
/* 042 */ }
/* 043 */
/* 044 */ if (rdd_isNull_1) {
/* 045 */ rdd_mutableStateArray_0[0].setNullAt(1);
/* 046 */ } else {
/* 047 */ rdd_mutableStateArray_0[0].write(1, rdd_value_1);
/* 048 */ }
/* 049 */ append((rdd_mutableStateArray_0[0].getRow()));
/* 050 */ if (shouldStop()) return;
/* 051 */ }
/* 052 */ }
/* 053 */
/* 054 */ }
22/08/12 15:38:24 INFO CodeGenerator: Code generated in 16.08309 ms
22/08/12 15:38:24 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$doExecute$4$adapted
22/08/12 15:38:24 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$doExecute$4$adapted) is now cleaned +++
22/08/12 15:38:24 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$write$21
22/08/12 15:38:24 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$write$21) is now cleaned +++
22/08/12 15:38:24 INFO SparkContext: Starting job: parquet at NativeMethodAccessorImpl.java:0
22/08/12 15:38:24 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 7 took 0.000116 seconds
22/08/12 15:38:24 DEBUG DAGScheduler: Merging stage rdd profiles: Set()
22/08/12 15:38:24 INFO DAGScheduler: Got job 3 (parquet at NativeMethodAccessorImpl.java:0) with 16 output partitions
22/08/12 15:38:24 INFO DAGScheduler: Final stage: ResultStage 3 (parquet at NativeMethodAccessorImpl.java:0)
22/08/12 15:38:24 INFO DAGScheduler: Parents of final stage: List()
22/08/12 15:38:24 INFO DAGScheduler: Missing parents: List()
22/08/12 15:38:24 DEBUG DAGScheduler: submitStage(ResultStage 3 (name=parquet at NativeMethodAccessorImpl.java:0;jobs=3))
22/08/12 15:38:24 DEBUG DAGScheduler: missing: List()
22/08/12 15:38:24 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[7] at parquet at NativeMethodAccessorImpl.java:0), which has no missing parents
22/08/12 15:38:24 DEBUG DAGScheduler: submitMissingTasks(ResultStage 3)
22/08/12 15:38:25 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 214.9 KiB, free 434.2 MiB)
22/08/12 15:38:25 DEBUG BlockManager: Put block broadcast_3 locally took 1 ms
22/08/12 15:38:25 DEBUG BlockManager: Putting block broadcast_3 without replication took 1 ms
22/08/12 15:38:25 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 77.9 KiB, free 434.1 MiB)
22/08/12 15:38:25 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_3_piece0 for BlockManagerId(driver, 10.1.255.235, 33757, None)
22/08/12 15:38:25 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.1.255.235:33757 (size: 77.9 KiB, free: 434.3 MiB)
22/08/12 15:38:25 DEBUG BlockManagerMaster: Updated info of block broadcast_3_piece0
22/08/12 15:38:25 DEBUG BlockManager: Told master about block broadcast_3_piece0
22/08/12 15:38:25 DEBUG BlockManager: Put block broadcast_3_piece0 locally took 1 ms
22/08/12 15:38:25 DEBUG BlockManager: Putting block broadcast_3_piece0 without replication took 1 ms
22/08/12 15:38:25 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1513
22/08/12 15:38:25 INFO DAGScheduler: Submitting 16 missing tasks from ResultStage 3 (MapPartitionsRDD[7] at parquet at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
22/08/12 15:38:25 INFO TaskSchedulerImpl: Adding task set 3.0 with 16 tasks resource profile 0
22/08/12 15:38:25 DEBUG TaskSetManager: Epoch for TaskSet 3.0: 0
22/08/12 15:38:25 DEBUG TaskSetManager: Adding pending tasks took 0 ms
22/08/12 15:38:25 DEBUG TaskSetManager: Valid locality levels for TaskSet 3.0: NO_PREF, ANY
22/08/12 15:38:25 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_3.0, runningTasks: 0
22/08/12 15:38:25 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 16) (10.1.255.235, executor driver, partition 0, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 17) (10.1.255.235, executor driver, partition 1, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 2.0 in stage 3.0 (TID 18) (10.1.255.235, executor driver, partition 2, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 3.0 in stage 3.0 (TID 19) (10.1.255.235, executor driver, partition 3, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 4.0 in stage 3.0 (TID 20) (10.1.255.235, executor driver, partition 4, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 5.0 in stage 3.0 (TID 21) (10.1.255.235, executor driver, partition 5, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 6.0 in stage 3.0 (TID 22) (10.1.255.235, executor driver, partition 6, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 7.0 in stage 3.0 (TID 23) (10.1.255.235, executor driver, partition 7, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 8.0 in stage 3.0 (TID 24) (10.1.255.235, executor driver, partition 8, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 9.0 in stage 3.0 (TID 25) (10.1.255.235, executor driver, partition 9, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 10.0 in stage 3.0 (TID 26) (10.1.255.235, executor driver, partition 10, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 11.0 in stage 3.0 (TID 27) (10.1.255.235, executor driver, partition 11, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 12.0 in stage 3.0 (TID 28) (10.1.255.235, executor driver, partition 12, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 13.0 in stage 3.0 (TID 29) (10.1.255.235, executor driver, partition 13, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 14.0 in stage 3.0 (TID 30) (10.1.255.235, executor driver, partition 14, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO TaskSetManager: Starting task 15.0 in stage 3.0 (TID 31) (10.1.255.235, executor driver, partition 15, PROCESS_LOCAL, 4471 bytes) taskResourceAssignments Map()
22/08/12 15:38:25 INFO Executor: Running task 1.0 in stage 3.0 (TID 17)
22/08/12 15:38:25 INFO Executor: Running task 2.0 in stage 3.0 (TID 18)
22/08/12 15:38:25 INFO Executor: Running task 3.0 in stage 3.0 (TID 19)
22/08/12 15:38:25 INFO Executor: Running task 5.0 in stage 3.0 (TID 21)
22/08/12 15:38:25 INFO Executor: Running task 6.0 in stage 3.0 (TID 22)
22/08/12 15:38:25 INFO Executor: Running task 8.0 in stage 3.0 (TID 24)
22/08/12 15:38:25 INFO Executor: Running task 9.0 in stage 3.0 (TID 25)
22/08/12 15:38:25 INFO Executor: Running task 0.0 in stage 3.0 (TID 16)
22/08/12 15:38:25 INFO Executor: Running task 10.0 in stage 3.0 (TID 26)
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 3
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 1
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 2
22/08/12 15:38:25 INFO Executor: Running task 7.0 in stage 3.0 (TID 23)
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 4
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 5
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 6
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 7
22/08/12 15:38:25 INFO Executor: Running task 4.0 in stage 3.0 (TID 20)
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 8
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 9
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 10
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 11
22/08/12 15:38:25 INFO Executor: Running task 12.0 in stage 3.0 (TID 28)
22/08/12 15:38:25 INFO Executor: Running task 11.0 in stage 3.0 (TID 27)
22/08/12 15:38:25 INFO Executor: Running task 14.0 in stage 3.0 (TID 30)
22/08/12 15:38:25 INFO Executor: Running task 13.0 in stage 3.0 (TID 29)
22/08/12 15:38:25 INFO Executor: Running task 15.0 in stage 3.0 (TID 31)
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 13
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 12
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 14
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 15
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 16
22/08/12 15:38:25 DEBUG BlockManager: Getting local block broadcast_3
22/08/12 15:38:25 DEBUG BlockManager: Level for block broadcast_3 is StorageLevel(disk, memory, deserialized, 1 replicas)
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242991889787839631523_0003}; taskId=attempt_202208121538242991889787839631523_0003_m_000011_27, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@31f6631f}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242991889787839631523_0003}; taskId=attempt_202208121538242991889787839631523_0003_m_000011_27, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242991889787839631523_0003}; taskId=attempt_202208121538242991889787839631523_0003_m_000011_27, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@3a211648}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242991889787839631523_0003}; taskId=attempt_202208121538242991889787839631523_0003_m_000011_27, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO PythonRunner: Times: total = 87, boot = -2026, init = 2113, finish = 0
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242079469126800442703_0003}; taskId=attempt_202208121538242079469126800442703_0003_m_000013_29, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@7d603642}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242079469126800442703_0003}; taskId=attempt_202208121538242079469126800442703_0003_m_000013_29, status=''}
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243594149530430168495_0003}; taskId=attempt_202208121538243594149530430168495_0003_m_000008_24, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@1b6b9665}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243594149530430168495_0003}; taskId=attempt_202208121538243594149530430168495_0003_m_000008_24, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242079469126800442703_0003}; taskId=attempt_202208121538242079469126800442703_0003_m_000013_29, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@29fa8db2}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242079469126800442703_0003}; taskId=attempt_202208121538242079469126800442703_0003_m_000013_29, status=''}
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243594149530430168495_0003}; taskId=attempt_202208121538243594149530430168495_0003_m_000008_24, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@aebc88f}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243594149530430168495_0003}; taskId=attempt_202208121538243594149530430168495_0003_m_000008_24, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246186025400026495128_0003}; taskId=attempt_202208121538246186025400026495128_0003_m_000002_18, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@faad229}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246186025400026495128_0003}; taskId=attempt_202208121538246186025400026495128_0003_m_000002_18, status=''}
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538247838847636316877994_0003}; taskId=attempt_202208121538247838847636316877994_0003_m_000015_31, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@575075f2}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538247838847636316877994_0003}; taskId=attempt_202208121538247838847636316877994_0003_m_000015_31, status=''}
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538241431370443743393950_0003}; taskId=attempt_202208121538241431370443743393950_0003_m_000014_30, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@711c895}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538241431370443743393950_0003}; taskId=attempt_202208121538241431370443743393950_0003_m_000014_30, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538248384345865724627642_0003}; taskId=attempt_202208121538248384345865724627642_0003_m_000000_16, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@65b11fe8}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538248384345865724627642_0003}; taskId=attempt_202208121538248384345865724627642_0003_m_000000_16, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538247838847636316877994_0003}; taskId=attempt_202208121538247838847636316877994_0003_m_000015_31, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@3a0bb717}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538247838847636316877994_0003}; taskId=attempt_202208121538247838847636316877994_0003_m_000015_31, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538248057729003051628769_0003}; taskId=attempt_202208121538248057729003051628769_0003_m_000009_25, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@4e9db9a3}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538248057729003051628769_0003}; taskId=attempt_202208121538248057729003051628769_0003_m_000009_25, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538248057729003051628769_0003}; taskId=attempt_202208121538248057729003051628769_0003_m_000009_25, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@8a0aa82}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538248057729003051628769_0003}; taskId=attempt_202208121538248057729003051628769_0003_m_000009_25, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243576078083490261803_0003}; taskId=attempt_202208121538243576078083490261803_0003_m_000003_19, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@594a0fbd}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243576078083490261803_0003}; taskId=attempt_202208121538243576078083490261803_0003_m_000003_19, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538245411134883447354195_0003}; taskId=attempt_202208121538245411134883447354195_0003_m_000006_22, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@29a2713e}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538245411134883447354195_0003}; taskId=attempt_202208121538245411134883447354195_0003_m_000006_22, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538241773204340993498424_0003}; taskId=attempt_202208121538241773204340993498424_0003_m_000005_21, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@2512e7f9}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538241773204340993498424_0003}; taskId=attempt_202208121538241773204340993498424_0003_m_000005_21, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538241773204340993498424_0003}; taskId=attempt_202208121538241773204340993498424_0003_m_000005_21, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@2ce438ce}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538241773204340993498424_0003}; taskId=attempt_202208121538241773204340993498424_0003_m_000005_21, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538248384345865724627642_0003}; taskId=attempt_202208121538248384345865724627642_0003_m_000000_16, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@904cdf7}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538248384345865724627642_0003}; taskId=attempt_202208121538248384345865724627642_0003_m_000000_16, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO PythonRunner: Times: total = 118, boot = -1999, init = 2117, finish = 0
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243621455509773921197_0003}; taskId=attempt_202208121538243621455509773921197_0003_m_000001_17, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@4a2fc758}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243621455509773921197_0003}; taskId=attempt_202208121538243621455509773921197_0003_m_000001_17, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538245411134883447354195_0003}; taskId=attempt_202208121538245411134883447354195_0003_m_000006_22, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@4637e863}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538245411134883447354195_0003}; taskId=attempt_202208121538245411134883447354195_0003_m_000006_22, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243621455509773921197_0003}; taskId=attempt_202208121538243621455509773921197_0003_m_000001_17, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@29f7bd59}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243621455509773921197_0003}; taskId=attempt_202208121538243621455509773921197_0003_m_000001_17, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243576078083490261803_0003}; taskId=attempt_202208121538243576078083490261803_0003_m_000003_19, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@39754010}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538243576078083490261803_0003}; taskId=attempt_202208121538243576078083490261803_0003_m_000003_19, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO PythonRunner: Times: total = 119, boot = -2062, init = 2181, finish = 0
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242218734324492501469_0003}; taskId=attempt_202208121538242218734324492501469_0003_m_000004_20, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@f30e251}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242218734324492501469_0003}; taskId=attempt_202208121538242218734324492501469_0003_m_000004_20, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246737245493563198477_0003}; taskId=attempt_202208121538246737245493563198477_0003_m_000012_28, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@4b6ccb96}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246737245493563198477_0003}; taskId=attempt_202208121538246737245493563198477_0003_m_000012_28, status=''}
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242218734324492501469_0003}; taskId=attempt_202208121538242218734324492501469_0003_m_000004_20, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@ce4bf95}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538242218734324492501469_0003}; taskId=attempt_202208121538242218734324492501469_0003_m_000004_20, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246737245493563198477_0003}; taskId=attempt_202208121538246737245493563198477_0003_m_000012_28, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@448528e7}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246737245493563198477_0003}; taskId=attempt_202208121538246737245493563198477_0003_m_000012_28, status=''}
22/08/12 15:38:25 INFO PythonRunner: Times: total = 117, boot = -2019, init = 2136, finish = 0
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO PythonRunner: Times: total = 122, boot = -2011, init = 2133, finish = 0
22/08/12 15:38:25 INFO PythonRunner: Times: total = 169, boot = -2004, init = 2173, finish = 0
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538241431370443743393950_0003}; taskId=attempt_202208121538241431370443743393950_0003_m_000014_30, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@449e909}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538241431370443743393950_0003}; taskId=attempt_202208121538241431370443743393950_0003_m_000014_30, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246252528645038300397_0003}; taskId=attempt_202208121538246252528645038300397_0003_m_000007_23, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@7d2282f3}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246252528645038300397_0003}; taskId=attempt_202208121538246252528645038300397_0003_m_000007_23, status=''}
22/08/12 15:38:25 INFO PythonRunner: Times: total = 101, boot = -2005, init = 2106, finish = 0
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246252528645038300397_0003}; taskId=attempt_202208121538246252528645038300397_0003_m_000007_23, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@459b5ce}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246252528645038300397_0003}; taskId=attempt_202208121538246252528645038300397_0003_m_000007_23, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO PythonRunner: Times: total = 163, boot = -2022, init = 2185, finish = 0
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_20220812153824469238366268405742_0003}; taskId=attempt_20220812153824469238366268405742_0003_m_000010_26, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@46566ea8}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_20220812153824469238366268405742_0003}; taskId=attempt_20220812153824469238366268405742_0003_m_000010_26, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_20220812153824469238366268405742_0003}; taskId=attempt_20220812153824469238366268405742_0003_m_000010_26, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@341c2d2}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_20220812153824469238366268405742_0003}; taskId=attempt_20220812153824469238366268405742_0003_m_000010_26, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO PythonRunner: Times: total = 156, boot = -2025, init = 2181, finish = 0
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246186025400026495128_0003}; taskId=attempt_202208121538246186025400026495128_0003_m_000002_18, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@5129ce67}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix4 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121538246186025400026495128_0003}; taskId=attempt_202208121538246186025400026495128_0003_m_000002_18, status=''}
22/08/12 15:38:25 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 15:38:25 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 15:38:25 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 15:38:25 INFO PythonRunner: Times: total = 160, boot = -1998, init = 2158, finish = 0
22/08/12 15:38:25 INFO CodecConfig: Compression: SNAPPY
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 2
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 3
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 4
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 6
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 5
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 5
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 6
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 6
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23 with change tracker null
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 7
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 7
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 7
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30 with change tracker null
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 8
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 8
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22 with change tracker null
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30 Headers: (amz-sdk-invocation-id: d0ef810b-e8bb-43d9-5dab-5605939f5d45, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22 Headers: (amz-sdk-invocation-id: 8892aefa-56be-c298-b01e-2dd0fe266648, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 8
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26 (nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 9
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 9
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26 with change tracker null
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22
amz-sdk-invocation-id:8892aefa-56be-c298-b01e-2dd0fe266648
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 INFO CodecConfig: Compression: SNAPPY
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
3fd585b63d4deb8e7d96291ee27df5b29001ccd8d6d1272f65cce991cfcc7ce5"
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26 Headers: (amz-sdk-invocation-id: a69ff9f3-26da-a735-0200-cf44c63343bb, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 5
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26
amz-sdk-invocation-id:a69ff9f3-26da-a735-0200-cf44c63343bb
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
f6d555d477efa60c86965e9c2626c434b3a7ff134516aa00303c981981dad642"
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 10
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 10
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20 with change tracker null
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20 Headers: (amz-sdk-invocation-id: 08521f58-74cd-8004-fa06-10ee84ac32f9, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 10
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 INFO CodecConfig: Compression: SNAPPY
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 11
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 11
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18 with change tracker null
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23 Headers: (amz-sdk-invocation-id: 5b3c648b-3b87-4c74-3443-ea583c4baed8, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26 HTTP/1.1
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20
amz-sdk-invocation-id:08521f58-74cd-8004-fa06-10ee84ac32f9
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
8880d38868c2c983b204227ae7b0dc228079bf46e427fb407b230107c0a65b8a"
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18 Headers: (amz-sdk-invocation-id: 969ea54c-33ea-4c5b-2014-e3a2651cd04e, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 12
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 12
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 11
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 13
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 13
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28 with change tracker null
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28 Headers: (amz-sdk-invocation-id: ab582557-5722-91e6-435b-0200ed4d257a, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 INFO CodecConfig: Compression: SNAPPY
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28
amz-sdk-invocation-id:ab582557-5722-91e6-435b-0200ed4d257a
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
5388da2c7ff9a40985bac9104c99fcaedba68d8787dd266cf12938c2b5a6d8bb"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30
amz-sdk-invocation-id:d0ef810b-e8bb-43d9-5dab-5605939f5d45
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 9
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
7c1e39ff74a1c9abe4af616ce8717734dac5ac387ef877c8dacfadb0680cc2cb"
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 15:38:25 INFO PythonRunner: Times: total = 250, boot = 45, init = 205, finish = 0
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 14
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 14
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17 with change tracker null
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 12
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 15
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 15
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29 with change tracker null
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17 Headers: (amz-sdk-invocation-id: e9d3f845-162f-64ba-9477-58fd0ce6c1eb, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29 Headers: (amz-sdk-invocation-id: a3e43c4e-6eba-5caa-71c3-f49ac056a22d, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29
amz-sdk-invocation-id:a3e43c4e-6eba-5caa-71c3-f49ac056a22d
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
3e692d29b3d5edc51bdd1cdbb99de45521c3275c246160672c2022ec86c19445"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17
amz-sdk-invocation-id:e9d3f845-162f-64ba-9477-58fd0ce6c1eb
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
3d02ff36aa5f1e830e32b9f3a4d0e10d19d753eb8c26b2d76c563ff40fbd2457"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18
amz-sdk-invocation-id:969ea54c-33ea-4c5b-2014-e3a2651cd04e
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27 with change tracker null
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26 HTTP/1.1
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21 with change tracker null
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
46b29c2663b73a91f15fddfd720c9d5a9c10b12ac70bafb419bec49df440aa1b"
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21 Headers: (amz-sdk-invocation-id: 21900bd9-6aa1-9aa1-6632-4f93a308396c, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21
amz-sdk-invocation-id:21900bd9-6aa1-9aa1-6632-4f93a308396c
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
7bdbc55a70bbde5cd89563b8914ea46f7ff90926feb4d32852c9ed1ec12a6bfb"
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: a69ff9f3-26da-a735-0200-cf44c63343bb
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=eb3fa901030275f987f45bbf7741dea5da6f60dfd88c4c9acd0a3c1da6109cbe
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: a69ff9f3-26da-a735-0200-cf44c63343bb[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=eb3fa901030275f987f45bbf7741dea5da6f60dfd88c4c9acd0a3c1da6109cbe[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23
amz-sdk-invocation-id:5b3c648b-3b87-4c74-3443-ea583c4baed8
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
6b20a417fb710425a4853e6b7cb40f380698e1f7317cdd59f56602d3573975fa"
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27 Headers: (amz-sdk-invocation-id: 20e99976-34cb-74fa-2b12-68366a6aca90, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27
amz-sdk-invocation-id:20e99976-34cb-74fa-2b12-68366a6aca90
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
d468897e051392b253c67c201634dbcf13cf9e19a54c90f9018056aac6949911"
22/08/12 15:38:25 INFO PythonRunner: Times: total = 245, boot = 29, init = 216, finish = 0
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 13
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 16
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 16
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25 with change tracker null
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25 Headers: (amz-sdk-invocation-id: c0131e5c-0a7d-ec52-931c-60c3efb253b2, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25
amz-sdk-invocation-id:c0131e5c-0a7d-ec52-931c-60c3efb253b2
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
df655b7449f40fc52406fd18ffd1d772e6a6817649777b80e416a8317eff86e2"
22/08/12 15:38:25 INFO PythonRunner: Times: total = 262, boot = 99, init = 163, finish = 0
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 14
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 17
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 17
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19 with change tracker null
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19 Headers: (amz-sdk-invocation-id: f0fe262f-0fd6-b22c-889b-91874569b204, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19
amz-sdk-invocation-id:f0fe262f-0fd6-b22c-889b-91874569b204
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
7313bb625f92d47822f53d99d41a32c19b16a6e652e3bd93894f85de59c0d629"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: b3d08785-6271-4e6b-b20c-b9ea52054dda[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: b3d08785-6271-4e6b-b20c-b9ea52054dda[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "Content-Length: 320[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: b3d08785-6271-4e6b-b20c-b9ea52054dda
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: b3d08785-6271-4e6b-b20c-b9ea52054dda
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << Content-Length: 320
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:25 INFO PythonRunner: Times: total = 300, boot = 85, init = 214, finish = 1
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 15
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 18
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 18
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24 with change tracker null
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24 Headers: (amz-sdk-invocation-id: 8e9f83d5-bf29-8e50-ef3c-958658ee9016, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24
amz-sdk-invocation-id:8e9f83d5-bf29-8e50-ef3c-958658ee9016
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
91358edaec3485811d9e06f4a33d52c7485567db8b539553a7efc9254d6c6fa2"
22/08/12 15:38:25 INFO ParquetOutputFormat: Parquet block size to 134217728
22/08/12 15:38:25 INFO ParquetOutputFormat: Parquet block size to 134217728
22/08/12 15:38:25 INFO ParquetOutputFormat: Validation is off
22/08/12 15:38:25 INFO ParquetOutputFormat: Validation is off
22/08/12 15:38:25 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
22/08/12 15:38:25 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
22/08/12 15:38:25 INFO ParquetOutputFormat: Parquet properties are:
Parquet page size to 1048576
Parquet dictionary page size to 1048576
Dictionary is true
Writer version is: PARQUET_1_0
Page size checking is: estimated
Min row count for page size check is: 100
Max row count for page size check is: 10000
Truncate length for column indexes is: 64
Truncate length for statistics min/max is: 2147483647
Bloom filter enabled: false
Max Bloom filter size for a column is 1048576
Bloom filter expected number of distinct values are: null
Page row count limit to 20000
Writing page checksums is: on
22/08/12 15:38:25 INFO ParquetOutputFormat: Parquet properties are:
Parquet page size to 1048576
Parquet dictionary page size to 1048576
Dictionary is true
Writer version is: PARQUET_1_0
Page size checking is: estimated
Min row count for page size check is: 100
Max row count for page size check is: 10000
Truncate length for column indexes is: 64
Truncate length for statistics min/max is: 2147483647
Bloom filter enabled: false
Max Bloom filter size for a column is 1048576
Bloom filter expected number of distinct values are: null
Page row count limit to 20000
Writing page checksums is: on
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 1][route: {}->http://localhost:30600][total available: 0; route allocated: 2 of 96; total allocated: 2 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 2][route: {}->http://localhost:30600][total available: 0; route allocated: 3 of 96; total allocated: 3 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59622<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 3 of 96; total allocated: 3 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 3 of 96; total allocated: 3 of 96]
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 3][route: {}->http://localhost:30600][total available: 0; route allocated: 4 of 96; total allocated: 4 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 4 of 96; total allocated: 4 of 96]
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59624<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-3: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17 HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 4][route: {}->http://localhost:30600][total available: 0; route allocated: 5 of 96; total allocated: 5 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17 HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> amz-sdk-invocation-id: e9d3f845-162f-64ba-9477-58fd0ce6c1eb
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 5 of 96; total allocated: 5 of 96]
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=170f259fe2283085d48de2babc4294bdca3bef6c4a61a335a1572db0af0b8545
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: b3d08785-6271-4e6b-b20c-b9ea52054dda; S3 Extended Request ID: b3d08785-6271-4e6b-b20c-b9ea52054dda; Proxy: null), S3 Extended Request ID: b3d08785-6271-4e6b-b20c-b9ea52054dda
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 11
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59626<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: b3d08785-6271-4e6b-b20c-b9ea52054dda; S3 Extended Request ID: b3d08785-6271-4e6b-b20c-b9ea52054dda; Proxy: null), S3 Extended Request ID: b3d08785-6271-4e6b-b20c-b9ea52054dda], RequestType=[GetObjectMetadataRequest], AWSRequestID=[b3d08785-6271-4e6b-b20c-b9ea52054dda], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[82.523], HttpClientSendRequestTime=[21.141], HttpRequestTime=[79.382], ApiCallLatency=[82.334], RequestSigningTime=[0.639], CredentialsRequestTime=[0.005, 0.004], HttpClientReceiveResponseTime=[31.463],
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 5 of 96; total allocated: 5 of 96]
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 5
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 5 of 96; total allocated: 5 of 96]
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 8722ecf8-aff7-8ce4-5e93-0552ffe7e3b5, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 0; route allocated: 6 of 96; total allocated: 6 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 6][route: {}->http://localhost:30600][total available: 0; route allocated: 7 of 96; total allocated: 7 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-1: set socket timeout to 200000
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 7 of 96; total allocated: 7 of 96]
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-2: set socket timeout to 200000
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 7][route: {}->http://localhost:30600][total available: 0; route allocated: 8 of 96; total allocated: 8 of 96]
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59628<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23 HTTP/1.1
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22 HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59630<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22 HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28 HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> amz-sdk-invocation-id: 8892aefa-56be-c298-b01e-2dd0fe266648
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28 HTTP/1.1
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20 HTTP/1.1
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 8 of 96; total allocated: 8 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59632<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-4: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19 HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 8][route: {}->http://localhost:30600][total available: 0; route allocated: 9 of 96; total allocated: 9 of 96]
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19 HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> amz-sdk-invocation-id: f0fe262f-0fd6-b22c-889b-91874569b204
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=dedd5874801463734474686fb20ae0c40ccc60ad5e549db9f5433db56567480c
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "amz-sdk-invocation-id: f0fe262f-0fd6-b22c-889b-91874569b204[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=dedd5874801463734474686fb20ae0c40ccc60ad5e549db9f5433db56567480c[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_20220812153824469238366268405742_0003_m_000010_26%2F
amz-sdk-invocation-id:8722ecf8-aff7-8ce4-5e93-0552ffe7e3b5
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=eadc17bae817a1485a8f130e9e688729dd4946156861236e2f7c3c17dd7b4f8e
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
329eb6869b776e05c4ef8787dcd063d2b2979d7959342c446516176259212605"
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "amz-sdk-invocation-id: 8892aefa-56be-c298-b01e-2dd0fe266648[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23 HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: 5b3c648b-3b87-4c74-3443-ea583c4baed8
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=a5d8d4d5bd7a8a1232cdf8581d87cac20c9cc0c4d8484fde61395b3c8e11b1ce
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: 5b3c648b-3b87-4c74-3443-ea583c4baed8[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=a5d8d4d5bd7a8a1232cdf8581d87cac20c9cc0c4d8484fde61395b3c8e11b1ce[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> amz-sdk-invocation-id: ab582557-5722-91e6-435b-0200ed4d257a
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "amz-sdk-invocation-id: e9d3f845-162f-64ba-9477-58fd0ce6c1eb[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c56a15c595405aede60b96fd8dfbc6e2c07d7e60474c19778cc3f48a1233541d
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59634<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-6: set socket timeout to 200000
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30 HTTP/1.1
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=170f259fe2283085d48de2babc4294bdca3bef6c4a61a335a1572db0af0b8545[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=eadc17bae817a1485a8f130e9e688729dd4946156861236e2f7c3c17dd7b4f8e[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30 HTTP/1.1
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20 HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> amz-sdk-invocation-id: 08521f58-74cd-8004-fa06-10ee84ac32f9
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "amz-sdk-invocation-id: ab582557-5722-91e6-435b-0200ed4d257a[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=7538f267c5c75c82148eb18d28d0629d967602ab713bac9abc52a8b4c70b4fe6
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c56a15c595405aede60b96fd8dfbc6e2c07d7e60474c19778cc3f48a1233541d[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 9 of 96; total allocated: 9 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 9][route: {}->http://localhost:30600][total available: 0; route allocated: 10 of 96; total allocated: 10 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59640<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-9: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27 HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27 HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> amz-sdk-invocation-id: 20e99976-34cb-74fa-2b12-68366a6aca90
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 10 of 96; total allocated: 10 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 10][route: {}->http://localhost:30600][total available: 0; route allocated: 11 of 96; total allocated: 11 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "amz-sdk-invocation-id: 08521f58-74cd-8004-fa06-10ee84ac32f9[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> amz-sdk-invocation-id: d0ef810b-e8bb-43d9-5dab-5605939f5d45
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59636<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=f0649676713467884760c02446a01830b0c3ebe33f77b8f0c14ea2bef5998700
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 11 of 96; total allocated: 11 of 96]
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 10 of 96; total allocated: 10 of 96]
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 11][route: {}->http://localhost:30600][total available: 0; route allocated: 12 of 96; total allocated: 12 of 96]
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 12 of 96; total allocated: 12 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 12][route: {}->http://localhost:30600][total available: 0; route allocated: 13 of 96; total allocated: 13 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59642<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-12: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_20220812153824469238366268405742_0003_m_000010_26%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_20220812153824469238366268405742_0003_m_000010_26%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> amz-sdk-invocation-id: 8722ecf8-aff7-8ce4-5e93-0552ffe7e3b5
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=19616e2707182e8dc4ee20da791ac22b6a0022e58839c7d09146c80f131a7fdb
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_20220812153824469238366268405742_0003_m_000010_26%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "amz-sdk-invocation-id: 8722ecf8-aff7-8ce4-5e93-0552ffe7e3b5[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=19616e2707182e8dc4ee20da791ac22b6a0022e58839c7d09146c80f131a7fdb[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-8: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25 HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29 HTTP/1.1
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 11 of 96; total allocated: 11 of 96]
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=baaed20977152bfc2d34e25bc4871df22ee5d70da76ff09a8e4c3005d1702de2
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 13][route: {}->http://localhost:30600][total available: 0; route allocated: 14 of 96; total allocated: 14 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25 HTTP/1.1
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59644<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "amz-sdk-invocation-id: 20e99976-34cb-74fa-2b12-68366a6aca90[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-10: set socket timeout to 200000
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18 HTTP/1.1
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59646<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=f0649676713467884760c02446a01830b0c3ebe33f77b8f0c14ea2bef5998700[\r][\n]"
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29 HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> amz-sdk-invocation-id: a3e43c4e-6eba-5caa-71c3-f49ac056a22d
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=44ddb2f4b5617e68816ea7ace08d81bd92d71476aab176d0838374c6dba0dfc8
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "amz-sdk-invocation-id: a3e43c4e-6eba-5caa-71c3-f49ac056a22d[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=44ddb2f4b5617e68816ea7ace08d81bd92d71476aab176d0838374c6dba0dfc8[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59648<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-11: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21 HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21 HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> amz-sdk-invocation-id: 21900bd9-6aa1-9aa1-6632-4f93a308396c
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=d0b982c3bce449b7afad961738e96cd0d788b17ce6ee9afd6108d22f51bef729
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "amz-sdk-invocation-id: 21900bd9-6aa1-9aa1-6632-4f93a308396c[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=d0b982c3bce449b7afad961738e96cd0d788b17ce6ee9afd6108d22f51bef729[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18 HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> amz-sdk-invocation-id: 969ea54c-33ea-4c5b-2014-e3a2651cd04e
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=3d3c578d61ed6ce43efa3e66f9854b5dea3d5bfc93e331e7fa2fd37b7f4df4c4
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "amz-sdk-invocation-id: 969ea54c-33ea-4c5b-2014-e3a2651cd04e[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=3d3c578d61ed6ce43efa3e66f9854b5dea3d5bfc93e331e7fa2fd37b7f4df4c4[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-13: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24 HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24 HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> amz-sdk-invocation-id: 8e9f83d5-bf29-8e50-ef3c-958658ee9016
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=934b4df3c8f05345fe76eed4a9e49a9e1d7daa5b665d931939097b14bf629108
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=7538f267c5c75c82148eb18d28d0629d967602ab713bac9abc52a8b4c70b4fe6[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: c0131e5c-0a7d-ec52-931c-60c3efb253b2
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=1d072b7b3af6c2ca5a59f9a7845de937a53c0a626b3b9061c1ee2960f738c522
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: c0131e5c-0a7d-ec52-931c-60c3efb253b2[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=1d072b7b3af6c2ca5a59f9a7845de937a53c0a626b3b9061c1ee2960f738c522[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30 HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "amz-sdk-invocation-id: d0ef810b-e8bb-43d9-5dab-5605939f5d45[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=baaed20977152bfc2d34e25bc4871df22ee5d70da76ff09a8e4c3005d1702de2[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "amz-sdk-invocation-id: 8e9f83d5-bf29-8e50-ef3c-958658ee9016[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=934b4df3c8f05345fe76eed4a9e49a9e1d7daa5b665d931939097b14bf629108[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "[\r][\n]"
22/08/12 15:38:25 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "a",
"type" : "long",
"nullable" : true,
"metadata" : { }
}, {
"name" : "b",
"type" : "double",
"nullable" : true,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
optional int64 a;
optional double b;
}
22/08/12 15:38:25 DEBUG EncryptionPropertiesFactory: EncryptionPropertiesFactory is not configured - name not found in hadoop config
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "X-Amz-Id-2: 5f8f1e20-f80f-4834-bba5-aa1cbd2209c5[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "X-Amz-Request-Id: 5f8f1e20-f80f-4834-bba5-aa1cbd2209c5[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << X-Amz-Id-2: 5f8f1e20-f80f-4834-bba5-aa1cbd2209c5
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << X-Amz-Request-Id: 5f8f1e20-f80f-4834-bba5-aa1cbd2209c5
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << Content-Length: 321
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 3][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-3: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 3][route: {}->http://localhost:30600][total available: 1; route allocated: 14 of 96; total allocated: 14 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "a",
"type" : "long",
"nullable" : true,
"metadata" : { }
}, {
"name" : "b",
"type" : "double",
"nullable" : true,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
optional int64 a;
optional double b;
}
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 5f8f1e20-f80f-4834-bba5-aa1cbd2209c5; S3 Extended Request ID: 5f8f1e20-f80f-4834-bba5-aa1cbd2209c5; Proxy: null), S3 Extended Request ID: 5f8f1e20-f80f-4834-bba5-aa1cbd2209c5
22/08/12 15:38:25 DEBUG EncryptionPropertiesFactory: EncryptionPropertiesFactory is not configured - name not found in hadoop config
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 12
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 5f8f1e20-f80f-4834-bba5-aa1cbd2209c5; S3 Extended Request ID: 5f8f1e20-f80f-4834-bba5-aa1cbd2209c5; Proxy: null), S3 Extended Request ID: 5f8f1e20-f80f-4834-bba5-aa1cbd2209c5], RequestType=[GetObjectMetadataRequest], AWSRequestID=[5f8f1e20-f80f-4834-bba5-aa1cbd2209c5], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=3, ClientExecuteTime=[116.64], HttpClientSendRequestTime=[8.696], HttpRequestTime=[56.07], ApiCallLatency=[116.112], RequestSigningTime=[0.912], CredentialsRequestTime=[0.016, 0.007], HttpClientReceiveResponseTime=[43.638],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 6
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 0d1136a8-1d60-f617-c6f6-58f3691c9ef7, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243621455509773921197_0003_m_000001_17%2F
amz-sdk-invocation-id:0d1136a8-1d60-f617-c6f6-58f3691c9ef7
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
0e103e6cb1ffcc88fbad464a56108762d2fd3f815bde3c9fd7d0bf65a70fdc64"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 14 of 96; total allocated: 14 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 3][route: {}->http://localhost:30600][total available: 0; route allocated: 14 of 96; total allocated: 14 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-3: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-3: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243621455509773921197_0003_m_000001_17%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243621455509773921197_0003_m_000001_17%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> amz-sdk-invocation-id: 0d1136a8-1d60-f617-c6f6-58f3691c9ef7
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=396b43d65f92ca2e85bb8a63c7533e06d843946faff980ebc15e07cf6021d6d1
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243621455509773921197_0003_m_000001_17%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "amz-sdk-invocation-id: 0d1136a8-1d60-f617-c6f6-58f3691c9ef7[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=396b43d65f92ca2e85bb8a63c7533e06d843946faff980ebc15e07cf6021d6d1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_create by 1 with final value 1
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_create by 1 with final value 2
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet); needEmptyDirectory=false
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
22/08/12 15:38:25 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 19
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 20
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 19
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 20
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet with change tracker null
22/08/12 15:38:25 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet with change tracker null
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet Headers: (amz-sdk-invocation-id: 191c3d7c-1133-927c-081f-49bf51814e33, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet Headers: (amz-sdk-invocation-id: 95b77d76-02b5-869c-a4c2-61b355feabfb, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
amz-sdk-invocation-id:95b77d76-02b5-869c-a4c2-61b355feabfb
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
amz-sdk-invocation-id:191c3d7c-1133-927c-081f-49bf51814e33
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
92b0fa6f9efcd6e2a181a28826c519a554417b06e5eec8f38b627e6c9853e399"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
a849b26135b01d66ec4185906cc136243801e854c393706ebb9fd07a4e356a75"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 14 of 96; total allocated: 14 of 96]
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 14][route: {}->http://localhost:30600][total available: 0; route allocated: 15 of 96; total allocated: 15 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 15 of 96; total allocated: 15 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 15][route: {}->http://localhost:30600][total available: 0; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59652<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-14: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> amz-sdk-invocation-id: 95b77d76-02b5-869c-a4c2-61b355feabfb
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59654<->127.0.0.1:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=d7af28f0e1879439ee7afee83d4aef93627848f4bb1bdc01d2953f9d0d8bc8c6
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "amz-sdk-invocation-id: 95b77d76-02b5-869c-a4c2-61b355feabfb[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=d7af28f0e1879439ee7afee83d4aef93627848f4bb1bdc01d2953f9d0d8bc8c6[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "X-Amz-Id-2: 5fc319ca-9c89-4a06-8dba-1aa7bbcb99db[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "X-Amz-Request-Id: 5fc319ca-9c89-4a06-8dba-1aa7bbcb99db[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << X-Amz-Id-2: 5fc319ca-9c89-4a06-8dba-1aa7bbcb99db
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << X-Amz-Request-Id: 5fc319ca-9c89-4a06-8dba-1aa7bbcb99db
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << Content-Length: 321
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 7][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 7][route: {}->http://localhost:30600][total available: 1; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 5fc319ca-9c89-4a06-8dba-1aa7bbcb99db; S3 Extended Request ID: 5fc319ca-9c89-4a06-8dba-1aa7bbcb99db; Proxy: null), S3 Extended Request ID: 5fc319ca-9c89-4a06-8dba-1aa7bbcb99db
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 13
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 5fc319ca-9c89-4a06-8dba-1aa7bbcb99db; S3 Extended Request ID: 5fc319ca-9c89-4a06-8dba-1aa7bbcb99db; Proxy: null), S3 Extended Request ID: 5fc319ca-9c89-4a06-8dba-1aa7bbcb99db], RequestType=[GetObjectMetadataRequest], AWSRequestID=[5fc319ca-9c89-4a06-8dba-1aa7bbcb99db], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=2, ClientExecuteTime=[149.225], HttpClientSendRequestTime=[6.999], HttpRequestTime=[75.149], ApiCallLatency=[149.014], RequestSigningTime=[0.374], CredentialsRequestTime=[0.005, 0.004], HttpClientReceiveResponseTime=[58.231],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 7
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 6a274340-6255-a66f-5f19-5646c093f528, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246737245493563198477_0003_m_000012_28%2F
amz-sdk-invocation-id:6a274340-6255-a66f-5f19-5646c093f528
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
16d758ee8f8d22812bd872a84c15b7f7886f41c087e981e2c26a44ca9c62d24f"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 7][route: {}->http://localhost:30600][total available: 0; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246737245493563198477_0003_m_000012_28%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246737245493563198477_0003_m_000012_28%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> amz-sdk-invocation-id: 6a274340-6255-a66f-5f19-5646c093f528
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6465ee2179f5ec40b3b2945c15acc10b5f22a3df37b9e096cf32f72023ad0d5e
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246737245493563198477_0003_m_000012_28%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "amz-sdk-invocation-id: 6a274340-6255-a66f-5f19-5646c093f528[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6465ee2179f5ec40b3b2945c15acc10b5f22a3df37b9e096cf32f72023ad0d5e[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-15: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> amz-sdk-invocation-id: 191c3d7c-1133-927c-081f-49bf51814e33
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=60c1c56691db2139f48cde0fa5ff3d837ef29ae7d275fe2e2c7a2b041a2f3dac
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "amz-sdk-invocation-id: 191c3d7c-1133-927c-081f-49bf51814e33[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=60c1c56691db2139f48cde0fa5ff3d837ef29ae7d275fe2e2c7a2b041a2f3dac[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "X-Amz-Id-2: 96cf3b9c-f187-4768-a8fb-108d4b70d908[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "X-Amz-Request-Id: 96cf3b9c-f187-4768-a8fb-108d4b70d908[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << X-Amz-Id-2: 96cf3b9c-f187-4768-a8fb-108d4b70d908
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << X-Amz-Request-Id: 96cf3b9c-f187-4768-a8fb-108d4b70d908
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << Content-Length: 321
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 2][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-2: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 2][route: {}->http://localhost:30600][total available: 1; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 96cf3b9c-f187-4768-a8fb-108d4b70d908; S3 Extended Request ID: 96cf3b9c-f187-4768-a8fb-108d4b70d908; Proxy: null), S3 Extended Request ID: 96cf3b9c-f187-4768-a8fb-108d4b70d908
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 14
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 96cf3b9c-f187-4768-a8fb-108d4b70d908; S3 Extended Request ID: 96cf3b9c-f187-4768-a8fb-108d4b70d908; Proxy: null), S3 Extended Request ID: 96cf3b9c-f187-4768-a8fb-108d4b70d908], RequestType=[GetObjectMetadataRequest], AWSRequestID=[96cf3b9c-f187-4768-a8fb-108d4b70d908], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[166.441], HttpClientSendRequestTime=[7.036], HttpRequestTime=[162.917], ApiCallLatency=[166.006], RequestSigningTime=[0.489], CredentialsRequestTime=[0.005, 0.005], HttpClientReceiveResponseTime=[70.826],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 8
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 243d89db-51b5-7b69-77e0-fc232e658cdd, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538245411134883447354195_0003_m_000006_22%2F
amz-sdk-invocation-id:243d89db-51b5-7b69-77e0-fc232e658cdd
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
95bb0e4bf5a783d69a6101182e517293da116fbd6d7a86a952da46b1643f03c9"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 2][route: {}->http://localhost:30600][total available: 0; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-2: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-2: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538245411134883447354195_0003_m_000006_22%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538245411134883447354195_0003_m_000006_22%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> amz-sdk-invocation-id: 243d89db-51b5-7b69-77e0-fc232e658cdd
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0bf364955575c7a16ec037d2f5c393d8d337f72a49435dcc8e092a1f37015ae3
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538245411134883447354195_0003_m_000006_22%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "amz-sdk-invocation-id: 243d89db-51b5-7b69-77e0-fc232e658cdd[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0bf364955575c7a16ec037d2f5c393d8d337f72a49435dcc8e092a1f37015ae3[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "X-Amz-Id-2: 49f28135-3b91-452e-9ad9-863ce007387f[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "X-Amz-Request-Id: 49f28135-3b91-452e-9ad9-863ce007387f[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << X-Amz-Id-2: 49f28135-3b91-452e-9ad9-863ce007387f
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << X-Amz-Request-Id: 49f28135-3b91-452e-9ad9-863ce007387f
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << Content-Length: 321
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 13][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-13: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 13][route: {}->http://localhost:30600][total available: 1; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "X-Amz-Id-2: 8ee7264a-fdfe-43c6-8542-cde66b44fb9b[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "X-Amz-Request-Id: 8ee7264a-fdfe-43c6-8542-cde66b44fb9b[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << X-Amz-Id-2: 8ee7264a-fdfe-43c6-8542-cde66b44fb9b
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << X-Amz-Request-Id: 8ee7264a-fdfe-43c6-8542-cde66b44fb9b
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 49f28135-3b91-452e-9ad9-863ce007387f; S3 Extended Request ID: 49f28135-3b91-452e-9ad9-863ce007387f; Proxy: null), S3 Extended Request ID: 49f28135-3b91-452e-9ad9-863ce007387f
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << Content-Length: 321
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 15
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 49f28135-3b91-452e-9ad9-863ce007387f; S3 Extended Request ID: 49f28135-3b91-452e-9ad9-863ce007387f; Proxy: null), S3 Extended Request ID: 49f28135-3b91-452e-9ad9-863ce007387f], RequestType=[GetObjectMetadataRequest], AWSRequestID=[49f28135-3b91-452e-9ad9-863ce007387f], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=4, ClientExecuteTime=[110.786], HttpClientSendRequestTime=[3.021], HttpRequestTime=[93.374], ApiCallLatency=[110.2], RequestSigningTime=[0.68], CredentialsRequestTime=[0.021, 0.011], HttpClientReceiveResponseTime=[70.992],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 4][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-4: set socket timeout to 0
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 9
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 4][route: {}->http://localhost:30600][total available: 2; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 8ee7264a-fdfe-43c6-8542-cde66b44fb9b; S3 Extended Request ID: 8ee7264a-fdfe-43c6-8542-cde66b44fb9b; Proxy: null), S3 Extended Request ID: 8ee7264a-fdfe-43c6-8542-cde66b44fb9b
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 16
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: b80804f5-488e-d0b4-6133-ff01e47fe7af, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 8ee7264a-fdfe-43c6-8542-cde66b44fb9b; S3 Extended Request ID: 8ee7264a-fdfe-43c6-8542-cde66b44fb9b; Proxy: null), S3 Extended Request ID: 8ee7264a-fdfe-43c6-8542-cde66b44fb9b], RequestType=[GetObjectMetadataRequest], AWSRequestID=[8ee7264a-fdfe-43c6-8542-cde66b44fb9b], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=4, ClientExecuteTime=[129.133], HttpClientSendRequestTime=[0.528], HttpRequestTime=[96.686], ApiCallLatency=[128.652], RequestSigningTime=[1.47], CredentialsRequestTime=[0.015, 0.005], HttpClientReceiveResponseTime=[88.147],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 10
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243594149530430168495_0003_m_000008_24%2F
amz-sdk-invocation-id:b80804f5-488e-d0b4-6133-ff01e47fe7af
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
79715468006676a7bf44c4ddeca5db42a0c3e4168900f77945f3188fd117d982"
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: c71a5fb7-827c-005b-3ac8-a65d10bdddbc, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 2; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243576078083490261803_0003_m_000003_19%2F
amz-sdk-invocation-id:c71a5fb7-827c-005b-3ac8-a65d10bdddbc
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 4][route: {}->http://localhost:30600][total available: 1; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
4bd4e272b0561d07990e71ea23c3e383fbecaf18cd04c3ae8a79b5ff92278814"
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-4: set socket timeout to 200000
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-4: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243594149530430168495_0003_m_000008_24%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243594149530430168495_0003_m_000008_24%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> amz-sdk-invocation-id: b80804f5-488e-d0b4-6133-ff01e47fe7af
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6c8162a3a3b7042d8201275b4fd8cb99019386f247de370363a8e6ba6a2c2840
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243594149530430168495_0003_m_000008_24%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "amz-sdk-invocation-id: b80804f5-488e-d0b4-6133-ff01e47fe7af[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6c8162a3a3b7042d8201275b4fd8cb99019386f247de370363a8e6ba6a2c2840[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 13][route: {}->http://localhost:30600][total available: 0; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-13: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-13: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243576078083490261803_0003_m_000003_19%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243576078083490261803_0003_m_000003_19%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> amz-sdk-invocation-id: c71a5fb7-827c-005b-3ac8-a65d10bdddbc
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=62b90dbaebc842f4d08375a6cc273f869eafcdda2256f12743734ec3db905d2e
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538243576078083490261803_0003_m_000003_19%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "amz-sdk-invocation-id: c71a5fb7-827c-005b-3ac8-a65d10bdddbc[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=62b90dbaebc842f4d08375a6cc273f869eafcdda2256f12743734ec3db905d2e[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: a5eb9589-1b49-4e81-8b13-27aa50eeeb05[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: a5eb9589-1b49-4e81-8b13-27aa50eeeb05[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: a5eb9589-1b49-4e81-8b13-27aa50eeeb05
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: a5eb9589-1b49-4e81-8b13-27aa50eeeb05
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << Content-Length: 321
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: a5eb9589-1b49-4e81-8b13-27aa50eeeb05; S3 Extended Request ID: a5eb9589-1b49-4e81-8b13-27aa50eeeb05; Proxy: null), S3 Extended Request ID: a5eb9589-1b49-4e81-8b13-27aa50eeeb05
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 17
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: a5eb9589-1b49-4e81-8b13-27aa50eeeb05; S3 Extended Request ID: a5eb9589-1b49-4e81-8b13-27aa50eeeb05; Proxy: null), S3 Extended Request ID: a5eb9589-1b49-4e81-8b13-27aa50eeeb05], RequestType=[GetObjectMetadataRequest], AWSRequestID=[a5eb9589-1b49-4e81-8b13-27aa50eeeb05], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=3, ClientExecuteTime=[149.096], HttpClientSendRequestTime=[6.64], HttpRequestTime=[111.638], ApiCallLatency=[148.585], RequestSigningTime=[0.632], CredentialsRequestTime=[0.021, 0.003], HttpClientReceiveResponseTime=[87.526],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 11
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: d63424ed-7cc4-61fd-f9e8-49b99bce1ddb, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248057729003051628769_0003_m_000009_25%2F
amz-sdk-invocation-id:d63424ed-7cc4-61fd-f9e8-49b99bce1ddb
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
8f4bf8769936c1ca45b493d473d33ae103f241d41ca5b05d546845879a0a3d1e"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248057729003051628769_0003_m_000009_25%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248057729003051628769_0003_m_000009_25%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: d63424ed-7cc4-61fd-f9e8-49b99bce1ddb
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9efe350c97135ed5173cd68216b55d63e5eb09e2f3a4db3ea12d77e3dbf903b9
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248057729003051628769_0003_m_000009_25%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: d63424ed-7cc4-61fd-f9e8-49b99bce1ddb[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9efe350c97135ed5173cd68216b55d63e5eb09e2f3a4db3ea12d77e3dbf903b9[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 << "X-Amz-Id-2: 958a780d-d3d2-49bc-89e5-915ff2bb227c[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 << "X-Amz-Request-Id: 958a780d-d3d2-49bc-89e5-915ff2bb227c[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 << "Content-Length: 358[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-12 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 << X-Amz-Id-2: 958a780d-d3d2-49bc-89e5-915ff2bb227c
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 << X-Amz-Request-Id: 958a780d-d3d2-49bc-89e5-915ff2bb227c
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-12 << Content-Length: 358
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "X-Amz-Id-2: d31230fa-be6e-4fb1-905a-07e578b44a19[\r][\n]"
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "X-Amz-Request-Id: d31230fa-be6e-4fb1-905a-07e578b44a19[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-3 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << X-Amz-Id-2: d31230fa-be6e-4fb1-905a-07e578b44a19
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << X-Amz-Request-Id: d31230fa-be6e-4fb1-905a-07e578b44a19
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-3 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 12][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-12: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 12][route: {}->http://localhost:30600][total available: 1; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: 958a780d-d3d2-49bc-89e5-915ff2bb227c
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: 958a780d-d3d2-49bc-89e5-915ff2bb227c
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: 958a780d-d3d2-49bc-89e5-915ff2bb227c
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 18
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[958a780d-d3d2-49bc-89e5-915ff2bb227c], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, HttpClientPoolLeasedCount=9, ResponseProcessingTime=[4.264], ClientExecuteTime=[128.179], HttpClientSendRequestTime=[0.903], HttpRequestTime=[119.166], ApiCallLatency=[127.782], RequestSigningTime=[2.483], CredentialsRequestTime=[0.015, 0.01], HttpClientReceiveResponseTime=[112.123],
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 3][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-3: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 3][route: {}->http://localhost:30600][total available: 2; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "X-Amz-Id-2: 30e0c8df-54cd-4d19-b1b6-620e13c07771[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "X-Amz-Request-Id: 30e0c8df-54cd-4d19-b1b6-620e13c07771[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "[\r][\n]"
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.133s
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << X-Amz-Id-2: 30e0c8df-54cd-4d19-b1b6-620e13c07771
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << X-Amz-Request-Id: 30e0c8df-54cd-4d19-b1b6-620e13c07771
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << Content-Length: 321
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_20220812153824469238366268405742_0003_m_000010_26
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 1][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-1: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 1][route: {}->http://localhost:30600][total available: 3; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 30e0c8df-54cd-4d19-b1b6-620e13c07771; S3 Extended Request ID: 30e0c8df-54cd-4d19-b1b6-620e13c07771; Proxy: null), S3 Extended Request ID: 30e0c8df-54cd-4d19-b1b6-620e13c07771
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 19
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 30e0c8df-54cd-4d19-b1b6-620e13c07771; S3 Extended Request ID: 30e0c8df-54cd-4d19-b1b6-620e13c07771; Proxy: null), S3 Extended Request ID: 30e0c8df-54cd-4d19-b1b6-620e13c07771], RequestType=[GetObjectMetadataRequest], AWSRequestID=[30e0c8df-54cd-4d19-b1b6-620e13c07771], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=1, ClientExecuteTime=[216.74], HttpClientSendRequestTime=[12.746], HttpRequestTime=[213.789], ApiCallLatency=[216.467], RequestSigningTime=[0.752], CredentialsRequestTime=[0.007, 0.005], HttpClientReceiveResponseTime=[114.883],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 12
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 639eeee5-f9ad-86e4-896f-dd2d495d99cb, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242218734324492501469_0003_m_000004_20%2F
amz-sdk-invocation-id:639eeee5-f9ad-86e4-896f-dd2d495d99cb
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
1f8edbd706ab5962e678d2eb2f4e79312c632657ec9436ee21b35a1aae75cde1"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 3; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 1][route: {}->http://localhost:30600][total available: 2; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-1: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-1: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242218734324492501469_0003_m_000004_20%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242218734324492501469_0003_m_000004_20%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> amz-sdk-invocation-id: 639eeee5-f9ad-86e4-896f-dd2d495d99cb
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0a1f61b08288449da5f2e8e1ea6c0f26667cd72ed30f6a03ccb51f98ebda7dd7
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242218734324492501469_0003_m_000004_20%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "amz-sdk-invocation-id: 639eeee5-f9ad-86e4-896f-dd2d495d99cb[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=0a1f61b08288449da5f2e8e1ea6c0f26667cd72ed30f6a03ccb51f98ebda7dd7[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: d31230fa-be6e-4fb1-905a-07e578b44a19
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: d31230fa-be6e-4fb1-905a-07e578b44a19
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: d31230fa-be6e-4fb1-905a-07e578b44a19
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 20
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 2
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[d31230fa-be6e-4fb1-905a-07e578b44a19], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=13, ResponseProcessingTime=[15.053], ClientExecuteTime=[88.255], HttpClientSendRequestTime=[0.824], HttpRequestTime=[70.694], ApiCallLatency=[87.834], RequestSigningTime=[0.695], CredentialsRequestTime=[0.016, 0.014], HttpClientReceiveResponseTime=[68.46],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.089s
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243621455509773921197_0003_m_000001_17
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 3
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_20220812153824469238366268405742_0003_m_000010_26
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "X-Amz-Id-2: eb98ab32-1e22-4053-bc7d-f5755e2f13d7[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "X-Amz-Request-Id: eb98ab32-1e22-4053-bc7d-f5755e2f13d7[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-2 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << X-Amz-Id-2: eb98ab32-1e22-4053-bc7d-f5755e2f13d7
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << X-Amz-Request-Id: eb98ab32-1e22-4053-bc7d-f5755e2f13d7
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-2 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "X-Amz-Id-2: b45f54c1-cae3-4222-8b86-d5c20a628441[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "X-Amz-Request-Id: b45f54c1-cae3-4222-8b86-d5c20a628441[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-4 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << X-Amz-Id-2: b45f54c1-cae3-4222-8b86-d5c20a628441
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << X-Amz-Request-Id: b45f54c1-cae3-4222-8b86-d5c20a628441
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-4 << Content-Length: 359
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 2][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-2: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 2][route: {}->http://localhost:30600][total available: 3; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: eb98ab32-1e22-4053-bc7d-f5755e2f13d7
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538243621455509773921197_0003_m_000001_17
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: eb98ab32-1e22-4053-bc7d-f5755e2f13d7
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: eb98ab32-1e22-4053-bc7d-f5755e2f13d7
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 21
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[eb98ab32-1e22-4053-bc7d-f5755e2f13d7], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=15, ResponseProcessingTime=[5.769], ClientExecuteTime=[70.172], HttpClientSendRequestTime=[0.717], HttpRequestTime=[61.851], ApiCallLatency=[69.787], RequestSigningTime=[0.621], CredentialsRequestTime=[0.009, 0.003], HttpClientReceiveResponseTime=[59.701],
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 4][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-4: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 4][route: {}->http://localhost:30600][total available: 4; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.071s
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538245411134883447354195_0003_m_000006_22
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 4
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538245411134883447354195_0003_m_000006_22
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "X-Amz-Id-2: 0c0b84a0-b7a5-4b5f-b938-841172589d60[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "X-Amz-Request-Id: 0c0b84a0-b7a5-4b5f-b938-841172589d60[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-13 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << X-Amz-Id-2: 0c0b84a0-b7a5-4b5f-b938-841172589d60
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << X-Amz-Request-Id: 0c0b84a0-b7a5-4b5f-b938-841172589d60
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-13 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 13][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-13: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 13][route: {}->http://localhost:30600][total available: 5; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: b45f54c1-cae3-4222-8b86-d5c20a628441
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: b45f54c1-cae3-4222-8b86-d5c20a628441
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: b45f54c1-cae3-4222-8b86-d5c20a628441
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 22
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[b45f54c1-cae3-4222-8b86-d5c20a628441], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=2, RequestCount=1, HttpClientPoolLeasedCount=14, ResponseProcessingTime=[9.649], ClientExecuteTime=[67.987], HttpClientSendRequestTime=[0.759], HttpRequestTime=[56.174], ApiCallLatency=[67.678], RequestSigningTime=[0.591], CredentialsRequestTime=[0.01, 0.006], HttpClientReceiveResponseTime=[52.092],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.068s
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243594149530430168495_0003_m_000008_24
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 5
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538243594149530430168495_0003_m_000008_24
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: 0c0b84a0-b7a5-4b5f-b938-841172589d60
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: 0c0b84a0-b7a5-4b5f-b938-841172589d60
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: 0c0b84a0-b7a5-4b5f-b938-841172589d60
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 23
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[0c0b84a0-b7a5-4b5f-b938-841172589d60], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=15, ResponseProcessingTime=[3.076], ClientExecuteTime=[67.62], HttpClientSendRequestTime=[0.795], HttpRequestTime=[61.44], ApiCallLatency=[67.359], RequestSigningTime=[1.38], CredentialsRequestTime=[0.008, 0.008], HttpClientReceiveResponseTime=[52.858],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.068s
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538243576078083490261803_0003_m_000003_19
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 6
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538243576078083490261803_0003_m_000003_19
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "X-Amz-Id-2: ef2e9382-566f-432a-9e31-527c1cfea14a[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "X-Amz-Request-Id: ef2e9382-566f-432a-9e31-527c1cfea14a[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << X-Amz-Id-2: ef2e9382-566f-432a-9e31-527c1cfea14a
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << X-Amz-Request-Id: ef2e9382-566f-432a-9e31-527c1cfea14a
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << Content-Length: 321
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 11][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-11: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 11][route: {}->http://localhost:30600][total available: 6; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: ef2e9382-566f-432a-9e31-527c1cfea14a; S3 Extended Request ID: ef2e9382-566f-432a-9e31-527c1cfea14a; Proxy: null), S3 Extended Request ID: ef2e9382-566f-432a-9e31-527c1cfea14a
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 24
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: ef2e9382-566f-432a-9e31-527c1cfea14a; S3 Extended Request ID: ef2e9382-566f-432a-9e31-527c1cfea14a; Proxy: null), S3 Extended Request ID: ef2e9382-566f-432a-9e31-527c1cfea14a], RequestType=[GetObjectMetadataRequest], AWSRequestID=[ef2e9382-566f-432a-9e31-527c1cfea14a], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=3, ClientExecuteTime=[230.01], HttpClientSendRequestTime=[0.858], HttpRequestTime=[171.385], ApiCallLatency=[229.665], RequestSigningTime=[0.461], CredentialsRequestTime=[0.015, 0.006], HttpClientReceiveResponseTime=[149.568],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 13
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 8e65319c-fd9b-a355-a0ae-42225e0001ae, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 INFO Executor: Finished task 6.0 in stage 3.0 (TID 22). 2828 bytes result sent to driver
22/08/12 15:38:25 INFO Executor: Finished task 3.0 in stage 3.0 (TID 19). 2785 bytes result sent to driver
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 15
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 14
22/08/12 15:38:25 INFO Executor: Finished task 1.0 in stage 3.0 (TID 17). 2828 bytes result sent to driver
22/08/12 15:38:25 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_3.0, runningTasks: 15
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538241773204340993498424_0003_m_000005_21%2F
amz-sdk-invocation-id:8e65319c-fd9b-a355-a0ae-42225e0001ae
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 13
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
23e7f310d6361a03b79cb3ce419281b306298febd8f05cd10c86fdf4f1f34001"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "X-Amz-Id-2: b772ba9d-2060-4f52-b3ca-7a5bc94503f4[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "X-Amz-Request-Id: b772ba9d-2060-4f52-b3ca-7a5bc94503f4[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-1 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << X-Amz-Id-2: b772ba9d-2060-4f52-b3ca-7a5bc94503f4
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << X-Amz-Request-Id: b772ba9d-2060-4f52-b3ca-7a5bc94503f4
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-1 << Content-Length: 359
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 6; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 11][route: {}->http://localhost:30600][total available: 5; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-11: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-11: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538241773204340993498424_0003_m_000005_21%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538241773204340993498424_0003_m_000005_21%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> amz-sdk-invocation-id: 8e65319c-fd9b-a355-a0ae-42225e0001ae
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=822adf1dc164ff17625127277d71928092ef3b21e870837274881e417e50b9ff
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538241773204340993498424_0003_m_000005_21%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "amz-sdk-invocation-id: 8e65319c-fd9b-a355-a0ae-42225e0001ae[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=822adf1dc164ff17625127277d71928092ef3b21e870837274881e417e50b9ff[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "[\r][\n]"
22/08/12 15:38:25 INFO Executor: Finished task 10.0 in stage 3.0 (TID 26). 2785 bytes result sent to driver
22/08/12 15:38:25 INFO Executor: Finished task 8.0 in stage 3.0 (TID 24). 2828 bytes result sent to driver
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 1][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-1: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 1][route: {}->http://localhost:30600][total available: 6; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "X-Amz-Id-2: 1069c3bb-7b05-4440-b356-c80f4e9ccd80[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "X-Amz-Request-Id: 1069c3bb-7b05-4440-b356-c80f4e9ccd80[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << X-Amz-Id-2: 1069c3bb-7b05-4440-b356-c80f4e9ccd80
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << X-Amz-Request-Id: 1069c3bb-7b05-4440-b356-c80f4e9ccd80
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << Content-Length: 321
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: b772ba9d-2060-4f52-b3ca-7a5bc94503f4
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: b772ba9d-2060-4f52-b3ca-7a5bc94503f4
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 10][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-10: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 10][route: {}->http://localhost:30600][total available: 7; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 12
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: b772ba9d-2060-4f52-b3ca-7a5bc94503f4
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 25
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 1069c3bb-7b05-4440-b356-c80f4e9ccd80; S3 Extended Request ID: 1069c3bb-7b05-4440-b356-c80f4e9ccd80; Proxy: null), S3 Extended Request ID: 1069c3bb-7b05-4440-b356-c80f4e9ccd80
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[b772ba9d-2060-4f52-b3ca-7a5bc94503f4], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=3, RequestCount=1, HttpClientPoolLeasedCount=13, ResponseProcessingTime=[7.236], ClientExecuteTime=[45.495], HttpClientSendRequestTime=[0.718], HttpRequestTime=[36.113], ApiCallLatency=[45.109], RequestSigningTime=[0.63], CredentialsRequestTime=[0.01, 0.005], HttpClientReceiveResponseTime=[34.177],
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 26
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.046s
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242218734324492501469_0003_m_000004_20
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 1069c3bb-7b05-4440-b356-c80f4e9ccd80; S3 Extended Request ID: 1069c3bb-7b05-4440-b356-c80f4e9ccd80; Proxy: null), S3 Extended Request ID: 1069c3bb-7b05-4440-b356-c80f4e9ccd80], RequestType=[GetObjectMetadataRequest], AWSRequestID=[1069c3bb-7b05-4440-b356-c80f4e9ccd80], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=3, ClientExecuteTime=[261.969], HttpClientSendRequestTime=[3.725], HttpRequestTime=[186.14], ApiCallLatency=[261.682], RequestSigningTime=[17.507], CredentialsRequestTime=[0.004, 0.005], HttpClientReceiveResponseTime=[163.508],
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 7
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538242218734324492501469_0003_m_000004_20
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 14
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 0029031c-cbbe-b929-85d0-605a0d6002b7, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 11
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246186025400026495128_0003_m_000002_18%2F
amz-sdk-invocation-id:0029031c-cbbe-b929-85d0-605a0d6002b7
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
03fbaf4da6cbfb31dd09cb3181c32263c0938cf7832a4bf852e0ed06a9e7b609"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 7; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 10][route: {}->http://localhost:30600][total available: 6; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-10: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-10: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246186025400026495128_0003_m_000002_18%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246186025400026495128_0003_m_000002_18%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> amz-sdk-invocation-id: 0029031c-cbbe-b929-85d0-605a0d6002b7
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=ed978e828ee2df79839a624ec71b80ff5a82399f6be3e1370f446bfbcef4599a
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246186025400026495128_0003_m_000002_18%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "amz-sdk-invocation-id: 0029031c-cbbe-b929-85d0-605a0d6002b7[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "X-Amz-Id-2: 1353a6a0-bbed-450e-b8f8-ca493b329331[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "X-Amz-Request-Id: 1353a6a0-bbed-450e-b8f8-ca493b329331[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "Content-Length: 389[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=ed978e828ee2df79839a624ec71b80ff5a82399f6be3e1370f446bfbcef4599a[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << X-Amz-Id-2: 1353a6a0-bbed-450e-b8f8-ca493b329331
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << X-Amz-Request-Id: 1353a6a0-bbed-450e-b8f8-ca493b329331
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << Content-Length: 389
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 15][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-15: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 15][route: {}->http://localhost:30600][total available: 7; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 1353a6a0-bbed-450e-b8f8-ca493b329331; S3 Extended Request ID: 1353a6a0-bbed-450e-b8f8-ca493b329331; Proxy: null), S3 Extended Request ID: 1353a6a0-bbed-450e-b8f8-ca493b329331
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 27
22/08/12 15:38:25 INFO TaskSetManager: Finished task 3.0 in stage 3.0 (TID 19) in 705 ms on 10.1.255.235 (executor driver) (1/16)
22/08/12 15:38:25 INFO Executor: Finished task 4.0 in stage 3.0 (TID 20). 2785 bytes result sent to driver
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 10
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@259c0eff)
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 1353a6a0-bbed-450e-b8f8-ca493b329331; S3 Extended Request ID: 1353a6a0-bbed-450e-b8f8-ca493b329331; Proxy: null), S3 Extended Request ID: 1353a6a0-bbed-450e-b8f8-ca493b329331], RequestType=[GetObjectMetadataRequest], AWSRequestID=[1353a6a0-bbed-450e-b8f8-ca493b329331], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=14, ClientExecuteTime=[123.684], HttpClientSendRequestTime=[0.926], HttpRequestTime=[121.121], ApiCallLatency=[123.243], RequestSigningTime=[0.519], CredentialsRequestTime=[0.015, 0.007], HttpClientReceiveResponseTime=[108.237],
22/08/12 15:38:25 INFO TaskSetManager: Finished task 6.0 in stage 3.0 (TID 22) in 710 ms on 10.1.255.235 (executor driver) (2/16)
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 15
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 15f48620-0dcf-579c-3d07-4b3325a83d22, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 17) in 713 ms on 10.1.255.235 (executor driver) (3/16)
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@15f187ff)
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2Fpart-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F
amz-sdk-invocation-id:15f48620-0dcf-579c-3d07-4b3325a83d22
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
e0ba912b4743cf99d3e2153448a2482e36c04197b402199d40862e89541d8541"
22/08/12 15:38:25 INFO TaskSetManager: Finished task 8.0 in stage 3.0 (TID 24) in 712 ms on 10.1.255.235 (executor driver) (4/16)
22/08/12 15:38:25 INFO TaskSetManager: Finished task 4.0 in stage 3.0 (TID 20) in 714 ms on 10.1.255.235 (executor driver) (5/16)
22/08/12 15:38:25 INFO TaskSetManager: Finished task 10.0 in stage 3.0 (TID 26) in 712 ms on 10.1.255.235 (executor driver) (6/16)
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 7; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 15][route: {}->http://localhost:30600][total available: 6; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-15: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-15: set socket timeout to 200000
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@e36c9f4)
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2Fpart-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2Fpart-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> amz-sdk-invocation-id: 15f48620-0dcf-579c-3d07-4b3325a83d22
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=19095cdb269f021a9b5fd296cf246f3456d26ad215250878fbad524c5fe35187
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2Fpart-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "amz-sdk-invocation-id: 15f48620-0dcf-579c-3d07-4b3325a83d22[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=19095cdb269f021a9b5fd296cf246f3456d26ad215250878fbad524c5fe35187[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@522c98d1)
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "X-Amz-Id-2: 9a0c7334-0b36-4676-9f43-149a9929c748[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "X-Amz-Request-Id: 9a0c7334-0b36-4676-9f43-149a9929c748[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << X-Amz-Id-2: 9a0c7334-0b36-4676-9f43-149a9929c748
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << X-Amz-Request-Id: 9a0c7334-0b36-4676-9f43-149a9929c748
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@17029f9d)
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 << "X-Amz-Id-2: e7c383ed-9122-4f47-8352-c79e9bc31a7d[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 << "X-Amz-Request-Id: e7c383ed-9122-4f47-8352-c79e9bc31a7d[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 << "Content-Length: 389[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-14 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 << X-Amz-Id-2: e7c383ed-9122-4f47-8352-c79e9bc31a7d
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 << X-Amz-Request-Id: e7c383ed-9122-4f47-8352-c79e9bc31a7d
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-14 << Content-Length: 389
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 14][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-14: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 14][route: {}->http://localhost:30600][total available: 7; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "X-Amz-Id-2: 6ac1b676-3be4-4295-afd0-8beff41eb66d[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "X-Amz-Request-Id: 6ac1b676-3be4-4295-afd0-8beff41eb66d[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << X-Amz-Id-2: 6ac1b676-3be4-4295-afd0-8beff41eb66d
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << X-Amz-Request-Id: 6ac1b676-3be4-4295-afd0-8beff41eb66d
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 << "X-Amz-Id-2: ad8e9658-bea7-46a9-bb89-fc762c56e001[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 << "X-Amz-Request-Id: ad8e9658-bea7-46a9-bb89-fc762c56e001[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-9 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 << X-Amz-Id-2: ad8e9658-bea7-46a9-bb89-fc762c56e001
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 << X-Amz-Request-Id: ad8e9658-bea7-46a9-bb89-fc762c56e001
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-9 << Content-Length: 321
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 9][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-9: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 9][route: {}->http://localhost:30600][total available: 8; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: ad8e9658-bea7-46a9-bb89-fc762c56e001; S3 Extended Request ID: ad8e9658-bea7-46a9-bb89-fc762c56e001; Proxy: null), S3 Extended Request ID: ad8e9658-bea7-46a9-bb89-fc762c56e001
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 11][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-11: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 11][route: {}->http://localhost:30600][total available: 9; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 28
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: ad8e9658-bea7-46a9-bb89-fc762c56e001; S3 Extended Request ID: ad8e9658-bea7-46a9-bb89-fc762c56e001; Proxy: null), S3 Extended Request ID: ad8e9658-bea7-46a9-bb89-fc762c56e001], RequestType=[GetObjectMetadataRequest], AWSRequestID=[ad8e9658-bea7-46a9-bb89-fc762c56e001], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=3, ClientExecuteTime=[282.642], HttpClientSendRequestTime=[5.484], HttpRequestTime=[223.624], ApiCallLatency=[271.253], RequestSigningTime=[3.799], CredentialsRequestTime=[0.017, 0.067], HttpClientReceiveResponseTime=[204.568],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 16
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: 6ac1b676-3be4-4295-afd0-8beff41eb66d
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: 6ac1b676-3be4-4295-afd0-8beff41eb66d
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: 6ac1b676-3be4-4295-afd0-8beff41eb66d
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 29
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[6ac1b676-3be4-4295-afd0-8beff41eb66d], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=6, RequestCount=1, HttpClientPoolLeasedCount=10, ResponseProcessingTime=[5.088], ClientExecuteTime=[52.493], HttpClientSendRequestTime=[0.784], HttpRequestTime=[41.713], ApiCallLatency=[52.025], RequestSigningTime=[1.952], CredentialsRequestTime=[0.014, 0.029], HttpClientReceiveResponseTime=[39.997],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.053s
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241773204340993498424_0003_m_000005_21
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 7dd3daf9-0737-0464-d594-f72b16b12564, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 8
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538241773204340993498424_0003_m_000005_21
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242991889787839631523_0003_m_000011_27%2F
amz-sdk-invocation-id:7dd3daf9-0737-0464-d594-f72b16b12564
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
3b832f9a7b4ebadcd080a19442ed98c53ddc03d231fd96f9ffc75156ba884835"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 9; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 11][route: {}->http://localhost:30600][total available: 8; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-11: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-11: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242991889787839631523_0003_m_000011_27%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242991889787839631523_0003_m_000011_27%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> amz-sdk-invocation-id: 7dd3daf9-0737-0464-d594-f72b16b12564
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=5c996fb8a2418c71b049771af41ebb616c3b495615bd02654732abd201389c57
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 INFO Executor: Finished task 5.0 in stage 3.0 (TID 21). 2785 bytes result sent to driver
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: e7c383ed-9122-4f47-8352-c79e9bc31a7d; S3 Extended Request ID: e7c383ed-9122-4f47-8352-c79e9bc31a7d; Proxy: null), S3 Extended Request ID: e7c383ed-9122-4f47-8352-c79e9bc31a7d
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 9
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242991889787839631523_0003_m_000011_27%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "amz-sdk-invocation-id: 7dd3daf9-0737-0464-d594-f72b16b12564[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 30
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=5c996fb8a2418c71b049771af41ebb616c3b495615bd02654732abd201389c57[\r][\n]"
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: e7c383ed-9122-4f47-8352-c79e9bc31a7d; S3 Extended Request ID: e7c383ed-9122-4f47-8352-c79e9bc31a7d; Proxy: null), S3 Extended Request ID: e7c383ed-9122-4f47-8352-c79e9bc31a7d], RequestType=[GetObjectMetadataRequest], AWSRequestID=[e7c383ed-9122-4f47-8352-c79e9bc31a7d], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=14, ClientExecuteTime=[159.609], HttpClientSendRequestTime=[1.106], HttpRequestTime=[144.934], ApiCallLatency=[159.157], RequestSigningTime=[0.541], CredentialsRequestTime=[0.015, 0.006], HttpClientReceiveResponseTime=[141.459],
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@ed52cfd)
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 17
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 31a2bab4-552d-a205-cdb0-43a6538f44a4, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 INFO TaskSetManager: Finished task 5.0 in stage 3.0 (TID 21) in 739 ms on 10.1.255.235 (executor driver) (7/16)
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 7][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 7][route: {}->http://localhost:30600][total available: 9; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2Fpart-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F
amz-sdk-invocation-id:31a2bab4-552d-a205-cdb0-43a6538f44a4
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
22e52b53f3511a40070d036606f7ca2cba187d61f5c256c26ce9165e13c2c2df"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@36143dba)
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: 9a0c7334-0b36-4676-9f43-149a9929c748
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: 9a0c7334-0b36-4676-9f43-149a9929c748
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: 9a0c7334-0b36-4676-9f43-149a9929c748
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 31
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 9; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 7][route: {}->http://localhost:30600][total available: 8; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[9a0c7334-0b36-4676-9f43-149a9929c748], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=15, ResponseProcessingTime=[3.651], ClientExecuteTime=[156.592], HttpClientSendRequestTime=[0.849], HttpRequestTime=[130.589], ApiCallLatency=[155.731], RequestSigningTime=[0.843], CredentialsRequestTime=[0.033, 0.012], HttpClientReceiveResponseTime=[128.275],
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2Fpart-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.158s
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2Fpart-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> amz-sdk-invocation-id: 31a2bab4-552d-a205-cdb0-43a6538f44a4
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246737245493563198477_0003_m_000012_28
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=7cad3242d53361e80fc2e8eca6e0fae8d419456859e30bec5d08d3d68f44a314
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 9
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Content-Length: 0
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538246737245493563198477_0003_m_000012_28
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2Fpart-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "amz-sdk-invocation-id: 31a2bab4-552d-a205-cdb0-43a6538f44a4[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=7cad3242d53361e80fc2e8eca6e0fae8d419456859e30bec5d08d3d68f44a314[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 >> "[\r][\n]"
22/08/12 15:38:25 INFO Executor: Finished task 12.0 in stage 3.0 (TID 28). 2785 bytes result sent to driver
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 8
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "X-Amz-Id-2: d5d08d0a-5bee-4b23-9f6e-5a2af62047b9[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "X-Amz-Request-Id: d5d08d0a-5bee-4b23-9f6e-5a2af62047b9[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "[\r][\n]"
22/08/12 15:38:25 INFO TaskSetManager: Finished task 12.0 in stage 3.0 (TID 28) in 743 ms on 10.1.255.235 (executor driver) (8/16)
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@48d1b9)
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: ae64a326-6b25-4d4d-80eb-8482ecd3f831[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: ae64a326-6b25-4d4d-80eb-8482ecd3f831[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: ae64a326-6b25-4d4d-80eb-8482ecd3f831
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: ae64a326-6b25-4d4d-80eb-8482ecd3f831
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-0 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 9; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << X-Amz-Id-2: d5d08d0a-5bee-4b23-9f6e-5a2af62047b9
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << X-Amz-Request-Id: d5d08d0a-5bee-4b23-9f6e-5a2af62047b9
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << Content-Length: 321
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: ae64a326-6b25-4d4d-80eb-8482ecd3f831
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: ae64a326-6b25-4d4d-80eb-8482ecd3f831
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: ae64a326-6b25-4d4d-80eb-8482ecd3f831
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 32
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[ae64a326-6b25-4d4d-80eb-8482ecd3f831], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=15, ResponseProcessingTime=[3.852], ClientExecuteTime=[132.849], HttpClientSendRequestTime=[0.673], HttpRequestTime=[121.847], ApiCallLatency=[132.331], RequestSigningTime=[4.715], CredentialsRequestTime=[0.018, 0.018], HttpClientReceiveResponseTime=[120.106],
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 8][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-8: set socket timeout to 0
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.134s
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 8][route: {}->http://localhost:30600][total available: 10; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248057729003051628769_0003_m_000009_25
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 10
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538248057729003051628769_0003_m_000009_25
22/08/12 15:38:25 INFO Executor: Finished task 9.0 in stage 3.0 (TID 25). 2785 bytes result sent to driver
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 7
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: d5d08d0a-5bee-4b23-9f6e-5a2af62047b9; S3 Extended Request ID: d5d08d0a-5bee-4b23-9f6e-5a2af62047b9; Proxy: null), S3 Extended Request ID: d5d08d0a-5bee-4b23-9f6e-5a2af62047b9
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 33
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: d5d08d0a-5bee-4b23-9f6e-5a2af62047b9; S3 Extended Request ID: d5d08d0a-5bee-4b23-9f6e-5a2af62047b9; Proxy: null), S3 Extended Request ID: d5d08d0a-5bee-4b23-9f6e-5a2af62047b9], RequestType=[GetObjectMetadataRequest], AWSRequestID=[d5d08d0a-5bee-4b23-9f6e-5a2af62047b9], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=3, ClientExecuteTime=[309.85], HttpClientSendRequestTime=[1.056], HttpRequestTime=[248.064], ApiCallLatency=[309.668], RequestSigningTime=[0.659], CredentialsRequestTime=[0.004, 0.005], HttpClientReceiveResponseTime=[225.98],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 INFO TaskSetManager: Finished task 9.0 in stage 3.0 (TID 25) in 757 ms on 10.1.255.235 (executor driver) (9/16)
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 18
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: d176207b-3c16-6763-1df9-5cee96e08c69, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@3b5585a)
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242079469126800442703_0003_m_000013_29%2F
amz-sdk-invocation-id:d176207b-3c16-6763-1df9-5cee96e08c69
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
1f7f14a4fc3baadc0a7ab233d7b1405caeb7c31ba164550cc19f5380b22b917e"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 10; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 8][route: {}->http://localhost:30600][total available: 9; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-8: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-8: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242079469126800442703_0003_m_000013_29%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242079469126800442703_0003_m_000013_29%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> amz-sdk-invocation-id: d176207b-3c16-6763-1df9-5cee96e08c69
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=486c8cd3b5c98ffbf1ad936ccb7339dc84e3d3d7ed013d456d6ab4720dd2500b
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538242079469126800442703_0003_m_000013_29%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "amz-sdk-invocation-id: d176207b-3c16-6763-1df9-5cee96e08c69[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=486c8cd3b5c98ffbf1ad936ccb7339dc84e3d3d7ed013d456d6ab4720dd2500b[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "X-Amz-Id-2: 02ccc5d7-6314-4c47-8105-e17ddb051841[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "X-Amz-Request-Id: 02ccc5d7-6314-4c47-8105-e17ddb051841[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "[\r][\n]"
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << X-Amz-Id-2: 02ccc5d7-6314-4c47-8105-e17ddb051841
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << X-Amz-Request-Id: 02ccc5d7-6314-4c47-8105-e17ddb051841
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << Content-Length: 321
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 6][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-6: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 6][route: {}->http://localhost:30600][total available: 10; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 02ccc5d7-6314-4c47-8105-e17ddb051841; S3 Extended Request ID: 02ccc5d7-6314-4c47-8105-e17ddb051841; Proxy: null), S3 Extended Request ID: 02ccc5d7-6314-4c47-8105-e17ddb051841
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 34
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 02ccc5d7-6314-4c47-8105-e17ddb051841; S3 Extended Request ID: 02ccc5d7-6314-4c47-8105-e17ddb051841; Proxy: null), S3 Extended Request ID: 02ccc5d7-6314-4c47-8105-e17ddb051841], RequestType=[GetObjectMetadataRequest], AWSRequestID=[02ccc5d7-6314-4c47-8105-e17ddb051841], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=3, ClientExecuteTime=[338.034], HttpClientSendRequestTime=[13.581], HttpRequestTime=[258.184], ApiCallLatency=[337.79], RequestSigningTime=[18.233], CredentialsRequestTime=[0.01, 0.006], HttpClientReceiveResponseTime=[232.488],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 19
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 73aca539-a3fc-3c98-e143-ba7c2ceeb193, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538241431370443743393950_0003_m_000014_30%2F
amz-sdk-invocation-id:73aca539-a3fc-3c98-e143-ba7c2ceeb193
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
232a42a37a72a10cc1eed2bded5964adc9117b5f4121b394d559f5bba82d6351"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 10; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 6][route: {}->http://localhost:30600][total available: 9; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-6: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-6: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538241431370443743393950_0003_m_000014_30%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538241431370443743393950_0003_m_000014_30%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> amz-sdk-invocation-id: 73aca539-a3fc-3c98-e143-ba7c2ceeb193
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=5e69e37701fffee1b1b9eedefe741020b3010a2282c33ed568acb8950025f46a
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 >> Connection: Keep-Alive
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538241431370443743393950_0003_m_000014_30%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "amz-sdk-invocation-id: 73aca539-a3fc-3c98-e143-ba7c2ceeb193[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=5e69e37701fffee1b1b9eedefe741020b3010a2282c33ed568acb8950025f46a[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 >> "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "X-Amz-Id-2: fde100c0-6b66-4673-871a-4faef96ce254[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "X-Amz-Request-Id: fde100c0-6b66-4673-871a-4faef96ce254[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-11 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << X-Amz-Id-2: fde100c0-6b66-4673-871a-4faef96ce254
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << X-Amz-Request-Id: fde100c0-6b66-4673-871a-4faef96ce254
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-11 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "X-Amz-Id-2: 49df14c5-e914-4a0b-9e15-f0a2437a3734[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "X-Amz-Request-Id: 49df14c5-e914-4a0b-9e15-f0a2437a3734[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "Content-Length: 427[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-15 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << X-Amz-Id-2: 49df14c5-e914-4a0b-9e15-f0a2437a3734
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << X-Amz-Request-Id: 49df14c5-e914-4a0b-9e15-f0a2437a3734
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-15 << Content-Length: 427
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 11][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-11: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 11][route: {}->http://localhost:30600][total available: 10; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 15][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-15: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 15][route: {}->http://localhost:30600][total available: 11; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: 49df14c5-e914-4a0b-9e15-f0a2437a3734
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: 49df14c5-e914-4a0b-9e15-f0a2437a3734
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: 49df14c5-e914-4a0b-9e15-f0a2437a3734
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 35
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[49df14c5-e914-4a0b-9e15-f0a2437a3734], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=7, RequestCount=1, HttpClientPoolLeasedCount=9, ResponseProcessingTime=[2.485], ClientExecuteTime=[71.477], HttpClientSendRequestTime=[1.676], HttpRequestTime=[65.542], ApiCallLatency=[70.994], RequestSigningTime=[0.718], CredentialsRequestTime=[0.036, 0.008], HttpClientReceiveResponseTime=[62.278],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.072s
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter files_created by 1 with final value 1
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: fde100c0-6b66-4673-871a-4faef96ce254
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: fde100c0-6b66-4673-871a-4faef96ce254
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: fde100c0-6b66-4673-871a-4faef96ce254
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 36
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[fde100c0-6b66-4673-871a-4faef96ce254], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=9, RequestCount=1, HttpClientPoolLeasedCount=7, ResponseProcessingTime=[6.146], ClientExecuteTime=[50.104], HttpClientSendRequestTime=[4.018], HttpRequestTime=[42.298], ApiCallLatency=[49.837], RequestSigningTime=[0.405], CredentialsRequestTime=[0.01, 0.004], HttpClientReceiveResponseTime=[36.913],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.050s
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242991889787839631523_0003_m_000011_27
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 11
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538242991889787839631523_0003_m_000011_27
22/08/12 15:38:25 INFO Executor: Finished task 11.0 in stage 3.0 (TID 27). 2785 bytes result sent to driver
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 6
22/08/12 15:38:25 INFO TaskSetManager: Finished task 11.0 in stage 3.0 (TID 27) in 785 ms on 10.1.255.235 (executor driver) (10/16)
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@24a02e15)
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "X-Amz-Id-2: 422725e6-d166-4d68-8958-8f3d34d211e4[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "X-Amz-Request-Id: 422725e6-d166-4d68-8958-8f3d34d211e4[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-8 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << X-Amz-Id-2: 422725e6-d166-4d68-8958-8f3d34d211e4
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << X-Amz-Request-Id: 422725e6-d166-4d68-8958-8f3d34d211e4
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-8 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "X-Amz-Id-2: fb9840a7-1eee-450b-8576-23fbb9566e03[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "X-Amz-Request-Id: fb9840a7-1eee-450b-8576-23fbb9566e03[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-6 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << X-Amz-Id-2: fb9840a7-1eee-450b-8576-23fbb9566e03
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << X-Amz-Request-Id: fb9840a7-1eee-450b-8576-23fbb9566e03
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-6 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 6][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 8][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-6: set socket timeout to 0
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-8: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 6][route: {}->http://localhost:30600][total available: 13; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 8][route: {}->http://localhost:30600][total available: 13; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: 422725e6-d166-4d68-8958-8f3d34d211e4
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: fb9840a7-1eee-450b-8576-23fbb9566e03
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: 422725e6-d166-4d68-8958-8f3d34d211e4
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: fb9840a7-1eee-450b-8576-23fbb9566e03
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: 422725e6-d166-4d68-8958-8f3d34d211e4
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: fb9840a7-1eee-450b-8576-23fbb9566e03
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: 0b7e61bd-be0a-479b-b8d0-3ad20abcac21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: 0b7e61bd-be0a-479b-b8d0-3ad20abcac21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 37
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 38
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << HTTP/1.1 404 Not Found
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: 0b7e61bd-be0a-479b-b8d0-3ad20abcac21
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: 0b7e61bd-be0a-479b-b8d0-3ad20abcac21
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << Content-Length: 321
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[422725e6-d166-4d68-8958-8f3d34d211e4], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=10, RequestCount=1, HttpClientPoolLeasedCount=6, ResponseProcessingTime=[3.249], ClientExecuteTime=[39.337], HttpClientSendRequestTime=[0.481], HttpRequestTime=[29.947], ApiCallLatency=[39.006], RequestSigningTime=[0.462], CredentialsRequestTime=[0.01, 0.004], HttpClientReceiveResponseTime=[28.349],
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[fb9840a7-1eee-450b-8576-23fbb9566e03], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=10, RequestCount=1, HttpClientPoolLeasedCount=6, ResponseProcessingTime=[3.725], ClientExecuteTime=[31.305], HttpClientSendRequestTime=[0.554], HttpRequestTime=[25.183], ApiCallLatency=[30.938], RequestSigningTime=[0.548], CredentialsRequestTime=[0.013, 0.014], HttpClientReceiveResponseTime=[23.54],
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.040s
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.032s
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538242079469126800442703_0003_m_000013_29
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538241431370443743393950_0003_m_000014_30
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 13
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 12
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538242079469126800442703_0003_m_000013_29
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538241431370443743393950_0003_m_000014_30
22/08/12 15:38:25 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 0b7e61bd-be0a-479b-b8d0-3ad20abcac21; S3 Extended Request ID: 0b7e61bd-be0a-479b-b8d0-3ad20abcac21; Proxy: null), S3 Extended Request ID: 0b7e61bd-be0a-479b-b8d0-3ad20abcac21
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 39
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 0b7e61bd-be0a-479b-b8d0-3ad20abcac21; S3 Extended Request ID: 0b7e61bd-be0a-479b-b8d0-3ad20abcac21; Proxy: null), S3 Extended Request ID: 0b7e61bd-be0a-479b-b8d0-3ad20abcac21], RequestType=[GetObjectMetadataRequest], AWSRequestID=[0b7e61bd-be0a-479b-b8d0-3ad20abcac21], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=3, ClientExecuteTime=[372.315], HttpClientSendRequestTime=[0.807], HttpRequestTime=[285.944], ApiCallLatency=[371.977], RequestSigningTime=[4.911], CredentialsRequestTime=[0.01, 0.014], HttpClientReceiveResponseTime=[280.242],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:25 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 20
22/08/12 15:38:25 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 84367185-83b6-9107-be0b-3d74ada7c28f, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:25 INFO Executor: Finished task 14.0 in stage 3.0 (TID 30). 2785 bytes result sent to driver
22/08/12 15:38:25 INFO Executor: Finished task 13.0 in stage 3.0 (TID 29). 2785 bytes result sent to driver
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 5
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 4
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246252528645038300397_0003_m_000007_23%2F
amz-sdk-invocation-id:84367185-83b6-9107-be0b-3d74ada7c28f
amz-sdk-request:ttl=20220812T144145Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143825Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:25 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143825Z
20220812/us-east-1/s3/aws4_request
c005d0cd71fe004e53df685b8881471a9c03fa505048e00889a84895775acdeb"
22/08/12 15:38:25 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:25 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 13; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:25 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246252528645038300397_0003_m_000007_23%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246252528645038300397_0003_m_000007_23%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: 84367185-83b6-9107-be0b-3d74ada7c28f
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=ea83ff30a0242abcbcf1eac68b7ef4dd4ca845470a07de64bcca161aeef6ccac
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143825Z
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> Content-Length: 0
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:25 INFO TaskSetManager: Finished task 14.0 in stage 3.0 (TID 30) in 801 ms on 10.1.255.235 (executor driver) (11/16)
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538246252528645038300397_0003_m_000007_23%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: 84367185-83b6-9107-be0b-3d74ada7c28f[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144145Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=ea83ff30a0242abcbcf1eac68b7ef4dd4ca845470a07de64bcca161aeef6ccac[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143825Z[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:25 INFO TaskSetManager: Finished task 13.0 in stage 3.0 (TID 29) in 801 ms on 10.1.255.235 (executor driver) (12/16)
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "X-Amz-Id-2: fab1f29d-10b3-4999-90ee-9b87513670b2[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "X-Amz-Request-Id: fab1f29d-10b3-4999-90ee-9b87513670b2[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-10 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << X-Amz-Id-2: fab1f29d-10b3-4999-90ee-9b87513670b2
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << X-Amz-Request-Id: fab1f29d-10b3-4999-90ee-9b87513670b2
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-10 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 10][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-10: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 10][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@334e2ed6)
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@75cde41f)
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: fab1f29d-10b3-4999-90ee-9b87513670b2
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: fab1f29d-10b3-4999-90ee-9b87513670b2
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: fab1f29d-10b3-4999-90ee-9b87513670b2
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 40
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[fab1f29d-10b3-4999-90ee-9b87513670b2], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=7, RequestCount=1, HttpClientPoolLeasedCount=9, ResponseProcessingTime=[2.984], ClientExecuteTime=[113.988], HttpClientSendRequestTime=[1.01], HttpRequestTime=[108.64], ApiCallLatency=[113.536], RequestSigningTime=[0.629], CredentialsRequestTime=[0.018, 0.007], HttpClientReceiveResponseTime=[106.618],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.114s
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246186025400026495128_0003_m_000002_18
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 14
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538246186025400026495128_0003_m_000002_18
22/08/12 15:38:25 INFO Executor: Finished task 2.0 in stage 3.0 (TID 18). 2785 bytes result sent to driver
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 3
22/08/12 15:38:25 INFO TaskSetManager: Finished task 2.0 in stage 3.0 (TID 18) in 817 ms on 10.1.255.235 (executor driver) (13/16)
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@5c130574)
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "X-Amz-Id-2: 247f3a01-16cc-4eac-bcc0-c54b60d7ac23[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "X-Amz-Request-Id: 247f3a01-16cc-4eac-bcc0-c54b60d7ac23[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "Content-Length: 427[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-7 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << X-Amz-Id-2: 247f3a01-16cc-4eac-bcc0-c54b60d7ac23
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << X-Amz-Request-Id: 247f3a01-16cc-4eac-bcc0-c54b60d7ac23
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-7 << Content-Length: 427
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 7][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 7][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: 247f3a01-16cc-4eac-bcc0-c54b60d7ac23
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: 247f3a01-16cc-4eac-bcc0-c54b60d7ac23
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: 247f3a01-16cc-4eac-bcc0-c54b60d7ac23
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 41
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[247f3a01-16cc-4eac-bcc0-c54b60d7ac23], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=9, RequestCount=1, HttpClientPoolLeasedCount=7, ResponseProcessingTime=[2.125], ClientExecuteTime=[83.833], HttpClientSendRequestTime=[0.867], HttpRequestTime=[79.217], ApiCallLatency=[83.442], RequestSigningTime=[0.627], CredentialsRequestTime=[0.018, 0.011], HttpClientReceiveResponseTime=[76.281],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.084s
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter files_created by 1 with final value 2
22/08/12 15:38:25 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: 9f5e2927-6976-4532-993d-b8b4a32541dd[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: 9f5e2927-6976-4532-993d-b8b4a32541dd[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:25 GMT[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:25 DEBUG wire: http-outgoing-5 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23/</Prefix></ListBucketResult>"
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << HTTP/1.1 200 OK
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: 9f5e2927-6976-4532-993d-b8b4a32541dd
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: 9f5e2927-6976-4532-993d-b8b4a32541dd
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG headers: http-outgoing-5 << Content-Length: 359
22/08/12 15:38:25 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:25 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:25 GMT
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:25 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:25 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:25 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:25 DEBUG request: Received successful response: 200, AWS Request ID: 9f5e2927-6976-4532-993d-b8b4a32541dd
22/08/12 15:38:25 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:25 DEBUG requestId: AWS Request ID: 9f5e2927-6976-4532-993d-b8b4a32541dd
22/08/12 15:38:25 DEBUG requestId: AWS Extended Request ID: 9f5e2927-6976-4532-993d-b8b4a32541dd
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 42
22/08/12 15:38:25 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[9f5e2927-6976-4532-993d-b8b4a32541dd], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=14, RequestCount=1, HttpClientPoolLeasedCount=2, ResponseProcessingTime=[2.247], ClientExecuteTime=[31.329], HttpClientSendRequestTime=[0.692], HttpRequestTime=[26.928], ApiCallLatency=[30.932], RequestSigningTime=[0.474], CredentialsRequestTime=[0.012, 0.006], HttpClientReceiveResponseTime=[23.171],
22/08/12 15:38:25 DEBUG S3AFileSystem: LIST: duration 0:00.032s
22/08/12 15:38:25 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538246252528645038300397_0003_m_000007_23
22/08/12 15:38:25 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 15
22/08/12 15:38:25 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538246252528645038300397_0003_m_000007_23
22/08/12 15:38:25 DEBUG S3ABlockOutputStream: Initialized S3ABlockOutputStream for nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet output to FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-7382960693842441093.tmp, state=Writing, dataSize=0, limit=67108864}
22/08/12 15:38:25 DEBUG S3ABlockOutputStream: Initialized S3ABlockOutputStream for nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet output to FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8794108122613463043.tmp, state=Writing, dataSize=0, limit=67108864}
22/08/12 15:38:25 INFO Executor: Finished task 7.0 in stage 3.0 (TID 23). 2785 bytes result sent to driver
22/08/12 15:38:25 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 2
22/08/12 15:38:25 INFO TaskSetManager: Finished task 7.0 in stage 3.0 (TID 23) in 835 ms on 10.1.255.235 (executor driver) (14/16)
22/08/12 15:38:25 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@79665865)
22/08/12 15:38:25 DEBUG ParquetFileWriter: 0: start
22/08/12 15:38:25 DEBUG ParquetFileWriter: 0: start
22/08/12 15:38:25 DEBUG MemoryManager: Allocated total memory pool is: 1020054720
22/08/12 15:38:25 INFO CodecPool: Got brand-new compressor [.snappy]
22/08/12 15:38:25 INFO CodecPool: Got brand-new compressor [.snappy]
22/08/12 15:38:25 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 1 initialCapacity 64
22/08/12 15:38:25 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 1 initialCapacity 64
22/08/12 15:38:25 DEBUG CapacityByteArrayOutputStream: initial slab of size 64
22/08/12 15:38:25 DEBUG CapacityByteArrayOutputStream: initial slab of size 64
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 15:38:26 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 1 initialCapacity 64
22/08/12 15:38:26 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 1 initialCapacity 64
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 64
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 64
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 15:38:26 DEBUG MemoryManager: Adjust block size from 134,217,728 to 134,217,728 for writer: org.apache.parquet.hadoop.InternalParquetRecordWriter@435cea10
22/08/12 15:38:26 DEBUG MemoryManager: Adjust block size from 134,217,728 to 134,217,728 for writer: org.apache.parquet.hadoop.InternalParquetRecordWriter@31572404
22/08/12 15:38:26 DEBUG MemoryManager: Adjust block size from 134,217,728 to 134,217,728 for writer: org.apache.parquet.hadoop.InternalParquetRecordWriter@435cea10
22/08/12 15:38:26 INFO PythonRunner: Times: total = 262, boot = 12, init = 250, finish = 0
22/08/12 15:38:26 DEBUG RecordConsumerLoggingWrapper: <!-- flush -->
22/08/12 15:38:26 DEBUG RecordConsumerLoggingWrapper: <!-- start message -->
22/08/12 15:38:26 DEBUG InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 0
22/08/12 15:38:26 DEBUG MessageColumnIO: < MESSAGE START >
22/08/12 15:38:26 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={}}: [] r:0
22/08/12 15:38:26 DEBUG ParquetFileWriter: 4: column indexes
22/08/12 15:38:26 DEBUG ParquetFileWriter: 4: offset indexes
22/08/12 15:38:26 DEBUG ParquetFileWriter: 4: bloom filters
22/08/12 15:38:26 DEBUG ParquetFileWriter: 4: end
22/08/12 15:38:26 DEBUG RecordConsumerLoggingWrapper: <a>
22/08/12 15:38:26 DEBUG MessageColumnIO: startField(a, 0)
22/08/12 15:38:26 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={}}: [a] r:0
22/08/12 15:38:26 DEBUG RecordConsumerLoggingWrapper: 1
22/08/12 15:38:26 DEBUG MessageColumnIO: addLong(1)
22/08/12 15:38:26 DEBUG MessageColumnIO: r: 0
22/08/12 15:38:26 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={}}: [a] r:0
22/08/12 15:38:26 DEBUG RecordConsumerLoggingWrapper: </a>
22/08/12 15:38:26 DEBUG MessageColumnIO: endField(a, 0)
22/08/12 15:38:26 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0}}: [] r:0
22/08/12 15:38:26 DEBUG RecordConsumerLoggingWrapper: <b>
22/08/12 15:38:26 DEBUG MessageColumnIO: startField(b, 1)
22/08/12 15:38:26 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0}}: [b] r:0
22/08/12 15:38:26 DEBUG RecordConsumerLoggingWrapper: 2.0
22/08/12 15:38:26 DEBUG MessageColumnIO: addDouble(2.0)
22/08/12 15:38:26 DEBUG MessageColumnIO: r: 0
22/08/12 15:38:26 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0}}: [b] r:0
22/08/12 15:38:26 DEBUG RecordConsumerLoggingWrapper: </b>
22/08/12 15:38:26 DEBUG MessageColumnIO: endField(b, 1)
22/08/12 15:38:26 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0, 1}}: [] r:0
22/08/12 15:38:26 DEBUG MessageColumnIO: < MESSAGE END >
22/08/12 15:38:26 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0, 1}}: [] r:0
22/08/12 15:38:26 DEBUG RecordConsumerLoggingWrapper: <!-- end message -->
22/08/12 15:38:26 INFO PythonRunner: Times: total = 216, boot = -1974, init = 2190, finish = 0
22/08/12 15:38:26 DEBUG RecordConsumerLoggingWrapper: <!-- flush -->
22/08/12 15:38:26 DEBUG InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 24
22/08/12 15:38:26 DEBUG ParquetFileWriter: 4: start block
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 64
22/08/12 15:38:26 DEBUG DictionaryValuesWriter: max dic id 0
22/08/12 15:38:26 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 0 initialCapacity 1024
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024
22/08/12 15:38:26 DEBUG DictionaryValuesWriter: rle encoded bytes 1
22/08/12 15:38:26 DEBUG BytesInput: BytesInput from array of 1 bytes
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024
22/08/12 15:38:26 DEBUG PlainValuesWriter: writing a buffer of size 8
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 0 bytes to out
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 6 bytes to out
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: {
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 4 bytes to out
22/08/12 15:38:26 DEBUG BytesUtils: write le int: 2 => 2 0 0 0
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 2 bytes to out
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: }
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 8 bytes to out
22/08/12 15:38:26 DEBUG BytesInput: converted 16 to byteArray of 16 bytes
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 23 bytes to out
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 16 bytes to out
22/08/12 15:38:26 DEBUG BytesInput: converted 39 to byteArray of 39 bytes
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 64
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 64
22/08/12 15:38:26 DEBUG DictionaryValuesWriter: max dic id 0
22/08/12 15:38:26 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 0 initialCapacity 1024
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024
22/08/12 15:38:26 DEBUG DictionaryValuesWriter: rle encoded bytes 1
22/08/12 15:38:26 DEBUG BytesInput: BytesInput from array of 1 bytes
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024
22/08/12 15:38:26 DEBUG PlainValuesWriter: writing a buffer of size 8
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 0 bytes to out
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 6 bytes to out
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: {
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 4 bytes to out
22/08/12 15:38:26 DEBUG BytesUtils: write le int: 2 => 2 0 0 0
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 2 bytes to out
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: }
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 8 bytes to out
22/08/12 15:38:26 DEBUG BytesInput: converted 16 to byteArray of 16 bytes
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 23 bytes to out
22/08/12 15:38:26 DEBUG BytesInput$SequenceBytesIn: write 16 bytes to out
22/08/12 15:38:26 DEBUG BytesInput: converted 39 to byteArray of 39 bytes
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 64
22/08/12 15:38:26 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 15:38:26 DEBUG ParquetFileWriter: 4: write data pages
22/08/12 15:38:26 DEBUG ParquetFileWriter: 4: write data pages content
22/08/12 15:38:26 DEBUG ParquetFileWriter: 43: end column
22/08/12 15:38:26 DEBUG ColumnChunkPageWriteStore: written 39B for [a] optional int64 a: 1 values, 14B raw, 16B comp, 1 pages, encodings: [PLAIN]
22/08/12 15:38:26 DEBUG ParquetFileWriter: 43: write data pages
22/08/12 15:38:26 DEBUG ParquetFileWriter: 43: write data pages content
22/08/12 15:38:26 DEBUG ParquetFileWriter: 82: end column
22/08/12 15:38:26 DEBUG ColumnChunkPageWriteStore: written 39B for [b] optional double b: 1 values, 14B raw, 16B comp, 1 pages, encodings: [PLAIN]
22/08/12 15:38:26 DEBUG ParquetFileWriter: 82: end block
22/08/12 15:38:26 DEBUG ParquetFileWriter: 82: column indexes
22/08/12 15:38:26 DEBUG ParquetFileWriter: 144: offset indexes
22/08/12 15:38:26 DEBUG ParquetFileWriter: 164: bloom filters
22/08/12 15:38:26 DEBUG ParquetFileWriter: 164: end
22/08/12 15:38:26 DEBUG ParquetFileWriter: 357: footer length = 353
22/08/12 15:38:26 DEBUG BytesUtils: write le int: 353 => 97 1 0 0
22/08/12 15:38:26 DEBUG S3ABlockOutputStream: S3ABlockOutputStream{WriteOperationHelper {bucket=master.rando2}, blockSize=67108864, activeBlock=FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8794108122613463043.tmp, state=Writing, dataSize=365, limit=67108864} Statistics=counters=((stream_write_exceptions=0) (stream_write_total_data=0) (action_executor_acquired.failures=0) (object_multipart_aborted=0) (op_abort=0) (multipart_upload_completed.failures=0) (op_hsync=0) (multipart_upload_completed=0) (op_abort.failures=0) (stream_write_exceptions_completing_upload=0) (stream_write_block_uploads=0) (action_executor_acquired=0) (stream_write_bytes=365) (stream_write_queue_duration=0) (object_multipart_aborted.failures=0) (op_hflush=0) (stream_write_total_time=0));
gauges=((stream_write_block_uploads_data_pending=0) (stream_write_block_uploads_pending=0));
minimums=((object_multipart_aborted.failures.min=-1) (multipart_upload_completed.min=-1) (op_abort.failures.min=-1) (object_multipart_aborted.min=-1) (action_executor_acquired.min=-1) (multipart_upload_completed.failures.min=-1) (op_abort.min=-1) (action_executor_acquired.failures.min=-1));
maximums=((object_multipart_aborted.max=-1) (multipart_upload_completed.max=-1) (action_executor_acquired.failures.max=-1) (multipart_upload_completed.failures.max=-1) (op_abort.max=-1) (object_multipart_aborted.failures.max=-1) (action_executor_acquired.max=-1) (op_abort.failures.max=-1));
means=((action_executor_acquired.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000)));
}: Closing block #1: current block= FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8794108122613463043.tmp, state=Writing, dataSize=365, limit=67108864}
22/08/12 15:38:26 DEBUG S3ABlockOutputStream: Executing regular upload for WriteOperationHelper {bucket=master.rando2}
22/08/12 15:38:26 DEBUG S3ADataBlocks: Start datablock[1] upload
22/08/12 15:38:26 DEBUG S3ADataBlocks: FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8794108122613463043.tmp, state=Writing, dataSize=365, limit=67108864}: entering state Upload
22/08/12 15:38:26 DEBUG IOStatisticsStoreImpl: Incrementing counter stream_write_block_uploads by 1 with final value 1
22/08/12 15:38:26 DEBUG ParquetFileWriter: 710: footer length = 546
22/08/12 15:38:26 DEBUG BytesUtils: write le int: 546 => 34 2 0 0
22/08/12 15:38:26 DEBUG S3ABlockOutputStream: Clearing active block
22/08/12 15:38:26 DEBUG S3ABlockOutputStream: S3ABlockOutputStream{WriteOperationHelper {bucket=master.rando2}, blockSize=67108864, activeBlock=FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-7382960693842441093.tmp, state=Writing, dataSize=718, limit=67108864} Statistics=counters=((op_hflush=0) (action_executor_acquired.failures=0) (multipart_upload_completed=0) (stream_write_total_data=0) (stream_write_bytes=718) (op_abort=0) (stream_write_exceptions_completing_upload=0) (stream_write_block_uploads=0) (object_multipart_aborted=0) (op_abort.failures=0) (op_hsync=0) (action_executor_acquired=0) (object_multipart_aborted.failures=0) (stream_write_queue_duration=0) (stream_write_total_time=0) (stream_write_exceptions=0) (multipart_upload_completed.failures=0));
gauges=((stream_write_block_uploads_pending=0) (stream_write_block_uploads_data_pending=0));
minimums=((multipart_upload_completed.failures.min=-1) (op_abort.min=-1) (object_multipart_aborted.min=-1) (action_executor_acquired.failures.min=-1) (object_multipart_aborted.failures.min=-1) (action_executor_acquired.min=-1) (op_abort.failures.min=-1) (multipart_upload_completed.min=-1));
maximums=((multipart_upload_completed.max=-1) (multipart_upload_completed.failures.max=-1) (action_executor_acquired.failures.max=-1) (object_multipart_aborted.failures.max=-1) (object_multipart_aborted.max=-1) (op_abort.max=-1) (op_abort.failures.max=-1) (action_executor_acquired.max=-1));
means=((object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=0, sum=0, mean=0.0000)));
}: Closing block #1: current block= FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-7382960693842441093.tmp, state=Writing, dataSize=718, limit=67108864}
22/08/12 15:38:26 DEBUG S3ABlockOutputStream: Executing regular upload for WriteOperationHelper {bucket=master.rando2}
22/08/12 15:38:26 DEBUG S3ADataBlocks: Start datablock[1] upload
22/08/12 15:38:26 DEBUG S3ADataBlocks: FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-7382960693842441093.tmp, state=Writing, dataSize=718, limit=67108864}: entering state Upload
22/08/12 15:38:26 DEBUG IOStatisticsStoreImpl: Incrementing counter stream_write_block_uploads by 1 with final value 1
22/08/12 15:38:26 DEBUG S3ABlockOutputStream: Clearing active block
22/08/12 15:38:26 DEBUG Invoker: Starting: Writing Object
22/08/12 15:38:26 DEBUG S3AFileSystem: PUT 365 bytes to nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
22/08/12 15:38:26 DEBUG S3AFileSystem: PUT start 365 bytes
22/08/12 15:38:26 DEBUG Invoker: Starting: Writing Object
22/08/12 15:38:26 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request by 1 with final value 2
22/08/12 15:38:26 DEBUG S3AFileSystem: PUT 718 bytes to nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
22/08/12 15:38:26 DEBUG S3AFileSystem: PUT start 718 bytes
22/08/12 15:38:26 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request by 1 with final value 3
22/08/12 15:38:26 DEBUG Mimetypes: Loading mime types from file in the classpath: mime.types
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/activemessage'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ez' to 'application/andrew-inset'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/applefile'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'atom' to 'application/atom+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/atomicmail'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/batch-smtp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/beep+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cals-1840'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cnrp+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/commonground'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cpl+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cybercash'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/dca-rft'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/dec-dx'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/dvcs'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/edi-consent'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/edifact'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/edi-x12'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/eshop'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/font-tdpfr'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/http'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/hyperstudio'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/iges'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.cmd'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.obj'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.response'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.vnd'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/iotp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/ipp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/isup'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'json' to 'application/json'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'hqx' to 'application/mac-binhex40'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'cpt' to 'application/mac-compactpro'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/macwriteii'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/marc'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/mathematica'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mathml' to 'application/mathml+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'doc' to 'application/msword'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/news-message-id'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/news-transmission'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/ocsp-request'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/ocsp-response'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'bin' to 'application/octet-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'dms' to 'application/octet-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'lha' to 'application/octet-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'lzh' to 'application/octet-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'exe' to 'application/octet-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'class' to 'application/octet-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'so' to 'application/octet-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'dll' to 'application/octet-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'dmg' to 'application/octet-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'oda' to 'application/oda'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ogg' to 'application/ogg'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/parityfec'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pdf' to 'application/pdf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pgp-encrypted'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pgp-keys'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pgp-signature'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkcs10'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkcs7-mime'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkcs7-signature'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkix-cert'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkix-crl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkixcmp'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ai' to 'application/postscript'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'eps' to 'application/postscript'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ps' to 'application/postscript'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.alvestrand.titrax-sheet'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.cww'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.nprend'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.plucker'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/qsig'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'rdf' to 'application/rdf+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/reginfo+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/remote-printing'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/riscos'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/rtf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sdp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-payment'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-payment-initiation'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-registration'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-registration-initiation'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sgml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sgml-open-catalog'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sieve'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/slate'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'smi' to 'application/smil'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'smil' to 'application/smil'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'gram' to 'application/srgs'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'grxml' to 'application/srgs+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/timestamp-query'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/timestamp-reply'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/tve-trigger'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vemmi'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.pic-bw-large'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.pic-bw-small'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.pic-bw-var'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.sms'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3m.post-it-notes'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.accpac.simply.aso'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.accpac.simply.imp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.acucobol'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.acucorp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.adobe.xfdf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.aether.imp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.amiga.ami'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.anser-web-certificate-issue-initiation'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.anser-web-funds-transfer-initiation'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.audiograph'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.blueice.multipass'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.bmi'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.businessobjects'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.canon-cpdl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.canon-lips'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cinderella'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.claymore'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.commerce-battelle'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.commonspace'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.contact.cmsg'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cosmocaller'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.criticaltools.wbs+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ctc-posml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cups-postscript'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cups-raster'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cups-raw'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.curl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cybank'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.data-vision.rdz'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dna'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dpgraph'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dreamfactory'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dxr'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecdis-update'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.chart'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.filerequest'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.fileupdate'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.series'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.seriesrequest'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.seriesupdate'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.enliven'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.esf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.msf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.quickanime'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.salt'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.ssf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ericsson.quickcall'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.eudora.data'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fdf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ffsns'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fints'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.flographit'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.framemaker'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fsc.weblaunch'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasys'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasys2'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasys3'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasysgp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasysprs'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujixerox.ddd'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujixerox.docuworks'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujixerox.docuworks.binder'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fut-misnet'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.grafeq'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-account'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-help'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-identity-message'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-injector'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-tool-message'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-tool-template'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-vcard'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hbci'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hhe.lesson-player'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-hpgl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-hpid'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-hps'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-pcl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-pclxl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.httphone'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hzn-3d-crossword'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.afplinedata'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.electronic-media'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.minipay'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.modcap'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.rights-management'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.secure-container'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.informix-visionary'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intercon.formnet'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intertrust.digibox'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intertrust.nncp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intu.qbo'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intu.qfx'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.irepository.package+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.is-xpr'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-directory-service'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-jpnstore-wakeup'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-payment-wakeup'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-registration'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-registration-wakeup'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-setstore-wakeup'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-verification'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-verification-wakeup'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.jisp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.karbon'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kchart'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kformula'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kivio'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kontour'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kpresenter'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kspread'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kword'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kenameaapp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.koan'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.liberty-request+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.llamagraphics.life-balance.desktop'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.llamagraphics.life-balance.exchange+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-1-2-3'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-approach'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-freelance'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-notes'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-organizer'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-screencam'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-wordpro'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mcd'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mediastation.cdkey'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.meridian-slingshot'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.micrografx.flo'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.micrografx.igx'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mif' to 'application/vnd.mif'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.minisoft-hp3000-save'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mitsubishi.misty-guard.trustweb'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.daf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.dis'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.mbk'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.mqy'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.msl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.plc'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.txf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mophun.application'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mophun.certificate'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.adsi'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.fis'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.gotap'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.kmr'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.ttc'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.wem'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xul' to 'application/vnd.mozilla.xul+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-artgalry'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-asf'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xls' to 'application/vnd.ms-excel'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-lrm'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ppt' to 'application/vnd.ms-powerpoint'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-project'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-tnef'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-works'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-wpl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mseq'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.msign'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.music-niff'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.musician'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.netfpx'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.noblenet-directory'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.noblenet-sealer'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.noblenet-web'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.novadigm.edm'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.novadigm.edx'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.novadigm.ext'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.obn'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.osa.netdeploy'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.palm'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pg.format'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pg.osasli'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder6'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder6-s'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder7'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder7-s'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder75'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder75-s'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.previewsystems.box'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.publishare-delta-tree'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pvi.ptid1'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pwg-multiplexed'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pwg-xhtml-print+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.quark.quarkxpress'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.rapid'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'rm' to 'application/vnd.rn-realmedia'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.s3sms'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sealed.net'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.seemail'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.formdata'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.formtemplate'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.interchange'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.package'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.smaf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sss-cod'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sss-dtf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sss-ntf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.street-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.svd'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.swiftview-ics'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.triscape.mxs'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.trueapp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.truedoc'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ufdl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.alert'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.alert-wbxml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.bearer-choice'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.bearer-choice-wbxml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.cacheop'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.cacheop-wbxml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.channel'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.channel-wbxml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.list'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.list-wbxml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.listcmd'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.listcmd-wbxml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.signal'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vcx'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vectorworks'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vidsoft.vidconference'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.visio'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.visionary'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vividence.scriptfile'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vsf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wap.sic'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wap.slc'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'wbxml' to 'application/vnd.wap.wbxml'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'wmlc' to 'application/vnd.wap.wmlc'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'wmlsc' to 'application/vnd.wap.wmlscriptc'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.webturbo'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wrq-hp3000-labelled'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wt.stf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wv.csp+wbxml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.xara'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.xfdl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yamaha.hv-dic'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yamaha.hv-script'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yamaha.hv-voice'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yellowriver-custom-menu'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'vxml' to 'application/voicexml+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/watcherinfo+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/whoispp-query'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/whoispp-response'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/wita'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/wordperfect5.1'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'bcpio' to 'application/x-bcpio'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'vcd' to 'application/x-cdlink'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pgn' to 'application/x-chess-pgn'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/x-compress'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'cpio' to 'application/x-cpio'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'csh' to 'application/x-csh'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'dcr' to 'application/x-director'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'dir' to 'application/x-director'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'dxr' to 'application/x-director'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'dvi' to 'application/x-dvi'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'spl' to 'application/x-futuresplash'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'gtar' to 'application/x-gtar'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'gz' to 'application/x-gzip'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'hdf' to 'application/x-hdf'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'js' to 'application/x-javascript'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'jnlp' to 'application/x-java-jnlp-file'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'skp' to 'application/x-koan'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'skd' to 'application/x-koan'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'skt' to 'application/x-koan'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'skm' to 'application/x-koan'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'latex' to 'application/x-latex'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'nc' to 'application/x-netcdf'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'cdf' to 'application/x-netcdf'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'sh' to 'application/x-sh'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'shar' to 'application/x-shar'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'swf' to 'application/x-shockwave-flash'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'sit' to 'application/x-stuffit'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'sv4cpio' to 'application/x-sv4cpio'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'sv4crc' to 'application/x-sv4crc'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'tar' to 'application/x-tar'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'tcl' to 'application/x-tcl'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'tex' to 'application/x-tex'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'texinfo' to 'application/x-texinfo'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'texi' to 'application/x-texinfo'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 't' to 'application/x-troff'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'tr' to 'application/x-troff'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'roff' to 'application/x-troff'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'man' to 'application/x-troff-man'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'me' to 'application/x-troff-me'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ms' to 'application/x-troff-ms'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ustar' to 'application/x-ustar'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'src' to 'application/x-wais-source'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/x400-bp'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xhtml' to 'application/xhtml+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xht' to 'application/xhtml+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xslt' to 'application/xslt+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xml' to 'application/xml'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xsl' to 'application/xml'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'dtd' to 'application/xml-dtd'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/xml-external-parsed-entity'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'zip' to 'application/zip'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/32kadpcm'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/amr'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/amr-wb'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'au' to 'audio/basic'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'snd' to 'audio/basic'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/cn'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/dat12'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/dsr-es201108'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/dvi4'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/evrc'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/evrc0'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g722'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g.722.1'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g723'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-16'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-24'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-32'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-40'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g728'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g729'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g729D'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g729E'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/gsm'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/gsm-efr'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l8'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l16'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l20'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l24'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/lpc'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mid' to 'audio/midi'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'midi' to 'audio/midi'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'kar' to 'audio/midi'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/mpa'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/mpa-robust'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'm4a' to 'audio/mp4a-latm'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'm4p' to 'audio/mp4a-latm'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mpga' to 'audio/mpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mp2' to 'audio/mpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mp3' to 'audio/mpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/parityfec'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/pcma'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/pcmu'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/prs.sid'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/qcelp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/red'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/smv'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/smv0'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/telephone-event'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/tone'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vdvi'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.3gpp.iufp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.cisco.nse'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.cns.anp1'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.cns.inf1'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.digital-winds'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.everad.plj'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.lucent.voice'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nortel.vbk'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nuera.ecelp4800'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nuera.ecelp7470'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nuera.ecelp9600'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.octel.sbc'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.qcelp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.rhetorex.32kadpcm'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.vmx.cvsd'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'aif' to 'audio/x-aiff'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'aiff' to 'audio/x-aiff'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'aifc' to 'audio/x-aiff'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/x-alaw-basic'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'm3u' to 'audio/x-mpegurl'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ram' to 'audio/x-pn-realaudio'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ra' to 'audio/x-pn-realaudio'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/x-pn-realaudio-plugin'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'wav' to 'audio/x-wav'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pdb' to 'chemical/x-pdb'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xyz' to 'chemical/x-xyz'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'bmp' to 'image/bmp'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'cgm' to 'image/cgm'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/g3fax'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'gif' to 'image/gif'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ief' to 'image/ief'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'jpeg' to 'image/jpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'jpg' to 'image/jpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'jpe' to 'image/jpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'jp2' to 'image/jp2'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/naplps'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pict' to 'image/pict'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pic' to 'image/pict'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pct' to 'image/pict'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'png' to 'image/png'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/prs.btif'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/prs.pti'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'svg' to 'image/svg+xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/t38'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'tiff' to 'image/tiff'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'tif' to 'image/tiff'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/tiff-fx'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.cns.inf2'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'djvu' to 'image/vnd.djvu'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'djv' to 'image/vnd.djvu'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.dwg'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.dxf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fastbidsheet'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fpx'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fst'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fujixerox.edmics-mmr'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fujixerox.edmics-rlc'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.globalgraphics.pgb'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.mix'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.ms-modi'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.net-fpx'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.svf'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'wbmp' to 'image/vnd.wap.wbmp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.xiff'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ras' to 'image/x-cmu-raster'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pntg' to 'image/x-macpaint'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pnt' to 'image/x-macpaint'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mac' to 'image/x-macpaint'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ico' to 'image/x-icon'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pnm' to 'image/x-portable-anymap'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pbm' to 'image/x-portable-bitmap'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'pgm' to 'image/x-portable-graymap'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ppm' to 'image/x-portable-pixmap'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'qtif' to 'image/x-quicktime'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'qti' to 'image/x-quicktime'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'rgb' to 'image/x-rgb'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xbm' to 'image/x-xbitmap'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xpm' to 'image/x-xpixmap'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'xwd' to 'image/x-xwindowdump'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/delivery-status'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/disposition-notification'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/external-body'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/http'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/news'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/partial'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/rfc822'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/s-http'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/sip'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/sipfrag'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'igs' to 'model/iges'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'iges' to 'model/iges'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'msh' to 'model/mesh'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mesh' to 'model/mesh'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'silo' to 'model/mesh'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.dwf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.flatland.3dml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.gdl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.gs-gdl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.gtw'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.mts'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.parasolid.transmit.binary'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.parasolid.transmit.text'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.vtu'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'wrl' to 'model/vrml'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'vrml' to 'model/vrml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/alternative'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/appledouble'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/byteranges'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/digest'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/encrypted'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/form-data'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/header-set'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/mixed'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/parallel'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/related'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/report'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/signed'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/voice-message'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ics' to 'text/calendar'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ifb' to 'text/calendar'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'css' to 'text/css'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/directory'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/enriched'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'html' to 'text/html'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'htm' to 'text/html'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/parityfec'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'asc' to 'text/plain'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'txt' to 'text/plain'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/prs.lines.tag'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/rfc822-headers'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'rtx' to 'text/richtext'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'rtf' to 'text/rtf'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'sgml' to 'text/sgml'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'sgm' to 'text/sgml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/t140'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'tsv' to 'text/tab-separated-values'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/uri-list'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.abc'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.curl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.dmclientscript'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.fly'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.fmi.flexstor'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.in3d.3dml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.in3d.spot'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.iptc.nitf'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.iptc.newsml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.latex-z'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.motorola.reflex'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.ms-mediapackage'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.net2phone.commcenter.command'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.sun.j2me.app-descriptor'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.wap.si'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.wap.sl'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'wml' to 'text/vnd.wap.wml'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'wmls' to 'text/vnd.wap.wmlscript'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'etx' to 'text/x-setext'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/xml'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/xml-external-parsed-entity'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension '3gp' to 'video/3gpp'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/bmpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/bt656'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/celb'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/dv'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h261'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h263'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h263-1998'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h263-2000'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/jpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp1s'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp2p'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp2t'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mp4' to 'video/mp4'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp4v-es'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mpv'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mpeg' to 'video/mpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mpg' to 'video/mpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mpe' to 'video/mpeg'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/nv'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ogv' to 'video/ogg'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/parityfec'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/pointer'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'qt' to 'video/quicktime'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mov' to 'video/quicktime'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/smpte292m'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.fvt'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.motorola.video'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.motorola.videop'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'mxu' to 'video/vnd.mpegurl'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'm4u' to 'video/vnd.mpegurl'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.nokia.interleaved-multimedia'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.objectvideo'
22/08/12 15:38:26 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.vivo'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'webm' to 'video/webm'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'dv' to 'video/x-dv'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'dif' to 'video/x-dv'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'flv' to 'video/x-flv'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'm4v' to 'video/x-m4v'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'wmv' to 'video/x-ms-wmv'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'avi' to 'video/x-msvideo'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'movie' to 'video/x-sgi-movie'
22/08/12 15:38:26 DEBUG Mimetypes: Setting mime type for extension 'ice' to 'x-conference/x-cooltalk'
22/08/12 15:38:26 DEBUG Mimetypes: Extension 'tmp' is unrecognized in mime type listing, using default mime type: 'application/octet-stream'
22/08/12 15:38:26 DEBUG Mimetypes: Extension 'tmp' is unrecognized in mime type listing, using default mime type: 'application/octet-stream'
22/08/12 15:38:26 DEBUG request: Sending Request: PUT http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet Headers: (amz-sdk-invocation-id: da4fe9b9-c13c-feb1-f83e-337692cd6cc6, Content-Length: 365, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:26 DEBUG request: Sending Request: PUT http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet Headers: (amz-sdk-invocation-id: c1f87863-3f41-4ec7-2c5e-419397cea555, Content-Length: 718, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:26 DEBUG AWS4Signer: AWS4 Canonical Request: '"PUT
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
amz-sdk-invocation-id:c1f87863-3f41-4ec7-2c5e-419397cea555
amz-sdk-request:attempt=1;max=21
amz-sdk-retry:0/0/500
content-length:892
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20220812T143826Z
x-amz-decoded-content-length:718
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD"
22/08/12 15:38:26 DEBUG AWS4Signer: AWS4 Canonical Request: '"PUT
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
amz-sdk-invocation-id:da4fe9b9-c13c-feb1-f83e-337692cd6cc6
amz-sdk-request:attempt=1;max=21
amz-sdk-retry:0/0/500
content-length:539
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20220812T143826Z
x-amz-decoded-content-length:365
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD"
22/08/12 15:38:26 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143826Z
20220812/us-east-1/s3/aws4_request
b24667ebf0599ca93e0b23d2bc317ffb870bb36b87b41dd7bdad73e0ba4f4865"
22/08/12 15:38:26 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143826Z
20220812/us-east-1/s3/aws4_request
ce8d95eb16ce79111f7bee6b7c1884bfcb9dd8ac6c77c524aa68e81cc3e6715a"
22/08/12 15:38:26 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:26 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:26 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:26 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:26 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:26 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:26 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:26 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:26 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 7][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:26 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:26 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:26 DEBUG MainClientExec: Executing request PUT /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:26 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:26 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:26 DEBUG MainClientExec: Executing request PUT /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:26 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> PUT /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: c1f87863-3f41-4ec7-2c5e-419397cea555
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> amz-sdk-request: attempt=1;max=21
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> PUT /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> Host: localhost:30600
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=4885c4e0f0005106c114fdce8ea3c0da7fa19e695e6af7a373fc751e80fe8599
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> amz-sdk-invocation-id: da4fe9b9-c13c-feb1-f83e-337692cd6cc6
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> amz-sdk-request: attempt=1;max=21
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=0c4b1a5d644e3c19a75e328b5c37992d1647b13d18ca3f8653725d29fa5141ab
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143826Z
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> Content-Type: application/octet-stream
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> x-amz-decoded-content-length: 718
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> Content-Length: 892
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> X-Amz-Date: 20220812T143826Z
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 >> Expect: 100-continue
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> x-amz-decoded-content-length: 365
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> Content-Length: 539
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> Connection: Keep-Alive
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 >> Expect: 100-continue
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "PUT /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: c1f87863-3f41-4ec7-2c5e-419397cea555[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "PUT /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: attempt=1;max=21[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "amz-sdk-invocation-id: da4fe9b9-c13c-feb1-f83e-337692cd6cc6[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "amz-sdk-request: attempt=1;max=21[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=4885c4e0f0005106c114fdce8ea3c0da7fa19e695e6af7a373fc751e80fe8599[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=0c4b1a5d644e3c19a75e328b5c37992d1647b13d18ca3f8653725d29fa5141ab[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143826Z[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "x-amz-decoded-content-length: 718[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "Content-Length: 892[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "X-Amz-Date: 20220812T143826Z[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "Expect: 100-continue[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "x-amz-decoded-content-length: 365[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "Content-Length: 539[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "Expect: 100-continue[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 << "HTTP/1.1 100 Continue[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 << "HTTP/1.1 100 Continue[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 << "[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:26 DEBUG headers: http-outgoing-5 << HTTP/1.1 100 Continue
22/08/12 15:38:26 DEBUG headers: http-outgoing-7 << HTTP/1.1 100 Continue
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "2ce;chunk-signature=330ac793c7866fd0ecc00c61f47e53eaef7fa2409f3ca0be25b3ca8738591287[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "16d;chunk-signature=d2c39236a9f0931586dbd6af3722c8e1172fcbb64da123bdda7e3e28d30b6069[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "PAR1[0x15][0x2][0x19]<H[0xc]spark_schema[0x15][0x4][0x0][0x15][0x4]%[0x2][0x18][0x1]a[0x0][0x15][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "PAR1[0x15][0x0][0x15][0x1c][0x15] [0x15][0xfe][0xc8][0xd0][0xb6][0xb][0x1c][0x15][0x2][0x15][0x0][0x15][0x6][0x15][0x8][0x0][0x0][0xe]4[0x2][0x0][0x0][0x0][0x3][0x1][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x15][0x0][0x15][0x1c][0x15] [0x15][0x9d][0xb3][0x9d][0xee][0x3][0x1c][0x15][0x2][0x15][0x0][0x15][0x6][0x15][0x8][0x0][0x0][0xe]4[0x2][0x0][0x0][0x0][0x3][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x19][0x11][0x2][0x19][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x19][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x15][0x2][0x19][0x16][0x0][0x0][0x19][0x11][0x2][0x19][0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x19][0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x15][0x2][0x19][0x16][0x0][0x0][0x19][0x1c][0x16][0x8][0x15]N[0x16][0x0][0x0][0x0][0x19][0x1c][0x16]V[0x15]N[0x16][0x0][0x0][0x0][0x15][0x2][0x19]<H[0xc]spark_schema[0x15][0x4][0x0][0x15][0x4]%[0x2][0x18][0x1]a[0x0][0x15][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "%[0x2][0x18][0x1]b[0x0][0x16][0x2][0x19][0x1c][0x19],&[0x8][0x1c][0x15][0x4][0x19]5[0x8][0x6][0x0][0x19][0x18][0x1]a[0x15][0x2][0x16][0x2][0x16]J[0x16]N&[0x8]<[0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x16][0x0]([0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x19][0x1c][0x15][0x0][0x15][0x0][0x15][0x2][0x0][0x0][0x16][0xa0][0x2][0x15][0x14][0x16][0xa4][0x1][0x15]>[0x0]&V[0x1c][0x15][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "%[0x2][0x18][0x1]b[0x0][0x16][0x0][0x19][0xc][0x19],[0x18][0x18]org.apache.spark.version[0x18][0x5]3.3.0[0x0][0x18])org.apache.spark.sql.parquet.row.metadata[0x18][0x90][0x1]{"type":"struct","fields":[{"name":"a","type":"long","nullable":true,"metadata":{}},{"name":"b","type":"double","nullable":true,"metadata":{}}]}[0x0][0x18]Jparquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94)[0x19],[0x1c][0x0][0x0][0x1c][0x0][0x0][0x0]a[0x1][0x0][0x0]PAR1[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "0;chunk-signature=fd7236215112fc3b5f872abc8dcf9c0b7e173aea98b150b240f05690fe30eb05[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-7 >> "[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "[0x19]5[0x8][0x6][0x0][0x19][0x18][0x1]b[0x15][0x2][0x16][0x2][0x16]J[0x16]N&V<[0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x16][0x0]([0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x0][0x19][0x1c][0x15][0x0][0x15][0x0][0x15][0x2][0x0][0x0][0x16][0xb4][0x2][0x15][0x14][0x16][0xe2][0x1][0x15]>[0x0][0x16][0x94][0x1][0x16][0x2]&[0x8][0x16][0x9c][0x1][0x14][0x0][0x0][0x19],[0x18][0x18]org.apache.spark.version[0x18][0x5]3.3.0[0x0][0x18])org.apache.spark.sql.parquet.row.metadata[0x18][0x90][0x1]{"type":"struct","fields":[{"name":"a","type":"long","nullable":true,"metadata":{}},{"name":"b","type":"double","nullable":true,"metadata":{}}]}[0x0][0x18]Jparquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94)[0x19],[0x1c][0x0][0x0][0x1c][0x0][0x0][0x0]"[0x2][0x0][0x0]PAR1[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "0;chunk-signature=15b43ca3265417e074fd9a28fb6c2cbe8be7809b79e1576b3c474ccebda389cb[\r][\n]"
22/08/12 15:38:26 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "Etag: "0a03a8cbced07244b6b9fdfaa86b9b37c3070344e57dde0757bc57e6a7ab045d"[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "X-Amz-Version-Id: 5be6c0fab64746a5986bb2599e3e88ac[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:27 GMT[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "Content-Length: 0[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << HTTP/1.1 200 OK
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << Etag: "0a03a8cbced07244b6b9fdfaa86b9b37c3070344e57dde0757bc57e6a7ab045d"
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << X-Amz-Version-Id: 5be6c0fab64746a5986bb2599e3e88ac
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:27 GMT
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << Content-Length: 0
22/08/12 15:38:27 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:27 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:27 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:27 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:27 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:27 GMT
22/08/12 15:38:27 DEBUG request: Received successful response: 200, AWS Request ID: null
22/08/12 15:38:27 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:27 DEBUG requestId: AWS Request ID: not available
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 43
22/08/12 15:38:27 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[PutObjectRequest], AWSRequestID=[null], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=16, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[0.164], ClientExecuteTime=[936.476], HttpClientSendRequestTime=[2.909], HttpRequestTime=[934.637], ApiCallLatency=[936.141], RequestSigningTime=[0.468], CredentialsRequestTime=[0.009, 0.004], HttpClientReceiveResponseTime=[930.405],
22/08/12 15:38:27 DEBUG S3AFileSystem: PUT completed success=true; 718 bytes
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_bytes by 718 with final value 718
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request_completed by 1 with final value 2
22/08/12 15:38:27 DEBUG S3AFileSystem: Finished write to nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet, len 718. etag 0a03a8cbced07244b6b9fdfaa86b9b37c3070344e57dde0757bc57e6a7ab045d, version 5be6c0fab64746a5986bb2599e3e88ac
22/08/12 15:38:27 DEBUG CallableSupplier: Starting: Waiting for task completion
22/08/12 15:38:27 DEBUG S3AFileSystem: Starting: Deleting 5 keys
22/08/12 15:38:27 DEBUG S3AFileSystem: Initiating delete operation for 5 objects
22/08/12 15:38:27 DEBUG S3AFileSystem: nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/
22/08/12 15:38:27 DEBUG S3AFileSystem: nonemptyprefix4/_temporary/0/_temporary/
22/08/12 15:38:27 DEBUG S3AFileSystem: nonemptyprefix4/_temporary/0/
22/08/12 15:38:27 DEBUG S3AFileSystem: nonemptyprefix4/_temporary/
22/08/12 15:38:27 DEBUG S3AFileSystem: nonemptyprefix4/
22/08/12 15:38:27 DEBUG S3AFileSystem: Starting: DELETE 5 keys
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter object_bulk_delete_request by 1 with final value 2
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter object_delete_objects by 5 with final value 7
22/08/12 15:38:27 DEBUG request: Sending Request: POST http://localhost:30600 /master.rando2/ Parameters: ({"delete":[null]}Headers: (amz-sdk-invocation-id: f7211adc-818c-b37f-3496-7b9d87c6a903, Content-Length: 387, Content-MD5: puxnAF3WXY9tsAZAu8TteQ==, Content-Type: application/xml, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:27 DEBUG AWS4Signer: AWS4 Canonical Request: '"POST
/master.rando2/
delete=
amz-sdk-invocation-id:f7211adc-818c-b37f-3496-7b9d87c6a903
amz-sdk-request:ttl=20220812T144147Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-length:387
content-md5:puxnAF3WXY9tsAZAu8TteQ==
content-type:application/xml
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:f5cb53cbea9f96019105c26ab3bfc9cfcf9c2999482dc995f17c2ff10e26252c
x-amz-date:20220812T143827Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
f5cb53cbea9f96019105c26ab3bfc9cfcf9c2999482dc995f17c2ff10e26252c"
22/08/12 15:38:27 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143827Z
20220812/us-east-1/s3/aws4_request
76dbf2fcef1abd90afadd531e7c6cb0509948f76cf5941c0e24f9b785149287b"
22/08/12 15:38:27 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:27 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:27 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:27 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:27 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:27 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:27 DEBUG MainClientExec: Executing request POST /master.rando2/?delete HTTP/1.1
22/08/12 15:38:27 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> POST /master.rando2/?delete HTTP/1.1
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: f7211adc-818c-b37f-3496-7b9d87c6a903
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144147Z;attempt=1;max=21
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=f540bddac29f146860fbb00659152548f848d1f0467da4b32f2381fd654bc157
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> Content-MD5: puxnAF3WXY9tsAZAu8TteQ==
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> Content-Type: application/xml
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: f5cb53cbea9f96019105c26ab3bfc9cfcf9c2999482dc995f17c2ff10e26252c
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143827Z
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> Content-Length: 387
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "POST /master.rando2/?delete HTTP/1.1[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: f7211adc-818c-b37f-3496-7b9d87c6a903[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144147Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=f540bddac29f146860fbb00659152548f848d1f0467da4b32f2381fd654bc157[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "Content-MD5: puxnAF3WXY9tsAZAu8TteQ==[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: f5cb53cbea9f96019105c26ab3bfc9cfcf9c2999482dc995f17c2ff10e26252c[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143827Z[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "Content-Length: 387[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "<Delete><Quiet>true</Quiet><Object><Key>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/</Key></Object><Object><Key>nonemptyprefix4/_temporary/0/_temporary/</Key></Object><Object><Key>nonemptyprefix4/_temporary/0/</Key></Object><Object><Key>nonemptyprefix4/_temporary/</Key></Object><Object><Key>nonemptyprefix4/</Key></Object></Delete>"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: 2a18ce36-0433-46a2-962b-534ae0446156[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: 2a18ce36-0433-46a2-962b-534ae0446156[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:27 GMT[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "Content-Length: 116[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 << "<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>"
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << HTTP/1.1 200 OK
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: 2a18ce36-0433-46a2-962b-534ae0446156
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: 2a18ce36-0433-46a2-962b-534ae0446156
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:27 GMT
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 << Content-Length: 116
22/08/12 15:38:27 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:27 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:27 GMT
22/08/12 15:38:27 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$DeleteObjectsHandler
22/08/12 15:38:27 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:27 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:27 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:27 DEBUG request: Received successful response: 200, AWS Request ID: 2a18ce36-0433-46a2-962b-534ae0446156
22/08/12 15:38:27 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:27 DEBUG requestId: AWS Request ID: 2a18ce36-0433-46a2-962b-534ae0446156
22/08/12 15:38:27 DEBUG requestId: AWS Extended Request ID: 2a18ce36-0433-46a2-962b-534ae0446156
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 44
22/08/12 15:38:27 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[DeleteObjectsRequest], AWSRequestID=[2a18ce36-0433-46a2-962b-534ae0446156], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=15, RequestCount=1, HttpClientPoolLeasedCount=1, ResponseProcessingTime=[1.48], ClientExecuteTime=[105.861], HttpClientSendRequestTime=[0.981], HttpRequestTime=[102.841], ApiCallLatency=[105.566], RequestSigningTime=[0.458], CredentialsRequestTime=[0.011, 0.005], HttpClientReceiveResponseTime=[100.985],
22/08/12 15:38:27 DEBUG S3AFileSystem: DELETE 5 keys: duration 0:00.107s
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter fake_directories_deleted by 5 with final value 7
22/08/12 15:38:27 DEBUG S3AFileSystem: Deleting 5 keys: duration 0:00.108s
22/08/12 15:38:27 DEBUG CallableSupplier: Waiting for task completion: duration 0:00.108s
22/08/12 15:38:27 DEBUG Invoker: Writing Object: duration 0:01.060s
22/08/12 15:38:27 DEBUG S3ADataBlocks: FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-7382960693842441093.tmp, state=Upload, dataSize=718, limit=67108864}: entering state Closed
22/08/12 15:38:27 DEBUG S3ADataBlocks: Closed FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-7382960693842441093.tmp, state=Closed, dataSize=718, limit=67108864}
22/08/12 15:38:27 DEBUG S3ADataBlocks: Closing FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-7382960693842441093.tmp, state=Closed, dataSize=718, limit=67108864}
22/08/12 15:38:27 DEBUG S3ADataBlocks: block[1]: closeBlock()
22/08/12 15:38:27 DEBUG S3ABlockOutputStream: Upload complete to nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet by WriteOperationHelper {bucket=master.rando2}
22/08/12 15:38:27 DEBUG S3ABlockOutputStream: Statistics: OutputStreamStatistics{counters=((stream_write_block_uploads=1) (object_multipart_aborted.failures=0) (multipart_upload_completed.failures=0) (stream_write_exceptions=0) (action_executor_acquired=0) (object_multipart_aborted=0) (op_abort.failures=0) (stream_write_exceptions_completing_upload=0) (multipart_upload_completed=0) (op_hflush=0) (stream_write_total_time=0) (action_executor_acquired.failures=0) (op_abort=0) (stream_write_total_data=718) (stream_write_queue_duration=0) (stream_write_bytes=718) (op_hsync=0));
gauges=((stream_write_block_uploads_data_pending=0) (stream_write_block_uploads_pending=1));
minimums=((multipart_upload_completed.failures.min=-1) (op_abort.min=-1) (action_executor_acquired.min=-1) (op_abort.failures.min=-1) (multipart_upload_completed.min=-1) (object_multipart_aborted.failures.min=-1) (object_multipart_aborted.min=-1) (action_executor_acquired.failures.min=-1));
maximums=((op_abort.max=-1) (multipart_upload_completed.max=-1) (op_abort.failures.max=-1) (object_multipart_aborted.max=-1) (multipart_upload_completed.failures.max=-1) (action_executor_acquired.failures.max=-1) (action_executor_acquired.max=-1) (object_multipart_aborted.failures.max=-1));
means=((multipart_upload_completed.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000)));
, blocksActive=0, blockUploadsCompleted=0, blocksAllocated=1, blocksReleased=1, blocksActivelyAllocated=0, transferDuration=0 ms, totalUploadDuration=0 ms, effectiveBandwidth=0.0 bytes/s}
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter stream_write_total_data by 718 with final value 718
22/08/12 15:38:27 DEBUG MemoryManager: Adjust block size from 134,217,728 to 134,217,728 for writer: org.apache.parquet.hadoop.InternalParquetRecordWriter@435cea10
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter op_get_file_status by 1 with final value 1
22/08/12 15:38:27 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet); needEmptyDirectory=false
22/08/12 15:38:27 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 21
22/08/12 15:38:27 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 21
22/08/12 15:38:27 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet with change tracker null
22/08/12 15:38:27 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet Headers: (amz-sdk-invocation-id: 0224943c-ac16-4b2a-2d85-2ba4f657876f, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:27 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
amz-sdk-invocation-id:0224943c-ac16-4b2a-2d85-2ba4f657876f
amz-sdk-request:ttl=20220812T144147Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143827Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:27 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143827Z
20220812/us-east-1/s3/aws4_request
705e001feb06f60bdde109dc954b766d09ec3f84f5122ffafcff4cbf0f0f728b"
22/08/12 15:38:27 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:27 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:27 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:27 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:27 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:27 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:27 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:27 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: 0224943c-ac16-4b2a-2d85-2ba4f657876f
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144147Z;attempt=1;max=21
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=70fe5122054da46ef05e1746ab2da3f1b244e3d918cf96cba6ee5a7e447985fd
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143827Z
22/08/12 15:38:27 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: 0224943c-ac16-4b2a-2d85-2ba4f657876f[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144147Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=70fe5122054da46ef05e1746ab2da3f1b244e3d918cf96cba6ee5a7e447985fd[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143827Z[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:27 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "Etag: "4f910321a51d23c0cf5e3c96276ed6aa3c9a9e02700ba2ce2945ea3bf6e2052b"[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "X-Amz-Version-Id: 38a0567050164db49cd07507b8d5c28d[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "Date: Fri, 12 Aug 2022 14:38:28 GMT[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "Content-Length: 0[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "[\r][\n]"
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << HTTP/1.1 200 OK
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << Etag: "4f910321a51d23c0cf5e3c96276ed6aa3c9a9e02700ba2ce2945ea3bf6e2052b"
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << X-Amz-Version-Id: 38a0567050164db49cd07507b8d5c28d
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << Date: Fri, 12 Aug 2022 14:38:28 GMT
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << Content-Length: 0
22/08/12 15:38:28 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:28 DEBUG PoolingHttpClientConnectionManager: Connection [id: 7][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:28 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 0
22/08/12 15:38:28 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 7][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:28 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:28 GMT
22/08/12 15:38:28 DEBUG request: Received successful response: 200, AWS Request ID: null
22/08/12 15:38:28 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:28 DEBUG requestId: AWS Request ID: not available
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 45
22/08/12 15:38:28 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[PutObjectRequest], AWSRequestID=[null], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=16, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[0.091], ClientExecuteTime=[2021.522], HttpClientSendRequestTime=[2.776], HttpRequestTime=[2019.657], ApiCallLatency=[2021.038], RequestSigningTime=[0.505], CredentialsRequestTime=[0.015, 0.006], HttpClientReceiveResponseTime=[2015.537],
22/08/12 15:38:28 DEBUG S3AFileSystem: PUT completed success=true; 365 bytes
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_bytes by 365 with final value 1083
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request_completed by 1 with final value 3
22/08/12 15:38:28 DEBUG S3AFileSystem: Finished write to nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet, len 365. etag 4f910321a51d23c0cf5e3c96276ed6aa3c9a9e02700ba2ce2945ea3bf6e2052b, version 38a0567050164db49cd07507b8d5c28d
22/08/12 15:38:28 DEBUG CallableSupplier: Starting: Waiting for task completion
22/08/12 15:38:28 DEBUG S3AFileSystem: Starting: Deleting 5 keys
22/08/12 15:38:28 DEBUG S3AFileSystem: Initiating delete operation for 5 objects
22/08/12 15:38:28 DEBUG S3AFileSystem: nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/
22/08/12 15:38:28 DEBUG S3AFileSystem: nonemptyprefix4/_temporary/0/_temporary/
22/08/12 15:38:28 DEBUG S3AFileSystem: nonemptyprefix4/_temporary/0/
22/08/12 15:38:28 DEBUG S3AFileSystem: nonemptyprefix4/_temporary/
22/08/12 15:38:28 DEBUG S3AFileSystem: nonemptyprefix4/
22/08/12 15:38:28 DEBUG S3AFileSystem: Starting: DELETE 5 keys
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter object_bulk_delete_request by 1 with final value 3
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter object_delete_objects by 5 with final value 12
22/08/12 15:38:28 DEBUG request: Sending Request: POST http://localhost:30600 /master.rando2/ Parameters: ({"delete":[null]}Headers: (amz-sdk-invocation-id: b1d56a70-a196-4086-4f5c-0dbcb8c5ad89, Content-Length: 387, Content-MD5: Pjwk9JqAh9Q+cFuuoUEllQ==, Content-Type: application/xml, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:28 DEBUG AWS4Signer: AWS4 Canonical Request: '"POST
/master.rando2/
delete=
amz-sdk-invocation-id:b1d56a70-a196-4086-4f5c-0dbcb8c5ad89
amz-sdk-request:ttl=20220812T144148Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-length:387
content-md5:Pjwk9JqAh9Q+cFuuoUEllQ==
content-type:application/xml
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:2f5d32cf659fc6fe9e851147a382cc84646b92f2ad393dbd810012ad99747758
x-amz-date:20220812T143828Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
2f5d32cf659fc6fe9e851147a382cc84646b92f2ad393dbd810012ad99747758"
22/08/12 15:38:28 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143828Z
20220812/us-east-1/s3/aws4_request
59300c35f037e75ddfc5b3cc6083117c4a0cd0af898d4291764530fd69f4387e"
22/08/12 15:38:28 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:28 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:28 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:28 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 7][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:28 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:28 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:28 DEBUG MainClientExec: Executing request POST /master.rando2/?delete HTTP/1.1
22/08/12 15:38:28 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> POST /master.rando2/?delete HTTP/1.1
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> Host: localhost:30600
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> amz-sdk-invocation-id: b1d56a70-a196-4086-4f5c-0dbcb8c5ad89
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> amz-sdk-request: ttl=20220812T144148Z;attempt=1;max=21
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=fde08ab485c57994ea01c700411600bbfa06ce3decd3eaa4fd430795954fe9b7
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> Content-MD5: Pjwk9JqAh9Q+cFuuoUEllQ==
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> Content-Type: application/xml
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> x-amz-content-sha256: 2f5d32cf659fc6fe9e851147a382cc84646b92f2ad393dbd810012ad99747758
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> X-Amz-Date: 20220812T143828Z
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> Content-Length: 387
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> Connection: Keep-Alive
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "POST /master.rando2/?delete HTTP/1.1[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "amz-sdk-invocation-id: b1d56a70-a196-4086-4f5c-0dbcb8c5ad89[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "amz-sdk-request: ttl=20220812T144148Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=fde08ab485c57994ea01c700411600bbfa06ce3decd3eaa4fd430795954fe9b7[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "Content-MD5: Pjwk9JqAh9Q+cFuuoUEllQ==[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "x-amz-content-sha256: 2f5d32cf659fc6fe9e851147a382cc84646b92f2ad393dbd810012ad99747758[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "X-Amz-Date: 20220812T143828Z[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "Content-Length: 387[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "<Delete><Quiet>true</Quiet><Object><Key>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/</Key></Object><Object><Key>nonemptyprefix4/_temporary/0/_temporary/</Key></Object><Object><Key>nonemptyprefix4/_temporary/0/</Key></Object><Object><Key>nonemptyprefix4/_temporary/</Key></Object><Object><Key>nonemptyprefix4/</Key></Object></Delete>"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "X-Amz-Id-2: 010a54e2-28e6-4e94-aadc-0ec833434094[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "X-Amz-Request-Id: 010a54e2-28e6-4e94-aadc-0ec833434094[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "Date: Fri, 12 Aug 2022 14:38:28 GMT[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "Content-Length: 116[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 << "<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>"
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << HTTP/1.1 200 OK
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << Content-Type: application/xml
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << X-Amz-Id-2: 010a54e2-28e6-4e94-aadc-0ec833434094
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << X-Amz-Request-Id: 010a54e2-28e6-4e94-aadc-0ec833434094
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << Date: Fri, 12 Aug 2022 14:38:28 GMT
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 << Content-Length: 116
22/08/12 15:38:28 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:28 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:28 GMT
22/08/12 15:38:28 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$DeleteObjectsHandler
22/08/12 15:38:28 DEBUG PoolingHttpClientConnectionManager: Connection [id: 7][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:28 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 0
22/08/12 15:38:28 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 7][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:28 DEBUG request: Received successful response: 200, AWS Request ID: 010a54e2-28e6-4e94-aadc-0ec833434094
22/08/12 15:38:28 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:28 DEBUG requestId: AWS Request ID: 010a54e2-28e6-4e94-aadc-0ec833434094
22/08/12 15:38:28 DEBUG requestId: AWS Extended Request ID: 010a54e2-28e6-4e94-aadc-0ec833434094
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 46
22/08/12 15:38:28 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[DeleteObjectsRequest], AWSRequestID=[010a54e2-28e6-4e94-aadc-0ec833434094], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=15, RequestCount=1, HttpClientPoolLeasedCount=1, ResponseProcessingTime=[1.543], ClientExecuteTime=[101.029], HttpClientSendRequestTime=[0.847], HttpRequestTime=[97.835], ApiCallLatency=[100.648], RequestSigningTime=[0.44], CredentialsRequestTime=[0.018, 0.022], HttpClientReceiveResponseTime=[96.101],
22/08/12 15:38:28 DEBUG S3AFileSystem: DELETE 5 keys: duration 0:00.102s
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter fake_directories_deleted by 5 with final value 12
22/08/12 15:38:28 DEBUG S3AFileSystem: Deleting 5 keys: duration 0:00.103s
22/08/12 15:38:28 DEBUG CallableSupplier: Waiting for task completion: duration 0:00.103s
22/08/12 15:38:28 DEBUG Invoker: Writing Object: duration 0:02.138s
22/08/12 15:38:28 DEBUG S3ADataBlocks: FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8794108122613463043.tmp, state=Upload, dataSize=365, limit=67108864}: entering state Closed
22/08/12 15:38:28 DEBUG S3ADataBlocks: Closed FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8794108122613463043.tmp, state=Closed, dataSize=365, limit=67108864}
22/08/12 15:38:28 DEBUG S3ADataBlocks: Closing FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8794108122613463043.tmp, state=Closed, dataSize=365, limit=67108864}
22/08/12 15:38:28 DEBUG S3ADataBlocks: block[1]: closeBlock()
22/08/12 15:38:28 DEBUG S3ABlockOutputStream: Upload complete to nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet by WriteOperationHelper {bucket=master.rando2}
22/08/12 15:38:28 DEBUG S3ABlockOutputStream: Statistics: OutputStreamStatistics{counters=((stream_write_total_time=0) (stream_write_queue_duration=0) (op_hsync=0) (op_abort=0) (object_multipart_aborted=0) (stream_write_exceptions=0) (stream_write_bytes=365) (action_executor_acquired.failures=0) (multipart_upload_completed=0) (stream_write_exceptions_completing_upload=0) (object_multipart_aborted.failures=0) (op_abort.failures=0) (multipart_upload_completed.failures=0) (op_hflush=0) (stream_write_total_data=365) (stream_write_block_uploads=1) (action_executor_acquired=0));
gauges=((stream_write_block_uploads_data_pending=0) (stream_write_block_uploads_pending=1));
minimums=((op_abort.failures.min=-1) (multipart_upload_completed.failures.min=-1) (multipart_upload_completed.min=-1) (action_executor_acquired.failures.min=-1) (object_multipart_aborted.min=-1) (op_abort.min=-1) (object_multipart_aborted.failures.min=-1) (action_executor_acquired.min=-1));
maximums=((object_multipart_aborted.failures.max=-1) (op_abort.failures.max=-1) (multipart_upload_completed.failures.max=-1) (action_executor_acquired.failures.max=-1) (multipart_upload_completed.max=-1) (op_abort.max=-1) (action_executor_acquired.max=-1) (object_multipart_aborted.max=-1));
means=((action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.mean=(samples=0, sum=0, mean=0.0000)));
, blocksActive=0, blockUploadsCompleted=0, blocksAllocated=1, blocksReleased=1, blocksActivelyAllocated=0, transferDuration=0 ms, totalUploadDuration=0 ms, effectiveBandwidth=0.0 bytes/s}
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter stream_write_total_data by 365 with final value 1801
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter op_get_file_status by 1 with final value 2
22/08/12 15:38:28 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet); needEmptyDirectory=false
22/08/12 15:38:28 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 22
22/08/12 15:38:28 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 22
22/08/12 15:38:28 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet with change tracker null
22/08/12 15:38:28 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet Headers: (amz-sdk-invocation-id: a58c11a6-3eba-1c50-b0e2-0403e11d2343, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:28 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
amz-sdk-invocation-id:a58c11a6-3eba-1c50-b0e2-0403e11d2343
amz-sdk-request:ttl=20220812T144148Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143828Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:28 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143828Z
20220812/us-east-1/s3/aws4_request
4e6509a85e28e91e5f03008b0a575060333d3a09c0c75066d2acdd7c08157e8f"
22/08/12 15:38:28 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:28 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:28 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:28 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 7][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:28 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:28 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:28 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:28 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> Host: localhost:30600
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> amz-sdk-invocation-id: a58c11a6-3eba-1c50-b0e2-0403e11d2343
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> amz-sdk-request: ttl=20220812T144148Z;attempt=1;max=21
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=d740e5f9e8fcd9d19947cd646426ffd0be8e31ea4d87d94d02e6780de422beb7
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> Content-Type: application/octet-stream
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> X-Amz-Date: 20220812T143828Z
22/08/12 15:38:28 DEBUG headers: http-outgoing-7 >> Connection: Keep-Alive
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet HTTP/1.1[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "amz-sdk-invocation-id: a58c11a6-3eba-1c50-b0e2-0403e11d2343[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "amz-sdk-request: ttl=20220812T144148Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=d740e5f9e8fcd9d19947cd646426ffd0be8e31ea4d87d94d02e6780de422beb7[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "X-Amz-Date: 20220812T143828Z[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:28 DEBUG wire: http-outgoing-7 >> "[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: 0efe8499-c32e-4d1f-92a1-5386a92932bd[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: 0efe8499-c32e-4d1f-92a1-5386a92932bd[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:29 GMT[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 << "Content-Length: 389[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 << HTTP/1.1 404 Not Found
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: 0efe8499-c32e-4d1f-92a1-5386a92932bd
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: 0efe8499-c32e-4d1f-92a1-5386a92932bd
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:29 GMT
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 << Content-Length: 389
22/08/12 15:38:29 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:29 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:29 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:29 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:29 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:29 GMT
22/08/12 15:38:29 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 0efe8499-c32e-4d1f-92a1-5386a92932bd; S3 Extended Request ID: 0efe8499-c32e-4d1f-92a1-5386a92932bd; Proxy: null), S3 Extended Request ID: 0efe8499-c32e-4d1f-92a1-5386a92932bd
22/08/12 15:38:29 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 47
22/08/12 15:38:29 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 0efe8499-c32e-4d1f-92a1-5386a92932bd; S3 Extended Request ID: 0efe8499-c32e-4d1f-92a1-5386a92932bd; Proxy: null), S3 Extended Request ID: 0efe8499-c32e-4d1f-92a1-5386a92932bd], RequestType=[GetObjectMetadataRequest], AWSRequestID=[0efe8499-c32e-4d1f-92a1-5386a92932bd], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=15, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=1, ClientExecuteTime=[2316.224], HttpClientSendRequestTime=[0.326], HttpRequestTime=[2315.049], ApiCallLatency=[2315.975], RequestSigningTime=[0.198], CredentialsRequestTime=[0.007, 0.002], HttpClientReceiveResponseTime=[2313.91],
22/08/12 15:38:29 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:29 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:29 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 21
22/08/12 15:38:29 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: ab61e9e8-8454-1c1d-a90d-86e184807087, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:29 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2Fpart-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F
amz-sdk-invocation-id:ab61e9e8-8454-1c1d-a90d-86e184807087
amz-sdk-request:ttl=20220812T144149Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143829Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:29 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143829Z
20220812/us-east-1/s3/aws4_request
462e192263ea62afd853c86618d63f474768526cfe24623a3c9fb2df961d9d69"
22/08/12 15:38:29 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:29 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:29 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:29 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:29 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:29 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:29 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2Fpart-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:29 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2Fpart-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: ab61e9e8-8454-1c1d-a90d-86e184807087
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144149Z;attempt=1;max=21
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=82c1551932c3c2fc8b31545c170038223f0ef09e109ef24d124f4fd1a9492e9f
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143829Z
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> Content-Length: 0
22/08/12 15:38:29 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2Fpart-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: ab61e9e8-8454-1c1d-a90d-86e184807087[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144149Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=82c1551932c3c2fc8b31545c170038223f0ef09e109ef24d124f4fd1a9492e9f[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143829Z[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:29 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:29 DEBUG ExecutorMetricsPoller: removing (1, 0) from stageTCMP
22/08/12 15:38:29 DEBUG ExecutorMetricsPoller: removing (2, 0) from stageTCMP
22/08/12 15:38:29 DEBUG ExecutorMetricsPoller: removing (0, 0) from stageTCMP
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: 0e0ee46a-74a6-4b52-87de-0974ca7fa557[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: 0e0ee46a-74a6-4b52-87de-0974ca7fa557[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Length: 427[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/</Prefix></ListBucketResult>"
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << HTTP/1.1 200 OK
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: 0e0ee46a-74a6-4b52-87de-0974ca7fa557
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: 0e0ee46a-74a6-4b52-87de-0974ca7fa557
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Length: 427
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:30 DEBUG request: Received successful response: 200, AWS Request ID: 0e0ee46a-74a6-4b52-87de-0974ca7fa557
22/08/12 15:38:30 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:30 DEBUG requestId: AWS Request ID: 0e0ee46a-74a6-4b52-87de-0974ca7fa557
22/08/12 15:38:30 DEBUG requestId: AWS Extended Request ID: 0e0ee46a-74a6-4b52-87de-0974ca7fa557
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 48
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[0e0ee46a-74a6-4b52-87de-0974ca7fa557], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=15, RequestCount=1, HttpClientPoolLeasedCount=1, ResponseProcessingTime=[2.387], ClientExecuteTime=[772.575], HttpClientSendRequestTime=[0.645], HttpRequestTime=[767.863], ApiCallLatency=[772.241], RequestSigningTime=[0.677], CredentialsRequestTime=[0.011, 0.007], HttpClientReceiveResponseTime=[766.068],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST: duration 0:00.773s
22/08/12 15:38:30 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter op_get_file_status.failures by 1 with final value 1
22/08/12 15:38:30 DEBUG BasicWriteTaskStatsTracker: File s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet is not yet visible
java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/part-00015-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$24(S3AFileSystem.java:3556)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3554)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.getFileSize(BasicWriteStatsTracker.scala:89)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.getFileSize(BasicWriteStatsTracker.scala:73)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.updateFileStats(BasicWriteStatsTracker.scala:151)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.closeFile(BasicWriteStatsTracker.scala:146)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.$anonfun$releaseCurrentWriter$1(FileFormatDataWriter.scala:65)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.$anonfun$releaseCurrentWriter$1$adapted(FileFormatDataWriter.scala:65)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseCurrentWriter(FileFormatDataWriter.scala:65)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:75)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:105)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:332)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:338)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$21(FileFormatWriter.scala:256)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 16
22/08/12 15:38:30 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31); needEmptyDirectory=false
22/08/12 15:38:30 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 23
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 23
22/08/12 15:38:30 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31 with change tracker null
22/08/12 15:38:30 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31 Headers: (amz-sdk-invocation-id: ba6bf310-6e28-a26a-819f-5591f34c5d6b, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "X-Amz-Id-2: 183c8f45-36a2-4a42-9db2-ababa8f0d9c8[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "X-Amz-Request-Id: 183c8f45-36a2-4a42-9db2-ababa8f0d9c8[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "Content-Length: 389[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "[\r][\n]"
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << HTTP/1.1 404 Not Found
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << X-Amz-Id-2: 183c8f45-36a2-4a42-9db2-ababa8f0d9c8
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31
amz-sdk-invocation-id:ba6bf310-6e28-a26a-819f-5591f34c5d6b
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << X-Amz-Request-Id: 183c8f45-36a2-4a42-9db2-ababa8f0d9c8
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
220d8e872cdbcfe2e5a136741f588e331d4c6fd4af66d6cea22c53cb17c5cc61"
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << Content-Length: 389
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 7][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 7][route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 7][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:30 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 183c8f45-36a2-4a42-9db2-ababa8f0d9c8; S3 Extended Request ID: 183c8f45-36a2-4a42-9db2-ababa8f0d9c8; Proxy: null), S3 Extended Request ID: 183c8f45-36a2-4a42-9db2-ababa8f0d9c8
22/08/12 15:38:30 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31 HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31 HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> amz-sdk-invocation-id: ba6bf310-6e28-a26a-819f-5591f34c5d6b
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 49
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bdd21388c4a5d7b27211de5e9f54903d6c479f208b729e910d342f7fbf92170b
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 183c8f45-36a2-4a42-9db2-ababa8f0d9c8; S3 Extended Request ID: 183c8f45-36a2-4a42-9db2-ababa8f0d9c8; Proxy: null), S3 Extended Request ID: 183c8f45-36a2-4a42-9db2-ababa8f0d9c8], RequestType=[GetObjectMetadataRequest], AWSRequestID=[183c8f45-36a2-4a42-9db2-ababa8f0d9c8], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=15, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=1, ClientExecuteTime=[2017.819], HttpClientSendRequestTime=[0.403], HttpRequestTime=[2016.666], ApiCallLatency=[2017.612], RequestSigningTime=[0.199], CredentialsRequestTime=[0.006, 0.002], HttpClientReceiveResponseTime=[2015.367],
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31 HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "amz-sdk-invocation-id: ba6bf310-6e28-a26a-819f-5591f34c5d6b[\r][\n]"
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=bdd21388c4a5d7b27211de5e9f54903d6c479f208b729e910d342f7fbf92170b[\r][\n]"
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 22
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 40925adb-029a-13f1-6d99-2ae0ecbb01a2, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2Fpart-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F
amz-sdk-invocation-id:40925adb-029a-13f1-6d99-2ae0ecbb01a2
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
c369558b44e4703a7f1f09ae8e2d4c96869ac88332b8928803d33f6ea00ef988"
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2Fpart-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2Fpart-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: 40925adb-029a-13f1-6d99-2ae0ecbb01a2
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=24f417d08fd2024309fd70456f8333804e30f2b1597f38553aeddaac575c97d3
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Length: 0
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2Fpart-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: 40925adb-029a-13f1-6d99-2ae0ecbb01a2[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=24f417d08fd2024309fd70456f8333804e30f2b1597f38553aeddaac575c97d3[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "X-Amz-Id-2: 46a40feb-4d8a-4373-8044-d133fef33706[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "X-Amz-Request-Id: 46a40feb-4d8a-4373-8044-d133fef33706[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "[\r][\n]"
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << HTTP/1.1 404 Not Found
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << X-Amz-Id-2: 46a40feb-4d8a-4373-8044-d133fef33706
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << X-Amz-Request-Id: 46a40feb-4d8a-4373-8044-d133fef33706
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << Content-Length: 321
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 7][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 7][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 46a40feb-4d8a-4373-8044-d133fef33706; S3 Extended Request ID: 46a40feb-4d8a-4373-8044-d133fef33706; Proxy: null), S3 Extended Request ID: 46a40feb-4d8a-4373-8044-d133fef33706
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 50
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 46a40feb-4d8a-4373-8044-d133fef33706; S3 Extended Request ID: 46a40feb-4d8a-4373-8044-d133fef33706; Proxy: null), S3 Extended Request ID: 46a40feb-4d8a-4373-8044-d133fef33706], RequestType=[GetObjectMetadataRequest], AWSRequestID=[46a40feb-4d8a-4373-8044-d133fef33706], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=15, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=1, ClientExecuteTime=[18.954], HttpClientSendRequestTime=[0.789], HttpRequestTime=[17.526], ApiCallLatency=[18.686], RequestSigningTime=[0.364], CredentialsRequestTime=[0.01, 0.004], HttpClientReceiveResponseTime=[15.647],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:30 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 23
22/08/12 15:38:30 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 75d04955-3aa4-b63e-21e4-7a362363d735, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2F
amz-sdk-invocation-id:75d04955-3aa4-b63e-21e4-7a362363d735
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
73a5b01ec54ce6d8ba62c6fe2310d63d4606aa5b4503ba42b54cdf60e840ce04"
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 7][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 200000
22/08/12 15:38:30 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> amz-sdk-invocation-id: 75d04955-3aa4-b63e-21e4-7a362363d735
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=5129e19c267eea00c3d986e6fee35dc120efac859bde90765bfd888dd31e38d5
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> Content-Length: 0
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538247838847636316877994_0003_m_000015_31%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "amz-sdk-invocation-id: 75d04955-3aa4-b63e-21e4-7a362363d735[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=5129e19c267eea00c3d986e6fee35dc120efac859bde90765bfd888dd31e38d5[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: 26371953-d84a-4cf1-abf4-5c5c26a1879b[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: 26371953-d84a-4cf1-abf4-5c5c26a1879b[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Length: 427[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet/</Prefix></ListBucketResult>"
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << HTTP/1.1 200 OK
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: 26371953-d84a-4cf1-abf4-5c5c26a1879b
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: 26371953-d84a-4cf1-abf4-5c5c26a1879b
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Length: 427
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:30 DEBUG request: Received successful response: 200, AWS Request ID: 26371953-d84a-4cf1-abf4-5c5c26a1879b
22/08/12 15:38:30 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:30 DEBUG requestId: AWS Request ID: 26371953-d84a-4cf1-abf4-5c5c26a1879b
22/08/12 15:38:30 DEBUG requestId: AWS Extended Request ID: 26371953-d84a-4cf1-abf4-5c5c26a1879b
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 51
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[26371953-d84a-4cf1-abf4-5c5c26a1879b], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=15, RequestCount=1, HttpClientPoolLeasedCount=1, ResponseProcessingTime=[1.645], ClientExecuteTime=[25.163], HttpClientSendRequestTime=[0.891], HttpRequestTime=[21.862], ApiCallLatency=[24.912], RequestSigningTime=[0.457], CredentialsRequestTime=[0.008, 0.005], HttpClientReceiveResponseTime=[19.895],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST: duration 0:00.026s
22/08/12 15:38:30 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter op_get_file_status.failures by 1 with final value 2
22/08/12 15:38:30 DEBUG BasicWriteTaskStatsTracker: File s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet is not yet visible
java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/part-00000-abc4d155-280f-4c55-9237-3ebebea73a2f-c000.snappy.parquet
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$24(S3AFileSystem.java:3556)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3554)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.getFileSize(BasicWriteStatsTracker.scala:89)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.getFileSize(BasicWriteStatsTracker.scala:73)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.updateFileStats(BasicWriteStatsTracker.scala:151)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.closeFile(BasicWriteStatsTracker.scala:146)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.$anonfun$releaseCurrentWriter$1(FileFormatDataWriter.scala:65)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.$anonfun$releaseCurrentWriter$1$adapted(FileFormatDataWriter.scala:65)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseCurrentWriter(FileFormatDataWriter.scala:65)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:75)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:105)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:332)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:338)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$21(FileFormatWriter.scala:256)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 17
22/08/12 15:38:30 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16 (nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16); needEmptyDirectory=false
22/08/12 15:38:30 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 24
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 24
22/08/12 15:38:30 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16 with change tracker null
22/08/12 15:38:30 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16 Headers: (amz-sdk-invocation-id: 2d798f7f-eea1-6380-3139-78d1739dc473, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16
amz-sdk-invocation-id:2d798f7f-eea1-6380-3139-78d1739dc473
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
2a2750ea1a0c5a9beb3b41134e90821e59cbd9728e88b31f06aa14c25aec482a"
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 14; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16 HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16 HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: 2d798f7f-eea1-6380-3139-78d1739dc473
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=aeb9f94ad8f16d0b6a414d313d25fc767921d53d4ca2f7c7e13411dd891f681a
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16 HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: 2d798f7f-eea1-6380-3139-78d1739dc473[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=aeb9f94ad8f16d0b6a414d313d25fc767921d53d4ca2f7c7e13411dd891f681a[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "X-Amz-Id-2: 06019c1d-8fe4-414c-8eb8-ac167ebb83c9[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "X-Amz-Request-Id: 06019c1d-8fe4-414c-8eb8-ac167ebb83c9[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-7 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31/</Prefix></ListBucketResult>"
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << HTTP/1.1 200 OK
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << X-Amz-Id-2: 06019c1d-8fe4-414c-8eb8-ac167ebb83c9
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << X-Amz-Request-Id: 06019c1d-8fe4-414c-8eb8-ac167ebb83c9
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-7 << Content-Length: 359
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 7][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-7: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 7][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:30 DEBUG request: Received successful response: 200, AWS Request ID: 06019c1d-8fe4-414c-8eb8-ac167ebb83c9
22/08/12 15:38:30 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:30 DEBUG requestId: AWS Request ID: 06019c1d-8fe4-414c-8eb8-ac167ebb83c9
22/08/12 15:38:30 DEBUG requestId: AWS Extended Request ID: 06019c1d-8fe4-414c-8eb8-ac167ebb83c9
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 52
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[06019c1d-8fe4-414c-8eb8-ac167ebb83c9], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=15, RequestCount=1, HttpClientPoolLeasedCount=1, ResponseProcessingTime=[2.513], ClientExecuteTime=[22.907], HttpClientSendRequestTime=[0.658], HttpRequestTime=[18.711], ApiCallLatency=[22.679], RequestSigningTime=[0.408], CredentialsRequestTime=[0.007, 0.005], HttpClientReceiveResponseTime=[17.16],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST: duration 0:00.024s
22/08/12 15:38:30 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538247838847636316877994_0003_m_000015_31
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 16
22/08/12 15:38:30 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538247838847636316877994_0003_m_000015_31
22/08/12 15:38:30 WARN BasicWriteTaskStatsTracker: Expected 1 files, but only saw 0. This could be due to the output format not writing empty files, or files being not immediately visible in the filesystem.
22/08/12 15:38:30 INFO Executor: Finished task 15.0 in stage 3.0 (TID 31). 2828 bytes result sent to driver
22/08/12 15:38:30 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 1
22/08/12 15:38:30 INFO TaskSetManager: Finished task 15.0 in stage 3.0 (TID 31) in 5392 ms on 10.1.255.235 (executor driver) (15/16)
22/08/12 15:38:30 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@3b9ce101)
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: e7a466d0-d31e-4f29-8aac-f48c42fa76b7[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: e7a466d0-d31e-4f29-8aac-f48c42fa76b7[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Length: 321[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << HTTP/1.1 404 Not Found
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: e7a466d0-d31e-4f29-8aac-f48c42fa76b7
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: e7a466d0-d31e-4f29-8aac-f48c42fa76b7
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Length: 321
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: e7a466d0-d31e-4f29-8aac-f48c42fa76b7; S3 Extended Request ID: e7a466d0-d31e-4f29-8aac-f48c42fa76b7; Proxy: null), S3 Extended Request ID: e7a466d0-d31e-4f29-8aac-f48c42fa76b7
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 53
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: e7a466d0-d31e-4f29-8aac-f48c42fa76b7; S3 Extended Request ID: e7a466d0-d31e-4f29-8aac-f48c42fa76b7; Proxy: null), S3 Extended Request ID: e7a466d0-d31e-4f29-8aac-f48c42fa76b7], RequestType=[GetObjectMetadataRequest], AWSRequestID=[e7a466d0-d31e-4f29-8aac-f48c42fa76b7], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=15, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=1, ClientExecuteTime=[20.426], HttpClientSendRequestTime=[0.395], HttpRequestTime=[19.201], ApiCallLatency=[20.222], RequestSigningTime=[0.247], CredentialsRequestTime=[0.007, 0.002], HttpClientReceiveResponseTime=[17.931],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:30 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 24
22/08/12 15:38:30 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: d7da1c30-445f-27a4-1696-f75703a35126, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2F
amz-sdk-invocation-id:d7da1c30-445f-27a4-1696-f75703a35126
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
b81f6cfaa9ec2ae45fe7df86b012b37e244b1cd31046c04e7827f8fbf402a4ef"
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: d7da1c30-445f-27a4-1696-f75703a35126
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=60600446ffb16c7cda6ddb11867e7fb1b2ab60543dd90268ff49fb90d321a57e
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Length: 0
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F_temporary%2Fattempt_202208121538248384345865724627642_0003_m_000000_16%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: d7da1c30-445f-27a4-1696-f75703a35126[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=60600446ffb16c7cda6ddb11867e7fb1b2ab60543dd90268ff49fb90d321a57e[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: e1c06836-3d12-4457-8f25-57d60b9d075c[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: e1c06836-3d12-4457-8f25-57d60b9d075c[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Length: 359[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16/</Prefix></ListBucketResult>"
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << HTTP/1.1 200 OK
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: e1c06836-3d12-4457-8f25-57d60b9d075c
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: e1c06836-3d12-4457-8f25-57d60b9d075c
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Length: 359
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:30 DEBUG request: Received successful response: 200, AWS Request ID: e1c06836-3d12-4457-8f25-57d60b9d075c
22/08/12 15:38:30 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:30 DEBUG requestId: AWS Request ID: e1c06836-3d12-4457-8f25-57d60b9d075c
22/08/12 15:38:30 DEBUG requestId: AWS Extended Request ID: e1c06836-3d12-4457-8f25-57d60b9d075c
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 54
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[e1c06836-3d12-4457-8f25-57d60b9d075c], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=16, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.759], ClientExecuteTime=[22.053], HttpClientSendRequestTime=[0.636], HttpRequestTime=[19.001], ApiCallLatency=[21.83], RequestSigningTime=[0.381], CredentialsRequestTime=[0.007, 0.004], HttpClientReceiveResponseTime=[17.616],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST: duration 0:00.023s
22/08/12 15:38:30 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0/_temporary/attempt_202208121538248384345865724627642_0003_m_000000_16
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 17
22/08/12 15:38:30 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121538248384345865724627642_0003_m_000000_16
22/08/12 15:38:30 WARN BasicWriteTaskStatsTracker: Expected 1 files, but only saw 0. This could be due to the output format not writing empty files, or files being not immediately visible in the filesystem.
22/08/12 15:38:30 INFO Executor: Finished task 0.0 in stage 3.0 (TID 16). 2785 bytes result sent to driver
22/08/12 15:38:30 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 0
22/08/12 15:38:30 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 16) in 5430 ms on 10.1.255.235 (executor driver) (16/16)
22/08/12 15:38:30 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
22/08/12 15:38:30 INFO DAGScheduler: ResultStage 3 (parquet at NativeMethodAccessorImpl.java:0) finished in 5.457 s
22/08/12 15:38:30 DEBUG DAGScheduler: After removal of stage 3, remaining stages = 0
22/08/12 15:38:30 INFO DAGScheduler: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/12 15:38:30 INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage finished
22/08/12 15:38:30 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@21eb1771)
22/08/12 15:38:30 INFO DAGScheduler: Job 3 finished: parquet at NativeMethodAccessorImpl.java:0, took 5.462559 s
22/08/12 15:38:30 INFO FileFormatWriter: Start to commit write Job 6a90b306-b8f5-41c1-9d5c-67e4cd7ef87c.
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter op_list_status by 1 with final value 1
22/08/12 15:38:30 DEBUG Invoker: Starting: listStatus
22/08/12 15:38:30 DEBUG S3AFileSystem: List status for path: s3a://master.rando2/nonemptyprefix4/_temporary/0
22/08/12 15:38:30 DEBUG S3AFileSystem: listStatus: doing listObjects for directory nonemptyprefix4/_temporary/0/
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/ delimiter=/ keys=5000 requester pays=false
22/08/12 15:38:30 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 25
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 1
22/08/12 15:38:30 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["5000"],"prefix":["nonemptyprefix4/_temporary/0/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: dd5d5b36-cb91-1a9f-1f9f-cf8d26a87b07, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=5000&prefix=nonemptyprefix4%2F_temporary%2F0%2F
amz-sdk-invocation-id:dd5d5b36-cb91-1a9f-1f9f-cf8d26a87b07
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
f5e92e4ad71696f6b3b057a9223645fd4fcf93703ca4bc0d07e77783bbf2669f"
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=5000&prefix=nonemptyprefix4%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=5000&prefix=nonemptyprefix4%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: dd5d5b36-cb91-1a9f-1f9f-cf8d26a87b07
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=f4628559474000256ff22d003274c13b3e0f2975904cf72fcc86e9074890a7dd
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Length: 0
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=5000&prefix=nonemptyprefix4%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: dd5d5b36-cb91-1a9f-1f9f-cf8d26a87b07[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=f4628559474000256ff22d003274c13b3e0f2975904cf72fcc86e9074890a7dd[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: 2d847202-2d95-4789-94d0-89698cc099ca[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: 2d847202-2d95-4789-94d0-89698cc099ca[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Length: 292[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>5000</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/</Prefix></ListBucketResult>"
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << HTTP/1.1 200 OK
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: 2d847202-2d95-4789-94d0-89698cc099ca
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: 2d847202-2d95-4789-94d0-89698cc099ca
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Length: 292
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:30 DEBUG request: Received successful response: 200, AWS Request ID: 2d847202-2d95-4789-94d0-89698cc099ca
22/08/12 15:38:30 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:30 DEBUG requestId: AWS Request ID: 2d847202-2d95-4789-94d0-89698cc099ca
22/08/12 15:38:30 DEBUG requestId: AWS Extended Request ID: 2d847202-2d95-4789-94d0-89698cc099ca
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 55
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[2d847202-2d95-4789-94d0-89698cc099ca], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=16, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.859], ClientExecuteTime=[17.598], HttpClientSendRequestTime=[0.609], HttpRequestTime=[14.297], ApiCallLatency=[17.369], RequestSigningTime=[0.435], CredentialsRequestTime=[0.008, 0.004], HttpClientReceiveResponseTime=[12.889],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST: duration 0:00.018s
22/08/12 15:38:30 DEBUG S3AFileSystem: Added 0 entries; ignored 0; hasNext=false; hasMoreObjects=false
22/08/12 15:38:30 DEBUG S3AFileSystem: All entries in batch were filtered...continuing
22/08/12 15:38:30 DEBUG S3AFileSystem: Start iterating the provided status.
22/08/12 15:38:30 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary/0 (nonemptyprefix4/_temporary/0); needEmptyDirectory=false
22/08/12 15:38:30 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary/0
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 25
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 25
22/08/12 15:38:30 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary/0 with change tracker null
22/08/12 15:38:30 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary/0 Headers: (amz-sdk-invocation-id: 5ea9e2c1-b78f-0c0b-a23a-ea63fb14dd72, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary/0
amz-sdk-invocation-id:5ea9e2c1-b78f-0c0b-a23a-ea63fb14dd72
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
69c5f121d13395d40b47875a0f7fd39fe23006e1b5f24cd99e8ae3e6bc6cda74"
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary/0 HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> HEAD /master.rando2/nonemptyprefix4/_temporary/0 HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: 5ea9e2c1-b78f-0c0b-a23a-ea63fb14dd72
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2642d290404f5d4c0bf214a10cfc9ada769a2d37e135e9a2ee3aa2ad278f8977
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "HEAD /master.rando2/nonemptyprefix4/_temporary/0 HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: 5ea9e2c1-b78f-0c0b-a23a-ea63fb14dd72[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2642d290404f5d4c0bf214a10cfc9ada769a2d37e135e9a2ee3aa2ad278f8977[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: c2acf51e-5592-4f00-8d71-66e0feecac23[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: c2acf51e-5592-4f00-8d71-66e0feecac23[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Length: 251[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << HTTP/1.1 404 Not Found
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: c2acf51e-5592-4f00-8d71-66e0feecac23
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: c2acf51e-5592-4f00-8d71-66e0feecac23
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Length: 251
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: c2acf51e-5592-4f00-8d71-66e0feecac23; S3 Extended Request ID: c2acf51e-5592-4f00-8d71-66e0feecac23; Proxy: null), S3 Extended Request ID: c2acf51e-5592-4f00-8d71-66e0feecac23
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 56
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: c2acf51e-5592-4f00-8d71-66e0feecac23; S3 Extended Request ID: c2acf51e-5592-4f00-8d71-66e0feecac23; Proxy: null), S3 Extended Request ID: c2acf51e-5592-4f00-8d71-66e0feecac23], RequestType=[GetObjectMetadataRequest], AWSRequestID=[c2acf51e-5592-4f00-8d71-66e0feecac23], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=16, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[17.408], HttpClientSendRequestTime=[0.528], HttpRequestTime=[16.173], ApiCallLatency=[17.166], RequestSigningTime=[0.345], CredentialsRequestTime=[0.008, 0.004], HttpClientReceiveResponseTime=[14.87],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/0/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:30 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 26
22/08/12 15:38:30 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/0/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 85001302-7e3c-d3f5-d31f-2235000c205d, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F
amz-sdk-invocation-id:85001302-7e3c-d3f5-d31f-2235000c205d
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
11a402fe8bbdfa848c8aab1b1f71fabb58461bf4112ae622ab70b9773bfec64a"
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: 85001302-7e3c-d3f5-d31f-2235000c205d
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=a9d3217f85ba88e4500dea6721b8c9f3578b36b0882796c97e710a4248d0f769
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Length: 0
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: 85001302-7e3c-d3f5-d31f-2235000c205d[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=a9d3217f85ba88e4500dea6721b8c9f3578b36b0882796c97e710a4248d0f769[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: 93c95388-f49b-4e34-bc67-b71971923250[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: 93c95388-f49b-4e34-bc67-b71971923250[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Length: 289[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/0/</Prefix></ListBucketResult>"
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << HTTP/1.1 200 OK
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: 93c95388-f49b-4e34-bc67-b71971923250
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: 93c95388-f49b-4e34-bc67-b71971923250
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Length: 289
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:30 DEBUG request: Received successful response: 200, AWS Request ID: 93c95388-f49b-4e34-bc67-b71971923250
22/08/12 15:38:30 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:30 DEBUG requestId: AWS Request ID: 93c95388-f49b-4e34-bc67-b71971923250
22/08/12 15:38:30 DEBUG requestId: AWS Extended Request ID: 93c95388-f49b-4e34-bc67-b71971923250
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 57
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[93c95388-f49b-4e34-bc67-b71971923250], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=16, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.565], ClientExecuteTime=[18.24], HttpClientSendRequestTime=[0.399], HttpRequestTime=[15.676], ApiCallLatency=[18.041], RequestSigningTime=[0.258], CredentialsRequestTime=[0.006, 0.002], HttpClientReceiveResponseTime=[14.718],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST: duration 0:00.018s
22/08/12 15:38:30 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary/0
22/08/12 15:38:30 DEBUG Invoker: listStatus: duration 0:00.064s
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter op_list_status.failures by 1 with final value 1
22/08/12 15:38:30 ERROR FileFormatWriter: Aborting job 6a90b306-b8f5-41c1-9d5c-67e4cd7ef87c.
java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix4/_temporary/0
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:3300)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$null$20(S3AFileSystem.java:3264)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$21(S3AFileSystem.java:3263)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3262)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2014)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:334)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:404)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:377)
at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:48)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:192)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$25(FileFormatWriter.scala:267)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:642)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:267)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:186)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:793)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
22/08/12 15:38:30 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/_temporary (nonemptyprefix4/_temporary); needEmptyDirectory=true
22/08/12 15:38:30 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/_temporary
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 26
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 26
22/08/12 15:38:30 DEBUG S3AFileSystem: HEAD nonemptyprefix4/_temporary with change tracker null
22/08/12 15:38:30 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/_temporary Headers: (amz-sdk-invocation-id: a1f08c0d-acd5-5cf4-6dc4-854c2299e286, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/_temporary
amz-sdk-invocation-id:a1f08c0d-acd5-5cf4-6dc4-854c2299e286
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
4bd6915955a9a8d3cabb2b1c580ee130c52784d3614ff33779204c2e470910b7"
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/_temporary HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> HEAD /master.rando2/nonemptyprefix4/_temporary HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: a1f08c0d-acd5-5cf4-6dc4-854c2299e286
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=993263f0227af04073f4dd6e03e87848183d28167ee83b817430ec22baf94cca
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "HEAD /master.rando2/nonemptyprefix4/_temporary HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: a1f08c0d-acd5-5cf4-6dc4-854c2299e286[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=993263f0227af04073f4dd6e03e87848183d28167ee83b817430ec22baf94cca[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: d26981c0-4815-4563-aabd-c048ea4e3da7[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: d26981c0-4815-4563-aabd-c048ea4e3da7[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Length: 249[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << HTTP/1.1 404 Not Found
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: d26981c0-4815-4563-aabd-c048ea4e3da7
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: d26981c0-4815-4563-aabd-c048ea4e3da7
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Length: 249
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: d26981c0-4815-4563-aabd-c048ea4e3da7; S3 Extended Request ID: d26981c0-4815-4563-aabd-c048ea4e3da7; Proxy: null), S3 Extended Request ID: d26981c0-4815-4563-aabd-c048ea4e3da7
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 58
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: d26981c0-4815-4563-aabd-c048ea4e3da7; S3 Extended Request ID: d26981c0-4815-4563-aabd-c048ea4e3da7; Proxy: null), S3 Extended Request ID: d26981c0-4815-4563-aabd-c048ea4e3da7], RequestType=[GetObjectMetadataRequest], AWSRequestID=[d26981c0-4815-4563-aabd-c048ea4e3da7], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=16, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[19.242], HttpClientSendRequestTime=[0.287], HttpRequestTime=[18.255], ApiCallLatency=[19.005], RequestSigningTime=[0.195], CredentialsRequestTime=[0.011, 0.002], HttpClientReceiveResponseTime=[17.282],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix4/_temporary/ delimiter=/ keys=2 requester pays=false
22/08/12 15:38:30 DEBUG S3AFileSystem: Starting: LIST
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 27
22/08/12 15:38:30 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix4/_temporary/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 3f699e07-6993-bb7e-2dc8-bcaa567592eb, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F
amz-sdk-invocation-id:3f699e07-6993-bb7e-2dc8-bcaa567592eb
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
6923920a67a453fd032bfd475ff8c0a13065d13285c8a88d3ae82875bd4d7639"
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F&fetch-owner=false HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: 3f699e07-6993-bb7e-2dc8-bcaa567592eb
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=3a35066576fc4a519df49d1bbeba7e76a197998024a968462b6ec3ef866d26ea
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Length: 0
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix4%2F_temporary%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: 3f699e07-6993-bb7e-2dc8-bcaa567592eb[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=3a35066576fc4a519df49d1bbeba7e76a197998024a968462b6ec3ef866d26ea[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Length: 0[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: a47601c7-5530-4139-b067-ec72b1268089[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: a47601c7-5530-4139-b067-ec72b1268089[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Length: 287[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix4/_temporary/</Prefix></ListBucketResult>"
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << HTTP/1.1 200 OK
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: a47601c7-5530-4139-b067-ec72b1268089
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: a47601c7-5530-4139-b067-ec72b1268089
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Length: 287
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 15:38:30 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 15:38:30 DEBUG request: Received successful response: 200, AWS Request ID: a47601c7-5530-4139-b067-ec72b1268089
22/08/12 15:38:30 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 15:38:30 DEBUG requestId: AWS Request ID: a47601c7-5530-4139-b067-ec72b1268089
22/08/12 15:38:30 DEBUG requestId: AWS Extended Request ID: a47601c7-5530-4139-b067-ec72b1268089
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 59
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[a47601c7-5530-4139-b067-ec72b1268089], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=16, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[2.257], ClientExecuteTime=[19.33], HttpClientSendRequestTime=[0.297], HttpRequestTime=[16.037], ApiCallLatency=[19.114], RequestSigningTime=[0.217], CredentialsRequestTime=[0.005, 0.003], HttpClientReceiveResponseTime=[15.189],
22/08/12 15:38:30 DEBUG S3AFileSystem: LIST: duration 0:00.020s
22/08/12 15:38:30 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix4/_temporary
22/08/12 15:38:30 DEBUG S3AFileSystem: Couldn't delete s3a://master.rando2/nonemptyprefix4/_temporary - does not exist: java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix4/_temporary
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter ignored_errors by 1 with final value 1
22/08/12 15:38:30 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix4/.spark-staging-abc4d155-280f-4c55-9237-3ebebea73a2f (nonemptyprefix4/.spark-staging-abc4d155-280f-4c55-9237-3ebebea73a2f); needEmptyDirectory=true
22/08/12 15:38:30 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix4/.spark-staging-abc4d155-280f-4c55-9237-3ebebea73a2f
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 27
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 27
22/08/12 15:38:30 DEBUG S3AFileSystem: HEAD nonemptyprefix4/.spark-staging-abc4d155-280f-4c55-9237-3ebebea73a2f with change tracker null
22/08/12 15:38:30 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix4/.spark-staging-abc4d155-280f-4c55-9237-3ebebea73a2f Headers: (amz-sdk-invocation-id: 2f69177b-de05-e676-e66c-9b05b4a5ed7f, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix4/.spark-staging-abc4d155-280f-4c55-9237-3ebebea73a2f
amz-sdk-invocation-id:2f69177b-de05-e676-e66c-9b05b4a5ed7f
amz-sdk-request:ttl=20220812T144150Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T143830Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 15:38:30 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T143830Z
20220812/us-east-1/s3/aws4_request
d72be09ed015d95adf5eacb214716383939a7b40a53a2e02a42ed6a87e465ec0"
22/08/12 15:38:30 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 15:38:30 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 5][route: {}->http://localhost:30600][total available: 15; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 200000
22/08/12 15:38:30 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix4/.spark-staging-abc4d155-280f-4c55-9237-3ebebea73a2f HTTP/1.1
22/08/12 15:38:30 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> HEAD /master.rando2/nonemptyprefix4/.spark-staging-abc4d155-280f-4c55-9237-3ebebea73a2f HTTP/1.1
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Host: localhost:30600
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-invocation-id: 2f69177b-de05-e676-e66c-9b05b4a5ed7f
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> amz-sdk-retry: 0/0/500
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=376425b2e625887b1e5dc6ee4aa9afc13dc1d007d0b986590f00d5b00c056b5e
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Content-Type: application/octet-stream
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> X-Amz-Date: 20220812T143830Z
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 >> Connection: Keep-Alive
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "HEAD /master.rando2/nonemptyprefix4/.spark-staging-abc4d155-280f-4c55-9237-3ebebea73a2f HTTP/1.1[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Host: localhost:30600[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-invocation-id: 2f69177b-de05-e676-e66c-9b05b4a5ed7f[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-request: ttl=20220812T144150Z;attempt=1;max=21[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=376425b2e625887b1e5dc6ee4aa9afc13dc1d007d0b986590f00d5b00c056b5e[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "X-Amz-Date: 20220812T143830Z[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 >> "[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Type: application/xml[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Id-2: 4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "X-Amz-Request-Id: 4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Date: Fri, 12 Aug 2022 14:38:30 GMT[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "Content-Length: 290[\r][\n]"
22/08/12 15:38:30 DEBUG wire: http-outgoing-5 << "[\r][\n]"
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << HTTP/1.1 404 Not Found
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Type: application/xml
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Id-2: 4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << X-Amz-Request-Id: 4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Date: Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG headers: http-outgoing-5 << Content-Length: 290
22/08/12 15:38:30 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection [id: 5][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 15:38:30 DEBUG DefaultManagedHttpClientConnection: http-outgoing-5: set socket timeout to 0
22/08/12 15:38:30 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 5][route: {}->http://localhost:30600][total available: 16; route allocated: 16 of 96; total allocated: 16 of 96]
22/08/12 15:38:30 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 14:38:30 GMT
22/08/12 15:38:30 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae; S3 Extended Request ID: 4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae; Proxy: null), S3 Extended Request ID: 4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae
22/08/12 15:38:30 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 60
22/08/12 15:38:30 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae; S3 Extended Request ID: 4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae; Proxy: null), S3 Extended Request ID: 4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae], RequestType=[GetObjectMetadataRequest], AWSRequestID=[4f1dc4c4-7bb5-42ea-b4c3-783d7ddeb4ae], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment