Skip to content

Instantly share code, notes, and snippets.

@lukemarsden

lukemarsden/gist:cb175735e1cdcab10b5b9ab1c95c7730

Created August 12, 2022 15:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lukemarsden/cb175735e1cdcab10b5b9ab1c95c7730 to your computer and use it in GitHub Desktop.
Save lukemarsden/cb175735e1cdcab10b5b9ab1c95c7730 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
22/08/12 16:24:39 WARN Utils: Your hostname, mind resolves to a loopback address: 127.0.1.1; using 10.1.255.235 instead (on interface enp6s0f0)
22/08/12 16:24:39 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
22/08/12 16:24:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/08/12 16:24:40 INFO SparkContext: Running Spark version 3.3.0
22/08/12 16:24:40 INFO ResourceUtils: ==============================================================
22/08/12 16:24:40 INFO ResourceUtils: No custom resources configured for spark.driver.
22/08/12 16:24:40 INFO ResourceUtils: ==============================================================
22/08/12 16:24:40 INFO SparkContext: Submitted application: spark.py
22/08/12 16:24:40 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/08/12 16:24:40 INFO ResourceProfile: Limiting resource is cpu
22/08/12 16:24:40 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/08/12 16:24:40 INFO SecurityManager: Changing view acls to: luke
22/08/12 16:24:40 INFO SecurityManager: Changing modify acls to: luke
22/08/12 16:24:40 INFO SecurityManager: Changing view acls groups to:
22/08/12 16:24:40 INFO SecurityManager: Changing modify acls groups to:
22/08/12 16:24:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(luke); groups with view permissions: Set(); users with modify permissions: Set(luke); groups with modify permissions: Set()
22/08/12 16:24:41 INFO Utils: Successfully started service 'sparkDriver' on port 45909.
22/08/12 16:24:41 INFO SparkEnv: Registering MapOutputTracker
22/08/12 16:24:41 INFO SparkEnv: Registering BlockManagerMaster
22/08/12 16:24:41 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/08/12 16:24:41 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/08/12 16:24:41 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
22/08/12 16:24:41 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-9de0714f-1adf-4b7d-bacc-77154eb0cf5a
22/08/12 16:24:41 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
22/08/12 16:24:41 INFO SparkEnv: Registering OutputCommitCoordinator
22/08/12 16:24:41 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/08/12 16:24:41 INFO SparkContext: Added JAR file:///home/luke/pp/pachyderm/spark/hadoop-aws-3.3.3.jar at spark://10.1.255.235:45909/jars/hadoop-aws-3.3.3.jar with timestamp 1660317880795
22/08/12 16:24:41 INFO SparkContext: Added JAR file:///home/luke/pp/pachyderm/spark/aws-java-sdk-bundle-1.12.264.jar at spark://10.1.255.235:45909/jars/aws-java-sdk-bundle-1.12.264.jar with timestamp 1660317880795
22/08/12 16:24:41 INFO Executor: Starting executor ID driver on host 10.1.255.235
22/08/12 16:24:41 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
22/08/12 16:24:41 INFO Executor: Fetching spark://10.1.255.235:45909/jars/hadoop-aws-3.3.3.jar with timestamp 1660317880795
22/08/12 16:24:41 INFO TransportClientFactory: Successfully created connection to /10.1.255.235:45909 after 32 ms (0 ms spent in bootstraps)
22/08/12 16:24:41 INFO Utils: Fetching spark://10.1.255.235:45909/jars/hadoop-aws-3.3.3.jar to /tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e/userFiles-1e00ef57-8902-4eb1-a9e5-60aee3067fb4/fetchFileTemp6439226125535153377.tmp
22/08/12 16:24:41 INFO Executor: Adding file:/tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e/userFiles-1e00ef57-8902-4eb1-a9e5-60aee3067fb4/hadoop-aws-3.3.3.jar to class loader
22/08/12 16:24:41 INFO Executor: Fetching spark://10.1.255.235:45909/jars/aws-java-sdk-bundle-1.12.264.jar with timestamp 1660317880795
22/08/12 16:24:41 INFO Utils: Fetching spark://10.1.255.235:45909/jars/aws-java-sdk-bundle-1.12.264.jar to /tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e/userFiles-1e00ef57-8902-4eb1-a9e5-60aee3067fb4/fetchFileTemp7017306744932639789.tmp
22/08/12 16:24:42 INFO Executor: Adding file:/tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e/userFiles-1e00ef57-8902-4eb1-a9e5-60aee3067fb4/aws-java-sdk-bundle-1.12.264.jar to class loader
22/08/12 16:24:42 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38343.
22/08/12 16:24:42 INFO NettyBlockTransferService: Server created on 10.1.255.235:38343
22/08/12 16:24:42 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/08/12 16:24:42 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.1.255.235, 38343, None)
22/08/12 16:24:42 INFO BlockManagerMasterEndpoint: Registering block manager 10.1.255.235:38343 with 434.4 MiB RAM, BlockManagerId(driver, 10.1.255.235, 38343, None)
22/08/12 16:24:42 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.1.255.235, 38343, None)
22/08/12 16:24:42 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.1.255.235, 38343, None)
[('spark.driver.extraJavaOptions', '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'), ('spark.hadoop.fs.s3a.connection.ssl.enabled', 'false'), ('spark.app.initial.jar.urls', 'spark://10.1.255.235:45909/jars/hadoop-aws-3.3.3.jar,spark://10.1.255.235:45909/jars/aws-java-sdk-bundle-1.12.264.jar'), ('spark.driver.host', '10.1.255.235'), ('spark.hadoop.fs.s3a.path.style.access', 'true'), ('spark.repl.local.jars', 'file:///home/luke/pp/pachyderm/spark/hadoop-aws-3.3.3.jar,file:///home/luke/pp/pachyderm/spark/aws-java-sdk-bundle-1.12.264.jar'), ('spark.app.id', 'local-1660317881500'), ('spark.executor.id', 'driver'), ('spark.hadoop.fs.s3a.change.detection.mode', 'none'), ('spark.jars', 'file:///home/luke/pp/pachyderm/spark/hadoop-aws-3.3.3.jar,file:///home/luke/pp/pachyderm/spark/aws-java-sdk-bundle-1.12.264.jar'), ('spark.hadoop.fs.s3a.impl', 'org.apache.hadoop.fs.s3a.S3AFileSystem'), ('spark.app.submitTime', '1660317880334'), ('spark.hadoop.fs.s3a.change.detection.version.required', 'false'), ('spark.app.name', 'spark.py'), ('spark.rdd.compress', 'True'), ('spark.hadoop.fs.s3a.endpoint', 'http://localhost:30600'), ('spark.executor.extraJavaOptions', '-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED'), ('spark.hadoop.fs.s3a.access.key', 'lemon'), ('spark.serializer.objectStreamReset', '100'), ('spark.master', 'local[*]'), ('spark.submit.pyFiles', ''), ('spark.driver.port', '45909'), ('spark.hadoop.fs.s3a.secret.key', 'lemon'), ('spark.submit.deployMode', 'client'), ('spark.app.startTime', '1660317880795')]
22/08/12 16:24:42 DEBUG FileSystem: Looking for FS supporting file
22/08/12 16:24:42 DEBUG FileSystem: looking for configuration option fs.file.impl
22/08/12 16:24:42 DEBUG FileSystem: Looking in service filesystems for implementation class
22/08/12 16:24:42 DEBUG FileSystem: FS for file is class org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem
22/08/12 16:24:42 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
22/08/12 16:24:42 INFO SharedState: Warehouse path is 'file:/home/luke/pp/pachyderm/spark/spark-warehouse'.
22/08/12 16:24:42 DEBUG FsUrlStreamHandlerFactory: Creating handler for protocol jar
22/08/12 16:24:42 DEBUG FileSystem: Looking for FS supporting jar
22/08/12 16:24:42 DEBUG FileSystem: looking for configuration option fs.jar.impl
22/08/12 16:24:42 DEBUG FileSystem: Looking in service filesystems for implementation class
22/08/12 16:24:42 DEBUG FsUrlStreamHandlerFactory: Unknown protocol jar, delegating to default implementation
22/08/12 16:24:43 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$pythonToJava$1
22/08/12 16:24:43 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$pythonToJava$1) is now cleaned +++
22/08/12 16:24:43 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$toJavaArray$1
22/08/12 16:24:43 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$toJavaArray$1) is now cleaned +++
22/08/12 16:24:43 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$applySchemaToPythonRDD$1
22/08/12 16:24:43 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$applySchemaToPythonRDD$1) is now cleaned +++
22/08/12 16:24:43 DEBUG CatalystSqlParser: Parsing command: spark_grouping_id
22/08/12 16:24:45 DEBUG WholeStageCodegenExec:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private scala.collection.Iterator rdd_input_0;
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[2];
/* 011 */
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) {
/* 013 */ this.references = references;
/* 014 */ }
/* 015 */
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) {
/* 017 */ partitionIndex = index;
/* 018 */ this.inputs = inputs;
/* 019 */ rdd_input_0 = inputs[0];
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0);
/* 021 */ rdd_mutableStateArray_0[1] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 64);
/* 022 */
/* 023 */ }
/* 024 */
/* 025 */ protected void processNext() throws java.io.IOException {
/* 026 */ while ( rdd_input_0.hasNext()) {
/* 027 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next();
/* 028 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 029 */ // common sub-expressions
/* 030 */
/* 031 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0);
/* 032 */ long rdd_value_0 = rdd_isNull_0 ?
/* 033 */ -1L : (rdd_row_0.getLong(0));
/* 034 */ boolean project_isNull_0 = rdd_isNull_0;
/* 035 */ UTF8String project_value_0 = null;
/* 036 */ if (!rdd_isNull_0) {
/* 037 */ project_value_0 = UTF8String.fromString(String.valueOf(rdd_value_0));
/* 038 */ }
/* 039 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1);
/* 040 */ double rdd_value_1 = rdd_isNull_1 ?
/* 041 */ -1.0 : (rdd_row_0.getDouble(1));
/* 042 */ boolean project_isNull_2 = rdd_isNull_1;
/* 043 */ UTF8String project_value_2 = null;
/* 044 */ if (!rdd_isNull_1) {
/* 045 */ project_value_2 = UTF8String.fromString(String.valueOf(rdd_value_1));
/* 046 */ }
/* 047 */ rdd_mutableStateArray_0[1].reset();
/* 048 */
/* 049 */ rdd_mutableStateArray_0[1].zeroOutNullBytes();
/* 050 */
/* 051 */ if (project_isNull_0) {
/* 052 */ rdd_mutableStateArray_0[1].setNullAt(0);
/* 053 */ } else {
/* 054 */ rdd_mutableStateArray_0[1].write(0, project_value_0);
/* 055 */ }
/* 056 */
/* 057 */ if (project_isNull_2) {
/* 058 */ rdd_mutableStateArray_0[1].setNullAt(1);
/* 059 */ } else {
/* 060 */ rdd_mutableStateArray_0[1].write(1, project_value_2);
/* 061 */ }
/* 062 */ append((rdd_mutableStateArray_0[1].getRow()));
/* 063 */ if (shouldStop()) return;
/* 064 */ }
/* 065 */ }
/* 066 */
/* 067 */ }
22/08/12 16:24:45 DEBUG CodeGenerator:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private scala.collection.Iterator rdd_input_0;
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[2];
/* 011 */
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) {
/* 013 */ this.references = references;
/* 014 */ }
/* 015 */
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) {
/* 017 */ partitionIndex = index;
/* 018 */ this.inputs = inputs;
/* 019 */ rdd_input_0 = inputs[0];
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0);
/* 021 */ rdd_mutableStateArray_0[1] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 64);
/* 022 */
/* 023 */ }
/* 024 */
/* 025 */ protected void processNext() throws java.io.IOException {
/* 026 */ while ( rdd_input_0.hasNext()) {
/* 027 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next();
/* 028 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 029 */ // common sub-expressions
/* 030 */
/* 031 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0);
/* 032 */ long rdd_value_0 = rdd_isNull_0 ?
/* 033 */ -1L : (rdd_row_0.getLong(0));
/* 034 */ boolean project_isNull_0 = rdd_isNull_0;
/* 035 */ UTF8String project_value_0 = null;
/* 036 */ if (!rdd_isNull_0) {
/* 037 */ project_value_0 = UTF8String.fromString(String.valueOf(rdd_value_0));
/* 038 */ }
/* 039 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1);
/* 040 */ double rdd_value_1 = rdd_isNull_1 ?
/* 041 */ -1.0 : (rdd_row_0.getDouble(1));
/* 042 */ boolean project_isNull_2 = rdd_isNull_1;
/* 043 */ UTF8String project_value_2 = null;
/* 044 */ if (!rdd_isNull_1) {
/* 045 */ project_value_2 = UTF8String.fromString(String.valueOf(rdd_value_1));
/* 046 */ }
/* 047 */ rdd_mutableStateArray_0[1].reset();
/* 048 */
/* 049 */ rdd_mutableStateArray_0[1].zeroOutNullBytes();
/* 050 */
/* 051 */ if (project_isNull_0) {
/* 052 */ rdd_mutableStateArray_0[1].setNullAt(0);
/* 053 */ } else {
/* 054 */ rdd_mutableStateArray_0[1].write(0, project_value_0);
/* 055 */ }
/* 056 */
/* 057 */ if (project_isNull_2) {
/* 058 */ rdd_mutableStateArray_0[1].setNullAt(1);
/* 059 */ } else {
/* 060 */ rdd_mutableStateArray_0[1].write(1, project_value_2);
/* 061 */ }
/* 062 */ append((rdd_mutableStateArray_0[1].getRow()));
/* 063 */ if (shouldStop()) return;
/* 064 */ }
/* 065 */ }
/* 066 */
/* 067 */ }
22/08/12 16:24:45 INFO CodeGenerator: Code generated in 173.205468 ms
22/08/12 16:24:45 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$doExecute$4$adapted
22/08/12 16:24:45 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$doExecute$4$adapted) is now cleaned +++
22/08/12 16:24:45 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$executeTake$2
22/08/12 16:24:45 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$executeTake$2) is now cleaned +++
22/08/12 16:24:45 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$runJob$5
22/08/12 16:24:45 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$runJob$5) is now cleaned +++
22/08/12 16:24:45 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0
22/08/12 16:24:45 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 6 took 0.000872 seconds
22/08/12 16:24:45 DEBUG DAGScheduler: Merging stage rdd profiles: Set()
22/08/12 16:24:45 INFO DAGScheduler: Got job 0 (showString at NativeMethodAccessorImpl.java:0) with 1 output partitions
22/08/12 16:24:45 INFO DAGScheduler: Final stage: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0)
22/08/12 16:24:45 INFO DAGScheduler: Parents of final stage: List()
22/08/12 16:24:45 INFO DAGScheduler: Missing parents: List()
22/08/12 16:24:45 DEBUG DAGScheduler: submitStage(ResultStage 0 (name=showString at NativeMethodAccessorImpl.java:0;jobs=0))
22/08/12 16:24:45 DEBUG DAGScheduler: missing: List()
22/08/12 16:24:45 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents
22/08/12 16:24:45 DEBUG DAGScheduler: submitMissingTasks(ResultStage 0)
22/08/12 16:24:45 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 12.4 KiB, free 434.4 MiB)
22/08/12 16:24:45 DEBUG BlockManager: Put block broadcast_0 locally took 32 ms
22/08/12 16:24:45 DEBUG BlockManager: Putting block broadcast_0 without replication took 33 ms
22/08/12 16:24:45 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 6.5 KiB, free 434.4 MiB)
22/08/12 16:24:45 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_0_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None)
22/08/12 16:24:45 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.1.255.235:38343 (size: 6.5 KiB, free: 434.4 MiB)
22/08/12 16:24:45 DEBUG BlockManagerMaster: Updated info of block broadcast_0_piece0
22/08/12 16:24:45 DEBUG BlockManager: Told master about block broadcast_0_piece0
22/08/12 16:24:45 DEBUG BlockManager: Put block broadcast_0_piece0 locally took 9 ms
22/08/12 16:24:45 DEBUG BlockManager: Putting block broadcast_0_piece0 without replication took 9 ms
22/08/12 16:24:45 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1513
22/08/12 16:24:45 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
22/08/12 16:24:45 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0
22/08/12 16:24:45 DEBUG TaskSetManager: Epoch for TaskSet 0.0: 0
22/08/12 16:24:45 DEBUG TaskSetManager: Adding pending tasks took 2 ms
22/08/12 16:24:45 DEBUG TaskSetManager: Valid locality levels for TaskSet 0.0: NO_PREF, ANY
22/08/12 16:24:45 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_0.0, runningTasks: 0
22/08/12 16:24:45 DEBUG TaskSetManager: Valid locality levels for TaskSet 0.0: NO_PREF, ANY
22/08/12 16:24:45 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (10.1.255.235, executor driver, partition 0, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:45 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY
22/08/12 16:24:45 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
22/08/12 16:24:45 DEBUG ExecutorMetricsPoller: stageTCMP: (0, 0) -> 1
22/08/12 16:24:45 DEBUG BlockManager: Getting local block broadcast_0
22/08/12 16:24:45 DEBUG BlockManager: Level for block broadcast_0 is StorageLevel(disk, memory, deserialized, 1 replicas)
22/08/12 16:24:46 INFO PythonRunner: Times: total = 427, boot = 367, init = 60, finish = 0
22/08/12 16:24:46 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (0, 0) -> 0
22/08/12 16:24:46 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 571 ms on 10.1.255.235 (executor driver) (1/1)
22/08/12 16:24:46 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
22/08/12 16:24:46 INFO PythonAccumulatorV2: Connected to AccumulatorServer at host: 127.0.0.1 port: 49069
22/08/12 16:24:46 INFO DAGScheduler: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0) finished in 0.785 s
22/08/12 16:24:46 DEBUG DAGScheduler: After removal of stage 0, remaining stages = 0
22/08/12 16:24:46 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/12 16:24:46 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
22/08/12 16:24:46 INFO DAGScheduler: Job 0 finished: showString at NativeMethodAccessorImpl.java:0, took 0.837916 s
22/08/12 16:24:46 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$executeTake$2
22/08/12 16:24:46 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$executeTake$2) is now cleaned +++
22/08/12 16:24:46 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$runJob$5
22/08/12 16:24:46 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$runJob$5) is now cleaned +++
22/08/12 16:24:46 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0
22/08/12 16:24:46 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 6 took 0.000216 seconds
22/08/12 16:24:46 DEBUG DAGScheduler: Merging stage rdd profiles: Set()
22/08/12 16:24:46 INFO DAGScheduler: Got job 1 (showString at NativeMethodAccessorImpl.java:0) with 4 output partitions
22/08/12 16:24:46 INFO DAGScheduler: Final stage: ResultStage 1 (showString at NativeMethodAccessorImpl.java:0)
22/08/12 16:24:46 INFO DAGScheduler: Parents of final stage: List()
22/08/12 16:24:46 INFO DAGScheduler: Missing parents: List()
22/08/12 16:24:46 DEBUG DAGScheduler: submitStage(ResultStage 1 (name=showString at NativeMethodAccessorImpl.java:0;jobs=1))
22/08/12 16:24:46 DEBUG DAGScheduler: missing: List()
22/08/12 16:24:46 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents
22/08/12 16:24:46 DEBUG DAGScheduler: submitMissingTasks(ResultStage 1)
22/08/12 16:24:46 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 12.4 KiB, free 434.4 MiB)
22/08/12 16:24:46 DEBUG BlockManager: Put block broadcast_1 locally took 1 ms
22/08/12 16:24:46 DEBUG BlockManager: Putting block broadcast_1 without replication took 2 ms
22/08/12 16:24:46 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.6 KiB, free 434.4 MiB)
22/08/12 16:24:46 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_1_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None)
22/08/12 16:24:46 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.1.255.235:38343 (size: 6.6 KiB, free: 434.4 MiB)
22/08/12 16:24:46 DEBUG BlockManagerMaster: Updated info of block broadcast_1_piece0
22/08/12 16:24:46 DEBUG BlockManager: Told master about block broadcast_1_piece0
22/08/12 16:24:46 DEBUG BlockManager: Put block broadcast_1_piece0 locally took 2 ms
22/08/12 16:24:46 DEBUG BlockManager: Putting block broadcast_1_piece0 without replication took 3 ms
22/08/12 16:24:46 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1513
22/08/12 16:24:46 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 1 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(1, 2, 3, 4))
22/08/12 16:24:46 INFO TaskSchedulerImpl: Adding task set 1.0 with 4 tasks resource profile 0
22/08/12 16:24:46 DEBUG TaskSetManager: Epoch for TaskSet 1.0: 0
22/08/12 16:24:46 DEBUG TaskSetManager: Adding pending tasks took 0 ms
22/08/12 16:24:46 DEBUG TaskSetManager: Valid locality levels for TaskSet 1.0: NO_PREF, ANY
22/08/12 16:24:46 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_1.0, runningTasks: 0
22/08/12 16:24:46 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1) (10.1.255.235, executor driver, partition 1, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2) (10.1.255.235, executor driver, partition 2, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3) (10.1.255.235, executor driver, partition 3, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 4) (10.1.255.235, executor driver, partition 4, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY
22/08/12 16:24:46 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
22/08/12 16:24:46 INFO Executor: Running task 1.0 in stage 1.0 (TID 2)
22/08/12 16:24:46 INFO Executor: Running task 2.0 in stage 1.0 (TID 3)
22/08/12 16:24:46 INFO Executor: Running task 3.0 in stage 1.0 (TID 4)
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 1
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 2
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 3
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 4
22/08/12 16:24:46 DEBUG BlockManager: Getting local block broadcast_1
22/08/12 16:24:46 DEBUG BlockManager: Level for block broadcast_1 is StorageLevel(disk, memory, deserialized, 1 replicas)
22/08/12 16:24:46 INFO PythonRunner: Times: total = 62, boot = 6, init = 56, finish = 0
22/08/12 16:24:46 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 3
22/08/12 16:24:46 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 88 ms on 10.1.255.235 (executor driver) (1/4)
22/08/12 16:24:46 INFO PythonRunner: Times: total = 67, boot = 9, init = 58, finish = 0
22/08/12 16:24:46 INFO Executor: Finished task 3.0 in stage 1.0 (TID 4). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 2
22/08/12 16:24:46 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 4) in 91 ms on 10.1.255.235 (executor driver) (2/4)
22/08/12 16:24:46 INFO PythonRunner: Times: total = 74, boot = -98, init = 172, finish = 0
22/08/12 16:24:46 INFO Executor: Finished task 1.0 in stage 1.0 (TID 2). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 1
22/08/12 16:24:46 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 98 ms on 10.1.255.235 (executor driver) (3/4)
22/08/12 16:24:46 INFO PythonRunner: Times: total = 84, boot = 3, init = 80, finish = 1
22/08/12 16:24:46 INFO Executor: Finished task 2.0 in stage 1.0 (TID 3). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (1, 0) -> 0
22/08/12 16:24:46 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 3) in 106 ms on 10.1.255.235 (executor driver) (4/4)
22/08/12 16:24:46 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
22/08/12 16:24:46 INFO DAGScheduler: ResultStage 1 (showString at NativeMethodAccessorImpl.java:0) finished in 0.125 s
22/08/12 16:24:46 DEBUG DAGScheduler: After removal of stage 1, remaining stages = 0
22/08/12 16:24:46 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/12 16:24:46 INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished
22/08/12 16:24:46 INFO DAGScheduler: Job 1 finished: showString at NativeMethodAccessorImpl.java:0, took 0.131962 s
22/08/12 16:24:46 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$executeTake$2
22/08/12 16:24:46 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$executeTake$2) is now cleaned +++
22/08/12 16:24:46 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$runJob$5
22/08/12 16:24:46 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$runJob$5) is now cleaned +++
22/08/12 16:24:46 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0
22/08/12 16:24:46 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 6 took 0.000154 seconds
22/08/12 16:24:46 DEBUG DAGScheduler: Merging stage rdd profiles: Set()
22/08/12 16:24:46 INFO DAGScheduler: Got job 2 (showString at NativeMethodAccessorImpl.java:0) with 11 output partitions
22/08/12 16:24:46 INFO DAGScheduler: Final stage: ResultStage 2 (showString at NativeMethodAccessorImpl.java:0)
22/08/12 16:24:46 INFO DAGScheduler: Parents of final stage: List()
22/08/12 16:24:46 INFO DAGScheduler: Missing parents: List()
22/08/12 16:24:46 DEBUG DAGScheduler: submitStage(ResultStage 2 (name=showString at NativeMethodAccessorImpl.java:0;jobs=2))
22/08/12 16:24:46 DEBUG DAGScheduler: missing: List()
22/08/12 16:24:46 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents
22/08/12 16:24:46 DEBUG DAGScheduler: submitMissingTasks(ResultStage 2)
22/08/12 16:24:46 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 12.4 KiB, free 434.4 MiB)
22/08/12 16:24:46 DEBUG BlockManager: Put block broadcast_2 locally took 1 ms
22/08/12 16:24:46 DEBUG BlockManager: Putting block broadcast_2 without replication took 1 ms
22/08/12 16:24:46 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 6.6 KiB, free 434.3 MiB)
22/08/12 16:24:46 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_2_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None)
22/08/12 16:24:46 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.1.255.235:38343 (size: 6.6 KiB, free: 434.4 MiB)
22/08/12 16:24:46 DEBUG BlockManagerMaster: Updated info of block broadcast_2_piece0
22/08/12 16:24:46 DEBUG BlockManager: Told master about block broadcast_2_piece0
22/08/12 16:24:46 DEBUG BlockManager: Put block broadcast_2_piece0 locally took 2 ms
22/08/12 16:24:46 DEBUG BlockManager: Putting block broadcast_2_piece0 without replication took 2 ms
22/08/12 16:24:46 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1513
22/08/12 16:24:46 INFO DAGScheduler: Submitting 11 missing tasks from ResultStage 2 (MapPartitionsRDD[6] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15))
22/08/12 16:24:46 INFO TaskSchedulerImpl: Adding task set 2.0 with 11 tasks resource profile 0
22/08/12 16:24:46 DEBUG TaskSetManager: Epoch for TaskSet 2.0: 0
22/08/12 16:24:46 DEBUG TaskSetManager: Adding pending tasks took 0 ms
22/08/12 16:24:46 DEBUG TaskSetManager: Valid locality levels for TaskSet 2.0: NO_PREF, ANY
22/08/12 16:24:46 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_2.0, runningTasks: 0
22/08/12 16:24:46 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 5) (10.1.255.235, executor driver, partition 5, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 6) (10.1.255.235, executor driver, partition 6, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 7) (10.1.255.235, executor driver, partition 7, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 8) (10.1.255.235, executor driver, partition 8, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 4.0 in stage 2.0 (TID 9) (10.1.255.235, executor driver, partition 9, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 5.0 in stage 2.0 (TID 10) (10.1.255.235, executor driver, partition 10, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 6.0 in stage 2.0 (TID 11) (10.1.255.235, executor driver, partition 11, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 7.0 in stage 2.0 (TID 12) (10.1.255.235, executor driver, partition 12, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 8.0 in stage 2.0 (TID 13) (10.1.255.235, executor driver, partition 13, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 9.0 in stage 2.0 (TID 14) (10.1.255.235, executor driver, partition 14, PROCESS_LOCAL, 4433 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 INFO TaskSetManager: Starting task 10.0 in stage 2.0 (TID 15) (10.1.255.235, executor driver, partition 15, PROCESS_LOCAL, 4471 bytes) taskResourceAssignments Map()
22/08/12 16:24:46 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY
22/08/12 16:24:46 INFO Executor: Running task 1.0 in stage 2.0 (TID 6)
22/08/12 16:24:46 INFO Executor: Running task 0.0 in stage 2.0 (TID 5)
22/08/12 16:24:46 INFO Executor: Running task 2.0 in stage 2.0 (TID 7)
22/08/12 16:24:46 INFO Executor: Running task 3.0 in stage 2.0 (TID 8)
22/08/12 16:24:46 INFO Executor: Running task 5.0 in stage 2.0 (TID 10)
22/08/12 16:24:46 INFO Executor: Running task 4.0 in stage 2.0 (TID 9)
22/08/12 16:24:46 INFO Executor: Running task 6.0 in stage 2.0 (TID 11)
22/08/12 16:24:46 INFO Executor: Running task 7.0 in stage 2.0 (TID 12)
22/08/12 16:24:46 INFO Executor: Running task 8.0 in stage 2.0 (TID 13)
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 3
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 2
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 4
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 1
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 5
22/08/12 16:24:46 INFO Executor: Running task 9.0 in stage 2.0 (TID 14)
22/08/12 16:24:46 INFO Executor: Running task 10.0 in stage 2.0 (TID 15)
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 6
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 8
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 7
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 9
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 10
22/08/12 16:24:46 DEBUG BlockManager: Getting local block broadcast_2
22/08/12 16:24:46 DEBUG BlockManager: Level for block broadcast_2 is StorageLevel(disk, memory, deserialized, 1 replicas)
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 11
22/08/12 16:24:46 INFO PythonRunner: Times: total = 66, boot = -51, init = 117, finish = 0
22/08/12 16:24:46 INFO Executor: Finished task 2.0 in stage 2.0 (TID 7). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 10
22/08/12 16:24:46 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 7) in 90 ms on 10.1.255.235 (executor driver) (1/11)
22/08/12 16:24:46 INFO PythonRunner: Times: total = 75, boot = -54, init = 129, finish = 0
22/08/12 16:24:46 INFO PythonRunner: Times: total = 76, boot = 11, init = 65, finish = 0
22/08/12 16:24:46 INFO PythonRunner: Times: total = 82, boot = -59, init = 141, finish = 0
22/08/12 16:24:46 INFO Executor: Finished task 6.0 in stage 2.0 (TID 11). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 9
22/08/12 16:24:46 INFO TaskSetManager: Finished task 6.0 in stage 2.0 (TID 11) in 99 ms on 10.1.255.235 (executor driver) (2/11)
22/08/12 16:24:46 INFO Executor: Finished task 7.0 in stage 2.0 (TID 12). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 8
22/08/12 16:24:46 INFO TaskSetManager: Finished task 7.0 in stage 2.0 (TID 12) in 102 ms on 10.1.255.235 (executor driver) (3/11)
22/08/12 16:24:46 INFO Executor: Finished task 3.0 in stage 2.0 (TID 8). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 7
22/08/12 16:24:46 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 8) in 110 ms on 10.1.255.235 (executor driver) (4/11)
22/08/12 16:24:46 INFO PythonRunner: Times: total = 96, boot = 8, init = 88, finish = 0
22/08/12 16:24:46 INFO PythonRunner: Times: total = 97, boot = 3, init = 93, finish = 1
22/08/12 16:24:46 INFO Executor: Finished task 9.0 in stage 2.0 (TID 14). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 6
22/08/12 16:24:46 INFO Executor: Finished task 5.0 in stage 2.0 (TID 10). 1789 bytes result sent to driver
22/08/12 16:24:46 INFO TaskSetManager: Finished task 9.0 in stage 2.0 (TID 14) in 116 ms on 10.1.255.235 (executor driver) (5/11)
22/08/12 16:24:46 INFO PythonRunner: Times: total = 98, boot = -44, init = 142, finish = 0
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 5
22/08/12 16:24:46 INFO PythonRunner: Times: total = 100, boot = 12, init = 87, finish = 1
22/08/12 16:24:46 INFO TaskSetManager: Finished task 5.0 in stage 2.0 (TID 10) in 120 ms on 10.1.255.235 (executor driver) (6/11)
22/08/12 16:24:46 INFO Executor: Finished task 0.0 in stage 2.0 (TID 5). 1789 bytes result sent to driver
22/08/12 16:24:46 INFO Executor: Finished task 1.0 in stage 2.0 (TID 6). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 4
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 3
22/08/12 16:24:46 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 5) in 128 ms on 10.1.255.235 (executor driver) (7/11)
22/08/12 16:24:46 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 6) in 128 ms on 10.1.255.235 (executor driver) (8/11)
22/08/12 16:24:46 INFO PythonRunner: Times: total = 112, boot = 24, init = 88, finish = 0
22/08/12 16:24:46 INFO PythonRunner: Times: total = 117, boot = 19, init = 98, finish = 0
22/08/12 16:24:46 INFO PythonRunner: Times: total = 106, boot = 16, init = 90, finish = 0
22/08/12 16:24:46 INFO Executor: Finished task 8.0 in stage 2.0 (TID 13). 1789 bytes result sent to driver
22/08/12 16:24:46 INFO Executor: Finished task 4.0 in stage 2.0 (TID 9). 1789 bytes result sent to driver
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 2
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 1
22/08/12 16:24:46 INFO Executor: Finished task 10.0 in stage 2.0 (TID 15). 1830 bytes result sent to driver
22/08/12 16:24:46 INFO TaskSetManager: Finished task 4.0 in stage 2.0 (TID 9) in 145 ms on 10.1.255.235 (executor driver) (9/11)
22/08/12 16:24:46 DEBUG ExecutorMetricsPoller: stageTCMP: (2, 0) -> 0
22/08/12 16:24:46 INFO TaskSetManager: Finished task 8.0 in stage 2.0 (TID 13) in 144 ms on 10.1.255.235 (executor driver) (10/11)
22/08/12 16:24:46 INFO TaskSetManager: Finished task 10.0 in stage 2.0 (TID 15) in 144 ms on 10.1.255.235 (executor driver) (11/11)
22/08/12 16:24:46 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
22/08/12 16:24:46 INFO DAGScheduler: ResultStage 2 (showString at NativeMethodAccessorImpl.java:0) finished in 0.162 s
22/08/12 16:24:46 DEBUG DAGScheduler: After removal of stage 2, remaining stages = 0
22/08/12 16:24:46 INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/12 16:24:46 INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished
22/08/12 16:24:46 INFO DAGScheduler: Job 2 finished: showString at NativeMethodAccessorImpl.java:0, took 0.168241 s
22/08/12 16:24:46 DEBUG GenerateSafeProjection: code for createexternalrow(input[0, string, true].toString, input[1, string, true].toString, StructField(a,StringType,true), StructField(b,StringType,true)):
/* 001 */ public java.lang.Object generate(Object[] references) {
/* 002 */ return new SpecificSafeProjection(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
/* 006 */
/* 007 */ private Object[] references;
/* 008 */ private InternalRow mutableRow;
/* 009 */
/* 010 */
/* 011 */ public SpecificSafeProjection(Object[] references) {
/* 012 */ this.references = references;
/* 013 */ mutableRow = (InternalRow) references[references.length - 1];
/* 014 */
/* 015 */ }
/* 016 */
/* 017 */ public void initialize(int partitionIndex) {
/* 018 */
/* 019 */ }
/* 020 */
/* 021 */ public java.lang.Object apply(java.lang.Object _i) {
/* 022 */ InternalRow i = (InternalRow) _i;
/* 023 */ org.apache.spark.sql.Row value_5 = CreateExternalRow_0(i);
/* 024 */ if (false) {
/* 025 */ mutableRow.setNullAt(0);
/* 026 */ } else {
/* 027 */
/* 028 */ mutableRow.update(0, value_5);
/* 029 */ }
/* 030 */
/* 031 */ return mutableRow;
/* 032 */ }
/* 033 */
/* 034 */
/* 035 */ private org.apache.spark.sql.Row CreateExternalRow_0(InternalRow i) {
/* 036 */ Object[] values_0 = new Object[2];
/* 037 */
/* 038 */ boolean isNull_2 = i.isNullAt(0);
/* 039 */ UTF8String value_2 = isNull_2 ?
/* 040 */ null : (i.getUTF8String(0));
/* 041 */ boolean isNull_1 = true;
/* 042 */ java.lang.String value_1 = null;
/* 043 */ if (!isNull_2) {
/* 044 */ isNull_1 = false;
/* 045 */ if (!isNull_1) {
/* 046 */
/* 047 */ Object funcResult_0 = null;
/* 048 */ funcResult_0 = value_2.toString();
/* 049 */ value_1 = (java.lang.String) funcResult_0;
/* 050 */
/* 051 */ }
/* 052 */ }
/* 053 */ if (isNull_1) {
/* 054 */ values_0[0] = null;
/* 055 */ } else {
/* 056 */ values_0[0] = value_1;
/* 057 */ }
/* 058 */
/* 059 */ boolean isNull_4 = i.isNullAt(1);
/* 060 */ UTF8String value_4 = isNull_4 ?
/* 061 */ null : (i.getUTF8String(1));
/* 062 */ boolean isNull_3 = true;
/* 063 */ java.lang.String value_3 = null;
/* 064 */ if (!isNull_4) {
/* 065 */ isNull_3 = false;
/* 066 */ if (!isNull_3) {
/* 067 */
/* 068 */ Object funcResult_1 = null;
/* 069 */ funcResult_1 = value_4.toString();
/* 070 */ value_3 = (java.lang.String) funcResult_1;
/* 071 */
/* 072 */ }
/* 073 */ }
/* 074 */ if (isNull_3) {
/* 075 */ values_0[1] = null;
/* 076 */ } else {
/* 077 */ values_0[1] = value_3;
/* 078 */ }
/* 079 */
/* 080 */ final org.apache.spark.sql.Row value_0 = new org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema(values_0, ((org.apache.spark.sql.types.StructType) references[0] /* schema */));
/* 081 */
/* 082 */ return value_0;
/* 083 */ }
/* 084 */
/* 085 */ }
22/08/12 16:24:46 DEBUG CodeGenerator:
/* 001 */ public java.lang.Object generate(Object[] references) {
/* 002 */ return new SpecificSafeProjection(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
/* 006 */
/* 007 */ private Object[] references;
/* 008 */ private InternalRow mutableRow;
/* 009 */
/* 010 */
/* 011 */ public SpecificSafeProjection(Object[] references) {
/* 012 */ this.references = references;
/* 013 */ mutableRow = (InternalRow) references[references.length - 1];
/* 014 */
/* 015 */ }
/* 016 */
/* 017 */ public void initialize(int partitionIndex) {
/* 018 */
/* 019 */ }
/* 020 */
/* 021 */ public java.lang.Object apply(java.lang.Object _i) {
/* 022 */ InternalRow i = (InternalRow) _i;
/* 023 */ org.apache.spark.sql.Row value_5 = CreateExternalRow_0(i);
/* 024 */ if (false) {
/* 025 */ mutableRow.setNullAt(0);
/* 026 */ } else {
/* 027 */
/* 028 */ mutableRow.update(0, value_5);
/* 029 */ }
/* 030 */
/* 031 */ return mutableRow;
/* 032 */ }
/* 033 */
/* 034 */
/* 035 */ private org.apache.spark.sql.Row CreateExternalRow_0(InternalRow i) {
/* 036 */ Object[] values_0 = new Object[2];
/* 037 */
/* 038 */ boolean isNull_2 = i.isNullAt(0);
/* 039 */ UTF8String value_2 = isNull_2 ?
/* 040 */ null : (i.getUTF8String(0));
/* 041 */ boolean isNull_1 = true;
/* 042 */ java.lang.String value_1 = null;
/* 043 */ if (!isNull_2) {
/* 044 */ isNull_1 = false;
/* 045 */ if (!isNull_1) {
/* 046 */
/* 047 */ Object funcResult_0 = null;
/* 048 */ funcResult_0 = value_2.toString();
/* 049 */ value_1 = (java.lang.String) funcResult_0;
/* 050 */
/* 051 */ }
/* 052 */ }
/* 053 */ if (isNull_1) {
/* 054 */ values_0[0] = null;
/* 055 */ } else {
/* 056 */ values_0[0] = value_1;
/* 057 */ }
/* 058 */
/* 059 */ boolean isNull_4 = i.isNullAt(1);
/* 060 */ UTF8String value_4 = isNull_4 ?
/* 061 */ null : (i.getUTF8String(1));
/* 062 */ boolean isNull_3 = true;
/* 063 */ java.lang.String value_3 = null;
/* 064 */ if (!isNull_4) {
/* 065 */ isNull_3 = false;
/* 066 */ if (!isNull_3) {
/* 067 */
/* 068 */ Object funcResult_1 = null;
/* 069 */ funcResult_1 = value_4.toString();
/* 070 */ value_3 = (java.lang.String) funcResult_1;
/* 071 */
/* 072 */ }
/* 073 */ }
/* 074 */ if (isNull_3) {
/* 075 */ values_0[1] = null;
/* 076 */ } else {
/* 077 */ values_0[1] = value_3;
/* 078 */ }
/* 079 */
/* 080 */ final org.apache.spark.sql.Row value_0 = new org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema(values_0, ((org.apache.spark.sql.types.StructType) references[0] /* schema */));
/* 081 */
/* 082 */ return value_0;
/* 083 */ }
/* 084 */
/* 085 */ }
22/08/12 16:24:46 INFO CodeGenerator: Code generated in 15.263034 ms
+---+---+
| a| b|
+---+---+
| 1|2.0|
+---+---+
22/08/12 16:24:46 DEBUG FileSystem: Starting: Acquiring creator semaphore for s3a://master.rando2/nonemptyprefix8
22/08/12 16:24:46 DEBUG FileSystem: Acquiring creator semaphore for s3a://master.rando2/nonemptyprefix8: duration 0:00.000s
22/08/12 16:24:46 DEBUG FileSystem: Starting: Creating FS s3a://master.rando2/nonemptyprefix8
22/08/12 16:24:46 DEBUG FileSystem: Looking for FS supporting s3a
22/08/12 16:24:46 DEBUG FileSystem: looking for configuration option fs.s3a.impl
22/08/12 16:24:46 DEBUG Configuration: Reloading 9 existing configurations
22/08/12 16:24:46 DEBUG FileSystem: Filesystem s3a defined in configuration option
22/08/12 16:24:46 DEBUG FileSystem: FS for s3a is class org.apache.hadoop.fs.s3a.S3AFileSystem
22/08/12 16:24:46 DEBUG AbstractService: Service: NoopAuditor entered state INITED
22/08/12 16:24:46 DEBUG AbstractService: Service NoopAuditor is started
22/08/12 16:24:46 DEBUG S3AFileSystem: Initializing S3AFileSystem for master.rando2
22/08/12 16:24:46 DEBUG S3AUtils: Propagating entries under fs.s3a.bucket.master.rando2.
22/08/12 16:24:46 DEBUG S3AUtils: Data is unencrypted
22/08/12 16:24:46 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms
22/08/12 16:24:46 DEBUG MetricsSystemImpl: from system property: null
22/08/12 16:24:46 DEBUG MetricsSystemImpl: from environment variable: null
22/08/12 16:24:46 DEBUG MetricsConfig: Could not locate file hadoop-metrics2-s3a-file-system.properties
org.apache.hadoop.shaded.org.apache.commons.configuration2.ex.ConfigurationException: Could not locate: org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocator@1191d4eb[fileName=hadoop-metrics2-s3a-file-system.properties,basePath=<null>,sourceURL=,encoding=<null>,fileSystem=<null>,locationStrategy=<null>]
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocatorUtils.locateOrThrow(FileLocatorUtils.java:346)
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:972)
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:702)
at org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:118)
at org.apache.hadoop.metrics2.impl.MetricsConfig.create(MetricsConfig.java:97)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:482)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.getMetricsSystem(S3AInstrumentation.java:251)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.registerAsMetricsSource(S3AInstrumentation.java:264)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.<init>(S3AInstrumentation.java:235)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:466)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461)
at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:558)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
22/08/12 16:24:46 DEBUG MetricsConfig: Could not locate file hadoop-metrics2.properties
org.apache.hadoop.shaded.org.apache.commons.configuration2.ex.ConfigurationException: Could not locate: org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocator@79eafac4[fileName=hadoop-metrics2.properties,basePath=<null>,sourceURL=,encoding=<null>,fileSystem=<null>,locationStrategy=<null>]
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileLocatorUtils.locateOrThrow(FileLocatorUtils.java:346)
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:972)
at org.apache.hadoop.shaded.org.apache.commons.configuration2.io.FileHandler.load(FileHandler.java:702)
at org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:118)
at org.apache.hadoop.metrics2.impl.MetricsConfig.create(MetricsConfig.java:97)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:482)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.getMetricsSystem(S3AInstrumentation.java:251)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.registerAsMetricsSource(S3AInstrumentation.java:264)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.<init>(S3AInstrumentation.java:235)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:466)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461)
at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:558)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
22/08/12 16:24:46 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: period
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: periodMillis
22/08/12 16:24:46 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableCounterLong org.apache.hadoop.metrics2.impl.MetricsSystemImpl.droppedPubAll with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Dropped updates by all sinks"})
22/08/12 16:24:46 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableStat org.apache.hadoop.metrics2.impl.MetricsSystemImpl.publishStat with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Publish", "Publishing stats"})
22/08/12 16:24:46 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableStat org.apache.hadoop.metrics2.impl.MetricsSystemImpl.snapshotStat with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Snapshot", "Snapshot stats"})
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: source.source.start_mbeans
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'MetricsConfig' for key: source.start_mbeans
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: *.source.start_mbeans
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Updating attr cache...
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Done. # tags & metrics=10
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Updating info cache...
22/08/12 16:24:46 DEBUG MetricsSystemImpl: [javax.management.MBeanAttributeInfo[description=Metrics context, name=tag.Context, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of active metrics sources, name=NumActiveSources, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of all registered metrics sources, name=NumAllSources, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of active metrics sinks, name=NumActiveSinks, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of all registered metrics sinks, name=NumAllSinks, type=java.lang.Integer, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Dropped updates by all sinks, name=DroppedPubAll, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of ops for publishing stats, name=PublishNumOps, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Average time for publishing stats, name=PublishAvgTime, type=java.lang.Double, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of ops for snapshot stats, name=SnapshotNumOps, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Average time for snapshot stats, name=SnapshotAvgTime, type=java.lang.Double, read-only, descriptor={}]]
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Done
22/08/12 16:24:46 DEBUG MBeans: Registered Hadoop:service=s3a-file-system,name=MetricsSystem,sub=Stats
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
22/08/12 16:24:46 INFO MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
22/08/12 16:24:46 INFO MetricsSystemImpl: s3a-file-system metrics system started
22/08/12 16:24:46 DEBUG MBeans: Registered Hadoop:service=s3a-file-system,name=MetricsSystem,sub=Control
22/08/12 16:24:46 DEBUG MetricsSystemImpl: S3AMetrics1-master.rando2,
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: source.source.start_mbeans
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'MetricsConfig' for key: source.start_mbeans
22/08/12 16:24:46 DEBUG MetricsConfig: poking parent 'PropertiesConfiguration' for key: *.source.start_mbeans
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Updating attr cache...
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Done. # tags & metrics=187
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Updating info cache...
22/08/12 16:24:46 DEBUG MetricsSystemImpl: [javax.management.MBeanAttributeInfo[description=Metrics context, name=tag.Context, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=A unique identifier for the instance, name=tag.s3aFileSystemId, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Hostname from the FS URL, name=tag.bucket, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Local hostname, name=tag.Hostname, type=java.lang.String, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of directories created through the object store., name=directories_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of directories deleted through the object store., name=directories_deleted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files copied within the object store., name=files_copied, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of bytes copied within the object store., name=files_copied_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files created through the object store., name=files_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files deleted from the object store., name=files_deleted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of files whose delete request was rejected, name=files_delete_rejected, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of fake directory entries created in the object store., name=fake_directories_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total number of fake directory deletes submitted to object store., name=fake_directories_deleted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Errors caught and ignored, name=ignored_errors, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of createNonRecursive(), name=op_create_non_recursive, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of hflush(), name=op_hflush, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of hsync(), name=op_hsync, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listLocatedStatus(), name=op_list_located_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of open(), name=op_open, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object copy requests, name=object_copy_requests, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Objects deleted in delete requests, name=object_delete_objects, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of requests for object metadata, name=object_metadata_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object put/multipart upload completed count, name=object_put_request_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=number of bytes uploaded, name=object_put_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of S3 Select requests issued, name=object_select_requests, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of times the TCP stream was aborted, name=stream_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Bytes read from an input stream in read() calls, name=stream_read_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes discarded by aborting an input stream, name=stream_read_bytes_discarded_in_abort, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes read and discarded when closing an input stream, name=stream_read_bytes_discarded_in_close, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of times the TCP stream was closed, name=stream_read_closed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total count of times an attempt to close an input stream was made, name=stream_read_close_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of exceptions raised during input stream reads, name=stream_read_exceptions, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of readFully() operations in an input stream, name=stream_read_fully_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total count of times an input stream to object store data was opened, name=stream_read_opened, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of read() operations in an input stream, name=stream_read_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of incomplete read() operations in an input stream, name=stream_read_operations_incomplete, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of version mismatches encountered while reading an input stream, name=stream_read_version_mismatches, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of executed seek operations which went backwards in a stream, name=stream_read_seek_backward_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes moved backwards during seek operations in an input stream, name=stream_read_bytes_backwards_on_seek, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes read and discarded during seek() in an input stream, name=stream_read_seek_bytes_discarded, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes skipped during forward seek operations an input stream, name=stream_read_seek_bytes_skipped, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of executed seek operations which went forward in an input stream, name=stream_read_seek_forward_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of seek operations in an input stream, name=stream_read_seek_operations, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of times the seek policy was dynamically changed in an input stream, name=stream_read_seek_policy_changed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total count of bytes read from an input stream, name=stream_read_total_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of stream write failures reported, name=stream_write_exceptions, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of failures when finalizing a multipart upload, name=stream_write_exceptions_completing_upload, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of block/partition uploads completed, name=stream_write_block_uploads, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of number of block uploads committed, name=stream_write_block_uploads_committed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of number of block uploads aborted, name=stream_write_block_uploads_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of total time taken for uploads to complete, name=stream_write_total_time, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of total data uploaded, name=stream_write_total_data, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes written to output stream (including all not yet uploaded), name=stream_write_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of files to commit created, name=committer_commits_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of files committed, name=committer_commits_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of successful jobs, name=committer_jobs_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of failed jobs, name=committer_jobs_failed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of successful tasks, name=committer_tasks_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of failed tasks, name=committer_tasks_failed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Amount of data committed, name=committer_bytes_committed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of bytes uploaded duing commit operations, name=committer_bytes_uploaded, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of commits failed, name=committer_commits.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of commits aborted, name=committer_commits_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of commits reverted, name=committer_commits_reverted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of files created under 'magic' paths, name=committer_magic_files_created, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store put one metadata path request, name=s3guard_metadatastore_put_path_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store initialization times, name=s3guard_metadatastore_initialization, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store records deleted, name=s3guard_metadatastore_record_deletes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store records read, name=s3guard_metadatastore_record_reads, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store records written, name=s3guard_metadatastore_record_writes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store retry events, name=s3guard_metadatastore_retry, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store throttled events, name=s3guard_metadatastore_throttled, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=S3Guard metadata store authoritative directories updated from S3, name=s3guard_metadatastore_authoritative_directories_updated, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=requests made of the remote store, name=store_io_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=retried requests made of the remote store, name=store_io_retry, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Requests throttled and retried, name=store_io_throttled, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Uploader Instantiated, name=multipart_instantiated, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Part Put Operation, name=multipart_upload_part_put, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Part Put Bytes, name=multipart_upload_part_put_bytes, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Aborted, name=multipart_upload_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Abort Unner Path Invoked, name=multipart_upload_abort_under_path_invoked, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Completed, name=multipart_upload_completed, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload Started, name=multipart_upload_started, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Audit access check was rejected, name=audit_access_check_failure, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Audit Span Created, name=audit_span_creation, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Audit failure/rejection, name=audit_failure, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=AWS request made, name=audit_request_execution, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Current number of active put requests, name=object_put_request_active, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=number of bytes queued for upload/being actively uploaded, name=object_put_bytes_pending, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of block/partition uploads active, name=stream_write_block_uploads_active, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Gauge of block/partitions uploads queued to be written, name=stream_write_block_uploads_pending, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Gauge of data queued to be written, name=stream_write_block_uploads_data_pending, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=gauge to indicate if client side encryption is enabled, name=client_side_encryption_enabled, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Executor acquired., name=action_executor_acquired, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Executor acquired., name=action_executor_acquired.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=HEAD request., name=action_http_head_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=HEAD request., name=action_http_head_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=GET request., name=action_http_get_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=GET request., name=action_http_get_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of abort(), name=op_abort, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of abort(), name=op_abort.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of access(), name=op_access, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of access(), name=op_access.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of copyFromLocalFile(), name=op_copy_from_local_file, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of copyFromLocalFile(), name=op_copy_from_local_file.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of create(), name=op_create, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of create(), name=op_create.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of delete(), name=op_delete, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of delete(), name=op_delete.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of exists(), name=op_exists, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of exists(), name=op_exists.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getContentSummary(), name=op_get_content_summary, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getContentSummary(), name=op_get_content_summary.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getDelegationToken(), name=op_get_delegation_token, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getDelegationToken(), name=op_get_delegation_token.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileChecksum(), name=op_get_file_checksum, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileChecksum(), name=op_get_file_checksum.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileStatus(), name=op_get_file_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getFileStatus(), name=op_get_file_status.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of globStatus(), name=op_glob_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of globStatus(), name=op_glob_status.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isDirectory(), name=op_is_directory, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isDirectory(), name=op_is_directory.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isFile(), name=op_is_file, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of isFile(), name=op_is_file.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listFiles(), name=op_list_files, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listFiles(), name=op_list_files.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listStatus(), name=op_list_status, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of listStatus(), name=op_list_status.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of mkdirs(), name=op_mkdirs, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of mkdirs(), name=op_mkdirs.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of rename(), name=op_rename, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of rename(), name=op_rename.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path), name=op_xattr_get_map, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path), name=op_xattr_get_map.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttr(Path, String), name=op_xattr_get_named, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttr(Path, String), name=op_xattr_get_named.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of xattr(), name=op_xattr_get_named_map, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of xattr(), name=op_xattr_get_named_map.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path, List<String> names), name=op_xattr_list, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Calls of getXAttrs(Path path, List<String> names), name=op_xattr_list.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object delete requests, name=object_delete_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object delete requests, name=object_delete_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object bulk delete requests, name=object_bulk_delete_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object bulk delete requests, name=object_bulk_delete_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of object listings made, name=object_list_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of object listings made, name=object_list_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of continued object listings made, name=object_continue_list_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of continued object listings made, name=object_continue_list_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload initiated, name=object_multipart_initiated, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload initiated, name=object_multipart_initiated.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload aborted, name=object_multipart_aborted, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object multipart upload aborted, name=object_multipart_aborted.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object put/multipart upload count, name=object_put_request, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Object put/multipart upload count, name=object_put_request.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total queue duration of all block uploads, name=stream_write_queue_duration, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Total queue duration of all block uploads, name=stream_write_queue_duration.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to commit an entire job, name=committer_commit_job, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to commit an entire job, name=committer_commit_job.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to materialize a file in job commit, name=committer_materialize_file, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of time to materialize a file in job commit, name=committer_materialize_file.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of files uploaded from a local staging path, name=committer_stage_file_upload, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Duration Tracking of files uploaded from a local staging path, name=committer_stage_file_upload.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Store Existence Probe, name=store_exists_probe, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Store Existence Probe, name=store_exists_probe.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of delegation tokens issued, name=delegation_tokens_issued, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Count of delegation tokens issued, name=delegation_tokens_issued.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload List, name=multipart_upload_list, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Multipart Upload List, name=multipart_upload_list.failures, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of ops for s3Guard metadata store put one metadata path latency with 1s interval, name=S3guard_metadatastore_put_path_latencyNumOps, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=50 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency50thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=75 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency75thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=90 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency90thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=95 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency95thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=99 percentile latency with 1 second interval for s3Guard metadata store put one metadata path latency, name=S3guard_metadatastore_put_path_latency99thPercentileLatency, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of events for s3Guard metadata store throttle rate with 1s interval, name=S3guard_metadatastore_throttle_rateNumEvents, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=50 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate50thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=75 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate75thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=90 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate90thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=95 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate95thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=99 percentile frequency (Hz) with 1 second interval for s3Guard metadata store throttle rate, name=S3guard_metadatastore_throttle_rate99thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Number of events for rate of S3 request throttling with 1s interval, name=Store_io_throttle_rateNumEvents, type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=50 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate50thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=75 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate75thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=90 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate90thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=95 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate95thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=99 percentile frequency (Hz) with 1 second interval for rate of S3 request throttling, name=Store_io_throttle_rate99thPercentileFrequency (Hz), type=java.lang.Long, read-only, descriptor={}]]
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: Done
22/08/12 16:24:46 DEBUG MBeans: Registered Hadoop:service=s3a-file-system,name=S3AMetrics1-master.rando2
22/08/12 16:24:46 DEBUG MetricsSourceAdapter: MBean for source S3AMetrics1-master.rando2 registered.
22/08/12 16:24:46 DEBUG MetricsSystemImpl: Registered source S3AMetrics1-master.rando2
22/08/12 16:24:46 DEBUG S3AFileSystem: Client Side Encryption enabled: false
22/08/12 16:24:46 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms
22/08/12 16:24:46 DEBUG S3GuardExistsRetryPolicy: Retrying on recoverable S3Guard table/S3 inconsistencies 7 times with an initial interval of 2000ms
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.paging.maximum is 5000
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.block.size is 33554432
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.readahead.range is 65536
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.max.total.tasks is 32
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.threads.keepalivetime is 60
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.executor.capacity is 16
22/08/12 16:24:46 DEBUG SignerManager: No custom signers specified
22/08/12 16:24:46 DEBUG AuditIntegration: auditing is disabled
22/08/12 16:24:46 DEBUG AbstractService: Service: NoopAuditManagerS3A entered state INITED
22/08/12 16:24:46 DEBUG CompositeService: NoopAuditManagerS3A: initing services, size=0
22/08/12 16:24:46 DEBUG CompositeService: Adding service NoopAuditor
22/08/12 16:24:46 DEBUG AbstractService: Service: NoopAuditor entered state INITED
22/08/12 16:24:46 DEBUG CompositeService: NoopAuditManagerS3A: starting services, size=1
22/08/12 16:24:46 DEBUG AbstractService: Service NoopAuditor is started
22/08/12 16:24:46 DEBUG AbstractService: Service NoopAuditManagerS3A is started
22/08/12 16:24:46 DEBUG AuditIntegration: Started Audit Manager Service NoopAuditManagerS3A in state NoopAuditManagerS3A: STARTED
22/08/12 16:24:46 DEBUG S3AUtils: Value of fs.s3a.internal.upload.part.count.limit is 10000
22/08/12 16:24:46 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms
22/08/12 16:24:46 DEBUG S3AUtils: Credential provider class is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
22/08/12 16:24:46 DEBUG S3AUtils: Credential provider class is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
22/08/12 16:24:46 DEBUG S3AUtils: Credential provider class is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
22/08/12 16:24:46 DEBUG S3AUtils: Credential provider class is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
22/08/12 16:24:47 DEBUG S3AUtils: For URI s3a://master.rando2/nonemptyprefix8, using credentials AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@717fe440]
22/08/12 16:24:47 DEBUG S3AFileSystem: Using credential provider AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@717fe440]
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.connection.maximum is 96
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.attempts.maximum is 20
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.connection.establish.timeout is 5000
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.connection.timeout is 200000
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.socket.send.buffer is 8192
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.socket.recv.buffer is 8192
22/08/12 16:24:47 DEBUG S3AUtils: Using User-Agent: Hadoop 3.3.2
22/08/12 16:24:47 DEBUG S3AUtils: Data is unencrypted
22/08/12 16:24:47 DEBUG AmazonWebServiceClient: Internal logging successfully configured to commons logger: true
22/08/12 16:24:47 DEBUG AwsSdkMetrics: Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics
22/08/12 16:24:47 DEBUG DefaultS3ClientFactory: Creating endpoint configuration for "http://localhost:30600"
22/08/12 16:24:47 DEBUG DefaultS3ClientFactory: Endpoint URI = http://localhost:30600
22/08/12 16:24:47 DEBUG DefaultS3ClientFactory: Endpoint http://localhost:30600 is not the default; parsing
22/08/12 16:24:47 DEBUG DefaultS3ClientFactory: Region for endpoint http://localhost:30600, URI http://localhost:30600 is determined as null
22/08/12 16:24:47 DEBUG FsUrlStreamHandlerFactory: Creating handler for protocol http
22/08/12 16:24:47 DEBUG FsUrlStreamHandlerFactory: Unknown protocol http, delegating to default implementation
22/08/12 16:24:47 DEBUG CsmConfigurationProviderChain: Unable to load configuration from com.amazonaws.monitoring.EnvironmentVariableCsmConfigurationProvider@6e2dc702: Unable to load Client Side Monitoring configurations from environment variables!
22/08/12 16:24:47 DEBUG CsmConfigurationProviderChain: Unable to load configuration from com.amazonaws.monitoring.SystemPropertyCsmConfigurationProvider@52ebaf37: Unable to load Client Side Monitoring configurations from system properties variables!
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Closing connections idle longer than 60000 MILLISECONDS
22/08/12 16:24:47 DEBUG CsmConfigurationProviderChain: Unable to load configuration from com.amazonaws.monitoring.ProfileCsmConfigurationProvider@9ad30ff: The 'default' profile does not define all the required properties!
22/08/12 16:24:47 DEBUG S3AFileSystem: skipping check for bucket existence
22/08/12 16:24:47 DEBUG S3AFileSystem: Input fadvise policy = normal
22/08/12 16:24:47 DEBUG S3AFileSystem: Change detection policy = ETagChangeDetectionPolicy mode=None
22/08/12 16:24:47 DEBUG S3AFileSystem: Filesystem support for magic committers is enabled
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.fast.upload.active.blocks is 4
22/08/12 16:24:47 DEBUG S3AFileSystem: Using S3ABlockOutputStream with buffer = disk; block=67108864; queue limit=4
22/08/12 16:24:47 DEBUG S3Guard: Metastore option source [core-default.xml]
22/08/12 16:24:47 DEBUG S3Guard: Using NullMetadataStore metadata store for s3a filesystem
22/08/12 16:24:47 DEBUG S3AFileSystem: S3Guard is disabled on this bucket: master.rando2
22/08/12 16:24:47 DEBUG DirectoryPolicyImpl: Directory markers will be deleted
22/08/12 16:24:47 DEBUG S3AFileSystem: Directory marker retention policy is DirectoryMarkerRetention{policy='delete'}
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.multipart.purge.age is 86400
22/08/12 16:24:47 DEBUG S3AUtils: Value of fs.s3a.bulk.delete.page.size is 250
22/08/12 16:24:47 DEBUG FileSystem: Creating FS s3a://master.rando2/nonemptyprefix8: duration 0:00.715s
22/08/12 16:24:47 DEBUG FileCommitProtocol: Creating committer org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol; job 31b8f587-d086-4d3b-bf6d-4ce13ea2fecc; output=s3a://master.rando2/nonemptyprefix8; dynamic=false
22/08/12 16:24:47 DEBUG FileCommitProtocol: Using (String, String, Boolean) constructor
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 1
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8 (nonemptyprefix8); needEmptyDirectory=false
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 1
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 1
22/08/12 16:24:47 DEBUG S3AFileSystem: HEAD nonemptyprefix8 with change tracker null
22/08/12 16:24:47 DEBUG Invoker: Starting: create credentials
22/08/12 16:24:47 DEBUG Invoker: create credentials: duration 0:00.004s
22/08/12 16:24:47 DEBUG AWSCredentialProviderList: No credentials from TemporaryAWSCredentialsProvider: org.apache.hadoop.fs.s3a.auth.NoAwsCredentialsException: Session credentials in Hadoop configuration: No AWS Credentials
22/08/12 16:24:47 DEBUG AWSCredentialProviderList: Using credentials from SimpleAWSCredentialsProvider
22/08/12 16:24:47 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8 Headers: (amz-sdk-invocation-id: daff2a9c-2d42-a289-c168-fd308a999145, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix8
amz-sdk-invocation-id:daff2a9c-2d42-a289-c168-fd308a999145
amz-sdk-request:attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152447Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152447Z
20220812/us-east-1/s3/aws4_request
09eb0b140b531b6a30b0a76354ce47f4c0aa77ef031f1d8595e5f14111629d1b"
22/08/12 16:24:47 DEBUG AWS4Signer: Generating a new signing key as the signing key not available in the cache for the date 1660262400000
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 0; route allocated: 0 of 96; total allocated: 0 of 96]
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG MainClientExec: Opening connection {}->http://localhost:30600
22/08/12 16:24:47 DEBUG DefaultHttpClientConnectionOperator: Connecting to localhost/127.0.0.1:30600
22/08/12 16:24:47 DEBUG DefaultHttpClientConnectionOperator: Connection established 127.0.0.1:59728<->127.0.0.1:30600
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8 HTTP/1.1
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8 HTTP/1.1
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: daff2a9c-2d42-a289-c168-fd308a999145
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: attempt=1;max=21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=7a1f1dab9b137bd736489828a6596cc4039d538e9e0c24fca5853bef77d74cd5
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8 HTTP/1.1[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: daff2a9c-2d42-a289-c168-fd308a999145[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: attempt=1;max=21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=7a1f1dab9b137bd736489828a6596cc4039d538e9e0c24fca5853bef77d74cd5[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 3daab1a0-6106-4800-bbde-d1a1b7688a30[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 3daab1a0-6106-4800-bbde-d1a1b7688a30[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 238[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 3daab1a0-6106-4800-bbde-d1a1b7688a30
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 3daab1a0-6106-4800-bbde-d1a1b7688a30
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 238
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30; S3 Extended Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30; Proxy: null), S3 Extended Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 1
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30; S3 Extended Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30; Proxy: null), S3 Extended Request ID: 3daab1a0-6106-4800-bbde-d1a1b7688a30], RequestType=[GetObjectMetadataRequest], AWSRequestID=[3daab1a0-6106-4800-bbde-d1a1b7688a30], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=0, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[136.637], HttpClientSendRequestTime=[2.668], HttpRequestTime=[69.853], ApiCallLatency=[120.482], RequestSigningTime=[32.408], CredentialsRequestTime=[5.573, 0.016], HttpClientReceiveResponseTime=[18.711],
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/ delimiter=/ keys=2 requester pays=false
22/08/12 16:24:47 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 1
22/08/12 16:24:47 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 22c79b53-2c3d-5be0-a595-a586ebb84541, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F
amz-sdk-invocation-id:22c79b53-2c3d-5be0-a595-a586ebb84541
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152447Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152447Z
20220812/us-east-1/s3/aws4_request
019acf4980f1ae26885efbc8e4913bd8ac4c94c2eaa3ccf2f282556c4ef43fb8"
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 22c79b53-2c3d-5be0-a595-a586ebb84541
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=359683b5b489e76389fdb97672339667f5e7a0471f09890bbe933ab7c1b410e6
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 22c79b53-2c3d-5be0-a595-a586ebb84541[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=359683b5b489e76389fdb97672339667f5e7a0471f09890bbe933ab7c1b410e6[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: cd3e943f-8e26-426b-9953-1dea8efbcf46[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: cd3e943f-8e26-426b-9953-1dea8efbcf46[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 276[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/</Prefix></ListBucketResult>"
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: cd3e943f-8e26-426b-9953-1dea8efbcf46
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: cd3e943f-8e26-426b-9953-1dea8efbcf46
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 276
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:47 DEBUG request: Received successful response: 200, AWS Request ID: cd3e943f-8e26-426b-9953-1dea8efbcf46
22/08/12 16:24:47 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:47 DEBUG requestId: AWS Request ID: cd3e943f-8e26-426b-9953-1dea8efbcf46
22/08/12 16:24:47 DEBUG requestId: AWS Extended Request ID: cd3e943f-8e26-426b-9953-1dea8efbcf46
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 2
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[cd3e943f-8e26-426b-9953-1dea8efbcf46], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[37.947], ClientExecuteTime=[80.903], HttpClientSendRequestTime=[1.577], HttpRequestTime=[23.382], ApiCallLatency=[80.641], RequestSigningTime=[0.526], CredentialsRequestTime=[0.008, 0.012], HttpClientReceiveResponseTime=[19.702],
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST: duration 0:00.083s
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 1
22/08/12 16:24:47 INFO ParquetFileFormat: Using default output committer for Parquet: org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 16:24:47 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624471812156148346658793_0000}; taskId=attempt_202208121624471812156148346658793_0000_m_000000_0, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@7b9e186f}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix8 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624471812156148346658793_0000}; taskId=attempt_202208121624471812156148346658793_0000_m_000000_0, status=''}
22/08/12 16:24:47 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 16:24:47 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 16:24:47 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 16:24:47 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624471812156148346658793_0000}; taskId=attempt_202208121624471812156148346658793_0000_m_000000_0, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@3ff4f746}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix8 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624471812156148346658793_0000}; taskId=attempt_202208121624471812156148346658793_0000_m_000000_0, status=''}
22/08/12 16:24:47 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 16:24:47 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 16:24:47 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter op_mkdirs by 1 with final value 1
22/08/12 16:24:47 DEBUG MkdirOperation: Making directory: s3a://master.rando2/nonemptyprefix8/_temporary/0
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0 (nonemptyprefix8/_temporary/0); needEmptyDirectory=false
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/ delimiter=/ keys=2 requester pays=false
22/08/12 16:24:47 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 2
22/08/12 16:24:47 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/0/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: a9501fd6-6372-0185-fe8b-85005b8d2afe, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F
amz-sdk-invocation-id:a9501fd6-6372-0185-fe8b-85005b8d2afe
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152447Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152447Z
20220812/us-east-1/s3/aws4_request
f4caaffa42a171e08d2295eab15dda7db3dac909fcde4b5507747310c177f6ee"
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: a9501fd6-6372-0185-fe8b-85005b8d2afe
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=caa695eb2a1be3f95e9101d66cb8389d82d23bdc653e4b1bdd50537606be4705
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: a9501fd6-6372-0185-fe8b-85005b8d2afe[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=caa695eb2a1be3f95e9101d66cb8389d82d23bdc653e4b1bdd50537606be4705[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 6aafe837-cd7b-40b7-838a-f3089fe62a59[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 6aafe837-cd7b-40b7-838a-f3089fe62a59[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 289[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/</Prefix></ListBucketResult>"
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 6aafe837-cd7b-40b7-838a-f3089fe62a59
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 6aafe837-cd7b-40b7-838a-f3089fe62a59
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 289
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:47 DEBUG request: Received successful response: 200, AWS Request ID: 6aafe837-cd7b-40b7-838a-f3089fe62a59
22/08/12 16:24:47 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:47 DEBUG requestId: AWS Request ID: 6aafe837-cd7b-40b7-838a-f3089fe62a59
22/08/12 16:24:47 DEBUG requestId: AWS Extended Request ID: 6aafe837-cd7b-40b7-838a-f3089fe62a59
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 3
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[6aafe837-cd7b-40b7-838a-f3089fe62a59], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[2.28], ClientExecuteTime=[26.321], HttpClientSendRequestTime=[1.173], HttpRequestTime=[22.087], ApiCallLatency=[26.062], RequestSigningTime=[0.581], CredentialsRequestTime=[0.009, 0.004], HttpClientReceiveResponseTime=[19.775],
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST: duration 0:00.027s
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0 (nonemptyprefix8/_temporary/0); needEmptyDirectory=false
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 2
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 2
22/08/12 16:24:47 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary/0 with change tracker null
22/08/12 16:24:47 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0 Headers: (amz-sdk-invocation-id: e8b4813b-fc7a-8967-7924-64c90534fc39, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix8/_temporary/0
amz-sdk-invocation-id:e8b4813b-fc7a-8967-7924-64c90534fc39
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152447Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152447Z
20220812/us-east-1/s3/aws4_request
eaf3b5deb5e1b4da4d9933c57201ae5d00e2b93686fc151bf229f841c3d7ef43"
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: e8b4813b-fc7a-8967-7924-64c90534fc39
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=73372ffd119ab8114dac440234fc87dcb379c69f423c1088ebf21092968a77cd
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: e8b4813b-fc7a-8967-7924-64c90534fc39[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=73372ffd119ab8114dac440234fc87dcb379c69f423c1088ebf21092968a77cd[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: ff19ddf7-5f6c-4726-9cc9-db2290704a16[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: ff19ddf7-5f6c-4726-9cc9-db2290704a16[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 251[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: ff19ddf7-5f6c-4726-9cc9-db2290704a16
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: ff19ddf7-5f6c-4726-9cc9-db2290704a16
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 251
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16; S3 Extended Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16; Proxy: null), S3 Extended Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 4
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16; S3 Extended Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16; Proxy: null), S3 Extended Request ID: ff19ddf7-5f6c-4726-9cc9-db2290704a16], RequestType=[GetObjectMetadataRequest], AWSRequestID=[ff19ddf7-5f6c-4726-9cc9-db2290704a16], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[22.965], HttpClientSendRequestTime=[0.845], HttpRequestTime=[21.581], ApiCallLatency=[22.746], RequestSigningTime=[0.311], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[19.578],
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary (nonemptyprefix8/_temporary); needEmptyDirectory=false
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/ delimiter=/ keys=2 requester pays=false
22/08/12 16:24:47 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 3
22/08/12 16:24:47 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 6f197e9e-ece8-147a-639b-49f0ce133788, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F
amz-sdk-invocation-id:6f197e9e-ece8-147a-639b-49f0ce133788
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152447Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152447Z
20220812/us-east-1/s3/aws4_request
4a87fed889492bff077761a34fbd8ca98c2c69f7ef878d776f8583cd8f2e691a"
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 6f197e9e-ece8-147a-639b-49f0ce133788
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=862667900d080a98e9cbbae8321a52d307bf7a38375c16e85475d94b6c426ff9
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 6f197e9e-ece8-147a-639b-49f0ce133788[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=862667900d080a98e9cbbae8321a52d307bf7a38375c16e85475d94b6c426ff9[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 91414309-86a8-4b18-af1b-2b7f37d37b21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 91414309-86a8-4b18-af1b-2b7f37d37b21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 287[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/</Prefix></ListBucketResult>"
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 91414309-86a8-4b18-af1b-2b7f37d37b21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 91414309-86a8-4b18-af1b-2b7f37d37b21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 287
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:47 DEBUG request: Received successful response: 200, AWS Request ID: 91414309-86a8-4b18-af1b-2b7f37d37b21
22/08/12 16:24:47 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:47 DEBUG requestId: AWS Request ID: 91414309-86a8-4b18-af1b-2b7f37d37b21
22/08/12 16:24:47 DEBUG requestId: AWS Extended Request ID: 91414309-86a8-4b18-af1b-2b7f37d37b21
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 5
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[91414309-86a8-4b18-af1b-2b7f37d37b21], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.872], ClientExecuteTime=[22.39], HttpClientSendRequestTime=[1.189], HttpRequestTime=[18.897], ApiCallLatency=[22.139], RequestSigningTime=[0.55], CredentialsRequestTime=[0.007, 0.005], HttpClientReceiveResponseTime=[16.639],
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST: duration 0:00.023s
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary (nonemptyprefix8/_temporary); needEmptyDirectory=false
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 3
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 3
22/08/12 16:24:47 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary with change tracker null
22/08/12 16:24:47 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary Headers: (amz-sdk-invocation-id: 51b43d6b-15fa-d3a7-0f7c-294df2b09065, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix8/_temporary
amz-sdk-invocation-id:51b43d6b-15fa-d3a7-0f7c-294df2b09065
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152447Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152447Z
20220812/us-east-1/s3/aws4_request
edc7e55f1ec0754e6c860fee6c88f79b470b0b5144e6d8c332c6b1c1b4d57f03"
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 51b43d6b-15fa-d3a7-0f7c-294df2b09065
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2574e43ca3e37e49613b59ca5b2ea39f6745f8b8c29c8e940813a73669a48211
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 51b43d6b-15fa-d3a7-0f7c-294df2b09065[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2574e43ca3e37e49613b59ca5b2ea39f6745f8b8c29c8e940813a73669a48211[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: b803373f-f269-4997-a6dd-b1da6a68a524[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: b803373f-f269-4997-a6dd-b1da6a68a524[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 249[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: b803373f-f269-4997-a6dd-b1da6a68a524
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: b803373f-f269-4997-a6dd-b1da6a68a524
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 249
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: b803373f-f269-4997-a6dd-b1da6a68a524; S3 Extended Request ID: b803373f-f269-4997-a6dd-b1da6a68a524; Proxy: null), S3 Extended Request ID: b803373f-f269-4997-a6dd-b1da6a68a524
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 6
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: b803373f-f269-4997-a6dd-b1da6a68a524; S3 Extended Request ID: b803373f-f269-4997-a6dd-b1da6a68a524; Proxy: null), S3 Extended Request ID: b803373f-f269-4997-a6dd-b1da6a68a524], RequestType=[GetObjectMetadataRequest], AWSRequestID=[b803373f-f269-4997-a6dd-b1da6a68a524], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[16.192], HttpClientSendRequestTime=[0.69], HttpRequestTime=[15.04], ApiCallLatency=[16.003], RequestSigningTime=[0.313], CredentialsRequestTime=[0.005, 0.003], HttpClientReceiveResponseTime=[13.422],
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8 (nonemptyprefix8); needEmptyDirectory=false
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/ delimiter=/ keys=2 requester pays=false
22/08/12 16:24:47 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 4
22/08/12 16:24:47 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 7aa1f9d8-63be-c804-2d80-759c8c87cc8d, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F
amz-sdk-invocation-id:7aa1f9d8-63be-c804-2d80-759c8c87cc8d
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152447Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152447Z
20220812/us-east-1/s3/aws4_request
b37845cd551c8cb5d2915212bbf0f39774dd03e3ab428c5cf88d16fdb5838b6e"
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 7aa1f9d8-63be-c804-2d80-759c8c87cc8d
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=79eb3148c7e5c30911c8b005724da88280c6e23d24e8e0ef46ed56b6584a8d02
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 7aa1f9d8-63be-c804-2d80-759c8c87cc8d[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=79eb3148c7e5c30911c8b005724da88280c6e23d24e8e0ef46ed56b6584a8d02[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 6131f265-28b9-4d23-a8f1-cc70617cb6af[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 6131f265-28b9-4d23-a8f1-cc70617cb6af[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 276[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/</Prefix></ListBucketResult>"
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 6131f265-28b9-4d23-a8f1-cc70617cb6af
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 6131f265-28b9-4d23-a8f1-cc70617cb6af
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 276
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:47 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:47 DEBUG request: Received successful response: 200, AWS Request ID: 6131f265-28b9-4d23-a8f1-cc70617cb6af
22/08/12 16:24:47 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:47 DEBUG requestId: AWS Request ID: 6131f265-28b9-4d23-a8f1-cc70617cb6af
22/08/12 16:24:47 DEBUG requestId: AWS Extended Request ID: 6131f265-28b9-4d23-a8f1-cc70617cb6af
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 7
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[6131f265-28b9-4d23-a8f1-cc70617cb6af], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.891], ClientExecuteTime=[17.214], HttpClientSendRequestTime=[0.746], HttpRequestTime=[14.048], ApiCallLatency=[17.015], RequestSigningTime=[0.384], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[12.559],
22/08/12 16:24:47 DEBUG S3AFileSystem: LIST: duration 0:00.018s
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8
22/08/12 16:24:47 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8 (nonemptyprefix8); needEmptyDirectory=false
22/08/12 16:24:47 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 4
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 4
22/08/12 16:24:47 DEBUG S3AFileSystem: HEAD nonemptyprefix8 with change tracker null
22/08/12 16:24:47 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8 Headers: (amz-sdk-invocation-id: 916bbbb3-be37-d245-ea23-ebeb9132a28b, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix8
amz-sdk-invocation-id:916bbbb3-be37-d245-ea23-ebeb9132a28b
amz-sdk-request:ttl=20220812T152807Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152447Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152447Z
20220812/us-east-1/s3/aws4_request
eaf3b75856e032fd560c7786eedad643c3aeec9f6ed34755f79636dfc263fbac"
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8 HTTP/1.1
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8 HTTP/1.1
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 916bbbb3-be37-d245-ea23-ebeb9132a28b
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=4bebe31525db92dca60f96ba1a28fde085378b487ee8cdf8ea5bf9e3b98397a7
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8 HTTP/1.1[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 916bbbb3-be37-d245-ea23-ebeb9132a28b[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152807Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=4bebe31525db92dca60f96ba1a28fde085378b487ee8cdf8ea5bf9e3b98397a7[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 38a86ef9-0270-4ffc-a709-d68818fccdaa[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 38a86ef9-0270-4ffc-a709-d68818fccdaa[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:47 GMT[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "Content-Length: 238[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 38a86ef9-0270-4ffc-a709-d68818fccdaa
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 38a86ef9-0270-4ffc-a709-d68818fccdaa
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << Content-Length: 238
22/08/12 16:24:47 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:47 GMT
22/08/12 16:24:47 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa; S3 Extended Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa; Proxy: null), S3 Extended Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 8
22/08/12 16:24:47 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa; S3 Extended Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa; Proxy: null), S3 Extended Request ID: 38a86ef9-0270-4ffc-a709-d68818fccdaa], RequestType=[GetObjectMetadataRequest], AWSRequestID=[38a86ef9-0270-4ffc-a709-d68818fccdaa], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[15.768], HttpClientSendRequestTime=[0.575], HttpRequestTime=[14.201], ApiCallLatency=[15.56], RequestSigningTime=[0.284], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[12.65],
22/08/12 16:24:47 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8
22/08/12 16:24:47 DEBUG Invoker: Starting: PUT 0-byte object
22/08/12 16:24:47 DEBUG S3AFileSystem: PUT 0 bytes to nonemptyprefix8/_temporary/0/
22/08/12 16:24:47 DEBUG S3AFileSystem: PUT start 0 bytes
22/08/12 16:24:47 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request by 1 with final value 1
22/08/12 16:24:47 DEBUG request: Sending Request: PUT http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0/ Headers: (amz-sdk-invocation-id: 798a1364-fafa-c571-a88f-423c786d6d8f, Content-Length: 0, Content-Type: application/x-directory, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 Canonical Request: '"PUT
/master.rando2/nonemptyprefix8/_temporary/0/
amz-sdk-invocation-id:798a1364-fafa-c571-a88f-423c786d6d8f
amz-sdk-request:attempt=1;max=21
amz-sdk-retry:0/0/500
content-length:86
content-type:application/x-directory
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20220812T152447Z
x-amz-decoded-content-length:0
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD"
22/08/12 16:24:47 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152447Z
20220812/us-east-1/s3/aws4_request
e148dd69e9e93d8a12a1c71aa0bda527f3f418edfce3dc8ddd0da013f327ca3a"
22/08/12 16:24:47 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:47 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:47 DEBUG MainClientExec: Executing request PUT /master.rando2/nonemptyprefix8/_temporary/0/ HTTP/1.1
22/08/12 16:24:47 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> PUT /master.rando2/nonemptyprefix8/_temporary/0/ HTTP/1.1
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 798a1364-fafa-c571-a88f-423c786d6d8f
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-request: attempt=1;max=21
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=7d6cd2284b14c1caccea796a2a3f89b0bc7b3d3ae388a85eea793152e131fbc4
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Type: application/x-directory
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152447Z
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> x-amz-decoded-content-length: 0
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Content-Length: 86
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 >> Expect: 100-continue
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "PUT /master.rando2/nonemptyprefix8/_temporary/0/ HTTP/1.1[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 798a1364-fafa-c571-a88f-423c786d6d8f[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: attempt=1;max=21[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=7d6cd2284b14c1caccea796a2a3f89b0bc7b3d3ae388a85eea793152e131fbc4[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Type: application/x-directory[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152447Z[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "x-amz-decoded-content-length: 0[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Content-Length: 86[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "Expect: 100-continue[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "HTTP/1.1 100 Continue[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:47 DEBUG headers: http-outgoing-0 << HTTP/1.1 100 Continue
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "0;chunk-signature=1417326b39cf805f273fb1b4b19e48c7a14f8b1e9acc0f1572f8b6f6b3d0c1c4[\r][\n]"
22/08/12 16:24:47 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Etag: "0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8"[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Version-Id: 2d00fbfd57b840f0863941d59e62cdad[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:48 GMT[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Length: 0[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Etag: "0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8"
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Version-Id: 2d00fbfd57b840f0863941d59e62cdad
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:48 GMT
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Length: 0
22/08/12 16:24:48 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:48 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:48 GMT
22/08/12 16:24:48 DEBUG request: Received successful response: 200, AWS Request ID: null
22/08/12 16:24:48 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:48 DEBUG requestId: AWS Request ID: not available
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 9
22/08/12 16:24:48 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[PutObjectRequest], AWSRequestID=[null], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[0.315], ClientExecuteTime=[186.683], HttpClientSendRequestTime=[3.475], HttpRequestTime=[182.177], ApiCallLatency=[186.265], RequestSigningTime=[1.006], CredentialsRequestTime=[0.009, 0.005], HttpClientReceiveResponseTime=[177.452],
22/08/12 16:24:48 DEBUG S3AFileSystem: PUT completed success=true; 0 bytes
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request_completed by 1 with final value 1
22/08/12 16:24:48 DEBUG S3AFileSystem: Finished write to nonemptyprefix8/_temporary/0/, len 0. etag 0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8, version 2d00fbfd57b840f0863941d59e62cdad
22/08/12 16:24:48 DEBUG CallableSupplier: Starting: Waiting for task completion
22/08/12 16:24:48 DEBUG S3AFileSystem: Starting: Deleting 2 keys
22/08/12 16:24:48 DEBUG S3AFileSystem: Initiating delete operation for 2 objects
22/08/12 16:24:48 DEBUG S3AFileSystem: nonemptyprefix8/_temporary/
22/08/12 16:24:48 DEBUG S3AFileSystem: nonemptyprefix8/
22/08/12 16:24:48 DEBUG S3AFileSystem: Starting: DELETE 2 keys
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter object_bulk_delete_request by 1 with final value 1
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter object_delete_objects by 2 with final value 2
22/08/12 16:24:48 DEBUG request: Sending Request: POST http://localhost:30600 /master.rando2/ Parameters: ({"delete":[null]}Headers: (amz-sdk-invocation-id: f4f9232d-9938-70d8-c32c-9c611681ca83, Content-Length: 135, Content-MD5: xFbx26Nz2FGJIP9zZ3RhAg==, Content-Type: application/xml, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 Canonical Request: '"POST
/master.rando2/
delete=
amz-sdk-invocation-id:f4f9232d-9938-70d8-c32c-9c611681ca83
amz-sdk-request:ttl=20220812T152808Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-length:135
content-md5:xFbx26Nz2FGJIP9zZ3RhAg==
content-type:application/xml
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:16f9f85365f43ddbe703d6dd461ae2d8d9ef1785e41655b9c0d396a819924aed
x-amz-date:20220812T152448Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
16f9f85365f43ddbe703d6dd461ae2d8d9ef1785e41655b9c0d396a819924aed"
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152448Z
20220812/us-east-1/s3/aws4_request
e99663c839db631169021f8ed01103d6016b2b9ea36cb07379745ab0ecafea7b"
22/08/12 16:24:48 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:48 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:48 DEBUG MainClientExec: Executing request POST /master.rando2/?delete HTTP/1.1
22/08/12 16:24:48 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> POST /master.rando2/?delete HTTP/1.1
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: f4f9232d-9938-70d8-c32c-9c611681ca83
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=73cf3673181055080f1d69875af40ac920a41fe7f1366c449457754ab3c59832
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-MD5: xFbx26Nz2FGJIP9zZ3RhAg==
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-Type: application/xml
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: 16f9f85365f43ddbe703d6dd461ae2d8d9ef1785e41655b9c0d396a819924aed
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152448Z
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-Length: 135
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "POST /master.rando2/?delete HTTP/1.1[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: f4f9232d-9938-70d8-c32c-9c611681ca83[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=73cf3673181055080f1d69875af40ac920a41fe7f1366c449457754ab3c59832[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-MD5: xFbx26Nz2FGJIP9zZ3RhAg==[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: 16f9f85365f43ddbe703d6dd461ae2d8d9ef1785e41655b9c0d396a819924aed[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152448Z[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-Length: 135[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "<Delete><Quiet>true</Quiet><Object><Key>nonemptyprefix8/_temporary/</Key></Object><Object><Key>nonemptyprefix8/</Key></Object></Delete>"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 7d9090db-110a-4ee5-8c26-ef863774f9d4[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 7d9090db-110a-4ee5-8c26-ef863774f9d4[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:48 GMT[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Length: 116[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>"
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 7d9090db-110a-4ee5-8c26-ef863774f9d4
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 7d9090db-110a-4ee5-8c26-ef863774f9d4
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:48 GMT
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Length: 116
22/08/12 16:24:48 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:48 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:48 GMT
22/08/12 16:24:48 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$DeleteObjectsHandler
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:48 DEBUG request: Received successful response: 200, AWS Request ID: 7d9090db-110a-4ee5-8c26-ef863774f9d4
22/08/12 16:24:48 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:48 DEBUG requestId: AWS Request ID: 7d9090db-110a-4ee5-8c26-ef863774f9d4
22/08/12 16:24:48 DEBUG requestId: AWS Extended Request ID: 7d9090db-110a-4ee5-8c26-ef863774f9d4
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 10
22/08/12 16:24:48 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[DeleteObjectsRequest], AWSRequestID=[7d9090db-110a-4ee5-8c26-ef863774f9d4], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[2.522], ClientExecuteTime=[50.971], HttpClientSendRequestTime=[0.714], HttpRequestTime=[45.489], ApiCallLatency=[50.678], RequestSigningTime=[1.65], CredentialsRequestTime=[0.009, 0.004], HttpClientReceiveResponseTime=[43.856],
22/08/12 16:24:48 DEBUG S3AFileSystem: DELETE 2 keys: duration 0:00.122s
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter fake_directories_deleted by 2 with final value 2
22/08/12 16:24:48 DEBUG S3AFileSystem: Deleting 2 keys: duration 0:00.125s
22/08/12 16:24:48 DEBUG CallableSupplier: Waiting for task completion: duration 0:00.126s
22/08/12 16:24:48 DEBUG Invoker: PUT 0-byte object : duration 0:00.323s
22/08/12 16:24:48 DEBUG Progress: PUT nonemptyprefix8/_temporary/0: 0 bytes
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter directories_created by 1 with final value 1
22/08/12 16:24:48 DEBUG WholeStageCodegenExec:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private scala.collection.Iterator rdd_input_0;
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1];
/* 011 */
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) {
/* 013 */ this.references = references;
/* 014 */ }
/* 015 */
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) {
/* 017 */ partitionIndex = index;
/* 018 */ this.inputs = inputs;
/* 019 */ rdd_input_0 = inputs[0];
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0);
/* 021 */
/* 022 */ }
/* 023 */
/* 024 */ protected void processNext() throws java.io.IOException {
/* 025 */ while ( rdd_input_0.hasNext()) {
/* 026 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next();
/* 027 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 028 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0);
/* 029 */ long rdd_value_0 = rdd_isNull_0 ?
/* 030 */ -1L : (rdd_row_0.getLong(0));
/* 031 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1);
/* 032 */ double rdd_value_1 = rdd_isNull_1 ?
/* 033 */ -1.0 : (rdd_row_0.getDouble(1));
/* 034 */ rdd_mutableStateArray_0[0].reset();
/* 035 */
/* 036 */ rdd_mutableStateArray_0[0].zeroOutNullBytes();
/* 037 */
/* 038 */ if (rdd_isNull_0) {
/* 039 */ rdd_mutableStateArray_0[0].setNullAt(0);
/* 040 */ } else {
/* 041 */ rdd_mutableStateArray_0[0].write(0, rdd_value_0);
/* 042 */ }
/* 043 */
/* 044 */ if (rdd_isNull_1) {
/* 045 */ rdd_mutableStateArray_0[0].setNullAt(1);
/* 046 */ } else {
/* 047 */ rdd_mutableStateArray_0[0].write(1, rdd_value_1);
/* 048 */ }
/* 049 */ append((rdd_mutableStateArray_0[0].getRow()));
/* 050 */ if (shouldStop()) return;
/* 051 */ }
/* 052 */ }
/* 053 */
/* 054 */ }
22/08/12 16:24:48 DEBUG CodeGenerator:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */ private Object[] references;
/* 008 */ private scala.collection.Iterator[] inputs;
/* 009 */ private scala.collection.Iterator rdd_input_0;
/* 010 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] rdd_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1];
/* 011 */
/* 012 */ public GeneratedIteratorForCodegenStage1(Object[] references) {
/* 013 */ this.references = references;
/* 014 */ }
/* 015 */
/* 016 */ public void init(int index, scala.collection.Iterator[] inputs) {
/* 017 */ partitionIndex = index;
/* 018 */ this.inputs = inputs;
/* 019 */ rdd_input_0 = inputs[0];
/* 020 */ rdd_mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(2, 0);
/* 021 */
/* 022 */ }
/* 023 */
/* 024 */ protected void processNext() throws java.io.IOException {
/* 025 */ while ( rdd_input_0.hasNext()) {
/* 026 */ InternalRow rdd_row_0 = (InternalRow) rdd_input_0.next();
/* 027 */ ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(1);
/* 028 */ boolean rdd_isNull_0 = rdd_row_0.isNullAt(0);
/* 029 */ long rdd_value_0 = rdd_isNull_0 ?
/* 030 */ -1L : (rdd_row_0.getLong(0));
/* 031 */ boolean rdd_isNull_1 = rdd_row_0.isNullAt(1);
/* 032 */ double rdd_value_1 = rdd_isNull_1 ?
/* 033 */ -1.0 : (rdd_row_0.getDouble(1));
/* 034 */ rdd_mutableStateArray_0[0].reset();
/* 035 */
/* 036 */ rdd_mutableStateArray_0[0].zeroOutNullBytes();
/* 037 */
/* 038 */ if (rdd_isNull_0) {
/* 039 */ rdd_mutableStateArray_0[0].setNullAt(0);
/* 040 */ } else {
/* 041 */ rdd_mutableStateArray_0[0].write(0, rdd_value_0);
/* 042 */ }
/* 043 */
/* 044 */ if (rdd_isNull_1) {
/* 045 */ rdd_mutableStateArray_0[0].setNullAt(1);
/* 046 */ } else {
/* 047 */ rdd_mutableStateArray_0[0].write(1, rdd_value_1);
/* 048 */ }
/* 049 */ append((rdd_mutableStateArray_0[0].getRow()));
/* 050 */ if (shouldStop()) return;
/* 051 */ }
/* 052 */ }
/* 053 */
/* 054 */ }
22/08/12 16:24:48 INFO CodeGenerator: Code generated in 23.15445 ms
22/08/12 16:24:48 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$doExecute$4$adapted
22/08/12 16:24:48 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$doExecute$4$adapted) is now cleaned +++
22/08/12 16:24:48 DEBUG ClosureCleaner: Cleaning indylambda closure: $anonfun$write$21
22/08/12 16:24:48 DEBUG ClosureCleaner: +++ indylambda closure ($anonfun$write$21) is now cleaned +++
22/08/12 16:24:48 INFO SparkContext: Starting job: save at NativeMethodAccessorImpl.java:0
22/08/12 16:24:48 DEBUG DAGScheduler: eagerlyComputePartitionsForRddAndAncestors for RDD 8 took 0.000745 seconds
22/08/12 16:24:48 DEBUG DAGScheduler: Merging stage rdd profiles: Set()
22/08/12 16:24:48 INFO DAGScheduler: Got job 3 (save at NativeMethodAccessorImpl.java:0) with 1 output partitions
22/08/12 16:24:48 INFO DAGScheduler: Final stage: ResultStage 3 (save at NativeMethodAccessorImpl.java:0)
22/08/12 16:24:48 INFO DAGScheduler: Parents of final stage: List()
22/08/12 16:24:48 INFO DAGScheduler: Missing parents: List()
22/08/12 16:24:48 DEBUG DAGScheduler: submitStage(ResultStage 3 (name=save at NativeMethodAccessorImpl.java:0;jobs=3))
22/08/12 16:24:48 DEBUG DAGScheduler: missing: List()
22/08/12 16:24:48 INFO DAGScheduler: Submitting ResultStage 3 (CoalescedRDD[8] at save at NativeMethodAccessorImpl.java:0), which has no missing parents
22/08/12 16:24:48 DEBUG DAGScheduler: submitMissingTasks(ResultStage 3)
22/08/12 16:24:48 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 215.2 KiB, free 434.1 MiB)
22/08/12 16:24:48 DEBUG BlockManager: Put block broadcast_3 locally took 1 ms
22/08/12 16:24:48 DEBUG BlockManager: Putting block broadcast_3 without replication took 1 ms
22/08/12 16:24:48 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 77.9 KiB, free 434.1 MiB)
22/08/12 16:24:48 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_3_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None)
22/08/12 16:24:48 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.1.255.235:38343 (size: 77.9 KiB, free: 434.3 MiB)
22/08/12 16:24:48 DEBUG BlockManagerMaster: Updated info of block broadcast_3_piece0
22/08/12 16:24:48 DEBUG BlockManager: Told master about block broadcast_3_piece0
22/08/12 16:24:48 DEBUG BlockManager: Put block broadcast_3_piece0 locally took 2 ms
22/08/12 16:24:48 DEBUG BlockManager: Putting block broadcast_3_piece0 without replication took 2 ms
22/08/12 16:24:48 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1513
22/08/12 16:24:48 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (CoalescedRDD[8] at save at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))
22/08/12 16:24:48 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks resource profile 0
22/08/12 16:24:48 DEBUG TaskSetManager: Epoch for TaskSet 3.0: 0
22/08/12 16:24:48 DEBUG TaskSetManager: Adding pending tasks took 0 ms
22/08/12 16:24:48 DEBUG TaskSetManager: Valid locality levels for TaskSet 3.0: NO_PREF, ANY
22/08/12 16:24:48 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_3.0, runningTasks: 0
22/08/12 16:24:48 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 16) (10.1.255.235, executor driver, partition 0, PROCESS_LOCAL, 5295 bytes) taskResourceAssignments Map()
22/08/12 16:24:48 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY
22/08/12 16:24:48 INFO Executor: Running task 0.0 in stage 3.0 (TID 16)
22/08/12 16:24:48 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 1
22/08/12 16:24:48 DEBUG BlockManager: Getting local block broadcast_3
22/08/12 16:24:48 DEBUG BlockManager: Level for block broadcast_3 is StorageLevel(disk, memory, deserialized, 1 replicas)
22/08/12 16:24:48 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624485385779855500771815_0003}; taskId=attempt_202208121624485385779855500771815_0003_m_000000_16, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@57e9a743}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix8 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624485385779855500771815_0003}; taskId=attempt_202208121624485385779855500771815_0003_m_000000_16, status=''}
22/08/12 16:24:48 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 16:24:48 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 16:24:48 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 16:24:48 DEBUG PathOutputCommitter: Instantiating committer FileOutputCommitter{PathOutputCommitter{context=TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624485385779855500771815_0003}; taskId=attempt_202208121624485385779855500771815_0003_m_000000_16, status=''}; org.apache.parquet.hadoop.ParquetOutputCommitter@3da9886}; outputPath=null, workPath=null, algorithmVersion=0, skipCleanup=false, ignoreCleanupFailures=false} with output path s3a://master.rando2/nonemptyprefix8 and job context TaskAttemptContextImpl{JobContextImpl{jobId=job_202208121624485385779855500771815_0003}; taskId=attempt_202208121624485385779855500771815_0003_m_000000_16, status=''}
22/08/12 16:24:48 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
22/08/12 16:24:48 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
22/08/12 16:24:48 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
22/08/12 16:24:48 INFO CodecConfig: Compression: SNAPPY
22/08/12 16:24:48 INFO CodecConfig: Compression: SNAPPY
22/08/12 16:24:48 INFO ParquetOutputFormat: Parquet block size to 134217728
22/08/12 16:24:48 INFO ParquetOutputFormat: Validation is off
22/08/12 16:24:48 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
22/08/12 16:24:48 INFO ParquetOutputFormat: Parquet properties are:
Parquet page size to 1048576
Parquet dictionary page size to 1048576
Dictionary is true
Writer version is: PARQUET_1_0
Page size checking is: estimated
Min row count for page size check is: 100
Max row count for page size check is: 10000
Truncate length for column indexes is: 64
Truncate length for statistics min/max is: 2147483647
Bloom filter enabled: false
Max Bloom filter size for a column is 1048576
Bloom filter expected number of distinct values are: null
Page row count limit to 20000
Writing page checksums is: on
22/08/12 16:24:48 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "a",
"type" : "long",
"nullable" : true,
"metadata" : { }
}, {
"name" : "b",
"type" : "double",
"nullable" : true,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
optional int64 a;
optional double b;
}
22/08/12 16:24:48 DEBUG EncryptionPropertiesFactory: EncryptionPropertiesFactory is not configured - name not found in hadoop config
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter op_create by 1 with final value 1
22/08/12 16:24:48 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet (nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet); needEmptyDirectory=false
22/08/12 16:24:48 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 5
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 5
22/08/12 16:24:48 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet with change tracker null
22/08/12 16:24:48 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet Headers: (amz-sdk-invocation-id: 932d78b1-791a-d543-8013-eb2cf915725f, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet
amz-sdk-invocation-id:932d78b1-791a-d543-8013-eb2cf915725f
amz-sdk-request:ttl=20220812T152808Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152448Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152448Z
20220812/us-east-1/s3/aws4_request
864a04a69c8c070f0d465e3922bd7968cdb900b53e289ac3f5b60d031fd148b8"
22/08/12 16:24:48 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:48 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:48 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1
22/08/12 16:24:48 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 932d78b1-791a-d543-8013-eb2cf915725f
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=8289e572a608af7c1197644d4efd48b52f8f8bcf7410daa22943c2a5d202e22d
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152448Z
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 932d78b1-791a-d543-8013-eb2cf915725f[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=8289e572a608af7c1197644d4efd48b52f8f8bcf7410daa22943c2a5d202e22d[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152448Z[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 4be2e478-732d-4a97-87e9-cf2ac708c76a[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 4be2e478-732d-4a97-87e9-cf2ac708c76a[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:48 GMT[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Length: 389[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 4be2e478-732d-4a97-87e9-cf2ac708c76a
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 4be2e478-732d-4a97-87e9-cf2ac708c76a
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:48 GMT
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Length: 389
22/08/12 16:24:48 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:48 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:48 GMT
22/08/12 16:24:48 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a; S3 Extended Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a; Proxy: null), S3 Extended Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 11
22/08/12 16:24:48 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a; S3 Extended Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a; Proxy: null), S3 Extended Request ID: 4be2e478-732d-4a97-87e9-cf2ac708c76a], RequestType=[GetObjectMetadataRequest], AWSRequestID=[4be2e478-732d-4a97-87e9-cf2ac708c76a], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[20.796], HttpClientSendRequestTime=[0.85], HttpRequestTime=[18.873], ApiCallLatency=[20.476], RequestSigningTime=[0.643], CredentialsRequestTime=[0.01, 0.009], HttpClientReceiveResponseTime=[16.92],
22/08/12 16:24:48 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/ delimiter=/ keys=2 requester pays=false
22/08/12 16:24:48 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 5
22/08/12 16:24:48 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 2abe7ab3-f0a3-f96d-99bb-99cce65be5bc, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F
amz-sdk-invocation-id:2abe7ab3-f0a3-f96d-99bb-99cce65be5bc
amz-sdk-request:ttl=20220812T152808Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152448Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:48 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152448Z
20220812/us-east-1/s3/aws4_request
3f30b94b744865d8f0e7128a4e36531e3ddc20b660b5749c7627ed39f44b5c8d"
22/08/12 16:24:48 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:48 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:48 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:48 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 2abe7ab3-f0a3-f96d-99bb-99cce65be5bc
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=23a08d83bb0abb3cc211e6589dac3c2c1fef887550c2906583b1c797c612eec9
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152448Z
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 2abe7ab3-f0a3-f96d-99bb-99cce65be5bc[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152808Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=23a08d83bb0abb3cc211e6589dac3c2c1fef887550c2906583b1c797c612eec9[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152448Z[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:48 GMT[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "Content-Length: 427[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:48 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/</Prefix></ListBucketResult>"
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:48 GMT
22/08/12 16:24:48 DEBUG headers: http-outgoing-0 << Content-Length: 427
22/08/12 16:24:48 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:48 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:48 GMT
22/08/12 16:24:48 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:48 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:48 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:48 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:48 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:48 DEBUG request: Received successful response: 200, AWS Request ID: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d
22/08/12 16:24:48 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:48 DEBUG requestId: AWS Request ID: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d
22/08/12 16:24:48 DEBUG requestId: AWS Extended Request ID: 7e04903a-9d8d-4db3-8b5d-ceeffb51682d
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 12
22/08/12 16:24:48 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[7e04903a-9d8d-4db3-8b5d-ceeffb51682d], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[32.111], ClientExecuteTime=[54.843], HttpClientSendRequestTime=[0.686], HttpRequestTime=[21.041], ApiCallLatency=[54.551], RequestSigningTime=[0.454], CredentialsRequestTime=[0.008, 0.005], HttpClientReceiveResponseTime=[19.491],
22/08/12 16:24:48 DEBUG S3AFileSystem: LIST: duration 0:00.055s
22/08/12 16:24:48 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet
22/08/12 16:24:48 DEBUG IOStatisticsStoreImpl: Incrementing counter files_created by 1 with final value 1
22/08/12 16:24:48 DEBUG S3ARetryPolicy: Retrying on recoverable AWS failures 7 times with an initial interval of 500ms
22/08/12 16:24:48 DEBUG S3ABlockOutputStream: Initialized S3ABlockOutputStream for nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet output to FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Writing, dataSize=0, limit=67108864}
22/08/12 16:24:48 DEBUG ParquetFileWriter: 0: start
22/08/12 16:24:48 DEBUG MemoryManager: Allocated total memory pool is: 1020054720
22/08/12 16:24:48 INFO CodecPool: Got brand-new compressor [.snappy]
22/08/12 16:24:48 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 1 initialCapacity 64
22/08/12 16:24:48 DEBUG CapacityByteArrayOutputStream: initial slab of size 64
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(36)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 36
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 36
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(21)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 21
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 21
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(33)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 33
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 33
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(1)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 1
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 1
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(41)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 41
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 41
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(85)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 85
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 85
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(59)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 59
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 59
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(58)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 58
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 58
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(15)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 15
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 15
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanBroadcast(0)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning broadcast 0
22/08/12 16:24:48 DEBUG TorrentBroadcast: Unpersisting TorrentBroadcast 0
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: removing broadcast 0
22/08/12 16:24:48 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 16:24:48 DEBUG BlockManager: Removing broadcast 0
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_0_piece0
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_0_piece0 of size 6705 dropped from memory (free 455149761)
22/08/12 16:24:48 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_0_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None)
22/08/12 16:24:48 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 10.1.255.235:38343 in memory (size: 6.5 KiB, free: 434.3 MiB)
22/08/12 16:24:48 DEBUG BlockManagerMaster: Updated info of block broadcast_0_piece0
22/08/12 16:24:48 DEBUG BlockManager: Told master about block broadcast_0_piece0
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_0
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_0 of size 12720 dropped from memory (free 455162481)
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Done removing broadcast 0, response is 0
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Sent response: 0 to 10.1.255.235:45909
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned broadcast 0
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(13)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 13
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 13
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(73)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 73
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 73
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(11)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 11
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 11
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(71)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 71
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 71
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(28)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 28
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 28
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(48)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 48
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 48
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(75)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 75
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 75
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanBroadcast(2)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning broadcast 2
22/08/12 16:24:48 DEBUG TorrentBroadcast: Unpersisting TorrentBroadcast 2
22/08/12 16:24:48 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 1 initialCapacity 64
22/08/12 16:24:48 DEBUG CapacityByteArrayOutputStream: initial slab of size 64
22/08/12 16:24:48 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: removing broadcast 2
22/08/12 16:24:48 DEBUG BlockManager: Removing broadcast 2
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_2_piece0
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_2_piece0 of size 6713 dropped from memory (free 455169194)
22/08/12 16:24:48 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_2_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None)
22/08/12 16:24:48 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 10.1.255.235:38343 in memory (size: 6.6 KiB, free: 434.3 MiB)
22/08/12 16:24:48 DEBUG BlockManagerMaster: Updated info of block broadcast_2_piece0
22/08/12 16:24:48 DEBUG BlockManager: Told master about block broadcast_2_piece0
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_2
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_2 of size 12720 dropped from memory (free 455181914)
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Done removing broadcast 2, response is 0
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Sent response: 0 to 10.1.255.235:45909
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned broadcast 2
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(76)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 76
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 76
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(77)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 77
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 77
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(16)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 16
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 16
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(26)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 26
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 26
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(83)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 83
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 83
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(7)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 7
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 7
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(44)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 44
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 44
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(82)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 82
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 82
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(50)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 50
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 50
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(66)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 66
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 66
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(43)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 43
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 43
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(68)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 68
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 68
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(17)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 17
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 17
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(78)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 78
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 78
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(60)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 60
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 60
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(57)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 57
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 57
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(23)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 23
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 23
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(72)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 72
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 72
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(12)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 12
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 12
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(46)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 46
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 46
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(18)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 18
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 18
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(2)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 2
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 2
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(5)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 5
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 5
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(49)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 49
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 49
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(25)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 25
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 25
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(14)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 14
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 14
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(19)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 19
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 19
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(6)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 6
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 6
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(4)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 4
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 4
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(62)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 62
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 62
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(38)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 38
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 38
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(35)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 35
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 35
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(22)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 22
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 22
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(45)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 45
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 45
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(47)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 47
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 47
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(20)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 20
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 20
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(70)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 70
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 70
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(64)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 64
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 64
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(3)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 3
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 3
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(32)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 32
22/08/12 16:24:48 DEBUG MemoryManager: Adjust block size from 134,217,728 to 134,217,728 for writer: org.apache.parquet.hadoop.InternalParquetRecordWriter@40045bfe
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 32
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(61)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 61
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 61
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(87)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 87
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 87
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(51)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 51
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 51
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(29)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 29
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 29
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(74)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 74
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 74
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(39)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 39
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 39
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(27)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 27
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 27
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(10)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 10
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 10
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(30)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 30
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 30
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(40)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 40
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 40
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(54)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 54
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 54
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(86)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 86
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 86
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(67)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 67
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 67
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(56)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 56
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 56
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(81)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 81
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 81
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(24)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 24
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 24
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(53)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 53
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 53
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(55)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 55
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 55
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(31)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 31
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 31
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(69)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 69
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 69
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(65)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 65
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 65
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(42)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 42
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 42
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(52)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 52
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 52
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(34)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 34
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 34
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(8)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 8
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 8
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanBroadcast(1)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning broadcast 1
22/08/12 16:24:48 DEBUG TorrentBroadcast: Unpersisting TorrentBroadcast 1
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: removing broadcast 1
22/08/12 16:24:48 DEBUG BlockManager: Removing broadcast 1
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_1
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_1 of size 12720 dropped from memory (free 455194634)
22/08/12 16:24:48 DEBUG BlockManager: Removing block broadcast_1_piece0
22/08/12 16:24:48 DEBUG MemoryStore: Block broadcast_1_piece0 of size 6711 dropped from memory (free 455201345)
22/08/12 16:24:48 DEBUG BlockManagerMasterEndpoint: Updating block info on master broadcast_1_piece0 for BlockManagerId(driver, 10.1.255.235, 38343, None)
22/08/12 16:24:48 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 10.1.255.235:38343 in memory (size: 6.6 KiB, free: 434.3 MiB)
22/08/12 16:24:48 DEBUG BlockManagerMaster: Updated info of block broadcast_1_piece0
22/08/12 16:24:48 DEBUG BlockManager: Told master about block broadcast_1_piece0
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Done removing broadcast 1, response is 0
22/08/12 16:24:48 DEBUG BlockManagerStorageEndpoint: Sent response: 0 to 10.1.255.235:45909
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned broadcast 1
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(80)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 80
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 80
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(63)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 63
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 63
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(79)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 79
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 79
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(9)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 9
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 9
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(37)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 37
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 37
22/08/12 16:24:48 DEBUG ContextCleaner: Got cleaning task CleanAccum(84)
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaning accumulator 84
22/08/12 16:24:48 DEBUG ContextCleaner: Cleaned accumulator 84
22/08/12 16:24:48 INFO PythonRunner: Times: total = 50, boot = -2330, init = 2379, finish = 1
22/08/12 16:24:48 INFO PythonRunner: Times: total = 51, boot = -2362, init = 2413, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 50, boot = -2419, init = 2469, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 51, boot = -2467, init = 2518, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 49, boot = -2490, init = 2539, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 49, boot = -2533, init = 2582, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 49, boot = -2594, init = 2643, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 46, boot = -2635, init = 2681, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 47, boot = -2676, init = 2723, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 50, boot = -2721, init = 2771, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 52, boot = -2774, init = 2826, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 75, boot = -502, init = 577, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 50, boot = -526, init = 576, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 55, boot = -526, init = 581, finish = 0
22/08/12 16:24:49 INFO PythonRunner: Times: total = 83, boot = -531, init = 614, finish = 0
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: <!-- start message -->
22/08/12 16:24:49 DEBUG MessageColumnIO: < MESSAGE START >
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={}}: [] r:0
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: <a>
22/08/12 16:24:49 DEBUG MessageColumnIO: startField(a, 0)
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={}}: [a] r:0
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: 1
22/08/12 16:24:49 DEBUG MessageColumnIO: addLong(1)
22/08/12 16:24:49 DEBUG MessageColumnIO: r: 0
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={}}: [a] r:0
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: </a>
22/08/12 16:24:49 DEBUG MessageColumnIO: endField(a, 0)
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0}}: [] r:0
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: <b>
22/08/12 16:24:49 DEBUG MessageColumnIO: startField(b, 1)
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0}}: [b] r:0
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: 2.0
22/08/12 16:24:49 DEBUG MessageColumnIO: addDouble(2.0)
22/08/12 16:24:49 DEBUG MessageColumnIO: r: 0
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0}}: [b] r:0
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: </b>
22/08/12 16:24:49 DEBUG MessageColumnIO: endField(b, 1)
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0, 1}}: [] r:0
22/08/12 16:24:49 DEBUG MessageColumnIO: < MESSAGE END >
22/08/12 16:24:49 DEBUG MessageColumnIO: 0, VistedIndex{vistedIndexes={0, 1}}: [] r:0
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: <!-- end message -->
22/08/12 16:24:49 INFO PythonRunner: Times: total = 51, boot = -563, init = 614, finish = 0
22/08/12 16:24:49 DEBUG RecordConsumerLoggingWrapper: <!-- flush -->
22/08/12 16:24:49 DEBUG InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 24
22/08/12 16:24:49 DEBUG ParquetFileWriter: 4: start block
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 64
22/08/12 16:24:49 DEBUG DictionaryValuesWriter: max dic id 0
22/08/12 16:24:49 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 0 initialCapacity 1024
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024
22/08/12 16:24:49 DEBUG DictionaryValuesWriter: rle encoded bytes 1
22/08/12 16:24:49 DEBUG BytesInput: BytesInput from array of 1 bytes
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024
22/08/12 16:24:49 DEBUG PlainValuesWriter: writing a buffer of size 8
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 0 bytes to out
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 6 bytes to out
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: {
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 4 bytes to out
22/08/12 16:24:49 DEBUG BytesUtils: write le int: 2 => 2 0 0 0
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 2 bytes to out
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: }
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 8 bytes to out
22/08/12 16:24:49 DEBUG BytesInput: converted 16 to byteArray of 16 bytes
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 23 bytes to out
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 16 bytes to out
22/08/12 16:24:49 DEBUG BytesInput: converted 39 to byteArray of 39 bytes
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 64
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 64
22/08/12 16:24:49 DEBUG DictionaryValuesWriter: max dic id 0
22/08/12 16:24:49 DEBUG RunLengthBitPackingHybridEncoder: Encoding: RunLengthBitPackingHybridEncoder with bithWidth: 0 initialCapacity 1024
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024
22/08/12 16:24:49 DEBUG DictionaryValuesWriter: rle encoded bytes 1
22/08/12 16:24:49 DEBUG BytesInput: BytesInput from array of 1 bytes
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: used 0 slabs, adding new slab of size 1024
22/08/12 16:24:49 DEBUG PlainValuesWriter: writing a buffer of size 8
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 0 bytes to out
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 6 bytes to out
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: {
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 4 bytes to out
22/08/12 16:24:49 DEBUG BytesUtils: write le int: 2 => 2 0 0 0
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 2 bytes to out
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: }
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 8 bytes to out
22/08/12 16:24:49 DEBUG BytesInput: converted 16 to byteArray of 16 bytes
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 23 bytes to out
22/08/12 16:24:49 DEBUG BytesInput$SequenceBytesIn: write 16 bytes to out
22/08/12 16:24:49 DEBUG BytesInput: converted 39 to byteArray of 39 bytes
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 64
22/08/12 16:24:49 DEBUG CapacityByteArrayOutputStream: initial slab of size 1024
22/08/12 16:24:49 DEBUG ParquetFileWriter: 4: write data pages
22/08/12 16:24:49 DEBUG ParquetFileWriter: 4: write data pages content
22/08/12 16:24:49 DEBUG ParquetFileWriter: 43: end column
22/08/12 16:24:49 DEBUG ColumnChunkPageWriteStore: written 39B for [a] optional int64 a: 1 values, 14B raw, 16B comp, 1 pages, encodings: [PLAIN]
22/08/12 16:24:49 DEBUG ParquetFileWriter: 43: write data pages
22/08/12 16:24:49 DEBUG ParquetFileWriter: 43: write data pages content
22/08/12 16:24:49 DEBUG ParquetFileWriter: 82: end column
22/08/12 16:24:49 DEBUG ColumnChunkPageWriteStore: written 39B for [b] optional double b: 1 values, 14B raw, 16B comp, 1 pages, encodings: [PLAIN]
22/08/12 16:24:49 DEBUG ParquetFileWriter: 82: end block
22/08/12 16:24:49 DEBUG ParquetFileWriter: 82: column indexes
22/08/12 16:24:49 DEBUG ParquetFileWriter: 144: offset indexes
22/08/12 16:24:49 DEBUG ParquetFileWriter: 164: bloom filters
22/08/12 16:24:49 DEBUG ParquetFileWriter: 164: end
22/08/12 16:24:49 DEBUG ParquetFileWriter: 710: footer length = 546
22/08/12 16:24:49 DEBUG BytesUtils: write le int: 546 => 34 2 0 0
22/08/12 16:24:49 DEBUG S3ABlockOutputStream: S3ABlockOutputStream{WriteOperationHelper {bucket=master.rando2}, blockSize=67108864, activeBlock=FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Writing, dataSize=718, limit=67108864} Statistics=counters=((action_executor_acquired=0) (stream_write_bytes=718) (stream_write_block_uploads=0) (stream_write_exceptions_completing_upload=0) (object_multipart_aborted.failures=0) (multipart_upload_completed=0) (op_hsync=0) (stream_write_queue_duration=0) (op_abort=0) (stream_write_total_time=0) (op_hflush=0) (object_multipart_aborted=0) (stream_write_exceptions=0) (stream_write_total_data=0) (action_executor_acquired.failures=0) (op_abort.failures=0) (multipart_upload_completed.failures=0));
gauges=((stream_write_block_uploads_pending=0) (stream_write_block_uploads_data_pending=0));
minimums=((op_abort.min=-1) (action_executor_acquired.failures.min=-1) (action_executor_acquired.min=-1) (object_multipart_aborted.failures.min=-1) (object_multipart_aborted.min=-1) (multipart_upload_completed.failures.min=-1) (multipart_upload_completed.min=-1) (op_abort.failures.min=-1));
maximums=((op_abort.max=-1) (object_multipart_aborted.max=-1) (op_abort.failures.max=-1) (object_multipart_aborted.failures.max=-1) (multipart_upload_completed.failures.max=-1) (action_executor_acquired.failures.max=-1) (multipart_upload_completed.max=-1) (action_executor_acquired.max=-1));
means=((multipart_upload_completed.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000)));
}: Closing block #1: current block= FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Writing, dataSize=718, limit=67108864}
22/08/12 16:24:49 DEBUG S3ABlockOutputStream: Executing regular upload for WriteOperationHelper {bucket=master.rando2}
22/08/12 16:24:49 DEBUG S3ADataBlocks: Start datablock[1] upload
22/08/12 16:24:49 DEBUG S3ADataBlocks: FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Writing, dataSize=718, limit=67108864}: entering state Upload
22/08/12 16:24:49 DEBUG IOStatisticsStoreImpl: Incrementing counter stream_write_block_uploads by 1 with final value 1
22/08/12 16:24:49 DEBUG S3ABlockOutputStream: Clearing active block
22/08/12 16:24:49 DEBUG Invoker: Starting: Writing Object
22/08/12 16:24:49 DEBUG S3AFileSystem: PUT 718 bytes to nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet
22/08/12 16:24:49 DEBUG S3AFileSystem: PUT start 718 bytes
22/08/12 16:24:49 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request by 1 with final value 2
22/08/12 16:24:49 DEBUG Mimetypes: Loading mime types from file in the classpath: mime.types
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/activemessage'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ez' to 'application/andrew-inset'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/applefile'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'atom' to 'application/atom+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/atomicmail'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/batch-smtp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/beep+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cals-1840'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cnrp+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/commonground'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cpl+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/cybercash'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/dca-rft'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/dec-dx'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/dvcs'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/edi-consent'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/edifact'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/edi-x12'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/eshop'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/font-tdpfr'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/http'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/hyperstudio'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/iges'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.cmd'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.obj'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.response'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/index.vnd'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/iotp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/ipp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/isup'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'json' to 'application/json'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'hqx' to 'application/mac-binhex40'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'cpt' to 'application/mac-compactpro'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/macwriteii'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/marc'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/mathematica'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mathml' to 'application/mathml+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'doc' to 'application/msword'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/news-message-id'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/news-transmission'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/ocsp-request'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/ocsp-response'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'bin' to 'application/octet-stream'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dms' to 'application/octet-stream'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'lha' to 'application/octet-stream'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'lzh' to 'application/octet-stream'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'exe' to 'application/octet-stream'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'class' to 'application/octet-stream'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'so' to 'application/octet-stream'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dll' to 'application/octet-stream'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dmg' to 'application/octet-stream'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'oda' to 'application/oda'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ogg' to 'application/ogg'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/parityfec'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pdf' to 'application/pdf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pgp-encrypted'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pgp-keys'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pgp-signature'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkcs10'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkcs7-mime'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkcs7-signature'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkix-cert'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkix-crl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/pkixcmp'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ai' to 'application/postscript'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'eps' to 'application/postscript'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ps' to 'application/postscript'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.alvestrand.titrax-sheet'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.cww'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.nprend'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/prs.plucker'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/qsig'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'rdf' to 'application/rdf+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/reginfo+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/remote-printing'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/riscos'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/rtf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sdp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-payment'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-payment-initiation'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-registration'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/set-registration-initiation'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sgml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sgml-open-catalog'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/sieve'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/slate'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'smi' to 'application/smil'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'smil' to 'application/smil'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'gram' to 'application/srgs'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'grxml' to 'application/srgs+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/timestamp-query'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/timestamp-reply'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/tve-trigger'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vemmi'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.pic-bw-large'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.pic-bw-small'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.pic-bw-var'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3gpp.sms'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.3m.post-it-notes'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.accpac.simply.aso'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.accpac.simply.imp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.acucobol'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.acucorp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.adobe.xfdf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.aether.imp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.amiga.ami'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.anser-web-certificate-issue-initiation'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.anser-web-funds-transfer-initiation'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.audiograph'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.blueice.multipass'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.bmi'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.businessobjects'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.canon-cpdl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.canon-lips'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cinderella'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.claymore'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.commerce-battelle'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.commonspace'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.contact.cmsg'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cosmocaller'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.criticaltools.wbs+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ctc-posml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cups-postscript'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cups-raster'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cups-raw'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.curl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.cybank'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.data-vision.rdz'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dna'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dpgraph'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dreamfactory'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.dxr'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecdis-update'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.chart'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.filerequest'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.fileupdate'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.series'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.seriesrequest'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ecowin.seriesupdate'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.enliven'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.esf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.msf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.quickanime'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.salt'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.epson.ssf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ericsson.quickcall'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.eudora.data'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fdf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ffsns'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fints'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.flographit'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.framemaker'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fsc.weblaunch'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasys'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasys2'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasys3'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasysgp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujitsu.oasysprs'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujixerox.ddd'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujixerox.docuworks'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fujixerox.docuworks.binder'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.fut-misnet'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.grafeq'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-account'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-help'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-identity-message'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-injector'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-tool-message'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-tool-template'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.groove-vcard'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hbci'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hhe.lesson-player'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-hpgl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-hpid'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-hps'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-pcl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hp-pclxl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.httphone'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.hzn-3d-crossword'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.afplinedata'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.electronic-media'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.minipay'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.modcap'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.rights-management'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ibm.secure-container'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.informix-visionary'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intercon.formnet'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intertrust.digibox'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intertrust.nncp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intu.qbo'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.intu.qfx'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.irepository.package+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.is-xpr'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-directory-service'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-jpnstore-wakeup'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-payment-wakeup'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-registration'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-registration-wakeup'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-setstore-wakeup'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-verification'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.japannet-verification-wakeup'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.jisp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.karbon'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kchart'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kformula'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kivio'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kontour'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kpresenter'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kspread'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kde.kword'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.kenameaapp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.koan'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.liberty-request+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.llamagraphics.life-balance.desktop'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.llamagraphics.life-balance.exchange+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-1-2-3'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-approach'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-freelance'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-notes'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-organizer'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-screencam'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.lotus-wordpro'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mcd'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mediastation.cdkey'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.meridian-slingshot'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.micrografx.flo'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.micrografx.igx'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mif' to 'application/vnd.mif'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.minisoft-hp3000-save'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mitsubishi.misty-guard.trustweb'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.daf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.dis'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.mbk'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.mqy'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.msl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.plc'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mobius.txf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mophun.application'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mophun.certificate'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.adsi'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.fis'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.gotap'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.kmr'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.ttc'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.motorola.flexsuite.wem'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xul' to 'application/vnd.mozilla.xul+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-artgalry'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-asf'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xls' to 'application/vnd.ms-excel'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-lrm'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ppt' to 'application/vnd.ms-powerpoint'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-project'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-tnef'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-works'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ms-wpl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.mseq'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.msign'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.music-niff'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.musician'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.netfpx'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.noblenet-directory'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.noblenet-sealer'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.noblenet-web'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.novadigm.edm'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.novadigm.edx'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.novadigm.ext'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.obn'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.osa.netdeploy'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.palm'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pg.format'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pg.osasli'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder6'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder6-s'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder7'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder7-s'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder75'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.powerbuilder75-s'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.previewsystems.box'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.publishare-delta-tree'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pvi.ptid1'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pwg-multiplexed'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.pwg-xhtml-print+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.quark.quarkxpress'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.rapid'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'rm' to 'application/vnd.rn-realmedia'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.s3sms'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sealed.net'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.seemail'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.formdata'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.formtemplate'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.interchange'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.shana.informed.package'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.smaf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sss-cod'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sss-dtf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.sss-ntf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.street-stream'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.svd'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.swiftview-ics'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.triscape.mxs'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.trueapp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.truedoc'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.ufdl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.alert'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.alert-wbxml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.bearer-choice'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.bearer-choice-wbxml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.cacheop'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.cacheop-wbxml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.channel'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.channel-wbxml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.list'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.list-wbxml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.listcmd'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.listcmd-wbxml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.uplanet.signal'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vcx'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vectorworks'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vidsoft.vidconference'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.visio'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.visionary'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vividence.scriptfile'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.vsf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wap.sic'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wap.slc'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wbxml' to 'application/vnd.wap.wbxml'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wmlc' to 'application/vnd.wap.wmlc'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wmlsc' to 'application/vnd.wap.wmlscriptc'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.webturbo'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wrq-hp3000-labelled'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wt.stf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.wv.csp+wbxml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.xara'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.xfdl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yamaha.hv-dic'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yamaha.hv-script'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yamaha.hv-voice'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/vnd.yellowriver-custom-menu'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'vxml' to 'application/voicexml+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/watcherinfo+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/whoispp-query'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/whoispp-response'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/wita'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/wordperfect5.1'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'bcpio' to 'application/x-bcpio'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'vcd' to 'application/x-cdlink'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pgn' to 'application/x-chess-pgn'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/x-compress'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'cpio' to 'application/x-cpio'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'csh' to 'application/x-csh'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dcr' to 'application/x-director'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dir' to 'application/x-director'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dxr' to 'application/x-director'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dvi' to 'application/x-dvi'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'spl' to 'application/x-futuresplash'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'gtar' to 'application/x-gtar'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'gz' to 'application/x-gzip'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'hdf' to 'application/x-hdf'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'js' to 'application/x-javascript'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'jnlp' to 'application/x-java-jnlp-file'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'skp' to 'application/x-koan'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'skd' to 'application/x-koan'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'skt' to 'application/x-koan'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'skm' to 'application/x-koan'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'latex' to 'application/x-latex'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'nc' to 'application/x-netcdf'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'cdf' to 'application/x-netcdf'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sh' to 'application/x-sh'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'shar' to 'application/x-shar'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'swf' to 'application/x-shockwave-flash'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sit' to 'application/x-stuffit'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sv4cpio' to 'application/x-sv4cpio'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sv4crc' to 'application/x-sv4crc'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tar' to 'application/x-tar'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tcl' to 'application/x-tcl'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tex' to 'application/x-tex'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'texinfo' to 'application/x-texinfo'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'texi' to 'application/x-texinfo'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 't' to 'application/x-troff'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tr' to 'application/x-troff'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'roff' to 'application/x-troff'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'man' to 'application/x-troff-man'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'me' to 'application/x-troff-me'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ms' to 'application/x-troff-ms'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ustar' to 'application/x-ustar'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'src' to 'application/x-wais-source'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/x400-bp'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xhtml' to 'application/xhtml+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xht' to 'application/xhtml+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xslt' to 'application/xslt+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xml' to 'application/xml'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xsl' to 'application/xml'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dtd' to 'application/xml-dtd'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'application/xml-external-parsed-entity'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'zip' to 'application/zip'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/32kadpcm'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/amr'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/amr-wb'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'au' to 'audio/basic'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'snd' to 'audio/basic'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/cn'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/dat12'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/dsr-es201108'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/dvi4'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/evrc'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/evrc0'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g722'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g.722.1'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g723'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-16'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-24'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-32'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g726-40'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g728'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g729'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g729D'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/g729E'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/gsm'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/gsm-efr'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l8'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l16'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l20'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/l24'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/lpc'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mid' to 'audio/midi'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'midi' to 'audio/midi'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'kar' to 'audio/midi'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/mpa'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/mpa-robust'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'm4a' to 'audio/mp4a-latm'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'm4p' to 'audio/mp4a-latm'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mpga' to 'audio/mpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mp2' to 'audio/mpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mp3' to 'audio/mpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/parityfec'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/pcma'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/pcmu'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/prs.sid'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/qcelp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/red'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/smv'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/smv0'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/telephone-event'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/tone'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vdvi'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.3gpp.iufp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.cisco.nse'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.cns.anp1'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.cns.inf1'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.digital-winds'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.everad.plj'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.lucent.voice'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nortel.vbk'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nuera.ecelp4800'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nuera.ecelp7470'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.nuera.ecelp9600'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.octel.sbc'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.qcelp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.rhetorex.32kadpcm'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/vnd.vmx.cvsd'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'aif' to 'audio/x-aiff'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'aiff' to 'audio/x-aiff'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'aifc' to 'audio/x-aiff'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/x-alaw-basic'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'm3u' to 'audio/x-mpegurl'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ram' to 'audio/x-pn-realaudio'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ra' to 'audio/x-pn-realaudio'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'audio/x-pn-realaudio-plugin'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wav' to 'audio/x-wav'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pdb' to 'chemical/x-pdb'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xyz' to 'chemical/x-xyz'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'bmp' to 'image/bmp'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'cgm' to 'image/cgm'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/g3fax'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'gif' to 'image/gif'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ief' to 'image/ief'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'jpeg' to 'image/jpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'jpg' to 'image/jpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'jpe' to 'image/jpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'jp2' to 'image/jp2'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/naplps'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pict' to 'image/pict'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pic' to 'image/pict'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pct' to 'image/pict'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'png' to 'image/png'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/prs.btif'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/prs.pti'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'svg' to 'image/svg+xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/t38'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tiff' to 'image/tiff'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tif' to 'image/tiff'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/tiff-fx'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.cns.inf2'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'djvu' to 'image/vnd.djvu'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'djv' to 'image/vnd.djvu'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.dwg'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.dxf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fastbidsheet'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fpx'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fst'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fujixerox.edmics-mmr'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.fujixerox.edmics-rlc'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.globalgraphics.pgb'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.mix'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.ms-modi'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.net-fpx'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.svf'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wbmp' to 'image/vnd.wap.wbmp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'image/vnd.xiff'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ras' to 'image/x-cmu-raster'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pntg' to 'image/x-macpaint'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pnt' to 'image/x-macpaint'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mac' to 'image/x-macpaint'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ico' to 'image/x-icon'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pnm' to 'image/x-portable-anymap'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pbm' to 'image/x-portable-bitmap'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'pgm' to 'image/x-portable-graymap'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ppm' to 'image/x-portable-pixmap'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'qtif' to 'image/x-quicktime'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'qti' to 'image/x-quicktime'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'rgb' to 'image/x-rgb'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xbm' to 'image/x-xbitmap'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xpm' to 'image/x-xpixmap'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'xwd' to 'image/x-xwindowdump'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/delivery-status'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/disposition-notification'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/external-body'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/http'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/news'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/partial'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/rfc822'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/s-http'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/sip'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'message/sipfrag'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'igs' to 'model/iges'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'iges' to 'model/iges'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'msh' to 'model/mesh'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mesh' to 'model/mesh'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'silo' to 'model/mesh'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.dwf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.flatland.3dml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.gdl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.gs-gdl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.gtw'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.mts'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.parasolid.transmit.binary'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.parasolid.transmit.text'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'model/vnd.vtu'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wrl' to 'model/vrml'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'vrml' to 'model/vrml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/alternative'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/appledouble'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/byteranges'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/digest'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/encrypted'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/form-data'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/header-set'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/mixed'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/parallel'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/related'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/report'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/signed'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'multipart/voice-message'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ics' to 'text/calendar'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ifb' to 'text/calendar'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'css' to 'text/css'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/directory'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/enriched'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'html' to 'text/html'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'htm' to 'text/html'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/parityfec'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'asc' to 'text/plain'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'txt' to 'text/plain'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/prs.lines.tag'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/rfc822-headers'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'rtx' to 'text/richtext'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'rtf' to 'text/rtf'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sgml' to 'text/sgml'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'sgm' to 'text/sgml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/t140'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'tsv' to 'text/tab-separated-values'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/uri-list'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.abc'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.curl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.dmclientscript'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.fly'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.fmi.flexstor'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.in3d.3dml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.in3d.spot'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.iptc.nitf'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.iptc.newsml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.latex-z'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.motorola.reflex'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.ms-mediapackage'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.net2phone.commcenter.command'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.sun.j2me.app-descriptor'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.wap.si'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/vnd.wap.sl'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wml' to 'text/vnd.wap.wml'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wmls' to 'text/vnd.wap.wmlscript'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'etx' to 'text/x-setext'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/xml'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'text/xml-external-parsed-entity'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension '3gp' to 'video/3gpp'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/bmpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/bt656'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/celb'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/dv'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h261'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h263'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h263-1998'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/h263-2000'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/jpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp1s'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp2p'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp2t'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mp4' to 'video/mp4'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mp4v-es'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/mpv'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mpeg' to 'video/mpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mpg' to 'video/mpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mpe' to 'video/mpeg'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/nv'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ogv' to 'video/ogg'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/parityfec'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/pointer'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'qt' to 'video/quicktime'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mov' to 'video/quicktime'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/smpte292m'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.fvt'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.motorola.video'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.motorola.videop'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'mxu' to 'video/vnd.mpegurl'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'm4u' to 'video/vnd.mpegurl'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.nokia.interleaved-multimedia'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.objectvideo'
22/08/12 16:24:49 DEBUG Mimetypes: Ignoring mimetype with no associated file extensions: 'video/vnd.vivo'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'webm' to 'video/webm'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dv' to 'video/x-dv'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'dif' to 'video/x-dv'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'flv' to 'video/x-flv'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'm4v' to 'video/x-m4v'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'wmv' to 'video/x-ms-wmv'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'avi' to 'video/x-msvideo'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'movie' to 'video/x-sgi-movie'
22/08/12 16:24:49 DEBUG Mimetypes: Setting mime type for extension 'ice' to 'x-conference/x-cooltalk'
22/08/12 16:24:49 DEBUG Mimetypes: Extension 'tmp' is unrecognized in mime type listing, using default mime type: 'application/octet-stream'
22/08/12 16:24:49 DEBUG request: Sending Request: PUT http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet Headers: (amz-sdk-invocation-id: 26236f27-4f41-11b2-2c3c-2d72077ebda7, Content-Length: 718, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:49 DEBUG AWS4Signer: AWS4 Canonical Request: '"PUT
/master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet
amz-sdk-invocation-id:26236f27-4f41-11b2-2c3c-2d72077ebda7
amz-sdk-request:attempt=1;max=21
amz-sdk-retry:0/0/500
content-length:892
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD
x-amz-date:20220812T152449Z
x-amz-decoded-content-length:718
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length
STREAMING-AWS4-HMAC-SHA256-PAYLOAD"
22/08/12 16:24:49 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152449Z
20220812/us-east-1/s3/aws4_request
77ff5913ac812a24ded91dc2ce594049c2445d92e2a6358cdaa66e84f10d6527"
22/08/12 16:24:49 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:49 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:49 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:49 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:49 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:49 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:49 DEBUG MainClientExec: Executing request PUT /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1
22/08/12 16:24:49 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> PUT /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 26236f27-4f41-11b2-2c3c-2d72077ebda7
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> amz-sdk-request: attempt=1;max=21
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=f5989455b323a7bedf2733a772ede181b7d343cde4703e03ab29a7af7e8067f0
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152449Z
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> x-amz-decoded-content-length: 718
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Content-Length: 892
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 >> Expect: 100-continue
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "PUT /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 26236f27-4f41-11b2-2c3c-2d72077ebda7[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: attempt=1;max=21[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=f5989455b323a7bedf2733a772ede181b7d343cde4703e03ab29a7af7e8067f0[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152449Z[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "x-amz-decoded-content-length: 718[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Content-Length: 892[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "Expect: 100-continue[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 << "HTTP/1.1 100 Continue[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:49 DEBUG headers: http-outgoing-0 << HTTP/1.1 100 Continue
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "2ce;chunk-signature=95afe3e597425c2053d0489a240992cf32b2c4bffce41f604e72055dcc9f824e[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "PAR1[0x15][0x0][0x15][0x1c][0x15] [0x15][0xfe][0xc8][0xd0][0xb6][0xb][0x1c][0x15][0x2][0x15][0x0][0x15][0x6][0x15][0x8][0x0][0x0][0xe]4[0x2][0x0][0x0][0x0][0x3][0x1][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x15][0x0][0x15][0x1c][0x15] [0x15][0x9d][0xb3][0x9d][0xee][0x3][0x1c][0x15][0x2][0x15][0x0][0x15][0x6][0x15][0x8][0x0][0x0][0xe]4[0x2][0x0][0x0][0x0][0x3][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x19][0x11][0x2][0x19][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x19][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x15][0x2][0x19][0x16][0x0][0x0][0x19][0x11][0x2][0x19][0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x19][0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x15][0x2][0x19][0x16][0x0][0x0][0x19][0x1c][0x16][0x8][0x15]N[0x16][0x0][0x0][0x0][0x19][0x1c][0x16]V[0x15]N[0x16][0x0][0x0][0x0][0x15][0x2][0x19]<H[0xc]spark_schema[0x15][0x4][0x0][0x15][0x4]%[0x2][0x18][0x1]a[0x0][0x15][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "%[0x2][0x18][0x1]b[0x0][0x16][0x2][0x19][0x1c][0x19],&[0x8][0x1c][0x15][0x4][0x19]5[0x8][0x6][0x0][0x19][0x18][0x1]a[0x15][0x2][0x16][0x2][0x16]J[0x16]N&[0x8]<[0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x16][0x0]([0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x18][0x8][0x1][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x0][0x19][0x1c][0x15][0x0][0x15][0x0][0x15][0x2][0x0][0x0][0x16][0xa0][0x2][0x15][0x14][0x16][0xa4][0x1][0x15]>[0x0]&V[0x1c][0x15][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "[0x19]5[0x8][0x6][0x0][0x19][0x18][0x1]b[0x15][0x2][0x16][0x2][0x16]J[0x16]N&V<[0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x16][0x0]([0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x18][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x0]@[0x0][0x19][0x1c][0x15][0x0][0x15][0x0][0x15][0x2][0x0][0x0][0x16][0xb4][0x2][0x15][0x14][0x16][0xe2][0x1][0x15]>[0x0][0x16][0x94][0x1][0x16][0x2]&[0x8][0x16][0x9c][0x1][0x14][0x0][0x0][0x19],[0x18][0x18]org.apache.spark.version[0x18][0x5]3.3.0[0x0][0x18])org.apache.spark.sql.parquet.row.metadata[0x18][0x90][0x1]{"type":"struct","fields":[{"name":"a","type":"long","nullable":true,"metadata":{}},{"name":"b","type":"double","nullable":true,"metadata":{}}]}[0x0][0x18]Jparquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94)[0x19],[0x1c][0x0][0x0][0x1c][0x0][0x0][0x0]"[0x2][0x0][0x0]PAR1[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "0;chunk-signature=5da53b38411f1cee0108e3369ee46ed8fd850bf4b032bdfecabb8b450bb5a019[\r][\n]"
22/08/12 16:24:49 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Etag: "0a03a8cbced07244b6b9fdfaa86b9b37c3070344e57dde0757bc57e6a7ab045d"[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "X-Amz-Version-Id: 519c48fef92047a29bc78fca8b6c1bf3[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:50 GMT[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Content-Length: 0[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Etag: "0a03a8cbced07244b6b9fdfaa86b9b37c3070344e57dde0757bc57e6a7ab045d"
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << X-Amz-Version-Id: 519c48fef92047a29bc78fca8b6c1bf3
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:50 GMT
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Content-Length: 0
22/08/12 16:24:50 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:50 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:50 GMT
22/08/12 16:24:50 DEBUG request: Received successful response: 200, AWS Request ID: null
22/08/12 16:24:50 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:50 DEBUG requestId: AWS Request ID: not available
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 13
22/08/12 16:24:50 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[PutObjectRequest], AWSRequestID=[null], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[0.19], ClientExecuteTime=[869.845], HttpClientSendRequestTime=[3.405], HttpRequestTime=[867.332], ApiCallLatency=[869.087], RequestSigningTime=[0.677], CredentialsRequestTime=[0.013, 0.007], HttpClientReceiveResponseTime=[862.463],
22/08/12 16:24:50 DEBUG S3AFileSystem: PUT completed success=true; 718 bytes
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_bytes by 718 with final value 718
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter object_put_request_completed by 1 with final value 2
22/08/12 16:24:50 DEBUG S3AFileSystem: Finished write to nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet, len 718. etag 0a03a8cbced07244b6b9fdfaa86b9b37c3070344e57dde0757bc57e6a7ab045d, version 519c48fef92047a29bc78fca8b6c1bf3
22/08/12 16:24:50 DEBUG CallableSupplier: Starting: Waiting for task completion
22/08/12 16:24:50 DEBUG S3AFileSystem: Starting: Deleting 5 keys
22/08/12 16:24:50 DEBUG S3AFileSystem: Initiating delete operation for 5 objects
22/08/12 16:24:50 DEBUG S3AFileSystem: nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/
22/08/12 16:24:50 DEBUG S3AFileSystem: nonemptyprefix8/_temporary/0/_temporary/
22/08/12 16:24:50 DEBUG S3AFileSystem: nonemptyprefix8/_temporary/0/
22/08/12 16:24:50 DEBUG S3AFileSystem: nonemptyprefix8/_temporary/
22/08/12 16:24:50 DEBUG S3AFileSystem: nonemptyprefix8/
22/08/12 16:24:50 DEBUG S3AFileSystem: Starting: DELETE 5 keys
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter object_bulk_delete_request by 1 with final value 2
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter object_delete_objects by 5 with final value 7
22/08/12 16:24:50 DEBUG request: Sending Request: POST http://localhost:30600 /master.rando2/ Parameters: ({"delete":[null]}Headers: (amz-sdk-invocation-id: a62c5d64-b53b-ef06-3b3b-bb32dff9b5b8, Content-Length: 387, Content-MD5: toKYq0RGHKQSGL9sB1/cdg==, Content-Type: application/xml, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:50 DEBUG AWS4Signer: AWS4 Canonical Request: '"POST
/master.rando2/
delete=
amz-sdk-invocation-id:a62c5d64-b53b-ef06-3b3b-bb32dff9b5b8
amz-sdk-request:ttl=20220812T152810Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-length:387
content-md5:toKYq0RGHKQSGL9sB1/cdg==
content-type:application/xml
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:3bbb784825b9f4bd2627b155ff84e230e583c52472a55c175691a5da26426020
x-amz-date:20220812T152450Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
3bbb784825b9f4bd2627b155ff84e230e583c52472a55c175691a5da26426020"
22/08/12 16:24:50 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152450Z
20220812/us-east-1/s3/aws4_request
f0d5d3665185b4539cd1e3482bc457e01e6cacdaec50d590325cedcabe563b5f"
22/08/12 16:24:50 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:50 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:50 DEBUG MainClientExec: Executing request POST /master.rando2/?delete HTTP/1.1
22/08/12 16:24:50 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> POST /master.rando2/?delete HTTP/1.1
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: a62c5d64-b53b-ef06-3b3b-bb32dff9b5b8
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152810Z;attempt=1;max=21
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=31376faf52d9c0f96abf9757bb37f9f955b52cc54a00c5d87c6a2d4e0d5fb41d
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Content-MD5: toKYq0RGHKQSGL9sB1/cdg==
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Content-Type: application/xml
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: 3bbb784825b9f4bd2627b155ff84e230e583c52472a55c175691a5da26426020
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152450Z
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Content-Length: 387
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "POST /master.rando2/?delete HTTP/1.1[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: a62c5d64-b53b-ef06-3b3b-bb32dff9b5b8[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152810Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-md5;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=31376faf52d9c0f96abf9757bb37f9f955b52cc54a00c5d87c6a2d4e0d5fb41d[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Content-MD5: toKYq0RGHKQSGL9sB1/cdg==[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: 3bbb784825b9f4bd2627b155ff84e230e583c52472a55c175691a5da26426020[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152450Z[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Content-Length: 387[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "<Delete><Quiet>true</Quiet><Object><Key>nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/</Key></Object><Object><Key>nonemptyprefix8/_temporary/0/_temporary/</Key></Object><Object><Key>nonemptyprefix8/_temporary/0/</Key></Object><Object><Key>nonemptyprefix8/_temporary/</Key></Object><Object><Key>nonemptyprefix8/</Key></Object></Delete>"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: f338b1a2-c6fc-4af3-935f-27189524aeb3[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: f338b1a2-c6fc-4af3-935f-27189524aeb3[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:50 GMT[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "Content-Length: 116[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 << "<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"></DeleteResult>"
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: f338b1a2-c6fc-4af3-935f-27189524aeb3
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: f338b1a2-c6fc-4af3-935f-27189524aeb3
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:50 GMT
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 << Content-Length: 116
22/08/12 16:24:50 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:50 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:50 GMT
22/08/12 16:24:50 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$DeleteObjectsHandler
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:50 DEBUG request: Received successful response: 200, AWS Request ID: f338b1a2-c6fc-4af3-935f-27189524aeb3
22/08/12 16:24:50 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:50 DEBUG requestId: AWS Request ID: f338b1a2-c6fc-4af3-935f-27189524aeb3
22/08/12 16:24:50 DEBUG requestId: AWS Extended Request ID: f338b1a2-c6fc-4af3-935f-27189524aeb3
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 14
22/08/12 16:24:50 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[DeleteObjectsRequest], AWSRequestID=[f338b1a2-c6fc-4af3-935f-27189524aeb3], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.572], ClientExecuteTime=[95.418], HttpClientSendRequestTime=[1.283], HttpRequestTime=[92.192], ApiCallLatency=[95.067], RequestSigningTime=[0.543], CredentialsRequestTime=[0.012, 0.008], HttpClientReceiveResponseTime=[89.956],
22/08/12 16:24:50 DEBUG S3AFileSystem: DELETE 5 keys: duration 0:00.097s
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter fake_directories_deleted by 5 with final value 7
22/08/12 16:24:50 DEBUG S3AFileSystem: Deleting 5 keys: duration 0:00.097s
22/08/12 16:24:50 DEBUG CallableSupplier: Waiting for task completion: duration 0:00.098s
22/08/12 16:24:50 DEBUG Invoker: Writing Object: duration 0:00.991s
22/08/12 16:24:50 DEBUG S3ADataBlocks: FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Upload, dataSize=718, limit=67108864}: entering state Closed
22/08/12 16:24:50 DEBUG S3ADataBlocks: Closed FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Closed, dataSize=718, limit=67108864}
22/08/12 16:24:50 DEBUG S3ADataBlocks: Closing FileBlock{index=1, destFile=/tmp/hadoop-luke/s3a/s3ablock-0001-8560763669495856341.tmp, state=Closed, dataSize=718, limit=67108864}
22/08/12 16:24:50 DEBUG S3ADataBlocks: block[1]: closeBlock()
22/08/12 16:24:50 DEBUG S3ABlockOutputStream: Upload complete to nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet by WriteOperationHelper {bucket=master.rando2}
22/08/12 16:24:50 DEBUG S3ABlockOutputStream: Statistics: OutputStreamStatistics{counters=((stream_write_exceptions=0) (multipart_upload_completed.failures=0) (multipart_upload_completed=0) (stream_write_block_uploads=1) (object_multipart_aborted.failures=0) (op_hsync=0) (op_abort=0) (stream_write_queue_duration=0) (stream_write_bytes=718) (object_multipart_aborted=0) (op_abort.failures=0) (stream_write_exceptions_completing_upload=0) (action_executor_acquired=0) (stream_write_total_data=718) (op_hflush=0) (action_executor_acquired.failures=0) (stream_write_total_time=0));
gauges=((stream_write_block_uploads_pending=1) (stream_write_block_uploads_data_pending=0));
minimums=((op_abort.failures.min=-1) (action_executor_acquired.failures.min=-1) (multipart_upload_completed.min=-1) (action_executor_acquired.min=-1) (multipart_upload_completed.failures.min=-1) (object_multipart_aborted.failures.min=-1) (object_multipart_aborted.min=-1) (op_abort.min=-1));
maximums=((op_abort.failures.max=-1) (action_executor_acquired.max=-1) (object_multipart_aborted.max=-1) (action_executor_acquired.failures.max=-1) (multipart_upload_completed.failures.max=-1) (op_abort.max=-1) (multipart_upload_completed.max=-1) (object_multipart_aborted.failures.max=-1));
means=((object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000)));
, blocksActive=0, blockUploadsCompleted=0, blocksAllocated=1, blocksReleased=1, blocksActivelyAllocated=0, transferDuration=0 ms, totalUploadDuration=0 ms, effectiveBandwidth=0.0 bytes/s}
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter stream_write_total_data by 718 with final value 718
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter op_get_file_status by 1 with final value 1
22/08/12 16:24:50 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet (nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet); needEmptyDirectory=false
22/08/12 16:24:50 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 6
22/08/12 16:24:50 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 6
22/08/12 16:24:50 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet with change tracker null
22/08/12 16:24:50 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet Headers: (amz-sdk-invocation-id: 27b050b9-f85f-e928-3680-5830e9e4c362, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:50 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet
amz-sdk-invocation-id:27b050b9-f85f-e928-3680-5830e9e4c362
amz-sdk-request:ttl=20220812T152810Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152450Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:50 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152450Z
20220812/us-east-1/s3/aws4_request
901b3afc5808d40b024f5b646f56c6bdc400a9e7f70fda96b91a1942a94f6e06"
22/08/12 16:24:50 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:50 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:50 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:50 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:50 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1
22/08/12 16:24:50 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 27b050b9-f85f-e928-3680-5830e9e4c362
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152810Z;attempt=1;max=21
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c436f5583498a8da5c7433ed0827e388de288d702b7f4553162ba2e2172cf68a
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152450Z
22/08/12 16:24:50 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet HTTP/1.1[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 27b050b9-f85f-e928-3680-5830e9e4c362[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152810Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c436f5583498a8da5c7433ed0827e388de288d702b7f4553162ba2e2172cf68a[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152450Z[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:50 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 7fb54d66-eb63-49b0-baf8-394beed0fb78[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 7fb54d66-eb63-49b0-baf8-394beed0fb78[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 389[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 7fb54d66-eb63-49b0-baf8-394beed0fb78
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 7fb54d66-eb63-49b0-baf8-394beed0fb78
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 389
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78; S3 Extended Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78; Proxy: null), S3 Extended Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 15
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78; S3 Extended Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78; Proxy: null), S3 Extended Request ID: 7fb54d66-eb63-49b0-baf8-394beed0fb78], RequestType=[GetObjectMetadataRequest], AWSRequestID=[7fb54d66-eb63-49b0-baf8-394beed0fb78], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[755.373], HttpClientSendRequestTime=[2.076], HttpRequestTime=[753.718], ApiCallLatency=[755.1], RequestSigningTime=[0.433], CredentialsRequestTime=[0.008, 0.007], HttpClientReceiveResponseTime=[750.435],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/ delimiter=/ keys=2 requester pays=false
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 6
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: f97ebf25-72c6-b816-2176-e7fb83f7ce8b, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F
amz-sdk-invocation-id:f97ebf25-72c6-b816-2176-e7fb83f7ce8b
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152451Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152451Z
20220812/us-east-1/s3/aws4_request
b47271e8efca2189609898dc4cac47b15df6d34d4c57b2d391b80c7fc8aec663"
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: f97ebf25-72c6-b816-2176-e7fb83f7ce8b
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=900733557a3486e91db8a71ed5763399cd3d754b52e257494b0f82e766107e81
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2Fpart-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: f97ebf25-72c6-b816-2176-e7fb83f7ce8b[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=900733557a3486e91db8a71ed5763399cd3d754b52e257494b0f82e766107e81[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 8eb704a4-3211-44e6-8fad-ee79fa440b71[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 8eb704a4-3211-44e6-8fad-ee79fa440b71[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 427[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet/</Prefix></ListBucketResult>"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 8eb704a4-3211-44e6-8fad-ee79fa440b71
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 8eb704a4-3211-44e6-8fad-ee79fa440b71
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 427
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: 8eb704a4-3211-44e6-8fad-ee79fa440b71
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: 8eb704a4-3211-44e6-8fad-ee79fa440b71
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: 8eb704a4-3211-44e6-8fad-ee79fa440b71
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 16
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[8eb704a4-3211-44e6-8fad-ee79fa440b71], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[2.01], ClientExecuteTime=[24.744], HttpClientSendRequestTime=[1.1], HttpRequestTime=[20.892], ApiCallLatency=[24.477], RequestSigningTime=[0.579], CredentialsRequestTime=[0.008, 0.007], HttpClientReceiveResponseTime=[17.786],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.026s
22/08/12 16:24:51 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter op_get_file_status.failures by 1 with final value 1
22/08/12 16:24:51 DEBUG BasicWriteTaskStatsTracker: File s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet is not yet visible
java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/part-00000-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc-c000.snappy.parquet
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$24(S3AFileSystem.java:3556)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3554)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.getFileSize(BasicWriteStatsTracker.scala:89)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.getFileSize(BasicWriteStatsTracker.scala:73)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.updateFileStats(BasicWriteStatsTracker.scala:151)
at org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker.closeFile(BasicWriteStatsTracker.scala:146)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.$anonfun$releaseCurrentWriter$1(FileFormatDataWriter.scala:65)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.$anonfun$releaseCurrentWriter$1$adapted(FileFormatDataWriter.scala:65)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseCurrentWriter(FileFormatDataWriter.scala:65)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:75)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:105)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:332)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:338)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$21(FileFormatWriter.scala:256)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists by 1 with final value 2
22/08/12 16:24:51 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 (nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16); needEmptyDirectory=false
22/08/12 16:24:51 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 7
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 7
22/08/12 16:24:51 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 with change tracker null
22/08/12 16:24:51 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 Headers: (amz-sdk-invocation-id: 6c3a185e-0f14-f0c3-2eed-ed33a6b8df5e, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16
amz-sdk-invocation-id:6c3a185e-0f14-f0c3-2eed-ed33a6b8df5e
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152451Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152451Z
20220812/us-east-1/s3/aws4_request
c697e3a3f9dd16fef539157287952cc455f77d3859c05e8352fc3d64774bc6ac"
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 HTTP/1.1
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 HTTP/1.1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 6c3a185e-0f14-f0c3-2eed-ed33a6b8df5e
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2d252e1819288bb106f29b54518889e14de29614d6e4bb66d866df1c1785aaa2
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16 HTTP/1.1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 6c3a185e-0f14-f0c3-2eed-ed33a6b8df5e[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=2d252e1819288bb106f29b54518889e14de29614d6e4bb66d866df1c1785aaa2[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: c7e4dc4b-fc95-406a-87c7-304916a96ca7[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: c7e4dc4b-fc95-406a-87c7-304916a96ca7[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 321[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: c7e4dc4b-fc95-406a-87c7-304916a96ca7
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: c7e4dc4b-fc95-406a-87c7-304916a96ca7
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 321
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7; S3 Extended Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7; Proxy: null), S3 Extended Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 17
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7; S3 Extended Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7; Proxy: null), S3 Extended Request ID: c7e4dc4b-fc95-406a-87c7-304916a96ca7], RequestType=[GetObjectMetadataRequest], AWSRequestID=[c7e4dc4b-fc95-406a-87c7-304916a96ca7], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[21.524], HttpClientSendRequestTime=[0.497], HttpRequestTime=[20.419], ApiCallLatency=[21.339], RequestSigningTime=[0.288], CredentialsRequestTime=[0.007, 0.003], HttpClientReceiveResponseTime=[19.029],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/ delimiter=/ keys=2 requester pays=false
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 7
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 8b9dc119-a5b5-1907-841a-40737bfed560, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2F
amz-sdk-invocation-id:8b9dc119-a5b5-1907-841a-40737bfed560
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152451Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152451Z
20220812/us-east-1/s3/aws4_request
a761af39b70452fa9e5f865c72ded9254ff2583b10356a1f0332125e26183463"
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 8b9dc119-a5b5-1907-841a-40737bfed560
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=8929be5086660625b3e189da58ff1d3c4546d1c660718febb3b4d5b379f955c8
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F_temporary%2Fattempt_202208121624485385779855500771815_0003_m_000000_16%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 8b9dc119-a5b5-1907-841a-40737bfed560[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=8929be5086660625b3e189da58ff1d3c4546d1c660718febb3b4d5b379f955c8[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: f21dc649-ab17-47ba-b9cf-9f4a27494ad7[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: f21dc649-ab17-47ba-b9cf-9f4a27494ad7[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 359[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16/</Prefix></ListBucketResult>"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: f21dc649-ab17-47ba-b9cf-9f4a27494ad7
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: f21dc649-ab17-47ba-b9cf-9f4a27494ad7
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 359
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: f21dc649-ab17-47ba-b9cf-9f4a27494ad7
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: f21dc649-ab17-47ba-b9cf-9f4a27494ad7
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: f21dc649-ab17-47ba-b9cf-9f4a27494ad7
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 18
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[f21dc649-ab17-47ba-b9cf-9f4a27494ad7], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.969], ClientExecuteTime=[19.841], HttpClientSendRequestTime=[0.513], HttpRequestTime=[16.61], ApiCallLatency=[19.577], RequestSigningTime=[0.321], CredentialsRequestTime=[0.007, 0.003], HttpClientReceiveResponseTime=[15.378],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.020s
22/08/12 16:24:51 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0/_temporary/attempt_202208121624485385779855500771815_0003_m_000000_16
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter op_exists.failures by 1 with final value 2
22/08/12 16:24:51 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_202208121624485385779855500771815_0003_m_000000_16
22/08/12 16:24:51 WARN BasicWriteTaskStatsTracker: Expected 1 files, but only saw 0. This could be due to the output format not writing empty files, or files being not immediately visible in the filesystem.
22/08/12 16:24:51 INFO Executor: Finished task 0.0 in stage 3.0 (TID 16). 2914 bytes result sent to driver
22/08/12 16:24:51 DEBUG ExecutorMetricsPoller: stageTCMP: (3, 0) -> 0
22/08/12 16:24:51 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 16) in 3418 ms on 10.1.255.235 (executor driver) (1/1)
22/08/12 16:24:51 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
22/08/12 16:24:51 INFO DAGScheduler: ResultStage 3 (save at NativeMethodAccessorImpl.java:0) finished in 3.452 s
22/08/12 16:24:51 DEBUG DAGScheduler: After removal of stage 3, remaining stages = 0
22/08/12 16:24:51 INFO DAGScheduler: Job 3 is finished. Cancelling potential speculative or zombie tasks for this job
22/08/12 16:24:51 INFO TaskSchedulerImpl: Killing all running tasks in stage 3: Stage finished
22/08/12 16:24:51 DEBUG SQLHadoopMapReduceCommitProtocol: onTaskCommit(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage@224dab57)
22/08/12 16:24:51 INFO DAGScheduler: Job 3 finished: save at NativeMethodAccessorImpl.java:0, took 3.459553 s
22/08/12 16:24:51 INFO FileFormatWriter: Start to commit write Job 1579bf8c-1bfb-448e-994b-bf13ffb6fe44.
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter op_list_status by 1 with final value 1
22/08/12 16:24:51 DEBUG Invoker: Starting: listStatus
22/08/12 16:24:51 DEBUG S3AFileSystem: List status for path: s3a://master.rando2/nonemptyprefix8/_temporary/0
22/08/12 16:24:51 DEBUG S3AFileSystem: listStatus: doing listObjects for directory nonemptyprefix8/_temporary/0/
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/ delimiter=/ keys=5000 requester pays=false
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 8
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 1
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["5000"],"prefix":["nonemptyprefix8/_temporary/0/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 3011fe8b-f41b-86d6-56f5-d169b3fffbc1, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=5000&prefix=nonemptyprefix8%2F_temporary%2F0%2F
amz-sdk-invocation-id:3011fe8b-f41b-86d6-56f5-d169b3fffbc1
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152451Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152451Z
20220812/us-east-1/s3/aws4_request
e3511709d2eee863d6c53c409365922ac12c78c26b0c9d1aa5782f5ab33d05c3"
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=5000&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=5000&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 3011fe8b-f41b-86d6-56f5-d169b3fffbc1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6c71d26c6694c471b3bb54ffeeecd8d246a0b64fa79abe27f20f0c188608d575
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=5000&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 3011fe8b-f41b-86d6-56f5-d169b3fffbc1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=6c71d26c6694c471b3bb54ffeeecd8d246a0b64fa79abe27f20f0c188608d575[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: af36717a-5c80-4d34-84f8-b041b08c5376[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: af36717a-5c80-4d34-84f8-b041b08c5376[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 292[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>5000</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/</Prefix></ListBucketResult>"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: af36717a-5c80-4d34-84f8-b041b08c5376
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: af36717a-5c80-4d34-84f8-b041b08c5376
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 292
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: af36717a-5c80-4d34-84f8-b041b08c5376
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: af36717a-5c80-4d34-84f8-b041b08c5376
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: af36717a-5c80-4d34-84f8-b041b08c5376
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 19
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[af36717a-5c80-4d34-84f8-b041b08c5376], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.787], ClientExecuteTime=[18.346], HttpClientSendRequestTime=[0.488], HttpRequestTime=[15.265], ApiCallLatency=[18.142], RequestSigningTime=[0.393], CredentialsRequestTime=[0.007, 0.004], HttpClientReceiveResponseTime=[14.104],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.019s
22/08/12 16:24:51 DEBUG S3AFileSystem: Added 0 entries; ignored 0; hasNext=false; hasMoreObjects=false
22/08/12 16:24:51 DEBUG S3AFileSystem: All entries in batch were filtered...continuing
22/08/12 16:24:51 DEBUG S3AFileSystem: Start iterating the provided status.
22/08/12 16:24:51 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary/0 (nonemptyprefix8/_temporary/0); needEmptyDirectory=false
22/08/12 16:24:51 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary/0
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 8
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 8
22/08/12 16:24:51 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary/0 with change tracker null
22/08/12 16:24:51 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary/0 Headers: (amz-sdk-invocation-id: e494962e-c10c-7eb4-0a66-d6491ec50ef4, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix8/_temporary/0
amz-sdk-invocation-id:e494962e-c10c-7eb4-0a66-d6491ec50ef4
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152451Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152451Z
20220812/us-east-1/s3/aws4_request
8b09a1ea04a37a1688c0ddb5c634fc72c81ac78ee150f8c6952bcf472c1f980e"
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: e494962e-c10c-7eb4-0a66-d6491ec50ef4
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=da3421c4076f13bd1eefe66623084ae3b32b8d9d1ce3d3562b00ba8a0722643c
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary/0 HTTP/1.1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: e494962e-c10c-7eb4-0a66-d6491ec50ef4[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=da3421c4076f13bd1eefe66623084ae3b32b8d9d1ce3d3562b00ba8a0722643c[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: d782dd59-cc6f-4369-9ffe-1014297ddb9d[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: d782dd59-cc6f-4369-9ffe-1014297ddb9d[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 251[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: d782dd59-cc6f-4369-9ffe-1014297ddb9d
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: d782dd59-cc6f-4369-9ffe-1014297ddb9d
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 251
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d; S3 Extended Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d; Proxy: null), S3 Extended Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 20
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d; S3 Extended Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d; Proxy: null), S3 Extended Request ID: d782dd59-cc6f-4369-9ffe-1014297ddb9d], RequestType=[GetObjectMetadataRequest], AWSRequestID=[d782dd59-cc6f-4369-9ffe-1014297ddb9d], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[15.068], HttpClientSendRequestTime=[0.654], HttpRequestTime=[13.776], ApiCallLatency=[14.845], RequestSigningTime=[0.319], CredentialsRequestTime=[0.007, 0.003], HttpClientReceiveResponseTime=[12.373],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/0/ delimiter=/ keys=2 requester pays=false
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 9
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/0/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 102525c2-7281-4568-38a0-0d644d4e8781, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F
amz-sdk-invocation-id:102525c2-7281-4568-38a0-0d644d4e8781
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152451Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152451Z
20220812/us-east-1/s3/aws4_request
b1b6315da502059e7d80bd950896f44a02b4be883c275342cb87f26ca4b9f0ef"
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 102525c2-7281-4568-38a0-0d644d4e8781
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c21dc7ada620ea45e9b848b24fc70f8f42e2da7b4d1e2e2f1846cb69a00aaeac
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F0%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 102525c2-7281-4568-38a0-0d644d4e8781[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c21dc7ada620ea45e9b848b24fc70f8f42e2da7b4d1e2e2f1846cb69a00aaeac[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 35778686-73a8-4b45-a002-e6315654ad1f[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 35778686-73a8-4b45-a002-e6315654ad1f[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 289[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/0/</Prefix></ListBucketResult>"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 35778686-73a8-4b45-a002-e6315654ad1f
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 35778686-73a8-4b45-a002-e6315654ad1f
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 289
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: 35778686-73a8-4b45-a002-e6315654ad1f
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: 35778686-73a8-4b45-a002-e6315654ad1f
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: 35778686-73a8-4b45-a002-e6315654ad1f
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 21
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[35778686-73a8-4b45-a002-e6315654ad1f], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.608], ClientExecuteTime=[17.536], HttpClientSendRequestTime=[0.496], HttpRequestTime=[14.694], ApiCallLatency=[17.336], RequestSigningTime=[0.331], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[13.589],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.018s
22/08/12 16:24:51 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary/0
22/08/12 16:24:51 DEBUG Invoker: listStatus: duration 0:00.057s
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter op_list_status.failures by 1 with final value 1
22/08/12 16:24:51 ERROR FileFormatWriter: Aborting job 1579bf8c-1bfb-448e-994b-bf13ffb6fe44.
java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix8/_temporary/0
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:3300)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$null$20(S3AFileSystem.java:3264)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$21(S3AFileSystem.java:3263)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3262)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2014)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:334)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:404)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:377)
at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:48)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:192)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$25(FileFormatWriter.scala:267)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:642)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:267)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:186)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
22/08/12 16:24:51 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/_temporary (nonemptyprefix8/_temporary); needEmptyDirectory=true
22/08/12 16:24:51 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/_temporary
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 9
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 9
22/08/12 16:24:51 DEBUG S3AFileSystem: HEAD nonemptyprefix8/_temporary with change tracker null
22/08/12 16:24:51 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/_temporary Headers: (amz-sdk-invocation-id: e7507793-ba60-18ef-a010-f51284433e19, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix8/_temporary
amz-sdk-invocation-id:e7507793-ba60-18ef-a010-f51284433e19
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152451Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152451Z
20220812/us-east-1/s3/aws4_request
98bdab4f625a5e1d2a55a8523f53e80a6d21f926bb162c9aebecda5f6f725761"
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: e7507793-ba60-18ef-a010-f51284433e19
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=1a966b377e5ec0ceaff0728e2ff5fe4458c42ac90422ff136eec6a6cdf8bd78d
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/_temporary HTTP/1.1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: e7507793-ba60-18ef-a010-f51284433e19[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=1a966b377e5ec0ceaff0728e2ff5fe4458c42ac90422ff136eec6a6cdf8bd78d[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 3de2dd19-68e9-4f52-aff3-04b834d9d45a[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 3de2dd19-68e9-4f52-aff3-04b834d9d45a[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 249[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 3de2dd19-68e9-4f52-aff3-04b834d9d45a
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 3de2dd19-68e9-4f52-aff3-04b834d9d45a
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 249
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a; S3 Extended Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a; Proxy: null), S3 Extended Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 22
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a; S3 Extended Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a; Proxy: null), S3 Extended Request ID: 3de2dd19-68e9-4f52-aff3-04b834d9d45a], RequestType=[GetObjectMetadataRequest], AWSRequestID=[3de2dd19-68e9-4f52-aff3-04b834d9d45a], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[15.174], HttpClientSendRequestTime=[0.456], HttpRequestTime=[14.17], ApiCallLatency=[14.979], RequestSigningTime=[0.217], CredentialsRequestTime=[0.007, 0.003], HttpClientReceiveResponseTime=[12.958],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/_temporary/ delimiter=/ keys=2 requester pays=false
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 10
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/_temporary/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 42665700-8a06-44b3-9b47-1dd927dcbcea, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F
amz-sdk-invocation-id:42665700-8a06-44b3-9b47-1dd927dcbcea
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152451Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152451Z
20220812/us-east-1/s3/aws4_request
b0507684371063ddfc5ff4dd6ab9000619ee52f59de7230c4d3df288590c111e"
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 42665700-8a06-44b3-9b47-1dd927dcbcea
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=ff64a0de247a9499296ab46858db31d9b09bc37dd52951ee12334699d9f86bb0
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F_temporary%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 42665700-8a06-44b3-9b47-1dd927dcbcea[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=ff64a0de247a9499296ab46858db31d9b09bc37dd52951ee12334699d9f86bb0[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 311c11a7-8c34-4e47-a231-4f55043280c7[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 311c11a7-8c34-4e47-a231-4f55043280c7[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 287[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/_temporary/</Prefix></ListBucketResult>"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 311c11a7-8c34-4e47-a231-4f55043280c7
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 311c11a7-8c34-4e47-a231-4f55043280c7
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 287
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: 311c11a7-8c34-4e47-a231-4f55043280c7
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: 311c11a7-8c34-4e47-a231-4f55043280c7
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: 311c11a7-8c34-4e47-a231-4f55043280c7
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 23
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[311c11a7-8c34-4e47-a231-4f55043280c7], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.769], ClientExecuteTime=[17.922], HttpClientSendRequestTime=[0.475], HttpRequestTime=[15.02], ApiCallLatency=[17.736], RequestSigningTime=[0.266], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[13.818],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.018s
22/08/12 16:24:51 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/_temporary
22/08/12 16:24:51 DEBUG S3AFileSystem: Couldn't delete s3a://master.rando2/nonemptyprefix8/_temporary - does not exist: java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix8/_temporary
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter ignored_errors by 1 with final value 1
22/08/12 16:24:51 DEBUG S3AFileSystem: Getting path status for s3a://master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc (nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc); needEmptyDirectory=true
22/08/12 16:24:51 DEBUG S3AFileSystem: S3GetFileStatus s3a://master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_metadata_request by 1 with final value 10
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter action_http_head_request by 1 with final value 10
22/08/12 16:24:51 DEBUG S3AFileSystem: HEAD nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc with change tracker null
22/08/12 16:24:51 DEBUG request: Sending Request: HEAD http://localhost:30600 /master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc Headers: (amz-sdk-invocation-id: b399a93c-20e2-c4fc-8e8f-fac69f36d235, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"HEAD
/master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc
amz-sdk-invocation-id:b399a93c-20e2-c4fc-8e8f-fac69f36d235
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152451Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152451Z
20220812/us-east-1/s3/aws4_request
15860be2bf2524f1fb7674bc176ab7da26428ecc133b5794ec5a51628b2f32ed"
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG MainClientExec: Executing request HEAD /master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc HTTP/1.1
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> HEAD /master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc HTTP/1.1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: b399a93c-20e2-c4fc-8e8f-fac69f36d235
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9ab50c61b7c0bc5ae1f2416adefad6e998ea1f56834c75b2fe9d83ec8a9a7dc8
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "HEAD /master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc HTTP/1.1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: b399a93c-20e2-c4fc-8e8f-fac69f36d235[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9ab50c61b7c0bc5ae1f2416adefad6e998ea1f56834c75b2fe9d83ec8a9a7dc8[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 1708d265-86bd-4376-8203-97c31bb17675[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 1708d265-86bd-4376-8203-97c31bb17675[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 290[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 404 Not Found
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 1708d265-86bd-4376-8203-97c31bb17675
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 1708d265-86bd-4376-8203-97c31bb17675
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 290
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 1708d265-86bd-4376-8203-97c31bb17675; S3 Extended Request ID: 1708d265-86bd-4376-8203-97c31bb17675; Proxy: null), S3 Extended Request ID: 1708d265-86bd-4376-8203-97c31bb17675
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 24
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], StatusCode=[404], ServiceEndpoint=[http://localhost:30600], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 1708d265-86bd-4376-8203-97c31bb17675; S3 Extended Request ID: 1708d265-86bd-4376-8203-97c31bb17675; Proxy: null), S3 Extended Request ID: 1708d265-86bd-4376-8203-97c31bb17675], RequestType=[GetObjectMetadataRequest], AWSRequestID=[1708d265-86bd-4376-8203-97c31bb17675], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[14.502], HttpClientSendRequestTime=[0.494], HttpRequestTime=[13.466], ApiCallLatency=[14.307], RequestSigningTime=[0.262], CredentialsRequestTime=[0.006, 0.003], HttpClientReceiveResponseTime=[12.258],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST List master.rando2:/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc/ delimiter=/ keys=2 requester pays=false
22/08/12 16:24:51 DEBUG S3AFileSystem: Starting: LIST
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter object_list_request by 1 with final value 11
22/08/12 16:24:51 DEBUG request: Sending Request: GET http://localhost:30600 /master.rando2/ Parameters: ({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc/"],"fetch-owner":["false"]}Headers: (amz-sdk-invocation-id: 4602ce4e-6b53-2da6-893c-d421f16d381c, Content-Type: application/octet-stream, User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy, )
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 Canonical Request: '"GET
/master.rando2/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=nonemptyprefix8%2F.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc%2F
amz-sdk-invocation-id:4602ce4e-6b53-2da6-893c-d421f16d381c
amz-sdk-request:ttl=20220812T152811Z;attempt=1;max=21
amz-sdk-retry:0/0/500
content-type:application/octet-stream
host:localhost:30600
user-agent:Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220812T152451Z
amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
22/08/12 16:24:51 DEBUG AWS4Signer: AWS4 String to Sign: '"AWS4-HMAC-SHA256
20220812T152451Z
20220812/us-east-1/s3/aws4_request
61f3f8c13b34f3760cc32e7971be556e54e9e09677dda81206ff2dec99d03626"
22/08/12 16:24:51 DEBUG RequestAddCookies: CookieSpec selected: default
22/08/12 16:24:51 DEBUG RequestAuthCache: Auth cache not set in the context
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection request: [route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://localhost:30600][total available: 0; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 200000
22/08/12 16:24:51 DEBUG MainClientExec: Executing request GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG MainClientExec: Proxy auth state: UNCHALLENGED
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc%2F&fetch-owner=false HTTP/1.1
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Host: localhost:30600
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-invocation-id: 4602ce4e-6b53-2da6-893c-d421f16d381c
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> amz-sdk-retry: 0/0/500
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=53e7ec49f932c5ef39cc90a689142c9d7de1139aa6c2e749b3259c6f256d863f
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Type: application/octet-stream
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> X-Amz-Date: 20220812T152451Z
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Content-Length: 0
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 >> Connection: Keep-Alive
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "GET /master.rando2/?list-type=2&delimiter=%2F&max-keys=2&prefix=nonemptyprefix8%2F.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc%2F&fetch-owner=false HTTP/1.1[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Host: localhost:30600[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-invocation-id: 4602ce4e-6b53-2da6-893c-d421f16d381c[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-request: ttl=20220812T152811Z;attempt=1;max=21[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=lemon/20220812/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=53e7ec49f932c5ef39cc90a689142c9d7de1139aa6c2e749b3259c6f256d863f[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "User-Agent: Hadoop 3.3.2, aws-sdk-java/1.12.264 Linux/5.15.0-40-generic OpenJDK_64-Bit_Server_VM/11.0.16+8-post-Ubuntu-0ubuntu122.04 java/11.0.16 scala/2.12.15 vendor/Ubuntu cfg/retry-mode/legacy[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "X-Amz-Date: 20220812T152451Z[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Content-Length: 0[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 >> "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Id-2: 5076f85d-0a42-4fd9-b066-797a75c1dc78[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "X-Amz-Request-Id: 5076f85d-0a42-4fd9-b066-797a75c1dc78[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Date: Fri, 12 Aug 2022 15:24:51 GMT[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "Content-Length: 328[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "[\r][\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
22/08/12 16:24:51 DEBUG wire: http-outgoing-0 << "<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Marker></Marker><MaxKeys>2</MaxKeys><Name>master.rando2</Name><Prefix>nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc/</Prefix></ListBucketResult>"
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << HTTP/1.1 200 OK
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Type: application/xml
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Id-2: 5076f85d-0a42-4fd9-b066-797a75c1dc78
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << X-Amz-Request-Id: 5076f85d-0a42-4fd9-b066-797a75c1dc78
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Date: Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG headers: http-outgoing-0 << Content-Length: 328
22/08/12 16:24:51 DEBUG MainClientExec: Connection can be kept alive for 60000 MILLISECONDS
22/08/12 16:24:51 DEBUG ClockSkewAdjuster: Reported server date (from 'Date' header): Fri, 12 Aug 2022 15:24:51 GMT
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Sanitizing XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection [id: 0][route: {}->http://localhost:30600] can be kept alive for 60.0 seconds
22/08/12 16:24:51 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: set socket timeout to 0
22/08/12 16:24:51 DEBUG PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://localhost:30600][total available: 1; route allocated: 1 of 96; total allocated: 1 of 96]
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Parsing XML response document with handler: class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListObjectsV2Handler
22/08/12 16:24:51 DEBUG XmlResponsesSaxParser: Examining listing for bucket: master.rando2
22/08/12 16:24:51 DEBUG request: Received successful response: 200, AWS Request ID: 5076f85d-0a42-4fd9-b066-797a75c1dc78
22/08/12 16:24:51 DEBUG requestId: x-amzn-RequestId: not available
22/08/12 16:24:51 DEBUG requestId: AWS Request ID: 5076f85d-0a42-4fd9-b066-797a75c1dc78
22/08/12 16:24:51 DEBUG requestId: AWS Extended Request ID: 5076f85d-0a42-4fd9-b066-797a75c1dc78
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter store_io_request by 1 with final value 25
22/08/12 16:24:51 DEBUG latency: ServiceName=[Amazon S3], StatusCode=[200], ServiceEndpoint=[http://localhost:30600], RequestType=[ListObjectsV2Request], AWSRequestID=[5076f85d-0a42-4fd9-b066-797a75c1dc78], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, HttpClientPoolLeasedCount=0, ResponseProcessingTime=[1.598], ClientExecuteTime=[19.183], HttpClientSendRequestTime=[0.469], HttpRequestTime=[16.476], ApiCallLatency=[18.99], RequestSigningTime=[0.326], CredentialsRequestTime=[0.007, 0.004], HttpClientReceiveResponseTime=[15.417],
22/08/12 16:24:51 DEBUG S3AFileSystem: LIST: duration 0:00.020s
22/08/12 16:24:51 DEBUG S3AFileSystem: Not Found: s3a://master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc
22/08/12 16:24:51 DEBUG S3AFileSystem: Couldn't delete s3a://master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc - does not exist: java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix8/.spark-staging-31b8f587-d086-4d3b-bf6d-4ce13ea2fecc
22/08/12 16:24:51 DEBUG IOStatisticsStoreImpl: Incrementing counter ignored_errors by 1 with final value 2
Traceback (most recent call last):
File "/home/luke/pp/pachyderm/spark/spark.py", line 48, in <module>
df.coalesce(1).write.format("parquet").mode("overwrite").save("s3a://master.rando2/nonemptyprefix8")
File "/home/luke/pp/pachyderm/venv/lib/python3.10/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 968, in save
File "/home/luke/pp/pachyderm/venv/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
File "/home/luke/pp/pachyderm/venv/lib/python3.10/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 190, in deco
File "/home/luke/pp/pachyderm/venv/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o243.save.
: org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:638)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:278)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:186)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://master.rando2/nonemptyprefix8/_temporary/0
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3866)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3688)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:3300)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$null$20(S3AFileSystem.java:3264)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$21(S3AFileSystem.java:3263)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3262)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2014)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:334)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:404)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:377)
at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:48)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:192)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$25(FileFormatWriter.scala:267)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:642)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:267)
... 41 more
22/08/12 16:24:51 INFO SparkContext: Invoking stop() from shutdown hook
22/08/12 16:24:51 INFO SparkUI: Stopped Spark web UI at http://10.1.255.235:4040
22/08/12 16:24:51 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/08/12 16:24:52 INFO MemoryStore: MemoryStore cleared
22/08/12 16:24:52 INFO BlockManager: BlockManager stopped
22/08/12 16:24:52 INFO BlockManagerMaster: BlockManagerMaster stopped
22/08/12 16:24:52 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/08/12 16:24:52 INFO SparkContext: Successfully stopped SparkContext
22/08/12 16:24:52 INFO ShutdownHookManager: Shutdown hook called
22/08/12 16:24:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-675a0fcb-380d-4f5f-8991-c288769c744f
22/08/12 16:24:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e/pyspark-3de0ca21-97fc-444f-ae94-cbae398755d5
22/08/12 16:24:52 INFO ShutdownHookManager: Deleting directory /tmp/spark-7ddbe69a-a202-4991-b257-71af8961db3e
22/08/12 16:24:52 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.FilterFileSystem.close(FilterFileSystem.java:529)); Key: (luke (auth:SIMPLE))@file://; URI: file:///; Object Identity Hash: 60fc0331
22/08/12 16:24:52 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.RawLocalFileSystem.close(RawLocalFileSystem.java:759)); Key: null; URI: file:///; Object Identity Hash: 3221f19
22/08/12 16:24:52 DEBUG S3AFileSystem: Filesystem s3a://master.rando2 is closed
22/08/12 16:24:52 DEBUG S3AFileSystem: IOStatistics: counters=((stream_read_exceptions=0) (object_metadata_request=10) (op_hsync=0) (object_multipart_aborted.failures=0) (s3guard_metadatastore_record_reads=0) (op_glob_status.failures=0) (op_list_located_status=0) (stream_read_fully_operations=0) (stream_read_operations=0) (files_copied=0) (object_continue_list_request.failures=0) (op_get_file_checksum.failures=0) (multipart_upload_part_put=0) (store_exists_probe.failures=0) (action_executor_acquired=0) (store_io_retry=0) (object_put_request=2) (multipart_upload_aborted=0) (multipart_instantiated=0) (stream_read_operations_incomplete=0) (object_put_request.failures=0) (stream_read_seek_policy_changed=0) (object_put_request_completed=2) (fake_directories_created=0) (stream_read_bytes_discarded_in_close=0) (s3guard_metadatastore_record_deletes=0) (stream_write_exceptions=0) (object_continue_list_request=0) (delegation_tokens_issued=0) (directories_deleted=0) (op_access.failures=0) (committer_bytes_uploaded=0) (committer_commit_job.failures=0) (op_get_delegation_token=0) (op_exists.failures=2) (op_list_files=0) (stream_read_seek_bytes_discarded=0) (files_copied_bytes=0) (op_create.failures=0) (committer_stage_file_upload=0) (s3guard_metadatastore_initialization=0) (op_get_file_checksum=0) (op_mkdirs.failures=0) (stream_read_version_mismatches=0) (multipart_upload_completed=0) (op_access=0) (object_delete_request=0) (stream_read_opened=0) (audit_span_creation=0) (op_exists=2) (s3guard_metadatastore_retry=0) (op_copy_from_local_file=0) (committer_tasks_failed=0) (stream_write_block_uploads=1) (committer_commit_job=0) (multipart_upload_list=0) (multipart_upload_part_put_bytes=0) (stream_write_total_data=1436) (multipart_upload_started=0) (object_bulk_delete_request=2) (committer_magic_files_created=0) (files_created=1) (stream_read_seek_operations=0) (op_get_content_summary=0) (op_create_non_recursive=0) (stream_read_bytes_discarded_in_abort=0) (stream_read_total_bytes=0) (committer_jobs_failed=0) (op_copy_from_local_file.failures=0) (object_multipart_aborted=0) (stream_write_block_uploads_committed=0) (op_is_file.failures=0) (op_is_directory.failures=0) (s3guard_metadatastore_put_path_request=0) (stream_write_queue_duration=0) (op_is_file=0) (committer_commits_created=0) (committer_tasks_completed=0) (action_executor_acquired.failures=0) (object_delete_objects=7) (ignored_errors=2) (stream_write_exceptions_completing_upload=0) (committer_bytes_committed=0) (stream_read_close_operations=0) (op_list_status=1) (object_copy_requests=0) (fake_directories_deleted=7) (stream_read_bytes=0) (s3guard_metadatastore_record_writes=0) (store_exists_probe=0) (op_hflush=0) (op_xattr_get_named=0) (object_multipart_initiated.failures=0) (action_http_get_request=0) (op_is_directory=0) (action_http_get_request.failures=0) (op_xattr_get_named_map.failures=0) (committer_materialize_file=0) (op_open=0) (s3guard_metadatastore_authoritative_directories_updated=0) (stream_read_bytes_backwards_on_seek=0) (audit_failure=0) (committer_commits.failures=0) (op_delete.failures=0) (files_delete_rejected=0) (files_deleted=0) (action_http_head_request=10) (op_rename=0) (s3guard_metadatastore_throttled=0) (stream_write_total_time=0) (committer_commits_completed=0) (op_get_file_status=1) (object_put_bytes=718) (stream_read_seek_bytes_skipped=0) (op_abort=0) (object_list_request.failures=0) (stream_read_seek_forward_operations=0) (op_create=1) (store_io_throttled=0) (multipart_upload_abort_under_path_invoked=0) (store_io_request=25) (object_list_request=11) (op_xattr_get_named.failures=0) (op_mkdirs=1) (committer_commits_reverted=0) (op_rename.failures=0) (op_delete=0) (op_list_status.failures=1) (op_xattr_get_map=0) (object_delete_request.failures=0) (op_xattr_get_named_map=0) (op_xattr_list.failures=0) (op_get_delegation_token.failures=0) (committer_commits_aborted=0) (object_bulk_delete_request.failures=0) (op_get_file_status.failures=1) (committer_stage_file_upload.failures=0) (delegation_tokens_issued.failures=0) (op_get_content_summary.failures=0) (op_glob_status=0) (stream_write_queue_duration.failures=0) (stream_write_bytes=718) (op_abort.failures=0) (committer_materialize_file.failures=0) (multipart_upload_list.failures=0) (op_xattr_list=0) (audit_request_execution=0) (object_multipart_initiated=0) (op_list_files.failures=0) (stream_write_block_uploads_aborted=0) (op_xattr_get_map.failures=0) (object_select_requests=0) (directories_created=1) (stream_aborted=0) (stream_read_seek_backward_operations=0) (committer_jobs_completed=0) (action_http_head_request.failures=0) (stream_read_closed=0) (audit_access_check_failure=0));
gauges=((stream_write_block_uploads_active=0) (object_put_bytes_pending=0) (object_put_request_active=0) (client_side_encryption_enabled=0) (stream_write_block_uploads_pending=1) (stream_write_block_uploads_data_pending=0));
minimums=((op_get_content_summary.failures.min=-1) (object_bulk_delete_request.failures.min=-1) (op_list_status.min=-1) (op_get_file_status.min=-1) (op_delete.failures.min=-1) (op_exists.failures.min=43) (op_copy_from_local_file.min=-1) (delegation_tokens_issued.failures.min=-1) (op_xattr_get_map.failures.min=-1) (op_is_directory.failures.min=-1) (committer_commit_job.min=-1) (multipart_upload_list.min=-1) (op_xattr_list.min=-1) (op_delete.min=-1) (op_exists.min=-1) (op_glob_status.min=-1) (object_put_request.failures.min=-1) (action_http_get_request.min=-1) (object_continue_list_request.failures.min=-1) (object_delete_request.failures.min=-1) (object_multipart_aborted.failures.min=-1) (object_multipart_initiated.min=-1) (committer_materialize_file.failures.min=-1) (store_exists_probe.min=-1) (op_xattr_get_named.min=-1) (object_multipart_aborted.min=-1) (op_access.min=-1) (store_exists_probe.failures.min=-1) (op_mkdirs.min=451) (op_is_file.min=-1) (op_is_file.failures.min=-1) (multipart_upload_list.failures.min=-1) (delegation_tokens_issued.min=-1) (op_xattr_list.failures.min=-1) (object_list_request.failures.min=-1) (action_http_get_request.failures.min=-1) (op_get_file_checksum.failures.min=-1) (op_rename.failures.min=-1) (op_glob_status.failures.min=-1) (action_http_head_request.failures.min=-1) (op_create.failures.min=-1) (op_list_files.failures.min=-1) (stream_write_queue_duration.min=-1) (action_executor_acquired.failures.min=-1) (op_xattr_get_named_map.min=-1) (op_abort.min=-1) (op_get_delegation_token.min=-1) (op_list_status.failures.min=57) (op_get_file_checksum.min=-1) (op_list_files.min=-1) (committer_stage_file_upload.min=-1) (op_abort.failures.min=-1) (op_get_delegation_token.failures.min=-1) (stream_write_queue_duration.failures.min=-1) (op_xattr_get_named.failures.min=-1) (committer_stage_file_upload.failures.min=-1) (op_rename.min=-1) (op_copy_from_local_file.failures.min=-1) (object_multipart_initiated.failures.min=-1) (op_create.min=98) (op_access.failures.min=-1) (action_http_head_request.min=14) (object_delete_request.min=-1) (committer_materialize_file.min=-1) (op_xattr_get_map.min=-1) (committer_commit_job.failures.min=-1) (object_continue_list_request.min=-1) (object_list_request.min=18) (op_get_content_summary.min=-1) (op_xattr_get_named_map.failures.min=-1) (object_put_request.min=189) (op_get_file_status.failures.min=782) (op_is_directory.min=-1) (object_bulk_delete_request.min=96) (action_executor_acquired.min=-1) (op_mkdirs.failures.min=-1));
maximums=((multipart_upload_list.failures.max=-1) (op_list_files.failures.max=-1) (committer_materialize_file.failures.max=-1) (op_get_file_status.failures.max=782) (object_multipart_initiated.max=-1) (object_delete_request.failures.max=-1) (op_access.failures.max=-1) (op_xattr_list.failures.max=-1) (op_get_delegation_token.max=-1) (op_glob_status.max=-1) (op_get_delegation_token.failures.max=-1) (object_multipart_initiated.failures.max=-1) (op_rename.max=-1) (op_get_file_status.max=-1) (op_exists.max=-1) (stream_write_queue_duration.failures.max=-1) (op_glob_status.failures.max=-1) (op_list_status.failures.max=57) (op_is_directory.failures.max=-1) (op_xattr_get_named.max=-1) (action_http_get_request.failures.max=-1) (op_get_file_checksum.max=-1) (action_executor_acquired.failures.max=-1) (op_get_content_summary.max=-1) (object_continue_list_request.failures.max=-1) (op_mkdirs.failures.max=-1) (op_xattr_get_named_map.max=-1) (object_multipart_aborted.failures.max=-1) (committer_commit_job.max=-1) (op_is_file.max=-1) (committer_stage_file_upload.failures.max=-1) (op_delete.failures.max=-1) (delegation_tokens_issued.max=-1) (op_copy_from_local_file.max=-1) (object_put_request.max=893) (op_delete.max=-1) (object_bulk_delete_request.failures.max=-1) (op_mkdirs.max=451) (committer_materialize_file.max=-1) (committer_commit_job.failures.max=-1) (op_xattr_get_map.failures.max=-1) (op_xattr_get_named_map.failures.max=-1) (object_put_request.failures.max=-1) (op_is_file.failures.max=-1) (object_list_request.failures.max=-1) (object_continue_list_request.max=-1) (op_create.max=98) (object_multipart_aborted.max=-1) (store_exists_probe.failures.max=-1) (action_http_head_request.failures.max=-1) (object_list_request.max=82) (op_get_content_summary.failures.max=-1) (op_xattr_list.max=-1) (delegation_tokens_issued.failures.max=-1) (object_bulk_delete_request.max=120) (op_copy_from_local_file.failures.max=-1) (store_exists_probe.max=-1) (action_executor_acquired.max=-1) (multipart_upload_list.max=-1) (op_xattr_get_map.max=-1) (op_xattr_get_named.failures.max=-1) (op_get_file_checksum.failures.max=-1) (action_http_head_request.max=756) (op_list_status.max=-1) (op_exists.failures.max=239) (op_access.max=-1) (op_is_directory.max=-1) (committer_stage_file_upload.max=-1) (op_abort.failures.max=-1) (op_create.failures.max=-1) (op_list_files.max=-1) (stream_write_queue_duration.max=-1) (op_abort.max=-1) (action_http_get_request.max=-1) (op_rename.failures.max=-1) (object_delete_request.max=-1));
means=((op_exists.failures.mean=(samples=2, sum=282, mean=141.0000)) (committer_commit_job.failures.mean=(samples=0, sum=0, mean=0.0000)) (committer_stage_file_upload.mean=(samples=0, sum=0, mean=0.0000)) (committer_commit_job.mean=(samples=0, sum=0, mean=0.0000)) (delegation_tokens_issued.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_is_directory.mean=(samples=0, sum=0, mean=0.0000)) (action_http_get_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_checksum.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_is_file.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named_map.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_initiated.mean=(samples=0, sum=0, mean=0.0000)) (object_put_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_list.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_status.mean=(samples=0, sum=0, mean=0.0000)) (op_is_directory.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_mkdirs.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_initiated.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_put_request.mean=(samples=2, sum=1082, mean=541.0000)) (op_access.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (op_delete.mean=(samples=0, sum=0, mean=0.0000)) (store_exists_probe.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_glob_status.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_copy_from_local_file.mean=(samples=0, sum=0, mean=0.0000)) (op_rename.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_list_files.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (store_exists_probe.mean=(samples=0, sum=0, mean=0.0000)) (object_delete_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (stream_write_queue_duration.mean=(samples=0, sum=0, mean=0.0000)) (op_create.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_is_file.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_list_status.failures.mean=(samples=1, sum=57, mean=57.0000)) (object_continue_list_request.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_checksum.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named.mean=(samples=0, sum=0, mean=0.0000)) (committer_materialize_file.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_list.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_copy_from_local_file.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_named_map.mean=(samples=0, sum=0, mean=0.0000)) (op_rename.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_list.mean=(samples=0, sum=0, mean=0.0000)) (committer_materialize_file.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_map.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_list.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_file_status.failures.mean=(samples=1, sum=782, mean=782.0000)) (op_exists.mean=(samples=0, sum=0, mean=0.0000)) (op_mkdirs.mean=(samples=1, sum=451, mean=451.0000)) (object_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_bulk_delete_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_http_head_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_delegation_token.failures.mean=(samples=0, sum=0, mean=0.0000)) (committer_stage_file_upload.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_get_content_summary.mean=(samples=0, sum=0, mean=0.0000)) (op_get_content_summary.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=0, sum=0, mean=0.0000)) (op_list_status.mean=(samples=0, sum=0, mean=0.0000)) (op_delete.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_create.mean=(samples=1, sum=98, mean=98.0000)) (object_delete_request.mean=(samples=0, sum=0, mean=0.0000)) (stream_write_queue_duration.failures.mean=(samples=0, sum=0, mean=0.0000)) (delegation_tokens_issued.mean=(samples=0, sum=0, mean=0.0000)) (object_bulk_delete_request.mean=(samples=2, sum=216, mean=108.0000)) (object_continue_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (op_xattr_get_map.mean=(samples=0, sum=0, mean=0.0000)) (op_glob_status.mean=(samples=0, sum=0, mean=0.0000)) (action_http_get_request.mean=(samples=0, sum=0, mean=0.0000)) (object_list_request.mean=(samples=11, sum=324, mean=29.4545)) (op_get_delegation_token.mean=(samples=0, sum=0, mean=0.0000)) (op_access.mean=(samples=0, sum=0, mean=0.0000)) (action_http_head_request.mean=(samples=10, sum=1047, mean=104.7000)) (op_list_files.failures.mean=(samples=0, sum=0, mean=0.0000)));
22/08/12 16:24:52 DEBUG FileSystem: FileSystem.close() by method: org.apache.hadoop.fs.s3a.S3AFileSystem.close(S3AFileSystem.java:4069)); Key: (luke (auth:SIMPLE))@s3a://master.rando2; URI: s3a://master.rando2; Object Identity Hash: 17051619
22/08/12 16:24:52 DEBUG PoolingHttpClientConnectionManager: Connection manager is shutting down
22/08/12 16:24:52 DEBUG IdleConnectionReaper: Reaper thread:
java.lang.InterruptedException: sleep interrupted
at java.base/java.lang.Thread.sleep(Native Method)
at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:188)
22/08/12 16:24:52 DEBUG IdleConnectionReaper: Shutting down reaper thread.
22/08/12 16:24:52 DEBUG DefaultManagedHttpClientConnection: http-outgoing-0: Close connection
22/08/12 16:24:52 DEBUG PoolingHttpClientConnectionManager: Connection manager shut down
22/08/12 16:24:52 DEBUG S3AFileSystem: Gracefully shutting down executor service. Waiting max 30 SECONDS
22/08/12 16:24:52 DEBUG S3AFileSystem: Succesfully shutdown executor service
22/08/12 16:24:52 DEBUG S3AFileSystem: Gracefully shutting down executor service. Waiting max 30 SECONDS
22/08/12 16:24:52 DEBUG S3AFileSystem: Succesfully shutdown executor service
22/08/12 16:24:52 DEBUG MBeans: Unregistering Hadoop:service=s3a-file-system,name=S3AMetrics1-master.rando2
22/08/12 16:24:52 DEBUG S3AInstrumentation: Shutting down metrics publisher
22/08/12 16:24:52 DEBUG MetricsSystemImpl: refCount=1
22/08/12 16:24:52 INFO MetricsSystemImpl: Stopping s3a-file-system metrics system...
22/08/12 16:24:52 DEBUG MBeans: Unregistering Hadoop:service=s3a-file-system,name=MetricsSystem,sub=Stats
22/08/12 16:24:52 INFO MetricsSystemImpl: s3a-file-system metrics system stopped.
22/08/12 16:24:52 DEBUG MBeans: Unregistering Hadoop:service=s3a-file-system,name=MetricsSystem,sub=Control
22/08/12 16:24:52 INFO MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
22/08/12 16:24:52 DEBUG SignerManager: Unregistering fs from 0 initializers
22/08/12 16:24:52 DEBUG AbstractService: Service: NoopAuditManagerS3A entered state STOPPED
22/08/12 16:24:52 DEBUG CompositeService: NoopAuditManagerS3A: stopping services, size=1
22/08/12 16:24:52 DEBUG CompositeService: Stopping service #0: Service NoopAuditor in state NoopAuditor: STARTED
22/08/12 16:24:52 DEBUG AbstractService: Service: NoopAuditor entered state STOPPED
22/08/12 16:24:52 DEBUG S3AFileSystem: Closing AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@717fe440] last provider: SimpleAWSCredentialsProvider
22/08/12 16:24:52 DEBUG AWSCredentialProviderList: Closing AWSCredentialProviderList[refcount= 0: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@717fe440] last provider: SimpleAWSCredentialsProvider
22/08/12 16:24:52 DEBUG S3AFileSystem: Statistics for s3a://master.rando2: counters=((action_http_head_request=10)
(directories_created=1)
(fake_directories_deleted=7)
(files_created=1)
(ignored_errors=2)
(object_bulk_delete_request=2)
(object_delete_objects=7)
(object_list_request=11)
(object_metadata_request=10)
(object_put_bytes=718)
(object_put_request=2)
(object_put_request_completed=2)
(op_create=1)
(op_exists=2)
(op_exists.failures=2)
(op_get_file_status=1)
(op_get_file_status.failures=1)
(op_list_status=1)
(op_list_status.failures=1)
(op_mkdirs=1)
(store_io_request=25)
(stream_write_block_uploads=1)
(stream_write_bytes=718)
(stream_write_total_data=1436));
gauges=((stream_write_block_uploads_pending=1));
minimums=((action_http_head_request.min=14)
(object_bulk_delete_request.min=96)
(object_list_request.min=18)
(object_put_request.min=189)
(op_create.min=98)
(op_exists.failures.min=43)
(op_get_file_status.failures.min=782)
(op_list_status.failures.min=57)
(op_mkdirs.min=451));
maximums=((action_http_head_request.max=756)
(object_bulk_delete_request.max=120)
(object_list_request.max=82)
(object_put_request.max=893)
(op_create.max=98)
(op_exists.failures.max=239)
(op_get_file_status.failures.max=782)
(op_list_status.failures.max=57)
(op_mkdirs.max=451));
means=((action_http_head_request.mean=(samples=10, sum=1047, mean=104.7000))
(object_bulk_delete_request.mean=(samples=2, sum=216, mean=108.0000))
(object_list_request.mean=(samples=11, sum=324, mean=29.4545))
(object_put_request.mean=(samples=2, sum=1082, mean=541.0000))
(op_create.mean=(samples=1, sum=98, mean=98.0000))
(op_exists.failures.mean=(samples=2, sum=282, mean=141.0000))
(op_get_file_status.failures.mean=(samples=1, sum=782, mean=782.0000))
(op_list_status.failures.mean=(samples=1, sum=57, mean=57.0000))
(op_mkdirs.mean=(samples=1, sum=451, mean=451.0000)));
22/08/12 16:24:52 DEBUG ShutdownHookManager: Completed shutdown in 0.067 seconds; Timeouts: 0
22/08/12 16:24:52 DEBUG ShutdownHookManager: ShutdownHookManager completed shutdown.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment