Skip to content

Instantly share code, notes, and snippets.

@joestein

joestein/gist:6eeceb4eb8b2f80c83c684a96ee31cd5

Created November 19, 2023 20:03
Show Gist options
  • Save joestein/6eeceb4eb8b2f80c83c684a96ee31cd5 to your computer and use it in GitHub Desktop.
Save joestein/6eeceb4eb8b2f80c83c684a96ee31cd5 to your computer and use it in GitHub Desktop.
Download ZIP
error with spark and iceberg with nessie
Raw
gistfile1.txt
charmalloc@Joes-MacBook-Pro getting-started-apachespark-apachekafka-apacheiceberg % ./create-table.sh anothertester2 yoyoma3
:: loading settings :: url = jar:file:/Users/charmalloc/spark-3.5.0-bin-hadoop3/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /Users/charmalloc/.ivy2/cache
The jars for the packages stored in: /Users/charmalloc/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
org.apache.spark#spark-sql-kafka-0-10_2.12 added as a dependency
org.apache.spark#spark-avro_2.12 added as a dependency
org.apache.iceberg#iceberg-spark-runtime-3.3_2.12 added as a dependency
software.amazon.awssdk#bundle added as a dependency
software.amazon.awssdk#url-connection-client added as a dependency
org.projectnessie.nessie-integrations#nessie-spark-extensions-3.5_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-dfba919b-b3c9-4874-80b4-7733434a5f07;1.0
confs: [default]
found org.apache.hadoop#hadoop-aws;3.2.2 in central
found com.amazonaws#aws-java-sdk-bundle;1.11.563 in central
found org.apache.spark#spark-sql-kafka-0-10_2.12;3.5.0 in central
found org.apache.spark#spark-token-provider-kafka-0-10_2.12;3.5.0 in central
found org.apache.kafka#kafka-clients;3.4.1 in local-m2-cache
found org.lz4#lz4-java;1.8.0 in local-m2-cache
found org.xerial.snappy#snappy-java;1.1.10.3 in central
found org.slf4j#slf4j-api;2.0.7 in central
found org.apache.hadoop#hadoop-client-runtime;3.3.4 in central
found org.apache.hadoop#hadoop-client-api;3.3.4 in central
found commons-logging#commons-logging;1.1.3 in local-m2-cache
found com.google.code.findbugs#jsr305;3.0.0 in local-m2-cache
found org.apache.commons#commons-pool2;2.11.1 in local-m2-cache
found org.apache.spark#spark-avro_2.12;3.5.0 in central
found org.tukaani#xz;1.9 in local-m2-cache
found org.apache.iceberg#iceberg-spark-runtime-3.3_2.12;1.4.1 in central
found software.amazon.awssdk#bundle;2.18.31 in central
found software.amazon.eventstream#eventstream;1.0.1 in central
found software.amazon.awssdk#url-connection-client;2.18.31 in central
found software.amazon.awssdk#utils;2.18.31 in central
found org.reactivestreams#reactive-streams;1.0.3 in local-m2-cache
found software.amazon.awssdk#annotations;2.18.31 in central
found software.amazon.awssdk#http-client-spi;2.18.31 in central
found software.amazon.awssdk#metrics-spi;2.18.31 in central
found org.projectnessie.nessie-integrations#nessie-spark-extensions-3.5_2.12;0.73.0 in central
:: resolution report :: resolve 303ms :: artifacts dl 8ms
:: modules in use:
com.amazonaws#aws-java-sdk-bundle;1.11.563 from central in [default]
com.google.code.findbugs#jsr305;3.0.0 from local-m2-cache in [default]
commons-logging#commons-logging;1.1.3 from local-m2-cache in [default]
org.apache.commons#commons-pool2;2.11.1 from local-m2-cache in [default]
org.apache.hadoop#hadoop-aws;3.2.2 from central in [default]
org.apache.hadoop#hadoop-client-api;3.3.4 from central in [default]
org.apache.hadoop#hadoop-client-runtime;3.3.4 from central in [default]
org.apache.iceberg#iceberg-spark-runtime-3.3_2.12;1.4.1 from central in [default]
org.apache.kafka#kafka-clients;3.4.1 from local-m2-cache in [default]
org.apache.spark#spark-avro_2.12;3.5.0 from central in [default]
org.apache.spark#spark-sql-kafka-0-10_2.12;3.5.0 from central in [default]
org.apache.spark#spark-token-provider-kafka-0-10_2.12;3.5.0 from central in [default]
org.lz4#lz4-java;1.8.0 from local-m2-cache in [default]
org.projectnessie.nessie-integrations#nessie-spark-extensions-3.5_2.12;0.73.0 from central in [default]
org.reactivestreams#reactive-streams;1.0.3 from local-m2-cache in [default]
org.slf4j#slf4j-api;2.0.7 from central in [default]
org.tukaani#xz;1.9 from local-m2-cache in [default]
org.xerial.snappy#snappy-java;1.1.10.3 from central in [default]
software.amazon.awssdk#annotations;2.18.31 from central in [default]
software.amazon.awssdk#bundle;2.18.31 from central in [default]
software.amazon.awssdk#http-client-spi;2.18.31 from central in [default]
software.amazon.awssdk#metrics-spi;2.18.31 from central in [default]
software.amazon.awssdk#url-connection-client;2.18.31 from central in [default]
software.amazon.awssdk#utils;2.18.31 from central in [default]
software.amazon.eventstream#eventstream;1.0.1 from central in [default]
:: evicted modules:
org.slf4j#slf4j-api;1.7.30 by [org.slf4j#slf4j-api;2.0.7] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 26 | 0 | 0 | 1 || 25 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-dfba919b-b3c9-4874-80b4-7733434a5f07
confs: [default]
0 artifacts copied, 25 already retrieved (0kB/8ms)
23/11/19 15:02:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/11/19 15:02:02 INFO SparkContext: Running Spark version 3.5.0
23/11/19 15:02:02 INFO SparkContext: OS info Mac OS X, 13.6, aarch64
23/11/19 15:02:02 INFO SparkContext: Java version 20.0.1
23/11/19 15:02:02 INFO ResourceUtils: ==============================================================
23/11/19 15:02:02 INFO ResourceUtils: No custom resources configured for spark.driver.
23/11/19 15:02:02 INFO ResourceUtils: ==============================================================
23/11/19 15:02:02 INFO SparkContext: Submitted application: app_name
23/11/19 15:02:02 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
23/11/19 15:02:02 INFO ResourceProfile: Limiting resource is cpu
23/11/19 15:02:02 INFO ResourceProfileManager: Added ResourceProfile id: 0
23/11/19 15:02:02 INFO SecurityManager: Changing view acls to: charmalloc
23/11/19 15:02:02 INFO SecurityManager: Changing modify acls to: charmalloc
23/11/19 15:02:02 INFO SecurityManager: Changing view acls groups to:
23/11/19 15:02:02 INFO SecurityManager: Changing modify acls groups to:
23/11/19 15:02:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: charmalloc; groups with view permissions: EMPTY; users with modify permissions: charmalloc; groups with modify permissions: EMPTY
23/11/19 15:02:02 INFO Utils: Successfully started service 'sparkDriver' on port 54682.
23/11/19 15:02:02 INFO SparkEnv: Registering MapOutputTracker
23/11/19 15:02:02 INFO SparkEnv: Registering BlockManagerMaster
23/11/19 15:02:02 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/11/19 15:02:02 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
23/11/19 15:02:02 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
23/11/19 15:02:02 INFO DiskBlockManager: Created local directory at /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/blockmgr-7024265e-457b-4d5d-81d4-75a63c101f7a
23/11/19 15:02:02 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
23/11/19 15:02:02 INFO SparkEnv: Registering OutputCommitCoordinator
23/11/19 15:02:02 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI
23/11/19 15:02:02 INFO Utils: Successfully started service 'SparkUI' on port 4040.
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-aws-3.2.2.jar at spark://192.168.1.9:54682/jars/org.apache.hadoop_hadoop-aws-3.2.2.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar at spark://192.168.1.9:54682/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-avro_2.12-3.5.0.jar at spark://192.168.1.9:54682/jars/org.apache.spark_spark-avro_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar at spark://192.168.1.9:54682/jars/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_bundle-2.18.31.jar at spark://192.168.1.9:54682/jars/software.amazon.awssdk_bundle-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_url-connection-client-2.18.31.jar at spark://192.168.1.9:54682/jars/software.amazon.awssdk_url-connection-client-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar at spark://192.168.1.9:54682/jars/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar at spark://192.168.1.9:54682/jars/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar at spark://192.168.1.9:54682/jars/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.apache.kafka_kafka-clients-3.4.1.jar at spark://192.168.1.9:54682/jars/org.apache.kafka_kafka-clients-3.4.1.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/com.google.code.findbugs_jsr305-3.0.0.jar at spark://192.168.1.9:54682/jars/com.google.code.findbugs_jsr305-3.0.0.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.apache.commons_commons-pool2-2.11.1.jar at spark://192.168.1.9:54682/jars/org.apache.commons_commons-pool2-2.11.1.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar at spark://192.168.1.9:54682/jars/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.lz4_lz4-java-1.8.0.jar at spark://192.168.1.9:54682/jars/org.lz4_lz4-java-1.8.0.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.xerial.snappy_snappy-java-1.1.10.3.jar at spark://192.168.1.9:54682/jars/org.xerial.snappy_snappy-java-1.1.10.3.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.slf4j_slf4j-api-2.0.7.jar at spark://192.168.1.9:54682/jars/org.slf4j_slf4j-api-2.0.7.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-api-3.3.4.jar at spark://192.168.1.9:54682/jars/org.apache.hadoop_hadoop-client-api-3.3.4.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/commons-logging_commons-logging-1.1.3.jar at spark://192.168.1.9:54682/jars/commons-logging_commons-logging-1.1.3.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.tukaani_xz-1.9.jar at spark://192.168.1.9:54682/jars/org.tukaani_xz-1.9.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/software.amazon.eventstream_eventstream-1.0.1.jar at spark://192.168.1.9:54682/jars/software.amazon.eventstream_eventstream-1.0.1.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_utils-2.18.31.jar at spark://192.168.1.9:54682/jars/software.amazon.awssdk_utils-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_annotations-2.18.31.jar at spark://192.168.1.9:54682/jars/software.amazon.awssdk_annotations-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_http-client-spi-2.18.31.jar at spark://192.168.1.9:54682/jars/software.amazon.awssdk_http-client-spi-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/org.reactivestreams_reactive-streams-1.0.3.jar at spark://192.168.1.9:54682/jars/org.reactivestreams_reactive-streams-1.0.3.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added JAR file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_metrics-spi-2.18.31.jar at spark://192.168.1.9:54682/jars/software.amazon.awssdk_metrics-spi-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-aws-3.2.2.jar at file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-aws-3.2.2.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-aws-3.2.2.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-aws-3.2.2.jar
23/11/19 15:02:02 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar at file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar
23/11/19 15:02:02 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-avro_2.12-3.5.0.jar at file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-avro_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.apache.spark_spark-avro_2.12-3.5.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-avro_2.12-3.5.0.jar
23/11/19 15:02:02 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar at file:///Users/charmalloc/.ivy2/jars/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar
23/11/19 15:02:02 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_bundle-2.18.31.jar at file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_bundle-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_bundle-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_bundle-2.18.31.jar
23/11/19 15:02:02 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_url-connection-client-2.18.31.jar at file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_url-connection-client-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_url-connection-client-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_url-connection-client-2.18.31.jar
23/11/19 15:02:02 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar at file:///Users/charmalloc/.ivy2/jars/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar with timestamp 1700424122496
23/11/19 15:02:02 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar at file:///Users/charmalloc/.ivy2/jars/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar at file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.apache.kafka_kafka-clients-3.4.1.jar at file:///Users/charmalloc/.ivy2/jars/org.apache.kafka_kafka-clients-3.4.1.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.apache.kafka_kafka-clients-3.4.1.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.kafka_kafka-clients-3.4.1.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/com.google.code.findbugs_jsr305-3.0.0.jar at file:///Users/charmalloc/.ivy2/jars/com.google.code.findbugs_jsr305-3.0.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/com.google.code.findbugs_jsr305-3.0.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/com.google.code.findbugs_jsr305-3.0.0.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.apache.commons_commons-pool2-2.11.1.jar at file:///Users/charmalloc/.ivy2/jars/org.apache.commons_commons-pool2-2.11.1.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.apache.commons_commons-pool2-2.11.1.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.commons_commons-pool2-2.11.1.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar at file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.lz4_lz4-java-1.8.0.jar at file:///Users/charmalloc/.ivy2/jars/org.lz4_lz4-java-1.8.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.lz4_lz4-java-1.8.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.lz4_lz4-java-1.8.0.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.xerial.snappy_snappy-java-1.1.10.3.jar at file:///Users/charmalloc/.ivy2/jars/org.xerial.snappy_snappy-java-1.1.10.3.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.xerial.snappy_snappy-java-1.1.10.3.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.xerial.snappy_snappy-java-1.1.10.3.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.slf4j_slf4j-api-2.0.7.jar at file:///Users/charmalloc/.ivy2/jars/org.slf4j_slf4j-api-2.0.7.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.slf4j_slf4j-api-2.0.7.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.slf4j_slf4j-api-2.0.7.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-api-3.3.4.jar at file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-api-3.3.4.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-api-3.3.4.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-client-api-3.3.4.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/commons-logging_commons-logging-1.1.3.jar at file:///Users/charmalloc/.ivy2/jars/commons-logging_commons-logging-1.1.3.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/commons-logging_commons-logging-1.1.3.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/commons-logging_commons-logging-1.1.3.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.tukaani_xz-1.9.jar at file:///Users/charmalloc/.ivy2/jars/org.tukaani_xz-1.9.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.tukaani_xz-1.9.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.tukaani_xz-1.9.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/software.amazon.eventstream_eventstream-1.0.1.jar at file:///Users/charmalloc/.ivy2/jars/software.amazon.eventstream_eventstream-1.0.1.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/software.amazon.eventstream_eventstream-1.0.1.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.eventstream_eventstream-1.0.1.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_utils-2.18.31.jar at file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_utils-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_utils-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_utils-2.18.31.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_annotations-2.18.31.jar at file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_annotations-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_annotations-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_annotations-2.18.31.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_http-client-spi-2.18.31.jar at file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_http-client-spi-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_http-client-spi-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_http-client-spi-2.18.31.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/org.reactivestreams_reactive-streams-1.0.3.jar at file:///Users/charmalloc/.ivy2/jars/org.reactivestreams_reactive-streams-1.0.3.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/org.reactivestreams_reactive-streams-1.0.3.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.reactivestreams_reactive-streams-1.0.3.jar
23/11/19 15:02:03 INFO SparkContext: Added file file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_metrics-spi-2.18.31.jar at file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_metrics-spi-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Copying /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_metrics-spi-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_metrics-spi-2.18.31.jar
23/11/19 15:02:03 INFO Executor: Starting executor ID driver on host 192.168.1.9
23/11/19 15:02:03 INFO Executor: OS info Mac OS X, 13.6, aarch64
23/11/19 15:02:03 INFO Executor: Java version 20.0.1
23/11/19 15:02:03 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
23/11/19 15:02:03 INFO Executor: Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@261a8ba5 for default.
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_utils-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_utils-2.18.31.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_utils-2.18.31.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.lz4_lz4-java-1.8.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.lz4_lz4-java-1.8.0.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.lz4_lz4-java-1.8.0.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/software.amazon.eventstream_eventstream-1.0.1.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/software.amazon.eventstream_eventstream-1.0.1.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.eventstream_eventstream-1.0.1.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/com.google.code.findbugs_jsr305-3.0.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/com.google.code.findbugs_jsr305-3.0.0.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/com.google.code.findbugs_jsr305-3.0.0.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.xerial.snappy_snappy-java-1.1.10.3.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.xerial.snappy_snappy-java-1.1.10.3.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.xerial.snappy_snappy-java-1.1.10.3.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-aws-3.2.2.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-aws-3.2.2.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-aws-3.2.2.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_bundle-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_bundle-2.18.31.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_bundle-2.18.31.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.tukaani_xz-1.9.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.tukaani_xz-1.9.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.tukaani_xz-1.9.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_url-connection-client-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_url-connection-client-2.18.31.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_url-connection-client-2.18.31.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.apache.commons_commons-pool2-2.11.1.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.apache.commons_commons-pool2-2.11.1.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.commons_commons-pool2-2.11.1.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.apache.kafka_kafka-clients-3.4.1.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.apache.kafka_kafka-clients-3.4.1.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.kafka_kafka-clients-3.4.1.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_http-client-spi-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_http-client-spi-2.18.31.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_http-client-spi-2.18.31.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-avro_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.apache.spark_spark-avro_2.12-3.5.0.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-avro_2.12-3.5.0.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_metrics-spi-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_metrics-spi-2.18.31.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_metrics-spi-2.18.31.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.slf4j_slf4j-api-2.0.7.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.slf4j_slf4j-api-2.0.7.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.slf4j_slf4j-api-2.0.7.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/commons-logging_commons-logging-1.1.3.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/commons-logging_commons-logging-1.1.3.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/commons-logging_commons-logging-1.1.3.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-api-3.3.4.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.apache.hadoop_hadoop-client-api-3.3.4.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-client-api-3.3.4.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/org.reactivestreams_reactive-streams-1.0.3.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/org.reactivestreams_reactive-streams-1.0.3.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.reactivestreams_reactive-streams-1.0.3.jar
23/11/19 15:02:03 INFO Executor: Fetching file:///Users/charmalloc/.ivy2/jars/software.amazon.awssdk_annotations-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: /Users/charmalloc/.ivy2/jars/software.amazon.awssdk_annotations-2.18.31.jar has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_annotations-2.18.31.jar
23/11/19 15:02:03 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.apache.hadoop_hadoop-aws-3.2.2.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO TransportClientFactory: Successfully created connection to /192.168.1.9:54682 after 13 ms (0 ms spent in bootstraps)
23/11/19 15:02:03 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.apache.hadoop_hadoop-aws-3.2.2.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp13346193918872381284.tmp
23/11/19 15:02:03 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp13346193918872381284.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-aws-3.2.2.jar
23/11/19 15:02:03 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-aws-3.2.2.jar to class loader default
23/11/19 15:02:03 INFO Executor: Fetching spark://192.168.1.9:54682/jars/commons-logging_commons-logging-1.1.3.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Fetching spark://192.168.1.9:54682/jars/commons-logging_commons-logging-1.1.3.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp370333549259066269.tmp
23/11/19 15:02:03 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp370333549259066269.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/commons-logging_commons-logging-1.1.3.jar
23/11/19 15:02:03 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/commons-logging_commons-logging-1.1.3.jar to class loader default
23/11/19 15:02:03 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.apache.commons_commons-pool2-2.11.1.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.apache.commons_commons-pool2-2.11.1.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp13744678481442411571.tmp
23/11/19 15:02:03 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp13744678481442411571.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.commons_commons-pool2-2.11.1.jar
23/11/19 15:02:03 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.commons_commons-pool2-2.11.1.jar to class loader default
23/11/19 15:02:03 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp10360797100428185545.tmp
23/11/19 15:02:03 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp10360797100428185545.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar
23/11/19 15:02:03 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.5.0.jar to class loader default
23/11/19 15:02:03 INFO Executor: Fetching spark://192.168.1.9:54682/jars/software.amazon.eventstream_eventstream-1.0.1.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Fetching spark://192.168.1.9:54682/jars/software.amazon.eventstream_eventstream-1.0.1.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp3673731967100770538.tmp
23/11/19 15:02:03 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp3673731967100770538.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.eventstream_eventstream-1.0.1.jar
23/11/19 15:02:03 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.eventstream_eventstream-1.0.1.jar to class loader default
23/11/19 15:02:03 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.lz4_lz4-java-1.8.0.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.lz4_lz4-java-1.8.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp3590133730801838051.tmp
23/11/19 15:02:03 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp3590133730801838051.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.lz4_lz4-java-1.8.0.jar
23/11/19 15:02:03 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.lz4_lz4-java-1.8.0.jar to class loader default
23/11/19 15:02:03 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp12034143384454749484.tmp
23/11/19 15:02:03 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp12034143384454749484.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar
23/11/19 15:02:03 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.iceberg_iceberg-spark-runtime-3.3_2.12-1.4.1.jar to class loader default
23/11/19 15:02:03 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp16060285960327691806.tmp
23/11/19 15:02:03 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp16060285960327691806.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar
23/11/19 15:02:03 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-client-runtime-3.3.4.jar to class loader default
23/11/19 15:02:03 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.tukaani_xz-1.9.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.tukaani_xz-1.9.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp6480969683247340273.tmp
23/11/19 15:02:03 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp6480969683247340273.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.tukaani_xz-1.9.jar
23/11/19 15:02:03 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.tukaani_xz-1.9.jar to class loader default
23/11/19 15:02:03 INFO Executor: Fetching spark://192.168.1.9:54682/jars/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar with timestamp 1700424122496
23/11/19 15:02:03 INFO Utils: Fetching spark://192.168.1.9:54682/jars/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp15131233074573092301.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp15131233074573092301.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/com.amazonaws_aws-java-sdk-bundle-1.11.563.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.apache.hadoop_hadoop-client-api-3.3.4.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.apache.hadoop_hadoop-client-api-3.3.4.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp9911738831164433897.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp9911738831164433897.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-client-api-3.3.4.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.hadoop_hadoop-client-api-3.3.4.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.reactivestreams_reactive-streams-1.0.3.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.reactivestreams_reactive-streams-1.0.3.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp12256753891109409752.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp12256753891109409752.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.reactivestreams_reactive-streams-1.0.3.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.reactivestreams_reactive-streams-1.0.3.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_url-connection-client-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_url-connection-client-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp2682141745072233434.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp2682141745072233434.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_url-connection-client-2.18.31.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_url-connection-client-2.18.31.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.apache.kafka_kafka-clients-3.4.1.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.apache.kafka_kafka-clients-3.4.1.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp17508849673993485939.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp17508849673993485939.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.kafka_kafka-clients-3.4.1.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.kafka_kafka-clients-3.4.1.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_utils-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_utils-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp15802966200724031987.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp15802966200724031987.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_utils-2.18.31.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_utils-2.18.31.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_annotations-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_annotations-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp15241050173186725867.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp15241050173186725867.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_annotations-2.18.31.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_annotations-2.18.31.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp14833436643218508794.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp14833436643218508794.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.projectnessie.nessie-integrations_nessie-spark-extensions-3.5_2.12-0.73.0.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.xerial.snappy_snappy-java-1.1.10.3.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.xerial.snappy_snappy-java-1.1.10.3.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp4167540884596294512.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp4167540884596294512.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.xerial.snappy_snappy-java-1.1.10.3.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.xerial.snappy_snappy-java-1.1.10.3.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_bundle-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_bundle-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp14009423441532945649.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp14009423441532945649.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_bundle-2.18.31.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_bundle-2.18.31.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_metrics-spi-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_metrics-spi-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp5800058955114080369.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp5800058955114080369.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_metrics-spi-2.18.31.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_metrics-spi-2.18.31.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_http-client-spi-2.18.31.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/software.amazon.awssdk_http-client-spi-2.18.31.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp6141140504532091947.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp6141140504532091947.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_http-client-spi-2.18.31.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/software.amazon.awssdk_http-client-spi-2.18.31.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp3516201225751901310.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp3516201225751901310.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-sql-kafka-0-10_2.12-3.5.0.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/com.google.code.findbugs_jsr305-3.0.0.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/com.google.code.findbugs_jsr305-3.0.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp13182356259049804131.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp13182356259049804131.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/com.google.code.findbugs_jsr305-3.0.0.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/com.google.code.findbugs_jsr305-3.0.0.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.slf4j_slf4j-api-2.0.7.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.slf4j_slf4j-api-2.0.7.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp15887285542952085125.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp15887285542952085125.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.slf4j_slf4j-api-2.0.7.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.slf4j_slf4j-api-2.0.7.jar to class loader default
23/11/19 15:02:04 INFO Executor: Fetching spark://192.168.1.9:54682/jars/org.apache.spark_spark-avro_2.12-3.5.0.jar with timestamp 1700424122496
23/11/19 15:02:04 INFO Utils: Fetching spark://192.168.1.9:54682/jars/org.apache.spark_spark-avro_2.12-3.5.0.jar to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp15268305788987280209.tmp
23/11/19 15:02:04 INFO Utils: /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/fetchFileTemp15268305788987280209.tmp has been previously copied to /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-avro_2.12-3.5.0.jar
23/11/19 15:02:04 INFO Executor: Adding file:/private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/userFiles-131e3eb0-830a-4998-8dd6-c0bbd59c910f/org.apache.spark_spark-avro_2.12-3.5.0.jar to class loader default
23/11/19 15:02:04 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54685.
23/11/19 15:02:04 INFO NettyBlockTransferService: Server created on 192.168.1.9:54685
23/11/19 15:02:04 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/11/19 15:02:04 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.1.9, 54685, None)
23/11/19 15:02:04 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.9:54685 with 434.4 MiB RAM, BlockManagerId(driver, 192.168.1.9, 54685, None)
23/11/19 15:02:04 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.1.9, 54685, None)
23/11/19 15:02:04 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.1.9, 54685, None)
23/11/19 15:02:05 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
23/11/19 15:02:05 INFO SharedState: Warehouse path is 'file:/Users/charmalloc/dev/bitsandbytes/getting-started-apachespark-apachekafka-apacheiceberg/spark-warehouse'.
Traceback (most recent call last):
File "/Users/charmalloc/dev/bitsandbytes/getting-started-apachespark-apachekafka-apacheiceberg/create_table.py", line 46, in <module>
spark.sql(f"""
File "/Users/charmalloc/spark-3.5.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/session.py", line 1631, in sql
File "/Users/charmalloc/spark-3.5.0-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
File "/Users/charmalloc/spark-3.5.0-bin-hadoop3/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 185, in deco
pyspark.errors.exceptions.captured.ParseException:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'CATALOG'.(line 2, pos 9)
== SQL ==
CREATE CATALOG nessiecat1test
---------^^^
WITH (
'type'='iceberg',
'catalog-impl'='org.apache.iceberg.nessie.NessieCatalog',
'uri'='http://localhost:19120/api/v1',
'ref'='main',
'warehouse' = 'spark-warehouse-nessie')
23/11/19 15:02:05 INFO SparkContext: Invoking stop() from shutdown hook
23/11/19 15:02:05 INFO SparkContext: SparkContext is stopping with exitCode 0.
23/11/19 15:02:05 INFO SparkUI: Stopped Spark web UI at http://192.168.1.9:4040
23/11/19 15:02:05 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/11/19 15:02:05 INFO MemoryStore: MemoryStore cleared
23/11/19 15:02:05 INFO BlockManager: BlockManager stopped
23/11/19 15:02:05 INFO BlockManagerMaster: BlockManagerMaster stopped
23/11/19 15:02:05 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/11/19 15:02:05 INFO SparkContext: Successfully stopped SparkContext
23/11/19 15:02:05 INFO ShutdownHookManager: Shutdown hook called
23/11/19 15:02:05 INFO ShutdownHookManager: Deleting directory /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-aea36b6c-f24e-43b2-8b8e-a19ec88869de
23/11/19 15:02:05 INFO ShutdownHookManager: Deleting directory /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181
23/11/19 15:02:05 INFO ShutdownHookManager: Deleting directory /private/var/folders/tn/fkzyhj6d0zn9m30j8btybtqh0000gn/T/spark-3829322e-b715-4fff-afe0-b2fbb2607181/pyspark-9967e68e-848c-46c5-a707-8880576df765
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment