Skip to content

Instantly share code, notes, and snippets.

View yuwtennis's full-sized avatar

Yu Watanabe yuwtennis

3 followers · 5 following
View GitHub Profile
@yuwtennis
yuwtennis / spark-3.2.1
Created February 13, 2022 10:44
[ywatanabe@laptop-archlinux spark3.2.1]$ bin/spark-shell
22/02/13 17:06:55 WARN Utils: Your hostname, laptop-archlinux resolves to a loopback address: 127.0.0.1; using 10.23.109.137 instead (on interface wlp0s20f3)
22/02/13 17:06:55 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark3.2.1/jars/spark-unsafe_2.12-3.2.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
@yuwtennis
yuwtennis / .env
Created August 15, 2021 16:23
SDK client for spark runner
export SDK_HARNESS_ENDPOINT=localhost:50000
export JOB_SERVICE_ENDPOINT=localhost:8099
@yuwtennis
yuwtennis / gist:7b0c1dc0dcf98297af1e3179852ca693
Created August 15, 2021 16:19
Spark Worker error
Spark Executor Command: "/usr/lib/jvm/java-11-openjdk/bin/java" "-cp" "/opt/spark3/conf/:/opt/spark3/jars/*" "-Xmx1024M" "-Dspark.driver.port=35601" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.11.2:35601" "--executor-id" "0" "--hostname" "192.168.11.2" "--cores" "1" "--app-id" "app-20210816011025-0000" "--worker-url" "spark://Worker@192.168.11.2:39677"
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/08/16 01:10:26 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 134769@laptop-archlinux
21/08/16 01:10:26 INFO SignalUtils: Registering signal handler for TERM
21/08/16 01:10:26 INFO SignalUtils: Registering signal handler for HUP
21/08/16 01:10:26 INFO SignalUtils: Registering signal handler for INT
21/08/16 01:10:26 WARN Utils: Your hostname, laptop-archlinux resolves to a loopback address: 127.0.0.1; using 192.168.11.2 instead (on interface enp0s20f