Skip to content

Instantly share code, notes, and snippets.

@ashishb888

ashishb888/spark-stand-alone-cluster

Last active August 24, 2019 06:34
Show Gist options
  • Save ashishb888/6561840c4222705a2530580ce29489a2 to your computer and use it in GitHub Desktop.
Save ashishb888/6561840c4222705a2530580ce29489a2 to your computer and use it in GitHub Desktop.
Download ZIP
Apache Spark stand alone cluster
Raw
spark-stand-alone-cluster
./sbin/start-master.sh
---------------------------
Spark Command: /opt/ngs/setups/jdk1.8.0_191/bin/java -cp /opt/ngs/ashishb/packages/spark-2.4.3-bin-hadoop2.7/conf/:/opt/ngs/ashishb/packages/spark-2.4.3-bin-hadoop2.7/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host hdpdev6 --port 7077 --webui-port 8080
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/08/14 19:18:48 INFO Master: Started daemon with process name: 51863@hdpdev6
19/08/14 19:18:48 INFO SignalUtils: Registered signal handler for TERM
19/08/14 19:18:48 INFO SignalUtils: Registered signal handler for HUP
19/08/14 19:18:48 INFO SignalUtils: Registered signal handler for INT
19/08/14 19:18:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/08/14 19:18:49 INFO SecurityManager: Changing view acls to: hdpusr
19/08/14 19:18:49 INFO SecurityManager: Changing modify acls to: hdpusr
19/08/14 19:18:49 INFO SecurityManager: Changing view acls groups to:
19/08/14 19:18:49 INFO SecurityManager: Changing modify acls groups to:
19/08/14 19:18:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hdpusr); groups with view permissions: Set(); users with modify permissions: Set(hdpusr); groups with modify permissions: Set()
19/08/14 19:18:49 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
19/08/14 19:18:49 INFO Master: Starting Spark master at spark://hdpdev6:7077
19/08/14 19:18:49 INFO Master: Running Spark version 2.4.3
19/08/14 19:18:49 INFO Utils: Successfully started service 'MasterUI' on port 8080.
19/08/14 19:18:49 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://hdpdev6:8080
19/08/14 19:18:50 INFO Master: I have been elected leader! New state: ALIVE
---------------------------
./sbin/start-slave.sh spark://hdpdev6:7077
---------------------------
Spark Command: /opt/ngs/setups/jdk1.8.0_191/bin/java -cp /opt/ngs/ashishb/packages/spark-2.4.3-bin-hadoop2.7/conf/:/opt/ngs/ashishb/packages/spark-2.4.3-bin-hadoop2.7/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://hdpdev6:7077
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/08/14 19:23:34 INFO Worker: Started daemon with process name: 187005@hdpdev4
19/08/14 19:23:34 INFO SignalUtils: Registered signal handler for TERM
19/08/14 19:23:34 INFO SignalUtils: Registered signal handler for HUP
19/08/14 19:23:34 INFO SignalUtils: Registered signal handler for INT
19/08/14 19:23:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/08/14 19:23:34 INFO SecurityManager: Changing view acls to: hdpusr
19/08/14 19:23:34 INFO SecurityManager: Changing modify acls to: hdpusr
19/08/14 19:23:34 INFO SecurityManager: Changing view acls groups to:
19/08/14 19:23:34 INFO SecurityManager: Changing modify acls groups to:
19/08/14 19:23:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hdpusr); groups with view permissions: Set(); users with modify permissions: Set(hdpusr); groups with modify permissions: Set()
19/08/14 19:23:35 INFO Utils: Successfully started service 'sparkWorker' on port 45542.
19/08/14 19:23:35 INFO Worker: Starting Spark worker 172.17.5.34:45542 with 32 cores, 124.7 GB RAM
19/08/14 19:23:35 INFO Worker: Running Spark version 2.4.3
19/08/14 19:23:35 INFO Worker: Spark home: /opt/ngs/ashishb/packages/spark-2.4.3-bin-hadoop2.7
19/08/14 19:23:35 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
19/08/14 19:23:35 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://hdpdev4:8081
19/08/14 19:23:35 INFO Worker: Connecting to master hdpdev6:7077...
19/08/14 19:23:35 INFO TransportClientFactory: Successfully created connection to hdpdev6/172.17.5.36:7077 after 57 ms (0 ms spent in bootstraps)
19/08/14 19:23:35 INFO Worker: Successfully registered with master spark://hdpdev6:7077
---------------------------
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment