Skip to content

Instantly share code, notes, and snippets.

@yaravind
Last active December 30, 2016 00:59
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yaravind/1dcb35bf84f2bddabeb7d583371a9294 to your computer and use it in GitHub Desktop.
Save yaravind/1dcb35bf84f2bddabeb7d583371a9294 to your computer and use it in GitHub Desktop.
TITLE Launcher - Spark Master, 1 Worker and History Server
set SPARK_HOME=C:\aravind\sw\spark-2.0.2-bin-hadoop2.7
:: - Find the IP Address and set it to IP_ADDR env var and reuse it while launching Worker
for /f "tokens=1-2 delims=:" %%a in ('ipconfig^|find "IPv4"') do set ip=%%b
set IP_ADDR=%ip:~1%
echo %IP_ADDR%
:: - Start master
START "Spark Master" %SPARK_HOME%\bin\spark-class.cmd org.apache.spark.deploy.master.Master
ECHO Started Spark MASTER - Web UI on http://localhost:8080
%% Wait for master to complete initialization so that worker can connect to it
PING -n 15 127.0.0.1 > NUL
:: - Start worker with 1 core and 64mb ram
START "Spark Worker" %SPARK_HOME%\bin\spark-class.cmd org.apache.spark.deploy.worker.Worker -c 1 -m 1g -d C:/aravind/sw/spark-worker-dir spark://%IP_ADDR%:7077
ECHO Started Spark WORKER - Web UI on http://localhost:8081
PING -n 5 127.0.0.1 > NUL
START "Spark History Server" %SPARK_HOME%\bin\spark-class.cmd org.apache.spark.deploy.history.HistoryServer file:///C:/aravind/sw/spark-shared-eventlog
ECHO Started Spark WORKER - Web UI on http://localhost:18080 and REST API on http://localhost:18080/api/v1/applications
PING -n 5 127.0.0.1 > NUL
:: - Launch WebUI's in defualt browser
start chrome http://%IP_ADDR%:8080
start chrome http://%IP_ADDR%:8081
start chrome http://%IP_ADDR%:18080
@yaravind
Copy link
Author

yaravind commented Dec 30, 2016

I chose to launch the WebUI's in chrome. If you want them to be launched in the default browser remove 'chrome' i.e. start http://%IP_ADDR%:8080

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment