Skip to content

Instantly share code, notes, and snippets.

@bigsnarfdude

bigsnarfdude/gist:d764e3586e17a290f4cf

Created January 20, 2015 20:02
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bigsnarfdude/d764e3586e17a290f4cf to your computer and use it in GitHub Desktop.
Save bigsnarfdude/d764e3586e17a290f4cf to your computer and use it in GitHub Desktop.
Download ZIP
Spark EC2 Output
Raw
gistfile1.txt
~/spark-1.2.0-bin-hadoop1/ec2 $./spark-ec2 -k spark-east -i /Users/home/spark-east.pem -t m1.small -z us-east-1a launch test-cluster
Setting up security groups...
Searching for existing cluster test-cluster...
Spark AMI: ami-5bb18832
Launching instances...
Launched 1 slaves in us-east-1e, regid = 1981cd
Launched master in us-east-1e, regid = 66fe1d
Waiting for all instances in cluster to enter 'ssh-ready' state...............
Generating cluster's SSH key on master...
Warning: Permanently added 'ec2-5-1-1-64.compute-1.amazonaws.com' (RSA) to the list of known hosts.
Connection to ec2-5-1-1-64.compute-1.amazonaws.com closed.
Transferring cluster's SSH key to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
Warning: Permanently added 'ec2-5.1.1.2.compute-1.amazonaws.com' (RSA) to the list of known hosts.
Cloning into 'spark-ec2'...
remote: Counting objects: 1698, done.
remote: Compressing objects: 100% (805/805), done.
remote: Total 1698 (delta 607), reused 1698 (delta 607)
Receiving objects: 100% (1698/1698), 273.67 KiB, done.
Resolving deltas: 100% (607/607), done.
Connection to ec2-5-1-1-64.compute-1.amazonaws.com closed.
Deploying files to master...
building file list ... done
root/spark-ec2/ec2-variables.sh
sent 1594 bytes received 42 bytes 1090.67 bytes/sec
total size is 1453 speedup is 0.89
Running setup on master...
Connection to ec2-5-1-1-64.compute-1.amazonaws.com closed.
Setting up Spark on ip-1.1.1.2.ec2.internal...
Setting executable permissions on scripts...
Running setup-slave on master to mount filesystems, etc...
Setting up slave on ip-1.1.1.2.ec2.internal...
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 18.8494 s, 57.0 MB/s
mkswap: /mnt/swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=4396-ba35-fcf6f1a66819
Added 1024 MB swap file /mnt/swap
SSH'ing to master machine(s) to approve key(s)...
ec2-5-1-1-64.compute-1.amazonaws.com
Warning: Permanently added 'ec2-5-1-1-64.compute-1.amazonaws.com,1.3.3.7' (ECDSA) to the list of known hosts.
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Warning: Permanently added 'ip-1.1.1.2.ec2.internal' (ECDSA) to the list of known hosts.
SSH'ing to other cluster nodes to approve keys...
ec2-5.1.1.2.compute-1.amazonaws.com
Warning: Permanently added 'ec2-5.1.1.2.compute-1.amazonaws.com,10.2.3.4' (ECDSA) to the list of known hosts.
RSYNC'ing /root/spark-ec2 to other cluster nodes...
ec2-5.1.1.2.compute-1.amazonaws.com
id_rsa 100% 1679 1.6KB/s 00:00
Running slave setup script on other cluster nodes...
ec2-5.1.1.2.compute-1.amazonaws.com
Setting up slave on ip-1-3-4-2.ec2.internal...
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 18.6761 s, 57.5 MB/s
mkswap: /mnt/swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=4959-a071-e5a02fe4bfee
Added 1024 MB swap file /mnt/swap
Connection to ec2-5.1.1.2.compute-1.amazonaws.com closed.
Initializing scala
~ ~/spark-ec2
Unpacking Scala
--2015-01-20 19:39:48-- http://s3.amazonaws.com/spark-related-packages/scala-2.10.3.tgz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 1.2.3.4
Connecting to s3.amazonaws.com (s3.amazonaws.com)|1.2.3.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30531249 (29M) [application/x-compressed]
Saving to: ‘scala-2.10.3.tgz’
100%[==================================================================================================================================>] 30,531,249 3.20MB/s in 9.8s
2015-01-20 19:39:58 (2.99 MB/s) - ‘scala-2.10.3.tgz’ saved [30531249/30531249]
~/spark-ec2
Initializing spark
~ ~/spark-ec2
--2015-01-20 19:39:59-- http://s3.amazonaws.com/spark-related-packages/spark-1.2.0-bin-hadoop1.tgz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 1.2.3.4
Connecting to s3.amazonaws.com (s3.amazonaws.com)|1.2.3.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 203977086 (195M) [application/x-compressed]
Saving to: ‘spark-1.2.0-bin-hadoop1.tgz’
100%[==================================================================================================================================>] 203,977,086 38.2MB/s in 4.8s
2015-01-20 19:40:04 (40.4 MB/s) - ‘spark-1.2.0-bin-hadoop1.tgz’ saved [203977086/203977086]
Unpacking Spark
~/spark-ec2
Initializing shark
~ ~/spark-ec2
ERROR: Unknown Shark version
~/spark-ec2
Initializing ephemeral-hdfs
~ ~/spark-ec2
--2015-01-20 19:40:14-- http://s3.amazonaws.com/spark-related-packages/hadoop-1.0.4.tar.gz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 1.2.3.4
Connecting to s3.amazonaws.com (s3.amazonaws.com)|1.2.3.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 62793050 (60M) [application/x-gzip]
Saving to: ‘hadoop-1.0.4.tar.gz’
100%[==================================================================================================================================>] 62,793,050 41.4MB/s in 1.4s
2015-01-20 19:40:15 (41.4 MB/s) - ‘hadoop-1.0.4.tar.gz’ saved [62793050/62793050]
Unpacking Hadoop
RSYNC'ing /root/ephemeral-hdfs to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
~/spark-ec2
Initializing persistent-hdfs
~ ~/spark-ec2
--2015-01-20 19:40:49-- http://s3.amazonaws.com/spark-related-packages/hadoop-1.0.4.tar.gz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 1.2.3.4
Connecting to s3.amazonaws.com (s3.amazonaws.com)|1.2.3.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 62793050 (60M) [application/x-gzip]
Saving to: ‘hadoop-1.0.4.tar.gz’
100%[==================================================================================================================================>] 62,793,050 44.8MB/s in 1.3s
2015-01-20 19:40:51 (44.8 MB/s) - ‘hadoop-1.0.4.tar.gz’ saved [62793050/62793050]
Unpacking Hadoop
RSYNC'ing /root/persistent-hdfs to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
~/spark-ec2
Initializing spark-standalone
Initializing tachyon
~ ~/spark-ec2
--2015-01-20 19:41:24-- https://s3.amazonaws.com/Tachyon/tachyon-0.4.1-bin.tar.gz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 1.2.3.4
Connecting to s3.amazonaws.com (s3.amazonaws.com)|1.2.3.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22751619 (22M) [application/x-gzip]
Saving to: ‘tachyon-0.4.1-bin.tar.gz’
100%[==================================================================================================================================>] 22,751,619 3.34MB/s in 7.1s
2015-01-20 19:41:32 (3.08 MB/s) - ‘tachyon-0.4.1-bin.tar.gz’ saved [22751619/22751619]
Unpacking Tachyon
~/spark-ec2
Initializing ganglia
Connection to ec2-5.1.1.2.compute-1.amazonaws.com closed.
Creating local config files...
Connection to ec2-5.1.1.2.compute-1.amazonaws.com closed.
Connection to ec2-5.1.1.2.compute-1.amazonaws.com closed.
Configuring /etc/ganglia/gmond.conf
Configuring /etc/ganglia/gmetad.conf
Configuring /etc/httpd/conf.d/ganglia.conf
Configuring /etc/httpd/conf/httpd.conf
Configuring /root/mapreduce/hadoop.version
Configuring /root/mapreduce/conf/core-site.xml
Configuring /root/mapreduce/conf/slaves
Configuring /root/mapreduce/conf/mapred-site.xml
Configuring /root/mapreduce/conf/hdfs-site.xml
Configuring /root/mapreduce/conf/hadoop-env.sh
Configuring /root/mapreduce/conf/masters
Configuring /root/persistent-hdfs/conf/core-site.xml
Configuring /root/persistent-hdfs/conf/slaves
Configuring /root/persistent-hdfs/conf/mapred-site.xml
Configuring /root/persistent-hdfs/conf/hdfs-site.xml
Configuring /root/persistent-hdfs/conf/hadoop-env.sh
Configuring /root/persistent-hdfs/conf/masters
Configuring /root/ephemeral-hdfs/conf/core-site.xml
Configuring /root/ephemeral-hdfs/conf/slaves
Configuring /root/ephemeral-hdfs/conf/mapred-site.xml
Configuring /root/ephemeral-hdfs/conf/hadoop-metrics2.properties
Configuring /root/ephemeral-hdfs/conf/hdfs-site.xml
Configuring /root/ephemeral-hdfs/conf/hadoop-env.sh
Configuring /root/ephemeral-hdfs/conf/masters
Configuring /root/spark/conf/core-site.xml
Configuring /root/spark/conf/spark-defaults.conf
Configuring /root/spark/conf/spark-env.sh
Configuring /root/tachyon/conf/slaves
Configuring /root/tachyon/conf/tachyon-env.sh
Configuring /root/shark/conf/shark-env.sh
Deploying Spark config files...
RSYNC'ing /root/spark/conf to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
Setting up scala
RSYNC'ing /root/scala to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
Setting up spark
RSYNC'ing /root/spark to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
Setting up shark
RSYNC'ing /root/shark to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
File or directory /root/hive* doesn't exist!
Setting up ephemeral-hdfs
~/spark-ec2/ephemeral-hdfs ~/spark-ec2
ec2-5.1.1.2.compute-1.amazonaws.com
Connection to ec2-5.1.1.2.compute-1.amazonaws.com closed.
RSYNC'ing /root/ephemeral-hdfs/conf to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
Formatting ephemeral HDFS namenode...
Warning: $HADOOP_HOME is deprecated.
15/01/20 19:42:35 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ip-1.1.1.2.ec2.internal/1.3.3.7
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
15/01/20 19:42:35 INFO util.GSet: VM type = 64-bit
15/01/20 19:42:35 INFO util.GSet: 2% max memory = 19.33375 MB
15/01/20 19:42:35 INFO util.GSet: capacity = 2^21 = 2097152 entries
15/01/20 19:42:35 INFO util.GSet: recommended=2097152, actual=2097152
15/01/20 19:42:36 INFO namenode.FSNamesystem: fsOwner=root
15/01/20 19:42:37 INFO namenode.FSNamesystem: supergroup=supergroup
15/01/20 19:42:37 INFO namenode.FSNamesystem: isPermissionEnabled=false
15/01/20 19:42:37 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
15/01/20 19:42:37 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
15/01/20 19:42:37 INFO namenode.NameNode: Caching file names occuring more than 10 times
15/01/20 19:42:37 INFO common.Storage: Image file of size 110 saved in 0 seconds.
15/01/20 19:42:37 INFO common.Storage: Storage directory /mnt/ephemeral-hdfs/dfs/name has been successfully formatted.
15/01/20 19:42:37 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ip-1.1.1.2.ec2.internal/1.3.3.7
************************************************************/
Starting ephemeral HDFS...
./ephemeral-hdfs/setup.sh: line 31: /root/ephemeral-hdfs/sbin/start-dfs.sh: No such file or directory
Warning: $HADOOP_HOME is deprecated.
starting namenode, logging to /mnt/ephemeral-hdfs/logs/hadoop-root-namenode-ip-1.1.1.2.ec2.internal.out
ec2-5.1.1.2.compute-1.amazonaws.com: starting datanode, logging to /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-1-3-4-2.ec2.internal.out
ec2-5.1.1.2.compute-1.amazonaws.com: Warning: $HADOOP_HOME is deprecated.
ec2-5.1.1.2.compute-1.amazonaws.com:
ec2-5-1-1-64.compute-1.amazonaws.com: Warning: $HADOOP_HOME is deprecated.
ec2-5-1-1-64.compute-1.amazonaws.com:
ec2-5-1-1-64.compute-1.amazonaws.com: starting secondarynamenode, logging to /mnt/ephemeral-hdfs/logs/hadoop-root-secondarynamenode-ip-1.1.1.2.ec2.internal.out
~/spark-ec2
Setting up persistent-hdfs
~/spark-ec2/persistent-hdfs ~/spark-ec2
Pseudo-terminal will not be allocated because stdin is not a terminal.
RSYNC'ing /root/persistent-hdfs/conf to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
Formatting persistent HDFS namenode...
Warning: $HADOOP_HOME is deprecated.
15/01/20 19:42:1 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ip-1.1.1.2.ec2.internal/1.3.3.7
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
15/01/20 19:42:55 INFO util.GSet: VM type = 64-bit
15/01/20 19:42:55 INFO util.GSet: 2% max memory = 19.33375 MB
15/01/20 19:42:55 INFO util.GSet: capacity = 2^21 = 2097152 entries
15/01/20 19:42:55 INFO util.GSet: recommended=2097152, actual=2097152
15/01/20 19:42:57 INFO namenode.FSNamesystem: fsOwner=root
15/01/20 19:42:58 INFO namenode.FSNamesystem: supergroup=supergroup
15/01/20 19:42:58 INFO namenode.FSNamesystem: isPermissionEnabled=false
15/01/20 19:42:58 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
15/01/20 19:42:58 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
15/01/20 19:42:58 INFO namenode.NameNode: Caching file names occuring more than 10 times
15/01/20 19:42:59 INFO common.Storage: Image file of size 110 saved in 0 seconds.
15/01/20 19:42:59 INFO common.Storage: Storage directory /vol/persistent-hdfs/dfs/name has been successfully formatted.
15/01/20 19:42:59 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ip-1.1.1.2.ec2.internal/1.3.3.7
************************************************************/
Persistent HDFS installed, won't start by default...
~/spark-ec2
Setting up spark-standalone
RSYNC'ing /root/spark/conf to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
RSYNC'ing /root/spark-ec2 to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
ec2-5.1.1.2.compute-1.amazonaws.com: no org.apache.spark.deploy.worker.Worker to stop
no org.apache.spark.deploy.master.Master to stop
starting org.apache.spark.deploy.master.Master, logging to /root/spark/sbin/../logs/spark-root-org.apache.spark.deploy.master.Master-1-ip-1.1.1.2.ec2.internal.out
ec2-5.1.1.2.compute-1.amazonaws.com: starting org.apache.spark.deploy.worker.Worker, logging to /root/spark/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-1-3-4-2.ec2.internal.out
Setting up tachyon
RSYNC'ing /root/tachyon to slaves...
ec2-5.1.1.2.compute-1.amazonaws.com
ec2-5.1.1.2.compute-1.amazonaws.com: Formatting Tachyon Worker @ ip-1-3-4-2.ec2.internal
ec2-5.1.1.2.compute-1.amazonaws.com: Removing local data under folder: /mnt/ramdisk/tachyonworker/
Formatting Tachyon Master @ ec2-5-1-1-64.compute-1.amazonaws.com
Formatting JOURNAL_FOLDER: /root/tachyon/libexec/../journal/
Formatting UNDERFS_DATA_FOLDER: hdfs://ec2-5-1-1-64.compute-1.amazonaws.com:9000/tachyon/data
Formatting UNDERFS_WORKERS_FOLDER: hdfs://ec2-5-5-1-4.compute-1.amazonaws.com:9000/tachyon/workers
TACHYON_LOGS_DIR: /root/tachyon/libexec/../logs
Killed 0 processes
Killed 0 processes
ec2-5-5-1-4.compute-1.amazonaws.com: Killed 0 processes
Starting master @ ec2-5-5-1-4.compute-1.amazonaws.com
ec2-5-5-1-4.compute-1.amazonaws.com: TACHYON_LOGS_DIR: /root/tachyon/libexec/../logs
ec2-5-5-1-4.compute-1.amazonaws.com: Formatting RamFS: /mnt/ramdisk (512mb)
ec2-5-5-1-4.compute-1.amazonaws.com: Starting worker @ ip-1-1-3-2.ec2.internal
Setting up ganglia
RSYNC'ing /etc/ganglia to slaves...
ec2-5-5-1-4.compute-1.amazonaws.com
Shutting down GANGLIA gmond: [FAILED]
Starting GANGLIA gmond: [ OK ]
Shutting down GANGLIA gmond: [FAILED]
Starting GANGLIA gmond: [ OK ]
Connection to ec2-5-5-1-4.compute-1.amazonaws.com closed.
Shutting down GANGLIA gmetad: [FAILED]
Starting GANGLIA gmetad: [ OK ]
Stopping httpd: [FAILED]
Starting httpd: [ OK ]
Connection to ec2-5-5-1-4.compute-1.amazonaws.com closed.
Spark standalone cluster started at http://ec2-5-5-1-4.compute-1.amazonaws.com:8080
Ganglia started at http://ec2-5-5-1-4.compute-1.amazonaws.com:5080/ganglia
Done!
~/Downloads/spark-1.2.0-bin-hadoop1/ec2 $
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment