smartnose

Spark internals through code

Nothing gives you more detail about spark internals than actually reading it source code. In addition, you get to learn many design techniques and improve your scala coding skills. These are the random notes I make while reading the spark code. The best way to comprehend the notes is to load spark code into an IDE, e.g. IntelliJ, and navigate the code on the side.

Genesis - creation of a spark cluster

The scripts for creating a spark cluster are: start-master.sh and start-slave.sh. Read them carefully, and you can see that both scripts are very similar except the values for $CLASS variable. For start-master.sh, the value is CLASS="org.apache.spark.deploy.master.Master", while the value for start-slave.sh is shown below with more context.

# NOTE: This exact class name is matched downstream by SparkSubmit.

Momoumar

#To download all your programming assignments including all files and notebooks, follow these steps:

#1 - Go to the root tree folder for instance: https://hub.coursera-notebooks.org/user/${user_token}/tree/
#2-  Open the terminal by clicking the + button on the right-hand corner
#3 - Enter the following command in the terminal: 
    tar cvfz allassignments.tar.gz *
#4 - The previous command will create a zip named allassignments containing all your programmings assignment
#5 - Select allassignments.tar.gz and download
#6 - Enjoy, don't forget to delete it afterward ;-)

akashpalrecha

mGBUfLn9

I was drawn to programming, science, technology and science fiction
ever since I was a little kid. I can't say it's because I wanted to
make the world a better place. Not really. I was simply drawn to it
because I was drawn to it. Writing programs was fun. Figuring out how
nature works was fascinating. Science fiction felt like a grand
adventure.

Then I started a software company and poured every ounce of energy
into it. It failed. That hurt, but that part is ok. I made a lot of
mistakes and learned from them. This experience made me much, much