Skip to content

Instantly share code, notes, and snippets.

Tianxing Li nex3z

Block or report user

Report or block nex3z

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View spark-internals-through-code.md

Spark internals through code

Nothing gives you more detail about spark internals than actually reading it source code. In addition, you get to learn many design techniques and improve your scala coding skills. These are the random notes I make while reading the spark code. The best way to comprehend the notes is to load spark code into an IDE, e.g. IntelliJ, and navigate the code on the side.

Genesis - creation of a spark cluster

The scripts for creating a spark cluster are: start-master.sh and start-slave.sh. Read them carefully, and you can see that both scripts are very similar except the values for $CLASS variable. For start-master.sh, the value is CLASS="org.apache.spark.deploy.master.Master", while the value for start-slave.sh is shown below with more context.

# NOTE: This exact class name is matched downstream by SparkSubmit.
@staltz
staltz / introrx.md
Last active Nov 18, 2019
The introduction to Reactive Programming you've been missing
View introrx.md
@Stanback
Stanback / nginx.conf
Last active Nov 18, 2019 — forked from michiel/cors-nginx.conf
Example Nginx configuration for adding cross-origin resource sharing (CORS) support to reverse proxied APIs
View nginx.conf
#
# CORS header support
#
# One way to use this is by placing it into a file called "cors_support"
# under your Nginx configuration directory and placing the following
# statement inside your **location** block(s):
#
# include cors_support;
#
# As of Nginx 1.7.5, add_header supports an "always" parameter which
You can’t perform that action at this time.