Skip to content

Instantly share code, notes, and snippets.

View dbrambilla's full-sized avatar

dbrambilla

32 followers · 14 following
View GitHub Profile
@dbrambilla
dbrambilla / spark-rest-submit.sh
Created October 31, 2017 00:29 — forked from yaravind/spark-rest-submit.sh
Submit apps (SparkPi as e.g.) to spark cluster using rest api
curl -X POST -d http://master-host:6066/v1/submissions/create --header "Content-Type:application/json" --data '{
"action": "CreateSubmissionRequest",
"appResource": "hdfs://localhost:9000/user/spark-examples_2.11-2.0.0.jar",
"clientSparkVersion": "2.0.0",
"appArgs": [ "10" ],
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass": "org.apache.spark.examples.SparkPi",
"sparkProperties": {
@dbrambilla
dbrambilla / get_job_status.sh
Created October 9, 2017 16:15 — forked from arturmkrtchyan/get_job_status.sh
Apache Spark Hidden REST API
curl http://spark-cluster-ip:6066/v1/submissions/status/driver-20151008145126-0000
@dbrambilla
dbrambilla / FIleSystemOperations.java
Created October 9, 2017 16:14 — forked from ashrithr/FIleSystemOperations.java
HDFS FileSystems API example
package com.cloudwick.mapreduce.FileSystemAPI;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;