Skip to content

Instantly share code, notes, and snippets.

@nbest937
nbest937 / sparkSessionOnEMR.scala
Last active October 3, 2018 22:11 — forked from dyno/sparkSessionOnEMR.scala
Working with SparkSession with Ammonite .on AWS EMR
// =============================================================================
// emr.sc -- import it in predef.sc
import ammonite.ops._
// add emr jars and config dir to interp class path
//
def loadEmrJars() = {
// /etc/zeppelin/conf/zeppelin-env.sh
val libDirs = List( //
"/usr/lib/hadoop-lzo/lib",
"/usr/lib/spark/jars",
library(maps)
library(geosphere)
library(plyr)
library(ggplot2)
library(sp)
airports <- read.csv("http://www.stanford.edu/~cengel/cgi-bin/anthrospace/wp-content/uploads/2012/03/airports.csv", as.is=TRUE, header=TRUE)
flights <- read.csv("http://www.stanford.edu/~cengel/cgi-bin/anthrospace/wp-content/uploads/2012/03/PEK-openflights-export-2012-03-19.csv", as.is=TRUE, header=TRUE)
# aggregate nunber of flights
@nbest937
nbest937 / grbSum.R
Last active December 11, 2015 06:08
A contrived example of R processing some raster data. It accepts the path to a NARR GRB file from ftp://nomads.ncdc.noaa.gov/NARR and the number of bands to scan and computes the sum of each layer.
#!/software/R-2.15-el6-x86_64/bin/Rscript --vanilla
##!/usr/bin/Rscript --vanilla
library( raster, quietly= TRUE)
args <- commandArgs(TRUE)
grbFile <- args[ 1]
scanHowMany <- args[ 2]
@nbest937
nbest937 / convertGridFolders.sh
Created October 1, 2012 16:36
shell script for restructuring pDSSAT grid directories to the new two-tier design required by the Linux filesystem
#!/bin/bash
# arguments should be relative paths from the working directory (not
# necessarily the directory where this script lives) to the top level
# of a grid/1234567 structure so that it can be converted to the
@nbest937
nbest937 / split.R
Created July 10, 2012 16:52
R script to split DSSAT RIX stanzas into individual files
#!/soft/R/gnu-4.1/2.14.0/bin/Rscript --vanilla
library( stringr)
args <- commandArgs()
args <- args[-(1:match("--args", args))]
inputLines <- readLines( args[ 1])
## inputLines <- readLines( "input/mail merge sample.txt")