Skip to content

Instantly share code, notes, and snippets.

@lossyrob
Last active August 29, 2015 14:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lossyrob/fda457994a4be6db598c to your computer and use it in GitHub Desktop.
Save lossyrob/fda457994a4be6db598c to your computer and use it in GitHub Desktop.
Spark submit for an ingest job in Accumulo
### INGEST SPATIAL GEOTIFF IN ACCUMULO ###
# geotrellis-spark JAR. Shouldn't have to change this one if running in the root folder (remember to run ./sbt "project spark" assembly)
JAR=/Users/rob/proj/gt/geotrellis/spark/target/scala-2.10/geotrellis-spark-assembly-0.10.0-SNAPSHOT.jar
# Directory with the input tiled GeoTIFF's
INPUT=file:/Users/rob/proj/climate/data/one-month-tiles
# Table to store tiles
TABLE=nexmonth
# Name of the layer. This will be used in conjunction with the zoom level to reference the layer (see LayerId)
LAYER_NAME=nexmonth
# This defines the destination spatial reference system we want to use
# (in this case, Web Mercator)
CRS=EPSG:3857
# true means we want to pyramid the raster up to larger zoom levels,
# so if our input rasters are at a resolution that maps to zoom level 11, pyramiding will also save
# off levels 10, 9, ..., 1.
PYRAMID=true
# true will delete the HDFS data for the layer if it already exists.
CLOBBER=true
# We need to remove some bad signatures from the assembled JAR. We're working on excluding these
# files as part of the build step, this is a workaround.
zip -d $JAR META-INF/ECLIPSEF.RSA
zip -d $JAR META-INF/ECLIPSEF.SF
# Run the spark submit job
spark-submit \
--class geotrellis.spark.ingest.AccumuloIngestCommand \
$JAR \
--instance gis --user root --password secret --zookeeper localhost \
--crs $CRS \
--pyramid $PYRAMID --clobber $CLOBBER \
--input $INPUT \
--layerName $LAYER_NAME \
--table $TABLE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment