jpolchlo

jpolchlo

jpolchlo

Core GEE Refernce: https://developers.google.com/earth-engine/api_docs

File: https://github.com/ianeece/hyperion_preproc_gee_code/blob/master/dn_to_radiance.txt

image.select(ee.List.sequence(0,49)).divide(40.0).float()

float: convert to floating point type divide: pixelwise division of images (divide(40.0) implicitly generates constant image of value 40.0) select: takes bands from an image

jpolchlo

jpolchlo

[
    {
        "condition": ["building"],
        "subtree": [
            {
                "condition": ["roof:material=wood,building:roof=wood"],
                "subtree": [
                    {
                        "condition": ["building:condition=good,building:condition=excellent,condition=good"],
                        "color": "#276035"

jpolchlo

import org.apache.spark.sql.{DataFrame, Row}
import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
import org.apache.spark.sql.expressions.MutableAggregationBuffer
import org.apache.spark.sql.expressions.UserDefinedAggregateFunction
import org.apache.spark.sql.types._
import collection.mutable.{Map => MMap}

class SparseMapAggregator extends UserDefinedAggregateFunction {
  type IxMap = Map[Row, Long]
  type MMap = collection.mutable.Map[Row, Long]

jpolchlo

Starting from the directory containing bathy_xyz.csv:

docker run -it --rm -v $(pwd):/work quay.io/geodocker/pointcloud bash

This will give you the proper working environment, in which we start up a spark shell (version 2.3 or better):

spark-shell --packages org.locationtech.geotrellis:geotrellis-spark_2.11:2.2.0,com.azavea.geotrellis:geotrellis-pointcloud_2.11:0.2.4,org.locationtech.geomesa:geomesa-spark-jts_2.11:2.2.1  --repositories http://dl.bintray.com/azavea/geotrellis/ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer

jpolchlo

Polygonal Summaries in RasterFrames

We want to use RasterFrames (RF), with JTS frames (thanks, GeoMesa), to produce a polygonal summary. In usual terms, this means that each polygon is going to be compared to some base raster that the polygon covers. The raster cells contained in the geometry (either entirely or partially, depending on application), will be visited and accumulated according to some operation. The challenge, of course, is that a geometry might span multiple cells. This is a fairly easy problem to solve with RF, but will require some UDF work.

To get an environment to play with, start up spark shell with

spark-shell --packages com.azavea:vectorpipe_2.11:1.0.0,com.azavea.geotrellis:geotrellis-contrib-vlm_2.11:0.9.0,org.locationtech.rasterframes:rasterframes_2.11:0.8.0-RC3 \
            --repositories http://dl.bintray.com/azavea/maven,http://dl.bintray.com/azavea/geotrellis \
            --driver-memory 4g

jpolchlo

"Abyei (disp)"
"Afghanistan"
"Akrotiri (UK)"
"Aksai Chin (disp)"
"Albania"
"Algeria"
"American Samoa (US)"
"Antarctica"
"Antigua & Barbuda"
"Argentina"

jpolchlo

Rasterframes and GeoPySpark

Pluses and minuses of Rasterframes

• Dataframe model is great
  • Generally better for analysis to have tabular model
  • Catalyst is a huge advantage
    • Better resource management means jobs complete more often
  • Extensions with UDTs of GT types (and JTS types) improves usability for many