lossyrob

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
 <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:8020</value>
  </property>
  <property>
    <name>fs.trash.interval</name>

lossyrob

#! /usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0

lossyrob

MAC

Hadoop

brew install hadoop

Go to https://gist.github.com/lossyrob/c492b0c81aacc31c7953 and copy in those config files to /usr/local/Cellar/hadoop/VERSION/libexec/etc/hadoop

Find any config that has "/Users/rob" and replace that directory with your own path.

lossyrob

Web services - Runs locally on one machine or on AWS

Data validation and metadata generation

• Request (JSON):
• Includes imagery to be processed -Includes metadata that can’t be read out of the image.
• Action:
• Validate each image file
• Generate metadata json for each imagery if set is valid.
• Overwrites any existing metadata.

lossyrob

// Find the data here: https://www.opendataphilly.org/dataset/schools/resource/b29d98a9-6b96-4742-aedc-80cac6398d1a
import scala.io._

import geotrellis.vector._
import geotrellis.vector.io.json._

import spray.json._

// { "OBJECTID": 1, "AUN": 126515001, "SCHOOL_NUM": 3712, "LOCATION_ID": "6250", "FACIL_NAME": "Henry, Charles W.", "FACILNAME_LABEL": "Henry, Charles W.", "FACIL_ADDRESS": "601 Carpenter La.", "ZIPCODE": "19119 - 3405", "FACIL_TELEPHONE": "951-4006", "ACTIVE": "Open", "GRADE_LEVEL": "Elem\/Middle", "GRADE_ORG": "K-8", "ENROLLMENT": 478, "TYPE": "District", "TYPE_SPECIFIC": "District", "GlobalID": "1eea0c90-80c1-4cd2-a8e6-40ea9973ea35" }
case class SchoolData(enrollment: Int)

lossyrob

import geotrellis.proj4._
import geotrellis.vector._
import geotrellis.raster._
import geotrellis.raster.io.geotiff._

import org.apache.spark.rdd._

trait Example {
  // Some information you'll need to supply...
  val crs = CRS.fromName("EPSG")

lossyrob

OAM Server update

Current work: Local tiler

• Wrote a local tiler in node js using swfr. Ran against a sample set of LandSat and PlanetLabs imagery. For zoom level 13 over 1.4 G of imagery, takes about 19 minutes.
• Wrote a GeoTrellis/Scala version of the tiler to compare, generates in about 9 minutes.
• Ran gdal2tilesp, ran in 8 minutes.
• Generates 471M of pngs.

Issues

• Image prioritization:

lossyrob

import geotrellis.raster._
import geotrellis.raster.render._
import geotrellis.raster.resample._
import geotrellis.raster.io.geotiff._
import geotrellis.vector._
import geotrellis.vector.io.json._
import geotrellis.vector.reproject._
import geotrellis.spark._
import geotrellis.spark.tiling._
import geotrellis.proj4._

lossyrob

"use strict";

var assert = require("assert"),
    fs = require("fs"),
    os = require("os"),
    path = require("path"),
    url = require("url"),
    util = require("util");

var async = require("async"),

lossyrob

import os
from subprocess import call

images = ["LC80160292015156LGN00",
"LC80160292015124LGN00",
"LC80160292015092LGN00",
"LC80160292015060LGN00",
"LC80160292015028LGN00",
"LC80160292014361LGN00",
"LC80160292014329LGN00",