tiling_notes.md

Notes on creating and serving raster tiles (Fall 2018)

Options to consider

1. Cutting edge - host file on S3 and serve tiles with lambda

Untested but very interesting, see:

• https://hi.stamen.com/stamen-aws-lambda-tiler-blog-post-76fc1138a145
• https://blog.mapbox.com/combining-the-power-of-aws-lambda-and-rasterio-8ffd3648c348

Probably best reserved for a larger project due to setup time and AWS costs.

2. Use a classic map server - Geoserver / Mapserver

Untested, I didn't want to manage the server application. In hindsight this may have been the best option due to unexpected requirement for ArcGIS desktop usage (Arc requires that the images be served as WMS/WMTS, it does not support XYZ tiles. This workaround exists - https://gis.stackexchange.com/questions/174569/adding-custom-web-tile-layer-to-arcmap - but does not work currently with a free AGOL account)

3. Convert to mbtiles and serve with tilestache / tilestream

Untested, see:

• https://github.com/mapbox/rio-mbtiles
• https://github.com/cutting-room-floor/tilestream

4. Cut tiles with gdal2tiles

This method chosen because:

• no server setup other than file transfer to existing web server
• zero additional cost within existing infrastructure
• easy consumption with Leaflet
• ArcMap requirement was not identified early on

gdal2tiles

https://www.gdal.org/gdal2tiles.html

• gdal2tiles.py only dumps to TMS (origin is lower left, see https://gist.github.com/tmcw/4954720)
• QGIS 3 load XYZ tiles easily, and even handles TMS with a check of a box
• convert existing gdal2tiles filenames from TMS to XYZ with https://gist.github.com/kannes/ebfe021458f96e4f30b5
• arcgis desktop can consume xyz tiles if you serve them as WMS using mapproxy
• various gdal2tiles implementations/forks are available on github (and elsewhere) but the latest GDAL (2.3.1) includes a working version that includes multithread support.

Data prep

Regardless of the method chosen, the data should generally be converted from source TIFFs to jpeg compressed COGs (to save space, speed serving)

What is a TIFF / COG?

• https://medium.com/planet-stories/a-handy-introduction-to-cloud-optimized-geotiffs-1f2c9e716ec3
• https://www.cogeo.org/
• https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF

Lazy process the rasters:

• https://www.perrygeo.com/lazy-raster-processing-with-gdal-vrts.html

Compression:

• http://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html
• https://kokoalberti.com/articles/geotiff-compression-optimization-guide/

Consider writing a python script with rasterio rather than using gdal translate:

• https://rasterio.readthedocs.io/en/latest/topics/profiles.html

Example

Process a folder of UTMZ10 tiffs:

# build vrt
gdalbuildvrt \
  test.vrt \
  path/to/tiffs/*.tif

# convert to single 8 bit COG
gdal_translate \
  test.vrt \
  test.tif \
  -ot Byte \
  -scale 0 65535 0 255 \
  -co TILED=YES \
  -co COPY_SRC_OVERVIEWS=NO \
  -co COMPRESS=JPEG \
  -co PHOTOMETRIC=YCBCR

# create overviews for COG
gdaladdo \
  --config COMPRESS_OVERVIEW JPEG \
  --config PHOTOMETRIC_OVERVIEW YCBCR \
  --config INTERLEAVE_OVERVIEW PIXEL \
  -r average \
  test.tif \
  2 4 8 16

# test that it is compliant
validate_cloud_optimized_geotiff.py test.tif

# create tiles
gdal2tiles.py \
  -e \
  -s EPSG:26910 \
  -z 12-20 \
  -w none \
  -r lanczos \
  -a 0 \
  --processes 12 \
  test.tif \
  test_tiles

When complete, convert file names to XYZ with https://gist.github.com/kannes/ebfe021458f96e4f30b5. Then transfer to web server.

Mapproxy

Config

Below config will serve XYZ tiles as WMS. See manual for more . https://mapproxy.org/.

services:
  wms:
    versions: ['1.1.1']
    image_formats: ['image/png']
    md:
        title: MYTITLE

layers:
 - name: cgpr2017
   title: TITLE
   sources: [cgpr2017cache]

caches:
  cgpr2017cache:
    grids: [GLOBAL_WEBMERCATOR]
    sources: [cgpr2017tilesource]

sources:
  cgpr2017tilesource:
    type: tile
    url: https://URL/%(z)s/%(x)s/%(y)s.png
    grid: 'GLOBAL_WEBMERCATOR'
    coverage:
      bbox: [xmin ymin xmax ymax]
      srs: 'EPSG:4326'

globals:
  image:
    formats:
      image/png:
        transparent: true
        opacity: 0.0

Deployment

Deployment with gunicorn behind nginx seems to work fine...although I haven't tested the performance beyond successfully opening imagery in an arcmap session. See https://mapproxy.org/docs/1.11.0/deployment.html#nginx

Some server config is necessary to get the python script to run on startup https://mapproxy.org/docs/1.11.0/deployment.html#python-http-server, which I haven't yet bothered with. It is probably an easy config but systemd docs are extensive: https://www.digitalocean.com/community/tutorials/systemd-essentials-working-with-services-units-and-the-journal