palmerj/rgb_bigtiff_cogs_notes.md

## rgb_bigtiff_cogs_notes.md

      
    Raw
  

              rgb_bigtiff_cogs_notes.md
            
          
    Creating a Cloud Optimised Geotiffs (COGs) for raster photo imagery

This process outlines the process for creating Cloud Optimised Geotiffs
suitable for hosting in services such as AWS S3. COGs enables more
efficient workflows use cases such as fast access from Functions as a Services
(E.g AWS Lambda), or comsumption into client desktop GIS systems (e.g QGIS).
For more details on COGs please see https://www.cogeo.org/in-depth.html
1. Create a mosaic

First create the virtual mosaic from the directory of tiles, ensuring that a alpha
band is created in the VRT to set transparency where there is no source
raster.
gdalbuildvrt -addalpha mosaic.vrt *.tif
gdal_translate -b 1 -b 2 -b 3 -mask 4 mosaic.vrt rgbmask.vrt

2. Create a BigTiff

Create a BigTiff in a lossless compression to avoid quality loss. Use all
available CPU cores (DEFLATE compression method can use multi-threading).
The GeoTiff has an internal 1-bit mask band to provide transparency for
parts of the mosaic raster extent that contain no source data
gdal_translate \
  -b 1 -b 2 -b 3 -mask 4 \
  -of GTiff \
  -co BIGTIFF=YES \
  -co TILED=YES \
  -co COMPRESS=DEFLATE \
  -co PREDICTOR=2 \
  -co NUM_THREADS=ALL_CPUS \
  --config GDAL_CACHEMAX 4096 \
  -co ALPHA=YES \
  --config GDAL_TIFF_INTERNAL_MASK YES \
  mosaic.vrt output.tif

3. Create Overviews

Create overviews for the mosaic.
Note: For the gdaladdo there is known issue that generating
multiple overviews in the same TIFF file is slow and causes
tiff directory thrashing. The libtiff library has to go
back-and-forth between  multiple TIFF internal images, and
load/unload the TIFF indexes each time. For a huge file, this
involes a lot of I/O. The workaround, which is especially fine
for the COG case, is to generate each overview level in its
own file by cascading calls to gdaladdo. See
https://trac.osgeo.org/gdal/ticket/5067#comment:2 for more
info

OVERVIEW=output.tif
for VARIABLE in 2 4 8 16 32 64 128 256 512
do
  gdaladdo \
    --config GDAL_CACHEMAX 4096 \
    --config COMPRESS_OVERVIEW DEFLATE \
    -ro \
    -r average \
    $OVERVIEW 2
  OVERVIEW = ${OVERVIEW}.ovr
done

4. Create Cloud Optimised Geotiff (COGS)

Create COGs, applying final JPEG compression, and  copying and
compressing the previously generated overview's IFD
(Image File Directory) index in the header of the file to be efficiently
fetchable via cloud web APIs. The GeoTiff is creates internal tiles
of 256x256 for the main resolution and 128x128 tiles for overviews
NOTES:

When compressing with JPEG multi-threading can not be used.
Increasing the block size can reduce the size of the
IFD. But larger blocks can cause more bytes to be pulled for random
access if the compression rate is not high. Going from teh default
of 256 to 512 will reduce the index by a factor of 4. The size of the
TIFF index arrays, for each pyramid level, is :
2 * ceil(xsize / blockxsize) * ceil(ysize / blockysize) * 8 bytes
Because we use an internal mask, this value has to be multiplied by 2.

gdal_translate \
  -of GTiff \
  -co BIGTIFF=YES \
  -co TILED=YES \
  -co BLOCKXSIZE=256 \
  -co BLOCKYSIZE=256 \
  -co COMPRESS=JPEG \
  -co JPEG_QUALITY=85 \
  -co PHOTOMETRIC=YCBCR \
  -co COPY_SRC_OVERVIEWS=YES \
  -co ALPHA=YES \
  --config GDAL_TIFF_INTERNAL_MASK YES \
  --config GDAL_TIFF_OVR_BLOCKSIZE 128 \
  --config GDAL_CACHEMAX 4096 \
  output.tif output_cogs.tif

5. Validate the COGs Geotiff

Check there are no errors or warnings from the following script
python validate_cloud_optimized_geotiff.py output_cogs.tif