grovduck/_readme.md

## _readme.md

      
    Raw
  

              _readme.md
            
          
    Overview

This is an attempt to create a simple script for downloading new GNN data for the NWFP visualization group through Google Earth Engine.  This code will only work for members of the nwfp-vis or nwfp25-users groups as we don't intend to make the GNN neighbor rasters available outside these groups.
Note that our standard GNN download tool will actually NOT use GEE given the complexities of managing exports from GEE to Google Drive and the possible demands put on specific user accounts when multiple downloads are requested.  But this is a way that folks from NWFP-25 working groups can access the data in a somewhat easy API (although this may be a bit too little, too late ...)
Note that this "API" will likely change in the future, although I will try to minimize these changes.  Please let me know if you come across any configurations that are not working and provide the configuration object that you used when you triggered the bug.  I have not tested many of the different combinations.
Available options for GNN download configuration

attr


Any default GNN attribute with quotes ('VEGCLASS' is default)

model_series


MODEL_SERIES.V43 (default - 25 year NWFP (1986-2017))
MODEL_SERIES.V52 (CCDC-based new GNN series (1986-2021))

mask


FOREST_MASKS.NWFP_FOREST_MASK (default - Davis mask in NWFP, GAP elsewhere)
FOREST_MASKS.GAP_FOREST_MASK (GAP mask, GNN default)
FOREST_MASKS.NONE (unmasked)

boundary


BOUNDARIES.NWFP_BOUND (default)
BOUNDARIES.ORCAWA_BOUND (all of Oregon/Washington/California)
BOUNDARIES.BIO_ASSESSMENT_BOUND (Bio-assessment boundary)
BOUNDARIES.CA_BIOMASS_BOUND (Boundary for California biomass study)
BOUNDARIES.MR224_BOUND (Model Region 224 - Oregon West Cascades)

projection


PROJECTIONS.NATIONAL_ALBERS (default)
PROJECTIONS.R6_ALBERS (Regional Albers for Region 6)
PROJECTIONS.UTM_10 (UTM Zone 10 / NAD 83)

scalar


Amount to multiply attribute by to avoid lost precision (default: 1.0)

year


Any valid year 1985-2017 (default is 2017)

pyramiding_policy


Only used when saving images as assets rather than to Google Drive
For categorical assets, likely change this to {".default": "mode"}
Defaults to {".default": "mean"}.

export_name


Function to customize name of exported asset (expects object parameters as only argument)
Defaults to <attr>_<year>.tif

Default parameters

The default parameters set all of the required fields, although properties can (and should) be overridden for the parameters desired.  These are the current defaults:
  o.defaults = {
    attr: 'VEGCLASS',
    model_series: o.MODEL_SERIES.V43,
    mask: o.FOREST_MASKS.NWFP_FOREST_MASK,
    boundary: o.BOUNDARIES.NWFP_BOUND,
    projection: o.PROJECTIONS.NATIONAL_ALBERS,
    scalar: 1.0,
    year: 2017,
    pyramiding_policy: {".default": "mean"},
    export_name: export_name_default
  };

Functions available

get_gnn_image(params)

Function takes a Javascript object with all above keywords defined and returns a predicted GNN raster. Note that the returned image is always of type int, so if extra precision is needed, be sure to set the scalar parameter to a value greater than 1 (and typically a power of 10).
export_gnn_image(image, params)

Function takes two arguments, the image created from the first step and the parameter object and exports the image to the user's Drive account.
save_as_asset(image, image_collection, params)

Function takes three arguments, the image created from the first step, the image collection where the asset will be saved, and the parameter object.  This function will save the GNN attribute as an asset in the users's GEE space.
Example template scripts

To download 2016 CANCOV and 2017 BA_GE_3 for the NWFP area (both boundary and mask) and scaling each attribute value by 100.0, the following code can be used:
// Bring in the required modules
var gnn_func = require('users/gregorma/gnn:functions.js');
var config = require('users/gregorma/gnn:config.js');

// Specify the objects in an array
var download_objs = [
  {
    attr: 'CANCOV',
    year: 2016,
    scalar: 100.0
  },
  {
    attr: 'BA_GE_3',
    year: 2017,
    scalar: 100.0
  }
];

// Download each of the objects
for (var i = 0; i < download_objs.length; i++) {

  // Combine the default configuration and user-defined configuration
  var obj = gnn_func.assign({}, config.defaults, download_objs[i]);
  
  // Get the GNN image
  var img = config.get_gnn_image(obj);
  
  // Download the GNN image
  config.export_gnn_image(img, obj);
}

To download an unmasked BA_GE_3 2017 image for all of OR/CA/WA, use this download_objs configuration instead:
var download_objs = [
  {
    attr: 'BA_GE_3',
    year: 2017,
    scalar: 100.0,
    boundary: config.BOUNDARIES.ORCAWA_BOUND,
    mask: config.FOREST_MASKS.NONE
  }
];

To loop through a bunch of attributes and years and set up export tasks, try something like this:
var gnn_func = require('users/gregorma/gnn:functions.js');
var config = require('users/gregorma/gnn:config.js');

var attrs = [
  'TPH_GE_3',
  'CANCOV',
  'QMD_GE_3',
  'QMD_DOM',
  'BA_GE_3'
];

var years = [
  1993,
  2017
];

attrs.forEach(function(attr) {
  years.forEach(function(year) {
    var o = {
      attr: attr,
      year: year,
      scalar: 100.0,
      boundary: config.BOUNDARIES.ORCAWA_BOUND,
      mask: config.FOREST_MASKS.GAP_FOREST_MASK
    };
    var obj = gnn_func.assign({}, config.defaults, o);
    var img = config.get_gnn_image(obj);
    config.export_gnn_image(img, obj);
  });
});

Additional processing steps required after download

There are steps typically required after downloading the GeoTiffs from your Google Drive account.  I try to handle most of these steps through GDAL tools.
Tiled output

When the output raster exceeds a certain size (something like 1GB), GEE will break up the output into tiles.  Individual tiles have ten-digit row and column addressing added to file names and all tiles need to be mosaicked to create the full raster.  An example file name might be ba_ge_3_2017-0000000000-00000032768.tif.  From what I've been able to discern the row and column splits are not always predictable.  Often they break on powers of 2 (e.g. 32768, 65536), but I've seen other values as well (e.g. 46592).  Regardless, you need to create a single GeoTiff from all tiles.  I use gdalbuildvrt and gdal_translate to do this.  Assuming you have a set of tiles that look like this:
ba_ge_3_2017-0000000000-0000000000.tif
ba_ge_3_2017-0000000000-0000032768.tif
ba_ge_3_2017-0000032768-0000000000.tif
ba_ge_3_2017-0000032768-0000032768.tif
ba_ge_3_2017-0000065536-0000000000.tif
ba_ge_3_2017-0000065536-0000032768.tif

create a text file that simply lists these tiles (one per line) called something like filelist.txt, and then run:
gdalbuildvrt -input_file_list filelist.txt ba_ge_3_2017.vrt

This creates a GDAL Virtual Raster (VRT) which is really just an XML file with pointers to these GeoTiff files.  To convert this VRT to a GeoTiff, run the following:
gdal_translate -of GTiff -co "COMPRESS=LZW" ba_ge_3_2017.vrt ba_ge_3_2017.tif

From Joel Thompson, an alternative method for processing the downloaded tiles is using the Mosaic To New Raster function is ArcGIS.  It is found under Data Management Tools > Raster > Raster Dataset > Mosaic To New Raster.  Be sure to set the Pixel Type to match your data. The default is 8 bit unsigned integer.
Change Log

2019-08-08:

Bug in projected coordinates when using .reproject.  Changed to specifying the crs and crsTransform in the export command
Shared ORCAWA_BOUND with groups