ra-tolson

Gippy 1.0 Conversion Guide for GIPS

API docs more or less here: https://gippy.readthedocs.io/en/latest/quickstart.html

The diffs here mostly represent actual changes to gips code for PR #401. The conversion wasn't perfect; see See Missing or Regressed Features below.

Many of the changes followed a regular pattern of naming changes. Classes are still in CamelCase. Methods and functions all changed to python_case. Examples:

ra-tolson

Because the datahandler's logging was written in a huge hurry at the last minute, it's going to need a rewrite:

• A custom object is used for logging instead of django's logging facility; it's already running if you orm.setup().
• The root logger ('') is always hardcoded to talk to a socket; sooner or later we'll need a configuration-based logging config (which again django already provides).
• The custom method, Logger.log, calls python's logging.log, thus losing information as funcName will always be log; the custom extra used instead, caller, is not reliable, as 3rd party libs don't know to use it (we had this problem with RQ).
• In dhd.serve_log, jobid and caller had to be taken out due to above issues with RQ. This is because logging.basicConfig sets a configuration for the root (ie global) logger that makes poor assumptions.
• It's a datahandler logger, not a gips logger. In the long run, there is no meaningful separation between gips logging and datahandler logging, only differ

ra-tolson

# This is to demonstrate https://github.com/Applied-GeoSolutions/gips/issues/147

import os
import datetime
import gippy
from gips.inventory import dbinv, orm
from gips.datahandler import worker

# manual setup before running this:  load standard modis assets into inventory, but set
# status='scheduled' for MOD10A1.A2012336.h12v04.005.2012339213007.hdf

ra-tolson

diff --git a/dev_requirements.txt b/dev_requirements.txt
index 08821ed..31c38d5 100644
--- a/dev_requirements.txt
+++ b/dev_requirements.txt
@@ -1,4 +1,3 @@
--r requirements.txt
 pytest
 pytest-timeout
 pytest-mock
diff --git a/install.sh b/install.sh

ra-tolson

diff --git a/gips/inventory/dbinv/api.py b/gips/inventory/dbinv/api.py
index 450e56d..1bd06b6 100644
--- a/gips/inventory/dbinv/api.py
+++ b/gips/inventory/dbinv/api.py
@@ -62,9 +62,8 @@ def rectify_assets(asset_class):
     def rectify_asset(f_name):  # work function for _chunky_transaction()
         a = asset_class(f_name)
         (asset, created) = mao.update_or_create(
-            asset=a.asset, sensor=a.sensor, tile=a.tile, date=a.date,
-            name=f_name, driver=driver, status='complete'

ra-tolson

GIPS Data Inventory (v0.8.2)
Retrieving inventory for site NHseacoast-0
Retrieving data for bounds (-180.0, -90.0) - (180.0, 90.0)
Traceback (most recent call last):
  File "/home/tolson/src/gips/gips/data/core.py", line 807, in fetch
    cls.Asset.fetch(a, t, d)
  File "/home/tolson/src/gips/gips/data/merra/merra.py", line 300, in fetch
    data = dataset[asset][:, iy0:iy1, ix0:ix1].astype('float32')
  File "/home/tolson/src/gips/.venv/local/lib/python2.7/site-packages/pydap/model.py", line 192, in __getitem__
    return self.data[index]

ra-tolson

command to produce this was $ py.test --cov=gips -s -vv 2>&1 | tee coverage.log

============================= test session starts ==============================
platform linux2 -- Python 2.7.11+, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /home/tolson/src/gips/.venv/bin/python2
cachedir: .cache
rootdir: /home/tolson/src/gips, inifile: pytest.ini
plugins: cov-2.2.1, timeout-1.0.0
collecting ... collected 11 items

gips/test/sys/t_modis.py::t_inventory PASSED