Illustrating apparent lag in IOOS Catalog OGC CSW endpoint temporal extent

iooscatalog_csw_oldtemporalextent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Illustrating apparent lag in IOOS Catalog OGC CSW endpoint temporal extent.\n",
    "Emilio Mayorga, 8/19/2018. Notice that a search for records from the last 5 days barely returns anything. But expanding the temporal window to 20 days expands the response to 28 records. This can not be the result of a single web service getting old, because the records come from multiple web services and providers. The only logical explanation I can think of is that the CSW records (or at least their temporal extent) are getting stale, by about two weeks (based on other tests)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the filters or filter set ups that will be reused"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spatial Bounding Box and filter first. Followed by temporal filter set up, variable, and other query set ups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mayorga/miniconda/envs/ioosaccess/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n",
      "/home/mayorga/miniconda/envs/ioosaccess/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n",
      "/home/mayorga/miniconda/envs/ioosaccess/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime, timedelta\n",
    "from ioos_tools.ioos import fes_date_filter, get_csw_records\n",
    "\n",
    "from owslib import fes\n",
    "\n",
    "from owslib.csw import CatalogueServiceWeb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_lon, max_lon = -127, -123.75\n",
    "min_lat, max_lat = 43, 48\n",
    "\n",
    "bbox = [min_lon, min_lat, max_lon, max_lat]\n",
    "crs = 'urn:ogc:def:crs:OGC:1.3:CRS84'\n",
    "\n",
    "bbox_crs = fes.BBox(bbox, crs=crs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "now_datetime_utc = datetime.utcnow()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "cf_names = [\n",
    "    'sea_water_temperature',\n",
    "    'sea_surface_temperature',\n",
    "    'sea_water_potential_temperature',\n",
    "    'equivalent_potential_temperature',\n",
    "    'sea_water_conservative_temperature',\n",
    "    'pseudo_equivalent_potential_temperature',\n",
    "]\n",
    "\n",
    "kw = dict(\n",
    "    wildCard='*',\n",
    "    escapeChar='\\\\',\n",
    "    singleChar='?',\n",
    "    propertyname='apiso:AnyText'\n",
    ")\n",
    "\n",
    "or_filt = fes.Or(\n",
    "    [fes.PropertyIsLike(literal=('*%s*' % val), **kw)\n",
    "        for val in cf_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up CSW Catalog end point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint = 'https://data.ioos.us/csw'\n",
    "\n",
    "csw = CatalogueServiceWeb(endpoint, timeout=60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Temporal Filter Range 1: 5 days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "days = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = now_datetime_utc - timedelta(days=days)\n",
    "stop = now_datetime_utc\n",
    "\n",
    "begin, end = fes_date_filter(start, stop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assemble the `fes` filter with `a`+`b`+`c`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter_list = [\n",
    "    fes.And(\n",
    "        [\n",
    "            bbox_crs,\n",
    "            begin, end,\n",
    "            or_filt,\n",
    "            fes.Not([fes.PropertyIsLike(literal='*cdip*', **kw)]),\n",
    "            fes.Not([fes.PropertyIsLike(literal='*grib*', **kw)]),\n",
    "        ]\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 1 records.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "get_csw_records(csw, filter_list, pagesize=10, maxrecords=1000)\n",
    "\n",
    "records = '\\n'.join(csw.records.keys())\n",
    "print('Found {} records.\\n'.format(len(csw.records.keys())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the records found:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[HYbrid Coordinate Ocean Model (HYCOM): Global]\n",
      "  hycom_global\n"
     ]
    }
   ],
   "source": [
    "for key, value in list(csw.records.items()):\n",
    "    print(u'[{}]\\n  {}'.format(value.title, key))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Only one record returned, and no observation assets!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Temporal Filter Range 2: 20 days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "days = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = now_datetime_utc - timedelta(days=days)\n",
    "stop = now_datetime_utc\n",
    "\n",
    "begin, end = fes_date_filter(start, stop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assemble the `fes` filter with `a`+`b`+`c`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "filter_list = [\n",
    "    fes.And(\n",
    "        [\n",
    "            bbox_crs,\n",
    "            begin, end,\n",
    "            or_filt,\n",
    "            fes.Not([fes.PropertyIsLike(literal='*cdip*', **kw)]),\n",
    "            fes.Not([fes.PropertyIsLike(literal='*grib*', **kw)]),\n",
    "        ]\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 28 records.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "get_csw_records(csw, filter_list, pagesize=10, maxrecords=1000)\n",
    "\n",
    "records = '\\n'.join(csw.records.keys())\n",
    "print('Found {} records.\\n'.format(len(csw.records.keys())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the records found:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[urn:ioos:station:NOAA.NOS.CO-OPS:9432780 station, Charleston, OR]\n",
      "  opendap.co-ops.nos.noaa.gov-urn_ioos_station_NOAA.NOS.CO-OPS_9432780\n",
      "[urn:ioos:station:NOAA.NOS.CO-OPS:9435380 station, South Beach, OR]\n",
      "  opendap.co-ops.nos.noaa.gov-urn_ioos_station_NOAA.NOS.CO-OPS_9435380\n",
      "[urn:ioos:station:NOAA.NOS.CO-OPS:9437540 station, Garibaldi, OR]\n",
      "  opendap.co-ops.nos.noaa.gov-urn_ioos_station_NOAA.NOS.CO-OPS_9437540\n",
      "[urn:ioos:station:NOAA.NOS.CO-OPS:9439040 station, Astoria, OR]\n",
      "  opendap.co-ops.nos.noaa.gov-urn_ioos_station_NOAA.NOS.CO-OPS_9439040\n",
      "[urn:ioos:station:NOAA.NOS.CO-OPS:9440581 station, Cape Disappointment]\n",
      "  opendap.co-ops.nos.noaa.gov-urn_ioos_station_NOAA.NOS.CO-OPS_9440581\n",
      "[urn:ioos:station:NOAA.NOS.CO-OPS:9440910 station, Toke Point, WA]\n",
      "  opendap.co-ops.nos.noaa.gov-urn_ioos_station_NOAA.NOS.CO-OPS_9440910\n",
      "[urn:ioos:station:NOAA.NOS.CO-OPS:9441102 station, Westport, WA]\n",
      "  opendap.co-ops.nos.noaa.gov-urn_ioos_station_NOAA.NOS.CO-OPS_9441102\n",
      "[urn:ioos:station:NOAA.NOS.CO-OPS:9442396 station, La Push, WA]\n",
      "  opendap.co-ops.nos.noaa.gov-urn_ioos_station_NOAA.NOS.CO-OPS_9442396\n",
      "[(CMOP) SATURN-02]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_cmop_ogi02\n",
      "[(CMOP) SATURN-03]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_cmop_saturn03\n",
      "[(CMOP) SATURN-04]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_cmop_mottb\n",
      "[(CMOP) SATURN-07]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_cmop_saturn07\n",
      "[CMOP Virtual Columbia River (SELFE); f33]\n",
      "  2d_ST\n",
      "[G1SST, 1km blended SST]\n",
      "  G1_SST_GLOBAL\n",
      "[Global SST & Sea Ice Analysis, L4 OSTIA, 0.05 deg daily (METOFFICE-GLO-SST-L4-NRT-OBS-SST-V2)]\n",
      "  AOOS_OSTIA\n",
      "[HYbrid Coordinate Ocean Model (HYCOM): Global]\n",
      "  hycom_global\n",
      "[(NERRS SOS) Tom's Creek (sostcmet), South Slough Reserve]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_nerrs_sostcmet\n",
      "[(NERRS) Station SOSCWQ - Charleston Bridge, South Slough Reserve]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_nerrs_soscwq\n",
      "[(NERRS) Station SOSECWQ - Elliot Creek, South Slough Reserve]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_nerrs_sosecwq\n",
      "[(NERRS) Station SOSVAWQ - Valino Island, South Slough Reserve]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_nerrs_sosvawq\n",
      "[(NERRS) Station SOSWIWQ - Winchester Arm, South Slough Reserve]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_nerrs_soswiwq\n",
      "[NOAA Coral Reef Watch Operational Daily Near-Real-Time Global 5-km Satellite Coral Bleaching Monitoring Products]\n",
      "  dhw_5km\n",
      "[(OSU) CB-06, 6NM W of Coos Bay]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_osu_cb06\n",
      "[(PSI) PCSGA - Bay Center Port mooring, Willapa Bay]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_psi_baycenter\n",
      "[(PSI) PCSGA - Nahcotta Port hatchery mooring, Willapa Bay]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_psi_nahcotta\n",
      "[Regional Ocean Modeling System (ROMS): Oregon Coast]\n",
      "  OCOS\n",
      "[Regional Ocean Modeling System (ROMS): Oregon Coast]\n",
      "  /opendap/hyrax/aggregated/ocean_time_aggregation.ncml\n",
      "[(SSNERR) SOSNSWQ Station - North Spit-BLM Boat Ramp near North Bend]\n",
      "  data.nanoos.org-urn_ioos_station_nanoos_nerrs_sosnswq\n"
     ]
    }
   ],
   "source": [
    "for key, value in list(csw.records.items()):\n",
    "    print(u'[{}]\\n  {}'.format(value.title, key))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**28 records returned, including plenty of observation assets plus more model forecast assets.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "_draft": {
   "nbviewer_url": "https://gist.github.com/45efe49edf6ff6f653ee9a41481ee665"
  },
  "gist": {
   "data": {
    "description": "IOOS_data_demo.ipynb",
    "public": true
   },
   "id": "45efe49edf6ff6f653ee9a41481ee665"
  },
  "kernelspec": {
   "display_name": "Python [conda env:ioosaccess]",
   "language": "python",
   "name": "conda-env-ioosaccess-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}