Skip to content

Instantly share code, notes, and snippets.

@rsignell-usgs

rsignell-usgs/comt_pycsw.ipynb

Last active August 29, 2015 14:13
Show Gist options
  • Save rsignell-usgs/58f80172a77b8182843f to your computer and use it in GitHub Desktop.
Save rsignell-usgs/58f80172a77b8182843f to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
comt_pycsw.ipynb
{
"worksheets": [
{
"cells": [
{
"metadata": {},
"cell_type": "markdown",
"source": "#Use CSW to find ROMS data at COMT pycsw"
},
{
"metadata": {},
"cell_type": "code",
"input": "from owslib.csw import CatalogueServiceWeb\nfrom owslib import fes",
"prompt_number": 1,
"outputs": [],
"language": "python",
"trusted": true,
"collapsed": false
},
{
"metadata": {},
"cell_type": "markdown",
"source": "##Find model results at COMT pycsw"
},
{
"metadata": {},
"cell_type": "code",
"input": "endpoint = 'http://comt.sura.org:8000/pycsw' # NODC/UAF Geoportal: granule level\ncsw = CatalogueServiceWeb(endpoint,timeout=60)\nprint csw.version",
"prompt_number": 2,
"outputs": [
{
"output_type": "stream",
"text": "2.0.2\n",
"stream": "stdout"
}
],
"language": "python",
"trusted": true,
"collapsed": false
},
{
"metadata": {},
"cell_type": "code",
"input": "csw.get_operation_by_name('GetRecords').constraints",
"prompt_number": 3,
"outputs": [
{
"text": "[Constraint: SupportedISOQueryables - ['apiso:DistanceValue', 'apiso:Abstract', 'apiso:RevisionDate', 'apiso:Subject', 'apiso:KeywordType', 'apiso:Title', 'apiso:CRS', 'apiso:PublicationDate', 'apiso:Type', 'apiso:AlternateTitle', 'apiso:BoundingBox', 'apiso:AnyText', 'apiso:ParentIdentifier', 'apiso:Modified', 'apiso:Operation', 'apiso:Format', 'apiso:TempExtent_end', 'apiso:DistanceUOM', 'apiso:OrganisationName', 'apiso:ServiceType', 'apiso:TempExtent_begin', 'apiso:ResourceLanguage', 'apiso:ServiceTypeVersion', 'apiso:OperatesOn', 'apiso:Denominator', 'apiso:HasSecurityConstraints', 'apiso:OperatesOnIdentifier', 'apiso:GeographicDescriptionCode', 'apiso:Language', 'apiso:Identifier', 'apiso:OperatesOnName', 'apiso:TopicCategory', 'apiso:CreationDate', 'apiso:CouplingType'],\n Constraint: AdditionalQueryables - ['apiso:Lineage', 'apiso:Classification', 'apiso:Creator', 'apiso:Relation', 'apiso:OtherConstraints', 'apiso:SpecificationTitle', 'apiso:ResponsiblePartyRole', 'apiso:SpecificationDateType', 'apiso:Degree', 'apiso:Contributor', 'apiso:ConditionApplyingToAccessAndUse', 'apiso:SpecificationDate', 'apiso:AccessConstraints', 'apiso:Publisher'],\n Constraint: SupportedDublinCoreQueryables - ['dc:contributor', 'dc:source', 'dc:language', 'dc:title', 'dc:subject', 'dc:creator', 'dc:type', 'ows:BoundingBox', 'dct:modified', 'dct:abstract', 'dc:relation', 'dc:date', 'dc:identifier', 'dc:publisher', 'dc:format', 'csw:AnyText', 'dc:rights']]",
"output_type": "pyout",
"metadata": {},
"prompt_number": 3
}
],
"language": "python",
"trusted": true,
"collapsed": false
},
{
"metadata": {},
"cell_type": "code",
"input": "val = 'ROMS'\nfilter1 = fes.PropertyIsLike(propertyname='apiso:AnyText',literal=('*%s*' % val),\n escapeChar='\\\\',wildCard='*',singleChar='?')\nfilter_list = [ filter1 ]",
"prompt_number": 4,
"outputs": [],
"language": "python",
"trusted": true,
"collapsed": false
},
{
"metadata": {},
"cell_type": "code",
"input": "csw.getrecords2(constraints=filter_list,maxrecords=100,esn='full')\nprint len(csw.records.keys())\nfor rec in list(csw.records.keys()):\n print csw.records[rec].title \n ",
"prompt_number": 5,
"outputs": [
{
"output_type": "stream",
"text": "16\nCOAWST Forecast System : USGS : US East Coast and Gulf of Mexico (Experimental)\nUCSC California Current System, version wc12.0_coamps_hycom_gls_v736_tave24_01 with npzd\nEstuarine Hypoxia : VIMS : CBOFS : 2004-2005\nEstuarine Hypoxia : VIMS : ChesROMS : 1term constR\nEstuarine Hypoxia : VIMS : ChesROMS : 1term ddR\nEstuarine Hypoxia : WHOI : ChesROMS-1term : 1991-2005 1termDO\nShelf Hypoxia : DAL : ROMS : 2004-2009 climatology obc with oxygen hnd soc 20layers\nShelf Hypoxia : DAL : ROMS : 2004-2009 climatology obc with oxygen inst rem 20layers\nShelf Hypoxia : DAL : ROMS : 2004-2009 hycom obc with oxygen hnd soc 30layers\nShelf Hypoxia : DAL : ROMS : 2004-2009 hycom obc with oxygen inst rem 30layers\nShelf Hypoxia : DAL : ROMS : 2004-2009 ias obc with oxygen hnd soc 30layers\nShelf Hypoxia : DAL : ROMS : 2004-2009 ias obc with oxygen inst rem 30layers\nShelf Hypoxia : TAMU : ROMS : 2004-2009 hycom obc 24h\nShelf Hypoxia : TAMU : ROMS : 2004-2009 iasnfs obc 24h\nShelf Hypoxia : TAMU : ROMS : 2004-2009 hycom obc 24h\nROMS/TOMS 3.0 - New Floria Shelf Application\n",
"stream": "stdout"
}
],
"language": "python",
"trusted": true,
"collapsed": false
},
{
"metadata": {},
"cell_type": "code",
"input": "choice=np.random.choice(list(csw.records.keys()))\nprint(csw.records[choice].title)\ncsw.records[choice].references",
"prompt_number": 6,
"outputs": [
{
"output_type": "stream",
"text": "Shelf Hypoxia : DAL : ROMS : 2004-2009 hycom obc with oxygen inst rem 30layers\n",
"stream": "stdout"
},
{
"text": "[{'scheme': 'WWW:LINK',\n 'url': 'http://comt.sura.org/thredds/dodsC/data/comt_1_archive/shelf_hypoxia/DAL_ROMS/2004-2009_hycom_obc_with_oxygen_inst_rem_30layers.html'},\n {'scheme': 'WWW:LINK',\n 'url': 'http://www.ncdc.noaa.gov/oa/wct/wct-jnlp-beta.php?singlefile=http://comt.sura.org/thredds/dodsC/data/comt_1_archive/shelf_hypoxia/DAL_ROMS/2004-2009_hycom_obc_with_oxygen_inst_rem_30layers'},\n {'scheme': 'OPeNDAP:OPeNDAP',\n 'url': 'http://comt.sura.org/thredds/dodsC/data/comt_1_archive/shelf_hypoxia/DAL_ROMS/2004-2009_hycom_obc_with_oxygen_inst_rem_30layers'},\n {'scheme': 'OGC:WCS',\n 'url': 'http://comt.sura.org/thredds/wcs/data/comt_1_archive/shelf_hypoxia/DAL_ROMS/2004-2009_hycom_obc_with_oxygen_inst_rem_30layers?service=WCS&version=1.0.0&request=GetCapabilities'},\n {'scheme': 'OGC:WMS',\n 'url': 'http://comt.sura.org/thredds/wms/data/comt_1_archive/shelf_hypoxia/DAL_ROMS/2004-2009_hycom_obc_with_oxygen_inst_rem_30layers?service=WMS&version=1.3.0&request=GetCapabilities'},\n {'scheme': 'UNIDATA:NCSS',\n 'url': 'http://comt.sura.org/thredds/ncss/data/comt_1_archive/shelf_hypoxia/DAL_ROMS/2004-2009_hycom_obc_with_oxygen_inst_rem_30layers/dataset.html'}]",
"output_type": "pyout",
"metadata": {},
"prompt_number": 6
}
],
"language": "python",
"trusted": true,
"collapsed": false
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Add bounding box constraint. To specify lon,lat order for bbox (which we want to do so that we can use the same bbox with either geoportal server or pycsw requests), we need to request the bounding box specifying the CRS84 coordinate reference system. The CRS84 option is available in `pycsw 1.1.10`+. The ability to specify the `crs` in the bounding box request is available in `owslib 0.8.12`+. For more info on the bounding box problem and how it was solved, see this [pycsw issue](https://github.com/geopython/pycsw/issues/287), this [geoportal server issue](https://github.com/Esri/geoportal-server/issues/124), and this [owslib issue](https://github.com/geopython/OWSLib/issues/201)"
},
{
"metadata": {},
"cell_type": "code",
"input": "bbox = [-87.40, 24.25, -74.70, 36.70]\nbbox_filter = fes.BBox(bbox,crs='urn:ogc:def:crs:OGC:1.3:CRS84')\nfilter_list = [filter1, bbox_filter]\ncsw.getrecords2(constraints=filter_list, maxrecords=1000)",
"prompt_number": 11,
"outputs": [],
"language": "python",
"trusted": true,
"collapsed": false
},
{
"metadata": {},
"cell_type": "code",
"input": "print len(csw.records.keys())\nfor rec in list(csw.records.keys()):\n print csw.records[rec].title ",
"prompt_number": 10,
"outputs": [
{
"output_type": "stream",
"text": "81\nCOAWST Forecast System : USGS : US East Coast and Gulf of Mexico (Experimental)\nUCSC California Current System, version wc12.0_coamps_hycom_gls_v736_tave24_01 with npzd\nEstuarine Hypoxia : VIMS : CBOFS : 2004-2005\nEstuarine Hypoxia : VIMS : ChesROMS : 1term constR\nEstuarine Hypoxia : VIMS : ChesROMS : 1term ddR\nEstuarine Hypoxia : VIMS : EFDC : 2004_DO\nEstuarine Hypoxia : WHOI : ChesROMS-1term : 1991-2005 1termDO\nInundation Tropical : MDL : SLOSH : Hurricane Ike final run egm3\nInundation Tropical : MDL : SLOSH : Hurricane Rita final run egm3\nInundation Tropical : UND : ADCIRC : Hurricane Ike 2D final run with waves\nInundation Tropical : UND : ADCIRC : Hurricane Ike 2D final run without waves\nInundation Tropical : UND : ADCIRC : Hurricane Ike 2D preliminary run 1 spinup\nInundation Tropical : UND : ADCIRC : Hurricane Ike 2D preliminary run 2 spinup without advection implicit 1s\nInundation Tropical : UND : ADCIRC : Hurricane Ike 2D preliminary run 3 spinup with levees\nInundation Tropical : UND : ADCIRC : Hurricane Ike 2D preliminary run 4 spinup with levees and river flow\nInundation Tropical : UND : ADCIRC : Hurricane Ike 2D preliminary run 5 tides only\nInundation Tropical : UND : ADCIRC : Hurricane Ike 3D final run with waves\nInundation Tropical : UND : ADCIRC : Hurricane Ike 3D final run without waves\nInundation Tropical : UND : ADCIRC : Hurricane Ike 3D preliminary run 1 spinup\nInundation Tropical : UND : ADCIRC : Hurricane Ike Timing with waves\nInundation Tropical : UND : ADCIRC : Hurricane Ike Timing without waves\nInundation Tropical : UND : ADCIRC : Hurricane Rita 2D final run with waves\nInundation Tropical : UND : ADCIRC : Hurricane Rita 2D final run without waves\nInundation Tropical : UND : ADCIRC : Hurricane Rita 2D preliminary run 1 spinup\nInundation Tropical : UND : ADCIRC : Hurricane Rita 2D preliminary run 2 spinup without advection implicit 1s\nInundation Tropical : UND : ADCIRC : Hurricane Rita 2D preliminary run 3 tides only\nInundation Tropical : UND : ADCIRC : Hurricane Rita 2D preliminary run 4 with waves without advection implicit 1s\nInundation Tropical : UND : ADCIRC : Hurricane Rita 2D preliminary run 5 without waves and advection implicit 1s\nInundation Tropical : UND : ADCIRC : Hurricane Rita 3D final run with waves\nInundation Tropical : UND : ADCIRC : Hurricane Rita 3D final run without waves\nInundation Tropical : UND : ADCIRC : Hurricane Rita 3D preliminary run 1 spinup\nInundation Tropical : UND : ADCIRC : Tides only 2D final run\nInundation Tropical : UND : ADCIRC : Tides only 2D preliminary run 1 without advection implicit 1s\nInundation Tropical : UND : ADCIRC : Tides only 3D final run\nInundation Tropical : UND : SLOSH : Hurricane Ike final run egm3\nInundation Tropical : UND : SLOSH : Hurricane Ike preliminary run 4 egm3 without tides\nInundation Tropical : UND : SLOSH : Hurricane Ike preliminary run 5 egm3 with SLOSH winds\nInundation Tropical : UND : SLOSH : Hurricane Ike preliminary run 6 egm3 with SLOSH winds without tides\nInundation Tropical : UND : SLOSH : Hurricane Rita final run egm3\nInundation Tropical : UND : SLOSH : Hurricane Rita preliminary run 4 egm3 without tides\nInundation Tropical : UND : SLOSH : Hurricane Rita preliminary run 5 egm3 with SLOSH winds\nInundation Tropical : UND : SLOSH : Hurricane Rita preliminary run 6 egm3 with SLOSH winds without tides\nInundation Tropical : USF : FVCOM : Hurricane Ike 2D final run with waves\nInundation Tropical : USF : FVCOM : Hurricane Ike 2D final run without waves\nInundation Tropical : USF : FVCOM : Hurricane Ike 3D final run with waves\nInundation Tropical : USF : FVCOM : Hurricane Ike 3D final run without waves\nInundation Tropical : USF : FVCOM : Hurricane Rita 2D final run with waves\nInundation Tropical : USF : FVCOM : Hurricane Rita 2D final run without waves\nInundation Tropical : USF : FVCOM : Hurricane Rita 3D final run with waves\nInundation Tropical : USF : FVCOM : Hurricane Rita 3D final run without waves\nInundation Tropical : USF : FVCOM : SWAN Ike final run with 2DSurge\nInundation Tropical : USF : FVCOM : SWAN Ike final run without Surge\nInundation Tropical : USF : FVCOM : SWAN Rita final run with 2DSurge\nInundation Tropical : USF : FVCOM : SWAN Rita final run without Surge\nInundation Tropical : VIMS : SELFE : Hurricane Ike 2D final run wave only\nInundation Tropical : VIMS : SELFE : Hurricane Ike 2D final run with waves\nInundation Tropical : VIMS : SELFE : Hurricane Ike 2D final run without waves\nInundation Tropical : VIMS : SELFE : Hurricane Ike 3D final run with waves\nInundation Tropical : VIMS : SELFE : Hurricane Ike 3D final run without waves\nInundation Tropical : VIMS : SELFE : Hurricane Rita 2D final run waves only\nInundation Tropical : VIMS : SELFE : Hurricane Rita 2D final run with waves\nInundation Tropical : VIMS : SELFE : Hurricane Rita 2D final run without waves\nInundation Tropical : VIMS : SELFE : Hurricane Rita 3D final run with waves\nInundation Tropical : VIMS : SELFE : Hurricane Rita 3D final run without waves\nShelf Hypoxia : DAL : ROMS : 2004-2009 climatology obc with oxygen hnd soc 20layers\nShelf Hypoxia : DAL : ROMS : 2004-2009 climatology obc with oxygen inst rem 20layers\nShelf Hypoxia : DAL : ROMS : 2004-2009 hycom obc with oxygen hnd soc 30layers\nShelf Hypoxia : DAL : ROMS : 2004-2009 hycom obc with oxygen inst rem 30layers\nShelf Hypoxia : DAL : ROMS : 2004-2009 ias obc with oxygen hnd soc 30layers\nShelf Hypoxia : DAL : ROMS : 2004-2009 ias obc with oxygen inst rem 30layers\nShelf Hypoxia : NOAA : NGOM : 2005-2011 NGOM\nShelf Hypoxia : TAMU : ROMS : 2004-2009 hycom obc 24h\nShelf Hypoxia : TAMU : ROMS : 2004-2009 iasnfs obc 24h\nShelf Hypoxia : TAMU : ROMS : 2004-2009 hycom obc 24h\nPr Inundation Tropical : EMC : ADCIRC-WW3 : Dec2013Storm 2D preliminary run 1 waves only\nPr Inundation Tropical : NRL : DELFT3D : Hurricane Ike 2D preliminary run 1 without waves\nNECOFS GOM3 (FVCOM) - Northeast US - Latest Forecast\nNECOFS GOM3 Wave - Northeast US - Latest Forecast\nUSF FVCOM - Nowcast Aggregation\nROMS/TOMS 3.0 - New Floria Shelf Application\nUSF Wave Forecast Products\n",
"stream": "stdout"
}
],
"language": "python",
"trusted": true,
"collapsed": false
},
{
"metadata": {},
"cell_type": "code",
"input": "",
"outputs": [],
"language": "python",
"trusted": true,
"collapsed": false
}
],
"metadata": {}
}
],
"metadata": {
"gist_id": "58f80172a77b8182843f",
"name": "",
"signature": "sha256:d6276bc28f6e257b87f168f3151ed8e6449f0399528fdde12e7dd4d4f50fccbd"
},
"nbformat": 3
}
Display the source blob
Display the rendered blob
Raw
usgs_pycsw.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment