Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
A Jupyter notebook showing how to use Python to access the NCEI radar archive on Amazon S3 through Unidata's THREDDS server.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@tjlang
Copy link

tjlang commented Oct 28, 2015

I get the following error when trying to execute line 2 (the RadarServer declaration):


KeyError Traceback (most recent call last)
in ()
1 from siphon.radarserver import RadarServer
----> 2 rs = RadarServer('http://thredds-aws.unidata.ucar.edu/thredds/radarServer/nexrad/level2/S3/')

/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/radarserver.pyc in init(self, url)
66 if url.endswith(xmlfile):
67 url = url[:-len(xmlfile)]
---> 68 super(RadarServer, self).init(url)
69
70 def _get_metadata(self):

/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/http_util.pyc in init(self, url)
345 self._base = url
346 self._session = create_http_session()
--> 347 self._get_metadata()
348
349 def get_query(self, query):

/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/radarserver.pyc in _get_metadata(self)
69
70 def _get_metadata(self):
---> 71 ds_cat = TDSCatalog(self.url_path('dataset.xml'))
72 self.metadata = ds_cat.metadata
73 self.variables = set(k.split('/')[0] for k in self.metadata['variables'].keys())

/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/catalog.pyc in init(self, catalog_url)
55
56 # If we were given an HTML link, warn about it and try to fix to xml
---> 57 if 'html' in resp.headers['content-type']:
58 import warnings
59 new_url = self.catalog_url.replace('html', 'xml')

/Users/tjlang/anaconda/lib/python2.7/site-packages/requests/structures.pyc in getitem(self, key)
52
53 def getitem(self, key):
---> 54 return self._store[key.lower()][1]
55
56 def delitem(self, key):

KeyError: 'content-type'

@dopplershift
Copy link
Author

dopplershift commented Oct 29, 2015

Are you, by chance, coming from a non-edu domain? Since the TDS is locked down to .edu only, siphon will only work there?

I need to catch the error and be more clear about what's going on (obviously), but the root cause is that you're getting a 403 (Unauthorized) error.

@drussellwx
Copy link

drussellwx commented Oct 29, 2015

I am having the same 'content-type' error. I am not on an edu domain. Any way to get around this TDS lock down?

@dopplershift
Copy link
Author

dopplershift commented Nov 2, 2015

Sorry, we kinda have to be careful of data charges. One option, if you're looking for data from the last two weeks is to use our server at http://thredds.ucar.edu/thredds/

The other option is that you can browse and downloda the data over http directly from the S3 bucket at https://noaa-nexrad-level2.s3.amazonaws.com/index.html

@dopplershift
Copy link
Author

dopplershift commented Apr 1, 2020

For anyone else encountering this, we have relocated the server to https://tds-nexrad.scigw.unidata.ucar.edu and it's no longer locked down.

@ebdavison
Copy link

ebdavison commented Mar 31, 2022

I am also getting an error trying to run thiswith:
Fedora 35
python 3.10.3
pyproj 3.3.0
cartopy 0.20.1

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb Cell 25' in <module>
      [1](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=0)[ import matplotlib.pyplot as plt
----> ]()[2](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=1)[ import cartopy.crs as ccrs
      ]()[3](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=2)[ import cartopy.feature as cfeature
      ]()[5](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=4)[ def new_map(fig, lon, lat):
      ]()[6](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=5)[     # Create projection centered on the radar. This allows us to use x
      ]()[7](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=6)[     # and y relative to the radar.

File /usr/lib64/python3.10/site-packages/cartopy/__init__.py:107, in <module>
    ]()[102](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=101)[     pass
    ]()[105](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=104)[ # Commonly used sub-modules. Imported here to provide end-user
    ]()[106](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=105)[ # convenience.
--> ]()[107](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=106)[ import cartopy.crs
    ]()[108](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=107)[ import cartopy.feature

File /usr/lib64/python3.10/site-packages/cartopy/crs.py:22, in <module>
     ]()[20](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=19)[ import numpy as np
     ]()[21](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=20)[ import shapely.geometry as sgeom
---> ]()[22](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=21)[ from pyproj import Transformer
     ]()[23](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=22)[ from pyproj.exceptions import ProjError
     ]()[24](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=23)[ from shapely.prepared import prep

File /usr/lib64/python3.10/site-packages/pyproj/__init__.py:57, in <module>
     ]()[50](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=49)[ from pyproj._datadir import (  # noqa: F401 pylint: disable=unused-import
     ]()[51](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=50)[     _pyproj_global_context_initialize,
     ]()[52](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=51)[     set_use_global_context,
     ]()[53](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=52)[ )
     ]()[54](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=53)[ from pyproj._show_versions import (  # noqa: F401 pylint: disable=unused-import
     ]()[55](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=54)[     show_versions,
     ]()[56](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=55)[ )
---> ]()[57](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=56)[ from pyproj.crs import CRS  # noqa: F401 pylint: disable=unused-import
     ]()[58](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=57)[ from pyproj.database import (  # noqa: F401 pylint: disable=unused-import
     ]()[59](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=58)[     get_authorities,
     ]()[60](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=59)[     get_codes,
     ]()[61](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=60)[     get_units_map,
     ]()[62](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=61)[ )
     ]()[63](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=62)[ from pyproj.exceptions import (  # noqa: F401 pylint: disable=unused-import
     ]()[64](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=63)[     DataDirError,
     ]()[65](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=64)[     ProjError,
     ]()[66](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=65)[ )

File /usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py:16, in <module>
      ]()[1](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=0)[ """
      ]()[2](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=1)[ This module interfaces with PROJ to produce a pythonic interface
      ]()[3](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=2)[ to the coordinate reference system (CRS) information through the CRS
      ]()[4](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=3)[ class.
      ]()[5](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=4)[ """
      ]()[7](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=6)[ from pyproj._crs import (  # noqa: F401  pylint: disable=unused-import
      ]()[8](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=7)[     CoordinateOperation,
      ]()[9](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=8)[     CoordinateSystem,
   (...)
     ]()[14](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=13)[     is_wkt,
     ]()[15](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=14)[ )
---> ]()[16](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=15)[ from pyproj.crs.crs import (  # noqa: F401  pylint: disable=unused-import
     ]()[17](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=16)[     CRS,
     ]()[18](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=17)[     BoundCRS,
     ]()[19](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=18)[     CompoundCRS,
     ]()[20](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=19)[     CustomConstructorCRS,
     ]()[21](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=20)[     DerivedGeographicCRS,
     ]()[22](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=21)[     GeocentricCRS,
     ]()[23](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=22)[     GeographicCRS,
     ]()[24](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=23)[     ProjectedCRS,
     ]()[25](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=24)[     VerticalCRS,
     ]()[26](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=25)[ )
     ]()[27](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=26)[ from pyproj.exceptions import CRSError

File /usr/lib64/python3.10/site-packages/pyproj/crs/crs.py:27, in <module>
     ]()[11](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=10)[ from typing import Any, Callable, Dict, List, Optional, Tuple, Union
     ]()[13](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=12)[ from pyproj._crs import (
     ]()[14](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=13)[     _CRS,
     ]()[15](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=14)[     AreaOfUse,
   (...)
     ]()[25](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=24)[     is_wkt,
     ]()[26](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=25)[ )
---> ]()[27](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=26)[ from pyproj.crs._cf1x8 import (
     ]()[28](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=27)[     _GEOGRAPHIC_GRID_MAPPING_NAME_MAP,
     ]()[29](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=28)[     _GRID_MAPPING_NAME_MAP,
     ]()[30](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=29)[     _INVERSE_GEOGRAPHIC_GRID_MAPPING_NAME_MAP,
     ]()[31](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=30)[     _INVERSE_GRID_MAPPING_NAME_MAP,
     ]()[32](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=31)[     _horizontal_datum_from_params,
     ]()[33](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=32)[     _try_list_if_string,
     ]()[34](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=33)[ )
     ]()[35](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=34)[ from pyproj.crs.coordinate_operation import ToWGS84Transformation
     ]()[36](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=35)[ from pyproj.crs.coordinate_system import Cartesian2DCS, Ellipsoidal2DCS, VerticalCS

File /usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py:11, in <module>
      ]()[8](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=7)[ import warnings
     ]()[10](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=9)[ from pyproj._crs import Datum, Ellipsoid, PrimeMeridian
---> ]()[11](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=10)[ from pyproj.crs.coordinate_operation import (
     ]()[12](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=11)[     AlbersEqualAreaConversion,
     ]()[13](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=12)[     AzimuthalEquidistantConversion,
     ]()[14](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=13)[     GeostationarySatelliteConversion,
     ]()[15](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=14)[     HotineObliqueMercatorBConversion,
     ]()[16](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=15)[     LambertAzimuthalEqualAreaConversion,
     ]()[17](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=16)[     LambertConformalConic1SPConversion,
     ]()[18](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=17)[     LambertConformalConic2SPConversion,
     ]()[19](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=18)[     LambertCylindricalEqualAreaConversion,
     ]()[20](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=19)[     LambertCylindricalEqualAreaScaleConversion,
     ]()[21](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=20)[     MercatorAConversion,
     ]()[22](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=21)[     MercatorBConversion,
     ]()[23](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=22)[     OrthographicConversion,
     ]()[24](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=23)[     PolarStereographicAConversion,
     ]()[25](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=24)[     PolarStereographicBConversion,
     ]()[26](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=25)[     PoleRotationNetCDFCFConversion,
     ]()[27](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=26)[     SinusoidalConversion,
     ]()[28](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=27)[     StereographicConversion,
     ]()[29](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=28)[     TransverseMercatorConversion,
     ]()[30](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=29)[     VerticalPerspectiveConversion,
     ]()[31](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=30)[ )
     ]()[32](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=31)[ from pyproj.crs.datum import CustomDatum, CustomEllipsoid, CustomPrimeMeridian
     ]()[33](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=32)[ from pyproj.exceptions import CRSError

File /usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py:12, in <module>
      ]()[9](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=8)[ from typing import Any
     ]()[11](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=10)[ from pyproj._crs import CoordinateOperation
---> ]()[12](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=11)[ from pyproj._transformer import PROJ_VERSION
     ]()[13](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=12)[ from pyproj.exceptions import CRSError
     ]()[16](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=15)[ class AlbersEqualAreaConversion(CoordinateOperation):

ImportError: /usr/lib64/python3.10/site-packages/pyproj/_transformer.cpython-310-x86_64-linux-gnu.so: undefined symbol: proj_trans_bounds]()

@dopplershift
Copy link
Author

dopplershift commented Apr 1, 2022

That looks like a conflict between the Fedora packages for pyproj and PROJ. Given that it's an ImportError when running import cartopy.crs as ccrs, it's definitely not a problem with the code on this gist. I'd recommend raising an issue on the Fedora issue tracker.

@ebdavison
Copy link

ebdavison commented Apr 6, 2022

Understood. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment