Skip to content

Instantly share code, notes, and snippets.

@dopplershift
Last active April 26, 2024 20:26
Show Gist options
  • Save dopplershift/356f2e14832e9b676207 to your computer and use it in GitHub Desktop.
Save dopplershift/356f2e14832e9b676207 to your computer and use it in GitHub Desktop.
A Jupyter notebook showing how to use Python to access the NCEI radar archive on Amazon S3 through Unidata's THREDDS server.
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@tjlang
Copy link

tjlang commented Oct 28, 2015

I get the following error when trying to execute line 2 (the RadarServer declaration):


KeyError Traceback (most recent call last)
in ()
1 from siphon.radarserver import RadarServer
----> 2 rs = RadarServer('http://thredds-aws.unidata.ucar.edu/thredds/radarServer/nexrad/level2/S3/')

/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/radarserver.pyc in init(self, url)
66 if url.endswith(xmlfile):
67 url = url[:-len(xmlfile)]
---> 68 super(RadarServer, self).init(url)
69
70 def _get_metadata(self):

/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/http_util.pyc in init(self, url)
345 self._base = url
346 self._session = create_http_session()
--> 347 self._get_metadata()
348
349 def get_query(self, query):

/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/radarserver.pyc in _get_metadata(self)
69
70 def _get_metadata(self):
---> 71 ds_cat = TDSCatalog(self.url_path('dataset.xml'))
72 self.metadata = ds_cat.metadata
73 self.variables = set(k.split('/')[0] for k in self.metadata['variables'].keys())

/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/catalog.pyc in init(self, catalog_url)
55
56 # If we were given an HTML link, warn about it and try to fix to xml
---> 57 if 'html' in resp.headers['content-type']:
58 import warnings
59 new_url = self.catalog_url.replace('html', 'xml')

/Users/tjlang/anaconda/lib/python2.7/site-packages/requests/structures.pyc in getitem(self, key)
52
53 def getitem(self, key):
---> 54 return self._store[key.lower()][1]
55
56 def delitem(self, key):

KeyError: 'content-type'

@dopplershift
Copy link
Author

Are you, by chance, coming from a non-edu domain? Since the TDS is locked down to .edu only, siphon will only work there?

I need to catch the error and be more clear about what's going on (obviously), but the root cause is that you're getting a 403 (Unauthorized) error.

@drussellwx
Copy link

I am having the same 'content-type' error. I am not on an edu domain. Any way to get around this TDS lock down?

@dopplershift
Copy link
Author

Sorry, we kinda have to be careful of data charges. One option, if you're looking for data from the last two weeks is to use our server at http://thredds.ucar.edu/thredds/

The other option is that you can browse and downloda the data over http directly from the S3 bucket at https://noaa-nexrad-level2.s3.amazonaws.com/index.html

@dopplershift
Copy link
Author

For anyone else encountering this, we have relocated the server to https://tds-nexrad.scigw.unidata.ucar.edu and it's no longer locked down.

@ebdavison
Copy link

ebdavison commented Mar 31, 2022

I am also getting an error trying to run thiswith:
Fedora 35
python 3.10.3
pyproj 3.3.0
cartopy 0.20.1

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb Cell 25' in <module>
      [1](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=0)[ import matplotlib.pyplot as plt
----> ]()[2](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=1)[ import cartopy.crs as ccrs
      ]()[3](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=2)[ import cartopy.feature as cfeature
      ]()[5](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=4)[ def new_map(fig, lon, lat):
      ]()[6](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=5)[     # Create projection centered on the radar. This allows us to use x
      ]()[7](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=6)[     # and y relative to the radar.

File /usr/lib64/python3.10/site-packages/cartopy/__init__.py:107, in <module>
    ]()[102](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=101)[     pass
    ]()[105](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=104)[ # Commonly used sub-modules. Imported here to provide end-user
    ]()[106](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=105)[ # convenience.
--> ]()[107](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=106)[ import cartopy.crs
    ]()[108](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=107)[ import cartopy.feature

File /usr/lib64/python3.10/site-packages/cartopy/crs.py:22, in <module>
     ]()[20](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=19)[ import numpy as np
     ]()[21](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=20)[ import shapely.geometry as sgeom
---> ]()[22](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=21)[ from pyproj import Transformer
     ]()[23](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=22)[ from pyproj.exceptions import ProjError
     ]()[24](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=23)[ from shapely.prepared import prep

File /usr/lib64/python3.10/site-packages/pyproj/__init__.py:57, in <module>
     ]()[50](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=49)[ from pyproj._datadir import (  # noqa: F401 pylint: disable=unused-import
     ]()[51](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=50)[     _pyproj_global_context_initialize,
     ]()[52](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=51)[     set_use_global_context,
     ]()[53](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=52)[ )
     ]()[54](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=53)[ from pyproj._show_versions import (  # noqa: F401 pylint: disable=unused-import
     ]()[55](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=54)[     show_versions,
     ]()[56](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=55)[ )
---> ]()[57](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=56)[ from pyproj.crs import CRS  # noqa: F401 pylint: disable=unused-import
     ]()[58](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=57)[ from pyproj.database import (  # noqa: F401 pylint: disable=unused-import
     ]()[59](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=58)[     get_authorities,
     ]()[60](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=59)[     get_codes,
     ]()[61](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=60)[     get_units_map,
     ]()[62](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=61)[ )
     ]()[63](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=62)[ from pyproj.exceptions import (  # noqa: F401 pylint: disable=unused-import
     ]()[64](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=63)[     DataDirError,
     ]()[65](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=64)[     ProjError,
     ]()[66](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=65)[ )

File /usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py:16, in <module>
      ]()[1](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=0)[ """
      ]()[2](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=1)[ This module interfaces with PROJ to produce a pythonic interface
      ]()[3](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=2)[ to the coordinate reference system (CRS) information through the CRS
      ]()[4](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=3)[ class.
      ]()[5](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=4)[ """
      ]()[7](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=6)[ from pyproj._crs import (  # noqa: F401  pylint: disable=unused-import
      ]()[8](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=7)[     CoordinateOperation,
      ]()[9](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=8)[     CoordinateSystem,
   (...)
     ]()[14](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=13)[     is_wkt,
     ]()[15](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=14)[ )
---> ]()[16](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=15)[ from pyproj.crs.crs import (  # noqa: F401  pylint: disable=unused-import
     ]()[17](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=16)[     CRS,
     ]()[18](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=17)[     BoundCRS,
     ]()[19](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=18)[     CompoundCRS,
     ]()[20](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=19)[     CustomConstructorCRS,
     ]()[21](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=20)[     DerivedGeographicCRS,
     ]()[22](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=21)[     GeocentricCRS,
     ]()[23](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=22)[     GeographicCRS,
     ]()[24](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=23)[     ProjectedCRS,
     ]()[25](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=24)[     VerticalCRS,
     ]()[26](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=25)[ )
     ]()[27](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=26)[ from pyproj.exceptions import CRSError

File /usr/lib64/python3.10/site-packages/pyproj/crs/crs.py:27, in <module>
     ]()[11](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=10)[ from typing import Any, Callable, Dict, List, Optional, Tuple, Union
     ]()[13](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=12)[ from pyproj._crs import (
     ]()[14](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=13)[     _CRS,
     ]()[15](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=14)[     AreaOfUse,
   (...)
     ]()[25](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=24)[     is_wkt,
     ]()[26](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=25)[ )
---> ]()[27](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=26)[ from pyproj.crs._cf1x8 import (
     ]()[28](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=27)[     _GEOGRAPHIC_GRID_MAPPING_NAME_MAP,
     ]()[29](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=28)[     _GRID_MAPPING_NAME_MAP,
     ]()[30](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=29)[     _INVERSE_GEOGRAPHIC_GRID_MAPPING_NAME_MAP,
     ]()[31](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=30)[     _INVERSE_GRID_MAPPING_NAME_MAP,
     ]()[32](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=31)[     _horizontal_datum_from_params,
     ]()[33](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=32)[     _try_list_if_string,
     ]()[34](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=33)[ )
     ]()[35](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=34)[ from pyproj.crs.coordinate_operation import ToWGS84Transformation
     ]()[36](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=35)[ from pyproj.crs.coordinate_system import Cartesian2DCS, Ellipsoidal2DCS, VerticalCS

File /usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py:11, in <module>
      ]()[8](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=7)[ import warnings
     ]()[10](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=9)[ from pyproj._crs import Datum, Ellipsoid, PrimeMeridian
---> ]()[11](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=10)[ from pyproj.crs.coordinate_operation import (
     ]()[12](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=11)[     AlbersEqualAreaConversion,
     ]()[13](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=12)[     AzimuthalEquidistantConversion,
     ]()[14](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=13)[     GeostationarySatelliteConversion,
     ]()[15](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=14)[     HotineObliqueMercatorBConversion,
     ]()[16](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=15)[     LambertAzimuthalEqualAreaConversion,
     ]()[17](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=16)[     LambertConformalConic1SPConversion,
     ]()[18](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=17)[     LambertConformalConic2SPConversion,
     ]()[19](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=18)[     LambertCylindricalEqualAreaConversion,
     ]()[20](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=19)[     LambertCylindricalEqualAreaScaleConversion,
     ]()[21](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=20)[     MercatorAConversion,
     ]()[22](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=21)[     MercatorBConversion,
     ]()[23](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=22)[     OrthographicConversion,
     ]()[24](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=23)[     PolarStereographicAConversion,
     ]()[25](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=24)[     PolarStereographicBConversion,
     ]()[26](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=25)[     PoleRotationNetCDFCFConversion,
     ]()[27](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=26)[     SinusoidalConversion,
     ]()[28](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=27)[     StereographicConversion,
     ]()[29](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=28)[     TransverseMercatorConversion,
     ]()[30](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=29)[     VerticalPerspectiveConversion,
     ]()[31](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=30)[ )
     ]()[32](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=31)[ from pyproj.crs.datum import CustomDatum, CustomEllipsoid, CustomPrimeMeridian
     ]()[33](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=32)[ from pyproj.exceptions import CRSError

File /usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py:12, in <module>
      ]()[9](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=8)[ from typing import Any
     ]()[11](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=10)[ from pyproj._crs import CoordinateOperation
---> ]()[12](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=11)[ from pyproj._transformer import PROJ_VERSION
     ]()[13](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=12)[ from pyproj.exceptions import CRSError
     ]()[16](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=15)[ class AlbersEqualAreaConversion(CoordinateOperation):

ImportError: /usr/lib64/python3.10/site-packages/pyproj/_transformer.cpython-310-x86_64-linux-gnu.so: undefined symbol: proj_trans_bounds]()

@dopplershift
Copy link
Author

That looks like a conflict between the Fedora packages for pyproj and PROJ. Given that it's an ImportError when running import cartopy.crs as ccrs, it's definitely not a problem with the code on this gist. I'd recommend raising an issue on the Fedora issue tracker.

@ebdavison
Copy link

Understood. Thanks.

@btang8
Copy link

btang8 commented Apr 1, 2024

Following the example script exactly, I am encountering an error where catalog.datasets returns an empty list for either real-time or archived queries. Perhaps just a temporary hiccup with the radar server or some change with the backend?

@jrudzin
Copy link

jrudzin commented Apr 18, 2024

Having the same issue as btang8 - receiving an empty list even though the data is there

@dopplershift
Copy link
Author

I've been digging in, but haven't had any luck so far figuring out what's wrong. I'll keep digging, though.

@btang8
Copy link

btang8 commented Apr 26, 2024

FYI, Kevin Tyle shared a sample script with me that accesses radar data in a different way, in case others are looking for a workaround: https://www.atmos.albany.edu/facstaff/ktyle/atm350_2023/core/week12/radar_AWS.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment