Last active
April 26, 2024 20:26
-
-
Save dopplershift/356f2e14832e9b676207 to your computer and use it in GitHub Desktop.
A Jupyter notebook showing how to use Python to access the NCEI radar archive on Amazon S3 through Unidata's THREDDS server.
Are you, by chance, coming from a non-edu domain? Since the TDS is locked down to .edu only, siphon will only work there?
I need to catch the error and be more clear about what's going on (obviously), but the root cause is that you're getting a 403 (Unauthorized) error.
I am having the same 'content-type' error. I am not on an edu domain. Any way to get around this TDS lock down?
Sorry, we kinda have to be careful of data charges. One option, if you're looking for data from the last two weeks is to use our server at http://thredds.ucar.edu/thredds/
The other option is that you can browse and downloda the data over http directly from the S3 bucket at https://noaa-nexrad-level2.s3.amazonaws.com/index.html
For anyone else encountering this, we have relocated the server to https://tds-nexrad.scigw.unidata.ucar.edu and it's no longer locked down.
I am also getting an error trying to run thiswith:
Fedora 35
python 3.10.3
pyproj 3.3.0
cartopy 0.20.1
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb Cell 25' in <module>
[1](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=0)[ import matplotlib.pyplot as plt
----> ]()[2](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=1)[ import cartopy.crs as ccrs
]()[3](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=2)[ import cartopy.feature as cfeature
]()[5](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=4)[ def new_map(fig, lon, lat):
]()[6](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=5)[ # Create projection centered on the radar. This allows us to use x
]()[7](vscode-notebook-cell:/opt/Data/Personal/NextCloud/Documents/IT/GIS/python_to_access_nexrad.ipynb#ch0000024?line=6)[ # and y relative to the radar.
File /usr/lib64/python3.10/site-packages/cartopy/__init__.py:107, in <module>
]()[102](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=101)[ pass
]()[105](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=104)[ # Commonly used sub-modules. Imported here to provide end-user
]()[106](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=105)[ # convenience.
--> ]()[107](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=106)[ import cartopy.crs
]()[108](file:///usr/lib64/python3.10/site-packages/cartopy/__init__.py?line=107)[ import cartopy.feature
File /usr/lib64/python3.10/site-packages/cartopy/crs.py:22, in <module>
]()[20](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=19)[ import numpy as np
]()[21](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=20)[ import shapely.geometry as sgeom
---> ]()[22](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=21)[ from pyproj import Transformer
]()[23](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=22)[ from pyproj.exceptions import ProjError
]()[24](file:///usr/lib64/python3.10/site-packages/cartopy/crs.py?line=23)[ from shapely.prepared import prep
File /usr/lib64/python3.10/site-packages/pyproj/__init__.py:57, in <module>
]()[50](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=49)[ from pyproj._datadir import ( # noqa: F401 pylint: disable=unused-import
]()[51](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=50)[ _pyproj_global_context_initialize,
]()[52](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=51)[ set_use_global_context,
]()[53](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=52)[ )
]()[54](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=53)[ from pyproj._show_versions import ( # noqa: F401 pylint: disable=unused-import
]()[55](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=54)[ show_versions,
]()[56](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=55)[ )
---> ]()[57](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=56)[ from pyproj.crs import CRS # noqa: F401 pylint: disable=unused-import
]()[58](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=57)[ from pyproj.database import ( # noqa: F401 pylint: disable=unused-import
]()[59](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=58)[ get_authorities,
]()[60](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=59)[ get_codes,
]()[61](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=60)[ get_units_map,
]()[62](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=61)[ )
]()[63](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=62)[ from pyproj.exceptions import ( # noqa: F401 pylint: disable=unused-import
]()[64](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=63)[ DataDirError,
]()[65](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=64)[ ProjError,
]()[66](file:///usr/lib64/python3.10/site-packages/pyproj/__init__.py?line=65)[ )
File /usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py:16, in <module>
]()[1](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=0)[ """
]()[2](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=1)[ This module interfaces with PROJ to produce a pythonic interface
]()[3](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=2)[ to the coordinate reference system (CRS) information through the CRS
]()[4](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=3)[ class.
]()[5](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=4)[ """
]()[7](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=6)[ from pyproj._crs import ( # noqa: F401 pylint: disable=unused-import
]()[8](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=7)[ CoordinateOperation,
]()[9](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=8)[ CoordinateSystem,
(...)
]()[14](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=13)[ is_wkt,
]()[15](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=14)[ )
---> ]()[16](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=15)[ from pyproj.crs.crs import ( # noqa: F401 pylint: disable=unused-import
]()[17](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=16)[ CRS,
]()[18](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=17)[ BoundCRS,
]()[19](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=18)[ CompoundCRS,
]()[20](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=19)[ CustomConstructorCRS,
]()[21](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=20)[ DerivedGeographicCRS,
]()[22](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=21)[ GeocentricCRS,
]()[23](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=22)[ GeographicCRS,
]()[24](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=23)[ ProjectedCRS,
]()[25](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=24)[ VerticalCRS,
]()[26](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=25)[ )
]()[27](file:///usr/lib64/python3.10/site-packages/pyproj/crs/__init__.py?line=26)[ from pyproj.exceptions import CRSError
File /usr/lib64/python3.10/site-packages/pyproj/crs/crs.py:27, in <module>
]()[11](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=10)[ from typing import Any, Callable, Dict, List, Optional, Tuple, Union
]()[13](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=12)[ from pyproj._crs import (
]()[14](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=13)[ _CRS,
]()[15](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=14)[ AreaOfUse,
(...)
]()[25](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=24)[ is_wkt,
]()[26](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=25)[ )
---> ]()[27](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=26)[ from pyproj.crs._cf1x8 import (
]()[28](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=27)[ _GEOGRAPHIC_GRID_MAPPING_NAME_MAP,
]()[29](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=28)[ _GRID_MAPPING_NAME_MAP,
]()[30](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=29)[ _INVERSE_GEOGRAPHIC_GRID_MAPPING_NAME_MAP,
]()[31](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=30)[ _INVERSE_GRID_MAPPING_NAME_MAP,
]()[32](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=31)[ _horizontal_datum_from_params,
]()[33](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=32)[ _try_list_if_string,
]()[34](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=33)[ )
]()[35](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=34)[ from pyproj.crs.coordinate_operation import ToWGS84Transformation
]()[36](file:///usr/lib64/python3.10/site-packages/pyproj/crs/crs.py?line=35)[ from pyproj.crs.coordinate_system import Cartesian2DCS, Ellipsoidal2DCS, VerticalCS
File /usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py:11, in <module>
]()[8](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=7)[ import warnings
]()[10](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=9)[ from pyproj._crs import Datum, Ellipsoid, PrimeMeridian
---> ]()[11](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=10)[ from pyproj.crs.coordinate_operation import (
]()[12](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=11)[ AlbersEqualAreaConversion,
]()[13](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=12)[ AzimuthalEquidistantConversion,
]()[14](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=13)[ GeostationarySatelliteConversion,
]()[15](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=14)[ HotineObliqueMercatorBConversion,
]()[16](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=15)[ LambertAzimuthalEqualAreaConversion,
]()[17](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=16)[ LambertConformalConic1SPConversion,
]()[18](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=17)[ LambertConformalConic2SPConversion,
]()[19](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=18)[ LambertCylindricalEqualAreaConversion,
]()[20](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=19)[ LambertCylindricalEqualAreaScaleConversion,
]()[21](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=20)[ MercatorAConversion,
]()[22](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=21)[ MercatorBConversion,
]()[23](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=22)[ OrthographicConversion,
]()[24](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=23)[ PolarStereographicAConversion,
]()[25](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=24)[ PolarStereographicBConversion,
]()[26](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=25)[ PoleRotationNetCDFCFConversion,
]()[27](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=26)[ SinusoidalConversion,
]()[28](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=27)[ StereographicConversion,
]()[29](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=28)[ TransverseMercatorConversion,
]()[30](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=29)[ VerticalPerspectiveConversion,
]()[31](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=30)[ )
]()[32](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=31)[ from pyproj.crs.datum import CustomDatum, CustomEllipsoid, CustomPrimeMeridian
]()[33](file:///usr/lib64/python3.10/site-packages/pyproj/crs/_cf1x8.py?line=32)[ from pyproj.exceptions import CRSError
File /usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py:12, in <module>
]()[9](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=8)[ from typing import Any
]()[11](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=10)[ from pyproj._crs import CoordinateOperation
---> ]()[12](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=11)[ from pyproj._transformer import PROJ_VERSION
]()[13](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=12)[ from pyproj.exceptions import CRSError
]()[16](file:///usr/lib64/python3.10/site-packages/pyproj/crs/coordinate_operation.py?line=15)[ class AlbersEqualAreaConversion(CoordinateOperation):
ImportError: /usr/lib64/python3.10/site-packages/pyproj/_transformer.cpython-310-x86_64-linux-gnu.so: undefined symbol: proj_trans_bounds]()
That looks like a conflict between the Fedora packages for pyproj and PROJ. Given that it's an ImportError when running import cartopy.crs as ccrs, it's definitely not a problem with the code on this gist. I'd recommend raising an issue on the Fedora issue tracker.
Understood. Thanks.
Following the example script exactly, I am encountering an error where catalog.datasets returns an empty list for either real-time or archived queries. Perhaps just a temporary hiccup with the radar server or some change with the backend?
Having the same issue as btang8 - receiving an empty list even though the data is there
I've been digging in, but haven't had any luck so far figuring out what's wrong. I'll keep digging, though.
FYI, Kevin Tyle shared a sample script with me that accesses radar data in a different way, in case others are looking for a workaround: https://www.atmos.albany.edu/facstaff/ktyle/atm350_2023/core/week12/radar_AWS.html
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I get the following error when trying to execute line 2 (the RadarServer declaration):
KeyError Traceback (most recent call last)
in ()
1 from siphon.radarserver import RadarServer
----> 2 rs = RadarServer('http://thredds-aws.unidata.ucar.edu/thredds/radarServer/nexrad/level2/S3/')
/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/radarserver.pyc in init(self, url)
66 if url.endswith(xmlfile):
67 url = url[:-len(xmlfile)]
---> 68 super(RadarServer, self).init(url)
69
70 def _get_metadata(self):
/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/http_util.pyc in init(self, url)
345 self._base = url
346 self._session = create_http_session()
--> 347 self._get_metadata()
348
349 def get_query(self, query):
/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/radarserver.pyc in _get_metadata(self)
69
70 def _get_metadata(self):
---> 71 ds_cat = TDSCatalog(self.url_path('dataset.xml'))
72 self.metadata = ds_cat.metadata
73 self.variables = set(k.split('/')[0] for k in self.metadata['variables'].keys())
/Users/tjlang/anaconda/lib/python2.7/site-packages/siphon/catalog.pyc in init(self, catalog_url)
55
56 # If we were given an HTML link, warn about it and try to fix to xml
---> 57 if 'html' in resp.headers['content-type']:
58 import warnings
59 new_url = self.catalog_url.replace('html', 'xml')
/Users/tjlang/anaconda/lib/python2.7/site-packages/requests/structures.pyc in getitem(self, key)
52
53 def getitem(self, key):
---> 54 return self._store[key.lower()][1]
55
56 def delitem(self, key):
KeyError: 'content-type'