Skip to content

Instantly share code, notes, and snippets.

@timtroendle

timtroendle/test-atlite-numerics.ipynb Secret

Last active May 15, 2019 06:15
Show Gist options
  • Save timtroendle/832ea97fc3f594560ce7dcaf95af4fe5 to your computer and use it in GitHub Desktop.
Save timtroendle/832ea97fc3f594560ce7dcaf95af4fe5 to your computer and use it in GitHub Desktop.
Download ZIP
test-atlite-numerics
Display the source blob
Display the rendered blob
Raw
test-atlite-numerics.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Numerics of ERA5 downloads with atlite\n",
"\n",
"There seem to be tiny numerical issues with ERA5 data downloaded using atlite. I show that by downloading the same data through atlite and directly from the CDSAPI and comparing the results."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Download"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"import atlite\n",
"import xarray as xr"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2019-05-15 07:30:19,009 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels\n",
"2019-05-15 07:30:19,476 INFO Request is queued\n",
"2019-05-15 07:31:35,465 INFO Request is running\n",
"2019-05-15 07:32:14,006 INFO Request is completed\n",
"2019-05-15 07:32:14,011 INFO Downloading request for 9 variables to /var/folders/77/hl9n2z_n6059xs8ml53n_tsr0000gp/T/tmpdr8e_fys.nc\n",
"2019-05-15 07:32:14,012 INFO Downloading http://136.156.132.201/cache-compute-0004/cache/data3/adaptor.mars.internal-1557898292.3789797-339-11-fe60ccbf-7dcc-42a4-b2c2-47f3d9d0b7d8.nc to /var/folders/77/hl9n2z_n6059xs8ml53n_tsr0000gp/T/tmpdr8e_fys.nc (1.2M)\n",
"2019-05-15 07:32:15,150 INFO Download rate 1.1M/s\n",
"2019-05-15 07:32:15,332 INFO Starting preparation of cutout 'test-numerics'\n",
"2019-05-15 07:32:15,370 INFO 1 tasks have been collected. Starting running them on all processors.\n",
"2019-05-15 07:32:15,406 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels\n",
"2019-05-15 07:32:15,871 INFO Request is queued\n",
"2019-05-15 07:34:10,481 INFO Request is running\n",
"2019-05-15 07:48:37,503 INFO Request is completed\n",
"2019-05-15 07:48:37,509 INFO Downloading request for 10 variables to /var/folders/77/hl9n2z_n6059xs8ml53n_tsr0000gp/T/tmpcgyapk1g.nc\n",
"2019-05-15 07:48:37,512 INFO Downloading http://136.156.132.105/cache-compute-0000/cache/data3/adaptor.mars.internal-1557898463.6191638-31309-5-ae8de4fe-fd03-41db-a993-8d075e41ffa3.nc to /var/folders/77/hl9n2z_n6059xs8ml53n_tsr0000gp/T/tmpcgyapk1g.nc (377.6M)\n",
"2019-05-15 08:00:52,833 INFO Download rate 525.8K/s\n",
"2019-05-15 08:00:52,851 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels\n",
"2019-05-15 08:00:53,221 INFO Request is queued\n",
"2019-05-15 08:02:47,820 INFO Request is running\n",
"2019-05-15 08:03:45,598 INFO Request is completed\n",
"2019-05-15 08:03:45,601 INFO Downloading request for 2 variables to /var/folders/77/hl9n2z_n6059xs8ml53n_tsr0000gp/T/tmpszimr8x1.nc\n",
"2019-05-15 08:03:45,604 INFO Downloading http://136.156.132.235/cache-compute-0006/cache/data2/adaptor.mars.internal-1557900172.6555233-26809-9-c09d8678-9669-4362-aefa-297e2a3fd273.nc to /var/folders/77/hl9n2z_n6059xs8ml53n_tsr0000gp/T/tmpszimr8x1.nc (2.4M)\n",
"2019-05-15 08:03:47,136 INFO Download rate 1.6M/s\n",
"2019-05-15 08:03:51,023 INFO Merging variables into monthly compound files\n",
"2019-05-15 08:03:51,026 INFO Cutout 'test-numerics' has been successfully prepared\n"
]
}
],
"source": [
"X_RANGE = slice(-10.7, 34.6)\n",
"Y_RANGE = slice(71.2, 34.5)\n",
"\n",
"cutout = atlite.Cutout(\n",
" name=\"test-numerics\",\n",
" module=\"era5\",\n",
" cutout_dir=\"atlite-cutouts/\",\n",
" xs=X_RANGE,\n",
" ys=Y_RANGE,\n",
" years=slice(2016, 2016),\n",
" months=slice(1, 1)\n",
")\n",
"cutout.prepare()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"atlite_download = xr.open_dataset(\"atlite-cutouts/test-numerics/201601.nc\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2019-05-15 08:04:40,741 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels\n",
"2019-05-15 08:04:41,255 INFO Request is completed\n",
"2019-05-15 08:04:41,256 INFO Downloading http://136.156.132.201/cache-compute-0004/cache/data1/adaptor.mars.internal-1557896528.9115837-23654-9-73fc8db6-ebd5-44fa-b38f-54c08fcffd16.nc to 201601.nc (1.2M)\n",
"2019-05-15 08:04:45,090 INFO Download rate 326K/s\n"
]
},
{
"data": {
"text/plain": [
"'201601.nc'"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import cdsapi\n",
"\n",
"product='reanalysis-era5-single-levels'\n",
"request = {\n",
" 'product_type': 'reanalysis', \n",
" 'format': 'netcdf', \n",
" 'day': 1, \n",
" 'time': ['00:00', '01:00', '02:00', '03:00', '04:00', '05:00', '06:00', '07:00', '08:00', '09:00', '10:00', '11:00', '12:00', '13:00', '14:00', '15:00', '16:00', '17:00', '18:00', '19:00', '20:00', '21:00', '22:00', '23:00'], \n",
" 'variable': 'runoff', \n",
" 'year': 2016, \n",
" 'month': 1, \n",
" 'area': [71.2, -10.7, 34.5, 34.6]\n",
"}\n",
"result = cdsapi.Client().retrieve(\n",
" product,\n",
" request\n",
")\n",
"result.download(\"201601.nc\")"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"direct_download = xr.open_dataset(\"201601.nc\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Compare"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"count 6.385680e+05\n",
"mean 1.552314e-05\n",
"std 5.738868e-05\n",
"min -4.656613e-10\n",
"25% -4.656613e-10\n",
"50% 3.972091e-07\n",
"75% 8.748844e-06\n",
"max 6.215543e-03\n",
"Name: runoff, dtype: float64"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"atlite_download.runoff.sel(time=\"2016-01-01\").to_series().describe()"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"count 6.385680e+05\n",
"mean 1.552658e-05\n",
"std 5.738928e-05\n",
"min 0.000000e+00\n",
"25% 0.000000e+00\n",
"50% 4.742760e-07\n",
"75% 8.820556e-06\n",
"max 6.215572e-03\n",
"Name: ro, dtype: float64"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"direct_download.ro.to_series().describe()"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<xarray.DataArray 'runoff' (time: 24)>\n",
"array([-4.656613e-10, -4.656613e-10, -4.656613e-10, -4.656613e-10,\n",
" -4.656613e-10, -4.656613e-10, -4.656613e-10, -4.656613e-10,\n",
" -4.656613e-10, -4.656613e-10, -4.656613e-10, -4.656613e-10,\n",
" -4.656613e-10, -4.656613e-10, -4.656613e-10, -4.656613e-10,\n",
" -4.656613e-10, -4.656613e-10, -4.656613e-10, -4.656613e-10,\n",
" -4.656613e-10, -4.656613e-10, -4.656613e-10, -4.656613e-10],\n",
" dtype=float32)\n",
"Coordinates:\n",
" x float32 -10.5\n",
" y float32 71.0\n",
" * time (time) datetime64[ns] 2016-01-01 ... 2016-01-01T23:00:00\n",
" lat float32 ...\n",
" lon float32 ...\n",
"Attributes:\n",
" units: m\n",
" long_name: Runoff"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"atlite_download.runoff.sel(time=\"2016-01-01\").isel(x=0, y=0)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<xarray.DataArray 'ro' (time: 24)>\n",
"array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
" 0., 0., 0., 0., 0., 0.], dtype=float32)\n",
"Coordinates:\n",
" longitude float32 -10.5\n",
" latitude float32 71.0\n",
" * time (time) datetime64[ns] 2016-01-01 ... 2016-01-01T23:00:00\n",
"Attributes:\n",
" units: m\n",
" long_name: Runoff"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"direct_download.ro.isel(latitude=0, longitude=0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are negative values in the atlite data that aren't in the raw ERA5 data. Also, the mean, median, and max are slightly different. Of course, those difference likely stem from numerical issues. But it isn't clear to me, where they are introduced and why. Because everything is supposed to stay as raw float32, there shouldn't be any numerical issues. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment