Skip to content

Instantly share code, notes, and snippets.

@stefsmeets

stefsmeets/Run recipe.ipynb

Last active January 6, 2021 11:32
Show Gist options
  • Save stefsmeets/1163eeea765906550f772c15b69e866e to your computer and use it in GitHub Desktop.
Save stefsmeets/1163eeea765906550f772c15b69e866e to your computer and use it in GitHub Desktop.
Download ZIP
run recipe in a notebook
Display the source blob
Display the rendered blob
Raw
Run recipe.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Running a recipe in a notebook\n",{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Running a recipe in a notebook\n",
"\n",
"The goal of this notebook is to demonstrate and test the functionality to load and run recipes directly from the notebook. See associated PR here: https://github.com/ESMValGroup/ESMValCore/pull/907"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Getting a recipe\n",
"\n",
"Recipes can be obtained by their name, similar to what one may already be familiar with through `esmvaltool recipes list`. To do so, use `get_recipe`."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/stef/r/esmvalcore/esmvalcore/experimental/_warnings.py:18: UserWarning: \n",
" Thank you for trying out the new ESMValCore API.\n",
" Note that this API is experimental and may be subject to change.\n",
" More info: https://github.com/ESMValGroup/ESMValCore/issues/498\n",
"/home/stef/r/esmvalcore/esmvalcore/experimental/config/_config_validators.py:254: ESMValToolDeprecationWarning: `write_plots` will be removed in 2.2.0.\n",
"/home/stef/r/esmvalcore/esmvalcore/experimental/config/_config_validators.py:255: ESMValToolDeprecationWarning: `write_netcdf` will be removed in 2.2.0.\n",
"/home/stef/r/esmvalcore/esmvalcore/experimental/config/_config_validators.py:256: ESMValToolDeprecationWarning: `output_file_type` will be removed in 2.2.0.\n"
]
},
{
"data": {
"text/markdown": [
"## Recipe python\n",
"\n",
"Example recipe that plots a map and timeseries of temperature.\n",
"\n",
"### Authors\n",
" - Bouwe Andela (NLeSC, Netherlands; https://orcid.org/0000-0001-9005-8940)\n",
" - Mattia Righi (DLR, Germany; https://orcid.org/0000-0003-3827-5950)\n",
"\n",
"### Maintainers\n",
" - Manuel Schlund (DLR, Germany; https://orcid.org/0000-0001-5251-0158)\n",
"\n",
"### Projects\n",
" - DLR project ESMVal\n",
" - Copernicus Climate Change Service 34a Lot 2 (MAGIC) project\n",
"\n",
"### References\n",
" - Please acknowledge the project\\(s\\)\\.\n"
],
"text/plain": [
"RecipeInfo('Recipe python')"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from esmvalcore.experimental import get_recipe\n",
"\n",
"recipe = get_recipe('examples/recipe_python.yml')\n",
"recipe"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Running a recipe\n",
"\n",
"Run the recipe by using the `.run` method. \n",
"\n",
"The session can be defined as an arguement, but if left out, a new session is created automatically.\n",
"\n",
"Under the hood it calls `esmvalcore._recipe.Recipe` with the session config. The logger is redirected automatically to `main_log_debug.txt` / `main_log.txt` in the appropriate directory."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'auxiliary_data_dir': PosixPath('/home/stef/auxiliary_data'),\n",
" 'check_level': <CheckLevels.DEFAULT: 3>,\n",
" 'compress_netcdf': False,\n",
" 'config_developer_file': None,\n",
" 'config_file': PosixPath('/home/stef/.esmvaltool/config-user.yml'),\n",
" 'diagnostics': set(),\n",
" 'drs': {'CMIP5': 'default', 'CMIP6': 'default'},\n",
" 'exit_on_warning': False,\n",
" 'log_level': 'info',\n",
" 'max_datasets': None,\n",
" 'max_parallel_tasks': None,\n",
" 'max_years': None,\n",
" 'output_dir': PosixPath('/home/stef/esmvaltool_output'),\n",
" 'output_file_type': 'png',\n",
" 'plot_dir': PosixPath('/home/stef/esmvaltool_output/recipe_python_20210106_112622/plots'),\n",
" 'preproc_dir': PosixPath('/home/stef/esmvaltool_output/recipe_python_20210106_112622/preproc'),\n",
" 'profile_diagnostic': False,\n",
" 'remove_preproc_dir': True,\n",
" 'rootpath': {'CMIP5': [PosixPath('/home/stef/data')],\n",
" 'CMIP6': [PosixPath('/home/stef/data')],\n",
" 'default': [PosixPath('/home/stef/data')]},\n",
" 'run_diagnostic': True,\n",
" 'run_dir': PosixPath('/home/stef/esmvaltool_output/recipe_python_20210106_112622/run'),\n",
" 'save_intermediary_cubes': False,\n",
" 'skip-nonexistent': False,\n",
" 'synda_download': False,\n",
" 'work_dir': PosixPath('/home/stef/esmvaltool_output/recipe_python_20210106_112622/work'),\n",
" 'write_netcdf': True,\n",
" 'write_plots': True}\n",
"Creating tasks from recipe\n",
"Creating tasks for diagnostic map\n",
"Creating preprocessor task map/tas\n",
"Creating preprocessor 'select_january' task for variable 'tas'\n",
"Using input files for variable tas of dataset BCC-ESM1:\n",
"/home/stef/data/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc\n",
"Using input files for variable tas of dataset CanESM2:\n",
"/home/stef/data/tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc\n",
"PreprocessingTask map/tas created. It will create the files:\n",
"/home/stef/esmvaltool_output/recipe_python_20210106_112622/preproc/map/tas/CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.nc\n",
"/home/stef/esmvaltool_output/recipe_python_20210106_112622/preproc/map/tas/CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.nc\n",
"Creating diagnostic task map/script1\n",
"Creating tasks for diagnostic timeseries\n",
"Creating preprocessor task timeseries/tas_amsterdam\n",
"Creating preprocessor 'annual_mean_amsterdam' task for variable 'tas'\n",
"Using input files for variable tas of dataset BCC-ESM1:\n",
"/home/stef/data/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc\n",
"Using input files for variable tas of dataset CanESM2:\n",
"/home/stef/data/tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc\n",
"PreprocessingTask timeseries/tas_amsterdam created. It will create the files:\n",
"/home/stef/esmvaltool_output/recipe_python_20210106_112622/preproc/timeseries/tas_amsterdam/CMIP5_CanESM2_Amon_historical_r1i1p1_tas_1850-2000.nc\n",
"/home/stef/esmvaltool_output/recipe_python_20210106_112622/preproc/timeseries/tas_amsterdam/CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_1850-2000.nc\n",
"Creating preprocessor task timeseries/tas_global\n",
"Creating preprocessor 'annual_mean_global' task for variable 'tas'\n",
"Using input files for variable tas of dataset BCC-ESM1:\n",
"/home/stef/data/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc\n",
"Using input files for variable tas of dataset CanESM2:\n",
"/home/stef/data/tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc\n",
"PreprocessingTask timeseries/tas_global created. It will create the files:\n",
"/home/stef/esmvaltool_output/recipe_python_20210106_112622/preproc/timeseries/tas_global/CMIP5_CanESM2_Amon_historical_r1i1p1_tas_1850-2000.nc\n",
"/home/stef/esmvaltool_output/recipe_python_20210106_112622/preproc/timeseries/tas_global/CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_1850-2000.nc\n",
"Creating diagnostic task timeseries/script1\n",
"These tasks will be executed: map/tas, timeseries/script1, timeseries/tas_global, timeseries/tas_amsterdam, map/script1\n",
"Running 5 tasks using 4 processes\n",
"Starting task map/tas in process [19207]\n",
"Starting task timeseries/tas_amsterdam in process [19208]\n",
"Starting task timeseries/tas_global in process [19209]\n",
"Progress: 3 tasks running, 2 tasks waiting for ancestors, 0/5 done\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/stef/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/iris/analysis/cartography.py:394: UserWarning: Using DEFAULT_SPHERICAL_EARTH_RADIUS.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Calculated grid area shape: (1812, 64, 128)\n",
"Successfully completed task map/tas (priority 0) in 0:00:10.898444\n",
"Progress: 2 tasks running, 2 tasks waiting for ancestors, 1/5 done\n",
"Starting task map/script1 in process [19210]\n",
"Running command ['/home/stef/miniconda3/envs/esmvaltool/bin/python', '/home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/stef/esmvaltool_output/recipe_python_20210106_112622/run/map/script1/settings.yml']\n",
"Writing output to /home/stef/esmvaltool_output/recipe_python_20210106_112622/work/map/script1\n",
"Writing plots to /home/stef/esmvaltool_output/recipe_python_20210106_112622/plots/map/script1\n",
"Writing log to /home/stef/esmvaltool_output/recipe_python_20210106_112622/run/map/script1/log.txt\n",
"To re-run this diagnostic script, run:\n",
"cd /home/stef/esmvaltool_output/recipe_python_20210106_112622/run/map/script1; MPLBACKEND=\"Agg\" /home/stef/miniconda3/envs/esmvaltool/bin/python /home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py /home/stef/esmvaltool_output/recipe_python_20210106_112622/run/map/script1/settings.yml\n",
"Progress: 3 tasks running, 1 tasks waiting for ancestors, 1/5 done\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/stef/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/iris/analysis/cartography.py:394: UserWarning: Using DEFAULT_SPHERICAL_EARTH_RADIUS.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Calculated grid area shape: (1812, 64, 128)\n",
"Maximum memory used (estimate): 0.2 GB\n",
"Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.\n",
"Successfully completed task map/script1 (priority 1) in 0:00:04.595631\n",
"Progress: 2 tasks running, 1 tasks waiting for ancestors, 2/5 done\n",
"Successfully completed task timeseries/tas_global (priority 3) in 0:00:16.353615\n",
"Progress: 1 tasks running, 1 tasks waiting for ancestors, 3/5 done\n",
"Generated PreprocessorFile: /home/stef/esmvaltool_output/recipe_python_20210106_112622/preproc/timeseries/tas_amsterdam/MultiModelMean_Amon_tas_1850-2000.nc\n",
"Successfully completed task timeseries/tas_amsterdam (priority 2) in 0:00:21.932366\n",
"Progress: 0 tasks running, 1 tasks waiting for ancestors, 4/5 done\n",
"Starting task timeseries/script1 in process [19207]\n",
"Running command ['/home/stef/miniconda3/envs/esmvaltool/bin/python', '/home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/stef/esmvaltool_output/recipe_python_20210106_112622/run/timeseries/script1/settings.yml']\n",
"Writing output to /home/stef/esmvaltool_output/recipe_python_20210106_112622/work/timeseries/script1\n",
"Writing plots to /home/stef/esmvaltool_output/recipe_python_20210106_112622/plots/timeseries/script1\n",
"Writing log to /home/stef/esmvaltool_output/recipe_python_20210106_112622/run/timeseries/script1/log.txt\n",
"To re-run this diagnostic script, run:\n",
"cd /home/stef/esmvaltool_output/recipe_python_20210106_112622/run/timeseries/script1; MPLBACKEND=\"Agg\" /home/stef/miniconda3/envs/esmvaltool/bin/python /home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py /home/stef/esmvaltool_output/recipe_python_20210106_112622/run/timeseries/script1/settings.yml\n",
"Progress: 1 tasks running, 0 tasks waiting for ancestors, 4/5 done\n",
"Maximum memory used (estimate): 0.2 GB\n",
"Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.\n",
"Successfully completed task timeseries/script1 (priority 4) in 0:00:05.227710\n",
"Progress: 0 tasks running, 0 tasks waiting for ancestors, 5/5 done\n",
"Successfully completed all tasks.\n"
]
}
],
"source": [
"recipe.run()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`.run` takes a session object as an optional parameter, so you can define your own session:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'auxiliary_data_dir': PosixPath('/home/stef/auxiliary_data'),\n",
" 'check_level': <CheckLevels.DEFAULT: 3>,\n",
" 'compress_netcdf': False,\n",
" 'config_developer_file': None,\n",
" 'config_file': PosixPath('/home/stef/.esmvaltool/config-user.yml'),\n",
" 'diagnostics': set(),\n",
" 'drs': {'CMIP5': 'default', 'CMIP6': 'default'},\n",
" 'exit_on_warning': False,\n",
" 'log_level': 'info',\n",
" 'max_datasets': None,\n",
" 'max_parallel_tasks': None,\n",
" 'max_years': None,\n",
" 'output_dir': PosixPath('/home/stef/esmvaltool_output'),\n",
" 'output_file_type': 'png',\n",
" 'plot_dir': PosixPath('/home/stef/esmvaltool_output/my_session_20210106_112650/plots'),\n",
" 'preproc_dir': PosixPath('/home/stef/esmvaltool_output/my_session_20210106_112650/preproc'),\n",
" 'profile_diagnostic': False,\n",
" 'remove_preproc_dir': True,\n",
" 'rootpath': {'CMIP5': [PosixPath('/home/stef/data')],\n",
" 'CMIP6': [PosixPath('/home/stef/data')],\n",
" 'default': [PosixPath('/home/stef/data')]},\n",
" 'run_diagnostic': True,\n",
" 'run_dir': PosixPath('/home/stef/esmvaltool_output/my_session_20210106_112650/run'),\n",
" 'save_intermediary_cubes': False,\n",
" 'skip-nonexistent': False,\n",
" 'synda_download': False,\n",
" 'work_dir': PosixPath('/home/stef/esmvaltool_output/my_session_20210106_112650/work'),\n",
" 'write_netcdf': True,\n",
" 'write_plots': True}\n",
"Creating tasks from recipe\n",
"Creating tasks for diagnostic map\n",
"Creating preprocessor task map/tas\n",
"Creating preprocessor 'select_january' task for variable 'tas'\n",
"Using input files for variable tas of dataset BCC-ESM1:\n",
"/home/stef/data/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc\n",
"Using input files for variable tas of dataset CanESM2:\n",
"/home/stef/data/tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc\n",
"PreprocessingTask map/tas created. It will create the files:\n",
"/home/stef/esmvaltool_output/my_session_20210106_112650/preproc/map/tas/CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.nc\n",
"/home/stef/esmvaltool_output/my_session_20210106_112650/preproc/map/tas/CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.nc\n",
"Creating diagnostic task map/script1\n",
"Creating tasks for diagnostic timeseries\n",
"Creating preprocessor task timeseries/tas_amsterdam\n",
"Creating preprocessor 'annual_mean_amsterdam' task for variable 'tas'\n",
"Using input files for variable tas of dataset BCC-ESM1:\n",
"/home/stef/data/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc\n",
"Using input files for variable tas of dataset CanESM2:\n",
"/home/stef/data/tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc\n",
"PreprocessingTask timeseries/tas_amsterdam created. It will create the files:\n",
"/home/stef/esmvaltool_output/my_session_20210106_112650/preproc/timeseries/tas_amsterdam/CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_1850-2000.nc\n",
"/home/stef/esmvaltool_output/my_session_20210106_112650/preproc/timeseries/tas_amsterdam/CMIP5_CanESM2_Amon_historical_r1i1p1_tas_1850-2000.nc\n",
"Creating preprocessor task timeseries/tas_global\n",
"Creating preprocessor 'annual_mean_global' task for variable 'tas'\n",
"Using input files for variable tas of dataset BCC-ESM1:\n",
"/home/stef/data/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc\n",
"Using input files for variable tas of dataset CanESM2:\n",
"/home/stef/data/tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc\n",
"PreprocessingTask timeseries/tas_global created. It will create the files:\n",
"/home/stef/esmvaltool_output/my_session_20210106_112650/preproc/timeseries/tas_global/CMIP5_CanESM2_Amon_historical_r1i1p1_tas_1850-2000.nc\n",
"/home/stef/esmvaltool_output/my_session_20210106_112650/preproc/timeseries/tas_global/CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_1850-2000.nc\n",
"Creating diagnostic task timeseries/script1\n",
"These tasks will be executed: timeseries/tas_global, timeseries/tas_amsterdam, map/tas, timeseries/script1, map/script1\n",
"Running 5 tasks using 4 processes\n",
"Starting task map/tas in process [19350]\n",
"Starting task timeseries/tas_global in process [19352]\n",
"Starting task timeseries/tas_amsterdam in process [19351]\n",
"Progress: 3 tasks running, 2 tasks waiting for ancestors, 0/5 done\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/stef/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/iris/analysis/cartography.py:394: UserWarning: Using DEFAULT_SPHERICAL_EARTH_RADIUS.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Calculated grid area shape: (1812, 64, 128)\n",
"Successfully completed task map/tas (priority 0) in 0:00:08.363885\n",
"Progress: 2 tasks running, 2 tasks waiting for ancestors, 1/5 done\n",
"Starting task map/script1 in process [19353]\n",
"Running command ['/home/stef/miniconda3/envs/esmvaltool/bin/python', '/home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/stef/esmvaltool_output/my_session_20210106_112650/run/map/script1/settings.yml']\n",
"Writing output to /home/stef/esmvaltool_output/my_session_20210106_112650/work/map/script1\n",
"Writing plots to /home/stef/esmvaltool_output/my_session_20210106_112650/plots/map/script1\n",
"Writing log to /home/stef/esmvaltool_output/my_session_20210106_112650/run/map/script1/log.txt\n",
"To re-run this diagnostic script, run:\n",
"cd /home/stef/esmvaltool_output/my_session_20210106_112650/run/map/script1; MPLBACKEND=\"Agg\" /home/stef/miniconda3/envs/esmvaltool/bin/python /home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py /home/stef/esmvaltool_output/my_session_20210106_112650/run/map/script1/settings.yml\n",
"Progress: 3 tasks running, 1 tasks waiting for ancestors, 1/5 done\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/stef/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/iris/analysis/cartography.py:394: UserWarning: Using DEFAULT_SPHERICAL_EARTH_RADIUS.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Calculated grid area shape: (1812, 64, 128)\n",
"Maximum memory used (estimate): 0.2 GB\n",
"Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.\n",
"Successfully completed task map/script1 (priority 1) in 0:00:03.537595\n",
"Progress: 2 tasks running, 1 tasks waiting for ancestors, 2/5 done\n",
"Successfully completed task timeseries/tas_global (priority 3) in 0:00:13.544399\n",
"Progress: 1 tasks running, 1 tasks waiting for ancestors, 3/5 done\n",
"Generated PreprocessorFile: /home/stef/esmvaltool_output/my_session_20210106_112650/preproc/timeseries/tas_amsterdam/MultiModelMean_Amon_tas_1850-2000.nc\n",
"Successfully completed task timeseries/tas_amsterdam (priority 2) in 0:00:18.346895\n",
"Progress: 0 tasks running, 1 tasks waiting for ancestors, 4/5 done\n",
"Starting task timeseries/script1 in process [19350]\n",
"Running command ['/home/stef/miniconda3/envs/esmvaltool/bin/python', '/home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/stef/esmvaltool_output/my_session_20210106_112650/run/timeseries/script1/settings.yml']\n",
"Writing output to /home/stef/esmvaltool_output/my_session_20210106_112650/work/timeseries/script1\n",
"Writing plots to /home/stef/esmvaltool_output/my_session_20210106_112650/plots/timeseries/script1\n",
"Writing log to /home/stef/esmvaltool_output/my_session_20210106_112650/run/timeseries/script1/log.txt\n",
"To re-run this diagnostic script, run:\n",
"cd /home/stef/esmvaltool_output/my_session_20210106_112650/run/timeseries/script1; MPLBACKEND=\"Agg\" /home/stef/miniconda3/envs/esmvaltool/bin/python /home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py /home/stef/esmvaltool_output/my_session_20210106_112650/run/timeseries/script1/settings.yml\n",
"Progress: 1 tasks running, 0 tasks waiting for ancestors, 4/5 done\n",
"Maximum memory used (estimate): 0.2 GB\n",
"Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.\n",
"Successfully completed task timeseries/script1 (priority 4) in 0:00:04.081022\n",
"Progress: 0 tasks running, 0 tasks waiting for ancestors, 5/5 done\n",
"Successfully completed all tasks.\n"
]
}
],
"source": [
"from esmvalcore.experimental import CFG\n",
"\n",
"my_session = CFG.start_session('my_session')\n",
"recipe.run(session=my_session)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "esmvaltool",
"language": "python",
"name": "esmvaltool"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
"\n",
"The goal of this notebook is to demonstrate and test the functionality to load and run recipes directly from the notebook. See associated PR here: https://github.com/ESMValGroup/ESMValCore/pull/907"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/stef/r/esmvalcore/esmvalcore/experimental/__init__.py:5: UserWarning: \n",
" Thank you for trying out the new ESMValCore API.\n",
" Note that this API is experimental and may be subject to change.\n",
" More info: https://github.com/ESMValGroup/ESMValCore/issues/498\n",
"/home/stef/r/esmvalcore/esmvalcore/experimental/config/_config_validators.py:254: ESMValToolDeprecationWarning: `write_plots` will be removed in 2.2.0.\n",
"/home/stef/r/esmvalcore/esmvalcore/experimental/config/_config_validators.py:255: ESMValToolDeprecationWarning: `write_netcdf` will be removed in 2.2.0.\n",
"/home/stef/r/esmvalcore/esmvalcore/experimental/config/_config_validators.py:256: ESMValToolDeprecationWarning: `output_file_type` will be removed in 2.2.0.\n",
"/home/stef/r/esmvalcore/esmvalcore/experimental/config/_validated_config.py:81: MissingConfigParameter: `drs` is not defined (https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html)\n",
"/home/stef/r/esmvalcore/esmvalcore/experimental/config/_validated_config.py:81: MissingConfigParameter: `rootpath` is not defined (https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html)\n"
]
}
],
"source": [
"from esmvalcore.experimental import CFG"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Getting a recipe\n",
"\n",
"Recipes can be obtained by their name, similar to what one may already be familiar with through `esmvaltool recipes list`. To do so, use `get_recipe`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/markdown": [
"## Recipe python\n",
"\n",
"Example recipe that plots a map and timeseries of temperature.\n",
"\n",
"### Authors\n",
" - Bouwe Andela (NLeSC, Netherlands; https://orcid.org/0000-0001-9005-8940)\n",
" - Mattia Righi (DLR, Germany; https://orcid.org/0000-0003-3827-5950)\n",
"\n",
"### Maintainers\n",
" - Manuel Schlund (DLR, Germany; https://orcid.org/0000-0001-5251-0158)\n",
"\n",
"### Projects\n",
" - DLR project ESMVal\n",
" - Copernicus Climate Change Service 34a Lot 2 (MAGIC) project\n",
"\n",
"### References\n",
" - Please acknowledge the project\\(s\\)\\.\n"
],
"text/plain": [
"RecipeInfo('Recipe python')"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from esmvalcore.experimental import get_recipe\n",
"\n",
"recipe_info = get_recipe('examples/recipe_python')\n",
"recipe_info"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Running a recipe\n",
"\n",
"Run the recipe by using the `.run` method. \n",
"\n",
"The session can be defined as an arguement, but if left out, a new session is created automatically.\n",
"\n",
"Under the hood it calls `esmvalcore._recipe.Recipe` with the session config. The logger is redirected automatically to `main_log_debug.txt` / `main_log.txt` in the appropriate directory."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'auxiliary_data_dir': PosixPath('/home/stef/auxiliary_data'),\n",
" 'check_level': <CheckLevels.DEFAULT: 3>,\n",
" 'compress_netcdf': False,\n",
" 'config_developer_file': None,\n",
" 'config_file': PosixPath('/home/stef/.esmvaltool/config-user.yml'),\n",
" 'diagnostics': set(),\n",
" 'drs': {'CMIP5': 'default', 'CMIP6': 'default'},\n",
" 'exit_on_warning': False,\n",
" 'log_level': 'info',\n",
" 'max_datasets': None,\n",
" 'max_parallel_tasks': 1,\n",
" 'max_years': None,\n",
" 'output_dir': PosixPath('/home/stef/esmvaltool_output'),\n",
" 'output_file_type': 'png',\n",
" 'plot_dir': PosixPath('/home/stef/esmvaltool_output/recipe_python_20201217_161632/plots'),\n",
" 'preproc_dir': PosixPath('/home/stef/esmvaltool_output/recipe_python_20201217_161632/preproc'),\n",
" 'profile_diagnostic': False,\n",
" 'remove_preproc_dir': True,\n",
" 'rootpath': {'CMIP5': [PosixPath('/home/stef/data')],\n",
" 'CMIP6': [PosixPath('/home/stef/data')],\n",
" 'default': [PosixPath('/home/stef/data')]},\n",
" 'run_diagnostic': True,\n",
" 'run_dir': PosixPath('/home/stef/esmvaltool_output/recipe_python_20201217_161632/run'),\n",
" 'save_intermediary_cubes': False,\n",
" 'skip-nonexistent': False,\n",
" 'synda_download': False,\n",
" 'work_dir': PosixPath('/home/stef/esmvaltool_output/recipe_python_20201217_161632/work'),\n",
" 'write_netcdf': True,\n",
" 'write_plots': True}\n",
"Creating tasks from recipe\n",
"Creating tasks for diagnostic map\n",
"Creating preprocessor task map/tas\n",
"Creating preprocessor 'select_january' task for variable 'tas'\n",
"Using input files for variable tas of dataset BCC-ESM1:\n",
"/home/stef/data/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc\n",
"Using input files for variable tas of dataset CanESM2:\n",
"/home/stef/data/tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc\n",
"PreprocessingTask map/tas created. It will create the files:\n",
"/home/stef/esmvaltool_output/recipe_python_20201217_161632/preproc/map/tas/CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.nc\n",
"/home/stef/esmvaltool_output/recipe_python_20201217_161632/preproc/map/tas/CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.nc\n",
"Creating diagnostic task map/script1\n",
"Creating tasks for diagnostic timeseries\n",
"Creating preprocessor task timeseries/tas_amsterdam\n",
"Creating preprocessor 'annual_mean_amsterdam' task for variable 'tas'\n",
"Using input files for variable tas of dataset BCC-ESM1:\n",
"/home/stef/data/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc\n",
"Using input files for variable tas of dataset CanESM2:\n",
"/home/stef/data/tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc\n",
"PreprocessingTask timeseries/tas_amsterdam created. It will create the files:\n",
"/home/stef/esmvaltool_output/recipe_python_20201217_161632/preproc/timeseries/tas_amsterdam/CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_1850-2000.nc\n",
"/home/stef/esmvaltool_output/recipe_python_20201217_161632/preproc/timeseries/tas_amsterdam/CMIP5_CanESM2_Amon_historical_r1i1p1_tas_1850-2000.nc\n",
"Creating preprocessor task timeseries/tas_global\n",
"Creating preprocessor 'annual_mean_global' task for variable 'tas'\n",
"Using input files for variable tas of dataset BCC-ESM1:\n",
"/home/stef/data/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc\n",
"Using input files for variable tas of dataset CanESM2:\n",
"/home/stef/data/tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc\n",
"PreprocessingTask timeseries/tas_global created. It will create the files:\n",
"/home/stef/esmvaltool_output/recipe_python_20201217_161632/preproc/timeseries/tas_global/CMIP5_CanESM2_Amon_historical_r1i1p1_tas_1850-2000.nc\n",
"/home/stef/esmvaltool_output/recipe_python_20201217_161632/preproc/timeseries/tas_global/CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_1850-2000.nc\n",
"Creating diagnostic task timeseries/script1\n",
"These tasks will be executed: map/tas, timeseries/tas_global, map/script1, timeseries/tas_amsterdam, timeseries/script1\n",
"Running 5 tasks sequentially\n",
"Starting task map/tas in process [19946]\n",
"Successfully completed task map/tas (priority 0) in 0:00:06.913743\n",
"Starting task map/script1 in process [19946]\n",
"Running command ['/home/stef/miniconda3/envs/esmvaltool/bin/python', '/home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/stef/esmvaltool_output/recipe_python_20201217_161632/run/map/script1/settings.yml']\n",
"Writing output to /home/stef/esmvaltool_output/recipe_python_20201217_161632/work/map/script1\n",
"Writing plots to /home/stef/esmvaltool_output/recipe_python_20201217_161632/plots/map/script1\n",
"Writing log to /home/stef/esmvaltool_output/recipe_python_20201217_161632/run/map/script1/log.txt\n",
"To re-run this diagnostic script, run:\n",
"cd /home/stef/esmvaltool_output/recipe_python_20201217_161632/run/map/script1; MPLBACKEND=\"Agg\" /home/stef/miniconda3/envs/esmvaltool/bin/python /home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py /home/stef/esmvaltool_output/recipe_python_20201217_161632/run/map/script1/settings.yml\n",
"Maximum memory used (estimate): 0.2 GB\n",
"Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.\n",
"Successfully completed task map/script1 (priority 1) in 0:00:03.432193\n",
"Starting task timeseries/tas_amsterdam in process [19946]\n",
"Generated PreprocessorFile: /home/stef/esmvaltool_output/recipe_python_20201217_161632/preproc/timeseries/tas_amsterdam/MultiModelMean_Amon_tas_1850-2000.nc\n",
"Successfully completed task timeseries/tas_amsterdam (priority 2) in 0:00:16.398000\n",
"Starting task timeseries/tas_global in process [19946]\n",
"Calculated grid area shape: (1812, 64, 128)\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/stef/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/iris/analysis/cartography.py:394: UserWarning: Using DEFAULT_SPHERICAL_EARTH_RADIUS.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Calculated grid area shape: (1812, 64, 128)\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/stef/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/iris/analysis/cartography.py:394: UserWarning: Using DEFAULT_SPHERICAL_EARTH_RADIUS.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Successfully completed task timeseries/tas_global (priority 3) in 0:00:11.466836\n",
"Starting task timeseries/script1 in process [19946]\n",
"Running command ['/home/stef/miniconda3/envs/esmvaltool/bin/python', '/home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py', '/home/stef/esmvaltool_output/recipe_python_20201217_161632/run/timeseries/script1/settings.yml']\n",
"Writing output to /home/stef/esmvaltool_output/recipe_python_20201217_161632/work/timeseries/script1\n",
"Writing plots to /home/stef/esmvaltool_output/recipe_python_20201217_161632/plots/timeseries/script1\n",
"Writing log to /home/stef/esmvaltool_output/recipe_python_20201217_161632/run/timeseries/script1/log.txt\n",
"To re-run this diagnostic script, run:\n",
"cd /home/stef/esmvaltool_output/recipe_python_20201217_161632/run/timeseries/script1; MPLBACKEND=\"Agg\" /home/stef/miniconda3/envs/esmvaltool/bin/python /home/stef/r/esmvaltool/esmvaltool/diag_scripts/examples/diagnostic.py /home/stef/esmvaltool_output/recipe_python_20201217_161632/run/timeseries/script1/settings.yml\n",
"Maximum memory used (estimate): 0.2 GB\n",
"Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.\n",
"Successfully completed task timeseries/script1 (priority 4) in 0:00:04.954120\n"
]
}
],
"source": [
"recipe_info.run()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "esmvaltool",
"language": "python",
"name": "esmvaltool"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment