Skip to content

Instantly share code, notes, and snippets.

@rsignell-usgs

rsignell-usgs/mitgcm_detiding.ipynb

Created March 16, 2021 15:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rsignell-usgs/2d8e283a7afe71c01d28fb4f711e3d0a to your computer and use it in GitHub Desktop.
Save rsignell-usgs/2d8e283a7afe71c01d28fb4f711e3d0a to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
mitgcm_detiding.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Calculation of simulated tidal waves in the MIT/GCM/LLC4320 model\n",
"\n",
"In this example, we will estimate the tidal waves generated by the model. To do this we have to process the time series consisting of a 2D grid set. This set is stored as a 3D cube of 242'611'200 pixels per face stacked in 7566 layers. Each layer represents a date of the simulation. This cube is stored in Zarr format and represents about 7 TB.\n",
"\n",
"![cube](cube.png \"Chart representation of the cube formed by the grids.\")\n",
"\n",
"This analysis consists of performing a harmonic analysis on a time series of the cube. This time series is shown in yellow in the figure below. To perform this analysis, we must perform this analysis on all the pixels of the faces, i.e. 242 million times. It will also be necessary to modify the shape of the cube in memory to optimize data access during analysis."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import dask.distributed\n",
"import intake\n",
"import numpy as np"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Definition of the calculation period of the analysis (the spin-up period is not included)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"START_DATE = np.datetime64('2011-11-13')\n",
"END_DATE = np.datetime64('2012-11-12')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Loading the dataset."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div><svg style=\"position: absolute; width: 0; height: 0; overflow: hidden\">\n",
"<defs>\n",
"<symbol id=\"icon-database\" viewBox=\"0 0 32 32\">\n",
"<path d=\"M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z\"></path>\n",
"<path d=\"M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
"<path d=\"M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
"</symbol>\n",
"<symbol id=\"icon-file-text2\" viewBox=\"0 0 32 32\">\n",
"<path d=\"M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z\"></path>\n",
"<path d=\"M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"<path d=\"M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"<path d=\"M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
"</symbol>\n",
"</defs>\n",
"</svg>\n",
"<style>/* CSS stylesheet for displaying xarray objects in jupyterlab.\n",
" *\n",
" */\n",
"\n",
":root {\n",
" --xr-font-color0: var(--jp-content-font-color0, rgba(0, 0, 0, 1));\n",
" --xr-font-color2: var(--jp-content-font-color2, rgba(0, 0, 0, 0.54));\n",
" --xr-font-color3: var(--jp-content-font-color3, rgba(0, 0, 0, 0.38));\n",
" --xr-border-color: var(--jp-border-color2, #e0e0e0);\n",
" --xr-disabled-color: var(--jp-layout-color3, #bdbdbd);\n",
" --xr-background-color: var(--jp-layout-color0, white);\n",
" --xr-background-color-row-even: var(--jp-layout-color1, white);\n",
" --xr-background-color-row-odd: var(--jp-layout-color2, #eeeeee);\n",
"}\n",
"\n",
"html[theme=dark],\n",
"body.vscode-dark {\n",
" --xr-font-color0: rgba(255, 255, 255, 1);\n",
" --xr-font-color2: rgba(255, 255, 255, 0.54);\n",
" --xr-font-color3: rgba(255, 255, 255, 0.38);\n",
" --xr-border-color: #1F1F1F;\n",
" --xr-disabled-color: #515151;\n",
" --xr-background-color: #111111;\n",
" --xr-background-color-row-even: #111111;\n",
" --xr-background-color-row-odd: #313131;\n",
"}\n",
"\n",
".xr-wrap {\n",
" display: block;\n",
" min-width: 300px;\n",
" max-width: 700px;\n",
"}\n",
"\n",
".xr-text-repr-fallback {\n",
" /* fallback to plain text repr when CSS is not injected (untrusted notebook) */\n",
" display: none;\n",
"}\n",
"\n",
".xr-header {\n",
" padding-top: 6px;\n",
" padding-bottom: 6px;\n",
" margin-bottom: 4px;\n",
" border-bottom: solid 1px var(--xr-border-color);\n",
"}\n",
"\n",
".xr-header > div,\n",
".xr-header > ul {\n",
" display: inline;\n",
" margin-top: 0;\n",
" margin-bottom: 0;\n",
"}\n",
"\n",
".xr-obj-type,\n",
".xr-array-name {\n",
" margin-left: 2px;\n",
" margin-right: 10px;\n",
"}\n",
"\n",
".xr-obj-type {\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-sections {\n",
" padding-left: 0 !important;\n",
" display: grid;\n",
" grid-template-columns: 150px auto auto 1fr 20px 20px;\n",
"}\n",
"\n",
".xr-section-item {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-section-item input {\n",
" display: none;\n",
"}\n",
"\n",
".xr-section-item input + label {\n",
" color: var(--xr-disabled-color);\n",
"}\n",
"\n",
".xr-section-item input:enabled + label {\n",
" cursor: pointer;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-section-item input:enabled + label:hover {\n",
" color: var(--xr-font-color0);\n",
"}\n",
"\n",
".xr-section-summary {\n",
" grid-column: 1;\n",
" color: var(--xr-font-color2);\n",
" font-weight: 500;\n",
"}\n",
"\n",
".xr-section-summary > span {\n",
" display: inline-block;\n",
" padding-left: 0.5em;\n",
"}\n",
"\n",
".xr-section-summary-in:disabled + label {\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-section-summary-in + label:before {\n",
" display: inline-block;\n",
" content: '►';\n",
" font-size: 11px;\n",
" width: 15px;\n",
" text-align: center;\n",
"}\n",
"\n",
".xr-section-summary-in:disabled + label:before {\n",
" color: var(--xr-disabled-color);\n",
"}\n",
"\n",
".xr-section-summary-in:checked + label:before {\n",
" content: '▼';\n",
"}\n",
"\n",
".xr-section-summary-in:checked + label > span {\n",
" display: none;\n",
"}\n",
"\n",
".xr-section-summary,\n",
".xr-section-inline-details {\n",
" padding-top: 4px;\n",
" padding-bottom: 4px;\n",
"}\n",
"\n",
".xr-section-inline-details {\n",
" grid-column: 2 / -1;\n",
"}\n",
"\n",
".xr-section-details {\n",
" display: none;\n",
" grid-column: 1 / -1;\n",
" margin-bottom: 5px;\n",
"}\n",
"\n",
".xr-section-summary-in:checked ~ .xr-section-details {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-array-wrap {\n",
" grid-column: 1 / -1;\n",
" display: grid;\n",
" grid-template-columns: 20px auto;\n",
"}\n",
"\n",
".xr-array-wrap > label {\n",
" grid-column: 1;\n",
" vertical-align: top;\n",
"}\n",
"\n",
".xr-preview {\n",
" color: var(--xr-font-color3);\n",
"}\n",
"\n",
".xr-array-preview,\n",
".xr-array-data {\n",
" padding: 0 5px !important;\n",
" grid-column: 2;\n",
"}\n",
"\n",
".xr-array-data,\n",
".xr-array-in:checked ~ .xr-array-preview {\n",
" display: none;\n",
"}\n",
"\n",
".xr-array-in:checked ~ .xr-array-data,\n",
".xr-array-preview {\n",
" display: inline-block;\n",
"}\n",
"\n",
".xr-dim-list {\n",
" display: inline-block !important;\n",
" list-style: none;\n",
" padding: 0 !important;\n",
" margin: 0;\n",
"}\n",
"\n",
".xr-dim-list li {\n",
" display: inline-block;\n",
" padding: 0;\n",
" margin: 0;\n",
"}\n",
"\n",
".xr-dim-list:before {\n",
" content: '(';\n",
"}\n",
"\n",
".xr-dim-list:after {\n",
" content: ')';\n",
"}\n",
"\n",
".xr-dim-list li:not(:last-child):after {\n",
" content: ',';\n",
" padding-right: 5px;\n",
"}\n",
"\n",
".xr-has-index {\n",
" font-weight: bold;\n",
"}\n",
"\n",
".xr-var-list,\n",
".xr-var-item {\n",
" display: contents;\n",
"}\n",
"\n",
".xr-var-item > div,\n",
".xr-var-item label,\n",
".xr-var-item > .xr-var-name span {\n",
" background-color: var(--xr-background-color-row-even);\n",
" margin-bottom: 0;\n",
"}\n",
"\n",
".xr-var-item > .xr-var-name:hover span {\n",
" padding-right: 5px;\n",
"}\n",
"\n",
".xr-var-list > li:nth-child(odd) > div,\n",
".xr-var-list > li:nth-child(odd) > label,\n",
".xr-var-list > li:nth-child(odd) > .xr-var-name span {\n",
" background-color: var(--xr-background-color-row-odd);\n",
"}\n",
"\n",
".xr-var-name {\n",
" grid-column: 1;\n",
"}\n",
"\n",
".xr-var-dims {\n",
" grid-column: 2;\n",
"}\n",
"\n",
".xr-var-dtype {\n",
" grid-column: 3;\n",
" text-align: right;\n",
" color: var(--xr-font-color2);\n",
"}\n",
"\n",
".xr-var-preview {\n",
" grid-column: 4;\n",
"}\n",
"\n",
".xr-var-name,\n",
".xr-var-dims,\n",
".xr-var-dtype,\n",
".xr-preview,\n",
".xr-attrs dt {\n",
" white-space: nowrap;\n",
" overflow: hidden;\n",
" text-overflow: ellipsis;\n",
" padding-right: 10px;\n",
"}\n",
"\n",
".xr-var-name:hover,\n",
".xr-var-dims:hover,\n",
".xr-var-dtype:hover,\n",
".xr-attrs dt:hover {\n",
" overflow: visible;\n",
" width: auto;\n",
" z-index: 1;\n",
"}\n",
"\n",
".xr-var-attrs,\n",
".xr-var-data {\n",
" display: none;\n",
" background-color: var(--xr-background-color) !important;\n",
" padding-bottom: 5px !important;\n",
"}\n",
"\n",
".xr-var-attrs-in:checked ~ .xr-var-attrs,\n",
".xr-var-data-in:checked ~ .xr-var-data {\n",
" display: block;\n",
"}\n",
"\n",
".xr-var-data > table {\n",
" float: right;\n",
"}\n",
"\n",
".xr-var-name span,\n",
".xr-var-data,\n",
".xr-attrs {\n",
" padding-left: 25px !important;\n",
"}\n",
"\n",
".xr-attrs,\n",
".xr-var-attrs,\n",
".xr-var-data {\n",
" grid-column: 1 / -1;\n",
"}\n",
"\n",
"dl.xr-attrs {\n",
" padding: 0;\n",
" margin: 0;\n",
" display: grid;\n",
" grid-template-columns: 125px auto;\n",
"}\n",
"\n",
".xr-attrs dt,\n",
".xr-attrs dd {\n",
" padding: 0;\n",
" margin: 0;\n",
" float: left;\n",
" padding-right: 10px;\n",
" width: auto;\n",
"}\n",
"\n",
".xr-attrs dt {\n",
" font-weight: normal;\n",
" grid-column: 1;\n",
"}\n",
"\n",
".xr-attrs dt:hover span {\n",
" display: inline-block;\n",
" background: var(--xr-background-color);\n",
" padding-right: 10px;\n",
"}\n",
"\n",
".xr-attrs dd {\n",
" grid-column: 2;\n",
" white-space: pre-wrap;\n",
" word-break: break-all;\n",
"}\n",
"\n",
".xr-icon-database,\n",
".xr-icon-file-text2 {\n",
" display: inline-block;\n",
" vertical-align: middle;\n",
" width: 1em;\n",
" height: 1.5em !important;\n",
" stroke-width: 0;\n",
" stroke: currentColor;\n",
" fill: currentColor;\n",
"}\n",
"</style><pre class='xr-text-repr-fallback'>&lt;xarray.Dataset&gt;\n",
"Dimensions: (face: 13, i: 4320, j: 4320, time: 9030)\n",
"Coordinates:\n",
" * face (face) int64 0 1 2 3 4 5 6 7 8 9 10 11 12\n",
" * i (i) int64 0 1 2 3 4 5 6 7 ... 4313 4314 4315 4316 4317 4318 4319\n",
" * j (j) int64 0 1 2 3 4 5 6 7 ... 4313 4314 4315 4316 4317 4318 4319\n",
" * time (time) datetime64[ns] 2011-09-13 ... 2012-09-23T05:00:00\n",
"Data variables:\n",
" Eta (time, face, j, i) float32 dask.array&lt;chunksize=(1, 1, 4320, 4320), meta=np.ndarray&gt;\n",
"Attributes:\n",
" Conventions: CF-1.6\n",
" history: Created by calling `open_mdsdataset(llc_method=&#x27;smallchunks...\n",
" source: MITgcm\n",
" title: netCDF wrapper of MITgcm MDS binary data</pre><div class='xr-wrap' hidden><div class='xr-header'><div class='xr-obj-type'>xarray.Dataset</div></div><ul class='xr-sections'><li class='xr-section-item'><input id='section-0e610aef-9095-4e67-93b5-1d217a7f44aa' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-0e610aef-9095-4e67-93b5-1d217a7f44aa' class='xr-section-summary' title='Expand/collapse section'>Dimensions:</label><div class='xr-section-inline-details'><ul class='xr-dim-list'><li><span class='xr-has-index'>face</span>: 13</li><li><span class='xr-has-index'>i</span>: 4320</li><li><span class='xr-has-index'>j</span>: 4320</li><li><span class='xr-has-index'>time</span>: 9030</li></ul></div><div class='xr-section-details'></div></li><li class='xr-section-item'><input id='section-779fcb75-d6e3-4ce4-b0d1-fa11365e5c91' class='xr-section-summary-in' type='checkbox' checked><label for='section-779fcb75-d6e3-4ce4-b0d1-fa11365e5c91' class='xr-section-summary' >Coordinates: <span>(4)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>face</span></div><div class='xr-var-dims'>(face)</div><div class='xr-var-dtype'>int64</div><div class='xr-var-preview xr-preview'>0 1 2 3 4 5 6 7 8 9 10 11 12</div><input id='attrs-d06c35e9-e021-477f-9f64-52fcdf016add' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-d06c35e9-e021-477f-9f64-52fcdf016add' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-b2fddcfb-a9cd-4160-8763-975d2f1ef8f4' class='xr-var-data-in' type='checkbox'><label for='data-b2fddcfb-a9cd-4160-8763-975d2f1ef8f4' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>standard_name :</span></dt><dd>face_index</dd></dl></div><div class='xr-var-data'><pre>array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>i</span></div><div class='xr-var-dims'>(i)</div><div class='xr-var-dtype'>int64</div><div class='xr-var-preview xr-preview'>0 1 2 3 4 ... 4316 4317 4318 4319</div><input id='attrs-1f75c527-def9-4d39-acaa-4053bd2c4afe' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-1f75c527-def9-4d39-acaa-4053bd2c4afe' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-11221253-09e1-44a4-9557-ea0bfe884f36' class='xr-var-data-in' type='checkbox'><label for='data-11221253-09e1-44a4-9557-ea0bfe884f36' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>axis :</span></dt><dd>X</dd><dt><span>long_name :</span></dt><dd>x-dimension of the t grid</dd><dt><span>standard_name :</span></dt><dd>x_grid_index</dd><dt><span>swap_dim :</span></dt><dd>XC</dd></dl></div><div class='xr-var-data'><pre>array([ 0, 1, 2, ..., 4317, 4318, 4319])</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>j</span></div><div class='xr-var-dims'>(j)</div><div class='xr-var-dtype'>int64</div><div class='xr-var-preview xr-preview'>0 1 2 3 4 ... 4316 4317 4318 4319</div><input id='attrs-c9b96ecc-5f8b-46de-975a-80baa768d0d2' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-c9b96ecc-5f8b-46de-975a-80baa768d0d2' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-0430c51b-692a-47ec-ae79-2fa2d2d26e32' class='xr-var-data-in' type='checkbox'><label for='data-0430c51b-692a-47ec-ae79-2fa2d2d26e32' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>axis :</span></dt><dd>Y</dd><dt><span>long_name :</span></dt><dd>y-dimension of the t grid</dd><dt><span>standard_name :</span></dt><dd>y_grid_index</dd><dt><span>swap_dim :</span></dt><dd>YC</dd></dl></div><div class='xr-var-data'><pre>array([ 0, 1, 2, ..., 4317, 4318, 4319])</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>time</span></div><div class='xr-var-dims'>(time)</div><div class='xr-var-dtype'>datetime64[ns]</div><div class='xr-var-preview xr-preview'>2011-09-13 ... 2012-09-23T05:00:00</div><input id='attrs-a20021ca-6900-498d-b75d-70e114f9995e' class='xr-var-attrs-in' type='checkbox' ><label for='attrs-a20021ca-6900-498d-b75d-70e114f9995e' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-8c7ff045-2a47-4da9-b851-1babf1f67ce5' class='xr-var-data-in' type='checkbox'><label for='data-8c7ff045-2a47-4da9-b851-1babf1f67ce5' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'><dt><span>axis :</span></dt><dd>T</dd><dt><span>long_name :</span></dt><dd>Time</dd><dt><span>standard_name :</span></dt><dd>time</dd></dl></div><div class='xr-var-data'><pre>array([&#x27;2011-09-13T00:00:00.000000000&#x27;, &#x27;2011-09-13T01:00:00.000000000&#x27;,\n",
" &#x27;2011-09-13T02:00:00.000000000&#x27;, ..., &#x27;2012-09-23T03:00:00.000000000&#x27;,\n",
" &#x27;2012-09-23T04:00:00.000000000&#x27;, &#x27;2012-09-23T05:00:00.000000000&#x27;],\n",
" dtype=&#x27;datetime64[ns]&#x27;)</pre></div></li></ul></div></li><li class='xr-section-item'><input id='section-78679c36-3fc1-45cc-a53d-aab1aa0d696c' class='xr-section-summary-in' type='checkbox' checked><label for='section-78679c36-3fc1-45cc-a53d-aab1aa0d696c' class='xr-section-summary' >Data variables: <span>(1)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span>Eta</span></div><div class='xr-var-dims'>(time, face, j, i)</div><div class='xr-var-dtype'>float32</div><div class='xr-var-preview xr-preview'>dask.array&lt;chunksize=(1, 1, 4320, 4320), meta=np.ndarray&gt;</div><input id='attrs-d606adbd-ce43-4a05-9452-8114a6736351' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-d606adbd-ce43-4a05-9452-8114a6736351' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-9a5a990a-df6f-4e40-93e4-fdf293aee723' class='xr-var-data-in' type='checkbox'><label for='data-9a5a990a-df6f-4e40-93e4-fdf293aee723' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><table>\n",
"<tr>\n",
"<td>\n",
"<table>\n",
" <thead>\n",
" <tr><td> </td><th> Array </th><th> Chunk </th></tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr><th> Bytes </th><td> 8.76 TB </td> <td> 74.65 MB </td></tr>\n",
" <tr><th> Shape </th><td> (9030, 13, 4320, 4320) </td> <td> (1, 1, 4320, 4320) </td></tr>\n",
" <tr><th> Count </th><td> 117391 Tasks </td><td> 117390 Chunks </td></tr>\n",
" <tr><th> Type </th><td> float32 </td><td> numpy.ndarray </td></tr>\n",
" </tbody>\n",
"</table>\n",
"</td>\n",
"<td>\n",
"<svg width=\"502\" height=\"122\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"0\" y1=\"0\" x2=\"120\" y2=\"0\" style=\"stroke-width:2\" />\n",
" <line x1=\"0\" y1=\"25\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"0\" y1=\"0\" x2=\"0\" y2=\"25\" style=\"stroke-width:2\" />\n",
" <line x1=\"3\" y1=\"0\" x2=\"3\" y2=\"25\" />\n",
" <line x1=\"7\" y1=\"0\" x2=\"7\" y2=\"25\" />\n",
" <line x1=\"11\" y1=\"0\" x2=\"11\" y2=\"25\" />\n",
" <line x1=\"14\" y1=\"0\" x2=\"14\" y2=\"25\" />\n",
" <line x1=\"18\" y1=\"0\" x2=\"18\" y2=\"25\" />\n",
" <line x1=\"22\" y1=\"0\" x2=\"22\" y2=\"25\" />\n",
" <line x1=\"26\" y1=\"0\" x2=\"26\" y2=\"25\" />\n",
" <line x1=\"29\" y1=\"0\" x2=\"29\" y2=\"25\" />\n",
" <line x1=\"33\" y1=\"0\" x2=\"33\" y2=\"25\" />\n",
" <line x1=\"37\" y1=\"0\" x2=\"37\" y2=\"25\" />\n",
" <line x1=\"41\" y1=\"0\" x2=\"41\" y2=\"25\" />\n",
" <line x1=\"44\" y1=\"0\" x2=\"44\" y2=\"25\" />\n",
" <line x1=\"48\" y1=\"0\" x2=\"48\" y2=\"25\" />\n",
" <line x1=\"52\" y1=\"0\" x2=\"52\" y2=\"25\" />\n",
" <line x1=\"56\" y1=\"0\" x2=\"56\" y2=\"25\" />\n",
" <line x1=\"60\" y1=\"0\" x2=\"60\" y2=\"25\" />\n",
" <line x1=\"63\" y1=\"0\" x2=\"63\" y2=\"25\" />\n",
" <line x1=\"67\" y1=\"0\" x2=\"67\" y2=\"25\" />\n",
" <line x1=\"71\" y1=\"0\" x2=\"71\" y2=\"25\" />\n",
" <line x1=\"74\" y1=\"0\" x2=\"74\" y2=\"25\" />\n",
" <line x1=\"78\" y1=\"0\" x2=\"78\" y2=\"25\" />\n",
" <line x1=\"82\" y1=\"0\" x2=\"82\" y2=\"25\" />\n",
" <line x1=\"86\" y1=\"0\" x2=\"86\" y2=\"25\" />\n",
" <line x1=\"89\" y1=\"0\" x2=\"89\" y2=\"25\" />\n",
" <line x1=\"93\" y1=\"0\" x2=\"93\" y2=\"25\" />\n",
" <line x1=\"97\" y1=\"0\" x2=\"97\" y2=\"25\" />\n",
" <line x1=\"101\" y1=\"0\" x2=\"101\" y2=\"25\" />\n",
" <line x1=\"104\" y1=\"0\" x2=\"104\" y2=\"25\" />\n",
" <line x1=\"108\" y1=\"0\" x2=\"108\" y2=\"25\" />\n",
" <line x1=\"112\" y1=\"0\" x2=\"112\" y2=\"25\" />\n",
" <line x1=\"116\" y1=\"0\" x2=\"116\" y2=\"25\" />\n",
" <line x1=\"120\" y1=\"0\" x2=\"120\" y2=\"25\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"0.0,0.0 120.0,0.0 120.0,25.412616514582485 0.0,25.412616514582485\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
"\n",
" <!-- Text -->\n",
" <text x=\"60.000000\" y=\"45.412617\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >9030</text>\n",
" <text x=\"140.000000\" y=\"12.706308\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(0,140.000000,12.706308)\">1</text>\n",
"\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"204\" y2=\"14\" style=\"stroke-width:2\" />\n",
" <line x1=\"190\" y1=\"57\" x2=\"204\" y2=\"72\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"190\" y2=\"57\" style=\"stroke-width:2\" />\n",
" <line x1=\"191\" y1=\"1\" x2=\"191\" y2=\"58\" />\n",
" <line x1=\"192\" y1=\"2\" x2=\"192\" y2=\"59\" />\n",
" <line x1=\"193\" y1=\"3\" x2=\"193\" y2=\"60\" />\n",
" <line x1=\"194\" y1=\"4\" x2=\"194\" y2=\"62\" />\n",
" <line x1=\"195\" y1=\"5\" x2=\"195\" y2=\"63\" />\n",
" <line x1=\"196\" y1=\"6\" x2=\"196\" y2=\"64\" />\n",
" <line x1=\"198\" y1=\"8\" x2=\"198\" y2=\"65\" />\n",
" <line x1=\"199\" y1=\"9\" x2=\"199\" y2=\"66\" />\n",
" <line x1=\"200\" y1=\"10\" x2=\"200\" y2=\"67\" />\n",
" <line x1=\"201\" y1=\"11\" x2=\"201\" y2=\"68\" />\n",
" <line x1=\"202\" y1=\"12\" x2=\"202\" y2=\"70\" />\n",
" <line x1=\"203\" y1=\"13\" x2=\"203\" y2=\"71\" />\n",
" <line x1=\"204\" y1=\"14\" x2=\"204\" y2=\"72\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"190.0,0.0 204.9485979497544,14.948597949754403 204.9485979497544,72.35723582350856 190.0,57.40863787375415\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"247\" y2=\"0\" style=\"stroke-width:2\" />\n",
" <line x1=\"191\" y1=\"1\" x2=\"248\" y2=\"1\" />\n",
" <line x1=\"192\" y1=\"2\" x2=\"249\" y2=\"2\" />\n",
" <line x1=\"193\" y1=\"3\" x2=\"250\" y2=\"3\" />\n",
" <line x1=\"194\" y1=\"4\" x2=\"252\" y2=\"4\" />\n",
" <line x1=\"195\" y1=\"5\" x2=\"253\" y2=\"5\" />\n",
" <line x1=\"196\" y1=\"6\" x2=\"254\" y2=\"6\" />\n",
" <line x1=\"198\" y1=\"8\" x2=\"255\" y2=\"8\" />\n",
" <line x1=\"199\" y1=\"9\" x2=\"256\" y2=\"9\" />\n",
" <line x1=\"200\" y1=\"10\" x2=\"257\" y2=\"10\" />\n",
" <line x1=\"201\" y1=\"11\" x2=\"258\" y2=\"11\" />\n",
" <line x1=\"202\" y1=\"12\" x2=\"260\" y2=\"12\" />\n",
" <line x1=\"203\" y1=\"13\" x2=\"261\" y2=\"13\" />\n",
" <line x1=\"204\" y1=\"14\" x2=\"262\" y2=\"14\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"190\" y1=\"0\" x2=\"204\" y2=\"14\" style=\"stroke-width:2\" />\n",
" <line x1=\"247\" y1=\"0\" x2=\"262\" y2=\"14\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"190.0,0.0 247.40863787375415,0.0 262.3572358235086,14.948597949754403 204.9485979497544,14.948597949754403\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"204\" y1=\"14\" x2=\"262\" y2=\"14\" style=\"stroke-width:2\" />\n",
" <line x1=\"204\" y1=\"72\" x2=\"262\" y2=\"72\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"204\" y1=\"14\" x2=\"204\" y2=\"72\" style=\"stroke-width:2\" />\n",
" <line x1=\"262\" y1=\"14\" x2=\"262\" y2=\"72\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"204.9485979497544,14.948597949754403 262.3572358235086,14.948597949754403 262.3572358235086,72.35723582350856 204.9485979497544,72.35723582350856\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Text -->\n",
" <text x=\"233.652917\" y=\"92.357236\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >4320</text>\n",
" <text x=\"282.357236\" y=\"43.652917\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,282.357236,43.652917)\">4320</text>\n",
" <text x=\"187.474299\" y=\"84.882937\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,187.474299,84.882937)\">13</text>\n",
"</svg>\n",
"</td>\n",
"</tr>\n",
"</table></div></li></ul></div></li><li class='xr-section-item'><input id='section-7d0cff3a-a818-4d59-b849-8b40a7118877' class='xr-section-summary-in' type='checkbox' checked><label for='section-7d0cff3a-a818-4d59-b849-8b40a7118877' class='xr-section-summary' >Attributes: <span>(4)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><dl class='xr-attrs'><dt><span>Conventions :</span></dt><dd>CF-1.6</dd><dt><span>history :</span></dt><dd>Created by calling `open_mdsdataset(llc_method=&#x27;smallchunks&#x27;, nz=None, ny=None, nx=None, default_dtype=dtype(&#x27;&gt;f4&#x27;), ignore_unknown_vars=True, chunks=None, endian=&#x27;&gt;&#x27;, swap_dims=None, grid_vars_to_coords=True, geometry=&#x27;llc&#x27;, calendar=&#x27;gregorian&#x27;, ref_date=None, delta_t=25.0, read_grid=True, prefix=None, iters=[10368], grid_dir=&#x27;/pleiades/u/dmenemen/llc_4320/grid/&#x27;, data_dir=&#x27;/pleiades/u/dmenemen/llc_4320/MITgcm/run/&#x27;)`</dd><dt><span>source :</span></dt><dd>MITgcm</dd><dt><span>title :</span></dt><dd>netCDF wrapper of MITgcm MDS binary data</dd></dl></div></li></ul></div></div>"
],
"text/plain": [
"<xarray.Dataset>\n",
"Dimensions: (face: 13, i: 4320, j: 4320, time: 9030)\n",
"Coordinates:\n",
" * face (face) int64 0 1 2 3 4 5 6 7 8 9 10 11 12\n",
" * i (i) int64 0 1 2 3 4 5 6 7 ... 4313 4314 4315 4316 4317 4318 4319\n",
" * j (j) int64 0 1 2 3 4 5 6 7 ... 4313 4314 4315 4316 4317 4318 4319\n",
" * time (time) datetime64[ns] 2011-09-13 ... 2012-09-23T05:00:00\n",
"Data variables:\n",
" Eta (time, face, j, i) float32 dask.array<chunksize=(1, 1, 4320, 4320), meta=np.ndarray>\n",
"Attributes:\n",
" Conventions: CF-1.6\n",
" history: Created by calling `open_mdsdataset(llc_method='smallchunks...\n",
" source: MITgcm\n",
" title: netCDF wrapper of MITgcm MDS binary data"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat_url = \"https://raw.githubusercontent.com/pangeo-data/pangeo-datastore/master/intake-catalogs/ocean/llc4320.yaml\"\n",
"cat = intake.open_catalog(cat_url)\n",
"\n",
"ssh = cat.LLC4320_SSH.to_dask()\n",
"ssh"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Selection of the computation time series (the spin-off period of the model is skipped)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"number of layers to process 7566\n",
"period [2011-11-13T00:00:00.000000000, 2012-09-23T05:00:00.000000000]\n"
]
}
],
"source": [
"time_series = ssh.time.values\n",
"period = (time_series >= START_DATE) & (time_series <= END_DATE)\n",
"print(\"number of layers to process %d\" % len(time_series[period]))\n",
"print(\"period [%s, %s]\" % (time_series[period].min(), time_series[period].max()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Creation of the object controlling the harmonic analysis of the waves M2, K1, O1, P1, S1, S2."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"6 tidal constituents to be analysed\n"
]
}
],
"source": [
"import pytide\n",
"\n",
"wave_table = pytide.WaveTable(['M2', 'K1', 'O1', 'P1', 'S1', 'S2'])\n",
"print(\"%d tidal constituents to be analysed\" % len(wave_table))"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "9b993901b807458ebc3e0ef072252d2b",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(HTML(value='<h2>GatewayCluster</h2>'), HBox(children=(HTML(value='\\n<div>\\n<style scoped>\\n …"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from dask_gateway import Gateway\n",
"from dask.distributed import Client\n",
"\n",
"gateway = Gateway()\n",
"cluster = gateway.new_cluster()\n",
"cluster.adapt(minimum=1, maximum=20)\n",
"cluster"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"client = Client(cluster)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Creation of the Dask cluster."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# from dask_kubernetes import KubeCluster\n",
"# from dask.distributed import Client\n",
"# cluster = KubeCluster()\n",
"# cluster.adapt(minimum=10, maximum=40)\n",
"# client = Client(cluster)\n",
"# cluster"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"def compute_nodal_modulations(client, waves, time_series):\n",
" \"\"\"Compute nodal corrections for a given time series\"\"\"\n",
" t = time_series.astype('datetime64[s]')\n",
" f, v0u = waves.compute_nodal_modulations(t)\n",
" return (dask.array.from_delayed(client.scatter(f, broadcast=True),\n",
" shape=f.shape,\n",
" dtype=f.dtype),\n",
" dask.array.from_delayed(client.scatter(v0u, broadcast=True),\n",
" shape=v0u.shape,\n",
" dtype=v0u.dtype))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Calculation of nodal corrections for the selected time series."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"f, v0u = compute_nodal_modulations(client, wave_table, time_series[period])"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"def load_faces(ds, face, period, indices=None):\n",
" \"\"\"Load a face from the time series\"\"\"\n",
" if indices is None:\n",
" indices = slice(0, None, 1)\n",
" ds = ds.Eta\n",
" ds = ds.transpose(\"face\", \"j\", \"i\", \"time\")\n",
" return ds.isel(face=face, time=period, i=indices, j=indices).data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The calculation takes a little time. In order to make it faster, the processed grid is sub-sampled. If you want to process it completely, replace the following line with:\n",
"\n",
"```python\n",
"indices=None\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"indices=slice(0, None, 8)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this example, we will analyze the part of the grid representing Europe (`face=2`). Since the tide is strong, near the northwest coast, the effects of the tide in this area can be better illustrated."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"face = 2"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table>\n",
"<tr>\n",
"<td>\n",
"<table>\n",
" <thead>\n",
" <tr><td> </td><th> Array </th><th> Chunk </th></tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr><th> Bytes </th><td> 8.82 GB </td> <td> 1.17 MB </td></tr>\n",
" <tr><th> Shape </th><td> (540, 540, 7566) </td> <td> (540, 540, 1) </td></tr>\n",
" <tr><th> Count </th><td> 251377 Tasks </td><td> 7566 Chunks </td></tr>\n",
" <tr><th> Type </th><td> float32 </td><td> numpy.ndarray </td></tr>\n",
" </tbody>\n",
"</table>\n",
"</td>\n",
"<td>\n",
"<svg width=\"201\" height=\"108\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"10\" y1=\"0\" x2=\"31\" y2=\"21\" style=\"stroke-width:2\" />\n",
" <line x1=\"10\" y1=\"36\" x2=\"31\" y2=\"58\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"36\" style=\"stroke-width:2\" />\n",
" <line x1=\"31\" y1=\"21\" x2=\"31\" y2=\"58\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"10.0,0.0 31.56166017942209,21.56166017942209 31.56166017942209,58.216482484439645 10.0,36.654822305017554\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"10\" y1=\"0\" x2=\"130\" y2=\"0\" style=\"stroke-width:2\" />\n",
" <line x1=\"31\" y1=\"21\" x2=\"151\" y2=\"21\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"10\" y1=\"0\" x2=\"31\" y2=\"21\" style=\"stroke-width:2\" />\n",
" <line x1=\"16\" y1=\"0\" x2=\"37\" y2=\"21\" />\n",
" <line x1=\"22\" y1=\"0\" x2=\"44\" y2=\"21\" />\n",
" <line x1=\"28\" y1=\"0\" x2=\"50\" y2=\"21\" />\n",
" <line x1=\"35\" y1=\"0\" x2=\"56\" y2=\"21\" />\n",
" <line x1=\"41\" y1=\"0\" x2=\"63\" y2=\"21\" />\n",
" <line x1=\"47\" y1=\"0\" x2=\"69\" y2=\"21\" />\n",
" <line x1=\"54\" y1=\"0\" x2=\"75\" y2=\"21\" />\n",
" <line x1=\"60\" y1=\"0\" x2=\"82\" y2=\"21\" />\n",
" <line x1=\"66\" y1=\"0\" x2=\"88\" y2=\"21\" />\n",
" <line x1=\"73\" y1=\"0\" x2=\"94\" y2=\"21\" />\n",
" <line x1=\"79\" y1=\"0\" x2=\"101\" y2=\"21\" />\n",
" <line x1=\"85\" y1=\"0\" x2=\"107\" y2=\"21\" />\n",
" <line x1=\"92\" y1=\"0\" x2=\"113\" y2=\"21\" />\n",
" <line x1=\"98\" y1=\"0\" x2=\"119\" y2=\"21\" />\n",
" <line x1=\"104\" y1=\"0\" x2=\"126\" y2=\"21\" />\n",
" <line x1=\"111\" y1=\"0\" x2=\"132\" y2=\"21\" />\n",
" <line x1=\"117\" y1=\"0\" x2=\"138\" y2=\"21\" />\n",
" <line x1=\"123\" y1=\"0\" x2=\"145\" y2=\"21\" />\n",
" <line x1=\"130\" y1=\"0\" x2=\"151\" y2=\"21\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"10.0,0.0 130.0,0.0 151.5616601794221,21.56166017942209 31.56166017942209,21.56166017942209\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"31\" y1=\"21\" x2=\"151\" y2=\"21\" style=\"stroke-width:2\" />\n",
" <line x1=\"31\" y1=\"58\" x2=\"151\" y2=\"58\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"31\" y1=\"21\" x2=\"31\" y2=\"58\" style=\"stroke-width:2\" />\n",
" <line x1=\"37\" y1=\"21\" x2=\"37\" y2=\"58\" />\n",
" <line x1=\"44\" y1=\"21\" x2=\"44\" y2=\"58\" />\n",
" <line x1=\"50\" y1=\"21\" x2=\"50\" y2=\"58\" />\n",
" <line x1=\"56\" y1=\"21\" x2=\"56\" y2=\"58\" />\n",
" <line x1=\"63\" y1=\"21\" x2=\"63\" y2=\"58\" />\n",
" <line x1=\"69\" y1=\"21\" x2=\"69\" y2=\"58\" />\n",
" <line x1=\"75\" y1=\"21\" x2=\"75\" y2=\"58\" />\n",
" <line x1=\"82\" y1=\"21\" x2=\"82\" y2=\"58\" />\n",
" <line x1=\"88\" y1=\"21\" x2=\"88\" y2=\"58\" />\n",
" <line x1=\"94\" y1=\"21\" x2=\"94\" y2=\"58\" />\n",
" <line x1=\"101\" y1=\"21\" x2=\"101\" y2=\"58\" />\n",
" <line x1=\"107\" y1=\"21\" x2=\"107\" y2=\"58\" />\n",
" <line x1=\"113\" y1=\"21\" x2=\"113\" y2=\"58\" />\n",
" <line x1=\"119\" y1=\"21\" x2=\"119\" y2=\"58\" />\n",
" <line x1=\"126\" y1=\"21\" x2=\"126\" y2=\"58\" />\n",
" <line x1=\"132\" y1=\"21\" x2=\"132\" y2=\"58\" />\n",
" <line x1=\"138\" y1=\"21\" x2=\"138\" y2=\"58\" />\n",
" <line x1=\"145\" y1=\"21\" x2=\"145\" y2=\"58\" />\n",
" <line x1=\"151\" y1=\"21\" x2=\"151\" y2=\"58\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"31.56166017942209,21.56166017942209 151.5616601794221,21.56166017942209 151.5616601794221,58.216482484439645 31.56166017942209,58.216482484439645\" style=\"fill:#8B4903A0;stroke-width:0\"/>\n",
"\n",
" <!-- Text -->\n",
" <text x=\"91.561660\" y=\"78.216482\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >7566</text>\n",
" <text x=\"171.561660\" y=\"39.889071\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,171.561660,39.889071)\">540</text>\n",
" <text x=\"10.780830\" y=\"67.435652\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,10.780830,67.435652)\">540</text>\n",
"</svg>\n",
"</td>\n",
"</tr>\n",
"</table>"
],
"text/plain": [
"dask.array<getitem, shape=(540, 540, 7566), dtype=float32, chunksize=(540, 540, 1), chunktype=numpy.ndarray>"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds = load_faces(ssh, face, period, indices=indices)\n",
"ds"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"def dask_array_rechunk(da, axis=2):\n",
" \"\"\"Search for the optimal block cutting without modifying the axis 'axis'\n",
" in order to optimize its access in memory.\"\"\"\n",
" nblocks = 1\n",
" \n",
" def calculate_chuncks_size(chunks, size):\n",
" result = np.array(chunks).prod() * size\n",
" return result / (1000**2)\n",
" \n",
" while True:\n",
" chunks = []\n",
" div = int(np.sqrt(nblocks))\n",
" for index, item in enumerate(da.chunks):\n",
" chunks.append(np.array(item).sum() * (div if index == axis else 1))\n",
" chunks = tuple(item // div for index, item in enumerate(chunks))\n",
" chuncks_size = calculate_chuncks_size(chunks, da.dtype.itemsize)\n",
" if chuncks_size > 100 and chuncks_size < 150:\n",
" return chunks\n",
" nblocks += 1"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table>\n",
"<tr>\n",
"<td>\n",
"<table>\n",
" <thead>\n",
" <tr><td> </td><th> Array </th><th> Chunk </th></tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr><th> Bytes </th><td> 8.82 GB </td> <td> 135.86 MB </td></tr>\n",
" <tr><th> Shape </th><td> (540, 540, 7566) </td> <td> (67, 67, 7566) </td></tr>\n",
" <tr><th> Count </th><td> 256870 Tasks </td><td> 81 Chunks </td></tr>\n",
" <tr><th> Type </th><td> float32 </td><td> numpy.ndarray </td></tr>\n",
" </tbody>\n",
"</table>\n",
"</td>\n",
"<td>\n",
"<svg width=\"201\" height=\"108\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"10\" y1=\"0\" x2=\"31\" y2=\"21\" style=\"stroke-width:2\" />\n",
" <line x1=\"10\" y1=\"4\" x2=\"31\" y2=\"26\" />\n",
" <line x1=\"10\" y1=\"9\" x2=\"31\" y2=\"30\" />\n",
" <line x1=\"10\" y1=\"13\" x2=\"31\" y2=\"35\" />\n",
" <line x1=\"10\" y1=\"18\" x2=\"31\" y2=\"39\" />\n",
" <line x1=\"10\" y1=\"22\" x2=\"31\" y2=\"44\" />\n",
" <line x1=\"10\" y1=\"27\" x2=\"31\" y2=\"48\" />\n",
" <line x1=\"10\" y1=\"31\" x2=\"31\" y2=\"53\" />\n",
" <line x1=\"10\" y1=\"36\" x2=\"31\" y2=\"57\" />\n",
" <line x1=\"10\" y1=\"36\" x2=\"31\" y2=\"58\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"36\" style=\"stroke-width:2\" />\n",
" <line x1=\"12\" y1=\"2\" x2=\"12\" y2=\"39\" />\n",
" <line x1=\"15\" y1=\"5\" x2=\"15\" y2=\"42\" />\n",
" <line x1=\"18\" y1=\"8\" x2=\"18\" y2=\"44\" />\n",
" <line x1=\"20\" y1=\"10\" x2=\"20\" y2=\"47\" />\n",
" <line x1=\"23\" y1=\"13\" x2=\"23\" y2=\"50\" />\n",
" <line x1=\"26\" y1=\"16\" x2=\"26\" y2=\"52\" />\n",
" <line x1=\"28\" y1=\"18\" x2=\"28\" y2=\"55\" />\n",
" <line x1=\"31\" y1=\"21\" x2=\"31\" y2=\"58\" />\n",
" <line x1=\"31\" y1=\"21\" x2=\"31\" y2=\"58\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"10.0,0.0 31.56166017942209,21.56166017942209 31.56166017942209,58.216482484439645 10.0,36.654822305017554\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"10\" y1=\"0\" x2=\"130\" y2=\"0\" style=\"stroke-width:2\" />\n",
" <line x1=\"12\" y1=\"2\" x2=\"132\" y2=\"2\" />\n",
" <line x1=\"15\" y1=\"5\" x2=\"135\" y2=\"5\" />\n",
" <line x1=\"18\" y1=\"8\" x2=\"138\" y2=\"8\" />\n",
" <line x1=\"20\" y1=\"10\" x2=\"140\" y2=\"10\" />\n",
" <line x1=\"23\" y1=\"13\" x2=\"143\" y2=\"13\" />\n",
" <line x1=\"26\" y1=\"16\" x2=\"146\" y2=\"16\" />\n",
" <line x1=\"28\" y1=\"18\" x2=\"148\" y2=\"18\" />\n",
" <line x1=\"31\" y1=\"21\" x2=\"151\" y2=\"21\" />\n",
" <line x1=\"31\" y1=\"21\" x2=\"151\" y2=\"21\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"10\" y1=\"0\" x2=\"31\" y2=\"21\" style=\"stroke-width:2\" />\n",
" <line x1=\"130\" y1=\"0\" x2=\"151\" y2=\"21\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"10.0,0.0 130.0,0.0 151.5616601794221,21.56166017942209 31.56166017942209,21.56166017942209\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Horizontal lines -->\n",
" <line x1=\"31\" y1=\"21\" x2=\"151\" y2=\"21\" style=\"stroke-width:2\" />\n",
" <line x1=\"31\" y1=\"26\" x2=\"151\" y2=\"26\" />\n",
" <line x1=\"31\" y1=\"30\" x2=\"151\" y2=\"30\" />\n",
" <line x1=\"31\" y1=\"35\" x2=\"151\" y2=\"35\" />\n",
" <line x1=\"31\" y1=\"39\" x2=\"151\" y2=\"39\" />\n",
" <line x1=\"31\" y1=\"44\" x2=\"151\" y2=\"44\" />\n",
" <line x1=\"31\" y1=\"48\" x2=\"151\" y2=\"48\" />\n",
" <line x1=\"31\" y1=\"53\" x2=\"151\" y2=\"53\" />\n",
" <line x1=\"31\" y1=\"57\" x2=\"151\" y2=\"57\" />\n",
" <line x1=\"31\" y1=\"58\" x2=\"151\" y2=\"58\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Vertical lines -->\n",
" <line x1=\"31\" y1=\"21\" x2=\"31\" y2=\"58\" style=\"stroke-width:2\" />\n",
" <line x1=\"151\" y1=\"21\" x2=\"151\" y2=\"58\" style=\"stroke-width:2\" />\n",
"\n",
" <!-- Colored Rectangle -->\n",
" <polygon points=\"31.56166017942209,21.56166017942209 151.5616601794221,21.56166017942209 151.5616601794221,58.216482484439645 31.56166017942209,58.216482484439645\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
"\n",
" <!-- Text -->\n",
" <text x=\"91.561660\" y=\"78.216482\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >7566</text>\n",
" <text x=\"171.561660\" y=\"39.889071\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,171.561660,39.889071)\">540</text>\n",
" <text x=\"10.780830\" y=\"67.435652\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,10.780830,67.435652)\">540</text>\n",
"</svg>\n",
"</td>\n",
"</tr>\n",
"</table>"
],
"text/plain": [
"dask.array<rechunk-merge, shape=(540, 540, 7566), dtype=float32, chunksize=(67, 67, 7566), chunktype=numpy.ndarray>"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds = ds.rechunk(dask_array_rechunk(ds))\n",
"ds"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Normally, we just have to call the function `dask.array.apply_along_axis`. This function tries to validate the arguments by executing once the function to be vectorized with arbitrary values. However, it does not work here, because one of our parameters is a matrix. To avoid this problem, the function has been copied and modified in this notebook."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"def _apply_along_axis(arr, func1d, func1d_axis, func1d_args, func1d_kwargs):\n",
" \"\"\"Wrap apply_along_axis\"\"\"\n",
" return np.apply_along_axis(func1d, func1d_axis, arr, *func1d_args,\n",
" **func1d_kwargs)\n",
"\n",
"\n",
"def apply_along_axis(func1d, axis, arr, *args, **kwargs):\n",
" \"\"\"Apply the harmonic analysis to 1-D slices along the given axis.\"\"\"\n",
" arr = dask.array.core.asarray(arr)\n",
"\n",
" # Validate and normalize axis.\n",
" arr.shape[axis]\n",
" axis = len(arr.shape[:axis])\n",
"\n",
" # Rechunk so that analyze is applied over the full axis.\n",
" arr = arr.rechunk(arr.chunks[:axis] + (arr.shape[axis:axis + 1], ) +\n",
" arr.chunks[axis + 1:])\n",
"\n",
" # Test out some data with the function.\n",
" test_data = np.ones(args[0].shape[1], dtype=arr.dtype)\n",
" test_result = np.array(func1d(test_data, *args, **kwargs))\n",
"\n",
" # Map analyze over the data to get the result\n",
" # Adds other axes as needed.\n",
" result = arr.map_blocks(\n",
" _apply_along_axis,\n",
" name=dask.utils.funcname(func1d) + '-along-axis',\n",
" dtype=test_result.dtype,\n",
" chunks=(arr.chunks[:axis] + test_result.shape + arr.chunks[axis + 1:]),\n",
" drop_axis=axis,\n",
" new_axis=list(range(axis, axis + test_result.ndim, 1)),\n",
" func1d=func1d,\n",
" func1d_axis=axis,\n",
" func1d_args=args,\n",
" func1d_kwargs=kwargs,\n",
" )\n",
"\n",
" return result"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, we perform our harmonic analysis on the face of the model loaded in memory.\n",
"\n",
"*Note: To analyze the entire time series, we would have to loop on the 13 faces storing the different geographical areas of the Earth.*"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"future = apply_along_axis(pytide.WaveTable.harmonic_analysis, 2, ds,\n",
" *(f, v0u))"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"ename": "KilledWorker",
"evalue": "(\"('harmonic_analysis-along-axis-8eca5ecff77faf95613d6a3504f7364e', 0, 1, 0)\", <Worker 'tls://10.49.132.7:35617', name: dask-worker-aafb9ced5a9d4d4a95b7185d85f06478-cqdpq, memory: 0, processing: 1>)",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mKilledWorker\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-21-db657b8a5761>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0manalysis\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfuture\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m/srv/conda/envs/notebook/lib/python3.8/site-packages/dask/base.py\u001b[0m in \u001b[0;36mcompute\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m 277\u001b[0m \u001b[0mdask\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbase\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcompute\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 278\u001b[0m \"\"\"\n\u001b[0;32m--> 279\u001b[0;31m \u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtraverse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 280\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 281\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/srv/conda/envs/notebook/lib/python3.8/site-packages/dask/base.py\u001b[0m in \u001b[0;36mcompute\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 565\u001b[0m \u001b[0mpostcomputes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dask_postcompute__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 566\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 567\u001b[0;31m \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mschedule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdsk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeys\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 568\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mrepack\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresults\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpostcomputes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 569\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(self, dsk, keys, restrictions, loose_restrictions, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)\u001b[0m\n\u001b[1;32m 2674\u001b[0m \u001b[0mshould_rejoin\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2675\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2676\u001b[0;31m \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgather\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpacked\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0masynchronous\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0masynchronous\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdirect\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdirect\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2677\u001b[0m \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2678\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mf\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mfutures\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py\u001b[0m in \u001b[0;36mgather\u001b[0;34m(self, futures, errors, direct, asynchronous)\u001b[0m\n\u001b[1;32m 1983\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1984\u001b[0m \u001b[0mlocal_worker\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1985\u001b[0;31m return self.sync(\n\u001b[0m\u001b[1;32m 1986\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_gather\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1987\u001b[0m \u001b[0mfutures\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py\u001b[0m in \u001b[0;36msync\u001b[0;34m(self, func, asynchronous, callback_timeout, *args, **kwargs)\u001b[0m\n\u001b[1;32m 829\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfuture\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 830\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 831\u001b[0;31m return sync(\n\u001b[0m\u001b[1;32m 832\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloop\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback_timeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcallback_timeout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 833\u001b[0m )\n",
"\u001b[0;32m/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py\u001b[0m in \u001b[0;36msync\u001b[0;34m(loop, func, callback_timeout, *args, **kwargs)\u001b[0m\n\u001b[1;32m 338\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0merror\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 339\u001b[0m \u001b[0mtyp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0merror\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 340\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwith_traceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 341\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 342\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py\u001b[0m in \u001b[0;36mf\u001b[0;34m()\u001b[0m\n\u001b[1;32m 322\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mcallback_timeout\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 323\u001b[0m \u001b[0mfuture\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0masyncio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait_for\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfuture\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback_timeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 324\u001b[0;31m \u001b[0mresult\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32myield\u001b[0m \u001b[0mfuture\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 325\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 326\u001b[0m \u001b[0merror\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexc_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/srv/conda/envs/notebook/lib/python3.8/site-packages/tornado/gen.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 760\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 761\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 762\u001b[0;31m \u001b[0mvalue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfuture\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 763\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 764\u001b[0m \u001b[0mexc_info\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexc_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py\u001b[0m in \u001b[0;36m_gather\u001b[0;34m(self, futures, errors, direct, local_worker)\u001b[0m\n\u001b[1;32m 1848\u001b[0m \u001b[0mexc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCancelledError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1849\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1850\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mexception\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwith_traceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtraceback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1851\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1852\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0merrors\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"skip\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mKilledWorker\u001b[0m: (\"('harmonic_analysis-along-axis-8eca5ecff77faf95613d6a3504f7364e', 0, 1, 0)\", <Worker 'tls://10.49.132.7:35617', name: dask-worker-aafb9ced5a9d4d4a95b7185d85f06478-cqdpq, memory: 0, processing: 1>)"
]
}
],
"source": [
"analysis = future.compute()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The result tensor is arranged in order to place the estimated tidal waves on the first dimension."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"analysis = np.transpose(analysis, [2, 0, 1])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Visualization of the amplitude and phase calculated for the M2 wave."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The tide present in the model is now calculated to correct it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"amp = np.absolute(analysis[0, :, :])\n",
"pha = np.angle(analysis[0, :, :], deg=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"fig, (ax_pha, ax_amp) = plt.subplots(1, 2, figsize=(24, 8))\n",
"\n",
"c = ax_pha.pcolormesh(pha, vmin=-180, vmax=180, cmap='bwr')\n",
"fig.colorbar(c, ax=ax_pha)\n",
"ax_pha.set_title(\"M2 phase (deg)\")\n",
"\n",
"c = ax_amp.pcolormesh(amp, vmin=-0.15, vmax=2, cmap='jet')\n",
"fig.colorbar(c, ax=ax_amp)\n",
"ax_amp.set_title(\"M2 amplitude (m)\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"del amp\n",
"del pha"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"nwaves, ni, nj = analysis.shape\n",
"tide = wave_table.tide_from_mapping(\n",
" time_series[0].astype('datetime64[s]').astype('float64'),\n",
" analysis.reshape(nwaves, ni*nj)).reshape(ni, nj)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"total_ssh = ssh.Eta.sel(face=face, time=time_series[0])[indices, indices]\n",
"corrected_ssh = total_ssh - tide"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Figures of the analysis performed."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(18, 6))\n",
"\n",
"ax1.pcolormesh(total_ssh, cmap='jet')\n",
"ax1.set_title(\"Total SSH\")\n",
"ax2.pcolormesh(corrected_ssh, cmap='jet')\n",
"ax2.set_title(\"SSH corrected\")\n",
"ax3.pcolormesh(tide, cmap='jet')\n",
"ax3.set_title(\"Tide estimated\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment