Benchmark for the performances on the same mesh with different models (TELEMAC/SCHISM) and resources (EuroHPC / Azure)

benchmark_seareport.md

Here global-v2 is the currently used mesh on EuroHPC Meluxina, that uses the L5 GSHHS coastline (with the ice shelf).

MODEL SPECS:

• Mesh: 3.539.584 nodes (6.898.577 elements)
• time span: 1 month
• model type: surge only

COMPUTE SPECS:

• Meluxina (EuroHPC): 128 cores (1024GB ram)
• HPC Azure PoC1: 32 cores (TEST1 6/3/2024)
• HPC Azure PoC2: 120 cores (TEST2 22/3/2024) Standard_HB120rs_v3

Test	Python Pre-processing	Domain Decomposition	Computation	Merging Files
global-v2 + Telemac on Meluxina	1h15min	57 min	55 min	15 min
global-v2 + Telemac on Azure (PoC1)	1h20min	1h53min	1h40min	29min
global-v2 + Telemac on Azure (PoC2)	1h20min	7h50min	43min	1h24min
global-v2 + Schism on Meluxina	TBD	TBD	TBD	TBD
global-v2 + Schism on Azure	TBD	TBD	TBD	TBD

Python pre-processing breakdown:

• Extracting, processing, adjusting DEM and simplifying coastlines: 8 minutes
• Detailed processing, assembling dataset, setting NaNs, reading mesh: 17 minutes
• Interpolating on mesh, adjustments, and preparing for output: 2.9 minutes
• Saving geometry, meteo, boundary, and CAS files: 48.4 minutes
• Setting observation locations: 13.973 seconds

Remarks on the results:

The domain decomposition of the wind input file (which is a sequential process) in TELEMAC becomes so long that it ruins the walltime of the whole computation.

There are a few solutions to make the domain decomposition faster:

option 1

support a parallel partitioner (like PARMETIS). But so far it was not planned for the upcoming developments in the TELEMAC. I opened a ticket on the forum for more details.

option 2

split the domain without the wind and then within the python processes assign to each subdomain the wind conditions.
The steps to do so would be the following:

telemac.2d.py t2d_cas_file.cas --ncsize 128 --split
*add the wind info in the cas file*
*generate atm forcing for each subdomain*
telemac2d.py --run

To save more time for the export we could also, instead of merging, convert directly the output selafin of each subdomain into zarr chunks

Option 2 for sure - it has been a long time my idea to only split the GEO and nothing more. Once this is done, just copy the values on the split for the other BINARY files. And all in Python. And in fact, we should be able to provide a split (from an old simulation split for exemple), for TELEMAC to copy it for all files including the new GEO.
There is optimisation to do on the merge as well.
Note that you can already do the split once - and then re-run using --split and --run given a fixed directory.

Ideally though, you would want not to have to merge, and rather deal with multiple files natively.

Do you have an update on SCHISM ? or is it too good to put TELEMAC to the shame ? ;-)