Benchmark of the high-resolution Nemo-SI3-XIOS configuration SEDNA on an ARM-based HPC system, Fugaku.

We have developed the ocean-sea ice Arctic regional configuration SEDNA (Sea ice - EDdy resolving ocean paN-Arctic configuration) at ultra-high resolution (800 m in the horizontal and 150 vertical levels) based on the NEMO-SI3 numerical platform, in order to investigate how the dynamics of mesoscale turbulent eddies in the Arctic Ocean interplay with sea ice. 

The configuration was initially developed on the AMD-based HPC system Joliot-Curie ROME based at CEA in France, which has achieved a Linpack performance of 12 PFlop/s ranked number 33 of June 2020 TOP500 list (https://top500.org/system/179700/).

Thanks to a European PRACE allocation of nearly 40 million CPU hours, we were able to run a 8 year-long simulation. Although promising to understand part of the small-scale dynamics at play, this length of simulation will likely be a limiting factor in the investigation of the eddy dynamics which is known to equilibrate over several decades.

To overcome this limitation, here we investigate the feasibility of running SEDNA over several decades but at a realistic time cost. To that aim, a benchmark has been performed on the ARM based HPC system Fugaku based at RIKEN in Japan, which has archived a Linpack performance of 442 PFlop/s ranked number 1 of November 2020 TOP500 list (https://www.top500.org/system/179807/). Such a benchmark has required adaptation of the compiling and placements of NEMO and XIOS MPI processes in order to fit on the Fugaku architecture compared to standard X86_64 based HPC systems like Joliot-Curie ROME. In this presentation we will share the tips and lessons learned from our benchmarks and will report the benchmark results.  Our insights on model MPI placements for efficient post processing of huge models will be discussed as well.