Optimizations of Multiscale Simulation with AGRIF, towards Exascale Applications

AGRIF (Adaptive Grid Refinement In Fortran) is a package for the integration of full adaptive mesh refinement features within a multidimensional finite difference model. This library is used in ocean models like NEMO (Nucleus for European Modelling of the Ocean) to offer the possibility to run multiple levels and 2-way nested embedded zooms. Within the ESiWACE2 project, AGRIF performance have been addressed toward high resolution simulations. 
First, a simple AGRIF configuration within NEMO has been set up to simplify benchmarking, profiling and testing new optimizations ideas. We selected on purpose a configuration with small MPI sudomains to mimic simulations running on high numbers of core. Second, a profiling analysis has let us identify an important overhead. Indeed, on a zoom with a refinement of a factor 3 in both latitude and longitude covering 1/9 of the simulated domain an overhead of 46% has been observed compared with the theoretical performance. The correction of land points used in the interpolation on the zoom has been found to be a major bottlenecks. Third, we implemented an optimization concerning the correction on land point limiting as much as possible the computations and taking advantage of the specificity of each interpolation. This adjustment provided us with a reduction of 25% of the time to solution in the aforementioned configuration. For future work, we identified numerous optimizations including further optimizations of the correction of land points.