A systematic methodology for directive-based GPU porting: NEMO ocean model as a case study

The constant growth in computational demands of scientific applications, combined with energy efficiency requirements, makes GPU acceleration an important factor to consider in high-performance computing environments. Therefore, GPU porting has become essential for efficiently utilizing modern heterogeneous systems that currently provide both multicore CPUs and GPU-accelerated partitions.

While small, relatively new projects might be candidates for complete rewrites in low-level GPU languages like CUDA or HIP, this approach becomes impractical for larger and more complex codebases. Thus, a more convenient way is provided by the directive-based approach, which allows developers to maintain their original C++ or Fortran code while adding extra OpenACC/OpenMP directives to generate energy-efficient GPU code. 

However, this seemingly straightforward method often presents significant challenges. For instance, dealing with code that employs layouts that are not well-suited for GPU architectures, such as deeply nested loop structures or complex memory access patterns that result in suboptimal performance, might lead to the need to reorganize the initial code after all.

In this work, we present a systematic approach to performing the GPU code transition through compiler directives. This several-step incremental process seeks to reach a significant performance and energy consumption improvement while preserving code maintainability, portability, and output accuracy. We demonstrate the effectiveness of our approach through a detailed case study of our ongoing project porting the NEMO ocean model, which represents an interesting example of a complex scientific Fortran code with numerous common computational patterns. Finally, we discuss the experiences, limitations, and trade-offs encountered during this process, providing useful insights for other porting efforts that could face similar GPU migration challenges.