EGU21-4771, updated on 04 Mar 2021
EGU General Assembly 2021
© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

The NICAM 3.5km-1024 ensemble simulation: Performance optimization and scalability of NICAM-LETKF on supercomputer Fugaku

Hisashi Yashiro1,2, Koji Terasaki2, Yuta Kawai2, Shuhei Kudo2, Takemasa Miyoshi2, Toshiyuki Imamura2, Kazuo Minami2, Masuo Nakano3, Chihiro Kodama3, Masaki Satoh4, and Hirofumi Tomita2
Hisashi Yashiro et al.
  • 1National Institute for Environmental Studies, Tsukuba, Japan
  • 2RIKEN Center for Computational Science, Kobe, Japan
  • 3JAMSTEC, Yokohama, Japan
  • 4Atmosphere and Ocean Research Institute, The University of Tokyo, Kashiwa, Japan

In parallel with the new Japanese flagship supercomputer, Fugaku, we have continued improving a nonhydrostatic icosahedral atmospheric model (NICAM). Here, we introduce the results of our system-application co-design since 2014. Fugaku's CPU (A64FX) is based on the Arm instruction-set architecture. This 48-core many-core CPU is equipped with 32GB of HBM2 memory, showing data transfer performance comparable to GPUs. We have implemented kernel-level optimizations to take advantage of Fugaku's high memory performance. Among them, we recognized trade-offs related to ensuring memory locality and parallelism, and register allocation. We improved the application's average arithmetic intensity through detailed loop-by-loop performance measurements and reduced memory pressure by actively using single-precision operations. We also redesigned the data layout and the file I/O component of the ensemble data assimilation (DA) system and achieved good scalability in the atmospheric simulation and DA. We performed a global 3.5km mesh, 1024-member ensemble simulation, and DA using 82% of the Fugaku system (131,072 nodes, 6,291,456 cores). In this world's most massive ensemble DA benchmark experiment, the simulation and the DA achieved 29 PFLOPS and 79 PFLOPS of effective performance.

How to cite: Yashiro, H., Terasaki, K., Kawai, Y., Kudo, S., Miyoshi, T., Imamura, T., Minami, K., Nakano, M., Kodama, C., Satoh, M., and Tomita, H.: The NICAM 3.5km-1024 ensemble simulation: Performance optimization and scalability of NICAM-LETKF on supercomputer Fugaku, EGU General Assembly 2021, online, 19–30 Apr 2021, EGU21-4771,, 2021.

Corresponding displays formerly uploaded have been withdrawn.