Workflows for numerical reproducibility in the OceanVar data assimilation model

Numerical reproducibility is a crucial yet often overlooked challenge in ensuring the credibility of computational results and the validity of Earth system models. In large-scale, massively parallel simulations, achieving numerical reproducibility is complicated by factors such as heterogeneous HPC architectures, floating point intricacies, complex hardware/software dependencies, and the non-deterministic nature of parallel execution.

This work addresses the challenges of debugging and ensuring bitwise reproducibility (BR) in parallel simulations, specifically for the MPI-parallelised OceanVar data assimilation model. We explore methods for detecting and resolving BR-related bugs, focusing on an automated debugging process. Currently mature tools to automate this process are lacking for bugs due to MPI-parallelisation, making automatic BR verification in scientific workflows involving such codebases a time-consuming challenge.

However, BR is sometimes considered unrealistic in workflows involving heterogeneous computing architectures. As an alternative, statistical reproducibility (SR) is proposed and explored by various research groups in the Earth system modelling community, for which automated tools have been developed. For example, the scientific workflow of CESM supports automatic verification of SR using the CESM-ECT framework/PyCECT software. In case of failure of SR a root-cause analysis tool exists, CESM-RUANDA, albeit currently not fully functional. We explore SR as an alternative and complementary approach to of BR focusing on its potential to support numerical reproducibility in workflows involving heterogeneous computing architectures.