MPI and OpenMP coherence testing and vaildation: the hybris of testing non-deterministic model code

With the advent of parallel programming in the late 1990s. A port of the than available Max Planck Institutes for Meteorology spectral atmospheric model echam5 to MPI and OpenMP was done. For testing and validation of the hybrid parallelization a coherence algorithm was developed. The implementation has been incorporated into todays NWP and climate model ICON as well. The coherence algoritm consists of several stages: first one MPI rank is running the serial model against an n-task MPI parallelized model. During runtime the state vector is checked for binary-identity. If successfull a m-task MPI version can be compared to an m-task MPI version for high processor counts. The same schema can be used OpenMP parallelization. ONe MPI task runs the model serial using one OpenMP thread and a second MPI task runs k OpenMP threads. Again, the results are compared for binary-identity. As the testing needs to be done automatically, bit-identity is important for testing not necessarily for production.

The tesing revealed plenty of problems during the initial parallelization work of echam5 and showed constant appearing problems in the ICON development phase.

However, far in a couple of century long simulation the bit-identity was just by accident found to be broken: the search of the cause started!