Latest Developments in Probtest: Probabilistic Testing for Robust CPU/GPU Validation of Scientific Models

Porting large numerical models to heterogeneous computing architectures introduces significant challenges for software validation and testing, as results from CPU- and GPU-based executions are typically not bit-identical. These differences arise from variations in floating-point arithmetic, execution order, and the use of architecture-specific mathematical libraries. Traditional regression testing approaches based on exact reproducibility therefore become inadequate, particularly in continuous integration (CI) workflows.

Probtest is a lightweight testing framework developed to address this problem in the ICON numerical weather and climate model. It implements a probabilistic, tolerance-based testing strategy that enables robust numerical consistency checks between CPU and GPU runs while remaining fast and resource-efficient. Tolerances are derived from ensembles generated by perturbing prognostic variables in the initial conditions. From a larger ensemble of CPU reference runs, a representative subset is selected to compute variable-specific tolerance ranges that define acceptable numerical deviations. This approach allows reliable validation across architectures without constraining model development or optimization.

Recent developments focus on improving extensibility, usability, and reproducibility. Support for Feedback Output Files (FOF) has been added, enabling consistency checks for observation-based diagnostics in addition to model state variables. Furthermore, Probtest has been fully containerized, with each release published on Docker Hub. This removes local installation barriers, ensures reproducible testing environments, and simplifies integration into CI pipelines and collaborative development workflows. These developments strengthen Probtest as a practical and portable tool for validating ICON across heterogeneous computing platforms.