Enabling reliable workflow development with an advanced Testing Suite

Climate simulations require complex workflows that often integrate multiple components and different configurations per experiment, typically involving high-performance computing resources. The exhaustive testing required for these workflows can be time and resource consuming, presenting significant challenges in terms of computational cost and human effort. However, robust Continuous Integration (CI) testing ensures the reliability and reproducibility of such complex workflows by validating the codebase and ensuring the integrity of all the components used when performing climate simulations. Additionally, CI testing facilitates both major and minor releases, enhancing the efficiency of the development lifecycle.

To address these challenges, we present our Testing Suite software, designed to automate the setup, configuration, and execution of integration tests using Autosubmit, a workflow manager developed at the BSC. Autosubmit is typically used for climate modelling experiments, but also atmospheric composition ones, and also constitutes the backbone of some operational systems and Digital Twin initiatives. The Testing Suite software allows Autosubmit commands to be executed in batches and the responses from the Workflow Manager to be bypassed in a structured manner. By streamlining this process, it minimizes the effort required for exhaustive testing while ensuring reliability.

Beyond integration testing, the Testing Suite offers advanced capabilities for scientific result verification. By automatically comparing output data bit by bit, it swiftly detects regressions during test execution. Additionally, it provides CPMIP performance metrics, offering insights into the efficiency of the workflows.

As a result, the Testing Suite plays an important role in quality assurance, particularly during releases, where extensive testing ensures the workflow meets required functionality and performance standards across different configurations. These integration tests act as a checkpoint, validating the stability and robustness of the software before release. They also identify stable points in the main codebase, enabling developers to create new branches with confidence. This approach minimizes compatibility issues and facilitates a smoother development process.

In conclusion, the Testing Suite is a crucial part of the development lifecycle for climate simulations. It mitigates risks, ensures stability, and fosters innovation, all while maintaining a robust and reliable foundation for scientific research and development.