ClimateDT Workflow: A containerized climate workflow

The Climate Adaptation Digital Twin (ClimateDT), a contract (DE_340) inside the Destination Earth (DestinE) flagship initiative from the European Commission, is a highly collaborative project where climate models are executed in an operational manner on different EuroHPC platforms. The workflow software supporting such executions, called ClimateDT Workflow, contains a model component and an applications component. The applications can be seen as elements that consume the data that is provided by the climate models. They aim to provide climate information to sectors that are critically dependent on climate change, such as renewable energy or wildfires, among others. This workflow relies on the Autosubmit workflow manager and is executed over different EuroHPC platforms that are part of the contract.

There are six lightweight applications that are run in this workflow, in parallel to the model and in a streaming fashion. Setting up and maintaining an environment for these applications for each EuroHPC platform (plus the development environments) is a time-consuming and cumbersome task. These machines are shared by multiple users, have different operating systems and libraries, some do not have internet access for all users on their login nodes, and there are different rules to install and maintain software on each machine.

In order to overcome these difficulties all the application-required dependencies of the workflow are encapsulated beforehand in a Singularity container and therefore the portability to the different platforms becomes merely an issue with path-binding inside the platform. Through the use of Singularity containers, their execution does not require administrator permissions, which allows anyone with access to the project to execute the desired application either on the EuroHPC machines, or on their local development environment.

This work shows the structure of the ClimateDT workflow and how it uses Singularity containers, how they contribute not only to portability but also to traceability and provenance, and finally the benefits and issues found during its implementation. We believe that the successful use of containers in this climate workflow, where applications run in parallel to the climate models in a streaming fashion and where the complete workflow runs on different HPC platforms, presents a good reference for other projects and workflows that must be platform-agnostic and that require agile portability of their components.