Cloud-based orchestration of the biogeochemical forecasting system for Italian seas within the MER project

We present the biogeochemical forecasting system developed within the MER (Marine Ecosystem Restoration) project (Actions B32-B35). This case study leverages cloud-based workflow orchestration and traditional HPC systems to deliver daily operational marine biogeochemistry forecasts for Italian seas, as a downscaling of the Copernicus Marine Service (CMS).

The basins are divided into 7 regional high-resolution domains at ~500 m resolution and further 10 selected very high-resolution nested sites at ~100 m resolution. The downscaling pipelines we implemented are responsible for retrieving heterogeneous input data from multiple third-party sources (CMS, EFAS, ItaliaMeteo, ECMWF), their preprocessing to feed the MITgcm-BFM coupled physical-biogeochemical model, the postprocessing of the outputs and the publication of the final products. The implementation further provides observability, failsafes and fallbacks in case of missing data, and notifications regarding the status of operations.

Such a complex operational oceanographic system faces competing requirements: on one hand, computationally intensive numerical simulations demand HPC resources. On the other, the orchestration of several interdependent extract-transform-load workflows whilst guaranteeing monitoring and observability require capable management systems. These are often incompatible with HPC cluster policies (e.g., length of standing processes, security, …) and better suited for a cloud environment. On top of this, care must be taken to manage large volumes of data between the orchestrator and the HPC cluster.

We address these competing requirements through a hybrid architecture that combines cloud computing with HPC infrastructures for workflow orchestration and compute-intensive simulations, respectively. Our system, rewritten following software engineering best practices (modular architecture, separation of concerns, CI testing, …), employs Apache Airflow as the workflow manager, deployed in a fully containerised fashion on CINECA's OpenStack-based cloud infrastructure. A custom integration layer allows interfacing with the Slurm workload manager, offloading computationally intensive tasks onto CINECA's Leonardo HPC cluster. Parallel computing and distributed filesystems are efficiently exploited through modern technologies, particularly the cloud-native Zarr data format in conjunction with xarray and dask as Python-based numerical computing libraries.

Our setup demonstrates the viability of hybrid cloud-HPC architectures for operational Earth system modelling. It meets efficiency and scalability goals that would be challenging with either infrastructure alone. The software is planned to be open-sourced in the second half of 2026.

This work is developed by eXact lab Srl in partnership with OGS and CINECA within the MER project, led by ISPRA, funded by the NextGenerationEU program (Italian National Recovery and Resilience Plan, investment M2C4 ‐ I3.5).