MultIO: Message-Driven Data Routing for Distributed Earth-System Models

Traditionally, in numerical weather prediction, the computational cost of performing floating-point operations (flops) has been the primary concern. However, in the past couple of decades throughput to storage has become a significant bottleneck – a phenomenon often referred to as the input/output (I/O) performance gap. ECMWF runs time-critical operational weather forecasts four times a day, where the entire workflow of each run must complete within one hour. A single run currently produces around 30TiB of data, and it is expected to increase to hundreds of TiB within the next five-to-ten years. This is impossible on existing infrastructure without re-organising how model data is handled and processed. 

The proposed solution to this problem in the field of weather and climate numerical models has two elements. One is to decouple data output from numerical computations and dedicate pre-defined processes, called I/O-servers, purely to data output. The other is to move computations of derived (post-processed) data closer to the original “raw” weather data, thus reducing the amount of data to be moved. The challenge then is how to route the combination of raw and post-processed data efficiently to storage without compromising the performance of the running model. 

We present MultIO, an open-source software library developed at ECMWF for data routing from distributed parallel meteorological and earth-system models. It supports two distinct functionalities. First, it allows the creation of post-processing pipelines to calculate derived meteorological products, such as temporal pointwise statistics, interpolation onto different grids, encoding of data into output formats and output of data storage systems or other consumers. Second, it can act as an I/O-server, creating aggregated horizontal fields from distributed parallel meteorological and earth-system models. 

MultIO is a key component of the ACROSS project, funded by the EuroHPC JU. It is also partly developed via ECMWF's participation in Destination Earth and is a component of the Digital Twin Engine (DTE).