Generic State Vector: streaming and accessing high resolution climate data from models to end users

The Climate Adaptation Digital Twin (ClimateDT) is a contract under the Destination Earth initiative (DestinE) that aims to develop a digital twin to account for climate change adaptation. This is achieved by running high-resolution simulations with different climate models by making use of the different EuroHPC platforms. In addition to the climate models, applications that consume data from models are also developed under the contract. A common workflow is used to execute the whole pipeline from the model launching to the data consumption by the applications in a user-friendly and automated way.

One of the challenges of this complex workflow is to handle the different outputs that each of the climate models initially offered. Each model works with its own grid, vertical levels, and variable set. These differences in format make it very complicated for applications to consume and compare data coming from different models in an automated and timely manner. This issue is resolved by introducing the concept of Generic State Vector (GSV), which defines a common output portfolio for all models to ensure a homogeneous output between models. The conversion from the model's native output to the GSV happens before the data is written in the HPC and it is automated in the workflow allowing transparent access to the data changing only the name of the model in the call.

Data in the GSV format can be read using a newly designed dedicated Python tool: the GSV Interface. This tool links the model part of the workflow with the applications part of the workflow, enabling running everything in a single complex workflow (end-to-end workflow). The GSV Interface allows to read data that has been previously converted to GSV, adding proper metadata. It also offers some extra features like interpolation to regular grids and area selection. All the workflow components that read data from the models rely on the GSV Interface. In addition to that, the GSV Interface can also be used to transparently retrieve and process data from the public Destination Earth Service Platform.