The data streaming in the Climate Adaptation Digital Twin: a fundamental piece to transform climate data into climate information

In the context of advancing towards high resolution climate projections (1km, sub-hourly) and the consequently large memory requirements, we are reaching the point that not all of the data produced can be stored. In this work, we present the technical infrastructure developed in the context of the Destination Earth ClimateDT project, in order to consume the data produced by the core engines as soon as it is available,  a method known as “data streaming”. This mechanism consists of three main steps that are included in an integrated workflow: the run of the climate models themselves , the applications (which convert the model output to actionable information) and the mechanism that links both sides. This solution is designed to be scalable; different applications can be run simultaneously and with as many different variables and statistics as needed,  in order to fully utilize the output  from the digital twin. The flexibility of the workflow allows different applications to run at their optimal frequency in a seamless way. Last but not least,  the workflow integrates statistical streaming   algorithms, allowing integrated applications to generate on-demand online statistics from streamed data, minimizing the memory footprint.