I/O Profiling and Optimisation to Improve Energy Efficiency: Insights from the Climate Digital Twin project

The continuous increase in resolution and complexity of Earth System Models (ESMs), aimed at improving the accuracy of simulations, significantly increases data handling demands. For this reason, scalable and efficient I/O solutions are critical to ensure that data storage, processing, and transfer do not become bottlenecks that hinder overall simulation throughput. Optimising this workflow is essential not only for improving performance but also for reducing the energy footprint of large-scale Earth system simulations.

Many ESMs, including the IFS-NEMO coupled model used in our case study within the Destination Earth Climate Digital Twin project, adopt a client-server I/O architecture to address these challenges. In this scheme, the model sends generated data to a server that handles complex post-processing tasks, such as interpolation, encoding, and data writing, while the model continues simulating the next time steps. However, this approach also requires continuous optimisation to ensure the overall efficiency of the output pipeline.

To achieve this, detailed I/O profiling was conducted, with a focus on the MultIO library, which manages the output pipeline, including ocean server creation and ocean data transport. On the client side, frequent and costly access to metadata was identified as a major source of I/O overhead, while on the server side, high interpolation times were observed, prompting further analysis and targeted optimisations that achieved a sixfold speedup in ocean interpolation. Additional pipeline actions were reviewed and optimised, contributing to a more efficient and scalable output workflow. Combined with tests to optimise server resource allocation, these efforts resulted in an overall efficiency improvement of up to 6.7% in simulation performance.