Coupling km-Scale Earth System Model to Hierarchical Output for Analysis-Ready Dataset

Kilometer-scale Earth System Model (ESM) simulations increasingly generate petabyte-scale datasets. The scientific return from such datasets still remains constrained by their accessibility and heterogenity as well as the cost of their downstream analysis. Analysts often rely on ad-hoc workflows and even analyses on reduced datasets require repeated access to high-resolution data, limiting scalability. We present Hiopy (Hierarchical Output in Python), a tool for generating cloud accessible, analysis-ready dataset directly from a km-scale ESM simulation using the ICON model by computing hierarchical temporal and spatial aggregations in situ. Building on the work of Kölling et al. (2024, EGU), Hiopy produces multi-resolution, self-describing datasets that enable seamless access from coarse to native resolution using the Zarr format. To mitigate the computational and communication overhead of in-situ aggregations, Hiopy uses YAC (Yet Another Coupler) to couple the model to the output component and configures the aggregates such that the model’s domain decomposition and the prefered Zarr chunking are aligned for even distribution of workload across the output processes. As a result of this, the communication overhead is reduced and efficient parallel computation is possible without penalising the simulation throughput. Additional optimisations reduce communication buffers, eliminate redundant duplications in metadata handling, allows streaming the data directly to its final location and eases configuration for varying requirements. Hiopy supports native ICON model grids, regular latitude–longitude grids and the HEALPix grid and has been validated by producing publicly accessible datasets from the km-scale ESM simulations across multiple projects. This work demonstrates a practical tool in the software stack of high resolution climate modelling.