Scaling Down ESS Datasets: Lessons from the EERIE Project on Compression

The rapid growth of Earth System Sciences (ESS) datasets, driven by high-resolution numerical modeling, has outpaced storage and data-sharing capabilities. To address these challenges, we investigated lossy compression techniques as part of the EERIE project, aiming to significantly reduce storage demands while maintaining the scientific validity of critical diagnostics.

Our study examined two key diagnostics: Sea Surface Height (SSH) variability and ocean density, essential for understanding climate dynamics. Leveraging tools such as SZ3 and enstools-compression, we achieved data volume reductions by orders of magnitude without compromising the diagnostics' accuracy. Compression-induced differences were found to be negligible compared to the inherent variability between model outputs and observational datasets, underscoring the robustness of these methods.

Additionally, our work highlighted inefficiencies in current workflows, including the prevalent use of double precision in post-processing. We proposed improvements to align data precision with the original model outputs, further optimizing storage and computation. Integrating lossy compression into existing workflows via widely used formats like NetCDF and HDF5 demonstrates a practical path forward for sustainable ESS data management.

This study showcases the transformative potential of lossy compression to make high-resolution datasets more manageable, ensuring they remain accessible and scientifically reliable for stakeholders while significantly reducing resource demands.