Earth System Sciences (ESS) datasets are continuing to increase in terms of resolution and size. They are crucial to support critical activities such as climate change policymaking and weather forecasting. The storage and shareability of these datasets have become a pressing concern within the scientific community. State-of-the-art applications and ongoing and upcoming community initiatives, such as digital twins and the CMIP7, produce outputs so large that even the most advanced data centres and infrastructures struggle not only to store them but also to ensure their usability and processability, including by downstream machine learning applications. Similarly, the Copernicus program’s archive of satellite observations will grow exponentially in the coming years. With future investment in hardware likely to remain constrained, data volumes are increasingly restricting scientific progress. An essential and viable way forward is to explore data compression that balances efficiency with the needs of diverse stakeholders. Nevertheless, many researchers remain cautious about lossy compression, concerned that critical information or features may be lost for specific downstream use cases. Identifying these requirements and ensuring they are preserved during compression are essential steps toward building trust so that compression can become widely adopted across the community.
This session will present and discuss recent advances in and concerns about data compression for ESS datasets, focusing on:
1) Advances in and reviews of methods, including classical, learning-based, and hybrid approaches, with attention to computational efficiency of compression and decompression
2) Inter-disciplinary case studies of using compressed data in scientific workflows, both with encouraging and cautionary conclusions
3) Methods and framework to evaluate datasets and data infrastructure
4) Understanding the domain- and use-case-specific requirements for safe and performant compression, and developing methods that provide these guarantees for lossy compression
5) Exploration of novel workflows enabled by machine learning that reduce pressure on disk space, e.g. through compressing input data into lower-dimensional latent spaces
This session aims to address the concerns shared by the community during EGU 2026, and will conclude with a debate about how data producers, compression researchers, and domain scientists can co-steer the future development of compression for ESS.