Compression Safeguards - Towards Safe and Fearless Lossy Compression of Earth System Data

The volume of data produced by Earth System Science models, e.g. high-resolution weather and climate models, is increasing faster than the methods and budgets for storing, sharing, and analysing this data. To reduce data sizes, lossy data compression methods discard some quality, details, or precision of the original data. Even though some lossy compressors promise size reductions of 100x or more, the lack of trust in lossy compression, rooted in the fear of losing important information, has so far limited their adoption.

We introduce compression safeguards to help overcome this trust gap by

(i) enabling scientist users to precisely express their (general or specific) safety requirements for lossy compression, e.g. preserving specific values, regionally varying error bounds on the data or quantities derived from it, or any logical combination thereof,

(ii) securing any (existing) (lossy) compressor with the corresponding safeguards, which then

(iii) guarantee that the safety requirements are always met by the safeguarded compressor.

Compression safeguards thus provide a unified and flexible interface for specifying and guaranteeing user safety requirements that works with any existing compressor. They therefore shift the burden of trust in fulfilling these requirements away from specific compressor implementations. With the appropriate safeguards, even untrusted, potentially unsafe compressors can be used safely and without fear. We hope that compression safeguards will provide Earth System scientists with the guarantees to use lossy compression safely and without fear, thereby helping to unlock the benefits of lossy compression in reducing data volumes for the Earth System Science community.

We will showcase how our reference implementation, compression-safeguards (https://compression-safeguards.readthedocs.io/en/latest/), can be applied to safeguard important properties in several real-world meteorological examples, evaluate the impact on compression ratio (only low for sparse corrections) and computational load at compression (major) and decompression (negligible) time, and discuss the future pathway towards safe and fearless lossy compression.