Understanding the effects of Modern Lossless and Lossy Compressors on the Community Earth Science Model

Large scale climate simulations such as the Community Earth Science Model (CESM) produce enormous volumes of data per run. Transferring and storing this volume of data can be challenging leading researchers to consider data compression in order to mitigate the performance, monetary and environmental costs. In this work, we survey 8 methods ranging from higher-order SVD, multigrid, transform, and prediction based lossy compressors as well as specialized floating point lossless and lossy compressors and general lossless compressors to determine which methods are most effective at reducing the storage footprint.  We consider four components (atmosphere, ice, land, and ocean) within CESM taking into account the stringent quality thresholds required to preserve the integrity of climate research data. Our work goes beyond existing studies of compressor performance by considering these newer compression techniques, and by accounting for the candidate quality thresholds identified in prior work by Hammerling et al.  This provides a more realistic picture of the performance of lossy compression methods relative to lossless compression methods subject to each of these constraints with up to a 5.2x improvement over the leading lossless compressor and 21x over no compression. Our work features an automated method to automatically identify a configuration that satisfies the quality requirements for the lossy compressors that is agnostic to compressor implementations.