Error bounded compression for weather and climate applications

Weather and climate simulations produce petabytes of high-resolution data that are later analyzed by researchers to understand climate change or severe weather. We propose a new error-bounded compression method targeting for weather and climate data. It contains a JPEG2000 compression layer to capture the bulk part of the data, and a sparse wavelet layer to record the sparse signal that excess the given error bound. The sparse wavelet layer encodes the wavelet coefficients using the SPIHT algorithm. We test our method with established compression methods on a suite of benchmarks including basic statistics, case study of the hurricane data, derived variable computation, and Lagrangian trajectory simulation. Our method is favourable in most benchmark cases at given range relative error targets from 0.1% to 10% achieving compression ratios from 10x to more than 800x. It can reconstruct the derivatives of the compressed data with a best fidelity and does not add high frequency artifacts found in other compression methods. In Lagrangian trajectory simulations, our method can produce less distortion in trajectories and distribution of particles compared with SZ3. We are able to produce a 16x compressed wind data achieving less error metric than adding 5% random noise to the data, making it ready for practical use.