Tree-Based Adaptive Data Reduction Techniques for Scientific Simulation Data

Large-scale Earth system model simulations produce huge amounts of data. Due to limited I/O bandwidth and available storage space this data often needs to be reduced before written to disk or stored permanently. Error-bounded lossy compression is an effective approach to tackle the trade-off between accuracy and storage space.

We are exploring and discussing lossless as well as error-bounded lossy compression based on tree-based adaptive mesh refinement/coarsening (AMR) techniques. Our lossy compression schemes allow for absolute and relative error bounds. The data reduction methods are closely linked to an underlying (adaptive) mesh which easily permits error regions of different error tolerances and criteria – in particular, we allow nested domains of varying error tolerances specified by the user. Moreover, some of the compressed data structures allow for an incremental decompression in the resolution of the data which may be favorable for transmission and visualization.

We implement these techniques as the open source tool cmc, which is based on the parallel AMR library t8code. The compression tool can be linked to and used by arbitrary simulation applications or executed as a post-processing step. We show different application results of the compression in comparison to current state-of-the-art compression techniques on several benchmark data sets.