Adaptive Data Reduction Techniques for Extreme-Scale Atmospheric Models

Large-scale earth system simulations produce huge amounts of data. Due to limited I/O bandwidth and available storage space this data often needs to be reduced before writing to disk or storing permanently. Error-bounded lossy compression is an effective approach to tackle the trade-off between accuracy and storage space.

We are exploring and discussing error-bounded lossy compression based on tree-based adaptive mesh refinement (AMR) techniques. According to flexible error-criteria the simulation data is coarsened until a given error bound is reached. This reduces the number of mesh elements and data points significantly.

The error criterion may for example be an absolute or relative point-wise error. Since the compression method is closely linked to the mesh we can additionally incorporate geometry information - for example varying the error by geospatial region.

We implement these techniques as the open source tool cmc, which is based on the parallel AMR library t8code. The compression tool can be linked to and used by arbitrary simulation applications or executed as a post-processing step. As a first example, we couple our compressor with the MESSy and MPTRAC libraries.

We show different results including the compression of ERA5 data. The compressed sample datasets show better results in terms of file size than conventional compressors such as SZ and ZFP. In addition, our method allows for a more fine-grained error control.