EGU22-9153
https://doi.org/10.5194/egusphere-egu22-9153
EGU General Assembly 2022
© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Exploring the SZ lossy compressor use for the XIOS I/O server

Xavier Yepes-Arbós1, Sheng Di2, Kim Serradell1, Franck Cappello2, and Mario C. Acosta1
Xavier Yepes-Arbós et al.
  • 1Barcelona Supercomputing Center, Earth Sciences Department, Barcelona, Spain
  • 2Argonne National Laboratory, Mathematics and Computer Science Division, Lemont, IL, USA

Earth system models (ESMs) have increased the spatial resolution to achieve more accurate solutions. As a consequence, the number of grid points increases dramatically, so an enormous amount of data is produced as simulation results. In addition, if ESMs manage to take advantage of the upcoming exascale computing power, their current data management system will become a bottleneck as the data production will grow exponentially.

The XML Input/Output Server (XIOS) is an MPI parallel I/O server designed for ESMs to efficiently post-process data inline as well as read and write data in NetCDF4 format. Although it offers a good performance in terms of computational efficiency for current resolutions, this could change for larger resolutions since the XIOS performance is very dependent on the output size. To address this problem we test the HDF5 compression in order to reduce the size of the data so that both I/O time and storage footprint can be improved. However, the default lossless compression filter of HDF5 does not provide a good trade-off between size reduction and computational cost. 

Alternatively, we consider using lossy compression filters that may allow reaching high compression ratios and enough compression speed to considerably reduce the I/O time while keeping high accuracy. In particular, we are exploring the feasibility of using the SZ lossy compressor developed by the Argonne National Laboratory (ANL) to write highly compressed NetCDF files through XIOS. As a case study, the Open Integrated Forecast System (OpenIFS) is used, an atmospheric general circulation model that can use XIOS to output data.

How to cite: Yepes-Arbós, X., Di, S., Serradell, K., Cappello, F., and C. Acosta, M.: Exploring the SZ lossy compressor use for the XIOS I/O server, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-9153, https://doi.org/10.5194/egusphere-egu22-9153, 2022.

Comments on the display material

to access the discussion