EGU22-13259
https://doi.org/10.5194/egusphere-egu22-13259
EGU General Assembly 2022
© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Adding Quantization to the NetCDF C and Fortran Libraries to Enable Lossy Compression

Edward Hartnett and Charles Zender
Edward Hartnett and Charles Zender

The increasing volume of Earth science data sets continue to present challenges for large data producers. In order to support lossy compression in the netCDF C and Fortran libraries, we have added a quantize feature for netCDF floating point variables. When the quantize feature is enabled, the data creator specifies the number of significant digits. As data are written, the netCDF libraries apply a quantization algorithm which guarantees that the number of significant digits (for BitGroom and Granular BitRound algorithms) or bits (for BitRound algorithm) will be preserved, while setting unneeded bits to a constant value. This allows zlib lossless compression (or any other lossless compression) to achieve better and faster compression.

How to cite: Hartnett, E. and Zender, C.: Adding Quantization to the NetCDF C and Fortran Libraries to Enable Lossy Compression, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-13259, https://doi.org/10.5194/egusphere-egu22-13259, 2022.

Displays

Display file