High-Efficiency Rainfall Data Compression Using Binarized Convolutional Autoencoder

In the era of big data, managing and storing large-scale meteorological datasets is a critical challenge. We focus on high-resolution rainfall data, which is crucial to atmospheric sciences, climate research, and real-time weather forecasting. This study introduces a deep learning-based approach to compress the German Radar-Online-Aneichung (RADOLAN) rainfall dataset. We achieve a compression ratio of 200:1 while maintaining a minimal mean squared reconstruction error (MSE). Our method combines a convolutional autoencoder with a novel binarization mechanism, to compress data from a resolution of 900x900 pixels at 32-bit depth to 180x180 pixels at 4-bit depth. Leveraging the ConvNeXt architecture (Zhuang Liu, et al., 'A ConvNet for the 2020s'), our method learns a convolutional autoencoder for enhanced meteorological data compression. ConvNeXt introduces key architectural modifications, such as revised layer normalization and expanded receptive fields, taking inspiration from Vision Transformer to form a modern ConvNet. Our novel binarization mechanism, pivotal for achieving the high compression ratio, operates by dynamically quantizing the latent space representations using a novel magnitude specific noise injection technique. This quantization not only reduces the data size but also preserves crucial meteorological information as our low reconstruction MSE demonstrates. Beyond rainfall data, our approach shows promise for other types of high-resolution meteorological datasets, such as temperature, humidity, etc. Adapting our method to these modalities could further streamline the data management processes in meteorological deep learning scenarios and thus facilitate efficient storage and processing of diverse meteorological datasets.