Neural Compression of Remote Sensing Data for the Pre-Training of Geospatial Foundation Models

Over the course of a decade, a single Earth observation satellite mission, such as Sentinel-2, can generate more than 10 petabytes of data. While this wealth of information offers a unique opportunity for pre-training geospatial foundation models, storing and processing such massive datasets is challenging, even for powerful HPC systems. One potential solution is the use of lossy compression techniques, which remove irrelevant information (e.g., noise and redundancy) while preserving as much relevant content as possible. This approach enables significantly larger training datasets for self-supervised learning, potentially offsetting the loss in data quality and yielding performance gains in downstream tasks. However, at the time of writing, the application of lossy compression in this context remains largely underexplored.

Here, we use Vector Quantized Variational Autoencoders (VQ-VAE) for compression of Sentinel-2 and Sentinel-1 data. We show that VQ-VAE is able to achieve compression rates of up to 65x with minimal reconstruction errors. Compared to classic, general-purpose compression techniques such as JPEG-2000, the VQ-VAE model attains 2-3x higher compression rates for the same reconstruction error. We also present ablation studies on the use of compressed versus uncompressed data for pre-training masked autoencoders (MAE) under a fixed physical storage budget, reflecting the constraints of resource-limited HPC systems. Finally, inspired by previous work in computer vision, we explore using the learned quantization scheme to construct a probabilistic masked autoencoder. Instead of predicting a deterministic reflectance or backscatter value, our probabilistic model predicts a categorical distribution over the learned codewords and is trained using cross-entropy loss. This formulation naturally incorporates uncertainty or bimodality into the masked autoencoding task, for example under cloudy conditions.

Our results demonstrate the potential and feasibility of neural compression techniques for the pre-training of large geospatial foundation models. Looking ahead, we aim to incorporate our findings into the training of WeatherGenerator-Land, an upcoming multi-modal foundation model for Earth's land surface. WeatherGenerator-Land will be used for vegetation forecasting, predicting land-atmosphere interactions, and high-resolution land surface temperature forecasts, with a particular focus on heat waves and urban heat islands.