Finetuning and Benchmarking an AI Foundation Model for Cloud Gap Imputation

Abstract: Cloud cover poses a significant obstacle in harnessing multi-spectral satellite imagery for various earth observation applications including disaster response, land use and land cover mapping. To address this issue, this study investigates the potential of Prithvi WxC foundation model (Johannes Schmude et al., 2024), a deep learning architecture designed for weather and climate applications, to perform cloud gap imputation. By leveraging its ability to capture atmospheric dynamics and predict missing data, Prithvi WxC offers a promising solution.

The primary objective is to assess the accuracy and efficiency of Prithvi WxC in reconstructing cloudy pixels in Moderate Resolution Imaging Spectroradiometer (MODIS) surface reflectance; MOD09 (Eric Vermote, 2015). MOD09 data provides valuable information about earth surface, cloud cover and atmospheric conditions, which are instrumental in informing the Prithvi WxC model during the finetuning and imputation process.

This research evaluates the Prithvi WxC foundation model for cloud gap imputation applications and benchmarks its performance against other foundation models, such as Prithvi EO (Jakubik et al., 2023, 2024). The process begins with preprocessing the MOD09 dataset, filtering out missing and cloudy pixels to create clean visible patches, while real-world cloudy patches are used as masks. The preprocessed data is then resampled to align with the temporal and spatial resolution requirements of both the Prithvi WxC and Prithvi EO foundation models. Through rigorous fine-tuning strategies, these models learn to reconstruct the masked regions, effectively filling the gaps caused by cloud cover. Finally, the fine-tuned foundation models are benchmarked using quantitative metrics, such as the Structural Similarity Index Measure (SSIM) and Mean Absolute Error (MAE), complemented by qualitative visual analysis.

This research explores the potential of Prithvi WxC foundation model, pre-trained on extensive weather and climate data, to improve cloud gap imputation in satellite imagery, and subsequently benchmarks it against earth observation foundation models, such as Prithvi EO. Through this evaluation, we aim to enhance scientific understanding via multi-modality and sensor-independent approaches.

 References

Johannes Schmude, Sujit Roy, Will Trojak, Johannes Jakubik, Daniel Salles Civitarese, Shraddha Singh, Julian Kuehnert, Kumar Ankur, Aman Gupta, Christopher E Phillips, Romeo Kienzler, Daniela Szwarcman, Vishal Gaur, Rajat Shinde, Rohit Lal, Arlindo Da Sil: Prithvi WxC: Foundation Model for Weather and Climate." arXiv preprint arXiv:2409.13598, 2024.

C. Roger, E. F. Vermote, J. P. Ray: https://modis-land.gsfc.nasa.gov/pdf/MOD09_UserGuide_v1.4.pdf. NASA, MODIS Surface Reflectance User’s Guide, Collection 6, 2015.

Daniela Szwarcman, Sujit Roy, Paolo Fraccaro, Þorsteinn Elí Gíslason, Benedikt Blumenstiel, Rinki Ghosal, Pedro Henrique de Oliveira, Joao Lucas de Sousa Almeida, Rocco Sedona, Yanghui Kang, Srija Chakraborty, Sizhe Wang, Ankur Kumar, Myscon Truong, Denys: Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications. https://arxiv.org/abs/2412.02732, 2024.

Johannes Jakubik, Sujit Roy, C. E. Phillips, Paolo Fraccaro, Denys Godwin, Bianca Zadrozny, Daniela Szwarcman, Carlos Gomes, Gabby Nyirjesy, Blair Edwards, Daiki Kimura, Naomi Simumba, Linsong Chu, S. Karthik Mukkavilli, Devyani Lambhate, Kamal Das, Ranji: Foundation Models for Generalist Geospatial Artificial Intelligence, 2023.

Eric Vermote: MOD09 MODIS/Terra L2 Surface Reflectance, 5-Min Swath 250m, 500m, and 1km. NASA LP DAAC., NASA GSFC and MODAPS SIPS, NASA, http://doi.org/10.5067/MODIS/MOD09.061, 2015.