SHRUG-FM: Reliability-Aware Foundation Models for Earth Observation

Following recent advances of foundation models in natural language processing and computer vision, there is growing interest in leveraging geospatial foundation models (GFMs) for Earth system monitoring and climate-relevant applications. In particular, GFMs promise to support large-scale observation of climate-driven extreme events such as wildfires, floods and landslides. However, despite strong benchmark results, recent studies indicate that GFMs for land-cover modelling and hazard mapping models can behave unreliably under real-world conditions. Pretraining datasets often underrepresent rare or extreme environmental regimes, leading to degraded model performance precisely in situations where robust predictions are most critical for climate risk assessment and disaster response. Furthermore, GFMs are often surpassed by simple supervised baselines, highlighting the need for systematic reliability analysis, including out-of-distribution (OOD) detection and uncertainty quantification.

We present SHRUG-FM (systematic handling of real-world uncertainty in geospatial foundation models), a reliability-aware prediction framework that integrates three complementary signals: (1) OOD detection in the input space, (2) OOD detection in the embedding space and (3) task-specific predictive uncertainty obtained from decoder ensembles. We evaluate SHRUG-FM on climate-relevant extreme-event applications, including burn-scar, flood and landslide segmentation. Our results show that elevated OOD scores consistently co-locate with degraded model performance, while uncertainty-based indicators successfully capture many low-confidence and erroneous predictions. By linking these reliability signals to hydro-environmental descriptors from HydroATLAS, we further demonstrate that model failures cluster in distinct geographic and hydroclimatic regimes, revealing interpretable gaps in the pretraining distribution and guiding future dataset design.

SHRUG-FM delivers practical, operationally relevant diagnostics for Earth system monitoring and prediction. It enables selective prediction, rejection strategies, and reliability-aware quality control. These capabilities are essential for integrating GFMs into real-world workflows for climate impact assessment, hazard monitoring and early warning systems. Future work will extend the framework to additional foundation models and climate-driven hazards.