An Explainable AI (XAI) Benchmark for Geospatial Foundation Models

There is a significant growth in development and utilization of foundation models for geospatial applications. These models are trained on large scale unlabeled data and commonly evaluated on downstream tasks using labeled datasets. While this approach provides a platform to assess the performance of the model for specific downstream tasks, there has been limited effort to quantify the characteristics of the foundation model after pre-training. 

Explainable AI (XAI) approaches aim to increase the accuracy and transparency of AI models and to make their results interpretable. In the case of geospatial foundation models, it is essential to assess if the model learns the spectral, spatial and temporal properties of geospatial data, and how this learning impacts the accuracy of model predictions. 

To this end, we introduce a new global XAI benchmark for geospatial foundation models using multispectral remote sensing imagery. This benchmark contains separate tasks that allows the user to test a foundation model’s properties in the embedding space, and demonstrate whether the model has learned spectral, spatial and temporal features. The spectral task consists of a set of chips with homogeneous spatial patterns from all major land cover classes. The spatial task consists of the same data used for spectral taks but regular spatial patterns are replaced with heterogeneous features representative of their true distribution. Finally, the temporal task includes a set of chips with time series imagery of pre- and post-event for disturbances such as wildfire and flood. 

In this presentation, we will demonstrate the results of using this benchmark to evaluate the properties of multiple geospatial foundation models.