A Practical Guide to Hyperspectral Foundation Models

Hyperspectral imagery (HSI) provides rich spectral information that is the basis for applications such as mineral mapping, trace gas identification, and precision agriculture. Yet, the development of HSI Foundation Models (FMs) is less advanced compared to multi-spectral remote sensing modalities.

In this study, we leverage the SpectralEarth dataset [1] to explore practical aspects of training robust HSI FMs. In particular, we shed light on the role of:

• the impact of model architecture (transformers vs. convolutional networks),
• self-supervised learning methods (contrastive vs. masked autoencoders),
• model size & training data volume,
• and the resulting computational requirements.

Through extensive experiments, this study aims to provide concrete guidelines for the development and effective application of FMs in the HSI domain. Moreover, we report on findings to identify downstream applications where hyperspectral imagery has an edge over multi-spectral photos [2], and where such an advantage is less likely to expect.

 

References

[1] Braham, Nassim Ait Ali, et al. "SpectralEarth: Training Hyperspectral Foundation Models at Scale." arXiv preprint arXiv:2408.08447 (2024)

[2] Bangalore, Ranjini, et al. "Hyperspectral foundation model trained by spectral reconstruction for greenhouse gas emission estimation", annual meeting of the American Geophysical Union (2024)