Modeling landscape-scale vegetation response to climate: Synthesis of the EarthNet challenge

The biosphere displays high heterogeneity at landscape-scale. Vegetation modelers struggle to represent this variability in process-based models because global observations of micrometeorology and plant traits are not available at such fine granularity. However, remote sensing data is available: the Sentinel 2 satellites with a 10m resolution capture aspects of localized vegetation dynamics. The EarthNet challenge (EarthNet2021, [1]) aims at predicting satellite imagery conditioned on coarse-scale weather data. Multiple research groups approached this challenge with deep learning [2,3,4]. Here, we evaluate how well these satellite image models simulate the vegetation response to climate, where the vegetation status is approximated by the NDVI vegetation index.

Achieving the new vegetation-centric evaluation requires three steps. First, we update the original EarthNet2021 dataset to be suitable for vegetation modeling: EarthNet2021x includes improved georeferencing, a land cover map, and a more effective cloud mask. Second, we introduce the interpretable evaluation metric VegetationScore: the Nash Sutcliffe model efficiency (NSE) of NDVI predictions over clear-sky observations per vegetated pixel aggregated through normalization to dataset level. The ground truth NDVI time series achieves a VegetationScore of 1, the target period mean NDVI a VegetationScore of 0. Third, we assess the skill of two deep neural networks with the VegetationScore: ConvLSTM [2,3], which combines convolutions and recurrency, and EarthFormer [4], a Transformer adaptation for Earth science problems. 

Both models significantly outperform the persistence baseline. They do not display systematic biases and generally catch spatial patterns. Yet, both neural networks achieve a negative VegetationScore. Only in about 20% of vegetated pixels, the deep learning models do beat a hypothetical model predicting the true target period mean NDVI. This is partly because models largely underestimate the temporal variability. However, the target variability may partially be inflated by the noisy nature of the observed NDVI. Additionally, increasing uncertainty for longer lead times decreases scores: the mean RMSE in the first 25 days is 50% lower than between 75 and 100 days lead time. In general, consistent with the EarthNet2021 leaderboard, the EarthFormer outperforms the ConvLSTM. With EarthNet2021x, a more narrow perspective to the EarthNet challenge is introduced. Modeling localized vegetation response is a task that requires careful adjustments of off-the-shelf computer vision architectures for them to excel. The resulting specialized approaches can then be used to advance our understanding of the complex interactions between vegetation and climate.

 [1] Requena-Mesa, Benson, Reichstein, Runge and Denzler. EarthNet2021: A large-scale dataset and challenge for Earth surface forecasting as a guided video prediction task. CVPR Workshops, 2021.

 [2] Diaconu, Saha, Günnemann and Zhu. Understanding the Role of Weather Data for Earth Surface Forecasting Using a ConvLSTM-Based Model. CVPR Workshops, 2022.

 [3] Kladny, Milanta, Mraz, Hufkens and Stocker. Deep learning for satellite image forecasting of vegetation greenness. bioRxiv, 2022.

 [4] Gao, Shi, Wang, Zhu, Wang, Li and Yeung. Earthformer: Exploring Space-Time Transformers for Earth System Forecasting. NeurIPS, 2022.