Foundation versus Task-Specific Models for Environmental Time-Series Prediction

Machine learning models for Earth system prediction tasks differ substantially in their training strategy and the type of context they encode. Task-specific models are trained from scratch using a limited set of variables assumed to directly influence the prediction target. These models lack broad spatial and cross-variable context. In contrast, Earth system foundation models are pre-trained on large and heterogeneous data sets and are expected to capture richer environmental context that can be transferred to downstream prediction tasks.

In other machine learning domains, such as natural language processing, fine-tuning pre-trained foundation models has become standard practice due to consistent performance gains over models trained from scratch. Whether similar benefits arise for Earth system time-series prediction tasks remains unclear.

To address this gap, we compare task-specific transformer encoder models operating on pixel-level time series with fine-tuned Earth system foundation models across a set of time-series prediction tasks describing vegetation response to environmental change, including Gross Primary Productivity. This comparison isolates the effect of pre-training on predictive performance by keeping the prediction targets fixed. 

Our aim is to determine which modelling approach yields higher predictive accuracy for environmental time-series analyses.