Exploring Pretrained Transformers for Ocean State Forecasting

Traditional physics-based numerical models have served and are serving as reliable tools to gain insights into spatiotemporal behavior of ocean states such as water levels and currents. However, they have significant computational demand that often translates to slower forecasting capabilities. Additionally, these models can encounter difficulties in capturing certain physical processes and struggle to effectively bridge various spatial and temporal scales.

Considering these challenges, machine learning-based surrogate models emerge as a promising alternative. Physical surrogate models that learn multiple physics (on different spatial and temporal scales) from large datasets during extensive pretraining (Multiple physics pretraining, MPP) can enable later applications to poorly observed data domains which are common in ocean sciences. Hence, transfer learning capabilities can help improve the oceanographic forecasting, especially in data-limited regimes.

In this work, we explore the capabilities of pretrained transformer models for prediction on a test case for the North Sea. The results from two-dimensional simulations are used for training and fine-tuning. We utilize 2D datasets from publicly available PDEBench together with domain-specific datasets from DHI’s historical records of simulated 2D metocean data. We forecast water levels and currents with pretrained models and evaluate MPP forecast results against in-situ point observations and numerical model results.

Initial findings suggest that pretraining poses potential for generalizing and transferring knowledge to novel regions and relevance in practical application. A challenge is posed by model interpretability, highlighting an area for further development.