Leveraging Differentiable Programming and Online Learning for the design of Hybrid Numerical Models

Earth system models (ESMs) are widely used to study climate changes resulting from both anthropogenic and natural perturbations. Over the past years, significant advances have been made through the development of new numerical schemes, refined physical parameterizations, and the use of increasingly powerful computers. Despite these advances, tuning ESMs to accurately reproduce historical data remains largely a manual process, and persistent errors and biases continue to challenge their accuracy. Reducing uncertainties in long-term climate projections and accurately estimating the spread of climate simulations continue to be critical challenges.

Recent advances in machine learning have motivated the development of learning-based methods for the calibration of ESMs. One emerging area of research is the design of hybrid modeling approaches, which combine a physical core with a machine learning model. Training these hybrid models end-to-end (or online) has the potential to unify various challenges in ESMs development, ranging from building subgrid scale parameterizations, to bias correction and parameter tuning.

Training hybrid models online requires working with an optimization problem that depends on the numerical integration of the system. Solving this optimization problem using gradient-based approaches requires the system to be differentiable, or to have access to the adjoint of the numerical model, which is not the case for most of the large-scale physical models. Beyond the need for differentiability, developing hybrid models requires interfacing a physical core that is implemented in low-abstraction languages that are running on CPUs, with AI-based models that are developed using high-abstraction, rapidly evolving languages that run on GPUs. While this interface is not a problem at inference time, doing this interface at calibration time, which is necessary when doing online learning, is not trivial as it would require an iterative communication between components that are implemented on different architectures.

In this work, we aim to investigate online learning and hybrid models to develop new computing paradigms, tools, and calibration methods for designing numerical models that are closely aligned with observations. We study the potential of online learning for deriving efficient and scalable solutions to the above-mentioned problems for applications that include both short-term forecasting and long-term simulations, which require stability considerations of the resulting hybrid systems. We explore learning configurations that include both fully differentiable and black-box physical cores. The latter configuration aims at evaluating the extent to which differentiable programming frameworks can upscale modeling capabilities in terms of accuracy, computational efficiency, and adaptability to represent diverse physical processes.