Training Deep Data Assimilation Networks on Sparse and Noisy Observations

Data Assimilation (DA) is a challenging and expensive computational problem targetting hidden variables in high-dimensional spaces. 4DVar methods are widely used in weather forecasting to fit simulations to sparse observations by optimization over numerical model input. The complexity of this inverse problem and the sequential nature of common 4DVar approaches lead to long computation times with limited opportunity for parallelization. Here we propose using machine learning (ML) algorithms to replace the entire 4DVar optimization problem with a single forward pass through a neural network that maps from noisy and incomplete observations at multiple time points to a complete system state estimate at a single time point. We train the neural network using a loss function derived from the weak-constraint 4DVar objective, including terms incorporating errors in both model and data. In contrast to standard 4DVar approaches, our method amortizes the computational investment of training to avoid solving optimization problems for each assimilation window, and its non-sequential nature allows for easy parallelization along the time axis for both training and inference. In contrast to most previous ML-based data assimilation methods, our approach does not require access to complete, noise-free simulations for supervised learning or gradient-free approximations such as Ensemble Kalman filtering. To demonstrate the potential of our approach, we show a proof-of-concept on the chaotic Lorenz'96 system, using a novel "1.5D Unet" architecture combining 1D and 2D convolutions.