Bridging linear state estimation and machine learning

Recent years have seen active efforts within the geophysical community to combine traditional Data Assimilation (DA) methods with emerging Machine Learning (ML) techniques. However, most of this past theoretical work has been centered on variational DA approaches due to their similarity with ML in terms of how the underlying optimization problem is formulated and solved. Here I will present a new and completely general nonlinear estimation theory that retains the flexibility of advanced sampling-based methods (e.g., the particle filter) and the analytical tractability of linear estimation algorithms (e.g., the ensemble Kalman filter). In particular, an alternative state space model will be constructed whose filtering and smoothing distributions remain closed under a wide class of nonlinear functions. Since these nonlinear functions are only required to be bijective and continuously differentiable, the new estimation theory serves an ideal framework for rigorously incorporating invertible neural networks in the DA design. There are two additional properties which make the proposed framework especially appealing. First, linear estimation results follow immediately upon substituting the invertible neural networks with the identity transformation. Second, the prior and posterior belong to the same distribution family, which implies that the correlation structure and the corresponding dynamical balances in the model state are preserved following the analysis step. During the upcoming EGU meeting, I will discuss the motivation behind the new estimation framework, place it in the context of existing nonlinear DA techniques and demonstrate some of its benefits through idealized numerical examples.