A fast, single-iteration ensemble Kalman smoother for sequential data assimilation

Ensemble-variational methods form the basis of the state-of-the-art for nonlinear, scalable data assimilation, yet current designs may not be cost-effective for reducing prediction error in online, short-range forecast systems. We propose a novel, outer-loop optimization of the Bayesian maximum a posteriori formalism for ensemble-variational smoothing in applications for which the forecast error dynamics are weakly nonlinear, such as synoptic meteorology. In addition to providing a rigorous mathematical derivation our technique, we systematically develop and inter-compare a variety of ensemble-variational schemes in the Lorenz-96 model using the open-source Julia package DataAssimilationBenchmarks.jl. This high-performance numerical framework, supporting our mathematical results, produces extensive benchmarks that demonstrate the significant performance advantages of our proposed technique versus several similar estimator designs. In particular, our single-iteration ensemble Kalman smoother (SIEnKS) is shown both to improve prediction / posterior accuracy and to simultaneously reduce the leading order cost of iterative, sequential smoothers in a variety of relevant test cases for operational short-range forecasts.  These results are currently in open review in Geoscientific Model Development (Preprint gmd-2021-306) and the Journal of Open Source Software (Preprint #3976).