Sampling error in the ensemble Kalman filter for small ensembles and high-dimensional states

Sampling error is a fundamental limitation of assimilation schemes, such as the EnKF, that employ the sample covariance from an ensemble of forecasts.  Despite the fact that the EnKF is typically applied in situations where the ensemble size is small compared to the system dimension, most of what is known about the effect of sampling error comes from low-dimensional examples or asymptotic results valid when the ensemble size is large. For high-dimensional systems and small ensembles, progress can be made by leveraging (i) the diagonal form of the Kalman-filter update in the optimal coordinates of Snyder and Hakim (2022) and (ii) basic approximations from the theory of random matrices. These yield novel, explicit expressions for the EnKF gain, the (sample) mean and covariance of the EnKF posterior ensemble, and the error covariance of that posterior mean. The expressions show that a single EnKF update will remove almost all variance from the ensemble, unless the observations are very uninformative.  They also identify those directions in the state space for which the EnKF update is effective, improving the state estimate despite sampling errors, and those directions for which sampling errors in the EnKF overwhelm observational information and degrade the state estimate.