Data Assimilation of visible and infrared cloud observations from pictures

We present an innovational way of assimilating visible and infrared observations of clouds into the weather forecasting model for regional scale: ICON-D2 (ICOsahedral Nonhydrostatic), which is operated by the German Weather Service (Deutscher Wetterdienst, DWD). For the visible camera photographs, a convolutional neural network is trained to detect clouds in pictures. The result is a greyscale picture, in which each pixel has a value between 0 and 1, describing the probability of the pixel belonging to a cloud. By averaging over a certain section of the picture one gets a value for the cloud cover of that region. To build the forward operator, which maps an ICON model state into the observation space, a three dimensional grid in space from the camera point of view had to be constructed and the ICON model variables were interpolated onto this grid. The pixels of the picture are modeled as rays, originating at the camera location and the maximum interpolated cloud cover (CLC) along each ray is taken as a model equivalent for each pixel. CLC is a diagnostic variable of an ICON model state describing the probability of the cloud coverage within the respective grid box. After superobbing, monitoring experiments have been conducted to compare the observations and model equivalents over time. The results of these experiments look promising with RMSE values below 0.32 and we continued by performing single assimilation steps as well longer experiments. 
For assimilating the infrared camera pictures we use a forward operator created by Leonhard Scheck at LMU Munich which provides a fast solution for the radiative transfer equations. Monitoring experiments as well as Data Assimilation experiments were conducted and will be presented.