Multi-label cloud type classification from ground-based RGB pictures with a residual neural network ensemble

Clouds of any kind play a substantial role in a wide variety of atmospheric processes. They are directly linked to the formation of precipitation, and significantly affect the atmospheric energy budget via radiative effects and latent heat. Moreover, both the amount and type of clouds is supposed to alter in a changing climate. Hence, the currently decreasing number of operational cloud observations limits not only the possibility and accuracy of short-term weather forecasts but also the availability of long-term cloud type records.

To show that automatized methods can close this emerging gap, we trained an ensemble of 10 identically initialized residual neural network architectures from scratch on ground-based RGB sky pictures to classify clouds into 30 different classes. Four different pictures taken in the main cardinal directions are used as input at each instance, so that the whole visible sky is covered. Operational manual cloud classification reports at the nearby station Vienna Hohe Warte are used as ground truth. For each instance up to 3 out of 30 categories are reported according to the state-of-the-art WMO cloud classification scheme for operational synoptic observations, making this a multi-label classification task. To the best of our knowledge we are the first to automatically classify clouds based on this elaborate classification scheme. We utilize class specific resampling to reduce prediction biases because of highly imbalanced observation frequencies among categories. Results show that precision and recall scores are high and that every member of our ensemble outperforms both random and climatological predictions in each class. A substantial ratio of wrongly assigned pictures is made up by false negative predictions, where the model recognized the correct class in the input but the assigned probability was too small. Although the WMO classification scheme is well-defined, cloud classification is subjective to some extent because of e.g. the occurrence of clouds in transitional stages. Therefore, we also investigate the reliability of ground-truth observations.

Autonomy and output consistency are the main advantages of such a trained classifier, hence we consider operational cloud monitoring as main application. Either for consistent cloud class observations or to observe the current state of the weather and its short time evolution with high temporal resolution, e.g. in proximity of solar power plants. There, upcoming clouds can substantially change the possible energy output, which leads to the necessity of taking precautions.