Quality control of weather observations using machine learning

A key component of a weather service’s mission is to capture accurate observations of weather conditions in real-time, which are subsequently fed into forecasting models for prediction, used in climate research, and form the basis of the official climate statistics. The accuracy of such measurements is ensured by following a set of robust procedures set out in the World Meteorological Organisation guidelines. In addition to appropriate maintenance and calibration of equipment, procedures for quality control (QC) of observations are necessary in order for users to have maximum confidence in the data. Much of this QC tends to be done manually by expert climatologists who examine data and flag erroneous values. However, due to the current proliferation of climate data, more and more organisations are seeking to implement automated, real-time QC to prevent poor quality data being used in operational products. One potential approach is to use Anomaly Detection algorithms to identify potentially erroneous values. Such methods are a well-established field in Machine Learning and are a potential tool for QC of environmental data.

This paper develops and evaluates models to perform automatic QC of Irish weather data (from the Irish meteorological agency Met Éireann) using appropriate Machine Learning techniques such as Convolutional Neural Networks (CNN’s). The outcomes are evaluated in comparison to standard statistical tests which have traditionally been used for these purposes (such as linear regression spatial variability tests). The dataset in question consists of over 10 years of automated temperature observations taken at one minute intervals using Platinum Resistance Thermometers (PRT’s) located at sites across the country. This is complemented by a related data set that has been manually quality-controlled using well established methods, which can be used for comparison and verification.

The paper aims to investigate whether anomaly detection algorithms can correctly identify erroneous values in the time-series to at least the same standard as that achieved through traditional manual approaches. Furthermore, it examines whether these anomalies, once discovered, can be classified into categories with interpretable physical meaning.