Data Access Made Easy: flexible, on the fly data standardization and processing

Automatic Weather Stations (AWS) deployed in the context of research projects provide very valuable point data thanks to the flexibility they offer in term of measured meteorological parameters and setup. However this flexibility is a challenge in terms of metadata and data management. Traditional approaches based on networks of standard stations struggle to accommodate these needs, leading to wasted data periods because of difficult data reuse, low reactivity in identifying potential measurement problems, and lack of metadata to document what happened.

The Data Access Made Easy (DAME) effort is our answer to these challenges. At its core, it relies on the mature and flexible open source MeteoIO meteorological pre-processing library. Originally developed for the needs of numerical models consuming meteorological data it has expanded as a data standardization engine for the Global Cryosphere Watch (GCW) of the World Meteorological Organization (WMO). For each AWS, a single configuration file describes how to read and parse the data, defines a mapping between the available fields and a set of standardized names and provides relevant Attribute Conventions Dataset Discovery (ACDD) metadata fields. Low level data editing is also available, such as excluding a given sensor, swapping sensors or merging data from another AWS, for any given time period. Moreover an arbitrary number of filters can be applied on each meteorological parameter, restricted to specific time periods if required. This allows to describe the whole history of an AWS within a single configuration file and to deliver a single, consistent, standardized output file possibly spanning many years, many input data files and many changes both in format and available sensors.

Through the EU project Arctic Passion, a web interface has been developed that allows data owners to manage the configuration files for their stations, refresh their data at regular intervals, inspect the data QA log files, receive notification emails and allow on-demand data generation. The same interface allows other users to request data on-demand for any time period.