ZAMG Data Hub – Open access to high value data sets

With new open data policies in place, the European Commission aims to speed up the development of innovative services and products, thereby playing a significant role in economic growth. In particular, the reuse of so-called high value data sets is expected to bring remarkable benefits to society and economics. By being defined as high value information, meteorological data sets need to be made available by public sector bodies free of charge and with minimal legal restrictions, published via APIs and in machine readable formats.

Austria's national weather service Zentralanstalt für Meteorologie und Geodynamik (ZAMG) is aiming to realize those directives by publishing public sector data on the ZAMG Data Hub. Data sets are made available for the public in a step wise approach, which has started in 2021 with providing station measurement data and raster data sets and continues with the preparation of forecasts. Besides the open data sector, the ZAMG Data Hub provides commercial data for registered users and serves as a data platform for universities and ZAMG internal users. The main services and interfaces for the end user are a CKAN based web portal, a REST API, and an associated Metadata Hub, which are connected to the underlying Zarr and S3 file storage infrastructure by state-of-the-art technologies and components. The numerous use cases and actors, the large amount of data to be stored and processed together with the demand of offering a highly available and reliable infrastructure brings challenging questions regarding the architecture of the system and its components. Key aspects are the need for a powerful access control system, scalable and performant data access and easy-to-use metadata information. Besides continuous improvements, ZAMG wants to provide further data sets and implement additional data services to fulfill open data directives and maximize customer satisfaction.