ES1.5 | Open Data - data, application development, impact

ES1.5

Open Data - data, application development, impact 

Open Data policies have become both popular and mandatory across Europe. While several countries and institutions have adopted already a wide open data policy, the EU Open Data Directive [1] is changing the landscape even more in the coming years.

In meteorology and climate science, a variety of European and international Open Data services grant access to a growing amount of open datasets. Open Data related to weather and climate consist of several different data sources and space/time coverages. For instance, near-real-time weather station measurements, radar-based and satellite-based observations and nowcasting products, model analyses and forecasts, climate data, as well as datasets for experts in emergency management, agriculture, road maintenance, and many more specialised fields are widely provided as open data.

To tame the variety and sheer amount of data, humans rely on computational support and standardised automated ways to treat data and metadata. Popular interfaces are data portals for human interaction and APIs for machine/automated interaction. RESTful APIs are a popular choice as well as GeoWebServices, e.g. in OGC compatible WMS and WFS formats.

Additionally, it is more and more common to exploit clouds to distribute and process Open Data. Initiatives like WEkEO [2], European Weather Cloud [3], and Open Data on AWS [4] are specially built to bring users to data and make processing large data sets easier.

Since all of this Open Data can be freely used, modified, and shared by anyone for any purpose, numerous applications based on these datasets have been developed in the public and private sectors, by met services, companies, research institutes, and open source developers.

The aim of the session is to bring together the enablers, providers, and current/future users of Open Data in meteorology and climate, to share their experiences and requirements.

We invite contributions on both technical and user-focused topics related to

- New Open Data sets including hosting Open Data on-premise and in the cloud
- Metadata management including FAIR principles [5]
- Effects of and preparing for the new EU Open Data Directive

- Tools and interfaces (APIs) for accessing and utilizing Open Data
- How open data cloud-formats, such as Zarr and COG, play together with the new OGC APIs [6]
- The development of data portals, including catalogue services, download services, visualisation services, transformation services

- Existing Open Data applications using weather or climate data
- New ideas where and how Open Data can serve society
- Opportunities and challenges regarding Open Data, including data sources, data formats, legal issues ...

- Community building: How open data in weather and climate can be used and reused in various organisations, and where people can easily build on each other's work, and easily go somewhere to ask questions.
- Whatever you feel is necessary to tell about Open Data

[1] https://digital-strategy.ec.europa.eu/en/policies/psi-open-data
[2] https://www.wekeo.eu
[3] https://europeanweather.cloud
[4] https://aws.amazon.com/opendata
[5] https://www.go-fair.org/fair-principles/
[6] https://www.ogc.org/blog/4607

Public information:

We are looking forward to your input for open data in weather and climate - be it as part of our session or by filling out our open data questionnaire to better understand the needs of users, creators and developers.

Share:
Convener: Hella Riede | Co-conveners: Emma Pidduck, Roope Tervo, Björn Reetz, Håvard Futsæter
Orals
| Tue, 05 Sep, 14:00–16:00 (CEST)|Lecture room B1.04
Posters
| Attendance Tue, 05 Sep, 16:00–17:15 (CEST) | Display Mon, 04 Sep, 09:00–Wed, 06 Sep, 09:00|Poster area 'Day room'
Orals |
Tue, 14:00
Posters
Tue, 16:00

Session assets

Session materials

Orals: Tue, 5 Sep | Lecture room B1.04

The oral presentations are given in a hybrid format supported by a zoom meeting featuring onsite and online presentations. Alternatively, you can watch the session's live stream. The buttons to access the zoom meeting or live stream appear here just before the time block starts.
Chairpersons: Hella Riede, Roope Tervo, Håvard Futsæter
14:00–14:05
5-minute convener introduction
Keynote presentation
14:05–14:35
|
EMS2023-477
|
solicited
|
Onsite presentation
RODEO Project – bringing more European meteorological data open for all users 
Minna Huuskonen and Willie McCairns

The RODEO project responds to the requirements of the EU Directive (2019/1024) on Open Data and the Reuse of Public-sector Information and its Implementing Regulation in order to boost the re-use and combination of open public data across the EU. The Implementing Regulation defines High Value Datasets (HVD) for six thematic categories of public data of which weather observation data, climate data, warnings, weather radar data and Numerical Weather Prediction (NWP) data are defined as meteorological High Value Datasets. These HVDs shall be shared free of charge, under the conditions of the Creative Commons BY 4.0 licence or a less restrictive open licence, and will be openly accessible via Application Programming Interfaces (APIs), machine-readable and bulk downloadable.  

The three-year RODEO project is a joint effort by 11 European meteorological institutions, the European Centre for Medium-Range Weather Forecasts (ECMWF) and the network of 31 European National Meteorological and Hydrological Services, EUMETNET. The project strengthens the capacity of the European meteorological data providers by  

  • Developing a user interface and a data catalogue for making data discoverable; 
  • Developing APIs, by using open licences, for accessing weather observation data, climate data, weather radar data, warnings, and AI datasets; 
  • Engaging with the data owners and user communities; and 
  • Supporting the deployment of national data portals and APIs.
  • Implementing a comprehensive user engagement strategy involving
    • A Project External Advisory Board, composed of public and private sector partners
    • Frequent communications updates through several channels 

The project contributes not only to the overall EU and Digital Europe Programme (DEP) objectives but also to objectives of the global meteorological community. The World Meteorological Organization (WMO) Unified Data Policy commits WMO Member Nations to supporting free and open exchange of meteorological data. The RODEO project builds upon the WMO and EUMETNET long-term plans for exchange of meteorological data under the WMO Information System 2.0 (WIS 2.0) and an existing EUMETNET design for a shared federated data infrastructure.  

Vastly increasing the European-wide real-time meteorological data available for the public, business, and other governmental institutions opens business opportunities for several sectors and application areas. Small and medium-sized enterprises (SMEs) will benefit from the greater data availability by creating new digital products and services and eventually attract new investors. Data will also support research and better-informed policymaking, especially in actions mitigating climate change. Overall, better data availability leads to better warnings, forecasts, and services to the public and weather-critical industries which contributes to the safe and efficient functioning of society with multiple benefits across the European economy, industry, and society. 

How to cite: Huuskonen, M. and McCairns, W.: RODEO Project – bringing more European meteorological data open for all users, EMS Annual Meeting 2023, Bratislava, Slovakia, 4–8 Sep 2023, EMS2023-477, https://doi.org/10.5194/ems2023-477, 2023.

Oral Presentations
14:35–14:50
|
EMS2023-83
|
Onsite presentation
From Data to Decision - Using Open Data in the Cloud 
Chris Stoner

Cloud computing has opened the doors to large scale analysis of environmental data.  AWS created the AWS Open Data program to help Governments, research institutions, and private companies share massive amounts of data publicly in the cloud. Sharing data in the cloud means that the data in the cloud can be utilized by many reseachers at the same time, without the time consuming download before research can start. Researchers no longer have to find storage for the data they want to analyze, rather they can use the data in place in the cloud. The AWS Open Data program hosts over 100 Petabytes of data for public use, with a broad range of compute and data analytics products, including Amazon EC2, Amazon Athena, AWS Lambda, Amazon SageMaker and SageMaker Studio Lab, and Amazon EMR to help the community use the data in the cloud at scale. Sharing data in the cloud lets data users spend more time on data analysis rather than data acquisition, and the Open Data program offers the Registry of Open Data to find and use these data for analysis. Join this session to hear first hand from organizations sharing data how and why they have put data into the Open Data Program, discover best practices for sharing data in the cloud, learn how to find publicly available datasets through the Registry of Open Data on AWS, find out how to use data in the cloud with templates and code samples that you can take home to try, and how you can share your own data through the AWS Open Data program.

How to cite: Stoner, C.: From Data to Decision - Using Open Data in the Cloud, EMS Annual Meeting 2023, Bratislava, Slovakia, 4–8 Sep 2023, EMS2023-83, https://doi.org/10.5194/ems2023-83, 2023.

14:50–15:05
|
EMS2023-472
|
Onsite presentation
|
KNMI Data Platform - A cloud-based data platform for searching and accessing FAIR and Open Data using APIs 
Rosina Derks and Joske Brandsema

The KNMI Data Platform (KDP) is a cloud-based data platform that provides a central location for finding and accessing open and FAIR data related to weather and climate. KDP brings a user-friendly and searchable data catalog based on metadata, as well as the option to browse through the files of a dataset, and if applicable a Web Mapping Service (WMS) preview.  

The data is accessible through multiple APIs. In addition to our file-based API and WMS API, we released an alpha version of our Environmental Data Retrieval EDR API in October 2022. Both the data and the APIs adhere to internationally recognized standards by for instance the Open Geospatial Consortium (OGC), allowing interoperability and easy integration of the data into applications.  

The platform is built with cloud-based technologies resulting in high scalability and reliability. Not only is the data stored on Amazon Web Services (AWS), but the whole infrastructure containing the platform and the APIs are hosted on AWS. This ensures large amounts of data are stored and processed while always being available to users.  

KDP has been operational since 2019, evolving into a robust and highly available platform. This opens up room for community building where we exchange experience and knowledge. Currently, KDP facilitates the community by connecting researchers, organizations, and companies from different fields beyond the traditional meteorological user base. The result is better user feedback to KNMI and an exchange of code between companies about the interpretation of KNMI’s data.  

The aim of this contribution is to present the KNMI Data Platform along with an overview of ongoing advancements and forthcoming features. This will be shown by using a representative dataset to introduce our different APIs, demonstrating this dataset can be accessed and visualized using our platform. Additionally, we hope to discuss the opportunities and challenges surrounding building an API-based Open Data Platform using cloud technology. 

How to cite: Derks, R. and Brandsema, J.: KNMI Data Platform - A cloud-based data platform for searching and accessing FAIR and Open Data using APIs, EMS Annual Meeting 2023, Bratislava, Slovakia, 4–8 Sep 2023, EMS2023-472, https://doi.org/10.5194/ems2023-472, 2023.

15:05–15:20
|
EMS2023-478
|
Online presentation
|
Association of temperature and total precipitation anomalies with West Nile Virus human cases in Central Macedonia, Greece 
Anastasia Angelou and Ioannis Kioutsioukis

West Nile Virus (WNV) belongs to the flavivirus genus, and it is a neurotropic mosquito-borne virus. The virus transmitted among amplifying hosts, such as birds, transferred through the bites of mosquitoes and incidentally humans and other mammals may become infected. In elderly and immunocompromised people, symptomatic infections can result in neurologic diseases. There is currently no specific treatment or vaccine available for WNV. The transmission dynamics of WNV are complex and affected by various environmental factors, including temperature and total precipitation. Understanding the relationship between the environmental factors and WNV transmission is crucial for predicting and preventing outbreaks of the disease., while the prediction of an infectious disease outbreak is critical for reducing the potential impact on human health. A large proportion of WNV infections in humans present either asymptomatically or with some non-specific clinical symptoms and are unrecorded.

The aim of this study is to explore the association between climatic factors and the occurrence of West Nile fever (WNF) in humans at a finer than NUTS-3 spatial scale (LAU). The investigation area is the region of Central Macedonia in Northern Greece, analyzing a unique dataset that includes meteorological data from ERA5 (European Centre for Medium-Range Weather Forecasts) and epidemiological data from the Hellenic National Public Health Organization for the period 2010-2022. The research focuses on this region because it is an area of great epidemiological interest. Specifically, at least one WNV human case has been recorded at all LAUs, while in 26% of the LAUs an extremely high number of cases have been recorded during the period of research. The analysis shows a strong correlation between the number of annual human cases of WNV and temperature and precipitation patterns in the months leading up to the outbreak. The results reveal the augmented forecast potential from temperature and precipitation anomalies in virus spread prediction models.

Acknowledgments

This research has been co‐financed by the European Regional Development Fund of the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH – CREATE – INNOVATE (project code: Τ2ΕΔΚ-02070).

How to cite: Angelou, A. and Kioutsioukis, I.: Association of temperature and total precipitation anomalies with West Nile Virus human cases in Central Macedonia, Greece, EMS Annual Meeting 2023, Bratislava, Slovakia, 4–8 Sep 2023, EMS2023-478, https://doi.org/10.5194/ems2023-478, 2023.

15:20–15:35
|
EMS2023-497
|
Onsite presentation
Analysis-ready climate data with ESMValCore and ESMValTool 
Bouwe Andela, Peter Kalverla, Remi Kazeroni, Saskia Loosveldt Tomas, Valeriu Predoi, Manuel Schlund, Stef Smeets, and Klaus Zimmermann

We present new features of ESMValCore, a Python package designed to work with large climate datasets available from ESGF and beyond. The Earth System Grid Federation (ESGF) offers a wealth of climate data that can be used to do interesting research. For example, the latest edition of the Coupled Model Intercomparison Project (CMIP6) output features 20 petabytes of data. However, the heterogeneity of the data can make it difficult to find and work with. ESMValCore now provides a Python interface that makes it easy to discover what data is available on ESGF and locally, download it if necessary, and make it analysis-ready. The analysis-ready data can then be used as input to the ESMValCore preprocessor functions, a collection of functions to perform commonly used analysis steps such as regridding and statistics. When searching for data on ESGF as well as when loading the NetCDF files, the software intelligently corrects small issues in the metadata that otherwise make working with this data a time-consuming, manual effort. Data and metadata issues are fixed in memory for fast performance. The search and download features are user-friendly and will automatically use a different server if one of the ESGF servers is unavailable for some reason. Several Jupyter notebooks demonstrating these new features are available at https://github.com/ESMValGroup/ESMValCore/tree/main/notebooks. 

ESMValCore has been designed for use on computing systems that are typically used by researchers: it works well on a laptop or desktop computer, but also comes with example configuration files for use on large compute clusters attached to ESGF nodes. For reliable computations, ESMValCore makes use of the Iris library developed by the UK Met Office. This in turn is built on top of Dask, a library for efficient parallel computations with a low memory footprint. In 2023, we aim to improve our use of Dask in collaboration with the Iris developers, for even better computational performance. 

For easy reproducibility, ESMValCore also offers "recipes" in which standard analyses can be saved. A large collection of such recipes is available in the Earth System Model Evaluation Tool (ESMValTool), including recipes for estimating future drought risk. ESMValTool started out as a set of community-developed diagnostics and performance metrics for the evaluation of Earth system models. Recently it has also turned out to be useful for other users of climate data, such as hydrologists and climate change impact researchers. Both ESMValCore and ESMValTool are developed by and for researchers working with climate data, with the support of several research software engineers. An important recent achievement is the use of these packages to produce the figures for several chapters of the IPCC AR6 report. Documentation for both ESMValCore and ESMValTool is available at https://docs.esmvaltool.org. 

How to cite: Andela, B., Kalverla, P., Kazeroni, R., Loosveldt Tomas, S., Predoi, V., Schlund, M., Smeets, S., and Zimmermann, K.: Analysis-ready climate data with ESMValCore and ESMValTool, EMS Annual Meeting 2023, Bratislava, Slovakia, 4–8 Sep 2023, EMS2023-497, https://doi.org/10.5194/ems2023-497, 2023.

15:35–16:00
Discussion and Interaction

Posters: Tue, 5 Sep, 16:00–17:15 | Poster area 'Day room'

Display time: Mon, 4 Sep 09:00–Wed, 6 Sep 09:00
Chairpersons: Roope Tervo, Hella Riede, Emma Pidduck
P1
|
EMS2023-314
|
Onsite presentation
Progress on the implementation of the ECMWF Open Data roadmap and its challenges 
Ilaria Parodi, Maartje Kuilman, Emma Pidduck, Victoria Bennett, Xiaobo Yang, and Umberto Modigliani

The European Centre for Medium-Range Weather Forecasts (ECMWF) is a major provider of weather information, and its products and weather data contribute to a broad range of activities in service provision and research. The use of this data is also crucial for the activities of protection of life and property by National Weather Services and, in case of emergencies, humanitarian agencies. 

ECMWF recognises that open data is an essential tool to maximise socio-economic benefits of investments in weather and climate data production. Open data is also important to promote shared actions to mitigate the effects of climate change. 

ECMWF has agreed a move to an open data policy, and this is taking place gradually. Over the last few years, a number of steps have been taken, such as: 

  • Applying the Creative Commons CC-BY-4 licence to all non-valid (historical) data in the Archive Catalogue for MARS as well as to static charts, 
  • Reducing the information cost for data, 
  • Introducing a limited redistribution scheme, 
  • Introducing small business discount for micro-organisations 
  • Releasing a subset of the real-time catalogue with open data policy and available through a free access service with supporting software and tools to adapt to FAIR principles. 

The process toward an open data policy presents different challenges, for example the aspect connected to the reduction in the information cost revenue, but also the growing amount of data volume that will be served, leading to the need for an improvement of the types of accesses to data, the need to guarantee data interoperability, and the need for the data to be distributed together with the appropriate documentation and self-help information, to help the users to process data, derive information and form valuable conclusions. 

ECMWF is committed to continue proceeding further towards an open data policy in the coming years. We will illustrate the proposed changes and associated challenges related to this transition, such as new open datasets and reductions in the cost of data. The general aim of these changes is to continue promoting collaboration, innovation, and advancement within the meteorological and climate science communities while aligning with WMO guidelines.

How to cite: Parodi, I., Kuilman, M., Pidduck, E., Bennett, V., Yang, X., and Modigliani, U.: Progress on the implementation of the ECMWF Open Data roadmap and its challenges, EMS Annual Meeting 2023, Bratislava, Slovakia, 4–8 Sep 2023, EMS2023-314, https://doi.org/10.5194/ems2023-314, 2023.

P2
|
EMS2023-677
|
Onsite presentation
DWD-Crowdsourcing: User Reports available on Open Data 
Arne Spitzer, Harald Kempf, Matthias Jerg, and Ulrich Blahak

Since July 2020 the DWD WarnWetter-App comprises the Crowdsourcing-module “User Reports”. This module provides users the functionality to report observations about current weather conditions and severe weather to DWD and other users. The data is daily collected and available on DWD’s Open Data portal (https://opendata.dwd.de/weather/crowdsourcing/warnwetter/).

The user reports represent the current meteorological conditions at a certain place at a certain point of time. The Crowdsourcing-module provides 10 different meteorological categories (lightning, wind, hail, rain, wet icy conditions, snowfall, snow cover, cloudiness, fog, tornado), each of which contains specific characteristic levels and optionally additional attributes. In addition, the user has the option of setting the location and time of the event manually.

The benefit of the data is that meteorological information at ground level is collected at places where no weather station is located in the immediate vicinity. The dataset is able to complement the existing synoptic station network. In the future, the data could improve the evaluation of the current meteorological conditions and the warning management particularly during extreme weather events.

There is no sophisticated quality control for the user reports. Instead, the users are expected to estimate and report the weather conditions as accurate as possible. Badly inaccurate and false reports are detected by reference data and are excluded instantly. Additionally, in the app users have the opportunity to manually flag meteorologically doubtful reports. Other quality assurance methods are under development.

This contribution contains some numbers and statistics on previous user reports, shows some meteorologically interesting cases, and gives an insight into quality control.

How to cite: Spitzer, A., Kempf, H., Jerg, M., and Blahak, U.: DWD-Crowdsourcing: User Reports available on Open Data, EMS Annual Meeting 2023, Bratislava, Slovakia, 4–8 Sep 2023, EMS2023-677, https://doi.org/10.5194/ems2023-677, 2023.