ES1.6

Finnish Meteorological Institute has a long history of providing open data. Partly as a result of the INSPIRE directive almost all important data was opened back in 2013. Because of this we have quite a long data usage record and experience on technical solutions and user needs. The presentation will open up the current status and future development keeping in mind the upcoming WMO WIS2 development as well as the Open Data directive with its High Value Dataset proposal.

Data is provided via machine-readable interfaces as well as human usable web interfaces. We use on-premise storage and interfaces and in addition offer cloud-based distribution via Amazon Public Dataset program. The current operational interfaces are based on WFS 2.0, WMS and Amazon S3. Most recently added datasets include weather and flood warnings in Common Alerting Protocol (CAP) format and radar data archive via Amazon S3 in GeoTIFF and HDF5 formats. There is development ongoing for providing data via even more developer-friendly interfaces such as the OGC EDR and OGC Features API. Also new data is being added continuously based on our own and user needs.

An impact study has also been conducted for the usage of data from 2018 which revealed some findings on what data is used and how it impacts the users and their potential businesses. Also valuable information on the future needs of users was gathered and the most important findings of this study will be presented during the session. Based on these findings an impact dashboard is currently being setup to continuously monitor the impact and usage of our open data offering.

How to cite: Visa, M.: Open Data from a mixed on-premise and cloud environment at the Finnish Meteorological Institute, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-347, https://doi.org/10.5194/ems2022-347, 2022.

How to cite: Dautz, E., Teubner, I., Auer, M., Beck, A., Pechstein, F., Schöberl, J., Stuxer, B., and Lang, D.: ZAMG Data Hub – Open access to high value data sets, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-380, https://doi.org/10.5194/ems2022-380, 2022.

Weather data and especially a precise weather forecast on mobile devices is of great public interest. However, many apps for mobile devices are closed-source and even a lot of open source weather apps rely on commercial weather data providers. An other drawback of many weather apps is a great number of requested permissions on the device.

The idea was to create a free, open source weather app based on the open data from the Deutscher Wetterdienst (DWD). An other requirement was a privacy-friendly design without user-tracking. Furthermore, the package size should remain low and the amount of data traffic should be limited as far as possible.

The app is continuously developed since 2020. The first version was released in July 2020, and it left beta status in September 2020. It started with a simple weather forecast, features added over time include weather warnings for Germany, weather texts provided by the DWD and a rain radar. Lately, notifications about warnings applying to the selected location were added.

The project is licensed under the “GNU General Public License v3.0 or later”, ensuring that it will remain open source. It uses only open source code, especially no binary or closed-source libraries were used. This contributes to transparency and keeps the size of the program package fairly low (approx. 7 MB, version 0.58.0). The number of required permissions is very limited. Internet traffic only takes place between the app and the open data server of the DWD in order to fetch weather forecast data and weather warnings.

The project also uses non-commercial infrastructures on the web: it is hosted on codeberg.org, published on fdroid.org and translations take place on a private platform. Many volunteers contributed to the project over time and translated it into many languages.

Challenges were a proper implementation of weather warnings without using proprietary push services, and also the correct implementation and use of the open data without any special knowledge on meteorology despite a very good documentation by the DWD.

How to cite: Dube, P.: Tiny Weather Forecast Germany - an open source weather app based on open data from the Deutscher Wetterdienst (DWD), EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-141, https://doi.org/10.5194/ems2022-141, 2022.

In November 2021, DWD launched the pilot phase of the new DWD Geoportal at dwd-geoportal.de. It offers user-friendly exploration of DWD's Open Data, combining access to data, metadata, and documentation along with fulltext search and interactive previews powered by OGC WMS. This new front end covers datasets hosted on opendata.dwd.de as well as other DWD Open Data sources, for instance the Climate Data Centre and the DWD GeoServer.

The contribution will outline the latest version of the new DWD Geoportal and discuss current development and features that are planned for integration in the near future.

Open Data has been a part of the DWD data distribution strategy since 2017, starting with a small selection of meteorological products, but the number of available datasets has grown continuously over the last years. Since the start, users can download file-based meteorological products without registration. Free access and the variety of products have been welcomed by the general public as well as private met service providers. However, the more datasets were provided in a directory structure, the more tedious it became to find and select among all available data. Also, metadata and documentation were available, but on separate DWD websites. The DWD Geoportal adressed these trends, especially having new users of DWD's open data in mind.

Cloud technology is a suitable way forward for hosting the geoportal in it's operational state along with the data. Benefits are expected for the easy integration of rich APIs with the DWD Geoportal, and the flexible and fast deployment and scaling of optional or prototypical services such as WMS-based previews. DWD focuses on cloud technology for new development projects.

How to cite: Reetz, B., Riede, H., Fuchs, D., Jerg, M., and Hagedorn, R.: DWD Geoportal – A central hub for Open Data, API and communication, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-468, https://doi.org/10.5194/ems2022-468, 2022.

How to cite: Lopez, A., Buontempo, C., Suttie, M., Raoult, B., Comyn-Platt, E., and Varndell, J.: A modernised Data Store infrastructure for improving the access to Copernicus Climate and Atmosphere data and services., EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-263, https://doi.org/10.5194/ems2022-263, 2022.

About the EMS Technology Achievement award (TAA): history of the TAA award and introduction to the 2022 winner "The Weather Observations Website (WOW)"

How to cite: Mureau, R.: Introduction to the EMS Technology Achievement Award (TAA) by Robert Mureau, Chair of the TAA Committee, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-726, https://doi.org/10.5194/ems2022-726, 2022.

The Weather Observations Website (WOW) is a cloud-based platform used by the citizen science community to upload and share personal weather observations with anyone around the globe. Currently there are approximately 30,000 users of WOW and more than 1 million observations submitted to WOW each day. Since its release in 2011, WOW has undergone significant technological developments to enhance the sharing of crowdsourced data making it hugely successful. As a result, WOW has served as a tool for creating partnerships between the Met Office and various other National Meteorological Services, who recognise the value of the system and its ability to increase data coverage across areas that are sparse in data. WOW will be awarded the European Meteorological Society (EMS) Technology Achievement Award to formally acknowledge its value at the EMS Annual Meeting in Bonn, Germany.

How to cite: Mylne, K., Gilbert, S., Male, H., Pavelin, E., Sugier, J., De Baar, J., Reyniers, M., Runbäck, J., Walker, J., Alder, K., and Gooding, D.: The Weather Observations Website, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-725, https://doi.org/10.5194/ems2022-725, 2022.

Infoclimat is a non-profit organization created 20 years ago, aiming at facilitating the use, production and dissemination of weather and climate data, to experts and to the citizens, while promoting scientific education towards the general public and children.

The organization, constitued of hundreds of volunteers and no employees at this time, has started its work by gathering sources of open weather data at the beginning of this century. From a few French weather stations under WMO Resolution 40, to more than 18.000 weather stations around the world today, and 6 billion climate records accessible. Today, there are more than 40 sources of data that are processed, from the GTS SYNOP records to national open-data APIs made accessible by NWS all over the world, to custom IoT data transmission for owned weather stations. Often, those records do not use the same standards, data granularity, or time granularity, which make merging them the trickiest part of the platform.

The citizens are also part of this development : more than 1800 of their weather stations, strictly quality-controlled, complement the official networks, maintained by the organization or its contributors in France, or its non-profit counterparts in other European countries. The data produced by the organization network is placed under open licenses, and its access is made possible through a standardized API. The metadata, available to all consumers of the data, is followed closely by a team of enthusiasts and algorithms, which monitors data quality, instruments calibration, and environmental changes. This way, high-quality data is obtained, in environments that are agreed with the national weather service Météo-France.

The platform maintained by the volunteers of Infoclimat provides access to data in a common form, whatever its source, with respect to the producers licenses. There are many uses of the data stored in the platform : climatological analyses, real-time or climate interactive maps, national indicators, and data fusion with gridded products (eg. reanalysis, Copernicus products,...). The tools are made to be accessible to the general public for educational purposes, on computers or smartphones, but they are design so that more professionnal users can dig out the data, analyze them further or download raw data.

Finally, the organization is now changing its scale : from 100% volunteers, to now a first developer, from a budget of 30.000€ in 2015 to 100.000€ in 2022. The platform is shifting towards better internationalization, to make those tools available in more languages, and aims at integrating more regionalized climate predictions (like CORDEX), for better user awareness on climate change. Also, there is a work in progress to make the platform available for scientific teams to store and analyze their data reliably and in a persistent way, for example for researchers on Urban Heat Islands.

How to cite: Ameye, F. and the Infoclimat team: Open-Data and the Citizens: gathering weather and climate data in a digital common, crowdsourcing from the community, and producing value-added tools for the ecosystem, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-189, https://doi.org/10.5194/ems2022-189, 2022.

GeoE3 (Geospatially Enabled Ecosystem for Europe) is a project co-financed by the Connecting Europe Facility of the European Union. It aims to connect existing national, regional and cross-border services and data sets including meteorological, statistical and geospatial such as building data or road network data. This simplifies meaningful analysis and visualization in a national and cross-border context.

GeoE3 develops tools and APIs that will merge available information from national sources. It demonstrates data and service interoperability and creates dashboards and visualizations for an improved understanding of data from a variety of sources. To produce tangible results that can serve as best practices for other domains, disciplines, and areas, GeoE3’s tools and services address compelling use cases. The action will simplify the discovery of relevant data and improve services through the adoption of the latest standards with emphasis on Web APIs and linked data principles.

The three use cases include Solar Energy, Smart Cities and Electric Cars. Investigation of the solar energy potential and energy efficiency of buildings is based on detailed 3D building data, digital elevation models, climate normals, observations and forecasts. The electric cars use case is based on evaluation and prediction of the energy consumption of electric cars as well as providing new services for the European C-ITS platform (Cooperative Intelligent Transport Systems). This use case will use 2D and 3D road data, weather and traffic data, road signs and speed limits. Smart Cities explores renewable energy potential where a planned development area is considered and deals with the optimization of the efficiency of urban expansion.

From a meteorological domain perspective GeoE3 will deliver an implementation of the OGC EDR API built on top of Finnish Meteorological Institute open source SmartMet Server data server. Focus during this year will be on datasets necessary to fulfill the GeoE3 project use cases.

How to cite: Visa, M.: GeoE3 - combining meteorological data with geospatial and statistical data, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-424, https://doi.org/10.5194/ems2022-424, 2022.

Artificial Intelligence (AI) and Machine Learning (ML)-applications have become a huge hype. What does it mean to serve data for AI and ML? EUMETSAT climate reprocessing data records try to meet following guidelines as far as possible.

In ML applications data is typically combined from several sources. Training ML model needs normally a long history of data. Typical environmental ML applications employ 1-5 years of historical data while for example impact forecasts require often at least 10 years of history to contain enough extreme weather samples. ML applications are often trained with history data but applied to near-real-time (NRT) data. Thus, corresponding NRT data should be always available.

The historical data series should obviously be as harmonised as possible. However, the harmonisation doesn’t need to be perfect. Small changes in the data are not necessary affecting the performance of ML model too much. The changes in the underlying data should well documented.

Data quality is also very important aspect as ML models are just as good as underlying data. Thus, quality flags should be always available and provided in a way that they can be used to filter out bad samples. While reasonable assumption for default is to provide only good quality data, also other samples should be available as sometimes more lower quality data yields better results than less higher quality data. Whenever possible, users should be provided with option to access the raw data as well since it may open avenues to new ways to apply ML models or pre-process.

Data access should be obviously as fast as possible and all data should always be served from online data storage. As datasets are almost always combined with each other, data formats should be as well-known and supported as possible, even that would mean loss of metadata. Typically, it’s better to provide metadata beside the actual data and keep the data as consist as possible.

Some of the ML methods, such as Random Forests (RF) are more often used for supervised learning to specific points while i.e. neural networks (NN) are used for images and gridded fields, tensors. Serving data for point-based applications greatly benefits from API capable to provide best representative samples for any given point so that it’s easy to be combined with labels. Serving data for grid-based applications, however, benefit of relatively raw interfaces, such as S3, with wide client support.  Critical requirements for the interface and the data model is to enable sub-setting and slicing.

Finally, providing well-known and documented reference datasets with ready labels would be highly beneficial for ML developers. Such general domain datasets, such as the Iris Dataset already exist. Meteorological community should publish such datasets along with ready methods in common libraries to load the dataset easily.

How to cite: Tervo, R. and Grant, M.: Providing AI- and ML-ready data, EMS Annual Meeting 2022, Bonn, Germany, 5–9 Sep 2022, EMS2022-17, https://doi.org/10.5194/ems2022-17, 2022.