The UK's open data journey, ignited in 2010, saw the public sector lead the charge, epitomised by Transport for London's (TfL) 2017 initiative, yielding £130 million in annual savings. Third-party developers thrived, leveraging TfL's data for innovative solutions, reshaping London's urban landscape. The Department for Environment, Food & Rural Affairs (DEFRA)'s 2015 open data surge birthed diverse applications, from flood risk assessments to archaeological discoveries. However this ‘big bang’ of open data was often done haphazardly and as a box-ticking exercise, so open data leaders shifted the paradigm from 'open it and they will come' to 'publish with a purpose'. The public sector responded with more powerful open data services such as DEFRA's Earth Observation Data Service, that remain as open as ever, but provide targeted around specific use cases.

On the other side, the private sector has come later to the party, and initially with less consistency, with open data rationales ranging from regulatory mandates to collaborative initiatives. Open Banking, despite its misleading moniker, epitomises this synergy and tension, spurred by government intervention demanding open data and secure data sharing. The ecosystem, fostered by open standards and APIs, burgeons with diverse offerings, from financial management to debt advice. Learning from this, initiatives like OpenActive, Open Energy and Stream, in the water sector, employ incentives to cultivate similar data ecosystems organically. Spearheaded by organisations like the ODI and Icebreaker One, these platforms facilitate open data and secure data sharing while fostering societal benefits. 

Recognizing the wealth of data held by private entities, visionary companies embrace collaboration as the cornerstone of innovation. Microsoft's Industry Data for Society Partnership, in partnership with ODI and GovLab, spearheads this movement, bridging the data divide globally. Through initiatives like the Planetary Computer, Microsoft champions open data infrastructure, fostering collaborations to tackle pressing issues like climate change and education. Guided by principles of openness, usability, and empowerment, these endeavours pave the way for responsible data sharing, propelling society towards a more data-inclusive future.

Currently, we are beginning to recognise the best stewards of data are often neither government nor private sector, but independent vehicles set up to steward data on behalf of society for a specific purpose related to social impact. These organisations, data institutions, can publish open data, share sensitive data responsibly, and maintain key infrastructure for open data such as standards, identifiers and policies. Although nascent in their conceptual understanding, many already exist in the environment and climate space, providing access to data about climatic trends, forest cover, extractive industries and more. These organisations, operating in more networked and decentralised ways may be the future of the open data, but their impact is only just beginning to be understood.

How to cite: D'Addario, J. and Achenbach, K.: The State of UK Open Data 2024: From big bang open data to responsibly stewarding data with a purpose, EMS Annual Meeting 2024, Barcelona, Spain, 1–6 Sep 2024, EMS2024-413, https://doi.org/10.5194/ems2024-413, 2024.

ECMWF is a leading provider of weather information, and its products and weather data make contributions to a huge range of activities in service provision, research and for the protection of life and property via National Weather Services and humanitarian agencies. 

Since 2020, ECMWF has been implementing an open data policy, recognising that open data under Creative Commons licensing is an essential tool in contributing to the development of new meteorological methods and is encouraging innovation and progress in the weather and climate domains.

ECMWF plans to achieve this goal by 2026. A number of steps have been taken to increase the open data available to users and in the last year ECMWF has built on this further. The data currently released as open data are a subset of ECMWF Real-time catalogue at 0.25 degree resolution, from both ECMWF Integrated Forecasting System (IFS) and Artificial Intelligence/Integrated Forecasting System (AIFS).

The usage statistics on ECMWF open data for 2023 show a significant increase of user adoption with respect to the previous year, indicating the users are engaging with ECMWF open data offerings.

ECMWF Open Data are provided through different mechanisms, all of which have seen a notable growth in usage in 2023:

• Open Charts Service, which provides chart visualisations of ECMWF products. Some of the increased uptake can be attributed to the introduction of charts based on Machine learning products in mid-2023.
• The Open Data Portal (FTP service) is available since January 2022 and shows a continuing increase in numbers of requests. ECMWF predicts even higher usage with the addition of new Open Data of IFS products at higher resolution and additional parameters, plus AIFS products (machine learning-based) in early 2024.
• WMO Essential and WMO Additional datasets (also provided via FTP service) are still increasingly popular. ECMWF will, in the future, optimise this provision, possibly merging it with the open data offering.
• Since January 2022, ECMWF Open Data is also redistributed by 3^rd ECMWF has partnered with different cloud providers as part of their public dataset program (e.g. Amazon, Google and Microsoft), but the data has also started being provided by smaller independent users (e.g. Open-meteo.de) and in research archives (e.g. NCAR). The benefit is not only a good user uptake but also that data can be shared with a larger user community with no additional load on ECMWF systems.

ECMWF is committed to moving forward to achieve the goal of an open data policy over the next few years. Creative Commons licences will be applied to further datasets, while reducing and eventually removing the cost of data. The main objective of these changes is to continue to encourage collaboration, innovation and progress within the meteorological and climate science communities.

How to cite: Parodi, I., Coughlan, R., Kuilman, M., Pidduck, E., Bennett, V., Yang, X., and Modigliani, U.: Uptake of the ECMWF Open Data Programme, EMS Annual Meeting 2024, Barcelona, Spain, 1–6 Sep 2024, EMS2024-749, https://doi.org/10.5194/ems2024-749, 2024.

ECMWF are moving towards an extensive open data policy, providing data to a broader user base extending beyond operational forecasters in Member and Co-operating states and commercial customers. Beginning in 2020, the first phase saw the opening of hundreds of web forecast charts (www.charts.ecmwf.int) and made archived data available under a Creative Commons (CC BY 4.0) open licence. This momentum continued in January 2022 with the introduction of an open subset of real-time medium-range forecast data, with ongoing updates incorporating new parameters and datasets. Notably, the latest update in February 2024 included increasing the resolution from 0.4° to 0.25° and included the new Artificial Intelligence/Integrated Forecasting System (AIFS) model data.

This phased move towards free and open data aims to support creativity, innovation and reproducibility in scientific research and weather applications. However, this can not be achieved by only opening the real time data. The users need to be able to find and easily use the data and integrate it into their own research work or application workflows.

To address this, additional efforts are underway to improve the data's FAIR (Findable, Accessible, Interoperable and Reusable) attributes. Key developments include the creation of open source Python libraries for data processing and visualisation, alongside the introduction of a set of Jupyter notebooks, each of which is reproducing one open weather forecast chart - from the downloading the data to the visualisation.

However, the tools and data constantly change, and keeping up with these changes presents a significant challenge if not designed with maintenance in mind.

This talk will provide an overview of the open forecast web charts and the use of Jupyter notebooks for their reproduction, followed by an exploration of the maintenance challenges and future plans.

How to cite: Vuckovic, M.: Making ECMWF open weather forecast data more accessible to users: maintenance challenges, EMS Annual Meeting 2024, Barcelona, Spain, 1–6 Sep 2024, EMS2024-815, https://doi.org/10.5194/ems2024-815, 2024.

The WegenerNet climate station network Feldbach Region has provided measurements of hydrometeorological near-surface and surface variables with very high spatial and temporal resolution since 2007. It was specifically built as a long-term weather and climate monitoring facility that provides an open-ended data record to help fill the gap between short-term, high-resolution observations and long-term observations at larger scales. In 2020, this ground station network was complemented by the WegenerNet 3D Open-Air Laboratory for Climate Change Research, which has added atmospheric sounding capabilities through a polarimetric X-band Doppler weather radar, a microwave radiometer for vertical profiling of temperature, humidity, and cloud liquid water, an infrared cloud structure radiometer, and a water-vapor-mapping GNSS station network. Together, these facilities provide a unique setup for studying extreme meteorological events in a changing climate.

To maximize the impact of these data an open data store and open science tools, based on widely adopted and recognized (metadata) standards, are in the process of being implemented as part of the project WEGN2OPEN. Its focus lies on the provision of innovative high-resolution data cubes with a high level of interoperability to allow for an easy integration into existing applications and workflows.

The aim of this contribution is to showcase the current state of the WEGN2OPEN data store, along with planned advancements and features. This includes an overview of the currently available APIs through which datasets can be discovered and accessed, the web application for browsing, viewing, and subsetting data, and an outlook to a future extension with cloud-based analysis and processing tools.

How to cite: Kvas, A., Fuchsberger, J., Gorfer, M., Kirchengast, G., Scheidl, D., Bichler, C., Galovic, R., and Foelsche, U.: WEGN2OPEN - The WegenerNet Open Data and Science Laboratory Region Southeastern Austria, EMS Annual Meeting 2024, Barcelona, Spain, 1–6 Sep 2024, EMS2024-958, https://doi.org/10.5194/ems2024-958, 2024.

The modernized Copernicus Climate and Atmosphere Data Store infrastructure – released in 2024 – uses state-of-the-art technology to provide a more scalable and open access to data and services with the adoption of the FAIR principles and Open Data at its core. The infrastructure, named the Common Data Stores – Engine (CDS-E), exposes to the public two different frontends, the Climate Data Store (CDS) in support of the Copernicus Climate Change Service (C3S) and the Atmosphere Data Store (ADS) for the Copernicus Atmosphere Monitoring Service (CAMS) whilst sharing all the backend components. Combined, the CDS and ADS provide service to more than 300k registered users, delivering over 180 TBs of data on daily basis in the form of more than 700k processed requests. It also seeks to strengthen interdependencies with other European platforms such as the WEKEO Data and Information Access Platform (DIAS), Climate-Adapt, ClimSA, and facilitates interactions with different platforms in the future E.U. Green Deal Data Space, thereby fostering a broader engagement with the user community.

The CDS and ADS allow free and open access to the wide range of climate and atmospheric monitoring datasets through both a web-based and programmatic interface. The modernized infrastructure makes use of modern cloud technology and an open framework that bring new components to the CDS and ADS. These include a variety of adaptors, modern standards and protocols, such as OGC compliant APIs, STAC metadata standards for programmatic interrogation of geospatial information, Serverless Analysis-Ready Cloud Optimized (ARCO) Services and Standard WMS/WMTS for improved integration, access, and visualisation. This allows the CDS and ADSs to support a seamless experience on the user's journey to explore, discover, retrieve and use the available resources whilst remaining scalable and versatile.

At the core of its versatility is the implementation of FAIR principles (Findable, Accessible, Interoperable, Reusable). The adoption to the FAIR principles has in fact followed a gradual and systematic approach that has kept the platform and its content under continuous evolution and improvement. Here we describe how the adoption of the FAIR principles and an Open Data philosophy was implemented and outline the practical benefits of this approach to the user community.

How to cite: Zanacchi, M., Alos, A., Fuentes, M., and Damasio-Da-Costa, E.: The modernized Climate and Atmosphere Data Store infrastructure – a practical implementation of FAIR and Open Data, EMS Annual Meeting 2024, Barcelona, Spain, 1–6 Sep 2024, EMS2024-1089, https://doi.org/10.5194/ems2024-1089, 2024.

Over the past year, we have conducted a feasibility study and initiated a pre-project phase for the Open Earth Platform Initiative, aimed at harnessing open data and technology to drive climate adaptations. This initiative seeks to stimulate local innovation underpinned by global investment, thereby facilitating projects that are not only of higher quality and lower risk but also have a shorter development time from idea to product.

A key challenge identified is the scarcity of platforms that truly embrace an open approach. Vendor lock-in hinders the accessibility of open data, which often leads to both duplication of efforts and a prolonged process from data acquisition to product development. To combat these issues, our platform adheres to the FAIR principles, promoting findability, accessibility, interoperability, and reusability of data.

In the realm of open-source software, we emphasize the benefits of reuse and active involvement. While addressing the common concerns of security and vulnerabilities, we advocate for the synergy between open source software and open data, which enhances the overall effectiveness and innovation of the platform.

OpenEPI is designed with software developers as the primary end-users. A central feature is our developer portal, which is pivotal for ensuring developer engagement and productivity. The architecture of the platform champions vendor independence and utilizes a multicloud strategy to optimize performance and scalability. We rely on best-of-breed technologies, predominantly Python and its modern libraries and frameworks, alongside robust containerization practices.

In this talk at EMS2024, we will delve into the findings from our pre-project phase and provide updates on the current status, discussing how our strategic frameworks and practical implementations are being tailored to collectively foster the development of sustainable and impactful climate adaptation solutions.

How to cite: Stigen, K.: Open Earth Platform Initiative, EMS Annual Meeting 2024, Barcelona, Spain, 1–6 Sep 2024, EMS2024-1047, https://doi.org/10.5194/ems2024-1047, 2024.

When data is made accessible in the cloud, you can shorten the time to insights and help build impactful offerings for climate modelers and stakeholders. When data is shared on the Registry of Open Data on AWS, anyone can analyze it and build services on top of it using a broad range of compute and data analytics products, including Amazon EC2, Amazon Athena, AWS Lambda, and Amazon EMR. Sharing data in the cloud lets data users spend more time on data analysis rather than data acquisition. Just putting data into the cloud isn’t enough. In this session, the Amazon Web Services (AWS) Open Data team will discuss different options for creating cloud-optimized versions of data, that can be more efficiently utilized in the cloud. We will discuss the various options and trade-offs, such as when to create a full cloud optimized copy versus when to create a subset for your user community. We discuss how to match up the different options based on different levels of traffic, newness of data, and typical types of processing of the data. We will show how to use high quality climate and environmental datasets with the Registry of Open Data on AWS when they are cloud optimized and when they are not, and discuss the time difference for the data consumer between each option. Additionally, we will also show how to harness these data using low- and no-code machine learning tools such as SageMaker and SageMaker Studio Lab, using cloud-optimized capabilities such as subsetting, and how these capabilities can speed up the time from data to decision.

How to cite: Stoner, C.: Open Data for Impact: how to use the Open Data on Amazon Web Services to advance analysis, EMS Annual Meeting 2024, Barcelona, Spain, 1–6 Sep 2024, EMS2024-1120, https://doi.org/10.5194/ems2024-1120, 2024.

The RODEO project responds to the requirements of the EU Directive (2019/1024) on Open Data and its Implementing Regulation in order to boost the re-use and combination of open public data across the EU. The Implementing Regulation defines High Value Datasets (HVD) for six thematic categories of public data of which weather observation data, climate data, warnings, weather radar data and Numerical Weather Prediction (NWP) data are defined as meteorological High Value Datasets. These HVDs shall be shared free of charge, under the conditions of the Creative Commons BY 4.0 licence or a less restrictive open licence, and will be openly accessible via Application Programming Interfaces (APIs), machine-readable and bulk downloadable.

The three-year RODEO project is a joint effort by 11 European meteorological institutions, the European Centre for Medium-Range Weather Forecasts (ECMWF) and the network of 31 European National Meteorological and Hydrological Services, EUMETNET. The project strengthens the capacity of the European meteorological data providers by

• Developing a user interface and a data catalogue for making data discoverable;
• Developing APIs, using open licences, for accessing weather observation data, climate data, weather radar data, warnings, and AI datasets;
• Engaging with the data owners and user communities; and
• Supporting the deployment of national data portals and APIs.
• Implementing a comprehensive user engagement strategy involving
  • A Project External Advisory Board, composed of public and private sector partners
  • Frequent communications updates through several channels

The project is now in its second year. It has moved on from its design phase into developing and testing some of the APIs that are going to enable a large increase in the open availability of near real time European meteorological data. This will bring large benefits across society as a whole from the public through to decision makers. In particular, small and medium-sized enterprises (SMEs) will benefit from the greater data availability by creating new digital products and services and eventually attract new investors.

How to cite: McCairns, W.: RODEO Project – bringing meteorological data open for all users, EMS Annual Meeting 2024, Barcelona, Spain, 1–6 Sep 2024, EMS2024-1071, https://doi.org/10.5194/ems2024-1071, 2024.

Recently there has been many initiatives in the meteorological community and outside on implementing OGC API interfaces, especially OGC API EDR. Based on these efforts it seems that OGC API EDR will be a foundation technology in many initiatives and domains including WMO WIS2 (1), EUMETNET FEMDI (2) (aiming to provide high-value datasets required to the EU open data directive) and ICAO SWIM (3) services in the aviation domain.  This presentation will touch on how the challenges are adressed in context of the RODEO project between participants in the project. In the project it was noticed that the implementations of different insitutes complied with the standard but were in fact still not compatible as the output was different. This led to the need and discussion on how we would achieve more interoperability between the actual output content provided by the API's. It was clear that some kind of agreement was needed between the partners in the project. Work on this challenge has also been started in the OGC context as well to add support for so called profiles. Challenges, current status of this work and lessons learned during the development will be presented.

(1) WMO Information System 2.0: framework for WMO data sharing

(2) EUMETNET FEMDI: Federated European Meteorological Data Infrastructure. Unified access to European weather data.

(3) ICAO SWIM: System Wide Information Management concept consists of standards, infrastructure and governance enabling the management of ATM related information and its exchange between qualified parties via interoperable services (ICAO Doc.10039). This includes weather information.

How to cite: Visa, M. and Rauhala, M.: Lessons learned on implemenation of the OGC EDR API, EMS Annual Meeting 2024, Barcelona, Spain, 1–6 Sep 2024, EMS2024-1024, https://doi.org/10.5194/ems2024-1024, 2024.

In this presentation, we explore the pivotal recommendations for climate data sharing developed by Creative Commons in collaboration with leading organizations including the European Centre for Medium-Range Weather Forecasts (ECMWF), International Organization for Standardization (ISO), NASA, NOAA, the World Meteorological Organization (WMO), and the World Resources Institute (WRI). Our focus is on enhancing the accessibility, interoperability, and reusability of climate data through comprehensive legal and licensing terms and the strategic use of metadata values centered on attribution.

Given the urgency of addressing the climate crisis, our recommendations are designed to advance the goals of EMS 2024 by providing actionable guidelines for data managers, researchers, and policymakers. These guidelines aim to maximize the impact of climate data sharing by ensuring that data is both open and usable, thus fostering more effective climate research and policy development.

Our recommendations, the result of a nine-month collaborative effort, mark a significant step forward in improving the accessibility and reuse of climate data. We offer two primary licensing options: Option A, which utilizes CC0 plus an attribution request to dedicate data to the public domain while requesting attribution, and Option B, which employs CC BY 4.0 to maintain data ownership with enforceable attribution. These options address common challenges related to license stacking and attribution stacking, providing clear pathways for managing multiple climate data sources.

Additionally, we propose standardized metadata values that enhance both human and machine readability, ensuring greater transparency and reducing ambiguity in data sharing. By building on existing metadata schemas and standards, our recommendations aim to fill critical gaps by standardizing the inclusion of essential information related to attribution, licensing, and provenance.

Effective climate data management is further supported by our emphasis on four key practices: appointing dedicated technical and legal/policy stewards, encouraging collaborative data sharing, and regularly updating data sharing policies in line with evolving open data standards.

We invite attendees to engage with these recommendations and contribute to an ongoing dialogue on best practices for climate data sharing. By working together, we can drive forward innovations in climate research and enhance our collective ability to tackle the climate crisis. For further information and to access the full report and summary, please visit the provided links or contact us at openclimatedata@creativecommons.org.

How to cite: Campbell, T., Ainali, J., Li, W., and Green, C.: Creative Commons Recommended Best Practices for Better Sharing of Climate Data, EMS Annual Meeting 2024, Barcelona, Spain, 1–6 Sep 2024, EMS2024-1159, https://doi.org/10.5194/ems2024-1159, 2024.