Enabling access to harmonised ECV-related observation datasets from environmental Research Infrastructures

In the ENVRI-Hub-NEXT (EHN) project, environmental Research Infrastructures (RIs) collaborate within the European Open Science Cloud (EOSC) to improve access to observation datasets related to Essential Climate Variables (ECVs). The main goal is to enable users from any Virtual Research Environment (VRE) to process and analyse ECV-related data using ENVRI-Hub components. To support this, EHN provides a GUI-based Catalogue of Services (CoS) that describes RI services and datasets using an extension of DCAT (EPOS-DCAT-AP), complemented by a Catalogue of Data based on FAIR Data Points. Despite this, dataset discovery and federation remain challenging due to heterogeneous machine-to-machine services and differing vocabularies for observed variables.

To address these issues, an ECV Working Group was established within EHN to define an approach for matching ECVs as defined by the Global Climate Observing System (GCOS) to the diverse variables managed by RIs across multiple environmental domains. An ECV is defined as a physical, chemical or biological variable, or a group of linked variables, that is critical for characterising Earth’s climate. A key outcome was the publication of ECV concepts linked to the GCOS definitions, in a machine-readable vocabulary in the NERC Vocabulary Server (NVS). It enabled mappings to RI-specific vocabularies using the I-ADOPT approach, and the use of SPARQL queries to establish dynamic “ECV to observable properties” translations. Python notebooks were developed to interact with RI data access services, including a central notebook that translates a single ECV request into multiple RI-specific queries and data access requests. This work exposed limitations in the vocabularies used for observed parameters, as well as in the availability of direct and harmonised data access services.

In the next phase of EHN, several upgrades are planned to improve data accessibility and usability. All RIs will receive training on describing observational datasets using I-ADOPT-compliant vocabularies following recommended practices. 

Because RI machine-to-machine services rely on different APIs and constraints, they cannot be queried uniformly. The ECV data access library developed earlier in the project translates a single ECV request into the multiple requests required to query relevant RI services, using I-ADOPT mappings to identify RI parameter sets. This library will be further optimised, while RIs work towards more harmonised and direct data access services.

Many RIs still lack direct data access and especially subsetting capabilities, instead offering file-based or aggregated access via metadata search. Experience gained through the notebooks will guide improved integration of available services into the CoS. All notebooks and scripts will be released as open source and integrated into the ENVRI-Hub Analytical Framework, including a JupyterLab extension. As analytical services require harmonised data chunks rather than heterogeneous files, the next stage will test subsetting solutions such as Beacon, ERDDAP and Zarr.

The presentation will highlight the implemented solutions and opportunities for broader uptake within the EOSC domain.