Research data infrastructures (RDIs) are evolving and driven by diverse initiatives ranging from international to local, e.g., institutional scales, who try to build sustainable and useful services. The German national data infrastructure for Earth System Sciences (NFDI4Earth) aims to support researchers in 1) discovering and exploring relevant data sources, 2) data publication and curation, 3) solving research data management problems and 4) creating and publishing information products. In the development of the  software architecture for NFDI4Earth, we face challenges of computational, social, cultural, and strategic nature. Here, we are going to present an overview of these challenges and early outcomes and to reflect  first lessons learned from the initial concept and development phase of the NFDI4Earth architecture.

Starting with the (meta) data layer, the landscape of existing ESS services and repositories is diverse and features various metadata and data, like governmental (meta) data following INSPIRE, OGC, and ISO19xxx. This diversity demands harmonising and linking concepts that fit to standards for metadata, data and services, such as OGC APIs or ISO19xxx, as well as to Semantic Web concepts, e.g., FAIR Digital Objects, and provide extension points for (newly developed) specific formats. At the same time, the software stack and technologies of the business layer should consider interoperability, openness and sustainability aspects while providing a flexible solution to manage the distributed metadata. Moreover, in our case, activities on developing (meta) data and business layer concepts also include coordinating a software developer team with different scientific and technological backgrounds spread across several institutions.

NFDI4Earth is located in a dynamic landscape of ESS services and repositories which are often not sustainably funded. Hence, we need to implement  practices and collaborations to link or integrate further software, services, and information products so that an up-to-date living and evolving architecture serves the needs of researchers.

How to cite: Henzen, C., Degbelo, A., and Nüst, D.: Challenges in Developing a Software Architecture for a National Research Data Infrastructure in Earth System Sciences, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-17491, https://doi.org/10.5194/egusphere-egu23-17491, 2023.

The InterPlanetary File System (IPFS) is a novel decentralized file storage network that allows users to store and share files in a distributed manner, which can make it more resilient if individual infrastructure components fail. It also allows for faster access to content as users can get files directly from other users instead of having to go through a central server. However, one of the challenges of using IPFS is ensuring that the files remain available over time. This is where an IPFS pinning service offers a solution. An IPFS pinning service is a type of service that allows users to store and maintain the availability of their files on the IPFS network. The goal of an IPFS pinning service is to provide a reliable and trusted way for users to ensure that their files remain accessible on the IPFS network. This is accomplished by maintaining a copy of the file on the service's own storage infrastructure, which is then pinned to the IPFS network. This allows users to access the file even if the original source becomes unavailable.

We explored the use of the IPFS for scientific data with a focus on climate data. We set up an IPFS node running on a cloud instance at the German Climate Computing Center where selected scientists can pin their data and make them accessible to the public via the IPFS infrastructure. IPFS is a good choice for climate data, because the open network architecture strengthens open science efforts and enables FAIR data processing workflows. Data within the IPFS is freely accessible to scientists regardless of their location and offers fast access rates to large files. In addition, data within the IPFS is immutable, which ensures that the content of a content identifier does not change over time. Due to the recent development of the IPFS, the project outcomes are novel data science developments for the earth system science and are potentially relevant building blocks to be included in the earth system science community.

How to cite: Kulüke, M., Kindermann, S., and Kölling, T.: IPFS Pinning Service for Open Climate Research Data, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-6311, https://doi.org/10.5194/egusphere-egu23-6311, 2023.

The European Plate Observing System (EPOS) is releasing a pan-European research infrastructure (RI) for Solid-Earth sciences that targets different scientific communities. The EPOS RI enables sharing of data and resources; promoting collaboration, harmonization of practices and methods, fosters innovation and novel scientific discoveries. These principles of scientific and technological collaboration are the basis of the concepts of Open Source and Open Collaboration.

EPOS consists of essentially two components: firstly, the so-called Thematic Core Services (TCS) representing the Data providers from the scientific domains (e.g., Seismology, Satellite data etc.); secondly, the central integration node, namely the ICS-C (Integrated Core Services – Central Hub), representing the integrating ICT (Information and Communication Technology) system underpinning the EPOS Data Portal.

EPOS is currently releasing the Open-Source version of its architecture, based on microservices, which includes a GUI (Graphical User Interface) implementing the Data Portal, i.e., the human oriented interface for accessing the assets made available by the TCS. It communicates with the ICS-C system by means of RESTful APIs which also implement AAAI (authentication, authorization, accounting infrastructure). Through the APIs, the Data Portal queries the metadata catalog to discover and contextualize assets of interest provided by the TCS and documented as metadata.

In order to integrate heterogeneous datasets from the TCS, appropriate metadata and semantic descriptions are used to drive interactions with TCS resources or to construct a workflow to be executed across TCS.

EPOS Open Source also includes a microservice to enable the interaction with large-scale computing resources or geoscience software services, represented in EPOS as ICS-D (Integrated Core Services – Distributed). The different processing done through the ICS-D on the TCS data are also metadata driven, the software executed on the ICS-D, which enable additional features and functionalities to the ICS-C core, have their own metadata description and through a plugin architecture run on the ICS-D.

The architecture is designed to integrate with e-Infrastructures such as GRID or CLOUD facilities and particularly ongoing work includes achieving interoperability with EOSC (European Open Science Cloud) by means of FAIR web services. The EPOS architecture has also been used as a template in other initiatives such as other Environmental Science RIs (e.g., ENVRI Catalogue of Services) and Jerico.

In the presentation we will describe the work done so far and the key concept that brought to the adoption of a microservice based, open-source released architecture, and provide perspectives for future extension of the project.

How to cite: Vinciarelli, V., Orfino, A., Paciello, R., Bailo, D., Goffi, C., Giuliacci, K., Sbarra, M., Michalek, J., Nedrebø, H., Roquencourt, J.-B., Retout, Y., Warren, D., and Lavrnja-Czapski, J.: EPOS Open Source: A platform for integrating high-quality research products and services., EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-6400, https://doi.org/10.5194/egusphere-egu23-6400, 2023.

Over the last years, several repositories with curated environmental datasets have been created so that scientific communities have gained access to large collections of data from various domains. The level of data harmonisation and FAIRness, technical readiness and scalability of these repositories differs substantially. This restricts data exploration opportunities and limits scientific exploration with modern data science methods, such as machine learning. In­ the domain of air quality research, we have pioneered a data infrastructure for global observations of surface ozone and other air pollutant measurements that comes with rich possibilities for online data analysis. The data in the Tropospheric Ozone Assessment Report (TOAR) database is collected from about 40 different resource providers, from national and international environmental agencies to individual research groups around the world.
One of these data providers is OpenAQ, the world's first open, real-time air quality platform. Due to the higher standards of curation, the need for data harmonization, and the enriched metadata in the TOAR database, we had to develop an automated workflow to transport archived and real-time data from this provider to the TOAR database. The primary step is to clean and format all the OpenAQ records, according to the TOAR database schema, and concurrently, refine the metadata. The workflow includes tests for data sanity and checks if time series and station metadata can be amended, or whether new time series or station records must be created. The automation manager triggers the workflow hourly, so the database provides clean and updated air quality data at any time. 
The presentation describes the automated workflow and its design principles and discusses how such a workflow might be re-used in other environmental domains. All TOAR-related codes are open source.

How to cite: Kreshpa, E., Schröder, S., Selke, N., and Schultz, M.: An Automated Data Ingestion Workflow for the TOAR Database, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-7455, https://doi.org/10.5194/egusphere-egu23-7455, 2023.

The interconnectivity of existing data infrastructures (DIS) across national and international initiatives (e.g. NFDI, EOSC and others) is an important goal to create a common interoperable data ecosystem. To achieve this, it is critical to harmonize the existing methods and concepts of research data collection, research data-reuse, among the DIS and along the FAIR principles.

Within the Helmholtz Association we maintain more than 50 data infrastructures in the field of Earth and Environment. Procedures of data handling, documentation and storage are hardly coordinated within Helmholtz, even less so within the larger community. To find out about the state of our infrastructures, the different approaches in data management procedures, technical capabilities, and concepts, we conducted a survey among all Helmholtz DIS. We asked questions related to their roles in the community, self-perception, quality control, curation, technology interfaces, data re-use and demands.

Based on this data we developed our vision to create a “Helmholtz data space”, unifying Earth and Environmental Centres and infrastructures and powering a new wave of large-scale, globally oriented, data driven research. The Helmholtz Metadata Collaboration’s (HMC) mission is, to federate (meta)data systems across Earth and Environment Centres and infrastructures throughout the Helmholtz Association, continuously aligning Helmholtz capacities to global norms and developments.

How to cite: Soeding, E., Poersch, A., Weinelt, M., Buttigieg, P. L., Kollai, H., Lorenz, S., and Razeghi, Y.: Towards a harmonized data ecosystem in Earth and Environment– a view on the Helmholtz Association’s Data Infrastructures, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-8353, https://doi.org/10.5194/egusphere-egu23-8353, 2023.

ESA Datalabs is a collaborative scientific platform of the European Space Agency to exploit data across the ESA science directorate missions’ archives (astronomy, planetary and heliophysics). It allows you to bring your code to the data under a private account, shareable with colleagues. Large amount of data such as the GAIA multi-billion stars catalogues can be easily mounted and searchable, allowing large scale scientific investigations impossible to achieve on a regular laptop. It also provides multiple tools to access, process and visualize JWST data and was used during the recent commissioning of JWST. In other words, it handles both public and restricted access data.

In the heliophysics domain, data from a few missions are already mounted, including all public data from the Solar Orbiter mission. A few Jupyter notebooks are already available to help the community making use of the full capabilities of the ESA archives. More will be made available in the future including tools such as JHelioviewer and data mining. Interoperability is at the heart of the ESA datalabs infrastructure and connection to clouds such as the NASA heliocloud and Amazon Web Services accounts (AWS) are in progress. Developed over the past few years, ESA Datalabs is scheduled to be public to the scientific community in 2023. 

How to cite: Masson, A., Merin, B., Navarro, V., Arviset, C., and Middleton, H.: ESA DataLabs: an open science platform relevant to Heliophysics, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-12254, https://doi.org/10.5194/egusphere-egu23-12254, 2023.

The world of geosciences is very broadly positioned on the path to Open Science. The development of a research infrastructure must therefore address the fundamental needs within each discipline. In the geosciences, the spectrum of requirements for the data alone ranges from highly standardized real-time, large data made available internationally (e.g. in seismology or geodesy) to the full spectrum of small and highly variable data from long-tail data communities with best practices of sharing data via tables printed in research articles.

FID GEO is the specialized information service for Geosciences in Germany offering publication services and consulting around the full spectrum of Open Science in the Geosciences since 2016. Funded by the German Research Foundation (DFG), FID GEO is a service of the GFZ German Research Centre for Geosciences (GFZ) and the Göttingen State and University Library (SUB). The project provides broad access to digital knowledge resources, contributing to an open information infrastructure. The service portfolio includes electronic publishing of research results in our domain repositories. GEO-LEOe-docs, the repository for texts and geological maps, is hosted at SUB and GFZ Data Services, the domain repository for research data and scientific software, is hosted at GFZ. In addition, we offer digitization services, especially for (older) journal series and reports and a broad consulting portfolio on Open Science topics. Hereby, FID GEO advocates a holistic view of the chain of scientific results – from sample to data and software to scientific articles – and promotes that the individual elements are digitally linked in the best possible way.

From the beginning on, FID GEO developed services that facilitate the shift towards Open Science by engaging the geoscience community. The FID GEO website, our newsletter and Twitter account are tools to connect us with the community. We inform the majority of the German geoscientists with regular publications in the journal “GMIT - Geowissenschaftliche Mitteilungen”, which is also being published on GEO-LEOe-docs since 2021. Over the last six years, active interactions with the community during conferences, workshops, talks and through online questionnaires, revealed that there still is a high demand for information on open science practices.  Workshops and talks have proven to be very successful tools to meet the large need for discussion. They not only allow us to directly address questions or uncertainties regarding practical aspects of open science practices, but they also offer the suitable framework to prepare the information specifically for each research group. To improve our publication services and to intensify the open information culture in the geosciences, FID GEO collaborates with strategic (inter)national initiatives (like NFDI4Earth), with German geosciences societies and other library-related projects supporting the development of open research data infrastructures.

How to cite: Lorenz, M., Elger, K., Achterberg, I., and Semmler, M.: Establish Open Science practices throughout the geoscience community in Germany with the FID GEO services, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-12272, https://doi.org/10.5194/egusphere-egu23-12272, 2023.

In recent years, the requirements for data from earth system sciences have increased massively. Data from observation systems needs to be transferred into larger research data infrastructures, evaluated and flagged via well-defined quality checks, enriched with standardized metadata and finally made available to the public via standard interfaces. And in order to fulfill the FAIR principles, we have to ensure transparency and reproducibility of all these steps. Moreover, the rising demand of near-real-time (NRT) data requires the whole data pipeline to run operationally with minimal manual effort.

However, in many cases, there are still heterogenous data landscapes to be found without centralized control of data, data processing, version control and QA/QC. This is often aggravated by to inconvenient, outdated and isolated tools and software solutions.

Therefore, we develop and implement an adaptable automated pipeline, which combines the assurance of data consistency, QA/QC (Quality Assurance / Quality Control), graphically supported validation and unified persistence and publication of data. User friendliness is achieved by making the system configurable and trackable through lightweight user interfaces over the complete data lifecycle. By only using open-source software solutions and applying community standards for data formats and interfaces, a high level of sustainability and independence can be ensured.

In this presentation, we hence want to demonstrate such an end-to-end data pipeline that finally allows for the FAIRification of typical environmental sensor data.

How to cite: Louisot, B., Lorenz, C., Hadizadeh, M., Völksch, I., and Schäfer, D.: User-friendly data pipeline for FAIRification of environmental sensordata, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-13659, https://doi.org/10.5194/egusphere-egu23-13659, 2023.

This presentation summarizes information on the needs of data producers and data users gathered from the combination of a March 2022 two-day on-line workshop and the nearly 30-year history of the NASA Earth Observing System Data and Information System (EOSDIS) Distributed Active Archive Centers (DAACs).  The workshop included over 100 participants, primarily from the United States, with the first day focusing on the needs of data users and the second day focusing on the needs of data producers.  Based on both the workshop and the collective experience of the DAACs, both data producers and data users benefit substantially when there is an early partnership between the research project producing the data and the data archive which will publish the data.  The highly heterogenous nature of airborne and field research data presents  particular challenges for discovery, particularly in the context of systems that are optimized for discovery and delivery of on-orbit Earth science data.  The DAACs experience also demonstrates an evolution of best practices for working with this kind of data.  However, systems which have been in operation for decades often have technical debt, which can constrain the evolution of the research data infrastructure.  The migration of EOSDIS into a commercial cloud environment presents several interesting opportunities for addressing the data producer and data user needs identified by the workshop and experience of the DAACs.  

How to cite: Wilson, B., Buzanowicz, M., Leon, A., Lubkin, S., Sinclair, L., Stano, G., Thornton, M., Tisdale, M., Walker, T., Wei, Y., and Wingo, S.: Data Producer and Data User Needs for Airborne and Field Earth Science Research Measurements, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-14950, https://doi.org/10.5194/egusphere-egu23-14950, 2023.

The UK is planning to implement a £38M (€43M) Flood and Drought Research Infrastructure to facilitate the hydrological science and innovation needed to underpin the UK’s adaptation and resilience to floods and droughts. The UKRI’s intent to invest from its infrastructure fund was published following a ~2-year scoping study that determined research community requirements through reviews of comparable infrastructures across the UK, Europe and globally, community workshops and questionnaires, and direct engagement with potential beneficiaries from research, industry and government bodies. Significantly, the scoping study identified the importance of a digital infrastructure to enable a step-change in access to hydrological monitoring data.  This would complement community access to the expected physical infrastructure for monitoring all phases of the water cycle across a range of catchment types.

Key requirements for the proposed digital infrastructure, to be delivered through 2023-2028, include access to UK-wide hydrological data alongside new catchment observatory data, supporting field monitoring and innovation through open digital systems, advancing the state of the art for sensor data management, linking monitoring activities more closely with research data archives and delivering support for open science. The digital infrastructure would leverage technological developments e.g. in cloud-based virtual research environments, and be delivered alongside a significant community capacity building effort to support cultural change and enable researchers to transform ways-of-working to maximise its potential benefits.

How to cite: Fry, M. and Old, G.: Design of a digital infrastructure for hydrological research, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-15820, https://doi.org/10.5194/egusphere-egu23-15820, 2023.

The International Generic Sample Number (IGSN) is a crucial tool for ensuring the traceability and preservation of physical specimens in the Earth Science community. As a persistent identifier (PID), IGSN serves as a link between published digital data and the physical samples stored in a repository, enabling the creation of synergies with other services through the harvesting of machine-readable data.

The IGSN can be assigned to a physical specimen at the time of collection, either on board a research vessel or during a field campaign. This unique identifier will follow the sample through the various stages of processing and analysis. In this use case, we demonstrate how IGSN can be minted for sediment cores directly on board a research vessel and then subsequently linked to relevant research data infrastructures (RDIs) such as DSHIP and PANGAEA. This allows for the traceability and easy identification of the samples as they are transported and stored in different repositories.

Incorporating IGSN into a RDI helps to broadcast the existence of physical material and makes it more easily discoverable by researchers. This is especially useful for marine field work, which can be expensive and may not be accessible to all researchers. By making information about samples available as open access, researchers are able to easily locate and reuse existing material, which can be particularly beneficial for smaller research projects or research communities with limited resources. This is especially relevant in times of crisis, when access to certain regions may be restricted and there is an increased demand for the reuse of existing samples.

In our research institutions, there is close collaboration between RDI providers and sample curators to manage both the digital data and the physical objects, such as plant samples in a herbarium, rocks and sediment cores, and biological material. In this presentation, we will use our case studies to discuss the successes of sample management in relation to IGSN. In addition, we will address the challenges that we have encountered and how we are working to overcome them. Our goal is to provide reliable services to our communities with a long-term perspective, and we believe that the incorporation of IGSN into RDIs can help to foster cultural change and encourage international collaboration in the Earth Science community. In addition, the use of IGSN and RDIs can contribute to the sustainability and reproducibility of research.

How to cite: Maicher, D., Petersen, T., Mehrtens, H., and Springer, P.: Leveraging IGSN to Enhance Data Management in Research Institutions, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-11116, https://doi.org/10.5194/egusphere-egu23-11116, 2023.