ESSI3.3

The number of publications in the field of Hydrology (and in other geoscience fields) is rising at an almost exponential rate. In 2021 alone, more than 25 000 articles were listed in Web of Science on the topic of Water Resources. There is a tremendous wealth of knowledge and data hidden in these articles, which capture our experience in studying places, datasets or models. Hidden, because we currently do not possess (or at least, do not use) the necessary tools to access this knowledge resource in an effective manner. It is increasingly difficult for an individual researcher to build on existing knowledge. New ways to approach this problem are urgently needed.  

One approach to address this problem of literature explosion might be to extend article metadata to include geoscience-specific information that can facilitate knowledge search, accumulation and synthesis in a domain specific manner. Imagine one could easily find all studies performed in a specific location/ climate/ land use thus allowing a full picture of the hydrology of that region/ climate/ land use. It is important for any geoscience, a field strongly depending on experience, that knowledge is not “forgotten” in a mountain of publications but can easily be integrated into larger understanding.

So what meta-information would be most useful in knowledge synthesis? Study location? Spatial and/or temporal scale? Models used? Here, we would like to (re-)start the discussion on geoscience-relevant metadata enrichment. With the recent advancement in text mining scholarly literature, it is critical to have this discussion now or fall behind.

The Geosciences strongly depend on experiences we gain, which we largely share through the articles we publish. Knowledge accumulation in our science is hindered if this exchange of knowledge becomes ineffective. We are afraid it already has!

How to cite: Stein, L. and Wagener, T.: Knowledge hidden in plain sight – Extending article metadata to support meta-analysis and knowledge accumulation, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-3590, https://doi.org/10.5194/egusphere-egu22-3590, 2022.

The FAIR data principles form the core OGC mission that renders in the open geospatial standards and the open-data initiatives that use them. Although OGC is best known for the technical interoperability, the domain modelling and semantic level play an inevitable role in the standards definition and the exploitation. On the one hand, we have a growing number of specialised profiles and implementations that selectively use the OGC modular specification model components. On the other hand, various domain ontologies exist already, enabling a better understanding of the data. As there could be multiple semantic representations, common data models support cross ontology traverses. Defining the service in the technical-semantic space requires fixing some flexibility points, including optional and mandatory elements, additional constraints and rules, and content including normalised vocabularies to be used.

The proposed solution of the OGC Definition Server is a multi-purpose application built around the triple store database engine integrated with the ingestion, validation, and entailment tools and exposing customized end-points. The models are available in the human-readable format and machine-2-machine aimed encodings. For manual processes, it enables understanding the technical and semantic definitions/relationships between entities. Programmatic solutions benefit from a precise referential system, validations, and entailment.

Currently, OGC Definition Server is hosting several types of definitions covering:

• Register of OGC bodies, assets, and its modules
• Ontological common semantic models (e.g., for Agriculture)
• Dictionaries of subject domains (e.g., PipelineML Codelists)

In practice, that is a step forward in defining the bridge between conceptual and logical models. The concepts can be expressed as instances of various ontological classes and interpreted within multiple contexts, with the definition translated into entities, relationships, and properties. In the future, it is linking the data to the reference model and external ontologies that may be even more significant. Doing so can greatly improve the quality of the knowledge produced based on the collected data. Ability to verify the research outcomes and explainable AI are just two examples where a precise log of inferences and unambiguous semantic compatibility of the data will play a key role.

How to cite: Zaborowski, P., Atkinson, R., Hempelmann, N., and Voidrot, M.-F.: Technical-semantic interoperability reference, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-10032, https://doi.org/10.5194/egusphere-egu22-10032, 2022.

As one of NASA's Science Mission Directorate data centers, the Goddard Earth Sciences Data and Information Services Center (GES-DISC) provides Earth science data, information, and services to the public. One of the objectives of our mission is to facilitate data discovery for users and systems that utilize our data. Metadata plays a very important role in data discovery. As a result, if a dataset is to be used efficiently, it needs to be enhanced with rich and comprehensive metadata. For example, most search engines rely on matching the search query with the indexed metadata in order to find relevant results. Here we present a tool that supports data custodians in the process of creating metadata by utilizing natural language processing (NLP).

Our approach involves combining several text corpora and training a semantic embedding. An embedding is a numerical representation of linguistic features that is aware of the semantics and context. The text corpora we use to train our embedding model contains publication abstracts, our data collections metadata, and ontologies. Our recommendations are based on keywords selected from the Global Change Master Directory (GCMD) and a collection of ontologies including SWEET and ENVO. GCMD offers a comprehensive collection of Earth Science vocabulary terms. This data lexicon enables data curators to easily search metadata and retrieve the data, services, and variables associated with each term. When a query is matched against various keywords in the GCMD branch, the probability of the query matching these keywords is calculated. A similarity score is then assigned to each of the branches of the GCMD, and each branch is sorted according to this similarity metric. In addition to unsupervised training, our approach has the advantage of being able to search for keyword recommendations of different sizes, ranging from sub-words to sentences and longer texts.

How to cite: Mehrabian, A., Gerasimov, I., and Khayat, M.: A Natural Language Processing-based Metadata Recommendation Tool for Earth Science Data, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-10850, https://doi.org/10.5194/egusphere-egu22-10850, 2022.

The complexity of the climate system calls for a combined approach of different knowledge areas. For that, increasingly larger projects need a coordinate effort that fosters an active collaboration between members. On the other hand, although the continuous improvement of numerical models and larger observational data availability provides researchers with a growing amount of data to analyze, the need for greater resources to host, access, and evaluate them efficiently through High Performance Computing (HPC) infrastructures is growing more than ever. Finally, the thriving emphasis on FAIR data principles [1] and the easy reproducibility of evaluation workflows also requires a framework that facilitates these tasks. Freva (Free Evaluation System Framework [2, 3]) is an efficient solution to handle customizable evaluation systems of large research projects, institutes or universities in the Earth system community [4-6] over the HPC environment and in a centralized manner.

Freva is a scientific software infrastructure for standardized data and analysis tools (plugins) that provides all its available features both in a shell and web environment. Written in python, is equipped with a standardized model database, an application-programming interface (API) and a history of evaluations, among others:

• An implemented metadata system in SOLR with its own search tool allows scientists and their plugins to retrieve the required information from a centralized database. The databrowser interface satisfies the international standards provided by the Earth System Grid Federation (ESGF, e.g. [7]).
• An API allows scientific developers to connect their plugins with the evaluation system independently of the programming language. The connected plugins are able to access from and integrate their results back to the database, allowing for a concatenation of plugins as well. This ecosystem increases the number of scientists involved in the studies, boosting the interchange of results and ideas. It also fosters an active collaboration between plugin developers.
• The history and configuration sub-system stores every analysis performed with Freva in a MySQL database. Analysis configurations and results can be searched and shared among the scientists, offering transparency and reproducibility, and saving CPU hours, I/O, disk space and time.

Freva efficiently frames the interaction between different technologies thus improving the Earth system modeling science.

This framework has undergone major refactoring and restructuring of the core that will also be discussed. Among others:

• Major core Python update (2.7 to 3.9).
• Easier deployment and containerization of the framework via Docker.
• More secure system configuration via Vault integration.
• Direct Freva function calls via python client (e.g. for jupyter notebooks).
• Improvements in the dataset incorporation.

References:

[1] https://www.go-fair.org/fair-principles/

[2] Kadow, C. et al. , 2021. Introduction to Freva – A Free Evaluation System Framework for Earth System Modeling. JORS. http://doi.org/10.5334/jors.253

[3] gitlab.dkrz.de/freva

[4] freva.met.fu-berlin.de

[5] https://www.xces.dkrz.de/

[6] www-regiklim.dkrz.de

[7] https://esgf-data.dkrz.de/projects/esgf-dkrz/

How to cite: Lucio-Eceiza, E. E., Kadow, C., Bergemann, M., Ramadoss, M., Lewis, B., Fast, A., Grieger, J., Richling, A., Kirchner, I., Ulbrich, U., Thiemann, H., and Ludwig, T.: Freva, a software framework for the Earth System community. Overview and and new features., EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-11270, https://doi.org/10.5194/egusphere-egu22-11270, 2022.

Conversion of dynamic bottom-hole temperatures (BHTs) into static ones and utilizing on the purpose of either calibration for basin modelling or drilling plan is a crucial step for hydrocarbon and geothermal exploration projects. However, records of temperature conversions might be ignored or might get lost from the archives due to various reasons, such as project team change, diversion of focus into other areas or simply deletion of data. Disappearance of previous studies does not only disrupt the geoscientific knowledge but also causes repetition for exploration geoscientists to start the time consuming BHT conversion process all over again.

NE Mediterranean Dashboard v1.0 provides a solution for the issue by benefiting from data science instruments of Python programming language. By implementing Plotly-Dash for the front-end, and PostgreSQL for the back end as the keeper of thermal records in datatables, this open-source project proposes a user-friendly web application displaying temperature, geothermal gradient and heat flow profiles in a dashboard style.

The application is consisted of three tabs. The Overview tab provides statistical information while 2D plots section allows users to interact with cross-plots demonstrating thermal conditions for all wells or a particular well selected by the user. It also compares the results of three different BHT conversion methods known as; Horner-plot method, AAPG correction and Harrison et al. (1983). The last tab, Map View, illustrates the temperature, geothermal gradient, and heat flow maps for every 500 meters from surface to 4.5 km depth. The maps reveal the effects of the regional tectonics and how it controls the subsurface thermal behaviour along the Cilicia and Latakia Basins dominating the NE Mediterranean region.

All maps and cross-plots are interactive, and their styles can be changed according to the user’s preferences. They can also be downloaded as images for possible use in scientific publishment and/or presentations. The same interface and visualisation style, accessed by username and password, can also provide consistency between all project workers.

The source code is available at Github repository with the link; https://github.com/Ayberk-Uyanik/NE-Mediterranean-Thermal-Conditions and can efficiently be implemented for exploration projects in other regions.

How to cite: Uyanik, A.: An open-source web application displaying present-day subsurface thermal conditions of the NE Mediterranean region, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-810, https://doi.org/10.5194/egusphere-egu22-810, 2022.

For precision agriculture (PA) applications that use aerial platforms, researchers are likely to be interested in extracting, study and understanding biophysical and structural properties in a spatio-temporal manner by using remotely sensed imagery to infer variations of vegetation biomass and/or plant vigor, irrigation strategies, nutrient use efficiency, stress, disease identification, among others. This requires measuring spectral responses of the crop at specific wavelengths by using, for instance, Vegetation Indices (VI). However, for the analysis of this spectral response and its heterogeneity and spatial variability, a large amount of aerial imagery (data) must be collected and processed using a photogrammetry software. Data extraction is often performed in a Geographic Information System (GIS) software and then analyzed using (in general) statistical software. On the one hand, a GIS is used for the collection of resources to manipulate, analyze, and display all forms of geographically referenced information. In this regard, Quantum GIS (QGIS) is one of the most well-known open-source software used which provides an integration of geoprocessing tools from a variety of different software libraries. QGIS is widely used to obtain VI computations through the raster calculator, although, this computation is performed with band rasters manually provided by the user; one by one, which is time-consuming. On the other hand, QGIS provides a Python interface to efficiently exploit the capabilities of a GIS to create similar plugins, but this can be a non-trivial task. In this work, we developed a specific and QGIS independent semi-automatic tool called ViCTool (Vegetation index Computation Tool) as a free open-source software (FOSS) for large amount of data extraction to derive VIs from aerial raster images in a certain region of interest. This tool has the option of extracting several multispectral and RGB VIs employing Blue, Green, Red, NIR, LWIR, or Red edge bands. The user must provide the input folder path containing one or more raster band folders, the shapefile with the regions of Interests, an output path to store the output VI rasters, and the file containing the VI computations. ViCTool was developed using Python PyQT for designing the User Interface (UI) and Python GDAL for raster processing to simplify and speed up the process of calculating a large amount of data intuitively.

How to cite: Triana-Martinez, J. C., Fernandez-Gallego, J. A., Barrero, O., Borra-Serrano, I., De Swaef, T., Lootens, P., and Roldan-ruiz, I.: ViCTool: An open-source tool for vegetation indices computation of aerial raster images using python GDAL, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-8914, https://doi.org/10.5194/egusphere-egu22-8914, 2022.

The Australian 2030 Geophysics Collections Project seeks to make accessible online a selection of rawer, high-resolution versions of geophysics datasets that comply with the FAIR and CARE principles, and ensure they are suitable for programmatic access in HPC environments by future 2030 next-generation scalable, data-intensive computation (including AI and ML). The 2030 project is not about building systems for the infrastructures and stakeholder requirements of today, rather it is about positioning geophysical data collections to be capable of taking advantage of next generation technologies and computational infrastructures by 2030.

There are already many known knowns of 2030 computing: high end computational power will be at exascale and today’s emerging collaborative platforms will continue to evolve as a mix of HPC and cloud. Data volumes will be measured in Zettabytes (10²¹ bytes), which is about 10 times more than today. It will be mandatory for data access to be fully machine-to-machine as envisaged by the FAIR principles in 2016. Whereas we currently discuss Big Data Vs (volume, variety, value, velocity, veracity, etc), by 2030 the focus will be on Big Data Cs (community, capacity, confidence, consistency, clarity, crumbs, etc).

So often today’s research is undertaken on pre-canned, analysis-ready datasets (ARD) that are tuned towards the highest common denominator as determined by the data owner. However, increased computational power colocated with fast-access storage systems will mean that geophysicists will be able to work on less processed data levels and then transparently develop their own derivative products that are more tuned to the parameters of their particular use case. By 2030, as research teams analyse larger volumes of high-resolution data they will be able to see the quality of their algorithms quickly and there will be multiple versions of open software being used as researchers fine tune individual algorithms to suit their specific requirements. We will be capable of more precise solutions and in hazards space and other relevant areas, analytics will be done in faster-than-real-time. 

The known unknowns emerging are how we will preserve and make transparent any result from this diversity and flexibility with regards to the exact software used, the precise version of the data accessed, and the platforms utilised, etc. When we obtain a scientific ‘product’, how will we vouch for its fidelity and ensure it can be consistently replicated to establish trust? How do we preserve who funded what so that sponsors can see which investments have had the greatest impact and uptake? 

To have any confidence in any data product, we will need to have transparency throughout the whole scientific process. We need to start working now on more automated systems that capture provenance through successive levels of processing, including how it was produced and which dataset/dataset extract was used. But how do we do this in a scaleable, machine readable way?

And then there will be the unknown unknowns of 2030 computing. Time will progressively expose these to us in the next decade as the scale and speed at which collaborative research is undertaken increases.

How to cite: Wyborn, L., Rees, N., Klump, J., Evans, B., Rawling, T., and Druken, K.: The Known Knowns, the Known Unknowns and the Unknown Unknowns of Geophysics Data Processing in 2030, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-11012, https://doi.org/10.5194/egusphere-egu22-11012, 2022.

Research in environmental data science is typically transdisciplinary in nature, with scientists, practitioners, and stakeholders creating data-driven solutions to the environment’s grand challenges, often using a large amount of highly heterogeneous data along with complex analytical methods. The concept of virtual labs allow collaborating scientists to explore big data, develop and share new methods, as well as communicate their results to stakeholders, practitioners, and decision-makers across different scales (individual, local, regional, or national).

Within the Data Science of the Natural Environment (DSNE) project, a transdisciplinary team of environmental scientists, statisticians, computer scientists and social scientists are collaborating to develop statistical/data science algorithms for environmental grand challenges through the medium of a virtual labs platform, named DataLabs. DataLabs, in continuous development by UKCEH in an agile approach, is a consistent and coherent cloud-based research environment that advocates open and collaborative science by providing the infrastructure and software tools to bring users of different areas of expertise (scientists, stakeholders, policy-makers, and the public) interested in environmental science into one virtual space to tackle environmental problems. DataLabs support end-to-end analysis from the assimilation and analysis of data through to the visualisation, interpretation, and discussion of the results.

DataLabs draw on existing technologies to provide a range of functionality and modern tools to support research collaboration, including: (i) parallel data cluster services, such as DASK and Spark; (ii) executable notebook technologies, such as Jupyter, Zepplin and R; (iii) lightweight applications such as RShiny to allow rapid collaboration among diverse research teams; and (iv) containerisation of application deployment (e.g. using Docker) so that technologies developed can be more easily moved to other cloud platforms as required. Following the principles of service-oriented architectures, the design enables selecting the most appropriate technology for each component and exposing any functions by other systems via HTTP as services. Within each component, a modular-layered architecture is used to ensure separation of concerns and separated presentation. DataLabs are using JASMIN as the host computing platform, giving researchers seamless access to HPC resources, while taking advantage of the cloud scalability. Data storage is available to all systems through shared block storage (NFS cluster) and object storage (QuoBye S3).

Research into and development of virtual labs for environmental data science are taking part within the DSNE project. This requires studying the current experiences, barriers and opportunities associated with virtual labs, as well as the requirements for future developments and extensions. For this purpose, we have conducted an online user engagement survey, targeting DSNE researchers and the wider user community, as well as the international research groups and organisations that contribute to virtual labs design. The survey results are considered are feeding into the continuous development of DataLabs. For instance, some of the researchers’ requirements include the ability to submit their own containers to DataLabs and the security issues to access external data storage. Other users have indicated the importance of having libraries of data science and data visualisation methods, which are currently being populated by DSNE researchers to be then explored in different environmental problems. 

How to cite: Salama, M., Blair, G., Brown, M., and Hollaway, M.: Virtual Labs for Collaborative Environmental Data Science, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-10719, https://doi.org/10.5194/egusphere-egu22-10719, 2022.