ESSI2.9

The SCIentific Qt application for Learning from Observations of Plasmas (SciQLop) project allows to easily discover, retrieve, plot and label in situ space physic measurements from remote servers such as Coordinated Data Analysis Web (CDAWeb) or Automated Multi-Dataset Analysis (AMDA).  Analyzing data from a single instrument on a given mission can rise some technical difficulties such as finding where to get them, how to get them and sometimes how to read them.  Thus building for example a machine-learning pipeline involving multiple instruments and even multiple spacecraft missions can be very challenging. Our goal here is to remove all these technical difficulties without sacrificing performances to allow scientist to focus on data analysis.
SciQLop development has started in 2015 as a C++ graphical application funded by the Paris-Saclay Center for Data Science (CDS) then by Paris-Saclay SPACEOBS and finally it joined the Plasma Physics Data Center (CDPP) in 2019. It has evolved from  a monolithic C++ graphical application to a collection of simple and reusable Python or C++ packages solving one problem at a time, increasing our chances to reach users and contributors.

The SciQLop project is composed of the following tools:

• Speasy: An easy to use Python package to retrieve data from remote servers with multi-layer cache support.
• Speasy_proxy: A self-hostable, chainable remote cache for Speasy written as a simple Python package.
• Broni: A Python package which finds intersections between spacecraft trajectories and simple shapes or physical models such as magnetosheath.
• Orbit-viewer: A Python graphical user interface (GUI) for Broni.
• TSCat: A Python package used as backend for catalogs of events storage.
• TSCat-GUI: A Python graphical user interface (GUI).
• SciQLop-GUI: An extensible and efficient user interface to visualize and label time-series with an embedded IPYthon terminal.

While some components are production ready and already used for science, SciQLop is still in development and the landscape is moving quite fast.
In this presentation we will give an overview of SciQLop, demonstrate its benefits using some specific cases studies and finally discuss the planned features development.

How to cite: Jeandet, A., Aunai, N., Génot, V., Schulz, A., Renard, B., Bayane, M. D. W., and Nguyen, G.: SciQLop: an open source project for in situ data analysis, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-5774, https://doi.org/10.5194/egusphere-egu22-5774, 2022.

Earth science data usually have distinct three-dimensional spatial characteristics, in which the state of the atmosphere changes rapidly, the time dimension is also very important, and the variables that describe various physical and chemical states together constitute the earth science cube. GIS, scientific computation and visualization tools are important to find and extract patterns and scientific views behind the data. MeteoInfo open-source software was developed as an integrated framework both for GIS application and scientific computation environment with two applications for end users: MeteoInfoMap and MeteoInfoLab. MeteoInfoMap makes it quick and easy to explore many kinds of geoscience data in the form of GIS layers, and includes spatial data editing, projection and analysis functions. MeteoInfoLab includes multidimensional array calculations, scientific calculations such as linear algebra, and 2D/3D plotting functions, which are suitable for completing the tasks of geoscience data analysis and visualization. The software was developed using Java and Jython, which makes it has good cross-platform capabilities and can run in operating systems such as Windows, Linux/Unix, and Mac OS with Java supporting.

The functions can be conveniently extended through development of plugin for MeteoInfoMap and toolbox for MeteoInfoLab. For example, TrajStat plugin was developed for air trajectory analysis and air pollution source identification, which has been widely used in air pollution transport pathway and spatial sources studies. Several MeteoInfoLab toolbox were also developed for model evaluation (IMEP), air pollution emission data processing (EMIPS) and machine learning (MIML). MeteoInfoLab has similar functions with Python scientific packages such as numpy, pandas and matplotlib, also Jython is just Python in Java. So, the users can learn MeteoInfoLab easily when they have Python experience, vice versa. 3D visualization functions are more powerful in MeteoInfoLab due to the usage of opengl acceleration. Also, 3D earth coordinate is supported to plot geoscience data on virtual earth.

References:

Wang, Y.Q., 2014. MeteoInfo: GIS software for meteorological data visualization and analysis. Meteorological Applications, 21: 360-368.

Wang, Y.Q., 2019. An Open Source Software Suite for Multi-Dimensional Meteorological Data Computation and Visualisation. Journal of Open Research Software, 7(1), p.21. DOI: http://doi.org/10.5334/jors.267

Wang, Y.Q., Zhang, X.Y. and Draxler, R., 2009. TrajStat: GIS-based software that uses various trajectory statistical analysis methods to identify potential sources from long-term air pollution measurement data. Environmental Modelling & Software, 24: 938-939

How to cite: Wang, Y.: MeteoInfo: An open-source GIS, scientific computation and visualization platform, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-3492, https://doi.org/10.5194/egusphere-egu22-3492, 2022.

The Italian national hydrological and meteorological information system is being operationally implemented by Regional Hydrological Services (SIR), under the coordination of the Italian Institute for Environmental Protection and Research (ISPRA) and the collaboration of National Research Council of Italy, Institute of Atmospheric Pollution Research (CNR-IIA) for architectural design and implementation of the central components and the National Institute for Nuclear Physics (INFN) as cloud computing service developer and infrastructure provider. This work is funded by the Italian Ministry of Ecological Transition under the national initiative “Piano Operativo Ambiente FSC 2014-2020” with the aim of providing a standardised and uniform access to hydro-meteorological data of Italy, allowing, among the others, calculating statistics, trends and indicators related to the hydrological cycle, weather and climate and water resources at national and sub-national (e.g., river basin districts, catchments, climatic areas) scales. A prototype of the system has been developed in the framework of the Italian National Board for Hydrological Operational Services, coordinated by ISPRA, which federates the Italian SIRs that are responsible for hydro-meteorological monitoring at local level. 

A hydrometeorological web portal will be the entry point for end users such as Institutional bodies, research institutions and universities to discover, access and download hydrological and meteorological data of Italy made available by the SIR services.  

Each SIR is already publishing online hydrological and meteorological regional data by means of Internet services. As a requirement, no obligation can be imposed on the specific communication protocols (and related data models) that will be implemented by such services, as the entry barrier for SIR to participate in the system of systems should be minimal.

CNR-IIA is responsible for the design of the architecture, which will be based on a brokering approach to enable interoperability amongst the heterogeneous systems. CNR-IIA is also responsible for the implementation of both the brokering component, based on the Discovery and Access Broker (DAB) technology and the hydrometeorological web portal.

The DAB is a software framework able to implement discovery and access functionalities across a distributed set of heterogeneous data publication systems, in a transparent way for the end user, by acting as a mediator of communication protocols and related data models.

Other service interfaces published by the brokering component will be used by different end-user tools and applications, enabling as well sharing of hydrological and meteorological data of Italy towards different national and international initiatives, in particular the WMO Hydrological Observing System (WHOS).

The brokering component will be deployed and managed operatively on a cloud infrastructure to optimize overall system performance and resource usage.

The system will be initially hosted on the CNR-IIA cloud infrastructure backed by Amazon Web Services (AWS), while the target hosting infrastructure and the related cloud computing services, to be ready for operative use by the end of 2025, will be provided by INFN.

How to cite: Boldrini, E., Roncella, R., Papeschi, F., Mazzetti, P., Casaioli, M., Mariani, S., Bussettini, M., Lastoria, B., and Pecora, S.: Brokering approach based implementation for the national hydrological and meteorological information system in Italy, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-12167, https://doi.org/10.5194/egusphere-egu22-12167, 2022.

We present the 'Mine-the-Gaps' geospatial web application and use it to present our recently published series of cleaned and imputed environmental datasets. The Mine-the-Gaps tool positions environment sensor and regional estimation data side-by-side on a map, allowing researchers to visually check sensor locations, regional coverage, and the likely accuracy of any regional approximation methods. Users can upload their own time-series geospatial data datasets, but as a proof-of-concept, we present an online version of this tool (http://minethegaps.manchester.ac.uk) pre-loaded with our recently published environmental dataset. This contains 5 years of cleaned and pre-processed UK sensor data, originally from DEFRA (air quality) and the Met Office (pollen and weather) monitoring stations alongside our basic region estimations.

Sensors can’t be installed everywhere, so many areas are sparsely covered, with some measurements (e.g. pollens and SO2) even more sparse than others. Mine-the-Gaps allows users to, for example, approximate the level of pollen for somebody reporting severe allergy symptoms in Manchester if the nearest sensor is in Chester. Two basic, distance-based estimation algorithms are pre-loaded but more importantly, we provide researchers with methods to visualize, evaluate and compare new regional estimation techniques. Estimated data can be loaded into the application via a CSV file, or new algorithms can be added to our region estimations Python package, used by Mine-the-Gaps to calculate estimations on the fly. Uploaded data is presented on a map and time-series charts can be displayed that compare sensor data with approximations for that location, had the sensor been missing.

If users have their own sensor and/or region datasets, these can be uploaded to Mine-the-Gaps, which can be cloned from our code repository. Just 2 lines of script are required to get it running locally and accessible from any browser. Time-series sensor data from any location can then be uploaded and estimations made for any regions.

We publish Mine-the-Gaps, our environmental data-sets, and associated tools with a focus on making these resources as accessible and adaptable as possible, thus allowing researchers to get on with the key job of evaluating the impact and variance of environmental data, rather than becoming bogged down in pre-processing. Importantly, the emphasis on a transparent data processing and visualization methodology enables researchers to determine the usefulness, or not, of any technique used for mapping single site measurements to represent a specific geographical region.

Potential use cases include the evaluation of current and new regional estimation methods for sensor data; evaluating the effects of variation in region shape and size when making estimations; helping to determine where a new sensor would be best positioned to fit with the existing networks; the visualization of irregularly spaced datasets; and visualizing sensor data over time.

How to cite: Gledson, A., Lowe, D., Reani, M., Topping, D., and Jay, C.: The 'Mine-the-Gaps' geospatial web application for visualizing and evaluating regional environmental data estimates, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-3059, https://doi.org/10.5194/egusphere-egu22-3059, 2022.

Virtual Laser Scanning (VLS) provides a remote sensing method to generate 3D point clouds, which can, in certain cases, replace real data acquisition. A prerequisite is a suitable substitute of reality for modelling the 3D scene, the scanning system, the platform, the laser beam transmission, the beam-scene interaction, and the echo detection. The suitability of simulated laser scanning data largely depends on the application, and simulations that are more realistic come with stricter requirements on input data quality and higher computational costs. It is therefore important to have a good capability for corresponding trade-offs in the simulation software.

With the scientific software HELIOS++ [1], we provide an open source solution to acquire VLS data, where this trade-off can be tuned easily. HELIOS++ is implemented in C++ for optimized runtimes, and provides bindings in Python to allow integration into scripting environments (e.g. GIS plugins, Jupyter Notebooks).

The HELIOS++ VLS concept is based on a modular design. This allows the user to quickly exchange single simulation components, such as the scanner or the 3D scene. The simulation of diverging laser beams and the recording of full waveforms is supported via a subray tracing approach: depending on the desired physical realism and accuracy, a user-defined number of concentric circles approximate a single laser beam. On each circle, individual subrays are cast into the scene, which can then intersect with a single object and produce a hit. The returned waveforms are subsequently added together. This allows the simulator to detect multiple echoes for each pulse. The waveforms can be exported for further analysis.

In this contribution, we present main applications of HELIOS++ as a general-purpose LiDAR simulator. The first application is forestry, where green vegetation can be represented by different 3D model types. As the simulation of individual leaves as 3D mesh models requires high computational power, voxel-based methods have recently been proposed. HELIOS++ also supports simulation of semitransparent voxels, where a subray has a certain probability of creating a return when traversing. This probability depends on the incidence angle and the leaf angle distribution (e.g., planophile, erectophile …), the traversal length through the voxel, and the leaf area density of the voxel, which can, e.g., be derived from a terrestrial laser scanning point cloud. Tuning of the subray-parameters allows recreating vertical point density profiles of real surveys.

A second use case is the generation of training data for machine learning algorithms. Recently, several methods for Deep Learning on point clouds have been presented. However, such methods require immense amounts of training data to achieve acceptable performance. We present how VLS can be used to generate training data in machine learning classifiers, and how different sensor settings influence the classification results.

This contribution provides an introduction to VLS, possible use cases, pitfalls and best practices for successful application of laser scanning simulation.

[1] Winiwarter et al., 2021, DOI: 10.1016/j.rse.2021.112772

How to cite: Winiwarter, L., Esmorís Pena, A. M., Zahs, V., Weiser, H., Searle, M., Anders, K., and Höfle, B.: Virtual Laser Scanning using HELIOS++ - Applications in Machine Learning and Forestry, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-8671, https://doi.org/10.5194/egusphere-egu22-8671, 2022.

Indexed 3D Scene Layers (I3S) is an open format for streaming and storing massive amounts of heterogeneously distributed geospatial content. Adopted as the first OGC (Open Geospatial Consortium) Community Standard for 3D streaming and storage since 2017, I3S has been rapidly evolving to capture new use cases and techniques, advancing geospatial visualization and analysis. I3S enables the efficient transmission of various 3D geospatial data types, including discrete 3D objects with attributes, integrated surface meshes and point cloud data covering vast geographic areas as well as highly detailed BIM (Building Information Model) content, to web browsers, mobile apps, and desktop.

In this session, we’ll explore various aspects of I3S, including the latest update to the standard, OGC I3S Version 1.2, that brought dramatic improvements in performance and scalability. We’ll describe and demonstrate various I3S features, such as paged node access pattern – a compact Bounding Volume Hierarchy (BVH) representation that significantly reduces client-server traffic, a more compact geometry layout – using a well-known quantization encoding (Draco), advanced material definitions property that supports PBR (physically based rendering of materials), as well as support for KTX2 (Basis) compressed textures – reducing the compressed texture size by over 60%. We’ll also demonstrate collaborative & research work done to dramatically improve the speed of KTX2/Basis creation leveraging both the CPU and GPU – contributions that benefit both the geospatial and 3d graphics communities.

We will also further demonstrate various examples of the different layer types and profiles that are supported in I3S and how the data structure and organization help to efficiently store segmentation/classification information as well as triangle/point level attribution. Technological advancements in 3D graphics, data structuring, mesh and texture compression, efficient client-side filtering and so forth have significantly contributed to a paradigm shift in how geospatial content is created and disseminated, regardless of size and scale. Formats such as I3S now allow 3d content to be authored/created once and be efficiently consumed in various platforms including desktop, web and mobile for both offline and online access. This phenomenon – create once and consume everywhere model, has encouraged the dissemination and sharing of geospatial content for both planetary (whole earth) and planar 3D visualization experiences. The session will show case numerous examples (for desktop, web, and mobile experience) illustrating the many advancements made in geospatial technologies that are ripe to be embraced in various geoscience disciplines.

Lastly, we’ll close by showcasing various patterns for distributing 3D content via I3S and demonstrate its consumption thru open-source solutions such as loaders.gl and CesiumJS, supporting the latest version of I3S. We’ll also show latest additions to Tile Converter, an open-source solution that allows a two-way conversion between 3D Tiles and I3S, further fostering interoperability in the geospatial community.

How to cite: Belayneh, T.: Indexed 3D Scene Layers (I3S) – an open standard for 3D GIS visualization on Web, Desktop and Mobile Platforms, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-686, https://doi.org/10.5194/egusphere-egu22-686, 2022.

WQeMS aims to provide an operational Water Quality Emergency Monitoring Service to the water utilities industry in relation with the quality of the ‘water we drink’. This Copernicus service focuses on monitoring lakes for the delivery of drinking water and will provide open geospatial data products structured in Essential Water Variables. While Essential Climate variables are fully defined, a set of Essential Water Variables was proposed by GEOSS but was never fully adopted. In this communication we will present a metadata manager tool called GeM+ that adopts a general framework for Essential Variables (EV) that includes a renewed proposal for Essential Water Variables. EVs are included in a keyword library that relates them to SDG indicators. In addition, the GEM+ includes a library of quality measures defined in QualityML (that inherits and extends the UncertML approach) vocabulary that will be used and tested for the Water Quality Emergency Monitoring Service. The quality need for a vocabulary proposed by QualityML is now being adopted by the ISO 19157-3 proposal. GEM+ also implements the ISO 19115-1 approach for lineage (provenance) that allows to carefully document data product workflows used to create the geospatial products as a mechanism to describe the traceability, quality and reproducibility of the dada. Examples of water related dataset have been prepared showing how EVs, quality measures and provenance is used to semantically tag data, document quantitative quality estimations and present the sources and processes used to elaborate this Copernicus service candidate products.

WQeMS has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101004157.

How to cite: Serral, I., Masó, J., Julià, N., Manakos, I., Milis, G., and Pesquer, L.: Applying water requirements into metadata in the era of SDGs and Essential Variables: semantics, quality parameters and discoverability in the GEM+, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-5490, https://doi.org/10.5194/egusphere-egu22-5490, 2022.

Providing information on the context of long-term measurements, i.e. the observation and research facilities where measurements were done, is key for re-using data generated for a defined area. This information is also needed to manage site networks of research infrastructures or research networks or to enable a link from in-situ measurements to earth observation. To cover these requirements, the Dynamic Ecological Information Management System – Site and Dataset Registry (DEIMS-SDR, https://www.deims.org/) allows the description of in-situ environmental observation or experimental sites implementing a multipurpose data model and generating persistent, unique and resolvable identifiers for each site. The aim of DEIMS-SDR is to collect site information in a catalogue describing a wide range of sites across the globe, providing information including each site's location, ecosystems, facilities, measured parameters and research themes and enabling that standardised information to be openly available. DEIMS-SDR currently stores over 1200 site records along a wide geographic, altitudinal and ecosystem gradient. To address research needs as well as to enhance interoperability and machine readability, the used site model has been revised and compared to existing de-facto standards resulting in a more modular structure.

The presentation describes the outcomes of the revision of the data model, the conceptual considerations behind it and how it is implemented. We illustrate the capabilities of the data model focussing on the description of multi-layered geographic information that is needed to accurately document sites and the various observation locations they might cover. This represents a major step forward in both the development of DEIMS-SDR as well as the interoperable and cross-disciplinary documentation of sites in general.

How to cite: Wohner, C. and Peterseil, J.: Managing multi-layered geographic information to describe environmental monitoring and research sites using DEIMS-SDR, EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-5451, https://doi.org/10.5194/egusphere-egu22-5451, 2022.