EGU2020-10660
https://doi.org/10.5194/egusphere-egu2020-10660
EGU General Assembly 2020
© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.

Connecting data streams with On-Demand Services in the Alpine Environmental Data Analysis Centre

Johannes Munke1, Alexander Götz1, Helmut Heller1, Stephan Hachinger1, Dominik Laux2, Oleg Goussev3, Jana Handschuh2,3, Sabine Wüst3, Michael Bittner2,3, Roland Mair4, Bianca Wittmann4, Till Rehm5, Inga Beck5, and Markus Neumann5
Johannes Munke et al.
  • 1Leibniz Supercomputing Centre (LRZ) of the Bavarian Academy of Sciences & Humanities, Garching b. München, Germany
  • 2University of Augsburg, Institute of Physics, Augsburg, Germany
  • 3German Aerospace Center (DLR), Earth Observation Center, Weßling, Germany
  • 4bifa Umweltinstitut GmbH, Augsburg, Germany
  • 5Environmental Research Station Schneefernerhaus (UFS), Garmisch, Germany

The AlpEnDAC (Alpine Environmental Data Analysis Center) platform (www.alpendac.eu) aims to collect scientific data measured on different high-altitude research stations in the alpine region and beyond. It provides research data management, analysis and simulation services and supports the research activities of the VAO (Virtual Alpine Observatory) community.

With funding from the Bavarian State Ministry of the Environment and Consumer Protection, a new development cycle of the platform was launched in 2019. Novel components for Computing on Demand (CoD), Service on Demand (SoD) and Operating on Demand (OoD) will be integrated into the system. These will help to implement a near-real-time (NRT) decision support for the scientist during measurement processes and a better control of the measurement process.

In this work, the authors present a stream processing architecture to couple the new CoD, SoD and OoD components. Data from measurements (or also simulations) are normally ingested via a representational state transfer application programming interface (REST API) into the AlpEnDAC system. Before such data are stored in the data base, they will be run through a central stream processing engine, based on a message queue (e.g. Apache Kafka) and a series of specialized workers to process the data. A rule engine and analytics tools are connected to this engine and allow the automatic triggering of, e.g., measurements, HPC simulations, or evaluation and notification services in NRT. The services will be usable and configurable, as much as possible, via the AlpEnDAC web portal where also certain measurement device settings can be adjusted. With these developments, we want to make environmental scientists profit from NRT data collection and processing, as it is already an everyday tool, e.g., in the Internet-of-Things sector and in commercial applications.

How to cite: Munke, J., Götz, A., Heller, H., Hachinger, S., Laux, D., Goussev, O., Handschuh, J., Wüst, S., Bittner, M., Mair, R., Wittmann, B., Rehm, T., Beck, I., and Neumann, M.: Connecting data streams with On-Demand Services in the Alpine Environmental Data Analysis Centre, EGU General Assembly 2020, Online, 4–8 May 2020, EGU2020-10660, https://doi.org/10.5194/egusphere-egu2020-10660, 2020.

Display materials

Display file