EGU2020-15961
https://doi.org/10.5194/egusphere-egu2020-15961
EGU General Assembly 2020
© Author(s) 2022. This work is distributed under
the Creative Commons Attribution 4.0 License.

Automatic quality control and quality control schema in the Observation to Archive

Brenner Silva1, Najmeh Kaffashzadeh2, Erik Nixdorf3, Sebastian Immoor1, Philipp Fischer1, Norbert Anselm1, Peter Gerchow1, Angela Schäfer1, Roland Koppe1, and the Computing and data center*
Brenner Silva et al.
  • 1Alfred-Wegener-Institute, Computing and data center, Germany (bsilva@awi.de)
  • 2Forschungszentrum Juelich
  • 3Helmholtz Centre for Environmental Research
  • *A full list of authors appears at the end of the abstract

The O2A (Observation to Archive) is a data-flow framework for heterogeneous sources, including multiple institutions and scales of Earth observation. In the O2A, once data transmission is set up, processes are executed to automatically ingest (i.e. collect and harmonize) and quality control data in near real-time. We consider a web-based sensor description application to support transmission and harmonization of observational time-series data. We also consider a product-oriented quality control, where a standardized and scalable approach should integrate the diversity of sensors connected to the framework. A review of literature and observation networks of marine and terrestrial environments is under construction to allow us, for example, to characterize quality tests in use for generic and specific applications. In addition, we use a standardized quality flag scheme to support both user and technical levels of information. In our outlook, a quality score should pair the quality flag to indicate the overall plausibility of each individual data value or to measure the flagging uncertainty. In this work, we present concepts under development and give insights into the data ingest and quality control currently operating within the O2A framework.

Computing and data center:

Stephan Frickenhaus (head, stephan.frickenhaus@awi.de)

How to cite: Silva, B., Kaffashzadeh, N., Nixdorf, E., Immoor, S., Fischer, P., Anselm, N., Gerchow, P., Schäfer, A., and Koppe, R. and the Computing and data center: Automatic quality control and quality control schema in the Observation to Archive, EGU General Assembly 2020, Online, 4–8 May 2020, EGU2020-15961, https://doi.org/10.5194/egusphere-egu2020-15961, 2020.

Display materials

Display file

Comments on the display material

AC: Author Comment | CC: Community Comment | Report abuse

Display material version 3 – uploaded on 07 May 2020, no comments
Slide update: quality control tests
Display material version 2 – uploaded on 06 May 2020
Hyperlink update: https://dashboard.awi.de/data-xxl/
  • AC1: Comment on EGU2020-15961, Brenner Silva, 07 May 2020


    Dear Eugene, 
    For sure, I enjoy writing and learning from code. We work with a GitLab instance at our institute. At the moment, only a few are public (e.g. github.com/o2a-data/o2a-data-dws), but we are working on that
    What would you suggest for code sharing? 
    Kind regards, 
    Brenner

Display material version 1 – uploaded on 05 May 2020
  • CC1: Code sharing interest?, Eugene F. Burger, 06 May 2020

    Hello Brenner, I liked your presentaiton and would like to know if there is any opportunity/interest in code sharing?

     

    Thanks

      Eugene