EGU22-11270
https://doi.org/10.5194/egusphere-egu22-11270
EGU General Assembly 2022
© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.

Freva, a software framework for the Earth System community. Overview and and new features.

Etor E. Lucio-Eceiza1,2, Christopher Kadow2, Martin Bergemann2, Mahesh Ramadoss1,2, Brian Lewis2, Andrej Fast2, Jens Grieger1, Andy Richling1, Ingo Kirchner1, Uwe Ulbrich1, Hannes Thiemann2, and Thomas Ludwig2
Etor E. Lucio-Eceiza et al.
  • 1Freie Universität Berlin, Institute for Meteorology, Berlin, Germany (e.lucio-eceiza@fu-berlin.de)
  • 2Deutsches Klimarechenzentrum GmbH (DKRZ), Hamburg, German

The complexity of the climate system calls for a combined approach of different knowledge areas. For that, increasingly larger projects need a coordinate effort that fosters an active collaboration between members. On the other hand, although the continuous improvement of numerical models and larger observational data availability provides researchers with a growing amount of data to analyze, the need for greater resources to host, access, and evaluate them efficiently through High Performance Computing (HPC) infrastructures is growing more than ever. Finally, the thriving emphasis on FAIR data principles [1] and the easy reproducibility of evaluation workflows also requires a framework that facilitates these tasks. Freva (Free Evaluation System Framework [2, 3]) is an efficient solution to handle customizable evaluation systems of large research projects, institutes or universities in the Earth system community [4-6] over the HPC environment and in a centralized manner.

 

Freva is a scientific software infrastructure for standardized data and analysis tools (plugins) that provides all its available features both in a shell and web environment. Written in python, is equipped with a standardized model database, an application-programming interface (API) and a history of evaluations, among others:

  • An implemented metadata system in SOLR with its own search tool allows scientists and their plugins to retrieve the required information from a centralized database. The databrowser interface satisfies the international standards provided by the Earth System Grid Federation (ESGF, e.g. [7]).
  • An API allows scientific developers to connect their plugins with the evaluation system independently of the programming language. The connected plugins are able to access from and integrate their results back to the database, allowing for a concatenation of plugins as well. This ecosystem increases the number of scientists involved in the studies, boosting the interchange of results and ideas. It also fosters an active collaboration between plugin developers.
  • The history and configuration sub-system stores every analysis performed with Freva in a MySQL database. Analysis configurations and results can be searched and shared among the scientists, offering transparency and reproducibility, and saving CPU hours, I/O, disk space and time.

Freva efficiently frames the interaction between different technologies thus improving the Earth system modeling science.

 

This framework has undergone major refactoring and restructuring of the core that will also be discussed. Among others:

  • Major core Python update (2.7 to 3.9).
  • Easier deployment and containerization of the framework via Docker.
  • More secure system configuration via Vault integration.
  • Direct Freva function calls via python client (e.g. for jupyter notebooks).
  • Improvements in the dataset incorporation.

 

References:

[1] https://www.go-fair.org/fair-principles/

[2] Kadow, C. et al. , 2021. Introduction to Freva – A Free Evaluation System Framework for Earth System Modeling. JORS. http://doi.org/10.5334/jors.253

[3] gitlab.dkrz.de/freva

[4] freva.met.fu-berlin.de

[5] https://www.xces.dkrz.de/

[6] www-regiklim.dkrz.de

[7] https://esgf-data.dkrz.de/projects/esgf-dkrz/

How to cite: Lucio-Eceiza, E. E., Kadow, C., Bergemann, M., Ramadoss, M., Lewis, B., Fast, A., Grieger, J., Richling, A., Kirchner, I., Ulbrich, U., Thiemann, H., and Ludwig, T.: Freva, a software framework for the Earth System community. Overview and and new features., EGU General Assembly 2022, Vienna, Austria, 23–27 May 2022, EGU22-11270, https://doi.org/10.5194/egusphere-egu22-11270, 2022.

Displays

Display file