[Back] [Session ITS4.1/NP4.2]
EGU2020-14241
https://doi.org/10.5194/egusphere-egu2020-14241
EGU General Assembly 2020
© Author(s) 2020. This work is distributed under
the Creative Commons Attribution 4.0 License.

Data management and analysis of the high-resolution multi-model climate dataset from the PRIMAVERA project

Jon Seddon1 and Ag Stephens2
Jon Seddon and Ag Stephens
  • 1Met Office, Exeter, UK (jon.seddon@metoffice.gov.uk)
  • 2Centre for Environmental Data Analysis, STFC Rutherford Appleton Laboratory, Didcot, UK

The PRIMAVERA project aims to develop a new generation of advanced and well evaluated high-resolution global climate models. An integral component of PRIMAVERA is a new set of simulations at standard and high-resolution from seven different European climate models. The expected data volume is 1.6 petabytes, which is comparable to the total volume of data in CMIP5.  

A comprehensive Data Management Plan (DMP) was developed to allow the distributed group of scientists to produce and analyse this volume of data during the project’s limited time duration. The DMP uses the approach of taking the analysis to the data. The simulations were run on HPCs across Europe and the data was transferred to the JASMIN super-data-cluster at the Rutherford Appleton Laboratory. A Data Management Tool (DMT) was developed to catalogue the available data and allow users to search through it using an intuitive web-based interface. The DMT allows users to request that the data they require is restored from tape to disk. The users are then able to perform all their analyses at JASMIN. The DMT also controls the publication of the data to the Earth System Grid Federation, making it available to the global community. 

Here we introduce JASMIN and the PRIMAVERA data management plan. We describe how the DMT allowed the project’s scientists to analyse this multi-model dataset. We describe how the tools and techniques developed can help future projects.

How to cite: Seddon, J. and Stephens, A.: Data management and analysis of the high-resolution multi-model climate dataset from the PRIMAVERA project , EGU General Assembly 2020, Online, 4–8 May 2020, EGU2020-14241, https://doi.org/10.5194/egusphere-egu2020-14241, 2020

Display materials

Display file

Comments on the display material

AC: Author Comment | CC: Community Comment | Report abuse

Display material version 1 – uploaded on 24 Apr 2020
  • CC1: Comment on EGU2020-14241, Leonid Zotov, 07 May 2020

    I tryed to install your github programm. it writes to me

    python manage.py makemigrations pdata_app
    Traceback (most recent call last):
      File "manage.py", line 10, in <module>
        execute_from_command_line(sys.argv)
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
        utility.execute()
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/base.py", line 328, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/base.py", line 366, in execute
        self.check()
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/base.py", line 395, in check
        include_deployment_checks=include_deployment_checks,
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/base.py", line 382, in _run_checks
        return checks.run_checks(**kwargs)
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/checks/registry.py", line 72, in run_checks
        new_errors = check(app_configs=app_configs)
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/checks/translation.py", line 60, in check_language_settings_consistent
        get_supported_language_variant(settings.LANGUAGE_CODE)
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/conf/__init__.py", line 76, in __getattr__
        self._setup(name)
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/conf/__init__.py", line 63, in _setup
        self._wrapped = Settings(settings_module)
      File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/conf/__init__.py", line 161, in __init__
        raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
    django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

     

    Looks like the last note is important...

    • AC1: Updated instructions now available, Jon Seddon, 07 May 2020

      Hi Leonid,

      I'm really sorry about that. Now that the DMT is deployed I've not followed those instructions for several years. In GitHub I've updated README.md with some improved instructions that work with the current version. Please let me know if you have any problems when following the new instructions.

      I'm afraid that as I mentioned in the display, the code currently makes some assumptions about the PRIMAVERA data request and JASMIN. We want to re-use the code for some internal projects at the Met Office later this year. This will give me the chance to refactor the code to make it more generic and easier to apply to all projects and sites.

      Thanks for letting me know about the problems that you encountered. Hopefully other users will have fewer problems as a result.

      Thanks,

      Jon