Data management and analysis of the high-resolution multi-model climate dataset from the PRIMAVERA project
- 1Met Office, Exeter, UK (jon.seddon@metoffice.gov.uk)
- 2Centre for Environmental Data Analysis, STFC Rutherford Appleton Laboratory, Didcot, UK
The PRIMAVERA project aims to develop a new generation of advanced and well evaluated high-resolution global climate models. An integral component of PRIMAVERA is a new set of simulations at standard and high-resolution from seven different European climate models. The expected data volume is 1.6 petabytes, which is comparable to the total volume of data in CMIP5.
A comprehensive Data Management Plan (DMP) was developed to allow the distributed group of scientists to produce and analyse this volume of data during the project’s limited time duration. The DMP uses the approach of taking the analysis to the data. The simulations were run on HPCs across Europe and the data was transferred to the JASMIN super-data-cluster at the Rutherford Appleton Laboratory. A Data Management Tool (DMT) was developed to catalogue the available data and allow users to search through it using an intuitive web-based interface. The DMT allows users to request that the data they require is restored from tape to disk. The users are then able to perform all their analyses at JASMIN. The DMT also controls the publication of the data to the Earth System Grid Federation, making it available to the global community.
Here we introduce JASMIN and the PRIMAVERA data management plan. We describe how the DMT allowed the project’s scientists to analyse this multi-model dataset. We describe how the tools and techniques developed can help future projects.
How to cite: Seddon, J. and Stephens, A.: Data management and analysis of the high-resolution multi-model climate dataset from the PRIMAVERA project , EGU General Assembly 2020, Online, 4–8 May 2020, EGU2020-14241, https://doi.org/10.5194/egusphere-egu2020-14241, 2020
Comments on the display
AC: Author Comment | CC: Community Comment | Report abuse
I tryed to install your github programm. it writes to me
python manage.py makemigrations pdata_app
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/base.py", line 328, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/base.py", line 366, in execute
self.check()
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/base.py", line 395, in check
include_deployment_checks=include_deployment_checks,
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/management/base.py", line 382, in _run_checks
return checks.run_checks(**kwargs)
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/checks/registry.py", line 72, in run_checks
new_errors = check(app_configs=app_configs)
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/core/checks/translation.py", line 60, in check_language_settings_consistent
get_supported_language_variant(settings.LANGUAGE_CODE)
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/conf/__init__.py", line 76, in __getattr__
self._setup(name)
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/conf/__init__.py", line 63, in _setup
self._wrapped = Settings(settings_module)
File "/home/leonid/anaconda3/lib/python3.7/site-packages/django/conf/__init__.py", line 161, in __init__
raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
Looks like the last note is important...
Hi Leonid,
I'm really sorry about that. Now that the DMT is deployed I've not followed those instructions for several years. In GitHub I've updated README.md with some improved instructions that work with the current version. Please let me know if you have any problems when following the new instructions.
I'm afraid that as I mentioned in the display, the code currently makes some assumptions about the PRIMAVERA data request and JASMIN. We want to re-use the code for some internal projects at the Met Office later this year. This will give me the chance to refactor the code to make it more generic and easier to apply to all projects and sites.
Thanks for letting me know about the problems that you encountered. Hopefully other users will have fewer problems as a result.
Thanks,
Jon