Easier Access to ESM Data: Implementation at Jülich Supercomputing Centre

Improvements in computational speed lead to better resolutions in Earth System Models (ESM) allowing them to resolve scales of a few kilometers. The volume of the resulting data greatly increases with the improvements in resolution and introduces a challenge to processing and storing these results.
While modern HPC systems provide petabyte-scale capacity for file storage, analyzing such data on local user systems can become a prohibitive bottleneck. Beyond the sheer demands on processing high-volume ESM data, there is also increasing demand to make them FAIR and in particular findable.

The goal of the “Easier” module of the Warmworld project aims to simplify the access to ESM data from different HPC centers, in particular the German Climate Computing Center (DKRZ) and the Jülich Supercomputing Centre (JSC). One aspect is the creation of a joined catalog following the SpatioTemporal Asset Catalogs (STAC) specification. This allows browsing data available at both centers. Details on the STAC implementation will be presented by Kameswar Rao Modali et al. at this General Assembly (EGU25-10288).

The catalog also contains links to access the data, either as a download or later as zarr-stream. The explicit implementation of the required REST-APIs depends on the infrastructure, hardware and software, of the data centers as well as the organization of the stored data.

As the first data backend at JSC, we set up a Fields DataBase (FDB), developed by ECMWF, to store the ESM results as multi-dimensional data cubes. For data retrieval, we provide a download service based on the MARS language for identifying data within the FDB and the UNiform Interface to COmputing REsources (UNICORE) for an automated access to our HPC system. The download service integrates the Helmholtz authentication and authorization infrastructure. This will allow a large number of institutions to access these services with the possibility to control access and resource use.

Our poster will provide details on the ongoing implementation and combination of the various tools and services used at JSC as well as further details on the implementation on our HPC systems. In addition, we will provide first information about the planned authorization for different data accesses and also show the performance of different components with a set of benchmarks.