The Kilometer-Scale Cloud by DKRZ

We present the Kilometer-Scale Cloud data server (https://km-scale-cloud.dkrz.de) and its underlying software stack Cloudify (https://gitlab.dkrz.de/data-infrastructure-services/cloudify) developed and deployed at the German Climate Computing Center (DKRZ). The km-scale cloud provides open and analysis-ready access to prominent climate datasets from projects such as the European Eddy-Rich Earth System Models (EERIE) stored across heterogeneous storage tiers through standardized cloud-native interfaces, without requiring physical data reformatting or migration.

Within the EERIE project, kilometer-scale Earth System Models (~10 km atmosphere and ~5 km ocean) generate petabyte-scale output that exceeds the capabilities of traditional file-based access patterns. Cloudify addresses this challenge by emulating Zarr data originating from sources stored on file system or tape enabling efficient HTTP access to large datasets. This approach allows users to interact with HPC-resident data using established cloud-native tools and workflows.

The km-scale cloud offers several key advantages: (i) seamless bridging of HPC and cloud ecosystems, enabling interactive and scalable analysis without data duplication; (ii) analysis-ready data access, supporting chunk-based and parallel I/O patterns suited for modern data analytics and machine-learning workflows; (iii) improved data discoverability and reuse, facilitated by standardized interfaces and metadata services such as STAC catalogs; and (iv) lower entry barriers for external users, who can access large climate datasets without requiring direct HPC accounts or specialized system knowledge.

By deploying Cloudify as a data service, the km-scale cloud demonstrates a scalable pathway towards interoperable, cloud-enabled access to next-generation climate model output.