EarthWorks is a high-resolution, coupled, global storm-resolving Earth System Model which is aimed at both weather and climate applications. All components share the same geodesic grid. The target grid spacing is 3.75 km for all components, and the target performance is one simulated year per wall clock day by 2025. 

While EarthWorks uses the CESM framework, its atmosphere, ocean and sea ice components are based on the MPAS (“Model for Prediction Across Scales”) dynamical cores. These components are coupled using the CMEPS (Community Mediator for Earth Prediction Systems) developed for CESM. 

Our presentation will focus on the results of both fully coupled and AMIP simulations with various horizontal grid spacings. We will also mention various problems encountered and overcome along the way.

How to cite: Randall, D., Hurrell, J., Gettelman, A., Skamarock, W., Dazlich, D., Hauser, T., Mickelson, S., Medeiros, B., and Sun, L.: EarthWorks, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-16797, https://doi.org/10.5194/egusphere-egu23-16797, 2023.

Adaptive global circulation models (GCMs) have the potential to significantly improve the computational efficiency and accuracy of climate simulations by dynamically adjusting the local grid resolution to ensure a specified numerical tolerance or to track features of interest.  We have developed the global dynamical cores WAVETRISK-ATMOSPHERE and WAVETRISK-OCEAN to explore the strengths and weaknesses of dynamical GCMs. 

The main open challenge of adaptive climate modelling is how to appropriately couple the dynamical core to the physics. The physics should ideally be “scale-aware”: adjusting the parameterization as necessary based on the current local resolution (or disabling it entirely if the physical phenomenon becomes fully resolved).  A related question is whether the grid adaptation criteria should be based on the physics as well as the dynamics. Such scale-aware physics parameterizations remain poorly understood.  In this talk we report on initial progress in coupling WAVETRISK-ATMOSPHERE to Hourdin’s (1992) “simple dry physics”.

A better understanding of scale-aware physics will also improve non-adaptive climate modelling, since such models currently require extensive tuning each time the resolution is increased. An additional goal of this project is to develop a set of test cases for the simple physics that could be used to compare dynamical cores using a well-understood and standardized physics package.

This is joint work with Gabrielle Ching-Johnson (MSc student, McMaster University, Canada) and Thomas Dubos (LMD, École Polytechnique, France)

How to cite: Kevlahan, N., Ching-Johnson, G., and Dubos, T.: Coupling simple dry physics to a dynamically adaptive global atmosphere model, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-2394, https://doi.org/10.5194/egusphere-egu23-2394, 2023.

We present the ongoing development of a Python implementation of a finite-volume non-hydrostatic dynamical core at ECMWF and its member state partners. The main drivers behind the model formulation are suitability for convective-scale resolutions and increasing multi-level parallelism. Sustainable software design with respect to emerging and future heterogeneous computing platforms is addressed by leveraging the GT4Py domain-specific framework. We further address aspects of implementing and coupling selected ECMWF model physical parametrizations using GT4Py. 

How to cite: Ubbiali, S., Ehrengruber, T., Krieger, N., Kühnlein, C., Papritz, L., and Wernli, H.: Towards a Performance-Portable Finite-Volume Dynamical Core for Numerical Weather Prediction, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-15904, https://doi.org/10.5194/egusphere-egu23-15904, 2023.

Emission sources and sinks of long-lived greenhouse gases (GHGs), such as CO2 and N2O, can be localized and scaled by inversely modelling existing distributions of these tracers in the atmosphere. This is particularly useful for monitoring GHG emissions at a global level, for comparison, for instance, with the national inventory reports of the United Nations Framework Convention on Climate Change (UNFCCC).

To achieve inverse transport numerically, multiple approaches can be taken, notably variational data assimilation. This involves the optimization of the Bayesian cost function accounting for prior state errors and observation errors. Variational data assimilation can be implemented by adjoint modelling. This method is based on the modification of the tracer transport equations of general circulation models (GCMs). The transpose of the tangent-linear operator, called the “adjoint”, is applied to find the initial sources and sinks. Eulerian backtracking, also called “retro-transport”, is a simplified approach to adjoint modelling, where the roles of updraughts and entrainment are switched with downdraughts and detrainment, respectively, and vice versa (Hourdin et al., 2005a).

In our presented work, we implement both the adjoint method for inverse modelling (Lions, 1971; Marchuk, 1974, 1982) and the retro-transport method put forth by Hourdin et al. (2005a). Our newfound approach consists of adapting these methods to a hexagonal mesh. For this, we use the DYNAMICO dynamical core of the the Laboratoire de Météorologie Dynamique-Zoom (LMDZ) GCM, which computes forward-in-time transport equations on a hexagonal mesh (Dubos et al., 2015). We add routines to DYNAMICO’s source code for the adjoint method, and alter the direction of fluxes to implement Eulerian backtracking.

The hexagonal mesh permits to reduce the computational cost traditionally attributed to inverse atmospheric modelling. Without the decreasing cell size as we approach the poles in a regular lon-lat grid, a hexagonal mesh covers the globe with a smaller number of nearly fixed-size cells. This directly reduces the size of input data and the number of operations. Further, the hexagonal mesh allows modellers to bypass the need for nonlinear cell-treatment at the poles of a regular lon-lat grid, increasing the accuracy of the retro-transport approximation. The increased computational efficiency of the hexagonal mesh paves the way for higher horizontal resolutions for global atmospheric inversion.

How to cite: Takache, S., Chevallier, F., Lloret, Z., and Cozic, A.: Higher-Efficiency Inverse Atmospheric Modelling by Virtue of a Global Hexagonal Mesh, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-13070, https://doi.org/10.5194/egusphere-egu23-13070, 2023.

The current edge handling of the cubed sphere grid introduces numerical errors in simulations and creates grid imprinting due to the discontinuity of the great­-circle grid lines between two adjacent tiles. In this work, we implement a new edge/corner handling method to greatly reduce the grid imprinting in GFDL's dynamical core FV3. First, we extend on the duo-grid method (Chen 2021) to support halo updates of staggered variables. Second, we implement a corner handling algorithm to fill the corner regions using a lagrangian polynomial interpolation. Results of idealized shallow water test cases show that the new halo update methods are able to reduce the numerical noise at the edges/corners and thus reduce the grid imprinting in the numerical solution. This improvement is especially useful for coarse-grid models such as climate models in which the cube edges are most noticeable.

How to cite: Mouallem, J. and Harris, L.: Implementation Of The Novel Duo-Grid Within The GFDL FV3 Dynamical Core, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-1688, https://doi.org/10.5194/egusphere-egu23-1688, 2023.

Physical imbalances introduced by local sequential Bayesian data assimilation pose a problem for numerical weather prediction. For example, fast-mode acoustic imbalances of the order of the relevant slower dynamics destroy solution quality. We introduce a novel dynamics-driven method that suppresses imbalances arising from data assimilation. Specifically, we employ a blended numerical model with seamless access to compressible, soundproof, and hydrostatic dynamics. After careful numerical and asymptotic analysis, we introduce a one-step blending strategy to switch between model regimes within a simulation run. Upon assimilation of data, the model configuration is switched for one timestep to the limit soundproof pseudo-incompressible or hydrostatic regime. After that, the model configuration is switched back to the compressible regime for the duration of the assimilation window. The switching between model regimes is repeated for each subsequent assimilation window. Idealised experiments involving the travelling vortex, buoyancy-driven rising thermals, and internal gravity wave pulses demonstrate that our method successfully eliminates imbalances from data assimilation, yielding up to two orders-of-magnitude improvements in the analysis fields. While our studies involved eliminating acoustic and hydrostatic imbalances, this novel dynamics-driven method of achieving balanced data assimilation can be extended to eliminate other undesired imbalances, with significant prospective applications in real-world weather prediction.

How to cite: Chew, R., Benacchio, T., Hastermann, G., and Klein, R.: Balanced data assimilation with a blended numerical model, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-1254, https://doi.org/10.5194/egusphere-egu23-1254, 2023.

Within the ESiWACE-2 project, a work package was dedicated toward providing services to the European earth system modeling community; the primary aspect of these services was the advancement of weather and climate model components towards exascale hardware architectures. As the bulk of this software is MPI-parallelized Fortran code, significant leaps have to be made in design and engineering to utilize the potential of e.g. GPU-equipped supercomputers that constitute the majority of (pre-)exascale systems that will emerge in the near future in Europe. The service was organized in the form of a call for projects, where awarded modeling groups would benefit from a 6 person-month collaboration with HPC experts within the ESiWACE consortium.

One such project has been the regional climate model RegCM, a state of the art limited area model, developed by the Earth System Physics section of the Abdus Salam International Centre for Theoretical Physics (ICTP) for long-term regional climate simulation. RegCM has participated in numerous intercomparison projects and is designed to be a public, open source, user-friendly and portable code that can be applied to any region of the world. The RegCM userbase extends beyond Europe, both toward industrialized countries (e.g. the US) as well as developing nations. Its development iteration has seen the addition of a non-hydrostatic dynamical core which, coupled with model 1D packages solving the sub-grid scale physics of convection, water phase change, boundary layer, short and long wave solar and long wave earth radiation interaction, permit the model a time integration to produce a climate scenario simulation. The model has an internal coupling with a surface community land model for atmosphere surface interaction description (CLM4.5).

Within the ESiWACE-2 service project, we have accelerated this dynamical core to GPUs using the OpenACC programming model. Within the limited timeframe we have adopted three main optimizations: (i) the restructuring of zonal and meridional advection loops to expose full three-dimensional parallelism, (ii) the use of direct GPU-to-GPU communication through device-aware MPI calls, and (iii) the minimization of GPU-CPU exchanges by excluding any data transfers back to the host, except for I/O and physics parameterizations.

Using these programming techniques, we were able to construct a dynamical core for RegCM that runs exclusively on the GPU. For benchmarking, we use the ‘Alps’ test case, a 3km-resolution mesh with ~14M grid columns over the Alpine region, representative of the future convection permitting model configurations. Benchmarks on the JUWELS-Booster supercomputer show an acceleration by more than a factor of two at low node counts (1-3) which diminishes when higher node allocations are used; at 8 nodes, both CPU- and GPU-versions have comparable speed. For the previous-generation system Marconi-100, the accelerated version is observed to be consistently faster by a factor ~2.7.

Looking forward, performance profiles indicate that the GPU-resident code is mostly bound by MPI-communication latency within the advection substepping. Techniques to mitigate these penalties are currently being investigated. Moreover, more fine-grained parallelisation of complex loops, such as tuned tiling instructions, can further improve the performance of the nonhydrostatic dynamical core of RegCM.

How to cite: Sclocco, A., van den Oord, G., Giuliani, G., Girotto, I., Raffin, E., and van Werkhoven, B.: Acceleration of the non-hydrostatic dynamical core of RegCM using GPUs, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-7333, https://doi.org/10.5194/egusphere-egu23-7333, 2023.

In the last few years, operational modeling of the environment at the National Oceanic and Atmospheric Administration  (NOAA) in the USA has been moving towards a Unified Forecast System (UFS) approach based on open source community models and tools. For NOAA, the main benefits of this approach are to more rapidly transition innovations into operations, and simplifying NOAA’s production suite of models around selected UFS applications. For the broader community, this approach makes operational models easily available for a broad range of research, as well as for testing new ideas in a vetted, near-operational environment. The collaboration and cooperation of the UFS community are powered by the Earth Prediction Innovation Center (EPIC). EPIC is a virtual center managed by the Weather Program Office at NOAA’s Oceanic and Atmospheric Research and designed to ensure that the UFS is an efficient, effective, and user-friendly community modeling system. Additionally, EPIC ensures that NOAA’s operational needs and the Research and Development community are supported with effective Research to Operations and Operations to Research processes. NOAA furthermore drives the development of the UFS by a focus of internal NOAA resources on UFS applications. And by generally requiring applicants to NOAA Funding Opportunities to perform their research and development with UFS tools and approaches.  The presentation will outline the basic principles of the UFS, as well as progress made so far. The latter will highlight code releases and operational implementations, UFS governance, and progress with pre-operational prototype coupled models. 

How to cite: Tolman, H., Jacobs, N., Nanace, L., and Alves, H.: The Unified Forecast System: linking operational and research environmental modeling through an open source community approach, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-10012, https://doi.org/10.5194/egusphere-egu23-10012, 2023.

Modern numerical weather prediction requires vast amounts of computing power, so highly scalable algorithms are essential on massively parallel modern hardware. However, once the strong scaling limit is reached for spatial parallelism, the wallclock time increases with the number of timesteps. This limits the ability to provide higher resolution forecasts on an operational schedule. Parallel-in-time methods overcome this limit by exposing time parallelism, in addition to the spatial parallelism exposed by traditional domain decomposition.

ParaDiag is one such method, which reduces the coupled system for multiple time-steps into a block-diagonal matrix that can be solved in parallel. We will present recent progress on the application of ParaDiag to compatible finite element discretisations of PDEs for atmospheric flow, which are particularly challenging for time-parallel methods due to their highly oscillatory nature. These solvers are implemented as an open source general library using Firedrake, an automated code generation framework for the solution of finite element methods.

Various ParaDiag formulations are explored for linear and non-linear models, and their parallel scaling is compared. Their performance is compared against time serial methods to find the parameter ranges where the greatest speedups are achieved. We identify the main difficulties faced by ParaDiag and describe our approaches to overcoming these, including solution strategies for the block systems within the ParaDiag matrix, and improving the convergence of nonlinear models.

How to cite: Hope-Collins, J. and Cotter, C.: Parallel-in-time solution of finite element atmospheric models using ParaDiag, EGU General Assembly 2023, Vienna, Austria, 23–28 Apr 2023, EGU23-12375, https://doi.org/10.5194/egusphere-egu23-12375, 2023.