The CMIP Rapid Evaluation Framework (REF) for automated and systematic benchmarking of coupled models

The goal of the Coupled Model Intercomparison Project (CMIP) is to better understand past, present, and future changes in the Earth system in a multi-model context. In an effort to increase the project’s scientific and societal relevance, improve accessibility, and widen participation, the CMIP Panel advocated for establishing a number of Task Teams aimed at supporting the design, scope, and definition of the next phase of CMIP, as well as the evolution of infrastructure for and operationalization of CMIP activities. 

An important prerequisite for providing credible information about the Earth system from models is to understand their capabilities and limitations. Thus, systematic and comprehensive assessment of models in comparison with best-available observational and reanalysis data is essential. For CMIP7 new model evaluation challenges stemming from higher resolution, enhanced complexity, and machine learning components need to be rigorously addressed. The Climate Model Benchmarking Task Team aims to provide a vision and concrete guidance for establishing a systematic, open, and rapid performance evaluation of the expected large number of models participating in CMIP7, including a variety of performance metrics and informative diagnostics. The Task Team designed a plan for a community Rapid Evaluation Framework (REF) that would leverage and integrate existing open source community model evaluation tools for benchmarking the performance of CMIP simulations contributed by participating modeling centers.

Based on community input, an initial set of metrics and diagnostics were identified to be applied to Intergovernmental Panel on Climate Change (IPCC) Seventh Assessment Report (AR7) Fast Track simulations. With co-sponsorship from the US Department of Energy (DOE) and the European Space Agency (ESA), the development of the REF was launched in November 2024. The REF delivery team will integrate evaluation tools and a workflow system that will be deployed at two or more primary Earth System Grid Federation (ESGF) node sites to provide automated production of diagnostic information for CMIP model developers, data users, and stakeholders. The REF is expected to evolve to provide additional metrics and diagnostics and to use more data products in the future through the guidance of a community panel or consortium that will be formed in the coming year.