Improving impact model intercomparison by developing and applying quality control and quality assessment tools – the example of the ISIMIP global water sector

Process-based impact models are frequently used for a range of applications and are valuable for simulating fundamental processes in a changing world. Model Intercomparison Projects like the Inter-Sectoral Impact Model Intercomparison Project (ISIMIP, www.isimip.org) act as an umbrella for various sectors (e.g. water, agriculture, health) and numerous modelling teams that are following a common modeling protocol that enables model intercomparison and (cross-) sectoral multi-model impact assessments. However, such assessments require reliable model outputs which can be checked from two perspectives.

First, a quality control (QC) check ensures that simulated files follow the standards defined in the modelling protocol and includes plausibility checks. For example, structural inconsistencies and correct metadata entries can be assessed, but also in cases where the range of a specific variable exceeds plausibility limits (e.g. negative precipitation values), such a tool can facilitate error checking which is very helpful especially in the case of high data volume simulation outputs (e.g., errors stemming from an erroneous unit conversion).

Second, a quality assessment (QA) tool compares model output to observation data or benchmark models. This is particularly important for model development and improvement as it can highlight benefits and limitations of models for e.g., specific model configurations, but it also informs the identification of models that are best suited for specific regions and research questions.

Within the EU COST-Action “Process-based models for climate impact attribution across sectors“ (PROCLIAS), the aim is to establish a QC/QA workflow for the ISIMIP models. A QC tool is already developed and in operation which checks the data format and, exemplarily for the global water sector, each variable for plausibility ranges. An operational QA tool does not yet exist within PROCLIAS and ISIMIP but some experiences have been gained with existing evaluation frameworks such as ILAMB and the ESMValTool.

This presentation provides experiences gained with the QC tool and the application of ISIMIP data to existing QA frameworks and outlines the next milestones. It is planned to extend the plausibility ranges to all ISIMIP sectors by a survey within the modelling teams. For the QA tool, specific developments are required to integrate sector-specific evaluation methods (e.g., basin outlines into ILAMB). To use ESMValTool, the model output data needs to be restructured to a CF-compliant format. With the ISIMIP global water sector as a pilot sector, experiences are gained that will then be transferred to other sectors. This activity also calls for an exchange of ideas and experiences from other modeling communities.