Topology-preserving Feature-Space Analysis for Diagnostic Comparison of Weather Forecasting Models

This study proposes a novel diagnostic framework for systematically and intuitively evaluating the performance of medium-range weather forecasting models from a multivariate perspective. Traditional evaluation methods have primarily relied on point-wise error metrics (e.g., RMSE) for single variables at specific altitudes, which limits the analysis of inter-variable correlations and the dynamic evolution of forecast structures over lead times. To address these limitations, we present a methodology that integrates multivariate data into a shared, topology-preserving feature space, enabling the comparison and diagnosis of model-specific prediction trajectories.

The framework first represents multivariate atmospheric variables as images to extract semantic feature representations. To ensure robustness against spatial shifts and noise, we employ contrastive learning with data augmentation, effectively capturing the core physical characteristics of the atmospheric state. Subsequently, we apply a parametric manifold embedding specifically designed to preserve both the local neighborhood relationships of the high-dimensional feature space and its temporal continuity. This approach allows for a coherent and aligned comparison of prediction trajectories from diverse forecasting models within a unified coordinate system.

For the experimental setup, the feature space was defined using ERA5 reanalysis data from 2020 to 2024, with the 2025 ECMWF analysis serving as the reference ground truth. We analyzed a total of nine forecast configurations, combining three AI-based models (FourCastNet, GraphCast, and Pangu-Weather) with three operational numerical weather prediction initializations (IFS, KIM, and UM). By tracking trajectories at 6-hour intervals for up to 48 lead times, we visually analyzed model-specific dispersion and bias characteristics. Furthermore, the diagnostic validity of the framework was verified by comparing trajectory evolutions across different pressure levels and analyzing structural changes induced by varying variable compositions.

The proposed framework supplements conventional univariate and direction-agnostic metrics by enabling structure-aware, directional diagnostics in a multivariate feature space. It provides deep analytical insights into model-specific behaviors, serving as a critical diagnostic tool for future research on atmospheric pattern analysis and inter-variable correlation structures.