ClimateFieldQA: Evaluating Vision–Language Models on Tropical Cyclone Hazard Fields with Physics-Aware Visual Abstractions

The analysis of the impacts due to climate extremes, such as extreme precipitation, heatwaves, and tropical cyclones, needs to rely on multimodal data, ranging from complex geophysical fields to textual and visual data.

While recent advances in vision-language models (VLMs) have stimulated interest in multimodal-driven climate analysis, their application to natural hazard analysis is still relatively limited.

In this work, we focus on tropical cyclones, and construct a new framework, namely Visual Object Representation for Tropical Cyclone Extremes and eXtent (VORTEX), a physics-aware, visual abstraction designed to support interpretable vision-language reasoning over hazard fields for tropical cyclones.

VORTEX transforms spatiotemporal reanalysis data associated to tropical cyclones into structured, visually identifiable representations by explicitly encoding cyclone-specific physical properties, including pressure-anchored storm geometry, wind and precipitation intensity extrema, spatial asymmetry, and field-scale footprint.

Building on VORTEX, we construct ClimateFieldQA, a structured evaluation framework for diagnosing VLM reasoning over tropical cyclone hazard fields. ClimateFieldQA comprises 4,978 high-resolution reanalysis heatmaps and 243,922 instruction samples spanning spatial localization, intensity estimation, structural pattern recognition, field-scale extent reasoning, and physical impact analysis.

ClimateFieldQA is designed to expose strengths, limitations, and failure modes of VLM-based reasoning under physically constrained geoscientific settings.

Using ClimateFieldQA, we show that physics-aware visual abstractions systematically improve structure-sensitive reasoning and reduce common interpretation errors observed when VLMs operate on raw hazard fields, highlighting the methodological importance of representation design for climate impact analysis and natural hazard assessment in Earth system science.