A Multi-Modal Observation-Driven Foundation Model for Global Data Assimilation

Accurate characterization of meteorological conditions across urban regions is crucial for developing equitable energy and climate solutions. However, this task faces significant challenges: observational data often lacks spatial and temporal continuity, while numerical weather prediction models can struggle to localize severe weather events due to substantial latency between observation collection and forecast production. To address these challenges we introduce a multi-modal foundation model that rapidly integrates heterogeneous in situ and satellite data to produce a gap-filled global state. Our results show that the atmospheric structures predicted by our model are consistent with observed phenomena such as liquid and frozen precipitation and convection. Further, we apply our model to produce reanalysis and forecast datasets of solar irradiance for renewable energy applications. We also discuss ongoing work to connect global and local systems through a regional high-resolution foundation model, which is driven by multi-modal observations and the dynamics captured by the global model. This research aims to build predictive understanding of environmental systems and their interactions with built environments by improving our ability to forecast phenomena such as cold and warm fronts, convective weather, and their impacts on health, safety, and energy supply and demand.