PINN-Cast: Exploring the Role of Continuous-Depth NODE in Transformers and Physics Informed Loss as Soft Physical Constraints in Weather Forecasting

Traditional operational weather prediction systems are driven by physics-based numerical simulations, which demand substantial computational resources. With the advancement of Artificial Intelligence (AI), modern transformer architectures have emerged as powerful alternatives, delivering high accuracy in data-driven weather forecasting. Despite this progress, transformers inherently operate on discrete representations and do not follow the underlying physical laws, thereby limiting their effectiveness in modelling the continuous spatio-temporal evolution of atmospheric processes. To mitigate this issue and inject physical structure, we introduce continuous-depth dynamics within the encoder and attention mechanism of a transformer. We propose the dual attention mechanism that jointly captures spatial and temporal dependencies. The spatial mode is modelled as a simple multi-head attention which is fused with the temporal component. The temporal attention operates on finite-difference derivatives of token embeddings across successive time steps, allowing the network to infer local temporal gradients and represent continuous evolution in feature space. Furthermore, we introduce continuous-depth Neural ODE layers in transformer encoder which models smooth transitions replacing the discrete residual updates. Finally, we propose a customized physics-informed loss function which is applied during training as a soft-constraint. This loss penalizes deviations from established thermodynamic and kinetic energy relationships governing temperature and wind evolution. By constraining the learned dynamics to respect these physical laws, the model produces forecasts that are not only data-accurate but also energetically consistent with the underlying principles of the atmospheric system.