Stabilized Neural Differential Equations for Hybrid Modeling with Conservation Laws

Neural Differential Equations (NDEs) provide a powerful framework for hybrid modeling. Unfortunately, the flexibility of the neural network component of the model comes at the expense of potentially violating known physical invariants, such as conservation laws, during inference. This shortcoming is especially critical for applications requiring long simulations, such as climate modeling, where significant deviations from the physical invariants can develop over time. It is hoped that enforcing physical invariants will help address two of the main barriers to adoption for hybrid models in climate modeling: (1) long-term numerical stability, and (2) generalization to out-of-sample conditions unseen during training, such as climate change scenarios. We introduce Stabilized Neural Differential Equations, which augment an NDE model with compensating terms that ensure physical invariants remain approximately satisfied during numerical simulations. We apply Stabilized NDEs to the double pendulum and Hénon–Heiles systems, both of which are conservative, chaotic dynamical systems possessing a time-independent Hamiltonian. We evaluate Stabilized NDEs using both short-term and long-term prediction tasks, analogous to weather and climate prediction, respectively. Stabilized NDEs perform at least as well as unstabilized models at the “weather prediction” task, that is, predicting the exact near-term state of the system given initial conditions. On the other hand, Stabilized NDEs significantly outperform unstabilized models at the “climate prediction” task, that is, predicting long-term statistical properties of the system. In particular, Stabilized NDEs conserve energy during long simulations and consequently reproduce the long-term dynamics of the target system with far higher accuracy than non-energy conserving models. Stabilized NDEs also remain numerically stable for significantly longer than unstabilized models. As well as providing a new and lightweight method for combining physical invariants with NDEs, our results highlight the relevance of enforcing conservation laws for the long-term numerical stability and physical accuracy of hybrid models.