Reinforcement Learning for a system-of-systems approach in water management

The water cycle connects many essential parts of the environment and is a key process supporting life on Earth. Amid climate change impacts and competing water consumptions from a growing population, there is a need for better management of this scarce resource. Yet, water management is complex. As a resource, water exists under various forms, from water droplets in the atmosphere to embodied water in consumer products. Its flows and existence transcend national and geographic borders; its management, however, are limited by boundaries. To date, machine learning has shown potentials in applications across domains, from showing skills in game plays to improving efficiencies and operation of real-life processes. The system-of-systems perspective has emerged in many fields as an attempt to capture the complexity arising from individual components. Within a system, the interactions and interdependencies across components can produce unintended consequences. Moreover, their effects that are not explainable just from studying a component on its own. Its concept intertwines with Complexity Science, and points to Wicked Problems, solutions of which are difficult to find and achieve. Climate change itself has been recognised as a ‘Super Wicked’ problem, for which deadlines are approaching but for which there are no clear solutions. Yet, there is often a lack of understanding of the interactions and dependencies, even from a physical modelling perspective. A comprehensive approach to capturing these interactions is through physical modelling of water processes, such as hydraulics and hydrological modelling. The structure and data pipelines of such an approach, nevertheless, is static and does not evolve unless reconfigured by model experts. 

We propose that a form of machine learning, Deep Reinforcement Learning, can be used to better capture the complex whole system interactions of components in the water cycle and assist in their management. This approach capitalises the rapid advances of Machine Learning in environmental applications and differ to traditional optimisation techniques in that it provides distributed learning, consistent models for components that can evolve to connect and continuously adapt to the operating environment. This is key in capturing the changes brought about by climate change and the subsequent environmental and human change in response.

1. Reinforcement Learning for improving process modelling to produce a spectrum of fully physical models for hybrid physical-neural networks to full Deep Learning models that can mimic the natural processes of interest, such as streamflows or rainfall-runoff. An example case study could be a hydrological model of a river catchment and its upstream-downstream dam operation. The components in this case can be individual reservoir models, neural network-based emulators, or differential equation models.

2. Reinforcement Learning for holistic modelling of physical processes in water managemen to capture the whole system. Since each component is modelled as a full or hybrid physical-neural network model, the components could be integrated to provide a whole system approach. Within this, Reinforcement Learning can act as the constructor or go beyond this to provide solutions for targeted problems.