The LUE software framework: develop scalable global hydrological models without having to think about high-performance computing

High-resolution continental scale hydrological modelling often requires distributed computing to solve the model equations. Developing and maintaining these models is an enormous challenge as current approaches require knowledge about parallel and distributed computing. A solution to the problem followed here is the development of a modelling framework that deals with parallelization of model equations under the hood. Using such a framework allows hydrologists who are not familiar with low level software development and high-performance computing to develop models.

Our modelling framework, called LUE (de Jong et al. 2022, 2021), allows models that are large in terms of data set size and the number of calculations used, to use all hardware available to them efficiently. LUE models can be written in Python or C++, and can be executed unchanged on laptops and computer clusters.

In the implementation of LUE model operations we use the asynchronous many-tasks (AMT) approach, as implemented in the HPX C++ library. This makes it possible for model developers to express their models using simple algebraic expressions, while all details related to scheduling computations on parallel and distributed hardware are hidden from view. Executing LUE models results in a relative large collection of tasks that are ready to be scheduled for execution on the available hardware. This, in turn, results in models that are finished sooner and that can use additional hardware efficiently, automatically.

We are currently busy porting the PyCatch modelling suite (Lana-Renault and Karssenberg 2013), which is an integrated set of process-based hydrological and soil-vegetation models, to LUE. PyCatch is implemented in terms of generic modelling operations inspired by map algebra (local, focal, zonal, global operations) and flow routing operations like flow accumulation and the kinematic wave.

In our presentation we will further explain the LUE modelling framework, including the operations that are specifically targeted at hydrological modelling, and show results of applying the PyCatch model to Africa at 3 arc-second resolution (~90 m at the equator) using the MERIT Hydro high resolution raster data set.

References
de Jong, K., D. Panja, D. Karssenberg, and M. van Kreveld. 2022. “Scalability and Composability of Flow Accumulation Algorithms Based on Asynchronous
Many-Tasks.” Computers & Geosciences. https://doi.org/10.1016/j.cageo.2022.105083.
de Jong, K., D. Panja, M. van Kreveld, and D. Karssenberg. 2021. “An Environmental Modelling Framework Based on Asynchronous Many-Tasks: Scalability and Usability.” Environmental Modelling & Software. https://doi.org/10.1016/j.envsoft.2021.104998.
Lana-Renault, N., and D. Karssenberg. 2013. “PyCatch: Component Based Hydrological Catchment Modelling.” Cuadernos de Investigación Geográfa. https://doi.org/10.18172/cig.1993.