Tackling practical challenges in anomaly detection for real-time monitoring of urban waste water networks

Waste water networks constitute a crucial element of urban infrastructure that are influenced by an observed increase in urban flooding events. To ensure regular network operation and minimal environmental impact, anomaly detection of urban waste water networks timeseries can serve as a real-time monitoring tool to detect a) sensor defects and b) system anomalies such as leaks or blockages. However, setting up such a monitoring system in practice can face significant challenges. These include limited amounts of labeled anomalies, heterogenous data quality, inconsistent measurement frequencies as well as instationarity of the system (sensor displacement and drop-out, changes in network layout). For the waste water network of a medium-sized German city, we set up machine learning based anomaly detection and present strategies to tackle aforementioned challenges. Our results show that autoencoder-based model architectures are valuable tools in such a context where only a minimal fraction (<0.01%) of the data is labeled. Both a well-parametrized interpolation strategy and a model architecture that is largely robust to missing values are essential prerequisites for adequate model performance. Based on our results, we derive general strategies to aid in setting up anomaly detection systems in real-world use cases.