Unsupervised pattern recognition for imperfect datasets: a visual workflow for plausibility checks and regime diagnosis in high-dimensional environmental time series

Data imperfection is characterized by fragmentation, sensor failures, and high-dimensional noise. This remains a persistent challenge in environmental monitoring. As observation networks expand to capture heterogeneous soil-atmosphere interactions, traditional quality control methods based on rigid statistical thresholds often struggle to distinguish between sensor errors and genuine, non-linear system dynamics. This study presents a methodological development for knowledge extraction from imperfect and fragmented data, employing a multivariate visualization workflow that combines Principal Component Analysis (PCA) and Self-Organizing Maps (SOM) with Sammon Mapping.

We applied this unsupervised learning approach to a high-dimensional dataset (~100 variables) from a field-scale agricultural system, including measurements of soil moisture and temperature, eddy covariance-derived CO₂, energy fluxes, radiation, wind, precipitation, groundwater level and discharge.

This allowed us to compare a discontinuous period in 2024 against a continuous period in 2025. The results demonstrate the method's robustness in extracting coherent structural patterns despite data incompleteness. While PCA effectively isolated the dominant thermodynamic baselines from high-frequency hydrologic events, the topological SOM projection provided a rapid, visual plausibility check.

The visualization facilitated the identification of possible irregularities in the sensors as spatial outliers in the 2024 dataset, facilitating instant anomaly detection without manual time-series inspection. Furthermore, the method successfully captured shifts in system dynamics, such as the decoupling of surface moisture from groundwater, validating its utility for identifying physical regimes in heterogeneous data. We conclude that this visual workflow offers a scalable, data-driven solution for moving from raw, imperfect observations toward actionable system diagnostics, bridging the gap between data acquisition and process understanding in complex environmental observatories.