Fusing UAV-based hyperspectral and RGB imagery for potato plant disease detection

Pest and pathogen pressure in potato cultivation is increasingly affecting the potato quality and yield. The Netherlands, as the largest seed potato producer around the world, is particularly threatened by blackleg disease and potato virus Y (PVY). Uncrewed aerial vehicle (UAV)-based imaging combined with machine- and deep-learning methods have shown clear potential for potato disease identification, offering advantages over conventional human inspections, which are labor-intensive, expertise-demanding, and often subjective. Most existing studies focused on RGB data and pixel-level classification, producing maps that have limited practical value for targeted removal of infected plants. Earlier work demonstrated the potential of plant-level disease detection approaches. For example, Jia[1] employed hyperspectral data (specifically the first three principal component analysis (PCA) bands) with a YOLOv5s model to distinguish the blackleg- and PVY-infected plants from healthy ones, yielding average mAP@.50 scores of 0.85 for blackleg detection and 0.82 for PVY detection. Gibson-Poole[2] applied object-based image analysis (OBIA) to detect blackleg disease with RGB imagery, achieving a total accuracy of 87%. The findings suggest that multi-modal data (combining hyperspectral and RGB imagery) hold strong potential for plant-level disease detection. We aim to identify the most informative features derived from hyperspectral data and to investigate their integration with RGB data to enhance potato disease detection performance.

We proposed early fusion (E), where data were concatenated channel-wise before network input, and middle fusion (M) architectures, where features were extracted separately within a two-branch network and then merged at an intermediate stage, to integrate hyperspectral features and RGB imagery for potato disease detection. To reduce hyperspectral dimensionality, two feature sets were extracted: (i) the first three PCA bands, and (ii) 10 vegetation indices (VIs) selected from 64 candidates using variance inflation factor analysis to mitigate multicollinearity. Consequently, four models were developed and evaluated: E-PCA-RGB, E-VI-RGB, M-PCA-RGB, and M-VI-RGB. Unlike previous studies that focused on a single disease, our models detected blackleg-infected, PVY-infected, and healthy plants simultaneously. E-VI-RGB achieved the highest mAP@.50 value of 86.65±1.53, followed by M-VI-RGB (85.74±1.75). E-PCA-RGB and M-PCA-RGB yielded mAP@.50 scores of 83.00±2.52 and 83.11±2.20, respectively. These results demonstrate that combining hyperspectral features with RGB imagery improves detection performance compared with single-modality approaches (RGB 83.21±1.31, PCA 79.71±1.30, VIs 85.31±2.11). Our findings highlight the potential of multimodal fusion for potato disease detection in practice. The methods could enable automated systems not only to identify infected plants but also to support timely removal with machinery, mitigating the spread of disease in potato fields. The generalizability of our approach will be further tested and analyzed in future work.

References

[1] Jia, T., Smigaj, M., Kootstra, G. and Kooistra, L., 2024. Detection of Diseased Potato Plants with UAV Hyperspectral Imagery. In 2024 14th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS) (pp. 1-5). IEEE.

[2] Gibson-Poole, S., Humphris, S., Toth, I. and Hamilton, A., 2017. Identification of the onset of disease within a potato crop using a UAV equipped with un-modified and modified commercial off-the-shelf digital cameras. Advances in Animal Biosciences, 8(2), pp.812-816.