Ramses-4-CE: Towards Enhanced Generalization of RGB/Hyperspectral Imaging Data Processing

In the quest for achieving the principles of a circular economy, our Helios Lab seeks to optimize the e-waste recycling industry with its innovative Ramses-4-CE project (KIC RM 19262). Ramses focuses on the development of a smart network comprising multimodal optical non-invasive sensors mimicking industrial scenarios with a conveyor belt that can attain speeds of several meters per second. The primary objective is to facilitate the comprehensive identification and characterization of e-waste materials, particularly printed circuit boards (PCBs), plastics, and rare earth elements (REE).

The sensor ensemble encompasses a laser profiler generating height maps, an RGB camera capturing surface spatial features, hyperspectral cameras capturing the spectral features, and chemical characteristics obtained with a Raman spectroscopy sensor affixed to a robotic arm. Each sensor type offers unique advantages and inherent challenges. RGB cameras with their data facilitate fast, highly accurate, and smart data processing e.g., using machine learning (ML) and deep learning (DL) object detection and segmentation techniques in shredded plastics, while hyperspectral imaging (HSI) aids in polymer identification based on spectral fingerprint libraries. Nonetheless, HSI poses challenges such as large data size due to its abundant information, noise interference, and overlong processing times.

To optimize the data processing pipeline, meticulous preprocessing and processing methods have been devised. Upon data acquisition of different objects and materials, data co-registration is executed on the resulting RGB images and hyperspectral cubes, followed by object detection and segmentation of valuable objects on both data types. For objects eluding identification via RGB and hyperspectral imagery, a Raman spectroscopy-based validation is involved for detailed chemical analysis.

Yet, exerting high accuracy in HSI pixel-wise classification on multi-unseen data cubes necessitates HSI classification models with robust generalization capabilities. Towards this aim, smart automated masking of undesired objects in the hyperspectral scene is developed. HS cube contains abundant data causing their large volume size. This abundance highlights the useful information, while concurrently amplifying noises and artifacts, detrimentally affecting both data processing speed and model generalization. Masking undesired objects in the HSI reduces the number of pixel vectors skewing calculations in preprocessing steps and DL models training routines, leading to enhanced segmentation models, i.e., masking unwanted data vectors from HSI allows exclusive processing for desired targets elevating processing speed without compromising accuracy.