Big data techniques for real-time hyperspectral core logging

Hyperspectral imaging is gaining widespread use in the resource sector, with applications in mineral exploration, geometallurgy, and mine mapping. However, the sheer size of many hyperspectral datasets (>1 Tb), and associated data correction and analysis challenges, limit the integration of this technique into time-critical exploration and mining workflows. We present an overview of several newly developed  real-time processing capabilities to mitigate these challenges, and so provide hyperspectral data and derived products (e.g., mineral abundance estimates) in near real-time. This allows for efficient, timely, and automated delivery of hyperspectral data to enhance geological activities.

Hyperspectral data generally needs to be corrected, coregistered, cropped and masked, before derivative results can be generated, visualized and stored. To achieve real-time processing, each of these steps, which can involve the computationally intense manipulation of several Gb worth of spectral data, need to be completed within the 1-3 minutes a typical instrument or scanner takes to capture a data cube. To help with this, we have developed a python-based asynchronous processing pipeline, crunchy, that uses a file-discovery-based triggering mechanism to spawn parallel processing workflows that automatically perform these tasks. Coregistered and radiometrically corrected results are then stored using a directory-based data structure managed by a second python utility, hycore, that facilitates (1) consistent data storage, (2) file-based out-of-core processing, and (3) management of the various metadata required to localize and give meaning to hyperspectral drill core data. We have also developed a third python tool, hywiz, to enable an easy browser-based interaction with hycore databases. This includes the visualisation of sensor results and analysis products for individual trays and drillhole mosaics. Additional data such as assays, logging notes or downhole geophysical data can be overlain on these to enable integrated interpretation of otherwise disparate datasets. 

We hope that these tools will enable greater use of hyperspectral data in research and industry, and facilitate e.g., hyperspectrally enhanced core-logging, sample selection, vectoring and, potentially, realize self-updating 3-D geological models.