An Adaptive Masking Time Series Transformer based Representation Learning Model for Well Log Curves

Well log curves, acquired from downhole logging tools during well logging, are pivotal for reservoir characterization and formation evaluation in oil and gas exploration and production. However, manual feature extraction from raw curves remains essential for constructing effective machine learning models, presenting time-consuming challenges and stringent labeling requirements. Concurrently, the transformer architecture, prevalent in NLP and computer vision, offers promise for representation learning. This paper proposes a self-supervised transformer based methodology for extracting well log curves representations, aiming to expedite downstream model development.

While transformer models have gained prominence in handling text and image data, well log curves present a distinct challenge as they resemble time series data. Despite the nascent development of time series transformer models, we conducted an extensive review of current progress and adopted the best-performing time series transformer model for extracting representations from well log curves. Importantly, given the challenges posed by factors such as borehole conditions and instrument failure, certain types of well log curves may occasionally be missing or distorted. To address this issue, our proposed methodology introduces an adaptive masking mechanism, which selectively applies masking to patches of curves where data quality is poor, thereby effectively mitigating data quality concerns.

Data from 2000 wells are utilized for model training, with an additional 100 wells reserved for validation purposes. Our study observed a consistent decrease in both training and test losses until convergence during the training stage. Initially, mean squared error (MSE) and mean absolute error (MAE) are employed to quantify reconstruction errors between reconstructed curves and raw curves, low values of MSE (0.08) and MAE (0.07) indicate effectiveness of the learned representations. Subsequently, a downstream task involving oil and gas identification is undertaken, wherein a classification model is developed based on representations learned by the transformer model. Performance comparison between models utilizing learned representation and those employing statistical features highlights the superior performance of the former (98% accuracy), emphasizing the efficacy of our representation learning methodology. This paper introduces a novel self-supervised methodology based on transformer architecture for well log curve representation learning. The method automates information extraction without requiring logging expertise and substantially enhances downstream machine learning model performance.