Using Multimodal LLMs for Digitising Handwritten Climate Records

Millions of historic handwritten weather observations remain locked in paper records, leaving this valuable information inaccessible for analysis and at risk of permanent loss. Manual transcription of these records is highly accurate but time-consuming and costly, making this a task where AI could play a pivotal role. Traditional optical character recognition (OCR) methods struggle with the irregularities of historical handwriting and tabular layouts. This study proposes a novel automated digitization pipeline that leverages multimodal large language models (MLLMs) alongside table structure recognition (TSR) and OCR techniques to transcribe handwritten climate records efficiently and accurately.

First, we compare two MLLMs, and find that by guiding the MLLM with structured prompts and validating outputs based on physical relationships between meteorological variables, we achieve transcription precisions of up to 97%. This rivals human accuracy, though at the cost of a lower inclusion rate due to strict filtering. Second, we link MLLM outputs to detected table structures to generate training data for fine-tuning a pretrained OCR model. Fine-tuning significantly enhances transcription quality, improving from 19% to 81% on unseen data. Challenges remain due to the complexities of TSR in historical documents, reducing the quality of our training data. Despite these limitations, our research establishes a viable framework for scaling data rescue efforts, bringing us one step closer to unlocking centuries of climate data for scientific analysis.