Combining human and AI approaches for effective digitization of historical atmospheric electricity records

Long-term measurements of the atmospheric electric field, measured as the potential gradient (PG), were obtained at Lerwick Observatory (Shetland Isles), UK, from 1925 to 1984, and provide a unique resource for studying links between atmospheric electricity, the global electric circuit (GEC), and climate variability. Most of these historical observations were originally made as handwritten or printed records, limiting their accessibility for modern analysis. In this project, we have undertaken a comprehensive digitization of the Lerwick PG dataset, at hourly resolution, by combining contributions from a citizen science platform and various AI tools.

The earliest handwritten records, made from 1927–1956, were digitized through the Zooniverse citizen science platform by engaging volunteers in transcribing data.  To digitize the later records, from 1957–1984, which are mainly printed and scanned tables, we utilized AI-based optical character recognition (OCR) tools from several software packages. An essential part of the transcription the use of multiple validation steps to assess and correct errors introduced by both the AI-based tools and the citizen science activity. By these techniques, we optimised the effectiveness of the digitisation to provide the most scientifically useful dataset.

This work presents a summary of the digitized historical dataset from Lerwick and provides insights into the reliability and limitations of AI-assisted digitization of scientific archives. The resulting new dataset generated will underpin modern investigations into long-term trends in atmospheric electricity and its connection to climate processes.