Efficient OCR Post-Correction and Annotation for Neo-Latin Corpora. A Human-in-the-Loop Workflow Integrating LLMs in Digital Scholarly Editing

Efficient OCR Post-Correction and Annotation for Neo-Latin Corpora A Human-in-the-Loop Workflow Integrating LLMs in Digital Scholarly Editing Sarah A. Lang Max Planck Institute for the History of Science (Berlin) Aarhus/online, 2025-09-24 1/19 Bridging Computational and Humanities Needs This presentation does not aim to provide research novel for digital humanities. Rather, it seeks to bridge digital methods with the actual needs of historians and humanities scholars. As computational approaches move closer to computer science, we should not forget about the more basic uses of digital methods that are more closely aligned with what scholars actually need. 2/19 Large Scale DH vs. Humanists Expectations High Accuracy Standards for Editions •Minimal Tolerance for error digital scholarly editing. • Lesser quality can render data unusable or not taken seriously by scholars. •Implication: Perfectly corrected data is valuable but resource-intensive. Can AI help? Starting Points: NOSCEMUS ERC & NOSCEMUS GM •Corpus: Representative for Neo-Latin scientific literature. •HTR Model: The NOSCEMUS GM on Transkribus. •Access: Transkribus is now a (mostly) closed service. •Criticisms: TEI-XML data based on imperfect OCR is met with resistance by scholars. 3/19 A Human-in-the-Loop Post-Correction Workflow Pressing Question How can current technologies assist in creating perfectly corrected editions more efficiently? Suggested Solution Corrections suggested by LLMs are manually reviewed using visual diff tools. This offers an efficient yet fully human-controlled process. Approach • Use the strong NOSCEMUS GM as base OCR/HTR. • Employ LLMs to suggest corrections. →this part can be automated via API. • Integrate visual diff tools (e.g. colordiff or eScriptorium) for manual verification. •→Manual correction remains central for accuracy and conformance to guidelines because LLMs often degrade OCR quality (Boros et al. 2024). They tend to ignore or mix-up transcription standards, making outputs unusable for OCR re-training / generating ground truth 4/19 Leveraging colordiff (1) 5/19 Leveraging colordiff (2) Model: gpt-3.5-turbo via API; script: https://github.com/sarahalang/ LLM-powered-OCR-correction/blob/main/korr-ocr.py 6/19 Diff functionality in eScriptorium 7/19 Making things more difficult 8/19 Case Study: Recognising Alchemical Symbols The Project •Actual Research Goal: Enable automatic recipe extraction – but the need for manual PCR correction remains a major bottleneck. •Aim: Retrain model for rare alchemical symbols. •Corpus: Complex page layouts from Magnet’s Pharmacopoeia = Johannes Jacobus Magnet, Pharmacopoea Schrödero-Hoffmanniana illustrata et aucta etc (1687). URL: http://data.onb.ac.at/rep/10542338 . •High correction time (30–60 minutes per page); pages rich in symbols are most valuable but most labor-intensive. Crowdsourcing by students does not work • Requires Latin proficiency. • Needs familiarity with abbreviations and OCR error types. • Demands high accuracy in symbol identification and adherence to transcription guidelines. 9/19 Current Limits of LLM-powered OCR Post-Correction Challenges • The task is highly specialised and hard to outsource. • LLMs fail due to lack of exposure to rare material and non-standard knowledge. They normalise data, ignore guidelines, and degrade OCR output. • Even with Unicode input, LLMs hallucinate corrections. Current Findings • Traditional machine learning is necessary for progress in this area. • Some success with more frequent measurement symbols. • Alchemical symbols are too rare for effective training on only single pages. 16/19 The Dangers of Hallucinations Arthur Dee and the fake preface • LLM generated a plausible early modern preface. • None of it matched the actual manuscript. • Hallucinations are especially dangerous due to their believability / plausibility. • Even when prompted with transcription segments, models fabricate. Conclusion This reinforces the necessity of a human-in-the-loop approach that leverages LLMs to help generate further training / fine-tuning data to feed into traditional ML models (more reliable for our tasks). 17/19 Conclusions Critical Reflection • Does AI reduce effort, or leave only tedious tasks? • Historians risk being reduced to AI fact-checkers of hard-to-check, highly plausible looking content. • Current state-of-the-art studies (e.g. Boros et al. 2024) question the efficiency of such workflows. AI vs. Humanities scholars • Historians’ skills remain indispensable. • AI can assist but not replace human judgement in critical tasks. • The human-in-the-loop approach remains simple yet effective. 18/19 References [1] Emanuela Boros et al. “Post-Correction of Historical Text Transcripts with Large Language Models: An Exploratory Study”. In: Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024). Ed. by Yuri Bizzoni et al. St. Julians, Malta: Association for Computational Linguistics, Mar. 2024, pp. 133–159. URL: https://aclanthology.org/2024.latechclfl-1.14/. 19/19