From Annotation to Insight: Human-in-the-Loop Machine Learning for Historical Archives in HAICu WP2

NWO FUNDING: NWA.1518.22.105 From Annotation to Insight: Human-in-the-Loop Machine Learning for Historical Archives in HAICu WP2 C.A. Romein1 , K. van Schuijlenburg2, S. Peeters1 , B.J. Wolf3 , S.J.L. Weggeman3 , R. van Koert 4 , K. Dijkstra3 ,M.A. Dhali2, A. Weber12 , L.R.B. Schomaker2* 1 University of Twente; 2 University of Groningen; 3 NHL Stenden; 4 KNAW Humanities Cluster/ Digital Infrastructure *The order is based on relative contributions., starting with the largest contribution. CRedIT for this poster: Conceptualisation: CAR, AW, KvS Data curation: BW, CAR, KvS, SP Funding acquisition: AW, LS Investigation: all Methodology: BW, CAR, KvS, SP, RvK Resources: all Supervision: AW, MD Visualisation: CAR, KvS, SP Writing orig.: CAR, KvS Writing rev.: all t-SNE features of a DINO-embedding. National Archives, The Hague 4 Million Scans of Colonial Governance The archives of the Dutch Ministry of Colonial Aairs (1850-1900) contain millions of pages with diverse layouts: printed tables, handwrien annotations, mixed-format registers, and correspondence. Traditional text recognition fails to capture the structural complexity of these documents. At the University of Groningen, we extract the page of interest from images, allowing subsequent methods to focus only on relevant features. Deep learning models are continuously trained during data labelling, and computer vision algorithms are utilized to make labelling eicient. Working with NHL-Stenden University of Applied Sciences, we employ advanced deep-learning technologies, including Laypa for document layout analysis, and DINOv2 vision transformers to create sophisticated latent space embeddings for the unsupervised clustering of structural features. Visualising this latent space reveals meaningful clusters: handwriting styles, printed tabular structures, implicit tables on mixed-layout pages, and varying writing densities across dierent column configurations. Simultaneously, researchers at the University of Twente exploit the archive’s inherent historical organisation—indices and klappers—to develop rapid keyword-based information retrieval systems. Continuous Learning Through Citizen Science Central to our approach is the recognition that fully automated systems cannot capture the nuanced complexity of historical documents. We develop new frameworks where machine-learning models improve lifelong machine learning through strategic human intervention. Active Learning Strategies Algorithms identify uncertain or ambiguous cases, prioritising these for expert review. Scholars and trained volunteers provide targeted annotations precisely where models need guidance, maximising the impact of human eort. Continual Learning Architecture Rather than training a model for each scenario, we aim to incrementally learn models new models, tasks and classes without forgeing what has previously been learned. Democratising Research This collaborative approach engages diverse stakeholders—from academic researchers to heritage volunteers —in preserving and interpreting cultural archives. Human expertise guides computational power, whilst machine analysis reveals paerns invisible to manual review. Collectie Overijssel - Resolutions Centuries of Political Voice (1578-1795) The Staten van Overijssel archives document political and social interactions across more than two centuries of Dutch history. Within these ±60,000 pages, historical petitions oer particularly rich insights into societal concerns, representing authentic voices from early modern communities. A researcher at the University of Twente focuses on generating innovative automatic metadata to make these documents accessible beyond specialist circles. Using tools such as ANNIF for subject indexing, we will create semantic descriptions that connect historical content to contemporary understanding. The challenge lies not merely in transcribing text but in understanding context: recognising petitioners, identifying recurring grievances, tracing administrative responses, and mapping networks of political interaction. Human-in-the-loop methods prove essential here, as historical expertise guides the machine in distinguishing between formulaic language and substantive content. This work makes regional archives interpretable for the general public, revealing how ordinary people engaged with political institutions centuries ago. The HAICu project (digital Humanities, Artificial Intelligence & Cultural heritage, 2024-2030) develops innovative multimodal machine learning architectures for cultural heritage collections. Work Package 2 researchers lifelong machine learning frameworks that evolve through ongoing collaboration between computational models and human expertise. Rather than treating digitisation as a one-time process, we create feedback loops where volunteers and scholars continuously refine algorithms, enabling analysis of complex historical documents. Our research addresses three diverse Dutch archival collections, each presenting unique challenges in layout analysis, content extraction, and semantic interpretation. Groninger Archives Our third use case develops complementary approaches for collections at dierent stages of digitisation, ensuring our methods remain adaptable to varying institutional capacities and metadata standards. Looking Forward: 2024-2030 The HAICu WP2 project oers a new avenues of digital heritage research, demonstrating how continual learning and human-machine collaboration can unlock complex historical sources. Over the coming years, we will refine these methods and expand their application across diverse archival contexts.