ORKGEx v2.0: Automated Multimodal Knowledge Extraction for Scalable Scientific Knowledge Graph Construction

AIKG-SD 2025 Summer School co-located with the NFDI4DS Conference 2025 November 25-26, 2025, Berlin, Germany 1 ORKGEx v2.0: Automated Multimodal Knowledge Extraction for Scalable Scientific Knowledge Graph Construction Hassan Hussein, Allard Oelen, and Sören Auer TIB Leibniz Information Centre for Science and Technology, Hannover, Germany Abstract The exponential growth of scientific literature has created an unprecedented challenge: traditional manual annotation and knowledge extraction processes have become fundamentally inadequate for meaningful scientific discovery. Current approaches to literature review and knowledge synthesis create significant bottlenecks in scientific progress and lead to knowledge fragmentation across disciplines. This work presents ORKGEx v2.0, a revolutionary Chrome extension that addresses critical limitations of the original system through fundamental architectural improvements. Building upon our award-winning WEB 2025 conference paper[1], this extended version introduces a guided five-step annotation workflow that systematically walks researchers through metadata extraction, research field identification, research problem discovery, template selection, and knowledge graph integration—significantly reducing cognitive load and annotation errors that plagued earlier version. Our system introduces a sophisticated multimodal approach with human-in-the-loop integration that processes textual content, images, tables, and figures using GPT-4 Vision. The human researcher remains central to the annotation process, providing domain expertise and validation, while the AI handles computationally intensive tasks such as content analysis and property suggestion. This collaborative approach ensures high-quality annotations while dramatically reducing manual effort, with the system learning from user feedback to improve subsequent suggestions and maintain scientific accuracy. The core innovation lies in our dualpronged approach combining Retrieval-Augmented Generation (RAG) architecture with advanced semantic embedding. The RAG system feeds contextual paper content to large language models for intelligent property suggestions, while our embedding-based similarity mechanism identifies semantically related research problems and properties across the Open Research Knowledge Graph (ORKG)— significantly reducing annotation redundancy and enabling researchers to discover hidden connections across disciplines. The multimodal capabilities extend beyond traditional text processing to include automatic extraction of structured knowledge from research figures, data visualizations, and tabular content, addressing a critical gap where valuable information embedded in visual elements typically remains inaccessible to automated processing systems. This work represents a significant advancement toward AI-assisted scientific knowledge construction, demonstrating how intelligent systems can augment human expertise rather than replace it. By addressing the fundamental challenges of scalability, accuracy, and user experience in scientific annotation. Our system provides a practical solution with the potential to accelerate scientific discovery and enhance the quality of research knowledge graphs. The seamless integration of multimodal AI processing, semantic similarity detection, and guided user workflows establishes a new paradigm for efficiently transforming unstructured scientific literature into actionable, interconnected knowledge. AIKG-SD 2025 Summer School co-located with the NFDI4DS Conference 2025 November 25-26, 2025, Berlin, Germany 2 References 1. H. Hussein, F. Ahmed, A. Oelen, R. Ewerth, and S. Auer, "ORKGEx: Leveraging Language and Vision Models with Knowledge Graphs for Research Contribution Annotation," presented at the IARIA WEB Conference, Lisbon, Portugal, Mar. 2025.