Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? (slides)

Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Jenifer Tabita Ciuciu-Kiss Daniel Garijo Ontology Engineering Group, Universidad Politécnica de Madrid, Spain jenifer[email protected] [email protected] Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Background Scientific Knowledge Graphs (SKGs) have been developed to organize research publications and their knowledge. General SKGs Domain specific SKGs … … Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Problem - Annotations of the same artifacts are inconsistent ●Title: “Self-Refinement of Language Models from External Proxy Metrics Feedback” ○OpenAlex ■Machine Learning ■Natural Language Processing Techniques ■Computer Sciences ○Papers with Code ■Question Answering ■Response Generation ○OpenAIRE ■Computer Science – Artificial Intelligence ■Computer Science – Computation and Language ■Computer Science – Machine Learning ○ORKG ■Self-refinement methods in LLMs ■MultiDoc2Dial / QuAC (Evaluation Context) Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Problem - Annotations of the same artifacts are inconsistent ●Title: “Self-Refinement of Language Models from External Proxy Metrics Feedback” ○OpenAlex ■Machine Learning ■Natural Language Processing Techniques ■Computer Sciences ○Papers with Code ■Question Answering ■Response Generation ○OpenAIRE ■Computer Science – Artificial Intelligence ■Computer Science – Computation and Language ■Computer Science – Machine Learning ○ORKG ■Self-refinement methods in LLMs ■MultiDoc2Dial / QuAC (Evaluation Context) How big is this issue? Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Research Questions ●RQ1: How do category annotations differ across SKGs? ○Different SKGs use different taxonomies & workflows. ○Labels quality and abstraction level vary widely across SKGs Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Research Questions ●RQ1: How do category annotations differ across SKGs? ○Different SKGs use different taxonomies & workflows. ○Labels quality and abstraction level vary widely across SKGs ●RQ2: How accurate are these annotations compared to a manually curated standard? ○Measures reliability. Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Research Questions ●RQ1: How do category annotations differ across SKGs? ○Different SKGs use different taxonomies & workflows. ○Labels quality and abstraction level vary widely across SKGs ●RQ2: How accurate are these annotations compared to a manually curated standard? ○Measures reliability. ●RQ3: What types of annotation inconsistencies occur most frequently? ○Identify what needs fixing: taxonomy, coverage, noise? ○Granularity of SKG annotations. Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Research Questions ●RQ1: How do category annotations differ across SKGs? ○Different SKGs use different taxonomies & workflows. ○Labels quality and abstraction level vary widely across SKGs ●RQ2: How accurate are these annotations compared to a manually curated standard? ○Measures reliability. ●RQ3: What types of annotation inconsistencies occur most frequently? ○Identify what needs fixing: taxonomy, coverage, noise? ○Granularity of SKG annotations. ●Manually curated dataset of 70 AI-related publications ●Each annotated in 4 SKGs Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Methodology ●Comparative analysis on 2 datasets: initial vs gold-standard ○Initial: Raw annotations collected directly from 4 SKGs. ○Gold standard: Manually validated labels kept only if relevant to the title and abstract. ●Identified inconsistency types: ○Coverage: Some papers received only very broad or incomplete labels. ○Granularity: Labels varied from very general to very specific. ○Label mismatch: Different terms used for the same concept. ○Errors: Redundant, irrelevant, or incorrect labels. Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Discussion - Takeaways ●RQ1: Category annotations across SKGs ○Annotation strategies differ widely in coverage and specificity. ○SKGs differ in how broad vs. specific their annotations are. ●RQ2: Accuracy compared to the gold-standard ○Approaches that annotate more tend to achieve high recall but low precision. ○More selective annotations result in higher precision but lower recall. ○Annotation quality depends on the balance between coverage and correctness. ●RQ3: Types of annotation inconsistencies ○Coverage inconsistency: missed relevant labels or too generic labels. ○Incorrect assignment: unrelated labels to the actual subject of the paper. ○Granularity mismatch: overly broad to very specific labels→difficult comparison. ○Label noise and duplication: redundant, inconsistent, or wrong labels. Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Summary & future work ●Cross-SKG comparison on the annotations of 70 publications. ●Results show structural differences in how categories are defined and applied per SKG. ●Annotation quality is affected by schema and manual vs. automated labelling. ●Level of annotation abstraction is not aligned across SKGs. ●There is a need for common annotation guidelines and shared taxonomies across SKGs. ●Future work: Expand the dataset to other domains and explore alignment strategies to harmonize category schemes. Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Sci-K November 2025 Acknowledgments The authors would like to thank the EVERSE project (GA 101129744) under the European Union’s Horizon Europe Programme (HORIZON-INFRA-2023-EOSC-01-02)