Are Scientific Annotations Consistently Represented across Science Knowledge Graphs?

Are Scientific Annotations Consistently Represented across Science Knowledge Graphs? Jenifer Tabita Ciuciu-Kiss1,Daniel Garijo1 1Universidad Politécnica de Madrid, Boadilla del Monte, Madrid, Spain Abstract Scientific Knowledge Graphs (SKGs) are increasingly used to annotate and interlink research outputs. However, little is known about how consistently they annotate the same publication. This paper presents a comparative analysis of category annotations across four major SKGs (ORKG, OpenAlex, OpenAIRE, and Papers with Code) using a manually curated gold-standard dataset of 70 AI-related papers. We examine differences in annotation coverage, granularity, and semantic alignment, highlighting frequent inconsistencies such as label mismatches, overly generic terms, and coverage gaps. Our analysis reveals that manual curation offers high-quality but sparse annotations, while automated systems achieve broader coverage at the cost of precision. This work contributes insights into the reliability of SKG metadata and outlines pathways for improving interoperability and annotation practices. Keywords Science Knowledge Graphs, Comparative Analysis, Metadata Quality 1. Introduction In recent years, Scientific Knowledge Graphs (SKGs) [ 1 , 2 ] have become essential infrastructures for representing scholarly information [ 3 ] in a machine-readable format [ 4 ]. By linking research entities [ 5 ] such as publications, datasets, software, authors, and their associated annotations, SKGs enable advanced services for scientific discovery [ 6 , 7 ], evaluation, and reuse. While these infrastructures do not fully realize all aspects of the FAIR principles [ 8 , 9 , 10 ], they fall short particularly in interoperability and reusability. Annotation vocabularies are rarely harmonized across platforms, documentation of classification pipelines is often incomplete, and provenance metadata is inconsistently recorded. Nonetheless, they contribute toward findability and partial interoperability through metadata enrichment [ 11 ], standardized identifiers, and the homogenization [ 12 ] of scholarly records. Examples include OpenAlex [ 13 ], OpenAIRE [ 14 , 15 , 16 , 17 ], the Open Research Knowledge Graph (ORKG) [ 18 ], AI-KG [ 19 ], Crossref [ 20 ], and Papers with Code (PwC) 1 among others, each offering its own approach to structuring and classifying research outputs. A core functionality of these graphs is the annotation of research publications [ 21 , 22 ], typically through labels such as subjects [ 16 ], concepts [ 13 ], or tasks [ 18 ] 1 with or without a hierarchy. These annotations are key for enabling semantic search [ 23 ], recommendation systems [ 24 ], benchmarking platforms [ 25 ], and large-scale meta-analyses [ 26 ]. However, SKGs differ substantially in how they generate and apply such labels, ranging from manual curation [ 18 ] 1 to automated topic modeling [ 13 , 18 ], resulting in inconsistent representations of the same scientific work among various sources. While prior studies have explored overlaps between SKGs through quantitative methods [ 27 ], such as measuring lexical similarity between annotations, there remains limited understanding of how SKG category annotations differ in practice when describing the same publication across multiple SKGs. In this paper we explore these differences, which may stem from divergent modeling assumptions, annotation pipelines, and classification goals. We present a comparative analysis of 70 AI-research 5th International Workshop on Scientific Knowledge: Representation, Discovery, and Assessment, Nov 2024, Nara, Japan Envelope-Openjenifer.ciuciu-[email protected] (J. T. Ciuciu-Kiss); [email protected] (D. Garijo) GLOBEhttps://jeniferciuciukiss.com/ (J. T. Ciuciu-Kiss); https://dgarijo.com/ (D. Garijo) Orcid0000-0002-3170-6730 (J. T. Ciuciu-Kiss); 0000-0003-0454-7145 (D. Garijo) © 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1https://paperswithcode.com/ papers from recent years, annotated across four major SKGs: ORKG, OpenAlex, OpenAIRE, and PwC. We examine the types of inconsistencies that emerge when annotating the same publications, such as mismatches in granularity, terminology, coverage, and discuss their implications for interoperability, metadata quality, and downstream applications. To better understand SKG-based annotations, we formulate the following research questions, each paired with the main contribution that addresses it: •RQ1: How do category annotations differ across SKGs? We construct and release a manually curated dataset of 70 AI-related publications, each annotated across four major SKGs: ORKG, OpenAlex, OpenAIRE, and PwC. These annotations include tasks, methods, subjects, and other topical labels. We use this dataset to compare how SKGs differ in annotation scope, specificity, and structural conventions. Throughout this paper, we refer to such labels as (category) annotations. •RQ2: How accurate are these annotations compared to a manually curated standard? We manually reviewed the title and abstract of each paper to determine whether the SKGassigned annotations accurately reflected the paper’s content. Based on this expert validation, we constructed a gold-standard dataset and evaluated each SKG’s annotations in terms of precision, recall, and F1-score [28]. •RQ3: What types of annotation inconsistencies occur most frequently? We conduct a comparative evaluation across the four SKGs, identifying frequent inconsistencies such as mismatches in granularity, label ambiguity, incomplete coverage, and semantic misalignment. We further analyze these issues through representative examples and quantitative summaries. The remainder of this paper is structured as follows. Section 2reviews related work on SKGbased classification and metadata annotation. Section 3outlines our methodology, including dataset construction, annotation guidelines, and evaluation metrics. Section 4describes the initial and goldstandard datasets used for the comparative analysis. Section 5presents empirical results on annotation coverage, accuracy, and overlap across SKGs. Section 6discusses the findings in light of key annotation challenges and provides representative examples. Finally, Section 7concludes with a summary of insights and recommendations for improving annotation practices in scientific knowledge graphs. 1.1. Background: Scientific Knowledge Graphs (SKGs) SKGs are structured representations of scholarly knowledge [ 29 ] that encode entities (e.g. publications and concepts) and their semantic relationships in a graph-based format. Their primary aim is to support advanced search, integration, and analysis of scientific information by making research outputs machine-interpretable and interlinked. Depending on their design goals, SKGs differ in scope, domain coverage, and update mechanisms, ranging from large-scale, automatically constructed graphs to smaller, community-curated platforms. In the following, we discuss the key characteristics of some of the most widely used SKGs that serve as the foundation for our analysis. 1.1.1. OpenAlex OpenAlex [ 13 ] is an open catalog of scholarly entities that emerged as the successor of the Microsoft Academic Graph [ 30 ]. It compiles metadata on publications, authors, institutions, venues, concepts, and more. OpenAlex applies machine learning (ML) models trained on titles, abstracts, and citation contexts to assign fine-grained topic annotations from a curated ontology of over 60,000 concepts. These topic concepts are assigned probabilistically, with each publication receiving a primary concept and possibly several secondary ones, each associated with confidence scores. The classification pipeline is documented and accessible through the OpenAlex API 2 . As one of the largest open scholarly KGs, OpenAlex 2https://docs.openalex.org/api-entities/topics prioritizes breadth and scalability but, due to its automated nature, it may introduce inconsistencies in category granularity and semantic relevance, particularly across disciplines. 1.1.2. OpenAIRE - Open Access Infrastructure for Research in Europe Open Access Infrastructure for Research in Europe (OpenAIRE) [ 14 , 15 , 16 , 17 ] is a major European Open Science infrastructure designed to foster open scholarship and improve the accessibility and reusability of scientific knowledge. The OpenAIRE Knowledge Graph aggregates metadata from a broad spectrum of sources, including publications, datasets, projects, and research organizations across Europe, thus providing an integrated view of the research landscape. The OpenAIRE APIs 3 offer access to this aggregated metadata, enabling structured queries over scholarly content. In this work, we used the Search API 45 to retrieve category-level annotations for individual publications. OpenAIRE’s metadata model is enriched with subject classifications based on taxonomies such as the OECD Fields of Science (FOS), SCINOBO 6 and others integrated across its pipeline. As a component of the European Open Science Cloud, OpenAIRE benefits from frequent updates and ongoing standardization efforts. It supports the discoverability and interoperability of scientific content through harmonized metadata ingestion from compliant repositories and data providers. 1.1.3. ORKG - The Open Research Knowledge Graph ORKG [ 18 ] 7 offers a semantic infrastructure for representing individual research contributions using RDF and structured templates. Unlike fully automated SKGs, ORKG relies on manual, communitydriven annotations where users describe publications through semantically rich triples that capture the problem, method, and result of a study. Annotations are made using predefined templates that align with scholarly discourse elements, enabling fine-grained semantic modeling of contributions. The system is designed to increase interpretability and transparency of research metadata, supporting both manual entry and semi-automated extraction tools. While its manual approach limits coverage compared to large-scale automated graphs, the semantic depth and precision of ORKG annotations make it especially valuable for comparative analyses. 1.1.4. PwC - Papers with Code PwC 8 is a domain-specific SKG focused on the AI/ML research landscape. It integrates scientific publications, benchmark datasets, evaluation results, and source code into a coherent, task-driven knowledge graph. Each paper is linked to tasks and methods, with annotations derived via a hybrid pipeline combining automated extraction and human curation. PwC sources its papers primarily from arXiv and Crossref, and then connects them to relevant benchmarks and method families, drawing from curated taxonomies that reflect the evolving state of the field. The PwC dataset is regenerated daily, ensuring that new papers and updated annotations are continuously incorporated. Labels are reviewed and maintained by moderators and contributors from the research community, which supports high-quality and fine-grained annotations useful for reproducibility studies and trend analysis. Metadata and category information are available through the PwC platform and associated GitHub repositories. 9 Previous work has examined the consistency and accuracy of existing method quality in PwC, highlighting both the strengths and the limitations in coverage and granularity [31]. 3https://graph.openaire.eu/docs/apis/home/ 4https://graph.openaire.eu/docs/apis/search-api/ 5https://api.openaire.eu/search/publications 6https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10192702/ 7https://www.orkg.org/ 8https://paperswithcode.com/ 9https://github.com/paperswithcode 2. Related Work This section describes the diverse resource categorization techniques employed by different SKGs, highlighting their methodological differences and implications for consistency in annotations. SKGs employ varied annotation strategies to assign semantic categories to scholarly works. These strategies differ in automation level, interpretability, and domain specificity. Below, we categorize these approaches into four main types: rule-based/metadata, NER/linking, topic modeling/classification, and hybrid/manual curation. 2.1. Metadataand taxonomy-based classification Platforms like OpenAIRE [ 16 , 17 ] and Crossref [ 20 ] rely on repository metadata and established taxonomies (e.g., OECD Fields of Science [ 32 ], SCINOBO [ 33 ], ACM CSS [ 34 , 35 ]) to assign broad subject categories to publications [ 14 , 15 ]. This strategy enables scalable and harmonized annotation of research outputs using well-established classification schemes, contributing to metadata interoperability and integration across repositories. However, it typically produces coarse-grained labels that may lack domain specificity. Notably, many SKGs using metadata-based strategies provide limited documentation about how repositories map local tags to global taxonomies, which introduces opacity into the categorization pipeline. 2.2. Named-entity recognition (NER) and entity linking Some SKGs extract entities directly from unstructured text using NER and linking to external knowledge bases (e.g., Wikidata [ 36 ], MeSH [ 37 ] 10 ) [ 38 ]. This approach enables direct annotation of domain-specific entities (e.g., genes, diseases, methods), which is particularly useful in specialized fields like biomedicine, for instance, biomedical SKGs often detect gene, disease, or method mentions via concept recognition tools, followed by normalization to identifiers. These annotations can enhance semantic granularity and support knowledge integration. However, such pipelines are unevenly documented, with some SKGs omitting details about their training data and linking heuristics [ 19 , 39 ]. This opacity complicates reproducibility and comparison across systems. 2.3. Topic modeling and supervised classification Large-scale SKGs such as OpenAlex apply ML to cluster publications and assign topics based on features extracted from titles, abstracts, venue names, and citation networks. These annotations can enhance semantic granularity and support knowledge integration, for example, OpenAlex’s topic pipeline uses network clustering to define topic communities, labels them via LLMs, and employs a deep learning classifier to annotate works 11 . Such approaches can surface emerging topics and latent structure in scientific literature. However, they may result in inconsistent granularity, with some topics being overly broad and others overly specific. 2.4. Hybrid human-in-the-loop annotation Domain-specific SKGs, such as PwC, combine automated matching of papers to predefined task/method taxonomies with manual curation by community moderators1. This hybrid approach leverages the scalability of automation while incorporating expert validation to improve annotation accuracy. Such strategies strike a balance between breadth and quality: automated systems offer scalability, while manual review ensures semantic precision. Similarly, ORKG uses user-defined templates (e.g., method, result, etc.), filled manually, to produce highly structured and semantically rich metadata 12 . These annotations provide detailed and interpretable representations of research contributions. This yields precise 10https://www.nlm.nih.gov/mesh/meshhome.html 11https://docs.openalex.org/api-entities/topics 12https://orkg.org/stats results, but coverage remains limited, and the user-dependent process is not uniformly documented across contributions. Moreover, manual curation may introduce subjective bias, as moderators may apply labels based on individual interpretations, experience, or familiarity with specific research areas. Implications of strategy diversity on annotation consistency The diversity of annotation strategies across SKGs introduces both strengths and challenges for metadata consistency. Metadata-based systems produce generalized labels with limited thematic depth; topic modeling yields probabilistic but uneven concept assignments, and NER/linking systems vary based on entity recognition quality and KB integration. Hybrid manual approaches deliver semantically rich labels but lack scalability and full traceability, particularly when documentation of contributor workflows is absent. Understanding the trade-offs of each strategy is essential for improving interoperability and annotation quality. Our work contributes to this area by performing a fine-grained, paper-level comparison across SKGs, focusing on semantic overlap, divergence, and contextual usage of categories. 3. Methodology To assess the consistency and semantic validity of annotations across SKGs, we designed a two-stage evaluation pipeline: (1) data collection and preparation, and (2) comparative analysis based on manual validation of annotation correctness. All annotations were verified by a domain expert against the content of each publication by manually inspecting titles and abstracts. To mitigate potential bias in manual validation, we followed predefined rules (see Section 3.1), avoided adding new annotations, and applied a conservative exclusion strategy. Future work will incorporate multiple annotators and inter-annotator agreement to strengthen the reproducibility of our results. 3.1. Data Collection In the first stage of our methodology, we assembled a dataset of AI-related research papers that are jointly annotated across four prominent Scientific Knowledge Graphs (SKGs): ORKG, OpenAlex, OpenAIRE, PwC. These SKGs were selected to represent a spectrum of annotation strategies: from fully manual (ORKG), to hybrid human-in-the-loop (PwC), to fully automated systems (OpenAlex and OpenAIRE). This diversity allowed us to conduct a balanced comparison of annotation behavior across different design paradigms. Each SKG defines its own scope and indexing strategy, focusing on different disciplines, sources, or publication types, which naturally leads to variations in which papers are included and how they are annotated. As a result, it is common for a paper to appear in one SKG but not another, or to be indexed without any associated annotations. This diversity in coverage is expected and reflects the design priorities of each graph rather than inconsistencies. To enable a controlled comparative analysis, we selected only papers that were indexed and annotated by all four SKGs. Although the SKGs use distinct terminology for annotations, such as “tasks” and “methods” (PwC), “research problems” (ORKG), or broader “subjects” (OpenAIRE, OpenAlex)—we refer to all such labels uniformly as annotations throughout this paper.. Constructing a dataset with full parallel annotations across multiple SKGs required scanning a large candidate pool of papers and applying an iterative filtering process. Only those papers for which each SKG provided at least one annotation were retained for the final analysis. The resulting dataset and its properties are described in detail in Section 4. Paper Selection To build the dataset, we began by compiling a broad pool of AI-related research papers published between 2023 and 2025. Candidate papers were selected based on the authors’ domain expertise and covered a wide range of topics within Artificial Intelligence. For each paper, we attempted to match entries across the four selected SKGs using persistent identifiers (primarily DOIs) and title matching. Because each SKG has a different scope and indexing strategy, many papers were not fully covered in all four. Some were absent from one or more SKGs, while others were indexed but lacked relevant annotations. To ensure a fair and controlled comparison, we retained only papers that (1) were indexed in all four SKGs and (2) had at least one annotation from each. This filtering process was applied iteratively to an initial, approximately 200 papers, resulting in a final set of 70 papers that satisfied the completeness criteria for comparative analysis. Categorization Retrieval Once the final set of papers was selected, we retrieved their corresponding annotations from each of the four SKGs. The retrieval process was adapted to the access mechanisms and data availability of each source. Specifically: • For OpenAlex, we used the official API 13 to retrieve topic and concept annotations associated with each paper’s DOI. • For OpenAIRE, we queried the Search API 14 to obtain subject classifications based on the OECD Fields of Science taxonomy. • For ORKG, we used the public SPARQL endpoint 15 to extract structured annotations based on predefined semantic templates. In particular, we retrieved the hasResearchProblem and hasMethod fields from each research contribution. • For PwC, we used a local data dump 16 accessed on July 1, 2025, to extract annotations for tasks and methods associated with each paper. Annotations were collected and stored separately for each SKG, preserving their original format, structure, and terminology. No filtering or transformation was applied during this stage, to ensure that the data remained faithful to its source. This raw annotation set served as the input for the normalization and comparative analysis steps described in the following sections. Initial Dataset: Normalization The initial dataset was constructed directly from the raw annotations retrieved from each SKG. To ensure basic consistency across sources, all annotation labels were transformed to lowercase. In addition, whenever annotations were expressed using OECD Fields of Science (FoS) codes or non-standard descriptors, these were replaced with their corresponding standard FoS labels. No further transformations, filtering, or reformatting were applied at this stage. This version of the dataset preserves the original annotation behavior of each SKG and is referred to as the initial dataset in the rest of the paper. Gold-Standard Dataset: Manual Validation To assess annotation correctness, we manually curated a gold-standard by reviewing each paper’s title and abstract. Validation was performed by the first author (a PhD researcher specializing in AI and SKGs with over 5 years of experience). Each annotation was evaluated against three criteria: (i) semantic relevance to the paper’s research problem or method, (ii) domain specificity (avoiding overly generic categories such as ‘science’), and (iii) contextual accuracy. Borderline cases were conservatively excluded. This rule-based approach was adopted to minimize subjective bias. Annotations that were relevant were marked as correct and therefore kept in the final gold-standard dataset, while those that were off-topic, overly generic, overly specific, or misleading were marked as incorrect. In borderline cases, we adopted a conservative approach and excluded such annotations from the gold-standard. During this process, we also identified cases where the abstracts retrieved from certain SKGs were incomplete, incorrect, or contained metadata artifacts. These cases were corrected manually using the official abstracts from publisher websites or arXiv [ 40 ] to ensure that our validation was based on accurate representations of the paper content. 13https://api.openalex.org/works 14https://api.openaire.eu/search/publications 15https://orkg.org/sparql 16https://paperswithcode.com/about No new annotations were introduced during this step; the gold-standard only reflects corrections to existing labels and underlying metadata. This version of the dataset [ 41 ] is used in our evaluation of annotation accuracy in Section 5. 3.2. Comparative Analysis To evaluate the consistency and semantic appropriateness of category annotations across SKGs, we conducted a comparative analysis that combines quantitative metrics with qualitative interpretation. This twofold approach allowed us to assess both the overall annotation performance of each SKG and the nature of discrepancies that emerge when multiple SKGs describe the same publication. Using the gold-standard dataset, we first evaluated annotation correctness in terms of precision, recall, and F1-score for each SKG. These metrics quantify the alignment between the annotations and expert-validated labels, forming the basis of the results presented in Section 5. To complement the evaluation, we then analyzed the types of inconsistencies that commonly arise across SKGs. For this purpose, we read and interpreted the title and abstract of each paper to identify recurring annotation issues and anomalies. In particular, we distinguish four main types of inconsistencies: • Coverage inconsistency: Cases where a paper was present in all four SKGs but one or more SKGs provided no annotation. Importantly, no new categories were introduced during our process; coverage was judged solely based on whether each SKG offered at least one valid label. • Label mismatch: Use of different terms to describe the same concept (e.g., “NER” vs. “Named Entity Recognition”), reflecting differences in vocabulary and annotation conventions. • Granularity difference: One SKG uses broad categories (e.g., “Computer Vision”) while another applies fine-grained concepts (e.g., “Panoptic Segmentation”), complicating direct comparison. • Incorrect category assignment: A category is clearly misaligned with the content of the paper—such as labeling an NLP paper as “Computer Vision”—often due to automatic inference errors or misinterpreted metadata. Each inconsistency was documented at the paper level, and summary statistics were compiled to capture their distribution across the dataset. Representative examples and edge cases are discussed in Section 6to illustrate common pitfalls, semantic drift, and limitations in current SKG annotation practices. All code used for dataset construction and analysis is available on GitHub [ 42 ] and also published as a snapshot on Zenodo [41]. 4. Dataset The two datasets used in our analysis were constructed following the methodology described in Section 3. Both the initial dataset and the manually curated gold-standard dataset are publicly available on Zenodo [ 41 ]. Both datasets include the exact same set of 70 AI-related research papers from 2023–2025, each annotated by all four SKGs. Table 1presents key statistics for the initial dataset and the manually curated gold-standard dataset. Table 1 Summary statistics for the initial and gold-standard datasets. Metric Initial Dataset Gold-Standard Dataset Total annotations (across SKGs) 2756 1046 Average annotations per paper 39.37 14.94 Average annotations per paper per SKG 9.84 3.78 Unique category labels 728 300 Initial dataset reflects the raw annotations retrieved from each SKG, with only minimal normalization applied, such as converting to lowercase and replacing classification codes when necessary. This version of the dataset contains a total of 2,756 annotations, which corresponds to an average of 39.37 annotations per paper and 9.84 annotations per paper per SKG. The dataset includes 728 unique category labels, illustrating the broad topical coverage and terminological diversity across SKGs. However, this volume also introduced substantial noise, redundancy, and inconsistency, particularly in cases where overly generic or highly specific terms inflated the annotation count. The Gold-standard dataset builds on the initial version by incorporating manual validation of each annotation. Using the title and abstract of each paper, we assessed whether the assigned categories accurately reflected the main research topic or contribution. Annotations deemed off-topic, overly broad, overly specific, or ambiguous were removed. In a few cases, missing/incorrect abstracts were also corrected manually. This refinement reduced the dataset to 1,046 total annotations—an average of 14.94 annotations per paper and 3.78 per paper per SKG. The number of unique category labels decreased to 300, resulting in a cleaner and more semantically coherent label set suitable for evaluation purposes. The contrast between the two datasets highlights the tendency of automated and hybrid SKG pipelines to overgenerate annotations. While the initial dataset captures the full breadth of current SKG outputs, the gold-standard version provides a human-validated benchmark that filters out noise and prioritizes interpretability. 5. Results This section presents the results of our analysis, structured around the research questions (RQs) introduced in Section 1. Each subsection restates the corresponding RQ and provides a detailed response based on comparative findings on the proposed gold-standard dataset consisting of 70 AI-related research papers, annotated across the four SKGs. 5.1. RQ1: How do annotation strategies differ across SKGs? To examine how annotation practices vary across SKGs, we analyzed the average number of annotations per paper and the number of unique category labels for each graph, both before and after gold-standard curation. Table 2summarizes these results. The initial dataset reveals significant differences in annotation strategies across SKGs. PwC assigns the highest number of categories per paper (16.73 on average), followed by OpenAlex (12.39), OpenAIRE (7.43), and ORKG (2.83). While OpenAlex exhibits the broadest vocabulary with 277 unique categories, ORKG, despite its lower per-paper average—maintains 133 distinct labels, pointing to a focused yet diverse annotation strategy. After manual curation, annotation counts dropped significantly across all SKGs, ranging from 4.66 (PwC) to 1.93 (ORKG), reflecting reductions of over 50% in all cases. Overall, the results confirm that SKGs differ widely in both the volume and nature of their annotations. Automated systems like OpenAlex and OpenAIRE aim for broad coverage, but differ in granularity and topical precision. OpenAlex produces more annotations and a broader vocabulary, while OpenAIRE remains more conservative. PwC, despite being hybrid, assigns the highest number of categories per paper, suggesting a bias toward exhaustive labeling that introduces redundancy. In contrast, ORKG assigns far fewer annotations on average, but maintains a high number of distinct category labels, indicating a more targeted and semantically diverse annotation strategy. To further explore how SKGs align in their annotation choices, we computed the number of overlapping categories assigned per paper for each pair and triplet of SKGs. Table 3summarizes the total number of shared annotations observed across 70 papers. Overlaps were relatively sparse, underscoring the inconsistency in how different SKGs annotate the same paper. The highest pairwise agreement occurred between OpenAlex and OpenAIRE (71 overlapping categories), likely due to their shared reliance on automated subject classification at a broad scope. In contrast, overlap between ORKG and OpenAIRE was negligible (1 overlap), reflecting their distinct coverage and semantic focus. Notably, there were zero papers where all four SKGs assigned at least one identical category, and only two triple Table 2 Annotation statistics per SKG in the initial and gold-standard datasets. Metric PwC OpenAlex OpenAIRE ORKG Initial Dataset Avg. categories per paper 16.73 12.39 7.43 2.83 # Unique categories 198 277 157 133 Gold-Standard Dataset Avg. categories per paper 4.66 4.84 3.67 1.93 # Unique categories 119 96 38 75 combinations (PwC–OpenAlex–ORKG and OpenAlex–OpenAIRE–ORKG) resulted in even a single shared category across 70 papers. These results reinforce the conclusion that while SKGs may annotate the same papers, they do so using divergent taxonomies and strategies, limiting interoperability and semantic alignment. Table 3 Category overlaps across SKGs in the gold-standard dataset. Values represent total number of overlapping categories across 70 papers. SKG Combination Pairwise Overlap PwC & OpenAlex 18 PwC & OpenAIRE 4 PwC & ORKG 7 OpenAlex & OpenAIRE 71 OpenAlex & ORKG 3 OpenAIRE & ORKG 1 5.2. RQ2: How accurate are the annotations compared to a manually curated gold-standard? To assess annotation correctness, we evaluated how well the labels assigned by each SKG aligned with the manually curated gold-standard. For each of the 70 AI-related papers, the gold-standard contains only those annotations that were present in the original SKG outputs and judged to be semantically correct based on the paper’s title and abstract. No new categories were added—evaluation was performed purely within the set of originally retrieved labels. Table 4 Annotation precision, recall and F1 score per SKG, including total and average annotations per paper in both datasets. Metric PwC OpenAlex OpenAIRE ORKG Initial Dataset Total Annotations 1171 867 520 198 Avg. Annotations per Paper 16.73 12.39 7.43 2.83 Gold-Standard Dataset Total Annotations 317 339 257 133 Avg. Annotations per Paper 4.66 4.84 3.67 1.93 Precision 0.27 0.39 0.35 0.66 Recall 0.99 1.00 0.72 0.98 F1-score 0.42 0.56 0.47 0.79 As shown in Table 4, SKGs vary significantly in annotation quality. PwC and OpenAlex achieve nearly