CAFE: Knowledge graph completion using neighborhood-aware features

Engineering Applications of Artificial Intelligence 103 (2021) 104302 Contents lists available at ScienceDirect Engineering Applications of Artificial Intelligence journal homepage: www.elsevier.com/locate/engappai CAFE: Knowledge graph completion using neighborhood-aware features Agustín Borregoa,∗, Daniel Ayalaa, Inma Hernándeza, Carlos R. Riverob, David Ruiza aUniversity of Seville, E.T.S. Ingeniería Informática, Avda. de la Reina Mercedes s/n, Seville, Spain bRochester Institute of Technology, 92 Lomb Memorial Drive, Rochester, NY, USA ARTICLE INFO Keywords: Knowledge Graphs Knowledge graph completion Link prediction Machine learning ABSTRACT Knowledge Graphs (KGs) currently contain a vast amount of structured information in the form of entities and relations. Because KGs are often constructed automatically by means of information extraction processes, they may miss information that was either not present in the original source or not successfully extracted. As a result, KGs might lack useful and valuable information. Current approaches that aim to complete missing information in KGs have two main drawbacks. First, some have a dependence on embedded representations, which impose a very expensive preprocessing step and need to be recomputed again as the KG grows. Second, others are based on long random paths that may not cover all relevant information, whereas exhaustively analyzing all possible paths between entities is very time-consuming. In this paper, we present an approach to complete KGs based on evaluating candidate triples using a set of neighborhood-based features. Our approach exploits the highly connected nature of KGs by analyzing the entities and relations surrounding any given pair of entities, while avoiding full recomputations as new entities are added. Our results indicate that our proposal is able to identify correct triples with a higher effectiveness than other state-of-the-art approaches, achieving higher average F1 scores in all tested datasets. Therefore, we conclude that the information present in the vicinities of the two entities within a candidate triple can be leveraged to determine whether that triple is missing from the KG or not. 1. Introduction Knowledge Graphs (KGs) are vast repositories of structured information that have gained increasing popularity over the past years (Hogan et al.,2020). Large volumes of information about different domains can be found in some well-known KGs, such as DBpedia (Lehmann et al., 2015), NELL (Mitchell et al.,2018), Freebase (Bollacker et al.,2008) or the Google Knowledge Vault (Dong et al.,2014), and can be used for tasks like question answering (Bordes et al.,2014a;Huang et al., 2019). KGs are most commonly built by extracting non-structured (Dong et al.,2014;Mitchell et al.,2018) or semi-structured (Glass and Gliozzo,2018;Lehmann et al.,2015) information from web sources, though some smaller KGs can be manually curated by domain experts (Miller,1995). When information extraction systems are applied to extract knowledge from online sources, that information is then semantized (Ayala et al.,2019b;Neumaier et al.,2016) and stored in a KG as triples (Schlegel and Freitas,2019), which represent facts in the form of two entities connected by means of a certain relation. Regardless of the specific process by which a KG is constructed, the resulting structure usually lacks a certain amount of information, either because said information was not originally present in the information ∗Corresponding author. E-mail addresses: [email protected] (A. Borrego), [email protected] (D. Ayala), [email protected] (I. Hernández), [email protected] (C.R. Rivero), [email protected] (D. Ruiz). source, or because it was incorrectly extracted or semantized (Bordes and Gabrilovich,2014). Because of this inherent incompleteness, KGs operate under the Open World Assumption, i.e., a piece of information that is not present in a KG is not considered to be incorrect, but rather just unknown (Galárraga et al.,2015). Therefore, it is mandatory to refine KGs after their creation in order to expand the knowledge they contain and to increase the accuracy of their information (Paulheim, 2017). Such is the example of the Google Knowledge Vault (Dong et al., 2014), a KG that is commonly used for question answering, which is under continuous expansion and refinement. Deriving additional knowledge from an existing KG to augment it is a task known as Knowledge Graph completion (Paulheim,2017). In KG completion, the goal is to identify triples that are missing from the KG and have a reasonable chance of being correct. This is usually done by creating, training and applying a prediction model in a task known as triple classification (Borrego et al.,2019;Lin et al.,2018). Clearly, it is desirable for these models to achieve a high precision to ensure that the triples that are considered to be missing from the KG represent correct facts in the real world (Nentwig et al.,2017). KG completion can be approached with two complementary goals in mind: completing information about entity types (Neelakantan and https://doi.org/10.1016/j.engappai.2021.104302 Received 30 July 2020; Received in revised form 23 April 2021; Accepted 11 May 2021 Available online 18 May 2021 0952-1976/©2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). A. Borrego, D. Ayala, I. Hernández et al. Engineering Applications of Artificial Intelligence 103 (2021) 104302 Chang,2015), or completing the relations between entities (Galárraga et al.,2013;Lao and Cohen,2010;Lin et al.,2015;Socher et al.,2013). Furthermore, KG completion techniques can use both external features, i.e., features that depend on external information sources; or internal features, which are computed using only the KG itself (Drumond et al., 2012;Ji et al.,2015). As a motivational example, Fig. 1 presents a KG that contains information about fictional works, actors, writers and characters. This KG is incomplete, as it is missing certain information from the real world. An example of completing relations between entities would be relating Daniel Radcliffe and Harry Potter by means of the relation plays, which represents a correct fact that is not included in this KG. In this work, we focus in KG completion using only internal features, since this line of research has shown promising results while avoiding dependencies to external sources of information (Paulheim,2017). There are several related proposals in the literature that use solely internal features (Galárraga et al.,2015;Gardner and Mitchell,2015; Ji et al.,2015;Lin et al.,2015;Socher et al.,2013;Wang et al.,2014). Unfortunately, they suffer from a number of drawbacks. On one hand, some of them are embedding-based, which imposes a very expensive pre-processing step to map the entities and relations present in the KG to the embedding space. On the other hand, other proposals are based on random walks that are non-deterministic by definition and, as a consequence, they may not cover relevant information in the vicinity of an entity. In this paper, we present CAFE, our approach for KG Completion using neighborhood-aware features, which uses a feature set that leverages information from the neighborhoods of the entities, i.e., other nearby entities and relations. This feature set transforms triples in the KG into feature vectors, which are then used to train neural prediction models. These models help to discern between correct triples that should be added to the KG, and incorrect ones that should be disregarded. The main contributions of our work are as follows: •A set of neighborhood-based features. We propose a set of features that can be applied directly to any KG, without the need for computing embedded representations. This, in addition to the fact that the proposed features do not require any pre-processing of the KG, means that KG completion can be performed in a more agile way. This is especially appealing for ever-growing KGs, since full recomputations are not needed. •Deterministic KG completion. Because the proposed features do not rely on random paths, all the contextual information surrounding any given entity is always taken into account. This is especially useful in dense KGs, in which relevant information may not be covered by random paths due to mere randomness. •A KG completion workflow. We propose a KG completion technique, CAFE, that leverages the previously discussed set of features to perform KG completion with a minimal amount of user intervention. •High effectiveness. Our experimental results show that CAFE is able to predict which relations should be added to a KG with a comparable or higher effectiveness than other state-of-the-art approaches. The rest of this paper is organized as follows: Section 2analyzes the related work in KG completion proposals, Section 3describes our terminology and the set of features that CAFE uses, Section 4presents our proposal for training classification models for triples in a KG, Section 5reports on our experimental results; finally, Section 6presents our conclusions. 2. Related work In the field of KG completion, the existing techniques can be classified into three types: rule-based, embedding-based and path-based proposals. Rule-based proposals rely on discovering logical rules that determine whether a given triple is correct. In this regard, some authors propose using Inductive Logic Programming to find Horn rules that express the relations between entities in a KG, and then applying these rules to produce new explicit knowledge (Galárraga et al.,2013; Kolthoff and Dutta,2015), under the assumption that all triples that match the rules are correct. The performance of the obtained rules is usually measured using metrics such as rule support or confidence, or a variant of these metrics (Galárraga et al.,2015). Proposals that are based on embeddings aim to evaluate possible relations in a KG by learning embedded representations of its entities and relations, either by using them as inputs for a binary classification model (Socher et al.,2013), or by performing different transformations in an embedding space (Kazemi and Poole,2018;Lin et al.,2015;Liu et al.,2017;Sun et al.,2019;Trouillon et al.,2016). The resulting embedding space – or spaces – is subsequently used to evaluate the likelihood of a candidate triple to be correct or incorrect, since entities that are supposed to be related by means of a certain relation are expected to be close to each other in the embedding space. A similar line of work includes representing a KG as a tensor, and then factorizing it to obtain latent, more compact representations of the triples contained within it (Drumond et al.,2012;Nickel et al.,2012). These proposals suffer from a performance drawback: due to the way in which the embedded representations are obtained, they need to be recomputed whenever new triples are added to the KG, which is a relatively frequent event (Dong et al.,2014;Mitchell et al.,2018). Generally, embedding-based proposals rank triples in order of decreasing likelihood, according to how close the two entities in the triples are in the generated space. Thus, they are commonly evaluated using rankingbased metrics, such as MRR or Hits@N (Wang et al.,2019); however, a likelihood threshold can be set to compute traditional classification measures such as precision, recall, or F1 (Han et al.,2018). Finally, path-based techniques exploit the highly relational nature of KGs to learn how to predict new relations between entities. Our approach, CAFE, belongs to this category. In this line of work, Lao and Cohen (2010) introduced the Path Ranking Algorithm (PRA), a two-step process to find which paths may be useful to predict a certain relation. An evolution of PRA named Subgraph Feature Extraction (SFE) is proposed by Gardner and Mitchell (2015). SFE achieves better performance than PRA and produces more expressive results. It also requires the creation of a handmade ‘‘Alias’’ relation, which relates entities in the same KG that refer to the same element in the real world. Mazumder and Liu (2017) propose a random walk-based approach using neighborhood-guided path finding, where semantic similarities between entities are computed by applying a Word2vec- based embedding model on the names of the entities. Reinforcement learning has also been used to find valuable paths that can help to successfully complete a KG (Xiong et al.,2017). Unfortunately, due to the non-deterministic way in which these paths are computed, they may miss relevant information by mere chance. As of recent, there also exist a number of KG completion proposals that combine the previous approaches. For example, Shen et al. (2019) propose computing embeddings of the entities and relations, and then combining these embeddings in the forms of paths. This can be helpful for determining whether a certain triple is correct. A different way of obtaining embedded representations of the elements in a KG is presented by Wang et al. (2019), by using graph neural networks to capture information pertaining to the structure of the graph. Entity embeddings have also been used to aid in the production of KG completion rules (Ho et al.,2018). While embedded representations can undoubtedly be helpful, CAFE avoids computing them for the sake of agility of application, and instead relies only on analyzing the structure of the KG as is. 2 A. Borrego, D. Ayala, I. Hernández et al. Engineering Applications of Artificial Intelligence 103 (2021) 104302 Fig. 1. An example of a KG describing works, actors, writers and characters. 3. Neighborhood-aware features In this section, we first introduce some preliminary concepts that are necessary to understand our proposal, such as reachability and neighborhood subgraphs. We then define our set of neighborhood-aware features. 3.1. Preliminaries The following definitions form the core of our proposal. Definition 1 (Triple).Let be a set of entities, and let be a set of relations. We define a triple as a 3-tuple that represents the existence of a relation 𝑟∈between a source entity 𝑠∈and a target entity 𝑡∈. We denote triples as (𝑠, 𝑟, 𝑡). In the sample KG depicted in Fig. 1, a sample triple is (Emma Watson, starred_in, Beauty and the Beast). Definition 2 (Knowledge Graph).Let be a set of entities, let be a set of relations, and let be a set of triples of the form {(𝑠, 𝑟, 𝑡) ∣ 𝑠, 𝑡 ∈ , 𝑟 ∈}. We define a Knowledge Graph as  = (,,). Fig. 1 graphically represents a KG with 13 entities, 7 distinct relations and 19 triples. Definition 3 (Path Between Entities).Let  = (,,)be a Knowledge Graph, and let 𝑠, 𝑡 ∈be two entities in . We define a path 𝑝 between 𝑠and 𝑡as a sequence of triples of the form 𝑝=⟨(𝑒𝑖, 𝑟𝑖, 𝑒𝑖+1)⟩ for 𝑖= 1..𝑛, where 𝑒1=𝑠,𝑒𝑛+1 =𝑡and (𝑒𝑖, 𝑟𝑖, 𝑒𝑖+1) ∈ for 𝑖= 1..𝑛. We define the length of a path as the number of triples it contains, i.e., |𝑝|. We denote a path 𝑝of length 𝑛between 𝑠and 𝑡using the relations 𝑟1…𝑟𝑛as 𝑝𝑎𝑡ℎ(𝑠, 𝑡, 𝑟1, 𝑟2,…, 𝑟𝑛), or 𝑝𝑎𝑡ℎ𝑛(𝑠, 𝑡)for short. We denote the set of all possible distinct paths of the form 𝑝𝑎𝑡ℎ(𝑠, 𝑡, 𝑟1, 𝑟2,…, 𝑟𝑛)as (𝑠, 𝑡, 𝑟1, 𝑟2,…, 𝑟𝑛). In the KG depicted in Fig. 1, an example of a path of length 2 between the entities J.K. Rowling and The Cuckoo’s Calling is ⟨(J.K. Rowling, created, Robin Ellacott),(Robin Ellacott, appears_in, The Cuckoo’s Calling)⟩. Definition 4 (Reachability).Let  = (,,)be a Knowledge Graph, let 𝑠, 𝑡 ∈be two entities in , let 𝑟∈be a relation in , and let 𝑛≥1be a natural number. We define reachability as a predicate that determines whether there exists a path of length 𝑛between 𝑠and 𝑡in  such that the relation 𝑟appears in the last triple of the path, i.e., 𝑅𝑒𝑎𝑐ℎ(,𝑠,𝑡,𝑟,𝑛)⟺∃𝑝𝑎𝑡ℎ𝑛(𝑠, 𝑡) ∧ ∃ 𝑎∈∣𝑙𝑎𝑠𝑡(𝑝𝑎𝑡ℎ𝑛(𝑠, 𝑡)) = (𝑎, 𝑟, 𝑡). We define the set of entities that can be reached from 𝑠through a relation 𝑟at distance 𝑛as the set of entities that match the predicate Reach under such circumstances, i.e., {𝑡∈∣𝑅𝑒𝑎𝑐ℎ(, s, t, r, n)}. We denote the previously defined set as 𝑅𝑒𝑎𝑐ℎ𝑎𝑏𝑙𝑒(𝑠, 𝑟, 𝑛). In the example KG depicted in Fig. 1, 𝑅𝑒𝑎𝑐ℎ𝑎𝑏𝑙𝑒(Hermione Granger, writer, 2) = {J.K. Rowling}. Definition 5 (Neighborhood Subgraph).Let  = (,,)be a Knowledge Graph, let 𝑒∈be an entity in , and let 𝑛≥1be a natural number. We define the neighborhood subgraph of 𝑒of size 𝑛as a Knowledge Graph 𝑛 𝑒= (𝑛 𝑒,,𝑛 𝑒)that contains the triples whose target entities can be reached from 𝑒at a distance of at most 𝑛through any relation, and the entity set that can be derived from such triples, where 𝑛 𝑒= {(𝑠′, 𝑟′, 𝑡′) ∈ ∣𝑅𝑒𝑎𝑐ℎ(, e, 𝑡′,𝑟′, i),i=1..n} and 𝑛 𝑒= ⋃{{𝑠, 𝑡}⊆∣ (𝑠, 𝑟, 𝑡) ∈ 𝑛 𝑒}. Fig. 2 illustrates this definition, with two possible neighborhood subgraphs for the KG shown in Fig. 1. 3.2. Feature set We propose a set of neighborhood-aware features that takes neighborhood subgraphs, reachable entities and paths into account. Due to the large number of possible variations of each feature, we present our feature set in terms of groups of features. Each group can be parameterized to obtain a specific feature, which we call an instance of the feature group. Definition 6 (Feature).Let  = (,,)be a Knowledge Graph. We define a feature 𝑓as a function 𝑓∶→Rthat assigns a real number to a triple. For example, a feature 𝑓may convert a triple into the number of entities in the neighborhood subgraph of size 2 of the source entity, i.e., 𝑓∶ (𝑠, 𝑟, 𝑡)↦|2 𝑠|. Definition 7 (Feature Group).Let  = (,,)be a Knowledge Graph. We define a feature group 𝑓𝑛as a function 𝑓𝑛∶→(→R)that receives a set of parameters and returns a feature. For example, a feature group 𝑓0may return a feature that converts a triple into the number of entities in the neighborhood subgraph of size 𝑛of the source entity, i.e., 𝑓0(𝑛) = 𝑓∶ (𝑠, 𝑟, 𝑡)↦|𝑛 𝑠|, where 𝑛is a parameter of the feature group. Thus, 𝑓0(2) ∶ (𝑠, 𝑟, 𝑡)↦|2 𝑠|, which is the feature shown in the previous example. Consequently, feature groups allow us to represent a set of very similar features in a more compact way, where the only distinction between said features is a given set of parameters. 3 A. Borrego, D. Ayala, I. Hernández et al. Engineering Applications of Artificial Intelligence 103 (2021) 104302 Fig. 2. Two neighborhood subgraphs for the sample KG depicted in Fig. 1. In the following, we present the feature groups that CAFE uses. For the sake of clarity, we illustrate a possible instance of every feature group and its value using the example triple 𝑒𝑥𝑎𝑚𝑝𝑙𝑒 = (Daniel Radcliffe, plays, Harry Potter), the KG shown in Fig. 1 and the neighborhood subgraphs described in Fig. 2. Feature group 𝑓1: Number of entities in the neighborhood subgraph of size 𝑛of the source entity in the triple. Features in this group are computed as: 𝑓1(𝑛)∶(𝑠, 𝑟, 𝑡)↦|𝑛 𝑠| In the example shown in Fig. Fig. 2(a),𝑓1(2) applied to the example triple is |{Daniel Radcliffe, Harry Potter and the Goblet of Fire (movie), Harry Potter and the Prisoner of Azkaban (movie), Harry Potter and the Goblet of Fire (book), Harry Potter and the Prisoner of Azkaban (book)}| = 5. Feature group 𝑓2: Number of entities in the neighborhood subgraph of size 𝑛of the target entity in the triple. Features in this group are computed as: 𝑓2(𝑛)∶(𝑠, 𝑟, 𝑡)↦|𝑛 𝑡| In the example shown in Fig. Fig. 2(b),𝑓2(3) applied to the example triple is |{Harry Potter, J.K. Rowling, Harry Potter and the Goblet of Fire (book), Harry Potter and the Prisoner of Azkaban (book), Robin Ellacott, Hermione Granger}|= 6. Feature group 𝑓3: Degree of n-path centrality of the source entity in the triple. Features in this group are computed as: 𝑓3(𝑛)∶(𝑠, 𝑟, 𝑡)↦|𝑛 𝑠| ||− 1 In the example shown in Fig. Fig. 2(a),𝑓3(1) applied to the example triple is |{Harry Potter and the Goblet of Fire (movie), Harry Potter and the Prisoner of Azkaban (movie)}|∕(13 − 1) = 2∕12 ≈ 0.17. Feature group 𝑓4: Degree of N-path centrality of the target entity in the triple. Features in this group are computed as: 𝑓4(𝑛)∶(𝑠, 𝑟, 𝑡)↦|𝑛 𝑡| ||− 1 In the example shown in Fig. Fig. 2(b),𝑓4(1) applied to the example triple is |{Harry Potter and the Goblet of Fire (book), Harry Potter and the Prisoner of Azkaban (book)}|∕(13 − 1) = 2∕12 ≈ 0.17. Feature group 𝑓5: Number of common entities between the neighborhood subgraph of size 𝑛of the source entity and the neighborhood subgraph of size 𝑚of the target entity in the triple. Features in this group are computed as: 𝑓5(𝑛, 𝑚)∶(𝑠, 𝑟, 𝑡)↦|𝑛 𝑠∩𝑚 𝑡| In the example shown in Fig. Fig. 2,𝑓5(2, 3) applied to the example triple is |{Harry Potter and the Goblet of Fire (book),Harry Potter and the Prisoner of Azkaban (book)}|= 2. Feature group 𝑓6: Jaccard index of similarity between the entities in the neighborhood subgraph of size 𝑛of the source entity and the neighborhood subgraph of size 𝑚of the target entity in the triple. Features in this group are computed as: 𝑓6(𝑛, 𝑚)∶(𝑠, 𝑟, 𝑡)↦𝑗𝑎𝑐𝑐𝑎𝑟𝑑(𝑛 𝑠,𝑚 𝑡) In the example shown in Fig. Fig. 2,𝑓6(2, 3) applied to the example triple is |{Harry Potter and the Goblet of Fire (book), Harry Potter and the Prisoner of Azkaban (book)}|∕|{Daniel Radcliffe, Harry Potter and the Goblet of Fire (movie), Harry Potter and the Prisoner of Azkaban (movie), Harry Potter and the Goblet of Fire (book), Harry Potter and the Prisoner of Azkaban (book), Harry Potter, J.K. Rowling, Robin Ellacott, Hermione Granger}|= 2 ∕ 9 = 0.22. Feature group 𝑓7: Adamic–adar index of closeness between the neighborhood subgraphs of size 𝑛of the source and target entities in the triple. Features in this group are computed as: 𝑓7(𝑛)∶(𝑠, 𝑟, 𝑡)↦∑ 𝑒∈𝑛 𝑠∩𝑛 𝑡 1 𝑙𝑜𝑔|𝑛 𝑒| In the example shown in Fig. Fig. 2,𝑓7(2) applied to the example triple is 1 𝑙𝑜𝑔|2 𝐻𝑎𝑟𝑟𝑦𝑃 𝑜𝑡𝑡𝑒𝑟𝑎𝑛𝑑𝑡ℎ𝑒𝐺𝑜𝑏𝑙𝑒𝑡𝑜𝑓𝐹 𝑖𝑟𝑒(𝑏𝑜𝑜𝑘)|=1 𝑙𝑜𝑔|3|≈ 2.09. Feature group 𝑓8: Number of reachable entities through the relation 𝑟at distance 𝑛from the source entity in the triple. Features in this group are computed as: 𝑓8(𝑟, 𝑛)∶(𝑠, 𝑟, 𝑡)↦|𝑅𝑒𝑎𝑐ℎ𝑎𝑏𝑙𝑒(𝑠, 𝑟, 𝑛)| In the example shown in Fig. Fig. 2(a),𝑓8(hasPrequel, 2) applied to the example triple is |{Daniel Radcliffe, Harry Potter and the Prisoner of Azkaban (movie)}|= 2. Feature group 𝑓9: Number of reachable entities through the relation 𝑟at distance 𝑛from the target entity in the triple. Features in this group are computed as: 𝑓9(𝑟, 𝑛)∶(𝑠, 𝑟, 𝑡)↦|𝑅𝑒𝑎𝑐ℎ𝑎𝑏𝑙𝑒(𝑡, 𝑟, 𝑛)| In the example shown in Fig. Fig. 2(b),𝑓9(created, 2) applied to the example triple is |{Harry Potter, Hermione Granger, Robin Ellacott}|= 3. 4 A. Borrego, D. Ayala, I. Hernández et al. Engineering Applications of Artificial Intelligence 103 (2021) 104302 Feature group 𝑓10: Number of common reachable entities through the relation 𝑟from the source entity at distance 𝑛and from the target entity at distance 𝑚. Features in this group are computed as: 𝑓10(𝑟, 𝑛, 𝑚)∶(𝑠, 𝑟, 𝑡)↦|𝑅𝑒𝑎𝑐ℎ𝑎𝑏𝑙𝑒(𝑠, 𝑟, 𝑛) ∩ 𝑅𝑒𝑎𝑐ℎ𝑎𝑏𝑙𝑒(𝑡, 𝑟, 𝑚)| In the example shown in Fig. Fig. 2,𝑓10(created, 2, 3) applied to the example triple is |{Daniel Radcliffe} ∩ {Harry Potter, Hermione Granger, Robin Ellacott}|= 0. Feature group 𝑓11: Jaccard index of similarity between the reachable entities through the relation 𝑟from the source entity at distance 𝑛, and those reachable through the relation 𝑟from the target entity at distance 𝑚. Features in this group are computed as: 𝑓11(𝑟, 𝑛, 𝑚)∶(𝑠, 𝑟, 𝑡)↦𝑗𝑎𝑐𝑐𝑎𝑟𝑑(𝑅𝑒𝑎𝑐ℎ𝑎𝑏𝑙𝑒(𝑠, 𝑟, 𝑛), 𝑅𝑒𝑎𝑐ℎ𝑎𝑏𝑙𝑒(𝑡, 𝑟, 𝑚)) In the example shown in Fig. Fig. 2,𝑓11(created, 2, 3) applied to the example triple is |∅|∕|{Harry Potter, Hermione Granger, Robin Ellacott}|= 0 ∕ 3 = 0. Feature group 𝑓12: Number of distinct paths of length 𝑛between the source and the target entity in the triple, using relations 𝑟1,…, 𝑟𝑛. Features in this group are computed as: 𝑓12(𝑛, 𝑟1,…, 𝑟𝑛)∶(𝑠, 𝑟, 𝑡)↦|(𝑠, 𝑡, 𝑟1,…, 𝑟𝑛)| In the example shown in Fig. Fig. 1,𝑓12(4, starred_in, based_on, writer, created) applied to the example triple is 1, as there is one path of length 4 between the entities Daniel Radcliffe and Harry Potter that matches the given relations. 3.3. Discussion of the feature set The rationale behind this set of features is manifold. Regarding 𝑓1 and 𝑓2, knowing the size of the neighborhood of an entity can be helpful to determine whether said neighborhood encompasses relevant information. For instance, our hypothesis is that very large neighborhoods tend to contain a higher amount of unrelated information. The same idea is leveraged in 𝑓3and 𝑓4, which provide normalized indices of centrality with respect to the total amount of entities in the KG. Following the previous reasoning, we hypothesize that entities with large indices of centrality (i.e. highly connected to other entities) will yield less useful information. Meanwhile, feature groups 𝑓5,𝑓6and 𝑓7measure the degree of overlap that exists in the neighborhoods of the two entities, both in absolute and in relative terms, under the assumption that correct triples have a higher degree of overlap between the neighborhoods of their entities. The previously discussed feature groups do not take into account the specific relations involved. However, we deem it reasonable to assume that some relations may be more useful than others to determine whether a triple is correct, depending on their specific semantics. For example, having one or more children in common can be an indication of a marriage, while having the same nationality is not. In order to exploit this fine-grained information, feature groups 𝑓8,𝑓9,𝑓10 and 𝑓11 are similar to the previously discussed groups but restrict themselves to only one relation. These groups are computed for every relation in a KG. Finally, feature group 𝑓12 allows CAFE to find the number of paths that exist between two entities for any given relations. It is our intuition that correct triples have more alternative paths between the two entities they contain than false ones. 4. Our proposal Our proposal, CAFE, receives a KG and a set of relations from that KG as input, and outputs a classification model for each of the provided relations. These models are able to determine if a given triple that represents an instance of the relation is correct and should belong to the KG. Our workflow is depicted in Fig. 3 and, in the following subsections, we describe each of its steps. 4.1. Loading the KG CAFE internally stores the input KG in the form of (𝑠, 𝑟, 𝑡)triples, using an efficient data structure based on hashable properties, which is suitable for a high frequency of read operations. The triples that contain relations for which a predictive model does not need to be generated are still taken into account when computing features, since they may provide valuable predictive information, but they are not transformed into feature vectors in the following steps. 4.2. Generating negative examples A KG contains only positive information, i.e., it contains examples of the occurrence of a relation 𝑟between two entities. However, it does not contain explicit information about pairs of entities for which 𝑟does not hold. Our proposal relies on a classification model that requires negative examples for training, which means that a number of negatives for each positive triple must be produced. To accomplish this, we follow the type-constrained local closed world assumption (Bansal et al.,2020), i.e., we generate negative examples from every triple (𝑠, 𝑟, 𝑡)present in a KG by replacing their target entity 𝑡with a different one, 𝑡′, such that the resulting triple (𝑠, 𝑟, 𝑡′)does not exist in the KG. Furthermore, to preserve the range of each relation, we randomly choose 𝑡′such that there exists some other triple in the KG where 𝑡′ appears as the target entity for the relation 𝑟. This is known as type constraint. In the example depicted in Fig. 1, a valid negative example is (Hermione Granger, appears_in, The Cuckoo’s Calling), since we know that The Cuckoo’s Calling is a valid target for the relation appears_in. However, (Hermione Granger, appears_in, Daniel Radcliffe) would not be allowed as a negative example, because Daniel Radcliffe never appears as the target of the relation 𝑎𝑝𝑝𝑒𝑎𝑟𝑠_𝑖𝑛. It can be argued that generating negative evidence in this manner can produce false negatives by mere chance, i.e., statements that are deemed incorrect but that are true in the real world. While this is indeed plausible we, as other authors (Ji et al.,2015;Lin et al.,2015; Socher et al.,2013), consider that the chances of this happening are low and, as a consequence, the possible effects on the final results are not significant. 4.3. Converting triples into feature vectors Once negative examples have been generated, our feature set is instantiated and applied to all triples. For all feature groups, we obtain all possible feature instances by applying all possible combinations of the values of their parameters. Each feature instance assigns a real number to each triple. Therefore, applying several features to a triple results in a feature vector. Each position of the feature vector represents that real number that the corresponding feature assigned to the triple. It is important to note that, to compute features on a positive training triple, we temporarily remove it from the KG, since not doing so would result in trivial prediction models such as ‘‘a person plays a character if there exists a triple in the KG stating that the person plays that character’’. 4.4. Grouping feature vectors The previous step computes feature vectors of triples. Since these triples can be either positive or negative, the feature vectors are labeled as positive or negative. Based of the labeled feature vectors, we train a classification model for each relation that predicts whether a triple should be added to the KG. We do this in order to allow the models to capture meaningful and distinctive information for every relation: even though the same set of features is applied to all triples, some features might have more predictive power for a relation, and other features may be more helpful for a different one. 5 A. Borrego, D. Ayala, I. Hernández et al. Engineering Applications of Artificial Intelligence 103 (2021) 104302 Fig. 3. Workflow of CAFE. 4.5. Training and evaluating the models For every relation that we predict, we create one or more neural models, where each model focuses only on the features that are obtained from a certain neighborhood size. Thus, using only neighborhood subgraphs of size 1 results in one model, using neighborhood subgraphs of size of up to 2 results in two models, and so on. This allows each model to capture the specific information that every neighborhood size may yield. To combine two or more models, we use an additional combination layer to produce a single output. The neural models are trained using the labeled feature vectors in the training split for the desired relation, where each model receives only the features corresponding to its assigned neighborhood size, and the label or ground truth is shared among them. Prior to training our models, we first remove any individual features that have the exact same value in every feature vector and thus lack any predictive power. Such a removal helps the model focus only on a subset of potentially more useful features, which has been shown to improve classification results (Karasu et al.,2020). An example of this are path-based features (𝑓12), since only a small subset of all possible paths of fixed length occur between two given entities, and as a consequence most of them have a value of 0. Note that we use neural classification models because they have been shown to consistently achieve satisfactory results in many different classification tasks (Aggarwal and Zhai,2012;Altan and Karasu, 2020;Yadav and Bethard,2019), although other classification models that make use of our features could be used in this step. 5. Evaluation In this section, we present the experimental results that we obtained after evaluating the classification effectiveness of CAFE using several datasets, and we compare it with respect to other state-of-the-art KG completion techniques. In the following, we introduce the datasets that we used for evaluation, our experimental setting and, finally, our results. 5.1. Datasets We evaluated our proposal using four datasets provided by the freely available AYNEC-DataGen (Ayala et al.,2019a) tool: FB13-A, WN11-AR, WN18-AR and NELL-AR. These datasets are based on the well-known FB13, WN11 (Socher et al.,2013), WN18 (Bordes et al., 2014b), and a subset of NELL proposed by Gardner and Mitchell (2015). However, they have been processed to remove reciprocal relations detected by AYNEC, i.e., relations 𝑟and 𝑟′such that, if (𝑠, 𝑟, 𝑡)exists, Table 1 Overview of the datasets used for evaluation. Dataset Training triples Test triples Ents. Rels. FB13-A-10 228,172 481,457 74,998 13 WN11-AR-10 77,948 198,231 38,195 9 WN18-AR-10 71,984 183,051 40,943 11 NELL-AR-10 86,971 (1451) 219,374 (5083) 53,934 148 (9) then (𝑡, 𝑟′, 𝑠)also exists very frequently. Additionally, relations that amount to less than 5% of the total number of triples in the graph have been removed. These datasets originally contained one negative example per each positive triple in both their training and testing splits. In order to study how the KG completion techniques perform when presented with a much higher volume of negative evidence, we created versions of these datasets whose testing splits contained 10 negative examples per positive, using the AYNEC-DataGen tool. We believe that this is a more realistic scenario, since a much higher number of negative examples per positive triple is typically expected in real-world KG completion tasks (Borrego et al.,2019). To avoid confusion, we denote these versions as FB13-A-10, WN11-AR-10, WN18-AR-10 and NELL-AR- 10. For the sake of reproducibility, our source code and datasets are publicly available on GitHub.1 For FB13-A-10, WN11-AR-10 and WN18-AR-10, we aimed to predict all possible relations, and for NELL-AR-10 we focused on the same subset of 10 relations that were used to evaluate SFE (Gardner and Mitchell,2015). However, in the latter dataset, one relation was removed by AYNEC for being the reciprocal of another relation, leaving 9 relations for evaluation. In the specific case of FB13-A-10, we transferred 25% of the training triples over to the testing set in order to provide testing examples for some relations, as they were not available in the original dataset as introduced in Socher et al. (2013). Table 1 provides an overview of the aforementioned datasets. In the case of NELL-AR-10, we show in parentheses the amount of triples and relations that were considered for evaluation, although the entire graph was used for computing features. 5.2. Evaluation method A neural prediction model was created for every relation of interest and trained using its corresponding training set. Then, the model was 1https://github.com/DEAL-US/CAFE. 6 A. Borrego, D. Ayala, I. Hernández et al. Engineering Applications of Artificial Intelligence 103 (2021) 104302 Fig. 4. F1 scores for all relations obtained by CAFE and other state-of-the-art proposals. applied to all feature vectors in the test set, and we compared the expected label (which denotes whether it represents a valid triple or not) against the label that was produced by our model. We report our results in terms of precision, recall and F1, in order to determine how effective our proposal is when determining the correctness of a given triple. We evaluated three versions of CAFE, denoted CAFE1to CAFE3, which were limited to using feature instances that exploited neighborhood subgraphs and paths of a maximum size of 1, 2, and 3, respectively. This was done in order to study how using larger neighborhoods affects the effectiveness of CAFE. There exist many different KG completion proposals and, as shown in Section 2, they often use different evaluation metrics. Due to this, it is very difficult to perform a comparison across a large number of them in a manner that is fair and rigorous. For this reason, we used TransE (Bordes et al.,2013), TransD (Ji et al.,2015), TransH (Wang et al.,2014), TransR (Lin et al.,2015), Analogy (Liu et al.,2017), SimplE (Kazemi and Poole,2018) and RotatE (Sun et al.,2019) as baselines for our evaluation, since they are some of the most wellknown state-of-the-art KG completion proposals. In order to provide a common evaluation environment for these different proposals, we used the OpenKE (Han et al.,2018) framework to train and evaluate these proposals using our datasets. Additionally, since these proposals usually report metrics like MRR and Precision@N, we used the utilities provided by OpenKE to obtain binary labels for the testing triples, by setting a likelihood threshold in a way that optimized the classification results. We selected the following values for the hyperparameters of our neural models: 3 layers with 1024, 512 and 256 neurons each, learning rate of 0.001, batch size of 16, dropout of 0.1 for all layers, 100 epochs and validation ratio of 10%. When two or more models were to be combined, we joined their results using a hidden layer with 3 neurons and an output layer with a single neuron. These values for the hyperparameters were chosen using a hold-out or ‘‘dev’’ set for the FB13-A-10 dataset, and all datasets were then evaluated using the same hyperparameters. We chose them because they provided satisfactory results in our empirical tests. All our experiments were conducted on a computer equipped with an Intel Core i9-9900K CPU, 32 GB of RAM and an Nvidia RTX 2080 Ti GPU. 5.3. Results and discussion In Fig. 4, we show the evaluation results for CAFE1, CAFE2, CAFE3, and the related state-of-the-art proposals. For the sake of clarity, this Figure only displays the F1 values for each technique. Additionally, Table 2 shows the detailed results for all relations in every dataset and for all metrics under evaluation. Our results show that CAFE is able to match the performance of state-of-the-art embedding-based proposals, and in many cases achieve higher values on the metrics under evaluation. In the cases of FB13- A-10 (Fig. 4(a)) and WN18-AR-10 (Fig. 4(d)), CAFE can reach or surpass the F1 scores achieved by other state-of-the-art proposals in a consistent manner. CAFE also provides better results in the WN11- AR-10 (Fig. 4(c)) dataset, although with a higher degree of variability, and matches the performance of the rest of the analyzed techniques in the NELL-AR-10 (Fig. 4(b)) dataset. These results show that CAFE can be more effective than other proposals in challenging classification scenarios. Table 2 displays that, in general, both a satisfactory precision and recall can be achieved, and thus we consider that CAFE is generally effective. However, there exists a number of relations for which a very high precision value is obtained at the expense of a lower recall or viceversa, resulting in a typical precision–recall trade-off. We also observe that the nature of every individual relation has a significant impact in the results, since some of them are harder to predict than others. Such is the case of the relation cause_of_death, since learning to predict the cause of the death of a person with a very high effectiveness would be a remarkable achievement that unfortunately falls out of the scope of this work. We further discuss these limitations in Section 5.4. Regarding the question of how using different neighborhood sizes affects the effectiveness of CAFE, Fig. 4 shows that the metrics under evaluation are generally higher for CAFE2compared to CAFE1, but the same cannot always be said for CAFE3and CAFE2. Indeed, metrics appear to remain stagnant or even decrease when using larger neighborhoods in some cases. For example, in FB13-A-10 (Fig. 4(a)), we do not observe a significant increase in effectiveness when using larger neighborhood subgraphs, which suggests that the most useful information is readily available in the immediate vicinities of the 7 A. Borrego, D. Ayala, I. Hernández et al. Engineering Applications of Artificial Intelligence 103 (2021) 104302 Table 2 Detailed results for all relations. The highest value for a metric in a relation is marked in bold. The highest F1 value for a relation is also highlighted in gray. Fig. 5. F1 scores in the WN11-AR-10 dataset. relevant entities. In the cases of WN11-AR-10 (Fig. 4(c)) and WN18-AR- 10 (Fig. 4(d)), an improvement is observed when using neighborhoods of size 2, but further expanding the neighborhood size does not seem to have a significant impact. A possible conclusion for this is that, at a certain point, larger neighborhood subgraphs do not provide additional value or predictive power over smaller ones. In a worst-case scenario, the number of features with little to no predictive power would greatly increase by using larger neighborhood sizes, negatively affecting the results. This effect actually occurs in the case of the NELL-AR-10 dataset (Fig. 4(b)), where effectiveness decreases when increasing the size of the neighborhood subgraphs for which we compute features. A plausible explanation for this is that it has been noted that NELL is a noisier dataset (Paulheim and Bizer,2014), and thus taking larger neighborhoods into account significantly increases the amount of noise that our classification model has to deal with, reducing its effectiveness. 5.4. Limitations and performance considerations Despite our proposal being generally effective, it has some limitations. CAFE does not work well for relations for which useful predictive information cannot always be found in the neighborhoods of the entities in a triple. In this regard, we identify two types of relations: those that can be predicted using other information present in the KG, and those for which the entities and relations in their neighborhoods do not provide useful information, and thus are much harder to predict. This dichotomy can be observed in the results shown in Table 2, and it is especially notable in the WN11-AR-10 dataset. For the sake of visualization, we display the results of applying CAFE to this dataset in Fig. 5. This difference is particularly visible in the Similar to and Domain topic relations, which have low F1 scores. Upon manual inspection, we found that Similar to tends to link words that are generally isolated within the KG and share very little common context. Also, Domain topic is extremely broad, causing the two entities in a triple to have few or no relevant common elements in their neighborhoods, e.g., (Britain, Domain topic,Surgery). In other cases, even if some information is present in the neighborhoods under consideration, it may be not successfully captured by CAFE due to the large amount of irrelevant data surrounding it. In this regard, the NELL-AR-10 dataset shows that larger neighborhood sizes are not always better for predictive purposes, since the amount of noise they may include can be detrimental for the performance of CAFE. This can be due to some source or target entities being present in many triples (for example, countries). Therefore, larger neighborhoods of these entities introduce many other entities that are not relevant to 8 A. Borrego, D. Ayala, I. Hernández et al. Engineering Applications of Artificial Intelligence 103 (2021) 104302 Table 3 Time in seconds (average ±standard deviation) needed to compute features for all training and testing triples in every dataset. Dataset CAFE1CAFE2CAFE3 FB13-A-10 17.21 ±0.52 67.98 ±2.87 183.38 ±7.71 NELL-AR-10 0.56 ±0.06 2.41 ±0.23 6.95 ±0.62 WN11-AR-10 8.69 ±0.26 50.85 ±0.83 195.50 ±16.77 WN18-AR-10 5.70 ±0.12 18.90 ±0.29 55.57 ±0.91 the triple under evaluation. In these cases, it is up to the users to decide which maximum neighborhood size best caters to their interests. Additionally, due to the fact that feature instances are obtained from feature groups by considering all possible combination of the values of their parameters, as explained in Section 3.2, the number of individual feature instances associated with every neighborhood size has a theoretically exponential growth with respect to the number of relations in the dataset. However, several optimizations can be leveraged to reduce the amount of time needed to compute the required features for all triples in a KG. Given that this computation can be done independently for each triple, our implementation uses multi-threading to parallelize this task, and could theoretically benefit from further performance improvements under a distributed computing architecture. Furthermore, since our features are deterministic and always return the same value for the same input, we use a caching strategy to minimize the amount of feature computations needed. To illustrate this, we display the average CPU times required by CAFE to transform all triples in a dataset into feature vectors, across 10 different executions for all datasets and context sizes, in Table 3. It can be noted that, in all cases, the computation times were quite reasonable, which suggests that our proposal is suitable to be applied in a real-world scenario. 6. Conclusions In this paper, we have introduced CAFE, a proposal that classifies triples for Knowledge Graph completion by using a feature set based upon neighborhoods and neural prediction models. This feature set, and by extension our proposal, is applicable to all KGs, and it is completely deterministic. Additionally, CAFE does not need any manually supplied or external information, nor does it need any previous preprocessing of the KG, and it is able to operate using a KG as its only input. We have performed a number of experiments to evaluate the effectiveness of CAFE in four well-known Knowledge Graphs, identifying candidate triples that should be added to each KG. To provide a more realistic evaluation scenario, we have performed our experiments using testing splits that contain a much lower amount of true triples than false ones. Our results show that applying the proposed feature set yields an effectiveness that is comparable to state-of-the-art techniques, and in many cases even higher. CAFE achieves higher average F1 values in all datasets under study, thus leading to a more trustworthy KG completion process. We also show that using information present in the neighborhoods of the two entities in a triple provides satisfactory results, and thus local information existing around any given entity appears to be of great value. CRediT authorship contribution statement Agustín Borrego: Conceptualization, Methodology, Software, Investigation, Writing - original draft. Daniel Ayala: Conceptualization, Methodology, Software, Investigation, Writing - original draft. Inma Hernández: Conceptualization, Writing - review & editing, Supervision, Funding acquisition, Project administration. Carlos R. Rivero: Conceptualization, Writing - review & editing, Supervision. David Ruiz: Conceptualization, Writing - review & editing, Supervision, Funding acquisition, Project administration. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgment This work was supported by the Spanish Ministry of Science and Innovation under grants TIN2016-75394-R and PID2019-105471RB- I00. References Aggarwal, Charu C., Zhai, ChengXiang, 2012. A survey of text classification algorithms. In: Mining Text Data. Springer, pp. 163–222. http://dx.doi.org/10.1007/978-1- 4614-3223-4_6. Altan, Aytaç, Karasu, Seçkin, 2020. Recognition of COVID-19 disease from X-ray images by hybrid model consisting of 2D curvelet transform, chaotic salp swarm algorithm and deep learning technique. Chaos Solitons Fractals 140, 110071. http://dx.doi.org/10.1016/j.chaos.2020.110071. Ayala, Daniel, Borrego, Agustín, Hernández, Inma, Rivero, Carlos R., Ruiz, David, 2019a. AYNEC: All you need for evaluating completion techniques in knowledge graphs. In: ESWC. pp. 397–411. http://dx.doi.org/10.1007/978-3-030-21348-0_26. Ayala, Daniel, Hernández, Inma, Ruiz, David, Toro, Miguel, 2019b. TAPON: A twophase machine learning approach for semantic labelling. Knowl.-Based Syst. 163, 931–943. http://dx.doi.org/10.1016/j.knosys.2018.10.017. Bansal, Iti, Tiwari, Sudhanshu, Rivero, Carlos R., 2020. The impact of negative triple generation strategies and anomalies on knowledge graph completion. In: CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management. ACM, pp. 45–54. http://dx.doi.org/10.1145/3340531.3412023. Bollacker, Kurt, Evans, Colin, Paritosh, Praveen, Sturge, Tim, Taylor, Jamie, 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In: SIGMOD. ACM, pp. 1247–1250. http://dx.doi.org/10.1145/1376616.1376746. Bordes, Antoine, Chopra, Sumit, Weston, Jason, 2014a. Question answering with subgraph embeddings. In: EMNLP. ACL, pp. 615–620. Bordes, Antoine, Gabrilovich, Evgeniy, 2014. Constructing and mining web-scale knowledge graphs. In: SIGKDD. ACM, p. 1967. http://dx.doi.org/10.1145/2623330. 2630803. Bordes, Antoine, Glorot, Xavier, Weston, Jason, Bengio, Yoshua, 2014b. A semantic matching energy function for learning with multi-relational data - Application to word-sense disambiguation. Mach. Learn. 94 (2), 233–259. http://dx.doi.org/10. 1007/s10994-013-5363-6. Bordes, Antoine, Usunier, Nicolas, García-Durán, Alberto, Weston, Jason, Yakhnenko, Oksana, 2013. Translating embeddings for modeling multi-relational data. In: NIPS. pp. 2787–2795. Borrego, Agustín, Ayala, Daniel, Hernández, Inma, Rivero, Carlos R., Ruiz, David, 2019. Generating rules to filter candidate triples for their correctness checking by knowledge graph completion techniques. In: K-CAP. pp. 115–122. http://dx. doi.org/10.1145/3360901.3364418. Dong, Xin, Gabrilovich, Evgeniy, Heitz, Geremy, Horn, Wilko, Lao, Ni, Murphy, Kevin, Strohmann, Thomas, Sun, Shaohua, Zhang, Wei, 2014. Knowledge vault: A webscale approach to probabilistic knowledge fusion. In: SIGKDD. ACM, pp. 601–610. http://dx.doi.org/10.1145/2623330.2623623. Drumond, Lucas, Rendle, Steffen, Schmidt-Thieme, Lars, 2012. Predicting RDF triples in incomplete knowledge bases with tensor factorization. In: SAC. ACM, pp. 326–331. http://dx.doi.org/10.1145/2245276.2245341. Galárraga, Luis Antonio, Teflioudi, Christina, Hose, Katja, Suchanek, Fabian, 2013. AMIE: Association rule mining under incomplete evidence in ontological knowledge bases. In: WWW. ACM, pp. 413–422. http://dx.doi.org/10.1145/2488388.2488425. Galárraga, Luis, Teflioudi, Christina, Hose, Katja, Suchanek, Fabian M., 2015. Fast rule mining in ontological knowledge bases with AMIE+. VLDB J. 24 (6), 707–730. http://dx.doi.org/10.1007/s00778-015-0394-1. Gardner, Matt, Mitchell, Tom, 2015. Efficient and expressive knowledge base completion using subgraph feature extraction. In: EMNLP. The Association for Computational Linguistics, pp. 1488–1498. Glass, Michael, Gliozzo, Alfio, 2018. A dataset for web-scale knowledge base population. In: ESWC. Springer, pp. 256–271. http://dx.doi.org/10.1007/978-3-319-93417- 4_17. Han, Xu, Cao, Shulin, Lv, Xin, Lin, Yankai, Liu, Zhiyuan, Sun, Maosong, Li, Juanzi, 2018. OpenKE: An open toolkit for knowledge embedding. In: EMNLP. pp. 139–144. Ho, Vinh Thinh, Stepanova, Daria, Gad-Elrab, Mohamed Hassan, Kharlamov, Evgeny, Weikum, Gerhard, 2018. Learning rules from incomplete kgs using embeddings. In: The 17th International Semantic Web Conference. ceur. ws. org. Hogan, Aidan, Blomqvist, Eva, Cochez, Michael, d’Amato, Claudia, de Melo, Gerard, Gutierrez, Claudio, Gayo, José Emilio Labra, Kirrane, Sabrina, Neumaier, Sebastian, Polleres, Axel, Navigli, Roberto, Ngomo, Axel-Cyrille Ngonga, Rashid, Sabbir M., Rula, Anisa, Schmelzeisen, Lukas, Sequeda, Juan F., Staab, Steffen, Zimmermann, Antoine, 2020. Knowledge graphs. CoRR, abs/2003.02320. 9