SpaceRL — A reinforcement learning-based knowledge graph driver

Contents lists available at ScienceDirect SoftwareX journal homepage: www.elsevier.com/locate/softx Original Software Publication SpaceRL — A reinforcement learning-based knowledge graph driver Miguel Bermudo∗, Daniel Ayala, Inma Hernández, David Ruiz, Miguel Toro University of Seville, ETSII, Avda. Reina Mercedes, s/n. Seville, Spain ARTICLE INFO Keywords: Knowledge graphs Reinforcement learning Explainability Link prediction Open source ABSTRACT Knowledge Graphs are powerful data structures used by large IT companies and the scientific community alike. They aid in the representation of related information by means of nodes connected through links indicating types of relations. These graphs are used as the basis for several smart applications, such as question answering or product recommendation. However, they are built in an automated unsupervised way, which leads to gaps in information, usually in the form of missing links between related entities in the original data source, which have to be added later by completion techniques. SpaceRL is an end-to-end Python framework designed for the generation of reinforcement learning (RL) agents, which can be used to complete knowledge graphs through link discovery. The purpose of the generated agents is to help identify missing links in a knowledge graph by finding paths that implicitly connect two nodes, incidentally providing a reasoned explanation for the inferred new link. The generation of such agents is a complex task, even more so for a non-expert user. SpaceRL is meant to overcome these limitations by providing a flexible set of tools designed with a wide variety of customization options, in order to adapt to different users’ needs, while also including a variety of state-of-the-art RL algorithms and several embedding models that can be combined to optimize the agents performance. Furthermore, SpaceRL offers different interfaces to make it available either locally (programmatically or via a GUI), or through an OpenAPI-compliant REST API. Code metadata Current code version 0.5.6 Permanent link to repository used for this code version https://github.com/ElsevierSoftwareX/SOFTX-D-23-00527 Permanent link to Reproducible Capsule n/a Legal Code License MIT License Code versioning system used Git Software code languages, tools, and services used Python, TCL, HTML Compilation requirements, operating environments & dependencies Unix (CUDA 11), Python = 3.8, pip, apt Link to developer documentation/manual https://github.com/DEAL-US/SpaceRL-KG/blob/master/README.MD Support email for questions [email protected] 1. Motivation and significance Knowledge graphs (KGs) are sources of structured information that have proven their worth as data structures for the scientific community [1] and the IT industry alike [2]. They provide efficient and versatile support for advanced tasks such as question answering [3,4] or recommendation systems [5], which is the reason that led large companies such as Amazon, Google, or Meta, among many others, ∗Corresponding author. E-mail addresses: [email protected] (Miguel Bermudo), [email protected] (Daniel Ayala), [email protected] (Inma Hernández), [email protected] (David Ruiz), [email protected] (Miguel Toro). adopt them. Information in KGs is generally expressed as a list of facts, which in turn are represented as subject–predicate–object triples made up of two entities and a relation which connects them [6]. KGs are usually built from unstructured sources by automated unsupervised processes that extract the information and transform it into triples [7,8]. However, even the most comprehensive of these processes are always prone to leaving some facts behind, resulting in partially https://doi.org/10.1016/j.softx.2025.102078 Received 23 October 2023; Received in revised form 27 January 2025; Accepted 28 January 2025 SoftwareX 30 (2025) 102078 Available online 12 February 2025 2352-7110/© 2025 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ ). Miguel Bermudo et al. Table 1 Methodology comparison between SpaceRL and other approaches. Name and computation Overview Datasets Is open source Customizable [17]DeepPath REINFORCE with terminal rewards Supervised policy learning Pre-computed paths Post processing verification FreeBase NELL yes no [18]MINERVA REINFORCE with terminal reward Allows for paths of variable length and stopping conditions with no pre-computing Countries UMLS FreeBase Wordnet NELL yes no [19]Shaping REINFORCE with terminal rewards calculated through NN Random action dropout Soft reward based on embedding models UMLS Kinship FreeBase WordNet NELL no n/a [20]PGPR REINFORCE with ‘‘learned’’ soft-terminal reward Soft reward strategy User-conditional action pruning a multi-hop scoring function specialized in product datasets Amazon (Clothing, Beauty, CD, Cell Phones) yes no [21]DAPath REINFORCE distance based and terminal rewards Self-attention mechanism with GRU to compute a distance reward table NELL FreeBase no n/a [24]SpaceRL* REINFORCE, SAC, PPO (Expandable algorithm support) customizable computation parameters and rewards selectable multi-rewards with customizable layer structure and reward computations. Bundled: WordNet FreeBase NELL Countries UMLS Accepts any. yes yes incomplete KGs [9], which has a negative effect on the performance of the applications that use them. Several software tools have been proposed to deal with the incompleteness in KGs. Most of them do so by leveraging the information in the graph to infer new triples that represent missing information [10]. While several approaches have been made to tackle this problem, including rule-based reasoning [11–13], embedding-based algorithms [14], or relation path reasoning [9,15, 16], reinforcement learning (RL) path finding [17] is gaining traction as the most promising approach to knowledge graph completion. There are some previous proposals in this field, such as DeepPath [17], MINERVA [18], Reward Shaping [19], PGPR [20], or DAPath [21]. However, these approaches are restricted to classical RL algorithms, and overlook the application of more modern RL algorithm such as Proximal Policy Optimization (PPO) [22] or Soft Actor Critic (SAC) [23]. Also, most of these proposals are not distributed as usable tools intended for final users. Even if they make their implementation publicly available, their code is merely intended for the sake of experimental results reproducibility, and they often lack any degree of customization or flexibility, meaning they usually can only work on a number of predefined datasets as input, Table 1showcases the details of these shortcomings. SpaceRL combines the benefits from RL pathfinding with the power of representational embeddings to infer fairly long and explainable paths such as other works that also aim for finding the shortest paths in a graph [25],SpaceRL proves its usefulness for KG-based applications. Our tool is highly configurable, allowing for reward calculation to be modified with a combination of several options, customizing the policy intermediate activation function and regularization, using the more classical approach of the REINFORCE [26] algorithm instead of PPO if required, computing the reward in one of several ways, or selecting the max depth of paths to explore, new reward features can be also easily added to the system to further customize it. SpaceRL also allows the user to apply state-of-the-art RL algorithms out of the box, namely Proximal Policy Optimization (PPO) [22] combined with Soft Actor Critic (SAC) [23], which improve performance and help avoid reward plateaus while training. Finally, SpaceRL, aims to provide a versatile tool intended for users with different levels of expertise; from experts, who may prefer to install SpaceRL as a standalone tool for local usage or to become a service provider for third parties, to novices who would benefit from a simple MLaaS interface. SpaceRL offers RL model generation and usage as a service capabilities, either locally or through its GUI or as a deployable REST API for third party consumption. Therefore, it is, to the best of our knowledge, the first turnkey tool to provide such RL KG completion and reasoning functionalities with such a wide variety of options. The rest of this paper is structured as follows: Section 2describes SpaceRL in detail, its internal architecture, how to operate it and the services it provides. Section 3focuses on analyzing a real-world example application of SpaceRL. Section 4discusses the impact that SpaceRL would have for potential users. Finally, Section 5concludes the article. 2. Software description SpaceRL is an end-to-end Knowledge Graph completion tool, entirely Python-based. It is publicly available at our GitHub page [27], and open to contribution. The application is divided into the following subsystems, as depicted in Diagram 1. 2.1. Configuration The configuration module of SpaceRL is responsible for the management of a key–value map which holds several global tuning parameters. These parameters allow to fine-tune the behavior of the trainer and tester classes, including the activation function and reward options, the number of training episodes, embedding generation options, or neural network settings, amongst others. SoftwareX 30 (2025) 102078 2 Miguel Bermudo et al. Fig. 1. SpaceRL system overview. Fig. 2. SpaceRL environment is based on the KG structure. The current state node ‘‘s’’ is connected through relations to all other nodes which can be chosen as an action. 2.2. Reinforcement learning A Reinforcement Learning environment represents the context in which an agent will act and learn. The environment has astate that can be manipulated by a number of agent operations called steps. However, the agent has only a partial view on the environment in each step, which puts a limit on the actions it can take; this is referred to as the action space available to the agent in the current state (cf. Fig. 2). In this context, we consider that the environment is the entirety of the KG to be completed, the state is a particular node in the KG, and the action space is comprised of every relation that links that node with its adjacent nodes (which may include itself). The RL subsystem is responsible for the generation of the training environment, creating the agent that will train on an input KG with the specified configuration options, and managing the data generated during training and testing. The subsystem is comprised of a number of classes, namely: Environment,Agent, and DataManager. The Environment class was implemented following the OpenAI Gym [28] standard for reinforcement learning tools, recently transferred to the Farama Foundation and its new drop-in replacement Gymnasium [29]. Once created, the Environment instance first invokes the KnowledgeGraph class, initializes the cache, and generates the KG embedding vectors if they are not present. In its current version, SpaceRL provides support for four classical embedding models in the literature, namely, ComplEx [30], DistMult [31], TransE [32], and TransR [33] as implemented by DGL-KE [34]. We chose these particular models as they are the most present in the literature and performed satisfactorily when training the models. SpaceRL also allows for easy expansion as it is a modular tool and embedding generation is a self-contained module. Every training episode begins with a query triple (𝑠, 𝑟, 𝑡), and the node that contains the head entity 𝑠as the initial state of the episode. For each training step, the environment encodes the initial state as a concatenation of embedding vectors, and calculates all possible actions starting from that state. Then, it relies on the Agent to select one of those actions in order to advance to the next state. The process is repeated for each state until the episode is complete and the reward is computed. The Agent class is tasked with building the neural network according to a given specification, as well as selecting the actions on each step according to the action_picking_policy and reward_computation values in the configuration, memorizing them, storing the rewards given by the environment and advancing to the next environment step. Any Agent instance can be configured to use either PPO with an Actor–Critic strategy requiring multiple neural networks, or a classical RL algorithm with a single network. Class Agent intimately interacts with class Environment during the training episodes. For each action generated by the Environment, the Agent calculates the output of the neural network, which represents the score assigned by the agent to that action. Using the scores, the Agent class evaluates the former actions and selects one of them to proceed, which updates and steps the environment into the next state. Finally, the DataManager class is responsible for the storage, modification, and organization of data during the testing, training, and embedding generation processes. 2.3. Core The main subsystem of the application acts as the entry point to run a new training or testing process. Its main components are the Trainer and Tester classes. SoftwareX 30 (2025) 102078 3 Miguel Bermudo et al. Fig. 3. SpaceRL GUI: Visualization — the tool showcases the path the agent took from the source node to the end node and the score the model gave each step as well as come other nodes present adjacent to each path step. The Trainer class instantiates the DataManager,Environment, and Agent classes. Then, the Trainer awaits for the run method to be called, which triggers the training episodes. Each episode starts by obtaining one triple from the Environment, and using RL pathfinding to select the most promising actions that traverse the graph maximizing the episode reward. When the environment activates the done flag, the episode ends, and the total reward per step is computed and passed onto the agent learning function. Finally, the neural network weights are updated based on the taken path and computed rewards. During the testing process, the system iterates through the list of tests in the configuration file (config.py). For each of them, an agent model and Tester instances are loaded for each embedding specified in the test configuration. Then, the Tester executes the specified amount of testing episodes. The textual representations of the resulting paths are stored in text files in order to be accessible to other processes. 2.4. Graphical user interface SpaceRL offers a GUI that provides easy access to the main functionalities for an average user. The GUI implementation is based on Python default GUI manager, Tkinter. SpaceRL GUI includes a visualization tool (cf. Fig. 3) to graphically display any previously computed test result, including the inferred path, the relevant KG nodes, and the available actions at each step. The user can navigate between path nodes with the arrow keys, thus getting extended information about each subsequent step, namely: the node in which the agent was (highlighted in red), and the score given to each outgoing relation, which corresponds to a possible action that the agent could have taken. The selected actions are highlighted in a bold dark red line. This tool relies on Pygame [35], the SDL kit [36], and the NetworkX package [37]. 2.5. API SpaceRL also provides an application programming interface that can be used by developers to implement their own applications on top of our functionalities. Our API service relies on FastAPI [38] to generate an OpenAPI [39] compliant application, and Uvicorn [40] as a backend webserver to host the app. The use of FastAPI makes it easier Fig. 4. API structure of SpaceRL. to deploy it to a production server or to exchange it for another backend if desired. As depicted in Fig. 4the FastAPI application handles user requests through the webserver while the ConnectionManager holds the internal client–server architecture responsible for managing resource intensive operations while delivering instant responses back to the main application in order to deliver fast progress to the end user. The ConnectionManager class does not rely on intermediate packages to handle requests, which makes it faster and independent from interoperability limitations given by other general purpose software. SpaceRL internally distinguishes between two types of endpoints, instant and process. Both provide fast responses to the end user; however, process endpoints correspond to computationally intensive background tasks, which require a significant amount of resources. Therefore, the response sent to the user in those cases is merely a message to inform whether the process could be launched. If the resources needed to attend aprocess request are available, the ConnectionManager underlying client sends a plain-text message to the internal server with the request in a particular format and the server answers back in the same way (cf. listings 1and 2for examples of these exchanges). Note that SpaceRL accounts for computing resources limitations and it will return a failure message to any request to a process endpoint SoftwareX 30 (2025) 102078 4 Miguel Bermudo et al. 1message: post; cache;{’ datasets’:[’ COUNTRIES’], ’ depth’:3} 2response: success; cache is being generated, please be patient Listing 1 Internal server cache generation request and response 1message: " post; embedding;{‘ dataset’: ‘ NELL-995’, ‘ models’: [], ‘ use_gpu’: True, ‘ regenerate_existing’: True, ‘ normalize’: True, ‘ add_inverse_path’: True, ‘ fast_mode’: False}" 2response: " success; Embedding Calculation Launched" Listing 2 Internal server embedding generation request and response if it determines that it will cause problems for the host machine. For instance, if there is a embedding generation endpoint running which is using the only available GPU in the system and the /experiments/run endpoint is invoked with configuration parameter use_gpu = True, the API will respond with aBusyError, notifying the user that there are not enough resources available. 3. Illustrative example In order to better understand the capabilities of SpaceRL, we will describe how to use it to complete the well-known NELL-995, a subset of the NELL [41] knowledge graph. NELL is built automatically from web data in a continuous and mostly unsupervised fashion, meaning that there are usually a large number of missing triples, which makes it ideal to validate KG completion proposals. The entry point to our system is the input KG in the expected format, i.e., one (𝑠, 𝑟, 𝑡)triple per line, being each triple a sequence of three tab-separated values (𝑠,𝑟, and 𝑡), as shown in the following example: 1newspaper_daily_record newspaperincity city_baltimore 2 3city_baltimore citylocatedinstate stateorprovince_maryland 4 5sportsteam_coppin_state_lady_eagles teamplaysincity city_baltimore 6 7sportsteam_johns_hopkins teamplaysincity city_baltimore 8 9athlete_don_zimmerman athleteplaysinteam sportsteam_johns_hopkins 10 11 politician_sheila_dixon personhasresidencein city_baltimore Listing 3 An extract of NELL dataset with the expected format. The first step is to choose the most suitable values for the configuration parameters, as shown in listing Section 3, which also displays other possible values that the parameters can take as Python-style comments. Of particular note for this example, we chose ‘‘terminal’’ and ‘‘embedding’’ as the options for the reward, we could alter the reward function by selecting other options, for example for knowledge graphs which we presume have longer information chains, we could set a distance reward and allow for longer path chains with a different value for the ‘‘path_length’’ option. 1config = { 2" available_cores":8, 3[...] 4" guided_reward":True, 5 6#" distance"," terminal"," embedding"," shaping" 7" guided_to_compute":[" terminal"," embedding"], 8[...] 9# [0.8-0.99] PPO prev step NN learning rate 10 " alpha":0.9, 11 12 # [0.90-0.99] decay rate of past observations 13 " gamma":0.99, 14 15 #[1 e-3, 1 e-5] NN learning rate. 16 " learning_rate":1e-3, 17 18 # relu, prelu, leaky_relu, elu, tanh 19 " activation":’ leaky_relu’, 20 21 #" kernel", " bias", " activity" 22 " regularizers":[’ kernel’], 23 " algorithm":" PPO",# BASE, PPO 24 " reward_type":" simple",# retropropagation, simple 25 26 # " probability", " max" 27 " action_picking_policy":" probability", 28 29 #" max_percent", " one_hot_max", " straight" 30 " reward_computation":" one_hot_max", 31 32 " path_length":3, 33 " random_seed":True, 34 " seed":0 35 } Listing 4 Excerpt of the configuration parameters used for the example The next step is embedding generation, which can be triggered by issuing an HTTP POST request to endpoint /embeddings, including in the request body the parameters shown in listing Section 3. 1{ 2" dataset" :" NELL995", 3" models" : [], 4" use_gpu" :true, 5" regenerate_existing" :true, 6" normalize" :true, 7" add_inverse_path" :true, 8" fast_mode" :false 9} Listing 5 The request body parameters to generate all embeddings for NELL KG. The response to this request is sent back to the API client with information regarding whether the operation has began correctly, or if any error has occurred, e.g., if the server resources are busy and the request cannot be attended. Finally, to generate the RL agents, new experiments must be added to the experiments list via HTTP POST requests to the /experiment endpoint. An example of a new experiment that uses TransE embeddings and 150 laps to train on the NELL-995 KG can be seen in Section 3. 1{ 2" name" :" My_new_NELL_Agent", 3" dataset" :" NELL995", 4" single_relation" :false, 5" embedding" :" TransE_l2", 6" laps" :150, 7" relation_to_train" :"" 8} Listing 6 The request body parameters to generate a NELL RL Agent SoftwareX 30 (2025) 102078 5 Miguel Bermudo et al. Fig. 5. SpaceRL Configuration menu (left), Training menu (top-right) and Main Menu while execution is running (bottom-right). Fig. 6. Testing submenu with NELL Agent being tested. In response to the former request, the server adds the experiment to a queue, for latter processing. Then, the experiments would be run by issuing a POST request to the /experiment/run endpoint, which expects a list of ids to run, or an empty list, in which case all queued experiments are run. The main menu, now displays the progress of the current operations. Note that in the text above the progress bar, ‘‘(1/1)’’ indicates that only 1 element is queued for testing while ‘‘(16/200)’’ expresses the current and total episodes being run. The GUI can access the configuration options serving and server as a more accessible way to queue training and testing episodes for novices (Cf. Fig. 5). For example Fig. 6depicts a test for a ‘‘My_new_NELL_Agent’’ and allows the user to pick the appropriate embeddings that the selected agent was trained with as well as the number of episodes and add it to the queue FiFo style. Once the execution is finished, a new folder is created in model/data/results/My_NELL_Agent_Test, containing ametrics.csv file with the Hits@ and MRR metric values, and a paths.txt file containing the inferred paths in this testing execution. 1(‘ concept_ceo_alan_greenspan’, 2‘ concept: topmemberoforganization’, 3‘ concept_bank_u_s__federal_reserve’, 4‘¬concept: topmemberoforganization’, 5‘ concept_politicianus_ben_bernanke’, 6‘ concept: worksfor’, 7‘ concept_bank_federal_reserve’) Listing 7 An example of a returned path by the agent. Listing depicts an example of a newly reasoned path, expressed as the alternating sequence of entities and relations that must be traversed to get from the source entity to the target entity. This path provides some explainability regarding the existence of the initial query triple. For this particular example, we would have the following logic: •Alan Greenspan is a top member of the US federal reserve. •Ben Bernake is also a top member for the US federal reserve. •Ben Bernake works for the federal reserve. •Therefore, Alan Greenspan also works for the federal reserve. SoftwareX 30 (2025) 102078 6 Miguel Bermudo et al. •As a consequence, fact ‘‘(concept_ceo_alan_greenspan, concept: worksfor, concept_bank_federal_reserve)’’ should be added to the KG. 4. Impact SpaceRL offers a set of reinforcement learning tools tailored towards Knowledge Graph completion and reasoning. SpaceRL provides some novel benefits, namely: a versatile tool with different available interfaces, a visualization tool to graphically display the results, a GUI that provides access to less experienced users, and a REST API that enables for it to be operated as a MLaaS. To the best of our knowledge, it is the first tool to do so. SpaceRL was designed not only as a tool for final users, but also as a base for other developers to create their own custom tools, which can be offered as a service to a third party through its API complying with the openAPI convention. This might appeal to companies who wish to either serve or consume the capabilities offered. SpaceRL is also highly customizable, offering multiple configuration options that allow tuning it to deal with different tasks related to KG reasoning, such as completion, question answering, or product recommendations, among others. SpaceRL is Open Source software, open for collaboration in its GitHub repository [27]. Therefore, it could also be potentially beneficial for researchers in data engineering areas who can use our functionalities as a solid support to build their own smart applications. Finally, our tool could also aid researchers in other areas, such as biomedicine, statistics, or data science, who lack computing skills as to build their own software, but still require this kind of tool to operate on their knowledge graphs. 5. Conclusions We have presented SpaceRL, a software suite that aims to bridge the existing gap in the realm of Knowledge Graph reasoning, completion and visualization for users of any level of expertise. SpaceRL provides both a graphical user interface and a collection of REST API endpoints, so that it can be easily managed by an end user, but more experienced users could also integrate its functionalities into their own software applications. We have distributed SpaceRL as Open Software, to make it easily expandable by keeping the dependencies to a minimum whenever possible, providing extensive documentation and following community standards such as OpenAPI and Gymnasium. The future steps in the life cycle of SpaceRL would be to design modular reward and policy elements which could be altered by experts to increase its reach even more, to offer a large scale implementation to support uncoupled operations and to increase the level of the documentation offered The next steps in SpaceRL life cycle would be the inclusion of a policy builder to automate the process of constructing an NN and to add further options such as self-attention mechanisms into the policy for more modern applications. CRediT authorship contribution statement Miguel Bermudo: Writing – review & editing, Writing – original draft, Software, Methodology, Investigation, Data curation, Conceptualization. Daniel Ayala: Writing – review & editing, Supervision, Resources, Formal analysis, Conceptualization. Inma Hernández: Writing – review & editing, Validation, Supervision, Resources, Project administration, Funding acquisition. David Ruiz: Supervision, Project administration, Methodology, Funding acquisition. Miguel Toro: Supervision, Funding acquisition. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgments This work was supported by the Spanish Ministry of Science, Innovation and Universities (grants PID2019-105471RB-I00, TED2021131023B-C21/ AEI/10.13039/501100011033/ Unión Europea NextGenerationEU/PRTR (IRIS), and TED2021-131023B-C22/ AEI/10.13039/501100011033/ Unión Europea NextGenerationEU/ PRTR (ORCHID)), and the office for Economy and Knowledge of Andalusia (grants P18-RT-1060, US-1380565). References [1] Ji S, Pan S, Cambria E, Marttinen P, Yu P. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Trans Neural Networks Learn Syst 2022;33:494–514. [2] Noy N, Gao Y, Jain A, Narayanan A, Patterson A, Taylor J. Industry-scale knowledge graphs: Lessons and challenges: Five diverse technology companies show how it’s done. Queue 2019;17:48–75. [3] Liu L, Du B, Xu J, Xia Y, Tong H. Joint knowledge graph completion and question answering. In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. 2022, p. 1098–108. http://dx.doi.org/10. 1145/3534678.3539289. [4] Huang X, Zhang J, Li D, Li P. Knowledge graph embedding based question answering. In: WSDM’19. 2019, p. 105–13. [5] Yang Y, Huang C, Xia L, Li C. Knowledge graph contrastive learning for recommendation. In: Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 2022, p. 1434–43. http: //dx.doi.org/10.1145/3477495.3532009. [6] Schlegel V, Freitas A. Dbee: A database for creating and managing knowledge graphs and embeddings. In: Proceedings of the thirteenth workshop on graphbased methods for natural language processing, textGraphs@eMNLP 2019. 2019, p. 177–85. [7] Paulheim H. Knowledge graph refinement: A survey of approaches and evaluation methods. Semant Web. 2017;8:489–508. [8] Hogan A, Blomqvist E, Cochez M, D’Amato C, Melo G, Gutierrez C, et al. Knowledge graphs. CoRR 2020. abs/2003.02320. [9] Borrego A, Ayala D, Hernández I, Rivero C, Ruiz D. CAFE: Knowledge graph completion using neighborhood-aware features. Eng Appl Artif Intell 2021;103:104302. http://dx.doi.org/10.1016/j.engappai.2021.104302. [10] Chen Z, Wang Y, Zhao B, Cheng J, Zhao X, Duan Z. Knowledge graph completion: A review. Ieee Access 2020;8:192435–56. [11] Galárraga L, Teflioudi C, Hose K, Suchanek F. Fast rule mining in ontological knowledge bases with amie+. VLDB J 2015;24:707–30. [12] Kolthoff K, Dutta A. Semantic relation composition in large scale knowledge bases. In: LD4IE@iSWC. vol. 1467, 2015, p. 34–47. [13] Borrego A, Ayala D, Hernández I, Rivero C, Ruiz D. Generating rules to filter candidate triples for their correctness checking by knowledge graph completion techniques. In: Proceedings of the 10th international conference on knowledge capture, k-CAP 2019. 2019, p. 115–22. http://dx.doi.org/10.1145/3360901. 3364418, November 19-21, 2019. [14] Dai Y, Wang S, Xiong N, Guo W. A survey on knowledge graph embedding: Approaches, applications and benchmarks. Electronics 2020;9:750. [15] Gardner M, Mitchell T. Efficient and expressive knowledge base completion using subgraph feature extraction. In: EMNLP. 2015, p. 1488–98. [16] Mazumder S, Liu B. Context-aware path ranking for knowledge base completion. In: IJCAI. 2017, p. 1195–201. [17] Xiong W, Hoang T, Wang W. Deeppath: A reinforcement learning method for knowledge graph reasoning. 2017, arXiv preprint arXiv:1707.06690. [18] Das R, Dhuliawala S, Zaheer M, Vilnis L, Durugkar I, Krishnamurthy A, et al. Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. 2017, arXiv preprint arXiv:1711.05851. [19] Lin X, Socher R, Xiong C. Multi-hop knowledge graph reasoning with reward shaping. 2018, arXiv preprint arXiv:1808.10568. [20] Xian Y, Fu Z, Muthukrishnan S, De Melo G, Zhang Y. Reinforcement knowledge graph reasoning for explainable recommendation. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval. 2019, p. 285–94. [21] Tiwari P, Zhu H, Pandey H. DAPath: Distance-aware knowledge graph reasoning based on deep reinforcement learning. Neural Netw 2021;135. SoftwareX 30 (2025) 102078 7 Miguel Bermudo et al. [22] Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. 2017, arXiv preprint arXiv:1707.06347. [23] Haarnoja T, Zhou A, Abbeel P, Levine S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International conference on machine learning. 2018, p. 1861–70. [24] Bermudo M, Ayala D, Hernández I, Ruiz D, Toro M. SpaceRL-KG: Searching paths automatically combining embedding-based rewards with reinforcement learning in knowledge graphs. In: Expert systems with applications. 2024, 124410. [25] Hasan R, Akawee M, Sutikno T. Improved GIS-t model for finding the shortest paths in graphs. Babylon J Mach Learn 2023;2023:7–16. [26] Sutton R, McAllester D, Singh S, Mansour Y. Policy gradient methods for reinforcement learning with function approximation. Adv Neural Inf Process Syst 1999;12. [27] SpaceRL. 2023, https://github.com/DEAL-US/SpaceRL-KG, [Accessed July 2023]. [28] OpenAIGym. 2023, https://github.com/openai/gym, [Accessed July 2023]. [29] Gymnasium. 2023, https://github.com/Farama-Foundation/Gymnasium, [Accessed July 2023]. [30] Trouillon T, Welbl J, Riedel S, Bouchard GGaussier. Complex embeddings for simple link prediction. In: ICML. vol. 48, 2016, p. 2071–80. [31] Yang B, Yih S, He X, Gao J, Deng L. Embedding entities and relations for learning and inference in knowledge bases. In: Proceedings of the international conference on learning representations, ICLR. vol. 2015, 2015. [32] Bordes A, Usunier N, García-Durán A, Weston J, Yakhnenko O. Translating embeddings for modeling multi-relational data. NIPS; 2013, p. 2787–95. [33] Lin Y, Liu Z, Sun M, Liu Y, Zhu X. Learning entity and relation embeddings for knowledge graph completion. AAAI 2015;15:2181–7. [34] Zheng D, Song X, Ma C, Tan Z, Ye Z, Dong J, et al. Dgl-ke: Training knowledge graph embeddings at scale. In: Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval. 2020, p. 739–48. [35] PyGame. 2023, https://github.com/pygame/pygame, [Accessed July 2023]. [36] SDL Kit. 2023, https://www.libsdl.org/, [Accessed July 2023]. [37] NetworkX. 2023, https://github.com/networkx/networkx, [Accessed July 2023]. [38] FastAPI. 2023, https://github.com/tiangolo/fastapi, [Accessed July 2023]. [39] OpenAPI. 2023, https://github.com/OAI/OpenAPI-Specification, [Accessed July 2023]. [40] Uvicorn. 2023, https://github.com/encode/uvicorn, [Accessed July 2023]. [41] Mitchell T, Cohen W, Hruschka E, Talukdar P, Yang B, Betteridge J, et al. Never-ending learning. Commun ACM. 2018;61:103–15. SoftwareX 30 (2025) 102078 8