From language models to large‐scale food and biomedical knowledge graphs

1 Vol.:(0123456789) Scientific Reports | (2023) 13:7815 | https://doi.org/10.1038/s41598-023-34981-4 www.nature.com/scientificreports From language models to large‑scale food and biomedical knowledge graphs Gjorgjina Cenikj 1,2*, Lidija Strojnik 1, Risto Angelski 3, Nives Ogrinc 1, Barbara Koroušić Seljak 1 & Tome Eftimov 1 Knowledge about the interactions between dietary and biomedical factors is scattered throughout uncountable research articles in an unstructured form (e.g., text, images, etc.) and requires automatic structuring so that it can be provided to medical professionals in a suitable format. Various biomedical knowledge graphs exist, however, they require further extension with relations between food and biomedical entities. In this study, we evaluate the performance of three state‑of‑the‑art relation‑mining pipelines (FooDis, FoodChem and ChemDis) which extract relations between food, chemical and disease entities from textual data. We perform two case studies, where relations were automatically extracted by the pipelines and validated by domain experts. The results show that the pipelines can extract relations with an average precision around 70%, making new discoveries available to domain experts with reduced human effort, since the domain experts should only evaluate the results, instead of finding, and reading all new scientific papers. Noncommunicable chronic diseases (NCDs) account for more than 70% of deaths worldwide. Cardiovascular diseases account for most NCD deaths (17.9 M people annually), followed by cancers (9.3 M), respiratory diseases (4.1 M), and diabetes mellitus (1.5 M)1,2. As the leading cause of death globally, most of the deaths that happen from cardiovascular diseases (CVDs) are due to heart attacks and strokes3. A lot of scientific evidence indicates that between the most important risk factors for heart disease and stroke are unhealthy diet, alcohol and tobacco consumption, and physical activity. Among all the factors that contribute to the development and progression of CVDs, diet is one of the major ones4,5. It has been shown that eating more fruit and vegetables and decreasing the salt in diet reduce the risk of CVDs. Further, although there is a lot of knowledge about dietary effects on CVDs and broadly on NCDs, there are still many unresolved research questions. Such questions are not easy to be answered because food and nutrition in relation to diseases are described by various concepts and entities that interact in various ways6. For instance, there are many foods (described by food entities) made up of components (described by chemical entities)7 that may fight NCDs (described by disease entities) while others can be harmful8. These impacts are dependent on the combination of foods and their chemicals, the state of the food (e.g., raw/cooked, fresh/molded, etc.), the cooking method (e.g., steamed, grilled, baked, etc.), the health status of the person consuming food (e.g., healthy, ill, allergic) and others9. As there are many combinations of these factors, collecting and structuring the relations between all the concepts and entities describing the impacts of food on NCDs is a very complex work exceeding human capabilities. And taking into account the fact that research in this field is still progressing, the related knowledge evolves on a daily basis, making it challenging to follow. Such knowledge further opens possibilities to use Artificial Intelligence (AI) methods to aid in the early detection (prediction) of NCDs as well as their progression. However, before developing predictive AI methods, unstructured (textual) data available in cohorts, electronic health records (EHRs), registries, and scientific and grey literature needs to be structured and normalized/linked to domain semantic resources and further included in knowledge bases (KBs) which can be utilized for predictive modeling and integrated into health systems which will make the information easily accessible to medical professionals. To this end, user interfaces play a critical role in ensuring that healthcare professionals can effectively utilize AI systems to provide high-quality care to their patients10. A Knowledge Graph (KG) is a type of KB, where knowledge is stored in the form of entities characterized by some attributes, and relations connecting the entities. Conventional methods of KG construction can be broadly categorized into manual, and automatic, or semi-automatic methods. The benefits of manual creation and curation approaches are their high precision and reliability11, however, due to the high amount of effort OPEN 1Jožef Stefan Institute, Ljubljana 1000, Slovenia. 2Jožef Stefan International Postgraduate School, Ljubljana 1000, Slovenia. 3Clinic Doctor 24-hours, Ljubljana 1000, Slovenia. *email: gjorg[email protected] 2 Vol:.(1234567890) Scientific Reports | (2023) 13:7815 | https://doi.org/10.1038/s41598-023-34981-4 www.nature.com/scientificreports/ required by domain experts, they also have lower recall rates, poor scalability and time efficiency12. Automatic and semi-automatic KG construction is enabled by text-mining methods, which are able to extract entities and relations which can be structured as a KG. In the biomedical domain, automatic and semi-automatic structuring of textual data in the form of KGs is an active research area, which typically involves the use of Information Extraction (IE) pipelines consisting of multiple components. These components include Named Entity Recognition (NER) methods, which extract specific types of entities from raw text, Named Entity Linking (NEL) methods, whose goal is to map entity mentions to entries in a given KB, and Relation Extraction (RE) methods, which aim to automatically detect relations between entities13. Over the past 20 years, significant progress has been made in creating multiple IE pipelines for the biomedical domain. These pipelines primarily concentrate on identifying genotype and phenotype entities, as well as health-related entities such as diseases, treatments, drugs, and others. To allow their development, several collaborative workshops, as part of conference events like BioNLP14, BioCreative15, i2b216, and DDIExtraction17, have been arranged to provide semantic resources (e.g., annotated corpora, ontologies) that will further allow the developing of biomedical IE pipelines. The efforts done in the biomedical domain are focused entirely on biomedical concepts and not investigating relations with food concepts. On the other side, most of the efforts done in IE in the food domain are focused on relations that do not involve health/biomedical concepts, and even more, are developed using static data that is already presented in some other resources (e.g., datasets, controlled vocabularies, ontologies), so they need to be updated when new data is available in these resources. In addition, only a few studies have concentrated on traditional text mining techniques that employ sentiment analysis through manual feature extraction18–20. Despite this, the food and nutrition domain is low-resourced in semantic data resources compared to the biomedical domain. There is a lack of annotated food-disease relation corpora that serve as a benchmark and help develop IE pipelines. Even more, food semantic resources such as FoodOn21, FoodEx222, are still under development (i.e., frequently updating them with new data) to support IE activities. To bridge the gap between the food and biomedical domains, we introduce an approach that uses language models to extract the relations that exist between food, chemical, and disease entities and further normalize them to allow the creation of a KG. In our case, we evaluate the approach to trace the new knowledge about CVDs and milk products. The benefit of our approach is that we are not using the information that already exists in some static resources (e.g., databases), but try to catch all relations from textual data related to CVDs and milk products (milk was selected as a case study since it is rich in nutrients, a resource of proteins, vitamins, minerals, and fatty acids, which have an important impact on human metabolism and health) available in scientific abstracts, where new findings are presented. This makes the methodology easy to apply on new corpora of scientific abstracts, where the results of the pipelines can point out areas where the KG should be updated with new entities or relations. Related work A recent survey on knowledge-based biomedical data science23 highlights the application of KGs in the biomedical and clinical domain in improving the retrieval of information from large sources of clinical data or literature24–26, providing evidence to support phenomena observed in data27,28, using link prediction to complete missing information and hypothesize previously unknown relationships29, and improving patient data representation30–32. In the biomedical domain, IE pipelines have been developed for the extraction of drug-disease relations33,34 and disease-symptom relations35 from biomedical literature. A Coronavirus KG has been constructed by merging the Analytical Graph, with a collection of published scientific articles36. A PubMed KG has been constructed by extracting biomedical entities from PubMed abstracts and enriching it with funding, author, and affiliation data37. A recent work12 proposes the construction of domain-specific KGs with minimal supervision, which is able to derive open-ended relations from unstructured biomedical articles without the need of extensive labeling. While this study is largely focused on data integration, and only uses NER to extract the biomedical entities from the literature, our study goes a step further in the RE task, to extract the relations between the entities based on the text in the scientific abstracts, so that new relations can be added between entities in existing resources. Apart from using biomedical scientific papers as a source of information, EHRs have also been used for extracting disease-symptom relations38 and constructing a medical KG with nine biomedical entity types39. In the food domain, FoodKG has been recently developed for representing food recipe data including their ingredients and nutritional content40 by enriching a large amount of recipe data from Recipe1M dataset with the nutritional information available from USDA’s National Nutrient Database for Standard Reference represented with FoodOn21 semantic meta-data. Additionally, FoodKG41 was developed by using the existing text and graph embedding techniques applied to a controlled vocabulary called AGROVOC, to model the relations that exist in a plethora of datasets related to food, energy and water. Results To trace the knowledge about food, chemical, and disease interactions, we have shown the creation of a KG centered around the impact of different foods and chemicals on CVDs, and the other targeting the composition of the selected food item “milk”, as well as its beneficial and detrimental effects on different NCDs. For this purpose, three NLP pipelines, called FooDis, FoodChem, and ChemDis, were combined to extract “food-disease”, “foodchemical”, and “chemical-disease” relations from textual data. Semantically, we distinguish two relations between food-disease and chemical-disease entity pairs, which are “treat” and “cause”. In the case of food-chemical entity pairs, we extracted only one relation which is “contains”. All three pipelines were executed twice, on two different corpora, one that was collected for CVDs and one collected for milk products. In both use cases, the searched keywords were selected by domain experts. In the CVDs case, a more general keyword was selected “heart disease food”, since we would like to retrieve broader aspects between different cardiovascular events and food products. 3 Vol.:(0123456789) Scientific Reports | (2023) 13:7815 | https://doi.org/10.1038/s41598-023-34981-4 www.nature.com/scientificreports/ This ends up with 9984 abstracts. In the milk use case, three keywords were selected by the domain experts i.e., “milk composition”, “milk disease”, and “milk health benefits”. Table1a presents the number of abstracts that were retrieved and used in the analysis for both use cases, together with the keywords used to retrieve them, while Table1b presents the number of relations that were extracted for both use cases. Figure1a features the KG constructed by running the three pipelines for the two application use cases. The same nodes are grouped together by normalizing the extracted food, chemical, and disease entities. To go into more detail how the KG is constructed, in Fig.1b we present an example using the relations extracted for the “heart failure” disease entity. The green nodes, “meat products”, “salt” and “dietary fish oil” represent the food entities for which the FooDis pipeline extracted a relation with the “heart failure” disease entity, meaning that they have some effect on its development or treatment. In particular, the red edges connecting the “heart failure” disease entity and the food entities “meat products”, and “salt” indicate that the pipeline identified a “cause” relation, i.e. meat products, and salt can contribute to the occurrence of heart failure. On the other hand, the green edge between the “dietary fish oil” entity and the “heart failure” disease entity indicates a “treat” relation, i.e. the pipeline identified that dietary fish oil has a beneficial effect to heart failure. Similarly, the ChemDis pipeline identified that the chemical entities “DHA”, “ester”, “acid, n-3 fatty”, “antidiabetics canagliflozin”, “omega-3 fatty acid” and “calcium” can be used for treating “heart failure”, while the chemical entities “(-)-cocaine” and “vitamin E” can contribute to the development of “heart failure”. Table2 presents the supporting sentences from scientific abstracts from which the relations were extracted and further used for constructing the graph presented in Fig.1b. Next, such graphs are connected based on the same entities to link the information from different abstracts. Further, to validate the extracted information, domain experts were involved to check the extracted relations for both use cases. Use case: cardiovascular diseases. For the CVDs use case, a highly-skilled domain expert (an MD with more than 40 years of working experience in cardiology) evaluated the extractions from the three pipelines. The relations that were evaluated are extracted after the “Final relation determination” step from the FooDis, FoodChem and ChemDis pipelines. All three pipelines utilized here follow the same workflow. Each extracted relation is determined by all sentences where information about it is presented. We called them “supporting sentences”. The sentences can be from the same or different abstracts, since information about the same relation can be investigated in different papers. Domain expert evaluation. Each pipeline provides the result as a 6-tuple i.e., (name of the first entity, named of the second entity, synonyms for the first entity, synonyms for the second entity, relation, supporting sentences), which is further evaluated by the domain expert. The domain expert was asked to assign a binary indicator of the truthfulness of the relation. The pipelines were then evaluated by taking the mean of the correctness indicators assigned by the annotator for each relation and pipeline, which we refer to as the precision in the remainder of this section. In particular, if a pipeline extracted three relations, and the expert marked two of these as correct (binary indicators 1,0,1), the reported precision would be 0.66. Figure2a presents the number of relations extracted by each of the pipelines for the CVDs study, and the number of relations that the domain expert evaluated. We need to point out here that all extracted relations were provided to the domain expert, however, the evaluation has been performed only on those relations for which the domain expert has expert knowledge. Because of this, the human evaluation process covers 44% of the “contains” Table 1. Number of processed paper abstracts and number of extracted relations for each case study. Case study Keyword Number of abstracts Number of paper abstracts processed for each search keyword Heart disease Heart disease food 9984 Milk Milk composition 13,500 Milk disease 17,268 Milk health benefits 2343 Case study IE pipeline Relation Number of relations extracted Number of relations extracted by each pipeline in each case study Heart disease FooDis Cause 516 Treat 699 ChemDis Cause 635 Treat 1079 FoodChem Contains 981 Milk FooDis Cause 1184 Treat 789 ChemDis Cause 670 Treat 597 FoodChem Contains 1875 4 Vol:.(1234567890) Scientific Reports | (2023) 13:7815 | https://doi.org/10.1038/s41598-023-34981-4 www.nature.com/scientificreports/ relations extracted by the FoodChem pipeline, 33% of the “treat” relations extracted by the FooDis pipeline, 26% of the “cause” relations extracted by the FooDis pipeline, 26% of the “cause” relations extracted by the ChemDis pipeline, and 23% of the “treat” relations extracted by the ChemDis pipeline. (a) Theentire knowledge graph containing allextracted relations (b) Egonetwork of the"heartfailure" node Figure1. Knowledge graph constructed using the FooDis, FoodChem and ChemDis pipelines. The nodes in green represent the food entities, the nodes in blue represent the chemical entities, and the nodes in red represent the disease entities. The red, green, and blue edges represent the “cause”, “treat” and “contains” relations, respectively. The figures have been generated using the pyvis python library42, version 0.1.8.2. 5 Vol.:(0123456789) Scientific Reports | (2023) 13:7815 | https://doi.org/10.1038/s41598-023-34981-4 www.nature.com/scientificreports/ The mean precision of each of the pipelines (FooDis, ChemDis, and FoodChem) in the CVDs use case is presented in Fig.2b. From it, the FooDis pipeline achieves the highest precision of 0.79 for the “cause” and 0.78 for the “treat” relation. The lowest precision of 0.68 is achieved by the ChemDis pipeline for the extraction of the “cause” relation. Since the three pipelines extract a relation based on supporting sentences, in the Supplementary Materials, we have presented the distribution of the number of relations versus their number of supporting sentences. All of the pipelines extract more than 74% of the relations based on a single supporting sentence. The ChemDis and FoodChem pipelines can find a larger number of supporting sentences for some relations compared to the FooDis pipeline. In particular, the ChemDis pipeline can find up to five supporting sentences to identify “cause” relations and up to 14 supporting sentences to identify “treat” relations, while the FooDis pipeline uses up to three, and four supporting sentences for the “cause” and “treat” relations, respectively. Next, to see how the mean precision is affected by the number of supporting sentences, we analyze for each semantic relation separately. The results are presented in Supplementary Materials. From the conducted analysis, we can conclude that the mean precision is proportional to the number of supporting sentences. Almost for all relations, a precision of 1.00 is reached when the number of supporting relations is sufficiently high. This indicates that when the number of supporting sentences for a relation increases, there is an agreement between the domain expert validation and the result provided by our pipelines, with some exceptions listed in the Supplementary Materials. Error analysis. Next, we analyze the types of false discoveries produced by FooDis, FoodChem, and ChemDis pipelines. Figure3 features the relations with the highest number of supporting sentences for four chemical entities: “carbohydrates”, “fatty acid”, “sodium” and “vitamin d”. Here the results for the selected chemical entity from the two pipelines that deal with chemical entities (i.e., ChemDis and FoodChem) are presented. The green bars refer to the number of sentences in which the relation was correctly identified, while the purple plots refer to the number of false positive sentences for that relation, i.e. sentences where the relation was identified, however, it was marked as incorrect by the experts. For the “carbohydrates” entity, the ChemDis pipeline produced the false positive relation “carbohydratestreat-cardiomyopathy” when the supporting sentences suggested that a low-carbohydrate diet is recommended for treating cardiomyopathy. In this case, the pipeline fails to identify that a reduction of the chemical entity is required to treat the disease. In addition, the FoodChem pipeline produces a false discovered relation “bulkcontains-carbohydrates”, when the supporting sentence was saying that these two entities are contained in another entity, “dry beans”. For the “fatty acid” chemical entity, the ChemDis pipeline produced the false positive relation “fatty acid-cause-dysfunction endothelial”, when the supporting sentence was saying that increased fatty acid levels and endothelial dysfunction were contributing to the development of another disease, “sepsis”. The FoodChem pipeline produced the false entities, “wine-contains-fatty acid” and “acid fatty trans-contains-fatty acid”. In the first case, the two entities were co-occurring in the supporting sentence without any relation, while in the second one, the sentence was saying that trans fatty acids are a subcategory of fatty acids. In the case of Table 2. Supporting sentences for the relations of entity “heart failure” to different food and chemical entities. Food/chemical name Relation Supporting sentences (−)-Cocaine Cause 1) Additionally, cocaine use has been associated with left ventricular hypertrophy, myocarditis, and dilated cardiomyopathy, which can lead to heart failure if drug use is continued Vitamin E Cause 1) Yet, high doses of supplemental vitamin E have been associated with an elevated risk of heart failure and all-cause mortality. 2) Vitamin E supplementation might be associated with an increase in total mortality, heart failure, and hemorrhagic stroke Salt Cause 1) In patients who already have heart failure, a high salt intake aggravates the retention of salt and water, thereby exacerbating heart failure symptoms and progression of the disease Meat products Cause 1) Thermal processing of meat products generates cardiotoxic compounds capable of inducing heart failure in both humans and laboratory animals Acid, n-3 fatty Treat 1) Evidence from epidemiological, clinical and experimental studies indicates a beneficial role of the omega-3 polyunsaturated fatty acids (omega-3 PUFA) found in fish oils in the prevention and management of heart failure. 2) This review summarise the data related to use of omega-3 PUFA supplementation as a potential treatment for heart failure and discussed possible mechanism of action. 3) The 2017 American Heart Association science advisory on omega-3 fatty acid supplements suggested that it is reasonable to use omega-3 fatty acids for secondary prevention in people with coronary heart disease and heart failure Antidiabetics canagliflozin Treat 1) It has been concluded that canagliflozin, dapagliflozin, empagliflozin, or ertugliflozin can be recommended for preventing hospitalization associated with heart failure in patients with type 2 diabetes and established cardiovascular disease or those at high cardiovascular risk DHA Treat 1) Intake of fish oil containing docosahexaenoic acid (DHA) and eicosapentaenoic acid (EPA) prevents heart failure; however, the mechanisms are unclear Ester Treat 1) Because L-carnitine and its esters help reduce oxidative stress, they have been proposed as a treatment for many conditions, i.e. heart failure, angina and weight loss Omega-3 fatty acid Treat 1) The 2017 American Heart Association science advisory on omega-3 fatty acid supplements suggested that it is reasonable to use omega-3 fatty acids for secondary prevention in people with coronary heart disease and heart failure Calcium Treat 1) Here we review the key observations, controversies, and discoveries that have led to the development of novel compounds targeting the RyR2/calcium release channel for treating heart failure and for preventing lethal arrhythmias Dietary fish oil Treat 1) Intake of fish oil containing docosahexaenoic acid (DHA) and eicosapentaenoic acid (EPA) prevents heart failure; however, the mechanisms are unclear 6 Vol:.(1234567890) Scientific Reports | (2023) 13:7815 | https://doi.org/10.1038/s41598-023-34981-4 www.nature.com/scientificreports/ the “sodium” chemical entity, most of the sentences extracted by the ChemDis pipeline express the correct relation, however, sodium is incorrectly extracted as a partial match of the entity “Sodium-glucose co-transporter 2 inhibitors (SGLT2is)”. In the case of “vitamin d”, all of the false positive “cause” relations extracted by the ChemDis pipeline are due to the pipeline not recognizing that the deficiency of the vitamin was causing the diseases. Figure4 features the top 10 relations with a maximal number of supporting sentences for three disease entities. Here, we present the results from pipelines that are dealing with disease entities (i.e., FooDis and ChemDis). For 635 1079 291 444 759 635 171 254 76 147 339 171 ChemDis cause ChemDis treat FooDis cause FooDis treat Fo odChem contains 0200 400600 8001000 legend extracted relations evaluated relation s (a) Number of extracted andevaluated relations pereachpipelinefor theheart diseasestudy 0.68 0.75 0.79 0.78 0.74 0.68 ChemDis cause ChemDis treat FooDis cause FooDis treat FoodChem contains 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 (b) Mean precision of therelations extracted by each pipelinefor theheart diseasestudy Figure2. Number of extracted and evaluated relations and mean precision of each pipeline for the heart disease study. The plots have been generated using the plotly python library43, version 5.7.0. 7 Vol.:(0123456789) Scientific Reports | (2023) 13:7815 | https://doi.org/10.1038/s41598-023-34981-4 www.nature.com/scientificreports/ the “general cardiovascular disorders” entity, the pipelines extracted the relations “dietary vegetable-cause-general cardiovascular disorders”, “acid, saturated fatty-treat-general cardiovascular disorders”, “acid fatty polyunsaturated-cause-general cardiovascular disorders”, “cholesterol-treat-general cardiovascular disorders” due to the 0 1 2 Number of supporting sentences cardiovascular sy... obesity obesities, visceral hypertension fatty infiltratio... dyslipidaemia dysfunction endot... 3-10 heart diseas... body fails to res... fatty liver, nona... Pipeline: ChemDis Relation: cause 0 2 Number of supporting sentences hyperlipidemia coronary artery d... cardiomyopathy dehydration weight loss coronary heart di... Pipeline: ChemDis Relation: treat 0.0 0.5 1.0 Number of supporting sentences potato bulk Pipeline: FoodChem Relation: contains correct incorrect Relations with maximal support for entity: carbohydrates Figure3. Top 10 “cause”, “treat”, and “contains” relations with maximum number of supporting sentences for four chemical entities: “carbohydrates”, “fatty acid”, “sodium” and “vitamin d”. The entities in the rows of the ChemDis pipeline are diseases caused or treated by the chemical, while the entities in the rows of the FoodChem pipeline are food entities in which the chemical is contained. 8 Vol:.(1234567890) Scientific Reports | (2023) 13:7815 | https://doi.org/10.1038/s41598-023-34981-4 www.nature.com/scientificreports/ fact that the pipelines were not able to recognize that the sentences were referring to the reduction of these food or chemical entities affecting the disease development or treatment of the general cardiovascular disorders. This is also the reason for false positive relations extraction for the other two disease entities featured in the figure. Use case: milk. For the use case related to the composition and health effects of milk, two highly-skilled domain experts evaluated the results from all three pipelines: a chemist and a food and nutritional scientist. Domain expert evaluation. From the 33,111 processed abstracts related to the milk case study, the three pipelines extracted a total of 6792 relations, from which 5139 were evaluated by the two domain experts. We need to point out again that all extracted relations were provided to the domain experts, however, they evaluated only those relations for which they have domain expertise. Figure5a features the number of relations extracted by each pipeline for the milk case study, and the number of relations the experts evaluated. The highest number of evaluated relations were the “contains” relations extracted by the FoodChem pipeline, and the experts were able to evaluate 96% of them (2849 out of 2754). The experts also evaluated 73% of the “treat” and 78% of the “cause” relations produced by the FooDis, 34% of the “cause” relations, and 35% of the “treat” relations produced by the ChemDis pipeline. The mean precision for each of the five semantic relations for both domain experts is presented in Fig.5b separately. In addition, we have also presented the mean precision for each type of relation by averaging the precision across both domain experts. From the figure, we can see that the first domain expert, who evaluated the relations which were supported by a single sentence, identified more incorrect relations than the second domain expert, who evaluated the relations supported by multiple sentences. The overall mean precision for each of the five relations averaged across both domain experts are as follows: Figure4. Top 10 “cause” and “treat” relations with maximal number of supporting sentences related to three disease entities: “general cardiovascular disorders”, “diabetes”, and “obesity”. The entities listed in the rows of the FooDis pipeline are food entities, while the entities listed in the rows of the ChemDis pipeline are chemical entities, that cause or treat the specified disease. 9 Vol.:(0123456789) Scientific Reports | (2023) 13:7815 | https://doi.org/10.1038/s41598-023-34981-4 www.nature.com/scientificreports/ • 0.51 for the “cause” relation extracted by the ChemDis pipeline, • 0.79 for the “treat” relation extracted by the ChemDis pipeline, • 0.65 for the “cause” relation extracted by the FooDis pipeline, • 0.70 for the “treat” relation extracted by the FooDis pipeline, • 0.70 for the “contains” relation extracted by the FoodChem pipeline. (a) Number of extracted andevaluated relations pereachpipeline for the milk study (b) Mean precision of the extracted relations for each of the reviewers (inblack andgreen), as well as overallmean precision (purple), for eachpipeline for the milk study. Figure5. Number of extracted and evaluated relations and mean precision of each pipeline for the milk study. The plots have been generated using the plotly python library43, version 5.7.0.