A semantic web methodological framework to evaluate the support of integrity in thesaurus tools

A Semantic Web methodological framework to evaluate the support of integrity in thesaurus tools M Mercedes Martínez-González University of Valladolid, Spain [email protected] María-Luisa Alvite-Díez University of Leon, Spain [email protected] The final, published version of this article is available online. Please check the final publication record for the latest revisions to this article. [Martínez-González, M. M., & Alvite-Díez, M.-L. (2019). A semantic web methodological framework to evaluate the support of integrity in thesaurus tools. Journal of Information Science. https://doi.org/10.1177/0165551519837195] Abstract With the Semantic Web, thesauri recover a relevant role supporting semantic searches and other added-value services. Thesaurus standards define what constructs a thesaurus can have and the integrity rules it must comply with. Thesaurus editors can be helped in their work if thesaurus tools offer them support for integrity, warning when integrity rules are violated and/or helping them to correct these mistakes. The most recent thesaurus standard is ISO 25964, which supersedes ISO 2788, evolving towards concept-based thesauri, better aligned with the Semantic Web approach than the term-based thesauri of ISO 2788. However, the W3C recommendation for KOS (Knowledge Organization System) representation in the semantic web context is SKOS, which is in fact prior to ISO 25964. This paper focuses on thesaurus integrity and the evolution from ISO 2788 to ISO 25964. Its effect on integrity issues is analyzed. A methodological proposal for evaluating integrity support in thesaurus tools, arising from the results of this work, is presented. Its target audience is professionals in charge of thesaurus edition. Besides being adapted to the most recent thesaurus standard, ISO 25964, it also includes the comparison of ISO standards with SKOS. The paper is completed with the presentation of the results of applying it to three thesaurus tools. Keywords ISO 2788; ISO 25964; Semantic Web; SKOS; software evaluation; thesaurus integrity; thesaurus tools 1. Introduction The Semantic web and linked data have brought about a renewed interest in thesauri as conceptual tools that can be used to improve semantic interoperability [1, 2, 3, 4]. This “revival” is reflected in the update of the thesaurus ISO standard. Between 2011 and 2013, ISO published ISO 25964 [5, 6], the thesaurus standard that supersedes ISO 2788:1986 [7]. It is an evolution from the approach of term-based thesauri, present in ISO 2788, to concept-based thesauri [8, 9]. This concept-based approach was already used in SKOS [10], and is closer to the idea of conceptual searches that underlies semantic information retrieval [11, 12]. With the evolution from ISO 2788 to ISO 25964, the thesaurus community embraces the Semantic Web vision. All of this in an environment of semantic interoperability, in which thesauri are reused in open contexts. Reuse is definitely strengthened when using standards to share thesauri [1, 8]1. This new ISO standard appeared after the W3C recommendation for representing KOS in the semantic web: SKOS (Simple Knowledge Organization Systems) [13]. During the period when ISO 2788 was the available standard for thesauri, several thesauri were represented using SKOS and tools were adapted to use RDF/SKOS as the underlying language2 to store and share the thesauri their users created. However, SKOS is intended for use in a wider set of KOS. An example of its significance and relevant spread can be found in the W3C website -SKOS Datasets-3, in which thesauri, subject headings, glossaries, lists of authorities, classifications, taxonomies, nomenclatures, etc., are collected. It should be noted at this point that the thesauri with the longest trajectory and managed by prestigious international institutions, such as EuroVoc (European Union Publications Office), UNESCO Thesaurus (United Nations Educational, Scientific and Cultural Organization -UNESCO-) or AGROVOC (Food and Agriculture Organization -FAO-), among others, are represented in SKOS (see Note 1). It is worth remarking that the restrictive orientation required by vocabulary control (e.g., polysemy is not allowed, ambiguity is controlled) prevails in thesauri standards. However, SKOS has a more general vocation (purpose), as it was created not only for representing controlled vocabularies, such as thesauri, but also for less restrictive vocabularies such as folksonomies and glossaries. With this context, the comparison of the ISO thesauri standards and the SKOS recommendations when using SKOS for thesauri representation is required. The equivalence between thesauri constructs, as defined by ISO 2788, and SKOS constructs was first covered by guidelines provided in the SKOS Primer Guide [14]. The equivalence between ISO 25964 and SKOS is dealt with in [15]. Integrity is the quality that guarantees that a given thesaurus is consistent with regard to the formal definition of what a thesaurus can be (data model). Integrity is a crucial issue during the process of thesauri creation and edition, because infringing integrity rules can be done unconsciously (as shown by the results of assessing SKOS vocabularies published in [16]). In an ideal scenario, a thesaurus tool would deal with integrity in such a way that any updates that would result in integrity violation would be automatically rejected by the tool. In addition, it should be able to insert automatically any relationships that can be derived from those inserted by end users or to delete, in cascade, the relationships in which a deleted construct participates. When tools automatically check integrity, users are greatly helped in correcting errors committed when creating a thesaurus, so they can be more confident about the work done. As with ontologies [17], the interest of assisting the end user with a suitable checking tool increases as the size of the KOS being edited is greater. In this paper, a methodological proposal for evaluating integrity support in thesaurus tools is presented. This integrity evaluation is part of a general framework [18] aimed at evaluating the functionalities required for thesaurus tools. Its main public is the thesaurus community, i.e., information management professionals involved in the development, maintenance and edition of thesauri, who need reliable tools for these tasks. It is worth noting that it is more oriented towards the proper design of thesaurus management tools and their users’ experience, rather than a purely theoretical or logical perspective. This paper focuses on the integrity issue, while the previous research presented the general framework and covered 7 issues (Purpose, System requirements, Functionalities, Thesaurus constructs supported, Integrity management, Information interoperability, Software interoperability and integration). Thus, this paper provides an enhanced, indepth and detailed analysis of the integrity problem and the tests designed to evaluate its support in thesaurus tools, none of which could be included in the general overview presented in reference [18]. Besides, in this paper, the framework adaptation from ISO 2788 (term-based thesauri) to ISO 25964 (concept-based thesauri) is discussed. The tests were initially designed according to the ISO 2788 standard, which is still supported by thesaurus tools. They were later revised in accordance with ISO 25964. As for the comparison of thesaurus integrity conditions with the integrity conditions of SKOS, it is worth analyzing thesaurus issues that are also present in SKOS and those that are specific to thesauri, as this indicates which tests should be supervised more carefully in a practical evaluation. These differences provide the key to identifying tests whose results may more easily differ from what would be expected according to thesauri standards, that is, producing negative results in tools using SKOS as the underlying representation language. This is a qualitative, not quantitative methodology. There are two main reasons for this. First, the important issue for a thesaurus editor professional is to be assisted in the creation of a semantically correct KOS (which is a question with a binary answer: yes, the tool supports it/no, the tool does not support it). Second, the size of thesauri is not a problem for tools4. As already shown in our previous work [18], the most popular thesauri are loaded and treated without problems by most thesaurus tools, which means that there are no performance problems when dealing with real thesauri. The methodological framework was applied to three tools, PoolParty, SKOSEd, and VocBench to check its applicability. For this work, in which the new ISO 25964 standard is taken into account, tools already using SKOS are considered more suitable. As has been verified with such thesauri as EuroVoc, AGROVOC or the UNESCO thesaurus, and the work of the ISO working group (see [15]) shows, the tendency is to use SKOS for the representation of thesauri. While SKOSEd is, as its name indicates, a tool created to edit and manage SKOS systems, PoolParty is a tool originally designed more specifically for thesauri, while VocBench is a software tool for managing OWL ontologies, SKOS(XL) thesauri and generic RDF datasets. There are other surveys of thesaurus tools [19, 20]. In these surveys, such features as the creation and management of thesauri, and those related to software output (display of thesauri on the screen or printer, for instance), were touched upon. The proposal presented here is original due to its Semantic Web oriented approach. Moreover, this proposal deals in depth with integrity, taking into account ISO 2788 and ISO 25964. Besides, it is worth noting that, in the Semantic Web, thesauri and ontologies have different roles [21], thus the comparison with ontology tools is beyond the scope of this work. This work also differs from works that test the quality of KOS represented with SKOS [16]. While, in these latter works, the goal is to analyze the quality of the SKOS representation of controlled vocabularies (which include other KOS than thesauri) and to provide suggestions to correct the problems detected, in our methodological framework, the interest focuses on helping thesauri editors to select tools that assist them in thesaurus edition. Section 2 provides a brief introduction so as to understand the scope of integrity in order to check the relationships between constructs in a thesaurus; likewise, integrity conditions in ISO standards and the SKOS Recommendation are compared in detail. Section 3 is devoted to the methodological framework, in which an in depth review of the framework for integrity support is offered. This research takes into account the evolution from the ISO 2788 to ISO 25964 thesaurus standard. The analysis of correspondences between ISO 25964 and SKOS is also included in this section. In section 4, the results of applying the methodological proposal to the experimental evaluation of PoolParty, SKOSEd and VocBench are discussed. Section 5 collects the main conclusions obtained. 2. Thesaurus integrity Next, the main constructs of thesauri are presented, as well as the meaning and importance of integrity in a thesaurus (section 2.1). Then (heading 2.2), the integrity conditions for thesauri, both in ISO 2788 and ISO 25964, are discussed. The integrity conditions established in SKOS follow them in heading 2.3. It is useful to have the listing of SKOS restrictions, as they are helpful to support the analysis of the possibilities of finding support by default on tools using SKOS for representing thesauri, and of characterizing the thesaurus restrictions that any thesaurus tool should guarantee on its own as an added value. This analysis is given under heading 2.4. 2.1 Preliminary approach Thesauri standards establish the types of constructs that make up the parts of a thesaurus. With ISO 2788, the main constructs of thesauri are Terms: words or phrases that represent an idea. In ISO 25964, the main constructs of thesauri are Concepts. Concepts are represented by terms. There are two types of terms: Preferred terms and Nonpreferred terms. One single preferred term is chosen to represent an idea or concept. Several nonpreferred terms can be associated with a concept. Equivalence relationships relate the terms associated to the same concept: UF (Used For), for Non-Preferred Terms, and USE (Uses), for Preferred Terms. Hierarchical relationships provide the structure of a thesaurus: BT (Broader Term), NT (Narrower Term), Top Term/Top Concept. A BT concept has a scope including the scope of its narrower concepts. Note that the name of the relationships is not changed from ISO 2788 to ISO 25964, while in this last recommendation they should be read as Broader concept and Narrower concept. ISO 25964 introduces the possibility of qualifying these relationships with optional “roles”, something not possible in ISO 2788. There are three types of roles, which can be used to distinguish the three types of hierarchical relationship: instantial (instance of), partitive (part of), and generic (kind of). Detailed discussion can be found in [15, 22]. A top concept is not narrower than any other preferred term/concept in the same thesaurus. In some thesauri, there is also a multiple hierarchy or polyhierarchy. Another important type of relationships is Associative relationships: RT (Related Term), used for preferred terms/concepts that are not related by hierarchical relationships but have, however, some type of relationship (e.g., book has something to do with author, while they should not be related by an NT/BT relationship). Concepts can be grouped into microthesauri (also called subject fields, fields of knowledge or domains). It is not common, but nesting between microthesauri is possible (see, e.g., the EuroVoc5 thesaurus). ISO 25964 introduces a new construct: concept groups. Microthesauri are concept groups. Another way to group concepts in ISO 25964 is Arrays: these are groups of sibling concepts. In addition, ISO 25964 introduces the compound equivalence, which was not in ISO 2788. These are relationships used to represent compound concepts that do not exist in a thesaurus, but can be expressed as a combination of two or more simple concepts (e.g., “coal mining” is a compound of “coal” and “mining”). Ultimately, Notes serve to clarify the meaning and application of a term or concept: Scope notes, Historical notes, Editor notes, Usage notes, etc. The vocabulary of SKOS includes various elements that work together to represent KOS. These elements include concepts, labels, relationships, mapping properties, collections, and notes. When mapping to SKOS, skos:ConceptScheme, a structure that compiles and organizes a set of concepts, can be used to represent a thesaurus. Each concept is represented with skos:Concept, and each of them in turn is identified with a URI. In SKOS, a label is the descriptor of a concept. Labels can be preferred (skos:prefLabel), alternative (skos:altLabel), or hidden (skos:hiddenLabel). In SKOS, it is possible to state that a concept is a top concept by using the skos:topConceptOf property. The labels skos:broader and skos:narrower assert hierarchical relationships between concepts. Further, the skos:related label can be used to assert an associative relationship between two concepts. The SKOS Collections, skos:Collection, are groups of concepts which do not need to be related by hierarchical relationships. Members of a SKOS Collection may come from different ConceptSchemes. For the interest of thesaurus management, this means that a SKOS Collection can be used to represent domains or to group concepts from different thesauri. Regarding integrity, such standards as ISO 2788, ISO 25964, or SKOS, establish certain integrity conditions that a thesaurus, or a KOS represented with SKOS, should always respect. The integrity conditions are “statements within the formal definition of a data model, which are used to establish whether or not given data are consistent with respect to the data model” [13]. Integrity conditions determine what is, or is not, valid in a thesaurus or KOS. 2.2 ISO integrity conditions The integrity conditions for thesauri can be derived from the ISO standard and are presented next. The new ISO 25964 keeps the restrictions that could be inferred from ISO 2788. However, they have to be adapted to the change from the term-based approach to the concept-based approach. In the following, the conditions are outlined. 1. Uniqueness: there cannot be duplicated elements in a thesaurus. This means: a) Each concept or idea has a single term that represents it in each language. With ISO 2788, this implies that each non-preferred term is linked to only one preferred term. In ISO 25964, there must be only one Preferred Term for each concept in each language. b) Microthesauri (concept groups in ISO 25964) cannot be repeated. c) Concepts cannot be repeated. In ISO 2788, preferred terms cannot be repeated. d) Preferred Terms/concepts, microthesauri/groups of concepts, and arrays, are disjoint, i.e., a Preferred Term/concept cannot also be a microthesauri/group of concepts or an array. e) Preferred terms cannot be used as non-preferred terms. In other words, preferred terms and non-preferred terms are disjoint sets. 2. Only Preferred terms/concepts can participate in hierarchical and associative relationships. This means that, e.g., a Non-preferred term participating in some of these relationships is not allowed according to ISO 2788, or that a term cannot participate in these relationships if stated according to ISO 25964. 3. Some relationships are incompatible: BT and NT relationships are not compatible with RT relationships. Two preferred terms/concepts already related by a BT or NT relationship cannot be stated to be related by an RT relationship. 4. Cycles involving hierarchical relationships are forbidden. For example, if A BT B is true, B BT A is forbidden. This also holds when hierarchical and associative relationships are involved in the cycle. In addition, a top term cannot be narrower than another term. 5. Some relationships require the existence of an inverse relationship. For example, if A NT B, then B BT A should be inferred. 6. When a term is deleted, all the relationships it participates in, should also be deleted. 2.3 SKOS integrity conditions SKOS establishes its own integrity conditions, expressed in the SKOS Recommendation [13] by a set of axioms. The ones relevant for this work are included6: S9 - skos:ConceptScheme is disjoint with skos:Concept. S13 - skos:prefLabel, skos:altLabel and skos:hiddenLabel are pairwise disjoint properties. S14 - A resource has no more than one value of skos:prefLabel per language tag. S19 - The rdfs:domain of skos:semanticRelation is the class skos:Concept. S20 - The rdfs:range of skos:semanticRelation is the class skos:Concept. S27 - skos:related is disjoint with the property skos:broaderTransitive. S37 - skos:Collection is disjoint with each of skos:Concept and skos:ConceptScheme. 2.4. Comparison of integrity restrictions in ISO and SKOS A comparison of the integrity restrictions between ISO standards and SKOS is not included in the documents produced by the ISO working group. However, it is helpful to preview which thesaurus integrity restrictions could be expected to be supported in SKOS tools, and which should be an added value of thesaurus tools. Table 1 summarizes the set of similarities and differences between SKOS and ISO that are relevant for integrity checking. While it is logical to expect to find support for the restrictions covered in both standards in any tool, it is not the same for restrictions specific to thesauri. Table 1: Integrity in ISO standards and SKOS: similarities and differences. ISO restriction SKOS equivalence Cond. 1a: Only one Preferred Term for each concept SKOS axiom S14 Cond. 1b: Microthesauri/concept groups cannot be repeated Not in SKOS Cond. 1c: Preferred terms/concepts cannot be repeated Not in SKOS Cond. 1d: Preferred Terms/concepts and Microthesauri/concept groups are disjoint SKOS axioms S9 and S37 Cond. 1e: PrefTerms, Non-PrefTerms, ... disjoint sets SKOS axiom S13 Cond. 2: Only PrefTerms/Concepts participate in semantic relationships SKOS axioms S19 and S20 Cond. 3: Incompatible semantic relationships SKOS axiom S27 Cond. 4: Forbidden cycles Not in SKOS Cond. 5: Inverse relationships Not in SKOS As can be noted from Table 1: Axiom S14 guarantees that there is no more than one preferred term (property hasPreferredLabel in ISO 25964) for each concept, that is, it guarantees condition 1a. SKOS axioms S9 and S37 guarantee that a preferred term/concept cannot be, at the same time, a microthesaurus/group of concepts (condition 1d). This is true both with thesauri in which microthesauri have been represented with skos:ConceptScheme, e.g., the UKAT thesaurus, and with thesauri in which skos:Collection has been used for this aim, which is the proposal made in [15]. Axiom S13 guarantees that terms used as preferred labels are not also used as non-preferred labels (condition 1e), or, in the ISO 2788 form, that a preferred term is not also a non-preferred term. Axioms S19 and S20 guarantee that only preferred terms/concepts participate in semantic relationships (condition 2). Axiom S27 ensures the incompatibility of basic relationships (condition 3). However, as mentioned before, it is interesting to detect the integrity conditions from thesauri that are not included in SKOS, as they are the ones that will more probably determine the differences between a SKOS-oriented tool and a thesaurus-oriented tool. In addition, the introduction of the relations USE+ and UF+ between terms to represent compound concepts in ISO 25964 adds a new challenge for thesaurus tools. The ISO restrictions that cannot find an equivalent restriction in SKOS, presented in Table 1, are interpreted as follows: 1. In SKOS, the repetition of labels does not imply a duplicity of concepts; hence, even if not recommended, duplicity is not restricted [14, Section 2.2.1, Appendix]. For example, if two SKOS concepts share the same Preferred label, it is not inconsistent in SKOS. However, two concepts sharing a Preferred term should not be possible in a thesaurus. Consequently, conditions 1b and 1c do not have an equivalent restriction in SKOS. Even if some works about SKOS validation have considered this possibility as an improvement in quality checking of KOS, of which their editors are warned [16, 23], this is not the same as a restriction in the sense established in ISO thesauri standards. SKOS does not have an equivalent integrity restriction and, consequently, it has not been assumed by SKOS tools. 2. In SKOS, cycles in hierarchical relationships are possible, in opposition to condition 4 of thesauri. For example, A skos:broader B and B skos:broader A (see section 8.6.8 of [13]) is consistent in SKOS, something that would be inconsistent in an ISO valid thesaurus. 3. In SKOS, there is no integrity condition associated to the fact that skos:narrower and skos:broader are inverse relationships. There are only logical dependencies that permit the presence of a statement such as A skos:narrower B entails B skos:broader A to be inferred (see section 1.5 of [13]). However, the presence of only one of them is consistent with the SKOS data model, which works with the “open world assumption”. That is, the integrity condition 5 is not required by the SKOS recommendation. 4. In SKOS, only concepts can participate in relationships, so it is not possible to relate labels, as this would imply the USE+ and UF+ relationships of ISO 25964. In SKOS-XL, labels are promoted to objects, which implies that they can be related; this is closer to the possibility of relating terms as considered in ISO 2594. However, even this extension of SKOS has limitations to offer the full power of ISO 25964 [24]. In short, a SKOS tool should not reject a KOS (Knowledge Organization System) with cycles or repeated labels, even if it checks and warns about these quality issues, as some tools already do [16, 23]. Indeed, cycles can be correctly included in some KOS (let us consider, for example, classifications, glossaries, etc.), and are not even a reason for warning. However, a thesaurus tool should reject a thesaurus with some of these problems. In addition, some entailments that should be automated for thesauri, such as the inclusion of reciprocal NT/BT relationships, should not be generalized for KOS. A thesaurus tool should automatically add reciprocal relationships, which cannot be requested of a KOS tool. In consequence, the type of tool used conditions the expectations a thesaurus editor should have during the process of creating/editing a thesaurus concerning the support provided by the tool. 3. Methodological framework The target issue of this paper is integrity management, a crucial issue to ensure the correctness and quality of any thesaurus. The methodological framework, as has already been pointed out, has its starting point in a previous proposal [18]. Now, however, a review of the framework for integrity support is offered. This research takes into account the ISO 25964 thesaurus standard, which was not available when the previous framework was developed. Section 3.1 presents the methodology followed for the adaptation of the evaluative framework. In heading 3.2, the changes from ISO 2788 to ISO 25964 are analyzed. In section 3.3, the tests in the integrity suite are shown, while heading 3.4 introduces the thesauri used for the evaluation. 3.1 Methodology for the adaptation of the evaluation framework from ISO 2788 to ISO 25964 ISO 25964 was used as a guide for the revision carried out. Other possibilities could have been based on the revision of the thesaurus tools. However, as the adaptation to the ISO 25964 standard is still in progress, using the standard as a guide was the only guarantee to obtain a revision in which all changes are included, irrespective of whether they have already been reflected in the tools. The steps followed for the revision are: 1. Revision of ISO 25964 and the publications treating the main novelties it includes with respect to ISO 2788. 2. Identifying the set of constructs introduced in ISO 25964 for the first time, or that have changes with respect to the previous standard. 3. Locating the integrity rules present in ISO 25964. 4. Updates to the integrity rules already present in ISO 2788 to be compliant with ISO 25964. 5. Comparison of ISO 25964 integrity rules and SKOS rules. 6. Revision of integrity tests to reflect the conclusions from the previous steps. 3.2 Analysis of the changes needed to adapt the integrity suite to ISO 25964 Some modifications from ISO 2788 to ISO 25964 have involved changes in the integrity suite that are really significant. With ISO 25964, it is necessary to check the elimination of constructs whose presence becomes more relevant in this new standard: arrays, compound equivalence, and terms. It is necessary to check that their deletion is carried out in an ordered manner, i.e., the relationships they participate in are deleted in cascade. The new tests, 15, 16, 19, and 20, have been included in the suite (see Table 2). desired situations and the responsibility of the consistency of the thesaurus is transferred to the editor him/herself. 4.3. VocBench VocBench11 is a web-based, multilingual, collaborative development platform for managing OWL ontologies, SKOS(XL) thesauri and generic RDF datasets. VocBench uses Semantic Turkey, an open-source platform for Knowledge Acquisition and Management, from the ART Research Group at the University of Rome Tor Vergata. The tests have been applied to VocBench 3, the version released in September 2018. The results for VocBench are shown in the sixth column of Table 3. This tool supports some tests referring to the ISO 25964 that did not support the previous ones, such as those related with arrays. The integrity check in VocBench is carried out in two ways, at the time of creation of the construct, which is reflected in the M+I values in Table 3, or through a specific menu called “Integrity Constraint Validator”, which collects a set of validation items that can be used to assess some integrity constraints, for example “Concepts related disjoint relations”, “Hierarchical redundancies”, or “Cyclic hierarchical concepts”. When the integrity validation was offered by means of this menu, it has been reflected in the table with the value ‘M’. According to the help of the program, the inference of relations in VocBench depends on the RDF store used. In the tests executed for this proof of concept, the implementation has been made following the default settings offered by VocBench, which are adequate for introducing the Minimal ad-hoc thesaurus (Figure 1 (b)). The support of integrity checking in VocBench is quite complete. Perhaps the most remarkable observation concerns its management of cycles. There is no option in this tool to detect the type of hierarchical cycles introduced in test 10, which concerns a combination of RT and NT relationships to make up a cycle. Nevertheless, an option appears in the “Integrity Constraint Validator” menu for validating the integrity when a cycle exits/occurs between two concepts related by skos:related and skos:broaderTransitive relationships. This is a bit puzzling because it is possible, in this tool, to use the properties skos:related and skos:transitive to obtain the type of cycle tested in test 10. Even if this does not imply that the tool fails in this test (as the inference offered by the RDF store used may be able to detect the cycle introduced with the skos:related - skos:broaderTransitive check), we believe that offering the possibility of checking integrity on the same properties used during the creation of the thesaurus, skos:related and skos:transitive in this case, would facilitate the end-user’s understanding of the functionality of the application. Finally, a reasonable behavior of VocBench is observed in the tests that affect the deletion actions. The application rejects deletions that require integrity and, in other cases, presents a complete set of validation items inside the menu “Integrity Constraint Validator”. 4.4. Analysis Some tests prepared for the new relevant constructs in ISO 25964, such as test 18, are not applicable in the tools evaluated (they have been marked as 'Not appl.' in table 3), as the tools were still not completely adapted to the ISO 25964 standard at the moment of these proofs. In fact, ISO 25964 is extremely wide, and more restrictive and specific than SKOS. The correspondences of the new constructs of ISO 25964 to SKOS constructs (it should be borne in mind that SKOS is used by these tools to represent and store thesauri) are not as clear as they are for basic constructs that were already in ISO 2788, and whose correspondences were documented in [14]. A question that could arise is: Is it worth including in the framework tests that cannot be applied to the tools evaluated? The answer is that it is worth having these tests in the framework: the difficulties to implement ISO 25964 in its entirety cannot condition the definition of the framework. Conversely, the framework should conform to the standard and be prepared for the moment when the tools will be ready to support ISO 25964 completely. After all, it is a framework to evaluate the support of ISO standards in thesaurus tools. Table 3: Integrity support in PoolParty, SKOSEd and VocBench. Test ISO 2788 ISO 25964 PoolPart y SKOSE d VocBench 1 2 PrefTerms for 1 NonPrefTerm 2 PrefLabels for a concept M+I M+I M+I 2 Microthesaurus = Microthesaurus Concept group = Concept group M-I M-I M+I 3 PrefTerm = PrefTerm Concept = Concept M M-I M+I 4 PrefTerm = Microthesaurus Concept = Concept group M-I (Not Appl.) M 5 PrefTerm = NonPrefTerm PrefLabel = NonPrefLabel M+I M-I M+I 6a RT/NT/BT + NonPrefTerm and Non PrefTerm RT/NT/BT + Term M+I M+I M+I 6b NT/BT/RT + PrefTerm and NonPrefTerm RT/NT/BT + Term M+I M+I M+I 7 RT + NT/BT RT + NT/BT M+I M-I M 8 NT + BT NT + BT M+I M-I M 9 NT + Top Term NT + Top Concept M+I M-I M 10 NT + RT (cyclic) NT + RT (cyclic) M M-I M 11 NT + NT (cyclic) NT + NT (cyclic) M+I M-I M 12 Delete NT/BT Delete NT/BT M+I M M+I 13 Delete RT Delete RT M+I M M+I 14 Delete PrefTerm Delete concept M+I M M+I 15 Delete NonPrefTerm Delete (Not (Not (Not SimpleNonPrefTerm Appl.) Appl.) Appl.) 16 Delete Microthesaurus Delete concept group M+I M M+I 17 Delete Top term Delete top concept M+I M M+I 18 Delete array M+I (Not Appl.) M+I 19 Delete compound concept (Not Appl.) (Not Appl.) (Not Appl.) 20 Delete submicrothesaurus Delete SubGroupOf/- SuperGroupOf rel (Not Appl.) (Not Appl.) (Not Appl.) Testing three different tools has helped to corroborate the interest of comparing the integrity restrictions of ISO 25964 and SKOS. In fact, some results of the experiments on these tools (differences in behavior in some tests) had been foreseen as possible, due to the comparison discussed in section 2.4. Moreover, thanks to that comparative analysis, an explanation about these differences in behavior was already available. The experiments have confirmed the suspected limitations with new constructs and structures in ISO 25964, or that get a greater relevance than they had in ISO 2788: collections, polyhierarchy, arrays, compound concepts, etc. PoolParty has a good integrity support, well aligned with thesaurus standards, and the user interface is clear and intuitive. Limitations are found in the duplication of labels when collections are involved, and in the consequences of the policy chosen for deletions when polyhierarchy is present in the thesaurus. SKOSEd does not guarantee some integrity that would be expected in thesaurus. Two reasons explain this. First, the SKOS recommendation is less restrictive than thesaurus recommendations; this comparison was developed in section 2.3. The second one is related to the limitations of the OWL SKOS ontology loaded in Protégé: some SKOS axioms are missing in the SKOS ontology that is effectively loaded on Protégé, a lack which is itself related to some limitations of OWL [28, 29]. The analysis of OWL limitations is beyond the scope of this paper, for which the interested reader is referred to these references and literature concerning OWL reasoning. VocBench supports new constructs of ISO 25964, such as arrays and groups of concepts, besides offering a set of validation utilities that can be used individually and an extensive and suitable list, although it does not match exactly with the tests used here. Checking the results of tests 6a, 6b and 7 would probably be easier for the end-user if the inverse relationships inferred were immediately presented in the interface that shows the description of each construct. This would make it easier for the end-user to check they have been correctly inferred. Finally, we consider that the test suite designed to check the integrity management in the thesaurus editing tools has been effective and therefore allows us to know the degree of help these tools provide the user to guarantee the consistency and quality of the thesaurus. 5. Conclusions This paper presents a methodological evaluation framework for integrity support in thesaurus tools, whose target community is thesauri editors, concerned with obtaining thesauri that are compliant with pertinent thesauri recommendations. It is oriented towards the design of thesaurus management tools and their users’ experience. Integrity is one of the more important issues when creating and editing thesauri, as violating integrity rules can be done unconsciously. Thus, the help provided by tools in detecting possible violations and preventing them is valuable, as this will contribute to the quality of the thesauri created with them. Being able to verify what help can be obtained from a prospective tool to be used for thesaurus editing can itself be helpful for thesaurus editors confronted with the task of selecting a tool. This is what is offered in this framework. In addition, this work can be relevant for SKOS or thesaurus developers interested in crossusing their abilities, applying a SKOS tool to take care of, or publish, a thesaurus in the semantic web or vice versa. This is a qualitative methodology, which is justified by two main arguments. First, the relevant issue for its target public (thesaurus editor professionals) is being able to detect and correct inconsistencies in the thesaurus being edited. Whether the tool used is capable or not of helping them in this task is the relevant question. Second, as has already been argued in the Introduction, the size of a thesaurus is not a problem for thesaurus management tools. The inclusion of SKOS and ISO 25964 is a relevant novelty of this proposal, as compared with some previous proposals related to the evaluation of thesauri or thesauri tools and general frameworks for the Semantic Web. This is a guide for comparing thesaurus tools that thesauri editors can apply directly, something which is otherwise not available. Even if SKOS is used by the tools evaluated to represent thesauri, it is important to remark that this proposal is not conceived for SKOS tools, but for thesauri tools. A proposal for SKOS tools should differ in the issues evaluated and the tests, and the differences in integrity would be significant. The analysis carried out shows that some issues characteristic of thesauri, such as restrictions in cycles, are not well supported when SKOS is used for their representation. This was also verified with the experimental evaluation on SKOSEd and PoolParty. This offers some indications of the effort needed in the mapping of ISO 25964 and SKOS. This proposal was first designed taking into account the ISO 2788 standard, which is still supported by thesaurus tools, and revised to conform to ISO 25964. The changes that ISO 25964 introduces with respect to ISO 2788, and the modifications it implies on this framework, have been analyzed. It was important to keep both perspectives in parallel, as a complete adaptation of thesauri and tools to ISO 25964 may be a long process. The future will show how thesauri tools adapt to the new ISO standard. The proposal made in this paper would find an ideal complement in the study of functionalities and user interaction with thesaurus management tools. We consider the powerful approach of VocBench to support the collaborative work essential in the edition of large thesauri. The support of integrity in these tools should always come with good quality in user interfaces. This is considered as future work. Notes 1. For example, EuroVoc, the thesaurus maintained by the EU Publications Office, is used in a large number of regional parliaments adapted to the specific needs of each institution related to indexing and retrieval, that is, they spawn variants or customizations of the original thesaurus. Adaptations or variants of the ERIC Thesaurus are created for various educational scenarios, as significant political or geographical differences should be addressed in the vocabulary, while variants from the UNESCO thesaurus, in a wider thematic area, serve as additional examples. These examples illustrate that reusing shared thesauri is indeed a common practice. 2. The term “language” is used here with the same meaning that the SKOS Recommendation uses it in its Introduction: "SKOS also provides a lightweight, intuitive language for developing and sharing new knowledge organization systems" [13]. 3. SKOS Datasets. W3C. https://www.w3.org/2001/sw/wiki/SKOS/Datasets 4. The size of some of the most well-known and used thesauri illustrate this: the UKAT thesaurus has 13,976 descriptors (preferred terms) and 6,639 non-preferred terms, and the file size is 9.9MB; EuroVoc, the thesaurus maintained by the Publications Office of the European Union, has 6,883 preferred terms and 8,348 non-preferred terms, the file being 20.4MB. 5. http://eurovoc.europa.eu/ 6. We have considered all those rules that can be used to validate the correction of the data according to the data model, even though they are not listed in the SKOS Recommendation as “Integrity Conditions”. 7. http://www.poolparty.biz/ 8. Date of evaluation: February 2017. 9. https://code.google.com/p/skoseditor/ 10. There is no equivalent property for related Concepts in the SKOS Recommendation, i.e., there is no skos:relatedTransitive property. 11. http://vocbench.uniroma2.it/ Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. References [1] García-Marco FJ. The evolution of thesauri and the history of knowledge organization: between the sword of mapping knowledge and the wall of keeping it simple. Brazilian Journal of Information Science: Research Trends 2016; 10(1): 111. doi: 10.22566/brajis.v10i1.5786 [2] García-Marco FJ. Enhancing the Visibility and Relevance of Thesauri in the Web: Searching for a Hub in the Linked Data Environment. Knowledge Organization 2016; 43(3):193-202. [3] Isaac A. and Baker T. Linked data practice at different levels of semantic precision: The perspective of libraries, archives and museums. Bulletin of the Association for Information Science and Technology 2015; 41(4): 34-39. doi:10.1002/bult.2015.1720410411. [4] Mastora A, Peponakis M and Kapidakis S. SKOS concepts and natural language concepts: An analysis of latent relationships in KOSs. Journal of Information Science 2016; 43(4): 492 - 508. doi:10.1177/0165551516648108. [5] ISO 25964-1:2011 Information and Documentation - Thesauri and Interoperability with Other Vocabularies - Part 1: Thesauri for Information Retrieval. Geneva, International Organization for Standardization, 2011. [6] ISO 25964-2:2011 Information and Documentation - Thesauri and Interoperability with Other Vocabularies - Part 2: Interoperability with other vocabularies. Geneva, International Organization for Standardization, 2013. [7] ISO 2788:1986 Documentation - Guidelines for the Establishment and Development of Monolingual Thesauri. Geneva, International Organization for Standardization, 1986. [8] Clarke SGD and Zeng ML. From ISO 2788 to ISO 25964: The Evolution of Thesaurus Standards towards Interoperability and Data Modeling. Information Standards Quarterly (ISQ) 2012; 24(1): 20-26. doi: 10.3789/isqv24n1.2012.04. [9] Pastor-Sánchez JA. Proposal to represent the UNESCO thesaurus for the Semantic Web applying ISO-25964. Brazilian Journal of Information Science: Research Trends 2015; 9 (2): 1-8. doi:10.5016/10.5016/1981-16. [10] Pastor-Sánchez JA, Martínez-Méndez FJ and Rodríguez-Muñoz JV. Advantages of thesauri representation with the Simple Knowledge Organization System (SKOS) compared with proposed alternatives. Information Research 14 (4), http://informationr.net/ir/14-4/paper422.html (2009, accessed 4 April 2017). [11] Hildebrand M, Ossenbruggen JV, Hardman L and Jacobs G. Supporting subject matter annotation using heterogeneous thesauri: A user study in web data reuse. International Journal of Human-Computer Studies 2009; 67(10): 887-902. doi:10.1016/j.ijhcs.2009.07.008. [12] Jurisica I, Mylopoulos, J and Yu, ESK. Ontologies for knowledge management: An information systems perspective. Knowledge and Information Systems 2004; 6(4): 380-401. doi:10.1007/s10115-003-0135-4. [13] Miles A and Bechhofer S. SKOS Simple Knowledge Organization System Reference. W3C Recommendation 18 August 2009, http://www.w3.org/TR/2009/PR-skos-reference-20090818/ (2009, accessed 22 March 2017). [14] Isaac A and Summers E. SKOS Simple Knowledge Organization System Primer. W3C Working Group, Note 18 August 2009, http://www.w3.org/TR/skos-primer/ (2009, accessed 25 March 2017). [15] Isaac A. Correspondence between ISO 25964 and SKOS/SKOS-XL models, http://www.niso.org/apps/group_public/document.php?document_id=12351 (2013, accessed 6 April 2017). [16] Suominen O and Mader C. Assessing and improving the quality of SKOS vocabularies. Journal of Data Semantics 2014; 3(1): 47-73. doi:10.1007/s13740-0130026-0. [17] Noy NF, Sintek M, Decker S, Crubézy M, Fergerson RW and Musen MA. Creating Semantic Web Contents with Protégé-2000. IEEE Intelligent Systems 2001; 16(2): 60-71. doi: 10.1109/5254.920601 [18] Martínez-González MM and Alvite-Díez ML. On the evaluation of thesaurus tools compatible with the semantic web. Journal of Information Science 2014; 40(6): 711722. doi:10.1177/0165551514545603. [19] Severino F. The Term Development in the Thesauri of International Organisations. The European Journal of Developmental Research 2007; 19: 327-351. doi:10.1080/09578810701289261. [20] Will LD. Software for Building and Editing Thesauri, http://www.taxobank.org/content/thesauri-and-vocabulary-control-thesaurussoftware (2017, accessed 22 March 2017). [21] Kless D, Milton S, Kazmierczak E and Lindenthal J. Thesaurus and ontology structure: Formal and pragmatic differences and similarities. Journal of the Association for Information Science and Technology 2015; 66 (7): 1348-1366. doi:10.1002/asi.23268. [22] Alexiev V, Isaac A and Lindenthal J. On the composition of ISO 25964 hierarchical relations (BTG, BTP, BTI). International Journal on Digital Libraries 2016; 17(1): 39-48. doi:10.1007/s00799-015-0162-2. [23] Mader C, Haslhofer B and Isaac A. Finding quality issues in SKOS vocabularies, arXiv:1206.1339 (2012, accessed 25 February 2017). [24] Pastor-Sánchez JA. ISO-THES: ampliando SKOS a partir de la norma de tesauros ISO 25964. Anuario ThinkEPI 2013; 7: 189-193. [25] Will L. The ISO 25964 data model for the structure of an information retrieval thesaurus. Bulletin of the American Society for Information Science and Technology 2012; 38(4): 48–51. doi:10.1002/bult.2012.1720380413 [26] Stellato A, Rajbhandari S, Turbati A, et al. Vocbench: A web application for collaborative development of multilingual thesauri. In: Gandon F, Sabou M, Sack H, D’Amato C, Cudré-Mauroux P and Zimmermann A (eds.) The Semantic Web. Latest Advances and New Domains - 12th European Semantic Web Conference, ESWC 2015, Portoroz, Slovenia, May 31 - June 4, 2015. Proceedings, Vol. 9088 of Lecture Notes in Computer Science. Berlin: Springer, 2015, pp. 38-53. doi:10.1007/978-3319-188188 3. [27] Schandl T and Blumauer A. Poolparty: SKOS thesaurus management utilizing linked data. In Aroyo L, Antoniou G and Hyvönen E, et al. (eds.) The Semantic Web: Research and Applications, 7th Extended Semantic Web Conference, ESWC 2010, Heraklion, Crete, Greece, May 30 - June 3, 2010, Proceedings, Part II, Vol. 6089 of Lecture Notes in Computer Science. Berlin: Springer, 2010, pp. 421-425. doi:10.1007/978-3-642-13489-0 36. [28] Baker T, Bechhofer S, Isaac A, Miles A, Schreiber G and Summers E. Key choices in the design of simple knowledge organization system (SKOS). Journal of Web Semantics 2013; 20:35-49. doi:10.1016/j.websem.2013.05.001. [29] Knublauch H. Where OWL fails, http://composing-thesemanticweb.blogspot.com.es/2010/04/where-owl-fails.html (2010, accessed 30 January 2017)