Enhancing complex XML documents with linguistic annotation

Enhancing complex XML documents with linguistic annotation Pavel Vondřička Department of Linguistics, Faculty of Arts, Charles University Prague, Czech Republic [email protected] Abstract When building corpora from annotated XML documents, the compilers are usually confronted with the incapability of most tools for linguistic analysis and parsing (tokenizers, lemmatizers, PoS-taggers, etc.) to process more than just plain text input. Various single purpose solutions have been created for this purpose. We tried to develop a general set of scripts to assist with the task of enriching documents containing complex XML annotation with linguistic annotation generated by automatic analyzers. We present the challenges we met and solutions we chose, discussing their advantages, disadvantages and limits. Keywords: annotation, TEI, XML, corpus compilation 1. Objectives When building corpora from XML documents with more or less complex annotation, the compilers are usually confronted with the incapability of most tools for linguistic analysis and parsing (tokenizers, lemmatizers, PoS-taggers, syntactic parsers, etc.) to process more than just plain text input. In addition, their output is also seldom compatible with XML. Various ad-hoc solutions have been created for this purpose at different institutions and within different projects. Within the project CLS INFRA, we tried to develop a more general solution: a set of scripts to support merging of complex XML documents with linguistic annotation generated by automatic analyzers. This task is usually hampered by several factors, such as the requirements (and limits) of the XML format and mutually conflicting requirements for annotation of different aspects of the texts. The more complex the XML annotation of the source, the more difficult is the process of combining it with additional annotation produced by tools for linguistic analysis. 2. Basic situation and requirements on the tagger The basic situation we had to solve at the beginning was not much more complex than the situation of most corpus compilers – as long as the source is just a simple XML document containing simple text elements such as paragraphs of plain text, the task is rather trivial: extract the contents of the text elements, process them by the analyser and replace them with the analyzed and annotated output, possibly also converted into an XML format. But our situation was still slightly more difficult: the paragraphs often contain text annotated with spans indicating highlighted contents that we wanted to keep. As the easiest solution, we decided to remove all the XML annotation, keep track of its original position in form of a stand-off annotation for a later reconstruction, and process the isolated plain text contents. This solution was loosely inspired by the PAULA XML format developed by the Potsdam University and the Humboldt University in Berlin, 1 using similar basic principle for complex multi-layer stand-off annotation of texts. Thus, the plain text contents was extracted from the XML source and all information about the original annotation was kept in a separate file in the form of a JSON list of tags and their attributes. Since taggers commonly do not produce an output compatible with XML, it must usually be converted before being used as replacement for the original text contents. However, we decided not to create an output directly from the output of the tagger for several reasons. Instead, we match the resulting tokens with the original plain text extracted from the XML in order to create additional layer of annotation of the original contents. The most important reason is the simple fact that the taggers often discard all information about the presence of white-space between the tokens. 2 Not just we want to keep the original white-space, but any changes in white-space would also break the recovery of the original XML annotation stored by means of their exact positions defined by the amount of characters in the original text they span (including white-space). Another issue is the occasional need of the taggers to normalize the contents to some degree. 3 The most common case is the normalization of punctua1 See Dipper (2005), Chiarcos et al. (2008) or https: //www.sfb632.uni-potsdam.de/paula.html 2The UD tagger is rather an exception, in this case. 3 In some cases, the taggers even tend to correcting "mistakes" in the source text. tion symbols, such as various typographical quotation marks defined by the Unicode standard, which usually represent the same (or similar) function from the linguistic point of view, irrespective of their particular visual form. Matching the output of the taggers with the original source text may sometimes be a complex task. The basic requirement for the tagger to output the original string of each annotated token (before any normalization) can’t always be fulfilled, especially if the tagger is developed and maintained by a different institution with different goals and priorities. Therefore, we implemented basic mechanisms to be able to match the resulting tokens with the original text strings also by means of additional replacement or transformation rules using regular expressions, applied either to the output of the tagger or the source text. These can help to identify the original source form of a normalized token as long as the output of the tagger is deterministic and predictable. In this way, taggers with somewhat normalized output may also be used. In any case, the tagger is expected to always produce some output for every non-white-space sequence of characters in the source text. On the other hand, white-space characters may also be part of the annotated tokens in the output, if desired. 3. XML document vs. its plain text contents One important difference between information provided by the XML annotation and the extracted pure text contents quickly materialized into an issue which required a solution: while XML delimits basic textual units just by means of tags, plain text can only do this by means of line-breaks, which delimit paragraphs and usually present a kind of hard breaks the tagger is not supposed to cross in the process of sentence segmentation. 4 In XML, however, the line-breaks can be present just everywhere – without marking any structural breaks – just for practical reasons. On the other hand, they may not be present between the XML elements at all. Due to this trivial difference, the plain text may contain random line-breaks anywhere in the text flow, but it may also contain no line-breaks between text elements such as paragraphs (sometimes no white-space separators at all), when the XML annotation is just mechanically removed. The simplest solution is to put a requirement on the input XML, so that it must already be preprocessed in a way that there are always line-breaks 4 In our texts, we usually don’t want the parser to span sentences across boundaries of items in a list or (possibly) verses in a poem. This is a decision depending on a particular type of texts or particular project and its goals, of course. present between the textual XML elements, but nowhere within the body of the text elements themselves. Since we wanted to make the tools more flexible and independent of the input, we integrated a solution for this problem as well. This required a significant change to the tools: to make them context aware. Since XML is a generic meta-format not specifying the names and meaning of tags used, information about the particular XML input must be provided by means of external configuration. The minimal requirement for such processing is providing it with a list of XML element names which should be treated as basic text elements usually separated by line-breaks in the visual presentation of the text. Consequently, any "technical" line-breaks within their contents can be eliminated and new provisional line-breaks can be inserted between the corresponding blocks of plain text extracted, where necessary. The information about any added linebreaks can be stored together with the information about the removed XML annotation, in order to remove such provisional line-breaks again in the last stage, when reconstructing the final XML.5 4. Extending the capabilities to process complex XML With the desire to process more complex XML documents such as TEI, 6 the context aware mode became a necessity needing even further development. Beside of the possibility to list names of text elements to be extracted and processed, there is also a need to specify elements to be completely excluded from the extraction, including all its contents. This concerns for example the <teiHeader> element containing metadata, sometimes also enclosed in common textual elements such as <p> that should not be analyzed in this case. Another typical candidate for exclusion is the TEI element <foreign>. The combination of a list of text elements and a list of excluded elements is already powerful enough for most common cases, but in order to make the possibilities even more flexible, we also implemented a simple XPath-like syntax for the specification of textual elements (or excluded elements), so that they may also be specified by 5 We want to have the resulting XML as similar to the original document as possible. Yet we decided not to keep the information about line-breaks removed from the text elements and not to recover them. Since the textual contents will be enhanced by the annotation produced by the analyzer, the purely practical purpose of the original formatting (e.g. using line-breaks for easier manual editing) would usually become pointless anyway. 6 See TEI Consortium (2025) or https://tei-c. org means of constraints on their attribute values or nesting within particular ancestors. This allows for much more detailed specification than a plain list of element names.7 5. Conversion of the tagger output into XML annotation As explained above, the output from a linguistic analyzer is not just directly converted into an XML. Instead, it is matched with the plain text data extracted from the original XML and another (independent) stand-off description of an additional layer of annotation spans and attributes is created, similar to the description of the original XML annotation extracted in the first step. This conversion must be adapted to the particular output format of the analyzer. We created a configurable parser for any vertical-based format common to most taggers, including a preprocessor for the more specific ConNLL-U format commonly used by the Universal Dependencies tagger. This parser creates <s> spans from sentence segmentation (usually delimited simply by empty lines in the vertical format) and <w> spans for the tokens identified by the tagger. Additional token attributes, such as a lemma, PoS or morphological tags, are converted into the attributes of the token span. The attribute names can be customized in the configuration; selected attributes may also be completely omitted from the final output, if desired. The CoNLL-U format produced by the UD tagger has some additional features that need special treatment. The most important distinction to a simple vertical format is the use of two-level tokenization, where subtokens can be created for tokens representing two syntactic words. We implemented two ways how to deal with such subtokens, which do not necessarily always match any identifiable part of the original text string: • attribute values of the subtokens are concatenated (using a specified separator) into the corresponding attribute values of the primary 7 Since the beginning, there was a difficult decision to be made: whether to use a fully-fledged XML parser for the extraction of the text contents, or a custom simple detection of the XML mark-up. The latter option was chosen as easier to implement and completely satisfactory for the original simple type of documents. Also, the priority to efficiently process large documents would require deployment of the more complex SAX parser instead of the simple DOM method. The former option would later easily enable the support for full specification of elements based on the XPath syntax, but it would still make the rest of the process more difficult in many other aspects. Therefore the idea to rewrite the scripts by deploying an XML parser was dismissed again. token; this solution is suitable for search engines not supporting multi-level tokenization, such as the vertical-based CQP engines; e.g. the annotation of the English token "can’t" may result into the element <w id="1|2" synword="ca|n’t" lemma="can|not" upos="AUX|PART">can’t</w> • virtual subtokens – XML elements with a configurable name and without any textual contents – are created as children of the primary token to keep their own attributes separately; this solution is suitable for XMLaware search engines such as TEITOK 8 ; e.g. the annotation of the English token "can’t" may result into a structure such as <w id="1-2" synword="Can’t" lemma="_" upos="_">Can’t<dtok form="Ca" id="1" synword="Ca" lemma="can" upos="AUX"/><dtok form="n’t" id="2" synword="n’t" lemma="not" upos="PART"/></w> The CoNLL-U format can also carry other levels of additional annotation. Lately, we added support for CoNLL-U enriched with the multi-layer annotation of named entities added by the NameTag parser9. For taggers with an output incompatible with the common vertical, a special parser (conversion tool) would have to be written, or the output would have to be converted into a vertical format first. 6. Merging the annotations and composing the resulting XML The crucial stage of the whole process (see fig. 1) is the final reconstruction of the original XML document enriched with additional layers of annotation produced by the analyzers. This process has two steps: 1) merging the original and the added annotation into one single annotation hierarchy; 2) insertion of the merged XML mark-up back into the stream of plain text contents. Both steps are currently integrated in a single script. The combination of two independent annotations is not a trivial task, since it has to comply with the requirements of the XML standard 10 and possibly also fulfil specific demands for the particular data processed. The new annotation is added layer by layer (from the topmost elements – sentences – to their terminal children – tokens) into the original XML annotation. Any conflicts need to be resolved 8 See Janssen (2016) or http://teitok. corpuswiki.org 9 Also called "CoNLL-U+NE". See https://lindat. mff.cuni.cz/services/nametag/ 10https://www.w3.org/TR/xml/ Original XML document split Plain text contents Original annotation (stand-off) tagger linguistic analysis match New annotation (stand-off) merge Merged XML document Figure 1: Workflow of the enhancement process according to the project-specific priorities. In case a newly added element crosses its parent’s span borders, the parent must be split and reopened beyond the end of the new span. Splitting original spans by new annotation layers complies well with the commonly conflicting annotation of highlighted text spans, which have a lower priority and lower demand for continuity than the added linguistic annotation with a more important structural function and they can thus be interrupted. However, in case some original structural element is unexpectedly broken by the new layers of annotation, the scripts can also issue a warning on demand. The conversion offers some other additional configurable functions for convenience: interrupted spans such as highlighting of text don’t usually need to be reopened also between sentences 11 and elements with the same name and exactly same span can be automatically merged into a single element with their attributes combined, if desired. The process of merging the independent annotations thus tries to do its best within the limits of the XML standard. However, there are cases where priorities may change. Usually, each new layer of nonoverlapping spans is preferably inserted into the old spans as their children. This works well within the scope of the secondary (added) annotation, but its relation the the primary (original) annotation may be more complex, if the original annotation includes more than just highlighting spans for em11 Highlighting the intermittent space between sentences only is usually rather pointless. phasis. For example, TEI supports annotation of deleted (corrected) contents which may appear at the very beginning or end of a token and which should also stay within the final span of the token, despite the fact they shouldn’t be sent to the tagger – this feature is frequently used e.g. in corrected texts in learners corpora. In that case, the preference can be set to include such annotation within the new annotation spans wherever possible. For example, learner corpora may contain a corrected sentence such as I will<del>e</del> write an essay. , where the original form wille was corrected to will by deleting the final letter e – and this form should also be presented to the tagger for correct linguistic analysis. For this purpose, we can add the element <del> to the list of excluded contents not to be extracted into the plain text for tagging. However, the process of merging the original and the added annotation would then still put the del -span outside of the token span: <w>will</w><del>e</del> . In such case, we can also change the nesting priority of the element <del> to obtain the desired result: <w>will<del>e</del></w>. 7. Supplemental utilities For further convenience, we also provide three additional scripts outside of the core scope of the tools: • a full-fledged command-line client to the UDPipe API 12 integrated with a client to NameTag to produce complete CoNLL-U(+NE) output from plain text files • customizable tool for conversion of the merged XML documents into vertical format suitable for indexing by CQP-like search engines; it also offers solution to the typical issues with XML features not supported by these engines, such as removal of XML annotation from the token string, flattening of nested elements with the same name or insertion of additional "glue" elements between tokens originally not separated by space • wrapper to run the whole pipeline automatically on a batch of files, showing a progress bar and cleaning up all temporary files when finished; it supports processing in multiple threads in order to improve efficiency The wrapper script does not implement a more complex utilization of multiprocessing as this task can probably be better implemented by running the 12https://lindat.mff.cuni.cz/services/ udpipe/ process in multiple batches by dedicated external tools such as GNU parallel.13 8. Configuration and customization of the process As indicated before, virtually all aspects of the processing – extraction of text contents for analysis, conversion of the tagger output and names of all the newly created XML elements and their attributes – are configurable both using command line options of the individual scripts and a common configuration file. The configuration file may contain multiple profiles (preconfigured sets of settings) for processing of different data sets. A particular profile may then be selected by a single command line option. 9. Availability and development The scripts are implemented in Python (without any unnecessary extensions) and they are available at GitHub 14 under the GNU GPL 2.0 license, including detailed documentation and practical examples. The tools have also been integrated into the VELD platform for reproducible text processing developed at the Austrian Centre for Digital Humanities and Cultural Heritage (2024).15 A more thorough testing with various types of data is still needed. We will also be grateful for any feedback and suggestions for further improvements or possibly other extensions covering more types of input data, output formats or integration of other linguistic analyzers. 10. Acknowledgements This paper and the original software resulted from the implementation of the Czech National Corpus project (LM2023044) funded by the Ministry of Education, Youth and Sports of the Czech Republic within the framework of Large Research, Development and Innovation Infrastructures. Further development of the software has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101004984 (CLS INFRA). We also want to thank Maarten Janssen, the author of TEITOK, for very useful feedback and suggestions for many important improvements necessary for processing of more advanced TEI XML documents or less common types of data. 13 See Tange (2018) or https://www.gnu.org/ software/parallel/ 14https://github.com/czcorpus/ xmlanntools 15https://github.com/veldhub/veld_code_ _xmlanntools 11. References Austrian Centre for Digital Humanities and Cultural Heritage. 2024. VELD: Versioned executable logic and data, a design pattern for reproducible and flexible workflows. Christian Chiarcos, Stefanie Dipper, Michael Götze, Ulf Leser, Anke Lüdeling, Julia Ritz, and Manfred Stede. 2008. A flexible framework for integrating annotations from different tools and tag sets. TAL, 49:217–246. Stefanie Dipper. 2005. XML-based stand-off representation and exploitation of multi-level linguistic annotation. In Berliner XML Tage 2005, Humboldt-Universität zu Berlin, 12. bis 14. September 2005, pages 39–50. Maarten Janssen. 2016. TEITOK: Text-faithful annotated corpora. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4037–4043, Portorož, Slovenia. European Language Resources Association (ELRA). Ole Tange. 2018. GNU Parallel 2018. Ole Tange. TEI Consortium. 2025. TEI P5: Guidelines for electronic text encoding and interchange.