Go to originalLocalized version was produced by AI. It may contain inaccuracies.
Preparing document pages…
Page 1
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ
Vytautas Magnus University
MORPHOLOGICALLY AND SYNTACTICALLY ANNOTATED LITHUANIAN CORPORA
KEYWORDS: corpus, automatic morphological analysis, automatic syntactic analysis, language technologies.
INTRODUCTION
For 25 years, language technologies have been developed and advanced at the Centre of Computational Linguistics (KLC)1 of Vytautas Magnus University (VDU) (Utka et al. 2017). The Lithuanian language resources and tools prepared at VDU KLC provide the foundation essential for computerizing Lithuanian and adapting new technologies to it. In early 2015, Lithuania became a full member of CLARIN ERIC, and in July 2015 VDU KLC and the Faculty of Informatics of VDU, together with partners from Vilnius University and Kaunas University of Technology, established the CLARIN-LT consortium and launched the project “Lithuania’s Membership in the International Research Infrastructure – Common Language Resources and Technology Infrastructure”2, thus joining the common European language technology system.
The aim of this article is to present two Lithuanian language resources prepared at VDU KLC: morphologically annotated corpus MATAS and syntactically annotated corpus ALKSNIS. These resources are publicly available in the CLARIN-LT repository; searches can be performed using the ANNIS system3 (see Section 3 for more details). Searches in the automatically morphologically annotated corpus can also be performed on the website http://corpus.vdu.lt4.
Annotated corpora are key resources that play an important role in advancing language technologies. Grammatical analysis and statistical data about language are essential for computerizing language and developing computational tools for automatic language analysis. Thus, corpora enriched with grammatical annotations serve as raw material for further automatic language analysis. Such corpora are used to create other natural language resources and tools in fields such as
1 Available online: http://tekstynas.vdu.lt.
2 Available online: http://clarin-lt.lt/.
3 Available online: http://158.129.51.247:8080/annis-gui-3.4.4/.
4 This is an automatically morphologically annotated corpus consisting of 208 million words. It is not discussed further in this article. The focus is mainly on the 1.6-million-word morphologically annotated corpus MATAS, which has been reviewed by a linguist.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 2
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 2
automatic speech recognition systems, machine learning, automated translation, information extraction, and so on.
The article next presents the aforementioned corpora separately and describes how to search their data using the ANNIS system. Researchers and students studying the system of Lithuanian grammar, lecturers and teachers preparing assignments, and anyone interested in language technologies can use the corpora and their data.
ANNOTATED LITHUANIAN CORPORA AND THEIR COMPILATION FEATURES
1. Morphologically Annotated Corpus MATAS
Morphological analysis is the first stage of language processing and often prepares a text for subsequent syntactic or semantic analysis. It is therefore very important that morphological annotators operate as accurately as possible, since errors made at the morphological level hinder automatic language analysis at other levels.
The first morphological annotator for Lithuanian was developed about 25 years ago. Its author was Vytautas Zinkevičius (see Zinkevičius 2000 for more details). At present, two morphological annotators for Lithuanian are publicly available: the aforementioned program by V. Zinkevičius, usually called Lemuoklis5 (from the word lemma—the citation form), and the annotator available through the portal Lithuanian Information System for Syntactic and Semantic Analysis (hereafter semantika.lt). In this article, the MATAS corpus is presented as annotated using Lemuoklis, while the syntactically annotated corpus was prepared using the morphological annotator from semantika.lt.6
Although Lemuoklis works quite well, the program cannot be updated because it is closed-source. The Semantika.lt annotator was developed on the Hunspell platform as an open-source program; it is regularly updated and expanded with new words (it now covers 171,000 lemmas), and its rules are maintained (see Dadurkevičius 2017 for more details). A study conducted in 2017 to evaluate the quality of both annotators found that the semantika.lt annotator performs more accurately (Kapočiūtė-Dzikienė et al. 2017).
The MATAS corpus, presented below, was compiled in 2002–2014. It was based on a one-million-word corpus compiled in 2006, which was supplemented with new texts and revised. First, the texts in MATAS were automatically annotated morphologically; they were later reviewed and corrected by a linguist.
5 Available online: http://tekstynas.vdu.lt/page.xhtml?id=morphological-annotator; this program not only annotates text morphologically but can also synthesize, i.e., generate the required forms; it is also adapted for analyzing old writings.
6 Available online: http://semantika.lt/SyntaticAndSemanticAnalysis/Analysis.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 3
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuo ti
tekstynai | 3
MATAS consists of 1.6 million texts drawn from documents, works of fiction, and scholarly texts. As in the Corpus of Contemporary Lithuanian, the largest share (36 percent) consists of journalistic texts (see Figure 1).
FIG. 1. Structure of the morphologically annotated MATAS corpus (percentages)
MATAS is available in several formats: the so-called KLC format, which uses somewhat unusual abbreviations for parts of speech and grammatical categories (the corpus available in the http://clarin-lt.lt/ portal repository is also annotated in this format), for example:
MATAS is also annotated in the international TEI P5 format. In this format, parts of speech and grammatical categories are abbreviated using a single letter or digit; for example, vatstdv3 means, respectively:
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 4
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 4
verb, finite form, affirmative form, reflexive, indicative mood, past frequentative tense, singular, third person. The same text excerpt is given below in TEI P5 format:
Another format in which morphologically annotated texts are available on the http://corpus.vdu.lt website is based on the example of the MULTEXT-East7 format. In this format, each part of speech has a different number of morphological categories (from 2 to 14), which are written as abbreviations. For example, in the tag Ncmpnn- N stands for a noun, c for a common noun, m for masculine gender, p for plural, n for nominative, and n for a non-reflexive noun. The hyphen at the end indicates that no semantic tag is assigned to this word, for example, universitetai8.
In the third section, we will present searches in both the MATAS and ALKSNIS corpora using the ANNIS system. It uses another format for morphological tags, based on Leipzig glossing conventions (see Section 3 for more details).
Different annotation systems are characteristic not only of Lithuanian language technologies. This problem prompted the development of the conversion tool Pepper (Zipser et al. 2010). This reflects the various stages in the development of resources and the annotation tools used.
As the examples above show, four systems of morphological tags are used in KLC. Several different annotation formats emerged because the morphologically annotated corpus was being compiled over a long period. At one point, no annotation standards were applied at all; instead, the abbreviations for parts of speech and grammatical categories created by V. Zinkevičius, the developer of the morphological analyzer, were simply used. They are easy for users to understand. Later, through international collaboration, the morphologically annotated corpus was re-annotated using
7 Available online: http://nl.ijs.si/ME/V4/msd/html/index.html.
8 For more information about these tags, see http://corpus.vdu.lt/lt/morph.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 5
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 5
international standards. As mentioned, the morphological analyzer of semantika.lt (the main aim of this system was to create online services) uses tags in the MULTEXT-East format. The tags are short but difficult for people to understand; they are more suitable for computer analysis.
The ALKSNIS morphologically and syntactically annotated corpus is annotated in the MULTEXT-East format. The corpus search tool ANNIS (see Section 3) made it possible to combine the morphological annotation formats of ALKSNIS and MATAS. The initial idea was to use Universal Dependency (UD) tags, since this is a fairly widespread annotation system and is easy to read (in English). However, UD tags are very long, which proved inconvenient when searching in ANNIS: each word’s tags take up a lot of space, so when analyzing the context of the word being searched for, it was necessary to scroll the window in one direction or the other. Because of this practical problem, it was decided to adapt the Leipzig glossing tags, since they are easy to read and fairly short. Despite the variety of annotation tags described here, it should be emphasized that the ANNIS interface for both corpora uses only one system (adapted Leipzig glossing), which should make things much easier for users.
It should be noted that MATE does not specify sentence boundaries. These can be identified quickly and automatically, but the results need to be checked. MATAS, like the corpus ALKSNIS discussed below, is available in the http://clarin-lt.lt/portal repository9. It can be searched using the ANNIS system10 (see Section 3 for more details).
2. Syntactically annotated corpus ALKSNIS
The syntactically annotated corpus of Lithuanian was prepared in 2016. Its preparation was supported by the Common Language Resources and Technology Infrastructure European Research Infrastructure Consortium (CLARIN ERIC) (MTI-02/2015). The corpus is publicly available in the http://clarin-lt.lt/portal repository11. Instructions for using it are provided in the user guide12.
The corpus was designed as a benchmark for Lithuanian syntactic analysis. Such corpora are typically small and optimally balanced by genre, and their syntactic analyses, although performed automatically, are reviewed and corrected by linguists. Based on a syntactic analysis benchmark, it is possible to develop a statistically-based parser (statistically-based parser) and expand automatic syntactic analysis (Bielinskienė et al. 2016).
9 Available online: https://clarin.vdu.lt/xmlui/handle/20.500.11821/9.
10 Available online: http://158.129.51.247:8080/annis-gui-3.4.4.
11 Available online: https://clarin.vdu.lt/xmlui/handle/20.500.11821/10.
12 Available online: https://youtu.be/PIEOPWurb4Y.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 6
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 6
Since ALKSNIS is a newer Lithuanian-language resource than MATAS and is less thoroughly described, this article will focus more on ALKSNIS and also discuss questions that arose during syntactic annotation.
2.1. Corpus components
The ALKSNIS corpus consists of 30,599 words and 2,355 syntactically annotated sentences. Compared with corpora of the languages of neighboring countries, this corpus is quite similar and can be considered large enough: the Latvian corpus13 consists of 3,800 sentences and 53,000 words (Predkalnina et al. 2016); the Estonian corpus 14 comprises 1,400 sentences and 10,600 words (Muischnek et al. 2014); the Polish corpus 15 comprises 8,227 sentences. The first syntactically annotated corpus, PENN (USA), is considerably larger, containing about 3 million words16.
The ALKSNIS texts cover four genres (see Fig. 2). The general and specialist periodicals and fiction sections are roughly equal in size, while the administrative-text section is the smallest, since this is a specialized form of Lithuanian: sentences are often long and contain many document titles, references to other documents, and so on. Annotating such sentences would not reveal a typical syntactic structure.
FIG. 2. Composition of the syntactically annotated ALKSNIS corpus
Complete, unabridged texts were used to compile the corpus. Sentences were selected from as wide a variety of text genres as possible. The fiction section consists of sentences from works by Lithuanian authors. The selected sentences were prepared for computer analysis: text encoding was checked, and tables and figures were deleted.
13 Available online: http://sintakse.korpuss.lv/index.html.
14 Available online: https://metashare.ut.ee/repository/browse/estonian-treebank/4eb86e5e463411e2a6e4005056b400242c46832883754ad7bd89d07f69d6a0fc/.
15 Available online: http://zil.ipipan.waw.pl/Sk%C5%82adnica.
16 Available online: https://catalog.ldc.upenn.edu/ldc99t42.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 7
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 7
2.2. Corpus annotation and visualization tools
Automatic syntactic analysis is based on the parser developed during the VDU KLC project “A Lithuanian syntactic-semantic analysis system for corpora, the Lithuanian internet, and public-sector applications,” which was created in Haskell17. The parser uses a rule-based method (rule-based). This tool reads files morphologically annotated by the segmentation and morphological analysis modules of semantika.lt and outputs sentences divided into clauses or phrases, generates syntactic trees, and specifies their thematic roles (thematic roles) and syntactic functions (Boizou et al. 2014: 69).
Online services generate only JSON files, but another KLC tool converts the results of syntactic analysis into PML (Prague Markup Language) format. Dependency trees (dependency trees) are generated for the syntactically annotated corpus. This format allows dependency trees in PML format to be visualized and edited using the TrED editor developed by UFAL at Charles University in Prague18 (see Fig. 3). All automatically annotated sentences have been checked and corrected by linguists. Thus, syntactic analysis comprises three levels: automatic word and sentence segmentation and morphological analysis, automatic syntactic analysis (using a rule-based method), and manual sentence processing (see Section 2.4 for details).
2.3. ALKSNIS data structure
After automatic syntactic analysis, sentences are presented graphically as trees (see Fig. 3). Each node in a tree corresponds to a word, punctuation mark, or other sentence unit (a symbol, digit, etc.). Dependency relations between words are indicated by “branches” or edges.
17 The parser was developed at the VDU Centre of Computational Linguistics by L. Boizou and F. Zamblera.
18 Available online: https://ufal.mff.cuni.cz/tred/.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 8
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 8
FIG. 3. Dependency tree of a syntactically annotated sentence in TrEd editor format
For each word, the following information is given in this order (see Fig. 3):
1) the specific word form used in the sentence (e.g., gyventojai);
2) the headword (dictionary form)—the lemma (e.g., gyventojas);
3) morphological tags (e.g., Ncmpnn-);
4) syntactic function (e.g., Sub) (see below for explanations of abbreviations).
2.4. Levels of analysis
Morphological analysis level. Morphological annotation is the first stage of automatic analysis and is therefore essential for higher-level syntactic or sentence analysis. Morphological analysis was performed using the semantika.lt annotator (Dad urkevičius 2017). The morphological tags used in ALKSNYS are based on the MULTEXT-East format (described in more detail in Section 1).
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 9
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 9
Syntactic analysis level. The syntactic parser processes morphologically annotated files and produces sentences divided into clauses or phrases, generates syntactic trees, and specifies the syntactic functions of their elements. The analysis is based on the dependency grammar model (dependency grammar). It is generally applied to typologically similar languages characterized by inflectional word forms and free word order, such as Slavic languages (PDT19, SynTagRus20), Latvian, and so on. Syntactic dependencies are expressed hierarchically. The process begins with the main clause—the root of the tree—to which the other sentence elements are attached according to their dependency (i.e., syntactic relations).
The abbreviations for syntactic tags and information about syntactic relations and dependencies are based on Czech work (Hajič et al. 1999); they were among the first to develop syntactic analysis for the synthetic-type Czech language. ALKSNYS currently uses 18 basic syntactic tags (not counting their variants): Sub – subject, Pred – predicate, Obj – object, etc. (see Table 1). Variants of tags arise when two distinct or duplicate tags are combined into one. For example, two tags are used to mark a compound nominal or verbal predicate, respectively: PredN and PredV; when marking an attributive subordinate clause in a complex sentence, a double tag Pred_Atr is assigned to the predicate of the subordinate clause; to indicate coordination, _Co is marked as part of a double tag for coordinated sentence elements, i.e., Obj_Co is written for each of the coordinated objects listed, and so on. In addition, triple tags may occur in complex sentences, for example, when marking coordinated subordinate clauses: Pred_Atr_Co – this is an attributive subordinate clause.
TABLE 1. Syntactic functions and their tags in the corpus
Tag
Syntactic function
Example
Sub
Subject
Jis [Sub] sakė
Pred
Predicate (or auxiliary word)
Jis ėjo [Pred]; buvo [Pred] patenkintas
PredN
Nominal part of the predicate
Was [Pred] satisfied [PredN]
PredV
Verbal part of the predicate
Must [Pred] respond [PredV]
Obj
Object
I am waiting for a guest [Obj]; a decision must be made [Obj]
Atr
Attribute (modifier)
Secondary [Atr] school; member country [Atr]
Adj
Adverbials
Now [Adj] decides; on Partizanų Street [Adj]
19 Prague Dependency Treebank (Prague Dependency Treebank), see in more detail https://ufal.mff.cuni.cz/pdt3.0.
20 Available online: http://www.ruscorpora.ru/en/.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 10
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 10
Aux
Auxiliary function (e.g., commas or other symbols)
AuxC
Subordination (subordinate clauses)
He said that [AuxC]...
AuxK
Sentence-final punctuation mark
AuxL
Auxiliary lexical unit (e.g., a word in a foreign language)
U.S. company North [AuxL] American [AuxL] Investment [AuxL] Consulting [AuxL] Inc. [Atr]
Manufacturers [Atr_Co] and [Coord] sellers [Atr_Co]
Par
Parentheticals, forms of address
In fact [Par], it seems
ExD
Ellipsis or other omissions in a sentence
A woman is a [Pred_ExD] being
Pred_ (e.g., Pred_Obj)
Type of subordinate clause in a complex sentence (marked on the predicate)
He wrote that properties are determined [Pred_Obj]
Manual sentence correction level. All automatically syntactically annotated sentences were checked by linguists. The annotation of many sentences, especially complex ones, had to be corrected and the correct dependencies specified. In some cases, morphological tags were also corrected. The sentences were corrected independently by three linguists, and later each sentence was additionally checked by all three linguists. The characteristics of syntactic annotation are described below.
Syntactic annotation features. When performing automatic syntactic analysis, as mentioned, the root of the syntactic tree (root) is considered to be the hierarchically highest sentence constituent (e.g., a verb, sometimes a conjunction at the beginning of a sentence) or another sentence element (e.g., in a complex sentence, its conjunction or connective word); the branch indicating sentence punctuation is the punctuation mark: a period, question mark, exclamation mark, ellipsis, semicolon, etc. Below, we provide specific examples to illustrate the features of syntactic annotation.
An example in which the sentence head is a verb [Pred] is shown in Figure 4, while an example in which it is a conjunction [Coord] is shown in Figure 5.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 11
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 11
FIG. 4. The head of the dependency tree is a verb
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 12
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 12
FIG. 5. The head of the dependency tree is a conjunction at the beginning of the sentence
An example in which the head of a complex sentence is a punctuation mark is shown in Figure 6.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 13
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 13
FIG. 6. The head of the dependency tree is the punctuation mark separating the clauses of a complex sentence
In prepositional constructions, the preposition is considered to occupy a higher position and is labeled AuxP, while the word that follows it is labeled according to its function (e.g., object): prices for [AuxP] electricity [Obj] (see Fig. 7).
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 14
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA BIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 14
FIG. 7. Annotation of prepositional constructions
Subordinate clauses depend on the predicate of the main clause, and the conjunction introducing the subordinate clause depends on that predicate (the comma depends on the conjunction). It is also important to indicate the function of the subordinate clause. This is indicated beside the predicate of the subordinate clause, e.g., Pred_Obj – a subordinate object clause (see Fig. 8).
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 15
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 15
FIG. 8. Example of annotating subordinate clauses in a subordinate sentence
The marker of a coordinating relation (a conjunction or cable) may be the head of a coordinating construction (e.g., suggested sitting down and [Coord] playing; suggested to the children and [Coord] their parents) (see Fig. 9).
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 16
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 16
FIG. 9. Example of annotating clauses participating in coordinating relations
If there is a multi-clause mixed coordination, the conjunction in the construction is the head, and all the other asyndetic clauses depend on it (commas and the enumerated words associated with them are at the same dependency level). Other punctuation marks are labelled Aux (see Fig. 10).
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351–7204
Page 17
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 17
FIG. 10. Example of annotating a mixed coordinating construction (the figure also shows additional cases of coordinating relations)
The head of a coordinate sentence is the coordinating conjunction. Connective words consisting of adverbs, pronouns, or particles (e.g., the adverb therefore) depend on the predicate of the second clause and function as an adverbial (Adj), while the coordinating Coord function is performed by the cable, as in asyndetic sentences or sentences with repeated conjunctions, which are treated here as particles (see Fig. 12), as is the case when sentence clauses are coordinated (see Figs. 11 and 12).
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 18
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 18
FIG. 11. Example of annotating a coordinate sentence with a connective word
FIG. 12. Annotation of a coordinating construction
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 19
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 19
Numerical expressions, such as hours and years, are difficult to annotate. They are annotated as adverbials with modifiers or as modifiers (see Figs. 13 and 14).
FIG. 13. Annotation of a date
FIG. 14. Annotation of dates and other numerical expressions (the figure shows only a fragment of a syntactically annotated sentence)
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 20
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 20
Complex predicates are not always easy to annotate. In some cases, they are annotated as they are understood in Lithuanian grammar books, i.e., as nominal and verbal predicates (e.g., is [Pred] simple [PredN]; may [Pred] be [PredV]), while in some cases the second constituent of the predicate is considered an object (e.g., intend [Pred] to convene [Obj]).
It was difficult to annotate constructions with correlative conjunctions, because both conjunctions are hierarchically equivalent. In such cases, the predicate of the main clause was considered to occupy the higher position in the hierarchy, with the subordinate clause dependent on it (see Fig. 15).
FIG. 15. Example of annotation of complex sentences with correlative conjunctions
Finally, the status of certain parts of speech was considered. For example, repeated coordinating conjunctions were decided to be treated as particles, because their function in the structure of the dependency tree is similar to that of particles (see Fig. 12).
When manually annotating syntax, many problems had to be addressed; only a few have been identified here. Many annotation difficulties were resolved by following the perspective of traditional grammar and simply agreeing on a particular method of annotation. In other cases, it was necessary to accommodate the specific features of syntactic annotation and to apply the agreed approach consistently. It is hoped that the problems encountered will be addressed as automatic syntactic analysis is further developed.
The next subsection presents data retrieval in annotated corpora using the ANNIS system. This is one of the challenges involved in using annotated corpora, since data retrieval in such corpora, which contain numerous labels and several levels of annotation, must be multifaceted. For this reason, the ANNIS tool, developed by researchers at Humboldt University (Germany), was chosen.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 21
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA BIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 21
3. Searching via the ANNIS system
Naudoti ANNIS įrankį nėra labai sudėtinga (plačiau žr. Zeldes 2016). Reikia pasirinkti anotuotą tekstyną (MATO tekstyno dalį (šis tekstynas suskirstytas į dokumentus, grožinę literatūrą, mokslinius tekstus ir periodiką) arba ALKSNIO tekstyną (jis įvardytas taip: Alksnis_paula_1.0) ir pateikti užklausą. Užklausų struktūra aprašyta 3.1 ir 3.2 poskyriuose.
ANNIS provides the corresponding concordance lines with related information (see Fig. 16). A concordance line can be expanded from 5 to 25 words (including punctuation marks) to both the left and right; the results can also be exported.
Both corpora use grammatical tags based on the Leipzig Glossing Rules21. Several tags not found in the aforementioned rules have been added; for example, ~COMP denotes the comparative degree. A tilde is written before such tags. Parts of speech are specified according to the Universal Dependency format.
FIG. 16. A fragment of search results using the ANNIS tool
Searching for the lemma įsakymas in the MATO documents
21 Available online: https://www.eva.mpg.de/lingua/resources/glossing-rules.php. Other annotated corpora often use Universal Dependency (UD) tags (see http://universaldependencies.org/; see also Fig. 18). As noted in the first chapter, UD tags are very long (e.g., accusative case, feminine gender, and plural number are indicated as follows: case=accusative|gender=feminine|number=plural); they take up too much space in the ANNIS interface, making the information inconvenient to read.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 22
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anuotuoti tekstynai | 22
FIG. 17. A fragment of search results using the ANNIS tool
Searching for the lemma įsakymas in the ALKSNIS corpus
(to display syntactic relations with arrows, click the plus sign next to deps)
3.1. Simple search
A search for a single feature is called a simple search. The annotation provided (names of grammatical categories, grammatical labels used, etc.) depends on the structure of each corpus, not on the ANNIS tool.
Please note that searches are case-sensitive.
3.1.1. Search for exact word forms (character strings)
A simple search is performed by specifying the category name, an equals sign, and the category value enclosed in English quotation marks, e.g.:
• lemma="kalba" (this query means that the lemma kalba is being searched for);
• pos="ADJ" (adjectives are being searched for);
• gram=".M.SG.GEN." (searching for words used with certain grammatical categories, in this case words in the masculine singular genitive form).
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 23
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 23
To search for a specific word form in the text, you need to specify the category name tok or simply enter the form you are looking for in quotation marks, e.g.:
• tok="pasakė" or "pasakė".
In both annotated corpora, it is possible to search for lemmas, parts of speech, grammatical categories, and specific word forms; in ALKSNIS, it is also possible to search for syntactic tags, e.g.:
• syfun="Atr" (searching for attributes).
In ALKSNIS, it is also possible to search for dependency relations (deps), but these are searched for according to a different principle (see 3.2.3).
3.1.2. Searching for character sequences
ANNIS allows searches using regular expressions (regular expressions); these are denoted by special characters. A regular-expression search is essentially written in the same way as a query for exact word forms, except that English quotation marks must be replaced with forward slashes.
3.1.2.1. Searching for any characters
The most important special character is the period; it replaces any single character (letter, digit, etc.), e.g.:
• lemma=/pl.t.s/ (searching for, e.g., platus, plotis, plitus);
• /p.sak.s/ or tok=/p.sak.s/ (searching for, e.g., pasakys, pasakos, pasakas, pasakęs, pasakis);
• /201./ or tok=/201./ (searching for, e.g., 2010, 2011, 2012...).
Since the period can replace any character, to search for the period itself, use the combination of a backslash followed by a period (\.), e.g.:
• gram=/^\..\SG\.GEN\./ (searching for, e.g., .M.SG.GEN.,.F.SG.GEN.,...).
3.1.2.2. Searching for one of several characters
Instead of searching for completely unspecified characters, a query can specify several particular characters to search for. They are written in square brackets, e.g.:
• lemma=/pl[ao]t.s/ (searches for, e.g., platus, plotis, but not plitus);
• /t[ei]lp./ or tok=/t[ei]lp./ (searches for, e.g., telpa, tilpo, but not talpa);
• /201[0124]/ or tok=/201./ (searches for 2010, 2011, 2012, 2014);
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 24
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anuotuoti tekstynai | 24
• gram=/\.[MFN]\.SG\.GEN\./ (searches for .M.SG.GEN.,.F.SG.GEN.,.N.SG.GEN.).
Note that this method searches for one character from the specified set of characters: /dirb[aius]/ finds dirba, dirbi, dirbu, dirbs, but not dirbau or dirbsi.
3.1.3. Repetition operators
Characters (ordinary and special) can be repeated, e.g.:
• /dirb[ao][mt]e/ or tok=/dirb[ao][mt]e/ (searches for dirbame, dirbate, dirbome, dirbote).
Thus, the number of characters searched for is fixed, e.g., two characters in the examples above (or [ao][mt]); therefore, the query lemma=/..važiuoti/ will not return results with the lemma įvažiuoti. Repetition operators offer more options; their meanings are as follows:
•? – a particular character occurs once or not at all;
• + – a particular character occurs at least once (i.e., once, twice, three times, four times, or more);
• * – a particular character occurs n times (i.e., it may not occur at all, or may occur once, twice, or more times)
These operators help search for characters that follow or precede them, e.g.:
• /šild?o/ (searches for wood or heats);
• /Ma+u/ (search for Mau, Maau, Maaau, Maaau, etc.);
• /oi*/ (search for o, oi, oii, oiii, etc.).
The operators mentioned above are compatible with searching for any or one of several characters, e.g.:
• /šauk[aiu]*/ or tok=/šauk[aiu]*/ (search for, e.g., šauk, šauki, šaukia, šaukiu);
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 25
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anot uoti tekstynai | 25
/šauk[aiu]+/ or tok=/šauk[aiu]+/ (search for, e.g., šauki, šaukia, šaukiu, but not šauk);
gram=/.*\.F\..*/ (search for the sequence.F. between any other sequences of characters).
It should be noted that structures such as [aiu]* or [aiu]+ do not affect the order of the characters being searched for. Thus, results are returned that contain a or i, or u + a; or i, or u + a; or i, or u.... Thus, the following character combinations are recognized: aaa, i, uuuuu, uiua, uua, etc.
3.1.4. Searching for alternatives
The vertical bar (|) allows you to search for alternatives, e.g.:
lemma="bad" | lemma="good";
/d(au|ū)ž.*/ or tok=/d(au|ū)ž.*/ (searches, e.g., for daužo, dūžta, daužymas);
gram=/.*\.~(COMP|SUP)\..*/ (searches for.~COMP. and.~SUP. between any sequences of characters);
"only" | "just" or tok="only" | tok="just" (searches for tik and tiktai).
3.1.5. Negatively formulated queries
Although this feature is more useful when combining features in complex searches, simple searches can also specify what not to search for, e.g.:
pos!="NOUN" (searches for all words that are not nouns);
tok!=/.*[ąaęeėįiyiouųū]/ (searches for all word forms that do not end in a vowel).
If a complex search query is not selected (see 3.2), negatively formulated simple queries can cause problems, especially if they cover many words (as in the first case—nouns), so this query should be used with caution.
3.2. Compound search
A compound search is made up of several simple searches. An additional part is also needed to describe the relations between the simple searches (specified by position). Each simple search component is joined with an ampersand (&) sign.
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 26
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoji tekstynai | 26
3.2.1. Combinations of features for a single word
A simple search looks for the same word, and the link between simple searches is expressed by the combination operator (_=_), e.g.:
• "mano" & pos="PRON" & #1 _=_ #2 (search for the pronoun mano, but not the third-person present-tense form of the verb manyti).
This query formulation means that the first simple search (#1, i.e. "mano" or tok="mano") and the second simple search (#2, i.e. pos="PRON") refer to the same word.
3.2.2. Word combinations
If a query is formulated for different words based on a simple search, the combination operator is replaced by the sequence operator (a dot), e.g.:
• lemma="pilnas" & gram=/.*\.GEN\..*/ & #1.1 #2 (search for a word used in the genitive case after the word pilnas).
The number 1 after the dot means that #2 immediately (with nothing intervening) follows #1. The number 2 would indicate that one word intervenes, 3 that two words intervene, and so on. Instead of.1, a dot without any number can be written (lemma="pilnas" & gram=/.*\.GEN\..*/ & #1. #2). The interval can be specified more precisely with a comma, e.g..2,4 (with one to three intervening words). If the size of the gap does not matter, a dot and an asterisk (*) should be written.
Sequence and combination operators can be used in one compound query, e.g.:
• lemma="pilnas" & gram=/.*\.GEN\..*/ & pos!="ADJ" & #1.1 #2 & #2 _=_ #3 (search for genitive-form words that are not adjectives after the word pilnas).
3.2.3. Searching for syntactic dependencies
Dependency structures can also be searched using a complex query with the direct dominance operator ->, e.g.:
• pos="NOUN" & pos="ADJ" & #1 ->deps #2 (searches for nouns that govern adjectives, e.g., daugiabutis namas);
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotouti tekstynai | 27
CONCLUDING REMARKS
The MATAS and ALKSNIS corpora are important resources for the Lithuanian language. They are needed for subsequent stages in the computerisation of language. These corpora can provide quantitative data on the distribution of parts of speech, grammatical forms, and syntactic functions, as well as the most frequent lemmas, grammatical forms, and so on (several specific quantitative studies can be found in the following works: Rimkutė 2006; Utka 2009; Brokaitė 2017). MATAS was used to train the semantika.lt parser (to assess annotation quality).
In the future, automatic syntactic analysis must be further developed. First, the quality of syntactic annotation could be improved: syntactic functions could be refined and new labels added (e.g., adverbials could be refined according to their semantics: place, manner, time, etc.). At the outset of syntactic annotation, a number of ambiguities in annotation were encountered; therefore, certain syntactic cases should be discussed, such as the annotation of complex predicates, the distinction between ellipsis, asyndetic clauses and explanatory constructions, and the status of certain parts of speech, among other issues.
Another line of work is to expand the corpus itself. The corpus sections by genre should be supplemented, for example, by including more academic and popular science texts. In addition, the corpus must be converted to the widely used international Universal Dependency format (see Fig. 18; grammatical labels are not shown here, and the example is a syntactically annotated example in Finnish).
FIG. 18. Universal Dependency format
One of the main goals of automatic syntactic analysis is to create a statistically based syntactic parser (statistically-based parser), based on ALKSNIS as a benchmark for syntactic analysis and used to annotate large quantities of text.
Where possible, syntactic analysis can also be extended in other directions, e.g., thematic role annotation (thematic role annotation). This is a certain kind of semantic representation associated with valency. Within the internal syntactic annotation system, this level is assigned to every word and is currently not publicly visible, as it is more experimental, not fully developed, and unverified. It would also be useful to annotate fixed expressions (multi-word
BENDRINĖ KALBA 90 (2017) www.bendrinekalba.lt ISSN 2351-7204
Page 28
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 28
expressions, MWE22), which include collocations and phraseological units. Until now, these have not been annotated in the corpus, except for morphological multiword units, whose components are joined into a single unit (e.g., for now, from afar) and have a single morphological and syntactic function (most often tagged as adverbs and adjuncts). It is also planned to annotate anaphoric relations (when a noun is used in one sentence and replaced by a pronoun in another sentence (or another part of the sentence), that is, to mark the links between a pronoun and the noun it replaces).
LITERATŪRA
Bielinskienė A., Boizou L., Kovalevskaitė J., Rimkutė E. 2016: Lithuanian dependency treebank ALKSNIS. – Human language technologies – the Baltic perspective: proceedings of the 7th international conference, Baltic HLT 2016. Amsterdam: IOS Press, 107–114. Prieiga internete: http://ebooks.iospress.nl/volumearticle/45523 (žiūrėta 2017 10 15).
Boizou L., Zamblera F. 2014. Syntactic engine for the Lithuanian language. – Human language technologies – the Baltic perspective: proceedings of the 6th international conference, Baltic HLT 2014. Amsterdam: IOS Press, 69–74. Prieiga internete: http://ebooks.iospress.nl/publication/38006 (žiūrėta 2017 10 15).
Bro į kaitė K. 2017: Tarinio raiška gramatinėmis formomis sintaksiškai anotuotame lietuvių kalbos tekstyne ALKSNIS. Magistro darbas. Vytauto Didžiojo universitetas.
Dadūrkevičius 2017: Lietuvių kalbos gramatika skaitmeniniame atvirojo kodo pasaulyje. – 24-osios Jono Jablonskio konferencijos Skaitmeniniai kalbos ištekliai jų plėtros kryptys ir panaudos galimybės pranešimų tezės. 2017 m. rugsėjo 29 d., Vilnius.
Hajič J., Panevová J., Buráňová E., Urešová Z., Bémová A. 1999: ANNOTATIONS AT ANALYTICAL LEVEL. Instructions for annotators (11.10.1999), UK MFF ÚFAL Praha. Prieiga internete https://ufal.mff.cuni.cz/pedt2.0/publications/a-man-en.pdf (žiūrėta 2017 11 05).
Kapočiūtė-Dzikienė J., Rimkutė E., Boizou L. 2017: A Comparison of Lithuanian Morphological Analyzers. – 20th International Conference Text, Speech, and Dialogue (TSD 2017). Springer International Publishing AG, 47–56.
Muischnek K., Müürisep K., Puolakainen T. 2014: Dependency Parsing of Estonian: Statistical and Rule-based Approaches. – Human Language Technologies – The Baltic Perspective: proceedings of the 6th international conference, Baltic HLT 2014. Amsterdam:
22 Pastoviųjų junginių tyrimams skirtas tarptautinis projektas PARSEME, žr. https://typo.uni-konstanz.de/parseme/.
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 29
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotuoti tekstynai | 29
Pretkalniņa L., Rituma L., Saulīte B. 2016: Universal Dependency Treebank for Latvian: A Pilot. – Proceedings of the 7th International Conference on Human Language Technologies – the Baltic Perspective: proceedings of the 7th international conference, Baltic HLT 2016. Amsterdam: IOS Press, 136–143. Prieiga internete: http://ebooks.iospress.nl/volumearticle/45527 (žiūrėta 2017 10 10).
Rimkutė E. 2006: Morfologinio daugiareikšmiškumo ribojimas kompiuteriniame tekstyne. Daktaro disertacija. Vytauto Didžiojo universitetas.
Utka A. 2009: Dažninis rašytinės lietuvių kalbos žodynas: 1 milijono žodžių morfologiškai anotuoto tekstyno pagrindu. Kaunas: Vytauto Didžiojo universiteto leidykla. Prieiga internete: http://donelaitis.vdu.lt/publikacijos/Dazninis_zodynas.pdf (žiūrėta 2017 12 07).
Utka A., Bielinskienė A., Boizou L., Kovalevskaitė J., Repečka V., Rimkutė E. 2017: Kalbos technologijos – būtina sąlyga kalbai egzistuoti. BNS Spaudos centras. 2017 m. spalio 5, 1–4. Prieiga internete: http://sc.bns.lt/view/item/246397 (žiūrėta 2017 10 20).
Zeldes A. 2016: ANNIS User Guide (Version 3.4.3). Prieiga internete: http://corpus-tools.org/annis/resources/ANNIS_User_Guide_3.4.3.pdf (žiūrėta 2017 10 31).
Zinkevičius V. 2000: Lemuoklis – morfologinei analizei. – Darbai ir dienos, 24, 245–274. Prieiga internete: http://donelaitis.vdu.lt/publikacijos/zinkevicius.pdf (žiūrėta 2017 10 07).
Zipser F., Romary L. 2010: A model oriented approach to the mapping of annotation formats using standards. – Proceedings of the Workshop on Language Resource and Language Technology Standards, LREC 2010. Malta. Prieiga internete: http://hal.archives-ouvertes.fr/inria-00527799/en/ (žiūrėta 2017 12 07).
Įteikta 2017 11 06
Priimta 2017 12 20
BENDRINĖ KALBA 90 (2017)
www.bendrinekalba.lt
ISSN 2351-7204
Page 30
AGNĖ BIELINSKIENĖ, LOÏC BOIZOU, ERIKA RIMKUTĖ. Lietuvių kalbos morfologiškai ir sintaksiškai anotouti tekstynai | 30
LITHUANIAN MORPHOLOGICALLY ANNOTATED CORPUS AND TREEBANK
Summary
Annotated corpora are fundamental resources, which are very useful to develop language technology. The size, quality, and structure of such annotated corpora has a direct influence on the development of other tools. This article describes two annotated corpora prepared by the Centre of Computational Linguistics at Vytautas Magnus University: MATAS, a morphologically annotated corpus, and ALKSNIS, a treebank. It mainly discusses the structure and the tag set of both corpora, as well as the annotation procedure and tools. Both corpora are available online through ANNIS interface, therefore the syntax of ANNIS simple and complex requests is summarised for the Lithuanian potential users.
