Boosting offline handwritten text recognition in historical documents with few labeled lines

Received May 5, 2021, accepted May 12, 2021, date of publication May 21, 2021, date of current version June 2, 2021. Digital Object Identifier 10.1109/ACCESS.2021.3082689 Boosting Offline Handwritten Text Recognition in Historical Documents With Few Labeled Lines JOSÉ CARLOS ARADILLAS 1, JUAN JOSÉ MURILLO-FUENTES 1, (Senior Member, IEEE), AND PABLO M. OLMOS 2,3, (Member, IEEE) 1Departamento de Teoría de la Señal y Comunicaciones, Escuela Técnica Superior de Ingeniería, Universidad de Sevilla, 41004 Sevilla, Spain 2Departamento de Teoría de la Señal y Comunicaciones, Universidad Carlos III de Madrid, 28903 Madrid, Spain 3Gregorio Marañón Health Research Institute, 28007 Madrid, Spain Corresponding author: Juan José Murillo-Fuentes ([email protected]) This work was supported in part by the Spanish Government Ministerio de Educación (MEC) under Grant FPU16/04190 and Project MINECOTEC2016-78434-C3-2-3-R; in part by the Comunidad de Madrid under Grant IND2017/TIC-7618, Grant IND2018/TIC-9649, Grant IND2020/TIC-17372, and Grant Y2018/TCS-4705; in part by the Banco Bilbao Vizcaya Argentaria (BBVA) Foundation through the Domain Alignment and Data Wrangling with Deep Generative Models (Deep-DARWiN) Project; and in part by the European Union by the Fondo Europeo de Desarrollo Regional (FEDER) and the European Research Council (ERC) through the European Union Horizon 2020 Research and Innovation Program under Grant 714161. ABSTRACT In this paper we address the problem of offline handwritten text recognition (HTR) in historical documents when few labeled samples are available and some of them contain errors in the train set. Our three main contributions are: first, we analyze how to perform transfer learning (TL) from a massive database to a smaller historical database, analyzing which layers of the model need fine-tuning. Second, we analyze methods to efficiently combine TL and data augmentation (DA). Finally, we propose an algorithm to mitigate the effects of incorrect labeling in the training set. The methods are analyzed over the ICFHR 2018 competition database, Washington and Parzival. Combining all these techniques, we demonstrate a remarkable reduction of CER (up to 6 percentage points in some cases) in the test set with little complexity overhead. INDEX TERMS Connectionist temporal classification (CTC), convolutional neural networks (CNN), data augmentation (DA), deep neural networks (DNN), historical documents, long-short-term-memory (LSTM), offline handwriting text recognition (HTR), outlier detection, transfer learning. I. INTRODUCTION The transcription of historical manuscripts is paramount for a better understanding of our history, as it allows for direct access to the contents, quite facilitating searches and studies. Also, the classification and indexing of transcript text can be easily automated. Handwritten text recognition (HTR) tasks in historical datasets have been studied by many authors in the last few years [1]–[10]. In HTR, transcribing each author can be considered a different task, since the distribution of both model input and output varies from writer to writer. At the input, we have variations not only in the calligraphy but also, depending on the digitization process, in the image resolution, contrast, color, or background. On the other hand, at the output, the labels usually correspond to different languages The associate editor coordinating the review of this manuscript and approving it for publication was Gang Mei . and historical periods, with differences in the character set, the semantics and the lexicon. Usually, the process for automatic transcription of a document comprises 4 phases: 1) digitization of the document to obtain an image of every page in the document in electronic format; 2) segmentation of each page into corresponding regions with lines of text; 3) transcription of each line of text and finally 4) application of a dictionary or language model to correct errors in the transcription of texts as well as in the composition of the complete texts from the lines obtained in step 3). While segmentation is an important issue in HTR [2], [11]–[13], this paper focuses on the transcription phase. In recent years, there has been a trend towards models based on deep neural networks (DNNs) [8]. Once the DNN model to be used has been designed, an enormous number of training samples are required to minimize the number of transcription errors, measured in character error rate 76674 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ VOLUME 9, 2021 J. C. Aradillas et al.: Boosting Offline HTR in Historical Documents With Few Labeled Lines (CER) or word error rate (WER), given by the Levenshtein distance [14] between the ground truth (GT) and the output of the model. However, in reality most of the times we might only have a limited number of lines for a given author and document. Besides, transcription of part of the documents to get labeled samples is expensive either in time or money. Take [1] as an example, where the manual transcription process of a document by an expert in paleography took an average of 35 minutes per page. In this scenario, allowing for a reduction in the transcript needed would greatly improve the viability and cost of the process. The goal of the methods proposed in this paper is to significantly reduce the number of annotated lines, thus reducing the monetary cost of the process. Contributions are as follows: •A DNN model with transfer learning (TL) by studying in detail which layers should be retained and which ones should be retrained. We found this model to provide the best performance within the models using long-short- term-memory (LSTM). •A thoughtful analysis of HTR when little labeled data is available, studying the evolution with the number of transcript lines, highlighting how sensitive the models are to the number of lines in the train set when this number is low. •A model combining TL and data augmentation (DA). We show that for a reduced number of lines, using DA can be counterproductive. •An algorithm to detect and avoid mislabeled lines is proposed and tested, with good results in small training datasets. Furthermore, in some scenarios, wrong labels are fixed. The remaining of the paper is organized as follows: in Section II previous works reporting solutions to the problem of HTR over small datasets are summarized; in Section III the architecture and models used in this paper are presented; in Section IV the application of TL and DA for HTR is analyzed; in Section Van algorithm is proposed to detect and prone mislabeled lines in the training set; the paper ends with Section VI, drawing main conclusions. II. RELATED WORK AND CONTRIBUTIONS A. DNN MODEL State-of-the-art architectures combine a convolutional neural network (CNN) [15] with a recurrent neural network (RNN) with LSTM cells [16]. This type of network models the conditioned probability, p(l|x), of a character sequence of arbitrary length, l, given an image, x, of fixed height and arbitrary width. Note that varying the number of characters yields a new design of the last layer of the DNN model. These models are configured to minimize the connectionist temporal classification (CTC) cost function proposed by Graves in [17]. In some works 2D-LSTM [16] networks are used [18]–[21]. This RNN has two main drawbacks. On the one hand, it has an extremely large number of parameters that make learning difficult. On the other hand it is not parallelizable [22]. For these reasons, it has been discarded here. Regarding the state-of-the-art DNN models for HTR, some recent works are in the line of avoiding recurrence in the models, developing models based in fully-convolutional networks such as the (Gated) Fully Convolutional Networks (G)FCN [23]–[26]. This kind of model reduces the number of parameters in the architecture. B. TRANSFER LEARNING In the HTR problem with a reduced training set, TL was applied by Soullard et al. in [7]. The main idea behind TL is initializing the parameters of a model by those learned from a huge dataset beforehand, denoted as source. Then, the available labeled set of samples of the dataset of interest, the target, is used to refine the parameters of the model, usually just a subset of them [27]–[30]. In [30], they analyze how to reduce the dataset shift and enhance the feature transferability in task-specific layers of deep networks. Hence, with TL we start learning a different task to avoid learning the whole set of parameters from scratch, preventing overfitting and favoring convergence. In [7], they proposed a method that applies TL in both the optical and the language model. In this and other similar previous proposals on TL, the authors applied DA in both training and test steps. C. DATA AUGMENTATION DA consists in augmenting the training set with synthetically generated samples. Similar to TL, it reduces the tendency to overfit when training models with a large number of parameters and limited labeled data. In DA for image classification problems, the training set is increased by modifying the original images through transformations such as scaling, rotation, or flipping images, among others [31]. Several authors have proposed specific DA techniques for HTR: in [32] the authors apply methods for augmentation and normalization to improve HTR by allowing the network to be more tolerant of variations in handwriting by profile normalization. In [33] they show some affine transformation methods for data augmentation in HTR. In [34] and [35] they synthesize new lines images by concatenating characters from different datasets. [34] does it from cursive characters while in [35] they do it from a database of handwritten Chinese characters. Similar to [32], in [36] they also apply some elastic distortions to the original images. In [4] the authors improve the performance by augmenting the training set with specially crafted multiscale data. They also propose a model-based normalization scheme that considers the variability in the writing scale at the recognition phase. In these works, they apply DA in relatively large well-known datasets, but here we show that the regularization effect of any DA technique has no impact when doing the fine-tuning adaptation to a singular writer in small databases. Accordingly, we conclude that the combination of TL and DA applied to small datasets has to be done carefully, to reduce the final error. VOLUME 9, 2021 76675 J. C. Aradillas et al.: Boosting Offline HTR in Historical Documents With Few Labeled Lines FIGURE 1. The adapted CRNN architecture from [39] used as baseline. The number of channels of each CNN layer is shown in this scheme. Pooling layers after the first, second and third CNN layer are also depicted. D. MISLABELED SAMPLES Mislabeled detection in HTR has been seldom faced. In [37] they face a specific problem in the IAM database: crossed out words that are labeled with the symbol ‘‘#’’. The authors propose a method to avoid how this specific label affects the performance. That method is focused on the specific problem of crossed-out text and how it is annotated in the GT. The algorithm we propose in Section Vis more general, addressing this and other possible problems. In further related work in [38], the authors apply a method to align the output of a segmentation process with the available GT. III. ARCHITECTURE AND DATABASES In the HTR pipeline, there are several ways to improve performance of a DNN model: preprocessing steps, the architecture used, regularization techniques, optimization, language model and dictionary, among others. The methods proposed in this paper are developed for an exemplary state-of-the-art DNN architecture but can be easily used in the pipeline of any other HTR system to reduce transcript errors. For fair comparison, in this paper, we use the same DNN model for all the experiments. Extra correction steps such as adding a language model (LM) are not included but could be applied to further improve the performance. In this section, we focus on the selection of the model of the DNN and the databases used. A. ARCHITECTURE In this work, we implement a network architecture based on the convolutional recurrent neural network (CRNN) presented in [39]. This approach avoids the use of two-dimensional LSTM (2D-LSTM) layers, applying convolutional layers as feature extractors and a stack of 1D bidirectional LSTM (BLSTM) layers to perform classification. Previous DNN architectures for HTR consisted of a combination of 2D-LSTM layers and convolutional layers, with a collapsing stage before the output layer in order to reshape the features tensors from 2D to 1D [18], [19]. The use of 2D-LSTM layers at the first stages has several drawbacks such as the need of more memory in the allocation of activations and buffers during back-propagation and a longer runtime is required to train the networks since parallel computation cannot be implemented in contrast to a CNN [22]. Recently, it has been shown that CNN in the lower layers of an HTR system obtains similar features than an RNN containing 2D-LSTM units [22]. The CRNN architecture proposed in [39] is comprised of seven CNN with a max-pooling step at the output of four of them, followed by a stack of two BLSTM layers at the top of the network. In [6] we have shown that the CRNN in Fig. 1, the one used in this work,1achieves better performance than the original one proposed in [39]. It uses a CNN with 5 layers at the bottom, with a 3 ×3 and 1 ×1 stride kernel, the number of filters are 16, 32, 48, 64 and 80, respectively. We use LeakyReLU as the activation function. A 2 ×2 maxpooling is also applied at the output of the first 3 layers, to reduce the size of the input sequence. At the output of the CNN, a column-wise concatenation is carried out with the purpose of transforming the 3D tensors of size w×h×d (width ×height ×depth) into 2D tensors of size w×(h×d) where wand hare the width and height of the input image divided by 8, i.e., after 3 stages of 2 ×2 max-pooling. The depth, d=80, is the number of features of the last CNN layer. Therefore, at the output of the CNN, we have sequences of length wand depth h×80 features. After the CNN stage, 5 1D BLSTM recurrent layers of 256 units with hyperbolic tangent functions and without peephole connections.. Since at the output of each BLSTM layer we 1Implementation is publicly available in https://github.com/josarajar/ HTRTF 76676 VOLUME 9, 2021 J. C. Aradillas et al.: Boosting Offline HTR in Historical Documents With Few Labeled Lines FIGURE 2. IAM handwritten text sample: image of a line and its transcript. FIGURE 3. Washington handwritten text sample: image of a line and its transcript. have 256 features in each direction, we perform a depth-wise concatenation to adapt the input of the next layer, to the overall size of 512. Dropout regularization [18], [40] is applied at the output of every layer, except for the first convolutional one, with rates 0.2 for the CNN layers and 0.5 for the BLSTM layers. Finally, each column of features after the 5th BLSTM layer, with depth 512, is mapped into the L+1 output labels with a fully connected layer, where Lis the number of characters in the alphabet of each database, e.g., 79, 83, 96 or 102 in the IAM, Washington, Parzival or International Conference on Frontiers in Handwriting Recognition (ICFHR) 2018 Competition databases, respectively. The additional dimension is needed for the blank symbol of the CTC [17], which concludes this architecture. Overall, this CNN-BLSTM-CTC model has 9.58 ×106parameters roughly, depending on the number of characters in each database. The architecture is implemented in the open-source framework TensorFlow in Python, using the GPU-enabled version. We use the Adam algorithm, a learning rate of 0.003, β1=0.9 and β2=0.999. The parameters are updated using the gradients of the CTC loss on each batch of 16 text lines. We apply an early stopping criterion of 10 epochs without average improvement. The selected model was the one with the best performance out of the 7 +3, 8 +0, 4 +4, 5 +5, and 6 +6 configurations, where A+Bcorresponds to Aconvolutional followed by BBLSTM layers. On the other hand, for the CTC we tried best path decoding and beam search decoding, with no significant improvement of the later, despite its computational complexity. B. DATABASES In this paper we focus on HTR over eight databases: IAM [41], Washington [42], Parzival [42], and the five ones provided at the ICFHR 2018 Competition [5]. 1) THE IAM DATABASE The IAM database [41] contains 13353 labeled text lines of modern English handwritten by 657 different writers. The images were scanned at a resolution of 300 dpi and saved as PNG images with 256 gray levels. An image of this database is included in Fig. 2alongside the GT transcript. The database is partitioned into training, validation, and test sets of 6161, 900, and 2801 lines, respectively.2Here, the validation and test sets provided are merged in a unique test set. There are 79 different characters in this database, including capital and small letters, numbers, some punctuation symbols, and white space. 2) THE RIMES DATABASE The RIMES database is a collection of French letters handwritten by 1,300 volunteers who have participated in the RIMES database creation by writing up to 5 emails. The RIMES database thus contains 12,723 pages corresponding to 5605 mails of two to three pages. In our experiments, we take a set of 12111 lines extracted from the International Conference on Document Analysis and Recognition (ICDAR) 2011 line-level competition. There are 100 different characters in this database. 3) THE WASHINGTON DATABASE The Washington database contains 565 text lines of the George Washington letters, handwritten by two writers in the 18th century. Although the language is also English, the text is written in longhand script and the images are binarized as illustrated in Fig. 3, see [3] for a description of the differences between binarized and binarization-free images when applying HTR tasks. In this database, four possible partitions are provided to train and validate. In this work, we have randomly chosen one of them. The train, validation, and test set contain 325, 168 and 163 handwritten lines, respectively. There are 83 different characters in the database. 4) THE PARZIVAL DATABASE The Parzival database contains 4477 text lines handwritten by three writers in the 13th century. In this case, the lines are binarized like in the Washington database, but the text is written in gothic script. A sample is included in Fig. 4. There are 96 different characters in this database. Note that the Parzival database has a large number of text lines in comparison to the Washington one. We have randomly chosen a training set of the approximately same size as in the Washington training to emulate learning with a small dataset, the main goal of this work. 2The names of the images of each set are provided in the Large Writer Independent Text Line Recognition Task. VOLUME 9, 2021 76677 J. C. Aradillas et al.: Boosting Offline HTR in Historical Documents With Few Labeled Lines FIGURE 4. Parzival handwritten text sample: image of a line and its transcript. 5) THE ICFHR 2018 COMPETITION OVER READ DATASET The set of documents of the ICFHR2018 Competition on Automated Text Recognition on a READ Dataset (https://readcoop.eu/) was proposed to compare the performance of approaches learning with few labeled pages. The dataset provided for the competition consists of 22 documents segmented at line level [5]. They are written in Italian and modern and medieval German. Each of them was written by only one writer but in different periods and various languages. The training data is divided into a general set (of 17 documents) and a document-specific set (of 5 documents) called Konzilsprotokolle_C, Schiller, Ricordi, Patzig, and Schwerin of an equal script as in the test set. Hereafter, general is used to denote available source labeled databases different from the one of interest, while document-specific denotes particular target documents. Also, the Konzilsprotokolle_C dataset, of the University of Greifswald, will be abbreviated as Konzil. The general database comprises roughly 25 pages per document (the precise number of pages varies such that the number of contained characters is almost equal per document). It will be denoted hereafter by ICFHR18-G. For the 5 document-specific databases the authors provide 16 labeled pages plus 15 unlabeled pages. One can check for the error in the transcription of these databases by sending the authors the transcription of these 15 pages. The results of the transcription are then published on the web of the contest. In Fig. 5, samples from five specific target documents are displayed. The standard Unicode Normalization Form Compatibility Decomposition - NFKD is applied to the GT to provide a common character set over such different documents, with 102 characters. The goal of the competition is to fit a model to transcript each of the 5 specific target documents with the lowest CER possible, using the 17 source documents available for training. For each document-specific target dataset, four experiments are conducted, simulating that you have 0, 1, 4, or 16 annotated pages available for training. In Table 1we include the number of training and test lines available for every database. While all lines in the test sets will be used, the number of lines of the training set used vary through the experiments and will be indicated in every case. IV. ON THE DATA AUGMENTATION AND TRANSFER LEARNING TRADEOFF As our first contribution, in this section, we analyze the joint performance of TL and DA methods when applied to HTR. A. TRANSFER LEARNING To cope with a reduced set of labeled inputs, we could first train the DNN model using as source available labeled large TABLE 1. Number of lines available for training and test in each dataset. datasets. Then, we could apply TL or domain adaptation strategies [43] to tune the learned model to transcript a target document. As discussed in Section I, we usually deal with different tasks, where TL has shown useful to share the results of the learning between tasks. Formally, in HTR, deep learning algorithms have been usually used to solve problems over a domain D= {X,P(x)}, where P(x) is the marginal probability. Typically xis the image for a segmented line in the text. The task consists of two components: a label space Yand an objective predictive function f(·) (denoted by T= {Y,f(·)}), which can be learned from the training data. The data consists of pairs {xi,yi}, where xi∈Xand yi∈Y[44] and f(x)=Q(y|x) can be interpreted as the conditional probability distribution. Given a source domain DSand a learning task TS, transfer learning aims to help improve the learning of another target predictive function fT(·) in DTusing the knowledge in DS and TS. In this work, we are interested in inductive transfer learning in which the target task is different from the source task, as the domains are different (DS6= DT). Here we perform TL by retraining a DNN model where 1) all weights are initialized to the ones of the DNN learned for DSand TS and 2) the parameters of lower layers can be frozen to the values of the ones obtained after training with other available source datasets, used as off-the-shelf feature extractors [45]. In [6] we analyzed preliminary TL results over the Washington and Parzival databases, by using the IAM database as the source, and we investigated which layers should be kept fixed to then apply a fine-tuning process to the others. We concluded that the best choice is to unfreeze all the layers, where the first one can be eventually fixed. In most cases, fixing only the first CNN layer leads to the best performance. In Table 2we extend the analysis in [6] to the five specific documents in the ICFHR 2018 Competition dataset, where the 17 documents of the general set of the database, in the ICFHR18-G, are used as the source. Results are included when fixing layers 1 to 3 of the CNN, as fixing other layers provided larger errors in all cases. The lowest achieved errors are highlighted in boldface. Training set size is given in the number of lines. It can be observed that, among all databases, the best performance is achieved when unfreezing all layers or, at most, only the first layer is kept frozen. Hereafter the TL is applied by freezing just the first layer of the DNN model. The results shown in all tables hereafter indicate mean values 76678 VOLUME 9, 2021 J. C. Aradillas et al.: Boosting Offline HTR in Historical Documents With Few Labeled Lines FIGURE 5. From top to bottom: Konzil, Schiller, Ricordi, Patzig and Schwerin handwritten text samples with their transcripts. TABLE 2. TL performance: Mean CER (%) and bootstrapped confidence interval at 95%, in brackets, of the model in Fig. 1using TL for the Washington, Parzival, Konzil, Schiller, Ricordi, Patzig and Schwerin datasets (see Section III) as target domains. of CER or WER. To get the statistics, the model in Fig. 1 is trained 10 times, where the parameters to initialize are independently and randomly set. In Table 2, a non-parametric bootstrapped confidence interval at 95% [46] is also included. For the remaining tables, the confidence intervals can be found in the supplementary material. In Table 2we analyze different strategies of applying TL, with no DA, for the Washington and Parzival target datasets with the IAM database as the source and Konzil, Schiller, Ricordi, Patzig, and Schwerin datasets (see Section III) as target domains with ICFHR18-G as the source. In each of ICFHR 2018 document-specific datasets, 12 pages are used for training, see the corresponding number of lines in Table 1. We conclude that the good choice is to freeze the first convolutional layer of the model (column ‘‘CNN1’’). This solution will be used later in Subsection IV-C. B. DATA AUGMENTATION In [32] the authors compare various DA approaches using both RIMES [42] and IAM [41] databases as benchmarks, where transcription is made on the word level. Note that these databases have a considerably large number of labeled lines. When not applying any augmentation technique, they get a CER of 5.35 % (IAM) and 3.69 % (RIMES). The best CER values reported in [32] by using various DA techniques are 3.93 % and 1.36 %, respectively. Which is equivalent to an improvement of approximately 2 percentage points in both databases. Let us now extend the same analysis to scenarios with small training datasets: Washington, Parzival, Konzil, Schiller, Ricordi, Patzig, and Schwerin databases. As throughout the paper, the transcriptions are made on line level. Results for the IAM, RIMES, and the ICFGH18-G, i.e., the 17 documents of the general dataset in the ICFHR 2018 database, are also analyzed as references. In Table 3we include the CER of our DNN model with no DA and two different DA techniques, affine transformation [33] and random warp grid distortion (RWGD) [32], for all databases in Subsection III-B. We augment the training set by generating ten copies of every line in the training set. One of these copies is the original line without distortions. In Table 3, for the largest databases, the DA improvement is around 2 percentage points (2 percentage points in IAM, 1.9 percentage points in RIMES, and 2.5 percentage points in ICFHR18-G). However, in the small databases, the CER reduction is remarkable, in the range of 5 percentage points to 23.6 percentage points, see CERs highlighted in boldface. Note that the results in [32] are different to the ones in Table 3because while in [32] transcription is done at the word level here whole lines are processed. This explains that in IAM without DA we get CER 7.2% while in [32] 5.35% VOLUME 9, 2021 76679 J. C. Aradillas et al.: Boosting Offline HTR in Historical Documents With Few Labeled Lines TABLE 3. DA performance: Mean CER and WER (%) with affine transformations [33] and RWGD [32] DA approaches evaluated for all datasets in Subsection III-B. The DNN is trained from scratch using the number of lines indicated by ‘Train size’. Largest DA CER reductions are highlighted in boldface. is reported. In any case, it can be concluded that, since the DA acts as a regularization technique to avoid overfitting, the CER reduction is greater as the size of the training set is reduced. At this point, it is most interesting to compare the results of TL and DA techniques when applied independently. It can be observed that TL exhibits, by far, a much larger CER reduction. Next, we face the design and analysis of both techniques combined, where RWGD will be used as the DA approach. C. COMBINING DATA AUGMENTATION AND TRANSFER LEARNING When comparing DA with TL, the large databases are excluded from the comparison. They play the role of source databases in the TL approach, specifically, the IAM is the source dataset when Washington and Parzival are targets and ICFHR18-G in the Konzil, Schiller, Ricordi, Patzig, and Schwerin case. The RIMES database is only used to enhance the comparisons in this section. In the combination of TL and DA techniques, there are several possible designs. Here we propose the following two schemes. In a first approach we perform DA at both the learning from the source dataset and the retraining of the model with the target one: 1) Train the model from scratch with a source dataset, applying DA. 2) Retrain the model with the target dataset, applying DA. We name this proposal DA-TL-DA. In a second proposal, denoted by DA-TL, no DA is applied to the target: 1) Train the model from scratch with a source dataset, applying DA. 2) Retrain the model with the target dataset, without applying DA. We perform the same experiments as in Subsection IV-A, obtaining the results included in Table 4. For the sake of completeness we also report the results in the two previous subsections for TL with the first layer frozen and DA with RWGD. In the first step of the DA-TL and DA-TL-DA methods, the model has been trained from scratch with the IAM database. After that, we fine-tune the model using data from the Parzival and Washington databases. In Table 5the results are shown when training the model in Fig. 1from scratch with TABLE 4. TL and DA combined performance: Mean CER (%) evaluated for Washington and Parzival datasets using TL and DA with IAM database as the source. The number of annotated lines used in training is included as ‘Train size’. TABLE 5. TL and DA combined performance: Mean CER (%) evaluated in ICFHR 2018 Competition Specific datasets as targets using TL and DA with ICFHR18-G as source. The number of annotated pages used in the training is included as ‘Train size’. the ICFHR18-G, and being fine-tuned on the 5 specific target data sets provided. For the sake of completeness we include in Table 5the results for 0 pages in the target dataset, i.e., when no labeled sampled from the target is used. Note that in this case DA-TL-DA cannot be applied. In the light of Table 4and Table 5, it can be concluded that applying DA over the target training set once TL is applied, i.e., DA-TL-DA, either does not reduce the CER or even it slightly increases it, compared to the result of the TL approach alone or the DA-TL method. Except for Schwerin, in which DA-TL-DA slightly improves DA-TL. Put in other words, in general, it is harmful to apply DA to the target dataset if TL has been applied, when just a reduced number of labeled lines are available in the target. On the other hand, DA+TL achieves improvements up to 5 % in the ICFHR 2018 target documents, usually increasing with the reduction of the training set. From the discussion above, and bearing Table 4and Table 5in mind, it can be concluded that DA-TL is a robust approach. When fine-tuning a DNN that has been previously trained with a similar task (a huge database of HTR samples), the starting point is reasonably good as we can observe in Table 5for the training set sizes of 0 pages in all datasets. We show a good generalization ability of the model for the TL and DA+TL without further training with the target dataset. Afterwards, the DNN model is trained with the target database. Only a few samples are available in the target set, which represent just a limited part of the support 76680 VOLUME 9, 2021 J. C. Aradillas et al.: Boosting Offline HTR in Historical Documents With Few Labeled Lines FIGURE 6. (a) CER (%) divided by the number of annotated lines, l, used and (b) decrement of CER (%) divided by the number of new labeled lines added to obtain it, 1l, in the training of the DNN model with DA-TL approach using the ICFHR18-G dataset as the source and the Konzil dataset as the target with no artificial errors (×) corrupted with artificial errors (). of its marginal distribution, PT(x). After TL, the parameters of the DNN encode information from both the source and the target training sets. At this point, we conjecture that by using DA in the target dataset and further refining the parameters, the DNN model overfits to the augmented versions of the target samples, forgetting the knowledge learned from the source one, that very much helps to transcript inputs out of the support generated by augmenting the target set. This leads to an increase of the final CER. V. THE CORRUPTED LABEL PURGING (CLP) ALGORITHM In this section, we focus on the impact of the number of lines and their quality in the target dataset on the learning process of the DNN model. We first analyze the impact of the performance on the number of healthy lines, i.e., lines with no transcription errors in the training dataset. Then we study how this performance degrades with label errors. Finally, we propose an algorithm to detect and remove potential label errors in the dataset. A. PERFORMANCE VARIATION WITH THE NUMBER OF LABELED SAMPLES When a small number of lines is available in the target training set, deep learning models are quite sensitive to small variations in the number of labeled lines. In this subsection, this sensitivity is evaluated on a specific dataset from the ICFHR 2018 Competition [5]. The chosen training dataset consists of 16 pages from the Konzil, which is segmented at line level. Similar results were obtained for the other datasets. The ICFHR 2018 target datasets have 16 labeled pages each. Unless otherwise indicated, 4 of them will be used for testing purposes while up to 12 pages will be used for training. Usually, 10 % out of the used training set is devoted to validation. The ICFHR18-G dataset is used as source database in the TL-DA approach. In Fig. 6.(a) the blue curve in ×represents the TL-DA CER versus the available number of lines, l, of the target training set in the range 29-350 lines, corresponding to 1 and 12 pages, respectively. In the left part of the figure, the CER decreases at a rate of 1 percentage point every 4 new lines added to the training set. After approximately 50 lines, the decreasing rate of the CER changes to approximately 1 percentage point every 100 lines. This is evidenced in Fig. 6.(b) where the absolute value of the variation of the CER in percentage points is depicted 1CER, with the increment of the number of annotated lines used in the target to achieve it, 1l. It can be concluded that the sensitivity to the number of samples in the training set is significantly larger for small training sets. In Fig. 6we also include the ‘‘Training set with errors’’ curve () which corresponds to the analysis above but where labeling errors have been artificially introduced, as follows. The annotation of a line is modified with probability L. Then, with modified labeling, a character is changed with probability R. In both cases following a Bernoulli distribution. Every changed character is replaced by an independently and VOLUME 9, 2021 76681 J. C. Aradillas et al.: Boosting Offline HTR in Historical Documents With Few Labeled Lines FIGURE 7. Sample of a completely mislabeled text at Ricordi dataset. FIGURE 8. Sample of special annotations in the GT at the Ricordi dataset. randomly selected character, following a discrete uniform probability. In Fig. 6, where L=0.2 and R=0.3, it is interesting to note that the impact of labeling errors in the CER value is more dramatic for small training sets while the rate at which the CER decreases with the number of lines added remains roughly unaltered. B. TYPES OF TRANSCRIPTION ERRORS Before proposing approaches to detect mislabels in the training set, we discuss three typical types and causes of errors in the datasets, as follows. 1) Mislabeled characters. When labeling a training set, the most common mistake is to confuse a character with another, usually a similar one. This can be seen in the well-known IAM database [41], where it is indicated that some lines could have some annotation errors in the labels. This type of error is the one simulated in Fig. 6. 2) Label Misalignment. The second kind of detected error happens due to a misalignment in the labels. This could be caused by, e.g., a mistake in the name given to some images in the database. This error is encountered several times in the Ricordi dataset from the ICFHR 2018 Competition [5] as illustrated in Fig. 7. It can be observed in this example that the transcript does not correspond to the handwritten text in the image above. On the contrary, it is quite close to the model output, after being trained with several lines of the dataset. 3) Special annotations in the ground truth. Perhaps, the most common source of error is due to special annotations that some transcribers or database managers introduce in some datasets to include some notes inline. In [37] they found this problem in the IAM database: crossed out words that are labeled with the symbol ‘‘#’’ followed by the word behind the blot. Training the model with this labeling might lead to unpredictable behavior, since the model could replace the text using ‘‘#’’ at different parts of the text. The model will either be able to recognize the text behind the blot or replace the word with the symbol ‘‘#’’, or both. Another special annotation is included in Fig. 8, where they write in brackets extra characters that are not in the handwritten text. The output of a model trained with samples of the same dataset is shown below the GT. Despite in this line, the CER is about 35%, it can be observed that the model output is quite similar to the handwritten text. Manually annotating historical documents remains a challenging task that is prone to errors, even for experts in the field. As discussed in the previous section, when a huge set of annotated samples is available, deep learning models do not suffer from a few mislabeled samples, as they better generalize. However, when a limited set of annotated lines of a specific writer is available to train, mislabeled lines induce overfitting to transcripts with errors, which are quite hard to tackle via regularization. In the example shown in Fig. 6we illustrate this problem when just a few mislabeled lines are introduced. Algorithm 1 Corrupted Labels Purging (CLP) Given inputs: source set xS∈XSand yS∈YS, target set xT∈XTand yT∈YTand threshold, . 1) Fit the prediction function fS(yS|xS) with the source training set {xS,yS}. 2) Split the target training set into Nsubsets {xT1,xT2,...,xTN},{yT1,yT2,...,yTN}. for n=1,...,Ndo 3) Initialize the prediction function fTn(·)=fS(·). 4) Fine tune the prediction function with the whole target set except for the nth, {xTi6=Tn},{yTi6=Tn}. 5) Include in the new target set, {xT0,yT0}, all pairs {x(i) Tn,y(i) Tn}whose predictions fTn(y(i) Tn|x(i) Tn) have errors below a CER threshold, . end for 6) Initialize the prediction function fT(·)=fS(·). 7) Fine tune the prediction function, fT0(yT0|xT0), to the modified target set {xT0,yT0}. Output: Function fT(yT|xT) over the target domain DT. C. MISLABEL DETECTION ALGORITHM As one of our main contributions, we propose an algorithm to detect and remove mislabeled lines from the training set, 76682 VOLUME 9, 2021