A neural network for semantic labelling of structured information

A neural network for semantic labelling of structured information Daniel Ayala a, ∗ , Agustín Borrego a , Inma Hernández a , David Ruiz a a Universidad de Sevilla, ETSI Informática. Avda. de la Reina Mercedes, s/n, Sevilla E-41012, Spain. Abstract Semantic labelling consists in assigning known labels to the data from a source of structured information. This can be useful in a variety of tasks related to information extraction and integration into information systems and their local ontologies. Semantic labelling can be seen as a classication problem in which the input is structured information from which features can be computed in order to apply machine learning techniques. The existing proposals, based on machine learning so far, have focused on what features should be computed while relying on simple classication models like logistic regression or random forest, and may not be powerful enough to properly classify some classes, especially in scenarios in which a large number of features contain the necessary information but it is hard for the classiers to properly combine them. In this paper, we propose and test the novel application of neural networks to semantic labelling, which benets from non-linearity and can deal with the increasing number of features. Our proposal has been validated with datasets from three real world sources, and our conclusion is that state-of-the-art neural networks consistently improve the accuracy of the labelling when compared to traditional classication. Keywords: Semantic labelling, Information Integration, Neural Networks 1. Introduction 1 The Web is a rich source of semi-structured data 2 which usually has to be integrated into information 3 systems before its exploitation (Knoblock et al., 4 1998). The rst step towards the integration in 5 one such system is the crawling of the Web to ob- 6 tain a set of HTML documents (Hernández et al., 7 2018, Batzios et al., 2008). The second step is to 8 extract structured information from them (Sleiman 9 and Corchuelo, 2013, Wang et al., 2007). The ex- 10 tracted structured information lacks semantics, so 11 the third step is to establish correspondences be- 12 tween the data and a known ontology. This is 13 the goal of semantic labelling, which consists in 14 labelling elements in data structures with known 15 classes from a Web ontology (Pham et al., 2016). 16 Semantic labelling proposals take the structured 17 elements as input, and assign them one or sev- 18 eral labels, which correspond to the classes that 19 ∗ Corresponding author Email addresses: [email protected] (Daniel Ayala), [email protected] (Agustín Borrego), [email protected] (Inma Hernández), [email protected] (David Ruiz) best describe each element according to its fea- 20 tures. Figure 1(a) shows an example of a structured 21 dataset from the Jisc repository (Jisc, 2018), dis- 22 playing labelled information about a R&D project 23 related to education. A semantic labelling pro- 24 posal would learn from the examples in this and 25 other datasets a classication model for each class, 26 such as "jisc:name", "jisc:title", or "jisc:start-date". 27 Then, when fed a new unlabelled dataset like the 28 one in Figure 1(b), it would iterate every element in 29 it and endow it with a known class. Consequently, 30 semantic labelling can be seen as a classication 31 problem in which the input is one of the elements in 32 the structure and the features are whatever aspect 33 are measured from them. In the former example, 34 instance I2 could be classied as a "jisc:title" after 35 an analysis of some of its features, including the 36 number of words that start with an uppercase let- 37 ter and the position of the instance in the structure, 38 I3 could be classied as a "jisc:start-date" because 39 of the number of digits, and I10 could be classi- 40 ed as a "jisc:status" because programme statuses 41 only have a few possible values ("Complete", "Run- 42 ning", etc.), and the value of the instance matches 43 Preprint submitted to Elsevier February 11, 2019 that of other known examples of the same class. 44 We can apply the same model to data from any 45 source in order to label it with the same known 46 classes, as long as the model was able to properly 47 learn what features can be used to identify each 48 class. Semantic labelling is therefore related to the 49 integration of heterogeneous information from dif- 50 ferent sources by modelling classes in structured 51 information. Beyond the direct integration of in- 52 formation, the modelling has other applications 53 such as information extraction (Banko et al., 2007) 54 (which, as we mentioned, is also a step of informa- 55 tion integration), information verication (Kushm- 56 erick, 2000, Lerman et al., 2003, McCann et al., 57 2005), or ontology matching (Euzenat and Shvaiko, 58 2013). These areas are all tightly related to the 59 Web and the integration of information from exter- 60 nal sources. 61 The current trend in the state of the art proposals 62 is to focus on feature engineering(Ayala et al., 2019, 63 Ramnandan et al., 2015, Neumaier et al., 2016, 64 Pham et al., 2016), that is, identifying new fea- 65 tures that endow the classier with enough power 66 as to discern between dierent classes, even when 67 those classes are highly similar like "jisc:name" and 68 "jisc:title". Devising elaborate features is crucial to 69 achieve good accuracy, and the most recent work 70 related to semantic labelling (Ayala et al., 2019) 71 has resulted in a large explosion of features, with 72 potentially hundreds of them. However, our study 73 of the literature reveals that existing proposals are 74 based on baseline classication techniques, neglect- 75 ing advanced classication techniques that use the 76 features eciently. The most recent proposals only 77 use random forest or logistic regression classiers, 78 and do not study more elaborate alternatives, leav- 79 ing room for improvement. 80 Our hypothesis is that neural networks can sig- 81 nicantly improve the accuracy of a semantic la- 82 belling model, while using the same initial low-level 83 features as a traditional classication model. While 84 some areas like Natural Language Processing, Com- 85 puter Vision, or even other tasks related to integrat- 86 ing information from external sources like informa- 87 tion retrieval from the Web have been transformed 88 by the successful application of modern neural net- 89 work technology (Deng and Yu, 2014), semantic 90 labelling has so far relied on the more traditional 91 machine learning techniques we have mentioned. 92 While the potential of neural networks has been 93 tested in some related tasks like information extrac- 94 tion, to the best of our knowledge it remains com- 95 pletely unexplored in the eld of semantic labelling, 96 which motivated us to study it as a novel applica- 97 tion, checking what strategies and architectures are 98 applicable and what results they achieve. Our ex- 99 periments, in which we use a neural network with 100 dense layers for semantic labelling in several scenar- 101 ios using real world data, reveal that the accuracy of 102 the labels improves consistently when compared to 103 four traditional classication techniques, even when 104 there is little margin for improvement. 105 The rest of the paper is organised as follows: Sec- 106 tion 2 reports on some preliminaries that are neces- 107 sary to understand the domain of the problem; Sec- 108 tion 3 describes the analysis of the relevant propos- 109 als we have identied in the literature; Section 4 de- 110 scribes the nature of features in semantic labelling; 111 Section 5 contains a detailed description of the ap- 112 plication of neural networks to semantic labelling; 113 Section 6 describes the experiments we used to test 114 our hypothesis and their result; nally, Section 7 115 recaps on our main conclusions. 116 2. Preliminaries 117 In this Section, we introduce denitions of con- 118 cepts related to the problem of semantic labelling. 119 Class: a piece of text that denotes semantics in 120 a Web ontology. The output of semantic la- 121 belling is a set of labels that should match 122 the class of every data item. Example: classes 123 "jisc:Project" and "jisc:start-date". 124 Attribute: A data item with a textual value that 125 can be an instance of a class and have a label 126 that denotes it. The textual value can repre- 127 sent a number, date, boolean, or any other data 128 type. Note that in this context, an attribute 129 does not refer to an element of the schema, 130 but to a specic data item. It may be possible 131 to have an attribute that does not belong to 132 any class in a particular ontology, i.e., a piece 133 of text that is automatically extracted from a 134 website by a crawler but does not correspond 135 to any known class. Example: in Figure 1(a), 136 one of the two attributes of class "jisc:name" 137 has a textual value of "Support & Synthesis 138 Project", and the attribute of class "jisc:start- 139 date" has a textual value of "01/08/2009". In 140 Figure 1(b) there are several attributes: I2 (a 141 name), I3 (a start date), I5 (a title), I6 (a de- 142 scription), I7 (a doi), I9 (a name), I10 (a home- 143 2 jisc:Project jisc:name – “ Support & Synthesis Project ” jisc:start-date - ”01/08/2009” jisc:Organization jisc:name - “ CETIS, University of Bolton ” jisc:Article jisc:title - “ Programme Definition ” jisc:description - “ Programme Definition document... ” jisc:status - “ In Progress ” jisc:Programme jisc:name - “ Flexible Service Delivery Programme ” jisc:homepage - “ http://www.jisc.ac.uk/whatwedo/... ” jisc:status - “ Running ” (a) Labelled dataset. I1 I2 - “ Physical Sciences Subject Portal for the RDN ” I3 - “19/12/2002” I4 I5 - “ Subject portal ” I6 - “ This proposal will develop a... ” I8 I9 - “ Infrastructure programme ” I10 - “ http://www.jisc.ac.uk/whatwedo/... ” I11 - “ Complete ” I7 - “ doi:12.3456/7890 ” (b) Unlabelled dataset. Figure 1: Dataset examples. page), and I11 (a status), but their class is un- 144 known by the system. Attribute I7 is clearly 145 a doi, but there is no doi class in the known 146 ontology, so it would have no class in it. 147 Record: a text-less data item that has other at- 148 tributes or records as children, may be an in- 149 stance of a class and have a label that de- 150 notes it. Record classes admit a certain de- 151 gree of variability in their schema, that is, 152 dierent records of the same class may have 153 variable attributes and records if some of 154 them are optional or have dierent multiplic- 155 ity. Example: in Figure 1(a) there are four 156 records. The "jisc:Project" record contains 157 instances of classes "jisc:name", "jisc:start- 158 date", "jisc:Organization", "jisc:Article", and 159 "jisc:Programme". Some of them are 160 also records with their own instances, like 161 the "jisc:Organization" record that has a 162 "jisc:name. Figure 1(b) also shows several 163 records: I1 (a project), I4 (an article), and 164 I8 (a programme). Note that I1 belongs to 165 class "jisc:Project", but it does not contain any 166 "jisc:Organization" record, since it is optional. 167 Dataset: a set of attributes and records in a hi- 168 erarchical structure. Usually, there is a single 169 root record at the rst level of the dataset, but 170 nothing prevents the presence of several ones, 171 having a forest-like structure. Example: Fig- 172 ure 1(a) displays a dataset with 4 records and 173 9 attributes, and the root is the "jisc:Project" 174 record. Figure 1(b) displays a dataset with 3 175 records and 8 attributes, and the root is the I1 176 record. 177 Model: a classier that takes attributes as the in- 178 put, and outputs their label. A model could 179 classify a single instance or a group of them. 180 Example: a random forest classier that takes 181 the attributes in Figure 1(b), computes some 182 features, and outputs a label for each of them. 183 Feature: a numeric or categorical measure that 184 can be taken from an attribute or group of at- 185 tributes. It can be seen as a function that takes 186 an instance or group of attributes as input and 187 outputs a feature value. Example: a feature 188 that computes the number of digits in the tex- 189 tual value of an attribute, which in Figure 1(b) 190 would output 0.0 for I2 and 8.0 for I3. 191 Internal model: a model that learns from a set 192 of examples (labelled attributes) by using fea- 193 tures obtained from the data item themselves, 194 without relying on external sources of data. 195 Example: a classier that computes features 196 related to the format of the attributes such as 197 the number of uppercase letters or the average 198 word length, and labels them using a random 199 forest or logistic regression classier. 200 External model: a model that learns from a set 201 of examples by using at least one feature 202 that requires an external knowledge base (e.g. 203 YAGO, DBpedia) to be computed. These fea- 204 tures are usually computed by mean of queries 205 to the knowledge base. Example: a classier 206 that queries DBPedia using the textual value 207 of attributes and labels them with the class of 208 the result with the highest score. 209 3 3. Related work 210 In the literature, there are several types of pro- 211 posals that are able to provide structured informa- 212 tion with labels that describe it. These propos- 213 als have dierent goals, but they can all be ap- 214 plied to the problem of semantic labelling, which 215 is why we include them in this analysis. Further- 216 more, these proposals work with dierent types of 217 features; however, in our analysis, we focus on the 218 type of classication technique on which they are 219 based, regardless of the specic features. Note that 220 none of them use neural networks, and instead use 221 more traditional techniques like random forest, lin- 222 ear regression, and nearest neighbour classiers. 223 The proposals by Limaye et al. (2010), Venetis 224 et al. (2011), Mulwad et al. (2013), Ritze et al. 225 (2015), and Zhang (2016) focus on labelling Web 226 tables, which may include labels for individual cells, 227 rows, columns, and relationships between columns. 228 Tables can be transformed into generic structures, 229 each row being a record, and its cells the attributes. 230 These proposals use knowledge bases to perform the 231 labelling. These contain a set of entities that belong 232 to classes, and usually oer the possibility of query- 233 ing them to obtain entities that seem to match the 234 query. In most cases, tables are labelled in an it- 235 erative process by rst obtaining a set of candidate 236 entities for each cell, then labelling the columns ac- 237 cording to the most frequent classes among the can- 238 didate entities, and then rening the candidates by 239 limiting them to the column classes. These propos- 240 als are based on external models, since the classi- 241 cation is ultimately based on the score of queries to 242 external sources, which in turn usually depends on 243 the TF-IDF score and cosine distances computed 244 from the documents in the knowledge base. The 245 labels are limited to the existing classes in the ex- 246 ternal source. 247 The proposals by Ramnandan et al. (2015), 248 Pham et al. (2016), Neumaier et al. (2016), and Ay- 249 ala et al. (2019) label attributes by comparing them 250 to sets of examples of known classes. The labels are 251 obtained through a classication process, based on 252 features such as the value of numeric attributes, 253 string distance metrics, similarity metrics, or fea- 254 tures related to the structure of the data. These 255 proposals are based on internal modes. The pro- 256 posal by Ramnandan et al. (2015) selects the class 257 with the highest score when querying a Lucene in- 258 dex that contains examples of a class in each stored 259 document. The proposal by Pham et al. (2016) 260 uses a one-vs-all logistic regression classier with 261 several similarity measures. The proposal by Neu- 262 maier et al. (2016) uses a nearest neighbour clas- 263 sier. The proposal by Ayala et al. (2019) uses a 264 one-vs-all random forest classier. 265 In addition to the former proposals, those by 266 Kushmerick (1999), Lerman et al. (2003) and Mc- 267 Cann et al. (2005) focus on information verication, 268 and their goal is to conrm that a dataset is correct 269 according to the reference model. They learn from 270 a number of veried labelled examples, they com- 271 pute the collections of values of each feature, and in- 272 fer the statistical normal distributions that best t 273 them. When a dataset must be veried, the values 274 of its features are compared to the inferred distribu- 275 tions. If some of the values associated to an element 276 or the entire dataset deviate too much from the ver- 277 ied ones according to statistical tests, the dataset 278 is considered to be anomalous. Information veri- 279 cation is very similar to semantic labelling, since 280 verifying an already labelled dataset amounts to ap- 281 plying semantic labelling to re-compute the set of 282 labels for the dataset and checking that the two sets 283 of labels are identical. 284 We have observed that the classication of in- 285 stances is not trivial when the number of classes is 286 large. The similarity between classes may be such 287 that even if the computed features hold enough in- 288 formation to dierentiate classes, their ecient use 289 by a model may require complex non-linear com- 290 binations that represent a challenge to most tech- 291 niques. For example, instances of classes "jisc:title" 292 and "jisc:name" are usually similar, and correctly 293 separating their classes could require a combina- 294 tion of several features related to their length, pres- 295 ence of certain characters or tokens, and other 296 measures. The existing proposals use techniques 297 that do not deal well with cases that require non- 298 linearity, which motivated us to implement the 299 novel application of neural network techniques to 300 semantic labelling. 301 4. Features 302 Features in the eld of semantic labelling do not 303 necessarily measure the occurrence of specic words 304 in the textual value of attributes; instead, they are 305 mostly related to its format, i.e., the kind of char- 306 acters and tokens it contains, how long it is, or how 307 similar it is to sets of examples according to dif- 308 ferent distance functions. The features catalogue 309 4 does not necessarily depend on the particular clas- 310 sication algorithm that is being applied, i.e., we 311 can create several classiers for semantic labelling 312 using the exact same features. 313 In the past, the features set used in related 314 proposals was limited to around a dozen fea- 315 tures (Kushmerick, 2000, Lerman et al., 2003, Mc- 316 Cann et al., 2005). However, the most recent work 317 has started to develop larger, more expressive sets 318 of features to include as much information as possi- 319 ble in the input. One of the recent additions are the 320 so-called parametric features (Ayala et al., 2019). 321 They are a kind of feature that ts well this need 322 to include as much low-level information as possi- 323 ble in the rst layer. They take a parameter, which 324 means that each parametric feature results in a fam- 325 ily of features, each of them related to a dierent 326 value of the parameter. The parameter can be one 327 of the known classes, so that each variant of the fea- 328 ture gives information related to it. For example, 329 feature F3 expands into 6 dierent features of the 330 same family. 331 Table 1 displays the nal features that we have 332 selected from the literature. Note that several fea- 333 tures are parametrical, three of them on a per class 334 basis. Features F1 , F2 , F3 , and F4 give information 335 about the textual format of the attribute. Fea- 336 tures F5 and F6 help detect starting and ending 337 patterns. Feature F7 measures overall similarity to 338 each class. Feature F8 gives additional informa- 339 tion when an attribute has a numeric value that 340 can be considered a feature itself. Features F9 , 341 F10 and F11 give information about the structure 342 in which the attribute is present. For example, 343 if we have trained a classier with three known 344 classes: "jisc:title", "jisc:name" and "jisc:start- 345 date", feature F7 , "Average edit distance", would 346 have three versions: "Average edit distance to ex- 347 amples of class jisc:title/jisc:name/jisc:start-date". 348 With three classes there would be a total of 35 fea- 349 tures. Since in the real world cases we have studied 350 there are usually several dozens of classes, paramet- 351 ric features can result in a features explosion which 352 is dicult to handle for traditional classiers. 353 5. Our proposal 354 In this Section we present the neural network we 355 have devised. First, we describe the application 356 workow in which the neural network is framed. 357 Then, we describe in detail the architecture of the 358 network. Finally, we justify the choices in the ar- 359 chitecture and analyse why some popular strategies 360 could not be applied. 361 5.1. Workow 362 Figure 2 summarizes the classication workow. 363 The original input is a dataset containing several 364 records and attributes. Each individial attribute 365 is fed to a features calculator that computes the 366 low-level features. The features must be any mea- 367 surement that we can take from the text of an at- 368 tribute and the structure of the dataset that con- 369 tains it. The neural network should benet from a 370 large number of low-level features that can later be 371 combined. 372 The features are used to create a vector that is 373 fed to the rst layer of the neural network, whose 374 size is always equal to the number of features. After 375 going through the hidden layers, the output layer, 376 whose size is always equal to the number of known 377 classes, gives a score to each class, which is used to 378 select the nal label. 379 A strengh of our proposal is that it labels individ- 380 ual instances as opposed to labelling a group of sev- 381 eral attribute instances that are known to share the 382 same class. For example, the proposal by Ramnan- 383 dan et al. (2015) would take as input a set of several 384 dozens or hundreds of instances and output a single 385 label for them. We consider individual labelling to 386 be a more challenging task due to the limited infor- 387 mation available during classication. One possible 388 real-world scenario in which the inputs are individ- 389 ual attributes is unsupervised information extrac- 390 tion (Roldán et al., 2017), which extracts general 391 useful information from web pages in generic vari- 392 able structures with no schema by means of univer- 393 sal rules that do not require training. However, the 394 application to groups of attributes would be trivial, 395 simply requiring a change of features, so that they 396 are computed from several instances instead of a 397 single one. 398 While structured datasets may include both 399 records and attributes, our application of neural 400 networks focuses on classifying attributes, so that 401 our results are comparable with those in the re- 402 lated work, which does not include the labelling 403 of records in many cases. However, the attributes 404 used for training and testing are still positioned in 405 a structured datasets, and consequently, features 406 can make use of the records or their structure (for 407 example, a feature could be "Number of adjacent 408 records"). 409 5 ID Feature Description F 1 (S) Number of occ. of symbol type S The number of occurrences in the attribute of symbols of type S (letters, numbers, punctuation, symbols, separators, other). The considered types can be customised. F 2 (T) Number of occ. of token type T The number of occurrences in the attribute of token of type T (words starting with a lowercase letter, words starting with an uppercase letter followed by a non-separator character, uppercase words, numeric strings, HTML tags). The considered types can be customized. F 3 (S) Density of symbol type S The density in the attribute of symbols of type S. The density is computed as the number of occurrences of a character type divided by the total number of symbols in the attribute. F 4 (T) Density of token type T The density in the attribute of token of type T. The density is computed as described in AF3 F 5 (C) Average shared prefix length for class C Average length of the shared prefix between the text of the attribute and a set of stored examples of class C. The shared prefix is the set of characters that two attributes have in common in the beginning. If the attributes start with a different character, the length is 0. F 6 (C) Average shared suffix length for class C Average length of the shared prefix between the text of the attribute and a set of stored examples of class C. The shared suffix is the set of characters that two attributes have in common in the end. If the attributes end with a different character, the length is 0. F 7 (C) Average edit distance to class C Average Jaro edit distance between the attribute and a set of stored examples of class C. F 8 Numeric Value The numeric value of the text of the attribute if it matches a number pattern. -1.0 otherwise F 9 Depth The depth in the dataset of the attribute. F 10 Same level attributes The number of attributes at the same structural level. F 11 Same level attributes The number of records at the same structural level. Table 1: Features. Numer of occurrences of numer symbols Ratio of numeric tokens Number of tokens starting with an uppercase letter Average shared prefix length for examples of class “ jisc:title ” Average shared prefix length for examples of class “ jisc:name ” Average shared prefix length for examples of class “ jisc:start-date ”  Average edit distance to examples of class “ jisc:title ” Average edit distance to examples of class “ jisc:name ” Average edit distance to examples of class “ jisc:start-date ” F 1 (number) F 2 (startinguppercase) F 3 (number) F 5 (jisc:title) F 5 (jisc:name) F 5 (jisc:startdate) F 7 (jisc:name) F 7 (jisc:title) F 7 (jisc:startdate) 0.00 1.00 0.00 0.57 0.61 0.13  0.68 0.87 0.92 0.00 1.00 0.00 0.23 0.72 0.19 0.93 0.23 0.95 8.00 0.00 0.80 0.11 0.03 0.77 0.89 0.98 0.50   “ S u b je c t p o r t a l ” “ I n f r a s t r u c t u r e p r o g r a m m e ” “ 1 9 / 1 2 / 2 0 0 2 ”  0.82 0.15 0.03 jisc:title jisc:name jisc:start-date 0.13 0.73 0.14 0.10 0.04 0.86 DescriptionFeature name Output score “ S u b je c t p o r t a l ” “ I n f r a s t r u c t u r e p r o g r a m m e ” “ 1 9 / 1 2 / 2 0 0 2 ” H id d e n la y e r s Feature vectors Input dataset Features calculator Figure 2: Workow. 6 5.2. Architecture 410 Figure 3 summarises the architecture of our net- 411 work. Keep in mind that we have devised a multi- 412 purpose architecture for any scenario. However, it 413 could be adapted for a specic situation. For exam- 414 ple, the size of the hidden layers could be increased 415 or decreased in concordance with the number of fea- 416 tures (the size of the input layer). The following 417 paragraphs describe the architecture, which is jus- 418 tied in the next subsection. 419 Our network has three wide, fully connected hid- 420 den layers (each neuron in a layer is connected to 421 every neuron in the next layer). Their sizes are 422 2048, 1024 and 512. The size of the input layer is 423 equal to the number of initial features, and that of 424 the output layer, equal to the number of classes. 425 We have applied dropout, a probability of setting 426 a value being transmitted between layers to 0 in 427 order to decrease overtting. The dropout rates 428 of the layers are 0.01 , 0.1 and 0.1 . We have set 429 ReLU as the activation function of all intermediary 430 layers, and cross entropy as the loss function, since 431 it is applicable to multiclass classication. 432 The nal layer outputs the score of each label 433 after a softmax function from which we select the 434 one with the highest score. The user could also 435 choose not to accept a label below a given threshold. 436 The softmax function takes a vector of real values 437 and turns it into a new vector of real values in the 438 (0,1) range that add up to 1. 439 5.3. Discussion 440 Next, we justify our choices with regards to the 441 architecture, and oer some insights on why we did 442 not include some popular neural network strategies. 443 A popular machine learning practise is data aug- 444 mentation (Witten et al., 2016), which consists in 445 expanding the number of data points (in this case, 446 attributes used for training) by creating new syn- 447 thetic ones, derived from the original ones by means 448 of transformations that create dierent but still 449 valid data. For example, in computer vision this 450 can be done by panning, zooming, or rotating the 451 input images. Implementing data augmentation in 452 semantic labelling would require manually creat- 453 ing transformation functions that slightly alter at- 454 tributes while keeping them valid. For example, 455 one such transformation could be to add the coun- 456 try code to phone numbers, so that apart from the 457 training example "954123456", there is the exam- 458 ple "+34 954123456". For dates, we could create 459 several training examples for a particular date by 460 changing the date format. 461 Transformations would have to be created for 462 each of, potentially, several dozens of classes. Their 463 creation is not trivial, and it would be needed to 464 check that a transformation does not worsen train- 465 ing, i.e., always adding the same country code to 466 phone numbers would lead to overtting. More- 467 over, while some attributes allow simple changes of 468 format like the aforementioned ones, others would 469 require more complex alterations, such as classes 470 "jisc:description" or "jisc:homepage". Altering a 471 description would require somehow changing its 472 contents while keeping it a valid description, and 473 altering a homepage would require changing some 474 parts of the url while keeping it a valid homepage. 475 At this point, it is clear that the necessary analysis 476 to determine when transformations of the original 477 data can be applied to attributes of a class, and the 478 manual work needed to create them is so large, that 479 it would be easier to manually dene rules to label 480 attributes. Therefore, data augmentations does not 481 seem to be applicable to semantic labelling. 482 Regarding the layer types, we decided not to in- 483 clude some layer types like convolution or pooling 484 layers (LeCun et al., 2015). These and other similar 485 layers aggregate the values of a region of "nearby", 486 related features from a features vector, for example 487 with a weighted mean (convolution) or by taking 488 the maximum value (pooling). Evidently, these op- 489 erations can only be performed when there is some 490 kind of relation between features of the input that 491 allows us to identify regions of nearby features, as 492 is the case with pictures and sounds: the features 493 from an image (the value of its pixels) have two 494 spatial dimensions, and the features of a sound sig- 495 nal (the value of the samples) have a temporal one. 496 Even in NLP tasks where the input is a sentence 497 of a xed size and there is a feature for each word 498 of the sentence, we can apply convolution or pool- 499 ing to groups of embeddings from nearby words. In 500 semantic labelling, however, features are mostly re- 501 lated to the format of attributes, and there is no 502 relation between them that makes it reasonable to 503 talk about a region of features from which the mean 504 or maximum is computed. 505 Regarding the amount and size of layers, since 506 the initial features already have some level of ab- 507 straction, the network should not require a large 508 depth to be eective, and three layers should be 509 enough. The number of layers is in line with other 510 architectures related to structured data in dier- 511 7  jisc:title jisc:name jisc:start-date 0.10 0.04 0.86 N = # of features S o f t m a x 8.00 0.00 0.80 0.11 0.03 0.77 0.98 0.50  “ 1 9 / 1 2 / 2 0 0 2 ” 0.89 D r o p o u t = 0 . 0 1 N = 2048 R e L U N = 1024 R e L U D r o p o u t = 0 . 1 N = 512 R e L U D r o p o u t = 0 . 1 N = # of classes R e L U Figure 3: Architecture of our network. ent tasks (Kazemi and Poole, 2018, Huang et al., 512 2015, Leng and Jiang, 2016), and is enough to allow 513 nonlinear combinations of the input features which 514 should correspond to more complex textual formats 515 and data structures. The decreasing size helps force 516 the abstraction of features and avoid overtting. 517 To the best of our knowledge, there is no way to 518 determine the optimal value for hyperparameters in 519 a completely unsupervised way. The dropout prob- 520 ability in the rst layer is very low to preserve most 521 of the information in low-level features, while it is 522 higher in the later layers that correspond to more 523 abstract features. The exact value of hyperparam- 524 eters were selected by ne-tuning the network in 525 tests, using values that seem to be popular and 526 make sense, i.e. a dropout value no bigger than 527 0.2 . Changing them (for example, adding some ad- 528 ditional layers or increasing dropout) did not seem 529 to have a signicant impact. 530 The softmax functions is an appropriate choice 531 for the output layer, since each input is only given 532 a single label. Note that, if several labels per in- 533 stance are wanted, it is enough to replace it with a 534 dierent function without altering the architecture 535 of the network. 536 6. Experimental analysis 537 The experimental validation of our proposal con- 538 sists in performing semantic labelling on individ- 539 ual attributes in three dierent scenarios with real- 540 world datasets, which have been selected for their 541 high number of classes: 542 NSF Datasets from the National Science Foun- 543 dation Awards database (Foundation, 2018a), 544 corresponding to the rst 500 awards with the 545 latest end date in 2017. 546 Newcastle Datasets from the Newcastle Univer- 547 sity repository (University, 2018), correspond- 548 ing to article references. We set up a SPARQL 549 server using the rdf dump, queried it to obtain 550 resources with class "akt:Article-Reference", 551 and used the rst 250 results, each as the root 552 of a dataset where linked resources are records 553 and data properties are attributes. 554 Jisc Datasets from the Jisc repository (Jisc, 2018), 555 corresponding to projects. We obtained 250 556 datasets in the same way as the Newcastle Uni- 557 versity datasets, using class "jisc:Project" as 558 the root of each dataset. 559 8 Scenario Root class # of datasets # of classes # of attributes # of features NSF nsf:award 500 34 17,723 135 Newcastle akt:Article-Reference 250 23 7,657 102 Jisc jisc:Project 250 18 9,985 87 All Variable 1,000 75 35,365 258 Table 2: Scenarios. All The datasets from the former 3 scenarios, 560 added up. 561 Table 2 summarises some statistics about them. 562 The number of features is obtained after fully com- 563 puting all the parametric features in Table 1 564 The data we used in our experiments, including 565 the computed features, have been made available 566 online 1 for the sake of reproducibility. 567 We compare the results obtained by the dense 568 network architecture we described to the following 569 one-vs-all classiers, which are common in the liter- 570 ature (Ayala et al., 2019, Pham et al., 2016), since 571 they ease the separation of one class from the rest 572 when there is a large number of classes: 573 • A random forest classier with 20 trees, and 574 maximum depth of 5 . 575 • A logistic regression classier. 576 • A linear SVC classier with a maximum of 20 577 iterations, and tolerance of 10−4 . 578 • A gradient boosted trees classier with a max- 579 imum of 20 iterations. 580 We used the Spark (Foundation, 2018b) implemen- 581 tation of all classiers, leaving all the unspecied 582 hyperparameters at their default value. 583 For the implementation of our neural network, 584 we used PyTorch (PyTorch, 2018). We used a sin- 585 gle neural network as a multiclass classier. The 586 training of the neural network consisted of 5 train- 587 ing cycles of length 3 (15 epochs total) with learning 588 rate 10−3 , 2 training cycles of lengths 4 and 8 (12 589 epochs total) with learning rate 0.5∗10−3 , and 2 590 training cycles of lengths 4 and 8 (12 epochs to- 591 tal) with learning rate 0.1∗10−3 . In each fold, we 592 took the best accuracy among all 39 epochs. The 593 starting learning rate was determined by using the 594 technique described by Smith (2017), in which the 595 1 http://www.tdg-seville.info//Download.ashx?id=490 learning rate is set to a small value and progres- 596 sively increased, showing the point at which the 597 loss starts to increase. We diminish the learning 598 rate in the later cycles to allow subtler changes in 599 the weights. Further cycles did not improve the 600 results. 601 We set the batch size to 16, which achieved the 602 best results in optimal time, though this value could 603 vary depending on the size of the training sets. 604 We have used 10-fold cross validation, measur- 605 ing accuracy (fraction of correct labels), since it is 606 the most appropiate metric for multiclass problems 607 such as semantic labelling. Figure 4 shows the ac- 608 curacy achieved by the traditional classiers and 609 the dense network implementation in a box plot, 610 with separated results for each scenario, applying 611 10-fold cross validation. Table 3 shows a numerical 612 summary. Dense networks achieve better accuracy 613 consistently, even in the cases in which traditional 614 classiers have a high accuracy ("Newcastle" and 615 "Jisc"), where there is a dierence of approximately 616 2.7 percent points (in the median) when compared 617 to the best traditional classier (random forest). In 618 the "NSF" scenario, where results are worse overall 619 showing a greater labelling diculty, the improve- 620 ment is of 4.6 points. In the "All" scenario, the 621 most complex one because of the high number of 622 classes, the improvement is of 8.9 points. It could 623 seem strange that classiers achieve very similar, 624 and in some cases even better results in the "All" 625 scenario than in the "NSF" scenario, which has a 626 lower number of existing classes. This is caused by 627 the fact that we add relatively easy to classify cases 628 from the "Jisc" and "Newcastle" scenarios to the 629 harder "NSF" scenario, increasing the average ac- 630 curacy. However, the easier cases become harder to 631 classify due to the higher number of classes. The 632 classication power of the dense network classier 633 is most visible in "dicult" scenarios, such as those 634 in which there is a large number of classes or highly 635 similar classes, in which the dierence in accuracy 636 is more noticeable. 637 Note that the dense network approach only 638 needed a single multiclass classier to outperform 639 the one-vs-all classiers despite the high number of 640 classes, which was a cause for concern. 641 To prove the signicance of the dierences, we 642 have applied the Wilcoxon signed ranked test. In 643 all scenarios, the p-value is below 0.002 . Since it 644 is lower than the standard signicance level of α= 645 0.05 , we reject the null hypothesis that dierences 646 in distributions are caused by chance. 647 9