Broad phonetic class definition driven by phone confusions

RESEARCH Open Access B oad phone ic class de ini ion d i en by phone con usions Ca la Lopes 1,2* and Fe nando Pe digão 1,3 Abs ac In e media e ep esen a ions be ween he speech signal and phones may be used o imp o e disc imina ion among phones ha a e o en con used. These ep esen a ions a e usually ound acco ding o b oad phone ic classes, which a e de ined by a phone ician. This a icle p oposes an al e na i e da a-d i en me hod o gene a e hese classes. Phone con usion in o ma ion om he analysis o he ou pu o a phone ecogni ion sys em is used o ind clus e s a high isk o mu ual con usion. A me ic is de ined o compu e he dis ance be ween phones. The esul s, using TIMIT da a, show ha he p oposed con usion-d i en phone clus e ing me hod is an a ac i e al e na i e o he app oaches based on human knowledge. A hie a chical classi ica ion s uc u e o imp o e phone ecogni ion is also p oposed using a disc imina i e weigh aining me hod. Expe imen s show imp o emen s in phone ecogni ion on he TIMIT da abase compa ed o a baseline sys em. In oduc ion B oad phone ic classes (BPC) ha e widely been used in speech ecogni ion esea ch as, o ins ance, au oma ic language iden i ica ion [1]; speaking a e es ima ion [2]; mul i lingual sys ems [3,4] and, especially, in phone ecog- ni ion [5-8]. I s success is due o he ca ied addi ional in- o ma ion ha con ibu es o imp o e he ecogni ion a es, especially unde noise condi ions [9]. In phone ec- ogni ion ask, BPC in o ma ion may be used as an add- i ional se o acous ic ea u es o i may be in eg a ed in he phone p edic ions. One success ul example is p e- sen ed by Siniscalchi e al. [6], one o he bes esul s epo ed on he phone TIMIT ecogni ion ask, whe e 15 b oad a icula o y classes a e used o p edic pos e io phone p obabili ies and o esco e phone la ices. Ano he in e es ing example is gi en by Mo is and Fosle -Lussie [7] whe e he ou pu s o eigh b oad class classi ie s a e used as inpu ea u es on a condi ional andom ield (CRF) model. In [8], a hie a chical classi ica ion om suc- cessi e b oad classes is used wi h imp o emen s in phone accu acy. Se e al o he s udies ela ed o BPC could be e- e ed, e.g. [5,10], some imes using di e en e minology. In li e a u e, BPC ake di e en names (such as b oad phone ic g oups, speech a ibu es, e en s, e c.) bu in all hese s udies hey a e always se s o phones wi h simila acous ic/phone ic ea u es d awn manually by an expe (knowledge-d i en in o ma ion). The b oad classes a e selec ed acco ding o acous ic-phone ic p ope ies ha de- i e om a icula o y cons ain s o om hea ing pe cep- ion. This selec ion may con ain some subjec i i y o may be di icul o ca y ou when dealing wi h o he kind o speech uni s like syllables o when conside ing coa icu- la ion phenomena. BPC o he English language is an issue ha has widely been add essed by he scien i ic communi y [5,7,8,11]. I s de ini ion is usually ela ed o he manne and place o a - icula ion, so ha all he b oad classes show good ag ee- men wi hin some phone ic, a icula o y and/o acous ic p ope ies. The e icien cons uc ion o smalle /mo e compac phone se s is no i ial, howe e , as is con i med by he lack o consensus be ween se e al p oposals. Fo example in he TIMIT co pus, in [12], phone [dx] is classi- ied as a sono an consonan , while in [13] he same phone is classi ied as a s op and Halbe s ad and Glass [10] classi y i as a nasal/ lap. The same happens wi h phone [h ]. In [12], i is a sono an consonan and in [5,10] i is a ica i e. Compa ing he “ ica i e”b oad classes o [5,14] i can be seen ha he i s p oposal includes he phone [h ] while he second does no . The abo e examples a e o show he subjec i e na u e o he app oaches based on human knowledge, e en when e e - ing o such a widely s udied language as Ame ican English. * Co espondence: [email p o ec ed] 1 Ins i u o de Telecomunicações, Polo II, Coimb a P-3030-290, Po ugal 2 ESTG, Ins i u o Poli écnico de Lei ia, Campus 2, Lei ia P-2411-901, Po ugal Full lis o au ho in o ma ion is a ailable a he end o he a icle © 2012 Lopes and Pe digao; licensee Sp inge . This is an Open Access a icle dis ibu ed unde he e ms o he C ea i e Commons A ibu ion License (h p://c ea i ecommons.o g/licenses/by/2.0), which pe mi s un es ic ed use, dis ibu ion, and ep oduc ion in any medium, p o ided he o iginal wo k is p ope ly ci ed. Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 h p://asp.eu asipjou nals.com/con en /2012/1/158 This subjec i i y a ec s no only he numbe o classes, bu also he se o phones in each ca ego y. These p oblems, ela ed o he expe -d i en app oaches, ha e p omp ed he eme gence o da a-d i en app oaches, whe e he classes’ composi ion is guided by da a. We explo e in his a icle a da a-d i en me hod whe e he b oad classes a e au oma ic- ally de ined acco ding o he ou pu o a speech ecogni ion sys em. The me hod can be applied in sys ems using all kinds o ecogni ion uni s (phones, syllables, subwo ds, phones o di e en languages, e c.) and do no conduc o a s a ic di ision. The ou pu o a classi ica ion sys em is usually e alu- a ed compa ing all he ecognized sequences wi h he co esponding e e ences. F om his compa ison, a con- usion ma ix can be compu ed. In he p oposed ap- p oach i is conside ed ha i a uni (phone o o he ) has much con usion wi h o he uni is because, o he ecognize , hey a e somehow simila . This app oach may no ully ag ee wi h phone ics p inciples o acous ic heo ies bu i can be e y help ul allowing o o e come deadlocks in some si ua ions. Take he case o a mul i- lingual sys em, whe e a knowledge-based app oach has o in ol e expe s om all he languages in ol ed. The same occu s when dealing wi h a ie ies o a language (e.g., Eu opean Po uguese o B azilian Po uguese). The p oposed au oma ic clus e ing me hod ex ends he pos- sibili y o using b oad classes’in o ma ion o sys ems based on o he han he phone uni . Da a-d i en clus e ing usually s ands o a s a is ical measu emen o a class. The key poin o all clus e ing algo i hms is he choice o a p oximi y o dis ance meas- u e. This measu e can be ob ained om acous ic mod- els, e.g., [3,15], o e en ely on he con usion ma ix, e.g., [4,16]. Model-d i en me hods and con usion- d i en a me hods a e hen he wo majo ca ego ies o da a- d i en phone clus e ing algo i hms. In model-d i en me hods, he acous ic simila i y be- ween wo phones can be achie ed om he heo e ical dis ance be ween he co esponding acous ic models. This dis ance can be he Bha acha yya dis ance be ween wo Gaussian mix u e models [3,15]; a ela i e en opy- based (Kullback–Leible di e gence) dis ance be ween wo Laplacian mix u es [17], e c. In [18], a da a-d i en phone ic b oad class gene a ion is p oposed whe e mu- ual in o ma ion is used o compu e simila i y be ween models, while in [19] a simila i y measu e based on he likelihood be ween he acous ic ames and he hidden Ma ko models (HMMs) is p oposed. In his a icle, we p opose a con usion-d i en me hod o gene a e phone clus e s. This app oach was al eady p oposed in [4], whe e he phones a e g ouped using ules depending on a se o weigh s and h esholds. Ou p oposal di e s om [4] in so a as we de ine a me ic o compu e phone dis ances. The emainde o he a icle is o ganized as ollows. In Sec ion 2, we in oduce he da a-d i en app oach and Sec- ion 3 p esen s he me ic de ined ha e alua es he phones simila i ies. I also desc ibes he way ha he con- usion ma ix (which is he base o phone simila i ies compu a ion) was achie ed. Sec ion 4 p esen s a sys em whe e b oad classes a e used in o de o enhance phone ecogni ion, and in Sec ion 5 expe imen al esul s, com- pa ing da a-d i en and knowledge-d i en app oaches, a e p esen ed. Finally, some conclusions and u u e imp o e- men s a e d awn in Sec ion 6. Da a-d i en b oad classes In con usion-d i en me hods, he simila i y measu e ap- p oach comes om he phone con usion ma ix M. This ma ix is compu ed om he ou pu o a phone ecognize by aligning he ecognized sen ence wi h he e e ence one, using a dynamic p og amming algo i hm ( he Le ensh ein algo i hm). I includes he concep o hi s and con usions, as well as inse ions (INS) and dele ions (DEL). An example o pa o a con usion ma ix o ou phone ecognize using TIMIT [20] da a is shown in Figu e 1. The diagonal e e s o he numbe o co ec ly ecognized phones (hi s) and he o -diagonal elemen s e e o he numbe o misclassi ica ions. The idea behind con usion-d i en me hods is ha simi- la phones end o be mo e con usable and should belong o he same class. Figu e 2a shows, in pseudo colou s, a con usion ma ix using he 61 o iginal TIMIT phones, whe e he phones a e in alphabe ical o de and wi h blue ep esen ing he lowe alue and da k ed he highe alue. I highly con usable phones a e eposi ioned in such a way ha hey become nea each o he , we ge he ma ix depic ed in Figu e 2b. In his second ma ix, we can easily dis inguish se e al se s (clus e s) o phones whe e con u- sion be ween all he elemen s o he clus e is much highe han be ween o he phones o phone clus e s. In an a emp o ind his se o clus e s, his s udy explo es RECOGNISED AS aa ae ah ao aw ax DEL SOURCE REFERENCE aa 125 ae 88 ah 111 ao 113 aw 14 ax 456 8 52 87 16 3 12 448 23 2 10 5 45 31 369 16 9 71 67 2 21 441 8 7 16 14 6 6 121 0 5 2 64 17 8 592 217 INS 21 21 19 17 14 30 Figu e 1 Pa o a con usion ma ix: ou pu o ou TIMIT-based phone ecogni ion sys em. Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 2 o 12 h p://asp.eu asipjou nals.com/con en /2012/1/158 an au oma ic classi ica ion me hod, whe e he phone ic classes a e ound acco ding o he ou pu o an au oma ic phone ecogni ion sys em. Phones a e g ouped acco ding o a simila i y measu e es ima ed om he con usion ma ix gi en by he ecognize pe o mance. Con usion-d i en phone dis ance measu e This sec ion desc ibes an au oma ic me hod o phone ic class gene a ion based on a con usion ma ix o a phone ecognize . Usually, he clus e ing echniques in ol e h ee concep s: 1. A da a model; 2. A p oximi y c i e ion (simila i y, dis ance, e c.); 3. A clus e ing algo i hm ha gene a es he clus e s (b oad classes) using he da a model and he simila i y measu e. These concep s a e discussed in ollowing sec ions. Da a model In he p oposed me hod, he da a comes om a con usion ma ix yielded by he pe o mance analysis o a phone ec- ogni ion sys em. This ma ix may con ain esul s a he ame le el, i a i icial neu al ne wo k (ANN), suppo ec o machines (SVM), o CRF-based ecognize s a e used, o a he segmen le el, i segmen models a e used, such as HMMs o hyb id sys ems, as in he p esen case. An MLP/HMM hyb id sys em is used ha combines an o e all HMM s uc u e wi h he class p edic ions gi en by a mul ilaye pe cep on (MLP) classi ie , hus bene i ing om he ime modelling abili ies o HMMs and he dis- c imina ion capabili ies o ANNs. In he TIMIT aining se , his in ol es he ecogni ion o 143 k segmen s. Hyb id MLP/HMM desc ip ion An MLP ne wo k, wi h a single hidden laye , was ained o phone classi ica ion a a ame le el. The las laye pe - o ms a 1- o-61 classi ica ion o e he se o (TIMIT) phones. Speech was analysed e e y 10 ms wi h a 25-ms Hamming window. Thi y-nine pa ame e s we e used as s anda d inpu ea u es ep esen ing 12 Mel F equency Ceps al Coe icien s (MFCCs), plus ene gy, and hei i s and second ime-de i a i e coe icien s. The con ex win- dow used was 170 ms bu only 9 ame ea u es we e used, as desc ibed in [8]. The cu en ame is in he cen e o he con ex window ( empo al in o ma ion o pas and u- u e is included). The so max unc ion was used as he ac- i a ion unc ion o he ou pu laye so ha he ou pu alues could be in e p e ed as pos e io p obabili ies. The hidden laye has 1,000 nodes and uses a sigmoid ac i a ion unc ion. All he ne wo k weigh s and bias a e adjus ed using ba ch aining wi h he esilien back-p opaga ion (RPROP) algo i hm [21], so as o educe he minimum- c oss-en opy e o be ween ne wo k ou pu and he a ge alues. The ne wo k has 413 k ee aining pa ame e s. HMMs we e buil be o ehand o each phone using HTK3.41, [22], in o de o es ima e he ansi ion p ob- abili ies be ween s a es. Each phone was modelled by a h ee-s a e le - o- igh HMM and each s a e was mod- elled by a single Gaussian model. The inpu ea u es we e he same as o he MLP. In he hyb id MLP/HMM sys em, he s a e likelihoods a e eplaced by he pos e io p obabili ies gi en by he ou pu p edic ions o he MLP. The h ee s a es sha e he same MLP ou pu . We used HTK [22], wi h some changes in o de o eplace he usual Gaussian mix u e models wi h he ou pu s o he MLP. The con usion ma ix (da a model) is compu ed using he en i e TIMIT aining se , which consis s o all si and sx sen ences o he o iginal aining se (3,698 u e - ances). The pe o mance o he hyb id sys em is usually e alua ed by means o Co ec ness (Co ) and Accu acy (Acc). We ha e used HTK e alua ion ool HResul s o Figu e 2 (a) Con usion ma ix o ou TIMIT-based phone ecogni ion sys em. (b) A eposi ioned e sion o he ma ix depic ed in (a) whe e highly con usable phones a e place nea each o he . Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 3 o 12 h p://asp.eu asipjou nals.com/con en /2012/1/158 compu e hem. Bu wi h his ool only he oken se- quence is aken o e alua ion. A ine e alua ion akes in o accoun no only he co ec iden i ied sequence o phones, bu also hei ime localiza ion. This was he c i- e ia used in he con usion ma ix gene a ion on which he p oposed clus e ing is based. A b ie desc ip ion o he e alua ion p ocedu e used is gi en below. A ull de- sc ip ion can be ound in [23]. The e alua ion p ocedu e uses a modi ied Le ensh ein algo i hm [24] in he alignmen be ween labelled and ecognized phones, whe e he deg ee o o e lapping be- ween hem is aken in he local dis ance de ini ion. This algo i hm inds he bes alignmen be ween wo s ings and inse s a penal y i an e o occu s (inse ion, dele- ion and subs i u ion), bu no penal y is applied i he labels ma ch. In ou p oposal, we include an addi ional penal y ha is p opo ional o he a e age o he le and igh misalignmen s. I he labels do no o e lap (T OV ≤ 0 in Figu e 3), his penal y is se o a maximum alue (p max ), such ha an inse ion o a dele ion will be p e- e ed o a misaligned subs i u ion. Taking i1; i2and j1; j2as he bounda ies o he es and e e ence labels, as indica ed in Figu e 3, hey o e lap i i2 > j1 o j2 > i1 and hen he o al ime o he labels is T¼max i2; j2  min i1; j1  ¼T1þT2þTOV ð1Þ and he o e lapping ime is TOV ¼min i2; j2  max i1; j1  ð2Þ The le and igh misalignmen s a e T1¼ j1 i1  and T2¼ j2 i2 . I he labels lab i and lab j ma ch hei name bu a e no pe ec ly aligned, hen we in oduce an addi ional associa ion penal y, p A (i,j), which is in e sely p opo ional o he o e lap be ween he labels, acco ding o he ollowing exp ession: pAi;jðÞ¼ T1þT2 ðÞ=2 TOV ¼1 2 T TOV 1  ð3Þ I he labels o e lap mo e ha 50%, p A is smalle han 0.5. As a as he o e lapping dec eases, his dis ance inc eases and is clipped o p max =15, which co esponds o 3.2% o o e lapping. In o de o p omo e con usions (subs i u ions) we penalize inse ions and dele ions signi i- can ly by se ing p INS =12 and p DEL =12. The op imal alignmen is ound by acing back he pa h o accumu- la ed penal ies om he las label pai o he o igin o he (i,j) g id. Wi h he alignmen o all labels a con usion ma ix can hen be compu ed. This ma ix ep esen s he da a model e e ed o in he beginning o Sec ion 3. Simila i y measu e The con usion ma ix is o en con e ed in o a symme - ic simila i y ma ix using he so-called Hou gas algo- i hm. I measu es he simila i y be ween e e ence phones iand jusing he numbe o con usions o hese phones wi h all o he phones k. Mo e speci ically, i Nis he o al numbe o classes (phones), he Hou gas simi- la i y be ween phones iand j,sij is gi en by sij ¼sji ¼X N k¼1 min ik; jk  ð4Þ wi h 1≤i;j≤N.I i=j, ij is he numbe o hi s ins ead o con usions. Acco ding o his measu e, wo phones iand j a e simila i hey bo h ha e many con usions wi h he same phones and hei simila i y is ze o only when phones iand jha e no simul aneous con usions wi h any phones. Figu e 4 gi es a simple example o his measu e. The con usion ma ix o a good phone ecognize is close o a diagonal ma ix. The ou -diagonal alues ep e- sen misclassi ied phones (con usions be ween phones). Since he numbe s o occu ences o each phone a e qui e di e en (due o phone ically unbalanced speech ma e ial) we no malize he con usion ma ix by di iding he e- quency coun s ij by he o al numbe o occu ences o he phone iin he speech da abase: pij ¼ ij P N n¼1 in ¼P^ cjci jÞ ð5Þ In his way, we ha e an es ima e o he p obabili y o ecognizing he model o clus e ^ cjwhen i s e e ence class is c i . In his case he Hou gas simila i y measu e becomes s′ ij ¼X N n¼1 min pin;pjn  ¼X N n¼1 min P^ ci ðjcn ðÞ;P^ cj cnÞÞ ð6Þ T T 1 T 2 T OV Re e ence label, labj Tes label, labi i1 j1 i2 j2 Figu e 3 Measu emen o he ime misalignmen be ween wo labels. Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 4 o 12 h p://asp.eu asipjou nals.com/con en /2012/1/158 This new measu e has he ollowing p ope ies: (i) s′ ij≤1 and (ii) s′ ii ¼1 . Because min a;bðÞ¼ 1 2aþbab jj ðÞ and because PN n¼1pin ¼1 , a dis ance measu e be ween phones iand jcan he e o e be de ined as d1ci;cj  ¼21s′ ij  ¼X N n¼1 pin pjn ð7Þ This dis ance o ms a me ic because i is he L 1 no m applied o ow di e ences o ma ix P (wi h elemen s p ij as de ined in (5)). Se e al simila i y measu e p oposals can be ound in li e a u e [4,16,25], bu as e e ed in [26], hey do no ul il he p ope ies o a p ope me ic. In he p esen p oposal, d1has he h ee me ic p ope ies: i is posi i e, symme ic and sa is ies he iangle inequali y. O he dis ance measu es can be de ined based on he same p inciple, in pa icula he Euclidean dis ance o ows, d2ci;cj  ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi XN n¼1pin pjn  2 ð8Þ Clus e ing me hod The p oposed clus e ing me hod g oups phones in a mul ile el hie a chy whe e clus e s a one le el a e com- bined as clus e s a he nex le el. Clus e ing can be achie ed ollowing hie a chical agglome a i e clus e ing pa adigm [27,28] as ollows. Ini ially, each phone will be conside ed as a dis inc clus e . S ep 1 !Compu e ma ix P using (4). S ep 2 !Find he dis ance be ween each pai o phones using (7). S ep 3 !Compu e he dis ances be ween all clus e s. The dis ance be ween clus e s and s can be compu ed wi h se e al c i e ia, o which he simples is he nea es neighbou :d ;sðÞ¼min dc ij ;cjs jÞÞ;i¼1...n ;  j¼ 1::nswhe e n k is he numbe o phones in clus e k and c i|k is he i h phone in clus e k. S ep4 !C ea e a new clus e by g ouping he wo nea es ones. S eps 3 and 4 will epea un il he numbe o desi ed clus e s o a dis ance h eshold is eached. Ano he pos- sibili y is o compu e clus e s un il all he phones belong o he same clus e . A dend og am can be buil om hese clus e s, o allow he decision, he le el o scale o clus e ing ha is mos app op ia e o he applica ion. TIMIT phone clus e ing esul s S a ing om he da a model desc ibed in Sec ion 3.1 and ollowing he s eps p oposed in Sec ion 3.3, we a - i e a a hie a chical phone clus e ing, depic ed as a bin- a y clus e ee (dend og am) in Figu e 5. The labels along he ho izon al axis ep esen he phones in he o iginal da a se and e ical axis e e s o he dis ance be ween he phones. This dis ance is compu ed om a simila i y measu e using d1ci;cj  . The numbe o clus- e s depends on whe e he ee is cu . In he example gi en in Figu e 5, he 61 TIMIT phones a e ep esen ed by he 9 clus e s p esen ed in Table 1. The consis ence o he clus e s, pa icula ly he ones in ol ing owels, is well known. I we compa e he e- sul an clus e s wi h he knowledge-based di ision o [5] ab cde a b c d e a b c d e 10 2 1 3 0 a 16 4 3 4 6 111101 b4146 2 4 0 5 12 0 0 c 3 6 17 0 3 10 093 d4 2 0136 0 2 1 3 10 e 6 4 3 6 16 Con usion Ma ix (M)Hou gas simila i y measu e (S) 10 2130 111 1 0 1 1 + 2 + 1 + 0 + 0 = 4 Minimum be ween /a/ and /b/ pai o con usions Figu e 4 Example o Hou gas simila i y compu a ion. Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 5 o 12 h p://asp.eu asipjou nals.com/con en /2012/1/158 p esen ed in Table 2, only [y], [w] and [oy] a e in a sep- a a e clus e . Ne e heless, hese phones s ay in a single clus e no because hey a e acous ically di e en bu due o he li le con usions wi h o he phones. Ano he s ong clus e is ha o nasals. In his case, he da a- d i en di ision is he same as he knowledge-based di - ision. A ica i es a e se in a sepa a e clus e ( he knowledge-based p oposal some imes placed a ica i es wi h s ops and a o he s wi h ica i es). Fo ica i es and s ops, he me hod elies on mo e han wo clus e s. The numbe o con usions be ween some ica i es and s ops sugges s ha acous ically hey exhibi simila i ies. The cophene ic co ela ion coe icien e e ed o in Figu e 5 legend is a measu e o how ai h ully he den- d og am p ese es he pai wise dis ances be ween he o iginal unmodelled dis ances [29]. The close he alue o he cophene ic co ela ion coe icien is o 1, he mo e accu a ely he clus e ing solu ion e lec s he da a. Since he goal is phone ecogni ion, he ques ion emains as o how can hese clus e s help o imp o e phone ecogni ion. The nex sec ion desc ibes he ol- lowed app oach—a hie a chical classi ica ion o di e en le els o phone ic in o ma ion. Se e al in e media e clas- si ie s o e pos e io p obabili y p edic ions o BPC, achie ing phone de ail a he end. Enhanced phone ecogni ion In his sec ion, we use he b oad classes gene a ed au o- ma ically by he me hod p oposed in he p e ious sec ion in a phone ecogni ion ask. Gi en he di icul y o inding a h eshold ha leads o an op imum numbe o clus e s, we decided o cu he ee in di e en places, which esul ed in se e al se s o clus e s. This p ocedu e o ms a hie a chical s uc u e, om b oad o ine phone ic de ail. The combina ion o se e al le els o phone ic de ail has ih ix ax ah uh ae eh ax−h iy uw ux aa ao ow ay aw ax e el l w oy b p d dh h g kbcl dcl cl pcl epi pau q g cl kcl hh h ydx nx em en m n n g ch jh sh s z zh en g h# 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 ey Figu e 5 Hie a chical phone clus e ing dend og am using he a e age dis ance be ween clus e s (cophene ic co ela ion coe icien = 0.873). Table 1 Six y-one TIMIT con usion-di ision esul s in e ms o nine clus e s Clus e s TIMIT-labelled phones Clus e 1 bcl dcl epi gcl kcl pau pcl q cl Clus e 2 b d dh g k p h Clus e 3 y Clus e 4 hh h Clus e 5 dx em en m n ng nx Clus e 6 aa ae ah ao aw ax ax-h ax ay eh el e ey ih ix iy l ow oy uh uw ux w Clus e 7 ch jh s sh z zh Clus e 8 eng Clus e 9 h# Table 2 Thi y-nine TIMIT knowledge-based di ision in o i e b oad classes, om [5] B oad classes TIMIT-labelled phones Vowels l, , w, y, e , ey, aw, ay, oy, ow, iy ,eh, ae, aa, uh, uw, ax, ix S ops p, , k, b, d, g, jh, ch F ica i es s, z, zh, , h, , dh, hh Nasals m, n, ng Silences sil, dx Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 6 o 12 h p://asp.eu asipjou nals.com/con en /2012/1/158 al eady been in es iga ed in se e al s udies, e.g. [5], whe e he ou pu s o ou b oad phone ic g oup classi ie s (knowledge-based gene a ed and ained sepa a ely) a e combined in o de o co ec o enhance a phone classi- ie . Ou p oposal ollows a simila app oach, bu by means o a hie a chical s uc u e and wi h b oad classes gene a ed by he con usion-d i en app oach. La e , we show ha his hie a chical classi ica ion akes ad an age o e he classi ie s ained sepa a ely. Hie a chical b oad classes We p opose a hie a chical classi ica ion sys em ha con- sis s o a hyb id MLP/HMM, whe e he neu al ne wo k a chi ec u e pe o ms phone classi ica ion wi h a hie - a chical se o b oad class phone ic classi ie s. The num- be o ou pu laye s in he MLP is he same as he cu places o he clus e ing ee, wi h he same o de . Each clus e is cha ac e ized by he se o phones g ouped by he ee cu and is called b oad class. The b oad class p edic ions om ea lie classi ie s a e ed o he nex ones in o de o enhance he class disc imina ion in he cu en classi ie . The las laye pe o ms a 1- o-61 clas- si ica ion o he se o phones. All laye s a e ained con- cu en ly so ha , in aining mode, a ge s a e p esen ed a all ou pu laye s. The se ial a angemen p oposed p o ides se e al b oad-class pos e io s along wi h he phone pos e io s. A be e phone classi ie may be achie ed i all hese pos e io s a e co ec ly combined. P e ious s udy [8] shows ha phone p edic ion may be mo e obus i class membe ship p obabili ies a e weigh ed and combined. A me hod o inding he bes se o weigh s based on dis- c imina i e aining in a hyb id MLP/HMM sys em is desc ibed below. Hie a chical MLP combina ion app oach The goal o a combina ion app oach is o ake ad an age o he b oad-class pos e io s along wi h he phone pos e - io s in o de o imp o e he global phone ecogni ion pe - o mance. Ou app oach conside s ha each phone can be p edic ed by combining all he b oad-class ou pu s associa ed wi h ha phone, wi h weigh s di e ing o each phone. These weigh s a e ound by means o a disc imina- i e aining me hod. Each weigh will be assigned o he loga i hm o he ne wo k ou pu which includes he phone. The global phone pos e io s a e ound by combin- ing he co esponding ou pu s o all ou pu laye s. The p oposed combina ion ule is exp essed by ^ Pp kjyðÞ¼ 1 Zexp X NL l¼1 αlðÞ cklog ylðÞ ck  ! ð9Þ ^ Pp ky jÞð is he k h phone p obabili y p edic ion, gi en he laye ou pu s, y, co esponding o he b oad-class p edic ions in each o he N L ou pu laye s (see Figu e 6). ylðÞ ckand αlðÞ cka e he ne wo k ou pu and co esponding weigh o laye land index c k , deno ing he b oad-class index (in laye l) o which he phone kbelongs. Each phone is p edic ed by weigh ing all he class ou pu s associa ed wi h he phone k,k21; ::; 61 g , which a e di e en o each phone. Re e ing o Figu e 6, he e a e ou ou pu laye s wi h 9, 16, 40 and 61 clus e s and acco ding o Table 3 he phone [zh] has indexes 3, 9, 23 and 61 in he laye s om 1 o 4. In his equa ion, Zis a no maliza ion ac o o he p edic o ^ Pp kyjÞð o he 61 phones sum up o one. The bes se o weigh s is he one which gi es he highes phone accu acy acco ding o ou hyb id MLP/ Figu e 6 Disc imina i e weigh aining me hod scheme. Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 7 o 12 h p://asp.eu asipjou nals.com/con en /2012/1/158 HMM ecogni ion sys em. An i e a i e aining me hod based on he pa adigm o disc imina i e aining is he e o e app op ia e. Disc imina i e aining o he weigh s E e y kind o e o should be conside ed (subs i u ions, inse ions and dele ions) in he de ini ion o a cos unc ion. Since hese e o s a e ound by he Le ensh ein dis ance, he objec i e unc ion should include a minimiza ion o his dis ance wi h mul iple hypo heses. Howe e , we used a simple 1-bes disc imina i e unc ion, he eby a oiding he e o coun ing, which i is be e han applying he phone a ge s o he ne wo k ou pu laye as is usually done. The Le ensh ein dis ance aligns wo label sequences. One is he e e ence (assumed co ec ) sequence, Wlab , and he o he is he bes decoding hypo hesis gi en by he ecognize , W ec . Using he Vi e bi algo i hm, we de ine an e o unc- ion as dW ec;Wlab  ¼gW ec ðÞgW lab ðÞ ð10Þ whe e gW lab ðÞand gW ec ðÞ ep esen he e e ence and bes acous ic log likelihood o he obse a ion sequence acco ding o he Vi e bi algo i hm. This di e ence co e- sponds o a likelihood a io. I is always g ea e han ze o and is only ze o i he wo ansc ip ions a e exac ly he same (i labels and ime alignmen s ma ch). I N BD is he o al numbe o aining u e ances, he global cos is hen gi en by E¼X NBD n¼1 dW nðÞ ec ;WnðÞ lab  ð11Þ In he hyb id MLP/HMM app oach, he a p io i p ob- abili y unc ion b s (x) is he likelihood o obse ing x in he HMM s a e, s, being ans o med in he pos e io p obabili y p edic ed by Equa ion (9). In o de o ind he app op ia e se o weigh s αlðÞ k no , a g adien descenden me hod is applied. In his case, i can be shown ha he e o g adien has e ms o he o m @ @αlðÞ ck log^ Pp ky jÞ¼ log ylðÞ ck  11=ZðÞ ð12Þ On he o he hand, he g adien o he weigh s is αlðÞ ck¼@E @αlðÞ ck ¼X NBD n¼1 @gW nðÞ ec  @αlðÞ ck  @gW nðÞ lab  @αlðÞ ck 0 @1 Að13Þ The g adien s o he log likelihoods in his exp ession depends on (12) because he HMM s a es o he pa h (WnðÞ ec o WnðÞ lab ) a e associa ed wi h he MLP ou pu s, ylðÞ ck. Figu e 6 illus a es he scheme o he p oposed dis- c imina i e weigh aining me hod. Table 3 Desc ip ion o da a-d i en b oad classes BPC9 BPC16 BPC40 l, el, w, , e , ax , ey, aw, ay, iy, ih, eh, ae, ah, ax, uh, ix, uw, ux, ax-h, aa, ao, ow, oy l, el, w l, el w , e , ax , e , ax ey, aw, ay, iy, ih, eh, ae, ah, ax, uh, ix, uw, ux, ax-h, aa, ao, ow ey aw ay iy ih, eh, ae, ah, ax, uh, ix uw, ux ax-h aa, ao, ow oy oy p, b, d, , , h, , dh, k, g p, b p b d, , , h, , dh d, , h , dh k, g k g jh, ch, sh, z, s, zh jh, ch, sh jh, ch sh z, s, zh z, s zh hh, h hh, h hh, h m, n, em, ng, en, nx, dx m, n, em, ng, en m , n em ng en nx, dx nx dx pcl, cl, dcl, kcl, q pcl, cl, dcl, kcl, q pcl bcl, gcl, pau, epi bcl, gcl, pau, epi cl, dcl kcl q bcl gcl pau, epi yyy eng eng eng h# h# h# Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 8 o 12 h p://asp.eu asipjou nals.com/con en /2012/1/158 We used he RPROP o accele a e he con e gence o a solu ion. Resul s o he p oposed hie a chical MLP a chi ec u e wi h b oad classes ound by he clus e ing me hod a e p esen ed in Sec ion 5.3. Expe imen al esul s All he expe imen s we e ca ied ou using hyb id MLP/ HMM sys ems. Speech was analyzed as in Sec ion 3.1.1. Bo h aining and es ing we e ca ied ou using he TIMIT da abase [20] and i s o iginal 61 phoneme se . This da abase is o en used in phone ecogni ion benchma k- ing, e.g. [5,7,30]. The ain and es se s co espond o he o iginal spli ing o he TIMIT da abase. While he ain- ing se wi h all si and sx sen ences has 3,698 u e ances, he es se consis s o all si and sx sen ences om he comple e 168-speake es se , which has 1,344 u e ances. The a ge s de i e om he phone bounda ies p o ided by he TIMIT da abase. Al hough he neu al ne wo k is ailo ed o disc imina e he ull 61 TIMIT phones, hese symbols a e some imes conside ed a oo na ow desc ip- ion o p ac ical use, and o e alua ion pu poses we col- lapsed he 61 TIMIT labels in o he s anda d 39 phones as p oposedbyLeeandHon[31].In hehyb idMLP/HMM sys ems, he a p io i s a e likelihoods a e eplaced by pos- e io p obabili ies, ^ Pp kx jÞð , gi en di ec ly om he MLP ou pu laye o acco ding o Equa ion (9) (Sec ion 4.2). The pe o mance o he MLPs was e alua ed by means o a ame e o a e (FER). The pe o mance o he hyb id sys em was e alua ed by means o Co ec ness (Co ) and Accu acy (Acc). Hie a chical e sus single laye classi ica ion As desc ibed in Sec ion 4.1, a hie a chical classi ica ion o di e en le els o phone ic in o ma ion is p oposed in o de o imp o e phone ecogni ion. The neu al ne wo k has abou 161 k pa ame e s. I comp ises eigh hidden laye s. The numbe o nodes in he laye s is (in nume ical o de ): 50-9-50-16-50-40-100-61. E en laye s gi e pos e - io p obabili y p edic ions o BPC and he las laye gi es pos e io p obabili y p edic ions o phones. Thus, he p oposed MLP sys em is ained as a unc ion o he 61 phones and 3 addi ional se s o BPC, consis ing o 9, 16, 40 TIMIT phone se s ( iz. BPC9, BPC16, BPC40) achie ed by he con usion-d i en clus e ing app oach p oposed. The hie a chical phone clus e ing dend og am in Figu e 5 gi es ise o he BPC9. The wo o he s BPCs esul om cu ing he da a-d i en dend og am ee in o wo o he le els. The esul ing se s o BPCs we e g ouped acco ding o he di ision p esen ed in Table 3. S anda d MFCC’s ea u es and de i a i es a e p esen ed o inpu (odd) hidden laye s. A con ex window o 290 ms was used using only 15 ame ea u es (de ails in [8]). All laye s we e ained concu en ly so ha , in aining mode, a ge s we e p esen ed a all e en laye s: laye s 2, 4, 6 and 8. Since e en laye s a e ained wi h a so max ac i- a ion unc ion, i s ou pu s can be seen as BPC p obabil- i ies. The odd uses a sigmoid ac i a ion unc ion. All he ne wo k weigh s and bias a e adjus ed using ba ch aining wi h an RPROP algo i hm [21] so as o minimize he minimum-c oss-en opy e o be ween he ne wo k ou - pu and he a ge alues. The choice o he e o unc ion ollowed Bishop’s sugges ion [32], which was la e cla i ied by Dunne and Campbell [33]. I s a es ha he so max ac- i a ion unc ion should couple wi h he c oss-en opy penal y unc ion. In o de o e alua e he pe o mance o he p oposed hie a chical ne wo k, we ained ou single laye ne - wo ks, each one specialized in one BPC. Each single laye ne wo k was ained wi h he same numbe o hid- den nodes as he hie a chical ne wo k (e.g. BPC16 MLP has 50 hidden nodes and an ou pu laye wi h 16 ou - pu s). The esul s in e ms o FER a e gi en in Figu e 7 and show ha he hie a chical ne wo k ou pe o med he equi alen single laye ne wo ks wi h espec o BPC16, BPC40 and 61 TIMIT phones. Rega ding he classi ica ion o BPC9 (9 clus e s o Table 1) he pe - o mance o he single laye ne wo k is simila o he hie a chical. These esul s encou age he use o he hie - a chical s uc u e in u he expe imen s. No e ha hese esul s ela e o an e alua ion (in e ms o FER) o he es se o e e y aining epoch. Hie a chical con usion-d i en e sus hie a chical knowledge-d i en phone ecogni ion Since con usion-d i en clus e ing is he opic o his a icle, he pe o mance o his kind o clus e ing has o be compa ed wi h ha achie ed by an expe -knowledge ap- p oach. The e o e, we ained wo hie a chical MLPs om 0 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 T ain Epochs FER FER 9 classes-hie a chical ne FER 16 classes-hie a chical ne FER 40 classes-hie a chical ne FER 61 classes-hie a chical ne FER 9 classes-single laye FER 16 classes-single laye FER 40 classes-single laye FER 61 classes-single laye Figu e 7 FER compa ison o hie a chical BPC’s classi ica ion and equi alen single laye BPC’s classi ica ion. Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 9 o 12 h p://asp.eu asipjou nals.com/con en /2012/1/158