RESEARCH Open Access
B oad phone ic class de ini ion d i en by
phone con usions
Ca la Lopes
1,2*
and Fe nando Pe digão
1,3
Abs ac
In e media e ep esen a ions be ween he speech signal and phones may be used o imp o e disc imina ion
among phones ha a e o en con used. These ep esen a ions a e usually ound acco ding o b oad phone ic
classes, which a e de ined by a phone ician. This a icle p oposes an al e na i e da a-d i en me hod o gene a e
hese classes. Phone con usion in o ma ion om he analysis o he ou pu o a phone ecogni ion sys em is used
o ind clus e s a high isk o mu ual con usion. A me ic is de ined o compu e he dis ance be ween phones. The
esul s, using TIMIT da a, show ha he p oposed con usion-d i en phone clus e ing me hod is an a ac i e
al e na i e o he app oaches based on human knowledge. A hie a chical classi ica ion s uc u e o imp o e phone
ecogni ion is also p oposed using a disc imina i e weigh aining me hod. Expe imen s show imp o emen s in
phone ecogni ion on he TIMIT da abase compa ed o a baseline sys em.
In oduc ion
B oad phone ic classes (BPC) ha e widely been used in
speech ecogni ion esea ch as, o ins ance, au oma ic
language iden i ica ion [1]; speaking a e es ima ion [2];
mul i lingual sys ems [3,4] and, especially, in phone ecog-
ni ion [5-8]. I s success is due o he ca ied addi ional in-
o ma ion ha con ibu es o imp o e he ecogni ion
a es, especially unde noise condi ions [9]. In phone ec-
ogni ion ask, BPC in o ma ion may be used as an add-
i ional se o acous ic ea u es o i may be in eg a ed in
he phone p edic ions. One success ul example is p e-
sen ed by Siniscalchi e al. [6], one o he bes esul s
epo ed on he phone TIMIT ecogni ion ask, whe e 15
b oad a icula o y classes a e used o p edic pos e io
phone p obabili ies and o esco e phone la ices. Ano he
in e es ing example is gi en by Mo is and Fosle -Lussie
[7] whe e he ou pu s o eigh b oad class classi ie s a e
used as inpu ea u es on a condi ional andom ield
(CRF) model. In [8], a hie a chical classi ica ion om suc-
cessi e b oad classes is used wi h imp o emen s in phone
accu acy. Se e al o he s udies ela ed o BPC could be e-
e ed, e.g. [5,10], some imes using di e en e minology.
In li e a u e, BPC ake di e en names (such as b oad
phone ic g oups, speech a ibu es, e en s, e c.) bu in all
hese s udies hey a e always se s o phones wi h simila
acous ic/phone ic ea u es d awn manually by an expe
(knowledge-d i en in o ma ion). The b oad classes a e
selec ed acco ding o acous ic-phone ic p ope ies ha de-
i e om a icula o y cons ain s o om hea ing pe cep-
ion. This selec ion may con ain some subjec i i y o may
be di icul o ca y ou when dealing wi h o he kind o
speech uni s like syllables o when conside ing coa icu-
la ion phenomena.
BPC o he English language is an issue ha has widely
been add essed by he scien i ic communi y [5,7,8,11]. I s
de ini ion is usually ela ed o he manne and place o a -
icula ion, so ha all he b oad classes show good ag ee-
men wi hin some phone ic, a icula o y and/o acous ic
p ope ies. The e icien cons uc ion o smalle /mo e
compac phone se s is no i ial, howe e , as is con i med
by he lack o consensus be ween se e al p oposals. Fo
example in he TIMIT co pus, in [12], phone [dx] is classi-
ied as a sono an consonan , while in [13] he same
phone is classi ied as a s op and Halbe s ad and Glass
[10] classi y i as a nasal/ lap. The same happens wi h
phone [h ]. In [12], i is a sono an consonan and in
[5,10] i is a ica i e. Compa ing he “ ica i e”b oad
classes o [5,14] i can be seen ha he i s p oposal
includes he phone [h ] while he second does no . The
abo e examples a e o show he subjec i e na u e o he
app oaches based on human knowledge, e en when e e -
ing o such a widely s udied language as Ame ican English.
* Co espondence: [email p o ec ed]
1
Ins i u o de Telecomunicações, Polo II, Coimb a P-3030-290, Po ugal
2
ESTG, Ins i u o Poli écnico de Lei ia, Campus 2, Lei ia P-2411-901, Po ugal
Full lis o au ho in o ma ion is a ailable a he end o he a icle
© 2012 Lopes and Pe digao; licensee Sp inge . This is an Open Access a icle dis ibu ed unde he e ms o he C ea i e
Commons A ibu ion License (h p://c ea i ecommons.o g/licenses/by/2.0), which pe mi s un es ic ed use, dis ibu ion, and
ep oduc ion in any medium, p o ided he o iginal wo k is p ope ly ci ed.
Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158
h p://asp.eu asipjou nals.com/con en /2012/1/158
This subjec i i y a ec s no only he numbe o classes, bu
also he se o phones in each ca ego y. These p oblems,
ela ed o he expe -d i en app oaches, ha e p omp ed he
eme gence o da a-d i en app oaches, whe e he classes’
composi ion is guided by da a. We explo e in his a icle a
da a-d i en me hod whe e he b oad classes a e au oma ic-
ally de ined acco ding o he ou pu o a speech ecogni ion
sys em. The me hod can be applied in sys ems using all
kinds o ecogni ion uni s (phones, syllables, subwo ds,
phones o di e en languages, e c.) and do no conduc o a
s a ic di ision.
The ou pu o a classi ica ion sys em is usually e alu-
a ed compa ing all he ecognized sequences wi h he
co esponding e e ences. F om his compa ison, a con-
usion ma ix can be compu ed. In he p oposed ap-
p oach i is conside ed ha i a uni (phone o o he )
has much con usion wi h o he uni is because, o he
ecognize , hey a e somehow simila . This app oach
may no ully ag ee wi h phone ics p inciples o acous ic
heo ies bu i can be e y help ul allowing o o e come
deadlocks in some si ua ions. Take he case o a mul i-
lingual sys em, whe e a knowledge-based app oach has
o in ol e expe s om all he languages in ol ed. The
same occu s when dealing wi h a ie ies o a language
(e.g., Eu opean Po uguese o B azilian Po uguese). The
p oposed au oma ic clus e ing me hod ex ends he pos-
sibili y o using b oad classes’in o ma ion o sys ems
based on o he han he phone uni .
Da a-d i en clus e ing usually s ands o a s a is ical
measu emen o a class. The key poin o all clus e ing
algo i hms is he choice o a p oximi y o dis ance meas-
u e. This measu e can be ob ained om acous ic mod-
els, e.g., [3,15], o e en ely on he con usion ma ix,
e.g., [4,16]. Model-d i en me hods and con usion-
d i en
a
me hods a e hen he wo majo ca ego ies o da a-
d i en phone clus e ing algo i hms.
In model-d i en me hods, he acous ic simila i y be-
ween wo phones can be achie ed om he heo e ical
dis ance be ween he co esponding acous ic models.
This dis ance can be he Bha acha yya dis ance be ween
wo Gaussian mix u e models [3,15]; a ela i e en opy-
based (Kullback–Leible di e gence) dis ance be ween
wo Laplacian mix u es [17], e c. In [18], a da a-d i en
phone ic b oad class gene a ion is p oposed whe e mu-
ual in o ma ion is used o compu e simila i y be ween
models, while in [19] a simila i y measu e based on he
likelihood be ween he acous ic ames and he hidden
Ma ko models (HMMs) is p oposed.
In his a icle, we p opose a con usion-d i en me hod
o gene a e phone clus e s. This app oach was al eady
p oposed in [4], whe e he phones a e g ouped using
ules depending on a se o weigh s and h esholds. Ou
p oposal di e s om [4] in so a as we de ine a me ic
o compu e phone dis ances.
The emainde o he a icle is o ganized as ollows. In
Sec ion 2, we in oduce he da a-d i en app oach and Sec-
ion 3 p esen s he me ic de ined ha e alua es he
phones simila i ies. I also desc ibes he way ha he con-
usion ma ix (which is he base o phone simila i ies
compu a ion) was achie ed. Sec ion 4 p esen s a sys em
whe e b oad classes a e used in o de o enhance phone
ecogni ion, and in Sec ion 5 expe imen al esul s, com-
pa ing da a-d i en and knowledge-d i en app oaches, a e
p esen ed. Finally, some conclusions and u u e imp o e-
men s a e d awn in Sec ion 6.
Da a-d i en b oad classes
In con usion-d i en me hods, he simila i y measu e ap-
p oach comes om he phone con usion ma ix M. This
ma ix is compu ed om he ou pu o a phone ecognize
by aligning he ecognized sen ence wi h he e e ence one,
using a dynamic p og amming algo i hm ( he Le ensh ein
algo i hm). I includes he concep o hi s and con usions,
as well as inse ions (INS) and dele ions (DEL). An
example o pa o a con usion ma ix o ou phone
ecognize using TIMIT [20] da a is shown in Figu e 1.
The diagonal e e s o he numbe o co ec ly ecognized
phones (hi s) and he o -diagonal elemen s e e o he
numbe o misclassi ica ions.
The idea behind con usion-d i en me hods is ha simi-
la phones end o be mo e con usable and should belong
o he same class. Figu e 2a shows, in pseudo colou s, a
con usion ma ix using he 61 o iginal TIMIT phones,
whe e he phones a e in alphabe ical o de and wi h blue
ep esen ing he lowe alue and da k ed he highe
alue. I highly con usable phones a e eposi ioned in such
a way ha hey become nea each o he , we ge he ma ix
depic ed in Figu e 2b. In his second ma ix, we can easily
dis inguish se e al se s (clus e s) o phones whe e con u-
sion be ween all he elemen s o he clus e is much
highe han be ween o he phones o phone clus e s. In
an a emp o ind his se o clus e s, his s udy explo es
RECOGNISED AS
aa ae ah ao aw ax DEL
SOURCE REFERENCE
aa 125
ae 88
ah 111
ao 113
aw 14
ax
456 8 52 87 16 3
12 448 23 2 10 5
45 31 369 16 9 71
67 2 21 441 8 7
16 14 6 6 121 0
5 2 64 17 8 592 217
INS 21 21 19 17 14 30
Figu e 1 Pa o a con usion ma ix: ou pu o ou TIMIT-based
phone ecogni ion sys em.
Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 2 o 12
h p://asp.eu asipjou nals.com/con en /2012/1/158
an au oma ic classi ica ion me hod, whe e he phone ic
classes a e ound acco ding o he ou pu o an au oma ic
phone ecogni ion sys em. Phones a e g ouped acco ding
o a simila i y measu e es ima ed om he con usion
ma ix gi en by he ecognize pe o mance.
Con usion-d i en phone dis ance measu e
This sec ion desc ibes an au oma ic me hod o phone ic
class gene a ion based on a con usion ma ix o a phone
ecognize . Usually, he clus e ing echniques in ol e
h ee concep s:
1. A da a model;
2. A p oximi y c i e ion (simila i y, dis ance, e c.);
3. A clus e ing algo i hm ha gene a es he clus e s
(b oad classes) using he da a model and he
simila i y measu e.
These concep s a e discussed in ollowing sec ions.
Da a model
In he p oposed me hod, he da a comes om a con usion
ma ix yielded by he pe o mance analysis o a phone ec-
ogni ion sys em. This ma ix may con ain esul s a he
ame le el, i a i icial neu al ne wo k (ANN), suppo
ec o machines (SVM), o CRF-based ecognize s a e
used, o a he segmen le el, i segmen models a e used,
such as HMMs o hyb id sys ems, as in he p esen case.
An MLP/HMM hyb id sys em is used ha combines an
o e all HMM s uc u e wi h he class p edic ions gi en by
a mul ilaye pe cep on (MLP) classi ie , hus bene i ing
om he ime modelling abili ies o HMMs and he dis-
c imina ion capabili ies o ANNs. In he TIMIT aining
se , his in ol es he ecogni ion o 143 k segmen s.
Hyb id MLP/HMM desc ip ion
An MLP ne wo k, wi h a single hidden laye , was ained
o phone classi ica ion a a ame le el. The las laye pe -
o ms a 1- o-61 classi ica ion o e he se o (TIMIT)
phones. Speech was analysed e e y 10 ms wi h a 25-ms
Hamming window. Thi y-nine pa ame e s we e used as
s anda d inpu ea u es ep esen ing 12 Mel F equency
Ceps al Coe icien s (MFCCs), plus ene gy, and hei i s
and second ime-de i a i e coe icien s. The con ex win-
dow used was 170 ms bu only 9 ame ea u es we e used,
as desc ibed in [8]. The cu en ame is in he cen e o
he con ex window ( empo al in o ma ion o pas and u-
u e is included). The so max unc ion was used as he ac-
i a ion unc ion o he ou pu laye so ha he ou pu
alues could be in e p e ed as pos e io p obabili ies. The
hidden laye has 1,000 nodes and uses a sigmoid ac i a ion
unc ion. All he ne wo k weigh s and bias a e adjus ed
using ba ch aining wi h he esilien back-p opaga ion
(RPROP) algo i hm [21], so as o educe he minimum-
c oss-en opy e o be ween ne wo k ou pu and he a ge
alues. The ne wo k has 413 k ee aining pa ame e s.
HMMs we e buil be o ehand o each phone using
HTK3.41, [22], in o de o es ima e he ansi ion p ob-
abili ies be ween s a es. Each phone was modelled by a
h ee-s a e le - o- igh HMM and each s a e was mod-
elled by a single Gaussian model. The inpu ea u es
we e he same as o he MLP.
In he hyb id MLP/HMM sys em, he s a e likelihoods
a e eplaced by he pos e io p obabili ies gi en by he
ou pu p edic ions o he MLP. The h ee s a es sha e
he same MLP ou pu . We used HTK [22], wi h some
changes in o de o eplace he usual Gaussian mix u e
models wi h he ou pu s o he MLP.
The con usion ma ix (da a model) is compu ed using
he en i e TIMIT aining se , which consis s o all si
and sx sen ences o he o iginal aining se (3,698 u e -
ances). The pe o mance o he hyb id sys em is usually
e alua ed by means o Co ec ness (Co ) and Accu acy
(Acc). We ha e used HTK e alua ion ool HResul s o
Figu e 2 (a) Con usion ma ix o ou TIMIT-based phone
ecogni ion sys em. (b) A eposi ioned e sion o he ma ix depic ed
in (a) whe e highly con usable phones a e place nea each o he .
Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 3 o 12
h p://asp.eu asipjou nals.com/con en /2012/1/158
compu e hem. Bu wi h his ool only he oken se-
quence is aken o e alua ion. A ine e alua ion akes
in o accoun no only he co ec iden i ied sequence o
phones, bu also hei ime localiza ion. This was he c i-
e ia used in he con usion ma ix gene a ion on which
he p oposed clus e ing is based. A b ie desc ip ion o
he e alua ion p ocedu e used is gi en below. A ull de-
sc ip ion can be ound in [23].
The e alua ion p ocedu e uses a modi ied Le ensh ein
algo i hm [24] in he alignmen be ween labelled and
ecognized phones, whe e he deg ee o o e lapping be-
ween hem is aken in he local dis ance de ini ion. This
algo i hm inds he bes alignmen be ween wo s ings
and inse s a penal y i an e o occu s (inse ion, dele-
ion and subs i u ion), bu no penal y is applied i he
labels ma ch. In ou p oposal, we include an addi ional
penal y ha is p opo ional o he a e age o he le and
igh misalignmen s. I he labels do no o e lap (T
OV
≤
0 in Figu e 3), his penal y is se o a maximum alue
(p
max
), such ha an inse ion o a dele ion will be p e-
e ed o a misaligned subs i u ion.
Taking i1; i2and j1; j2as he bounda ies o he es
and e e ence labels, as indica ed in Figu e 3, hey o e lap
i
i2
>
j1
o
j2
>
i1
and hen he o al ime o he labels is
T¼max i2; j2

min i1; j1

¼T1þT2þTOV ð1Þ
and he o e lapping ime is
TOV ¼min i2; j2

max i1; j1
 ð2Þ
The le and igh misalignmen s a e T1¼ j1 i1

and T2¼ j2 i2
. I he labels lab
i
and lab
j
ma ch hei
name bu a e no pe ec ly aligned, hen we in oduce an
addi ional associa ion penal y, p
A
(i,j), which is in e sely
p opo ional o he o e lap be ween he labels, acco ding
o he ollowing exp ession:
pAi;jðÞ¼
T1þT2
ðÞ=2
TOV
¼1
2
T
TOV
1
 ð3Þ
I he labels o e lap mo e ha 50%, p
A
is smalle han
0.5. As a as he o e lapping dec eases, his dis ance
inc eases and is clipped o p
max
=15, which co esponds
o 3.2% o o e lapping. In o de o p omo e con usions
(subs i u ions) we penalize inse ions and dele ions signi i-
can ly by se ing p
INS
=12 and p
DEL
=12. The op imal
alignmen is ound by acing back he pa h o accumu-
la ed penal ies om he las label pai o he o igin o he
(i,j) g id. Wi h he alignmen o all labels a con usion
ma ix can hen be compu ed. This ma ix ep esen s he
da a model e e ed o in he beginning o Sec ion 3.
Simila i y measu e
The con usion ma ix is o en con e ed in o a symme -
ic simila i y ma ix using he so-called Hou gas algo-
i hm. I measu es he simila i y be ween e e ence
phones iand jusing he numbe o con usions o hese
phones wi h all o he phones k. Mo e speci ically, i Nis
he o al numbe o classes (phones), he Hou gas simi-
la i y be ween phones iand j,sij is gi en by
sij ¼sji ¼X
N
k¼1
min ik; jk
 ð4Þ
wi h 1≤i;j≤N.I i=j, ij is he numbe o hi s ins ead o
con usions. Acco ding o his measu e, wo phones iand j
a e simila i hey bo h ha e many con usions wi h he
same phones and hei simila i y is ze o only when phones
iand jha e no simul aneous con usions wi h any phones.
Figu e 4 gi es a simple example o his measu e.
The con usion ma ix o a good phone ecognize is
close o a diagonal ma ix. The ou -diagonal alues ep e-
sen misclassi ied phones (con usions be ween phones).
Since he numbe s o occu ences o each phone a e qui e
di e en (due o phone ically unbalanced speech ma e ial)
we no malize he con usion ma ix by di iding he e-
quency coun s
ij
by he o al numbe o occu ences o
he phone iin he speech da abase:
pij ¼ ij
P
N
n¼1
in
¼P^
cjci
jÞ
ð5Þ
In his way, we ha e an es ima e o he p obabili y o
ecognizing he model o clus e ^
cjwhen i s e e ence
class is c
i
. In his case he Hou gas simila i y measu e
becomes
s′
ij ¼X
N
n¼1
min pin;pjn

¼X
N
n¼1
min P^
ci
ðjcn
ðÞ;P^
cj
cnÞÞ ð6Þ
T
T
1
T
2
T
OV
Re e ence label,
labj
Tes label, labi
i1 j1 i2 j2
Figu e 3 Measu emen o he ime misalignmen be ween wo
labels.
Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 4 o 12
h p://asp.eu asipjou nals.com/con en /2012/1/158
This new measu e has he ollowing p ope ies: (i) s′
ij≤1
and (ii) s′
ii ¼1 . Because min a;bðÞ¼
1
2aþbab
jj
ðÞ
and because PN
n¼1pin ¼1 , a dis ance measu e be ween
phones iand jcan he e o e be de ined as
d1ci;cj

¼21s′
ij

¼X
N
n¼1
pin pjn
ð7Þ
This dis ance o ms a me ic because i is he L
1
no m
applied o ow di e ences o ma ix P (wi h elemen s p
ij
as de ined in (5)). Se e al simila i y measu e p oposals can
be ound in li e a u e [4,16,25], bu as e e ed in [26],
hey do no ul il he p ope ies o a p ope me ic. In he
p esen p oposal, d1has he h ee me ic p ope ies: i is
posi i e, symme ic and sa is ies he iangle inequali y.
O he dis ance measu es can be de ined based on he
same p inciple, in pa icula he Euclidean dis ance o ows,
d2ci;cj

¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
XN
n¼1pin pjn

2
ð8Þ
Clus e ing me hod
The p oposed clus e ing me hod g oups phones in a
mul ile el hie a chy whe e clus e s a one le el a e com-
bined as clus e s a he nex le el. Clus e ing can be
achie ed ollowing hie a chical agglome a i e clus e ing
pa adigm [27,28] as ollows. Ini ially, each phone will be
conside ed as a dis inc clus e .
S ep 1 !Compu e ma ix P using (4).
S ep 2 !Find he dis ance be ween each pai o
phones using (7).
S ep 3 !Compu e he dis ances be ween all clus e s. The
dis ance be ween clus e s and s can be compu ed wi h
se e al c i e ia, o which he simples is he nea es
neighbou :d ;sðÞ¼min dc
ij ;cjs
jÞÞ;i¼1...n ;
 j¼
1::nswhe e n
k
is he numbe o phones in clus e k and
c
i|k
is he i h phone in clus e k.
S ep4 !C ea e a new clus e by g ouping he wo
nea es ones.
S eps 3 and 4 will epea un il he numbe o desi ed
clus e s o a dis ance h eshold is eached. Ano he pos-
sibili y is o compu e clus e s un il all he phones belong
o he same clus e . A dend og am can be buil om
hese clus e s, o allow he decision, he le el o scale o
clus e ing ha is mos app op ia e o he applica ion.
TIMIT phone clus e ing esul s
S a ing om he da a model desc ibed in Sec ion 3.1
and ollowing he s eps p oposed in Sec ion 3.3, we a -
i e a a hie a chical phone clus e ing, depic ed as a bin-
a y clus e ee (dend og am) in Figu e 5. The labels
along he ho izon al axis ep esen he phones in he
o iginal da a se and e ical axis e e s o he dis ance
be ween he phones. This dis ance is compu ed om a
simila i y measu e using d1ci;cj

. The numbe o clus-
e s depends on whe e he ee is cu . In he example
gi en in Figu e 5, he 61 TIMIT phones a e ep esen ed
by he 9 clus e s p esen ed in Table 1.
The consis ence o he clus e s, pa icula ly he ones
in ol ing owels, is well known. I we compa e he e-
sul an clus e s wi h he knowledge-based di ision o [5]
ab cde a b c d e
a
b
c
d
e
10 2 1 3 0 a 16 4 3 4 6
111101 b4146 2 4
0 5 12 0 0 c 3 6 17 0 3
10 093 d4 2 0136
0 2 1 3 10 e 6 4 3 6 16
Con usion Ma ix (M)Hou gas simila i
y
measu e (S)
10 2130
111 1 0 1
1 + 2 + 1 + 0 + 0 = 4
Minimum be ween /a/ and
/b/ pai o con usions
Figu e 4 Example o Hou gas simila i y compu a ion.
Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 5 o 12
h p://asp.eu asipjou nals.com/con en /2012/1/158

p esen ed in Table 2, only [y], [w] and [oy] a e in a sep-
a a e clus e . Ne e heless, hese phones s ay in a single
clus e no because hey a e acous ically di e en bu
due o he li le con usions wi h o he phones. Ano he
s ong clus e is ha o nasals. In his case, he da a-
d i en di ision is he same as he knowledge-based di -
ision. A ica i es a e se in a sepa a e clus e ( he
knowledge-based p oposal some imes placed a ica i es
wi h s ops and a o he s wi h ica i es). Fo ica i es
and s ops, he me hod elies on mo e han wo clus e s.
The numbe o con usions be ween some ica i es and
s ops sugges s ha acous ically hey exhibi simila i ies.
The cophene ic co ela ion coe icien e e ed o in
Figu e 5 legend is a measu e o how ai h ully he den-
d og am p ese es he pai wise dis ances be ween he
o iginal unmodelled dis ances [29]. The close he alue
o he cophene ic co ela ion coe icien is o 1, he mo e
accu a ely he clus e ing solu ion e lec s he da a.
Since he goal is phone ecogni ion, he ques ion
emains as o how can hese clus e s help o imp o e
phone ecogni ion. The nex sec ion desc ibes he ol-
lowed app oach—a hie a chical classi ica ion o di e en
le els o phone ic in o ma ion. Se e al in e media e clas-
si ie s o e pos e io p obabili y p edic ions o BPC,
achie ing phone de ail a he end.
Enhanced phone ecogni ion
In his sec ion, we use he b oad classes gene a ed au o-
ma ically by he me hod p oposed in he p e ious sec ion
in a phone ecogni ion ask. Gi en he di icul y o inding
a h eshold ha leads o an op imum numbe o clus e s,
we decided o cu he ee in di e en places, which
esul ed in se e al se s o clus e s. This p ocedu e o ms a
hie a chical s uc u e, om b oad o ine phone ic de ail.
The combina ion o se e al le els o phone ic de ail has
ih ix ax ah uh ae eh ax−h iy uw ux aa ao ow ay aw ax e el l w oy b p d dh h
g
kbcl dcl cl pcl epi pau q
g
cl kcl hh h ydx nx em en m n n
g
ch jh sh s z zh en
g
h#
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
ey
Figu e 5 Hie a chical phone clus e ing dend og am using he a e age dis ance be ween clus e s (cophene ic co ela ion coe icien = 0.873).
Table 1 Six y-one TIMIT con usion-di ision esul s in
e ms o nine clus e s
Clus e s TIMIT-labelled phones
Clus e 1 bcl dcl epi gcl kcl pau pcl q cl
Clus e 2 b d dh g k p h
Clus e 3 y
Clus e 4 hh h
Clus e 5 dx em en m n ng nx
Clus e 6 aa ae ah ao aw ax ax-h ax ay
eh el e ey ih ix iy l ow oy uh uw ux w
Clus e 7 ch jh s sh z zh
Clus e 8 eng
Clus e 9 h#
Table 2 Thi y-nine TIMIT knowledge-based di ision in o
i e b oad classes, om [5]
B oad classes TIMIT-labelled phones
Vowels l, , w, y, e , ey, aw, ay, oy, ow, iy ,eh,
ae, aa, uh, uw, ax, ix
S ops p, , k, b, d, g, jh, ch
F ica i es s, z, zh, , h, , dh, hh
Nasals m, n, ng
Silences sil, dx
Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 6 o 12
h p://asp.eu asipjou nals.com/con en /2012/1/158
al eady been in es iga ed in se e al s udies, e.g. [5], whe e
he ou pu s o ou b oad phone ic g oup classi ie s
(knowledge-based gene a ed and ained sepa a ely) a e
combined in o de o co ec o enhance a phone classi-
ie . Ou p oposal ollows a simila app oach, bu by
means o a hie a chical s uc u e and wi h b oad classes
gene a ed by he con usion-d i en app oach. La e , we
show ha his hie a chical classi ica ion akes ad an age
o e he classi ie s ained sepa a ely.
Hie a chical b oad classes
We p opose a hie a chical classi ica ion sys em ha con-
sis s o a hyb id MLP/HMM, whe e he neu al ne wo k
a chi ec u e pe o ms phone classi ica ion wi h a hie -
a chical se o b oad class phone ic classi ie s. The num-
be o ou pu laye s in he MLP is he same as he cu
places o he clus e ing ee, wi h he same o de . Each
clus e is cha ac e ized by he se o phones g ouped by
he ee cu and is called b oad class. The b oad class
p edic ions om ea lie classi ie s a e ed o he nex
ones in o de o enhance he class disc imina ion in he
cu en classi ie . The las laye pe o ms a 1- o-61 clas-
si ica ion o he se o phones. All laye s a e ained con-
cu en ly so ha , in aining mode, a ge s a e p esen ed
a all ou pu laye s.
The se ial a angemen p oposed p o ides se e al
b oad-class pos e io s along wi h he phone pos e io s.
A be e phone classi ie may be achie ed i all hese
pos e io s a e co ec ly combined. P e ious s udy [8]
shows ha phone p edic ion may be mo e obus i class
membe ship p obabili ies a e weigh ed and combined. A
me hod o inding he bes se o weigh s based on dis-
c imina i e aining in a hyb id MLP/HMM sys em is
desc ibed below.
Hie a chical MLP combina ion app oach
The goal o a combina ion app oach is o ake ad an age
o he b oad-class pos e io s along wi h he phone pos e -
io s in o de o imp o e he global phone ecogni ion pe -
o mance. Ou app oach conside s ha each phone can
be p edic ed by combining all he b oad-class ou pu s
associa ed wi h ha phone, wi h weigh s di e ing o each
phone. These weigh s a e ound by means o a disc imina-
i e aining me hod. Each weigh will be assigned o he
loga i hm o he ne wo k ou pu which includes he
phone. The global phone pos e io s a e ound by combin-
ing he co esponding ou pu s o all ou pu laye s. The
p oposed combina ion ule is exp essed by
^
Pp
kjyðÞ¼
1
Zexp X
NL
l¼1
αlðÞ
cklog ylðÞ
ck

!
ð9Þ
^
Pp
ky
jÞð is he k h phone p obabili y p edic ion, gi en
he laye ou pu s, y, co esponding o he b oad-class
p edic ions in each o he N
L
ou pu laye s (see Figu e 6).
ylðÞ
ckand αlðÞ
cka e he ne wo k ou pu and co esponding
weigh o laye land index c
k
, deno ing he b oad-class
index (in laye l) o which he phone kbelongs. Each
phone is p edic ed by weigh ing all he class ou pu s
associa ed wi h he phone k,k21; ::; 61
g
, which a e
di e en o each phone. Re e ing o Figu e 6, he e a e
ou ou pu laye s wi h 9, 16, 40 and 61 clus e s and
acco ding o Table 3 he phone [zh] has indexes 3, 9, 23
and 61 in he laye s om 1 o 4. In his equa ion, Zis a
no maliza ion ac o o he p edic o ^
Pp
kyjÞð o he 61
phones sum up o one.
The bes se o weigh s is he one which gi es he
highes phone accu acy acco ding o ou hyb id MLP/
Figu e 6 Disc imina i e weigh aining me hod scheme.
Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 7 o 12
h p://asp.eu asipjou nals.com/con en /2012/1/158
HMM ecogni ion sys em. An i e a i e aining me hod
based on he pa adigm o disc imina i e aining is
he e o e app op ia e.
Disc imina i e aining o he weigh s
E e y kind o e o should be conside ed (subs i u ions,
inse ions and dele ions) in he de ini ion o a cos unc ion.
Since hese e o s a e ound by he Le ensh ein dis ance,
he objec i e unc ion should include a minimiza ion o his
dis ance wi h mul iple hypo heses. Howe e , we used a
simple 1-bes disc imina i e unc ion, he eby a oiding he
e o coun ing, which i is be e han applying he phone
a ge s o he ne wo k ou pu laye as is usually done. The
Le ensh ein dis ance aligns wo label sequences. One is he
e e ence (assumed co ec ) sequence, Wlab , and he o he
is he bes decoding hypo hesis gi en by he ecognize ,
W ec . Using he Vi e bi algo i hm, we de ine an e o unc-
ion as
dW
ec;Wlab

¼gW
ec
ðÞgW
lab
ðÞ ð10Þ
whe e gW
lab
ðÞand gW
ec
ðÞ ep esen he e e ence and
bes acous ic log likelihood o he obse a ion sequence
acco ding o he Vi e bi algo i hm. This di e ence co e-
sponds o a likelihood a io. I is always g ea e han ze o
and is only ze o i he wo ansc ip ions a e exac ly he
same (i labels and ime alignmen s ma ch).
I N
BD
is he o al numbe o aining u e ances, he
global cos is hen gi en by
E¼X
NBD
n¼1
dW
nðÞ
ec ;WnðÞ
lab
 ð11Þ
In he hyb id MLP/HMM app oach, he a p io i p ob-
abili y unc ion b
s
(x) is he likelihood o obse ing x in
he HMM s a e, s, being ans o med in he pos e io
p obabili y p edic ed by Equa ion (9).
In o de o ind he app op ia e se o weigh s αlðÞ
k
no
,
a g adien descenden me hod is applied. In his case, i
can be shown ha he e o g adien has e ms o he
o m
@
@αlðÞ
ck
log^
Pp
ky
jÞ¼ log ylðÞ
ck

11=ZðÞ
ð12Þ
On he o he hand, he g adien o he weigh s is
αlðÞ
ck¼@E
@αlðÞ
ck
¼X
NBD
n¼1
@gW
nðÞ
ec

@αlðÞ
ck

@gW
nðÞ
lab

@αlðÞ
ck
0
@1
Að13Þ
The g adien s o he log likelihoods in his exp ession
depends on (12) because he HMM s a es o he pa h
(WnðÞ
ec o WnðÞ
lab ) a e associa ed wi h he MLP ou pu s, ylðÞ
ck.
Figu e 6 illus a es he scheme o he p oposed dis-
c imina i e weigh aining me hod.
Table 3 Desc ip ion o da a-d i en b oad classes
BPC9 BPC16 BPC40
l, el, w, , e , ax , ey,
aw, ay, iy, ih, eh, ae,
ah, ax, uh, ix, uw, ux,
ax-h, aa, ao, ow, oy
l, el, w l, el
w
, e , ax , e , ax
ey, aw, ay, iy, ih, eh,
ae, ah, ax, uh, ix,
uw, ux, ax-h, aa, ao, ow
ey
aw
ay
iy
ih, eh, ae, ah,
ax, uh, ix
uw, ux
ax-h
aa, ao, ow
oy oy
p, b, d, , , h, , dh, k, g p, b p
b
d, , , h, , dh d,
, h
, dh
k, g k
g
jh, ch, sh, z, s, zh jh, ch, sh jh, ch
sh
z, s, zh z, s
zh
hh, h hh, h hh, h
m, n, em, ng, en, nx, dx m, n, em, ng, en m , n
em
ng
en
nx, dx nx
dx
pcl, cl, dcl, kcl, q pcl, cl, dcl, kcl, q pcl
bcl, gcl, pau, epi bcl, gcl, pau, epi cl, dcl
kcl
q
bcl
gcl
pau, epi
yyy
eng eng eng
h# h# h#
Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 8 o 12
h p://asp.eu asipjou nals.com/con en /2012/1/158
We used he RPROP o accele a e he con e gence o
a solu ion.
Resul s o he p oposed hie a chical MLP a chi ec u e
wi h b oad classes ound by he clus e ing me hod a e
p esen ed in Sec ion 5.3.
Expe imen al esul s
All he expe imen s we e ca ied ou using hyb id MLP/
HMM sys ems. Speech was analyzed as in Sec ion 3.1.1.
Bo h aining and es ing we e ca ied ou using he
TIMIT da abase [20] and i s o iginal 61 phoneme se . This
da abase is o en used in phone ecogni ion benchma k-
ing, e.g. [5,7,30]. The ain and es se s co espond o he
o iginal spli ing o he TIMIT da abase. While he ain-
ing se wi h all si and sx sen ences has 3,698 u e ances,
he es se consis s o all si and sx sen ences om he
comple e 168-speake es se , which has 1,344 u e ances.
The a ge s de i e om he phone bounda ies p o ided
by he TIMIT da abase. Al hough he neu al ne wo k is
ailo ed o disc imina e he ull 61 TIMIT phones, hese
symbols a e some imes conside ed a oo na ow desc ip-
ion o p ac ical use, and o e alua ion pu poses we col-
lapsed he 61 TIMIT labels in o he s anda d 39 phones as
p oposedbyLeeandHon[31].In hehyb idMLP/HMM
sys ems, he a p io i s a e likelihoods a e eplaced by pos-
e io p obabili ies, ^
Pp
kx
jÞð , gi en di ec ly om he MLP
ou pu laye o acco ding o Equa ion (9) (Sec ion 4.2).
The pe o mance o he MLPs was e alua ed by means o
a ame e o a e (FER). The pe o mance o he hyb id
sys em was e alua ed by means o Co ec ness (Co ) and
Accu acy (Acc).
Hie a chical e sus single laye classi ica ion
As desc ibed in Sec ion 4.1, a hie a chical classi ica ion o
di e en le els o phone ic in o ma ion is p oposed in
o de o imp o e phone ecogni ion. The neu al ne wo k
has abou 161 k pa ame e s. I comp ises eigh hidden
laye s. The numbe o nodes in he laye s is (in nume ical
o de ): 50-9-50-16-50-40-100-61. E en laye s gi e pos e -
io p obabili y p edic ions o BPC and he las laye gi es
pos e io p obabili y p edic ions o phones. Thus, he
p oposed MLP sys em is ained as a unc ion o he 61
phones and 3 addi ional se s o BPC, consis ing o 9, 16,
40 TIMIT phone se s ( iz. BPC9, BPC16, BPC40) achie ed
by he con usion-d i en clus e ing app oach p oposed.
The hie a chical phone clus e ing dend og am in Figu e 5
gi es ise o he BPC9. The wo o he s BPCs esul om
cu ing he da a-d i en dend og am ee in o wo o he
le els. The esul ing se s o BPCs we e g ouped acco ding
o he di ision p esen ed in Table 3.
S anda d MFCC’s ea u es and de i a i es a e p esen ed
o inpu (odd) hidden laye s. A con ex window o 290 ms
was used using only 15 ame ea u es (de ails in [8]).
All laye s we e ained concu en ly so ha , in aining
mode, a ge s we e p esen ed a all e en laye s: laye s 2, 4,
6 and 8. Since e en laye s a e ained wi h a so max ac i-
a ion unc ion, i s ou pu s can be seen as BPC p obabil-
i ies. The odd uses a sigmoid ac i a ion unc ion. All he
ne wo k weigh s and bias a e adjus ed using ba ch aining
wi h an RPROP algo i hm [21] so as o minimize he
minimum-c oss-en opy e o be ween he ne wo k ou -
pu and he a ge alues. The choice o he e o unc ion
ollowed Bishop’s sugges ion [32], which was la e cla i ied
by Dunne and Campbell [33]. I s a es ha he so max ac-
i a ion unc ion should couple wi h he c oss-en opy
penal y unc ion.
In o de o e alua e he pe o mance o he p oposed
hie a chical ne wo k, we ained ou single laye ne -
wo ks, each one specialized in one BPC. Each single
laye ne wo k was ained wi h he same numbe o hid-
den nodes as he hie a chical ne wo k (e.g. BPC16 MLP
has 50 hidden nodes and an ou pu laye wi h 16 ou -
pu s). The esul s in e ms o FER a e gi en in Figu e 7
and show ha he hie a chical ne wo k ou pe o med
he equi alen single laye ne wo ks wi h espec o
BPC16, BPC40 and 61 TIMIT phones. Rega ding he
classi ica ion o BPC9 (9 clus e s o Table 1) he pe -
o mance o he single laye ne wo k is simila o he
hie a chical. These esul s encou age he use o he hie -
a chical s uc u e in u he expe imen s. No e ha
hese esul s ela e o an e alua ion (in e ms o FER) o
he es se o e e y aining epoch.
Hie a chical con usion-d i en e sus hie a chical
knowledge-d i en phone ecogni ion
Since con usion-d i en clus e ing is he opic o his a icle,
he pe o mance o his kind o clus e ing has o be
compa ed wi h ha achie ed by an expe -knowledge ap-
p oach. The e o e, we ained wo hie a chical MLPs om
0 10 20 30 40 50 60 70 80 90 100
10
20
30
40
50
60
70
80
90
100
T ain Epochs
FER
FER 9 classes-hie a chical ne
FER 16 classes-hie a chical ne
FER 40 classes-hie a chical ne
FER 61 classes-hie a chical ne
FER 9 classes-single laye
FER 16 classes-single laye
FER 40 classes-single laye
FER 61 classes-single laye
Figu e 7 FER compa ison o hie a chical BPC’s classi ica ion
and equi alen single laye BPC’s classi ica ion.
Lopes and Pe digão EURASIP Jou nal on Ad ances in Signal P ocessing 2012, 2012:158 Page 9 o 12
h p://asp.eu asipjou nals.com/con en /2012/1/158