Optimization algorithms for estimating modulation spectrum domain filters

OPTIMIZATION ALGORITHMS FOR ESTIMATING MODULATION SPECTRUM DOMAIN FILTERS Pau Paches-Leal  y , z ,Richa dC.Rose y , and Climen Nadeu z y AT&T Labs-Resea ch, Flo ham Pa k, NJ, USA, z Uni . Poli ecnica de Ca alunya, Ba celona, Spain ABSTRACT The goal o he wo k desc ib ed in his pap e is o de elop and e alua e p o cedu es o au oma ic es ima ion o mod- ula ion sp ec um l e s o compensa e o dis o ions in he modula ion sp ec um domain. The mo dula ion sp ec- um (MS) is o en used o desc ib e he ime sequence o sp ec al pa ame e s (TSSPs) ha a e de i ed om he sp eechwa e o m, and is hough o be a go o d ep esen a- ion o many sou ces o a iabili y in speech. These p o- cedu es will b e used in he con ex o au oma ic sp eech ecogni ion (ASR) applica ions whe e he e is likely o b e a signican misma ch in he MS cha ac e is ics ha exis o sys em aining and e alua ion. Resul s a e p esen ed desc ibing applica ion o he algo i hm o one ask in ol - ing an a icially in oduced MS dis o ion and o ano he ask in ol ing die ences in sp eaking s yles o aining and es ing. I is shown in he pap e ha hese ech- niques a e able o comp ensa e o he eec s o a icially in o duced dis o ions ha appea in es ing. I is also shown ha a small deg ee o comp ensa ion is ob ained o sp eaking s yle misma ch, and his esul is compa ed wi h he measu ed eec s o he sp eaking s yle die ences in he MS domain. 1. INTRODUCTION The ime sequence o sp ec al ec o s de i ed om he sp eech signal can b e ep esen ed by he Mo dula ion Sp ec- um (MS) [5]. I has been p op osed o use in many ap- plica ions. These include cha ac e izing mul ipa h dis o - ions o ccu ing in e e b e an en i onmen s [2], desc ib- ing he eec s o channel dis o ion and sp ec al es ima- ion e o s in ASR [5], and desc ibing he eec s o a ying sp eaking s yles o ASR [5]. When used as a ep esen- a ion o he long{ e m a e aged spec um o ceps um pa ame e s in ASR, he MS is dened as he p owe sp ec- um o he ime sequence o he ea u e ec o s ha a e inpu o he ecognize . In sp eech ecogni ion ea u e ex- ac ion, MS l e s ha e b een designed o he pu p ose o selec i ely emo ing hose p o ions o he modula ion sp ec um ep esen ing noise o channel dis o ions, and e aining he p o ion o he MS con aining sp eech[5]. MS l e s a e commonly applied o l e ing he se- quence o ceps um co eÆcien s and also o compu ing he ceps um die ence dynamic co eÆcien s. I is imp o an o no e ha he MS l e s used o b o h he ceps um and dynamic ceps um coeÆcien s a e gene ally ob ained empi ically. As a esul , i is o en he case ha MS l- e s designed o op imize pe o mance o one ask unde a gi en se o condi ions p o e o be sub op imal when ap- plied o ano he ask. The au oma ic p o cedu e p esen ed he e o es ima ing mo dula ion sp ec um l e s is an a - emp o imp o e sp eech ecogni ion p e o mance unde highly misma ched ecogni ion / aining scena ios.  This esea chwas conduc ed a AT&T Shannon Labo a- o y as pa o P.Paches-Leal's Ph.D. hesis wi h he UPC The pap e is o ganized as ollows. Sec ion 2. desc ibes he mo dula ion compensa ion algo i hm (MCA) and dis- cusses implemen a ion issues ela ing o he algo i hm. In Sec ion 3., he algo i hm is implemen ed on an a i- cially in o duced dis o ion applied o he es da a. Fi- nally, in Sec ion 4., he issue o au oma ic comp ensa ion o sp eaking a e mis{ma ch using he MCA algo i hm is in es iga ed. 2. MODULATION COMPENSATION ALGORITHM The mo dula ion compensa ion algo i hm (MCA) es i- ma es a se o l e coeÆcien s o maximize he likeli- ho o d o he l e ed obse a ion sequence wi h esp ec o a gi en HMM model. The scena io unde which he al- go i hm is applied is illus a ed by he blo ck diag am in Figu e 1. The unde lying assump ion in his scena io is ha condi ions ha migh aec he MS cha ac e is ics o sp eech du ing ecogni ion may no b e p esen du ing HMM mo del aining. Discussion o he MCA algo i hm is p esen ed in his sec ion in wo pa s. Fi s , i is in o- duced as an ex ension o a class o echniques de elop ed by Chengal a ayan and Deng o simul aneous es ima ion o HMM and obse a ion sequence l e pa ame e s [1 ]. Second, he algo i hm is desc ib ed in de ail, along wi h algo i hmic issues ela ing o he op imiza ion c i e ion and he o m o he l e s ha a e used in he algo i hm. DOMAIN SPECIFIC MODULATION SPECTRAL SHAPING T aining Speech Tes Speech MODULATION COMPENSATION ALGORITHM (MCA) HMM Model λ Es ima ed Modula ion Spec um FIl e FORWARD− BACKWARD TRAINING Figu e 1. Mo dula ion comp ensa ion algo i hm (MCA) applied whe e mo dula ion sp ec um dis- o ions maybe in oduced in es condi ions. In [1 ], a class o echniques o simul aneous es ima ion o HMM and obse a ion sequence l e pa ame e s was de elop ed. This was used only o ob aining die ence ceps um pa ame e s om he absolu e ceps um. The comp onen s o a D dimensional dynamic ceps um ec o Y = y 1 ;:::;y D we e compu ed om a s a ic ec o X a ime acco ding o Y = X k =  b ! k;i;m X + k ; 1   T (1) whe e he mo dula ion" l e co eÆcien s ! k;i;m a e de- p enden on s a e i and Gaussian mix u e comp onen m . Equa ion 1 was in oduced in o he mo del ees ima ion equa ions o con inuous Gaussian obse a ion densi y HMMs and simul aneous ees ima ion o he l e co e - cien s and he HMM mo del pa ame e s was p e o med. 6 h Eu opean Con e ence on Speech Communica ion and Technology (EUROSPEECH’99) Budapes , Hunga y, Sep embe 5-9, 1999 ISCAA chi e h p://www.isca-speech.o g/a chi e Wellekens p op osed a mo e cons ained mo dula ion sp ec- um l e ees ima ion p o cedu e aimed a op imizing only he cu {o equency o he MS l e s [7]. Simul aneous es ima ion o hese pa ame e s has he desi able eec o p o iding a close coupling be ween mo del es ima ion and ea u e analysis in sp eech ecog- ni ion. Howe e , b ecause o he coupling o mo del es i- ma ion, i is unlikely ha he spec al cha ac e is ics o he l e pa ame e s es ima ed in his way will ac ually eec any meaning ul s uc u e asso cia ed wi h he MS o sp eech. The goal in his wo k is o es ima e he MS l- e pa ame e s sepa a e om he HMM model acili a ing he scena io depic ed by he blo ck diag am in Figu e 1. Since he mo del emains xed, MS misma chbe ween u - e ances used o ain he mo del and he u e ances used o es ing can b e educed. The MCA algo i hm and i s ma hema ical p ope ies a e desc ib ed in mo e de ail b e- low. The algo i hm es ima es a l e ha , when applied o he absolu e ceps um co eÆcien s, inc eases he likeliho o d o he l e ed da a, Y ! , wi h esp ec o he HMM model,  . This in heo y could b e accomplished byenume a ing an ensemble o MS l e s  and choosing he mos likely l e in he ensemble, ^ ~! = a g max ~! 2  P ( Y ! j ~!;  ) : (2) Howe e , in p ac ice i is e y diÆcul o speci y a man- ageable ensemble o l e s ha would b e sui able o ep- esen ing an a bi a y se o mo dula ion sp ec um dis- o ions and i is no clea ha a maximum likeliho o d c i e ion would b e sui able o selec ing he op imum MS l e om his ensemble. The algo i hm desc ib ed he e assumes a ni e impulse esponse l e , whose leng h mus be chosen, and es ima es he pa ame e s o his l e us- ing he exp ec a ion maximiza ion (EM) algo i hm. I can b e applied o l e ing he absolu e ceps um ea u es o o ob aining he l e pa ame e s used o compu e he dy- namic ea u es om he absolu e ceps um. The ollowing discussion will e e sp ecically o he o me case. Gi en an ini ial HMM mo del,  , and an ini ial leng h o he MS l e , ~! , his EM based algo i hm maximizes he exp ec ed alue o he log o he l e ed da a likeli- hood, P ( Y ! j ~!;  ), wi h esp ec o ~! . The da a o which he l e ~! is applied can be he unl e ed absolu e da a, X , o he absolu e da a l e ed wi h an ini ial l e . In he la e case, he o e all l e ha needs o b e applied o he es da a so as o educe he MS mis{ma chbe- ween he unl e ed aining da a and he es da a is he con olu ion o he ini ial l e and he l e op imized by he MCA, ~! . Unless o he wise sp ecied, no ini ial l e is used and he MCA is s a ed wi h unl e ed da a. The p o ion o he op imiza ion equa ion ha is dep en- den on he l e ed da a is gi en by X i;m;  ;i;m [ Y  ~ x;i;m ] T   1 x;i;m [ Y  ~ x;i;m ] ; (3) whe e  ;i;m is he a p os e io i p obabili yo o ccupying Gaussian mix u e componen m and s a e i a ime . The quan i ies ~ x;i;m and   1 x;i;m in Equa ion 3 a e he HMM mo del means and a iances o he unl e ed da a, X , and a e no ees ima ed as pa o his p o cedu e. Fo he pu p oses o his de elopmen , Y in Equa ion 1 can b e w i en in ec o no a ion as: Y = ( X  b ::: X + ) T ( !  b;i;m :::! ;i;m )) T = X +  b ~! i;m : (4) No e ha , while Equa ion 4 demons a es ha i is p os- sible o use MS l e s ~! i;m , ha a e HMM s a e and mix- u e comp onen dep enden , his is no done he e. By subs i u ing he exp ession o Y in Equa ion 3 and di - e en ia ing wi h espec o ~! , which does no dep end on i and m ,we ob ain X l;i;m;  ;i;m  X +  b  T   1 x;i;m X +  b ~! = X l;i;m;  ;i;m  X +  b  T   1 x;i;m ~ x;i;m (5) Finally, ~! can b e ob ained by sol ing he ma ix equa ion gi en by Equa ion 5 which equi es he mo del means and a iances, he a p os e io i p obabili ies compu ed in he o wa d{backwa d algo i hm, and he unl e ed obse a- ions. While he ab o e p o cedu e can b e i e a ed, using he es ima ed ~! ob ained by sol ing Equa ion 5 as inpu o a ollowing i e a ion, he e a numb e o issues ha mus b e deal wi h. The  s issue ela es o he ac ha i is he l e ed da a, as opp osed o he o iginal da a, ha is inpu o he nex i e a ion o he algo i hm. As a e- sul , he l e applied o he da a o a gi en i e a ion mus b e he con olu ion o all he l e s ob ained in he p eceding i e a ions. As has b een seen, a con olu ion is also necessa y o he case whe e he MCA p o cedu e is s a ed wi h da a l e ed wi h an ini ial l e . A second issue ela es o a i ac s ha a ise due o lack o ene gy cons ain s in he mo del es ima ion p o cedu e. An em- pi ical p o cedu e o no malizing l e ene gies be ween i e a ions is desc ib ed in Sec ion 3.. 3. COMPENSATING FOR MS DOMAIN MISMATCH The MCA algo i hm was  s applied o a simula ed dis- o ion. The goal o his  s applica ion was o imple- men a scena io as depic ed in he block diag am in Fig- u e 1. The o m o he mo dula ion sp ec um misma ch was in ended o inco p o a e a lo ose app oxima ion o ex- is ing MS mo dels o how sp eaking a e a iabili y migh b e eec ed in he mo dula ion sp ec um domain (e.g. [5]). These models cha ac e ize he MS o sp eechasha ing a p eak a app oxima ely ou Hz wi h some a ia ion in he lo ca ion o ha peak p ossibly esul ing om speaking a e die ences. The mo dula ion spec um dis o ion o ok he o m o an FIR l e o leng h 7 wi h a p eak a 10 Hz and a s ong a enua ion o mo dula ion equencies b eyond 20 Hz . This l e was ac ually applied o he aining u e ances o he high SNR, noise{ ee TI digi s da abase [4 ]. So he mo du- la ion sp ec um shaping indica ed in Figu e 1 would ac u- ally b e he in e se o ha sp ec um. The HMM mo del,  , was hen ained om he l e ed da a using he o wa d{ backwa d algo i hm. The 8623 digi s ings u e ed bya p opula ion o adul sp eake s in he aining se o he TI digi s da abase was used o aining digi mo dels wi h a mix u e o a mos 16 con inuous Gaussian comp onen s p e HMM s a e. The TI digi s es se was spli in o a 3306 u e ance de elopmen se , whichwas used as inpu o he MCA algo i hm, and a 5376 u e ance es se o e alua ing ASR wo d accu acy. The inpu s o MCA a e he model and a subse o he unl e ed de elopmen u e ances and he ou pu is a l- e ha educes he misma chbe ween he u e ances and he mo del. Figu e 2 shows he mo dula ion sp ec a o he aining u e ances and he unl e ed de elopmen u e ances o ceps al co eÆcien c 4 . The l e ha was applied o he aining u e ances is also shown. A sig- nican misma ch can be obse ed in he MS domain. The magni ude sp ec um o he MS l e es ima ed us- ing Equa ion 5 was ound o p o ide a easonably goo d app oxima ion magni ude sp ec um o he l e used on 0 5 10 15 20 10−1 100Modula ion Spec um o Dimension 4 dB Hz ain es il e Figu e 2. Mo dula ion Sp ec a o he T aining U - e ances and o he Unl e ed De elopmen U - e ances, along wi h he Spec um o he Fil e Applied o he T aining U e ances he aining u e ances. Howe e , since he e is no inhe - en cons ain on he ceps um ene gy le els in he MCA, signican misma ch in he ene gy le els o he l e ed ceps a and he o iginal ceps a can occu . To deal wi h his, he l e co eÆcien s a e scaled wi h a dimension spe- cic scale ac o so ha a e age ene gy o he aining and adap a ion u e ances a e no malized o he same le el. The es ima ed l e , scaled sepa a ely o each comp onen o he inpu ceps um ec o , is hen applied o he es u e ances. Table 1 displays he ecogni ion p e o mance a e he MS l e de e mined by he MCA was applied o he es u e ances. Pe o mance is p esen ed as a unc ion o he numb e o de elopmen o adap a ion u e ances ha we e used by he MCA. The algo i hm was implemen ed he e in a sup e ised mo de wi h he u e ance ansc ip- ions made a ailable o he MCA. The Table shows ha when he whole de elopmen se is used o de e mine he b es l e wi h which o educe he misma ch be ween ain and es u e ances, wo d accu acy app oaching he ma ched condi ion can be ob ained. When less da a is a ailable ( om 60 u e ances h ough jus one), p e o - mance deg ades g ace ully wi h esp ec o he whole de- elopmen se and s ill alle ia es mos o he ecogni ion a e educ ion due o he MS die ences b e ween aining and es ing. Resul s a e gi en in all cases o he l e ou pu by he  s i e a ion o MCA, which has leng h 3. Su p isingly, l e s wi h o he leng hs, esul ing ei he om unning se e al i e a ions wi h a l e leng h equal o 3, o om cho osing ano he l e leng h (e.g. 5,7o 9) also gi e goo d p e o mance bu sho o he esul s when he leng h is 3. A leng h o 3 seems o s ike he igh balance b e ween he numb e o deg ees o eedom and a cons ained es ima ion. Mo dula ion Sp ec um Adap a ion %Wo d Comp ensa ion U e ances Accu acy Ma ched - 96.30 Misma ched - 15.04 MCA 3306 92.10 MCA 10 88.78 MCA 1 87.93 Table 1. Resul s o MCA wi h a simula ed dis- o ion MCA seems o wo k well o he expe imen s ha we e done wi h a simula ed dis o ion. MCA was applied o es ima e he l e applied o he absolu e co eÆcien s, no dynamic ea u es we e used. The nex sec ion desc ibes an exp e imen whe e MCA was used o op imize he l e used o ob ain he del a coeÆcien s. 4. MCA AND SPEAKING RATE In o de o es MCA in a mo e ealis ic en i onmen , a da abase con aining sp eech elici ed a mul iple speaking a es was used [6]. This da abase, e e ed o as DB1, was eco ded in an anechoic o om wi h a high quali y mic ophone. I includes sp eech om 22 emale, and 24 male alke s, each o which eco ded 120 sen ences. Bo h no mal a e and as a e u e ances o each sen ence was elici ed om each speake . Fas a e sp eechwas elici ed om each sp eake by asking sp eake s o sp eak sen ences as apidly as p ossible wi hou g oss misp onouncia ions. Fo each sp eaking a e, c oss-wo d iphones backed o o monophones we e ained om he sen ences om 38 sp eake s. The emainde we e ese ed o es ing. Each sys em had 4903 s a es and o e 28000 Gaussians. In ecogni ion, each phone may be ecognized ia a iphone o he co esp onding monophone a any place wi hin a wo d. All 120 sen ences o all he es sp eake s we e ecognized. Figu e 3 desc ib es he exp e imen ha was ca ied ou . Modula ion Spec um Fil e ω Fas Speaking Ra e: De elopmen U e ances λ S Speake Dependen Model No mal Speaking Ra e: MODULATION COMPENSATION ALGORITHM Fas Speaking Ra e: Tes U e ances ASR FILTER TIME SEQUENCE OF CEPSTRUM PARAMETERS Figu e 3. MCA and sp eaking a e The goal was o s udy how well MCA can cop e wi h sp eaking a e misma ches and up o wha p oin hese can b e ep esen ed in he mo dula ion sp ec um. The ecog- ni ion esul s o all 8 es speake s we e analysed. The u e ances om a single male sp eake whe e mo e d a- ma ic die ences b e ween as and no mal a e sp eaking s yles we e obse ed we e selec ed o he exp e imen al s udy. Since MCA migh imp o ewo d accu acy by p e o m- ing sp eake adap a ion, a sp eake adap ed mo del was ob ained. To do his, he gaussians in he no mal- a e mo del we e clus e ed in o 4 eg ession classes. On he  s 60 no mal- a e sen ences, one Maximum Likeliho o d Linea Reg ession ull ma ix ans o ma ion (MLLR) was lea ned o each eg ession class [3]. These we e applied o he means o he no mal- a e no mal- a e mo del, which esul s in a sp eake adap ed no mal- a e mo del. This mo del was hen inpu o he MCA along wi h he  s 60 as sen ences, ega ded as de elopmen o adap- a ion sen ences. The l e ou pu by MCA was applied o he 60 las as u e ances. In his case, MCA was used o op imize he l e used o ob ain he del a ea u es (no o he dynamic ea u es we e used apa om hem). An- o he die ence wi h he p e ious expe imen is ha he l e used o ob ain he del a ea u es o he aining u - e ances was used as he ini ial l e o MCA. The leng h o he l e ou pu by each i e a ion was he ini ial model l e leng h, i.e. 5, plus 2 o each i e a ion. Resul s o his exp e imen can be ound in Table 2. Tes Condi ion %Wo d Acc. Sp eake Indep. No mal Ra e 57.28 Sp eake Adap . (SA) No mal Ra e 80.67 Baseline SA Fas Ra e 61.58 Po oled" MCA Fas Ra e 61.10 Dim. Specic" MCA Fas Ra e 63.96 Table 2. Resul s o MCA wi h die en sp eaking a es Two mo des o MCA we e in es iga ed. The  s , e- e ed o as p o oled MCA" in Table 2, es ima es he MCA l e co eÆcien s acco ding o he p o cedu e ou lined in Sec ion 2. The second, e med dimension specic MCA", es ima es a sepa a e se o MS l e co eÆcien s o each dimension o he ceps um obse a ion ec o . This ep e- sen s an ex ension o he MS l e desc ib ed in Equa ion 1 and indep enden ly es ima es l e pa ame e s o op imize dimension specic ML c i e ion. I is clea om Table 2 ha he MCA yields only a mo des imp o emen in WAC o e he baseline sys em on he as a e o his ask. 0 5 10 15 20 10−2 10−1 100Modula ion Spec um o Dimension 2 dB Hz as no mal Figu e 4. Measu ed MS om as and no mal sp eech u e ances In o de o ob ain some pe sp ec i e o he deg ee o which mo dula ion sp ec um based echniques may a - ec pe o mance o sp eaking a e misma ch, he a e - age mo dula ion spec um was measu ed o u e ances wi h die en sp eaking a es. Fo each ceps um com- p onen , he magni ude o he a e aged modula ion sp ec- um was compu ed o e 60 sen ences o no mal and as a e sp eech. Figu e 4 displays he magni ude spec a o one o he comp onen s o he acous ical ec o . I is clea om he cu es ha he as a e speech has only sligh ly mo e ene gy in he highe mo dula ion equencies han he no mal a e sp eech. This sugges s ha one migh ex- p ec only small changes in p e o mance using MS domain echniques o his ask, which supp o s he ai ly mino imp o emen s ha we e obse ed he e. 5. CONCLUSION The mo dula ion comp ensa ion algo i hm was p esen ed as a maximum likeliho o d echnique o es ima ing l e s o he ime sequence o spec al pa ame e s in o de o educe misma ch in he mo dula ion sp ec um. The p o ce- du e was desc ib ed as an ex ension o a class o echniques de elop ed in [1]. I can be applied o es ima ing MS l- e s o he absolu e ceps um co eÆcien s o o ob aining he dynamic ceps um co eÆcien s om he absolu e co- eÆcien s. Two applica ions o he MCA we e desc ib ed. The  s applica ion was o a ask whe e a simula ed mo dula ion sp ec um domain dis o ion was in oduced du ing es ing. I was shown ha he MCA can eec- i ely educe misma ch in he MS b e ween aining and es ing u e ances. In a second applica ion, he MCA was applied o educing misma ch a ibu able o die ences in sp eaking a e. The MCA was shown o ha e only a small impac on p e o mance o his ask. Fu he wo k is di ec ed owa ds applica ion o he algo i hm o o he sou ces o a iabili y whe e dis o ions in he mo dula ion sp ec um a e mo e p onounced. REFERENCES [1] Ra hina elu Chengal a ayan and Li Deng. Use o Gene alized Dynamic Fea u e Pa ame e s o Sp eech Recogni ion". IEEE T ans. on Speech and Audio P o- cessing , 5(3):232{242, May 1997. [2] T. Hou gas and H.J.M. S eeneken. A e iew o he MTF concep in o om acous ics and i s use o es- ima ing speech in elligibili y in audi o ia". JASA , 77(3):1069{1077, Ma ch 1985. [3] C.J. Legge e and P.C. Wo o dland. Maximum likeli- ho o d linea eg ession o speake adap a ion o con- inuous densi y hidden Ma ko mo dels". Compu e Speech and Language , 9:171{185, 1995. [4] R.G. Leona d. A Da abase o Sp eake -Independen Digi Recogni ion". In P oc. ICASSP'84 , pages 42.11.1{4, 1984. [5] Climen Nadeu, Pau Paches-Leal, and Biing-Hwang Juang. Fil e ing he ime sequences o sp ec al pa- ame e s o sp eech ecogni ion". Speech Communica- ion , 22:315{332, 1997. [6] Jue gen Sch o e e . A High-Quali y Sp eech Da abase". Technical ep o , AT&T Bell Lab o a o- ies, Mu ay Hill, NJ, USA, 1992. [7] Ch is J. Wellekens. Enhanced ASR by Acous ic Fea- u e Fil e ing". In P oc. ICSLP'98 , pages 2995{2998, Sydney, Aus alia, 1998.