Mixture of experts distributional regression: implementation using robust estimation with adaptive first-order methods

Rügame , Da id; P is e e , Flo ian; Bischl, Be nd; G ün, Be ina A icle — Published Ve sion Mix u e o expe s dis ibu ional eg ession: implemen a ion using obus es ima ion wi h adap i e i s -o de me hods AS A Ad ances in S a is ical Analysis P o ided in Coope a ion wi h: Sp inge Na u e Sugges ed Ci a ion: Rügame , Da id; P is e e , Flo ian; Bischl, Be nd; G ün, Be ina (2023) : Mix u e o expe s dis ibu ional eg ession: implemen a ion using obus es ima ion wi h adap i e i s - o de me hods, AS A Ad ances in S a is ical Analysis, ISSN 1863-818X, Sp inge , Be lin, Heidelbe g, Vol. 108, Iss. 2, pp. 351-373, h ps://doi.o g/10.1007/s10182-023-00486-8 This Ve sion is a ailable a : h ps://hdl.handle.ne /10419/313158 S anda d-Nu zungsbedingungen: Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen Zwecken und zum P i a geb auch gespeiche und kopie we den. Sie dü en die Dokumen e nich ü ö en liche ode komme zielle Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich machen, e eiben ode ande wei ig nu zen. So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen (insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en, gel en abweichend on diesen Nu zungsbedingungen die in de do genann en Lizenz gewäh en Nu zungs ech e. Te ms o use: Documen s in EconS o may be sa ed and copied o you pe sonal and schola ly pu poses. You a e no o copy documen s o public o comme cial pu poses, o exhibi he documen s publicly, o make hem publicly a ailable on he in e ne , o o dis ibu e o o he wise use he documen s in public. I he documen s ha e been made a ailable unde an Open Con en Licence (especially C ea i e Commons Licences), you may exe cise u he usage igh s as speci ied in he indica ed licence. h ps://c ea i ecommons.o g/licenses/by/4.0/ Vol.:(0123456789) AS A Ad ances in S a is ical Analysis (2024) 108:351–373 h ps://doi.o g/10.1007/s10182-023-00486-8 1 3 ORIGINAL PAPER Mix u e o expe s dis ibu ional eg ession: implemen a ion using obus es ima ion wi hadap i e i s ‑o de me hods Da idRügame 1,2,3 · Flo ianP is e e 1,3· Be ndBischl1,3· Be inaG ün4 Recei ed: 17 No embe 2022 / Accep ed: 24 Oc obe 2023 / Published online: 15 No embe 2023 © The Au ho (s) 2023 Abs ac In his wo k, we p opose an e icien implemen a ion o mix u es o expe s dis- ibu ional eg ession models which exploi s obus es ima ion by using s ochas ic i s -o de op imiza ion echniques wi h adap i e lea ning a e schedule s. We ake ad an age o he lexibili y and scalabili y o neu al ne wo k so wa e and imple- men he p oposed amewo k in mixdis eg, an R so wa e package ha allows o he de ini ion o mix u es o many di e en amilies, es ima ion in high-dimen- sional and la ge sample size se ings and obus op imiza ion based on Tenso Flow. Nume ical expe imen s wi h simula ed and eal-wo ld da a applica ions show ha op imiza ion is as eliable as es ima ion ia classical app oaches in many di e en se ings and ha esul s may be ob ained o complica ed scena ios whe e classical app oaches consis en ly ail. Keywo ds Mix u e models· Deep lea ning· S uc u ed addi i e eg ession· Neu al ne wo ks * Da id Rügame da id. uegame @s a .uni-muenchen.de Flo ian P is e e [email p o ec ed] Be nd Bischl be nd.bisc[email p o ec ed] Be ina G ün be ina.g [email p o ec ed] 1 Depa men o S a is ics, LMU Munich, Munich, Ge many 2 Depa men o S a is ics, TU Do mund, Do mund, Ge many 3 Munich Cen e o Machine Lea ning, Munich, Ge many 4 Ins i u e o S a is ics andMa hema ics, WU Vienna, Vienna, Aus ia 352 D.Rügame e al. 1 3 1 In oduc ion Mix u e models a e a common choice o model he join dis ibu ion o se e al sub- popula ions o subclasses on he basis o da a om he pooled popula ion whe e he subclass membe ships a e no obse ed ( o an in oduc ion see McLachlan and Peel 2004). Each subclass is assumed o ollow a pa ame ic p obabili y dis ibu ion such ha he pooled obse a ions a e om a mix u e o hese dis ibu ions. Many appli- ca ions o mix u e models aim a es ima ing he dis ibu ions o he la en subclasses and iden i ying subclass membe ships o he obse a ions (McLachlan e al. 2019). Mix u e models ha e also been used in machine lea ning, e.g., o clus e ing (Vi oli and McLachlan 2019), o build gene a i e models o images (Van den Oo d and Sch auwen 2014) o as a hyb id app oach o unsupe ised ou lie de ec ion (Zong e al. 2018). Di e en kinds o mix u e models a e used depending on he a ailable da a s uc- u e and he pa ame ic model which is assumed o each subpopula ion o subclass. In he ollowing, we assume a supe ised lea ning ask and ha eg ession mod- els a e used o model he subpopula ions. This leads o mix u e eg ession models which de ine a mix u e o (condi ional) models o he ou come o in e es , whe e he mix u e componen s a e s ill unknown and hus conside ed la en a iables. This model class is also e e ed o as mix u es o expe s (Go mley and F ühwi h-Schn- a e 2019). Mix u es o expe s allow o he inclusion o co a ia es when mod- eling he mix u e componen s as well as o he mix u e weigh s. In he ollowing, we conside a mix u e o expe s model whe e he eg ession models do no only include he mean pa ame e bu also o he dis ibu ional pa ame e s ha may depend on he co a ia es o ea u es. In addi ion, we assume ha he subpopula ion sizes a y wi h co a ia es (o ea u es). This leads o he class o mix u e o expe s dis i- bu ional eg ession models. 1.1 Mix u e models and hei es ima ion As o classical s a is ical eg ession models, he goal o mix u es o eg essions o mix u e eg ession models is o desc ibe he condi ional dis ibu ion o a esponse (o ou come), condi ional on a se o co a ia es (o ea u es). Mix u es o eg es- sions ha e been i s in oduced by Quand (1958) unde he e m swi ching egimes whe e only wo-componen mix u es o linea eg ession models we e conside ed, i.e., he numbe o subclasses was ixed o wo. This was ex ended o gene al mix- u es o linea eg ession models by DeSa bo and C on (1988) who e e ed o his app oach as a model-based e sion o clus e wise eg ession (Spä h 1979). The ex ension o mix u es o gene alized linea eg ession models was p oposed in Wedel and DeSa bo (1995). Aiming o a se ing beyond mean eg ession (Kneib 2013), a dis ibu ional eg ession se ing can be used such as gene alized addi i e models o loca ion, scale and shape (GAMLSS; Rigby and S asinopoulos 2005) which also allow o nonlinea smoo h ela ionships be ween co a ia es and he dis- ibu ional pa ame e o in e es (S asinopoulos e al. 2018). 353 1 3 Mix u e o expe s dis ibu ional eg ession: implemen a ion… Mix u e models can be es ima ed using a ious echniques, wi h he expec a- ion–maximiza ion (EM) algo i hm based on he maximum-likelihood p inciple being he mos p ominen one. O he app oaches include Bayesian me hods such as MCMC algo i hms wi h da a augmen a ion (Diebol and Robe 1994). An al e na- i e way o model speci ica ion and es ima ion was p oposed by Bishop (1994) who in oduced mix u e densi y ne wo ks (MDNs). MDNs use a mix u e o expe s dis- ibu ional eg ession model speci ica ion. Bu he ela ionship be ween he co a i- a es (o ea u es) and he mix u e weigh s o he dis ibu ional eg ession models is lea ned using a neu al ne wo k. The aining o MDNs is done using highly op i- mized s ochas ic g adien descen (SGD) ou ines wi h adap i e lea ning a es and momen um. These p ocedu es ha e p o en o be e y e ec i e in he op imiza ion o la ge neu al ne wo ks wi h millions o pa ame e s and complex model s uc u es. While o en MDNs a e ad an ageous in e ms o p edic ion pe o mance, hey lack in e p e abili y as he ela ionship be ween inpu s ( ea u es) and dis ibu ion pa am- e e s is modeled by a deep neu al ne wo k. 1.2 Ou con ibu ion 1.2.1 No el modeling app oach In his wo k, we combine he ideas o in e p e able mix u es o eg ession models and MDNs o allow o a mix u e o expe s dis ibu ional eg ession models in a e y gene al se up. In pa icula , his app oach enables modeling mix u es o sub- popula ions whe e he dis ibu ion o e e y subpopula ion is modeled using a dis i- bu ional eg ession. P edic o s o e e y dis ibu ion pa ame e in e e y subpopula- ion can be de ined by linea e ec s o ( enso p oduc ) splines and he eby allow no only o complex ela ionships be ween he ea u es and he dis ibu ions’ mean bu also o o he dis ibu ion cha ac e is ics such as scale o skewness. Fu he mo e, he subpopula ion sizes may a y wi h ea u es in a lexible way using again a com- bina ion o linea e ec s as well as ( enso p oduc ) splines. 1.2.2 Robus es ima ion Es ima ing mix u e eg ession models wi hin a maximum-likelihood amewo k on he basis o he EM algo i hm wo ks well o smalle p oblems. Howe e , highe - dimensional se ings whe e he numbe o pa ame e s is simila o o exceeds he numbe o obse a ions a e o en in easible o es ima e. In ac , he con e gence and s abili y o classical algo i hms a e mo e and mo e ad e sely a ec ed by an inc ease in model complexi y. Maximum-likelihood es ima ion o dis ibu ional eg ession models i sel induces a non-con ex op imiza ion p oblem in all bu special cases. Thus, ex ending mix u es o mean eg ession models o mix u es o dis ibu ional eg ession models u he complica es he op imiza ion. In his wo k, inspi ed by MDNs, we sugges and analyze he usage o s o- chas ic i s -o de op imiza ion echniques using adap i e lea ning a e schedul- e s o ( egula ized) maximum-likelihood es ima ion. Ou esul s show ha his 354 D.Rügame e al. 1 3 app oach is as eliable as es ima ion ia classical app oaches in many di e en se ings and e en p o ides es ima ion esul s in complex cases whe e classical app oaches consis en ly ail. 1.2.3 Flexible andscalable implemen a ion Common implemen a ions o EM op imiza ion ou ines a e limi ed in hei lexi- bili y o speci y a mix u e o (many) po en ially di e en dis ibu ions bu in gen- e al ocus on he case whe e all componen s ha e a pa ame ic dis ibu ion om he same dis ibu ional amily. Ano he (compu a ional) limi a ion o exis ing app oaches is aced o la ge amoun s o da a (obse a ions) as me hods usually scale a leas quad a ic in he numbe o obse a ions. On he o he hand, SGD op imize s om he ield o deep lea ning a e ained on mini-ba ches o da a allowing la ge da ase applica ions and he use in a gene ic ashion o all model classes. We ake ad an age o his lexibili y and scalabili y and implemen he i ing o he p oposed model class using mixdis eg, an R (R Co e Team 2022) so wa e package based on he R package deep eg ession (Rügame e al. 2023) ha allows o he de ini ion o mix u es wi h componen s om many di e en dis ibu ional amilies, es ima ion in high-dimensional and la ge sample size se - ings and obus op imiza ion based on he deep lea ning pla o m Tenso Flow (Abadi e al. 2015). In o de o use Tenso Flow wi hin R, he package e icula e (Ushey e al. 2022) is used o connec R o Py hon (Van Rossum and D ake J 1995). 1.2.4 Summa y o ou app oach ando e iew on hepape s uc u e Ou amewo k uni es neu al densi y ne wo ks (Magdon-Ismail and A iya 1998) wi h (dis ibu ional) eg ession app oaches, ex ends MDNs by inco po a ing penalized smoo h e ec s and comp ises a ious amewo ks p oposed in he s a- is ical communi y such as Leisch (2004); G ün and Leisch (2007); S asinopou- los and Rigby (2007) o es ima e mix u es o linea , gene alized linea , gene - alized addi i e o dis ibu ional eg ession models. In con as o many exis ing app oaches, ou amewo k u he allows o es ima e he mix u e weigh s hem- sel es on he basis o an addi i e s uc u ed p edic o . We e e o his combina- ion o he model class o mix u es o expe s dis ibu ional eg ession wi h s uc- u ed addi i e p edic o s and he es ima ion using neu al ne wo k so wa e as he neu al mix u e o expe s dis ibu ional eg ession (NMDR) app oach. The emainde o his pape is s uc u ed as ollows. In Sec .2, we p esen ou model de ini ion. In Sec .3, we in oduce he a chi ec u e o SGD-based op i- miza ion in neu al ne wo ks and discuss penalized es ima ion app oaches includ- ing mix u e spa si ica ion. We hen demons a e he amewo k’s p ope ies using ex ensi e nume ical expe imen s in Sec .4 and i s applica ion o eal-wo ld da a in Sec .5. We conclude wi h a discussion in Sec .6. 355 1 3 Mix u e o expe s dis ibu ional eg ession: implemen a ion… 2 Me hodology Ou goal is o model he condi ional dis ibu ion o Y∣x whe e Y is he uni a ia e ou come (o esponse) o in e es . We assume ha Y∣x∼F , whe e F is a mix- u e o pa ame ic dis ibu ions Fm,m∈{1, …,M} =∶ M . These dis ibu ions, in u n, depend on unknown pa ame e s 𝜽m(x)∈ ℝ km ha a e in luenced by a se o ea u es (o co a ia es) x∈ℝp . Fu he mo e, he nonnega i e mix u e weigh s a e also allowed o depend on ea u es (o co a ia es) x∈ℝp such ha 𝜋m(x) o all m=1, …,M wi h ∑M m=1 𝜋 m (x)= 1 . 2.1 Model de ini ion Fo an obse a ion y in (a sui able subse o ) ℝ cons i u ing he suppo om he condi ional dis ibu ion o Y∣x , we de ine he condi ional densi y by i.e., by a mix u e o densi y unc ions m o he dis ibu ions Fm . Each componen densi y m has i s own km dis ibu ion pa ame e s 𝜽m =(𝜃 m,1 ,…,𝜃 m,km ) ⊤ . 𝜋m∈[0, 1] a e he mix u e weigh s wi h ∑M m=1 𝜋 m = 1 . The ec o 𝝑=(𝜽⊤ ,𝝅⊤)⊤∈ ℝ K+M wi h K = ∑M m=1 k m comp ises all dis ibu ion pa ame e s 𝜽=(𝜽1,…,𝜽M)⊤ and all mix- u e weigh s 𝝅=( 𝜋 1,…, 𝜋 M)⊤ . Each o he pa ame e s 𝜗j in 𝝑 is assumed o depend on he ea u es x h ough an addi i e p edic o 𝜂j,j=1, …,K+M . Fo he dis ibu ion pa ame e s 𝜽, a mono onic and di e en iable unc ion hj,j=1, …,K, is assumed o p o ide a map be ween he addi i e s uc u ed p edic o and he dis ibu ion pa ame e , i.e., 𝜗j=hj(𝜂j(x)) . The pa ame e - ee ans o ma ion unc ion hj ensu es he co ec domain o each 𝜗j . Fo example, i 𝜗j ep esen s a scale pa ame e he unc ion, hj ensu es ha 𝜗j is posi i e while he addi i e p edic o 𝜂j may ake a bi a y alues in ℝ . Fo he mix u e weigh s, 𝝅 a single mono onic and di e en iable unc ion hK+1 is assumed o map he M addi i e p edic o s o he (M−1) -dimensional simplex, i.e., hK+1 maps ℝM → [0, 1]M unde he condi ion ha he sum o he weigh s is 1. This links he las M addi i e p edic o s 𝜼𝜋∶= ( 𝜂 K+1,…, 𝜂 K+M)⊤ o he se o mix u e weigh s 𝝅 . The mos common choice in his espec o hK+1 is he so max unc ion wi h (1) Y∣x(y∣𝝑(x)) = M ∑ m=1 𝜋m(x) m(y∣𝜽m(x)) , hK+1(𝜼𝜋)=(so max1(𝜼𝜋),…, so maxM(𝜼𝜋)) so max j(𝜼)= exp(𝜂 j ) ∑ M l=1 exp(𝜂 l ) . 356 D.Rügame e al. 1 3 This implies E ec s in he addi i e p edic o s 𝜂K+l,l=1, …,M , in (2) a e no iden i iable wi h- ou u he cons ain s. We do no en o ce any cons ain s du ing model aining. Model in e p e a ion is s ill possible in ela i e e ms (e.g., using a log-odds in e - p e a ion). Howe e , some cons ain s would need o be imposed i iden i iable eg ession coe icien s o he addi i e p edic o s a e o be ob ained. Fo all pa ame e s in 𝝑 , he addi i e s uc u ed p edic o s 𝜂j(x) ensu e in e p e - abili y o he ela ionship be ween he pa ame e 𝜗j and he co a ia es. Fo example, i 𝜂K+M is a linea model, i.e., 𝜂K+M ( x )= x⊤𝜷 , he eg ession coe icien s 𝜷 can be in e p e ed as linea con ibu ions o each o he ea u es o he logi s o he mix u e weigh o he las mix u e componen M. 2.2 Addi i e p edic o s uc u e The model (1) ela es all model pa ame e s 𝝑 o ea u es x h ough addi i e p edic- o s 𝜂j,j=1, …,K+M . As di e en densi ies m and also di e en pa ame e s 𝜽m ha e po en ially di e en in luences on he condi ional dis ibu ion o Y∣x , e e y pa ame e in 𝜽 is de ined by i s own addi i e s uc u ed p edic o 𝜂j . He e we assume he addi i e p edic o s o ha e he ollowing s uc u e: whe e 𝛽0,j co esponds o he model in e cep , 𝜷j a e he linea e ec s o p e-de ined co a ia es x L (j) wi h L(j)⊆{1, …,p}∪� being a subse o all possible p edic o s and 𝜙l,j a e nonlinea smoo h unc ions o one o mo e co a ia es in x wi h S(j) being a (po en ially emp y) se o indices indica ing he co a ia es wi h nonlinea p edic o e ec s. We assume ha e e y 𝜙l,j(x) can be ep esen ed by ( enso p oduc ) basis unc ions Bl,j,o,o=1, …,O , aking one o se e al columns o x as inpu and mapping hese on o he space spanned by he basis unc ions. Deno ing zl,j ∶= B l,j (x)=(B l,j,1 (x),…,B l,j,O (x)) ⊤ ∈ℝ O , he nonlinea smoo h e ms can be w i en as 𝜙 l,j(x)=z ⊤ l,j 𝜸l, j whe e 𝜸l,j ∈ℝ O a e he co esponding basis coe icien s o zl,j . In p inciple, such a lexible speci ica ion may also be used o 𝜂K+1,…,𝜂K+M , i.e., o he addi i e s uc u ed p edic o s ela ed o 𝝅 . While his is echnically possible (using a di e en subne wo k o e e y 𝜋m ), hese p edic o s a e usually assumed o sha e one and he same addi i e s uc u e, i.e., x L (j) and zl,j wi h l∈S(j) a e he same o all j ela ed o he M mix u e weigh s. This u he enables a s aigh o wa d in e p e a ion o he p edic o –mix u e weigh ela ionship, as sha ing one addi- i e p edic o ac oss all 𝜋m s esembles a mul inomial logis ic eg ession model. We will hus assume he same speci ica ion o all hese p edic o s. We summa ize all (2) 𝜋 m(xi)= exp(𝜂 K+m (x i )) ∑ M l=1 exp(𝜂 K+l (x i )) , o m∈M . (3) 𝜂 j=𝛽0,j+x⊤ L(j)𝜷j+ ∑ l∈ S (j) 𝜙l,j(x) , 357 1 3 Mix u e o expe s dis ibu ional eg ession: implemen a ion… model coe icien s o he linea e ms by 𝜷 =(𝛽 0,1 ,𝜷 ⊤ 1 ,𝛽 0,2 ,𝜷 ⊤ 2 ,…,𝛽 0,K+M ,𝜷 ⊤ K+M ) ⊤ and all model coe icien s o ep esen ing he nonlinea smoo h unc ions by 𝜸 =(𝛾 0,1 ,𝜸 ⊤ 1 ,𝛾 0,2 ,𝜸 ⊤ 2 ,…,𝛾 0,K + M ,𝜸 ⊤ K+M ) ⊤ . In he ollowing, we summa ize hese wo pa ame e ec o s as 𝝍=(𝜷⊤ ,𝜸⊤)⊤ . 2.3 Model log likelihood Based on model (1) and he s uc u es imposed on he p edic o s in (3), he nega i e log likelihood o he model pa ame e s 𝝍 o n independen obse a- ions y =( y1, … ,yn ) ⊤ ∈ℝ n and hei co esponding n obse ed ea u e ec o s x ∶= (x ⊤ 1 ,…,x ⊤ n ) ⊤ ∈ℝ n×p is gi en by he sum o he nega i e log-likelihood con i- bu ions 𝓁i(𝝍) o each obse a ion i=1, …,n : The nega i e log likelihood can be ew i en as he nega i e sum o he exponen i- a ed log likelihoods o bo h he mix u e weigh s 𝜋m and he componen densi ies m using he log-sum-exp (LSE) unc ion: In p ac ice, o mula ion (5) is o en p e e ed o e (4) as i is less a ec ed by unde - o o e low p oblems. 2.4 Iden i iabili y Iden i iabili y is o conce n o he mix u e o expe s dis ibu ional eg ession model because o he iden i iabili y p oblems which could occu due o he mix u e speci ica ion, due o he addi i e s uc u ed p edic o s and due o he so max map- ping o he mix u e weigh s. 2.4.1 Mix u e models Fo any mix u e model, i ial iden i iabili y p oblems may a ise due o label swi ch- ing and o e i ing wi h ei he emp y o duplica ed componen s. In addi ion, also gene ic iden i iabili y p oblems may occu o mix u es o dis ibu ions bu also o mix u es o eg essions. A de ailed o e iew o iden i iabili y p oblems in he ini e mix u e case is gi en in F ühwi h-Schna e (2006). (4) 𝓁 (𝝍) ∶= n ∑ i=1 𝓁i(𝝍)=− n ∑ i=1 log { M ∑ m=1 𝜋m(xi) m(yi∣𝜽m(xi)) }. (5) − n ∑ i=1 log { M ∑ m=1 exp [ log 𝜋m(xi)+log m(yi∣𝜽m(xi)) ]}. 358 D.Rügame e al. 1 3 2.4.2 Addi i e s uc u ed p edic o s The componen s o he addi i e model wi h p edic o s uc u es o he o m (3) a e in gene al only iden i iable up o a cons an and also equi e u he es ic ions i bo h linea and nonlinea smoo h e ec s a e de ined o one and he same co a ia e. Fo example, 𝜂j=𝛽0,j+𝜙1,j(x1) can be equally ep esen ed by  𝛽0,j +  𝜙 1,j (x 1) by de ining  𝛽0,j =𝛽 0,j + c and  𝜙1,j (x 1 )=𝜙 1,j (x 1 )− c wi h c∈ℝ , i.e., by adding and sub ac ing a cons an c in bo h e ms. We ensu e he iden i iabili y o nonlinea smoo h e ms 𝜙l,j by using sum- o-ze o cons ain s o all nonlinea addi i e unc ions such as splines o enso p oduc splines. The iden i iabili y o linea e ec s x𝛽 in he p esence o a uni a ia e nonlinea e ec 𝜙(x) mus also be ensu ed. In his case, se e al di e en op ions exis (see, e.g., Rügame e al. 2023b). The mos s aigh o wa d way is o use a nonlinea e ec 𝜙(x) wi h a basis ep esen a ion ha includes he linea e ec as null space (and hence, o enough penaliza ion as discussed in Sec .3.3, esul s in a linea e ec ). In his case, L(j) only consis s o a iables ha a e only modeled using a linea componen (e.g., ca ego ical e ec s) and L(j)∩S(j)=� . 3 Model ep esen a ion and obus es ima ion The obse ed da a log likelihood is, in gene al, no conca e and hus di icul o op i- mize. We sugges o use op imize s om he ield o deep lea ning by aming ou model as a neu al ne wo k. This enables he i ing o he ull model class o mix u e o expe s dis ibu ional eg ession models wi h addi i e s uc u e p edic o s in a s aigh o wa d way. Nume ical expe imen s con i m ha his choice—when p op- e ly ained—is no only mo e lexible and obus han EM-based op imiza ion bu also makes la ge da ase applica ions easible due o mini-ba ch aining. 3.1 Neu al ne wo k ep esen a ion Models desc ibed in (1) and (3) can be ep esen ed as neu al ne wo ks in he ol- lowing way. An exempla y a chi ec u e is depic ed in Fig.1. The ne wo k a chi ec- u e implemen ing model (1) de ines a mos K subne wo ks, whe e each subne wo k models one o mo e addi i e s uc u ed p edic o s 𝜂j o a dis ibu ion pa ame e 𝜃j . Using an app op ia e pa ame e - ee ans o ma ion unc ion, hese addi i e s uc- u ed p edic o s a e mapped o he dis ibu ion pa ame e s and passed o a dis ibu- ion laye (Dillon e al. 2017). Each dis ibu ion laye co esponds o a mix u e com- ponen m ha is u he passed o a mul inomial o ca ego ical dis ibu ion laye , modeling he mix u e o all de ined dis ibu ions. The mix u e weigh s can ei he be di ec ly es ima ed o also lea ned on he basis o inpu ea u es using addi i e s uc- u ed p edic o s in ano he subne wo k. A classical linea mix u e eg ession com- bining M linea eg essions would, e.g., be gi en by M subne wo ks, each lea ning he expec a ion o a no mal dis ibu ion and a mix u e subne wo k ha only akes a cons an inpu (a bias) and lea ns he M mix u e weigh s. The indi idual addi i e p edic o s o each subne wo k a e, in mos cases, ully connec ed laye s wi h only 365 1 3 Mix u e o expe s dis ibu ional eg ession: implemen a ion… eplica ions o he wo di e en mix u e weigh s and he wo numbe o noise a i- ables se ings. Resul s sugges ha ou app oach leads o be e p edic ions measu ed by he PLS, bu is in e io in e ms o LS. The smalle LS alues a e possibly due o ewe da a poin s o i he model because o he need o a alida ion se and due o he sh inkage induced by ea ly s opping he p ocedu e. The median pe o mance o he clus e ing induced based on he es ima ed pos e io p obabili ies is in gene al on pa wi h he EM-based app oach in e ms o ARI and ACC wi h lexmix, while showing e en sligh ly be e pe o mance in he Gaussian case. 4.5 Misspeci ied mix u es andspa si y In his simula ion, we use a no mal mix u e wi h pm =10 ixed linea p edic o s o each dis ibu ion and dis ibu ion pa ame e (mean and a iance), whe e all ea u es a e again d awn om a s anda d no mal dis ibu ion and eg ession coe - icien s om a uni o m U(−2, 2) -dis ibu ion. The da a a e hen gene a ed using M=2 ac ual mix u e componen s wi h 𝜋1 d awn (independen ly o ea u es) om a uni o m dis ibu ion on he in e al (0.06, 0.094) and 𝜋2=1−𝜋1 o ensu e ha he minimum alue o bo h p obabili ies is a leas 6%. We hen e alua e he es i- ma ion o mix u e p obabili ies by NMDR o n∈{300, 2500} when inc easing he numbe o speci ied dis ibu ions M†∈{3, 5, 10} . To allow o spa si y in 𝝅 , we use he objec i e unc ion 𝓁en in oduced in Sec .3.3. Fo each scena io, 10 eplica ions a e pe o med. ACC ARI PLS LS no mal (2) no mal (4) poisson (2)poisson (4) no mal (2) no mal (4) poisson (2)poisson (4) no mal (2) no mal (4) poisson (2)poisson (4) no mal (2) no mal (4) poisson (2)poisson (4) −10 −8 −6 −4 0.00 0.25 0.50 0.75 1.00 −8 −6 −4 −2 0.5 0.7 0.9 Value Me hod O acle EM NMDR Fig. 3 Compa ison o a e age PLS and LS as well as ACC and ARI o a s a e-o - he-a implemen a ion (EM), an o acle a ying coe icien model wi h known class membe ships and ou app oach (NMDR) in di e en colo s o he wo dis ibu ions and wo scales (x-axis). The boxplo s con ain he esul s o 10 eplica ions o e wo se ings o he mix u e weigh s and wo se ings o he numbe o noise a iables 366 D.Rügame e al. 1 3 4.5.1 Resul s Resul s o a ious se ings o he en opy penal y pa ame e 𝜉 a e depic ed in Fig.4. While se ing 𝜉 o small alues la ge han ze o can imp o e he p edic- i e pe o mance and e en ou pe o m he co ec ly speci ied model wi hou 1 3 8 PLS RMSE Coe . RMSE P ob. 300/10 2500/10 300/10 2500/10 300/10 2500/10 −50 −40 −30 −20 −10 0 2 4 6 0.0 0.1 0.2 0.3 n / p Value En opy Penal y 0 1e−05 0.001 0.01 0.1 0.3 co ec model Fig. 4 Model quali y o misspeci ied models wi h speci ied mix u es M†∈{3, 5, 10} (columns, coun - ing he addi ional componen s) ins ead o ac ual M=2 mix u es and di e en goodness o i measu es ( ows) o 10 eplica ions (boxes). Colo s co espond o di e en se ings o 𝜉 o ep esen es ima ion esul s o he co ec model (black) T ue P ob.:0.3923 T ue P ob.:0.6077 0.00 0.25 0.50 0.75 1.00 1e−051e−03 1e−01 En opy Penal y Es . P ob. Fig. 5 Coe icien pa h (es ima ed mix u e p obabili ies) o di e en en opy penal ies. A alue o a ound 1e-02 yields he bes ade-o be ween spa si y and es ima ion pe o mance 367 1 3 Mix u e o expe s dis ibu ional eg ession: implemen a ion… addi ional dis ibu ion componen s, he bias induced by he penal y gene ally dec eases he es ima ion pe o mance. In p ac ice, an app op ia e amoun o penaliza ion can be ound by unning c oss- alida ion along a g id o di e en 𝜉 alues as, e.g., done o he Lasso (Tibshi ani 1996). We addi ionally in es iga e he coe icien pa h ob ained om a ying al- ues o he penal y pa ame e 𝜉 o one simula ed example. Resul s o he n=2500, M†=5 se ing a e depic ed in Fig.5. 𝜉 is a ied be ween 0 and 1 on a loga i hmic scale. The ue model has wo nonze o p obabili ies 0.6077 and 0.3923 while he o he 3 en ies in 𝝅 a e 0. 5 Cell cycle‑ egula ed genes o yeas In o de o demons a e he lexibili y o ou app oach, we in es iga e i s applica ion o he yeas cell cycle da ase om Spellman e al. (1998). In his s udy, genome- wide mRNA le els we e measu ed o 6178 yeas open eading ames (ORFs) o 119min a 7-min in e als. We he e analyze he subse o da a whe e all 18 ime poin s o he alpha ac o a es a e a ailable. The esul ing longi udinal da ase consis s o 80,802 obse a ions o he s anda dized exp ession le els. A subse o his da ase was also analyzed using mix u e models in G ün e al. (2011). 5.1 Dis ibu ional mix u e eg ession As bo h he mean and s anda d de ia ion o he s anda dized exp ession le els o genes change o e ime, we apply a mix u e o dis ibu ional eg essions model whe e he mean 𝜇 and he s anda d de ia ion 𝜎 o he no mally dis ibu ed mix u e componen s depend on ime, i.e., Y ∼ ∑M m=1 𝜋 m N(𝜇 m, ,𝜎2 m, ) o ∈[0, 119] . The addi i e s uc u ed p edic o s o hese dis ibu ion pa ame e s a e de ined as whe e he nonlinea smoo h unc ions 𝜙 a e modeled by hin-pla e eg ession splines (Wood 2003). P e ious app oaches o modeling his da ase in es iga ed he use o a mix u e o mixed models, i.e., he inclusion o gene-speci ic andom e ec s (Luan and Li 2003; G ün e al. 2011). We in es iga e he e an al e na i e op ion o mod- eling his addi ional he e ogenei y by allowing o ime- a ying s anda d de ia ions. We use M=6 which co esponds o he numbe o mix u e componen s iden i ied by Spellman e al. (1998). 5.2 Resul s In o de o plo he es ima ed smoo h e ec s oge he wi h he ue obse a ions, we i s de i e he componen membe ship o e e y gene. As done in he E-s ep o mix u e model app oaches (see, e.g., G ün e al. 2011), we calcula e he a pos e io i h−1 (𝜇 m, )=𝛽 0,m,1 +𝜙 m,1 ( )and h −1 (𝜎 m, )=𝛽 0,m,2 +𝜙 m,2 ( ) , 368 D.Rügame e al. 1 3 p obabili y o e e y gene o belong o componen m and hen ake he maximum o all componen s 1, …,M=6 . Fo his applica ion, obse a ions we e only assigned o 5 o he 6 assumed componen s. No e ha due o he na u e o ou op imiza ion ou ine, no all componen s necessa ily con ain a leas one obse a ion. Compa ing he numbe o genes assigned o each clus e , one sees ha he mos common componen in ou esul s is clus e 6 wi h 39,078 obse a ions. Clus e 4 con ains 25,722 obse a ions, clus e 1 9306 and clus e 5 6552 obse a ions. Leas numbe o obse a ions is assigned o clus e 2 which con ains only 144 obse a ions. Figu e6 isualizes he esul s ob ained in a panel plo whe e in each panel he ajec o ies o he ORFs o all genes assigned o his clus e a e shown oge he wi h he componen -speci ic es ima es o he ime- a ying means and s and- a d de ia ions. The iden i ied clus e s clea ly a y in showing ei he an ini ial dec ease o inc ease in hei means. In addi ion, one also sees ha he s anda d de ia ions o he clus e s a y o e ime wi h some clus e s exhibi ing a pa icu- la ly la ge amoun o he e ogenei y a la e ime poin s. 6 Ou look We ha e in oduced he class o mix u es o expe s dis ibu ional eg ession wi h addi i e s uc u al p edic o s and in es iga ed i s embedding in o neu al ne wo ks o obus model es ima ion. O e all his leads o he neu al mix u e o expe s dis ibu ional eg ession (NMDR) app oach. We show ha popula i s -o de adap i e upda e ou ines a e well sui ed o lea ning hese mix u e o expe s (dis ibu ional) eg ession models and also highligh ha he embedding in o a neu al ne wo k es ima ion amewo k allows o s aigh o wa d ex ensions o he 56 124 0255075 100 0255075100 0255075 100 −4 0 4 −4 0 4 Time S anda dized exp ession le el Fig. 6 T ajec o ies o ORFs (y-axis) o all 4489 genes (black lines) o e he cou se o he 18 ime poin s (x-axis) wi h he es ima ed mean end ( ed solid line) pe componen ( ace s) and unce ain y isualized by wo imes he es ima ed ( ime- a ying) s anda d de ia ion (shaded ed a ea) 369 1 3 Mix u e o expe s dis ibu ional eg ession: implemen a ion… gene al mix u e model class and ( egula ized) maximum-likelihood es ima ion using op imiza ion ou ines sui able also o big da a applica ions due o mini- ba ch aining. Using he p oposed a chi ec u e o mix u e o expe s dis ibu- ional eg ession, a possible ex ension o ou app oach is he e o e he combina- ion wi h o he (deep) neu al ne wo ks. This allows lea ning bo h he dis ibu ion componen s and he mix u e weigh s ei he by (a) a s uc u ed model, such as a linea o addi i e model, (b) a cus om (deep) neu al ne wo k o (c) a combina ion he eo . A simila app oach has been in es iga ed by F i z e al. (2022) using a ze o-in la ed Poisson model (i.e., a mix u e including a poin mass dis ibu ion) which includes bo h addi i e e ec s and a g aph neu al ne wo k in he addi i e p edic o . Appendix A: Op imiza ion ou ines In o de o p o ide insigh s in o he a ious op imize s’ pe o mance, we conduc a small benchma k s udy o assess he in luence o he choice o an op imize and o ind a good de aul . Figu e7 isualizes he compa ison based on he anks o each op imize ac oss all 48 se ings om Sec .4.3. In all ou expe imen s, we use he Glo o ini ialize o ini ialize weigh s (Glo o and Bengio 2010) (ini ial- ized di e en ly o e e y op imize and es a ) a ba ch size o 50 and a maximum o 1500 epochs. Resul s Figu e 7 indica es ha con e gence p oblems a e p ima ily encoun e ed o SGD wi h a subs an ial amoun o uns di e ging du ing op imiza ion. An o e all pe o - mance assessmen based on anks indica es ha he RMSp op op imize achie es he lowes o e all ank. Howe e , he igu e highligh s ha in ac , no clea bes op i- mize eme ges ac oss all scena ios. The anks ob ained o he di e en op imize s also a y conside ably wi h he pe o mance c i e ion. O e all one migh conclude ha bo h he RMSp op op imize and he Adam op i- mize pe o m in gene al well, also when used wi h hei de aul se ings. We no e, howe e , ha uning he op imize and i s lea ning a e would in gene al also be bene icial in e ms o bo h p edic i e pe o mance and es ima ion quali y. A u he speed-up o some o hese ou ines can be achie ed by addi ionally inco po a ing momen um, which also p o ed o be e ec i e in he op imiza ion o addi i e models using boos ing (Schalk e al. 2022). Appendix B: Concomi an a iables Finally, we compa e ou app oach wi h lexmix in e ms o es ima ion quali y when he model includes concomi an a iables. 370 D.Rügame e al. 1 3 Da a gene a ing p ocess We simula e a mix u e eg ession model wi h M∈{2, 3} mix u e componen s, each ollowing a no mal dis ibu ion wi h ixed a iance, 𝜎2=1 , and mean de ined by p=10 co a ia es ha a e d awn om a s anda d no mal dis ibu ion and mul iplied by ixed coe icien also gene a ed om a s anda d no mal dis ibu ion. In con as o p e ious simula ions, his nume ical expe imen is based on a da a gene a ing p ocess ha uses addi ional 2, 4 o 8 co a ia es o de ine he addi i e p edic o s RMSE Coe .RMSE P ob. PLSARI P ob. ACC P ob. Adadel a (0.01) Adadel a (0.01) w/ CLR Adadel a (0.1) Adadel a (0.1) w/ CLR Adadel a (1) Adam (0.01) Adam (0.01) w/ CLR Adam (0.1) Adam (0.1) w/ CLR Range (0.01) Range (0.01) w/ CLR Range (0.1) Range (0.1) w/ CLR RMSp op (0.01) RMSp op (0.01) w/ CL R RMSp op (0.1) RMSp op (0.1) w/ CLR SGD (0.01) SGD (0.01) w/ CLR SGD (0.1) SGD (0.1) w/ CLR 0% 25% 50% 75% 100% 0% 25% 50% 75% 100% 0% 25% 50% 75% 100% 0% 25% 50% 75% 100% 0% 25% 50% 75% 100% Pe cen age Rank 1234 o highe Fig. 7 Compa ison o a ious op imize s (x-axis; wi h lea ning a e in b acke s) anked by pe o mance o di e en me ics (di e en ows), po en ially wi h addi ional cyclic lea ning a e schedule (CLR), ac oss simula ion se ings s udied in Sec .4.3. Ranks a e compu ed on pe o mance a e aged o e 3 ep- e i ions. Ba s om SGD uns do no sum up o 100% as some models di e ged du ing op imiza ion 371 1 3 Mix u e o expe s dis ibu ional eg ession: implemen a ion… 𝜂K+1,…,𝜂K+M . In o he wo ds, we do no se he mix u e p obabili ies o a ixed cons an bu make hem co a ia e-dependen . The p edic o s 𝜂K+1,…,𝜂K+M a e gen- e a ed independen ly om p edic o s 𝜂1,…,𝜂K by d awing co a ia es and e ec s om a s anda d no mal dis ibu ion and de ining hei e ec o be linea . Expe imen We se n=5000 and un he expe imen 10 imes o in es iga e he es ima ion pe o mance o he wo me hods o all eg ession coe icien s using he RMSE. mixdis eg is op imized using Adam wi h a lea ning a e 1e-3, ba ch size o 32, ea ly s opping on a 10% alida ion da a se and a pa ience o 250 epochs. Resul s Figu e8 shows he esul ing RMSE alues o di e en alues o M and he wo addi i e p edic o ypes (means and p obabili ies). Bo h app oaches pe o m well wi h RMSE alues in he ange 0.01 o 0.02, which is compa ably small compa ed o he ue coe icien alues which ange om −2 o 1.5. As in p e ious s udies, he es ima ion pe o mance o he EM-based app oach is sligh ly be e han he one using SGD. 2 mix u e componen s 3 mix u e componen s Mean p edic o P obabili y p edic o 248248 0.02 0.03 0.04 0.01 0.02 0.03 Numbe o co a ia es (in mix u e p obabili y p edic o ) RMSE lexmix mixdis eg Fig. 8 Compa ison o lexmix and mixdis eg es ima ion pe o mance using he RMSE ( isualized by boxplo s summa izing he 10 di e en uns) o coe icien s in he mean p edic o s o dis ibu ion membe s ( op ow) as well as addi i e p edic o o mix u e p obabili ies (bo om ow) o a mix u e o M∈{2, 3} dis ibu ions (columns) 372 D.Rügame e al. 1 3 Funding Open Access unding enabled and o ganized by P ojek DEAL. Decla a ions Compe ing inancial in e es s The au ho s decla e ha hey ha e no known compe ing inancial in e es s o pe sonal ela ionships ha could ha e appea ed o in luence he wo k epo ed in his pape . Open Access This a icle is licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License, which pe mi s use, sha ing, adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as long as you gi e app op ia e c edi o he o iginal au ho (s) and he sou ce, p o ide a link o he C ea i e Commons licence, and indica e i changes we e made. The images o o he hi d pa y ma e ial in his a icle a e included in he a icle’s C ea i e Commons licence, unless indica ed o he wise in a c edi line o he ma e ial. I ma e ial is no included in he a icle’s C ea i e Commons licence and you in ended use is no pe mi ed by s a u o y egula ion o exceeds he pe mi ed use, you will need o ob ain pe mis- sion di ec ly om he copy igh holde . To iew a copy o his licence, isi h p://c ea i ecommons.o g/ licenses/by/4.0/. Re e ences Abadi, M., Aga wal, A., Ba ham, P., e al.: Tenso Flow: la ge-scale machine lea ning on he e ogeneous sys ems (2015). h ps:// www. enso low. o g/ Bishop, C.M.: Mix u e densi y ne wo ks. As on Uni e si y (1994) DeSa bo, W.S., C on, W.L.: A maximum likelihood me hodology o clus e wise linea eg ession. J. Classi . 5(2), 249–282 (1988) Diebol , J., Robe , C.P.: Es ima ion o ini e mix u e dis ibu ions h ough Bayesian sampling. J. R. S a . Soc.: Se . B (Me hodol.) 56(2), 363–375 (1994) Dillon, J.V., Langmo e, I., T an, D., e  al.: Tenso Flow dis ibu ions. (2017). h ps:// doi. o g/ 10. 48550/ a Xi . 1711. 10604 F i z, C., Do iga i, E., Rügame , D.: Combining g aph neu al ne wo ks and spa io- empo al disease mod- els o imp o e he p edic ion o weekly co id-19 cases in Ge many. Sci. Rep. 12(1), 1–18 (2022) F ühwi h-Schna e , S.: Fini e Mix u e and Ma ko Swi ching Models. Sp inge , Be lin (2006) Gel and, A.E., Dey, D.K.: Bayesian model choice: asymp o ics and exac calcula ions. J. R. S a . Soc.: Se . B (Me hodol.) 56(3), 501–514 (1994) Glo o , X., Bengio, Y.: Unde s anding he di icul y o aining deep eed o wa d neu al ne wo ks. In: P oceedings o he Thi een h In e na ional Con e ence on A i icial In elligence and S a is ics, P o- ceedings o Machine Lea ning Resea ch, ol.9, pp. 249–256. PMLR (2010) Go mley, I.C., F ühwi h-Schna e , S.: Mix u e o expe s models. In: F ühwi h-Schna e , S., Celeux, G., Robe , C.P. (eds.) Handbook o Mix u e Analysis, pp. 271–307. Chapman & Hall/CRC Hand- books o Mode n S a is ical Me hods, Chapman and Hall/CRC (2019) G ün, B., Leisch, F.: Fi ing ini e mix u es o gene alized linea eg essions in R. Compu . S a . Da a Anal. 51(11), 5247–5252 (2007) G ün, B., Scha l, T., Leisch, F.: Modelling ime cou se gene exp ession da a wi h ini e mix u es o linea addi i e models. Bioin o ma ics 28(2), 222–228 (2011) Hin on, G., S i as a a, N., Swe sky, K.: Neu al ne wo ks o machine lea ning. Cou se a, ideo lec u es 264(1) (2012) Hube , L., A abie, P.: Compa ing pa i ions. J. Classi . 2(1), 193–218 (1985) Kingma, D.P., Ba, J.: Adam: A me hod o s ochas ic op imiza ion (2014). h ps:// doi. o g/ 10. 48550/ a Xi . 1412. 6980 Kneib, T.: Beyond mean eg ession. S a . Model. 13(4), 275–303 (2013) Leisch, F.: FlexMix: A gene al amewo k o ini e mix u e models and la en class eg ession in R. J. S a . So w. 11(8), 1–18 (2004). h ps:// doi. o g/ 10. 18637/ jss. 011. i08 Luan, Y., Li, H.: Clus e ing o ime-cou se gene exp ession da a using a mixed-e ec s model wi h B-splines. Bioin o ma ics 19(4), 474–482 (2003) Magdon-Ismail, M., A iya, A.: Neu al ne wo ks o densi y es ima ion. In: Ad ances in Neu al In o ma- ion P ocessing Sys ems (1998) 373 1 3 Mix u e o expe s dis ibu ional eg ession: implemen a ion… McLachlan, G.J., Peel, D.: Fini e Mix u e Models. John Wiley & Sons, London (2004) McLachlan, G.J., Lee, S.X., Ra hnayake, S.I.: Fini e mix u e models. Annu. Re . S a . Appl. 6(1), 355– 378 (2019) Quand , R.E.: The es ima ion o he pa ame e s o a linea eg ession sys em obeying wo sepa a e egimes. J. Am. S a . Assoc. 53(284), 873–880 (1958) R Co e Team: R: A Language and En i onmen o S a is ical Compu ing. R Founda ion o S a is ical Compu ing, Vienna, Aus ia (2022). h ps:// www.R- p oje c . o g/ Rigby, R.A., S asinopoulos, M.D.: GAMLSS: a dis ibu ional eg ession app oach. J. R. S a . Soc.: Se . C (Appl. S a .) 54(3), 507–554 (2005) Rügame , D., Kolb, C., F i z, C., e al.: deep eg ession: A lexible neu al ne wo k amewo k o semi- s uc u ed deep dis ibu ional eg ession. J. S a . So w. 105(1), 1–31 (2023a) Rügame , D., Kolb, C., Klein, N.: Semi-s uc u ed dis ibu ional eg ession. Am. S a . 1–12 (2023b) Schalk, D., Bischl, B., Rügame , D.: Accele a ed componen wise g adien boos ing using e icien da a ep esen a ion and momen um-based op imiza ion. J. Compu . G aph. S a . 1–11 (2022) Spä h, H.: Algo i hm39 clus e wise linea eg ession. Compu ing 22(4), 367–373 (1979) Spellman, P.T., She lock, G., Zhang, M.Q., e al.: Comp ehensi e iden i ica ion o cell cycle- egula ed genes o he yeas Saccha omyces ce e isiae by mic oa ay hyb idiza ion. Mol. Biol. Cell 9(12), 3273–3297 (1998) S asinopoulos, D.M., Rigby, R.A.: Gene alized addi i e models o loca ion scale and shape (GAMLSS) in R. J. S a . So w. 23(7), 1–46 (2007) S asinopoulos, M., Rigby, B.: gamlss.mx: Fi ing Mix u e Dis ibu ions wi h GAMLSS. R package e - sion 4.3-5 (2016) S asinopoulos, M.D., Rigby, R.A., Bas iani, F.D.: GAMLSS: a dis ibu ional eg ession app oach. S a . Model. 18(3–4), 248–273 (2018) Tibshi ani, R.: Reg ession sh inkage and selec ion ia he lasso. J. R. S a . Soc.: Se . B (Me hodol.) 58(1), 267–288 (1996) Ushey, K., Allai e, J., Tang, Y.: e icula e: In e ace o ‘Py hon’. h ps:// CRAN.R- p oje c . o g/ packa ge= e ic ula e, package e sion 1.26 (2022) Vanden Oo d, A., Sch auwen, B.: Fac o ing a ia ions in na u al images wi h deep Gaussian mix u e models. In: Ad ances in Neu al In o ma ion P ocessing Sys ems, pp. 3518–3526 (2014) Van Rossum, G., D ake J , F.L.: Py hon e e ence manual. Cen um oo Wiskunde en In o ma ica Ams e dam (1995) Vi oli, C., McLachlan, G.J.: Deep Gaussian mix u e models. S a . Compu . 29(1), 43–51 (2019) Wedel, M., DeSa bo, W.S.: A mix u e likelihood app oach o gene alized linea models. J. Classi . 12(1), 21–55 (1995) Wood, S.N.: Thin pla e eg ession splines. J. R. S a . Soc. Se . B (S a . Me hodol.) 65(1), 95–114 (2003) Wood, S.N.: Gene alized Addi i e Models: An In oduc ion wi h R. CRC P ess (2017) Wood, S.N., Fasiolo, M.: A gene alized Fellne –Schall me hod o smoo hing pa ame e op imiza ion wi h applica ion o Tweedie loca ion, scale and shape models. Biome ics 73(4), 1071–1081 (2017). h ps:// doi. o g/ 10. 1111/ biom. 12666 W igh , L.: New deep lea ning op imize , ange : Syne gis ic combina ion o adam lookahead o he bes o ... Medium (2019) Zeile , M.D.: ADADELTA: An adap i e lea ning a e me hod (2012). h ps:// doi. o g/ 10. 48550/ a Xi . 1212. 5701 Zong, B., Song, Q., Min, M.R., e  al.: Deep au oencoding Gaussian mix u e model o unsupe ised anomaly de ec ion. In: In e na ional Con e ence on Lea ning Rep esen a ions (2018) Publishe ’s No e Sp inge Na u e emains neu al wi h ega d o ju isdic ional claims in published maps and ins i u ional a ilia ions.