Rügame , Da id; P is e e , Flo ian; Bischl, Be nd; G ün, Be ina
A icle — Published Ve sion
Mix u e o expe s dis ibu ional eg ession:
implemen a ion using obus es ima ion wi h adap i e
i s -o de me hods
AS A Ad ances in S a is ical Analysis
P o ided in Coope a ion wi h:
Sp inge Na u e
Sugges ed Ci a ion: Rügame , Da id; P is e e , Flo ian; Bischl, Be nd; G ün, Be ina (2023) : Mix u e
o expe s dis ibu ional eg ession: implemen a ion using obus es ima ion wi h adap i e i s -
o de me hods, AS A Ad ances in S a is ical Analysis, ISSN 1863-818X, Sp inge , Be lin, Heidelbe g,
Vol. 108, Iss. 2, pp. 351-373,
h ps://doi.o g/10.1007/s10182-023-00486-8
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/313158
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h ps://c ea i ecommons.o g/licenses/by/4.0/
Vol.:(0123456789)
AS A Ad ances in S a is ical Analysis (2024) 108:351–373
h ps://doi.o g/10.1007/s10182-023-00486-8
1 3
ORIGINAL PAPER
Mix u e o expe s dis ibu ional eg ession:
implemen a ion using obus es ima ion wi hadap i e
i s ‑o de me hods
Da idRügame 1,2,3 · Flo ianP is e e 1,3· Be ndBischl1,3· Be inaG ün4
Recei ed: 17 No embe 2022 / Accep ed: 24 Oc obe 2023 / Published online: 15 No embe 2023
© The Au ho (s) 2023
Abs ac
In his wo k, we p opose an e icien implemen a ion o mix u es o expe s dis-
ibu ional eg ession models which exploi s obus es ima ion by using s ochas ic
i s -o de op imiza ion echniques wi h adap i e lea ning a e schedule s. We ake
ad an age o he lexibili y and scalabili y o neu al ne wo k so wa e and imple-
men he p oposed amewo k in mixdis eg, an R so wa e package ha allows o
he de ini ion o mix u es o many di e en amilies, es ima ion in high-dimen-
sional and la ge sample size se ings and obus op imiza ion based on Tenso Flow.
Nume ical expe imen s wi h simula ed and eal-wo ld da a applica ions show ha
op imiza ion is as eliable as es ima ion ia classical app oaches in many di e en
se ings and ha esul s may be ob ained o complica ed scena ios whe e classical
app oaches consis en ly ail.
Keywo ds Mix u e models· Deep lea ning· S uc u ed addi i e eg ession· Neu al
ne wo ks
* Da id Rügame
da id. uegame @s a .uni-muenchen.de
Flo ian P is e e
[email p o ec ed]
Be nd Bischl
be nd.bisc[email p o ec ed]
Be ina G ün
be ina.g [email p o ec ed]
1 Depa men o S a is ics, LMU Munich, Munich, Ge many
2 Depa men o S a is ics, TU Do mund, Do mund, Ge many
3 Munich Cen e o Machine Lea ning, Munich, Ge many
4 Ins i u e o S a is ics andMa hema ics, WU Vienna, Vienna, Aus ia
352
D.Rügame e al.
1 3
1 In oduc ion
Mix u e models a e a common choice o model he join dis ibu ion o se e al sub-
popula ions o subclasses on he basis o da a om he pooled popula ion whe e he
subclass membe ships a e no obse ed ( o an in oduc ion see McLachlan and Peel
2004). Each subclass is assumed o ollow a pa ame ic p obabili y dis ibu ion such
ha he pooled obse a ions a e om a mix u e o hese dis ibu ions. Many appli-
ca ions o mix u e models aim a es ima ing he dis ibu ions o he la en subclasses
and iden i ying subclass membe ships o he obse a ions (McLachlan e al. 2019).
Mix u e models ha e also been used in machine lea ning, e.g., o clus e ing (Vi oli
and McLachlan 2019), o build gene a i e models o images (Van den Oo d and
Sch auwen 2014) o as a hyb id app oach o unsupe ised ou lie de ec ion (Zong
e al. 2018).
Di e en kinds o mix u e models a e used depending on he a ailable da a s uc-
u e and he pa ame ic model which is assumed o each subpopula ion o subclass.
In he ollowing, we assume a supe ised lea ning ask and ha eg ession mod-
els a e used o model he subpopula ions. This leads o mix u e eg ession models
which de ine a mix u e o (condi ional) models o he ou come o in e es , whe e
he mix u e componen s a e s ill unknown and hus conside ed la en a iables. This
model class is also e e ed o as mix u es o expe s (Go mley and F ühwi h-Schn-
a e 2019). Mix u es o expe s allow o he inclusion o co a ia es when mod-
eling he mix u e componen s as well as o he mix u e weigh s. In he ollowing,
we conside a mix u e o expe s model whe e he eg ession models do no only
include he mean pa ame e bu also o he dis ibu ional pa ame e s ha may depend
on he co a ia es o ea u es. In addi ion, we assume ha he subpopula ion sizes
a y wi h co a ia es (o ea u es). This leads o he class o mix u e o expe s dis i-
bu ional eg ession models.
1.1 Mix u e models and hei es ima ion
As o classical s a is ical eg ession models, he goal o mix u es o eg essions o
mix u e eg ession models is o desc ibe he condi ional dis ibu ion o a esponse
(o ou come), condi ional on a se o co a ia es (o ea u es). Mix u es o eg es-
sions ha e been i s in oduced by Quand (1958) unde he e m swi ching egimes
whe e only wo-componen mix u es o linea eg ession models we e conside ed,
i.e., he numbe o subclasses was ixed o wo. This was ex ended o gene al mix-
u es o linea eg ession models by DeSa bo and C on (1988) who e e ed o his
app oach as a model-based e sion o clus e wise eg ession (Spä h 1979). The
ex ension o mix u es o gene alized linea eg ession models was p oposed in
Wedel and DeSa bo (1995). Aiming o a se ing beyond mean eg ession (Kneib
2013), a dis ibu ional eg ession se ing can be used such as gene alized addi i e
models o loca ion, scale and shape (GAMLSS; Rigby and S asinopoulos 2005)
which also allow o nonlinea smoo h ela ionships be ween co a ia es and he dis-
ibu ional pa ame e o in e es (S asinopoulos e al. 2018).
353
1 3
Mix u e o expe s dis ibu ional eg ession: implemen a ion…
Mix u e models can be es ima ed using a ious echniques, wi h he expec a-
ion–maximiza ion (EM) algo i hm based on he maximum-likelihood p inciple
being he mos p ominen one. O he app oaches include Bayesian me hods such as
MCMC algo i hms wi h da a augmen a ion (Diebol and Robe 1994). An al e na-
i e way o model speci ica ion and es ima ion was p oposed by Bishop (1994) who
in oduced mix u e densi y ne wo ks (MDNs). MDNs use a mix u e o expe s dis-
ibu ional eg ession model speci ica ion. Bu he ela ionship be ween he co a i-
a es (o ea u es) and he mix u e weigh s o he dis ibu ional eg ession models is
lea ned using a neu al ne wo k. The aining o MDNs is done using highly op i-
mized s ochas ic g adien descen (SGD) ou ines wi h adap i e lea ning a es and
momen um. These p ocedu es ha e p o en o be e y e ec i e in he op imiza ion
o la ge neu al ne wo ks wi h millions o pa ame e s and complex model s uc u es.
While o en MDNs a e ad an ageous in e ms o p edic ion pe o mance, hey lack
in e p e abili y as he ela ionship be ween inpu s ( ea u es) and dis ibu ion pa am-
e e s is modeled by a deep neu al ne wo k.
1.2 Ou con ibu ion
1.2.1 No el modeling app oach
In his wo k, we combine he ideas o in e p e able mix u es o eg ession models
and MDNs o allow o a mix u e o expe s dis ibu ional eg ession models in a
e y gene al se up. In pa icula , his app oach enables modeling mix u es o sub-
popula ions whe e he dis ibu ion o e e y subpopula ion is modeled using a dis i-
bu ional eg ession. P edic o s o e e y dis ibu ion pa ame e in e e y subpopula-
ion can be de ined by linea e ec s o ( enso p oduc ) splines and he eby allow no
only o complex ela ionships be ween he ea u es and he dis ibu ions’ mean bu
also o o he dis ibu ion cha ac e is ics such as scale o skewness. Fu he mo e,
he subpopula ion sizes may a y wi h ea u es in a lexible way using again a com-
bina ion o linea e ec s as well as ( enso p oduc ) splines.
1.2.2 Robus es ima ion
Es ima ing mix u e eg ession models wi hin a maximum-likelihood amewo k on
he basis o he EM algo i hm wo ks well o smalle p oblems. Howe e , highe -
dimensional se ings whe e he numbe o pa ame e s is simila o o exceeds he
numbe o obse a ions a e o en in easible o es ima e. In ac , he con e gence and
s abili y o classical algo i hms a e mo e and mo e ad e sely a ec ed by an inc ease
in model complexi y. Maximum-likelihood es ima ion o dis ibu ional eg ession
models i sel induces a non-con ex op imiza ion p oblem in all bu special cases.
Thus, ex ending mix u es o mean eg ession models o mix u es o dis ibu ional
eg ession models u he complica es he op imiza ion.
In his wo k, inspi ed by MDNs, we sugges and analyze he usage o s o-
chas ic i s -o de op imiza ion echniques using adap i e lea ning a e schedul-
e s o ( egula ized) maximum-likelihood es ima ion. Ou esul s show ha his
354
D.Rügame e al.
1 3
app oach is as eliable as es ima ion ia classical app oaches in many di e en
se ings and e en p o ides es ima ion esul s in complex cases whe e classical
app oaches consis en ly ail.
1.2.3 Flexible andscalable implemen a ion
Common implemen a ions o EM op imiza ion ou ines a e limi ed in hei lexi-
bili y o speci y a mix u e o (many) po en ially di e en dis ibu ions bu in gen-
e al ocus on he case whe e all componen s ha e a pa ame ic dis ibu ion om
he same dis ibu ional amily. Ano he (compu a ional) limi a ion o exis ing
app oaches is aced o la ge amoun s o da a (obse a ions) as me hods usually
scale a leas quad a ic in he numbe o obse a ions. On he o he hand, SGD
op imize s om he ield o deep lea ning a e ained on mini-ba ches o da a
allowing la ge da ase applica ions and he use in a gene ic ashion o all model
classes. We ake ad an age o his lexibili y and scalabili y and implemen he
i ing o he p oposed model class using mixdis eg, an R (R Co e Team 2022)
so wa e package based on he R package deep eg ession (Rügame e al. 2023)
ha allows o he de ini ion o mix u es wi h componen s om many di e en
dis ibu ional amilies, es ima ion in high-dimensional and la ge sample size se -
ings and obus op imiza ion based on he deep lea ning pla o m Tenso Flow
(Abadi e al. 2015). In o de o use Tenso Flow wi hin R, he package e icula e
(Ushey e al. 2022) is used o connec R o Py hon (Van Rossum and D ake J
1995).
1.2.4 Summa y o ou app oach ando e iew on hepape s uc u e
Ou amewo k uni es neu al densi y ne wo ks (Magdon-Ismail and A iya 1998)
wi h (dis ibu ional) eg ession app oaches, ex ends MDNs by inco po a ing
penalized smoo h e ec s and comp ises a ious amewo ks p oposed in he s a-
is ical communi y such as Leisch (2004); G ün and Leisch (2007); S asinopou-
los and Rigby (2007) o es ima e mix u es o linea , gene alized linea , gene -
alized addi i e o dis ibu ional eg ession models. In con as o many exis ing
app oaches, ou amewo k u he allows o es ima e he mix u e weigh s hem-
sel es on he basis o an addi i e s uc u ed p edic o . We e e o his combina-
ion o he model class o mix u es o expe s dis ibu ional eg ession wi h s uc-
u ed addi i e p edic o s and he es ima ion using neu al ne wo k so wa e as he
neu al mix u e o expe s dis ibu ional eg ession (NMDR) app oach.
The emainde o his pape is s uc u ed as ollows. In Sec .2, we p esen ou
model de ini ion. In Sec .3, we in oduce he a chi ec u e o SGD-based op i-
miza ion in neu al ne wo ks and discuss penalized es ima ion app oaches includ-
ing mix u e spa si ica ion. We hen demons a e he amewo k’s p ope ies using
ex ensi e nume ical expe imen s in Sec .4 and i s applica ion o eal-wo ld da a
in Sec .5. We conclude wi h a discussion in Sec .6.

355
1 3
Mix u e o expe s dis ibu ional eg ession: implemen a ion…
2 Me hodology
Ou goal is o model he condi ional dis ibu ion o
Y∣x
whe e Y is he uni a ia e
ou come (o esponse) o in e es . We assume ha
Y∣x∼F
, whe e
F
is a mix-
u e o pa ame ic dis ibu ions
Fm,m∈{1, …,M} =∶ M
. These dis ibu ions, in
u n, depend on unknown pa ame e s
𝜽m(x)∈
ℝ
km
ha a e in luenced by a se o
ea u es (o co a ia es)
x∈ℝp
. Fu he mo e, he nonnega i e mix u e weigh s a e
also allowed o depend on ea u es (o co a ia es)
x∈ℝp
such ha
𝜋m(x)
o all
m=1, …,M
wi h
∑M
m=1
𝜋
m
(x)=
1
.
2.1 Model de ini ion
Fo an obse a ion y in (a sui able subse o )
ℝ
cons i u ing he suppo om he
condi ional dis ibu ion o
Y∣x
, we de ine he condi ional densi y by
i.e., by a mix u e o densi y unc ions
m
o he dis ibu ions
Fm
. Each componen
densi y
m
has i s own
km
dis ibu ion pa ame e s
𝜽m
=(𝜃
m,1
,…,𝜃
m,km
)
⊤
.
𝜋m∈[0, 1]
a e he mix u e weigh s wi h
∑M
m=1
𝜋
m
=
1
. The ec o
𝝑=(𝜽⊤
,𝝅⊤)⊤∈
ℝ
K+M
wi h
K
=
∑M
m=1
k
m
comp ises all dis ibu ion pa ame e s
𝜽=(𝜽1,…,𝜽M)⊤
and all mix-
u e weigh s
𝝅=(
𝜋
1,…,
𝜋
M)⊤
.
Each o he pa ame e s
𝜗j
in
𝝑
is assumed o depend on he ea u es
x
h ough
an addi i e p edic o
𝜂j,j=1, …,K+M
. Fo he dis ibu ion pa ame e s
𝜽,
a
mono onic and di e en iable unc ion
hj,j=1, …,K,
is assumed o p o ide a
map be ween he addi i e s uc u ed p edic o and he dis ibu ion pa ame e , i.e.,
𝜗j=hj(𝜂j(x))
. The pa ame e - ee ans o ma ion unc ion
hj
ensu es he co ec
domain o each
𝜗j
. Fo example, i
𝜗j
ep esen s a scale pa ame e he unc ion,
hj
ensu es ha
𝜗j
is posi i e while he addi i e p edic o
𝜂j
may ake a bi a y alues in
ℝ
.
Fo he mix u e weigh s,
𝝅
a single mono onic and di e en iable unc ion
hK+1
is
assumed o map he M addi i e p edic o s o he
(M−1)
-dimensional simplex, i.e.,
hK+1
maps
ℝM
→
[0, 1]M
unde he condi ion ha he sum o he weigh s is 1. This
links he las M addi i e p edic o s
𝜼𝜋∶= (
𝜂
K+1,…,
𝜂
K+M)⊤
o he se o mix u e
weigh s
𝝅
. The mos common choice in his espec o
hK+1
is he so max unc ion
wi h
(1)
Y∣x(y∣𝝑(x)) =
M
∑
m=1
𝜋m(x) m(y∣𝜽m(x))
,
hK+1(𝜼𝜋)=(so max1(𝜼𝜋),…, so maxM(𝜼𝜋))
so max
j(𝜼)=
exp(𝜂
j
)
∑
M
l=1
exp(𝜂
l
)
.
356
D.Rügame e al.
1 3
This implies
E ec s in he addi i e p edic o s
𝜂K+l,l=1, …,M
, in (2) a e no iden i iable wi h-
ou u he cons ain s. We do no en o ce any cons ain s du ing model aining.
Model in e p e a ion is s ill possible in ela i e e ms (e.g., using a log-odds in e -
p e a ion). Howe e , some cons ain s would need o be imposed i iden i iable
eg ession coe icien s o he addi i e p edic o s a e o be ob ained.
Fo all pa ame e s in
𝝑
, he addi i e s uc u ed p edic o s
𝜂j(x)
ensu e in e p e -
abili y o he ela ionship be ween he pa ame e
𝜗j
and he co a ia es. Fo example,
i
𝜂K+M
is a linea model, i.e.,
𝜂K+M
(
x
)=
x⊤𝜷
, he eg ession coe icien s
𝜷
can be
in e p e ed as linea con ibu ions o each o he ea u es o he logi s o he mix u e
weigh o he las mix u e componen M.
2.2 Addi i e p edic o s uc u e
The model (1) ela es all model pa ame e s
𝝑
o ea u es
x
h ough addi i e p edic-
o s
𝜂j,j=1, …,K+M
. As di e en densi ies
m
and also di e en pa ame e s
𝜽m
ha e po en ially di e en in luences on he condi ional dis ibu ion o
Y∣x
, e e y
pa ame e in
𝜽
is de ined by i s own addi i e s uc u ed p edic o
𝜂j
. He e we assume
he addi i e p edic o s o ha e he ollowing s uc u e:
whe e
𝛽0,j
co esponds o he model in e cep ,
𝜷j
a e he linea e ec s o p e-de ined
co a ia es
x
L
(j)
wi h
L(j)⊆{1, …,p}∪�
being a subse o all possible p edic o s
and
𝜙l,j
a e nonlinea smoo h unc ions o one o mo e co a ia es in
x
wi h
S(j)
being a (po en ially emp y) se o indices indica ing he co a ia es wi h nonlinea
p edic o e ec s. We assume ha e e y
𝜙l,j(x)
can be ep esen ed by ( enso p oduc )
basis unc ions
Bl,j,o,o=1, …,O
, aking one o se e al columns o
x
as inpu and
mapping hese on o he space spanned by he basis unc ions. Deno ing
zl,j
∶= B
l,j
(x)=(B
l,j,1
(x),…,B
l,j,O
(x))
⊤
∈ℝ
O
, he nonlinea smoo h e ms can be
w i en as
𝜙
l,j(x)=z
⊤
l,j
𝜸l,
j
whe e
𝜸l,j
∈ℝ
O
a e he co esponding basis coe icien s o
zl,j
.
In p inciple, such a lexible speci ica ion may also be used o
𝜂K+1,…,𝜂K+M
, i.e.,
o he addi i e s uc u ed p edic o s ela ed o
𝝅
. While his is echnically possible
(using a di e en subne wo k o e e y
𝜋m
), hese p edic o s a e usually assumed o
sha e one and he same addi i e s uc u e, i.e.,
x
L
(j)
and
zl,j
wi h
l∈S(j)
a e he same
o all j ela ed o he M mix u e weigh s. This u he enables a s aigh o wa d
in e p e a ion o he p edic o –mix u e weigh ela ionship, as sha ing one addi-
i e p edic o ac oss all
𝜋m
s esembles a mul inomial logis ic eg ession model. We
will hus assume he same speci ica ion o all hese p edic o s. We summa ize all
(2)
𝜋
m(xi)=
exp(𝜂
K+m
(x
i
))
∑
M
l=1
exp(𝜂
K+l
(x
i
))
, o m∈M
.
(3)
𝜂
j=𝛽0,j+x⊤
L(j)𝜷j+
∑
l∈
S
(j)
𝜙l,j(x)
,
357
1 3
Mix u e o expe s dis ibu ional eg ession: implemen a ion…
model coe icien s o he linea e ms by
𝜷
=(𝛽
0,1
,𝜷
⊤
1
,𝛽
0,2
,𝜷
⊤
2
,…,𝛽
0,K+M
,𝜷
⊤
K+M
)
⊤
and all model coe icien s o ep esen ing he nonlinea smoo h unc ions by
𝜸
=(𝛾
0,1
,𝜸
⊤
1
,𝛾
0,2
,𝜸
⊤
2
,…,𝛾
0,K
+
M
,𝜸
⊤
K+M
)
⊤
. In he ollowing, we summa ize hese wo
pa ame e ec o s as
𝝍=(𝜷⊤
,𝜸⊤)⊤
.
2.3 Model log likelihood
Based on model (1) and he s uc u es imposed on he p edic o s in (3), he
nega i e log likelihood o he model pa ame e s
𝝍
o n independen obse a-
ions
y
=(
y1,
…
,yn
)
⊤
∈ℝ
n
and hei co esponding n obse ed ea u e ec o s
x
∶= (x
⊤
1
,…,x
⊤
n
)
⊤
∈ℝ
n×p
is gi en by he sum o he nega i e log-likelihood con i-
bu ions
𝓁i(𝝍)
o each obse a ion
i=1, …,n
:
The nega i e log likelihood can be ew i en as he nega i e sum o he exponen i-
a ed log likelihoods o bo h he mix u e weigh s
𝜋m
and he componen densi ies
m
using he log-sum-exp (LSE) unc ion:
In p ac ice, o mula ion (5) is o en p e e ed o e (4) as i is less a ec ed by unde -
o o e low p oblems.
2.4 Iden i iabili y
Iden i iabili y is o conce n o he mix u e o expe s dis ibu ional eg ession
model because o he iden i iabili y p oblems which could occu due o he mix u e
speci ica ion, due o he addi i e s uc u ed p edic o s and due o he so max map-
ping o he mix u e weigh s.
2.4.1 Mix u e models
Fo any mix u e model, i ial iden i iabili y p oblems may a ise due o label swi ch-
ing and o e i ing wi h ei he emp y o duplica ed componen s. In addi ion, also
gene ic iden i iabili y p oblems may occu o mix u es o dis ibu ions bu also o
mix u es o eg essions. A de ailed o e iew o iden i iabili y p oblems in he ini e
mix u e case is gi en in F ühwi h-Schna e (2006).
(4)
𝓁
(𝝍) ∶=
n
∑
i=1
𝓁i(𝝍)=−
n
∑
i=1
log
{
M
∑
m=1
𝜋m(xi) m(yi∣𝜽m(xi))
}.
(5)
−
n
∑
i=1
log
{
M
∑
m=1
exp
[
log 𝜋m(xi)+log m(yi∣𝜽m(xi))
]}.
358
D.Rügame e al.
1 3
2.4.2 Addi i e s uc u ed p edic o s
The componen s o he addi i e model wi h p edic o s uc u es o he o m (3) a e in
gene al only iden i iable up o a cons an and also equi e u he es ic ions i bo h
linea and nonlinea smoo h e ec s a e de ined o one and he same co a ia e. Fo
example,
𝜂j=𝛽0,j+𝜙1,j(x1)
can be equally ep esen ed by

𝛽0,j
+

𝜙
1,j
(x
1)
by de ining

𝛽0,j
=𝛽
0,j
+
c
and

𝜙1,j
(x
1
)=𝜙
1,j
(x
1
)−
c
wi h
c∈ℝ
, i.e., by adding and sub ac ing a
cons an c in bo h e ms. We ensu e he iden i iabili y o nonlinea smoo h e ms
𝜙l,j
by using sum- o-ze o cons ain s o all nonlinea addi i e unc ions such as splines
o enso p oduc splines. The iden i iabili y o linea e ec s
x𝛽
in he p esence o a
uni a ia e nonlinea e ec
𝜙(x)
mus also be ensu ed. In his case, se e al di e en
op ions exis (see, e.g., Rügame e al. 2023b). The mos s aigh o wa d way is o
use a nonlinea e ec
𝜙(x)
wi h a basis ep esen a ion ha includes he linea e ec
as null space (and hence, o enough penaliza ion as discussed in Sec .3.3, esul s
in a linea e ec ). In his case,
L(j)
only consis s o a iables ha a e only modeled
using a linea componen (e.g., ca ego ical e ec s) and
L(j)∩S(j)=�
.
3 Model ep esen a ion and obus es ima ion
The obse ed da a log likelihood is, in gene al, no conca e and hus di icul o op i-
mize. We sugges o use op imize s om he ield o deep lea ning by aming ou
model as a neu al ne wo k. This enables he i ing o he ull model class o mix u e
o expe s dis ibu ional eg ession models wi h addi i e s uc u e p edic o s in a
s aigh o wa d way. Nume ical expe imen s con i m ha his choice—when p op-
e ly ained—is no only mo e lexible and obus han EM-based op imiza ion bu
also makes la ge da ase applica ions easible due o mini-ba ch aining.
3.1 Neu al ne wo k ep esen a ion
Models desc ibed in (1) and (3) can be ep esen ed as neu al ne wo ks in he ol-
lowing way. An exempla y a chi ec u e is depic ed in Fig.1. The ne wo k a chi ec-
u e implemen ing model (1) de ines a mos K subne wo ks, whe e each subne wo k
models one o mo e addi i e s uc u ed p edic o s
𝜂j
o a dis ibu ion pa ame e
𝜃j
.
Using an app op ia e pa ame e - ee ans o ma ion unc ion, hese addi i e s uc-
u ed p edic o s a e mapped o he dis ibu ion pa ame e s and passed o a dis ibu-
ion laye (Dillon e al. 2017). Each dis ibu ion laye co esponds o a mix u e com-
ponen
m
ha is u he passed o a mul inomial o ca ego ical dis ibu ion laye ,
modeling he mix u e o all de ined dis ibu ions. The mix u e weigh s can ei he be
di ec ly es ima ed o also lea ned on he basis o inpu ea u es using addi i e s uc-
u ed p edic o s in ano he subne wo k. A classical linea mix u e eg ession com-
bining M linea eg essions would, e.g., be gi en by M subne wo ks, each lea ning
he expec a ion o a no mal dis ibu ion and a mix u e subne wo k ha only akes
a cons an inpu (a bias) and lea ns he M mix u e weigh s. The indi idual addi i e
p edic o s o each subne wo k a e, in mos cases, ully connec ed laye s wi h only
365
1 3
Mix u e o expe s dis ibu ional eg ession: implemen a ion…
eplica ions o he wo di e en mix u e weigh s and he wo numbe o noise a i-
ables se ings. Resul s sugges ha ou app oach leads o be e p edic ions measu ed
by he PLS, bu is in e io in e ms o LS. The smalle LS alues a e possibly due
o ewe da a poin s o i he model because o he need o a alida ion se and due
o he sh inkage induced by ea ly s opping he p ocedu e. The median pe o mance
o he clus e ing induced based on he es ima ed pos e io p obabili ies is in gene al
on pa wi h he EM-based app oach in e ms o ARI and ACC wi h lexmix, while
showing e en sligh ly be e pe o mance in he Gaussian case.
4.5 Misspeci ied mix u es andspa si y
In his simula ion, we use a no mal mix u e wi h
pm
=10
ixed linea p edic o s
o each dis ibu ion and dis ibu ion pa ame e (mean and a iance), whe e all
ea u es a e again d awn om a s anda d no mal dis ibu ion and eg ession coe -
icien s om a uni o m
U(−2, 2)
-dis ibu ion. The da a a e hen gene a ed using
M=2
ac ual mix u e componen s wi h
𝜋1
d awn (independen ly o ea u es) om
a uni o m dis ibu ion on he in e al (0.06, 0.094) and
𝜋2=1−𝜋1
o ensu e ha
he minimum alue o bo h p obabili ies is a leas 6%. We hen e alua e he es i-
ma ion o mix u e p obabili ies by NMDR o
n∈{300, 2500}
when inc easing
he numbe o speci ied dis ibu ions
M†∈{3, 5, 10}
. To allow o spa si y in
𝝅
,
we use he objec i e unc ion
𝓁en
in oduced in Sec .3.3. Fo each scena io, 10
eplica ions a e pe o med.
ACC ARI
PLS LS
no mal (2) no mal (4) poisson (2)poisson (4) no mal (2) no mal (4) poisson (2)poisson (4)
no mal (2) no mal (4) poisson (2)poisson (4) no mal (2) no mal (4) poisson (2)poisson (4)
−10
−8
−6
−4
0.00
0.25
0.50
0.75
1.00
−8
−6
−4
−2
0.5
0.7
0.9
Value
Me hod O acle EM NMDR
Fig. 3 Compa ison o a e age PLS and LS as well as ACC and ARI o a s a e-o - he-a implemen a ion
(EM), an o acle a ying coe icien model wi h known class membe ships and ou app oach (NMDR) in
di e en colo s o he wo dis ibu ions and wo scales (x-axis). The boxplo s con ain he esul s o 10
eplica ions o e wo se ings o he mix u e weigh s and wo se ings o he numbe o noise a iables

366
D.Rügame e al.
1 3
4.5.1 Resul s
Resul s o a ious se ings o he en opy penal y pa ame e
𝜉
a e depic ed in
Fig.4. While se ing
𝜉
o small alues la ge han ze o can imp o e he p edic-
i e pe o mance and e en ou pe o m he co ec ly speci ied model wi hou
1 3 8
PLS RMSE Coe . RMSE P ob.
300/10
2500/10
300/10
2500/10
300/10
2500/10
−50
−40
−30
−20
−10
0
2
4
6
0.0
0.1
0.2
0.3
n / p
Value
En opy Penal y
0
1e−05
0.001
0.01
0.1
0.3
co ec model
Fig. 4 Model quali y o misspeci ied models wi h speci ied mix u es
M†∈{3, 5, 10}
(columns, coun -
ing he addi ional componen s) ins ead o ac ual
M=2
mix u es and di e en goodness o i measu es
( ows) o 10 eplica ions (boxes). Colo s co espond o di e en se ings o
𝜉
o ep esen es ima ion
esul s o he co ec model (black)
T ue P ob.:0.3923
T ue P ob.:0.6077
0.00
0.25
0.50
0.75
1.00
1e−051e−03 1e−01
En opy Penal y
Es . P ob.
Fig. 5 Coe icien pa h (es ima ed mix u e p obabili ies) o di e en en opy penal ies. A alue o
a ound 1e-02 yields he bes ade-o be ween spa si y and es ima ion pe o mance
367
1 3
Mix u e o expe s dis ibu ional eg ession: implemen a ion…
addi ional dis ibu ion componen s, he bias induced by he penal y gene ally
dec eases he es ima ion pe o mance. In p ac ice, an app op ia e amoun o
penaliza ion can be ound by unning c oss- alida ion along a g id o di e en
𝜉
alues as, e.g., done o he Lasso (Tibshi ani 1996).
We addi ionally in es iga e he coe icien pa h ob ained om a ying al-
ues o he penal y pa ame e
𝜉
o one simula ed example. Resul s o he
n=2500, M†=5
se ing a e depic ed in Fig.5.
𝜉
is a ied be ween 0 and 1 on
a loga i hmic scale. The ue model has wo nonze o p obabili ies 0.6077 and
0.3923 while he o he 3 en ies in
𝝅
a e 0.
5 Cell cycle‑ egula ed genes o yeas
In o de o demons a e he lexibili y o ou app oach, we in es iga e i s applica ion
o he yeas cell cycle da ase om Spellman e al. (1998). In his s udy, genome-
wide mRNA le els we e measu ed o 6178 yeas open eading ames (ORFs) o
119min a 7-min in e als. We he e analyze he subse o da a whe e all 18 ime
poin s o he alpha ac o a es a e a ailable. The esul ing longi udinal da ase
consis s o 80,802 obse a ions o he s anda dized exp ession le els. A subse o
his da ase was also analyzed using mix u e models in G ün e al. (2011).
5.1 Dis ibu ional mix u e eg ession
As bo h he mean and s anda d de ia ion o he s anda dized exp ession le els o
genes change o e ime, we apply a mix u e o dis ibu ional eg essions model
whe e he mean
𝜇
and he s anda d de ia ion
𝜎
o he no mally dis ibu ed mix u e
componen s depend on ime, i.e.,
Y
∼
∑M
m=1
𝜋
m
N(𝜇
m,
,𝜎2
m, )
o
∈[0, 119]
. The
addi i e s uc u ed p edic o s o hese dis ibu ion pa ame e s a e de ined as
whe e he nonlinea smoo h unc ions
𝜙
a e modeled by hin-pla e eg ession splines
(Wood 2003). P e ious app oaches o modeling his da ase in es iga ed he use o
a mix u e o mixed models, i.e., he inclusion o gene-speci ic andom e ec s (Luan
and Li 2003; G ün e al. 2011). We in es iga e he e an al e na i e op ion o mod-
eling his addi ional he e ogenei y by allowing o ime- a ying s anda d de ia ions.
We use
M=6
which co esponds o he numbe o mix u e componen s iden i ied
by Spellman e al. (1998).
5.2 Resul s
In o de o plo he es ima ed smoo h e ec s oge he wi h he ue obse a ions,
we i s de i e he componen membe ship o e e y gene. As done in he E-s ep o
mix u e model app oaches (see, e.g., G ün e al. 2011), we calcula e he a pos e io i
h−1
(𝜇
m,
)=𝛽
0,m,1
+𝜙
m,1
( )and h
−1
(𝜎
m,
)=𝛽
0,m,2
+𝜙
m,2
( )
,
368
D.Rügame e al.
1 3
p obabili y o e e y gene o belong o componen m and hen ake he maximum o
all componen s
1, …,M=6
. Fo his applica ion, obse a ions we e only assigned
o 5 o he 6 assumed componen s. No e ha due o he na u e o ou op imiza ion
ou ine, no all componen s necessa ily con ain a leas one obse a ion.
Compa ing he numbe o genes assigned o each clus e , one sees ha he mos
common componen in ou esul s is clus e 6 wi h 39,078 obse a ions. Clus e
4 con ains 25,722 obse a ions, clus e 1 9306 and clus e 5 6552 obse a ions.
Leas numbe o obse a ions is assigned o clus e 2 which con ains only 144
obse a ions.
Figu e6 isualizes he esul s ob ained in a panel plo whe e in each panel he
ajec o ies o he ORFs o all genes assigned o his clus e a e shown oge he
wi h he componen -speci ic es ima es o he ime- a ying means and s and-
a d de ia ions. The iden i ied clus e s clea ly a y in showing ei he an ini ial
dec ease o inc ease in hei means. In addi ion, one also sees ha he s anda d
de ia ions o he clus e s a y o e ime wi h some clus e s exhibi ing a pa icu-
la ly la ge amoun o he e ogenei y a la e ime poin s.
6 Ou look
We ha e in oduced he class o mix u es o expe s dis ibu ional eg ession wi h
addi i e s uc u al p edic o s and in es iga ed i s embedding in o neu al ne wo ks
o obus model es ima ion. O e all his leads o he neu al mix u e o expe s
dis ibu ional eg ession (NMDR) app oach. We show ha popula i s -o de
adap i e upda e ou ines a e well sui ed o lea ning hese mix u e o expe s
(dis ibu ional) eg ession models and also highligh ha he embedding in o a
neu al ne wo k es ima ion amewo k allows o s aigh o wa d ex ensions o he
56
124
0255075 100 0255075100
0255075 100
−4
0
4
−4
0
4
Time
S anda dized exp ession le el
Fig. 6 T ajec o ies o ORFs (y-axis) o all 4489 genes (black lines) o e he cou se o he 18 ime poin s
(x-axis) wi h he es ima ed mean end ( ed solid line) pe componen ( ace s) and unce ain y isualized
by wo imes he es ima ed ( ime- a ying) s anda d de ia ion (shaded ed a ea)
369
1 3
Mix u e o expe s dis ibu ional eg ession: implemen a ion…
gene al mix u e model class and ( egula ized) maximum-likelihood es ima ion
using op imiza ion ou ines sui able also o big da a applica ions due o mini-
ba ch aining. Using he p oposed a chi ec u e o mix u e o expe s dis ibu-
ional eg ession, a possible ex ension o ou app oach is he e o e he combina-
ion wi h o he (deep) neu al ne wo ks. This allows lea ning bo h he dis ibu ion
componen s and he mix u e weigh s ei he by (a) a s uc u ed model, such as a
linea o addi i e model, (b) a cus om (deep) neu al ne wo k o (c) a combina ion
he eo . A simila app oach has been in es iga ed by F i z e al. (2022) using a
ze o-in la ed Poisson model (i.e., a mix u e including a poin mass dis ibu ion)
which includes bo h addi i e e ec s and a g aph neu al ne wo k in he addi i e
p edic o .
Appendix A: Op imiza ion ou ines
In o de o p o ide insigh s in o he a ious op imize s’ pe o mance, we conduc
a small benchma k s udy o assess he in luence o he choice o an op imize and
o ind a good de aul . Figu e7 isualizes he compa ison based on he anks o
each op imize ac oss all 48 se ings om Sec .4.3. In all ou expe imen s, we
use he Glo o ini ialize o ini ialize weigh s (Glo o and Bengio 2010) (ini ial-
ized di e en ly o e e y op imize and es a ) a ba ch size o 50 and a maximum
o 1500 epochs.
Resul s
Figu e 7 indica es ha con e gence p oblems a e p ima ily encoun e ed o SGD
wi h a subs an ial amoun o uns di e ging du ing op imiza ion. An o e all pe o -
mance assessmen based on anks indica es ha he RMSp op op imize achie es he
lowes o e all ank. Howe e , he igu e highligh s ha in ac , no clea bes op i-
mize eme ges ac oss all scena ios. The anks ob ained o he di e en op imize s
also a y conside ably wi h he pe o mance c i e ion.
O e all one migh conclude ha bo h he RMSp op op imize and he Adam op i-
mize pe o m in gene al well, also when used wi h hei de aul se ings. We no e,
howe e , ha uning he op imize and i s lea ning a e would in gene al also be
bene icial in e ms o bo h p edic i e pe o mance and es ima ion quali y. A u he
speed-up o some o hese ou ines can be achie ed by addi ionally inco po a ing
momen um, which also p o ed o be e ec i e in he op imiza ion o addi i e models
using boos ing (Schalk e al. 2022).
Appendix B: Concomi an a iables
Finally, we compa e ou app oach wi h lexmix in e ms o es ima ion quali y when
he model includes concomi an a iables.
370
D.Rügame e al.
1 3
Da a gene a ing p ocess
We simula e a mix u e eg ession model wi h
M∈{2, 3}
mix u e componen s, each
ollowing a no mal dis ibu ion wi h ixed a iance,
𝜎2=1
, and mean de ined by
p=10
co a ia es ha a e d awn om a s anda d no mal dis ibu ion and mul iplied
by ixed coe icien also gene a ed om a s anda d no mal dis ibu ion. In con as
o p e ious simula ions, his nume ical expe imen is based on a da a gene a ing
p ocess ha uses addi ional 2, 4 o 8 co a ia es o de ine he addi i e p edic o s
RMSE Coe .RMSE P ob. PLSARI P ob. ACC P ob.
Adadel a (0.01)
Adadel a (0.01) w/ CLR
Adadel a (0.1)
Adadel a (0.1) w/ CLR
Adadel a (1)
Adam (0.01)
Adam (0.01) w/ CLR
Adam (0.1)
Adam (0.1) w/ CLR
Range (0.01)
Range (0.01) w/ CLR
Range (0.1)
Range (0.1) w/ CLR
RMSp op (0.01)
RMSp op (0.01) w/ CL
R
RMSp op (0.1)
RMSp op (0.1) w/ CLR
SGD (0.01)
SGD (0.01) w/ CLR
SGD (0.1)
SGD (0.1) w/ CLR
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
Pe cen age
Rank 1234 o highe
Fig. 7 Compa ison o a ious op imize s (x-axis; wi h lea ning a e in b acke s) anked by pe o mance
o di e en me ics (di e en ows), po en ially wi h addi ional cyclic lea ning a e schedule (CLR),
ac oss simula ion se ings s udied in Sec .4.3. Ranks a e compu ed on pe o mance a e aged o e 3 ep-
e i ions. Ba s om SGD uns do no sum up o 100% as some models di e ged du ing op imiza ion

371
1 3
Mix u e o expe s dis ibu ional eg ession: implemen a ion…
𝜂K+1,…,𝜂K+M
. In o he wo ds, we do no se he mix u e p obabili ies o a ixed
cons an bu make hem co a ia e-dependen . The p edic o s
𝜂K+1,…,𝜂K+M
a e gen-
e a ed independen ly om p edic o s
𝜂1,…,𝜂K
by d awing co a ia es and e ec s
om a s anda d no mal dis ibu ion and de ining hei e ec o be linea .
Expe imen
We se
n=5000
and un he expe imen 10 imes o in es iga e he es ima ion
pe o mance o he wo me hods o all eg ession coe icien s using he RMSE.
mixdis eg is op imized using Adam wi h a lea ning a e 1e-3, ba ch size o 32,
ea ly s opping on a 10% alida ion da a se and a pa ience o 250 epochs.
Resul s
Figu e8 shows he esul ing RMSE alues o di e en alues o M and he wo
addi i e p edic o ypes (means and p obabili ies). Bo h app oaches pe o m well
wi h RMSE alues in he ange 0.01 o 0.02, which is compa ably small compa ed
o he ue coe icien alues which ange om
−2
o 1.5. As in p e ious s udies, he
es ima ion pe o mance o he EM-based app oach is sligh ly be e han he one
using SGD.
2 mix u e componen s 3 mix u e componen s
Mean p edic o P obabili y p edic o
248248
0.02
0.03
0.04
0.01
0.02
0.03
Numbe o co a ia es (in mix u e p obabili y p edic o )
RMSE
lexmix
mixdis eg
Fig. 8 Compa ison o lexmix and mixdis eg es ima ion pe o mance using he RMSE ( isualized
by boxplo s summa izing he 10 di e en uns) o coe icien s in he mean p edic o s o dis ibu ion
membe s ( op ow) as well as addi i e p edic o o mix u e p obabili ies (bo om ow) o a mix u e o
M∈{2, 3}
dis ibu ions (columns)
372
D.Rügame e al.
1 3
Funding Open Access unding enabled and o ganized by P ojek DEAL.
Decla a ions
Compe ing inancial in e es s The au ho s decla e ha hey ha e no known compe ing inancial in e es s
o pe sonal ela ionships ha could ha e appea ed o in luence he wo k epo ed in his pape .
Open Access This a icle is licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License,
which pe mi s use, sha ing, adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as long
as you gi e app op ia e c edi o he o iginal au ho (s) and he sou ce, p o ide a link o he C ea i e
Commons licence, and indica e i changes we e made. The images o o he hi d pa y ma e ial in his
a icle a e included in he a icle’s C ea i e Commons licence, unless indica ed o he wise in a c edi line
o he ma e ial. I ma e ial is no included in he a icle’s C ea i e Commons licence and you in ended
use is no pe mi ed by s a u o y egula ion o exceeds he pe mi ed use, you will need o ob ain pe mis-
sion di ec ly om he copy igh holde . To iew a copy o his licence, isi h p://c ea i ecommons.o g/
licenses/by/4.0/.
Re e ences
Abadi, M., Aga wal, A., Ba ham, P., e al.: Tenso Flow: la ge-scale machine lea ning on he e ogeneous
sys ems (2015). h ps:// www. enso low. o g/
Bishop, C.M.: Mix u e densi y ne wo ks. As on Uni e si y (1994)
DeSa bo, W.S., C on, W.L.: A maximum likelihood me hodology o clus e wise linea eg ession. J.
Classi . 5(2), 249–282 (1988)
Diebol , J., Robe , C.P.: Es ima ion o ini e mix u e dis ibu ions h ough Bayesian sampling. J. R. S a .
Soc.: Se . B (Me hodol.) 56(2), 363–375 (1994)
Dillon, J.V., Langmo e, I., T an, D., e  al.: Tenso Flow dis ibu ions. (2017). h ps:// doi. o g/ 10. 48550/
a Xi . 1711. 10604
F i z, C., Do iga i, E., Rügame , D.: Combining g aph neu al ne wo ks and spa io- empo al disease mod-
els o imp o e he p edic ion o weekly co id-19 cases in Ge many. Sci. Rep. 12(1), 1–18 (2022)
F ühwi h-Schna e , S.: Fini e Mix u e and Ma ko Swi ching Models. Sp inge , Be lin (2006)
Gel and, A.E., Dey, D.K.: Bayesian model choice: asymp o ics and exac calcula ions. J. R. S a . Soc.:
Se . B (Me hodol.) 56(3), 501–514 (1994)
Glo o , X., Bengio, Y.: Unde s anding he di icul y o aining deep eed o wa d neu al ne wo ks. In:
P oceedings o he Thi een h In e na ional Con e ence on A i icial In elligence and S a is ics, P o-
ceedings o Machine Lea ning Resea ch, ol.9, pp. 249–256. PMLR (2010)
Go mley, I.C., F ühwi h-Schna e , S.: Mix u e o expe s models. In: F ühwi h-Schna e , S., Celeux,
G., Robe , C.P. (eds.) Handbook o Mix u e Analysis, pp. 271–307. Chapman & Hall/CRC Hand-
books o Mode n S a is ical Me hods, Chapman and Hall/CRC (2019)
G ün, B., Leisch, F.: Fi ing ini e mix u es o gene alized linea eg essions in R. Compu . S a . Da a
Anal. 51(11), 5247–5252 (2007)
G ün, B., Scha l, T., Leisch, F.: Modelling ime cou se gene exp ession da a wi h ini e mix u es o linea
addi i e models. Bioin o ma ics 28(2), 222–228 (2011)
Hin on, G., S i as a a, N., Swe sky, K.: Neu al ne wo ks o machine lea ning. Cou se a, ideo lec u es
264(1) (2012)
Hube , L., A abie, P.: Compa ing pa i ions. J. Classi . 2(1), 193–218 (1985)
Kingma, D.P., Ba, J.: Adam: A me hod o s ochas ic op imiza ion (2014). h ps:// doi. o g/ 10. 48550/ a Xi .
1412. 6980
Kneib, T.: Beyond mean eg ession. S a . Model. 13(4), 275–303 (2013)
Leisch, F.: FlexMix: A gene al amewo k o ini e mix u e models and la en class eg ession in R. J.
S a . So w. 11(8), 1–18 (2004). h ps:// doi. o g/ 10. 18637/ jss. 011. i08
Luan, Y., Li, H.: Clus e ing o ime-cou se gene exp ession da a using a mixed-e ec s model wi h
B-splines. Bioin o ma ics 19(4), 474–482 (2003)
Magdon-Ismail, M., A iya, A.: Neu al ne wo ks o densi y es ima ion. In: Ad ances in Neu al In o ma-
ion P ocessing Sys ems (1998)
373
1 3
Mix u e o expe s dis ibu ional eg ession: implemen a ion…
McLachlan, G.J., Peel, D.: Fini e Mix u e Models. John Wiley & Sons, London (2004)
McLachlan, G.J., Lee, S.X., Ra hnayake, S.I.: Fini e mix u e models. Annu. Re . S a . Appl. 6(1), 355–
378 (2019)
Quand , R.E.: The es ima ion o he pa ame e s o a linea eg ession sys em obeying wo sepa a e
egimes. J. Am. S a . Assoc. 53(284), 873–880 (1958)
R Co e Team: R: A Language and En i onmen o S a is ical Compu ing. R Founda ion o S a is ical
Compu ing, Vienna, Aus ia (2022). h ps:// www.R- p oje c . o g/
Rigby, R.A., S asinopoulos, M.D.: GAMLSS: a dis ibu ional eg ession app oach. J. R. S a . Soc.: Se . C
(Appl. S a .) 54(3), 507–554 (2005)
Rügame , D., Kolb, C., F i z, C., e al.: deep eg ession: A lexible neu al ne wo k amewo k o semi-
s uc u ed deep dis ibu ional eg ession. J. S a . So w. 105(1), 1–31 (2023a)
Rügame , D., Kolb, C., Klein, N.: Semi-s uc u ed dis ibu ional eg ession. Am. S a . 1–12 (2023b)
Schalk, D., Bischl, B., Rügame , D.: Accele a ed componen wise g adien boos ing using e icien da a
ep esen a ion and momen um-based op imiza ion. J. Compu . G aph. S a . 1–11 (2022)
Spä h, H.: Algo i hm39 clus e wise linea eg ession. Compu ing 22(4), 367–373 (1979)
Spellman, P.T., She lock, G., Zhang, M.Q., e al.: Comp ehensi e iden i ica ion o cell cycle- egula ed
genes o he yeas Saccha omyces ce e isiae by mic oa ay hyb idiza ion. Mol. Biol. Cell 9(12),
3273–3297 (1998)
S asinopoulos, D.M., Rigby, R.A.: Gene alized addi i e models o loca ion scale and shape (GAMLSS)
in R. J. S a . So w. 23(7), 1–46 (2007)
S asinopoulos, M., Rigby, B.: gamlss.mx: Fi ing Mix u e Dis ibu ions wi h GAMLSS. R package e -
sion 4.3-5 (2016)
S asinopoulos, M.D., Rigby, R.A., Bas iani, F.D.: GAMLSS: a dis ibu ional eg ession app oach. S a .
Model. 18(3–4), 248–273 (2018)
Tibshi ani, R.: Reg ession sh inkage and selec ion ia he lasso. J. R. S a . Soc.: Se . B (Me hodol.)
58(1), 267–288 (1996)
Ushey, K., Allai e, J., Tang, Y.: e icula e: In e ace o ‘Py hon’. h ps:// CRAN.R- p oje c . o g/ packa ge=
e ic ula e, package e sion 1.26 (2022)
Vanden Oo d, A., Sch auwen, B.: Fac o ing a ia ions in na u al images wi h deep Gaussian mix u e
models. In: Ad ances in Neu al In o ma ion P ocessing Sys ems, pp. 3518–3526 (2014)
Van Rossum, G., D ake J , F.L.: Py hon e e ence manual. Cen um oo Wiskunde en In o ma ica
Ams e dam (1995)
Vi oli, C., McLachlan, G.J.: Deep Gaussian mix u e models. S a . Compu . 29(1), 43–51 (2019)
Wedel, M., DeSa bo, W.S.: A mix u e likelihood app oach o gene alized linea models. J. Classi .
12(1), 21–55 (1995)
Wood, S.N.: Thin pla e eg ession splines. J. R. S a . Soc. Se . B (S a . Me hodol.) 65(1), 95–114 (2003)
Wood, S.N.: Gene alized Addi i e Models: An In oduc ion wi h R. CRC P ess (2017)
Wood, S.N., Fasiolo, M.: A gene alized Fellne –Schall me hod o smoo hing pa ame e op imiza ion
wi h applica ion o Tweedie loca ion, scale and shape models. Biome ics 73(4), 1071–1081 (2017).
h ps:// doi. o g/ 10. 1111/ biom. 12666
W igh , L.: New deep lea ning op imize , ange : Syne gis ic combina ion o adam lookahead o he
bes o ... Medium (2019)
Zeile , M.D.: ADADELTA: An adap i e lea ning a e me hod (2012). h ps:// doi. o g/ 10. 48550/ a Xi .
1212. 5701
Zong, B., Song, Q., Min, M.R., e  al.: Deep au oencoding Gaussian mix u e model o unsupe ised
anomaly de ec ion. In: In e na ional Con e ence on Lea ning Rep esen a ions (2018)
Publishe ’s No e Sp inge Na u e emains neu al wi h ega d o ju isdic ional claims in published maps
and ins i u ional a ilia ions.