Bernstein flows for flexible posteriors in variational Bayes

Dü , Oli e ; Hö ling, S e an; Dold, Danil; Ko ylo , I onne; Sick, Bea e
A icle — Published Ve sion
Be ns ein lows o lexible pos e io s in a ia ional Bayes
AS A Ad ances in S a is ical Analysis
P o ided in Coope a ion wi h:
Sp inge Na u e
Sugges ed Ci a ion: Dü , Oli e ; Hö ling, S e an; Dold, Danil; Ko ylo , I onne; Sick, Bea e (2024) :
Be ns ein lows o lexible pos e io s in a ia ional Bayes, AS A Ad ances in S a is ical Analysis,
ISSN 1863-818X, Sp inge , Be lin, Heidelbe g, Vol. 108, Iss. 2, pp. 375-394,
h ps://doi.o g/10.1007/s10182-024-00497-z
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/315187
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h p://c ea i ecommons.o g/licenses/by/4.0/
Vol.:(0123456789)
AS A Ad ances in S a is ical Analysis (2024) 108:375–394
h ps://doi.o g/10.1007/s10182-024-00497-z
1 3
ORIGINAL PAPER
Be ns ein lows o  lexible pos e io s in a ia ional Bayes
Oli e Dü 1 · S e anHö ling1· DanilDold1· I onneKo ylo 1· Bea eSick2,3
Recei ed: 18 No embe 2022 / Accep ed: 6 Feb ua y 2024 / Published online: 3 Ap il 2024
© The Au ho (s) 2024
Abs ac
Black-box a ia ional in e ence (BBVI) is a echnique o app oxima e he pos e io
o Bayesian models by op imiza ion. Simila o MCMC, he use only needs o
speci y he model; hen, he in e ence p ocedu e is done au oma ically. In con as
o MCMC, BBVI scales o many obse a ions, is as e o some applica ions, and
can ake ad an age o highly op imized deep lea ning amewo ks since i can be
o mula ed as a minimiza ion ask. In he case o complex pos e io s, howe e ,
o he s a e-o - he-a BBVI app oaches o en yield unsa is ac o y pos e io
app oxima ions. This pape p esen s Be ns ein low a ia ional in e ence (BF-
VI), a obus and easy- o-use me hod lexible enough o app oxima e complex
mul i a ia e pos e io s. BF-VI combines ideas om no malizing lows and Be ns ein
polynomial-based ans o ma ion models. In benchma k expe imen s, we compa e
BF-VI solu ions wi h exac pos e io s, MCMC solu ions, and s a e-o - he-a BBVI
me hods, including no malizing low-based BBVI. We show o low-dimensional
models ha BF-VI accu a ely app oxima es he ue pos e io ; in highe -dimensional
models, BF-VI compa es a o ably agains o he BBVI me hods. Fu he , using
BF-VI, we de elop a Bayesian model o he semi-s uc u ed melanoma challenge
da a, combining a CNN model pa o image da a wi h an in e p e able model
pa o abula da a, and demons a e, o he i s ime, he use o BBVI in semi-
s uc u ed models.
Keywo ds Va ia ional in e ence· Deep lea ning· T ans o ma ion models· Bayesian
neu al ne wo k
1 In oduc ion
Unce ain y quan i ica ion is essen ial, especially i model p edic ions a e used
o suppo high-s akes decision-making. Quan i ying unce ain y in s a is ical o
machine lea ning models is o en achie ed by Bayesian app oaches, whe e pos-
e io dis ibu ions ep esen he unce ain y o he es ima ed model pa ame e s.
Oli e Dü and Bea e Sick ha e con ibu ed equally o his wo k.
Ex ended au ho in o ma ion a ailable on he las page o he a icle
376
O.Dü e al.
1 3
De e mining he exac pos e io dis ibu ions is o en impossible when he pos e-
io akes a complex shape and he model has many pa ame e s. This is especially
ue o complex models such as Bayesian neu al ne wo ks (NNs) o semi-s uc-
u ed models ha combine an in e p e able model pa wi h deep NNs. Va ia ional
in e ence (VI) is a commonly used app oach o app oxima e complex dis ibu ions
h ough op imiza ion (Jo dan e al. 1999; Blei e al. 2017). In VI, he complex pos e-
io is app oxima ed by a a ia ional dis ibu ion by minimizing a di e gence meas-
u e be ween he a ia ional and he ue pos e io dis ibu ion. VI is cu en ly a e y
ac i e esea ch ield ackling di e en challenges, which can be ca ego ized in o he
ollowing g oups: (1) cons uc ing a ia ional dis ibu ions ha a e lexible enough
o ma ch he ue pos e io dis ibu ion, (2) de ining op imal a ia ional objec i e
o uning he a ia ional dis ibu ion, which boils down o inding he mos sui ed
di e gence measu e quan i ying he di e ence be ween a a ia ional dis ibu ion
and pos e io , and (3) de eloping obus and accu a e s ochas ic op imiza ion ame-
wo ks o he a ia ional objec i e (Dhaka e al. 2020; Blei e al. 2016; Welandawe
e al. 2022). He e, we ocus on challenge (1) and p opose a me hod o cons uc a
a ia ional dis ibu ion ha is lexible enough o accu a ely and obus ly app oxi-
ma e complex mul idimensional pos e io s.
To a oid model-speci ic calcula ions, we design ou me hod as a black-
box VI (BBVI) app oach (Rangana h e  al. 2014). In BBVI, he app oxima i e
pos e io is de e mined by s ochas ic g adien descen . The use simply de ines
he Bayesian model by speci ying he likelihood and he p io , a e which all
subsequen calcula ions a e ca ied ou au oma ically. Due o i s simplici y, BBVI
is implemen ed in many packages o Bayesian modeling, like S an (Ca pen e e al.
2017) and Py o (Bingham e al. 2019) as an al e na i e o MCMC. Gi en BBVI’s
scalabili y o la ge da ase s and i s widesp ead applicabili y, i has eme ged as he
p e e ed echnique in he ield o machine lea ning (Welandawe e al. 2022).
Ou app oach uses ans o ma ion models (TMs) o cons uc complex pos e io s.
T ans o ma ion models (TMs) ha e been in oduced o i ing po en ially complex
ou come dis ibu ions o p obabilis ic eg ession models (Ho ho n e  al. 2014).
Since hen, hey ha e been mainly used o model di e en ou come ypes, such
as o dinal (Kook e  al. 2022; Bu i e  al. 2020), coun (Sieg ied and Ho ho n
2020), con inuous (Lohse e  al. 2017), o ime- o-e en ou comes (Campanella
e al. 2022) based on abula p edic o s. Mo eo e , TMs ha e been used o model
mul idimensional dis ibu ions (Klein e al. 2019). Neu al ne wo ks can be used o
ex end TMs o model ou comes o uns uc u ed p edic o s (e.g., images o ex ) o a
combina ion o abula and uns uc u ed p edic o s (Sick e al. 2021; Baumann e al.
2021; Kook e al. 2022; Rügame e al. 2021).
The basic idea o TMs is o lea n a lexible and mono one ans o ma ion
unc ion ha ans o ms be ween a simple la en dis ibu ion and a po en ially
complex condi ional ou come dis ibu ion. In TMs, he ans o ma ion unc ion is
pa ame e ized as an expansion o basis unc ions. In he case o con inuous a ge
dis ibu ions, mos o en, Be ns ein polynomials (Be nš eın 1912) a e used because
hey can easily be cons ained o be s ic ly mono one, and hei lexibili y can be
uned ia he o de M. A la ge o de M ensu es an accu a e app oxima ion o he
dis ibu ion, which is obus agains a u he inc ease o M (Ho ho n e al. 2018;
377
1 3
Be ns ein lows o  lexible pos e io s in a ia ional Bayes
Ramasinghe e al. 2021); his is also demons a ed in ou expe imen s o he BBVI
se ing.
Independen ly o TMs, no malizing lows (NFs) ha e been de eloped in he deep
lea ning communi y. NFs and TMs ely on he same idea, bu NFs usually cons uc
he ans o ma ion by chaining many simple unc ions, while TMs cons uc one
a he complex ans o ma ion unc ion. In NFs, each simple unc ion, such as
shi ing and scaling, inc emen ally adds o he complexi y o he inal ans o ma ion.
Among he p ominen NF implemen a ions a e RealNVP (Dinh e  al. 2016) and
Masked Au o eg essi e Flow (MAF) (Papamaka ios e al. 2017). RealNVP s ands
ou o i s e icien , in e ible ans o ma ions acili a ed by a specialized neu al
ne wo k a chi ec u e. I s key ad an age lies in he e icien compu a ion o he
Jacobian ma ix’s de e minan , essen ial o di ec densi y es ima ion in he change
o a iable unc ion (see6). This e iciency is achie ed by i e a i ely spli ing he
componen s o he da a in o wo pa s. In each s ep, he i s pa o he componen s
is used o ain a neu al ne wo k compu ing he scale and shi pa ame e s o he
ans o ma ion. This ans o ma ion is hen applied o he o he componen s, while
he i s pa emains unal e ed. This p ocedu e is epea ed mul iple imes wi h
di e en pa i ioning, leading o a iangula Jacobian, hus enabling e icien and
in e ible ans o ma ions. In con as , MAF adop s a undamen ally di e en
app oach o cons uc ans o ma ions (Papamaka ios e  al. 2017). I u ilizes a
sequen ial (au o eg essi e) amewo k, acili a ed by neu al ne wo ks. In MAF
each ou pu componen elies exclusi ely on i s p eceding componen s, a concep
o en e e ed o as causali y in his con ex . This design also leads o a iagonal
Jacobian ma ix and hus a as compu a ion o he change o a iable equa ion. The
MAF ensu es ha he n h ou pu o NN is solely dependen on he i s
n−1
inpu s,
yielding an au o eg essi e model. Howe e , some NF app oaches use a single
lexible ans o ma ion, such as sum-o -squa es polynomials (Jaini e al. 2019) o
splines (Du kan e al. 2019). Recen ly, also Be ns ein-based polynomials ha e been
used o modeling uncondi ional mul i a ia e densi y dis ibu ions (Ramasinghe
e al. 2021).
NFs we e ini ially in oduced o a ia ional in e ence o app oxima e po en ially
complex dis ibu ions o la en a iables in models such as a ia ional au oencode s
(Rezende and Mohamed 2015; Van DenBe g e al. 2018). In he pas , o en membe s
om simple dis ibu ion amilies ha e been used o app oxima e he pos e io
in BBVI. In he "Bayes by Backp op" me hod, Blundell e  al. used independen
Gaussians o app oxima e he pos e io o he weigh s in a Bayesian Neu al
Ne wo k (BNN). They de e mined he pa ame e s o hese Gaussians using BBVI
(Blundell e al. 2015). This app oach was made mo e lexible by using a mul i a ia e
Gaussians (Louizos and Welling 2017) as a ia ional dis ibu ion. While i is clea
ha TMs o NFs ha e he po en ial o cons uc lexible a ia ional dis ibu ions, he
i s a emp s o use NF-based BBVI we e p oposed only ecen ly (Ag awal e al.
2020). These NF-based BBVI app oaches compa e a o ably agains exis ing BBVI
me hods bu equi e a complex aining scheme and some imes exhibi pa hological
beha io (Dhaka e al. 2021).
He e, we in oduce Be ns ein low a ia ional in e ence (BF-VI), which, o he i s
ime, uses TMs in BBVI. We use TMs based on Be ns ein polynomials o cons uc a
378
O.Dü e al.
1 3
a ia ional dis ibu ion ha closely app oxima es a po en ially complex pos e io in
Bayesian models. The p oposed me hod is compu a ionally e icien and applicable o
ypical s a is ical models. The p oposed me hod yields supe io esul s in ou expe i-
men s compa ed o exis ing NF app oaches (Dhaka e al. 2021). Using BF-VI we u -
he demons a e, o he i s ime, how VI can be used o i Bayesian semi-s uc u ed
models whe e in e p e able s a is ical model pa s (based on abula da a) and deep
NN model pa s (based on images) a e join ly i ed. We de ine ou me hod in Sec .2.1
o one-dimensional examples and gene alize i o Bayesian models wi h mul i a ia e
pos e io s in Sec .2.2. In Sec .3, we benchma k ou BF-VI app oach agains exac
Bayesian models, MCMC-Simula ions, Gaussian-VI, and NF-based BBVI, showing
accu a e pos e io app oxima ions in low dimensions and supe io app oxima ions in
highe dimensions when compa ed o NF-based BBVI and summa ize in Sec .4.
2 Be ns ein low a ia ional in e ence
In he ollowing, we desc ibe he Be ns ein Flow-VI (BF-VI) app oach, which we
p opose o accu a ely and obus ly app oxima ing po en ially complex pos e io s
in Bayesian models. The main idea is o enable he VI p ocedu e o app oxima e
he joined pos e io o he p model pa ame e s by a lexible a ia ional dis ibu ion.
This is done by modeling he ans o ma ion unc ion om a p ede ined simple
la en dis ibu ion o a po en ially complex a ia ional dis ibu ion. The numbe o
pa ame e s p in he Bayesian model de e mines he dimension o bo h he la en and
he a ia ional dis ibu ion.
We i s explain BF-VI o Bayesian models wi h a single pa ame e and hence
a one-dimensional pos e io and hen gene alize o models wi h mul i a ia e
pos e io s. The code is publicly a ailable on Gi Hub.1
2.1 One‑dimensional Be ns ein lows
BF-VI app oxima es he bijec i e ans o ma ion unc ion
g∶Z
→
𝜃
be ween a
la en a iable
Z∈ℝ
wi h p ede ined dis ibu ion
FZ∶ℝ
→
[0, 1]
wi h log-conca e
and con inuous densi y
Z
, and he model pa ame e
𝜃∈ℝ
wi h a po en ially com-
plex dis ibu ion
F𝜃∶ℝ
→
[0, 1]
so ha
FZ(z)=F𝜃(g(z))
. Figu e1 isualizes his
ans o ma ion on he scale o he densi ies, whe e Z(z)= 𝜃(g(z)) ∣
𝜕g(z)
𝜕z∣
acco ding
o he change-o - a iable o mula.
Ho ho n e al. (2018) gi e heo e ical gua an ees o he exis ence and uniqueness
o
g
=F
−1
𝜃
◦F
Z
. Howe e , g canno be compu ed di ec ly i
F𝜃
is no known (in ou
applica ion
F𝜃
is he unknown dis ibu ion o he pos e io ). The co e o BF-VI is o
app oxima e g, shown in Fig.1, by
BP
Be ns ein polynomials (BP)2 as
1 h ps:// gi hub. com/ enso chie s/ b i_ pape .
2 Some au ho s make a dis inc ion be ween Be ns ein polynomials in which
𝜗i
is ixed by he alues
o he unc ion o be app oxima ed and call exp essions like exp essions like in (1), whe e
𝜗i
is a i ing
pa ame e , polynomials o Be ns ein ype.

379
1 3
Be ns ein lows o  lexible pos e io s in a ia ional Bayes
wi h
Be i(z)= Be (i+1,M−i+1)(z)
being he densi y o a Be a dis ibu ion wi h
pa ame e s
i+1
and
M−i+1
. To p ese e he bijec i i y o g, we use in BF-VI
w.l.o.g. a s ic mono one inc easing BP o app oxima e g. Wi h he app oxima ion
o he ans o ma ion unc ion g by
BP
, i holds ha
Z(z)
can be app oxima ed by
𝜃( BP(z)) ∣
𝜕
BP
(z)
𝜕z∣
.
Using a BP o app oxima ing g gi es he ollowing heo e ical gua an ees
(Fa ouki 2012) (1) Wi h inc easing o de M o he BP, he app oxima ion
BP
o g
ge s a bi a ily close ( he BP ha e been in oduced o his e y pu pose in he con-
s uc i e p oo o he Weie s ass heo em by Be nš eın (1912)); (2) he equi ed
(1)
BP(z)=
M
∑
i=0
Be i(z)
𝜗i
M+
1
Fig. 1 O e iew o he ans o ma ion model. A Shows he bijec i e ans o ma ion unc ion
g∶Z
→
𝜃
(o i s app oxima ion
BP
) mapping be ween, B a p ede ined la en densi y
Z
, and C a po en ially com-
plex pos e io (o i s a ia ional dis ibu ion)
380
O.Dü e al.
1 3
s ic mono onici y o he app oxima ion
BP
can be easily achie ed by cons aining
he coe icien s
𝜗i
o he BP o be inc easing; (3) BPs a e obus e sus pe u ba ions
o he coe icien s
𝜗i
; (4) he app oxima ion e o dec eases wi h 1/M (Vo ono s-
kaya’s heo em). See Be nš eın (1912); Fa ouki (2012) o mo e de ailed discussions
o he bene icial p ope ies o BPs in gene al and Ho ho n e al. (2018); Ramasinghe
e al. (2021) o ans o ma ion models.
While he ou pu o
BP(z)
is un es ic ed, a BP equi es a inpu z wi hin [0,1].
We expe imen ed wi h se e al app oaches o ensu e he es ic ion
z∈[0, 1]
esul -
ing in sligh ly di e en beha io du ing he aining (see AppendixB.2.2). Based
on hese expe imen s, we decided o ob ain
z∈[0, 1]
by sampling alues om
a s anda d no mal dis ibu ion,
z�� ∼N(0, 1)
, hen apply he a ine ans o ma-
ion
l(z��)=𝛼
⋅
z�� +𝛽
, ollowed by a sigmoid
𝜎(z�)=1∕(1+e−
z
�)
. Al oge he , we
app oxima e he ans o ma ion g by
∶Z
→
𝜃
ia
= BP
◦
𝜎
◦
l
, which we call
Be ns ein low.
To allow he applica ion o uncons ained s ochas ic g adien descen op imi-
za ion, which is ypically used in he deep lea ning domain, we en o ce he s ic
mono onici y o he low as ollows: We op imize un es ic ed pa ame e s o , i.e.,
𝜗�
0
,…𝜗
�
M
,
𝛼′
,𝛽′
, and apply he ollowing ans o ma ions o de e mine he pa am-
e e s o he bijec i e low:
𝜗0
=𝜗
�
0
, and
𝜗i
=𝜗
i−1
+so plus (𝜗
�
i)
o
i=1, …,M
o
ge ing a s ic ly inc easing BP and
𝛼=so plus (𝛼�)
,
𝛽=𝛽�
o ge ing an inc eas-
ing a ine ans o ma ion.
In Appendix A, we show ha he esul ing a ia ional dis ibu ion is a igh
app oxima ion o he pos e io in he sense ha he KL di e gence be ween
q𝜆(𝜃)
and
p(𝜃∣D)
dec eases wi h he o de o he BP ia 1/M.
2.2 Mul i a ia e gene aliza ion
In he case o a Bayesian model wi h p pa ame e s,
𝜃1,𝜃2,…,𝜃p
, he Be ns ein low
bijec i ely maps a p-dimensional
Z′
o a p-dimensional
𝜽
. We ealize his low by
choosing p independen s anda d no mal Gaussians as simple la en dis ibu ion o
he p-dimensional
Z′
and apply on each componen an a ine ans o ma ion ollowed
by a sigmoid unc ion o achie e a [0,1] es ic ed
Z
. The possible complex depend-
encies in
𝜽
a e modeled in he mul i a ia e gene aliza ion
BP
o he one-dimensional
Be ns ein polynomial (see Eq.2 o he de ini ion o he j h componen o
BP
).
To achie e an e icien compu a ion, we use a iangula map o cons uc ing coe -
icien s
𝜗j
i
j=2, …p,i=0,
⋯
,M
om
Z
. This ensu es ha he j h BP de e mining
𝜃j
only depends on he i s j-1 componen s o
Z
(see Eq.2). I is known ha bijec-
i e iangula maps wi h su icien lexibili y can map a simple p-dimensional dis-
ibu ion in o a bi a y complex p-dimensional a ge dis ibu ions (Bogache e al.
(2)
𝜃
j= BPj(z1∶j)= 1
M+1
M
∑
i=0
𝜗j
i(z1,…,zj−1)Be i(zj
)
381
1 3
Be ns ein lows o  lexible pos e io s in a ia ional Bayes
2005). We use a masked au o eg essi e low (MAF) (Papamaka ios e al. 2017) o
map
Z
o he BP coe icien s
𝜗j
i
j=2, …p,i=0,
⋯
,M
om
Z
. The MAF a chi ec-
u e ensu es ha
ha
𝜗j
i
depend only on hose componen s o he la en a iables
zj′
wi h
j′≤j
(as
equi ed in Eq.2). No e ha he i s coe icien s in all BPs
𝝑1
do no depend on z and
a e he e o e no modeled ia he MAF. The e o e, he Jacobian
∇ BP
w. . .
z
is a i-
angula ma ix, and hence
de ∇ BP
is gi en by he p oduc o he diagonal elemen s
o he Jacobian allowing o e icien compu a ion o he esul ing p-dimensional
a ia ional dis ibu ion
q𝝀(𝜽)
ia he mul i a ia e e sion o he change o a iable
o mula (see Eq.6). The lexibili y o such a p-dimensional bijec i e Be ns ein low
is only limi ed by he o de M o he Be ns ein polynomial and he complexi y o
he MAF. In ou expe imen s, we use an MAF wi h wo hidden laye s, each wi h 10
neu ons. The weigh s
w
o he MAF a e pa o he a ia ional pa ame e s o
BP
. In
o al, we ha e
𝝀=(𝝑1,
w
,𝜶,𝜷)
a ia ional pa ame e s.
2.3 Va ia ional in e ence p ocedu e
In VI he a ia ional pa ame e s
𝝀
a e uned such ha he esul ing a ia ional dis-
ibu ion
q𝝀(𝜽)
is as close o he pos e io
p(𝜽∣D)
as possible. He e, we do his
by minimizing he KL di e gence be ween he a ia ional dis ibu ion and he
(unknown) pos e io :
The KL di e gence is commonly used in VI, and a ecen s udy showed ha i
is easie o ain han o he di e gences and applicable o highe -dimensional
dis ibu ions (Dhaka e al. 2021).
Ins ead o minimizing (3) usually only he e idence lowe bound (ELBO) is max-
imized (Blundell e al. 2015) which consis s o he expec ed alue o he log-likeli-
hood,
𝔼𝜽∼q𝝀(log(p(D∣𝜽)))
, minus he KL di e gence be ween he a ia ional dis i-
bu ion
q𝝀(𝜽)
and he p io
p(𝜽)
. No e ha he ELBO does no explici ly con ain he
unknown pos e io . In p ac ice, we minimize he nega i e ELBO using s ochas ic
g adien descen acili a ed by au oma ic di e en ia ion. Fo consis ency wi h Dhaka
e al. (2021), we use Tenso Flow’s RMSp op in all ou expe imen s, con igu ed wi h
he de aul se ings. We ollow he BBVI app oach and app oxima e he expec ed
log-likelihood by a e aging o e S samples
𝜽s∼q𝝀(𝜽)
ia
(3)
KL
(q𝝀(𝜽)||p(𝜽∣D)) = ∫q𝝀(𝜽)log
(
q𝝀
(𝜽)
p(𝜽∣D)
)
d𝜽
=log(p(D)) − (𝔼𝜽∼q𝝀(log(p(D∣𝜽))) − KL(q𝝀(𝜽)
||
p(𝜽))
)
⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞
⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏞⏟
ELBO (𝝀)
(4)
𝔼
𝜽∼q𝝀(log(p(Di∣𝜽))) ≈
1
S
∑
s,i
log
(
p(Di∣𝜽s)
).
382
O.Dü e al.
1 3
He eby, we also assume he usual independence o he
i=1, …N
aining da a
poin s
Di
. To ge hese samples
𝜽s
, we use S samples
z′
s
om he la en dis ibu ion,
apply he ans o ma ion
= BP
◦
𝜎
◦
l
, and hen compu e he co esponding pa am-
e e samples ia
𝜽s= (zs)
. We use he same samples
𝜽s∼q𝝀(𝜽)
o app oxima e he
Kullback–Leible di e gence be ween he a ia ional dis ibu ion
q𝝀(𝜽)
and he p io
p(𝜽)
ia:
whe e he p obabili y densi y
q𝝀(𝜽s)
can be calcula ed, om he samples
𝜽s
using he
change o a iable unc ion as:
2.4 E alua ion
E alua ing he quali y o he i ed a ia ional dis ibu ions equi es a compa ison
o he ue pos e io . In he case o low-dimensional p oblems, he wo dis ibu ions
can be compa ed isually. In he case o highe -dimensional p oblems, his is no
possible anymo e. While he e idence lowe bound (ELBO) is a aluable me ic o
op imizing he pa ame e s in VI, i is less help ul in compa ing di e en app oxi-
ma ions because i depends on he speci ic pa ame iza ion o he model (Yao e al.
2018). The e o e, Yao e al. (2018) in oduced

k
as a mo e sui ed app oach o com-
pa ison, which since hen has been used in o he s udies like (Dhaka e al. 2021) o
which we compa e. The compu a ion o

k
is based on he impo ance a ios which
a e de ined as
I he a ia ional dis ibu ion
q𝝀(𝜽)
would be a pe ec app oxima ion o he pos e-
io
p(𝜽∣D)∝p(D∣𝜽)p(𝜽)
, hen impo an a ios
s
would be cons an . Howe e ,
because o he asymme y o he KL di e gence used in he op imiza ion objec-
i e (see Eq.3), he i ed
q𝝀(𝜽)
ends o ha e ligh e ails han
p(𝜽∣D)
, wi h he
e ec ha he dis ibu ion o
s
is hea ily igh - ailed. To quan i y he se e i y o he
unde es ima ed ails, a gene alized Pa e o dis ibu ion is i ed o he igh ail o he
s
. The es ima ed shape pa ame e

k
o he Pa e o dis ibu ion can be used as a diag-
nos ic ool. A la ge

k
indica es a p onounced ail in he
s
dis ibu ion and, hence, a
bad pos e io app oxima ion. Acco ding o Yao e al. (2018) alues o

k<0.5
indi-
ca e ha he a ia ional app oxima ion
q𝝀
is good. Values o
0.5 <
k<0.7
indica e
he a ia ional app oxima ion
q𝝀
is no pe ec bu s ill use ul.
(5)
KL
(q𝝀(𝜽)
||
p(𝜽)) ≈ 1
S
∑
s
log
(
q𝝀
(𝜽
s
)
p(𝜽
s
)
)
(6)
q𝝀
(𝜽
s
)=p(z
�
s
)⋅∣de ∇z�
BP
(𝜎(l(z
�
s
))) ∣
−1
(7)
s=
p(𝜽
s
,D)
q
𝝀(
𝜽
s)
=
p(D∣𝜽
s
)p(𝜽
s
)
q
𝝀(
𝜽
s)
389
1 3
Be ns ein lows o  lexible pos e io s in a ia ional Bayes
NNs and abula da a by in e p e able model componen s. Please no e ha he e, bo h
he condi ional dis ibu ion o he ou come
(y∣B,x)
and he uncondi ional pos e io
o he pa ame e s a e modeled by ans o ma ion models. Because o he deep NN
model componen s in ol ed, MCMC is no easible anymo e o de e mine he pos e-
io . As a da ase , we use he SIIM-ISIC Melanoma Classi ica ion Challenge4 da a.
The da a come om 33,126 pa ien s (6626 as es se , 21,200 ain, and 5300 alida-
ion se ) wi h a con i med diagnosis o hei skin lesions, which is in
≈98
% benign
(
y=0
) and in
≈2
% malignan (
y=1
). The p o ided da a
D=(B,x)
a e semi-s uc-
u ed since i comp ises (uns uc u ed) image da a B om he pa ien ’s lesion along
wi h (s uc u ed) abula da a x, i.e., he pa ien ’s age.
We i he condi ional ou come dis ibu ion
(
Y∣D)∼ Be
(
𝜋
D)
by modeling he
p obabili y o a lesion o be malignan
𝜋D=
p
(
y
=1∣
D
)=𝜎(
h
)
applying he sig-
moid unc ion
𝜎(
⋅
)
o a i ed ans o ma ion unc ion
h∶Y
→
Z
. We s udy h ee
models o h depending on x alone, B alone, and in combina ion B and x:
M1 (DL-Model)
h=𝜇(B)
: As a baseline, we use a deep con olu ional neu al
ne wo k (CNN) based on he melanoma image da a (see Fig. 6c) wi h a o al o
419,489 weigh s o ake ad an age o he p edic i e powe o DL on complex image
da a. Fo his DL model, we use deep ensembling (Lakshmina ayanan e al. 2017)
by i ing h ee CNN models wi h di e en andom ini ializa ions and a e aging he
Fig. 6 The a chi ec u e o he used NN models o model pa s. a Dense NN wi h one hidden laye o
model nonlinea dependencies om he inpu (used in he NN-based nonlinea eg ession example). b
Dense NN wi hou hidden laye o model linea dependencies om abula inpu da a (used in M1 and
M3 o he melanoma expe imen ). c CNN o model nonlinea dependencies om he image inpu (used
in M1 and M3)
4 h ps:// chall enge2 020. isic- a chi e. com

390
O.Dü e al.
1 3
p edic ed p obabili ies. The achie ed es p edic i e pe o mance and i s compa ison
o o he models a e discussed in he las pa ag aph o his sec ion.
M2 (Logis ic Reg ession)
h=𝜇0+𝛽1
⋅
x
: When using only abula ea u es x,
in e p e able models can be buil . We conside a Bayesian logis ic eg ession wi h
age as he only explana o y a iable x and use a BNN wi hou a hidden laye o se
up he model (see Fig.6b wi h only one inpu ea u e x). In logis ic eg ession, a
la en a iable is modeled by a linea p edic o
h=𝜇0+𝛽1
⋅
x
, which de e mines
he p obabili y o a lesion o be malignan ia
𝜋x
=𝜎
(
𝜇
0
+𝛽
1
⋅x
)
allowing o in e
e
e𝛽1
as he odds a io, i.e., he ac o by which he odds o lesions o be malignan
changes when inc easing he p edic o x by one uni . In Fig.7, we compa e he exac
MCMC pos e io o
𝛽1
wi h he BF-VI app oxima ion, demons a ing ha BF-VI
accu a ely app oxima es he pos e io .
M3 (semi-s uc u ed)
h=𝜇(B)+𝛽1
⋅
x
: This model in eg a es image and abu-
la da a and combines he p edic i e powe o M1 wi h he in e p e abili y o M2.
We use a (non-Bayesian) CNN ha de e mines
𝜇(B)
and BF-VI o he NN wi hou
a hidden laye ha de e mines
𝛽1
(see Fig.6b and c). Bo h NNs a e join ly ained
by op imizing he ELBO. The esul ing pos e io o
𝛽1
di e s om he simple logis-
ic eg ession (see Fig.7), indica ing a diminished e ec o age a e including he
image. Again,
e𝛽1
can be in e p e ed as he ac o by which he odds o a lesion o
be malignan change when inc easing he p edic o age by one uni and holding he
image cons an .
While he main in e es o ou s udy is on he pos e io s, we also de e mine he
p edic i e pe o mance on he es se . To quan i y and compa e he es p edic ion
pe o mances, we look a he achie ed log sco es (M1:
−
0.076, M2:
−
0.085, M3:
−
0.076) and he AUCs wi h 95
%
CI (M1: 0.83(0.79,0.86), M2: 0.66(0.61, 0.71),
M3: 0.82(0.79,0.85)). Fo bo h measu es, highe is be e . In e es ingly, he image-
based models (M1, M3) ha e highe p edic i e powe han M2, which only uses
abula da a. The semi-s uc u ed model M3, including abula and image in o ma-
ion, has a simila p edic i e powe compa ed o M1, which only uses images. The
bene i o he semi-s uc u ed model he e is ha i p o ides in e p e able pa ame e s
o he abula da a along wi h unce ain y quan i ica ion wi hou losing p edic i e
pe o mance.
Fig. 7 Pos e io s o he age-
e ec pa ame e
𝛽1
in he mela-
noma models M2 and M3
0
2
4
6
0.0 0.
40
.8
β
1
densi y
M2: BF−VI
M2: MCMC
M3: CNN+BF−VI
391
1 3
Be ns ein lows o  lexible pos e io s in a ia ional Bayes
4 Summa y andou look
The p oposed BF-VI is lexible enough o app oxima e any pos e io in p inciple
wi hou being es ic ed o a ia ional dis ibu ions wi h known pa ame ic
dis ibu ion amilies like Gaussians. In benchma k expe imen s, BF-VI accu a ely
i s non- i ial pos e io s in low-dimensional p oblems in a BBVI se ing. Fo
highe -dimensional models, BF-VI ou pe o ms published esul s om o he NF-VI
me hods (Dhaka e al. 2021) on he s udied benchma k da ase s. S ill, we obse e
ha he pos e io canno be i ed pe ec ly in high dimensions by BF-VI, especially
since he ails o he app oxima ion a e oo sho . We a ibu e his limi a ion
o known di icul ies in he op imiza ion p ocess and he asymme y o he KL
di e gence. These challenges o VI we e no in he ocus o ou s udy, and we lea e
i o u he esea ch.
To he bes o ou knowledge, we a e he i s o demons a e how BBVI can be
used in semi-s uc u ed models. We used BF-VI on he public melanoma challenge
da ase , in eg a ing image da a and abula da a by combining a deep CNN and an
in e p e able model pa . We see a aluable applica ion o BF-VI in models wi h
in e p e able pa ame e s, i.e., s a is ical o semi-s uc u ed models whe e we can
model complex pos e io dis ibu ions o he in e p e able pa ame e s. Especially
in semi-s uc u ed models wi h deep NN componen s ha canno be i ed wi h
MCMC, BF-VI allows de e mining he a ia ional dis ibu ion o he in e p e able
model pa s. Mo eo e , e icien SGD op imize s can be used in BF-VI o i all
model pa s join ly. We plan o ex end ou esea ch on BF-VI o semi-s uc u ed
models in he u u e and in es iga e he quali y o he pos e io app oxima ions.
Supplemen a y In o ma ion The online e sion con ains supplemen a y ma e ial a ailable a h ps:// doi.
o g/ 10. 1007/ s10182- 024- 00497-z.
Acknowledgemen s The esea ch o BS was suppo ed by he No a is Resea ch Founda ion (F eeNo a-
ion2019). The esea ch o he OD and DD was pa ly suppo ed by he Fede al Minis y o Educa ion
and Resea ch o Ge many (BMBF) in he p ojec DeepDoub (g an no. 01IS19083A). We, u he , would
like o hank Nadja Klein, Lucas Kook, and Rebekka Ax helm o ui ul discussions.
Funding Open Access unding enabled and o ganized by P ojek DEAL. This s udy was unded by Bun-
desminis e ium ü Bildung, Wissenscha , Fo schung und Technologie (01IS19083A), No a is S i ung
ü Medizinisch-Biologische Fo schung.
Decla a ions
Con lic o in e es None.
Open Access This a icle is licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License,
which pe mi s use, sha ing, adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as long
as you gi e app op ia e c edi o he o iginal au ho (s) and he sou ce, p o ide a link o he C ea i e
Commons licence, and indica e i changes we e made. The images o o he hi d pa y ma e ial in his
a icle a e included in he a icle’s C ea i e Commons licence, unless indica ed o he wise in a c edi line
392
O.Dü e al.
1 3
o he ma e ial. I ma e ial is no included in he a icle’s C ea i e Commons licence and you in ended
use is no pe mi ed by s a u o y egula ion o exceeds he pe mi ed use, you will need o ob ain pe mis-
sion di ec ly om he copy igh holde . To iew a copy o his licence, isi h p://c ea i ecommons.o g/
licenses/by/4.0/.
Re e ences
Ag awal, A., Sheldon, D.R., Domke, J.: Ad ances in black-box VI: no malizing lows, impo ance
weigh ing, and op imiza ion. Ad . Neu al In . P ocess. Sys . 33, 17358–17369 (2020)
Baumann, P.F.M., Ho ho n, T., Rügame , D.: Deep condi ional ans o ma ion models. In: Machine
Lea ning and Knowledge Disco e y in Da abases. Resea ch T ack, pp. 3–18. Sp inge , Cham (2021)
Be nš eın, S.: Démons a ion du héo eme de weie s ass ondée su le calcul des p obabili ies. Commun.
Soc. Ma h. Kha ko 13, 1–2 (1912)
Bingham, E., Chen, J.P., Jankowiak, M., Obe meye , F., P adhan, N., Ka ale sos, T., Singh, R., Sze lip,
P.A., Ho s all, P., Goodman, N.D.: Py o: deep uni e sal p obabilis ic p og amming. J. Mach. Lea n.
Res. 20, 28–1286 (2019)
Blei, D., Rangana h, R., Mohamed, S.: Va ia ional in e ence: ounda ions and mode n me hods. In: NIPS
u o ial (2016)
Blei, D.M., Kucukelbi , A., McAuli e, J.D.: Va ia ional in e ence: a e iew o s a is icians. J. Am. S a .
Assoc. 112(518), 859–877 (2017)
Blundell, C., Co nebise, J., Ka ukcuoglu, K., Wie s a, D.: Weigh unce ain y in neu al ne wo k. In:
In e na ional Con e ence on Machine Lea ning, pp. 1613–1622. PMLR (2015)
Blundell, C., Co nebise, J., Ka ukcuoglu, K., Wie s a, D.: Weigh unce ain y in neu al ne wo k. In:
In e na ional Con e ence on Machine Lea ning, pp. 1613–1622. PMLR (2015)
Bogache , V.I., Kolesniko , A.V., Med ede , K.V.: T iangula ans o ma ions o measu es. Sbo nik
Ma h. 196(3), 309 (2005)
Bu i, M., Cu , A., S ee es, J., Ho ho n, T.: Baseline-adjus ed p opo ional odds models o he quan i i-
ca ion o ea men e ec s in ials wi h o dinal sum sco e ou comes. BMC Med. Res. Me hodol. 20,
1–14 (2020)
Campanella, G., Kook, L., Häggs öm, I., Ho ho n, T., Fuchs, T.J.: Deep condi ional ans o ma ion mod-
els o su i al analysis (2022). a Xi p ep in a Xi : 2210. 11366
Ca pen e , B., Gelman, A., Ho man, M.D., Lee, D., Good ich, B., Be ancou , M., B ubake , M., Guo,
J., Li, P., Riddell, A.: S an: a p obabilis ic p og amming language. J. S a . So w. (2017). h ps:// doi.
o g/ 10. 18637/ jss. 076. i01
Dhaka, A.K., Ca alina, A., Ande sen, M.R., Magnusson, M., Huggins, J., Veh a i, A.: Robus , accu a e
s ochas ic op imiza ion o a ia ional in e ence. Ad . Neu al In . P ocess. Sys . 33, 10961–10973
(2020)
Dhaka, A.K., Ca alina, A., Welandawe, M., Ande sen, M.R., Huggins, J., Veh a i, A.: Challenges and
oppo uni ies in high dimensional a ia ional in e ence. Ad . Neu al In . P ocess. Sys . 34, 7787–
7798 (2021)
Dinh, L., Sohl-Dicks ein, J., Bengio, S.: Densi y es ima ion using eal NVP. In: In e na ional Con e ence
on Lea ning Rep esen a ions (2016)
Du kan, C., Bekaso , A., Mu ay, I., Papamaka ios, G.: Neu al spline lows. Ad . Neu al In . P ocess.
Sys . 32, 7511–7522 (2019)
Fa ouki, R.T.: The Be ns ein polynomial basis: a cen ennial e ospec i e. Compu . Aided Geom. Des.
29(6), 379–419 (2012)
Ho ho n, T., Kneib, T., Bühlmann, P.: Condi ional ans o ma ion models. J. R. S a . Soc. Se . B S a .
Me hodol. 76(1), 3–27 (2014)
Ho ho n, T., Moes , L., Buehlmann, P.: Mos likely ans o ma ions. Scand. J. S a . 45(1), 110–134 (2018)
Huggins, J., Kasp zak, M., Campbell, T., B ode ick, T.: Valida ed a ia ional in e ence ia p ac ical pos-
e io e o bounds. In: In e na ional Con e ence on A i icial In elligence and S a is ics, pp. 1792–
1802. PMLR (2020)
393
1 3
Be ns ein lows o  lexible pos e io s in a ia ional Bayes
Jaini, P., Selby, K.A., Yu, Y.: Sum-o -squa es polynomial low. In: In e na ional Con e ence on Machine
Lea ning, pp. 3009–3018. PMLR (2019)
Jo dan, M.I., Ghah amani, Z., Jaakkola, T.S., Saul, L.K.: An in oduc ion o a ia ional me hods o
g aphical models. Mach. Lea n. 37(2), 183–233 (1999)
Klein, N., Ho ho n, T., Ba ban i, L., Kneib, T.: Mul i a ia e condi ional ans o ma ion models. Scand. J.
S a . 49(1), 116–142 (2019)
Kook, L., He zog, L., Ho ho n, T., Dü , O., Sick, B.: Deep and in e p e able eg ession models o o di-
nal ou comes. Pa e n Recogni . 122, 108263 (2022)
Lakshmina ayanan, B., P i zel, A., Blundell, C.: Simple and scalable p edic i e nce ain y es ima ion
using deep ensembles. In: Ad ances in Neu al In o ma ion P ocessing Sys ems, ol. 30 (2017)
Lohse, T., Roh mann, S., Faeh, D., Ho ho n, T.: Con inuous ou come logis ic eg ession o analyzing
body mass index dis ibu ions. F1000Resea ch 6, 1933 (2017)
Louizos, C., Welling, M.: Mul iplica i e no malizing lows o a ia ional Bayesian neu al ne wo ks. In:
In e na ional Con e ence on Machine Lea ning, pp. 2218–2227. PMLR (2017)
Magnusson, M., Bü kne , P., Veh a i, A.: Pos e io DB: a se o pos e io s o Bayesian in e ence and
p obabilis ic p og amming
Papamaka ios, G., Pa lakou, T., Mu ay, I.: Masked au o eg essi e low o densi y es ima ion. In: P o-
ceedings o he 31s In e na ional Con e ence on Neu al In o ma ion P ocessing Sys ems, pp. 2335–
2344 (2017)
Ramasinghe, S., Fe nando, K., Khan, S., Ba nes, N.: Robus no malizing lows using Be ns ein- ype pol-
ynomials (2021). a Xi p ep in a Xi : 2102. 03509
Rangana h, R., Ge ish, S., Blei, D.: Black box a ia ional in e ence. In: A i icial In elligence and S a is-
ics, pp. 814–822. PMLR, Camb idge (2014)
Rezende, D., Mohamed, S.: Va ia ional in e ence wi h no malizing lows. In: In e na ional Con e ence on
Machine Lea ning, pp. 1530–1538. PMLR (2015)
Rügame , D., Baumann, P.F.M., Kneib, T., Ho ho n, T.: P obabilis ic ime se ies o ecas s wi h au o e-
g essi e ans o ma ion models. a Xi p ep in (2021)
Sick, B., Ha ho n, T., Dü , O.: Deep ans o ma ion models: ackling complex eg ession p oblems wi h
neu al ne wo k based ans o ma ion models. In: 2020 25 h In e na ional Con e ence on Pa e n
Recogni ion (ICPR), pp. 2476–2481. IEEE (2021)
Sieg ied, S., Ho ho n, T.: Coun ans o ma ion models. Me hods Ecol. E ol. 11(7), 818–827 (2020).
h ps:// doi. o g/ 10. 1111/ 2041- 210X. 13383
Van DenBe g, R., Hasencle e , L., Tomczak, J.M., Welling, M.: Syl es e no malizing lows o a i-
a ional in e ence. In: 34 h Con e ence on Unce ain y in A i icial In elligence 2018, UAI 2018, pp.
393–402. Associa ion Fo Unce ain y in A i icial In elligence (AUAI) (2018)
Welandawe, M., Ande sen, M.R., Veh a i, A., Huggins, J.H.: Robus , au oma ed, and accu a e black-box
a ia ional in e ence (2022). a Xi p ep in a Xi : 2203. 15945
Yao, Y., Veh a i, A., Simpson, D., Gelman, A.: Yes, bu did i wo k?: E alua ing a ia ional in e ence. In:
In e na ional Con e ence on Machine Lea ning, pp. 5581–5590. PMLR (2018)
Yao, Y., Veh a i, A., Gelman, A.: S acking o non-mixing Bayesian compu a ions: he cu se and blessing
o mul imodal pos e io s. J. Mach. Lea n. Res. 23, 1–45 (2022)
Publishe ’s No e Sp inge Na u e emains neu al wi h ega d o ju isdic ional claims in published maps
and ins i u ional a ilia ions.
394
O.Dü e al.
1 3
Au ho s and A ilia ions
Oli e Dü 1 · S e anHö ling1· DanilDold1· I onneKo ylo 1· Bea eSick2,3
* Oli e Dü
oli e .due @h wg-kons anz.de
* Bea e Sick
bea e.sic[email p o ec ed]; [email protected]
S e an Hö ling
S e anhoe [email p o ec ed]
Danil Dold
danil.dold@h wg-kons anz.de
I onne Ko ylo
i [email p o ec ed]
1 IOS, Kons anz Uni e si y o Applied Sciences, Al ed Wach el S aße 8, 78462Kons anz,
Ge many
2 IDP, Zu ich Uni e si y o Applied Sciences, Technikums asse 81, 8401Win e hu ,
Swi ze land
3 EBPI, Uni e si y o Zu ich, Hi scheng aben 84, 8001Zu ich, Swi ze land