Deducing neighborhoods of classes from a fitted model

Ge ha z, Alexande ; G oll, And eas; Schaube ge , Gun he
A icle — Published Ve sion
Deducing neighbo hoods o classes om a i ed model
AS A Ad ances in S a is ical Analysis
P o ided in Coope a ion wi h:
Sp inge Na u e
Sugges ed Ci a ion: Ge ha z, Alexande ; G oll, And eas; Schaube ge , Gun he (2024) : Deducing
neighbo hoods o classes om a i ed model, AS A Ad ances in S a is ical Analysis, ISSN 1863-818X,
Sp inge , Be lin, Heidelbe g, Vol. 108, Iss. 2, pp. 395-425,
h ps://doi.o g/10.1007/s10182-024-00502-5
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/315086
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h p://c ea i ecommons.o g/licenses/by/4.0/
Vol.:(0123456789)
AS A Ad ances in S a is ical Analysis (2024) 108:395–425
h ps://doi.o g/10.1007/s10182-024-00502-5
ORIGINAL PAPER
Deducing neighbo hoods o classes oma i ed model
Alexande Ge ha z1 · And easG oll1· Gun he Schaube ge 2
Recei ed: 22 No embe 2022 / Accep ed: 16 Feb ua y 2024 / Published online: 8 May 2024
© The Au ho (s) 2024, co ec ed publica ion 2024
Abs ac
In his a icle, a new kind o in e p e able machine lea ning me hod is p esen ed,
which can help o unde s and he pa i ion o he ea u e space in o p edic ed classes
in a classi ica ion model using quan ile shi s, and his way make he unde lying s a-
is ical o machine lea ning model mo e us wo hy. Basically, eal da a poin s (o
speci ic poin s o in e es ) a e used and he changes o he p edic ion a e sligh ly
aising o dec easing speci ic ea u es a e obse ed. By compa ing he p edic ions
be o e and a e he shi s, unde ce ain condi ions he obse ed changes in he p e-
dic ions can be in e p e ed as neighbo hoods o he classes wi h ega d o he shi ed
ea u es. Cho ddiag ams a e used o isualize he obse ed changes. Fo illus a-
ion, his quan ile shi me hod (QSM) is applied o an a i icial example wi h medi-
cal labels and a eal da a example.
Keywo ds In e p e able machine lea ning· Explainable a i icial in elligence·
Classi ica ion ask· Fea u e space pa i ion· Cho ddiag ams
1 In oduc ion
Wi h he inc easing demand o e y complex models in he a eas o da a analysis
and p edic i e modeling, he numbe o blackbox models is g owing s eadily. The
p oblem wi h hese models is ha by aising he p edic i e powe o a model o an
algo i hm by adding mo e complexi y o lexibili y o i , he loss o in e p e abili y
can be emendous. While mos ly i is ai ly easy o unde s and he i ing algo i hm,
unde s anding he i ed p edic ion model is p e y ha d. In a andom o es wi h
500 ees, o example, i is easy o unde s and a single classi ica ion ee, bu o
comple ely unde s and he whole ensemble model i is necessa y o look a e e y
* Alexande Ge ha z
ge ha z@s a is ik. u-do mund.de
1 Chai o S a is ical Me hods o Big Da a, Facul y o S a is ics, TU Do mund Uni e si y,
Do mund, Ge many
2 Technical Uni e si y o Munich, TUM School o Medicine andHeal h, Chai o Epidemiology,
Munich, Ge many
396
A.Ge ha z e al.
spli in e e y ee, which ge s oo expensi e i he co esponding classi ica ion ask
was e y huge and complex (B eiman 2001).
The wo ld o in e p e able machine lea ning (IML) me hods ies o open a
doo o unde s and he in e nals o hese complex models wi hou ha ing o unde -
s and e e y single in e nal de ail o hem. Al oge he , his way IML me hods y
o inc ease he us wo hiness o such complex models. A amous IML me hod is
he compu a ion o he pe mu a ion ea u e impo ance as desc ibed by B eiman
(2001). He e, he inpu is andomly pe mu ed ea u e by ea u e and i is measu ed
how much wo se a model pe o ms a e pe mu a ion in o de o de e mine he ea-
u es’ impo ance in he model. In con as , he pa ial dependence plo (F iedman
2001), o example, does no calcula e he impo ance o a ea u e in a model, bu i
is a well-known me hod o es ima e he mean e ec o a speci ic ea u e on he a ge
alue by shi ing he inpu s o some da a and o obse e how he ou pu changes. A
ypical s uc u e o his kind o IML me hods is displayed in Fig.1.
Ano he in e p e able machine lea ning me hod ha is based on his s uc u e
is he indi idual condi ional expec a ions (ICE) plo , which, simila o he pa ial
dependence plo , desc ibes he e ec o a speci ic ea u e on he a ge alue, bu
ins ead o displaying a mean e ec i p esen s he indi idual changes o e e y
obse a ion (Golds ein e al. 2015). Ano he comple ely di e en IML me hod is he
usage o ancho s (Ribei o e al. 2018). Ancho s a e used o ind speci ic ea u es
and hei espec i e ea u e alues ha de e mine he p edic ion o an obse a ion,
while he o he ea u es could be andomly al e ed wi hou a ec ing he p edic ion
oo much.
An o e iew o he s ill e ylimi ed ange o IML me hods can be ound in he
publicly a ailable book o Molna (2019), which lis s e enmo e IML me hods and
explains hei usages on e e y day examples. Mos o hose me hods a e applicable
Fig. 1 Basic concep behind mos in e p e able machine lea ning me hods/explainable a i icial
in elligence
397
Deducing neighbo hoods o classes oma i ed model
on bo h eg ession and classi ica ion asks (o e en mo e), while he me hod p o-
posed in his a icle is speci ically designed o classi ica ion asks only.
The quan ile shi me hod (QSM) p esen ed in his wo k is based on he basic
concep o IML (see Fig.1) and is used o de e mine which classes a e modeled
as neighbo s by a i ed model wi h ega d o speci ic ea u es o in e es . QSM is
used o de e mine, which small changes in he ea u es lead o a subs an ial change
in he p edic ions in he sense ha he p edic ed class labels change. These changes
can hen be in e p e ed as neighbo hoods o he di e en classes o an obse a ion
be o e and a e he shi . In con as , he ancho s me hod is used o ind he ea-
u es and hei espec i e alues, which de e mine a speci ic p edic ion and in e p e
hem as subs an ial o his speci ic p edic ion. While bo h me hods obse e whe he
sligh changes in he ea u es change a p edic ion, he in e p e a ion is subs an ially
di e en .
In an applica ion se ing, QSM can be used whene e he in luence o a ea u e on
he p edic ion o a a ge class wi hin a ixed classi ica ion model is o in e es . This
could be use ul, e.g., in medicine, when a physician uses a classi ica ion model o
pain le els and wan s o in es iga e wha in luence an inc ease o a ea u e, e.g., he
hea a e, can ha e on he pain le el o he pa ien . Fo some classic s a is ical mod-
els, his could be de i ed om he eg ession coe icien s. Howe e , i he model has
many a ge classes and many ea u es, he inally p edic ed label is ha d o de i e
om he coe icien s. Also, i he unde lying model is a blackbox model wi h no
easy way o in e p e a ion, i is a challenge o in es iga e he in luence o a speci ic
ea u e. He e, QSM can help as i will indica e how he p edic ion will change i a
speci ic ea u e (o e en mul iple ea u es) o in e es a e modi ied.
The emainde o his a icle is s uc u ed as ollows: In Sec .2, we in oduce he
ma hema ical de ails o he me hod and de i e he co esponding mig a ion ma ix
con aining hese changes, which will la e be p esen ed as a cho ddiag am. Addi-
ionally, we illus a e he me hod’s ele ance wi h an a i icially c ea ed example
wi h labels om he ield o medicine and also p o ide an in-dep h discussion and
explana ion o how o gene ally in e p e he me hod’s esul s. In Sec .4, a eal da a
example is used o illus a e how he me hod wo ks, and di e en ways how o use
QSM a e shown. Finally, Sec .5 concludes and discusses ad an ages and disad an-
ages o he p oposed me hod.
2 Me hodology
In his sec ion, we i s se he ma hema ical backg ound o QSM and explain how
o in e p e i . As he e a e ce ain condi ions, which ha e o be kep in mind o
assu e a meaning ul in e p e a ion o he esul s, we will hen explain some possible
pi alls and how hey can be sol ed.
398
A.Ge ha z e al.
2.1 Mo i a ion
The gene al idea o he p oposed me hod is o ake he p edic ions o a model o
a gi en da a se , hen inc ease o dec ease he alue o one o mul iple ea u e(s)
o each obse a ion and in e p e he changes in he p edic ed classes.
Wi h he p oposed me hod, i can be in es iga ed, which classes o a ca ego i-
cal a ge a e p edic ed nex o each o he based on he pa i ion o he ea u e
space by a speci ic classi ica ion model. This way, i can be lea ned which a ge
classes di e wi h ega d o he ea u e(s) o in e es , bu ha e e y simila (o
equal) alues in hei o he ea u es. I can also be in es iga ed, which o he a -
ge classes ha e la ge o lowe alues o he ea u e(s) o in e es . Fu he mo e,
his me hod can also be used o in es iga e whe he he e is an o dinal s uc u e in
he ca ego ical a ge a iable.
Fo his me hod, an in ui i e way o conduc he shi s o a speci ic ea u e is
o choose a ixed absolu e amoun and jus add o sub ac i o/ om he ea u e
o all obse a ions. Howe e , he e a e wo majo issues wi h his p ocess, which
we will desc ibe in he ollowing.
I he dis ibu ion o a ea u e is e y complex and con ains some low-densi y
a eas and some high-densi y a eas, hen a model can usually lea n mo e decision
bounda ies in a high-densi y a ea han in a low-densi y a ea (Fig.2). He e, i is
al eady clea , why changing he ea u e alues by a ixed amoun is no always a
good idea, because a small shi size migh be enough o change he p edic ion
in a high-densi y a ea, bu no in a low-densi y a ea. I he shi size is chosen
o be la ge, hen i migh be app op ia e o he low-densi y a ea, bu he shi s
migh be oo big o he high-densi y a ea and some impo an neighbo s migh
be skipped. The yellow a ows in his igu e display a possible shi size. In he
high-densi y a ea, he da a poin s could al eady be shi ed a enough o possibly
skip wo decision bounda ies, while in he low-densi y a ea some da a poin s do
no e en skip a single decision bounda y.
Ano he issue is ha by modi ying he ea u e alues he ma ginal dis ibu ion
ge s changed a lo , leading o alues ha ha e no exis ed be o e and migh e en
be impossible o exis . This can easily lead o ex apola ion, whe e he model is
used o p edic he class o obse a ions wi h unlikely o e en impossible ea-
u e alues (see Hooke 2004), which will esul in in e p e a ions which a e no
meaning ul.
Fig. 2 Example o a ea u e wi h high- and low-densi y a eas and possible decision bounda ies; yellow
a ows display a common possible shi size o all ea u e

399
Deducing neighbo hoods o classes oma i ed model
The p oposed me hod ackles also hese issues by using he unde lying da a
o dic a e he size o he shi o each a ea o he ange o a ea u e. This can be
achie ed by conduc ing he shi s based on he empi ical cumula i e dis ibu ion
unc ion. This way, smalle shi s will be conduc ed in high-densi y a eas and la ge
shi s in low-densi y a eas.
P incipally, QSM can help o answe he ollowing ques ions:
• Wha happens o he p edic ed label, when a speci ic ea u e inc eases o
dec eases? This ques ion can be ele an in he ield o Medicine. QSM could
help a physician who is conce ned wha e ec a speci ic pha maceu ical, which
has an impac on a ea u e o he classi ica ion model, could ha e, e.g., on he
pain le el o a speci ic g oup o pa ien s. Then, QSM can indica e he e ec .
Al hough he me hod is designed o wo k globally, a ained classi ica ion model
can be used wi h ocus on a e y speci ic g oup o obse a ions o explain he
model locally. Resea che s o o he ields, e.g., Spo s o Poli ics, could also be
in e es ed in his kind o ques ion as o en a new aining me hod (Spo s) o a
new law o poli ical guideline (Poli ics) can ha e an impac on a ea u e wi hin
a classi ica ion model. I one can es ima e he impac on he ea u es, hen wi h
QSM he impac on he p edic ion o he a ge classes can be in es iga ed.
• How a e he a ge classes ela ed? This ques ion can be ele an in, e.g., Social
Sciences. I one models he impac o social-economic ea u es on, e.g., he hap-
piness in li e ( a ge a iable - measu ed in ca ego ies), hen QSM can help o
in es iga e wha kind o impac an inc ease in, e.g., he income o he li ing
expenses has. Relying on quan iles o in es iga e he impac , as p oposed in his
me hod, migh imp o e he in e p e a ion.
• Is he e an o dinal s uc u e wi hin he a ge classes? He e, he e a e wo ypes
o cases in which his can be ele an :
– In he i s case, i one assumes ha he e migh be some kind o o dinal s uc-
u e wi hin he a ge classes based on a speci ic ea u e o in e es , one could
in es iga e i inc easing o dec easing ha ea u e con i ms he di ec ion o
he assumed o de . E.g., i he e is a model o he ela ion o he physical
i ness o a oo ball playe and he league he playe is playing in, one could
assume ha as e playe s a e playing in highe leagues. O cou se, he e
migh be some mo e complex s uc u es ha o some playe s being as e
migh no necessa ily help o play in a highe league (goalkeepe s?).
– The second case is when he a ge ac ually has an o dinal s uc u e, bu he
me hod used can only do classi ica ion wi hou aking he o dinal s uc u e
in o accoun . In Economics, he c edi a ing o a coun y o a company is
usually indica ed wi h o dinal ca ego ies. I he classi ica ion me hod used
can only ea his a ge a iable as ca ego ical and no o dinal, hen QSM
can help o in es iga e his s uc u e. I QSM indica es ha sligh ly weaking
one o he ea u es makes he p edic ed c edi a ing ac ually jump by mul iple
le els, hen he i ed model migh ac ually need u he in es iga ion. Fo his
case, by no means does QSM indica e how he model needs o be modi ied,
bu i migh e eal some pi alls o he model a hand.
400
A.Ge ha z e al.
2.2 Ma hema ical backg ound
In he ollowing, we will se he ma hema ical backg ound o QSM. The aim is o
sligh ly inc ease o dec ease he alue o he ea u es o in e es and obse e he
changes in he p edic ed classes.
Suppose

(x)
is a inal model i ed o a classi ica ion ask on a sample o size
n
wi h
K
di e en classes,
K≥2
, and
L
be he se o all he ea u es used o his clas-
si ica ion wi h a speci ic se size
p=|L|
. Then,
𝜋

,k(x)
deno es he es ima ed p obabil-
i y by he model

(
⋅
)
o an obse a ion
x
o belong o a speci ic class
k∈{1, …,K}
.
Nex , we de e mine
such ha
k∗

(x)
is he class wi h he highes p obabili y as es ima ed by he model

(
⋅
)
o he obse a ion
x
( om he e on we will always alk abou he same i ed
model, which is why we d op index

in he ollowing o be e eadabili y).
Nex , we choose a subse
M⊆L
con aining he ea u e(s) o in e es . Mos ly, he
subse
M
has a size o
|M|=1
, i.e.,we ocus on a single speci ic ea u e. Le now

xi
ep esen he ea u e- ec o o obse a ion
i
,
i=1, ..., n
, whe e hose ea u es om
M
each we e shi ed componen wisely by a small amoun .
The shi is done by sligh ly inc easing o dec easing he esul o he empi ical
cumula i e dis ibu ion unc ion (ecd ) o he subse
M
con aining he ea u es o in e -
es be o e applying he quan ile- unc ion (see Fig.3). Fo his pu pose, a small alue
ql
, he quan ile shi size, is added componen wisely o

Fl
(⋅
)
deno ing he ecd o all
ea u es
l=1, ..., p
, wi h
k∗

(x) = a g max
k
𝜋

,k
(x)
q
l=
{
ul, o Ll∈M,wi h ul∈ [−1, 1
]
0, else.
Fig. 3 Example o a ea u e shi o a single ea u e (con inuous case)
401
Deducing neighbo hoods o classes oma i ed model
The empi ical cumula i e dis ibu ion unc ion can be buil ei he on he o iginal
da a se , used o i ing he unde lying model, o on a e y speci ic da a se which
can be a subse o he o iginal da a o a comple ely new da a se . Depending on he
choice o he da a used o in e p e a ion, he use can choose be ween a global o a
e y a ge ed local explana ion o he model. To p e en ex apola ion in he quan-
ile unc ion

F
−
1
l
(𝛼
)
,
𝛼
is limi ed o he in e al
[0, 1]
. Then, o a posi i e shi wi h
ql∈(0, 1]
, we de ine:
The modi ying alues
ql
o each
l∈M
a e se by he use . As his is a c ucial poin
o he me hod, in he ollowing we p o ide some examples and ecommenda ions
o a easonable choice o
q
.
The in e se o he ecd

F
−
1
l
does no necessa ily exis , as

Fl
ypically is no con-
inuous. Hence, o a posi i e shi we ha e o de ine
Equa ion(2) de e mines each alue o ea u e
l
a e he shi as one ou o he uly
obse ed alues o he espec i e ea u e, which we e used o es ima e he ecd .
Due o he de ini ion o he in e se o he ecd as de ined in Eq.(2), a posi i e
shi is gene ally no compa able o a nega i e shi , i i is done he exac same way.
While e en a sligh posi i e shi esul s in a change o he co esponding ea u e’s
alues, sligh nega i e shi s ypically change no hing a all.
In Fig.4, bo h a posi i e (le ) and nega i e ( igh ) shi a e shown o a ea u e
l
wi h alues
xl=(1, 2, 2, 3, 4, 4, 5)T
. Now, QSM is applied wi h
0
<
|
ql
|
<
1
7
. In
he ecd as de ined abo e, o he poin
xl=2
o example, we ha e

F
l(2)=
3
7
(bluea ow in he le pa o Fig.4). I now
ql
is added, his esul s in
𝛼
whe e
(1)

Fl(xi,l)=min{

Fl(xi,l)+ql,1}
⟹
x
i,l
=

F−1
l
(min{

F
l
(x
i,l
)+q
l
,1})
.
(2)

F−1
l
(𝛼)=in {x∶

F
l
(x)≥𝛼}
.
Fig. 4 Compa ison o posi i e (le ) and nega i e ( igh ) shi wi h he same
|ql|
402
A.Ge ha z e al.
3
7
<𝛼<
4
7
(blue a ow ). Due o he de ini ion o Eq. (2), he posi i e shi
esul s in

F
−1
l
(𝛼)=
3
(bluea ow ), which means ha a e y small posi i e shi
esul s in a change o he ea u e’s alue.
I an equally small, bu nega i e shi is used o he same alue
xl=2
, hen

F
l(2)=
3
7
( ed a ow in he igh pa o Fig. 4), which is he same as o
he posi i e shi . Nex , he amoun
|ql|
is sub ac ed, which esul s in
𝛼
whe e
2
7
<𝛼<
3
7
( ed a ow ). When he shi ed alue is calcula ed now, due o he
de ini ion om Eq.(2),

F
−1
l
(𝛼)=
2
( eda ow ), which means he alue has no
changed a all. In ac , in he p esen example no a single alue would change
when pe o ming he nega i e shi s, which shows ha nega i e and posi i e
shi s o he same absolu e amoun
q
a e no necessa ily compa able as conduc ed
igh now. Hence, a nega i e shi has o be de ined in ano he way.
A nega i e shi can be done by using a posi i e shi in he same manne as
be o e a e lipping he whole dis ibu ion o he co esponding ea u e. In pa -
icula , in a i s s ep a whole ea u e
l
is lipped by mul iplying all o i s alues
wi h
(−1)
. Nex , he ecd is cons uc ed o he lipped ea u e
l
and he absolu e
amoun o
ql
is added o i s alues. Las , he esul ing quan iles gi e he new al-
ues o he lipped ea u e
l
which ha e o be ans o med back o ge meaning-
ul alues on he o iginal scale. The e o e, o any
ql
om
ql∈ [−1, 0)
Eq.(1)
changes o
wi h

F−l(
⋅
)
as he empi ical cumula i e dis ibu ion unc ion o he lipped ea u e
l
and

F−1
−l
(⋅
)
being he co esponding quan ile unc ion. Wi h his small di e ence in
he p ocess o he shi s he shi sizes become compa able in bo h di ec ions.
Combining all equa ions om abo e, he shi s become well de ined as
Now,

xi
is he new shi ed obse a ion, which has he same alue o hose co a i-
a es om
L⧵M
as
xi
, bu di e en alues o he ea u es om
M
. These ea u es
we e inc eased o dec eased by he alue ha co esponded o he componen wise
aise o educ ion o he espec i e ecd by he amoun
ql
, no exceeding he mini-
mum o maximum o he empi ical dis ibu ion o he ea u es om
M
.
P incipally, his modus ope andi does no only wo k o me ic ea u es, bu
also o (o de ed o nominal) ca ego ical ea u es o he o m
xl∈{1, …,c}
,
whe e
c
is he numbe o ca ego ies. Fo his kind o ea u es, a shi om one
g oup o ano he has o be chosen manually, e.g., swi ching om g oup
o
ano he g oup
s
wi hin a speci ic ea u e
l
, i.e.,changing om
xl=
o
xl=s
.
Finally, o obse a ion
i,i=1, ..., n
,
M⊆L
and
q
=(q
1
, ..., q
p
)
T
(possi-
bly including ze os i
M⊂L
) le
Cq,M(xi)
de ine he pai o he o iginal and he
(po en ially) new class p edic ion esul ing om his shi , i.e.,
(3)

xi,l
=−

F
−1
−l
(min{

F
−l
(−x
i,l
)+
|
q
l|
,1})
,
xi,l=
{
F
−1
l(min{

Fl(xi,l)+ql,1}), o ql∈(0, 1]
−

F−1
−l
(min{

F
−l
(−x
i,l
)+
|
q
l|
,1}), o q
l
∈ [−1, 0)
.
409
Deducing neighbo hoods o classes oma i ed model
No e ha in Fig. 6, ea u es
x1
and
x2
we e in en ionally p esen ed wi hou a
scale. The in en ion is o illus a e by his a i icial da a example why using quan-
ile-based ea u e shi s can be ad an ageous compa ed o using plain ( a he a bi-
a y) numbe s. The main eason is ha i is o en ha d o de e mine wha size a
“sligh ” inc ease o dec ease migh be in he ypical case when he exac dis ibu-
ion o he ea u es is unknown. This is pa icula ly ele an when he ask is o ind
di ec neighbo hoods. Mo eo e , i many p edic o s a e p esen , a as and au o-
ma ic me hod o he compu a ion o he co esponding cho ddiag ams is essen-
ial, ins ead o de e mining o e e y ea u e and obse a ion manually, wha a sligh
shi migh be. Di e en ea u es usually ha e di e en scales and, depending on
hei loca ion in he ea u e space, a “sligh ” shi could ha e a di e en meaning
o di e en obse a ions, especially i a ea u e o in e es has a e y complex dis-
ibu ion (e.g., a mul imodal dis ibu ion). To a oid he p oblem o di e en scales,
i could be a good idea o de e mine he shi sizes based on he ea u e’s s and-
a d de ia ion, bu especially o ea u es wi h high-densi y and low-densi y a eas,
his s ill does no sol e he p oblem (see Fig.2). All o hese cases can be handled
by using small amoun s on he quan ile scale, which a e compa able o all me ic
ea u es.
3.1 Rema ks abou  heme hod’s in e p e a ion
In he ollowing, we gi e some impo an ema ks ega ding he in e p e a ion o he
esul s.
1. I we de ine he p e e ence o de
hen due o he ac ha pa icula ly complex models can p oduce also complex
pa i ions o he ea u e space, i ollows
This exp esses ha he esul s o QSM can no be in e p e ed ansi i ely. In
pa icula , a a he complex model could classi y a speci ic class spo wise in he
ea u e space, in which case one could ge esul s ha seem ansi i e, bu in ac
a e no ( o mo e de ails, see Sec .3.2).
2. I he classi ica ion me hod used o modeling is only designed o es ima e (lin-
ea ) mono onous e ec s, hen mig a ions be ween wo classes can only be ound
in one di ec ion, bu no he o he . I , ins ead, he unde lying me hod can also
es ima e mo e complica ed e ec s, hen mig a ions om a ce ain class A in o
ano he class B, bu also in he o he di ec ion, om B in o A, can be ound a he
same ime. This indica es ha he alues o he o iginal ea u e o in e es , bu
also he alues o he co a ia es, can be a deciding ac o on wha is happening i
he ea u e o in e es is inc eased. In hese cases, a local applica ion o QSM can
help o imp o e he unde s anding on wha migh happen i he ea u e o in e es
is inc eased.
A≻B∶= class A is di ec ly (o gene ally) nex o class B in he di ec ion o he shi ,
A≻B∧B≻C⇏A≻C.

410
A.Ge ha z e al.
3. As indica ed, QSM is buil o ind neighbo hoods as desc ibed by he model,
bu no o p oo ha he e is no neighbo hood be ween wo classes ega ding he
shi o
xM
. I he goal is o p oo ha a ce ain class has no di ec neighbo hoods
wi hin he i ed model, hen one would ha e o ill he comple e modeled space
o he class o in e es wi h da a poin s and hen had o shi
xM
wi h in ini ely
small s eps om he s a ing poin s un il he limi s o he ea u e ange. I one
also conside s he model o ex apola ion, hen hese in ini ely small s eps would
need o be done ei he un il
+∞
o
−∞
, espec i ely, in di ec ion o he shi , o
un il all poin s ha e swi ched classes.
4. The shi o
xM
could be oo big, such ha an in e media e class was skipped, and
consequen ly, no di ec neighbo hood was ound. Hence, he “neighbo hoods”
om abo e should be ega ded mo e gene ally as an “exis s abo e” (i he shi
was done by aising
xM
) o as an “exis s below” (i he shi was done by dec eas-
ing
xM
). To ind di ec neighbo hoods one would ha e o s a wi h a e y small
shi o
xM
and inc ease (o dec ease) i con inuously. In con as o his, i one has
a speci ic shi in mind, one could jus use his speci ic shi and hen he esul ing
mig a ion ma ix shows he co esponding class changes (i any).
5. I speci ic ea u es a e shi ed and a neighbo hood is ound be ween wo classes
wi h espec o he magni ude o he shi , his neighbo hood can be in e p e ed
as a neighbo hood be ween he o iginal class and he class a e he shi , i he
ask a hand is o ind ou , how he model desc ibes he neighbo hoods (o cou se,
keeping in mind ha an in e media e class could ha e been skipped). Bu i he
ask a hand is o ind ealis ic and p ac ical neighbo hoods be ween modeled
classes, hen hese neighbo hoods should always be in es iga ed in wo ways. I
a neighbo hood is ound by he in ended shi be ween wo classes, his means
ha he e exis da a poin s on one side nea he bo de be ween hese wo classes.
This does no necessa ily mean ha on he o he side o his bo de da a poin s
can also exis . I simila shi s a e ca ied ou in he opposi e di ec ion and his
neighbo hood is no con i med, hen his migh mean ha due o he shi impos-
sible ea u e combina ions ha e been c ea ed and, hus, he ound neighbo hood
has no p ac ical use. Conside , e.g., a classi ica ion model o he posi ion o
oo ball playe s con aining sho s on a ge and sco ed goals as ea u es. I he
playe had a single sho on a ge , which esul ed in a goal, hen inc easing he
amoun o goals leads o impossible da a poin s, bu a classi ica ion bo de , and
hus a neighbo , could s ill be ound i he model does ex apola ion in his a ea
o he ea u e space. A eason o his migh be ha he model simply ex apola es
in o his a ea o he ea u e space (gene al p oblem o ex apola ion, which can
lead o un easonable in e p e a ions; Hooke 2004).
6. In many cases, mul iple da a poin s could occu wi h equal alues o a possible
ea u e o in e es . I hose poin s a e di ec ly a a bo de be ween wo classes,
di e en p oblems can be obse ed, as shi ing all da a poin s changes he unde -
lying dis ibu ion no jus a he edges. Fo mos applica ions, his is ypically no
a p oblem, bu i he ac ual numbe o obse a ions changing class is in es iga ed,
hen ha ing ies migh lead o misleading esul s. In cases wi h ies, a bulk o da a
poin s ied in he ea u e o in e es could ind a neighbo hood, while i he da a
411
Deducing neighbo hoods o classes oma i ed model
poin s would no be ied, hen jus a small amoun o da a poin s would ind his
neighbo hood. A de ailed discussion can be ound in Sec .3.3.
7. Finally, a a he s aigh o wa d and undamen al ema k: I he model a hand
is a he bad and inapp op ia e, he ound neighbo hoods be ween he classes
a e co ec o desc ibing and unde s anding he model, bu would no e lec
he eali y. Hence, i is impo an o p ope ly e alua e he model i s , be o e i is
in e p e ed.
3.2 T ansi i e in e p e a ions
To show possible p oblems ega ding a ansi i e in e p e a ion o his me hod, he e
is a small a i icial da a example. In Fig.8, a ea u e space wi h wo ea u es
x1
and
x2
is shown. A model now labels mos o his ea u e space as class B, while a small
a ea wi h a low alue o bo h ea u es is labeled as class C and a small a ea wi h
high alues o bo h ea u es is labeled as class A. In addi ion o ha he e a e 10 ed
and g ay da a poin s, which a e used o desc ibe he pa i ion o he ea u e space
wi h QSM.
Now, he ea u e space pa i ion should be de e mined by choosing
q
x
1
=
1
11
and
q
x
2
=0
(keeping
x2
cons an ). Hence, only neighbo hoods wi h ega d o
x1
a e
looked o . Wi h 10 di e en da a poin s, his means ha holding
x2
cons an each
da a poin ge s assigned he nex la ge alue o
x1
con ained in he da a se . The da a
poin wi h he highes
x1
does no change, as i is al eady a he uppe bound o he
ange o
x1
. This shi esul s in he mig a ion ma ix shown in Table5.
In Fig.8, he wo poin s, which change hei p edic ion, a e ma ked in ed and
swi ch he modeled class h ough he dashed lines. These a e he wo poin s shown
Fig. 8 Example o ansi i i y p oblem in a 2-dimensional ea u e space
412
A.Ge ha z e al.
in he mig a ion ma ix in Table 5 as one has swi ched om class A o class B
and he o he one om class B o class C. As he e is an a ea o class B modeled
in he di ec ion o he shi abo e an a ea o class A, and hen he e is an a ea o
class C modeled in he di ec ion o he shi abo e an a ea o class B, based on
he co esponding mig a ion ma ix one could conclude ha in he di ec ion o he
ea u e shi he e is some kind o “hie a chy”. Pa icula ly, he e one migh conclude
ha along he di ec ion o his shi class A is below class B, which i sel is below
class C, bu as shown in Fig.8 his is no he case. E en i he dimensionali y would
be oo la ge o g aphically isualize i , jus by checking he espec i e ea u e o
in e es o he g oups sepa a ely would likely con i m he non- ansi i i y o his
example.
In pa icula , he espec i e cho d diag am as shown in Fig. 9 can (w ongly)
sugges his al eady men ioned kind o “hie a chy”. The cho ddiag am shows he
Table 5 Mig a ion ma ix
o QSM wi h he shi o
q
=(
1
11
,0
)
om Fig.8
Aa e
Ba e
Ca e
Abe o e
1 1 0
Bbe o e
0 5 1
Cbe o e
0 0 2
Fig. 9 Cho ddiag am o mig a ion ma ix in Table5
413
Deducing neighbo hoods o classes oma i ed model
mig a ion o one obse a ion om class A o class B and he mig a ion o one
obse a ion om class B o class C, which looks like a hie a chical s uc u e ha
ac ually does no exis . To conclude, QSM esul s should no be in e p e ed wi h
ega d o ansi i i y.
In his speci ic case, QSM jus desc ibes ha he e is in ac an a ea o class B
modeled in he di ec ion o he shi abo e an a ea o class A and he e is an a ea o
class C modeled in he di ec ion o he shi abo e an a ea o class B. Thus, wi h his
speci ic shi hese wo neighbo hoods we e ound. When using o he shi s in o he
di ec ions, hen o he neighbo hoods could be ound.
3.3 Mul iple da a poin s wi h hesame alue
When mul iple da a poin s wi h he same alue occu in a da a se and QSM is used,
some unexpec ed p oblems can occu . In he le g aph o Fig.10, he e a e 3 da a
poin s wi h an equal alue o
x1
. When choosing he shi size
q
x
1
=
1
n+1
, so he
smalles possible numbe ha allows he da a poin s o shi , hen all 3 da a poin s
would change hei p edic ions. In his case a neighbo hood be ween class A and
class B and ano he neighbo hood be ween class A and class C would be ound. As
shown in he igh g aph o Fig. 10, i he same h ee da a poin s would no be
exac ly equal bu jus sligh ly di e en , only one o hese da a poin s would change
i s p edic ion and he neighbo hood be ween class A and class B would no e en be
ound in his case, which is ano he p oblem.
I he shi size
q
x
1
=
2
n+1
is chosen, such ha e e y da a poin is shi ed by 2
unique alues o
x1
, hen bo h neighbo hoods would be ound e en in he si ua ion
o he igh g aph in Fig.10, when he da a poin s a e no exac ly equal, as one o he
da a poin s changes i s p edic ion om class A o class B.
Fo any con inuous (and andom) ea u e, he p obabili y o a speci ic alue is
ze o. Hence, mul iple da a poin s wi h he same co a ia e alue heo e ically should
ne e occu . Bu in eal wo ld applica ions, o example due o ounding, equal
co a ia e alues a e possible and could subs an ially change he in e p e a ion o
QSM (see Fig.10).
Fig. 10 QSM wi h
qx1
=
1
n+1
wi h some equal (le ) and sligh ly ji e ed ( igh ) da a poin s
414
A.Ge ha z e al.
I some obse a ions ha e exac ly he same alue o a ea u e o in e es (e.g.,
due o ounding), hen he ma ginal dis ibu ion o he co esponding ea u e ge s
changed subs an ially, which is no desi ed. To a oid his p oblem, he e a e some
ways o adjus QSM o ea obse a ions wi h equal alues mo e ai ly compa ed
o hose, which ha e unique alues o a speci ic ea u e.
1. Shi all ies: One possibili y is o shi all o he da a poin s ha sha e a speci ic
alue o a ea u e o in e es and obse e he changes in he p edic ions (le g aph
in Fig.10). In his example, his gua an ees ha all da a poin s a e shi ed, and
neighbo hoods can be ound mo e easily. As men ioned abo e, his me hod migh
lead o disp opo iona ely la ge shi s compa ed o obse a ions wi h unique al-
ues. Also, he ma ginal dis ibu ion o he ea u e o in e es ge s changed a lo by
his e y as , which is a high indica o o c ea ing imp obable co a ia e combina-
ions.
2. Repea edly shi ies andomly: Ano he al e na i e in he case o mul iple obse -
a ions wi h equal alues o a speci ic ea u e o in e es , is o epea edly shi
he obse a ions a e de e mining an a i icial andom o de . As shown in he
igh g aph o Fig.10, when he obse a ions we e jus sligh ly ji e ed, in his
example wo obse a ions we e changed by almos no amoun and one by a la ge
amoun . As a consequence, hese obse a ions a e ea ed mo e ai compa ed
o he o he obse a ions han in he shi all ies me hod, bu e y unequally
among hemsel es. Fo he a i icial example, he p edic ion o jus one o hese
h ee obse a ions changes. The epea edly shi ies andomly app oach does
exac ly ha , bu ins ead o ji e ing and hus adding some blu edness o he
da a, i epea edly execu es QSM and andomly de e mines an o de o he ied
obse a ions. While all obse a ions wi h unique alues o he ea u e o in e es
a e shi ed in he same way o all he epea ed shi s, he obse a ions sha ing
hei alue o he ea u e o in e es wi h o he obse a ions a e shi ed in jus a
ac ion o he epe i ions by he ull shi as in he shi all ies app oach. Hence,
obse a ions wi h equal alues o he speci ic ea u e o in e es a e in mos epe i-
ions shi ed less, and i |q| is a he small, hey migh no be shi ed a all. Thus,
in compa ison o he shi all ies me hod, his me hod ea s he obse a ions
wi h he same alues o he speci ic ea u e o in e es mo e ai compa ed o he
obse a ions, which ha e unique alues o his ea u e o in e es . Addi ionally,
he pa ame e q is mo e sensi i e as a smalle change o q has mo e impac on he
ou pu o he me hod. Thus, i can be used mo e lexible by he use .
The di e ences o hese me hods a e illus a ed in Sec .4 in a eal da a example.

415
Deducing neighbo hoods o classes oma i ed model
4 Real da a applica ion
In his sec ion, QSM is applied o a eal da a se in o de o p o ide some insigh
on he me hod’s po en ial. Fo his pu pose, a da a se o classi ying he loca ion
o p o eins in yeas is used. Fi s , he da a se is used o show how QSM should
be applied and how o in e p e he esul s. Nex , he wo p oposed app oaches o
handle ies om Sec .3.3 a e compa ed.
4.1 Applica ion andin e p e a ion o QSM
The chosen da a se is he yeas da a se (Ho on and Nakai 1996), which was
aken om he OpenML da abase (Vanscho en e al. 2013). The da a se con ains
p o eins om yeas and he a ge o he classi ica ion ask is o localize he p o-
ein in o one o en possible loca ions wi hin he yeas . The en loca ions a e u he
desc ibed by Ho on and Nakai (1996) and con ain cy oplasmic, including cy oskel-
e al (CYT); nuclea (NUC); acuola (VAC); mi ochond ial (MIT); pe oxisomal
(POX); ex acellula , including hose localized o he cell wall (EXC); p o eins
localized o he lumen o he endoplasma ic e iculum (ERL); memb ane p o eins
wi h a clea ed signal (ME1); memb ane p o eins wi h an unclea ed signal (ME2);
and memb ane p o eins wi h no N- e minal signal (ME3), whe e ME1, ME2 and
ME3 p o eins may be localized o he plasma memb ane, he endoplasma ic e icu-
lum memb ane o he memb ane o a Golgi body.
The da a se also con ains eigh di e en ea u es, which a e used o classi y he
a ge a iable. These ea u es a e also u he desc ibed in Ho on and Nakai (1996)
and in he UCI Reposi o y (Dua and G a 2017) and ep esen :
• mi : sco e o disc iminan analysis o he amino acid con en o he N- e minal
egion (20 esidues long) o mi ochond ial and non-mi ochond ial p o eins.
• e l: p esence o “HDEL” subs ing ( hough o ac as a signal o e en ion in he
endoplasmic e iculum lumen); bina y a ibu e.
• pox: pe oxisomal a ge ing signal in he C- e minus.
• ac: sco e o disc iminan analysis o he amino acid con en o acuola and
ex acellula p o eins.
• nuc: sco e o disc iminan analysis o nuclea localiza ion signals o nuclea and
non-nuclea p o eins.
• mcg: McGeoch’s me hod o signal sequence ecogni ion.
• g h: on Heijne’s me hod o signal sequence ecogni ion.
• alm: sco e o he ALOM memb ane spanning egion p edic ion p og am.
Fo his classi ica ion ask, a mul inomial eg ession model using he logi link unc-
ion is now compu ed wi h he nne package (Venables and Ripley 2002). When
using a mul inomial eg ession model we ob ain coe icien s as well as an in e cep
o e e y class o he a ge excep o he chosen e e ence class o e e y ea u e.
O e all, his esul s in 81 di e en coe icien s o his exempla y da a se .
416
A.Ge ha z e al.
As in e p e ing a mul inomial logis ic eg ession model can be e y complex,
he e exis echniques o help wi h he in e p e a ion. One me hod is he isuali-
za ion wi h E ec S a s (Tu z and Schaube ge 2012). E ec S a s condense he
po en ially huge amoun o coe icien s in o one e y clea ly a anged g aphic.
The aim is o gi e a nice o e iew o he coe icien s o all he a ge classes
in o one single s a plo pe ea u e. This o e iew s ill has i s limi a ions as he
coe icien s o e e y ea u e a e isualized wi hou conside ing he complex ela-
ions be ween all he ea u es including he in e cep . Ano he way o assis in
he in e p e a ion o a mul inomial logis ic eg ession model is he use o E ec
g aphics (Fox and Weisbe g, 2019; Fox and Hong, 2009). E ec g aphics dis-
play how he p edic ed p obabili ies o all he a ge classes change o a single
speci ic ea u e o a small amoun o ea u es a he same ime. In his p ocess,
all o he ea u es a e ixed o speci ic alues, while he ea u es o in e es a e
a ied o e hei ange o alues. This does no conside he complex s uc u e
be ween he ea u es as his p ocess can p oduce ea u e combina ions ha a e
e y unlikely o exis o a e e en impossible. Also, as he addi ional ea u es a e
ixed o speci ic alues, he g aphic only isualizes he beha io o he model in
a e y na ow ange o he ea u e space igno ing e e y hing a ound i . Conside
a 2-dimensional example as, e.g., in Sec .3. I he e ec o
x1
should be in es i-
ga ed, hen
x2
is ixed o i s mean (de aul ), and he e ec o
x1
is in es iga ed on a
hin ho izon al line in he middle o Fig.6. All he o he a eas a e no conside ed
o in es iga ing he e ec . Imbalanced a ge classes may also lead o p oblems
in he in e p e a ion o bo h me hods. E ec S a s a e isualized wi hou linkage
be ween all he ea u es, bu especially he di e en in e cep s o he imbalanced
a ge classes can ha e a huge impac on whe he a class has a high likelihood o
be p edic ed. The E ec isualiza ions may also sugges ha a speci ic small a -
ge class would no be p edic ed o any alue o he ea u e(s) o in e es , bu he
cause o his could be he ixed alues o he emaining ea u es.
QSM has he ad an age ha he in e dependence be ween he ea u es is con-
side ed as i uses eal da a poin s. Also, he imbalance o classes does no a ec
QSM, as eal da a poin s wi h ea u e combina ions o he smalle classes a e
used ins ead o one ixed combina ion like in he E ec g aphics. Fo he mul i-
nomial logis ic eg ession model, QSM can help o isualize which classes will
ealis ically be p edic ed i a speci ic ea u e o a small amoun o ea u es o he
empi ical da a would ha e sligh ly la ge o lowe alues based on he unde lying
model.
In he ollowing, QSM is applied o his mul inomial model o lea n which a -
ge classes a e modeled closely and simila o each o he wi h sligh di e ences
conce ning a speci ic ea u e. Fo exempla y pu pose, we choose o use jus he
single ea u e
mi
, bu also o he single ea u es o e en mul iple ea u es a he
same ime could be used. He e, i is in es iga ed how he p edic ed a ge class
would change i he compu ed
mi
sco e would be sligh ly la ge . This helps o
g asp he ela ions be ween he p edic ed a ge classes ega ding his
mi
sco e.
An impo an eason why his da a se is used he e is ha i con ains a lo o
obse a ions wi h equal ea u e alues. E en hough he da a se con ains
n=1484
obse a ions,
xmi
con ains jus 78 di e en alues. Fo he i ed model,
417
Deducing neighbo hoods o classes oma i ed model
he quan ile shi size o
q
mi =
75
1484+1
≈
0.05
is chosen and
(
q
mcg
,q
g h
,q
alm
,q
e l
,q
pox
,q
ac
,q
nuc
)
⊤
=
0
. This means ha all ea u e alues o
xmi
a e inc eased by a sligh amoun . By his choice we simula e a da a se , whe e he
sco e o he espec i e disc iminan analysis unde lying he mi alue would ha e
been sligh ly la ge .
The ma ginal dis ibu ion o he ea u e
xmi
is shown in Fig.11. This indica es
ha la ge alues migh lead o a classi ica ion o a ge class MIT. Recall ha
ea u e
xmi
is ep esen ing a sco e o a disc iminan analysis o he amino acid
con en o he N- e minal egion (20 esidues long) o di ide mi ochond ial and
non-mi ochond ial p o eins. The coe icien s o he mul inomial model can be
used o check i acco ding o he model a la ge alue o
xmi
does indeed esul
in a la ge p obabili y o being p edic ed as a ge class MIT. QSM can now add
he in o ma ion, which o he a ge classes migh be simila o he a ge class
MIT ega ding he unchanged o he ea u es, bu wi h a lowe alue o
xmi
acco ding o he model. O cou se, depending on he s uc u e o he da a se and
he complexi y o he model, e en he a ge class MIT migh ha e ano he a ge
class close o i in which obse a ions migh mig a e i
xmi
is inc eased. Also, i
is globally in es iga ed i o he a ge classes a e modeled simila o each o he
ega ding he o he ea u es, bu wi h sligh ly di e en alues in
xmi
. Two a ge
classes ha a e ound o be simila in he desc ibed manne a e wha we e e o
as a neighbo s.
Fig. 11 Ma ginal dis ibu ions o
xmi
o each o he a ge classes
418
A.Ge ha z e al.
In Table 6, i is shown ha by using he shi all ies app oach a lo o
obse a ions change hei p edic ed class and indica e neighbo hoods. These
neighbo hoods a e isualized in he cho ddiag ams in Figs.12 and 13. We ind
ha 21 obse a ions change hei p edic ion om he a ge class CYT o MIT.
This means ha o 21 o he p o eins he model p edic s hem o be mi ochond ial
Table 6 Mig a ion ma ix o p edic ion changes when aising
xmi
by
q
mi =
75
1485
wi h he shi all ies
app oach; he obse ed class changes a e in bold on
CYT ERL EXC ME1 ME2 ME3 MIT NUC POX VAC
CYT 608 0 0 0 0 0 21 1 0 0
ERL 0 5 0 0 0 0 0 0 0 0
EXC 0 0 27 0 0 0 40 0 0
ME1 0 0 0 42 0 0 0 0 0 0
ME2 0 0 0 0 41 0 60 0 0
ME3 0 0 0 0 2170 20 0 0
MIT 0 0 0 0 0 0 227 0 0 0
NUC 0 0 0 0 0 0 21 293 0 0
POX 0 0 0 0 0 0 0 0 13 0
VAC 0 0 0 0 0 0 10 0 0
Fig. 12 Changes in p edic ion when inc easing
xmi
wi h
q
mi =
75
1485
; all obse a ions po ayed
425
Deducing neighbo hoods o classes oma i ed model
Decla a ions
Con lic o in e es The au ho s decla e he e is no con lic o in e es . The code used o he examples can
be ound in a public gi hub eposi o y: h ps://gi hub.com/habbeda1/QSM.
Open Access This a icle is licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License,
which pe mi s use, sha ing, adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as long as
you gi e app op ia e c edi o he o iginal au ho (s) and he sou ce, p o ide a link o he C ea i e Com-
mons licence, and indica e i changes we e made. The images o o he hi d pa y ma e ial in his a icle
a e included in he a icle’s C ea i e Commons licence, unless indica ed o he wise in a c edi line o he
ma e ial. I ma e ial is no included in he a icle’s C ea i e Commons licence and you in ended use is
no pe mi ed by s a u o y egula ion o exceeds he pe mi ed use, you will need o ob ain pe mission
di ec ly om he copy igh holde . To iew a copy o his licence, isi h p:// c ea i eco mmons. o g/ licen
ses/ by/4. 0/.
Re e ences
B eiman, L.: Random o es s. Mach. Lea n. 45(1), 5–32 (2001)
Dua, D., G a , C.: UCI machine lea ning eposi o y (2017)
Fox, J., Hong, J.: E ec displays in o mul inomial and p opo ional-odds logi models: ex ensions
o he e ec s package. J. S a . So w. 32(1), 1–24 (2009)
Fox, J., Weisbe g, S.: An R Companion o Applied Reg ession, 3 d edn. Sage, Thousand Oaks (2019)
F iedman, J.H.: G eedy unc ion app oxima ion: a g adien boos ing machine. Ann. S a . 29(5), 1189–
1232 (2001)
Golds ein, A., Kapelne , A., Bleich, J., Pi kin, E.: Peeking inside he black box: isualizing s a is ical
lea ning wi h plo s o indi idual condi ional expec a ion. J. Compu . G aph. S a . 24(1), 44–65
(2015)
Gu, Z., Gu, L., Eils, R., Schlesne , M., B o s, B.: ci clize implemen s and enhances ci cula isualiza ion
in . Bioin o ma ics 30, 2811–2812 (2014)
Hooke , G.: Diagnos ics and ex apola ion in machine lea ning. Ph.D. hesis, S an o d, CA, USA (2004)
Ho on, P., Nakai, K.: A p obablis ic classi ica ion sys em o p edic ing he cellula localiza ion si es o
p o eins. In ell. Sys . Mol. Biol. 8, 109–115 (1996)
Kelle , F., Mulle , E., Bohm, K.: HiCS: high con as subspaces o densi y-based ou lie anking. IEEE
(2012)
Molna , C.: In e p e able machine lea ning: a guide o making black box models explainable (2019)
R Co e Team.: R: A Language and En i onmen o S a is ical Compu ing. R Founda ion o S a is ical
Compu ing, Vienna (2021)
Ribei o, M.T., Singh, S., Gues in, C.: Ancho s: high-p ecision model-agnos ic explana ions. In:
P oceedings o he AAAI Con e ence on A i icial In elligence, ol. 32(1) (2018)
Tu z, G., Schaube ge , G.: Visualiza ion o ca ego ical esponse models: om da a glyphs o pa ame e
glyphs. J. Compu . G aph. S a . 22(1), 156–177 (2012)
Vanscho en, J., an Rijn, J.N., Bischl, B., To go, L.: Openml: ne wo ked science in machine lea ning.
SIGKDD Explo . 15(2), 49–60 (2013)
Venables, W.N., Ripley, B.D.: Mode n Applied S a is ics wi h S, 4 h edn. Sp inge , New Yo k (2002)
Publishe ’s No e Sp inge Na u e emains neu al wi h ega d o ju isdic ional claims in published maps
and ins i u ional a ilia ions.