G eybox XAI: a Neu al-Symbolic lea ning amewo k o p oduce
in e p e able p edic ions o image classi ica ion
Ad ien Benne o a,b,c, Gianni F anchia, Ja ie Del Se d,e, Raja Cha ilac, Na alia D´
ıaz-Rod ´
ıguez
aU2IS, ENSTA, Ins i u Poly echnique Pa is and In ia Flowe s, 91762, Palaiseau, F ance
bSegula Technologies, Pa c d’ac i i ´
e de Pissaloup, T appes, F ance
cSo bonne Uni e si ´
e, Pa is, F ance
dTECNALIA, Basque Resea ch and Technology Alliance (BRTA), 48160 De io, Bizkaia, Spain
eUni e si y o he Basque Coun y (UPV/EHU), 48013 Bilbao, Spain
Andalusian Resea ch Ins i u e in Da a Science and Compu a ional In elligence (DaSCI), Uni e si y o G anada, Spain
Abs ac
Al hough Deep Neu al Ne wo ks (DNNs) ha e g ea gene aliza ion and p edic ion capabili ies, hei
unc ioning does no allow a de ailed explana ion o hei beha io . Opaque deep lea ning models a e
inc easingly used o make impo an p edic ions in c i ical en i onmen s, and he dange is ha hey make
and use p edic ions ha canno be jus i ied o legi imized. Se e al eXplainable A i icial In elligence (XAI)
me hods ha sepa a e explana ions om machine lea ning models ha e eme ged, bu ha e sho comings
in ai h ulness o he model ac ual unc ioning and obus ness. As a esul , he e is a widesp ead ag eemen
on he impo ance o endowing Deep Lea ning models wi h explana o y capabili ies so ha hey can
hemsel es p o ide an answe o why a pa icula p edic ion was made. Fi s , we add ess he p oblem
o he lack o uni e sal c i e ia o XAI by o malizing wha an explana ion is. We also in oduced a
se o axioms and de ini ions o cla i y XAI om a ma hema ical pe spec i e. Finally, we p esen he
G eybox XAI, a amewo k ha composes a DNN and a anspa en model hanks o he use o a symbolic
Knowledge Base (KB). We ex ac a KB om he da ase and use i o ain a anspa en model (i.e., a
logis ic eg ession). An encode -decode a chi ec u e is ained on RGB images o p oduce an ou pu
simila o he KB used by he anspa en model. Once he wo models a e ained independen ly, hey
a e used composi ionally o o m an explainable p edic i e model. We show how his new a chi ec u e is
accu a e and explainable in se e al da ase s.
Keywo ds:
Explainable A i icial In elligence, Compu e Vision, Deep Lea ning, Pa -based Objec Classi ica ion,
Composi ional models, Neu al-symbolic lea ning and easoning
1. In oduc ion
Deep Neu al Ne wo ks (DNNs), conside ed as black-boxes, a e being inc easingly used o make
impo an p edic ions in c i ical con ex s. A he same ime, he demand o anspa ency om he a ious
s akeholde s in AI [
1
] is also g owing. The dange o using black box models would be o c ea e and
apply decisions ha a e no explainable and could no be jus i ied [
2
]. Common needs o equi ing
in e p e abili y in Deep Lea ning a e he need o e hics [
3
], sa e y when using AI in c i ical se ings [
4
]
and he need o enable he end use o us he sys em [
5
]. In e p e abili y can be de ined as he abili y o
p oduce explana ions o he model’s beha io ha can be unde s ood by a human use [6].
In [
7
], Mille highligh ed ou majo indings abou explana ions. A p e equisi e o a ”good” expla-
na ion is ha i does no only indica e why he model made a ce ain decision, bu also why i made his
decision a he han ano he . This e e s o he abili y o p oduce coun e ac uals. In addi ion, ”good”
P ep in submi ed o Else ie Sep embe 30, 2022
a Xi :2209.14974 1 [cs.CV] 26 Sep 2022
explana ions a e selec i e, meaning ha ocusing solely on he main causes o a decision-making p ocess
is su icien . Fu he mo e, i a causal explana ion o he gene aliza ion i sel is no p o ided, u ilizing
s a is ical gene aliza ions o explain why occu ences occu is insu icien . Finally, explana ions a e social,
meaning ha hey a e a ans e o knowledge be ween an explaine and an explainee.
Deep Lea ning models su e om wo kind o bias. The i s one is a lea ning bias, due o he p esence
o a bias in he aining da a se . This can happen when associa ions o concep s a e o e - o unde -
ep esen ed in he aining se . Fo example he e we e cases o a da ase wi h women unde - ep esen ed
in o ices compa ed o men, leading a cap ioning algo i hm o assume ha a pe son in an o ice was
necessa ily a man while i could also be a woman. One o he applica ions o XAI is highligh ing his bias
o co ec i [
8
]. The second ype o bias is he human induced one, when using common sense knowledge
abou he wo ld o explain he ou pu o a DNN o when using pa icula pa ame e s, a chi ec u es o loss
unc ions o model a p oblem [9].
A la ge numbe o me hods o model p obing ha e eme ged in ecen yea s. Some ha e he ad an age
o being model-agnos ic, i.e. sepa a ing he explana ion om he machine lea ning model. This has
he ad an age o p o iding lexibili y o he use as ools a e a ailable o ex ac explana o y elemen s
om each model [
10
]. Some o hese me hods, he bes known o which a e LIME [
11
] and SHAP [
12
],
a e based on he use o su oga e models. These p oxy models will locally mimic he beha iou o he
black-box in o de o explain indi idual p edic ions. While his has he ad an age o being easy o use,
he e a e p oblems o obus ness [
13
,
14
]. Mo eo e , i is no possible o ha e a global iew o he model’s
beha iou since he explana ion is local.
In image ecogni ion, ano he amily o me hods widely used is based on isualiza ion. They exp ess
an explana ion by highligh ing cha ac e is ics o he image ha objec i ely in luence he ou pu o a DNN
[
15
]. The bes known o hem, G ad-CAM [
16
], c ea es class ac i a ion map using he g adien s o he
DNN’s ou pu wi h espec o he las con olu ional laye . This p o ides a isual explana ion easy o
unde s and as i allows ecognizing he impo an egions o he image. Howe e i is di icul o know
whe he an explana ion is alid, in he sense ha a human non-expe in he ield does no necessa ily know
wha he impo an poin s o an image a e, and a pa o he e alua ion is subjec i e. Fu he mo e, i has
been shown ha some o he mos used me hods a e insensi i e o model and da a [
17
]. In addi ion, he e
is also a isk o in oducing a human-induced bias when a use is ying o in e p e he isual explana ion.
His o he unde s anding would depend on his o he own backg ound knowledge. Thus, i is necessa y
ha he explana o y elemen s o an AI model come di ec ly om he da a seen by he ne wo k, o i o be
ai h ul wi h espec o wha i ac ually lea ned [18].
One o he goals o ha ing in e p e abili y in a model is o explain i s easoning by exp essing i in
a way ha is unde s andable and eadable by human beings, while highligh ing he biases lea ned by
he model, in o de o alida e o in alida e i s decision a ionale [
19
]. The e is a ade-o be ween he
pe o mance o a model and i s anspa ency [
20
] bu i is also possible o conside ha he ad ocacy
o in e p e abili y may lead o a gene ic pe o mance imp o emen o 3 easons: i) i will help ensu e
impa iali y in decision-making, i.e. o highligh , and consequen ly, co ec om bias in he aining
da a-se , ii) in e p e abili y acili a es he p o ision o obus ness by highligh ing po en ial ad e sa ial
pe u ba ions ha could change he p edic ion, and inally, iii) in e p e abili y can ac as an insu ance
ha only meaning ul a iables in e he ou pu , i.e., gua an eeing ha an unde lying u h ul causali y
exis s in he model easoning. Combining he p edic ion capabili ies o connec ionis models wi h he
anspa ency o symbolic ones could pu aside he ade-o by inc easing ei he he in e p e abili y o he
pe o mance o AI models, he challenge being o inc ease one wi hou sac i icing oo much o he o he .
I has been p o en ha using backg ound knowledge wi hin a DNN can b ing obus ness o he lea ning
sys em [
21
,
22
,
23
]. The use o a Knowledge Base o lea n and eason wi h symbolic ep esen a ions has
he ad an age o p omo ing he p oduc ion o explana ions while making a p edic ion [
24
]. The abili y o
e e o es ablished easoning ules allows symbolic me hods o ul ill his p ope y.
In o de o ob ain a model ha mee s he abo e c i e ia, we in oduce he G eybox XAI amewo k.
This new a chi ec u e is anspa en by design when used o an image classi ica ion ask. I combines an
encode -decode used o he c ea ion o an Explainable La en Space which is hen used by a logis ic
eg ession. The Explainable La en Space allows knowing o which easons an image has been classi ied
in a ce ain way wi h he help o logis ic eg ession. Mo eo e , we p opose a o maliza ion o he no ion o
explana ion and we pose de ini ions allowing o judge i s quali y.
The con ibu ion o ou pape is h ee old:
• A heo y o explainabili y o deep lea ning models o quali y wha is a ”good” explana ion.
• An explainable by design composi ional amewo k called G eybox XAI.
•
We show ha his new amewo k p o ides s a e-o - he-a esul s on an image classi ica ion ask
ega ding he explainabili y/accu acy ade-o in a ious da ase s, as i s accu acy is close o he
exis ing models while being mo e explainable.
This pape is o ganized as ollows: i s we p esen he li e a u e a ound XAI and pa -based classi ie s
in Sec ion 2. We p esen he di e en no ions and e minology used in XAI and p opose ou de ini ions in
Sec ion 3. We desc ibe ou amewo k in Sec ion 4and we illus a e i s use by expe imen s on se e al
da ase s in Sec ion 5.
2. Rela ed Wo k: Explainable AI o maliza ion, composi ional pa -based classi ica ion and neu al-
symbolic compu a ion.
The li e a u e [
6
,
25
,
26
] dis inguishes Deep Lea ning’s XAI me hods in o wo ca ego ies: anspa en
models and opaque models ha need o be explained hanks o pos -hoc me hods. As ou model is a
composi ion o a anspa en model and an opaque model, we will pu a pa icula ocus on composi ional
models in Sec ion 2.1. We will also alk abou he use o KBs o XAI in Sec ion 2.2.
2.1. Composi ional Pa -based Classi ica ion Models
Composi ionali y in compu e ision e e s o he abili y o ep esen complex concep s by combining
simple pa s [
27
,
28
]. Composi ionali y is a desi able p ope y o CNNs as i can imp o e gene aliza ion
by encou aging ne wo ks o o m ep esen a ions ha disen angle he p edic ion o objec s om hei
su oundings and om each o he [
29
]. Fo example, handw i en symbols can be lea ned om only a ew
examples using a composi ional ep esen a ion o s okes [
30
]. The composi ionali y o neu al ne wo ks is
also seen as key o he in eg a ion o symbolism and connec ionism [31,32].
Pa -based objec ecogni ion is an example o seman ic composi ionali y and a classical pa adigm
whe e he idea is o collec in o ma ion a he local le el in o de o make a global classi ica ion. In [
33
],
he au ho s p opose a pipeline ha i s g oups pixels in o supe pixels, hen pe o ms a supe pixel-le el
segmen a ion, con e s his segmen a ion in o a ea u e ec o , and inally classi ies he global image
hanks o his ea u e ec o . A simila me hod is p oposed by [
34
], ex ending i o 3D da a. He e, he
idea is o classi y a pa o he image in o a p ede ined class and hen use hese in e media e p edic ions
o c ea e a classi ica ion o he whole image. The au ho s o [
35
] also de ine in e media e-le el ea u es
ha cap u e local s uc u es such as e ical o ho izon al edges, hai il e s, and so on. Howe e , hey a e
close o dic iona y lea ning han o he app oach we p opose in his pape .
One o he bes known models o objec pa ecogni ion is [
36
]. I p o ides objec ecogni ion
based on mix u es o de o mable pa models wi h mul iple scales based on da a mining o ha d nega i e
examples wi h pa ially labeled da a o ain a la en SVM. The e alua ion is pe o med in he PASCAL
objec de ec ion challenge (PASCAL VOC benchma k [37]).
Semi-supe ised me hods ha e been de eloped mo e ecen ly, such as [
38
]. They p opose a wo-s age
neu al a chi ec u e o ine-g ained image classi ica ion suppo ed by local de ec ions. The idea is ha
posi i e p oposal egions highligh di e en complemen a y in o ma ion and ha all o his in o ma ion
should be used. Fo his pu pose, an unsupe ised ecogni ion model is i s buil by al e na ely applying a
CRF and a Mask-RCNN (conside ing an ini ial app oxima ion wi h CAM). Then, he ecogni ion model
and he posi i e egion p oposal a e ed o a bidi ec ional LSTM, which gene a es a meaning ul ea u e
ec o ha collec s in o ma ion abou all egions and is hen able o classi y he image. This can be
conside ed as unsupe ised pa -based classi ica ion.
As in oduced in [
39
], he e is a need o causabili y in ce ain domains such as in he medical ield
o example. Causabili y is he measu able ex en o which an explana ion o a human expe achie es a
speci ied le el o causal unde s anding [
40
]. This no ion e e s o usabili y and mus no be con used wi h
causali y. The la e is he ela ionship be ween cause and e ec [
41
]. Causabili y can be measu ed wi h
he Sys em Causabili y Scale, a sys em o measu e he quali y o explana ions based on causabili y and
usabili y [42].
Cu en me hods o image classi ica ion wi h DNNs is o add a en ion mechanisms in o he lea ning
p ocess o au oma ically ex ac ele an ea u es o a gi en inpu . This mechanism is designed o ocus
DNNs on he mos impo an ea u es o a gi en classi ica ion ask.[
43
]. T ans o me a chi ec u e was i s
in oduced o machine ansla ion bu he compu e ision communi y is wo king on hei implemen a ion
o compu e ision [
44
,
45
,
46
]. A en ion-based neu al ne wo ks such as he Vision T ans o me (ViT)
ha e ecen ly a ained s a e-o - he-a esul s, in e m o accu acy, on many compu e ision benchma ks
[
47
,
48
]. Many esea ches ha e eme ged o imp o e hese ans o me s o compu e ision, no ably
wo king on echniques o mo e e icien ly scale he size o ision ans o me s[
49
], o make hem mo e
ac able o in e ence [
50
] o o imp o e gene aliza ion o domain shi [
51
]. By adding human isual
a en ion maps a e as an inpu o a DNN, i has been shown ha a en ion mechanisms in DNNs wo k as
human isual a en ion[52].
Finally, [
53
] p oposes a me hodology designed o lea n bo h symbolic and deep ep esen a ions. I
in ol es a composi ional con olu ional neu al ne wo k ha makes use o symbolic ep esen a ions called
EXPLANe and SHAP-Backp op, an explainable AI-in o med aining p ocedu e ha co ec s and guides
he DL p ocess o align wi h such symbolic ep esen a ions in o m o knowledge g aphs. To he bes o
ou knowledge, his model ep esen s he s a e o he a in e ms o composi ional lea ning models.
2.2. The use o Knowledge base o Explainable AI
The use o backg ound knowledge in he o m o logical s a emen s in KBs has shown o no only
imp o e explainabili y bu also pe o mance wi h espec o pu ely da a-d i en app oaches [
21
,
23
]. A
posi i e side e ec shown is ha his hyb id app oach p o ides obus ness o he lea ning sys em when
e o s a e p esen in he aining da a labels. O he app oaches ha e shown o be able o join ly lea n and
eason wi h bo h symbolic and sub-symbolic ep esen a ions and in e ence [
54
]. The in e es ing aspec
is ha his blend allows o exp essi e p obabilis ic-logical easoning in an end- o-end ashion [
55
]. An
example o use case is on die a y ecommenda ions, whe e explana ions a e ex ac ed om he easoning
behind (non deep bu KB-based) models [24].
A di e en pe spec i e on hyb id XAI models consis s o en iching black-box models knowledge
wi h ha one o anspa en ones, as p oposed in [
9
] and u he e ined in [
18
]. I allows he ne wo k
o exp ess wha is con iden o con used abou , in a con ex ha helps o ackle bias. O he examples o
hyb id symbolic and sub-symbolic me hods whe e a knowledge-based ool o g aph-pe spec i e enhances
he neu al (e.g., language [56]) model a e in [57,58].
Ano he hyb id app oach consis s o mapping an unin e p e able black-box sys em o a whi e-box win
ha is mo e in e p e able. Fo example, an opaque A i icial Neu al Ne wo k (ANN) can be combined
wi h a anspa en Case Based Reasoning (CBR) sys em [
59
,
60
]. In [
61
], he ANN (in his case a DNN)
and he CBR (in his case a k-NN) a e pai ed in o de o imp o e in e p e abili y while keeping he same
accu acy. The explana ion by example consis s o analyzing he ea u e weigh s o he ANN which a e
hen used in he CBR, in o de o e ie e nea es -neighbo cases o explain he ANN’s p edic ion.
Desc ip ion Logics [
62
] ha e success ully been used o enhancing deep lea ning models o image
in e p e a ion h ough he use o knowledge bases [
63
]. I can also help de ec inconsis encies in au oma ed
knowledge ep esen a ion and easoning. An example o au oma ed symbol design and in e p e a ion is in
[
64
]. Some XAI sys ems conside coun e ac ual ule lea ning and causal signal ex ac ions. Examples o
ule lea ning app oaches can include lea ning om noisy o uns uc u ed da a, o lea ning wi h cons ain s
[65].
As i seems in ui i e ha he p esence o a KB is use ul o p o ide an explana ion, how o use i o
image classi ica ion is no ob ious because KBs use a e y conc e e o malism which is in opposi ion o
he abs ac ea u es used by ne wo ks. Some me hods such as Logic Tenso Ne wo ks [
22
] o LYRICS, a
Gene al In e ace Laye o In eg a e AI and Deep Lea ning [
66
] show p omising esul s bu we did no
ind any execu able implemen a ion example o hei end- o-end use in he li e a u e.
3. XAI de ini ions and o maliza ion
We i s es ablish a common poin o unde s anding on wha he di e en e ms s ands o in he con ex
o XAI, based on [
6
]. We ecall he de ini ion o anspa ency and p opose a ious c i e ia o judge he
quali y o an explana ion. De ini ions o subsec ions 3.1 and 3.2 a e no con ibu ions bu a necessa y s ep
o p ope ly unde s and ou p oposed G eybox XAI amewo k.
3.1. De ini ion o anspa ency
T anspa ency e e s o a passi e p ope y o a model, which e e s o he le el a which a gi en model
”makes sense” o a human obse e . A model is conside ed opaque when i is no anspa en . The e a e 3
deg ees o anspa ency, om he leas o he mos anspa en [67]:
•
Algo i hmic T anspa ency. I deals wi h he abili y o he use o unde s and he p ocess ollowed by
he model o p oduce any gi en ou pu om i s inpu da a.
•
Decomposabili y. A model wi h algo i hmic anspa ency is decomposable i e e y pa o he model
is unde s andable by a human wi hou he need o addi ional ools. I means ha i is possible o
explain each o he pa s o a model (inpu , pa ame e and calcula ion). I equi es e e y inpu o be
in e p e able.
•
Simula abili y. I deno es he abili y o a model o be o ally simula ed by a human, meaning ha i s
complexi y is low. A decomposable model is he e o e simula able i i sel -con ained enough o a
human o hink and eason abou i as a whole.
To conclude, hese le els o anspa ency depend on he unde s andabili y o he model. The unde -
s andabili y o a model is i s abili y o make a human unde s and i s in e nal s uc u e o he algo i hmic
means by which he model p ocesses da a in e nally [
68
]. The ype o models conside ed as anspa en
a e Linea and Logis ic Reg ession [
69
], Decision T ees [
70
], K-Nea es Neighbo s [
71
], Rule Based
Lea ne s [
72
], Gene al Addi i e Models [
73
] and Bayesian Models [
74
]. These 6 ypes o models a e
always algo i hmically anspa en . They can each highe le els o in e p e abili y i he a iables a e
unde s andable and no oo nume ous, i he e a e no oo many ules, e c.
3.2. Cla i ying he concep o Explainabili y
As s a ed in [
6
], explainabili y can be conside ed as an ac i e cha ac e is ic o a model, which e e s o
any ac ion o p ocedu e implemen ed in o de o cla i y i s in e nal unc ions. The explainabili y o a model
hus deno es i s capaci y o p oduce an explana ion. In o de o make a non- anspa en model explainable,
many pos -hoc me hods we e designed. Pos -hoc me hods a e used on a model a e i s aining and a e
designed o p obe he model in o de o imp o e i s explainabili y.

While mos o he di e en e ms used in explainable AI ha e been widely deba ed in he li e a u e, he
one abou Wha cons i u es an explana ion has no been widely ma hema ized. We p opose a o maliza ion
o he no ion o explana ion, inspi ed by [
75
] in o de o es ablish objec i e c i e ia o a i m ha an
explana ion is ”good” o no . Whe he i comes om he anspa ency o a model o om a pos -hoc
me hod applied on an opaque model, an explana ion mus mee ce ain indispensable cha ac e is ics o be
conside ed as a ”good” explana ion.
3.3. Explana ion Fo malisa ion o an image classi ica ion p oblem
Le us deno e
E={(e)|e∈ {0,1}∗}
a se o explana ions
e
. This se is bina y in o de o be able o
encode any ype o communica ion because an explana ion can be gi en in a ious o ms: a ex , an image,
a g aph, e c.
De ini ion 3.1.
Le
:X → Y
be a classi ica ion model wi h
X
he inpu space and
Y
he label space.
The explana ion unc ion Φon a (x)p edic ion wi h x∈ X is de ined by:
Φ : Y → E(1)
(x)→Φ( (x)) (2)
On he basis o his de ini ion o an explana ion unc ion, we de ine se e al axioms cha ac e izing an
explana ion. These axioms in end o o mally quali y wha a ”good” explana ion is, ollowing desi ed
p ope ies. Based on he li e a u e [
6
,
9
,
7
] and he abo e de ini ions, we highligh 3 p ope ies ha we
conside necessa y o ob ain a ”good” explana ion:
1. Objec i i y
. Explainabili y is i s ly abou humans, as i e e s o he de ails and easons a model
elici s o make i s unc ioning clea o easy o unde s and gi en a ce ain audience [
6
]. The
poin o p oducing explana ions ha a e as objec i e as possible is o minimize he amoun o
subjec i i y a human migh ha e when in e p e ing he explana ions. Designing explana ions based
on symbols/concep s ha a e known and ela ed o he ask a hand allows o an unbiased explana ion
ha can be unde s ood in he same way by wo di e en use s wi h he same backg ound knowledge.
In o de o an explana ion o be mo e objec i e, we conside ha i mus be exp essed in such a way
ha i is unde s ood in he same way by he majo i y o membe s o a gi en audience. The eal wo ld
con ains objec s and we wan compac ep esen a ions o hose objec s [
76
]. We assume his can be
ob ained in an explana ion by he use o logical seman ics, using symbols and ela ions ha can be
concep ualized by he human use o he explana ion. This implies he use o on ologies, speci ying
wha indi iduals ( hings, objec s) and ela ionships a e assumed o exis and wha e minology is
used o hem.
De ini ion 3.2.
An explana ion
e
o a classi ica ion model
:X → Y
is said o be objec i e i
e
does con ain symbols and/o ela ionships.
Example 1. Objec i i y
Figu e 1shows he example o one explana ion making use o symbols
and/o ela ionships and ano he one no making use o hem. The less subjec i e explana ion mini-
mizes he amoun o in e p e a ion le o he explainee because i uses symbols (wo ds) commonly
employed o ep esen objec s. Mo eo e he subjec i e explana ion based on a isualiza ion o he
a en ion a eas o he model lea es a lo o he explainee’s in e p e a ion. F om one use o ano he ,
some will say ha he ho a ea is he head o he abbi while o he s will alk abou he colo o i s
muzzle, he ca o s and i is holding o i s eyes.
Figu e 1: Subjec i e and less subjec i e/mo e objec i e example o explana ions. The subjec i e explana ion is a supe imposed
isualiza ion o G ad-CAM’s hea map and he inpu image, showing he model mos ly used he cen e - igh o he image (whe e he
head o he abbi is) in o de o make i s p edic ion. The mo e objec i e explana ion is a ex ual explana ion using a ibu es de ec ed
on he abbi o ca ego ize and desc ibe i .
2. In insicali y
. The comple e explana ion o a p edic ion should come di ec ly om he model (o i s
in insic elemen s) ha p oduced he p edic ion. In o de o he explana ion o be o ally ai h ul o
wha happened in he model, i is necessa y ha only he inpu s, pa ame e s and ope a ions p esen
in he model ha we a e ying o explain a e used. This is essen ial o ensu e ha he explana ion
ha is gi en is wha ac ually happened in he model du ing i s in e ence a he han he expec ed
beha io . The alue o ha ing an in insic explana ion is o be su e ha he explana ion exac ly
desc ibes how he model wo ks, a he han an app oxima e o desi ed ope a ion. As a ma e o ac ,
i he explana ion depends on some hing ha is no ela ed o he model we wish o explain, i is
impossible o ensu e ha his explana ion does no dis o he eal easons o which a decision was
aken.
De ini ion 3.3.
An explana ion
e
o a classi ica ion model
:X → Y
is said o be in insic i
e
only depend o elemen s, pa ame e s and ope a ions p esen in ,Xo Y.
Example 2. In insicali y
In o de o p oduce an explana ion o a black box model
, he pos -hoc me hod LIME [
11
]
gene a es a new da ase consis ing o pe u bed samples and he co esponding p edic ions o
.
On his new da ase , LIME ains an in e p e able model
h
, which is weigh ed by he p oximi y o
he pe u bed samples o he ins ance o in e es . The p edic ion o he model
h
should be a good
app oxima ion o he p edic ions o he model
locally, bu i does no ha e o be a good global
app oxima ion o he model . The p oduced explana ion can be exp essed as ollows:
e= Φ( (x)) = a g min
h
L( , h, πx) + Ω(h)(3)
wi h
L( , h, πx)
he local ideli y, i.e. how close he p edic ions om
h
a e close o he p edic ions
om
. The p oximi y measu e
πx
de ines how la ge is he neighbo hood a ound he explained
ins ance. The e o e, explana ion
e
does no only depend o elemen s, pa ame e s and ope a ions
p esen in
,
X
o
Y
since he explana ion depends on he su oga e model
h
. Consequen ly, his
explana ion is no in insic. I is he p edic ion o he model
h
ha is explained, no ha o he model
.
Example 3. In insicali y
In opposi ion, he explana ion esul ing om a linea eg ession
h
can
be conside ed as in insic because he lea ned ela ionships be ween he inpu s and he labels can be
w i en as ollows:
e= Φ( (xi)) = θ ×xi(4)
wi h θ he se o ainable pa ame e s o and xian ins ance.
3. Validi y and Comple eness
. An explana ion o a p edic ion mus be alid, meaning ha i should
asse ha he model is Righ (o w ong) o he Righ Reason (RRR). I mus show ha he
unc ioning o he model is consis en , ha i is no biased by he aining da a. The explana ion
should be simila o wha an expe in he ield would gi e. To his no ion o alidi y we could
add a deside a um no ion o comple eness: an explana ion could be judged as incomple e i i does
no con ain enough alid elemen s in i s cons i u ion, meaning ha we wan a su icien numbe o
disc imina ing elemen s. Howe e , i has been es ablished in he li e a u e ha a ”good” explana ion
is selec i e [
7
]. Selec i i y means ha humans a e adep a selec ing a ew causes om a some imes
in ini e numbe o causes.
De ini ion 3.4.
Gi en a ield expe human being
h:X → Y
, we de ine
E alid
as he se o alid
explana ions Φ : Y → E alid
De ini ion 3.5.
An explana ion
e
o a classi ica ion model
:X → Y
is said o be alid i
e∈E alid and comple e i eis disc imina i e enough.
Example 4. Validi y:
To illus a e his, we ake he example o he image 2o a am abbi ha we see in i s en i e y, om
he on , and ha he model classi ies as a abbi . We can say ha he explana ion is alid i i
con ains elemen s explaining why i is abbi , i.e. he ones ha an expe looking a he image would
use o jus i y he ac ha i is a am abbi .
Example 5. Comple eness
To illus a e he no ion o comple eness, closely ela ed o alidi y, we ake he example on 3o a
am abbi ha we see in i s en i e y, om he on , and ha he model classi ies as a abbi . We
can say ha he explana ion is mo e comple e i i con ains enough elemen s explaining why i is a
abbi , i.e., some o he mos impo an ea u es ha a human expe looking a he image would
no ice and use o jus i y he ac ha i is a am abbi . Technically, i is no ”w ong” o say ha a
abbi can be o di e en colo s. Howe e , colou is no a disc imina ing elemen o ecognize a am
abbi and i would be imp obable o see a human gi ing his explana ion. I his jus i ica ion on he
colo s had been accompanied by disc imina i e elemen s o he abbi , such as i s big d oopy ea s,
he explana ion would ha e been mo e comple e.
Figu e 2: Valid and in alid examples o explana ions. The alid explana ion con ains elemen s ha a e p esen in he expe
explana ion se and ha a e he e o e ”good” easons o jus i y why a abbi is p esen in he pic u e. On he con a y, he in alid
explana ion con ains elemen s ou o he expe explana ion se .
Figu e 3: Valid and in alid examples o explana ions. The alid explana ion con ains elemen s ha a e p esen in he expe
explana ion se and ha a e he e o e ”good” easons o jus i y why a abbi is p esen in he pic u e. On he con a y, he in alid
explana ion con ains elemen s ou o he expe explana ion se .
•
Objec i e: as he explana ion
ei
uses symbols (
za ,i
) and ela ionships (
θh,i
) ha can be concep ual-
ized by he human use .
•
In insic: as he explana ion
ei
only depends o elemen s (
za ,i
), pa ame e s (
θh,i
) and ope a ions
p esen in o iginal model h
The alidi y o he explana ion will ha e o be measu ed when applying he G eybox XAI amewo k
o a use-case, as his c i e ion is da ase -dependan . This no ion can be easily measu ed by compa ing he
weigh s θh,i and an expe knowledge base.
4.2.3. T aining o he La en Space P edic o
The La en Space P edic o mus p edic a segmen a ion map
zi
om a RGB inpu image
xi
hanks o
an Encode -Decode a chi ec u e. This segmen a ion map will hen be ec o ized in an a ibu e ec o
za ,i
o cons i u e he Explainable La en Space
{zi, za ,i}
. We choose o use a DeepLab 3+ [
87
] as a
La en Space P edic o wi h a ResNe 101 as backbone model. The speci ici y o DeepLab 3+ is o use an
a ous con olu ion, allowing he de elope o adjus il e ’s ield-o - iew in o de o cap u e mul i-scale
in o ma ion, and a dep hwise sepa able con olu ion. As i is a seman ic segmen a ion ask, we use an
ou pu s ide o 16 o dense ea u e ex ac ion as i is he bes ade-o be ween speed and accu acy. The
objec i e is o make a pixel-wise p edic ion o e an en i e image and he pe o mance is measu ed in e ms
o pixel in e sec ion-o e -union a e aged ac oss he a ibu es (mIOU).
In o de o es he pe o mance o ou model, we do no ain i wi h images o seman ic segmen a ion
masks bu wi h bounding boxes showing he p esence o absence o a ibu es, in a weekly supe ised
manne [
88
]. The main objec i e is o ha e an Explainable La en Space ha accoun s o he p esence o
each a ibu e as much as possible. I is he e o e necessa y ha each a ibu e p esen on he
xi
image is
segmen ed bu he segmen a ion map does no need o be e y accu a e because i is a e wa ds ec o ized.
Pixels ha do no belong o any bounding box a e conside ed as backg ound pixels, while he ones
belonging o se e al bounding boxes a e accoun ed as belonging only o he smalles one. We selec he
smalles a he han he la ges in o de o no lose he small a ibu es encompassed by la ge ones (like
eyes in he middle o a ace o example).
The inpu o he La en Space p edic o is an RGB image
xi
while i s ou pu is a segmen a ion map
zi
o dimension
h∗w
wi h
h
and
w
he dimensions o
xi
. As he spa ial in o ma ion is no used by he
T anspa en Classi ie , we ex ac om
zi
an a ibu e ec o
za ,i
con aining he lis o each unique alue
con ained in
zi
. A con idence mask is applied in o de o keep only he a ibu es ha we e p edic ed wi h
a con idence abo e a ce ain h eshold. We inally use a one-ho encoding o ob ain a ec o o 0s and 1s,
desc ibing he p edic ion o p esence o absence o each a ibu e in he inpu RGB image
xi
. Because his
ex ac ion does no allow backp opaga ion, he La en Space P edic o is ained by maximising i s mIOU.
I is he e o e no di ec ly ained o p edic a good a ibu e ec o
za ,i
bu a good segmen a ion map
zi
.
No e ha his segmen a ion map p edic ion is opaque, no explana ion is gi en as o why a ce ain pixel has
been p edic ed as ep esen ing a ce ain a ibu e.
4.3. In e ence p edic ion and i s explana ion ende ing h ough a Na u al Language Explana ion
When he La en Space P edic o
g
and he T anspa en Classi ie
h
a e ained and p o ide good
esul s on hei own, we eeze hei weigh s and compose hem o e alua e he unc ion
(h◦g) : XY
in
o de o p edic a class
yi
om
xi
. As he T anspa en Classi ie
h
is anspa en by design, we a e able o
gene a e a na u al language explana ion while making he p edic ion
yi=h(g(xi))
. As explained ea lie
we p oduce an explana ion o p edic ion ˆy om 3 elemen s:
•
The in insic anspa ency o
h
, allowing o know wha would imply a change o
za ,i
on he inal
p edic ion ˆy hanks o he lea ned weigh s in θh.

•za ,i which akes he o m o a lis o a ibu es ha can be named in na u al language.
•zi
which is a segmen a ion map, showing he posi ion o a ibu es on he RGB image
xi
, hus aking
o e he ease o unde s anding o he usual isual explana ions.
We de ine he explana ion unc ion
Φ : Y → E
wi h
Y
he label space and
E
he explana ion space.
The T anspa en Classi ie
h
and he Explainable La en Space
{zi, za ,i}
make i possible o p oduce
explana ion ein na u al language.
ei= Φ(h(za ,i)) (24)
The e o e, we ha e he ollowing explana ion:
ei
= ”Image
xi
ep esen s a
yi
because a ibu es
za ,i,1
,
za ,i,...
and
za ,i,m
a e p esen , and he
classi ie hleads hose a ibu es espec i ely wi h weigh s θh,1,θh,... and θh,n o class yi.”
In addi ion, he segmen a ion map
zi
can be displayed as a isual explana ion o show he posi ion o
a ibu es za ,i.
As a summa y, he G eybox XAI amewo k makes a p edic ion
yi
o a andom RGB image
xi∈X
and p oduces an explana ion eiby ollowing Algo i hm 1:
Algo i hm 1
G eybox XAI amewo k pseudo-algo i hm o p oduce a na u al language explana ion o a
p edic ion
Requi e: Inpu Image xi, La en Space P edic o g, T anspa en Classi ie h, Explana ion Func ion Φ
1: S ep 1: P edic Explainable La en Space
2: zi←g(xi)
3: S ep 2: Vec o ize Explainable La en Space o ob ain an a ibu e ec o
4: o j∈zido
5: Append(za ,i,j) i j /∈za ,i
6: end o
7: S ep 3: P edic Objec Class
8: yi←h(za ,i)
9: S ep 4: Gene a e a Na u al Language Explana ion
10: ei←Φ(h(za ,i))
11: e u n P edic ion yi, Na u al Language Explana ion ei, Segmen ed Image zi
5. Expe imen al S udy
We illus a e he use o ou amewo k wi h 2 da ase s: MonuMAI, PASCAL-Pa . The ex ensi e use
case p o ing he u ili y o he model is de eloped on MonuMAI because his da ase has al eady been
used in he s a e o he a [
53
] o p o e he u ili y o composi ional models. The hypo hesis es ed is o
e i y ha he G eybox XAI amewo k is able o p oduce accu a e and explainable p edic ions. Ou goal
is o sol e a composi ional classi ica ion p oblem and o be able o p edic o each image which objec is
p esen , jus i ying his p edic ion by he pa s-o objec (a ibu es) o his objec p esen on he image.
MonuMAI da ase [
89
] allows o classi y a chi ec u al s yle classi ica ion om acade images. The
idea he e is o be able o classi y an image by p edic ing which ype o monumen is p esen in he image
based on he dis inc i e a chi ec u al a ibu es o he di e en ypes o monumen s. This da ase con ains
app oxima ely 1500 images labelled wi h 4 classes (a chi ec u al s yles) and con aining bounding boxes
desc ibing he p esence o 15 di e en a ibu es ( isible cha ac e is ics o hese a chi ec u al s yles).
Each image is labelled wi h he a chi ec u al s yle o he monumen p esen on he image and bounding
boxes in o m abou he p esence and posi ion o he a ibu es p esen on he image. We call his da ase
D iple ={(xi, zi, yi)}N
i=1
wi h a se
X
o RGB images ep esen ing a chi ec u al monumen s, a se
Z
o
bounding boxes ep esen ing a chi ec u al a ibu es and a se Yo a chi ec u al s yles.
A Knowledge Base 2is buil based on he expe knowledge o he MonuMAI da ase [89].
KB RDF (s,p,o) iple examples
TBox (Ogee A ch, isPa O , Go hic Monumen )
(Poin ed A ch, isPa O , Go hic Monumen )
(T e oil A ch, isPa O , Go hic Monumen )
(Go hic Pinnacle, isPa O , Go hic Monumen )
(Fla A ch, isPa O , Hispanic-Muslim Monumen )
(Lobed A ch, isPa O , Hispanic-Muslim Monumen )
(Ho seshoe A ch, isPa O , Hispanic-Muslim Monumen )
(B oken Pedimen A ch, isPa O , Ba oque Monumen )
(Solomonic Column, isPa O , Ba oque Monumen )
(Rounded A ch, isPa O , Ba oque Monumen )
(Rounded A ch, isPa O , Renaissance Monumen )
(Po hole A ch, isPa O , Ba oque Monumen )
(Po hole A ch, isPa O , Renaissance Monumen )
(Lin elled Doo way A ch, isPa O , Ba oque Monumen )
(Lin elled Doo way A ch, isPa O , Renaissance Monumen )
(Se liana, isPa O , Renaissance Monumen )
(Segmen al Pedimen , isPa O , Renaissance Monumen )
(T iangula Pedimen , isPa O , Renaissance Monumen )
Table 2: Examples o RDF iples ex ac ed om he MonuMAI da ase and con ained in a KB e minological (TBox) and asse ional
(ABox) componen s
We epea he a ious s eps desc ibed in he Sec ion 4.2 in o de o ain and use he G eybox XAI
amewo k:
•
A anspa en model
h
is ained o p edic an a chi ec u al s yle, using as inpu a ec o encoding
he p esence and absence o a chi ec u al a ibu es.
•
A Deep Neu al Ne wo k
g
is ained o p edic a segmen a ion map om an RGB image inpu . I s
pu pose is o de ec he di e en a chi ec u al a ibu es ha cons i u e he image.
5.1. Logis ic Reg ession as a T anspa en Classi ie
The pu pose o he T anspa en Classi ie is o ep esen a mo e accu a e and close e sion o he
da ase han he Knowledge Base 2i sel . In ac , he knowledge con ained in he knowledge base is
e y gene ic ( o example, a Hispanic-Muslim monumen has a la a ch, an ho seshoe a ch and a lobed
a ch). Howe e , while his is ue in a gene al case, no all such monumen s ha e all hese a ibu es and
some ep esen a ions (i.e. images) o hese monumen s may ha e some a ibu es missing. Also, in his
pa icula case o monumen s, i is possible ha some images ha e a chi ec u al a ibu es belonging o
se e al a chi ec u al s yles. I is he esul o he p og essi e e olu ion o he cons uc ion o econs uc ion
p ocesses.
In bina y logis ic eg ession he unc ion
h(za , θh)
used o model he dependence o a eg ession
a ge yi∈ {0,1}on ea u es za ,i whe e yi≈h(za ,i, θh)can be w i en as:
Following Sec ion 4.2.2, a logis ic eg ession
h(za , θh)
is ained on he se o a ibu es
‡a
o he
da ase o p edic classes om he se o labels
Y
. We compa e he pe o mance o he logis ic eg ession
o a Nai e Bayes Classi ie .
Model Accu acy
Logis ic Reg ession 97.65%
Nai e Bayes Classi ie 94.32%
Table 3: Mean Accu acy o 2 anspa en models on MonuMAI da ase . The logis ic eg ession ha e a be e accu acy han he Nai e
Bayes Classi ie . We he e o e choose o use i as he T anspa en Classi ie o he G eybox XAI amewo k
Figu e 5 ep esen he se o ainable weigh s
θh
o he logis ic eg ession, linking a ibu es om
za
and classes om
Y
hanks o he ela ionship
yi≈θ|
h×za i
. These weigh s p o ide a s a is ical link
be ween a ibu es and classes.
Figu e 5: Weigh s o he logis ic eg ession model i ed o link a ibu es and classes om he MonuMAI da ase .
F om he se o ainable weigh s
θh
o he T anspa en Classi ie we ex ac a Knowledge G aph (KG)
(Figu e 6) ep esen ing he link be ween a ibu es and classes. I is a isual ep esen a ion o he weigh s
om Figu e 5. I a weigh is supe io o 0, an edge is d awn be ween he 2 conce ned nodes. Rep esen ing
knowledge his way has a simple explana o y in e es : when an a ibu e is de ec ed, i is s aigh o wa d o
see which classes a e linked o his a ibu e. We see o example ha he a ibu es T e oil A ch,Poin ed
A ch and Ogee A ch a e only linked o he a chi ec u al s yle Go hic. The e o e, i hose a ibu es a e
p esen in he ec o za i, he class Go hic will be p edic ed by he T anspa en Classi ie .
5.2. Deeplab 3+ as a La en Space P edic o
Logis ic eg ession canno emain anspa en when aking images as inpu because he numbe o
pa ame e s and a iables would be a oo la ge. Mo eo e , hese a iables would be pixels a he han
symbols. To o e come his and o make he logis ic eg ession ake as inpu a ec o o a ibu es, we ain
an Encode -Decode on he images on a seman ic segmen a ion ask. We chose o use a DeepLab 3+ [
87
]
as i gi es he bes pe o mance on his da ase .
On his da ase we do no ha e a seman ic segmen a ion image ep esen ing he g ound u h bu only
bounding boxes a ound each a ibu e. We use hese bounding boxes o p edic a segmen a ion map hanks
o a c oss-en opy loss. As he model was ained wi h images anno a ed wi h bounding boxes ins ead
o seman ic segmen a ion images, he segmen ed a ibu es ha e a squa e shape. I is no a p oblem as
Figu e 6: Knowledge G aph ep esen a ion au oma ically ex ac ed om he Logis ic Reg ession weigh s on he MonumAI da ase .
G een nodes ep esen a ibu es while yellow nodes ep esen classes. Dis ances be ween nodes ep esen weigh s linking a ibu es
and classes in he
θ
ma ix o he i ed logis ic eg ession model: he close 2 nodes a e, he la ge he weigh linking hem. An edge
is se o black i he associa ed weigh is supe io o ze o and anspa en o he wise.
unlike mos seman ic segmen a ion models, wha we a e in e es ed in he e is no he mIoU o he accu acy
o he p edic ion a he pixel le el bu a he how well he a ibu e ec o
za ,i
is p edic ed. Since he
T anspa en Classi ie akes as inpu a one-ho encoded ec o o a ibu es, he mIoU and he a e age
p ecision o he Deeplab 3+ ha e no in luence on he classi ica ion esul : wha G eybox model ocuses on
is on de ec ing a leas once each a ibu e, no on de ec ing all pixels o each occu ence o each a ibu e.
Whe he we de ec one pixel o a gi en a ibu e o 3000 pixels o an a ibu e is he same because he
segmen a ion map is pu in he o m o a bina y ec o o a ibu e p esence in o de o be used by he
T anspa en Classi ie .
To gene a e his a ibu e ec o , we compa e he lis o so ed unique elemen s o he p edic ed
seman ic segmen a ion image and he lis o a ibu es p esen in he image. Figu e 7 ep esen s on he
le an RGB image used as inpu o he Deeplab 3+ and on he igh he p edic ed seman ic segmen a ion
image. The a ibu e ec o associa ed wi h his seman ic segmen a ion image would be, o example,
[1,0,1,1,0, ..., 0]
ep esen ing he 3 a ibu es (in da k cyan, pink and blue) p esen in he image. I is his
ec o ha is subsequen ly used by he p e- ained T anspa en Classi ie o p edic he class o he image.
5.3. Pe o mance o he G eybox XAI amewo k: Accu acy and Explainabili y
We judge he pe o mance o ou amewo k acco ding o 2 no ions: i s accu acy du ing an image
classi ica ion ask and he explainabili y o i s p edic ion du ing his same classi ica ion ask.
5.3.1. Accu acy
We e alua e ou model on he image classi ica ion ask and compa e i o se e al baselines. Ou model
is he G eybox XAI F amewo k, composed by a Deeplab 3+ and a logis ic eg ession. The i s baseline
is a Da a-e icien image T ans o me s (DeiT) [
90
], a con olu ion- ee ans o me buil upon he Vision
T ans o me (ViT) a chi ec u e [47] and achie ing s a e-o - he-a esul s o e baseline da ase s. In o de
o ob ain he bes pe o ming compa ison baseline, we used he la ges possible DeiT a chi ec u e (simila
o ha o VIT-B), as well as he highes possible image esolu ion (384x384). The second baseline is a
ResNe 101 classi ie , in o de o ha e a con olu ion-based baseline. We hen compa e o he EXPLANe
[
53
] model which is he s a e o he a o composi ional XAI models on his da abase. We also build a
new baseline by modi ying he DeepLab 3+ o make i a classi ie . This Deeplab Classi ie is a usual
Deeplab 3+ a chi ec u e, wi h a ResNe 101 as backbone, wi h skip connec ions and a ous con olu ions,
bu ins ead o p edic ing he class o each pixel as in a seman ic segmen a ion we use i in a classi ica ion
ole by modi ying he las laye . Ins ead o e alua ing i by he MiOU, i is now judged by he accu acy o
he global class. By adding an A e agePooling and a So max a e he decode , we ob ain an end- o-end
classi ica ion model. Since his is a classi ica ion ask, we compa e ou sel es in e ms o accu acy. We also
add he esul s on he PASCAL-Pa da ase , which con ains mo e a ibu es and mo e classes. As ou model
is anspa en by design, we compa e ou sel es sepa a ely o models conside ed as opaque (T ans o me s
like DeiT and CNNs such as ResNe o Encode -Decode such as Deeplab 3+) and explainable models
(such as EXPLANe ).
The esul s o G eybox XAI
4
classi ica ion model based on Deeplab 3+ seman ic segmen a ion
backbone and a logis ic eg ession, oge he wi h hese baseline classi ica ion ne wo ks a e shown in Table
4 o MonuMAI and PASCAL-Pa da ase . The esul s show ha he G eybox XAI classi ie achie es
accu acy sligh ly in e io o he opaque models, being ou pe o med by app oxima ely 2.5% each ime.
Howe e , i s accu acy is a supe io o he explainable baseline, which is he EXPLANe model. We
can he e o e conside ha he e is a sligh loss in accu acy compa ed o he opaque models bu a gain
compa ed o he composi ional models.
4h ps://gi hub.com/Ad ienBenne o /G eybox-

Figu e 7: Example o esul s ob ained using seman ic segmen a ion. Image a) ep esen s he RGB inpu da a and Image b) ep esen s
he G ound T u h o seman ic segmen a ion masks. Image c) ep esen s he esul o his segmen a ion by he La en Space P edic o
and image d) is an o e lay o he p edic ion on he inpu image, in o de o see whe e he de ec ed a ibu es a e. The black pixels
ep esen he backg ound, he cyan and pink pixels ep esen wo a chi ec u al a ibu es.
Da ase Model Accu acy (%)
Compa ison wi h Opaque Models
MonuMAI G eybox XAI (ou ) 94.04
Deeplab 3+ Classi ie 96.02
DeiT-B 96.48
ResNe 101 95.69
MonuNe Classi ie 83.11
PASCAL-Pa G eybox XAI (ou ) 88.30
Deeplab 3+ Classi ie 90.18
DeiT-B 90.85
ResNe 101 90.12
Compa ison wi h Explainable Models
MonuMAI G eybox XAI (ou ) 94.04
KG De e minis ic Classi ie 54.79
EXPLANe 90.40
PASCAL-Pa G eybox XAI (ou ) 88.30
EXPLANe 82.4
Table 4: Explainable composi ional s opaque di ec classi ica ion: Resul s o he G eybox XAI model (using seman ic segmen a ion
Deeplab 3+ and a logis ic eg ession) on MonuMAI and PASCAL-Pa da ase s, and compa ison wi h embedded e sion o he
baseline model MonuNe , a anilla classi ie baseline wi h ResNe 101, a ans o me model wi h DeiT, an expe KG-based
de e minis ic (non- ained) classi ie , he composi ional model EXPLANe and a classi ie de i ed om Deeplab 3+
The loss in accu acy compa ed o opaque models should be coun e balanced by a bene i in explain-
abili y because he G eybox XAI is anspa en by design when used o a classi ica ion ask (see Sec ion
4.2.2) and p oduces ”good” explana ions (objec i e and in insic).
5.3.2. Explainabili y
In o de o e i y whe he he explana ions gene a ed by he G eybox XAI a e alid we compa e he
Knowledge G aph ex ac ed di ec ly om he T anspa en Classi ie weigh s in Figu e 6and he expe
Knowledge G aph Figu e 8. This igu e is aken om [
53
] and ep esen s MonuMAI Knowledge G aph
cons uc ed based on a his o ians expe knowledge [
89
]. The KG is ex ac ed om he T anspa en
Classi ie by c ea ing a node o e e y pa -o and objec s o he weigh s ma ix and by d awing an edge
be ween nodes o e e y weigh s supe io o 0. The G aph Edi Dis ance (GED) [
91
] be ween he wo KG
is equal o ze o, meaning ha he explana ion o he T anspa en Classi ie is he same as he ones a
his o ians expe s would ha e p oduced. The e o e, we can conside ha hese explana ions a e globally
alid i he seman ic segmen a ion is co ec .
We also e i y i ou amewo k is a sel -explaining p edic ion model acco ding o he de ini ion o
[75], i.e. i i has he o m:
(x) = g(θ(x)1h(x)1, ..., θ(x)kh(x)k)(25)
whe e:
•gis mono one and comple ely addi i ely sepa able
• Fo e e y zi:= θi(x)hi(x),gsa is ies ∂g
∂zi≥0
•θis locally di e ence bounded by h
•hi(x)is an in e p e able ep esen a ion o x
Figu e 8: Simpli ied MonuMAI knowledge g aph cons uc ed based on a his o ians expe knowledge [
53
,
89
]. I links a ibu es
and classes in a g aphical ep esen a ion and shows, o example, ha a Hispanic-Muslim monumen is composed by Fla A ches,
Ho seshoe A ches and Lobed A ches. This g aph ep esen ing he expe knowledge o he da ase is simila o he one ex ac ed by
he T anspa en Classi ie .
•kis small
Taking Equa ion 12,G eybox XAI amewo k can be w i en:
(x)=(h◦g)(x) = h(g(xi, θg), θh)≈θh,1za ,1, ..., θh,nza ,n (26)
whe e:
•
The T anspa en Classi ie
h
is mono one and comple ely addi i e sepa able as i can be app oxi-
ma ed wi h he mul iplica ion be ween he weigh ma ix θhand he ea u es.
• Pa ial de i a i e o hwi h espec o θh,iza ,i is posi i e.
•θhis locally di e ence-bounded by za ,i.
•za ,i is an in e p e able ep esen a ion o xas za ,i a e nameable ea u es.
•n=
is small as in e ac ions o he logis ic eg ession a e kep o a minimum o espec he de ini ion
o simula abili y.
F om hese di e en elemen s, we can conclude ha he G eybox XAI model is anspa en and
p oduces ”good” explana ions when used o he ask o image classi ica ion. Mo eo e , i is possible
o accompany his ex ual explana ion by a isualiza ion, showing he seman ic segmen a ion image
masks used o de e mine he a ibu e ec o
‡a
employed by he T anspa en Classi ie o pe o m he
classi ica ion. Figu e 9shows an example o he isualiza ion ha can be p oduced. This image has been
classi ied as Go hic based on he di e en a ibu es de ec ed by he La en Space P edic o . The p edic ed
seman ic segmen a ion map can be o e laid on op o he RGB inpu o isualize whe e hese ea u es a e
loca ed. The image on he igh shows all pixels ha we e used du ing classi ica ion. These a e ob ained
by emo ing e e y pixel segmen ed as being a backg ound pixel.
Figu e 9: Visual explana ion o he G eybox XAI model. The image in he uppe le co ne is he RGB inpu ha he model mus
classi y. In he bo om le co ne is he seman ic segmen a ion image p edic ed by he model, showing in black he pixels classi ied
as pa o he backg ound. Cyan and black pixels ep esen s wo a ibu es. In he middle he supe posi ion o he wo images on he
le , emo ing black pixels om he backg ound o keep only he elemen s ha will be used in he logis ic eg ession. Finally, he
image on he igh is he same sample image bu eplacing he a ibu es by hei RGB alue and hiding he backg ound pixels, which
a e no used du ing he classi ica ion by he T anspa en Classi ie as i only uses as inpu an a ibu e ec o .
[22]
I. Donadello, Seman ic image in e p e a ion-in eg a ion o nume ical da a and logical knowledge
o cogni i e ision, Ph.D. hesis, Uni e si y o T en o (2018).
[23]
A. S. d’A ila Ga cez, M. Go i, L. C. Lamb, L. Se a ini, M. Sp ange , S. N. T an, Neu al-symbolic
compu ing: An e ec i e me hodology o p incipled in eg a ion o machine lea ning and easoning,
Jou nal o Applied Logics - I CoLog Jou nal o Logics and hei Applica ions (FLAP) 6 (4) (2019)
611–632.
URL h ps://collegepublica ions.co.uk/i colog/?00033
[24]
I. Donadello, M. D agoni, C. Ecche , Pe suasi e explana ion o easoning in e ences on die a y
da a, in: Fi s Wo kshop on Seman ic Explainabili y @ ISWC 2019, 2019.
[25]
R. Guido i, A. Mon eale, S. Ruggie i, F. Tu ini, F. Gianno i, D. Ped eschi, A su ey o me hods
o explaining black box models, ACM compu ing su eys (CSUR) 51 (5) (2018) 1–42.
[26]
V. Buh mes e , D. M
¨
unch, M. A ens, Analysis o explaine s o black box deep neu al ne wo ks o
compu e ision: A su ey, a Xi p ep in a Xi :1911.12116 (2019).
[27]
J. And eas, Measu ing composi ionali y in ep esen a ion lea ning, a Xi p ep in a Xi :1902.07181
(2019).
[28] J. A. Fodo , E. Lepo e, Composi ionali y Pape s, Ox o d Uni e si y P ess UK, 2002.
[29]
A. S one, H. Wang, M. S a k, Y. Liu, D. Sco Phoenix, D. Geo ge, Teaching composi ionali y o
CNNs, in: P oceedings o he IEEE Con e ence on Compu e Vision and Ra e n Recogni ion, 2017,
pp. 5058–5067.
[30]
B. M. Lake, R. Salakhu dino , J. B. Tenenbaum, Human-le el concep lea ning h ough p obabilis ic
p og am induc ion, Science 350 (6266) (2015) 1332–1338.
[31]
D. Hupkes, V. Danke s, M. Mul, E. B uni, The composi ionali y o neu al ne wo ks: in eg a ing
symbolism and connec ionism, a Xi p ep in a Xi :1908.08351 (2019).
[32]
J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, J. Wu, The neu o-symbolic concep lea ne : In e p e ing
scenes, wo ds, and sen ences om na u al supe ision, a Xi p ep in a Xi :1904.12584 (2019).
[33]
R. De Kok, T. Schneide , U. Amme , Objec -based classi ica ion and applica ions in he alpine
o es en i onmen , In e na ional A chi es o Pho og amme y and Remo e Sensing 32 (Pa 7)
(1999) 4–3.
[34]
D. Hube , A. Kapu ia, R. Donamukkala, M. Hebe , Pa s-based 3d objec classi ica ion, in: P oceed-
ings o he 2004 IEEE Compu e Socie y Con e ence on Compu e Vision and Pa e n Recogni ion.
CVPR 2004., Vol. 2, IEEE, 2004, pp. II–II.
[35]
E. J. Be ns ein, Y. Ami , Pa -based s a is ical models o objec classi ica ion and de ec ion, in:
IEEE Compu e Socie y Con e ence on Compu e Vision and Pa e n Recogni ion (CVPR’05),
Vol. 2, IEEE, 2005, pp. 734–740.
[36]
P. F. Felzenszwalb, R. B. Gi shick, D. McAlles e , D. Ramanan, Objec de ec ion wi h disc imina-
i ely ained pa -based models, IEEE T ansac ions on Pa e n Analysis and Machine In elligence
32 (9) (2009) 1627–1645.
[37]
M. E e ingham, L. Van Gool, C. K. I. Williams, J. Winn, A. Zisse man, The PASCAL Visual
Objec Classes Challenge 2012 (VOC2012) Resul s.
URL
h p://www.pascal-ne wo k.o g/challenges/VOC/ oc2012/
wo kshop/index.h ml

[38]
W. Ge, X. Lin, Y. Yu, Weakly supe ised complemen a y pa s models o ine-g ained image
classi ica ion om he bo om up, in: P oceedings o he IEEE Con e ence on Compu e Vision and
Ra e n Recogni ion, 2019, pp. 3034–3043.
[39]
A. Holzinge , G. Langs, H. Denk, K. Za loukal, H. M
¨
ulle , Causabili y and explainabili y o a i icial
in elligence in medicine, Wiley In e discip. Re . Da a Min. Knowl. Disco . 9 (4) (2019) e1312.
[40]
A. Holzinge , B. Malle, A. Sa an i, B. P ei e , Towa ds mul i-modal causabili y wi h g aph neu al
ne wo ks enabling in o ma ion usion o explainable ai, In o ma ion Fusion 71 (2021) 28–37.
doi:h ps://doi.o g/10.1016/j.in us.2021.01.008.
URL
h ps://www.sciencedi ec .com/science/a icle/pii/
S1566253521000142
[41] J. Pea l, Causali y, Camb idge uni e si y p ess, 2009.
[42]
A. Holzinge , A. Ca ing on, H. M
¨
ulle , Measu ing he quali y o explana ions: The sys em
causabili y scale (SCS), KI - K
¨
uns liche In elligenz 34 (2) (2020) 193–198.
doi:10.1007/
s13218-020-00636-z.
URL h ps://doi.o g/10.1007%2Fs13218-020-00636-z
[43]
J. Hu, L. Shen, G. Sun, Squeeze-and-exci a ion ne wo ks, in: 2018 IEEE/CVF Con e ence on
Compu e Vision and Pa e n Recogni ion, 2018, pp. 7132–7141. doi:10.1109/CVPR.2018.
00745.
[44]
A. S eine , A. Kolesniko , , X. Zhai, R. Wigh man, J. Uszko ei , L. Beye , How o ain you i ?
da a, augmen a ion, and egula iza ion in ision ans o me s, a Xi p ep in a Xi :2106.10270
(2021).
[45]
I. Tols ikhin, N. Houlsby, A. Kolesniko , L. Beye , X. Zhai, T. Un e hine , J. Yung, A. S eine ,
D. Keyse s, J. Uszko ei , M. Lucic, A. Doso i skiy, Mlp-mixe : An all-mlp a chi ec u e o ision,
a Xi p ep in a Xi :2105.01601 (2021).
[46]
J. Zhuang, B. Gong, L. Yuan, Y. Cui, H. Adam, N. D o nek, S. Ta ikonda, J. Duncan, T. Liu,
Su oga e gap minimiza ion imp o es sha pness-awa e aining, ICLR (2022).
[47]
A. Doso i skiy, L. Beye , A. Kolesniko , D. Weissenbo n, X. Zhai, T. Un e hine , M. Dehghani,
M. Minde e , G. Heigold, S. Gelly, J. Uszko ei , N. Houlsby, An image is wo h 16x16 wo ds:
T ans o me s o image ecogni ion a scale, ICLR (2021).
[48]
X. Chen, C.-J. Hsieh, B. Gong, When ision ans o me s ou pe o m esne s wi hou p e aining o
s ong da a augmen a ions, a Xi p ep in a Xi :2106.01548 (2021).
[49]
X. Zhai, A. Kolesniko , N. Houlsby, L. Beye , Scaling ision ans o me s, in: P oceedings o he
IEEE/CVF Con e ence on Compu e Vision and Pa e n Recogni ion (CVPR), 2022, pp. 12104–
12113.
[50]
A. Cha an, Z. Shen, Z. Liu, Z. Liu, K.-T. Cheng, E. P. Xing, Vision ans o me slimming: Mul i-
dimension sea ching in con inuous op imiza ion space, in: P oceedings o he IEEE/CVF Con e ence
on Compu e Vision and Pa e n Recogni ion (CVPR), 2022, pp. 4931–4941.
[51]
C. Zhang, M. Zhang, S. Zhang, D. Jin, Q. Zhou, Z. Cai, H. Zhao, X. Liu, Z. Liu, Del ing deep in o
he gene aliza ion o ision ans o me s unde dis ibu ion shi s, in: P oceedings o he IEEE/CVF
Con e ence on Compu e Vision and Pa e n Recogni ion (CVPR), 2022, pp. 7277–7286.
[52]
A. M. Obeso, J. Benois-Pineau, M. S. Ga c
´
ıa V
´
azquez, A.
´
Al a o Ram
´
ı ez Acos a,
Visual s in e nal a en ion mechanisms in deep neu al ne wo ks o image classi i-
ca ion and objec de ec ion, Pa e n Recogni ion 123 (2022) 108411.
doi:h ps:
//doi.o g/10.1016/j.pa cog.2021.108411.
URL
h ps://www.sciencedi ec .com/science/a icle/pii/
S0031320321005872
[53]
N. D
´
ıaz-Rod
´
ıguez, A. Lamas, J. Sanchez, , G. F anchi, I. Donadello, S. Tabik, D. Fillia , P. C uz,
R. Mon es, F. He e a, EXplainable Neu al-Symbolic Lea ning (X-NeSyL) me hodology o use
deep lea ning ep esen a ions wi h expe knowledge g aphs: he MonuMAI cul u al he i age use
case (2021).
[54]
M. Ga nelo, M. Shanahan, Reconciling deep lea ning wi h symbolic a i icial in elligence: ep e-
sen ing objec s and ela ions, Cu en Opinion in Beha io al Sciences 29 (2019) 17–23.
[55]
R. Manhae e, S. Dumancic, A. Kimmig, T. Demees e , L. De Raed , DeepP obLog: Neu al
p obabilis ic logic p og amming, in: P oceedings o he In e na ional Con e ence on Neu al
In o ma ion P ocessing Sys ems, Vol. 31, 2018, pp. 3749–3759.
URL
h ps://p oceedings.neu ips.cc/pape /2018/ ile/
dc5d637ed5e62c36ecb73b654b05ba2a-Pape .pd
[56]
F. Pe oni, T. Rock
¨
aschel, P. Lewis, A. Bakh in, Y. Wu, A. H. Mille , S. Riedel, Language models
as knowledge bases? (2019). a Xi :1909.01066.
[57]
K. Bollacke , N. D
´
ıaz-Rod
´
ıguez, X. Li, Ex ending knowledge g aphs wi h subjec i e in luence
ne wo ks o pe sonalized ashion, in: E. Po mann, M. E. Tabacchi, R. Seising, A. Habens ein
(Eds.), Designing Cogni i e Ci ies, Sp inge In e na ional Publishing, 2019, pp. 203–233.
[58]
W. Shang, A. T o , S. Zheng, C. Xiong, R. Soche , Lea ning wo ld g aphs o accele a e hie a chical
ein o cemen lea ning (2019). a Xi :1907.00664.
[59]
A. Aamod , E. Plaza, Case-based easoning: Founda ional issues, Me hodological Va ia ions, and
Sys em App oaches 7 (1) (1994) 39–59.
[60]
R. Ca uana, Case-based explana ion o a i icial neu al ne s, in: A i icial Neu al Ne wo ks in
Medicine and Biology, P oceedings o he ANNIMAB-1 Con e ence, 2000, pp. 303–308.
[61]
M. T. Keane, E. M. Kenny, The Twin-Sys em App oach as One Gene ic Solu ion o XAI: An
O e iew o ANN-CBR Twins o Explaining Deep Lea ning (2019). a Xi :1905.08069.
[62]
The Desc ip ion Logic Handbook: Theo y, Implemen a ion and Applica ions, 2nd Edi ion, Cam-
b idge Uni e si y P ess, 2007. doi:10.1017/CBO9780511711787.
[63]
I. Donadello, L. Se a ini, In eg a ion o nume ic and symbolic in o ma ion o seman ic image
in e p e a ion, In elligenza A i iciale 10 (1) (2016) 33–47.
[64]
J.-B. Lamy, L. F. Soualmia, Fo maliza ion o he seman ics o iconic languages: An on ology-based
me hod and ou seman ic-powe ed applica ions, Knowledge-Based Sys ems 135 (2017) 159–179.
[65]
G. Ma a, F. Giannini, M. Diligen i, M. Go i, Ly ics: a gene al in e ace laye o in eg a e ai and
deep lea ning, a Xi p ep in a Xi :1903.07534 (2019).
[66]
G. Ma a, F. Giannini, M. Diligen i, M. Go i, In eg a ing lea ning and easoning wi h deep logic
models (2019). a Xi :1901.04195.
[67]
Z. C. Lip on, The my hos o model in e p e abili y, Queue 16 (3) (2018) 30:31–30:57.
doi:
10.1145/3236386.3241340.
URL h p://doi.acm.o g/10.1145/3236386.3241340
[68]
G. Mon a on, W. Samek, K.-R. M
¨
ulle , Me hods o in e p e ing and unde s anding deep neu al
ne wo ks, Digi al Signal P ocessing 73 (2018) 1–15. doi:10.1016/j.dsp.2017.10.011.
[69]
Z. Bu sac, C. H. Gauss, D. K. Williams, D. W. Hosme , Pu pose ul selec ion o a iables in logis ic
eg ession, Sou ce code o biology and medicine 3 (1) (2008) 17.
[70]
L. Rokach, O. Z. Maimon, Da a mining wi h decision ees: heo y and applica ions, Vol. 69, Wo ld
scien i ic, 2014.
[71]
S. B. Imandous , M. Boland a a , Applica ion o k-nea es neighbo (knn) app oach o p edic ing
economic e en s: Theo e ical backg ound, In e na ional Jou nal o Enginee ing Resea ch and
Applica ions 3 (5) (2013) 605–610.
[72]
J. R. Quinlan, Gene a ing p oduc ion ules om decision ees., in: P oceedings o he In e na ional
Join Con e ence on A i ical In elligence, Vol. 87, Ci esee , 1987, pp. 304–307.
[73]
D. Be g, Bank up cy p edic ion by gene alized addi i e models, Applied S ochas ic Models in
Business and Indus y 23 (2) (2007) 129–143.
[74]
T. L. G i i hs, C. Kemp, J. B. Tenenbaum, Bayesian models o cogni ion. (4 2008).
doi:10.1184/R1/6613682. 1.
URL
h ps://kil hub.cmu.edu/a icles/Bayesian_models_o _
cogni ion/6613682
[75]
D. Al a ez-Melis, T. S. Jaakkola, Towa ds obus in e p e abili y wi h sel -explaining neu al ne -
wo ks, in: P oceedings o he 32nd In e na ional Con e ence on Neu al In o ma ion P ocessing
Sys ems, NIPS’18, Cu an Associa es Inc., Red Hook, NY, USA, 2018, p. 7786–7795.
[76] E. B. Baum, Wha Is Though ?, Camb idge MA: B ad o d Book/MIT P ess, 2004.
[77]
C. Blundell, J. Co nebise, K. Ka ukcuoglu, D. Wie s a, Weigh Unce ain y in Neu al Ne wo ks,
a Xi e-p in s (2015) a Xi :1505.05424a Xi :1505.05424.
[78]
P. K emen, M. Bla
ˇ
sko, Z. Kouba, Seman ic Anno a ion o Objec s, Handbook o Resea ch on Social
Dimensions o Seman ic Technologies and Web Se ices, IGI Global, He shey, PA, USA, 2009, pp.
223–238. doi:10.4018/978-1-60566-650-1.ch011.
URL h ps://doi.o g/10.4018/978-1-60566-650-1.ch011
[79]
F. Baade , W. Nu , The desc ip ion logic handbook, Camb idge Uni e si y P ess, New Yo k, NY,
USA, 2003, Ch. Basic Desc ip ion Logics, pp. 43–95.
URL h p://dl.acm.o g/ci a ion.c m?id=885746.885749
[80]
S. Aue , C. Bize , G. Kobila o , J. Lehmann, R. Cyganiak, Z. I es, Dbpedia: A nucleus o a web o
open da a, in: The seman ic web, Sp inge , 2007, pp. 722–735.
[81]
G. A. Mille , R. Beckwi h, C. Fellbaum, D. G oss, K. J. Mille , In oduc ion o wo dne : An on-line
lexical da abase, In e na ional jou nal o lexicog aphy 3 (4) (1990) 235–244.
[82]
C. Kiddon, P. Domingos, Knowledge ex ac ion and join in e ence using ac able Ma ko logic, in:
P oceedings o he Join Wo kshop on Au oma ic Knowledge Base Cons uc ion and Web-scale
Knowledge Ex ac ion (AKBC-WEKEX), Associa ion o Compu a ional Linguis ics, Mon
´
eal,
Canada, 2012, pp. 79–83.
[83]
N. Balasub amanian, S. Sode land, O. E zioni, e al., Rel-g ams: a p obabilis ic model o ela ions
in ex , in: P oceedings o he Join Wo kshop on Au oma ic Knowledge Base Cons uc ion and
Web-scale Knowledge Ex ac ion, Associa ion o Compu a ional Linguis ics, 2012, pp. 101–105.
[84]
P. Hi zle , M. K zsch, S. Rudolph, Founda ions o Seman ic Web Technologies, 1s Edi ion,
Chapman & Hall/CRC, 2009.
[85]
G. An oniou, F. Van Ha melen, Web on ology language: Owl, in: Handbook on on ologies, Sp inge ,
2004, pp. 67–92.
[86]
E. C. No on, B. E. Dowd, Log odds and he in e p e a ion o logi models, Heal h se ices esea ch
53 (2) (2018) 859–878, pMC5867187[pmcid]. doi:10.1111/1475-6773.12712.
URL h ps://doi.o g/10.1111/1475-6773.12712
[87]
L.-C. Chen, Y. Zhu, G. Papand eou, F. Sch o , H. Adam, Encode -decode wi h a ous sepa able
con olu ion o seman ic image segmen a ion (2018). a Xi :1802.02611.
[88]
H. Ke adec, J. Dolz, S. Wang, E. G ange , I. ben Ayed, Bounding boxes o weakly supe ised
segmen a ion: Global cons ain s ge close o ull supe ision, in: Medical Imaging wi h Deep
Lea ning, 2020.
URL h ps://open e iew.ne / o um?id=VOQMC3 Z L
[89]
A. Lamas, S. Tabik, P. C uz, R. Mon es,
´
A. Ma
´
ınez-Se illa, T. C uz, F. He e a, MonuMAI:
Da ase , deep lea ning pipeline and ci izen science based app o monumen al he i age axonomy
and classi ica ion, Neu ocompu ing 420 (2020) 266–280.
[90]
H. Tou on, M. Co d, M. Douze, F. Massa, A. Sablay olles, H. J
´
egou, T aining da a-e icien image
ans o me s and dis illa ion h ough a en ion (2020).
doi:10.48550/ARXIV.2012.12877
.
URL h ps://a xi .o g/abs/2012.12877
[91]
A. San eliu, K.-S. Fu, A dis ance measu e be ween a ibu ed ela ional g aphs o pa e n ecogni-
ion, IEEE T ansac ions on Sys ems, Man, and Cybe ne ics (1983) 353–362.
[92]
S. Jiang, H. Qin, B. Zhang, J. Zheng, Op imized loss unc ions o objec de ec ion and ap-
plica ion on nigh ime ehicle de ec ion, P oceedings o he Ins i u ion o Mechanical Engi-
nee s, Pa D: Jou nal o Au omobile Enginee ing 236 (7) (2021) 1568–1578.
doi:10.1177/
09544070211036366.
URL h ps://doi.o g/10.1177%2F09544070211036366
[93]
R. Qin, K. Qiao, L. Wang, L. Zeng, J. Chen, B. Yan, Weigh ed ocal loss: An e ec i e loss unc ion
o o e come unbalance p oblem o ches x- ay14, IOP Con e ence Se ies: Ma e ials Science and
Enginee ing 428 (2018) 012022. doi:10.1088/1757-899x/428/1/012022.
URL h ps://doi.o g/10.1088/1757-899x/428/1/012022
[94]
S. Wach e , B. Mi els ad , C. Russell, Coun e ac ual explana ions wi hou opening he black box:
Au oma ed decisions and he gdp (2017). doi:10.48550/ARXIV.1711.00399.
URL h ps://a xi .o g/abs/1711.00399
[95]
R. K. Mo hilal, A. Sha ma, C. Tan, Explaining machine lea ning classi ie s h ough di e se coun-
e ac ual explana ions, in: P oceedings o he 2020 Con e ence on Fai ness, Accoun abili y, and
T anspa ency, 2020, pp. 607–617.
[96]
J. Del Se , A. Ba edo-A ie a, N. D
´
ıaz-Rod
´
ıguez, F. He e a, A. Holzinge , Explo ing he ade-o
be ween plausibili y, change in ensi y and ad e sa ial powe in coun e ac ual explana ions using
mul i-objec i e op imiza ion (2022). doi:10.48550/ARXIV.2205.10232.
URL h ps://a xi .o g/abs/2205.10232
[97]
S. Ve ma, J. Dicke son, K. Hines, Coun e ac ual explana ions o machine lea ning: A e iew
(2020). doi:10.48550/ARXIV.2010.10596.
URL h ps://a xi .o g/abs/2010.10596
[98]
S. Dandl, C. Molna , M. Binde , B. Bischl, Mul i-objec i e coun e ac ual explana ions, in: T. B
¨
ack,
M. P euss, A. Deu z, H. Wang, C. Doe , M. Emme ich, H. T au mann (Eds.), Pa allel P oblem
Sol ing om Na u e – PPSN XVI, Sp inge In e na ional Publishing, Cham, 2020, pp. 448–469.
[99]
A. Van Loo e en, J. Klaise, In e p e able coun e ac ual explana ions guided by p o o ypes (2019).
doi:10.48550/ARXIV.1907.02584.
URL h ps://a xi .o g/abs/1907.02584
[100]
A.-H. Ka imi, G. Ba he, B. Balle, I. Vale a, Model-agnos ic coun e ac ual explana ions o
consequen ial decisions (2019). doi:10.48550/ARXIV.1905.11190.
URL h ps://a xi .o g/abs/1905.11190
[101]
T. Laugel, M.-J. Leso , C. Ma sala, X. Rena d, M. De yniecki, In e se classi ica ion o compa ison-
based in e p e abili y in machine lea ning (2017). doi:10.48550/ARXIV.1712.08443.
URL h ps://a xi .o g/abs/1712.08443
[102]
M. T. Ribei o, S. Singh, C. Gues in, Ancho s: High-p ecision model-agnos ic explana ions,
P oceedings o he AAAI Con e ence on A i icial In elligence 32 (1) (Ap . 2018).
URL h ps://ojs.aaai.o g/index.php/AAAI/a icle/ iew/11491
[103]
H. M
¨
ulle , A. Holzinge , Kandinsky pa e ns, A i icial In elligence 300 (2021) 103546.
doi:h ps://doi.o g/10.1016/j.a in .2021.103546.
URL
h ps://www.sciencedi ec .com/science/a icle/pii/
S0004370221000977
[104]
A. Holzinge , M. Kickmeie -Rus , H. M
¨
ulle , Kandinsky pa e ns as iq- es o machine lea ning,
in: A. Holzinge , P. Kiesebe g, A. M. Tjoa, E. Weippl (Eds.), Machine Lea ning and Knowledge
Ex ac ion, Sp inge In e na ional Publishing, Cham, 2019, pp. 1–14.