INTERNATIONAL DOCTORAL
SCHOOL OF THE USC
Ilia
S epin
PhD Thesis
A gumen a i e Con e sa ional
Agen s o Explainable A i icial
In elligence
San iago de Compos ela, 2023
Doc o al P og amme in In o ma ion Technology Resea ch
DOCTORAL THESIS
ARGUMENTATIVE
CONVERSATIONAL AGENTS
FOR EXPLAINABLE ARTIFICIAL
INTELLIGENCE
Ilia S epin
INTERNATIONAL PHD SCHOOL OF THE UNIVERSITY OF SANTIAGO DE COMPOSTELA
DOCTORAL PROGRAMME IN INFORMATION TECHNOLOGY RESEARCH
SANTIAGO DE COMPOSTELA
2023
Decla ación del au o de la esis
D. Ilia S epin
Tí ulo de la esis: A gumen a i e Con e sa ional Agen s o Explainable A i icial In elligence
P esen o mi esis, siguiendo el p ocedimien o adecuado al Reglamen o, y decla o que:
1. La esis aba ca los esul ados de la elabo ación de mi abajo.
2. De se el caso, en la esis se hace e e encia a las colabo aciones que u o es e abajo.
3. Con i mo que la esis no incu e en ningún ipo de plagio de o os au o es ni de abajos
p esen ados po mí pa a la ob ención de o os í ulos.
4. La esis es la e sión de ini i a p esen ada pa a su de ensa y coincide la e sión imp esa
con la p esen ada en o ma o elec ónico.
Y me comp ome o a p esen a el Comp omiso Documen al de Supe isión en caso de que el
o iginal no es é en la Escuela.
En San iago de Compos ela, 30 de Junio de 2023
Fdo. Ilia S epin
Au o ización del Di ec o /Tu o de la Tesis
A gumen a i e Con e sa ional Agen s o Explainable A i icial
In elligence
D. José Ma ía Alonso Mo al, P o eso Ti ula de la Uni e sidad de San iago de Compos ela
D. Alejand o Ca alá, P o eso Ayudan e Doc o de la Uni e sidad de San iago de Compos ela
INFORMAN:
Que la p esen e esis, se co esponde con el abajo ealizado po D. Ilia S epin, bajo nues a
di ección/ u o ización, y au o izamos su p esen ación, conside ando que eúne los equisi os
exigidos en el Reglamen o de Es udios de Doc o ado de la USC, y que como di ec o es/ u o es
de es a no incu e en las causas de abs ención es ablecidas en la Ley 40/2015.
De acue do con lo indicado en el Reglamen o de Es udios de Doc o ado, decla amos ambién
que la p esen e esis doc o al es idónea pa a se de endida en base a la modalidad de COM-
PENDIO DE PUBLICACIONES, en las que la pa icipación del doc o ando/a ue decisi a pa a
su elabo ación y las publicaciones se ajus an al Plan de In es igación. La u ilización de es os
a ículos en es a memo ia, es á en conocimien o de que ninguno de los abajos aquí eunidos
pod á se p esen ado en ninguna o a esis doc o al.
En San iago de Compos ela, 30 de Junio de 2023
Fdo. José Ma ía Alonso Mo al
Di ec o /a esis
Fdo. Alejand o Ca alá
Di ec o /a esis
ACKNOWLEDGEMENTS
Fi s and o emos , I am ex emely g a e ul o my supe iso s, D . José Ma ía Alonso Mo al
and D . Alejand o Ca alá o hei guidance and suppo h oughou my en i e doc o al s udy. I
would like o ex end my g a i ude o D . Ma ín Pe ei a-Fa iña o co-supe ising my doc o al
p ojec o abou wo yea s. The comple ion o his hesis would ha e been impossible wi hou
hei help ul ad ice o e he yea s.
I would like o ex end my since e hanks o all my ellow (ex-)colleagues om he Singula
Resea ch Cen e o In o ma ion Technology (Cen o Singula de In es igación en Tecnoloxías
In elixen es) o he Uni e si y o San iago de Compos ela whose company made me eel g ea ly
in eg a ed o he wo kplace and a beyond i . I also g a e ully acknowledge he assis ance o
he s a o he Uni e si y o San iago de Compos ela whose guidance was pa icula ly help ul
when esol ing a ious adminis a i e issues ha eme ged along he way.
Special hanks o he unding agencies ha p o ided inancial suppo o my doc o al p ojec .
Namely, I would like o exp ess my g a i ude o he Spanish S a e Resea ch Agency (Agencia
Es a al de In es igación) which unded, in i s en i e y, my p edoc o al con ac ( e .: FPI2019-
090153) as well as h ee na ionwide-le el p ojec s (ADHERE - U: RTI2018-099646-B-I00;
XAI4SOC: PID2021-123152OB-C21; DeepR3.gal: TED2021-130295B-C33) whe e I pa ic-
ipa ed as a esea ch assis an . In addi ion, I would like o ex end hanks o he Depa men o
Educa ion and Uni e si y Managemen (Conselle ía de Educación e O denación Uni e si a ia)
and he Eu opean Regional De elopmen Fund (ERDF), which unded he p ojec o excellence
“Explica-IA” (ED431F 2018/02) ha I had g ea pleasu e o co-wo king on.
Impo an ly, he p esen piece o esea ch is, in pa , a p oduc o ac i e in e na ional col-
labo a ion. I would hus like o ecognise he e o o membe s o he Labo a o y o he New
E hos (Wa saw Uni e si y o Technology, Poland) who played an impo an ole in shaping up
he me hodological basis o he a gumen a i e amewo k p oposed in his hesis. I would pa -
icula ly like o hank P o . D . Ka a zyna Budzynska, my secondmen supe iso , and D .
Ma cin Koszowy o hei ac i e coope a ion on he inal pa o my doc o al p ojec as well as
nume ous ou doo ac i i ies.
ILIA STEPIN
a i ma que as explicacións deben sa is ace unha se ie de p opiedades pa a que sexan e icaces.
En p imei o luga , as explicacións e ec i as deben se con as i as, é dici , non só explican
po que a decisión ou p edición au oma izada dada é o caso, senón que amén dan azóns po-
las que non son aplicables os esul ados al e na i os. Ademais, deben se selec i as, é dici ,
deben incluí só un núme o su icien emen e pequeno das causas ou ac o es máis ele an es que
conducen á decisión ou p edición dada. Po úl imo, pe o non menos impo an e, as explicacións
considé anse sociais, é dici , son p odu o da in e acción en e o explicado (o axen e que explica
o enómeno dado) e o explicado (o des ina a io da explicación). Como esul ado des a hipó ese,
as explicacións au oma izadas e án a máxima u ilidade se se modelan de aco do cos equisi os
desc i os an e io men e.
De aco do coa no ma i a ixen e da Escola de Dou o amen o In e nacional da Uni e sidade
de San iago de Compos ela, a p esen e ese p esén ase en o ma de compendio de publicacións.
En xe al, a ese di ídese en no e capí ulos. O Capí ulo 1 in oduce o ema da ese. No Capí-
ulo 2 exponse a hipó ese p obada e os obxec i os xe ais e especí icos da ese. No Capí ulo 3
desc íbese a me odoloxía aplicada pa a acada os obxec i os da ese. O capí ulo 4 o ece unha
discusión xe al sob e as p opiedades de explicación modeladas e as e amen as de xe ación
de explicacións desen ol idas nes a ese. O capí ulo 5 ecolle as p incipais con ibucións que
xu di on nes e p oxec o. Os capí ulos 6-8 es án elacionados coa me odoloxía e as conclusións
ecollidas nos a igos de e is as que o man o núcleo des a ese. O capí ulo 9 ex ae as p incipais
conclusións da p esen e ese. A con inuación, esumimos capí ulos 6-8 con máis de alle.
Nes a ese, le ouse a cabo o deseño, implemen ación e alidación dun no o ma co de xe ación
de explicacións que cump e odos os equisi os xe ais de explicación mencionados an e io men e
(é dici , as explicacións xe adas son con as i as, selec i as e sociais). De endemos o uso de
modelos in e p e ables baseados en eg as que se poidan u iliza (e cuxas p edicións poidan se
explicadas pos e io men e) de o ma independen e ou como ep esen a es pa a explica as p edi-
cións de algo i mos de “caixa neg a”. De calque a xei o, o ma co p opos o se e pa a explica
o esul ado dun clasi icado in e p e able (po exemplo, unha á bo e de decisións ou un sis ema
de clasi icación di uso baseado en eg as).
No con ex o de XAI, a p edición dun clasi icado pode explica se (non necesa iamen e de
o ma con as i a) en e mos dos azos máis ca ac e ís icos da ins ancia que conduci on á p edi-
ción dada. En dian e, e e i émonos a ales explicacións como ac uais. Pa a que as explicacións
esul an es sexan con as i as, buscamos modela explicacións complemen a ias ás ac uais, é
dici , que opoñen explíci amen e o esul ado da clasi icación ealmen e p edi o a esul ados al-
e na i os hipo é icos. Nou as palab as, a p edición do clasi icado dado explícase non só en
unción das ca ac e ís icas que son máis ele an es pa a a p edición, senón amén en e mos de
clasi icacións non p edi as. Ademais, ales explicacións poden suxe i cambios mínimos nos
alo es das ca ac e ís icas pa a que o esul ado p e is o cambie da o ma desexada. En XAI,
2
Resumo
es as explicacións amén se denominan comúnmen e como con a ac uais (CF). Nó ese que con-
a ac uais e í ense a exemplos xa obse ados no pasado, men es que ans ac uais e í ense a
exemplos sin é icos aínda non is os pe o que se espe a que se obse en no u u o. Non obs an e,
pa a simpli ica a no ación, emi i émonos ás explicacións CF no es o des a ese, sen impo a
se os cambios suxe idos in oluc an a c eación de exemplos sin é icos.
Pa a ins ilus a i os, conside emos un escena io banca io común. Unha clien e dun banco,
unha mulle de mediana idade cuxos ing esos son 4.000 eu os mensuais, solici a un p és amo
a un ano de 30.000 eu os. Ades ado pa a p e e se as solici udes de p és amo deben se acep-
adas ou exei adas, o sis ema de clasi icación banca ia suxi e que o unciona io banca io debe
exei a a solici ude da clien e. Pa a p opo ciona lle ao clien e a ecomendación máis ele an e
sob e como se pode ía modi ica a decisión, o sis ema p esen a a seguin e suxes ión de CF: “A
solici ude de p és amo da clien e se ía ap obada se os seus ing esos mensuais osen polo menos
5.000 eu os e i ese polo menos un p és amo ac i o menos”. Como se desp ende do exemplo
an e io , as explicacións de CF no con ex o de p oblemas de clasi icación son inhe en emen e
con as i as, xa que se opoñen de xei o explíci o a di e en es esul ados de clasi icación.
As explicacións con as i as e, máis especí icamen e, CF son es udadas dende hai moi o
empo nunha ampla gama de ciencias. Po exemplo, dise que o man pa e in eg an e do a-
zoamen o humano. A gumén ase amén que os con a ac uais ep esen an o ni el máis al o de
causalidade. Ademais, pódense xe a pa a calque a clasi icado . Po es es mo i os, as expli-
cacións CF chama on a a ención de nume osos in es igado es da XAI nos úl imos anos. Con-
encionalmen e, os con a ac uais considé anse explicacións agnós icas do modelo, pos -hoc e
locais. Son locais po que explican o compo amen o do sis ema a pa i das súas p edicións
indi iduais. Sábese que os con a ac uais explican as p edicións de o ma pos -hoc, xa que se
xe an despois de ob e a saída do sis ema. No ablemen e, es a amilia de explicacións é coñe-
cida, en xe al, po se independen e do modelo, xa que os co esponden es mé odos de xe ación
de explicacións es án deseñados pa a ope a só na en ada dada e na saída p e is a do sis ema
sen accede necesa iamen e aos elemen os in e nos do sis ema. Non obs an e, a di e sidade dos
mé odos de xe ación de explicacións con as i as e CF ecen emen e eme xen es mos a que
non se limi an necesa iamen e a es a de inición con encional.
No Capí ulo 6, e isamos as eo ías exis en es sob e a explicación con as i a e CF dunha
ampla gama de ciencias. Ademais, analizamos os ma cos compu acionais de úl ima xe ación
deseñados pa a a xe ación dos dous ipos de explicación mencionados an e io men e. Ademais,
inspeccionamos o g ao de sine xía en e os en oques eó icos da explicación con as i a e CF e
as súas con apa es compu acionais de úl ima xe ación.
Cabe des aca que as explicacións CF posúen unha se ie de p opiedades impo an es que
se poden u iliza como medidas de u ilidade da explicación ( alidez, p oximidade, accionabi-
lidade, di e sidade, po ci a algunhas). Nes a ese, cen ámonos na modelización de CFs que
3
ILIA STEPIN
se p esume que son su icien es co seguin e subconxun o de p opiedades. En p imei o luga ,
os CF deben se álidos, é dici , deben le a a p edicións co ec as co esponden es ao esul-
ado al e na i o desexado. En segundo luga , espé ase que unha explicación CF inclúa só un
conxun o de cambios mínimos nos pa es ca ac e ís ica- alo da ins ancia pa a que cambie a
clasi icación p e is a. De ei o, o axen e explicado es á in e esado en ecibi a explicación CF
máis ele an e pa a a ins ancia que se es á a conside a . Tendo en con a o exemplo banca io
an e io , men es que un CF que indica que os ing esos mensuais debe ían se de 6.000 eu os
ou máis segui án sendo álidos nes e escena io, o usua io inal es á in e esado en man e es e
alo o máis p óximo posible aos seus ing esos eais, polo que a explicación CF se ía p e e ible
indica que os ing esos mensuais debe ían se de 5.500 eu os (semp e que os dous CF sexan
álidos). En e cei o luga , espé ase que unha explicación CF sexa accionable, é dici , só as
ca ac e ís icas que se poidan modi ica de xei o iable o man pa e da explicación. De ei o,
se o sis ema suxi e que se eduza a idade do clien e, o CF co esponden e é inú il, aínda que
odos os demais cambios se poidan ace con éxi o. Po úl imo, espé ase que os con a ac uais
sexan di e sos, é dici , que ab angan a ios CF álidos dis in os (un pun o único ou ag upados
en conxun os) que eñan un pode explica i o equi alen e e, idealmen e, que se a open dispe -
sos nas exións de da os que cob en. De ei o, o usua io inal pode a opa explicacións máis
sa is ac o ias que con eñan non só alo es especí icos das ca ac e ís icas CF dadas, senón in-
e alos de ales alo es. A p esen ación de CFs an di e sas pode aumen a a lexibilidade do
compo amen o do usua io, xa que o des ina a io da explicación en a posibilidade a escolle o
escena io a segui que mello lle con eña. A di e sidade de explicacións de CF baseadas en
eg as pode mani es a se de a ias manei as. Po unha banda, as ca ac e ís icas de explicación
pódense ep esen a nume icamen e, en o ma de in e alos (po exemplo, “5.500 ≤ enda ≤
6.000”). Po ou a banda, pódense o ece no seu luga as co esponden es desc icións ex uais
(po exemplo, “a enda é al a”). En ambos casos, os CF xe ados au oma icamen e inclúen un
conxun o de da os CF que pe mi en ao usua io inal escolle o alo al e na i o máis axei ado
pa a as uncións dadas en e o in e alo de alo es suxe ido. Non obs an e, non es á cla o se
ales e ique as lingüís icas (é dici , “al a” do exemplo an e io ) an as ca ac e ís icas explica i as
co esponden es máis comp ensibles ou áciles de usa e, polo an o, a explicación xe al máis
e ec i a.
Pa a ace as explicacións selec i as, con iamos no uso de elemen os in e nos de modelos
baseados en eg as. Algúns des es algo i mos de clasi icación in e p e ables po deseño ag egan
in o mación sob e as ca ac e ís icas que son máis ele an es pa a a p edición dada. Po exem-
plo, as á bo es de decisión con eñen os alo es de ca ac e ís icas máis ele an es no camiño de
decisión. Polo an o, a explicación ác ual pódese econs uí esumindo a in o mación ag egada
no camiño desde a aíz a a o nodo olla p e is o. Non obs an e, a xe ación de explicacións CF
e ec i as pa a amilias especí icas de algo i mos in e p e ables, como á bo es de decisión ou
4
Resumo
sis emas de clasi icación baseados en eg as di usas, seguen sendo pouco es udadas. Ademais,
es es clasi icado es baseados en lóxica di usa o ecen e amen as que, po deseño, pe mi en aos
desen ol edo es cons i uí explicacións ex uais equi alen es u ilizando o epe o io de e mos
lingüís icos. Así, po encia á bo es de decisión (di usas) e clasi icado es baseados en eg as di-
usas con no os mé odos de xe ación de explicacións CF pe mí enos mello a aínda máis o seu
po encial explica i o.
No Capí ulo 7, p opoñemos es algo i mos de xe ación de explicacións CF que p oducen
explicacións ex uais ac uais e CF pa a clasi icado es baseados en eg as p eseleccionadas. En
p imei o luga , deseñamos un algo i mo (en dian e, denomínase XOR) que xe a explicacións CF
o denando as ep esen acións ec o izadas das eg as CF de aco do co seu g ao de ele ancia
pa a a ins ancia de p oba. Supoñemos que ales explicacións baseadas en eg as le an á xe ación
de explicacións CF álidas. Pos e io men e, in oducimos unha a ian e al e na i a do mesmo
algo i mo (en dian e, denomínase EUC) que elaciona os ec o es de uncións de pe enza di usa
coas eg as CF median e a medición da dis ancia euclidiana en e odos os pa es de ales ec-
o es. Pa a ambos algo i mos, p opoñemos ademais o mecanismo de ap oximación lingüís ica, é
dici , un mé odo pa a asocia in e alos de ca ac e ís icas numé icas a e mos lingüís icos. Es a
ex ensión pe mí enos xe a di e sas explicacións au omá icas equi alen es en o ma o numé ico
ou pu amen e ex ual.
A pesa de que os dous algo i mos in oducidos an e io men e o ecen explicacións acil-
men e in e p e ables, son especí icos do modelo, é dici , equi en acceso aos elemen os in e nos
do modelo e non se poden aplica di ec amen e a calque a clasi icado . Non obs an e, os algo i -
mos de xe ación de explicacións CF independen es do modelo son capaces de explica uni e -
salmen e a saída de calque a clasi icado an o de xei o ac ual como con a ac ual. Así, amén
p opoñemos un algo i mo de xe ación de explicacións CF xené ica independen e do modelo
(en dian e, denomínase GEN) que p oduce explicacións CF op imizando a poboación inicial de
o ma i e a i a a a que se iden i ique o único pun o de da os máis p óximo á ins ancia de p oba.
En conxun o, ambos g upos de algo i mos de xe ación de explicacións CF (é dici , aqueles es-
pecí icos do modelo e os agnós icos do modelo) poden usa se de o ma complemen a ia en e
si, especialmen e se as explicacións esul an es eñen en di e en es o ma os.
Co in de compa a a u ilidade das explicacións especí icas do modelo baseadas en coñece-
men os imp ecisos on e a ou as que apun an a pun os de da os especí icos, ealizamos dous
es udos de a aliación humana (é dici , Su ey GM eSu ey TS) onde compa amos a e icacia
das explicacións esul an es pa a as ins ancias de p oba p eseleccionadas. En ambos es udos,
ades amos sis emas de in e encia di usa de Mamdani que an p edicións u ilizando o algo-
i mo FURIA e xe an as explicacións co esponden es u ilizando odos os algo i mos p opos os
(é dici , XOR, EUC e GEN). Ades amos aos clasi icado es nun conxun o de da os de clasi i-
cación de ipos de ce exa pa a xe a pos e io men e explicacións lingüís icas pa a os es ímulos
5
ILIA STEPIN
da enquisa p edi os co ec amen e. Cabe sinala que odas as explicacións xe adas supoñense
accionables debido á es u u a do conxun o de da os u ilizado nos expe imen os: en calque a
caso, semp e é posible modi ica os alo es das ca ac e ís icas den o dos in e alos de alo es
suxe idos.
Su ey GM es á deseñado pa a que o usua io poida a alia a calidade da explicación au-
oma izada en base ás ca o máximas de G ice (can idade, calidade, ele ancia e o ma), que
ans o mamos en cinco aspec os explica i os (in o ma i idade, iabilidade, p ecisión, ele an-
cia e lexibilidade) pa a que os pa icipan es do es udo comp endan máis acilmen e a súa a e a
de a aliación. Du an e o es udo, os pa icipan es a alia on cada aspec o das es explicacións
(unha po cada mé odo de explicación p opos o) emp egando unha escala Like de 7 pun os.
Pola súa banda, Su ey TS é unha a ian e simpli icada de 5 pun os baseada na escala Like
emp egada en Su ey GM onde se a alía unha única explicación en e mos de iabilidade e
sa is acción xe al. Ambas enquisas le á onse a cabo pa a un público obxec i o que iña su i-
cien e coñecemen o do dominio e un al o ni el de expe iencia. A in es igación e os p o ocolos
expe imen ais o on ap obados polo comi é é ico da Uni e sidade de San iago de Compos ela.
Men es que se p opuxe on un g an núme o de mé icas compu ables au oma icamen e pa a
es ima a calidade das explicacións au oma izadas, es as adoi an se i pa a a alia a calidade
desde o pun o de is a algo í mico (po exemplo, a dis ancia xeomé ica á ins ancia de p oba).
Non obs an e, as mé icas au omá icas que es iman aspec os da pe cepción do usua io seguen
sendo escasas. Pa a abo da es e p oblema, p opoñemos a mé ica da complexidade da expli-
cación pe cibida, é dici , unha es imación do complexa que pa ece se unha explicación desde o
pun o de is a do usua io ao lela. A no a mé ica p opos a es á inspi ada no Gunning Fog Index
(un indicado da acilidade de en ende o ex o po pa e do público des ina a io). En pa icula ,
baséase en dous ac o es que es án p esen es nas explicacións ex uais baseadas en eg as xe-
adas median e os mé odos de xe ación de explicacións p opos os: a lonxi ude da explicación
e a elación ag egada do núme o de e mos lingüís icos de odas as ca ac e ís icas u ilizadas na
explicación. Os esul ados dos es udos de a aliación humana ealizados mos an que a com-
plexidade da explicación pe cibida en unha co elación posi i a mode ada coa in o ma i idade
es imada polo usua io e unha o e co elación nega i a coa ele ancia e a lexibilidade es imadas
polo usua io, men es que as pun uacións mé icas p opos as non se co elacionan coa iabilidade
ou a p ecisión. Polo an o, pódese concluí que a mé ica p opos a comp ende es dos aspec os
de explicación mencionados an e io men e pa a usua ios que eñan coñecemen os e expe iencia
su icien es no dominio. Ademais, usa a pun uación de complexidade da explicación pe cibida
pode se ú il pa a diminuí os cus os de a aliación humana (pa a o público obxec i o), xa que
se pode calcula pa a subs i uí (en pa e) os es udos de usua ios co esponden es.
Comp e sinala que os algo i mos p opos os no Capí ulo 7 limí anse á xe ación de expli-
cacións con as i as selec i as non sociais, é dici , ca ecen de calque a in e acción di ec a co
6
Resumo
usua io inal e só ep esen an os da os CF máis ele an es desde o pun o de is a algo í mico.
Nes as con igu acións, o usua io inal en que oma unha decisión sob e a iabilidade das expli-
cacións au oma izadas o ecidas sob e a base dunha única in o mación. Se a explicación non se
conside a o su icien emen e iable ou sa is ac o ia, o usua io inal pode que e desca ala aínda
que sexa álida. Polo an o, é indispensable que o usua io inal explo e o espazo de explicación
se o conside a necesa io.
Pa a ace sociais as explicacións esul an es, modelamos a in e acción en e o sis ema e o
usua io en o ma de diálogo explica i o a gumen a i o onde o usua io é capaz de discu i sob e
pezas de explicación especí icas e, des e xei o, explo a o espazo de explicación a a que poida
oma unha decisión in o mada sob e a p edición do sis ema. Pa a iso, ampliamos os nosos algo-
i mos de xe ación de explicacións cun módulo de xe ación de diálogos explica i os. Deseñado
como un axen e con e sacional, o modelo de diálogo esul an e ga an e unha comunicación di-
alóxica in e ac i a en e o sis ema e o usua io onde es e es á habili ado pa a solici a e p ocesa
as explicacións necesa ias o ecidas de o ma comp ensible e humana. Ademais, es a ex en-
sión mello a o ma co de xe ación de explicacións p opos o coa opción de o ma explicacións
in e ac i as dinámicas en con as e coas xené icas es á icas.
No Capí ulo 8 p opoñemos o denominado “xogo de diálogo explica i o”, un modelo o mal
de diálogo explica i o baseado no en oque co esponden e á modelización do diálogo a pa i
da eo ía da a gumen ación. O modelo de diálogo es á deseñado de o ma descenden e, é dici ,
baséase nun p o ocolo de diálogo p ede inido que con én ca o posibles solici udes de usua-
ios (as de explicación ac ual ou CF, de alle, acla ación e explicación al e na i a), con a ias
posibles espos as do sis ema asociadas a dis in as solici udes do usua io. Ademais, xene aliza-
mos o modelo de diálogo explica i o na o ma dunha g amá ica de diálogo sen con ex o pa a
acelo uni e salmen e aplicable á saída de calque a sis ema de clasi icación baseado en eg as
mello ado cun explicado que é capaz de p oduci explicacións ex uais baseadas en eg as.
Validamos o modelo de diálogo esul an e ealizando un es udo de a aliación humana me-
dian e es casos de uso: clasi icación da posición do xogado de balonces o, clasi icación do ipo
da ce exa e clasi icación da en e midade da i oide. Nes es escena ios de diálogo de busca de
in o mación, unha das pa es do diálogo (nes e caso, o usua io) é inicialmen e in o mada sob e
os da os que se es án p ocesando e despois p e ende da sen ido á in o mación dada pola ou a
pa e (nes e caso, a p edición do sis ema). Nos es casos de uso, os diálogos p e enden explica
a p edición dun único sis ema pa a unha ins ancia de da os p e iamen e seleccionada e clasi i-
cada co ec amen e. Como nes e expe imen o se abo da o aspec o comunica i o da xe ación de
explicacións, u ilízanse á bo es de decisión ní idas como clasi icado es, xun o co mé odo XOR
emp egado pa a xe a as explicacións co esponden es, co in de ga an i a anspa encia dos
esul ados expe imen ais.
Pa a analiza as ansc icións de diálogos ecollidas, aplicamos écnicas de mine ía de p o-
7
ILIA STEPIN
cesos que a an as ins ancias de diálogo explica i o como íos de p oceso. En pa icula , eali-
zamos a denominada comp obación de con o midade pa a elaciona o p o ocolo de diálogo e o
co pus de diálogos explica i os ealmen e exis ados. Concluimos que os pa icipan es no es-
udo an un uso ac i o de odos os ipos de solici udes o e adas. Polo an o, o p ocedemen o de
e i icación da con o midade con i ma a u ilidade do modelo de diálogo p opos o inicialmen e
na súa o alidade. Ademais, exis amos un g an núme o de solici udes de explicacións de CF
al e na i as (segunda e e cei a mello clasi icadas polo sis ema) pa a case odas as clases de CF
en odos os casos de uso. Es a obse ación apun a ademais á necesidade de p esen a di e sos
CFs múl iples en diálogos de busca de in o mación.
Cabe des aca que odos os aballos de e is as e so wa e que cons i úen a base da p esen e
ese es án a disposición do público. Finalmen e, ealizamos obse acións inais e esbozamos
di eccións pa a o aballo u u o no Capí ulo 9.
8
Summa y
A i icial In elligence (AI) plays an inc easingly impo an ole in a la ge numbe o daily li e
ac i i ies. AI applica ions a e ound in nume ous p oduc s, om banking o heal h ca e o manu-
ac u ing o educa ion. Howe e , he as p og ess o he p esen -day AI aises conce ns ela ed
o i s in e p e abili y and explainabili y. On he one hand, AI models a e apidly becoming
o e ly complex o a gene al audience o unde s and he na u e o he decision-making sys ems
ha hey make pa o . This may unde mine us in au oma ed decisions p oduced by such
sys ems and inc eases eluc ance o use hem. On he o he hand, shi ing om ha d-coded ule-
based o da a-d i en machine lea ning (ML)-based AI algo i hms has esul ed in he na u e o
such algo i hms being concealed e en om hei de elope s.
In o de o demys i y such “black-box” algo i hms o bo h lay use s and domain expe s,
esea che s om nume ous ields o science called o making he p esen -day AI explainable.
This esul ed in coun less esea ch p ojec s o ming he basis o he ecen ly eme ged eXplain-
able AI (XAI) communi y. In line wi h scien i ic aspi a ions, he ubiqui ous use o AI has led
o majo changes in legal egula ion, which a e e lec ed in, o example, he Eu opean Union’s
(EU) Gene al Da a P o ec ion Regula ion (GDPR) o he ecen ly p oposed A i icial In elli-
gence Ac (AIA) which was o ed o by he EU Pa liamen in June 2023, being in he inal
s age be o e becoming law and coming in o o ce in each membe s a e.
Poo explana o y capaci ies o “black-box” AI models ha e mo i a ed discussions on a
a ou able use o so-called in e p e able models, ins ead. He eina e , he concep o in e -
p e able models e e s o he amily o algo i hms ha g an access o hei human-comp ehensi e
in e nals. Indeed, mo e in e p e able bu (possibly) less accu a e ML models may appea o be
mo e e icien ly applicable o sol ing a ious challenging p oblems han mo e obus bu less
anspa en algo i hms, especially in cases o high-s akes decisions. Ne e heless, such explana-
ion gene a ion- ela ed sub- asks as, o example, e alua ion and communica ion emain being
demanding asks e en o in e p e able models.
A la ge body o in e disciplina y esea ch on he na u e o explana ion claims ha explana-
ions should sa is y a numbe o p ope ies o hem o be e ec i e. Fi s , e ec i e explana ions
a e claimed o be con as i e, i.e. hey do no only explain why he gi en au oma ed decision
o p edic ion is he case bu also gi e easons why al e na i e ou comes a e no applicable. In
ILIA STEPIN
addi ion, explana ions should be selec ed, i.e. hey should include only an adequa ely small
numbe o he mos ele an causes o ac o s ha lead o he gi en decision o p edic ion. Las
bu no leas , explana ions a e deemed social, i.e. hey a e a p oduc o in e ac ion be ween he
explaine ( he agen ha explains he gi en phenomenon) and he explainee ( he ecipien o he
explana ion). As a esul , au oma ed explana ions a e hypo hesised o ha e maximal u ili y i
modelled in acco dance wi h he equi emen s ou lined abo e.
In acco dance wi h he ac ual egula ions o he in e na ional doc o al school o he Uni e -
si y o San iago de Compos ela, his hesis is p esen ed in o m o a compendium o publica ions.
O e all, i is di ided in o nine chap e s. Chap e 1 in oduces he opic o he hesis. Chap e 2
s a es he hypo hesis es ed and he gene al and speci ic objec i es o he hesis. Chap e 3 de-
sc ibes he me hodology applied o each he hesis objec i es. Chap e 4 p o ides he eade
wi h a gene al discussion on he explana ion p ope ies modelled and he explana ion gene a ion
ools de eloped in his hesis. Chap e 5 lis s main con ibu ions ha eme ged as pa o he doc-
o al p ojec . Chap e s 6-8 ela e o he me hodology and indings epo ed in he jou nal pape s
ha o m he co e o his hesis. Chap e 9 d aws main conclusions om he p esen hesis. Le
us now summa ise Chap e s 6-8 in mo e de ail.
We design, implemen , and alida e a no el explana ion gene a ion amewo k whose ou pu
explana ions a e claimed o mee all he a o emen ioned gene al equi emen s o explana ion (i.e.
being con as i e, selec ed, and social). We ad oca e he use o in e p e able ule-based models
ha can be used (and whose p edic ions can be subsequen ly explained) independen ly o as
p oxies o explain p edic ions o “black-box” algo i hms. Ei he way, he p oposed amewo k
se es he pu pose o explaining he ou come o a gi en ule-based in e p e able classi ie (e.g.,
a decision ee o a uzzy ule-based classi ica ion sys em).
In he con ex o XAI, a classi ie ’s p edic ion can (no necessa ily con as i ely) be ex-
plained in e ms o he mos cha ac e is ic ea u es o he es ins ance ha led o he gi en
p edic ion. He eina e , we e e o such explana ions as ac ual. To make he ou pu explana-
ions con as i e, we pay pa icula a en ion o modelling explana ions ha a e complemen a y
o ac ual ones, i.e. hey explici ly oppose he ac ually p edic ed classi ica ion ou come o hy-
po he ical al e na i e ou comes. In o he wo ds, he gi en classi ie ’s p edic ion is explained
no only in e ms o he ea u es ha a e he mos ele an o he p edic ion bu also in e ms o
non-p edic ed classi ica ions. Fu he , such explana ions can sugges minimal changes in ea u e
alues so ha he p edic ed ou come changes in a desi ed way. In XAI, hese a e also commonly
e e ed o as he so-called coun e ac ual (CF) explana ions. No ice ha , coun e ac uals e e
o examples al eady obse ed in he pas while ans ac uals e e o syn he ic examples no seen
ye bu expec ed o be obse ed in he u u e. Anyway, o simplici y o no a ion, we will e e
o CF explana ions in he es o his hesis, no ma e i sugges ed changes in ol e he c ea ion
o syn he ic examples.
10
Summa y
Fo illus a i e pu poses, le us conside a common banking scena io. A clien o a bank, a
middle-aged woman whose income equals €4.000 pe mon h, is applying o a one-yea loan o
€30.000. T ained o p edic whe he loan applica ions should be accep ed o ejec ed, he bank-
ing classi ica ion sys em sugges s ha he bank o ice should decline he clien ’s eques . To
p o ide he clien wi h he mos ele an ecommenda ion o how he decision can be changed,
he sys em ou pu s he ollowing CF sugges ion: “The clien ’s loan applica ion would be ap-
p o ed i he mon hly income we e a leas €5.000 and i she had a leas one ac i e loan less.”
As ollows om he example abo e, CF explana ions in he con ex o classi ica ion p oblems
a e inhe en ly con as i e, as hey explici ly oppose di e en classi ica ion ou comes.
Con as i e and, mo e na owly, CF explana ions ha e long been s udied in a wide ange o
sciences. Fo ins ance, hey a e claimed o make an in eg a i e pa o human easoning. Fu he ,
coun e ac uals a e a gued o ep esen he opmos le el o causa ion. In addi ion, hey can be
gene a ed o any classi ie unde conside a ion. Fo hese easons, hey ha e a ac ed a en ion
o nume ous XAI esea che s in ecen yea s. Con en ionally, coun e ac uals a e conside ed
local pos -hoc model-agnos ic explana ions. They a e local because hey explain he sys em’s
beha iou on he basis o i s indi idual p edic ions. Coun e ac uals a e known o explain p e-
dic ions in a pos -hoc manne , as hey a e gene a ed a e he sys em’s ou pu has been ob ained.
Rema kably, his amily o explana ions is, in gene al, known o be model-agnos ic, since he
co esponding explana ion gene a ion me hods a e designed o ope a e only on he gi en inpu
and p edic ed ou pu o he sys em wi hou necessa ily accessing he sys em’s in e nals. How-
e e , he di e si y o he newly eme ging con as i e and CF explana ion gene a ion me hods
shows ha hey a e no necessa ily limi ed o his con en ional de ini ion.
In Chap e 6, we e iew exis ing heo ies o con as i e and CF explana ion om a wide
ange o sciences. Fu he , we analyse he s a e-o - he-a compu a ional amewo ks designed
o gene a ion o he wo a o emen ioned kinds o explana ion. In addi ion, we he ein inspec
he deg ee o syne gy be ween he heo e ical app oaches o con as i e and CF explana ion and
hei s a e-o - he-a compu a ional coun e pa s.
No ewo hy, CF explana ions possess a numbe o impo an p ope ies ha can be used as
measu es o explana ion u ili y ( alidi y, p oximi y, ac ionabili y, di e si y, o name a ew). In
his hesis, we ocus on modelling CFs ha a e hypo hesised o su ice he ollowing subse o
such p ope ies. Fi s , CFs mus be alid, i.e. hey mus lead o co ec p edic ions co esponding
o he desi ed al e na i e ou come. Second, a CF explana ion is expec ed o include only a se
o minimal changes o he es ins ance ea u e- alue pai s o he p edic ed classi ica ion o
change. Indeed, he explainee is in e es ed in ecei ing he piece o CF explana ion ha is he
mos ele an o he es ins ance unde conside a ion. Conside ing he banking example abo e,
whe eas he CF s a ing ha he mon hly income should be €6.000 o mo e will s ill be alid in his
scena io, he end use is in e es ed in keeping his alue as close as possible o he ac ual income,
11
ILIA STEPIN
In his ega d, he p ope y o con as i eness is a he co e o he so-called coun e ac ual ex-
plana ions (o coun e ac uals, o CFs, o sho ), a sub-g oup o con as i e explana ions ha
sugges minimal changes o he inpu ea u e alues so ha he ou pu changes in he desi ed
way [33]. Con as i e by na u e, CFs a e ound o be inhe en o human easoning [4] and can
he e o e g ea ly acili a e explana ion p ocessing by end use s [5]. Fo hese easons, explain-
ing p edic ions coun e ac ually has become among key explainabili y issues, especially when
explaining “black-box” models [21].
Gi en an e iden lack o anspa ency in he easoning o many complex AI models (e.g.
neu al ne wo ks), he use o possibly less accu a e o obus bu mo e in e p e able models has
been ac i ely a gued o [32]. In ligh o his, we explo e he po en ial o ule-based classi ica ion
sys ems o p o ide hei end use s wi h au oma ed explana ions o hei p edic ions. Indeed, he
po en ial o in e p e able models o explana ions is le la gely unde explo ed [19]. Enhancing
ule-based classi ica ion sys ems wi h e ec i e me hods o CF explana ion gene a ion allows
hem o become sel -explana o y while p o iding hei end use s wi h con as i e selec ed ex-
plana ions. Fu he , such sel -explana o y ule-based in e p e able classi ie s can hen be used
as pa o mo e complex explaine s o add ess he issue o explanabili y o “black-box” mod-
els. In o de o p ese e he s a e-o - he-a le els o pe o mance while gaining explainabili y,
“black-box” models can be enhanced wi h explana ion gene a ion modules ha make use o
(possibly, su oga e) in e p e able models [9]. Such in e p e able models (e.g., decision ees
o uzzy ule-based classi ica ion sys ems) [2] ha e shown o e ec i ely explain “black-box”
models in a pos -hoc manne when, o example, ained on a local neighbou hood a ound he
es ins ance [41]. In his ega d, hey can se e as a p oxy o app oxima e gi en single “black-
box”-based p edic ions.
As he need o explaining decisions made by AI-based sys ems is ecognised legally, a ious
esea che s a e u ging o making a s ep o wa d owa ds esponsible, human-cen ic AI [6].
Whe eas se e al au oma ic me ics ha e been designed o es ima e he quali y o au oma ed CF
explana ions wi h espec o hei compu a ional aspec s [26], human e alua ion emains among
he key challenges o uly e ec i e CF explana ion gene a ion [43]. Indeed, only a limi ed
numbe o s a e-o - he-a CF explana ion algo i hms ha e unde gone assessmen by po en ial
bene icia ies o such explana ions [16].
Human e alua ion o au oma ed explana ions is closely connec ed wi h he social aspec
o explana ion. I is o en add essed in XAI by means o engaging he end use in explana-
o y dialogue wi h he sys em [42]. Fu he , insigh s om humani ies and social sciences (e.g.,
a gumen a ion) allow us o p opose explana o y dialogue models ha ely on a consolida ed
body o knowledge abou human easoning and connec i o ha o an AI-based agen . In ac ,
a gumen a ion makes an in eg a i e pa o ce ain explana ion heo ies and he e o e appea s
o be a sui able me hodological i o b idge he gap be ween he explaine ( he explana ion
18
Chap e 1. In oduc ion
gene a ion module) and he explainee ( he end use ). Despi e speci ic me hodological di e -
ences, a gumen a ion and explana ion a e ound o g ea ly comple emen each o he [3]. Fo
example, some heo ies o explana ion concep ualise explana ions as a gumen s [12]. Whe eas
a gumen a ion heo ies p o ide a di e se epe oi e o amewo ks ha is capable o gene a -
ing explana ions o au oma ic p edic ions in a wide ange o asks [46], we aim o explo e i s
po en ial as a communica ion channel be ween he end use and he sys em o enhance he p e i-
ously designed amewo k o con as i e-coun e ac ual selec ed explana ions o in e p e able
ule-based classi ica ion sys ems wi h a social dimension.
1.2 THESIS STRUCTURE
The p esen hesis con ains nine chap e s. The emainde o he hesis is s uc u ed as ollows.
Chap e 2 s a es he hypo hesis es ed in his hesis as well as he gene al and speci ic objec-
i es. As we lis he objec i es o he hesis, we e e he eade o he publica ions whe e he
objec i es we e eached.
Chap e 3 desc ibes he gene al me hodology applied h oughou he hesis and desc ibes
speci ic ools ha we e used in o de o each he hesis objec i es.
Chap e 4 p o ides he eade wi h a gene al discussion on explana ion p ope ies in he
con ex o XAI and analyses in de ail he s eng hs and weaknesses o hei modelling in his
hesis.
Chap e 5 lis s he con ibu ions o his hesis, i.e., he so wa e de eloped o each he hesis
objec i es and all he publica ions ha eme ged du ing he doc o al p ojec .
Chap e 6 p o ides he eade wi h he backg ound in o ma ion on con as i e and CF ex-
plana ions. In addi ion, i examines heo e ical ounda ions he eo , he ela ed s a e-o - he-a
compu a ional amewo ks, and inspec s he deg ee o syne gy be ween he o me and he la e .
Chap e 7 in oduces h ee algo i hms o CF explana ion gene a ion (namely, XOR, EUC,
and GEN) used o explain p edic ions o an FRBCS. In addi ion o discussing echnicali ies o
he a o emen ioned algo i hms, i e alua es he algo i hms ia wo human e alua ion s udies.
Fu he , i p oposes a no el me ic o pe cei ed explana ion complexi y (PEC) ha aims o
acili a e e alua ion o au oma ically gene a ed explana ions.
Chap e 8 p oposes an a gumen a i e amewo k o communica ion o au oma ically gen-
e a ed ule-based explana ions. In pa icula , i o malises explana o y dialogue in o m o he
so-called “dialogue game” and desc ibes in de ail he co esponding dialogue p o ocol. Fu he ,
i addi ionally ep esen s he p o ocol in o m o con ex - ee dialogue g amma o make he
p o ocol uni e sally applicable o o he explaine -classi ie pai s ha a e capable o gene a ing
ex ual ule-based explana ions. Las bu no leas , i epo s he esul s o a human e alua ion
expe imen ha se es he pu pose o alida ion o he p oposed explana o y dialogue model.
19
ILIA STEPIN
Finally, Chap e 9 p esen s main conclusions de i ed om he esul s o he doc o al p ojec
and ou lines p ospec i e di ec ions o u u e wo k ha a e ele an o he p oblems o CF ex-
plana ion gene a ion, communica ion, and e alua ion.
20
2 Hypo hesis and objec i es
In his hesis, we de elop an explana ion gene a ion amewo k o in e p e able (i.e., “whi e-
box”) classi ie s (e.g., decision ees) and semi-in e p e able (i.e., “g ey-box”) ule-based clas-
si ica ion sys ems (e.g., uzzy in e ence sys ems). Despi e he ac ha such models p o ide
p edic ions ha can be easily in e p e ed ac ually, hei CF po en ial emains unde s udied. We
o mula e he main hypo hesis es ed in he p esen hesis as ollows: “By modelling explana-
ions sa is ying he p ope ies speci ically ele an o XAI and enhancing hem wi h dialogic
in e ac i e acili ies, we can con ey bo h ac ual and CF explana ions ha a e appealing o a
good numbe o use s in di e en applica ion domains”.
The gene al objec i e o he p esen doc o al hesis is o ad ance s a e-o - he-a XAI ech-
nologies o (1) au oma ic gene a ion o ac ual and CF explana ions o in e p e able ule-based
classi ie s and (2) e ec i e and comp ehensi e communica ion o such explana ions. The im-
plemen ed explana ion gene a ion amewo k is expec ed o ou pu explana ions ha sa is y he
a o emen ioned equi emen s o e ec i e explana ions (i.e. being con as i e, selec ed, and so-
cial). Mo e p ecisely, he ollowing speci ic objec i es a e conside ed o achie e he o e all
goal:
O1. Design, implemen , and alida e a amewo k o ac ual and CF explana ion gene a ion
applied o gi en p e ained (semi-)in e p e able ule-based classi ie s. This objec i e has
been success ully eached in he ollowing publica ions:
• Ilia S epin, Jose M. Alonso, Alejand o Ca ala, and Ma ín Pe ei a-Fa iña. “A Su -
ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explain-
able A i icial In elligence”. IEEE Access, ol. 9, pp. 11974–12001, 2021. DOI:
10.1109/ACCESS.2021.3051315;
• Ilia S epin, Jose M. Alonso-Mo al, Alejand o Ca ala, Ma ín Pe ei a-Fa iña. “An
empi ical s udy on how humans app ecia e au oma ed coun e ac ual explana ions
which emb ace imp ecise in o ma ion”. In o ma ion Sciences, ol. 618, pp. 379–
399, 2022. DOI: 10.1016/j.ins.2022.10.098.
O2. Design, implemen , and alida e a con e sa ional agen endowed wi h c edibili y ia na -
u al language p ocessing and a gumen a ion echnologies in he con ex o XAI in o de
ILIA STEPIN
o communica e and cus omise au oma ically gene a ed explana ions. This objec i e has
been success ully eached in he ollowing publica ion:
• Ilia S epin, Ka a zyna Budzynska, Alejand o Ca ala, Ma ín Pe ei a-Fa iña, Jose
M. Alonso-Mo al. “In o ma ion-seeking dialogue o explainable a i icial in elli-
gence: Modelling and analy ics”. A gumen and Compu a ion, in p ess, 2023. DOI:
10.3233/AAC-220011.
O3. De elop a human e alua ion amewo k o he pu pose o alida ion o he algo i hms de-
signed o achie e O1 and O2. This e alua ion amewo k se es he pu pose o es ima ing
a ious aspec s o au oma ically gene a ed explana ions and ha o assessing he quali y
o he p ocess o communica ion o such explana ions, espec i ely. This objec i e has
been success ully eached in he publica ions add essing bo h O1 and O2.
22
3 Me hodology
The me hodology employed in his hesis bases on he i e a i e de elopmen app oach. In o de
o achie e he objec i es lis ed in Sec ion 2, he wo k on each o hem p esupposes he ollowing
consecu i e s eps:
1. Requi emen speci ica ion: de ining he esea ch objec i e while conside ing possible
limi a ions o he co esponding s a e-o - he-a AI echniques and so wa e. Special a -
en ion is paid o aspec s o au oma ic explana ion gene a ion and communica ion (i.e.,
p ope ies o explana ion, equi emen s o he communica i e aspec s o he language
used o modelling a gumen a i e explana o y dialogue, use cases, a ge audience, e c.);
2. Li e a u e e iew: a bibliog aphic s udy ha se es o iden i y he s a e-o - he-a con-
cep ual, heo e ical, and compu a ional amewo ks designed o add ess he goal-speci ic
p oblems. This s udy includes an analysis o he ad an ages and disad an ages o he
iden i ied me hods;
3. Implemen a ion: design and de elopmen o concep ual models and algo i hms aimed
a p oducing ad ances wi h espec o he s a e-o - he-a . Theo e ical con ibu ions a e
ollowed by so wa e implemen a ions o be alida ed empi ically a he nex s ep;
4. Valida ion: he p ocess o e i ica ion o he so wa e as well as e ision o he algo i hm
o model implemen ed i he expe imen al esul s ob ained a e no sa is ac o y;
5. In eg a ion: once alida ed, he new algo i hms o p o o ypes a e in eg a ed wi h hose
alida ed p e iously so ha hey make pa o he uni ied amewo k.
In o de o achie e objec i e O1, we i s pe o m a sys ema ic li e a u e e iew (SLR) o
he s a e-o - he-a me hods o con as i e and CF explana ion gene a ion in he con ex o XAI
ollowing he guidelines o pe o ming SLRs in so wa e enginee ing [17, 18]. In pa icula , we
o mula e a se ies o esea ch ques ions o be add essed, design a sea ch s a egy, and ex ac and
syn hesise da a in acco dance wi h he p ede ined inclusion and exclusion c i e ia. In addi ion,
we pe o m he so-called “snowballing” p ocedu e, i.e. a e ision o he bibliog aphy lis s o he
p e iously collec ed s udies, ollowing he co esponding guidelines [45]. Subsequen ly, based
ILIA STEPIN
on he insigh s om ou SLR, we de elop a concep ual amewo k o ac ual and CF explana ion
gene a ion o in e p e able ule-based classi ie s ha ou pu s con as i e selec ed explana ions.
The amewo k includes one model-agnos ic and wo model-speci ic explana ion gene a ion
algo i hms. To gene a e au oma ic explana ions in na u al language, we adap one o he mos
commonly used na u al language gene a ion (NLG) pipelines [7, 31]. We op ed o using a
empla e-based NLG app oach ins ead o an end- o-end neu al NLG app oach o maximise he
ideli y o he da a o he gene a ed na a i es e sus hei na u alness. Acco dingly, he use o
p e- ained la ge language models o NLG alls ou side he scope o his hesis. Consequen ly,
we e alua e he p oposed me hods in a se ies o human e alua ion expe imen s con ibu ing o
eaching objec i e O3, as we adhe e o he human e alua ion guidelines designed speci ically
o XAI [13].
To accomplish objec i e O2, we de elop an a gumen a i e con e sa ional agen elying
on he “dialogue game”-based heo e ical app oach o a gumen a i e dialogue modelling [30].
Mo e speci ically, we design a se o o iginal eques s and esponses ha cons i u e a newly
p oposed explana o y dialogue p o ocol. Simila ly o O1, we e alua e he designed a gumen-
a i e amewo k by pe o ming a human e alua ion s udy and analyse he collec ed dialogue
ansc ip s. In addi ion, we employ concep s om p ocess mining o pe o m he so-called “con-
o mance checking” ea ing ins ances o he collec ed explana o y dialogues as p ocesses [27].
To each objec i e O3, we design a so wa e amewo k elying on human e alua ion guide-
lines o XAI [13] and implemen i as a s and-alone applica ion. I s lexible s uc u e allows
us o adap i o he needs o speci ic expe imen s ca ied ou as pa o he p esen hesis. As
a esul , we use i o e alua e all he CF explana ion gene a ion and communica ion me hods
p oposed in his hesis.
In o de o see he connec ion be ween he esea ch ques ions posed and he ools de eloped
o answe hem (as well as he publica ions add essing hem), we kindly e e he eade o
Fig. 5.1 om Chap e 5 which lis s he main con ibu ions o his hesis.
24
4 Gene al discussion
In his hesis, we explo e he po en ial o in e p e able ule-based classi ica ion sys ems o gen-
e a e e ec i e au oma ed explana ions in acco dance wi h ecen equi emen s o he quali y o
such explana ions. Recall ha e ec i e explana ions a e claimed o be con as i e, selec ed, and
social [23]. In his sec ion, we discuss peculia i ies o modelling au oma ed explana ions ensu -
ing encapsula ion o hese key p ope ies in he con ex o he explana ion gene a ion amewo k
o in e p e able ule-based classi ie s de eloped in his hesis. In pa icula , we i s inspec
compu a ional aspec s o con as i e and CF explana ions o classi ica ion p oblems. Then, we
discuss how one-sho selec ed CF explana ions can be gene a ed o p edic ions made by in e -
p e able ule-based classi ica ion sys ems. Finally, we inspec how a social laye can be added
on op o he p e iously p oposed explana ion amewo k. Mo e speci ically, we ocus on ex-
plana o y dialogue modelling as a means o explana ion communica ion. We hus examine (1)
he mos ele an aspec s o o mal explana o y dialogue modelling o ex ual ule-based expla-
na ions and (2) a gene al a chi ec u e o he co esponding a gumen a i e con e sa ional agen .
As a esul , we p opose an explana ion gene a ion amewo k whose ou pu emb aces all he
a o emen ioned p ope ies o explana ions o in e p e able ule-based classi ie s (see Fig. 4.1).
The gene al discussion on enhancing au oma ed explana ions wi h he a o emen ioned p op-
e ies is s uc u ed in he ollowing manne . Sec ion 4.1 discusses compu a ional aspec s o CF
explana ions and hei connec ion wi h he amily o con as i e explana ions as de ined in he
li e a u e. Sec ion 4.2 examines aspec s o one-sho , selec ed CF explana ion gene a ion o
in e p e able ule-based classi ica ion sys ems. Sec ion 4.3 explo es how insigh s om a gu-
men a ion heo y can enhance he social aspec o au oma ed explana ions in he con ex o
explana o y dialogue sys ems.
4.1 MAKING EXPLANATIONS CONTRASTIVE(-COUNTERFACTUAL)
Explana ions a e a gued o be con as i e, as he ac s ha hey explain a e (some imes, im-
plici ly) opposed o pieces o in o ma ion ela ed o a con as case [23]. They can be de ined
as answe s o he con as i e why-ques ion (i.e., “Why P a he han Q?”) whe e Pis he ac
being explained and Qis an al e na i e non-obse ed oil. Then, Pis said o be explained
ILIA STEPIN
Figu e 4.1: Explana ion p ope ies ha he p oposed a gumen a i e con e sa ional agen emb aces.
con as i ely i a causal di e ence be ween Pand no -Q is e i ied ( he so-called “di e ence
condi ion”) [20]. Ensu ing ha au oma ically gene a ed explana ions a e con as i e is claimed
o g ea ly con ibu e o hei eadabili y, as con as i e explana ions a e said o p une he space
o all causal ac o s aiding ine -g ained unde s anding [15].
Theo ies o con as i e explana ion la gely ely on causal accoun s o explana ion. How-
e e , a numbe o exis ing explana ion gene a ion algo i hms o e pu ely non-causal explana-
ions [25]. Fu he , he no ion o con as i e explana ion is ound o g ea ly o e lap wi h ha o
CF explana ion in he XAI communi y, despi e ce ain me hodological di e ences. Thus, causal
accoun s o con as i e explana ion imply ha CF explana ions can be deemed con as i e so
long as hey espec he di e ence condi ion [24]. Ne e heless, CF explana ions a e imposed
a numbe o addi ional cons ain s ha make hem a sub-g oup o con as i e explana ions i
causali y is no being conside ed.
In he con ex o classi ica ion p oblems, con as i e and/o CF explana ions di e en ia e he
gi en piece o ac ual in o ma ion (e.g., “You loan applica ion has been ejec ed because you
mon hly income is oo low”) om some CF in o ma ion (e.g., “You loan applica ion would ha e
been app o ed i you had a leas one ac i e loan less”). Al oge he , a combina ion o ac ual
and CF explana ions enables he end use o cons uc a men al ep esen a ion o he AI-based
agen ’s easoning o all possible ou comes.
CF explana ions a e a powe ul ool o explaining p edic ions o decisions made by AI-
based agen s. In addi ion o ac ual explana ions, hey p o ide complemen a y in o ma ion o
26
Chap e 4. Gene al discussion
explain he AI-based agen ’s easoning, so ha a combina ion o ac ual and CF explana ions
can explain all possible p edic ions o he AI-based agen i espec i e o whe he hey ook
place o no . CF explana ions a e said o be pos -hoc (i.e., hey explain p edic ions o p e ained
models) and local (i.e., hey a e designed o explain indi idual p edic ions). In addi ion, CFs
can be model-agnos ic (i.e., he co esponding explana ion gene a ion me hod ope a es only on
he inpu ea u e alues and he ou pu p edic ion o he classi ie ) o model-speci ic (i.e., he
explaine also has access o he classi ie ’s in e nals). Au oma ed CF explana ions a e claimed
o ha e a numbe o p ope ies (see [8, 26] o an exhaus i e lis he eo ):
•Validi y. A CF explana ion is said o be alid i he co esponding CF leads o he desi ed
CF p edic ion;
•P oximi y. A CF is said o be p oxima e i he dis ance be ween he es ins ance and he
CF da a poin ha he gi en CF explana ion is ela ed o is as small as possible;
•Spa si y. A CF is said o be spa se i i con ains he minimal numbe o ea u es in
compa ison o o he alid CFs;
•Di e si y. CFs a e said o be di e se i hey o m a se o alid CF da a poin s ha
a e a he same ime maximally di e en om each o he so ha he explainee is o e ed
a numbe o legi sugges ions on how o change he gi en p edic ion o he al e na i e
desi ed one.
Whe eas he quan i a i e me ics de ined o measu e he p ope ies lis ed abo e a e com-
monly used o e alua ing CFs [43], some may be incompa ible wi h o he s. Fo example, he e
exis s a ade-o be ween p oximi y and di e si y: no CF explana ion gene a ion me hod is
claimed o maximise bo h due o hei di e gen ea men o CFs wi h espec o he dis ance
om he es ins ance [26]. In his case, o he ac o s (e.g., he a ge audience o he applica ion
domain) may become c ucial o es ima e he quali y o au oma ed CFs.
In addi ion, esea che s dis inguish a ious o he p ope ies ha au oma ed CF explana ions
a e desi ed o ha e. These include ac ionabili y (i.e., he me e abili y o change he ea u e alues
as he gi en CF explana ion sugges s), causali y (i.e., es ablishing causal ela ions be ween he
ea u es wi h espec o a gi en causal model and ensu ing ha such ela ions a e main ained in
he gi en CF), and ai ness (i.e., he gi en CF explana ion is unbiased wi h espec o speci ic
p o ec ed ea u es, e.g., gende o ace). Indeed, heo e ical accoun s o con as i e and CF
explana ions a e ound o la gely di e om each o he in e ms o hei ela ion o he causal
aspec o explana ion. A as majo i y o heo e ical academic s udies appeal o he causal na u e
o con as i e and/o CF explana ion [36]. Howe e , modelling CFs ha possess hese p ope ies
alls ou side he scope o his hesis and is le o u u e wo k.
27
ILIA STEPIN
inds o be he mos ele an o he gi en ( ac ual o CF) class may no coincide wi h use
expec a ions o p e e ences, leading o dec eased u ili y o such an explana ion. I is he e o e
o pa amoun impo ance o enable he end use o inqui e al e na i e explana ions so ha he
use can p une he explana ion space un il she is ully sa is ied wi h he in o ma ion accumula ed
along he explana o y dialogue.
To ensu e ha he desi ed a o emen ioned ad an ages o use -sys em in e ac ion a e achie ed
ully, a gumen a ion heo y me hods can be used o model a communica ion channel be ween
he use and he explaine . In his hesis, we design explana o y use -sys em dialogue applying
he so-called “dialogue game” app oach o a gumen a ion [22]. This mechanism allows us o
(1) o malise and in eg a e eques ypes desc ibed below, (2) enable he end use o i e a i ely
explo e he explana ion space by a guing o e he explana ions o e ed p e iously, (3) pe son-
alise explana ions gi ing o he end use ull eedom o eques only necessa y and su icien
in o ma ion abou he da ase , p edic ion, o explana ion componen s.
The co esponding dialogue p o ocol es ablishes a ypology o use ’s eques s, explaine ’s
esponses, and ansi ions be ween he dialogue s a es. Thus, he se o he p oposed dialogue
eques s includes he ollowing ca ego ies:
•Fac ual and CF explana ion eques s. These include why and why-no ques ions o he
explaine o ac ual and CF classes, espec i ely.
•De ailisa ion eques s. These ackle he swi ch om pu ely linguis ic alues o speci ic
ea u es ha make pa o he gi en piece o explana ion o hei nume ical coun e pa s.
•Cla i ica ion eques s. These eques s a e mean o ques ion de ini ions o speci ic ea-
u es ha make pa o he gi en piece o explana ion.
•Al e na i e explana ion eques s. These eques s a e designed o enable he use o
explo e he explana ion space. No ably, hey a e made una ailable o ac ual explana ions
in he case o DTs, as al e na i e decision pa hs could be e oneous wi h espec o he
ac ual p edic ion and do no adequa ely explain he classi ie ’s easoning o he gi en
es ins ance.
The p oposed o mal model o explana o y dialogue has been implemen ed in o m o a
ask-o ien ed dialogue sys em1. The dialogue sys em’s main asks a e o (1) communica e o he
end use explana ions gene a ed au oma ically by an explaine and (2) handle ollow-up use
eques s conce ning explana ion- ela ed de ails. Adap ing a classic pipeline o ask-o ien ed
1The sou ce code is made publicly a ailable a h ps://gi lab.ci ius.usc.es/ilia.s epin/
c expgen, b anches “dialgame” and “dialgame_nlu”.
34
Chap e 4. Gene al discussion
Figu e 4.2: The dialogue sys em pipeline making use o he p oposed a gumen a i e explana o y model.
spoken dialogue sys ems [44] o ( ex ual) explana o y dialogue modelling, we designed an a -
gumen a i e con e sa ional agen ha handles use eques s sequen ially in acco dance wi h he
pipeline o dialogue sys em componen s ha comp ises he ollowing ou componen s:
•The na u al language unde s anding (NLU) module. I se es wo pu poses: (1) o
iden i y use ’s in en and (2) ecognise all en i ies ha he eques has (i any).
•The dialogue s a e acke (DST). I de ines he s a e ha he dialogue is cu en ly in
based on he use ’s in en ecognised by he NLU module.
•The dialogue policy manage (DPM). I selec s he mos app op ia e esponse among
hose a ailable a he gi en dialogue s a e passed by DST.
•The NLG module. I gene a es well- o med, g amma ical sys em’s esponses based on
he in o ma ion ecei ed om DPM.
Al oge he , DST and DPM a e said o cons i u e he dialogue manage . The a gumen a i e
dialogue p o ocol se es as he basis o he dialogue manage , as i acks he s a e o he dialogue
and de ines all possible ansi ions among dialogue s a es.
Fig. 4.2 illus a es he pipeline o he componen s o he implemen ed con e sa ional agen .
Le us conside a bee s yle classi ica ion p oblem o illus a e eques p ocessing. Gi en a p e-
ained bee s yle classi ie , he use passes he cha ac e is ics o a speci ic bee (e.g., colou ,
bi e ness, and s eng h) o he classi ica ion sys em and ob ains he classi ie ’s p edic ion (e.g.,
“This bee is Blanche”). Then, he use seeks an explana ion o he gi en p edic ion and sub-
mi s he co esponding eques o he dialogue sys em (e.g., “Why is his bee Blanche?”). A
35
ILIA STEPIN
i s , he NLU module analyses he use ’s eques o (1) iden i y wha ca ego y i belongs o
and (2) ecognise all en i ies ha he eques con ains. In ou example, he NLU module i s
a emp s o sol e he use in en classi ica ion p oblem whe e i assigns p obabili ies o each
ca ego y o use eques s, e.g., (“why-explain”, 0.95). The same p ocedu e is applied o en i y
ecogni ion. The in en and en i y ankings a e passed on o DST. Then, DST selec s he mos
p obable in en (in his case, “why-explain”) and swi ches he dialogue o he co esponding s a e
(in ou example, ha o a ac ual explana ion eques ). Gi en he dialogue s a e, DPM seeks he
mos adequa e esponse among he possible op ions. In acco dance wi h he dialogue p o ocol,
he sys em is allowed o espond o a ac ual explana ion eques (i.e., “why-explain”) by ei he
o e ing a ac ual explana ion o he end use (i.e., “explain- ”) i i is able o gene a e i o
e using o o e i , o he wise (i.e., “no-explain- ”). In ou example, he sys em inds ha he
decision pa h esponsible o he gi en p edic ion can be summa ised in e ms o wo ea u es
(i.e., colou and bi e ness) wi h he co esponding alues (i.e., black and high, espec i ely).
The explana o y ea u e- alue pai s a e hen passed on o he NLG module, which gene a es a
well- o med, g amma ical u e ance. Once he NLG module e u ns he sys em’s u e ance (in
his example, “The bee is Blanche because i s colou is black and bi e ness is high.”), i is
p esen ed o he end use .
Simila ly o he ac ual explana ion eques om he example abo e, he explana o y di-
alogue sys em ollows he a o emen ioned pipeline o handle all o he ypes o use eques s
de ined in he a gumen a i e dialogue p o ocol (i.e., CF explana ion, de ailisa ion, cla i ica ion,
and al e na i e explana ion eques s). In each case, DPM makes calls o ex e nal modules i
necessa y (e.g., he explaine when gene a ing explana ions o he knowledge base con aining
domain knowledge when p ocessing cla i ica ion eques s).
The p oposed explana o y dialogue p o ocol p o ides a anspa en means o explana ion
communica ion in in o ma ion-seeking se ings. Being anspa en , he p oposed model can be
aligned wi h egula o y equi emen s o au oma ic explana ion gene a ion. Fu he mo e, i is
shown o ake in o accoun use p e e ences, as i a ou s di e si y o he ou pu explana ions
and allows he use o u he ques ion all a iable componen s o such explana ions. The p o-
posed a gumen a i e dialogue model can be po en ially used as a ool o assessing e ec i eness
o CF explana ions gene a ed by o he ule-based CF explana ion gene a ion algo i hms. To a-
cili a e adap a ion o di e en ule-based CF explaine s, he p oposed dialogue model is u he
concep ualised using he o malism o dialogue g amma s.
No ably, he p oposed dialogue p o ocol also allows us o quan i a i ely es ima e he ne-
cessi y in di e si y o quali a i e CF explana ions. Recall ha he p oposed dialogue p o ocol
ep esen s he explana ion space o each (possibly, CF) class as a lis o explana ions anked by
ele ance, as measu ed by he explaine . Gi en a co pus o explana o y dialogues collec ed, we
can compa e he explaine -measu ed ele ance o he demand in he gi en explana ions based on
36
Chap e 4. Gene al discussion
he da a collec ed om ac ual use s. S a is ics o al e na i e explana ion eques s o each class
can be use ul o his pu pose. On he one hand, a la ge numbe o al e na i e explana ions asked
o may be a signal o li le u ili y o he ini ially o e ed explana ions. On he o he hand, he
empi ical da a om end use s may no adequa ely e lec hei sa is ac ion wi h he explana ion
space in i s en i e y, as end use s a e a all imes exposed o only a pa o he explana ion space
unless hey sequen ially eques all he explana ions ha he sys em can o e o hem. In his
ega d, a me ic o simila i y be ween he se o explana ions (a leas , po en ially) gene a ed and
ha o ac ually eques ed may be use ul o assessing au oma ically he use sa is ac ion wi h
he explana ion space, in gene al.
Finally, he concep o di e si y o CF explana ions ega ded in e ms o a se o single
al e na i e explana ions allows us o ques ion he na u e o he me e de ini ion o a CF explana-
ion. Indeed, au oma ed CFs gene a ed ollowing he con en ional de ini ion sea ch minimally
di e en ea u e- alue pai s ha ensu e a di e en classi ica ion. Howe e , i he e is a s ong
endency o dis ega d minimally di e en CF da a poin s ha o m he basis o a CF explana ion
and i is shown o be consis en o di e en explaine s and audiences, i may be imely o econ-
side he de ini ion o a CF explana ion o empowe i wi h he p ope y o human-cen ici y ha
goes beyond exis ing au oma ically compu able me ics. Whe eas he piece o wo k p esen ed
in his hesis only makes he i s s ep in his di ec ion, i can se e an inspi ing sou ce o ideas
o elabo a ion on he use -cen ic p ospec s o au oma ed CF explana ions.
37
5 Con ibu ions
The wo k on he p esen hesis esul ed in (1) se e al pieces o so wa e de eloped o each
he hesis objec i es and (2) a ious publica ions ha eme ged as a esul o he s udies ca ied
ou du ing he doc o al p ojec . Sec ion 5.1 de ails he so wa e p oduced as pa o he hesis.
Sec ion 5.2 lis s all he publica ions ha his hesis bases upon.
5.1 SOFTWARE
The ollowing pieces o so wa e ha e been de eloped in o de o achie e he hesis objec i es:
C1: FCFExpGen1– a amewo k o ac ual and CF explana ion gene a ion o in e p e able
ule-based classi ica ion sys ems. The p oposed amewo k includes he ollowing h ee
algo i hms:
XOR: a model-speci ic algo i hm ha selec s candida e CF ules and anks hem by ele-
ance o he es ins ance using he eXclusi e-OR (XOR) unc ion. The ule claimed
o be he mos ele an ep esen s a se o CF da a poin s ha a e minimally di e en
om he es ins ance in e ms o hei ea u es. This CF se o ms he basis o he
ou pu CF explana ion;
EUC: a model-speci ic a ian o he XOR algo i hm ha u ilises Euclidean dis ance as
a me ic o ele ance o he candida e CF ule o he es ins ance (i.e., i measu es
p oximi y o he ec o ep esen a ion o he CF ule o he es ins ance in he se-
lec ed n-dimensional space);
GEN: a model-agnos ic gene ic algo i hm ha sol es an op imisa ion p oblem looking o
he closes single CF da a poin w. . . he es ins ance unde conside a ion.
C2: DialGame2– an a gumen a i e con e sa ional agen o communica ion o au oma ically
gene a ed ex ual ule-based ac ual and CF explana ions. No ewo hy, he dialogue man-
1h ps://gi lab.ci ius.usc.es/ilia.s epin/ c expgen (b anch “xo _euc_gen”)
2h ps://gi lab.ci ius.usc.es/ilia.s epin/ c expgen (b anches “dialgame” and “dial-
game_nlu”)
ILIA STEPIN
age (i.e., an in eg a i e pa o he con e sa ional agen ) implemen s a dialogue p o ocol
basing on he a gumen a ion heo y-based echnique e e ed o as “dialogue game” [30].
C3: Su eyGene a o 3– a web- ool o ca ying ou human e alua ion expe imen s designed
o assessing a ious explana ion aspec s as well as he quali y o explana ion communi-
ca ion. Examples o he expe imen s ca ied ou include:
–Su ey GM: he explana ion e alua ion su ey ha enables he end-use o a e a
se ies o dis inc explana ions o he same es ins ance in e ms o in o ma i eness,
us wo hiness, accu acy, ele ance, and eadabili y;
–Su ey TS: a simpli ied e sion o Su ey GM which welcomes he end-use o a e
a single explana ion o he gi en es ins ance in e ms o us wo hiness and sa -
is ac ion.
I is wo h no ing ha all he sou ce code, he da a used in he human e alua ion expe imen s
and he co esponding expe imen al esul s a e made publicly a ailable and can be eached a a
public Gi lab eposi o y.
5.2 PUBLICATIONS
The wo k on he p esen doc o al hesis has esul ed in h ee jou nal pape s, h ee pape s p e-
sen ed a in e na ional con e ences and included in con e ence p oceedings (bo h main and o he
acks), and one book chap e . Namely, he ollowing jou nal publica ions co e he algo i hms
de eloped and e alua ed wi hin he doc o al p ojec [36, 37, 38]:
• Ilia S epin, Jose M. Alonso, Alejand o Ca ala, Ma ín Pe ei a-Fa iña. “A su ey o con-
as i e and coun e ac ual explana ion gene a ion me hods o explainable a i icial in el-
ligence”. IEEE Access, ol. 9, pp. 11974–12001, 2021. DOI: 10.1109/ACCESS.2021.3051315;
• Ilia S epin, Jose M. Alonso-Mo al, Alejand o Ca ala, Ma ín Pe ei a-Fa iña. “An empi -
ical s udy on how humans app ecia e au oma ed coun e ac ual explana ions which em-
b ace imp ecise in o ma ion”. In o ma ion Sciences, ol. 618, pp. 379–399, 2022. DOI:
10.1016/j.ins.2022.10.098;
• Ilia S epin, Ka a zyna Budzynska, Alejand o Ca ala, Ma ín Pe ei a-Fa iña, Jose M. Alonso-
Mo al. “In o ma ion-seeking dialogue o explainable a i icial in elligence: Modelling
and analy ics”. A gumen and Compu a ion, in p ess. DOI: 10.3233/AAC-220011.
3h ps://gi lab.ci ius.usc.es/jose.alonso/su eygene a o
40
Chap e 5. Con ibu ions
Fu he , he wo k in p og ess has been p esen ed a se e al in e na ional con e ences (in-
cluding main acks, wo kshops, and doc o al conso ia), which esul ed in he ollowing publi-
ca ions [34, 35, 39]:
• Ilia S epin, Alejand o Ca ala, Jose M. Alonso, Ma ín Pe ei a-Fa iña. “Pa ing he way
owa ds coun e ac ual gene a ion in a gumen a i e con e sa ional agen s”. In P oceed-
ings o he 1s Wo kshop on In e ac i e Na u al Language Technology o Explainable
A i icial In elligence (NL4XAI) colla ed wi h he Con e ence on In e na ional Na u al
Language Gene a ion (INLG), pp. 20-25, Tokyo (Japan), 2019. DOI: 10.18653/ 1/W19-
8405;
• Ilia S epin, Jose M. Alonso, Alejand o Ca ala, M. Pe ei a-Fa iña. “Gene a ion and e alu-
a ion o ac ual and coun e ac ual explana ions o decision ees and uzzy ule-based
classi ie s”. In P oceedings o he IEEE In e na ional Con e ence on Fuzzy Sys ems
(FUZZ-IEEE), Glasgow (UK), 2020. DOI: 10.1109/FUZZ48607.2020.9177629;
• Ilia S epin. “A gumen a ion-based in e ac i e ac ual and coun e ac ual explana ion gen-
e a ion”. In P oceedings o he 1s Doc o al Conso ium a he Eu opean Con e ence
on A i icial In elligence (DC-ECAI 2020), pp. 61-62, San iago de Compos ela (Spain),
2020.
No ably, u he expe imen s conce ning speci ic echnicali ies o he XOR algo i hm o au-
oma ed ac ual and CF explana ion gene a ion can be ound in he ollowing book chap e [40]:
• Ilia S epin, Alejand o Ca ala, Ma ín Pe ei a-Fa iña, Jose M. Alonso. “Fac ual and coun-
e ac ual explana ion o uzzy in o ma ion g anules”. In: Ped ycz, W., Chen, SM. (eds)
In e p e able A i icial In elligence: A Pe spec i e o G anula Compu ing. S udies in
Compu a ional In elligence, ol 937. Sp inge , Cham. DOI: 10.1007/978-3-030-64949-
4_6
In addi ion, a p oo o concep o he a gumen a i e amewo k o ac ual and CF expla-
na ion communica ion was p esen ed o ally a he Eu opean Con e ence on A gumen a ion in
Rome (I aly) on Sep embe 30, 2022. The ull pape is cu en ly unde e iew, pending o be
inally published by College Publica ions in hei “S udies in Logic and A gumen a ion” book
se ies4in 2023. Besides, one o he model-speci ic in e p e able uzzy ule-based explana ion
gene a ion algo i hms (i.e., EUC) is p esen ed in an imme si e a icle en i led “How o build sel -
explaining uzzy sys ems: F om in e p e abili y o explainabili y” and submi ed o he special
issue “A i icial In elligence eXplained” (AI-X) o he IEEE Compu a ional In elligence Mag-
azine. A he momen o w i ing, he manusc ip is unde going a second e iew ound.
4h ps://www.collegepublica ions.co.uk/logic/sla/
41
ILIA STEPIN
Fig. 5.1 isualises he main con ibu ions (i.e., so wa e and jou nal pape s) lis ed in Sec-
ions 5.1-5.2, espec i ely.
Figu e 5.1: Main esea ch ques ions and answe s ( ega ding echniques, so wa e and publica ions).
42
6 S a e-o - he-a me hods o con as i e and
coun e ac ual explana ion gene a ion
As discussed in Chap e 4, he p oblem o con as i e and CF explana ion gene a ion is among
he mos ending XAI opics [21]. The ecen ise o a en ion o con as i e and CF explana ion
encou ages a discussion abou he na u e o hese concep s and hei (mis-)use in XAI. Whe eas
XAI is a newly eme ged esea ch ield, he no ion o explana ion, in gene al, and i s con as i e
and CF sub- ypes, in pa icula , ha e long been discussed in social sciences. I is he e o e
impo an o know how heo e ical concep s ela ed o con as i e and CF explana ion can guide
XAI esea che s in de eloping e ec i e explana ion gene a ion me hods.
In his chap e , we (1) inspec heo e ical ounda ions o he concep s o con as i e and CF
explana ion, (2) explo e echnical aspec s o he s a e-o - he-a compu a ional amewo ks o
gene a ion o con as i e and/o CF explana ions, and (3) discuss how heo e ically g ounded he
compu a ional amewo ks a e. We show ha a majo i y o s a e-o - he-a con as i e and CF
explana ion gene a ion amewo ks only loosely (i a all) ely on heo e ical models o explana-
ion om social sciences. Ins ead, hey mainly ep esen da a-d i en solu ions o op imisa ion
p oblems o en imes neglec ing a wide body o knowledge abou he na u e o explana ion ac-
cumula ed o e he cen u ies. As a consequence, i some imes leads o e minological con usion
and makes us s i e o s anda disa ion o he explana ion- ela ed no ions used ac oss sub- ields
o XAI.
The esul s om his chap e a e published in he ollowing pape [36]:
Ilia S epina, Jose M. Alonsoa, Alejand o Ca alaa, Ma ín Pe ei a-Fa iñab. “A su ey o con-
as i e and coun e ac ual explana ion gene a ion me hods o explainable a i icial in elligence”.
In IEEE Access (Open Access), ol. 9, pp. 11974-12001, ISSN: 2169-3536, 2021. DOI:
10.1109/ACCESS.2021.3051315
aCen o Singula de In es igación en Tecnoloxías In elixen es (CiTIUS), Uni e sidade de San i-
ago de Compos ela, Rúa de Jena o de la Fuen e Domínguez, s/n, 15782 San iago de Compos ela,
Spain
ILIA STEPIN
aCen o Singula de In es igación en Tecnoloxías In elixen es (CiTIUS), Uni e sidade de San i-
ago de Compos ela, Rúa de Jena o de la Fuen e Domínguez, s/n, 15782 San iago de Compos ela,
A Co uña, Spain
bLabo a o y o The New E hos, Wa saw Uni e si y o Technology, plac Poli echniki 1, 00-661,
Wa saw, Poland
cDepa amen o de Elec ónica e Compu ación, Uni e sidade de San iago de Compos ela, Rúa
Lope Gómez de Ma zoa, s/n, 15782 San iago de Compos ela, A Co uña, Spain
dDepa amen o de Filoso ía e An opoloxía, Uni e sidade de San iago de Compos ela, Plaza
de Maza elos s/n, 15705 San iago de Compos ela, A Co uña, Spain
Scien i ic p oduc ion indica o s:
A he momen o w i ing, he scien i ic p oduc ion indica o s o he yea o publica ion
(i.e., 2023) a e una ailable o A gumen and Compu a ion, he jou nal whe e Chap e 8 was
published. In he yea immedia ely p eceding ha o publica ion (i.e., 2022), he jou nal had
a Ci eSco e index o 3.6 (calcula ed by Scopus on 05 May, 2023) and an impac ac o o 1.4
(2022 Jou nal Ci a ion Repo s). In addi ion, i had he ollowing posi ions in he ca ego ies
lis ed below:
• Scopus: Q1 ( ank #95/1078) in Linguis ics and Language ( he 91s pe cen ile), Q2 ( ank
#57/172) in Compu a ional Ma hema ics ( he 67 h pe cen ile), Q2 ( ank #366/792) in
Compu e Science Applica ions ( he 53 d pe cen ile), Q3 ( ank #159/301) in A i icial
In elligence ( he 47 h pe cen ile);
• JCR: Q4 ( ank #155/192) in Compu e Science and A i icial In elligence.
As o 29 June 2023, he publica ion has no been ci ed ye .
Pe sonal au ho ship s a emen :
In acco dance wi h he Con ibu o Roles Taxonomy (CRediT), he pe sonal au ho ship con-
ibu ion comp ises he ollowing oles: me hodology, so wa e, alida ion, in es iga ion, da a
cu a ion, w i ing - o iginal d a , isualiza ion.
Publishing igh s:
The jou nal pape whe e he esul s o his Chap e a e published is licensed unde a C e-
a i e Commons A ibu ion-NonComme cial 4.0 License. The indi idual o en i y exe cising
he licensed igh s (he eina e , “ he use ”) is ee o copy and edis ibu e he ma e ial in any
medium o o ma unde he ollowing e ms1:
1Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by-nc/4.0/
50
Chap e 8. A gumen a i e explana ion communica ion o ule-based classi ica ion sys ems
•A ibu ion: The use mus gi e app op ia e c edi , p o ide a link o he license, and
indica e i changes we e made. The use may do so in any easonable manne , bu no in
any way ha sugges s he licenso endo ses he use o he use ’s use.
•NonComme cial: The use may no use he ma e ial o comme cial pu poses.
51
9 Conclusion
In his hesis, we add essed he p oblem o explana ion gene a ion and communica ion o in-
e p e able ule-based sys ems. Mo e speci ically, we ocused on he ask o gene a ion o in-
e ac i e CF explana ions, which mee heo e ically g ounded equi emen s o quali y explana-
ions o XAI. To add ess his challenge, we i s pe o med a li e a u e e iew o heo e ical
ounda ions o he amily o con as i e and CF explana ions and he s a e-o - he-a me hods
o hei au oma ic gene a ion, which esul ed in a wo-le el axonomy o con as i e and CF
explana ions. Taking in o conside a ion he insigh s om he e iew, we hen designed, imple-
men ed, and alida ed a compu a ional amewo k o gene a ing ac ual and CF explana ions
associa ed o in e p e able ule-based classi ica ion sys ems. I includes one model-agnos ic and
wo model-speci ic algo i hms, all o which o e human-comp ehensi e explana ions in na u-
al language. Fu he , we enhanced he amewo k wi h an a gumen a i e dialogue gene a ion
module, which allows o in e ac i e explana ions in ag eemen wi h end use ’s needs. All in
all, he gene a ed explana ions ha e been shown o be con as i e, selec ed, and social.
In wha ollows, we summa ise main lessons lea ned om he wo k ca ied ou du ing his
doc o al p ojec . Sec ion 9.1 encapsula es ou concluding ema ks o each piece o esea ch
epo ed in Chap e s 6-8. Sec ion 9.2 ou lines p ospec i e di ec ions o u u e wo k.
9.1 CONCLUDING REMARKS
The li e a u e e iew on con as i e and CF explana ions e ealed se e al gaps in he inspec ed
sub- ield o XAI. Fi s , he s a e-o - he-a compu a ional amewo ks o con as i e and CF ex-
plana ion gene a ion a e sca cely g ounded on explana ion heo ies om social sciences. This
is, in pa , due o he ac ha he exis ing heo ies and compu a ional amewo ks mainly ad-
d ess dis inc aspec s o explana ion gene a ion. Thus, heo ies o con as i e explana ion o en
discuss p oduc s o he explana o y p ocess in e ms o cause-and-e ec ela ionships whe eas
a la ge numbe o compu a ional con as i e and/o CF explana ion gene a ion me hods ocus
on non-causal (e.g., spacial) ela ions be ween he es poin whose p edic ion is o be explained
and po en ial CF da a poin s. Second, i u ns ou ha he e ms “con as i e” and “coun e -
ac ual” a e o en used in e changeably in he XAI communi y despi e ce ain me hodological
ILIA STEPIN
di e ences. This obse a ion calls o s anda isa ion o he e minology used in he ield. In
o de o uni y he e minology (whe e applicable), we sugges ha he e m “con ac ual” be
p omo ed o con as i e-CF explana ions. We belie e ha his e m adequa ely encompasses
he aspec o con as i eness in CF explana ions and ice e sa. Thi d, he s a e-o - he-a com-
pu a ional amewo ks ha e been obse ed o g ea ly lack human e alua ion suppo . In ac , a
as majo i y o con as i e and CF explana ion gene a ion me hods ha e only been e alua ed
using da a-d i en au oma ically compu ed me ics. Ne e heless, human e alua ion s udies a e
indispensable o shi ing owa ds human-cen ic AI despi e being expensi e and di icul o
design.
In o de o educe he gap be ween he au oma ic da a-d i en and human e alua ion-based
me ics, we p oposed he me ic o pe cei ed explana ion complexi y (PEC), i.e. a measu e o
how complex he gi en piece o ex ual explana ion seems o be o he end use o p ocess i .
Fo he a ge audience (in his case, use s who ha e a high deg ee o expe ise in XAI o e-
la ed ields), human e alua ion expe imen s ha e shown ha he compu ed PEC sco es co ela e
wi h in o ma i eness, ele ance, and eadabili y o au oma ed gene a ions. Thus, he p oposed
me ic can e ec i ely eplace human e alua ion expe imen s measu ing he a o emen ioned ex-
plana ion aspec s o domain expe s o highly quali ied specialis s. O e all, he quali y o he
explana ions gene a ed by all he p oposed algo i hms was posi i ely e alua ed in he human
e alua ion s udies ha we ca ied ou in his hesis. In e ms o all he assessed explana ion as-
pec s (i.e. in o ma i eness, us wo hiness, accu acy, ele ance, eadabili y, and sa is ac ion),
he esul ing sco es we e abo e a e age o all he p oposed CF explana ion gene a ion me h-
ods. Howe e , none o hese me hods has been ound o consis en ly ou pe o m he o he s in
all he explana ion aspec s. In his ega d, we conclude ha he me hods modelling imp ecise
knowledge (e.g., XOR and EUC) and hose making use o p ecise ea u e alues (e.g., GEN)
a e bes used complemen a ily o each o he o sa is y he needs o a wide audience o use s.
Whe eas such complemen a y in o ma ion can be agg ega ed in a single piece o ex , he
social aspec o explana ion emains unadd essed in case o one-sho explana ions. To o e -
come his issue, we p oposed an a gumen a i e dialogue p o ocol o model in o ma ion-seeking
explana o y dialogues and de eloped a co esponding con e sa ional agen . The human e al-
ua ion esul s o he dialogue p o ocol alida ion p o e he necessi y o all he p oposed ypes
o eques s and esponses o e ec i e explana ion communica ion. Fu he , a la ge numbe
o eques s o al e na i e CF explana ions es i y ha he mos ele an CF explana ions om
he algo i hmic poin o iew may o en imes no seem op imal om he use ’s poin o iew.
Whe eas u he compa a i e s udies a e necessa y o analyse explana o y powe o dis inc ex-
plana ion gene a ion algo i hms om he cogni i e poin o iew, i can be concluded a his s age
ha he bes - anked CFs (i.e., mos ele an o minimally di e en CFs om he es ins ance)
may ha e o be combined wi h one o mo e al e na i es gi en a se o mul iple candida e CFs.
54
Chap e 9. Conclusion
S i ing o enabling he end use o play a decisi e ole in he p ocess o explana ion commu-
nica ion, we belie e ha i is indispensable o u he emphasise he social aspec o au oma ed
explana ions, e.g., by designing addi ional me ics o he quali y o explana ion communica ion
simila ly o he PEC sco e p oposed in his hesis.
In ligh o he s a emen s made abo e, we conclude ha he explana ion gene a ion ame-
wo k p oposed in his hesis o e s ac ual and CF explana ions ha u n ou o be appealing o
end use s. Thus, ex ual explana ions gene a ed in bo h modali ies ( hose modelling imp ecise
knowledge and hose ou pu ing speci ic nume ical alues o in e als he eo ) ecei ed highe -
han-a e age es ima es om he a ge audiences in all he expe imen s ca ied ou . In addi ion,
end use s app ecia ed hei in e ac ion wi h he a gumen a i e con e sa ional agen which was
ca e ully designed o communica e au oma ed ac ual and CF explana ions.
9.2 FUTURE WORK
The esea ch esul s p esen ed in his hesis indica e se e al di ec ions o u u e wo k. F om
he heo e ical poin o iew, he p oposed amewo k can be ex ended o in oduce causal ela-
ions be ween he p edic ed da a and ela ed ea u es. In ac , his could u he b idge he gap
be ween heo e ical and compu a ional pa adigms o con as i e and CF explana ion gene a ion
and he e o e appea s highly desi able in ligh o he esul s ob ained om he pe o med li e -
a u e e iew. F om he algo i hmic poin o iew, he p oposed explana ion gene a ion ame-
wo k should be u he ex ended wi h a su oga ion app oach o handle o he ypes o classi ie s
(including hose non-in e p e able and non- ule-based). Ne e heless, in he absence o access
o he in e nals o he classi ie and/o he ea u e space, i may be essen ial o ans o m he p o-
posed model-speci ic explana ion gene a ion algo i hms in o hei model-agnos ic equi alen s.
Fu he , di e en se ings may equi e changes in he dialogue p o ocol ha models he com-
munica ion p ocess be ween he explaine and he use o he sake o deepe cus omisa ion. In
addi ion, u he ex ension is equi ed o gua an ee p ope ies o CF explana ions no add essed
in his hesis. Fo example, he p esen ed algo i hms do no allow o assess s aigh o wa dly
how ac ionable he gene a ed CFs a e. Hence, enhancing ou pu CF explana ions wi h o he
desi ed CF p ope ies is belie ed o u he inc ease e ec i eness o such explana ions.
Impo an ly, i seems impossible o achie e he s a e o human-cen ic AI wi hou o malis-
ing and modelling e hical ela ions on he basis o he da a being p ocessed. Thus, bias mi iga-
ion, ye ano he highly ele an line o esea ch in he XAI communi y, is ano he algo i hmic
challenge o add ess. Finally, i is o ou pa icula in e es o u he adap he designed human
e alua ion amewo k o u u e expe imen s on explana ion, us wo hiness, and sa is ac ion.
Al oge he , he p ospec i e ex ensions o he wo k p esen ed in his hesis a e belie ed o ha e
g ea po en ial o mo ing o wa d om XAI o T us wo hy AI.
55
E hical conside a ions
The expe imen s whose esul s a e epo ed in he p esen hesis we e designed o in ol e human
e alua ion. The e o e, a p io pe mission o ca y hem ou had been ob ained om he E hics
Commi ee o he Uni e si y o San iago de Compos ela (a copy o he co esponding ce i ica e
is a ached below).
All he in o ma ion collec ed om he human e alua ion s udy pa icipan s was in ag ee-
men wi h he Eu opean Union’s Gene al Da a P o ec ion Regula ion (GDPR). Fu he , human
e alua ion was based solely on non-pe sonal o anonymous da a. In addi ion, all he pa icipan s
ga e in o med consen con i ming he ollowing:
• he pa icipan eached he age o majo i y;
• pa icipa ion in he s udy was comple ely olun a y;
• pa icipa ion in he s udy could be e mina ed a any ime;
• pa icipan ’s anonymous esponses would be used o esea ch pu poses in acco dance
wi h he GDPR.
None o he expe imen al esul s could anyhow be used agains he s udy pa icipan s. Risks
o misuse o he collec ed esul s we e minimal. In ligh o an inc easing use o AI-based appli-
ca ions o au oma ed ex and/o image gene a ion, i is impo an o men ion ha no piece o
ex o pic u es om he p esen hesis o any o he published o accep ed a icles was gene a ed
using any gene a i e AI-based applica ions (e.g., Cha GPT).
Bibliog aphy
[1] S. Ali, T. Abuhmed, S. El-Sappagh, K. Muhammad, Jose M. Alonso-Mo al, R. Con-
alonie i, R. Guido i, J. Del Se , N. Díaz-Rod íguez, and F. He e a. Explainable a i-
icial in elligence (XAI): Wha we know and wha is le o a ain us wo hy a i icial
in elligence. In o ma ion Fusion, in p ess, 2023. DOI: 10.1016/j.in us.2023.101805.
[2] J. M. Alonso, C. Cas iello, L. Magdalena, and C. Menca . Explainable Fuzzy Sys ems -
Pa ing he Way om In e p e able Fuzzy Sys ems o Explainable AI Sys ems, olume 970.
Sp inge In e na ional Publishing, 2021. DOI: 10.1007/978-3-030-71098-9.
[3] F. Bex and D. Wal on. Combining explana ion and a gumen a ion in dialogue. A gumen
& Compu a ion, 7(1):55–68, 2016. DOI: 10.3233/AAC-160001.
[4] R. M. J. By ne. Coun e ac ual hough . Annual Re iew o Psychology, 67:135–157, 2016.
DOI: 10.1146/annu e -psych-122414-033249.
[5] R. M. J. By ne. Coun e ac uals in explainable a i icial in elligence (XAI): E idence om
human easoning. In P oceedings o he 28 h In e na ional Join Con e ence on A i icial
In elligence (IJCAI), pages 6276–6282, 2019. DOI: 10.24963/ijcai.2019/876.
[6] V. Dignum. Responsible a i icial in elligence: how o de elop and use AI in a esponsible
way. A i icial In elligence: Founda ions, Theo y, and Algo i hms. Sp inge , Cham, 2019.
DOI: 10.1007/978-3-030-30371-6.
[7] A. Ga and E. K ahme . Su ey o he s a e o he a in na u al language gene a ion: Co e
asks, applica ions and e alua ion. Jou nal o A i icial In elligence Resea ch, 61:65–170,
2018. DOI: 10.1613/jai .5477.
[8] R. Guido i. Coun e ac ual explana ions and how o ind hem: li e a u e e iew and
benchma king. Da a Mining and Knowledge Disco e y, pages 1–55, 2022. DOI:
10.1007/s10618-022-00831-6.
[9] R. Guido i, A. Mon eale, F. Gianno i, D. Ped eschi, S. Ruggie i, and F. Tu ini. Fac ual and
Coun e ac ual Explana ions o Black Box Decision Making. IEEE In elligen Sys ems,
34(6):14–23, 2019. DOI: 10.1109/MIS.2019.2957223.
Lis o Ac onyms
AI a i icial in elligence
AIA a i icial in elligence ac
CF coun e ac ual
DPM dialogue policy manage
DST dialogue s a e acke
DT decision ee
FRBCS uzzy ule-based classi ica ion sys em
FURIA uzzy uno de ed ule induc ion algo i hm
GDPR gene al da a p o ec ion egula ion
ML machine lea ning
NLU na u al language unde s anding
NLG na u al language gene a ion
XAI explainable a i icial in elligence
ILIA STEPIN
APPENDIX. PUBLISHED OR ACCEPTED ARTICLES
This appendix con ains h ee jou nal pape s ha o m he basis o he p esen hesis. All o hem
a e Open Access publica ions, making hem publicly a ailable o he in e es ed eade . In his
appendix, hey appea in he ollowing o de :
• Ilia S epin, Jose M. Alonso, Alejand o Ca ala, Ma ín Pe ei a-Fa iña. “A su ey o con-
as i e and coun e ac ual explana ion gene a ion me hods o explainable a i icial in el-
ligence”. In IEEE Access, ol. 9, pp. 11974-12001, ISSN: 2169-3536. IEEE Inc. (Open
Access), 2021. DOI: 10.1109/ACCESS.2021.3051315
• Ilia S epin, Jose M. Alonso-Mo al, Alejand o Ca ala, Ma ín Pe ei a-Fa iña. “An empi -
ical s udy on how humans app ecia e au oma ed coun e ac ual explana ions which em-
b ace imp ecise in o ma ion”. In In o ma ion Sciences, ol. 618, pp. 379-399. ISSN:
0020-0255. Else ie (Open Access), 2022. DOI: 10.1016/j.ins.2022.10.098
• Ilia S epin, Ka a zyna Budzynska, Alejand o Ca ala, Ma ín Pe ei a-Fa iña, Jose M. Alonso-
Mo al. “In o ma ion-seeking dialogue o explainable a i icial in elligence: Modelling
and analy ics”. In A gumen and Compu a ion, ISSN p in : 1946-2166; ISSN online:
1946-2174, in p ess. IOS P ess (Open Access). DOI: 10.3233/AAC-220011
68
Recei ed Decembe 16, 2020, accep ed Janua y 3, 2021, da e o publica ion Janua y 13, 2021, da e o cu en e sion Janua y 22, 2021.
Digi al Objec Iden i ie 10.1109/ACCESS.2021.3051315
A Su ey o Con as i e and Coun e ac ual
Explana ion Gene a ion Me hods o Explainable
A i icial In elligence
ILIA STEPIN 1, JOSE M. ALONSO 1, (Membe , IEEE), ALEJANDRO CATALA 1,
AND MARTÍN PEREIRA-FARIÑA 2
1Cen o Singula de In es igación en Tecnoloxías In elixen es (CiTIUS), Uni e sidade de San iago de Compos ela, 15782 San iago de Compos ela, Spain
2Depa amen o de Filoso ía e An opoloxía, Uni e sidade de San iago de Compos ela, 15705 San iago de Compos ela, Spain
Co esponding au ho : Ilia S epin ([email p o ec ed])
This wo k was suppo ed in pa by he Spanish Minis y o Science, Inno a ion and Uni e si ies unde G an RTI2018-099646-B-I00 and
G an RED2018-102641-T, in pa by he Galician Minis y o Educa ion, Uni e si y and P o essional T aining unde G an ED431F
2018/02, G an ED431C 2018/29, G an ED431G/08, and G an ED431G2019/04; and in pa by he Eu opean Regional De elopmen
Fund (ERDF/FEDER P og am).
ABSTRACT A numbe o algo i hms in he ield o a i icial in elligence o e poo ly in e p e able decisions.
To disclose he easoning behind such algo i hms, hei ou pu can be explained by means o so-called
e idence-based (o ac ual) explana ions. Al e na i ely, con as i e and coun e ac ual explana ions jus i y
why he ou pu o he algo i hms is no any di e en and how i could be changed, espec i ely. I is o c ucial
impo ance o b idge he gap be ween heo e ical app oaches o con as i e and coun e ac ual explana ion
and he co esponding compu a ional amewo ks. In his wo k we conduc a sys ema ic li e a u e e iew
which p o ides eade s wi h a ho ough and ep oducible analysis o he in e disciplina y esea ch ield unde
s udy. We i s examine heo e ical ounda ions o con as i e and coun e ac ual accoun s o explana ion.
Then, we epo he s a e-o - he-a compu a ional amewo ks o con as i e and coun e ac ual explana ion
gene a ion. In addi ion, we analyze how g ounded such amewo ks a e on he insigh s om he inspec ed
heo e ical app oaches. As a esul , we highligh a a ie y o p ope ies o he app oaches unde s udy and
e eal a numbe o sho comings he eo . Mo eo e , we de ine a axonomy ega ding bo h heo e ical and
p ac ical app oaches o con as i e and coun e ac ual explana ion.
INDEX TERMS Compu a ional in elligence, con as i e explana ions, coun e ac uals, explainable a i icial
in elligence, sys ema ic li e a u e e iew.
I. INTRODUCTION
In he las ew decades, he ield o A i icial In elligence
(AI) has wi nessed majo changes. As a ailable compu a-
ional esou ces ha e g own signi ican ly, AI algo i hms a e
a ac ing a signi ican amoun o a en ion in indus y and
esea ch [1]. While a g ea numbe o such algo i hms p esen
s ikingly accu a e decisions, hei decision-making appa a-
us is equen ly le unclea o use s o such applica ions.
In pa icula , a numbe o Machine Lea ning (ML)-based
algo i hms a e o en pe cei ed as ‘‘black-box’’ algo i hms
because hey a e o e loaded wi h millions o ha dly in e -
p e able pa ame e s o be op imized a he aining s age. This
ac makes he algo i hm’s ou pu ha d o explain. A lack o
The associa e edi o coo dina ing he e iew o his manusc ip and
app o ing i o publica ion was F ancesco Piccialli.
he abili y o explain such au oma ic decisions unde mines
use s’ us and hence dec eases usabili y o such sys ems [2].
Fu he mo e, i p e en s use s om a esponsible exploi a-
ion o hei decisions [3]. In addi ion, many o he exis ing
eXplainable AI (XAI1) me hods p o ide summa ies o au o-
ma ically made p edic ions a he han ue explana ions [4].
As a esul , he need o mo i a e au oma ic decisions wi h a
clea explana ion o why he algo i hm ou pu s a pa icula
decision has made he XAI esea ch ield g ow quickly [5].
Since he numbe o high-s akes AI applica ions ound
in daily li e inc eases, he equi emen s o hei explana-
1XAI s ands o eXplainable A i icial In elligence. This ac onym was
made popula by he USA De ense Ad anced Resea ch P ojec s Agency
when launching o he esea ch communi y he challenge o designing
sel -explana o y AI sys ems (h ps://www.da pa.mil/p og am/
explainable-a i icial-in elligence).
11974
This wo k is licensed unde a C ea i e Commons A ibu ion-NonComme cial-NoDe i a i es 4.0 License.
Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/ VOLUME 9, 2021
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
o y capaci y inc ease acco dingly. This also p o okes he
in oduc ion o egula ions and laws conce ned wi h expla-
na ion equi emen s o AI-based applica ions. Fo ins ance,
he need o explaining easoning mechanisms behind such
applica ions is now legally egula ed in he Eu opean Union
by means o he Gene al Da a P o ec ion Regula ion.2
Acco ding o hese legal p o isions, he da a subjec mus
be p o ided wi h ‘‘meaning ul in o ma ion abou he logic
in ol ed’’ in he au oma ic decision making p ocess, which is
commonly e e ed o as he ‘‘ igh o explana ion’’ [6]. Thus,
an AI applica ion is expec ed no only o p o ide accu a e
decisions bu also o jus i y hem in a comp ehensi e manne
o end-use s.
The goal o app oaching human-cen ic AI has led owa ds
a deepe esea ch on he na u e o explana ion. Howe e ,
no ag eemen abou a de ini ion o explana ion has been
eached despi e he ac ha explana ion has called a signi -
ican amoun o a en ion in, e.g., philosophy o science [7],
[8]. In i s mos gene al o m, explana ion is no mally ea ed
as ‘‘an answe o he ques ion o why some hing is he case’’
[9]. In he con ex o AI, i o en bases on judgmen s abou
why a ce ain ou come is p edic ed by an AI algo i hm and
hypo heses abou causes wi h espec o gi en e ec s [10].
The need o gene a ing mo e human-like explana ions has
a ac ed AI esea che s’ a en ion o pa icula p ope ies o
explana ion as well as i s sub- ypes [11]. Thus, i appea s
pa icula ly challenging o explain a gi en algo i hm’s ou pu
in e ms o easonable ye non-occu ing al e na i es gi en a
possibly in ini e se o such op ions. Fu he mo e, his can be
enhanced wi h he abili y o sugges ing ele an changes in
he inpu so ha he algo i hm ou pu s a di e en decision.
Gi en a ising in e es owa ds hese ypes o explana ion
( e e ed o as con as i e and coun e ac ual, espec i ely)
wi hin he XAI communi y, i is o c ucial impo ance o
e iew he exis ing heo e ical accoun s o con as i e and
coun e ac ual explana ion as well as s a e-o - he-a compu-
a ional amewo ks o au oma ic gene a ion he eo . Thus,
he aim o his s udy is o ul ill he nex h ee objec i es:
(1) o sc u inize heo e ical wo ks on he con as i e and
coun e ac ual accoun s o explana ion; (2) o summa ize
s a e-o - he-a me hods in he ield o au oma ic explana ion
gene a ion he eo ; and (3) o discuss a deg ee o syne gy
be ween he e ised heo ies and hei ela ed up- o-da e
implemen a ions.
The es o he manusc ip is o ganized as ollows.
Sec ion II in oduces he no ions o con as i e and coun-
e ac ual explana ion as well as hei main applica ion
a eas. Sec ion III p esen s he e minology used h ough-
ou he e iew, poses he esea ch ques ions, and desc ibes
he me hodology employed o add ess he gi en ques ions.
Sec ion IV p esen s he main indings collec ed wi hin he
p esen su ey and he eme ging axonomy he eo . Sec ion V
discusses peculia i ies o he exis ing heo e ical and compu-
2h ps://eu -lex.eu opa.eu/legal-con en /EN/TXT/?u i=CELEX:
02016R0679-20160504
a ional amewo ks o con as i e and coun e ac ual expla-
na ion. Finally, we conclude in Sec ion VI.
II. BACKGROUND
A. CONTRASTIVE EXPLANATION
Findings on explana ion accumula ed in humani ies and
social sciences show ha i is in insically con as i e [11].
The p ope y o con as i eness p esupposes ha an expla-
na ion answe s he gi en why-ques ion ega ding he cause
o he e en in ques ion (‘‘Why did Phappen?’’) in e ms o
hypo hesized non-occu ing al e na i es (‘‘Why did Phappen
a he han Q?’’) [12]. Thus, suppo e s o he p agma ic
app oach o explana ion a gue ha i is exac ly he abili y
o dis inguish he answe o an explana o y ques ion om
a se o con as i e hypo hesized al e na i es ha p o ides
he explainee wi h su icien ly comp ehensi e in o ma ion on
he easoning behind he ques ion [13]. This app oach is also
claimed o se a minimum c i e ion ha an explana ion mus
ul ill: i mus a o he p obabili y o he obse ed e en P
o all he hypo he ical al e na i es (Q1,Q2,...,Qn) [14].
Con as i e explana ion is among in luen ial opics in cog-
ni i e science [15]–[17]. Thus, con as i e explana ions a e
claimed o be inhe en o human cogni ion [16]. Indeed,
we a e used o ques ion hose decisions ha we once made,
especially i such decisions o coinciding ci cums ances
esul ed in agic e en s [18].
In addi ion, con as i e easoning o ms he basis o abduc-
i e in e ence [19], i.e., he p ocess o in e ing ce ain ac s
ha ende some obse a ion plausible [20]. In o he wo ds,
a gi en obse a ion can be explained on he basis o he mos
likely among a pool o compe ing hypo heses [21].
B. COUNTERFACTUAL EXPLANATION
Gi en he p ope y o con as i eness, i is possible o imagine
explana o y al e na i es o how hings would s and i a di e -
en decision had been made a some poin . They can se e o
explain po en ial consequences o such con as i e non- aken
al e na i e decisions. In his case, he mind is assumed o
cons uc and compa e men al ep esen a ions o an ac ually
happened e en and ha o some e en al e na i e o i [22].
Cogni i e scien is s e e o such men al ep esen a ions o
al e na i es o pas e en s as coun e ac uals (‘‘con a y- o-
ac ’’) [15]. The p ocess o ‘‘ hinking abou pas possibili-
ies and pas o p esen impossibili ies’’ is he e o e called
coun e ac ual hinking [23]. Al e na i ely, he combina ion
o imagining an al e na i e scena io in ela ion o he one ha
ac ually happened and he explo a ion o i s consequences
is e e ed o as coun e ac ual easoning [24]. In addi ion,
coun e ac ual easoning is claimed o be a key mechanism
o explaining adap i e beha io in a changing en i onmen
[25], [26].
Coun e ac uals desc ibe e en s o s a es o he wo ld ha
did no occu and implici ly o explici ly con adic ac ual
wo ld knowledge [27]. Fo mula ed in na u al language, coun-
e ac uals a e usually p esen ed in he o m o condi ional
VOLUME 9, 2021 11975
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
s a emen s. B oadly speaking, hey con ain: (1) an an eceden
desc ibing an ou come al e na i e o an ac ual e en ; (2)
a consequen desc ibing (a se o ) consequences, had he
an eceden been he case; and (3) a bina y coun e ac ual
dependency ela ion be ween hem. Thus, G ahne de ines
a coun e ac ual o be a condi ional s a emen whe e he
an eceden ‘‘can con adic he cu en s a e o a ai s, o ou
cu en knowledge he eo ’’ [28]. Howe e , despi e a gen-
e al ag eemen on s uc u al p ope ies o coun e ac uals,
exis ing in e p e a ions o coun e ac ual condi ionals s ill
compe e. As such, u he cons ain s imposed on hei s uc-
u e di e depending on he app oach adop ed. Acco ding o
Ginsbe g [29], a coun e ac ual is a condi ional s a emen o
he o m ‘‘I P, hen Q’’ whe e Pis ‘‘expec ed o be alse’’.
Aumann limi s a coun e ac ual o be a condi ional wi h a
alse an eceden only [30]. In con as , Spohn a gues ha bo h
he an eceden and he consequen o a coun e ac ual mus
be alse [31]. All in all, coun e ac ual condi ional s a emen s
a e claimed o enable people o p oduce u e ances ha a e
ac ually alse ye u h ul i espec i e o he in e p e a ion
adop ed [32].
A line o esea ch de o ed o modeling human coun e ac-
ual easoning has been ho oughly in es iga ed in compu e
science. Thus, coun e ac ual easoning in compu e science
is de ined as he p ocess o e alua ing condi ional claims
abou al e na i e possibili ies and hei consequences [33].
I is a gued o be alid a ising om an eceden s ha a e
ue in a hypo he ical model bu alse in eali y [34]. In his
se ing, he u h o a coun e ac ually in e ed s a emen is
esol ed by: (1) modeling a si ua ion whe e he smalles
possible change in ea u es o he ac ual wo ld (as se in he
an eceden ) leads o a di e en (possibly, desi ed) s a e o
hings ( he so-called ‘‘closes ’’ o ‘‘nea es ’’ possible wo ld);
and (2) es ima ing wha is ue in ha se ing [35].
Mo eo e , coun e ac uali y is among he mos undamen-
al concep s in heo ies o causa ion [36], [37]. Indeed, coun-
e ac uals a e a gued o ep esen a causal ela ion be ween
he e en happened in eali y and i s imagina y coun e pa .
A coun e ac ual de ini ion o a cause o an a bi a y e en
aces back o Hume [38]. Acco ding o him, a cause is
an objec (an eceden ) ha jus i ies he exis ence o ano he
objec (consequen ) which i is ollowed by: ‘‘I he i s
objec had no been, hen he second ne e had exis ed’’.
The e o e, once a causal connec ion be ween he an eceden
and he consequen is es ablished, a coun e ac ual condi-
ional can be gene alized o be a condi ional claim abou an
al e na e possibili y and i s consequences o he o m ‘‘I X
we e o occu , hen Ywould (o migh ) occu ’’ [33]. Simila ly,
Kmen applies a simila i y-based app oach be ween possible
wo lds o o mula e a gene al accoun o coun e ac uals [39]
d i en by a non-epis emic in e p e a ion o explana ion (i.e.,
ac o s ha se e as easons o some ac o ob ain a e
esponsible o ha ac ).
The condi ional s uc u e o coun e ac ual s a emen s ga e
ise o a p obabilis ic accoun o such s a emen s. Thus, Pea l
ex ended he de ini ion o he causal coun e ac ual o es i-
ma e he p obabili y o he u h o he consequen caused by
he an eceden (‘‘a p obabili y s a emen abou he u h o
y, had xbeen ue, when i is known ha yhad been alse
when xwas alse’’) [37]. This app oach o coun e ac uals
mo i a ed a numbe o expe imen s on he exis ence o he
ela ion be ween coun e ac uals and condi ional p obabili y.
In suppo o his assump ion, O e e al. [40] showed he
exis ence o connec ion be ween coun e ac uals and con-
di ional p obabili y, as hey expe imen ed wi h p obabili y
judgmen s abou coun e ac uals. Thus, hey p oposed ha
he subjec i e p obabili y o he coun e ac ual a he p esen
ime is he same as he condi ional p obabili y P(y|x) a some
ea lie ime. Twen y-six subjec s we e asked o es ima e he
p obabili y o u h o hi y- wo coun e ac ual condi ionals
wi h bo h a i ma i e and nega i e an eceden s and conse-
quen s. Thei indings poin o a s ong co ela ion be ween
he p obabili y o he coun e ac ual condi ional and causal
s eng h judgmen s. On a simila no e, Edging on ega ded
coun e ac ual judgmen s as unce ain condi ional s a emen s
and he e o e e alua ed hem by es ima ing hei condi ional
p obabili y gi en some endo sing e en [41].
C. DISTINCTION BETWEEN CONTRASTIVE AND
COUNTERFACTUAL EXPLANATION
I is impo an o no e ha some esea che s end o ei he
collapse o in en ionally dis inguish con as i e easoning
om coun e ac ual easoning despi e hei concep ual sim-
ila i y. Fo ins ance, Lomb ozo ea ed coun e ac ual and
con as i e explana ions as equi alen assuming hypo he-
sized e en s non-occu ed in eali y o be ‘‘coun e ac ual
cases’’ whe e a subse o hese cases o ms a con as i e
explana ion [10]. In con as , McGill and Klein dis inguished
con as i e easoning om i s coun e ac ual coun e pa
[42]. Acco ding o hem, con as i e easoning is conce ned
wi h si ua ions whe e di e en a ge si ua ions a e analyzed
(‘‘Wha made he di e ence be ween he employee who
ailed and he employees who did no ail?’’). On he o he
hand, coun e ac ual easoning is claimed o deal wi h cases
whe e he an eceden is al e ed o accoun o changes in
he ou come (‘‘Would he employee ha e ailed had she no
been a woman?’’). Al e na i ely, Fang e al. [43] e e ed
o con as i e easoning as a p ocedu e ope a ing on ‘‘bu -
s a emen s’’, as in ‘‘all ca s a e pollu ing, bu hyb id ca s a e
no pollu ing’’, which se es a p incipally di e en explana-
ion gene a ion ask in compa ison wi h he o he a o emen-
ioned app oaches.
D. CONTRASTIVE AND COUNTERFACTUAL EXPLANATION
IN THE CONTEXT OF XAI
The s ochas ic na u e o p edic ions made by a ious AI
algo i hms is claimed o be among he main obs acles in
eaching a ue explana ion [44]. Resea ch on au oma ic
con as i e and coun e ac ual explana ion gene a ion shows
a numbe o conside able obse a ions ha help o e come
his issue. Thus, empi ical s udies p o e ha inco po a ing
con as i eness imp o es he quali y o explana ions o e ed
11976 VOLUME 9, 2021
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
o he end-use [45]. Fu he mo e, con as i e explana ions
can be used o pe sonalize human-machine in e ac ion when
a use is engaged in an explana o y dialogue wi h an AI
applica ion. Thus, hey can be employed wi h he aim o
adjus ing he con en s o he explana ion o he algo i hm’s
ou pu in acco dance wi h he use ’s p e e ences [46]. Finally,
he abili y o explain a decision con as i ely is claimed o
lead o esponsible decision-making [47].
I is impo an o no e ha con as i e explana ions poin
o he di e ence be ween he ac ual and a hypo he ical deci-
sion. On he o he hand, coun e ac ual explana ions speci y
necessa y minimal changes in he inpu so ha a con as i e
ou pu is ob ained. Howe e , hese e ms a e some imes used
in e changeably in he con ex o XAI [48], [49].
Va ious amilies o echniques ha e been p oposed o
gene a e con as i e and coun e ac ual explana ions o AI
algo i hm ou pu . In he con ex o XAI, an explana ion o an
au oma ic decision o p edic ion, ea ed as an obse a ion,
can be ob ained abduc i ely by a emp ing he sea ch p ob-
lem o e he se o he known in o ma ion conce ning ha
obse a ion [50]. Al e na i ely, coun e ac ual explana ion is
widely add essed in he pa adigm o case-based easoning,
i.e., a amily o p oblem sol ing me hods based on appeals
o p eceden solu ions. In his se ing, gene a ing he mos
sui able coun e ac ual may be iewed as a sea ch p oblem
whe e he mos simila p eceden is looked o among hose
making pa o he case da abase [14]. Fu he mo e, Keane
e al. a gue ha applying case-based easoning echniques
o gene a ing coun e ac uals inc eases hei explana o y
compe ence [51].
Coun e ac ual explana ions a e no mally conside ed con-
as i e by na u e and he e o e p esen a sou ce o aluable
complemen a y in o ma ion o a gi en au oma ic p edic-
ion [52]. Fo ins ance, a coun e ac ual explana ion o an
ML-based algo i hm p edic ion may desc ibe ‘‘ he small-
es change o he ea u e alues ha changes he p edic-
ion o a p ede ined ou pu ’’ [53]. An impo an ad an age
o coun e ac ual explana ions o e hei non-coun e ac ual
analogs is ha hey a e de oid o any p e equisi es o he
da a o model. Indeed, coun e ac ual explana ions a e da a-
agnos ic as hey can be based on he ea u es o he neigh-
bou ing da a examples ex ac ed om he same aining se
and/o on he da a gene a ed syn he ically a ound he da a
ins ance in ques ion. In addi ion, coun e ac ual explana ions
a e, in p inciple, model-agnos ic, as hey a e sui able o
explain he ou pu o any black-box algo i hm in a pos -hoc
manne .
Whe eas coun e ac ual explana ion gene a ion is con-
ce ned wi h a numbe o echnical challenges, i also equi es
o ake in o accoun se e al e hical aspec s. Fo ins ance,
hei use is expec ed o be sa e ( e ealing model’s in e nals
h ough coun e ac uals may lead o model s ealing) [54], ai
(disc imina o y explana ions should be a oided) [55], ac ion-
able (sugges ed changes in he inpu should be easible) [56],
and accoun able (ensu ing esponsibili y o he explana ions
p o ided) [57].
III. METHODOLOGY
The p esen su ey has been unde aken as a sys ema ic li -
e a u e e iew ollowing he guidelines by Ki chenham and
Cha e s [58], Ki chenham e al. [59], and Wohlin [60]. The
backg ound no a ion necessa y o ollow he indings o he
e iew is speci ied in Sec ion III-A.
In sho , he s udy comp ises h ee phases as es ablished in
he esea ch me hod by Ki chenham and Cha e s [58]: (1)
planning he e iew p ocedu e; (2) conduc ing he e iew;
and (3) epo ing he esul s. Du ing he i s phase, h ee
esea ch ques ions (RQ1,RQ2, and RQ3) we e speci ied
(see Sec ion III-B). Subsequen ly, we de e mined a sea ch
s a egy o e ie e p ima y s udies, i.e., we collec ed all
he ele an publica ions in es iga ing he esea ch ques-
ions (see Sec ion III-C). Then, we de eloped inclusion and
exclusion c i e ia (see Sec ion III-D) in o de o selec he
s udies ele an o his a icle. When he same publica ion
was e ie ed om mul iple sou ces, all-bu -one ins ances
o he publica ion (duplica es) we e disca ded. In addi ion,
we iden i ied and added manually o he ele an publica-
ions ex ac ed om he bibliog aphy lis s o he p e iously
selec ed manusc ip s o ensu e a maximum co e age o he
ela ed subjec a eas. I is wo h no ing ha his addi ional
p ocedu e is in o mally known as snowballing [60]. Finally,
we ex ac ed and syn hesized he da a necessa y o add ess
he esea ch ques ions (see Sec ion III-E).
A. PRELIMINARY TERMINOLOGY
As has been shown in Sec ion II, con as i e and coun e ac-
ual explana ions p esuppose a di e se na u e ac oss a ious
applica ion domains. Hence, le us now de ine he gene al
e ms used hence o h in his manusc ip . As we a e p ima ily
conce ned wi h explainabili y o AI algo i hms, we de ine
explana ion in e ms o he obse ed ou pu o such an algo-
i hm. Thus, we ega d an explana ion as a non-emp y se o
pieces o in o ma ion jus i ying he gi en algo i hm’s ou pu
o an inpu da a ins ance. The explana ion o he gi en
ou pu on he basis o he ea u es o he inpu da a ins ance
is deemed as ac ual. An explana ion opposing he ac ual
ou come o one o possible o he ou comes is conside ed o be
con as i e (e.g., ‘‘The da a ins ance is o class A and no B
because ...’’). An explana ion con aining ins uc ions on how
he ou pu could ha e been changed cons i u es a coun e ac-
ual explana ion (e.g., ‘‘The da a ins ance would be o class B
i ...’’). Explana ions exhibi ing pa e ns o bo h con as i e
and coun e ac ual explana ion a e deemed o be con as i e-
coun e ac ual explana ions (e.g., ‘‘The da a ins ance is o
class A and no B because .... Howe e , i would be o class
B i ...’’).
We dis inguish be ween con as i e and coun e ac ual
explana ion h oughou he es o he manusc ip i and
only i only one o hese wo e ms is used in he gi en
p ima y s udy. In con as , we uni y he no ions o coun e ac-
ual and con as i e explana ion in oducing he e m ‘‘con-
ac ual explana ion’’ o ‘‘con ac ual’’ o iden i y po en ial
simila i ies and di e ences o bo h ypes o explana ion
VOLUME 9, 2021 11977
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
wi hin a b oade scope o li e a u e. This e m is used he e-
a e whe e e bo h e ms o con as i e and coun e ac ual
explana ion can be used in e changeably. The e ms ‘‘con-
as i e explana ion’’ and ‘‘coun e ac ual explana ion’’ a e
only used when hey a e ound in he co esponding s udy and
canno be used in e changeably in he gi en con ex . No ice
ha he e m ‘‘con ac ual explana ion’’ is no equi alen
o ‘‘con as i e-coun e ac ual explana ion’’ bu co e s bo h
independen ly used ypes o explana ion as well as hei
usion.
A heo e ical amewo k p o iding jus i ica ion and a ea-
soning mechanism o ob aining a con ac ual explana ion is
ega ded as a heo y o con ac ual explana ion. Al oge he ,
we use he e m con ac ual explana ion gene a ion o e e
o he p ocess o au oma ic composi ion o con ac ual expla-
na ions o a gi en ou pu o an AI algo i hm in he o m o
a complemen a y piece o in o ma ion associa ed o a ac ual
explana ion.
B. RESEARCH QUESTIONS
In o de o each he h ee objec i es o he s udy as o mu-
la ed in Sec ion I, he ollowing h ee esea ch ques ions we e
speci ied:
•RQ1: How a e con ac ual explana ions de ined in he
li e a u e?
•RQ2: Wha a e he s a e-o - he-a me hods o con ac-
ual explana ion gene a ion?
•RQ3: How g ounded a e he s a e-o - he-a con ac-
ual explana ion gene a ion me hods on he heo e ical
app oaches o con ac ual explana ion?
C. SEARCH STRATEGY
We selec ed he digi al lib a ies Scopus and Web o Science
(WoS) o e ie e ele an publica ions om. These lib a ies
do no only include esea ch publica ions in compu ing bu
also index s udies ac oss all scien i ic ields, which allows
o an objec i e analysis o he in e disciplina y li e a u e
ele an o he esea ch ques ions posed.
Subsequen ly, we pe o med six que ies o e he i le,
abs ac , and au ho keywo ds in he a o emen ioned lib a ies
(see he o e all s uc u e o he que y pipeline in Fig. 1).
I is wo h no ing ha he p oximi y ope a o NEAR is used
ollowing he WoS no a ion whe eas he equi alen p oximi y
ope a o Wis used o he same que ies in Scopus. The
ollowing sea ch s ings we e used o que ying he digi al
lib a ies:
q1=coun e ac ual* W/3 expla*
q2=con as i e* W/3 expla*
q3=q1OR q2
q4=q3AND (de in* OR heo * OR in e * OR implic*)
q5=q3AND (gene a * OR implemen * OR amewo k*
OR de elop* OR so wa e* OR model* OR a i icial in el-
ligence OR AI) AND SUBJAREA(Compu e Science OR
Ma hema ics OR Enginee ing)
q6=q4AND q5
FIGURE 1. A pipeline o he que ies execu ed. The que ies ound in he
dashed a ea a e conside ed p epa a o y o hose di ec ly add essing he
esea ch ques ions.
The sea ch was pe o med on Oc obe 2nd , 2020. The
sea ch web ools o he selec ed digi al lib a ies allow
esea che s o ep oduce he o iginal s udy. Fu he mo e, hei
use gua an ees pe o ming equi alen que ies ac oss bo h
lib a ies. In o de o cap u e all ele an publica ions, we only
used he co esponding wo d-s ems o allow o maximal
di e si y o he e ie ed pape s. Fo ins ance, he sea ch i em
‘‘expla*’’ was used o co e all publica ions con aining such
wo d- o ms as ‘‘explana ion’’, ‘‘explaining’’, ‘‘explana o y’’,
and so on and so o h.
Que ies q1and q2emb ace all he up- o-da e publica-
ions con aining men ions o coun e ac ual and con as i e
explana ion, espec i ely, ound ac oss all subjec a eas.
In addi ion, we used a window span o h ee wo ds (i.e.,
‘‘NEAR/3’’) o ensu e ha he a ibu es ‘‘coun e ac ual’’
and ‘‘con as i e’’ ela e o explana ion. The esul ing se s o
publica ions we e hen uni ied (q3).
Subsequen ly, he p ep ocessed collec ion o publica ions
was spli in o wo o e lapping subse s aiming o dis inguish
he publica ions co e ing heo e ical accoun s o con ac ual
explana ion wi h he aim o ex ac ing he ela ed de ini-
ions, heo ies (o hei in e ences o implica ions) (q4) and
exis ing compu a ional amewo ks o con ac ual expla-
na ion gene a ion (q5). The e ms ‘‘de ini ion’’, ‘‘ heo y’’,
‘‘in e ence’’, and ‘‘implica ion’’ as well as hei co espond-
ing wo d- o ms (q4) we e expec ed o app op ia ely limi
he pool o he uni ied se o publica ions wi h he aim o
e ie ing de ini ions as equi ed o add essing RQ1. Sim-
ila ly, we used he e ms ‘‘gene a ion’’, ‘‘implemen a ion’’,
‘‘ amewo k’’, ‘‘de elopmen ’’, ‘‘so wa e’’, ‘‘model’’, and
hei co esponding wo d- o ms (q5) o e ie e publica-
ions conce ning con ac ual explana ion gene a ion ame-
wo ks. In addi ion, he e ms ‘‘a i icial in elligence’’ and
‘‘AI’’ we e used o ensu e e ie ing ele an AI- ela ed pub-
lica ions. Since RQ2add esses pu ely echnical issues o
11978 VOLUME 9, 2021
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
s a e-o - he-a implemen a ions o such ools, we u he
imposed an addi ional es ic ion on q5so ha i would
e u n only publica ions om such subjec a eas as compu e
science, ma hema ics, and enginee ing. Las bu no leas ,
he indings om q4and q5we e me ged o examine he
connec ion be ween he exis ing heo ies o con ac ual expla-
na ion and amewo ks o au oma ic con ac ual explana ion
gene a ion in he con ex o XAI (q6).
I is impo an o no e ha publica ions e ie ed as a
esul o q4 o m an exhaus i e se o pape s add ess-
ing RQ1. Simila ly, publica ions ob ained as a esul o
q5add ess RQ2. Finally, he pape s ha q6 e u ned
add ess RQ3.
D. INCLUSION AND EXCLUSION CRITERIA
The publica ions e ie ed du ing he ini ial sea ch we e sub-
sequen ly inspec ed on he basis o he ollowing inclusion
and exclusion c i e ia. To add ess he epis emology o con-
ac ual explana ion, we il e ed he e ie ed publica ions
o include in he collec ion o p ima y s udies only hose
sa is ying he ollowing c i e ia: (1) a publica ion p oposes
a con as i e o coun e ac ual o con as i e-coun e ac ual
app oach o explana ion o (2) i con ains a clea ly o mula ed
de ini ion o coun e ac ual o con as i e explana ion e e -
ing o o he publica ions in he co esponding ield. In o de
o cap u e exis ing compu a ional amewo ks o con ac ual
explana ion gene a ion, we included publica ions ha : (1)
p esen a no el app oach, me hod, o amewo k o con ac-
ual explana ion gene a ion whose ou pu can se e o explain
he easoning o an AI algo i hm and (2) a e ound in such
subjec a eas as compu e science, ma hema ics, enginee ing
as well as in hei sub- ields.
In con as , we excluded duplica e epo s o he same
s udies appea ed in bo h Scopus and WoS. As o he pub-
lica ions ela ed o RQ1, we also emo ed: (1) he s udies
whose con en s did no in oduce any con ac ual heo y o
explana ion o (2) hose con aining no o mal o in o mal
de ini ion o con as i e o coun e ac ual o con as i e-
coun e ac ual explana ion. As o he publica ions ela ed
o RQ2, we disca ded: (1) he publica ions which we e
no ela ed o AI algo i hms o applica ions as well as
(2) hose whe e he p oposed amewo k did no p o-
ide any human-comp ehensible con ac ual explana ions as
ou pu .
E. DATA EXTRACTION AND SYNTHESIS
Table 1shows he numbe o publica ions e ie ed a e each
independen que y, duplica es ound among hem in Scopus
and WoS, as well as Candida e P ima y S udies (CPS). No e
ha he numbe s o duplica es indica ed in Table 1 e e
only o wi hin-que y duplica es, i.e., he same publica ions
e ie ed om Scopus and WoS o he gi en single que y.
Recall ha q4and q5exhaus i ely co e all he h ee esea ch
ques ions. Hence, he numbe s o CPS a e calcula ed as a sum
o he publica ions e ie ed a e q4and q5. Fu he mo e,
CPS a e educed by he numbe o publica ions add essing
TABLE 1. Numbe s o publica ions e ie ed a e each single que y as
well as hose o ming he pool o candida e p ima y s udies. The numbe s
o publica ions making pa o he p ima y s udies a e highligh ed in bold.
FIGURE 2. A low diag am o he p ima y s udy selec ion on he basis o
que ies q4and q5(nis he numbe o publica ions a each s age).
RQ3because hey a e ound in bo h se s o publica ions
collec ed o RQ1and RQ2and a e he e o e duplica es.
Fig. 2displays he low diag am o he p ima y s udy
selec ion. A sum o 338 publica ions (207 om Scopus and
131 om WoS) made up he collec ion o CPS add essing he
esea ch ques ions. 107 wi hin-que y duplica es we e iden i-
ied and emo ed om u he analysis. In addi ion, 29 mo e
duplica es we e excluded when me ging he se s o publica-
ions e ie ed a e q4and q5. All in all, 136 duplica es we e
emo ed.
The i le, abs ac , and au ho keywo ds o each candida e
p ima y s udy we e sc eened o disca d he s udies i ele-
an o he esea ch ques ions posed. As shown in Fig. 2,
75 publica ions we e deemed i ele an and il e ed ou a his
s age. A deepe analysis o he emaining 127 publica ions
VOLUME 9, 2021 11979
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
FIGURE 3. A axonomy o con ac ual explana ion eme ging om ou sys ema ic li e a u e e iew.
TABLE 2. The exhaus i e lis o all he p ima y s udies in ela ion o each
esea ch ques ion.
en o ced us o disca d 33 s udies which did no sa is y he
inclusion c i e ia. Finally, 19 pape s we e added o he e iew
upon inspec ing he bibliog aphy o he p ima y s udies. As a
esul , 113 unique publica ions o med he exhaus i e pool o
p ima y s udies.
Table 2p esen s he lis o p ima y s udies selec ed o he
e iew. Thus, a collec ion o 74 ou o 113 (65.49%) o igi-
nal p ima y s udies we e ound o o mula e de ini ions o
con ac ual explana ion and/o add ess heo e ical accoun s
he eo (RQ1). In addi ion, 52 ou o 113 (46.02%) publi-
ca ions desc ibe amewo ks (o ex ensions o o he ame-
wo ks) o con ac ual explana ion gene a ion (RQ2). No e
ha 13 ou o 113 (11.50%) p ima y s udies we e ound o
add ess bo h RQ1and RQ2and he e o e answe RQ3.
The ollowing da a we e ex ac ed om each p ima y
s udy: i le, au ho s, yea o publica ion, au ho keywo ds.
In addi ion, all publica ions ela ed o RQ1we e ead o ana-
lyze con ac ual heo ies o explana ion and, subsequen ly,
ex ac he sough - o de ini ions o con ac ual explana ion.
As RQ2conce ned a b oade numbe o echnical cha ac e -
is ics o con ac ual explana ion amewo ks, we addi ionally
ex ac ed he ollowing in o ma ion: (1) he p oblem ha he
e ie ed amewo k aims o sol e; (2) he me hod p oposed
o con ac ual explana ion gene a ion; (3) he o m o ou pu
explana ion ( o ins ance, ex ual o isual); and (4) he co -
esponding e alua ion me hods. Based on he da a ex ac ed
om he p ima y s udies, he publica ions we e g ouped and
classi ied in acco dance wi h he a o emen ioned c i e ia.
IV. RESULTS
P io o answe ing he esea ch ques ions, we ca ied ou a
bibliome ic analysis o e he esul s o he gene al indepen-
den que ies on coun e ac ual and con as i e explana ion (q1
and q2, espec i ely) as well as hei union (q3). We epo
he esul s o he bibliome ic analysis in Sec ion IV-A. The
indings ela ed o he heo e ical accoun s o con ac ual
explana ion (RQ1) a e p esen ed in Sec ion IV-B. The analy-
sis o he compu a ional amewo ks o con ac ual explana-
ion gene a ion (RQ2) can be ound in Sec ion IV-C. Finally,
he publica ions desc ibing heo e ically g ounded compu a-
ional amewo ks (RQ3) a e epo ed in Sec ion IV-D.
An eme ging axonomy o con ac ual explana ion ame-
wo ks is depic ed in Fig. 3and o ms he co e o he esul s
discussed in he es o he manusc ip .
A. BIBLIOMETRIC ANALYSIS
The bibliome ic analysis o e he que ies q1,q2, and q3
allows us o ob ain a big pic u e o he esea ch a ea o con -
ac ual explana ion gene a ion and spo i s key cha ac e is ics.
To illus a e he s a e o a ai s wi hin he ield, we epo
annual scien i ic p oduc ion and maps o au ho keywo ds
e ealing he main p oblem-speci ic no ions. The e e ence
11980 VOLUME 9, 2021
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
TABLE 6. A classi ica ion o he con ac ual explana ion gene a o s by AI
p oblem.
hey emphasize he need o causal a ibu ion, as igno -
ing causal ela ions may lead o gene a ing un easible
coun e ac uals. The e o e, hey sugges a hyb id ame-
wo k o coun e ac ual explana ion gene a ion.
•Hyb id con as i e-coun e ac ual explana ion.
Kuo ikoski and Ylikoski poin o he mul i ace ed na u e
o con as i e-coun e ac ual explana ion. They a gue
‘‘ he e exis cons i u i e and possibly o mal coun e -
ac ual dependencies as well as combina ions o hese’’
[98]. Simila ly, Pex on sugges s a wo-le el hie a chy
o explana ion [108]: mic ophysical explana ions a e
non-causal and o m he lowe -le el o he hie a chy
whe eas manipulable causal explana ions a e placed a
he highe -le el.
C. CONTFACTUAL EXPLANATIONS AS DEFINED IN
AUTOMATIC GENERATION FRAMEWORKS (ANSWER TO
RQ2)
The analysis o he p ima y s udies ela ed o RQ2allows us
o ca ego ize he s a e-o - he-a con ac ual explana ion gen-
e a ion amewo ks in acco dance wi h he ollowing c i e ia:
(1) he p oblem he solu ion o which is o be explained (i.e.,
he AI p oblem); (2) he me hod employed o gene a e such
an explana ion (i.e., he explainabili y me hod); (3) he ou -
pu ep esen a ion o he explana ion; and (4) he e alua ion
me hod he eo .
1) AI PROBLEM
Con ac ual explana ions a e used o jus i y au oma ic deci-
sions ob ained o a a ie y o AI- ela ed p oblems. Table 6
p o ides he eade wi h a axonomy o he s a e-o - he-
a amewo ks om he p ima y s udies. I is de i ed om
he conside ed publica ions in e ms o he domain asks
ha hese amewo ks a e used o . As depic ed in Fig. 9,
mos con ac ual explana ion gene a ion amewo ks deal
wi h coun e ac ual explana ion (31 ou o 52 amewo ks;
59.62%). In con as , 17 ou o 52 (32.69%) gene a e con-
as i e explana ions. Only ou s udies (7.69%) use con-
as i e and coun e ac ual explana ions. One o hese s udies
[129] deals wi h bo h classi ica ion and eg ession.
•Con ac uals o classi ica ion. A as majo i y o
s a e-o - he-a AI applica ions ha gene a e con ac-
uals (42 ou o 52; 80.77%) a e used o explain he
FIGURE 9. Numbe s o amewo ks g ouped by AI p oblem wi h espec
o he ype o con ac ual explana ion gene a ed.
ou come o ML-based classi ie s, i.e., algo i hms ha
lea n a mapping unc ion :X−→ Y om a aining
da ase o nlabeled examples X= {xi|1≤i≤n} o
a disc e e ou pu a iable (class) Y= {yj|1≤j≤m}
whe e mis he numbe o classes. Indeed, con ac uals
a e pa icula ly sui able o in o ming he end-use why
a gi en da a example is assigned a pa icula class label.
Thus, he ou lined classi ica ion-o ien ed amewo ks
a e e alua ed on classi ie s based on logis ic eg ession
[55], [136], [153], [158], decision ees [46], [80], [122],
[140], [150], [155], [159], g adien boos ed decision
ees [147], suppo ec o machines [131], [138], [146],
andom o es s [81], [86], [142]–[144], neu al ne wo ks
[6], [48], [49], [91], [129], [130], [133], [135], [139],
[141], [145], [148], [151], o combina ions o hese
[100], [105], [134], [152], [154], [160]. In h ee s udies
[67], [128], [137], he classi ie s used in he expe imen s
a e no speci ied.
•Con ac uals o eg ession. One o he classi ica ion-
o ien ed amewo ks [129] is ex ended o also handle he
eg ession p oblem, i.e., lea ning a mapping unc ion
om a aining da ase X o a con inuous ou pu a i-
able Y. Howe e , he con inuous ou pu is, in his case,
subsequen ly con e ed o a lowe -scale disc e e alue
mapped o a ex ual desc ip ion simila o ha ypical o
a classi ica ion p oblem. The o he amewo ks add ess-
ing he eg ession p oblem aim o le e age g adien -
boos ed decision ees [61] and indica e how la ge e o s
in eg ession asks could be o e come [56].
•Con ac uals o knowledge enginee ing. The i s o
he conside ed amewo ks (in ch onological o de ) [93]
o e s explana ions by easoning abduc i ely o e he
in o ma ion ex ac ed om a gi en knowledge base o
answe a speci ic con as i e ques ion. In his se ing,
an explana ion is conside ed o be a consis en se o
disjunc i e li e als o he explana ion-seeking ques ion.
I is wo h no ing ha he amewo k is no designed o
p o ide explana ions o ML algo i hms.
VOLUME 9, 2021 11987
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
TABLE 7. A classi ica ion o he con ac ual explana ion gene a o s by
explainabili y me hod.
•Con ac uals o planning. Con ac ual explana ion
gene a ion appea s highly ele an o sequen ial asks
in obo ics such as au oma ic planning [94], [157],
[161]. Mo eo e , some o he obo ics- ela ed ame-
wo ks ound in ein o cemen lea ning se ings p o ide
explana ions o policies ha a obo selec s a a gi en
ime s ep [132], [156].
•Con ac uals o ecommenda ion. Ghazima in e al.
p opose a g aph-based ecommenda ion sys em in a
coun e ac ual se up [84]. They ob ain a coun e ac ual
explana ion by emo ing a minimal se o use ac ions
so ha he ou pu ecommenda ion changes.
•Con ac uals o con lic esolu ion. Mosca e al.
in oduce an a gumen a ion-based amewo k o social
ne wo k managemen [149]. They use con as i e expla-
na ions o answe c i ical ques ions abou agen ac ions
in he con ex o mul i-use p i acy con lic .
2) EXPLAINABILITY METHOD
All he amewo ks gene a ing con ac ual explana ions can
be classi ied by hei explainabili y me hod as ei he model-
speci ic o model-agnos ic. The o me ype o implemen-
a ions is mean o explaining decisions o pa icula AI
algo i hms. The amewo ks o he la e ype gene a e expla-
na ions i espec i e o he na u e o he unde lying algo i hm.
Table 7p esen s he publica ions unde s udy g ouped in
e ms o he explainabili y me hod ha hey apply. The dis-
ibu ion o model-speci ic and model-agnos ic explainabil-
i y me hods o gene a ion o di e en ypes o con ac ual
explana ion is shown in Fig. 10. Mos amewo ks deal wi h
coun e ac ual model-agnos ic me hods.
•Model-speci ic con ac ual explana ion gene a o s.
Se e al model-speci ic amewo ks gene a e coun e -
ac uals o explain he ou pu o decision ees [46], [80],
[122], [155]. Fo ins ance, Fe nández e al. [80] p esen a
ecu si e algo i hm which ex ac s coun e ac uals in he
o m o con as -class decision ee nodes. The ele ance
o he gene a ed coun e ac uals is hen measu ed by cal-
cula ing a a ian o he Gowe dis ance. The p oposed
me ic penalizes he numbe o ea u e changes when
a e sing he ee so ha spa si y is p omo ed. Al e na-
i ely, Sokol and Flash ely on he Manha an dis ance
measu ing lea - o-lea dis ance in he ee o e ie e
he mos ele an coun e ac uals [46], [155]. Designed
speci ically o decision ees, hei ‘‘Glass-box’’ ame-
FIGURE 10. Numbe s o amewo ks g ouped by explainabili y me hod
wi h espec o he ype o con ac ual explana ion gene a ed.
wo k is a gued o be easily ex endable o cap u e he
ou pu o o he logical ( ule-based) models. Aguila -
Palacios e al. gene a e con as i e explana ions using
g adien boos ed decision ees o o ecas p omo ional
sales [61]. The esea che s make use o he weigh ed
Euclidean dis ance o p esen he o ecas as a con as
o he neighbou ing ec o ized p omo ions. S epin e al.
e ie e coun e ac uals om a ule ma ix whe e each
ule is encoded in e ms o all possible ea u e alues
[122]. Subsequen ly, he gene a ed coun e ac uals a e
anked using a XOR-based dis ance o ind he mos
ele an coun e ac ual pe inen o he gi en con as
class. This me hod is u he ex ended o gene a ing
coun e ac uals o uzzy decision ees.
A numbe o amewo ks add ess speci ic p ope ies o
coun e ac uals. Thus, Us un e al. ackle he p oblem
o ac ionabili y, i.e., cons aining he gene a ed coun-
e ac uals in such a manne ha he imposed changes
‘‘do no al e immu able ea u es’’ and ha hey ‘‘do
no al e mu able ea u es in an in easible way’’ [158].
To app oach his p oblem, a mixed in ege p og am-
ming me hod is employed. Russell e al. adop a simila
app oach o encompass con inuous and disc e e a i-
ables as well as he combina ion o he wo [153]. The
main ocus o he wo k is howe e placed on assessing
cohe ence and di e si y o gene a ed coun e ac uals.
In o de o gua an ee he cohe ence o he coun e -
ac ual da a example used o explana ion, an in ege
p og amming-based me hod is p oposed. In addi ion,
he gene a ed coun e ac ual explana ions a e claimed
o be di e se, as di e si y cons ain s a e applied i e -
a i ely o a se o candida e coun e ac uals. Howe e ,
his amewo k is limi ed o: (1) explaining p edic ions
o only linea classi ie s and (2) a simple s uc u e o he
ex ual explana ion empla e.
A la ge numbe o amewo ks a e limi ed o explain-
ing he ou pu o pa icula models due o ask-speci ic
cons ain s. Fo ins ance, se e al explana ion gene a o s
add ess compu e ision asks. Hend icks e al. bind
11988 VOLUME 9, 2021
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
an inpu isual image wi h a pai ed ex ual coun e ac-
ual explana ion gene a ed by a ecu en neu al ne -
wo k [86]. In hei amewo k, a numbe o candida e
explana ions (bo h image- ele an and non- ele an ) a e
gene a ed, pai ed, and anked. The bes coun e ac ual
explana ion is hen selec ed o be he mos class-speci ic
o he coun e ac ual image while being he mos el-
e an o he inpu image. Goyal e al. a gue ha hei
model is mo e ai h ul by design, as i gene a es isual
explana ions di ec ly om ‘‘ he a ge model based on
he ecep i e ield o he model’s neu ons’’ [139].
Two model-speci ic amewo ks a e ound in he con-
ex o ideo p ocessing. Fo ins ance, Akula e al.
[128] p esen an empi ical s udy whe e inpu ideo
ames a e pai ed wi h he co esponding AND-OR
g aphs, i.e., composi ional ecu si ely de ined g aph-
based knowledge ep esen a ions cap u ing con ex ual
in o ma ion. The explana ions based on such g aphs
a e passed on o human subjec s o e alua e he con-
as i e answe s o he p ede ined ques ions. Al e na-
i ely, Kanehi a e al. ain a pos -hoc explana o y model
o jus i y a ideo classi ie ’s ou pu [91]. A coun e ac-
ual explana ion is, in his case, dependen on how likely
a selec ed egion in he gi en ame is classi ied posi i e
and no nega i e, hence all such egions a e sco ed and
no malized.
In acco dance wi h he indings in he p e ious
Sec ion IV-C1, con ac ual explana ions ha e a g ea
po en ial o au oma ic planning- ela ed asks. Mos
explana ion gene a o s mean o planning-based asks
a e model-speci ic due o he p oblem- and app oach-
speci ic es ic ions p e en ing hem om being used o
o he AI challenges. Fo ins ance, Kim e al. employ a
Bayesian p obabilis ic model o gene a ing con as i e
explana ions [94]. Thus, he amewo k ope a es on a
pai o plan aces de ined in e ms o linea empo al
logic empla es. The p oblem o ob aining con as i e
explana ions is designed as a Bayesian in e ence p ob-
lem, wi h he pos e io dis ibu ion o be maximized
de ined as he p obabili y o a con as i e explana ion
gi en a se o posi i e and nega i e plan aces. Con-
e sely, S eedha an e al. conside he ask o au oma ic
analysis o coun e ac ual explana ions in hei ‘‘Hie -
a chical Expe ise-Le el Modeling’’ amewo k [156].
A obo p o ides a use wi h a plan o he nex ac ion
o ake. Then, he obo expec s he use o espond wi h
a se o oils. The obo ’s ask is hen o con incingly
e u e he oils by o e ing a minimal explana ion o
why he oils a e no accep able unde he gi en ci -
cums ances. In addi ion, Chak abo i e al. o mula e
he mul i-model planning p oblem as a uple consis ing
o he planne ’s model o he p oblem and he co e-
sponding human app oxima ion he eo [132]. As plan
explicabili y is e o mula ed in e ms o i s comp ehen-
sibili y by an end-use . The obo ’s model is adap ed o
he upda es o human’s model o he p oblem.
The p oblem o con as i e explana ion gene a ion o
planning is also ound o be amed in he ein o cemen
lea ning se ing. Fo ins ance, Sukke d e al. o mula e
he planning p oblem as he sho es s ochas ic pa h
p oblem and de elop he co esponding p oblem sol e
o ob ain a con as i e explana ion [157]. Hence, hei
objec i e is o ind an op imal policy ‘‘ ha minimizes
he expec ed cumula i e cos o eaching a goal s a e
o e all closed policies’’. The explana ion is belie ed
o jus i y he ejec ion o he policies al e na i e o he
op imal one. In addi ion, Zhao and Sukke d explain
an au onomous sys em’s beha iou modeling i as a
Ma ko decision p ocess [161]. Thus, a con as i e
explana ion is p esen ed as a p oduc o he analysis o
he op imal policy a he nex ime s ep and an opposing
policy on he basis o he objec i e alues.
•Model-agnos ic con ac ual explana ion gene a o s.
A la ge numbe o model-agnos ic amewo ks ea con-
ac ual explana ion gene a ion as an op imiza ion p ob-
lem in a pos -hoc manne . Wach e e al. design a gene ic
coun e ac ual explana ion amewo k o ind he closes
poin o he es da a example [6]. Fixing he op imal
se o weigh s o a ained classi ie , he objec i e unc-
ion minimizes he dis ance be ween he nea es da a
poin s o opposing classes. No e ha coun e ac ual da a
poin s can be syn hesized a i icially. The esea che s
sugges he use o he Manha an dis ance weigh ed by
he in e se median absolu e de ia ion o calcula e he
p oximi y o a coun e ac ual o he inpu da a example.
Ano he case o coun e ac ual explana ion gene a ion
ega ded as an op imiza ion p oblem is he ‘‘Cons ained
Ad e sa ial Examples’’ amewo k [148]. Ad e sa ial
examples ha could se e as he basis o he coun e ac-
ual explana ion o he ou pu o deep lea ning models
a e sea ched o wi h he aim o minimizing he loss
wi h espec o he a ibu es ( ea u es) be ween he o ig-
inal and coun e ac ual da a examples. The esea che s
a emp o ind he bes coun e ac ual explana ion by
minimizing he numbe o a ibu es changed. Fu he -
mo e, he g adien di ec ion is cons ained o ensu e he
e hical adequacy o he explana ion gene a ed. Dandl
e al. [134] o mula e coun e ac ual sea ch as a mul i-
objec i e op imiza ion p oblem using a dis ance me ic
o mixed ea u e spaces aiming o ob ain spa se and
mos plausible coun e ac uals. Labaien e al. gene -
a e con as i e explana ions o ime-se ies da a [141].
The explana ion gene a ion is conside ed a wo- old
op imiza ion p oblem o inding pe inen posi i es and
nega i es. Pawelczyk e al. make use o an au oencode
a chi ec u e o a p e ained classi ie pe o ming coun-
e ac ual sea ch in he nea es neighbo s yle [151].
Model-agnos ic amewo ks a e la gely ound o use
decision ees as pa o he easoning mechanism
ins ead o explaining hei ou pu . In con as o he
model-speci ic amewo ks ope a ing on decision ee
ou pu , Guido i e al. employ decision ees as pa
VOLUME 9, 2021 11989
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
o econs uc ing he easoning behind any a bi a y
classi ie in a pos -hoc ashion [140]. In hei ‘‘Local
Rule-based Explana ion’’ amewo k, hey gene a e a
local neighbou hood o he gi en p e-classi ied da a
example using a gene ic algo i hm and subsequen ly
ain a decision ee on ha newly ob ained da ase
o selec a minimally dis an oil wi hin ha local
neighbou hood. Simila ly, an de Waa e al. andomly
sample o gene a e a da a se in he neighbou hood
local o he da a poin in ques ion [159]. A decision
ee is hen ained o selec he oil based on he
minimum numbe o nodes be ween he o iginal da a
poin and he candida e oils. Fu he mo e, hei ‘‘Foil-
T ees’’ amewo k p o ides he me hodological basis
o pe cep ual-le el con as i e explana ion gene a ion
wi hin he ‘‘Pe cep ual-Cogni i e Explana ion’’ ame-
wo k [150]. Subsequen ly, he gene a ed con as i e
explana ions a e a ibu ed o a speci ic g oup o use s
by means o on ology enginee ing a he cogni i e le el
o he amewo k o make he explana ions adap i e. In
con as , Ma ens and P o os a gue ha decision ees
a e an inadequa e ool o ep esen ing, e.g., la ge doc-
umen s [146]. Hence, hey sugges he model-agnos ic
‘‘Sea ch o Explana ions o Documen Classi ica ion’’
algo i hm o e ie ing coun e ac ual explana ions.
Howe e , i is only di ec ly applicable o bina y linea
classi ie s, whe eas heu is ics a e p oposed o non-
linea models.
Se e al model-agnos ic amewo ks aim a measu ing
speci ic p ope ies o con ac uals. Anjomshoae e al.
[129] ocus on con as i e explana ions ha maximize
con ex ual impo ance and con ex ual u ili y. On he
one hand, con ex ual impo ance measu es he ex en o
which he inpu ea u e alues a ec he black-box algo-
i hm’s ou pu . On he o he hand, con ex ual u ili y es i-
ies how a o able he alues o he selec ed ea u es a e
o a gi en decision. Thus, he con ex -based alues a e
calcula ed o each ea u e used by a black-box model
obse ing he changes in he ou pu as he inpu a ies
ac oss he ange o all possible inpu alues. Being based
on model-agnos ic and p oblem-independen concep s,
his amewo k is shown o be uni e sally applicable o
a ious classi ica ion and eg ession algo i hms. How-
e e , he scalabili y o such an algo i hm is limi ed o
he use-cases ope a ing on a small numbe o ea u es.
A simila limi a ion is obse ed due o possibly high
a iabili y o he inpu . Laugel e al. aise he issue o
jus i ica ion o coun e ac ual explana ion [144]. They
a gue ha a syn hesized coun e ac ual da a poin mus
be connec ed o he aining da a. Coun e ac uals a e
selec ed om a local neighbou hood ci cling a ound
he es example wi h he adius o he dis ance o he
closes co ec ly p edic ed da a poin o a con as -class.
The candida e coun e ac uals a e hen clus e ed, as he
ini ial local neighbou hood is upda ed o become a mo e
ex ensi e hype sphe ical laye , un il i can no longe
be ex ended. Laugel e al. [100] enhance he wo k on
jus i ied coun e ac ual explana ions. They a gue ha
he dis ance om he es ins ance o a coun e ac ual
does no su icien ly measu e coun e ac ual’s ele ance,
as he coun e ac ual in ques ion may appea discon-
nec ed om he g ound- u h da a. Thus, a coun e ac-
ual is deemed jus i ied i i can be connec ed o an asso-
cia ed g ound- u h da a ins ance wi hou c ossing he
decision bounda y. Fe nández e al. in oduce he no ion
o coun e ac ual se s o enhance coun e ac ual di e -
si y [81]. They explain andom o es p edic ions by
using di e en ee p edic o s so ha he esul ing coun-
e ac ual se con ains he mos ele an coun e ac ual.
The o he neighbo ing coun e ac uals se e o di e si y
he ou pu explana ion. Mo hilal e al. a e also con-
ce ned wi h coun e ac ual di e si y [105]. They design
a loss unc ion wi h a di e si y me ic o e he gene a ed
coun e ac uals o p o ide end-use s wi h mul iple ele-
an coun e ac ual explana ions. Kusne e al. p opose
a causal model o assess he so-called ‘‘coun e ac ual
ai ness’’ [55]. I is wo h no ing ha coun e ac uals a e
p esen ed in he o m o condi ional dis ibu ions and no
s uc u al equa ions despi e he ac ha he causal model
employed ollows Pea l’s o malism [37].
Simila ly o he model-speci ic amewo ks, nume ous
model-agnos ic explana ion gene a o s a e ound o be
ask-speci ic. In compu e ision- ela ed classi ica ion
asks, Chang e al. ind he smalles egion in he image
whose subs i u ion would change he classi ie ’s p edic-
ion [133]. They employ a gene a i e model o cons uc
a saliency map while masking he o he egions o he
inpu image. Simila ly, Dhu andha e al. add ess an
op imiza ion p oblem o e a pe u ba ion a iable o
p oduce a con as i e explana ion o he image classi i-
ca ion ask [135]. Howe e , he p oximi y o he selec ed
coun e example o he es poin is, in his case, gua an-
eed by using an au oencode .
3) OUTPUT REPRESENTATION
The conside ed amewo ks ou pu con ac ual explana ions
in se e al ways. Depending on he p oblem conside ed, con -
ac uals a e p esen ed in he o m o : (1) in e als o speci ic
alues o he app op ia e ea u e alues whose al e a ion
would ha e changed he ou pu (i.e., nume ical o ea u e-
based ou pu ); (2) single- o mul iple-sen ence cohe en ex
(i.e., linguis ic ou pu ); (3) speci ic egions in he inpu image
(i.e., isual ou pu ); o (4) a mul i-modal combina ion o
(some o ) he abo e (see Table 8). As depic ed in Fig. 11,
mos amewo ks ocus on nume ical coun e ac ual ou pu .
•Nume ical ( ea u e-based) con ac ual explana ion.
Nume ical alues (o in e als o alues) associa ed o
he mos ele an ea u es usually explain he beha io
o AI algo i hms. They can be ep esen ed as logical
o mulas [67], [93], [94] o in abula o m e lec ing
necessa y changes o a ec he decision [55], [56], [61],
11990 VOLUME 9, 2021
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
TABLE 8. A classi ica ion o he con ac ual explana ion amewo ks by
ou pu ep esen a ion.
FIGURE 11. Numbe s o amewo ks g ouped by ou pu ep esen a ion
wi h espec o he ype o con ac ual explana ion gene a ed.
[81], [105], [136], [147], [148], [151], [154], [158],
[160]. They can be ex ac ed om in e p e able ea u e-
alue pai s as a esul o p uning in he sea ch space
[132]. In addi ion, hey can eplica e he in e nals o he
classi ie ’s s uc u e, e.g., in he o m o decision ee
nodes o ules [80], [140], [152].
•Linguis ic con ac ual explana ion is a piece o g am-
ma ical single- o mul iple-sen ence ex in na u al lan-
guage. Single-sen ence ex ual explana ions combine
a ex ual desc ip ion wi h explici ly s a ed nume ical
ea u e alues [6]. Such explana ions sugges ea u e-
alue based ins uc ions [48], [122], [131] o al e na-
i e ac ions o a possible ou pu change [84], [149].
They also answe end-use ’s inqui ies wi h espec o he
au oma ic decision in ques ion [128], [159]. In con as ,
mul iple-sen ence explana ions p o ide end-use s wi h
speci ic de ails o he gi en decision [153], [156], [157],
[161] o explain mul iple decisions a once [146].
•Visual con ac ual explana ion. On he one hand,
isual explana ions o non- isual inpu da a (i.e.,
da ase s con aining con inuous o ca ego ical ea u e-
alue pai s) plo ea u e- alue pai dependencies
[134], [141], [142]. On he o he hand, isual inpu
da a (i.e., images) a e associa ed wi h saliency maps
[133] o explained by con as i e pa e ns be ween he
gi en da a example and ha o an opposing class in one
i e a ion [130], [144] o a se ies he eo [49]; by depic -
ing c i ical egions absen in he inpu da a example ha
de e mine wha lacks in he image o be classi ied di e -
en ly [135]; o by isualizing spa ial egions associa ed
o da a examples o opposed classes [139].
•Mul i-modal con ac ual explana ion is a combina ion
o nume ical and/o linguis ic and/o isual explana-
ions. Mul i-modal explana ions a e claimed o enhance
human- obo in e ac ion [46]. They a e o en selec ed o
be he mos app op ia e whe e he p oblem add essed
is conce ned wi h pai ing a compu e ision p oblem
wi h a na u al language p ocessing ask such as objec
de ec ion and language g ounding [86]. In addi ion,
isual-linguis ic explana ions iden i y coun e ac uali y
in ideos [91] and allow o dialogic in e ac ion [150].
Such hyb id coun e ac uals (in e ms o hei ou pu ep-
esen a ion) may as well complemen each o he while
add essing he same ask. Fo ins ance, Gomez e al.
isualize he gene a ed explana ions in he o m o ba
plo s combining hem wi h explici ly s a ed nume ical
alues [138]. Al e na i ely, Liu e al. combine ea u e
impo ance ba plo s wi h isual inpu and ou pu [145].
While a ex ual explana ion summa izes he deg ee o
impo ance o he selec ed ea u es, a isual explana ion
may p esen con ex ual in-me hod me ics ha jus i y
he classi ie ’s easoning [129]. Con ac ual explana-
ions, as a mix u e o abula and isual ou pu ep esen-
a ions, appea also in an augmen ed eali y amewo k
[137]. Ne e heless, explana ions o di e en modali ies
a e no necessa ily me ged. To ensu e he uni e sali y o
he p oposed app oaches, speci ic ea u e alues a e p e-
sen ed o asks wi h da ase s con aining only con inuous
ea u es, i.e., whe e he same me hod is used o ou pu
images o a handw i en digi classi ica ion p oblem
[143]. Finally, in e ac ion wi h use s can be enhanced
by means o oice-based explana ions combined wi h
ex ual explana ions [46], [155].
4) EVALUATION METHOD
E alua ion o gene a ed con ac ual explana ions is an issue
o main conce n. Un o una ely, despi e an inc easingly
expanding use o con ac ual explana ions, no uni o m se o
e alua ion me hods has been adop ed so a . Hence, i is wo h
aking a look a e alua ion me hods om o he gene a ion-
o ien ed sub-a eas o AI. Fo ins ance, i is common o dis-
inguish be ween in insic and ex insic e alua ion me hods
in na u al language gene a ion [163]. In insic e alua ion
implies assessing he pe o mance o a na u al language gen-
e a ion sys em (o i s modules) as an isola ed uni . In con as ,
ex insic ( ask-based) me hods a e designed o es ima e how
success ully he sys em pe o ms wi h espec o an ex e -
nal ask. In addi ion, Ga and K ahme make a dis inc ion
be ween ‘‘objec i e’’ (au oma ic, co pus-based) and ‘‘subjec-
i e’’ (human judgemen s) me ics [164]. Objec i e me ics
include (bu a e no limi ed o) p ecision- and/o ecall-
o ien ed sco es, numbe o inse ions/dele ions/subs i u ions,
VOLUME 9, 2021 11991
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
TABLE 9. A classi ica ion o he con ac ual explana ion gene a o s by
e alua ion me hod.
FIGURE 12. Numbe s o amewo ks g ouped by e alua ion me hod wi h
espec o he ype o con ac ual explana ion gene a ed.
e c. In u n, subjec i e me ics measu e eadabili y, accu acy,
ele ance o he gene a ed ex , as pe cei ed by humans.
Thanks o hei me hodological uni e sali y, hey can be
ex apola ed o o he (non-linguis ic) modali ies o gene a ed
explana ions (e.g., nume ical, isual, o mul i-modal) and a e
he e o e used o o m he basis o he e alua ion me hod
classi ica ion in his e iew. I is wo h no ing ha all he
conside ed amewo ks a e e alua ed by means o in insic
(ei he subjec i e o objec i e) me ics. Hence, we only make
a clea dis inc ion be ween subjec i e and objec i e e alua-
ion me hods in his s udy (see Table 9 o de ails). A dis inc-
ion be ween he use o di e en ypes o e alua ion me ics
can be seen in Fig. 12. I is easy o app ecia e how mos
amewo ks deal wi h objec i e e alua ion o coun e ac uals.
Le us gi e u he de ails below, ega ding he ou g oups o
publica ions in Table 9.
•No e alua ion de ails p o ided. 17 ou o 52 (32.69%)
o he conside ed publica ions do no e alua e hei
amewo ks, i.e., nei he au oma ic me ics o con ac-
ual explana ion gene a ion a e sugges ed no a human
e alua ion su ey is p esen ed in such publica ions.
Howe e , whe eas ce ain publica ions do no p o ide
any speci ic e alua ion me hod, some do s ess ha
human e alua ion should be encou aged o es ima e he
quali y and e ec i eness o he gene a ed coun e ac u-
als [46], [150], [153].
•Subjec i e e alua ion. The subjec i e me hods include
human p e e ences o ce ain ypes o con ac ual
explana ion o e o he s. Fo ins ance, Akula e al. show
ha ha con as i e explana ion-seeking ques ions a e
in gene al be e answe ed by means o con ac ual
explana ions [128]. They classi y con as i e ques ions
in he ollowing 10 ca ego ies sugges ing he empla e
ques ions o a bi a y objec s x,x1, and x2(all being o
some class X) and y,y1, and y2(all being o some o he
class Y):
–WH-X: ‘‘Why x a he han no x?’’;
–WH-X-NOT-Y: ‘‘Why x a he han y?’’;
–WH-X1-NOT-X2: ‘‘Why x1 a he han x2?’’;
–WH-NOT-Y: ‘‘Why no y?’’;
–NOT-X: ‘‘Is i x a he han no x?’’;
–NOT-X1-BUT-X2: ‘‘Is i x1 a he han x2?’’;
–NOT-X-BUT-Y: ‘‘Is i x a he han y?’’;
–DO-X-NOT-Y: ‘‘Wha i i is x a he han y?’’;
–DO-NOT-X: ‘‘Wha i i is no x?’’.
–DO-X1-NOT-X2: ‘‘Wha i i is x1and no x2?’’
I is wo h no ing ha 6 ou o 10 ques ion ypes
(WH-NOT-Y, NOT-X, NOT-X1-BUT-X2, NOT-X-BUT-
Y, DO-NOT-X, and DO-X1-NOT-X2) ma ched wi h
au oma ically gene a ed con ac uals a e shown o be
highly p e e ed o ac ual explana ions.
In addi ion, Fe a io e al. p opose an augmen ed eali y-
based se ing o a o in e ac i i y and acili a e explain-
ing ML algo i hm ou pu o non-expe s [137].
Howe e , he wo a o emen ioned s udies [128], [137]
lack an e alua ion o he quali y o he gene a ed con -
ac ual explana ions hemsel es.
Lucic e al. asked 75 subjec s o judge in e p e abil-
i y, ac ionabili y, and us wo hiness o he gene a ed
con ac ual explana ions [56]. They concluded con ac-
ual explana ions a e highly in e p e able and ac ionable.
In addi ion, hey help use s unde s and why he model
makes la ge e o s while sol ing a eg ession p oblem
bu do no suppo use s’ us in he model’s ou pu .
In addi ion, Hend icks e al. p o ide esul s o human
e alua ion o he gene a ed explana ions [86]. Howe e ,
hese only include e alua ions o ac ual explana ions
and a e he e o e excluded om he axonomy g oup
being discussed.
•Objec i e e alua ion. A majo i y o he esea che s
p opose objec i e (au oma ic) me hods o e alua ing
au oma ically gene a ed con ac uals. A numbe o he
amewo ks a e e alua ed by means o accu acy-based
me ics [61], [91], [94], [159]. Kanehi a e al. p opose
one accu acy-based e alua ion me ic o isual and lin-
guis ic explana ions each: nega i e class accu acy and
concep accu acy, espec i ely [91]. Nega i e class accu-
acy es ima es he quali y o he isual explana ion as
he a io o he p obabili y o he con as class a e he
image egion in ques ion is masked ou . In u n, concep
accu acy es ima es how compa ible he ou pu linguis ic
explana ion is o i s isual coun e pa . I is calcula ed
as he in e sec ion o e union be ween a gi en egion
11992 VOLUME 9, 2021
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
and all bounding boxes in he image. Kim e al. de ine
he domain-speci ic accu acy o he au oma ic planning
p oblem o unique con as i e explana ions as a sum o
he numbe o aces in a posi i e se o aces sa is ying
a con ain o a con as i e explana ion and hose o he
nega i e se whe e he cons ain is unsa is ied o e all
plan aces [94]. The consis ency o ou pu explana ions
is o he wise shown by measu ing hei accu acy on he
basis o mean e o , mean absolu e e o , and mean
absolu e pe cen age e o [61].
An ex ensi e numbe o e alua ion me hods a e ound
o be s ic ly ask- o app oach-speci ic. Hend icks e al.
measu e wo d de ec ion (i.e., which wo ds a e no image
ele an by holding ou one wo d a a ime om he sen-
ence o de e mine he leas ele an wo d in he explana-
ion) and wo d co ec ion (i.e., a numbe o eplacemen s
o he oiled wo d wi h wo ds om a se o a ge wo ds)
[86]. Simila ly, Ma ens and P o os es ima e explana-
ion complexi y by calcula ing he a e age numbe o
wo ds in he sho es explana ion and p oblem complex-
i y acco ding o he o e all numbe o gene a ed expla-
na ions [146]. Fe nández e al. e alua e he ele ance
o he gene a ed coun e ac uals measu ing a Gowe
dis ance-based me ic in compa ison wi h he numbe
o ea u e changes and minimum dis ance (in e ms o
decision ee nodes) be ween he lea es in he gi en
decision ee classi ie [80]. Van de Waa e al. e alua e
he gene a ed explana ions by means o such model-
speci ic me ics as he a e age leng h o he explana ion
in e ms o decision ee nodes and he F1-sco e o he
oil- ee on he es se compa ed o he model’s ou pu
[159]. Kusne e al. es ima e coun e ac ual ai ness on
he basis o he densi y o he p edic ed da a o hei
causal models [55]. Laugel e al. claim ha unde s and-
abili y o he gene a ed explana ions can be es ima ed
by means o hei spa si y de ined as he numbe o non-
ze o coo dina es o he explana ion ec o [143]. Moo e
e al. measu e he numbe o solu ions, he dis ances o
he nea es aining se da a poin s, and he ans e abil-
i y o he gene a ed coun e ac uals o o he da ase s and
classi ie s [148]. S eedha an e al. calcula e he numbe
o p edica es ha a e used o gene a e he model la ice
[156]. Simila ly, Chak abo i e al. calcula e he numbe
o nodes in he sea ch space emaining a e p uning
[132]. Goyal e al. epo how o en he disc imina i e
egions lie inside he es da a example segmen a ions
as well as ele an speci ic key egions [139]. Labaien
e al. calcula e he numbe o changes o swi ch om
he o iginal o he selec ed con as i e sample ollowing
he da ase cons ain s [141]. To es ima e ai h ulness
o he gene a ed coun e ac uals, Pawelczyk e al. sug-
ges calcula ing he so-called deg ee o di icul y o a
coun e ac ual sugges ion o measu e how cos ly i is o
achie e he s a e o he gi en sugges ion [151]. Aiming
o p o ide ealis ic coun e ac uals, Sha ma e al. in o-
duce he coun e ac ual explana ion obus ness-based
sco e de ined as he expec ed dis ance be ween he inpu
ins ances and hei co esponding coun e ac uals [154].
In addi ion, he gene a ed coun e ac uals a e inspec ed
in e ms o ai ness which is calcula ed as he expec ed
dis ance be ween he inpu and a coun e ac ual o e
dis inc alues o a speci ied ea u e se . Me ick and
Taly e alua e ou pu explana ions in e ms o mean
ea u e a ibu ions o show he impo ance o ele an
e e ences [147]. Gomez e al. e alua e coun e ac uals
in e ms o da a dis ibu ion, ea u e impo ance, as well
as possible and ac ionable changes o he inpu [138].
Dandl e al. use he hype olume indica o me ic o
es ima e he quali y o he es ima ed Pa e o on du ing
coun e ac ual sea ch [134]. In addi ion, Chang e al.
measu e he weakly supe ised localiza ion e o o an
image de ec ion ask – he in e sec ion-o e -union a io
o e 0.5 wi h any o he g ound u h bounding boxes
and he saliency me ic, i.e., ‘‘ he log a io be ween he
bounding box a ea and he in-class classi ie p obabili y
a e upscaling’’ [133].
Se e al me ics can be ex ended o be applied o o he
app oaches. Lash e al. es ima e how much he p ob-
abili y o a gi en p edic ion educes gi en a ea u e
pe u ba ion as de e mined by a con as i e explana ion
[142]. Dhu andha e al. employ he concep o pe inen
posi i es (i.e., ‘‘ ac o s whose p esence is minimally
su icien in jus i ying he inal classi ica ion’’ [135])
and pe inen nega i es (i.e., ‘‘ ac o s whose absence is
necessa y in asse ing he inal classi ica ion’’) o e alu-
a e ac uals and coun e ac uals, espec i ely, o a gi en
classi ica ion ask. Bo h ypes o e alua ion me hods
highligh he ea u es suppo ing e idence as o mula ed
in he con as i e explana ion on he basis o he alues
ha a pe u ba ion a iable akes on. Fe nández e al.
e alua e coun e ac uals in e ms o he a e age o he
pai wise dis ances based on he ea u e ype and he
pe cen age o alid coun e ac uals [81].
Mo hilal e al. s ess ha coun e ac uals should be
e alua ed in e ms o alidi y (i.e., whe he a gene a ed
coun e ac ual eally leads o a di e en ou come), p ox-
imi y (i.e., ea u e-wise dis ance be ween he o iginal
and coun e ac ual samples), spa si y (i.e., numbe o
ea u es di e ing in he o iginal and coun e ac ual sam-
ples), and di e si y (i.e., ea u e-wise dis ance be ween
each pai o coun e ac uals) [105]. Simila ly, S epin
e al. calcula e ac ual and coun e ac ual explana ion
leng h o es ima e conciseness o he gene a ed expla-
na ions [122]. They also compu e he numbe o coun-
e ac uals and hei bes minimal dis ance o he ac ual
explana ion o assess he ele ance o coun e ac uals.
Rajapaksha e al. conside co e age (as an indica o o
ep esen a i eness o a ule o a gi en da ase ), con-
idence (i.e., he pe cen age o ins ances in he da ase
which con ain he consequen and an eceden oge he
o e he numbe o ins ances which only con ain he
an eceden ), li (i.e., an associa ion be ween an eceden
VOLUME 9, 2021 11993
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
and consequen ), le e age (i.e., he obse ed equency
be ween he an eceden and consequen ), and he num-
be o ea u es in explana ion o e alua ing hei ame-
wo k agains o he ule-based me hods [152]. Also,
Whi e and Ga cez ein oduce ideli y o he unde lying
classi ie on he basis o dis ance o he decision bound-
a y [160].
In addi ion, some o he model-agnos ic amewo ks
[140], [159] allow o measu ing how well he ou pu
o black boxes (i.e., ac ual ou pu o be explained) and
g ey boxes (i.e. in e p e able in e media e p edic o s)
mimic he local neighbou hood (i.e., ideli y) and he
da a example o be explained (i.e., hi ). Laugel e al.
measu e how jus i ied coun e ac uals a e by a e aging a
bina y sco e (one i he explana ion is jus i ied ollowing
he p oposed de ini ion, ze o o he wise) o e all he
gene a ed explana ions [100], [144].
I is wo h no ing ha he un- ime o explana ion gen-
e a ion algo i hms is epo ed in addi ion o he e alua-
ion me ics o se e al amewo ks [132], [139], [146],
[152], [156], [159].
•Hyb id e alua ion. Two amewo ks a e e alua ed
in e ms o bo h au oma ic me ics and human judg-
men s. Ghazima in e al. calcula e explana ion leng h
o discuss comp ehensi eness o explana ions as well
as es ima e hei use ulness and c edibili y by su ey-
ing 500 subjec s [84]. In addi ion, Le e al. compu e
ideli y, conciseness, in o ma ion gain, and in luence
[48]. Au oma ic me ics a e complemen ed wi h a use
s udy on in ui i eness, iendliness, comp ehensibili y,
and unde s andabili y o gene a ed explana ions.
D. LINKS BETWEEN THEORETICAL AND PRACTICAL
CONTRIBUTIONS TO CONTFACTUAL EXPLANATION
GENERATION (ANSWER TO RQ3)
We ind ha only ew o he exis ing compu a ional ame-
wo ks a e g ounded on heo ies o con ac ual explana ion.
Indeed, only 13 ou o 113 s udies (11.50%) we e p esen in
bo h o he pools o p ima y s udies ela ed o RQ1and RQ2.
Table 10 summa izes he cha ac e is ics o such heo e ically
g ounded con ac ual explana ion gene a ion amewo ks.
Mo eo e , only 3 ou o he 13 (23.08%) s udies in e p e
he insigh s om he heo e ical ounda ions o p opose hei
own con ac ual explana ion de ini ion o p oblem-o ien ed
pu poses. Kean s a es ha ‘‘explana ion in a i icial in el-
ligence is based on he in e ence o deduc ion’’ [93]. He
complemen s a deduc i e e idence-based explana ion wi h a
ede ined abduc i e con as i e explana ion d awing pa allels
o he ‘‘in e ence o he bes explana ion’’ [103]. He models
Lip on’s heo e ical amewo k dis inguishing wo ypes o
con as i e explana ion: non-p eclusi e (i.e., non- es ic i e)
and p eclusi e. The key aspec dis inguishing he wo ypes o
con as i e explana ion is in ega d o how a model explains
he con as gi en an explana ion-seeking ques ion. Thus,
a non-p eclusi e con as i e explana ion is ‘‘i ele an o he
model o explaining he con as ’’ being ‘‘necessa y in he
model o explaining he ques ion’’. On he con a y, a p eclu-
si e con as i e explana ion is assumed o be es ic ed by a
nega ed model o he con as .
Aguila -Palacios e al. [61] e e o Lip on’s de ini ion o
con as i e explana ion [12]. Re e ing o Pea l [37], Be ossi
ede ines causal explana ion in he con ex o XAI o be ‘‘a
se o ea u e alues o he en i y unde classi ica ion ha is
mos esponsible o he ou come’’ [67].
The es o wo ks ede ine con ac ual explana ion on
he basis o he p oblem-speci ic cons ain s wi hou explic-
i ly e e ing o he heo e ical ounda ions desc ibed in
Sec ion IV-B. D i en by he ask o au oma ic planning, Kim
e al. de ine a con as i e explana ion o be a cons ain sa is-
ied by a speci ic se plan aces [94]. Fe nández e al. de ine
a coun e ac ual o be a se o ea u e changes ha u n he
gi en da a example o be classi ied di e en ly [80]. Whe eas
his de ini ion is applicable o he classi ica ion p oblem in
gene al, he applicabili y o he amewo k is es ic ed o
decision ees only. Simila ly, Hend icks e al. explain isual
concep s o he image classi ica ion ask on he basis o he
so-called coun e ac ual e idence (i.e., an a ibu e disc imi-
na i e enough o ano he class o objec s in he image absen
in he gi en image) [86]. Ghazima in e al. [84] de ine a
coun e ac ual on he basis o hei model’s in e nal s uc u e:
an explana ion is deemed coun e ac ual i a e emo ing he
edges om he ecommenda ion g aph, he use ecei es a
di e en op- anked ecommenda ion. In addi ion, Kanehi a
e al. only speci y he linguis ic o m o a coun e ac ual
explana ion wi hou de ining i explici ly [91].
Finally, he e is a numbe o ma ginal in e p e a ions o
con ac uals among he RQ3- ela ed s udies. Laugel e al.
deno e a coun e ac ual as a speci ic da a ins ance ha
changes he algo i hm’s p edic ion [100]. Poyiadzi e al.
deno e a coun e ac ual o be ‘‘ he new s a e o he objec ’’
[49]. Ne e heless, he mos commonly accep able de ini ion
o a con ac ual in he obse ed RQ3- ela ed s udies s a es
ha a con ac ual explana ion is a se o minimal ea u e
modi ica ions ha makes he model change he p edic ion
[81], [105], [122].
V. DISCUSSION
The indings show ha a la ge body o esea ch has been
elabo a ed on heo e ical accoun s o con as i e, coun e ac-
ual, and con as i e-coun e ac ual explana ion. In addi ion,
he opic has ecen ly a ac ed a en ion om esea che s
in XAI (see Fig. 13). Thus, 50 ou o he 52 conside ed
s a e-o - he-a con ac ual explana ion gene a ion ame-
wo ks (96.15%; RQ2) ha e been de eloped om 2017 o
2020.
The esul s o he s udy, in ela ion o RQ1, show ha a
majo i y o he conside ed heo e ical accoun s o con ac ual
explana ion (49 ou o 74; 66.22%) specula e on he causal
na u e o explana ion. Howe e , whe eas mos esea che s
in philosophy o science ha e mainly used he concep o
coun e ac uali y o explain causal ela ions be ween en i-
ies in ques ion, causal in e ence is poo ly add essed in he
11994 VOLUME 9, 2021
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
TABLE 10. A summa y o cha ac e is ics o heo e ically g ounded compu a ional amewo ks o con ac ual explana ion gene a ion. CT s ands o
con as i e explana ion, CF means coun e ac ual explana ion, and CT-CF is con as i e-coun e ac ual explana ion.
FIGURE 13. Numbe s o heo e ical and compu a ional con ac ual explana ion gene a ion amewo ks
g ouped by yea o publica ion. Fo illus a i e pu poses, only he s udies published om Janua y 1990 o
Sep embe 2020 a e displayed.
pool o publica ions conce ning compu a ional amewo ks
o con ac ual explana ion gene a ion. Kean di ec ly e e s
o a causal accoun o con as i e explana ion o add ess he
p oblem o abduc i e easoning [93]. In addi ion, Lucic e al.
[56] explici ly speci y ha hei me hod is based on p e ious
wo k on philosophical accoun s o con as i e explana ion
[12] as well as on causal a ibu ion [165], [166]. Kusne e al.
[55] make use o causal in e ence models and he co espond-
ing ools p o ided by Pea l [37]. They assess how disc imi-
na o y he gene a ed coun e ac ual explana ions a e o he
gi en classi ica ion ask ou pu . On he o he hand, Be ossi
ede ines he concep o causal explana ion [67]. Following
a causal accoun o con ac ual explana ion, Fe nández e al.
in oduce weakly causal i educible coun e ac ual explana-
ion [136]. As mos o he cu en ML- asks a e cen e ed
a ound singling meaning ul pa e ns ou om uns uc u ed
da a, es ablishing causal ela ions appea s o be among he AI
p oblems ha a e ye o a ac global a en ion. This pa ly
explains why mos o he mode n con ac ual explana ion
gene a o s ocus on ea u e pe u ba ion when sea ching o
he mos ele an con ac uals and no es ablishing causal
ela ions be ween hem.
A he same ime, he o he compu a ional amewo ks
a e p ima ily non-causal. Fu he mo e, a s ikingly low num-
be o such amewo ks appea o be oo ed in heo e ical
accoun s o explana ion due o an imbalance in a o o
causali y-o ien ed heo e ical accoun s. Howe e , he amoun
o publica ions o RQ3may be somewha misleading, as con-
ac ual explana ions a e o en ede ined wi hou speci ically
e e ing o heo e ical con ibu o s in explana ion. This is
VOLUME 9, 2021 11995
I. S epin e al.: Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable AI
hypo hesized o be due o he p oblem-speci ic necessi ies
igno ed in p e ious heo e ical wo ks om di e en b anches
o science. Fo ins ance, Laugel e al. emphasize ha he
minimal pe u ba ions equi ed o change he p edic ed class
o a gi en obse a ion enable a use o unde s and which
ea u es locally impac he p edic ion and he e o e how i
can be changed [144]. In his in e p e a ion, coun e ac ual
explana ions a e concep ually mos simila o coun e ac-
uals as de ined by Lewis [36]. Indeed, while he concep
o ‘‘ he closes possible wo ld’’ does no always appea in
he ela ed publica ions, i u ns ou o be implici ly wi ed
in almos all wo ks. Ins ead, o he conside ed amewo ks
do no appeal di ec ly o any o he heo e ical accoun s
o explana ion add essed in RQ1. Hence, i is wo hwhile
aking a look a how con as i e and coun e ac ual expla-
na ions a e ede ined in he amewo ks no appea ing in
Sec ion IV-D.
In gene al, a consensus among esea che s has been
obse ed on how con ac ual explana ions a e de ined i e-
spec i e o he heo e ical amewo k p oposed by indi idual
au ho s. In humani ies and social sciences, a majo di e ence
in a ious con ac ual heo ies o explana ion is obse ed o
conce n he causal na u e o explana ion and i s ex apola ion
o non-causal cases. In compu e science and AI, no ions o
coun e ac ual and con as i e explana ion a e ound o be he
mos dissimila when applied o non-o e lapping p oblems.
Ne e heless, he co esponding line o esea ch in AI makes
li le use o he ich heo e ical backg ound accumula ed by
now. While some ule-based app oaches used in expe sys-
ems a e jus i ied heo e ically (e.g., see [93]), newly eme g-
ing asks p esen no el challenges o heo is s and call o
upda ing he heo ies de eloped so a .
Mo e p ecisely, ML-speci ic con ac ual explana ions a e
designed o answe he ques ion: ‘‘Why was he ou come Y
obse ed ins ead o Y0?’’ [148]. Anjomshoae e al. de ine
inding a con as i e explana ion as ‘‘con as ing ins ance
agains he ins ance o in e es ’’ [129]. Fe nández e al. spec-
i y ha a coun e ac ual is gene ally ega ded as a hypo-
he ical ins ance simila o an example whose explana ion
is o in e es bu wi h a di e en p edic ed class [80]. Also,
coun e ac ual explana ions ‘‘show a di e ence in a pa icula
scena io ha causes an algo i hm o change i s mind’’ [155].
As a majo i y o he conside ed amewo ks a e designed
o ackling classi ica ion p oblems, con ac ual explana ions
ope a e on he no ion o a con as -class (e.g., see [155])
answe ing he ques ion: ‘‘How is he p edic ion al e ed when
he obse a ion changes, gi en a classi ie and an obse a-
ion?’’ Fu he mo e, hese changes a e no mally expec ed o
be minimal [143].
Howe e , ce ain applica ion domains as well as he selec-
ion o a classi ie equi e esea che s o ede ine con ac uals
imposing ask-dependen cons ain s, which makes i nea ly
impossible o connec hem o any o he exis ing heo ies o
con ac ual explana ion. Fo ins ance, Ma ens and P o os
de ine a con as i e explana ion o a documen classi ica ion
ask o be a minimal se o wo ds such ha emo ing all wo ds
wi hin his se om he documen changes he p edic ed class
om he class o in e es [146]. In addi ion, Guido i e al.
e o mula e a coun e ac ual o be a se o spli condi ions o a
decision ee desc ibing he minimal numbe o changes in he
ea u e alues o a es example [140]. In image classi ica ion,
i is ound necessa y o de ec speci ic egions in he gi en
es image. Fo his ype o asks, he con as i e explana ion-
seeking ques ion is o mula ed as ollows: ‘‘Which pa s o
he image, i hey we e no seen by he classi ie , would mos
change i s decision? o which inpu s, when eplaced by an
unin o ma i e e e ence alue, maximally change he classi-
ie ou pu ?’’ [133], [139]. Simila ly, Dhu andha e al. ask
wha should be minimally and necessa ily p esen and absen
in he gi en image o jus i y i s classi ica ion [135]. Al e -
na i ely, coun e ac uals a e iewed as ‘‘solu ions ha a e
gua an eed o map back on o he unde lying da a s uc u e’’
[153]. Rede ined con as i e explana ions a e also ound in
he domain o obo ics and au oma ic planning. Acco ding
o Sukke d e al., a con as i e explana ion answe s he
ques ion why a gene a ed beha io is op imal wi h espec
o he planning objec i es o an au onomous sys em [157].
Al e na i ely, con as i e explana ions a e used o answe
why-no ques ions abou he sys em’s beha io in which he
consequences o he coun e ac uals in ques ion a e poin ed
ou [161].
In addi ion, he na u e o he explana ion-seeking ques ions
o compu a ional amewo ks dese es u he discussion.
Sokol and Flash dis inguish h ee ypes o coun e ac ual
explana ions: (1) a plain coun e ac ual (‘‘Why?’’) gene a ed
as he sho es possible class-con as i e coun e ac ual; (2) a
coun e ac ual explana ion no condi ioned on he indica ed
ea u e(s) (‘‘Why despi e?’’); and (3) a (pa ially) ixed coun-
e ac ual explana ion (‘‘Why gi en?’’) which is condi ioned
on a p ede e mined se o ea u es [46]. Hil on p oposes
di e en ypes o con as i e ques ions such as: (1) ‘‘Why X
a he han no X?’’; (2) ‘‘Why X a he han he de aul alue
o X?’’ and (3) ‘‘Why X a he han Y?’’ [166]. Following
his dis inc ion, Akula e al. ex end his se o con as i e
ques ions o o mula e en con as i e ques ion ypes o
coun e ac ual explana ion gene a ion [128] (see Sec ion IV-
C4). Al e na i ely, only linguis ic empla es o such expla-
na ions a e de ined wi hou any heo e ical g ounding in
acco dance wi h any accoun s desc ibed in Sec ion IV-B. Fo
ins ance, Sokol and Flash de ine a coun e ac ual explana ion
o be a piece o ex ollowing he empla e: ‘‘The p edic ion
is hp edic ioni. Had a small subse o ea u es been di e -
en h oili, he p edic ion would ha e been hcoun e ac ual
p edic ioniins ead’’ [155].
Rema kably, a wide ange o amewo ks a o au oma ic
e alua ion me hods. Thus, hey a ely place he end-use in
he cen e o he explana ion e alua ion p ocess. Howe e ,
we ind an inc easing numbe o in e ac i e amewo ks ha
a emp no only o p esen he au oma ically gene a ed expla-
na ions o he end-use bu also in e ac wi h him o he [46],
[150], [155]. P omo ing in e ac i i y (e.g., by engaging he
end-use o pa icipa e in an explana o y dialogue wi h he
11996 VOLUME 9, 2021
ha ing hem explained, legal egula ions conce ning da a p ocessing a e becoming widely adop ed, e.g. he Gene al Da a
P o ec ion Regula ion (GDPR) in he Eu opean Union [33]. Mo eo e , a new Eu opean egula ion on AI is in p og ess and
highligh s he impo ance o p ese ing he Eu opean alues by p omo ing us wo hy and esponsible human-cen ic AI
[9,34].
The gap be ween obscu i y o au oma ic decisions and hei explainabili y can be o e come by using in e p e able models
[37]. Among all AI ools, such so compu ing echniques as uzzy se s and sys ems ha e been shown o be no only in e -
p e able bu also explainable [3]. Thus, wo key ad an ages a e dis inguished when ela ing he p ope ies o in e p e abili y
and explainabili y o uzzy sys ems. Fi s , hei anspa en (i.e., in e p e able) s uc u e allows o making unambiguous
in e ences o why he gi en ou pu was p oduced. Second, he use o linguis ic a iables and ules enables such sys ems
o be explainable, i.e., o p oduce comp ehensible explana ions in na u al language.
Ne e heless, he abili y o demons a e e idence on why speci ic ou pu is p oduced (i.e., explain he ac ual ou pu ) may
no be su icien o display he unde lying easoning o he end use . The e o e, a ac ual explana ion may need o be com-
plemen ed wi h an explana ion o why some o he ou pu was no p oduced. Opposed o ac ual explana ions jus i ying he
gi en p edic ion, coun e ac ual (CF) explana ions (o coun e ac uals) in o m he end use abou minimally di e en al e -
a ions o he inpu ea u es o he ou come o change [41]. In he con ex o classi ica ion p oblems, CF explana ions a e
ypically designed as answe s o he empla e ques ion ‘‘Why was Pp edic ed a he han Q?” whe e Pis he ou pu ( ac ual)
class and Qis a non-p edic ed hypo hesized al e na i e CF class [29].
CF explana ion gene a ion is o en ega ded as an op imiza ion p oblem in sea ch o he da a poin o ano he class which
ep esen s he closes da a poin al e na i e o he es ins ance in an n-dimensional Euclidean space [46]. In he con ex o
uzzy se s and sys ems, howe e , such minimal changes may be desc ibed no only by means o a con inuous a iable ep-
esen ing nume ical ea u e alues (which we call ‘‘quan i a i e CFs” in his pape ) bu also by a disc e e linguis ic a iable
whose alues a e linguis ic e ms (which we e e o as ‘‘quali a i e CFs” in his pape ). In he o me case, dis inc i e (nu-
me ical) ea u es poin o speci ic alues, which a e minimally di e en om hose he es ins ance has, ha should be se
o he ou come o change. In he la e case, linguis ic e ms ep esen se s o sui able CF ea u e alues in o m o ex and
conceal he unde lying nume ical in e als.
The di e ence in end use ’s pe cep ion o hese ypes o CF explana ions emains unclea [45]. On he one hand, i may be
a ec ed by peculia i ies o he s uc u e o explana ion, such as he numbe o explana o y ea u es o explana ion leng h.
On he o he hand, use ’s pe cep ion may be in luenced by a deg ee o p ecision o he explana ion con en . Thus, quali a i e
CFs may be ega ded as pieces o imp ecise in o ma ion which can acili a e unde s anding o he communica ed explana-
ion bu may, howe e , be unde in o ma i e o e en misleading o he end use . Con e sely, quan i a i e CFs speci y ine-
g ained changes o alues o ea u es. Las bu no leas , exis ing me ics o measu ing quali y o CF explana ions (e.g., alid-
i y, p oximi y, di e si y, among o he s) a e s ongly ela ed o he da a used o explana ion gene a ion [31]. Howe e , hose
me ics igno e pe cep ual skills o he explana ion’s ecipien and may no be su icien o assessing he o e all explana ion
e ec i eness. In o de o make ano he s ep owa ds human-cen ic AI, i he e o e appea s necessa y o p opose no el
means o cap u ing and assessing human pe cep ion o explana ions.
As pa o p e ious wo k [41], we in oduced a me hod o gene a ing quali a i e CF explana ions applied o decision ees
(DT). Then, we gene alized his me hod o uzzy in o ma ion g anules [43]. In his pape , ou con ibu ion is ou old. Fi s ,
we ex end ou p e ious wo k wi h a gene alized Euclidean dis ance-based me ic o CF explana ion gene a ion which be e
g asps membe ship unc ion alues. Second, we p opose a no el gene ic-based quan i a i e CF explana ion gene a ion
me hod. Thi d, we de ine a new me ic o assessing he complexi y o au oma ed explana ions. Fou h, we ca e ully alida e
bo h quali a i e and quan i a i e CF explana ions ia human e alua ion in ag eemen wi h he bes known p ac ices o ai
and sound e alua ion o Na u al Language Gene a ion (NLG) and analyze he indings in e ms o explana ion complexi y as
expec ed o be pe cei ed by he end use .
The es o he manusc ip is s uc u ed as ollows. Sec ion 2p esen s a b ie o e iew o exis ing me hods o quan i a-
i e and quali a i e CF explana ion gene a ion. Sec ion 3in oduces ou me hods o gene a ing CF explana ions associa ed
o uzzy ule-based classi ica ion sys ems (FRBCS). Sec ion 4desc ibes he key cha ac e is ics o he expe imen al design o
subsequen human e alua ion s udies. Sec ion 5goes in de ail wi h he analysis o he da a collec ed in wo e alua ion su -
eys. Sec ion 6discusses he indings and o e s sugges ions on how hey can be exploi ed. Finally, we ou line di ec ions o
u u e wo k and conclude in Sec ion 7.
2. Rela ed wo k
CF explana ion gene a ion has in ecen yea s a ac ed inc easing a en ion om esea che s in he AI ield. As CFs
oppose ac ual and po en ial ou comes, hey a e mos widely used o explain he ou pu o a ious classi ie s, om linea
machine lea ning models o deep neu al ne wo ks [42]. Fu he , hey a e ex ensi ely ound ac oss di e en applica ion
domains. Fo example, CFs a e ound applicable in heal hca e whe e hey, e.g., se e o p o ide a pa ien wi h a bigge pic u e
o he isk o de eloping diabe ic e inopa hy [26] o in banking whe e CFs sugges ecommenda ions on necessa y changes
o ha e a loan applica ion app o ed i p e iously ejec ed [16].In addi ion, CF explana ions a e as well ex ensi ely used in
obo ics (e.g., in planning – o jus i y he choice o a obo o e o he easible bu un a o ed possible solu ions [44]). Despi e
nume ous po en ial applica ion domains, he use o CFs is ad ised o be con oled due o possible malicious implica ions. As
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
380
such, hey ha e been misused o misin e p e ed (wha may lead o da a b eaches) in cases o , e.g. passwo d masking o e-
o ing [20]. O he p i acy conce ns include in e ing sensi i e pa e ns o he aining da a o manipula ions wi h he
e ealed in e nals o he model [40].
In he con ex o quali a i e CFs, a numbe o gene a ion me hods ou pu CF se s o suppo di e si y. Fo example, Sokol
and Flash inspec he in e nal s uc u e o DTs in hei ‘‘Glass-Box” amewo k o gene a ing CF se s [40]. Thus, he au ho s
e ie e CF se s om he decision pa hs anking hem by hei lea - o-lea dis ance o he ac ual p edic ion. On a simila no e,
S epin e al. gene a e se -based (i.e., quali a i e) CFs om ei he c isp o uzzy DTs [41] bu also ega ding uzzy in o ma ion
g anules [43] while in oducing an ex a-linguis ic laye o app oxima e nume ical in e als o membe ship unc ion alues,
espec i ely, using p ede ined linguis ic e ms.
Whe eas he a o emen ioned me hods a e model-speci ic, i.e., hey only allow o explaining coun e ac ually he gi en
ou pu o he DT i sel , DT-based app oaches a e also used o model-agnos ic me hods. In hei LOcal Rule-based Explana-
ion (LORE) me hod, Guido i e al. employ a gene ic algo i hm o i s syn hesize a local neighbo hood a ound he es
ins ance which is subsequen ly used o ain a DT and gene a e CF se s [17]. The collec ion o CF se s is hen econs uc ed
om he decision pa hs. Then, he minimally di e en CF se is selec ed on he basis o he (minimal) numbe o Boolean spli
condi ions o he DT ha he gi en CF pa h does no sa is y. Maa oo e al. ex end LORE o uzzy logic-based applica ions by
p oposing Con ex ualised LORE o Fuzzy a ibu es (C-LORE-F) [26]. Al e na i ely o LORE, he esea che s o mula e a local
neighbo hood gene a ion app oach o sol ing he uni o m cos sea ch p oblem. Po en ial neighbo s a e gene a ed by apply-
ing i e a i e changes o e a single ea u e aking in o accoun in e sec ions be ween wo co esponding uzzy se s. Fu he ,
he au ho s p opose o induce he ules ins ead o building up a DT using he Dominance-based Rough Se App oach (DRSA)
whe e he decision ules ake in o conside a ion he p e e ence di ec ions o he inpu a iables. In addi ion, Fe nández e al.
ex ac CF se s om a andom o es classi ie by pa ly using indi idual ee p edic o s [12]. Fu he , hei Random Fo es
Op imal Coun e ac ual Se Ex ac o (RF-OCSE) p unes he sea ch space o candida e CFs using he minimum obse able
app oach o il e ou CFs whose dis ance o he es ins ance exceeds he bes up- o-now dis ance.
On he o he hand, quan i a i e (i.e., single-poin -ou pu ) CF explana ion gene a ion me hods add ess he op imiza ion
p oblem sea ching o an indi idual da a poin ound o be minimally di e en om he es poin unde conside a ion in
acco dance wi h he selec ed dis ance unc ion, e.g., Manha an dis ance weigh ed by he in e se median absolu e de ia ion
[46]. Simila ly, Moo e e al. use a di e en iable model on he basis o a g adien -based me hod o e he c oss en opy loss
unc ion o iden i y a single minimally dis an CF da a poin [30].
Al e na i ely, gene ic algo i hms a e also equen ly used o gene a e CFs [39]. Model-agnos ic gene ic algo i hms a e
used no only o gene a e a local neighbo hood bu also o iden i y a speci ic op imal CF da a poin . In addi ion o he s an-
da d gene ic algo i hm, Lash e al. apply local sea ch o non-mu a ed child en so ha he bes solu ion is p ese ed o he
nex gene a ion [24]. Sha ma e al. p opose ano he app oach called Coun e ac ual Explana ions o Robus ness, T ans-
pa ency, In e p e abili y, and Fai ness o A i icial In elligence (CERTIFAI) whe e a gene ic algo i hm based on na u al selec-
ion, mu a ion, and c osso e appeals o use eedback ( ega ding ea u e mu a ion, ea u e ange speci ica ion, and
enqui ies o a speci ic numbe o explana ions) [39]. Whe eas hese use cons ain s allow o gene a ing ac ionable
human-cen ic explana ions, imposing oo se e e es ic ions may o e educe he sea ch space esul ing in gene a ing null
explana ions. In addi ion, Schleich e al. make use o a comple e sea ch space in hei GeCo amewo k [38]. Thus, he au ho s
p esen a cus omizable gene ic algo i hm enhanced wi h wo op imiza ion echniques o educe memo y cos s and unning
ime. The comp essed d- ep esen a ion o he inpu ea u es educes he memo y s o age equi ed o mu a ion- ela ed cal-
cula ions whe eas he so-called pa ial e alua ion op imizes he e alua ion o he classi ie , as s a ic componen s o he clas-
si ie can be p e-e alua ed using an equi alen sub-model o he same classi ie [38].
Finally, bo h quali a i e and quan i a i e gene a ion me hods a e p ima ily e alua ed wi h au oma ically compu able
me ics (e.g., ideli y, alidi y, p oximi y, o di e si y) [12,17,31]. Un o una ely, empi ical s udies in ol ing human e alua-
ion o assessing he goodness o au oma ed CFs a e sca cely ound in he li e a u e. Baaj and Poli show ha explana ions
based on he use o linguis ic e ms appea a he sa is ac o y and con incing despi e being o e ly epe i i e o a gene al
audience [5]. Wang and Yin s a e ha CFs inc ease unde s anding o use s who ha e su icien domain knowledge bu ail o
calib a e us in he model [47]. Fu he , Lucic e al. demons a e ha CFs help use s unde s and why a model makes la ge
e o s [25]. Olson e al. show ha CFs can be also e ec i e o non-expe use s in he iden i ica ion o lawed agen s [32].In
addi ion, Woodcock e al. s ess ha lay use s us CFs only i he in o ma ion gap in he exis ing domain knowledge
be ween hem and expe use s is no signi ican , speci ically in he heal hca e domain [48]. Ne e heless, unlike ou wo k,
none o he a o emen ioned s udies con as s he ou pu o single-poin -ou pu quan i a i e gene a ion me hods and se -
based quali a i e ones.
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
381
3. Explana ion gene a ion me hods
3.1. No a ion
The me hods p oposed in his s udy add ess a mul i-class classi ica ion p oblem, i.e., lea ning a mapping unc ion
h:X!Y om a da ase X¼x
i
gj
n
i¼1
con aining nlabeled ins ances o a disc e e ou pu a iable (class) Y¼y
j
j
m
j¼1
whe e
mis he numbe o classes. The da ase is cha ac e ized by he se o pnume ical
1
ea u es F¼ k
gj
p
k¼1, which a e mapped
o he co esponding linguis ic a iables. By de ini ion [49], each ea u e is a uple k¼L k;T k
L;U k;G k;M k
DE
;8 k2Fwhe e
L kis he name o he ea u e k;T k
L¼ k
l
no
js
l¼1is he se o linguis ic e ms de ined in he uni e se o discou se U k;G kand
M kbeing syn ac ic and seman ic ules, espec i ely. Le VT¼ST k
L;8 k2Fdeno e he se o all linguis ic e ms.
In ou expe imen s (see Sec ions 4 and 5), we aim o explain (bo h ac ually and coun e ac ually) he ou pu o an FRBCS
[23] which is de ined by he ollowing componen s:
a knowledge base con aining a se o inpu and ou pu a iables and a ule base which ep esen s a se R¼
i
w
i
ðÞ
gj
jRj
i¼1
o
weigh ed uzzy ules o he o m
i
w
i
ðÞ:IFL
1
is
1
1
AND ... L
k
is
k
k
... AND ...
hi
THENyISy
i
, whe e
i
2R;w
i
20;1½is he
ule weigh (i.e., he highe w
i
he mo e ele an
i
),
k
k
2T
k
L
;
k
2F;y
i
2Y;
a uzzy p ocessing s uc u e con aining uzzi ica ion and de uzzi ica ion in e aces as well as a uzzy easoning mecha-
nism. Gi en an inpu ec o x¼x
1
;...;x
p
and a ule
i
2R, i s ac i a ion deg ee a
i
is compu ed as a
i
(x)
¼
l
1
1
x
1
ðÞ...
l
k
k
x
k
ðÞ...
l
p
p
x
p
, being
l
k
k
x
k
ðÞ he membe ship deg ee o he alue x
k
o he linguis ic e m
k
associa ed o ea u e
k
, and is a -no m such as minimum o p oduc .
Any ule
i
can be deno ed as a uple
i
w
i
ðÞ¼AC
i
;cq
i
hiwhe e AC
i
is an an eceden (i.e., a non-emp y se o ea u e- alue
pai s) and cq
i
is a consequen (i.e., a class label).
The ou pu class y
FAC
2Yp edic ed by an FRBCS is said o be he ac ual explana ion class. All he ules om he ule base
ha lead o he p edic ed ou come o m a se o ac ual explana ion ules R
FAC
¼S
j
2R
j
jcq
j
¼y
FAC
, being R
FAC
#R. Simila ly,
all he non-p edic ed classes o m a se o CF classes, wi h a collec ion o he co esponding ules mapped o each o hem:
R
CF
¼S
j
2R
j
jcq
j
¼y
CF
no
;Y
CF
¼y
CF
jy
CF
2Yny
FAC
g.
Gi en an FRBCS s, a da a ins ance x2X, and he classi ica ion ou pu y
FAC
p edic ed by s, each class y
j
2Yis associa ed
wi h a single explana ion o why xis classi ied in he gi en way. Hence, he e exis s only one ac ual explana ion E
FAC
sð,
x,y
FAC
Þ. In addi ion, he e is a non-emp y se o CF explana ions E
CF
sð,x,Y
CF
Þ¼ S
y
CF
2Y
CF
E
CF
sð,x,y
CF
Þ o each non-p edic ed class
y
CF
2Y
CF
.
Th oughou he manusc ip , we assume ha he ou pu is explained in i s en i e y i he co esponding explana ion con-
ains a ac ual explana ion speci ying why he gi en decision is made as well as jYj1 CF explana ions indica ing why all he
al e na i e classi ica ion op ions a e disca ded. The e o e, a ( ull) explana ion o a da a ins ance x2Xis assumed o con ain
one ac ual explana ion and a non-emp y se o CF explana ions: Es
ð,x,YÞ¼E
FAC
s
ð,x,y
FAC
Þ[E
CF
s
ð,x,Y
CF
Þ. Acco dingly, explana-
ion gene a ion me hods aim o p oduce (1) a ac ual explana ion o he es ins ance and (2) he mos ele an CF expla-
na ions o all he CF classes.
3.2. Fac ual explana ion gene a ion
We design he p ocess o explana ion gene a ion o include h ee main s ages ( ex planning, sen ence planning, and su -
ace ex ealiza ion) as in he NLG pipeline p oposed by Rei e and Dale [35]. We selec ed his NLG pipeline because i is by
a he mos commonly used in he scien i ic communi y [14]. I is wo h no ing ha we apply he same NLG pipeline no
ma e i we conside ei he ac ual o CF explana ions:
Tex planning, whe e he in o ma ion o be con eyed in he ex is iden i ied (con en de e mina ion), as well as some
o de and gene al s uc u e o he ex is planned. In he case o CF explana ions, con en de e mina ion elies on ele-
ance es ima ion (as desc ibed in he nex sec ion).
Sen ence planning, which includes g ouping o messages when needed (sen ence agg ega ion) and decisions abou he
wo ds/exp essions o be used ( e e ing exp ession gene a ion and/o lexicaliza ion). This s age is c ucial o a oid epe-
i ions and make he ou pu ex mo e na u al.
1
The use o ca ego ical ea u es is ou o he scope o his wo k.
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
382
Su ace ex ealiza ion, which consis s o gene a ing a syn ac ically, mo phologically, and o hog aphically co ec ex .
This las s age is implemen ed using a pool o empla es dynamically ins an ia ed, popula ed and mixed wi h a Py hon
w appe o he SimpleNLG lib a y [6].
Speci ically, he ac ual explana ion gene a ion p ocess p esupposes he ollowing s eps: ac ual explana ion ule selec-
ion, linguis ic app oxima ion o he ea u e alues used in he an eceden (op ionally), and linguis ic ealiza ion. Fi s ,
he ac ual explana ion ule is selec ed om all he ules whose consequen is he p edic ed class. To do so, we calcula e
he p oduc o he ac i a ion deg ee a
j
o each ule
j
2Rand i s associa ed ule weigh w
j
, s. . a gmax w
j
a
j
, i.e., he ac ual
explana ion ule has he maximum p oduc o he ac i a ion deg ee a
j
and ule weigh w
j
. Second, i he ules a e seman i-
cally g ounded, i.e., i hey use meaning ul s ong uzzy pa i ions (SFP), he ea u e alues in he ac ual explana ion a e
eadily a ailable and mapped o he co esponding linguis ic e ms (e.g., ‘‘IF Colo IS Pale AND S eng h IS S anda d THEN
Bee s yle IS Blanche” whe e Pale and S anda d a e expe -de ined linguis ic e ms). O he wise, i.e., i only local seman ics
a e a ailable (e.g., ‘‘IF Colo IS MF0 AND S eng h IS MF1 THEN Bee -s yle IS Blanche” whe e MF0 and MF1 a e wo membe -
ship unc ions wi h local seman ics), linguis ic app oxima ion is necessa y o gene a e a meaning ul explana ion. No ice ha
he mechanism o linguis ic app oxima ion is also used o quali a i e CF explana ion gene a ion and will be desc ibed in
de ail in he nex sec ion. Finally, once he ele an pieces o in o ma ion a e iden i ied, linguis ic ealiza ion is pe o med.
3.3. Quali a i e coun e ac ual explana ion gene a ion
In his sec ion, we in oduce a new me hod o gene a ing quali a i e CF explana ions (he eina e deno ed as EUC). This
me hod can be ega ded as an ex ension o ou p e iously p oposed me hod (he eina e deno ed as XOR)[43]. The EUC
me hod aims o be mo e sensi i e han XOR o a ia ions in membe ship unc ions. Despi e ce ain me hodological di e -
ences, bo h me hods o m a pipeline con aining he ollowing s eps o be desc ibed in de ail below (see Fig. 1): CF ule ep-
esen a ion, ele ance es ima ion, linguis ic app oxima ion (op ional in e ms o he local/global seman ics a ached o he
FRBCS), and ex ual explana ion gene a ion.
CF ule ep esen a ion. Fi s o all, he es ins ance (as well as all he CF candida es) mus be ep esen ed in a compa ible
o m. Bo h EUC and XOR me hods eason o e he in o ma ion e ie ed om he ule base. Mul iple candida es o m CF se s
which a e labeled in acco dance wi h he selec ed linguis ic e ms o he gi en ea u es. Thus, we ega d CF se s as collec-
ions o da a ins ances co e ed by he ules leading o he desi ed CF class. In his sense, he e exis as many po en ial CFs as
he e a e ules ha lead o he desi ed CF class.
Fo a gi en FRBCS, a es ins ance x2Xcan be ep esen ed as a ec o x¼x
1jV
T
j
¼
l
x
i
ðÞj
jV
T
j
i¼1
hi
o membe ship unc ion
alues o each linguis ic a iable. Simila ly, each CF ule can be ega ded in e ms o he membe ship unc ion alues ha
he linguis ic a iables ake on. The e o e, each CF ule
CF
2R
CF
is ec o ized o e V
T
o compa ibili y pu poses so ha
he collec ion o such ec o ized ules makes up a ule- e m ma ix M
jR
CF
jjV
T
j
whe e he i- h ow co esponds o a CF ule
and he j- h column co esponds o he gi en linguis ic e m
j
2V
T
. Hence, he ule- e m ma ix is popula ed wi h such
membe ship alues as unc ions o a gi en linguis ic e m M
ij
¼
l
x
ij
.
Fig. 1. CF explana ion gene a ion pipeline. The shadowed building blocks in luence he su ace ealiza ion o he ou pu explana ion.
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
383
I is wo h no ing ha he XOR me hod addi ionally bina izes bo h he es ins ance ec o and he ule- e m ma ix, a
he cos o in o ma ion loss because o he es ins ance and ule ec o s being app oxima ed. Ins ead, he EUC me hod ep-
esen s he o iginal in o ma ion wi hou u he app oxima ion. This is claimed o be e cap u e uzzy a iable ambigui y
and a oid po en ial in o ma ion loss.
Rele ance es ima ion. Gi en ec o ep esen a ions o he candida e CF ules, i becomes essen ial o iden i y he CF se
ha is minimally di e en om (and he e o e mos ele an o) he es ins ance. Whe eas XOR calcula es ele ance by min-
imizing he numbe o di e en bi s, EUC ela es each ec o ized CF ule o he es ins ance ec o in a jV
T
j-dimensional
space and measu es CF ele ance as he Euclidean dis ance dbe ween pai s o ec o s x;
CF
i
;16i6jR
CF
j; being
CF
i
¼M
i;
he ec o associa ed o ow iin ma ix M, i.e., he ec o which co esponds o CF ule i.
d
XOR
x;
CF
i
¼P
j
jx
j
j
CFi
j
jV
T
j
20;1
½
;
d
EUC
x;
CF
i
¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
P
j
x
j
j
CF
i
2
20;1
½Þ
.
whe e x
j
and
j
CF
i
a e he j- h elemen s in ec o s xand
CF
i
, espec i ely.
The candida e CF ules a e hen anked in acco dance wi h he gi en dis ance me ic. Subsequen ly, we include he min-
imally dis an (o mos ele an ) CF ules o each CF class in he pool E
CF
o he esul ing CF explana ions o he gi en es
ins ance x. I mul iple CF ules a e equally minimally dis an om x, such ules a e deemed equally explana o y. In his case,
he mos ele an CF is selec ed andomly. Rep esen ing he es ins ance and CF ules in a Euclidean jV
T
j-dimensional space
is hypo hesized o be e cap u e uzzy-speci ic p ope ies o an FRBCS. Fo example, he Euclidean dis ance appea s mo e
sensi i e o changes in membe ship unc ion alues. The numbe o unique alues ha he XOR-based dis ance can ake
on is limi ed by jV
T
j. In consequence, se e al CF ules may esul in ha ing he same ele ance sco e while being dis inc
in he numbe o ea u es o hei labeling. On he con a y, EUC p o ides a mo e lexible and di e se measu e o ele ance
o di e en CF ules and he e o e gi es a be e insigh in o he uzzy sys em’s beha io .
Linguis ic app oxima ion. I he linguis ic e ms a e no based on a SFP and he e o e no seman ically g ounded, he
selec ed CF ule mus be enhanced wi h an addi ional linguis ic laye so ha he ou pu explana ion is meaning ul o he
end use . Once he CF ules a e anked by ele ance and he mos ele an CF is iden i ied, i mus he e o e be linguis ically
app oxima ed. To do so, each uzzy se co esponding o he linguis ic e m o he selec ed CF ule is mapped o he gold
s anda d anno a ions. No e ha his mapping is ac ionable i he
a
-cu is applied o such a uzzy se gi en some h eshold
alue d. To illus a e he p ocess o linguis ic app oxima ion, conside a uzzy se FS cha ac e ized by a apezoidal membe -
ship unc ion and h ee linguis ic e ms (T¼
1
;
2
;
3
g
) which a e candida es o be associa ed wi h FS (see Fig. 2 o de ails).
Gi en some cu -o h eshold alue d
1
, he uzzy se FS can be p ojec ed o an in e al o nume ical alues L¼
d
1
;
d
2
.In
addi ion, each linguis ic e m
i
2Tcan be p ojec ed o an in e al
id1
16i6jTjðÞ. Then, he in e al Lcan be compa ed wi h
he in e als
id1
using he Jacca d Simila i y Index [13]:
8L
a
2V
T
:S
id1
;LðÞ¼
id1
L
id1
[L20;1½;ð1Þ
Fig. 2. Illus a i e example o he linguis ic app oxima ion mechanism.
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
384
whe e
id1
is he nume ical in e al close o he linguis ic e m
i
, and Lis he nume ical in e al associa ed o he selec ed
a
-
cu . As ollows om Fig. 2,S
3d1
;LðÞ>S
2d1
;LðÞ>S
1d1
;LðÞ. Hence, he ea u e
j
cha ac e ized by uzzy se FS is e balized as
‘‘
j
is
3
” in his case.
No e ha he h eshold alue d o he
a
-cu se es as a hype pa ame e . The p e iously p oposed XOR me hod uses
heu is ics o speci y dmanually. Ins ead, bo h quali a i e CF gene a ion me hods now use majo o ing in o de o educe
possible app oxima ion e o . Thus, gi en some small enough s ep, we inspec all he app oxima ed linguis ic e ms o e
he cu -o in e al 0;1½ o each e m in he gi en CF ule and assign a con idence sco e o each e m
i
as ollows:
c
i
ðÞ¼
#
i
1þ
1
s ep
, being #
i
he numbe o imes
i
is he winne .
Fo each ea u e
j
in ol ed in he classi ica ion and conside ed in he ou pu explana ion, we apply majo o ing o iden-
i y which linguis ic e m is co e ed by he wides ange o he inspec ed app oxima ions using he app oxima ion con i-
dence sco e c
i
ðÞas a e e ence, so ha he selec ed linguis ic e m is
j
2V
T
ja gmax c
j
. Conside ing he example in
Fig. 2, le s ep be 0.01. We he e o e pe o m n¼1þ1=0:01 ¼101 linguis ic app oxima ions. Suppose ha he e m unde
conside a ion is mapped o he se o linguis ic e ms as indica ed in Table 1.
App oxima ion con idence sco es a e calcula ed o all he compe ing linguis ic e ms. Since we aim o use he mos e-
quen ly ound e m among all he conside ed h eshold alues, he linguis ic e m ha has he highes sco e (in his case,
3
)
is selec ed o he ou pu explana ion. I is wo h no ing ha in his illus a i e example, he selec ed linguis ic e m is he
same as he one selec ed when conside ing only d
1
. Howe e , in he gene al case hey may be di e en . The e o e, i is ec-
ommended o ollow he majo o ing app oach ins ead o elying only on a single d alue selec ed heu is ically.
As only wo building blocks ( ele ance es ima ion and linguis ic app oxima ion) in luence he ou pu explana ion (see
he shadowed blocks in Fig. 1), XOR and EUC gene a e CFs ollowing one o he h ee scena ios below:
he wo me hods selec he same ule o be he mos ele an , he app oxima ion algo i hm ge s he same seman ically
g ounded linguis ic e ms;
he wo me hods selec wo di e en CF ules (e.g., ‘‘IF
1
IS MF
0
and
2
IS MF
0
THEN y
c
” and ‘‘IF
1
IS MF
1
and
2
IS MF
1
THEN y
c
”) which ne e heless gene a e iden ical CF explana ions due o a la ge enough o e lap be ween he co espond-
ing uzzy se s. This scena io is possible when all he ea u es used in bo h ules a e iden ical and hei non-seman ically
g ounded alues o e lap o a la ge enough ex en ;
he wo me hods selec wo di e en CF ules (e.g., ‘‘IF
1
IS MF
0
and
2
IS MF
0
THEN y
c
” and ‘‘IF
1
IS MF
2
and
3
IS MF
4
THEN y
c
”) whe e ea u e alues a e app oxima ed o di e en linguis ic e ms.
Tex ual explana ion ealiza ion. A he las s age, he selec ed ac ual and CF pieces o in o ma ion a e con e ed o
explana ions in na u al language while applying he NLG pipeline in oduced in he p e ious sec ion. I is wo h no ing ha
he ex and sen ence planning along wi h ex ealiza ion o a ac ual explana ion ollow he s uc u e o he co esponding
winne ule om he ule base. Thus, a ac ual explana ion is assumed o include a subo dina e clause o cause (e.g., ‘‘The
da a ins ance xis o class y
because
1
is
1
and
2
is
2
”), which lis s he ea u es and he co esponding alues o linguis ic
e ms ha in luenced he ac ual decision. On he o he hand, a CF explana ion is e balized in na u al language as a complex
condi ional sen ence ha adop s he s uc u e o he ule, e.g., ‘‘xwould be o class y
c
i
1
we e
2
and
3
we e
4
” o he
gi en CF class y
c
.
Implemen a ion de ails. The XOR and EUC me hods a e implemen ed as open sou ce so wa e in Py hon and a e made
publicly a ailable a a Gi lab eposi o y
2
.
3.4. Quan i a i e coun e ac ual explana ion gene a ion
In his sec ion, we p esen a new me hod o CF explana ion gene a ion which is g ounded in e olu iona y and bio-
inspi ed compu a ion algo i hms o explainable AI [11]. Mo e p ecisely, we ha e implemen ed a Gene ic Algo i hm (he e-
a e deno ed as GEN) which akes as he s a ing poin he gene ic uzzy uning app oach p e iously p oposed by Alonso
e al. [4]. Indeed, he o iginal algo i hm was i s in oduced by Co don and He e a [7] and la e adap ed o explainable
SFP uning in [4].
GEN manages a popula ion Pwi h Nindi iduals which e ol e in ggene a ions. The gi en es ins ance xis used o build-
ing he i s indi idual o he popula ion. Each indi idual is associa ed o a eal-coded ch omosome which is made up o p
Table 1
App oxima ion con idence sco e calcula ion.
Te m dApp oxima ion con idence
1
[0.0, 0.3) 30/101 = 0.297
2
[0.3, 0.5) 20/101 = 0.198
3
[0.5, 1.0] 51/101 = 0.505
2
h ps://gi lab.ci ius.usc.es/ilia.s epin/ c expgen (b anch ‘‘xo _euc_gen”)
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
385
genes, wi h each gene ep esen ing one o he ea u es in F. Since all he ea u es a e nume ical, gene i21;p½encodes he
double alue associa ed o ea u e i. The es o he popula ion is gene a ed andomly. Thus, a andom alue is assigned o
each gene iwi hin i s a ia ion in e al which is de e mined by he nume ical ange associa ed o ea u e i. The pseudocode
o he de eloped algo i hm is as ollows (see he GEN shadowed block in Fig. 1):
1. Ini ialize he gene a ion coun e , g¼0, and e alua e he ini ial popula ion, P
0ðÞ
. E alua ing a popula ion means compu -
ing Fi ness o each indi idual in he popula ion. He e, Fi ness is compu ed as he Euclidean dis ance be ween he da a
ins ance ^
xassocia ed o he cu en ch omosome and he o iginal es ins ance x, i he in e ed ou pu is in ag eemen
wi h he a ge CF class. O he wise, Fi ness equals he maximum dis ance which comes ou om he Euclidean dis ance
be ween he wo ec o s ep esen ing he ex eme alues (min/max) o he a ia ion in e als associa ed o each ea u e.
Hence, he smalle Fi ness, he be e .
2. while g<MaxGene and Fi ness PS opTh es and Nbes 6N epTh es
g:¼gþ1
Selec P
ðgÞ
om P
ðg1Þ
C osso
e P
ðgÞ
Mu a e P
ðgÞ
Eli is selec ion P
ðg1Þ
E
alua e P
ðgÞ
end while
The p ocedu e ends ei he when he maximum numbe o gene a ions (MaxGene ) is eached, o Fi ness is unde he p e-
de ined h eshold (S opTh es), o he numbe o consecu i e gene a ions o which he bes i ness alue emains he same
(Nbes ) is g ea e han he p ede ined h eshold (N epTh es). On he one hand, MaxGene should be de ined empi ically in
e ms o he complexi y o he da ase unde conside a ion. I mus be la ge enough o gua an ee ha GEN con e ges o a
good enough solu ion. On he o he hand, S opTh es and N epTh es a e h eshold alues o speed up he p ocedu e, so ha
he algo i hm s ops be o e MaxGene is eached in case Fi ness is small enough o becomes cons an o a la ge enough num-
be o gene a ions. Fo each gene a ion, he ollowing s eps a e epea ed:
The selec ion o P
gðÞ
om P
g1ðÞ
is made as a de e minis ic ou namen selec ion p ocedu e. Each indi idual in he new
popula ion, P
gðÞ
, is chosen om he p e ious one, P
g1ðÞ
, a e making a ou namen ha in ol es TS indi iduals andomly
selec ed om P
g1ðÞ
. The bes indi idual is selec ed in any ou namen . The selec ion p essu e can be adjus ed by changing
TS 6N. The la ge TS, he smalle he chance o weak indi iduals o be selec ed. Fo example, i TS ¼N, hen all he indi-
iduals in P
gðÞ
a e equal o he bes one in P
g1ðÞ
, wha is unsa is ac o y om he poin o iew o di e si y in he
popula ion.
The BLX
a
c osso e ope a o [10] is applied o P
g
ðÞ
. The pa en s, i.e., he selec ed ch omosomes in he cu en popula-
ion, a e c ossed o e in pai s. Each pai o pa en s, dad ¼d
1
;;d
p
and mom ¼m
1
;;m
p
, is eplaced in he new pop-
ula ion by wo o sp ings, O
d
¼o
d1
;;o
dp
and O
m
¼o
m1
;;o
mp
, whe e o
dj
and o
mj
a e andom alues om he
in e als [min
dj
;max
dj
] and [min
mj
;max
mj
], espec i ely. I
j
=[I
l
j
;I
u
j
] is he a ia ion in e al o gene j. Acco ding o he axon-
omy o he c osso e ope a o p esen ed by [21],
a
¼0:3 is a sui able alue o le ing BLX
a
exploi he na u e o eal
coding as ollows:
min
dj
=maximum I
l
j
;d
j
a
jd
j
m
j
j
max
dj
=minimum d
j
þ
a
jd
j
m
j
j;I
u
j
min
mj
=maximum I
l
j
;m
j
a
jm
j
d
j
j
max
mj
=minimum m
j
þ
a
jm
j
d
j
j;I
u
j
A uni o m mu a ion ope a o is conside ed. The alue o he selec ed gene is changed by ano he one gene a ed andomly
wi hin i s a ia ion in e al.
The eli is selec ion ensu es pe pe ua ing he bes indi idual om he gi en gene a ion o he nex one. I he bes indi-
idual, B
i
in P
g1ðÞ
, is no included in P
gðÞ
, hen he wo s indi idual in P
gðÞ
is eplaced by B
i
.
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
386
Once GEN ends, we ha e iden i ied a new da a ins ance ^
x ha is assumed o minimally change he o iginal es ins ance x
while making he FRBCS in e he desi ed CF ou pu
3
. Then, i is ime o gene a ing he ela ed CF explana ion in na u al lan-
guage. To do so, we once again apply he NLG pipeline desc ibed p e iously. Fi s o all, we compu e he pe cen age o modi-
ica ion Dj¼100 ^
xjxj
Ijassocia ed o each ea u e j o go om x o ^
x. The ex which desc ibes Djis as ollows: xjis [sligh ly]
inc eased jdec eased; whe e inc eased appea s i Dj>0. On he con a y, dec eased is used i Dj<0. In addi ion, he linguis ic
modi ie sligh ly appea s only in case o small modi ica ions, i.e., only i 0:96Dj65, which means he pe cen age o modi ica-
ion is smalle o equal han 5%. No ice ha no hing is said abou ea u e ji Dj<0:9. In his case, we conside he ea u e j o
emain he same assuming ha such a small change (less han 0.9%) does no ha e su icien explana o y powe o he ecip-
ien o he explana ion. This assump ion is made heu is ically in acco dance wi h ou p e ious expe ience wi h designing NLG
sys ems while keeping in mind he limi ed p ocessing capabili y o human beings [28]. As a esul , he gene a ed ex ual expla-
na ions a e sho e and easie o p ocess while e e ing only o ele an changes.
A e wa ds, a he sen ence planning s age, o he sake o simplici y and na u alness, we agg ega e hose pieces o in o -
ma ion associa ed o di e en ea u es which a e a ec ed by he same ype o modi ica ion (e.g., ‘‘
1
and
2
a e sligh ly
inc eased” eplaces o ‘‘
1
is sligh ly inc eased and
2
is sligh ly inc eased”). We also apply lexicaliza ion o each ea u e o
be desc ibed in a ully meaning ul way. The e o e, inc eased and dec eased a e eplaced by mo e meaning ul e ms (e.g.,
s eng h is bigge o colo is da ke ).
Finally, ex ealiza ion is done again using he ollowing empla e and he SimpleNLG lib a y wi h he aim o ensu ing
syn ac ically, mo phologically and o hog aphically co ec inal ex : ‘‘[Ou pu Class Name] would be [CF Class Name] i
[Name o he mos Rele an Fea u e
j
]we e [linguis ic desc ip ion o D
j
] (new da a alue) [AND...]”. No ice ha he new al-
ues o he ea u es associa ed wi h he mos ele an changes a e gi en in b acke s.
Implemen a ion de ails. The GEN me hod is implemen ed as a piece o open sou ce so wa e in Py hon and is made pub-
licly a ailable a a Gi lab eposi o y
4
. I is also in eg a ed wi h he open sou ce so wa e GUAJE
5
which is de o ed o acili a ing
he design o explainable uzzy sys ems [3]. The ollowing GEN pa ame e s a e conside ed when gene a ing he quan i a i e CF
explana ions unde e alua ion in he es o he pape : popula ion leng h (N¼30), ou namen size (TS ¼2), mu a ion p oba-
bili y (mp ob ¼0:1), c osso e p obabili y (cp ob ¼0:8), a-c osso e (a¼0:3), MaxGene = 1000, S opTh es =0,N epTh es = 30.
The in e es ed eade is kindly e e ed o Appendix A o u he de ails abou how such pa ame e s we e selec ed.
4. E alua ion design
In his sec ion, we speci y some o he key ea u es ha subsequen human e alua ion s udies ely upon. Sec ion 4.1
in oduces he da ase and FRBCS whose classi ica ions a e explained. Then, Sec ion 4.2 p esen s a no el me ic o measu -
ing he complexi y o au oma ed explana ions.
4.1. Da ase and uzzy in e ence sys em
The expe imen s ha e been ca ied ou using he BEER da ase
6
. I con ains cha ac e is ics o 400 ins ances o bee each o
which belongs o one o 8 classes (Blanche, Lage , Pilsne , IPA, S ou , Ba leywine, Po e , o Belgian S ong Ale). All da a ins ances
a e desc ibed in e ms o h ee ea u es: colo , s eng h, and bi e ness. The co esponding linguis ic e ms and hei anges o
alues a e displayed in Table 2. I is wo h no ing ha all linguis ic e ms a e commonsense and ully meaning ul because hey
we e p o ided by expe b ewe s.
In ou expe imen s, we gene a e explana ions o an FRBCS associa ed wi h he Fuzzy Uno de ed Rule Induc ion Algo-
i hm (FURIA) [22]. The min–max in e ence mechanism [27] is applied so ha bo h conjunc ion (AND) and implica ion
(THEN) a e implemen ed by he -no m minimum, and he ou pu accumula ion is done by he -cono m maximum. All
membe ship unc ions a e apezoidal. All ule weigh s a e se o he de aul alue o 1. In addi ion, i is necessa y o apply
linguis ic app oxima ion as pa o he explana ion gene a ion pipeline because FURIA ules a e endowed only wi h local
seman ics. I is wo h no ing ha such a linguis ic app oxima ion makes use o meaning ul SFP-based linguis ic e ms as well
as hei combina ions. Thus, explana ions may con ain combina ions o adjacen e ms (e.g., ‘‘Fea u e
1
is Te m
1
o Te m
2
”)
wi h he aim o enhancing u he hei explana o y capaci y. Fig. 3 illus a es he SFP associa ed o colo .
In his wo k, we use he same FRBCS ha was p e iously designed and e alua ed in [43] wi h 10- old c oss- alida ion,
achie ing 95.5% o co ec ly classi ied ins ances and F1-sco e equals 0.954 (see he con usion ma ix in Table 3 o u he
de ails). No ice ha , wi h he aim o a oiding gene a ion o misleading explana ions and mainly because he p esen wo k
ocuses on he in ended human e alua ion, he misclassi ied es ins ances a e excluded om u he analysis in he es o
his manusc ip . Whe eas explaining misclassi ica ion is a challenging p oblem, i alls ou side he scope o his wo k.
3
Due o he well-known andom heu is ic na u e o gene ic algo i hms, hey a oid s acking in a local minimum bu hey can no always gua an ee he
con e gence o he global minimum. Anyway, as shown in Appendix A, GEN succeeds o be e ec i e in he sea ch o ‘‘sub-op imal” solu ions which a e
expec ed o be close enough o he op imal one.
4
h ps://gi lab.ci ius.usc.es/ilia.s epin/ c expgen (b anch ‘‘xo _euc_gen”)
5
h ps://gi lab.ci ius.usc.es/jose.alonso/guaje/
6
The BEER da ase is publicly a ailable a h ps://dx.doi.o g/10.13140/RG.2.2.20313.67680
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
387
4.2. Pe cei ed explana ion complexi y
The use o explana ions in na u al language poses he p oblem o adequa e es ima ion o explana ion complexi y. Fo
example, i emains unclea whe he he use o adjacen linguis ic e ms in an explana ion (e.g., ‘‘...i colo we e pale o
s aw”) inc eases o dec eases unde s andabili y (and he e o e e ec i eness and usabili y) o such an explana ion.
As he s a ing inspi ing poin o ou p oposal o au oma ic calcula ion o explana ion complexi y, we e e o exis ing
eadabili y es s in linguis ics, which es ima e how easily a ex can be ead by he in ended audience. Mo e p ecisely,
he well-known Gunning Fog Index [19] is he weigh ed a e age o he no malized sen ence leng h and he pe cen age o
complex wo ds in he ex . Simila ly, an es ima e o complexi y o a ea u e-based linguis ic explana ion (as pe cei ed by
he end use ) may ely on he explana ion leng h as well as on he numbe o ea u es and linguis ic e ms used in he
explana ion.
Fig. 3. In e p e a ion o SFP-based linguis ic e ms associa ed o Colo .
Table 3
FURIA con usion ma ix. UC s ands o Unclassi ied ins ances.
P edic ed class
Obse ed class BLA LAG PIL IPA STO BAR POR BSA UC
Blanche (BLA) 50
Lage (LAG) 48 1 1
Pilsne (PIL) 1 49
IPA 1 43 5 1
S ou (STO) 50
Ba leywine (BAR) 5 43 1 1
Po e (POR) 1 1 47 1
Belgian S ong Ale (BSA) 1 1 1 47
Table 2
Nume ical in e als associa ed o each SFP-based linguis ic e m.
Fea u e Linguis ic e m Range o alues
Colo Pale [0.0, 3.0]
S aw [3.0, 7.5]
Ambe [7.5, 19.0]
B own [19.0, 29.0]
Black [29.0, 45.0]
Bi e ness Low [7.0, 21.0]
Low-medium [21.0, 32.5]
Medium–high [32.5, 47.5]
High [47.5, 250.0]
S eng h Session [0.035, 0.052]
S anda d [0.052, 0.067]
High [0.067, 0.090]
Ve y high [0.090, 0.136]
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
388
In ligh o he abo e, we o mally de ine he pe cei ed explana ion complexi y (PEC) o an au oma ed explana ion eas
ollows:
PEC eðÞ¼kmin l eðÞ;
ðÞ
þ1kðÞ
1
jFjX
F
e
i¼1
i
jT
i
L
jð2Þ
whe e k20;1½is he weigh egula izing he impac o he explana ion leng h and numbe o ea u es and e ms used in he
explana ion, leðÞis he explana ion leng h in cha ac e s,
is a no maliza ion hype pa ame e o e he explana ion leng h, jFj
is he o al numbe o ea u es in he da ase , F
e
is he numbe o unique ea u es used in he gi en explana ion,
i
is he
numbe o e ms associa ed wi h he i- h ea u e used in he explana ion, jT
i
L
jis he powe o he se o linguis ic e ms
o he i- h ea u e.
In he case o he quali a i e me hods XOR and EUC, he basic linguis ic e ms o ake in o accoun a e hose al eady
desc ibed in Table 2. Howe e , in o de o gua an ee a ai compa ison be ween quan i a i e and quali a i e CF explana ions,
i is necessa y o linguis ically ep esen nume ical ea u e alue changes sugges ed by he quan i a i e me hod GEN. The
se s o linguis ic e ms associa ed o each ea u e by he GEN me hod a e he ollowing:
T
L
Colo ðÞ¼
da ke , sligh ly da ke , ligh e , sligh ly ligh e g.
T
L
Bi e nessðÞ¼
smalle , sligh ly smalle , bigge , sligh ly bigge g.
T
L
S eng h
ðÞ
¼ smalle , sligh ly smalle , bigge , sligh ly bigge g.
To illus a e compu a ion o PEC(e), le us conside he ollowing example: gi en a da a ins ance, k¼0:5 and
¼150, we
ha e h ee al e na i e CF explana ions wi h hei co esponding complexi y sco es.
XOR: ‘‘Bee s yle would be S ou i colo we e black.”
PEC eðÞ¼0:5
46
150
þ0:5
1
3
1
5
¼0:153 þ0:033 ¼0:186
EUC: ‘‘Bee s yle would be S ou i bi e ness we e low o low-medium, colo we e black, and s eng h we e s anda d o
high o e y high.”
PEC eðÞ¼0:5
130
150
þ0:5
1
3
2
4
þ
1
5
þ
3
4
¼0:433 þ0:242 ¼0:675
GEN: ‘‘Bee s yle would be S ou i colo we e bigge (30.501) and s eng h we e smalle (0.078).”
PEC eðÞ¼0:5
90
150
þ0:5
1
3
1
4
þ
1
4
¼0:300 þ0:083 ¼0:383
No ewo hy, i always holds ha PEC eðÞ20;1½.PEC(e) is null only i he explana ion is emp y and he associa ed weigh
k¼1. On he con a y, he highes alue o PEC(e) is ob ained when he explana ion leng h is equal o he no maliza ion
hype pa ame e
o all he da ase ea u es and all he linguis ic e ms a e included in he explana ion. Howe e , bo h
o hese special cases a e o no in e es , as he emp y explana ion has go null explana o y powe whe eas explana ion
including all he possible ca ego ies o ea u es is clea ly misleading.
5. Human e alua ion
The human e alua ion s udy consis ed o wo online ques ionnai es ha allowed us o assess how he me ic PEC is
ela ed o di e en explana ion aspec s. Sec ion 5.1 p esen s he ins umen s and design o he i s ques ionnai e (he e-
ina e e e ed o as Su ey GM because he i ems o a e a e associa ed o he so-called G icean Maxims [15] as we will show
below) as well as he analysis o collec ed da a and he discussion o main esul s. In he ligh o lessons lea ned om his
su ey, we de eloped a subsequen one (he eina e e e ed o as Su ey TS because he ocus is on assessing T us wo hiness
and Sa is ac ion o he gi en explana ions) whose expe imen al design and main disco e ies a e desc ibed in Sec ion 5.2.In
bo h su eys, all he subjec s pa icipa ed olun a ily and anonymously. This esea ch ob ained e hics app o al om he
Uni e si y E hics commi ee.
5.1. Su ey GM: E alua ing CF explana ions in e ms o G icean Maxims
5.1.1. Expe imen al se ings
The i s expe imen was designed as a wi hin-subjec s udy. In o de o pe o m a compa a i e analysis o quali a i e and
quan i a i e CF explana ions, we conside ed only hose es ins ances o which he quali a i e me hods (XOR and EUC) gen-
e a ed dis inc explana ions ( hus a oiding misleading epe i ions).
Since he BEER da ase has 8 classes, gi en a es ins ance we ha e 1 ac ual class and 7 al e na i e CF classes. Because he
FURIA ules we e ained and e alua ed wi h 10- old c oss- alida ion, he 400 da a ins ances in he BEER da ase we e spli
10 imes in o aining se (90%) and es se (10%). As a esul , we buil 10 se s o FURIA ules. They we e used o make p e-
dic ions o all es ins ances in each old (see de ails in Table 4). Then, we il e ed ou unclassi ied and misclassi ied es
ins ances wi h he aim o a oiding he inclusion o oid o misleading explana ions o be e alua ed in he su ey. No ewo -
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
389
su ey is cos ly as i equi es highe cogni i e load and mo e ime om he pa icipan s. The e o e, calcula ing PEC au oma -
ically allows he su ey designe o se up and deploy a sho e ques ionnai e and hus easie o ill (Su ey TS). I is wo h
no ing ha PEC s ongly co ela es wi h se e al explana ion aspec s bu does so in di e en di ec ions, so an FRBCS designe
is ad ised o ca e ully selec he me hod o explana ion gene a ion based on he peculia i ies o he applica ion domain and/
o in ended audience.
All in all, he insigh s om his wo k a e expec ed o ad ance me hods o gene a ion and e alua ion o a ious explana-
ion app oaches. As such, hey a e expec ed o be help ul o designing u u e human e alua ion su eys in he a ea o
explainable AI. Mo eo e , as pa o u u e wo k, we will go deepe wi h selec ing and using CF explana ions wi h he
aim o cus omizing hem o use s ha ing di e en p o iles in di e en applica ion scena ios. Fu he esea ch is he e o e
necessa y: (1) o ex end he p oposed CF explana ion gene a ion me hods beyond nume ical ea u es; (2) o be e assess he
impac o he PEC hype pa ame e s (
and k); and (3) o be e unde s and he connec ion be ween complexi y and us -
wo hiness o au oma ed explana ions. No ice ha , he conclusions de i ed om he cu en s udy a e only applicable o he
a ge popula ion unde conside a ion. As pa o u u e wo k, o he sake o gene aliza ion, we in end o design and ca y
ou o he simila expe imen s wi h a la ge and wide panel o esponden s, including non-expe lay use s. Finally, we plan
o use PEC as one o he c i e ia o op imize when designing explainable mul i-objec i e e olu iona y uzzy sys ems.
Funding
Ilia S epin is an FPI esea che (g an PRE2019-090153). Jose M. Alonso-Mo al is a Ramon y Cajal esea che (g an RYC-
2016–19802). This wo k was suppo ed by he Spanish Minis y o Science and Inno a ion (g an s RTI2018-099646-B-I00,
PID2021-123152OB-C21, and TED2021-130295B-C33) and he Galician Minis y o Cul u e, Educa ion, P o essional T aining
and Uni e si y (g an s ED431F2018/02, ED431G2019/04, and ED431C2022/19). All he g an s we e co- unded by he Eu o-
pean Regional De elopmen Fund (ERDF/FEDER p og am).
Decla a ion o Compe ing In e es
The au ho s decla e ha hey ha e no known compe ing inancial in e es s o pe sonal ela ionships ha could ha e
appea ed o in luence he wo k epo ed in his pape .
Appendix A
In addi ion o he human e alua ion s udy on he au oma ically gene a ed CFs, we pe o med h ee independen expe i-
men s on he gene ic algo i hm hype pa ame e ine- uning. In pa icula , we es ima ed he impac o he ollowing hype -
pa ame e s associa ed o he GEN me hod: (i) he size o he popula ion, (ii) he c osso e p obabili y and he co esponding
alpha alue, and (iii) he mu a ion p obabili y. All he expe imen s we e un o he i e su ey s imuli whe e bo h he p e-
dic ed classes and he CF classes we e known. The expe imen al esul s we e assessed in e ms o he bes achie ed i ness
sco es.
Fig. 4 summa izes he impac o he popula ion size (10, 20, 30, 40, 50). I can be obse ed ha he de aul popula ion size
(30) p o ides good esul s, on a e age, o all he es ins ances unde conside a ion.
Fig. 4. An empi ical assessmen o he impac o he popula ion size in he GEN me hod.
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
396
Fig. 5 shows he esul s o he expe imen on he c osso e p obabili y alues (0.7, 0.8, 0.9), conside ing di e en
a
alues
(0.2, 0.3, 0.4). In sho , he combina ion o he c osso e p obabili y (0.8) and
a
¼0:3 yields he bes esul s o he consid-
e ed CF da a poin s.
Fig. 6 illus a es he impac o he selec ed mu a ion p obabili y alues (0.05, 0.1, 0.15, 0.2). I can be seen ha doubling
he de aul mu a ion p obabili y alue may esul in wo sened pe o mance o he algo i hm.
To sum i up, he analysis ca ied ou allows us o conclude ha he selec ed hype pa ame e alues do no only ag ee
wi h he guidelines ound in he li e a u e (e.g., [21]) bu also p o e o be e ec i e in he gi en expe imen s and can indeed
be ecommended o u u e use. All he de ailed calcula ions as well as addi ional plo s and he sou ce code o eplica ing
his expe imen al analysis can be ound in ou Gi lab eposi o y:h ps://gi lab.ci ius.usc.es/ilia.s epin/ c expgen (b anch
‘‘xo _euc_gen”).
Fig. 5. An empi ical assessmen o he impac o he c osso e hype pa ame e s ( he c osso e p obabili y and he ac osso e ope a o ) in he GEN
me hod.
Fig. 6. An empi ical assessmen o he impac o he mu a ion p obabili y in he GEN me hod.
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
397
Re e ences
[1] A. Abdul, J. Ve meulen, D. Wang, B.Y. Lim, and M. Kankanhalli. T ends and ajec o ies o explainable, accoun able and in elligible sys ems: An HCI
esea ch agenda. In P oceedings o he Con e ence on Human Fac o s in Compu ing Sys ems (CHI), pages 1–18, Mon eal QC, Canada, 2018. Associa ion
o Compu ing Machine y. h ps://doi.o g/10.1145/3173574.3174156.
[2] A. Adadi, M. Be ada, Peeking inside he black-box: A su ey on explainable a i icial in elligence (XAI), IEEE Access 6 (2018) 52138–52160, h ps://doi.
o g/10.1109/ACCESS.2018.2870052.
[3] J.M. Alonso, C. Cas iello, L. Magdalena, C. Menca , Explainable Fuzzy Sys ems - Pa ing he Way om In e p e able Fuzzy Sys ems o Explainable AI
Sys ems, olume 970, Sp inge In e na ional Publishing (2021), h ps://doi.o g/10.1007/978-3-030-71098-9.
[4] J.M. Alonso, O. Co dón, S. Guillaume, and L. Magdalena. Highly in e p e able linguis ic knowledge bases op imiza ion: Gene ic uning e sus solis-
we s. Looking o a good in e p e abili y-accu acy ade-o . In P oceedings o he IEEE In e na ional Con e ence on Fuzzy Sys ems, pages 901–906,
London, UK, 2007. h ps://doi.o g/10.1109/FUZZY.2007.4295485.
[5] I. Baaj and J.-P. Poli. Na u al language gene a ion o explana ions o uzzy in e ence decisions. In P oceedings o he IEEE In e na ional Con e ence on
Fuzzy Sys ems, pages 1–6, New O leans, LA, USA, 2019. h ps://doi.o g/10.1109/FUZZ-IEEE.2019.8858994.
[6] A. Cascalla -Fuen es, A. Ramos-So o, A. Buga ín, Adap ing SimpleNLG o Galician Language, in: In P oceedings o he In e na ional Con e ence on
Na u al Language Gene a ion, Associa ion o Compu a ional Linguis ics (ACL), 2018, h ps://doi.o g/10.18653/ 1/W18-6507.
[7] O. Co dón, F. He e a, A Th ee-S age E olu iona y P ocess o Lea ning Desc ip i e and App oxima e Fuzzy Logic Con olle Knowledge Bases om
Examples, In e na ional Jou nal o App oxima e Reasoning 17 (4) (1997) 369–407, h ps://doi.o g/10.1016/S0888-613X(96)00133-8.
[8] R. Dale, E. Rei e , Compu a ional In e p e a ions o he G icean Maxims in he Gene a ion o Re e ing Exp essions, Cogni i e science 19 (2) (1995)
233–263, h ps://doi.o g/10.1016/0364-0213(95)90018-7.
[9] V. Dignum, Responsible A i icial In elligence: How o De elop and Use AI in a Responsible Way. A i icial In elligence: Founda ions, Theo y, and
Algo i hms, Sp inge , Cham, 2019, h ps://doi.o g/10.1007/978-3-030-30371-6.
[10] L.J. Eshelman and J.D. Scha e . Real-Coded Gene ic Algo i hms and In e al-Schema a. In L. Da ell Whi ley, edi o , Founda ions o Gene ic Algo i hms,
olume 2 o Founda ions o Gene ic Algo i hms, pages 187–202. Else ie , 1993. h ps://doi.o g/10.1016/B978-0-08-094832-4.50018-0.
[11] A. Fe nandez, F. He e a, O. Co don, M. Jose del Jesus, F. Ma celloni, E olu iona y Fuzzy Sys ems o Explainable A i icial In elligence: Why, When,
Wha o , and Whe e o?, IEEE Compu a ional In elligence Magazine 14 (1) (2019) 69–81, h ps://doi.o g/10.1109/MCI.2018.2881645.
[12] R.R. Fe nández, I.M. de Diego, V. Aceña, A. Fe nández-Isabel, J.M. Mogue za, Random o es explainabili y using coun e ac ual se s, In o ma ion Fusion
63 (2020) 196–207, h ps://doi.o g/10.1016/j.in us.2020.07.001.
[13] S. Fle che , M.Z. Islam, Compa ing se s o pa e ns wi h he Jacca d index, Aus alasian Jou nal o In o ma ion Sys ems 22 (2018), h ps://doi.o g/
10.3127/ajis. 22i0.1538.
[14] A. Ga , E. K ahme , Su ey o he S a e o he A in Na u al Language Gene a ion: Co e asks, applica ions and e alua ion, Jou nal o A i icial
In elligence Resea ch 61 (2018) 65–170, h ps://doi.o g/10.1613/jai .5477.
[15] H.P. G ice, Logic and Con e sa ion, in: P. Cole, J.L. Mo gan (Eds.), Syn ax and Seman ics: Speech Ac s, Academic P ess, 1975, pp. 41–58, h ps://doi.o g/
10.1163/9789004368811_003.
[16] R. Guido i, Coun e ac ual explana ions and how o ind hem: li e a u e e iew and benchma king, Da a Mining and Knowledge Disco e y (2022) 1–
55, h ps://doi.o g/10.1007/s10618-022-00831-6.
[17] R. Guido i, A. Mon eale, F. Gianno i, D. Ped eschi, S. Ruggie i, F. Tu ini, Fac ual and Coun e ac ual Explana ions o Black Box Decision Making, IEEE
In elligen Sys ems 34 (6) (2019) 14–23, h ps://doi.o g/10.1109/MIS.2019.2957223.
[18] D. Gunning, E. Vo m, J.Y. Wang, M. Tu ek, DARPA’s explainable AI (XAI) p og am: A e ospec i e, Applied AI Le e s 2 (4) (2021), h ps://doi.o g/
10.1002/ail2.61 e61.
[19] R. Gunning, Technique o clea w i ing, McG aw-Hill, 1968.
[20] C. He ley and W. Pie e s. I You We e A acked, You’d Be So y: Coun e ac uals as Secu i y A gumen s. In P oceedings o he 2015 New Secu i y
Pa adigms Wo kshop, NSPW ’15, pages 112–123, New Yo k, NY, USA, 2015. Associa ion o Compu ing Machine y. h ps://doi.o g/10.1145/2841113.
2841122.
[21] F. He e a, M. Lozano, A.M. Sánchez, A Taxonomy o he C osso e Ope a o o Real-Coded Gene ic algo i hms: An Expe imen al S udy, In e na ional
Jou nal o In elligen Sys ems 18 (3) (2003) 309–338, h ps://doi.o g/10.1002/in .10091.
[22] J. Hühn, E. Hülle meie , FURIA: an algo i hm o uno de ed uzzy ule induc ion, Da a Mining and Knowledge Disco e y 19 (3) (2009) 293–319, h ps://
doi.o g/10.1007/s10618-009-0131-8.
[23] H. Ishibuchi, T. Nakashima, M. Nii, Classi ica ion and modeling wi h linguis ic in o ma ion g anules: Ad anced app oaches o linguis ic Da a Mining,
Sp inge Science & Business Media (2004), h ps://doi.o g/10.1007/b138232.
[24] M. Lash, Q. Lin, N. S ee , J. Robinson, and J. Ohlmann. Gene alized In e se Classi ica ion. In P oceedings o he In e na ional Con e ence on Da a Mining
(SDM), pages 162–170. Socie y o Indus ial and Applied Ma hema ics, 2017. h ps://doi.o g/10.1137/1.9781611974973.19.
[25] A. Lucic, H. Haned, and M. de Rijke. Why Does My Model Fail? Con as i e Local Explana ions o Re ail Fo ecas ing. In P oceedings o he 2020
Con e ence on Fai ness, Accoun abili y, and T anspa ency, FAT* ’20, pages 90–98, Ba celona, Spain, 2020. Associa ion o Compu ing Machine y.
h ps://doi.o g/10.1145/3351095.3372824.
[26] N. Maa oo , A. Mo eno, A. Valls, M. Jab eel, M. Szelag, A Compa a i e S udy o Two Rule-Based Explana ion Me hods o Diabe ic Re inopa hy Risk
Assessmen , Applied Sciences 12 (7) (2022) 1–18, h ps://doi.o g/10.3390/app12073358.
[27] E.H. Mamdani, Applica ion o Fuzzy Logic o App oxima e Reasoning Using Linguis ic Sys ems, IEEE T ansac ions on Compu e s 26 (12) (1977) 1182–
1191, h ps://doi.o g/10.1109/TC.1977.1674779.
[28] G.A. Mille , The magical numbe se en, plus o minus wo: Some limi s on ou capaci y o p ocessing in o ma ion, Psychological Re iew 63 (2) (1956)
81–97, h ps://doi.o g/10.1037/0033-295x.101.2.343.
[29] T. Mille , Explana ion in a i icial in elligence: Insigh s om he social sciences, A i icial In elligence 267 (2019) 1–38, h ps://doi.o g/10.1016/j.
a in .2018.07.007.
[30] J. Moo e, N. Hamme la, and C. Wa kins. Explaining deep lea ning models wi h cons ained ad e sa ial examples. In P oceedings o he Paci ic Rim
In e na ional Con e ence on A i icial In elligence (PRICAI), pages 43–56. Sp inge , 2019. h ps://doi.o g/10.1007/978-3-030-29908-8_4.
[31] R.K. Mo hilal, A. Sha ma, and C. Tan. Explaining Machine Lea ning Classi ie s h ough Di e se Coun e ac ual Explana ions. In P oceedings o he 2020
Con e ence on Fai ness, Accoun abili y, and T anspa ency, FAT* ’20, pages 607–617, Ba celona, Spain, 2020. Associa ion o Compu ing Machine y.
h ps://doi.o g/10.1145/3351095.3372850.
[32] M.L. Olson, R. Khanna, L. Neal, F. Li, W.-K. Wong, Coun e ac ual s a e explana ions o ein o cemen lea ning agen s ia gene a i e deep lea ning,
A i icial In elligence 295 (2021) 1–29, h ps://doi.o g/10.1016/j.a in .2021.103455.
[33] Pa liamen and Council o he Eu opean Union. Gene al Da a P o ec ion Regula ion (GDPR), 2016. URL:h p://da a.eu opa.eu/eli/ eg/2016/679/oj.
[34] Pa liamen and Council o he Eu opean Union. A Eu opean App oach o A i icial In elligence, 2022. URL:h ps://digi al-s a egy.ec.eu opa.eu/en/
policies/eu opean-app oach-a i icial-in elligence.
[35] E. Rei e , R. Dale, Building Na u al Language Gene a ion Sys ems, in: S udies in Na u al Language P ocessing, Camb idge Uni e si y P ess, 2000,
h ps://doi.o g/10.1017/CBO9780511519857.
[36] M.T. Ribei o, S. Singh, and C. Gues in. ”Why should I us you?”: Explaining he p edic ions o any classi ie . In P oceedings o he In e na ional
Con e ence on Knowledge Disco e y and Da a Mining (SIGKDD), pages 1135–1144, San F ancisco, Cali o nia, USA, 2016. Associa ion o Compu ing
Machine y. h ps://doi.o g/10.1145/2939672.2939778.
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
398
[37] C. Rudin, S op explaining black box machine lea ning models o high s akes decisions and use in e p e able models ins ead, Na u e Machine
In elligence 1 (5) (2019) 206–215, h ps://doi.o g/10.1038/s42256-019-0048-x.
[38] M. Schleich, Z. Geng, Y. Zhang, and D. Suciu. GeCo: Quali y Coun e ac ual Explana ions in Real Time. In P oceedings o he Ve y La ge Da a Bases
(VLDB) Endowmen , olume 14(9), pages 1681–1693, 2021. h ps://doi.o g/10.14778/3461535.3461555.
[39] S. Sha ma, J. Hende son, J. Ghosh. CERTIFAI, A Common F amewo k o P o ide Explana ions and Analyse he Fai ness and Robus ness o Black-
Box Models, Associa ion o Compu ing Machine y, 2020, pp. 166–172, h ps://doi.o g/10.1145/3375627.3375812.
[40] K. Sokol and P. Flach. One Explana ion Does No Fi All: The P omise o In e ac i e Explana ions o Machine Lea ning T anspa ency. KI – Küns liche
In elligenz, 2020. h ps://doi.o g/10.1007/s13218-020-00637-y.
[41] I. S epin, J.M. Alonso, A. Ca ala, and M. Pe ei a-Fa iña. Gene a ion and e alua ion o ac ual and coun e ac ual explana ions o decision ees and uzzy
ule-based classi ie s. In P oceedings o he IEEE Wo ld Cong ess on Compu a ional In elligence (WCCI), Glasgow, UK, 2020. h ps://doi.o g/10.1109/
FUZZ48607.2020.9177629.
[42] I. S epin, J.M. Alonso, A. Ca ala, M. Pe ei a-Fa iña, A Su ey o Con as i e and Coun e ac ual Explana ion Gene a ion Me hods o Explainable A i icial
In elligence, IEEE Access 9 (2021) 11974–12001, h ps://doi.o g/10.1109/ACCESS.2021.3051315.
[43] I. S epin, A. Ca ala, M. Pe ei a-Fa iña, J.M. Alonso, Fac ual and Coun e ac ual Explana ion o Fuzzy In o ma ion G anules, in: In e p e able A i icial
In elligence: A Pe spec i e o G anula Compu ing, Sp inge In e na ional Publishing, 2021, pp. 153–185, h ps://doi.o g/10.1007/978-3-030-64949-
4_6.
[44] R. Sukke d, R. Simmons, D. Ga lan, Towa d Explainable Mul i-Objec i e P obabilis ic Planning, in: IEEE/ACM 4 h In e na ional Wo kshop on So wa e
Enginee ing o Sma Cybe -Physical Sys ems (SEsCPS), Go henbu g, Sweden, 2018, pp. 19–25.
[45] S. Ve ma, J. Dicke son, and K. Hines. Coun e ac ual explana ions o machine lea ning: A e iew. In P oceedings o he Machine Lea ning:
Re ospec i es, Su eys and me a-Analyses (ML-RSA) Wo kshop a he Con e ence on Neu al In o ma ion P ocessing Sys ems (Neu IPS), 2020.
[46] S. Wach e , B. Mi els ad , C. Russell, Coun e ac ual Explana ions Wi hou Opening he Black Box: Au oma ed Decisions and he GDPR, Ha a d
Jou nal o Law & Technology 31 (2) (2018) 841–887, h ps://doi.o g/10.2139/ss n.3063289.
[47] X. Wang, M. Yin, A e Explana ions Help ul? A Compa a i e S udy o he E ec s o Explana ions in AI-Assis ed Decision-Making, in: 26 h In e na ional
Con e ence on In elligen Use In e aces, IUI ’21, Associa ion o Compu ing Machine y, 2021, pp. 318–328, h ps://doi.o g/10.1145/
3397481.3450650.
[48] C. Woodcock, B. Mi els ad , D. Busb idge, G. Blank, e al, The Impac o Explana ions on Laype son T us in A i icial In elligence-D i en Symp om
Checke Apps: Expe imen al S udy, Jou nal o Medical In e ne Resea ch 23 (11) (2021), h ps://doi.o g/10.2196/29386 e29386.
[49] L.A. Zadeh, The concep o a linguis ic a iable and i s applica ion o app oxima e easoning, In o ma ion Sciences 8 (3) (1975) 199–249, h ps://doi.
o g/10.1016/0020-0255(75)90036-5.
I. S epin, J.M. Alonso-Mo al, A. Ca ala e al. In o ma ion Sciences 618 (2022) 379–399
399
CORRECTED PROOF
A gumen & Compu a ion -1 (2023) 1–59 1
DOI 10.3233/AAC-220011
IOS P ess
In o ma ion-seeking dialogue o explainable
a i icial in elligence: Modelling and
analy ics
Ilia S epin a,c,∗, Ka a zyna Budzynska b, Alejand o Ca ala a,c, Ma ín Pe ei a-Fa iña dand
Jose M. Alonso-Mo al a,c
aCen o Singula de In es igación en Tecnoloxías In elixen es (CiTIUS), Uni e sidade de San iago de
Compos ela, Rúa de Jena o de la Fuen e Domínguez s/n, 15782 San iago de Compos ela, A Co uña,
Spain
E-mails: [email p o ec ed],alejand [email p o ec ed],[email p o ec ed]
bLabo a o y o The New E hos, Wa saw Uni e si y o Technology, plac Poli echniki 1, 00-661, Wa saw,
Poland
E-mail: ka a zyna.b[email p o ec ed]
cDepa amen o de Elec ónica e Compu ación, Uni e sidade de San iago de Compos ela, Rúa Lope
Gómez de Ma zoa, s/n, 15782 San iago de Compos ela, A Co uña, Spain
dDepa amen o de Filoso ía e An opoloxía, Uni e sidade de San iago de Compos ela, Plaza de
Maza elos s/n, 15705 San iago de Compos ela, A Co uña, Spain
E-mail: ma in.pe ei [email p o ec ed]
Abs ac . Explainable a i icial in elligence has become a i ally impo an esea ch ield aiming, among o he asks, o jus i y
p edic ions made by in elligen classi ie s au oma ically lea ned om da a. Impo an ly, e iciency o au oma ed explana ions
may be unde mined i he end use does no ha e su icien domain knowledge o lacks in o ma ion abou he da a used o
aining. To add ess he issue o e ec i e explana ion communica ion, we p opose a no el in o ma ion-seeking explana o y
dialogue game ollowing he mos ecen equi emen s o au oma ically gene a ed explana ions. Fu he , we gene alise ou
dialogue model in o m o an explana o y dialogue g amma which makes i applicable o in e p e able ule-based classi ie s
ha a e enhanced wi h he capabili y o p o ide ex ual explana ions. Finally, we ca y ou an explo a o y use s udy o alida e
he co esponding dialogue p o ocol and analyse he expe imen al esul s using insigh s om p ocess mining and a gumen
analy ics. A high numbe o eques s o al e na i e explana ions es i ies he need o ensu ing di e si y in he con ex o
au oma ed explana ions.
Keywo ds: Explainable A i icial In elligence, in o ma ion-seeking dialogue game, explana ion locu ions, coun e ac ual
explana ion, p ocess mining analy ics, a gumen analy ics
1. In oduc ion
Explainabili y in he con ex o A i icial In elligence (AI) has long a ac ed a en ion o esea che s
om compu e science [57] and a gumen a ion [21]. The i s explana ion gene a ion me hods u ned up
*Co esponding au ho . Tel.: +34 8818 16394; E-mail: [email p o ec ed].
1946-2166 © 2023 – The au ho s. Published by IOS P ess. This is an Open Access a icle dis ibu ed unde he e ms o he
C ea i e Commons A ibu ion-NonComme cial License (CC BY-NC 4.0).
CORRECTED PROOF
2I. S epin e al. / In o ma ion-seeking dialogue o XAI
in he 1980s along wi h he so-called Expe Sys ems [74]. Mo e p ecisely, he i s explaine s add essed
he challenge o explaining he ou pu o expe sys ems and logic p og ams [7], which e en ually led
o he eme gence o he esea ch ield ha we now call Compu a ional A gumen a ion. Recen yea s
ha e wi nessed a new boos o in e es in de eloping eXplainable AI (XAI), as no el machine lea ning
(ML) algo i hms p oduce highly accu a e ye o en imes poo ly explainable p edic ions [1]. As de ined
a p esen , XAI aims o (1) gene a e explainable models p ese ing a high le el o accu acy and (2)
enable he end use , e.g., a clien o a bank o a pa ien o a hospi al, wi h he oppo uni y o unde s and,
us , and manage he gi en AI-based sys ems [2,29] (e.g., que ying a bank loan managemen sys em
o iden i y easons o he loan applica ion being ejec ed o a hospi al in o ma ion sys em o ecei e
ea men - ela ed ecommenda ions).
The obscu e na u e o he unde lying easoning o he s a e-o - he-a p edic i e algo i hms has gi en
way o he so-called “ igh o explana ion” [80]. The co esponding legal egula ions a e being inc eas-
ingly adop ed wo ldwide [87]. Fo example, he Eu opean Union (EU)’s Gene al Da a P o ec ion Reg-
ula ion (GDPR) acknowledges he igh o he use “no o be subjec o a decision e alua ing pe sonal
aspec s ela ing o him o he which is based solely on au oma ed p ocessing and which p oduces ad-
e se legal e ec s conce ning, o signi ican ly a ec s, him o he ” [51]. In addi ion, cu en EU’s le-
gal egula ions in, o example, he inancial domain equi e ha algo i hmic anspa ency be p o ided
o au oma ic ading echniques (see he Di ec i e 2014/65/EU on Ma ke s in Financial Ins umen s,
commonly known as MiFID II [52] o de ails). Being a con o e sial opic o p ima y impo ance o
nume ous s akeholde s, i s ju idical basis is cons an ly upda ed. Thus, he newly p oposed EU’s AI Ac
(AIA) [53] es ablishes a axonomy o AI-based sys ems and equi es ha high- isk AI applica ions o e
explana ions o hei decisions o ecommenda ions o hei end use s.
In o de o mi iga e algo i hmic anspa ency issues o he s a e-o - he-a AI algo i hms, a use o
in e p e able models is ad ised [59]. In e p e able ule-based models (such as, e.g., decision ees (DT)
o decision ules) a e known o p o ide use - iendly explana ions [47]. Rema kably, DTs can be used
as pa o mo e complex model-agnos ic explaine s ha a e able o jus i y p edic ions o o he a bi a y
classi ie s i hey a e, o example, ained on a local syn he ically gene a ed neighbou hood a ound he
es ins ance [28]. Despi e he ac ha only ew XAI amewo ks o e explana ions in na u al language
[12], DTs ha e also been shown o be a powe ul ool o communica ing ex ual explana ions o end
use s, e.g., by engaging he use in an explana o y dialogue [70,79].
Explana ions a e claimed o ha e o necessa ily be embedded in a dialogical in e ac ion so ha he end
use is able o challenge he aspec s o an explana ion ha ha e no been unde s ood [63]. Fo illus a i e
pu poses, le us conside a bee s yle classi ica ion p oblem (see Table 1 o de ails). Gi en a numbe o
p ede ined classes (i.e., bee s yles) and an ins ance o bee , he classi ie (Sys em) makes a p edic ion
on wha bee s yle he es ins ance is (mo e m1). The end use (Use ) engages in a dialogue wi h he
Sys em o inqui e an explana ion o he gi en p edic ion (m2). Then, he Sys em p o ides he Use
wi h a ac ual explana ion based on he mos dis inc i e ea u es, pe inen o he es ins ance, ha led
o he gi en p edic ion (m3). Then, gi en he ac ual explana ion, he Use may wan o cla i y speci ic
ea u es (e.g., bi e ness) ha he o e ed explana ion con ains (m4). In esponse, he Sys em o e s a
de ini ion o he eques ed ea u e (m5). Al e na i ely, he Use may inqui e he Sys em o explain why
he gi en es ins ance is no p edic ed o be o ano he class (m6,m10). The pieces o such coun e ac ual
explana ions o e ed in esponse sugges minimal changes o he inpu so ha i s classi ica ion changes
in he desi ed manne (m7,m11). Gi en he ini ial explana ion, he Use may wan o speci y a mo e
p ecise desc ip ion o he gi en ea u e (m8). Then, he Sys em may p esen de ailed in o ma ion o e he
co esponding nume ical in e al (subjec o a ailabili y) o e which he ea u e is de ined (m9). Fu he ,
CORRECTED PROOF
I. S epin e al. / In o ma ion-seeking dialogue o XAI 3
Table 1
An illus a i e explana o y dialogue
Mo e Agen U e ance
m1Sys em The p esen bee is Blanche.
m2Use Could you explain why you hink so?
m3Sys em I is Blanche because colou is pale and bi e ness is low.
m4Use Wha do you mean by bi e ness?
m5Sys em Bi e ness is a ha sh, ac id as e ha is one o he ou basic as e sensa ions.
m6Use Bu is he bee no s ou ?
m7Sys em I would be S ou i colou we e black and bi e ness we e high.
m8Use Could you speci y how high bi e ness is de ined?
m9Sys em Su e! I de ine bi e ness as high because i lies in he ange om 47 o 250.
m10 Use Bu why is i no Po e ?
m11 Sys em I would be Po e i colou we e b own.
m12 Use I am no qui e sa is ied wi h you explana ion. Could you o e me ano he one?
m13 Sys em Su e! I would be o class Po e i colou we e b own and s eng h we e session.
m14 Use Ok, now I us you p edic ion.
m15 Sys em Thank you o you us in me. Bye!
he Use may disag ee wi h he explana ion o e ed and a gue o e i (m12). The Sys em should hen
o e an al e na i e explana ion ha would sa is y he Use ’s needs (m13). When he Use is su icien ly
in o med abou he easons ha led o he gi en p edic ion, he o she makes an in o med decision on
whe he he Sys em’s p edic ion should be us ed o no (m14). The explana o y dialogue ends wi h he
Sys em’s a ewell locu ion (m15).
As ollows om Table 1, we conside wo ypes o explana ions: ac ual and coun e ac ual. Assuming
knowledge o he ea u e space, ac ual explana ions (illus a ed wi h mo e m3in Table 1) aim o explain
he gi en classi ie ’s p edic ion in e ms o he mos ele an ea u e alues ha led o ha p edic ion. On
he con a y, coun e ac uals (illus a ed wi h mo es m7,m11,andm13 in Table 1) a e pos -hoc example-
based explana ions ha sugges a minimal change in ea u e alues o hose o he gi en da a ins ance so
ha he sys em’s p edic ion changes as desi ed [71].
This pape in oduces an explana o y dialogue game o communica ing ac ual and coun e ac ual
explana ions o in e p e able ule-based classi ie s. We assume ha he classi ie is associa ed wi h an
explaine ha is capable o p o iding ex ual ( ule-based) explana ions. Based on he dialogue ypology
p oposed by Wal on and K abbe [82], we model he in o ma ion-seeking ype o explana o y dialogue
equipping i wi h a speci ic collec ion o locu ions ailo ed o he a o emen ioned ypes o explana ion
ha he use may ask he sys em. As a s a ing poin , we conside he ypology o dialogue mo es p o-
posed by Budzynska e al. [9]. In ou wo k, we ex end his ypology o dialogue mo es wi h a epe oi e
o locu ions allowing o communica ion o ac ual and coun e ac ual explana ions o enable he end
use o in e ac i ely explo e he explana ion space. Then, we p opose a con ex - ee dialogue g amma
o gene alise he o mal s uc u e o he esul ing dialogue model. Despi e an empi ically shown s ong
need in bo h ac ual and coun e ac ual explana ions [41] and a leas a hund ed o coun e ac ual expla-
na ion gene a ion me hods p oposed by now in he con ex o XAI, less han a hi d o hese me hods
a e e alua ed in use s udies [37]. To add ess his issue, we subsequen ly pe o m a pilo use s udy o
e alua e he p oposed dialogue model. Mo eo e , we analyse he collec ed dialogue ansc ip s ea ing
ins ances o explana o y dialogue as p ocesses using he s a e-o - he-a echniques om p ocess mining
and a gumen analy ics [43].
CORRECTED PROOF
4I. S epin e al. / In o ma ion-seeking dialogue o XAI
As a esul , we b idge he gap be ween ML p ac i ione s and he a gumen a ion communi y by making
he ollowing con ibu ions:
•we model in o ma ion-seeking explana o y dialogue based on he undamen al no ions om he
a gumen a ion heo y and apply he dialogue model in he con ex o XAI;
•we p opose a se o o iginal dialogue locu ion ypes ha a e ound speci ically sui able o e ec i e
communica ion o ac ual and coun e ac ual explana ions;
•we demons a e he explana o y u ili y o he p oposed dialogue p o ocol ia a human e alua ion
s udy based on h ee use cases o an in e p e able ule-based classi ie lea ing open-sou ce imple-
men a ions o he dialogue game and he human e alua ion oolki a ailable o public use;
•we sugges o mal means o ex ending he p oposed p o ocol o make i applicable o modelling
dialogic human-machine in e ac ion o classi ica ion asks in o he applica ions.
The es o he manusc ip is s uc u ed as ollows. Sec ion 2in oduces he classi ica ion p oblem
o mally and ou lines he common p ope ies o explana ions claimed o be essen ial o explaining so-
lu ions o such a p oblem. In addi ion, we subsequen ly discuss possible disc epancy be ween au oma i-
cally gene a ed explana ions and use -p e e ed explana ions. Sec ion 3de ines an explana o y dialogue
game as an in e ace be ween an explana ion gene a ion module and he end use . Sec ion 4in oduces
essen ial p ocess mining concep s and shows how we apply hem o explana o y dialogue analysis. Sec-
ion 5p esen s he expe imen al se ings o he human e alua ion s udy ca ied ou o assess he u ili y o
he p oposed dialogue p o ocol. Sec ion 6 epo s he expe imen al esul s ob ained om he human e al-
ua ion s udy. Sec ion 7discusses he dialogue model alida ion esul s. Sec ion 8p esen s an o e iew o
ela ed wo k ega ding o mal explana o y dialogue models as well as ecen a gumen a ion-based ech-
niques o explana o y dialogue modelling. Finally, we ou line p ospec i e di ec ions o u u e wo k
and conclude in Sec ion 9.
2. P elimina ies
In his sec ion, we i s ou line a de ini ion o he classi ica ion p oblem and assump ions abou he
na u e o classi ie s and explaine s ha we a e d i en by (see Sec ion 2.1 o de ails). Then, we o mally
de ine essen ial explana ion- ela ed concep s ha we u ilise h oughou he manusc ip in Sec ion 2.2.
Finally, we d aw eade ’s a en ion o possible disc epancies be ween he use -p e e ed explana ions
and hose o e ed o him o he by he explaine in Sec ion 2.3.
2.1. The classi ica ion p oblem
As ou lined in Sec ion 1, we ocus on communica ing o he end use au oma ed explana ions o
he ou pu o an in e p e able ule-based ML classi ie . Figu e 1depic s a gene al a chi ec u e o he
modelled explana ion communica ion p ocess. The Sys em is assumed o include, a leas , he ollowing
co e componen s: an in e p e able ule-based classi ie , an explaine , a knowledge base, and a da ase ha
he classi ie is ained on. The Use s a s he communica ion p ocess by sending a classi ica ion eques
o a speci ic es ins ance o he Sys em in o m o he es ins ance’s cha ac e is ics (i.e., ea u es). The
classi ie is p e ained on a gi en da ase X={xi}|n
i=1con aining nlabelled ins ances o lea n a mapping
unc ion c:X−→ Ywhe e Y={yj}|m
j=1is a disc e e ou pu a iable (class), mbeing he numbe o
classes.
CORRECTED PROOF
I. S epin e al. / In o ma ion-seeking dialogue o XAI 5
Fig. 1. A schema o he modelled sys em-use explana ion communica ion p ocess. This pape ocuses on designing an ex-
plana o y dialogue game o communica ion o ac ual and coun e ac ual explana ions o in e p e able ule-based classi ie s
( he shaded block).
In his wo k, we assume knowledge o he ea u e space: he da ase is said o con ain linea ly scaled
nume ical ea u es. In addi ion, all he nume ical ea u e alues a e said o be mapped o he co e-
sponding ea u e-dependen linguis ic a iable [86]. The e o e, each da a ins ance xi∈X=Fi,y
iis
associa ed o class yi∈Yand de ined o e he se o p3- uple ea u es Fi={ k, k, k}|p
k=1whe e
each ea u e kis assigned o he co esponding nume ical alue kand linguis ic e m k(e.g., age,
20, young). The alues o he linguis ic a iables (i.e., he so-called linguis ic e ms) may be de ined
by an expe . In his case, hey a e mapped o expe knowledge-based nume ical in e als co e ing all
he alues o he co esponding ea u e. O he wise, he linguis ic a iable is assigned o a se o ex ual
alues and mapped o equal-size nume ical in e als. In his espec , he se o ex ual alues ha he
linguis ic a iable can ake on is o a bi a y ca dinali y.
The classi ie p edic s he class label ˆy o he gi en es ins ance x es =F es ,y
es on he basis o he
lea ned mapping unc ion c. The es ins ance classi ica ion is p edic ed co ec ly i he p edic ed class
label and he ac ual es ins ance class label a e he same (i.e., ˆy=y es ). O he wise, he p edic ed class
is deemed w ong (i.e., ˆy= y es ). Al oge he , he in e p e able ule-based classi ie and he explaine a e
said o o m an explainable classi ie . Once he classi ie ou pu s a p edic ion, he associa ed explaine
a emp s o gene a e an explana ion in na u al language o ha p edic ion. Upon eques , he explana ion
is passed o he Use ia he explana o y dialogue game, which se es as a communica ion channel
be ween he explainable classi ie and he Use . Du ing hei in e cou se, he Use is assumed o be
able o submi u he explana ion- ela ed eques s and ecei e esponses p ocessed by he dialogue
game module whe eas he dialogue game module can que y he explaine o u he explana ion- ela ed
in o ma ion.
2.2. Explana ion o he classi ica ion
The upsu ging need o explaining a classi ie ’s ou pu is aising in e es in he me e na u e o he ex-
plana ion. Fo ins ance, social sciences es i y ha explana ions a e expec ed o be con as i e, selec ed,
CORRECTED PROOF
6I. S epin e al. / In o ma ion-seeking dialogue o XAI
and social [45]. Fi s , he p ope y o con as i eness implies es ablishing a ela ion no only be ween
he cause and e ec o he phenomenon unde conside a ion bu also ano he ela ion be ween he cause
and a gi en non-obse ed e ec (i.e., ano he al e na i e e ec ). Second, explana ions a e as well a gued
o be selec ed, i.e., only he mos ele an causes should make pa o a speci ic explana ion. Thi d, ex-
plana ions a e claimed o be social, i.e., hey a e a p oduc o in e ac ion be ween he explaine and he
explainee.
Con as i eness plays an impo an ole when explaining a solu ion o he classi ica ion p oblem, as
di e en classes a e opposed o he o he s on he basis o dis inc i e ea u e alues. Fu he , con as i e-
ness is inhe en o coun e ac ual (CF) explana ions (o coun e ac uals, o sho ). In he con ex o XAI,
coun e ac uals sugges minimal changes in ea u e- alue pai s o a di e en ou come o be ob ained
[71]. CFs a e said o be pos -hoc (i.e., hey a e gene a ed o p e ained classi ie s) and local (i.e., hey
explain he classi ie ’s ou pu w. . . a speci ic es ins ance) [27]. CFs may be (1) model-agnos ic i hey
ope a e only on he gi en inpu (i.e., a es ins ance) and ou pu (i.e., a p edic ion) o he classi ie o (2)
model-speci ic i hey u ilise he in e nals o he classi ie o explain he gi en ou pu [47,71].
CF explana ions a e claimed o ha e a numbe o desi ed p ope ies agains which CF explana ion
me hods can be e alua ed [27]. Fo example, CFs should be alid (i.e., CFs should uly lead o he
desi ed hypo he ical ou come), p oxima e (i.e., CFs should sugges only minimal changes o he es
ins ance w. . . he selec ed dis ance me ic), spa se (i.e., CFs should minimise he numbe o ea u es
whose alues a e o be changed), ac ionable (i.e., CFs should sugges easible changes), and di e se
(i.e., CFs should o e mul iple al e na i es). An exhaus i e lis o such p ope ies can be ound in ecen
su eys on CF explana ion gene a ion and e alua ion [27,49,78]).
A la ge numbe o explana ion gene a ion me hods a e e alua ed using au oma ically compu able
me ics ha assess he a o emen ioned p ope ies o CF explana ions [49]. Howe e , such me ics o -
en imes do no ake in o conside a ion use eedback a all. Whe eas conside ing he social ac o may
no be necessa y when, e.g., measu ing alidi y, es ima ing CF di e si y may ha e o di ec ly in ol e
cap u ing e ec s o he in e ac ion be ween he sys em and he use . Indeed, CF explana ions sugges ing
minimal changes in ea u e alues may no always be equally app ecia ed by end use s. Gi en a a ie y
o po en ial CFs, di e en use s may p e e dis inc CFs o he same hypo he ical ou pu . Fu he , he
social aspec o explana ion becomes c ucially impo an when wo al e na i e au oma ically gene a ed
pieces o explana ion a e deemed equally explana o y (e.g., when he dis ances om he es ins ance
o wo o mo e closes CF da a poin s a e he same o when wo CF se s ha e he same co e age). As
he s a e-o - he-a AI echnologies a e shi ing owa ds being use -cen ic [83], i appea s indispensable
o enhance exis ing explana ion gene a ion modules wi h a sys em-use communica ion in e ace ha
would allow end use s o p oduce such inqui ies o al e na i e CFs in he cou se o an explana o y
dialogue, e en i he use is no awa e o he da ase - ela ed peculia i ies.
Va ious s a e-o - he-a CF explana ion gene a ion amewo ks a e known o o e di e se CFs
([15,17,35,49,60,62,75,85], among o he s). Howe e , he o ma o such CFs aises se e al impo an
conce ns. Fi s , mos o such amewo ks lack any in e ac ion wi h end use s lea ing he use s wi hou
u he guidance when in e p e ing he gene a ed explana ions. Second, some explaine s ou pu a se o
dis inc CFs al oge he [49,60]. In hese se ings, he G ice’s maxim o quan i y [25] may be iola ed,
as only a subse o he o e ed explana ions can be su icien o he end use . Thi d, a la ge numbe o
di e se CF explana ion gene a ion amewo ks p o ide hei ou pu in abula o m [15,17,35,49,62,75].
Whe eas na u al language gene a ion ools can be used o ans o m abula da a in o ex , a axonomy
o necessa y explana ion- ela ed eques s and esponses emains missing. To add ess hese issues, we
p opose a anspa en explana o y dialogue model o di e se ac ual and coun e ac ual explana ion
CORRECTED PROOF
I. S epin e al. / In o ma ion-seeking dialogue o XAI 13
On he one hand, he se o eques s om he use o he sys em REQ={REQ-explana ion( ˆy), eq-
de ailisa ion( ˆy,e,), eq-cla i ica ion( ˆy,e,), eq-al e na i e( ˆy,e)} consis s o he ollowing i ems:2
•REQ-explana ion( ˆy): he se o use eques s o explana ion o sys em’s p edic ion ˆy;
• eq-de ailisa ion( ˆy,e,): he use eques o u he de ails on ea u e (i.e., he co esponding
nume ical in e als) ha makes pa o a high-le el (ei he ac ual o CF) explana ion e o p edic-
ion ˆy;
• eq-cla i ica ion( ˆy,e,): he use eques o cla i ica ion o he meaning o a speci ic ea u e
ha makes pa o (ei he ac ual o CF and ei he high-le el o low-le el) explana ion e o p edic-
ion ˆy;
• eq-al e na i e( ˆy,e): he use eques o an al e na i e (ei he ac ual o CF and ei he high-le el o
low-le el) explana ion p o ided ha he use is no sa is ied wi h he p e iously o e ed explana ion
e o sys em’s p edic ion ˆy.
Fu he , he se o use explana ion eques s REQ-explana ion( ˆy)consis s o he ollowing possible
locu ions:
• eq-why( ˆy): he use eques o a ac ual explana ion o he sys em’s p edic ion ˆy;
• eq-why-no ( ˆy,y): he use eques o a CF explana ion conce ning he CF class y∈Y {ˆy} o
p edic ion ˆy(i.e., o speci y why some CF class ywas no p edic ed ins ead o ˆy).
On he o he hand, he se o esponses ( eplies) ha he sys em sends back o he use REP={REP-
explana ion( ˆy),REP-de ailisa ion( ˆy,e,),REP-cla i ica ion( ˆy,e,),REP-al e na i e( ˆy,e)} mi o s
he se o use eques s:
•REP-explana ion( ˆy): he se o sys em esponses in an a emp o explain p edic ion ˆy;
•REP-de ailisa ion( ˆy,e,): he se o sys em esponses in an a emp o p o ide de ails (i.e., nume -
ical in e als) wi h espec o ea u e o explana ion e o sys em’s p edic ion ˆy;
•REP-cla i ica ion( ˆy,e,): he se o sys em esponses in an a emp o cla i y ea u e making
pa o (ei he ac ual o CF) explana ion e o p edic ion ˆy;
•REP-al e na i e( ˆy,e): he se o sys em esponses in an a emp o p o ide he use wi h an expla-
na ion al e na i e o he p e iously o e ed (ei he ac ual o CF and ei he high-le el o low-le el)
explana ion e o p edic ion ˆy.
In addi ion, he se o eplies o eques s o (ini ial, non-al e na i e) explana ion REP-explana ion( ˆy)
consis s o he ollowing i ems:
• ep-why( ˆy): he sys em a emp s o ac ually explain he p edic ion ˆyon he basis o he known
ea u es ha led o ha decision and o e s a ac ual explana ion i i is able o, o e uses o o e i ,
o he wise;
• ep-why-no ( ˆy,y): he sys em a emp s o p o ide he use wi h a CF explana ion o p edic ion ˆy
o he gi en CF class yo e uses o o e i , o he wise.
The se o eplies o de ailisa ion eques s REP-de ailisa ion( ˆy,e,)consis s o he ollowing i ems:
• ep-de ailisa ion( ˆy,e,): he sys em p o ides he nume ical in e als o e which he co esponding
linguis ic e m o he eques ed explana ion ea u e making pa o explana ion eis de ined;
2Se s o eques s a e deno ed using uppe case le e s (as in, e.g., REQ-explana ion) whe eas single ins ances o eques s a e
deno ed using only lowe case le e s (as in, e.g., eq-de ailisa ion).
CORRECTED PROOF
14 I. S epin e al. / In o ma ion-seeking dialogue o XAI
• ep-no-de ailisa ion( ˆy,e,): he sys em e uses o p o ide nume ical in e als on he eques ed
ea u e’s linguis ic e m in explana ion e, e.g. due o hei una ailabili y.
The se o eplies o cla i ica ion eques s REP-cla i ica ion( ˆy,e,)consis s o he ollowing i ems:
• ep-cla i ica ion( ˆy,e,): he sys em p o ides he use wi h a de ini ion o he eques ed ea u e
making pa o explana ion e o p edic ion ˆy e ie ing i om he knowledge base;
• ep-no-cla i ica ion( ˆy,e,): he sys em e uses o cla i y he eques ed ea u e making pa o
explana ion e o p edic ion ˆydue o, e.g., i s absence in he knowledge base.
The se o eplies o al e na i e explana ion eques s REP-al e na i e( ˆy,e)consis s o he ollowing
i ems:
• ep-al e na i e( ˆy,e): he sys em ecognises he ac ha he use is no sa is ied wi h he o e ed
( ac ual o CF) explana ion e o p edic ion ˆy, seeks he mos ele an al e na i e o i , gene a es
and o e s an al e na i e explana ion o he use ;
• ep-no-al e na i e( ˆy,e): he sys em ecognises he ac ha he use is no sa is ied wi h he o e ed
( ac ual o CF) explana ion e o p edic ion ˆy, seeks he mos ele an al e na i e o i , bu is unable
o gene a e i .
4) Dialogue p o ocol. An explana o y dialogue be ween he sys em and he use is modelled ollowing
he ules speci ied in he dialogue p o ocol. The p o ocol de e mines u n aking ules, he ules go e ning
use ’s and sys em’s allowed mo es a each s age o he explana o y dialogue, and he e mina ion s a es
o he dialogue. Thus, he locu ion ypes abo e a e di ec ly mapped o he speech ac s p oduced by he
sys em and he use as speci ied in he dialogue p o ocol. All o he a o emen ioned p o ocol ules a e
speci ied in Appendix B.
5) Knowledge s o e. Le Kbe he knowledge s o e which accumula es use ’s knowledge w. . . expla-
na ions eques ed du ing his o he in e ac ion wi h he sys em. Knowledge s o e Kis ini ialised o be
an emp y se : K=∅. When he sys em gene a es a ac ual o CF explana ion (locu ions explain- (ˆy,
E,e )and explain-c (ˆy,E,y,ec ), as speci ied in he dialogue p o ocol), he co esponding piece o
explana ion is added o he knowledge s o e: K=K∪e (ˆy) o K=K∪ec (ˆy,y), espec i ely. The
same applies o al e na i e explana ions o ei he kind (locu ions al e - (ˆy,E,e ,e
)and al e -c (ˆy,E,
y,ec ,e
c )).
6) Explana ion s o e. Le Ebe he explana ion s o e which acks he cu en s a e o he explainee-
p e e ed explana ion h oughou he dialogue. Explana ion s o e Eis ini ialised o be an emp y se :
E=∅. Simila ly o he knowledge s o e, a ac ual o CF explana ion is added o he explana ion s o e
once gene a ed: E=E∪e (ˆy) o E=E∪ec (ˆy,y), espec i ely. I he use inds he o e ed
ac ual o CF explana ion no sa is ac o y enough and asks o an al e na i e explana ion (locu ions why-
al e na i e(ˆy,E,e )and why-no -al e na i e(ˆy,E,y,ec ), espec i ely), he co esponding explana ion
is emo ed om he explana ion s o e: E=E e (ˆy) o E=E ec (ˆy,y), espec i ely. No ewo hy,
he use canno eques an al e na i e explana ion o any explana ion non-o e ed p e iously. Fu he , he
use can only submi explana ion- ela ed eques s (de ailisa ion, cla i ica ion, al e na i e) o he piece o
explana ion being p ocessed. The esul ing explainee-p e e ed explana ion is he union o all he pieces
o explana ion ound in he explana ion s o e when a e minal dialogue s a e is eached.
7) De ailisa ion s o e. Le DET be he s o e ha con ains he ea u es o he cu en ly p ocessed
high-le el explana ion o which u he de ails can be eques ed. DET is ini ialised o be emp y, as
he explana o y dialogue s a s: DET =∅. The use can submi a de ailisa ion eques o he sys em
only i a high-le el (ei he ac ual o CF) explana ion e=eh
|eh
c is being p ocessed. Recall ha o
CORRECTED PROOF
I. S epin e al. / In o ma ion-seeking dialogue o XAI 15
each ea u e o he cu en ly p ocessed high-le el explana ion e, he ea u e is de ined in e ms o
a linguis ic a iable mapped o he co esponding linguis ic e ms. When a new piece o high-le el
explana ion is o e ed o he end use , DET is eini ialised wi h he se o ea u es ha he cu en ly
p ocessed explana ion con ains: DET ={},∀∈e. The use can ask he sys em o p o ide him o he
wi h he nume ical in e als o he linguis ic e m o he gi en explana ion ea u e only once du ing a
sub-dialogue conce ning a speci ic piece o explana ion. Thus, he co esponding ea u e is elimina ed
om he de ailisa ion s o e once he sys em has gene a ed a esponse θ(locu ion elabo a e(ˆy,E[,y],
e,,θ)): DET =DET {}.I DET =∅, i is p ohibi ed o he use o submi a de ailisa ion eques
(locu ion wha -de ails(ˆy,E[,y], e,)). When he use makes he inal decision w. . . he sys em’s claim
(i.e., ei he accep s o ejec s i ), he de ailisa ion s o e is nulli ied: DET =∅.
8) Cla i ica ion s o e. Le CLAR be he cla i ica ion s o e ha con ains he explana ion ea u es whose
meaning can be cla i ied. Simila ly o he de ailisa ion s o e, CLAR is ini ialised o be emp y: CLAR =∅.
When a new piece o explana ion is o e ed, CLAR is popula ed wi h all he ea u es ha he explana ion
being p ocessed e=eh
|eh
c |el
|el
c con ains: CLAR ={},∀∈e. No ewo hy, he de ini ions o all
he ea u es ha he da ase con ains a e p ecollec ed, mapped o one ano he by an expe o e ie ed
om a dic iona y, and s o ed in he knowledge base. The use can ask o cla i y a speci ic ea u e om he
cla i ica ion s o e only once du ing a sub-dialogue conce ning a speci ic piece o explana ion. Then, he
co esponding ea u e is elimina ed om he cla i ica ion s o e a e he sys em’s esponse υ(locu ion
cla i y(ˆy,E[,y], e,, υ)): CLAR =CLAR {}.I CLAR =∅, i is p ohibi ed o he end use o submi
a cla i ica ion eques (locu ion wha -is(ˆy,E[,y], e,)). When he use makes he inal decision w. . .
he sys em’s claim (i.e., ei he accep s o ejec s i ), he cla i ica ion s o e is nulli ied: CLAR =∅.
9) CF class s o e. Le CFS be he CF class s o e ha con ains all CF classes. I is ini ialised upon
he success ul execu ion o he ac ual explana ion eques (locu ion explain- (ˆy,E,e ))so ha CFS
=Y {ˆy} o some p edic ion ˆy∈Y. The use is allowed o eques a CF explana ion o each class
om CFS only once (locu ion why-no -explain(ˆy,E,y)). In addi ion, he use is allowed o ask o a
(se ies o ) al e na i e CF explana ion(-s) o he same CF class (locu ion why-no -al e na i e(ˆy,E,y,
ec )as many imes as he e a e al e na i e CFs o ha class. Once a CF explana ion is eques ed o
some CF class y, i is elimina ed om he CFS s o e: CFS =CFS {y}. When he use makes he
inal decision w. . . he sys em’s claim (i.e., ei he accep s o ejec s i ), he CF class s o e is nulli ied:
CFS =∅.
10) Knowledge Base. The knowledge base con ains he da ase - ela ed domain knowledge including a
speci ica ion o all he da ase ea u es (e.g., linguis ic e ms, he co esponding in e als, and de ini ions
o all he ea u es ha he da ase con ains).
3.2. Illus a i e example
Ha ing in oduced he p oposed o malism o explana o y in o ma ion-seeking dialogue modelling,
le us now illus a e i aking he p e iously conside ed example o e e ence (see Table 1 o de ails).
Thus, we a e conside ing he bee s yle classi ica ion p oblem o he bee da ase ha con ains he ol-
lowing classes: Ybee ={Blanche, Lage , Pilsne , IPA, Ba leywine, S ou , Po e , Belgian s ong ale}.
Table 2ou lines he s a es o he de ailisa ion, cla i ica ion, and CF class s o es o he example explana-
o y dialogue a e each dialogue mo e. Table 3ou lines he s a es o he knowledge and explana ion
s o es o he same example dialogue.
Ini ially, he sys em claims ha some ins ance o bee is o class Blanche (mo e m1). All he s o es
ha make pa o he dialogue model (K, E, DET, CLAR, CFS) a e ini ialised o be emp y. A he nex
CORRECTED PROOF
16 I. S epin e al. / In o ma ion-seeking dialogue o XAI
Table 2
A mo e-by-mo e o mal desc ip ion o he s o es go e ning he example o explana o y dialogue om Table 1
Mo e Locu ion DET CLAR CFS
m1claim (ˆy,E)∅∅∅
m2why-explain (ˆy,E)∅∅∅
m3explain- (ˆy,E,e ){colou , bi e ness}{colou , bi e ness}{Lage , Pilsne , IPA, Ba leywine,
S ou , Po e , Belgian s ong ale}
m4wha -is(ˆy,E,e ,) {colou , bi e ness}{colou , bi e ness}{Lage , Pilsne , IPA, Ba leywine,
S ou , Po e , Belgian s ong ale}
m5cla i y (ˆy,E,e ,,υ) {colou , bi e ness}{colou }{Lage , Pilsne , IPA, Ba leywine,
S ou , Po e , Belgian s ong ale}
m6why-no -explain (ˆy,E,y){colou , bi e ness}{colou }{Lage , Pilsne , IPA, Ba leywine,
S ou , Po e , Belgian s ong ale}
m7explain-c (ˆy,E,y,ec ){colou , bi e ness}{colou , bi e ness}{Lage , Pilsne , IPA, Ba leywine,
Po e , Belgian s ong ale}
m8wha -de ails (ˆy,E,ec ,) {colou , bi e ness}{colou , bi e ness}{Lage , Pilsne , IPA, Ba leywine,
Po e , Belgian s ong ale}
m9elabo a e (ˆy,E,ec ,,θ) {colou }{colou , bi e ness}{Lage , Pilsne , IPA, Ba leywine,
Po e , Belgian s ong ale}
m10 why-no -explain(ˆy,E,y){colou }{colou , bi e ness}{Lage , Pilsne , IPA, Ba leywine,
Po e , Belgian s ong ale}
m11 explain-c (ˆy,E,y,ec ){colou }{colou }{Lage , Pilsne , IPA, Ba leywine,
Belgian s ong ale}
m12 why-no -al e na i e(ˆy,E,y,ec ){colou }{colou }{Lage , Pilsne , IPA, Ba leywine,
Belgian s ong ale}
m13 al e -c (ˆy,E,y,ec ,e
c ){colou , s eng h}{colou , s eng h}{Lage , Pilsne , IPA, Ba leywine,
Belgian s ong ale}
m14 accep -u (ˆy,E)∅∅∅
m15 accep -s (ˆy,E)∅∅∅
s ep, he use eques s a ac ual explana ion o he gi en p edic ion (m2). The sys em p o ides he use
wi h a ac ual explana ion (m3). As he ac ual explana ion is gene a ed, bo h DET and CLAR s o es a e
popula ed wi h he co esponding ea u es (colou and bi e ness). Fu he , he piece o ac ual expla-
na ion e (ˆy=Blanche)is placed o bo h he knowledge s o e and he explana ion s o e. In addi ion,
he CF s o e CFS is popula ed wi h all he CF classes. A he nex s age, he use asks he sys em o
cla i y he no ion o bi e ness (m4) and ecei es he co esponding de ini ion om he sys em (m5). As
he cla i ica ion eques o a gi en ea u e can only be submi ed once while p ocessing a speci ic piece
o explana ion, bi e ness is hen elimina ed om he CLAR s o e.
Once he ac ual explana ion is o e ed, he use may commi o he ac ual explana ion o e ed and
inqui e a CF explana ion o some CF class. In he p esen example, he use seeks, a his s age, o know
why he classi ie did no p edic he gi en bee o be S ou (m6). Then, he classi ie p esen s he mos
ele an piece o CF explana ion o his CF class in acco dance wi h i s anking (m7). The CF expla-
na ion ec (y=S ou )is hen added o bo h he knowledge and explana ion s o es, whe eas he class
S ou is emo ed om he CFS s o e. Then, he DET and CLAR s o es a e upda ed wi h he ea u es ha
he newly o e ed CF explana ion con ains. As he use equi es mo e de ailed in o ma ion on bi e ness
(m8), he sys em e ie es he eques ed nume ical in e al o e which he alue o bi e ness is de ined
o be high (m9). The ea u e bi e ness is hen emo ed om he DET s o e. Then, he use p oceeds o
eques a CF explana ion o class Po e (m10). Simila ly o he p e iously o e ed explana ions, DET
and CLAR a e upda ed acco dingly, as he mos ele an piece ( om explaine ’s poin o iew) o CF
CORRECTED PROOF
I. S epin e al. / In o ma ion-seeking dialogue o XAI 17
Table 3
An example explana o y dialogue schema
Block Mo e U e ance K E
Cm1Sys em: The es ins ance is o class y.∅∅
Em2Use : Could you explain why you hink so? ∅∅
m3Sys em: I is o class ybecause ea u e1is e m1.{e (ˆy)}{e (ˆy)}
m4Use : Wha do you mean by ea u e1?{e (ˆy)}{e (ˆy)}
m5Sys em: ea u e1is de ini ion o ea u e1.{e (ˆy)}{e (ˆy)}
m6Use : Bu why is i no o class y?{e (ˆy)}{e (ˆy)}
m7Sys em: I would be o class yi ea u e1{e (ˆy),ec (y)}{e (ˆy),ec (y)}
we e e m2and ea u e2we e e m3.
m8Use : Could you speci y how ea u e1is de ined? {e (ˆy),ec (y)}{e (ˆy),ec (y)}
m9Sys em: ea u e1is de ined o be e m2because {e (ˆy),ec (y)}{e (ˆy),ec (y)}
i is ound in he in e al [ e m2min, e m2max].
m10 Use : Bu why is he es ins ance no o class y?{e (ˆy),ec (y)}{e (ˆy),ec (y)}
m11 Sys em: I would be o class y i ea u e1{e (ˆy),ec (y), ec (y)}{e (ˆy),ec (y), ec (y)}
we e e m3and ea u e3we e e m3.
m12 Use : I am no qui e sa is ied wi h you explana ion. {e (ˆy),ec (y), ec (y)}{e (ˆy),ec (y)}
Could you o e me ano he one?
m13 Sys em: Su e!I wouldbeo classy i ... {e (ˆy),ec (y), ec (y), e
c (y)}{e (ˆy),ec (y), e
c (y)}
Tm14 Use : Okay, I us you p edic ion. {e (ˆy),ec (y), ec (y), e
c (y)}{e (ˆy),ec (y), e
c (y)}
m15 Sys em: Thank you o you us in me. Bye! {e (ˆy),ec (y), ec (y), e
c (y)}{e (ˆy),ec (y), e
c (y)}
In he le -hand side column (“Block”), C s ands o claim, E – o explana ion, T – o e mina ion).
explana ion is gene a ed and o e ed o he class Po e (m11). Then, he class Po e is excluded om
he CFS s o e whe eas he newly o e ed CF explana ion is added o he knowledge and explana ion
s o es. Howe e , as he use is le dissa is ied o no con inced enough wi h he o e ed explana ion, he
o she inqui es an al e na i e explana ion o he p e iously o e ed one (m12). Then, he la es o e ed
explana ion is emo ed om he explana ion s o e. Subsequen ly, i he nex bes anked al e na i e can
be o e ed, i is added o he explana ion s o e (m13). The DET and CLAR s o es a e hen upda ed acco d-
ingly. Ha ing p ocessed he p esen ed explana ions in hei en i e y, he use makes an in o med decision
ha he classi ie ’s p edic ion can be accep ed (m14). The sys em e mina es he dialogue ou pu ing a
a ewell locu ion (m15).
Table 3gene alises he p esen ed example o explana o y dialogue o any da ase whe e ea u es, lin-
guis ic e ms, and classes se e as da ase -speci ic a iables. I is possible o gene alise any explana o y
dialogue modelled in acco dance wi h he p oposed amewo k using he sugges ed empla e u e ances.
No ewo hy, h ee main building blocks o such explana o y dialogue (C – claim, E – explana ion, and
T – e mina ion) can be dis inguished. Figu e 5p esen s he co esponding (pa ial, o illus a i e pu -
poses) pa se ee o such a gene alised explana o y dialogue.
3.3. Explana o y dialogue g amma (EDG)
As ollows om he example o dialogue p esen ed in Sec ion 3.2, he p oposed dialogue model has a
hie a chical s uc u e wi h espec o i s main building blocks. This obse a ion allows us o e lec he
modula composi ion o explana o y dialogue ( ollowing ou model) in a con ex - ee dialogue g amma .
As he ansi ions be ween he s a es o he dialogue a e ini e and p ede ined, he use o he co espond-
CORRECTED PROOF
18 I. S epin e al. / In o ma ion-seeking dialogue o XAI
Fig. 5. A pa se ee o he example o explana o y dialogue. Shaded nodes a e non- e minals co esponding o speci ic speech ac s. The sub ees in he dashed
egions ep esen dialogue mo es.
CORRECTED PROOF
I. S epin e al. / In o ma ion-seeking dialogue o XAI 19
ing EDG allows us o (1) gene a e any explana o y dialogue ha is alid in acco dance wi h he dialogue
p o ocol es ic ions and (2) pa se any ac ually alid explana o y dialogue o make a conclusion ha
he p esen explana o y dialogue is in alid wi h espec o he dialogue model cons ain s. Fu he , a
g amma -based dialogue model can ake in o accoun modi ica ions in he dialogue p o ocol i hose a e
deemed necessa y.
In ligh o he abo e, we de ine an EDG ollowing Chomsky’s de ini ion o a con ex - ee g amma as
a uple G=T,N,P,Swhe e Tis he se o e minals, Nis he se o non- e minals, Pis he se o
p oduc ion ules (p oduc ions), and Sis he s a oken. In ou model, Tco esponds o a sen ence ac u-
ally u e ed by each pa icipan in he cou se o a dialogue. Nencompasses he in e nal building blocks
o he dialogue as well as he speech ac s in ol ed (see he shaded nodes in Fig. 5 o de ails). Thus,
any explana o y dialogue is said o ha e h ee main building blocks ( hose co esponding o he non-
e minals CLAIM, EXPLANATION, TERMINATION). In acco dance wi h cu en legal equi emen s o
explana ion o AI, he block EXPLANATION enables he use o exe cise he igh o explana ion and
is made op ional. All he non- e minals p oduced om he non- e minal EXPLANATION a e designed
in acco dance wi h he p ede ined eques s and esponses (see Sec ion 3.1 o de ails). In addi ion, P
is composed in acco dance wi h he dialogue p o ocol se ings (see Appendix B o de ails). No e ha
p oduc ions can be subdi ided in wo g oups, i.e., da ase -independen and da ase -speci ic p oduc ions.
Da ase -independen p oduc ion ules o m he co e o he p oposed explana o y dialogue model and
can be used in any applica ion domain so long as i mee s he se ings o he classi ica ion p oblem as
desc ibed in Sec ion 2.1. The da ase -independen ules alid o he illus a i e example o an explana-
o y dialogue a e ou lined in Appendix C. In u n, da ase -speci ic ules ollow he s uc u e o he gi en
da ase and hey a e es ic ed by he in o ma ion p o ided by he gi en in e p e able ule-based classi ie
and he co esponding knowledge base. Finally, he s a oken Sis known o always be he non- e minal
DIALOGUE node, i.e., he oo node in he ee depic ed in Fig. 5.
4. P ocess mining o dialogue analy ics
The p oposed model o explana o y dialogue is designed in a op-down manne , which signals ce ain
sho comings. Thus, he dialogue p o ocol bases on he assump ion ha he axonomy o eques s and
esponses p oposed in Sec ion 3inspi ed by indings om he li e a u e exhaus i ely co e s use ’s needs
and sys em’s abili ies when engaged in an explana o y dialogue. Howe e , in he absence o any em-
pi ical e alua ion, such assump ions may esul being pu ely specula i e. Fo example, speci ic eques s
may be u ilised o a e y limi ed ex en o e en no u ilised a all. Al e na i ely, he e may exis eques s
ha a e no included in he o iginal model, which may ne e heless be conside ed essen ial o human-
machine in e ac ion by he explainees. Ei he way, modi ica ions o he model should be g ounded on
he da a ob ained om he end use s. As such da a-d i en conclusions on he u ili y o he op-down
dialogue model can only be made upon empi ical e alua ion, a use s udy is necessa y o alida e he
p oposed model.
In addi ion o analysis o ee- o m use eedback, e alua ion o a dialogue model can be au oma ed
by inspec ing dialogue pa e ns in he collec ed dialogue ansc ip s. In hese se ings, dialogues can be
ea ed as i e a i e p ocesses whose key pa e ns allow us o disce n s eng hs and weaknesses o he
dialogue model. To analyse dialogues as p ocesses, we p opose a use o p ocess mining echniques.
P ocess mining is he sub ield o da a science ha aims o p o ide ools o disco e ing insigh s in o
ope a ional p ocesses and hus suppo s p ocess imp o emen s [76]. Following he p ocess mining e -
minology [50], an ins ance o a p ocess (i.e., a speci ic explana o y dialogue) is deno ed as a ace τ.
CORRECTED PROOF
20 I. S epin e al. / In o ma ion-seeking dialogue o XAI
Table 4
An example o an e en log ( he ac i i ies in bold a e hose p oduced by he sys em; he use -p oduced ac i i ies a e hose in
i alics)
Case Ac i i y S a End
Dialogue1claim 2022-06-09 11:54:12 2022-06-09 11:54:12
Dialogue1why-explain 2022-06-09 11:54:12 2022-06-09 11:54:21
Dialogue1explain- 2022-06-09 11:54:21 2022-06-09 11:54:22
Dialogue1wha -de ails 2022-06-09 11:54:22 2022-06-09 11:54:42
Dialogue1elabo a e 2022-06-09 11:54:42 2022-06-09 11:54:42
Dialogue1why-no -explain 2022-06-09 11:54:42 2022-06-09 11:55:58
Dialogue1explain-c 2022-06-09 11:55:58 2022-06-09 11:56:00
Dialogue1wha -de ails 2022-06-09 11:56:00 2022-06-09 11:56:32
Dialogue1elabo a e 2022-06-09 11:56:32 2022-06-09 11:56:33
Dialogue1accep -u 2022-06-09 11:56:33 2022-06-09 11:57:28
Dialogue1accep -s 2022-06-09 11:57:28 2022-06-09 11:57:28
Dialogue2claim 2022-06-15 17:03:34 2022-06-15 17:03:34
Dialogue2why-explain 2022-06-15 17:03:34 2022-06-15 17:04:22
Dialogue2explain- 2022-06-15 17:04:22 2022-06-15 17:04:23
Dialogue2wha -is 2022-06-15 17:04:23 2022-06-15 17:04:50
Dialogue2cla i y 2022-06-15 17:04:50 2022-06-15 17:04:50
Dialogue2why-no -explain 2022-06-15 17:04:50 2022-06-15 17:05:38
Dialogue2explain-c 2022-06-15 17:05:38 2022-06-15 17:05:40
Dialogue2why-no -al e na i e 2022-06-15 17:05:40 2022-06-15 17:06:12
Dialogue2al e -c 2022-06-15 17:06:12 2022-06-15 17:06:13
Dialogue2wha -de ails 2022-06-15 17:06:13 2022-06-15 17:06:59
Dialogue2elabo a e 2022-06-15 17:06:59 2022-06-15 17:07:00
Dialogue2 ejec -u 2022-06-15 17:07:00 2022-06-15 17:07:49
Dialogue2 ejec -s 2022-06-15 17:07:49 2022-06-15 17:07:49
Subsequen ly, each ace consis s o he se o ac i i ies A(in his case, locu ions). In u n, a speci ic
ins ance ( ealisa ion) o an ac i i y α∈A(i.e., a dialogue mo e) is e e ed o as an e en ε. Al oge he ,
a collec ion o explana o y dialogues makes up he so-called e en log.
An example o an e en log basing on a collec ion o explana o y dialogues is depic ed in Table 4.
I con ains wo aces (i.e., Dialogue1and Dialogue2) ha ep esen ins ances o he eco ded explana-
o y dialogues be ween (possibly, di e en ) use (-s) and he gi en sys em (i.e., an in e p e able ule-
based classi ie ). In o al, he p ocess model con ains 22 e en s each o which is essen ially a speci ic
dialogue mo e pai ed wi h he co esponding locu ion. Figu e 6illus a es he co esponding p ocess
model g aph. The isual ep esen a ion o he p ocess model acili a es de ec ion o he ac i i y pa e ns
(i.e., subp ocesses cha ac e ising common pa s o dis inc dialogues) aking place in he collec ion o
dialogues.
A dialogue p o ocol can be ep esen ed as a ini e s a e machine whose nodes a e he locu ions mod-
elled, edges being legi ima e ansi ions be ween di e en s a es o he dialogue (e.g., om a eques
o all possible esponses). In e ms o p ocess mining, one can ep esen he dialogue p o ocol as he
so-called p ocess model – a di ec ed g aph M=N,Ewhe e he se o nodes N⊆A∪{S a ,End}is
composed o he p ocess ac i i ies and he se o edges E⊆N×N ep esen s (possibly, causal) ela ions
be ween pai s o ac i i ies whe e S a and End a e, espec i ely, he s a and end ime o execu ion o
he co esponding ac i i y.
CORRECTED PROOF
I. S epin e al. / In o ma ion-seeking dialogue o XAI 21
Fig. 6. The g aphical iew o he p ocess model co esponding o he example Dialogue1in Table 4.
To analyse he ac ually eco ded dialogues quan i a i ely, we sugges ha he so-called con o mance
checking p ocedu e be applied. In p ocess mining, con o mance checking is applied o ela e he e en s
in he ac ually egis e ed p ocesses and he p ocess model in o de o iden i y commonali ies and dis-
c epancies be ween he o me and he la e . In he case o e alua ing he p oposed dialogue game, all
he mo es made by bo h dialogue game playe s ollow he p e iously de ined dialogue p o ocol. Hence,
no de ia ion om he p o ocol can be obse ed. Ins ead, con o mance checking allows us o highligh
he mos (and he leas ) equen dialogue pa e ns in he e en log and e alua e i agains he p ocess
model (i.e., he dialogue p o ocol). Con o mance checking can lead o ob aining da a-d i en knowledge
o he leas equen ly submi ed eques s and/o dialogue s a e ansi ions, which can be used o modi y
he o iginally p oposed dialogue p o ocol in o de o inc ease i s quali y.
To sum i up, he p oposed dialogue model can be e alua ed in wo complemen a y ways: quali a-
i ely and quan i a i ely. On he one hand, quali a i e ee- o m use eedback (e.g., in he o m o a
pos -expe imen su ey) can poin o missing eques s o ansi ions be ween exis ing eques s in he
dialogue p o ocol. On he o he hand, he leas equen dialogue pa e ns may signal hei u ili y o
explana o y pu poses o he dialogue model. In p ocess mining, a equency h eshold alue can, o
example, be se o subsequen ly op imise he p ocess model by emo ing he leas obse ed model pa -
e ns. Simila ly, he leas equen eques s o esponses may be emo ed om he dialogue p o ocol i
he empi ically g ounded h eshold alue is a ailable and se p io o e alua ion. As a esul , p ocess
mining is shown o se e as a me hodological basis o quan i a i e e alua ion o he p oposed dialogue
model. In combina ion wi h ee- o m use eedback o quali a i e e alua ion o he dialogue p o ocol,
p ocess mining is able o p o ide us wi h u he insigh s w. . . he quali y o a dialogue model.
5. Expe imen al se ings
In o de o e alua e he p oposed model o explana o y dialogue ollowing he a o emen ioned e alu-
a ion amewo k, we ca ied ou an explo a o y use s udy. In he emainde o his sec ion, we desc ibe
he se up o he human e alua ion s udy. Thus, Sec ion 5.1 desc ibes he da ase s used as he basis o
aining he classi ie s o he s udy. Sec ion 5.2 ou lines echnicali ies o he explana ion gene a ion
me hod used in he gi en expe imen . Sec ion 5.3 ou lines he dis inc i e cha ac e is ics o he classi ie s
ained on he a o emen ioned da ase s. Sec ion 5.4 discusses he s imuli selec ion as well as he design
o he dialogue sys em used in he expe imen .
5.1. Da ase s
In ou s udy, we used he ollowing h ee da ase s: baske ball playe posi ion [3], bee s yle [13],
and hy oid disease diagnosis [19]. All h ee da ase s se e o sol e a mul iclass classi ica ion p ob-
lem in h ee di e en applica ion domains. Fi s , he baske ball playe s posi ion da ase p esupposes
CORRECTED PROOF
22 I. S epin e al. / In o ma ion-seeking dialogue o XAI
i e classes ela ed o he ollowing playe posi ions: Ybaske ball ={poin -gua d, shoo ing-gua d, small-
o wa d, powe - o wa d, cen e }. Second, he bee s yle da ase (as was used in he illus a i e ex-
ample in Sec ion 3.2) ca ego ises ins ances o bee o belong o one o he ollowing eigh classes:
Ybee ={Blanche, Lage , Pilsne , IPA, Ba leywine, S ou , Po e , Belgian s ong ale}. Thi d, he hy-
oid disease da ase p esupposes he ollowing ou po en ial labels: Y hy oid ={no hypo hy oid, p ima y
hypo hy oid, compensa ed hypo hy oid, seconda y hypo hy oid}.
To gua an ee consis en and compa able esul s, only nume ical con inuous ea u es we e used o
aining he co esponding classi ie s. Fu he , all he ea u es we e mapped o linguis ic e ms as ollows.
The bee s yle da ase was anno a ed by an expe b ewe , he e o e i con ains o iginal ea u e- alue
pa i ions. The ea u es om he o he da ase s we e spli in h ee uni o m in e als o equal leng h,
each o which was mapped o he ollowing linguis ic e ms: low, medium, high(excep o he ea u e
heigh , which is desc ibed wi h 5 linguis ic e ms, in he baske ball playe posi ion da ase ). Table 5
summa ises in o ma ion on he ea u es om all he da ase s as well as he co esponding linguis ic
e ms, wi h he nume ical in e als a ached.
5.2. Explana ion gene a ion me hod
To e alua e he dialogue game p oposed in his pape as a communica ion in e ace be ween he sys-
em and he use , we gene a e mul iple ac ual and CF explana ions using he XOR me hod [72]. This
explana ion gene a ion me hod ope a es on he ule base (i.e., a se o decision pa hs o each class) o
a ule-based in e p e able classi ie (e.g., a uzzy ule-based classi ica ion sys em o a decision ee DT
whe e b anches a e i s ans o med in o a lis o ules). All au oma ic explana ions ollow he s uc u e
o he decision pa h (in he case o he ac ual explana ion) o he minimally di e en decision pa h
leading o he gi en CF class (in he case o he CF explana ion). The ollowing pipeline o ou s eps
cons i u es he explana ion gene a ion p ocess:
(1) Rule ec o isa ion. Each ule ound in he ule base is ep esen ed as a (bina y, in he case o
he XOR me hod) ec o o all possible ea u e- alue pai s. In he case o a DT, he alues o he
ec o a e all he unique condi ions (e.g., “bi e ness ⩽10”) ound in he se o DT nodes.
(2) Rele ance es ima ion. Once he ules a e ec o ised, a dis ance is calcula ed be ween ec o s
ep esen ing he decision pa h ec o ( esponsible o he p edic ion) and each ule leading o he
gi en ( ac ual o CF) class. In he case o he XOR me hod, he exclusi e-OR unc ion calcula es
he dis ance be ween he ec o s. The ec o s a e hen anked in acco dance wi h he dis ances.
The minimally dis an ule is selec ed as a empla e o he ou pu explana ion ollowing he con-
en ional de ini ion o a CF explana ion.
(3) Linguis ic app oxima ion. Each in e al ound in he selec ed ule is mapped o he p ede ined
linguis ic e ms by measu ing he simila i y be ween he se o nume ical alues co esponding
o his in e al and each se o nume ical alues o he co esponding ea u e. The mos simila
linguis ic e m is selec ed o he gi en ea u e.
(4) Su ace ealisa ion. The linguis ically app oxima ed ule is passed on o he su ace ealisa ion
module ha ou pu s a empla e-based g amma ically co ec high-le el explana ion. Simila ly, he
co esponding nume ical in e als a e used o gene a e a low-le el explana ion.
Fo DTs, ac ual explana ions a e essen ially he ea u e- alue in e als agg ega ed along he decision
pa h. This explana ion gene a ion me hod p esupposes ha al e na i e ac ual explana ions canno be
gene a ed because al e na i e decision pa hs leading o he same p edic ed class would no adequa ely
