E alua ion o Explainable A i icial In elligence me hods
in Language Lea ning Classi ica ion o Spanish Te ia y
Educa ion S uden s
G igo ios Tzionis1, Ge asimos An zoula os1, Pe iklis Papaioannou1, A hanasios
Ma opoulos1, Ilias Gialampoukidis1, Ma a González Bu gos2, S e anos V ochidis1,
Ioannis Kompa sia is1, Ma o Vlachopoulou3
1Cen e Fo Resea ch & Technology Hellas; 2Me odo Es udios Consul o es, 3Uni e si y o
Macedonia
Abs ac . Wi h he inc easing p e alence o AI, signi ican ad ancemen s ha e
been made ac oss a ious domains, such as heal hca e, lea ning, indus y, e c.
Howe e , challenges pe sis in e ms o us ing and comp ehending he ou -
comes gene a ed by hese echnologies. Speci ically in he language lea ning
domain, eache s ace challenges ega ding he classi ica ion o he s uden s’
lea ning capabili ies and build he app op ia e lea ning pa h o hem. To ad-
d ess hese challenges, he concep o Explainable A i icial In elligence (XAI)
was adop ed, which is a se o p ocesses and me hods ha allows human use s
o in e p e , unde s and and us he esul s de i ed om machine lea ning
models. In his s udy, we adop wo well-known XAI algo i hms, PFI and
SHAP in a p oposed Knowledge Gene a ion Model equipped wi h ML models
o de i e hidden knowledge. The whole amewo k has been applied and e alu-
a ed on he Language Lea ning Classi ica ion o Spanish Te ia y Educa ion
S uden s acqui ed om he CEDEL2 da abase. The analysis concludes ha in
e ms o explaining he black-box models, he SHAP model-agnos ic me hod is
he mos comp ehensi e and dominan o isualizing ea u e in e ac ions and
ea u e impo ance and be applicable o any ype o da a.
Keywo ds: Language Lea ning, Machine Lea ning, Explainable A i icial In-
elligence, In e p e abili y, Compa a i e Analysis.
1 In oduc ion
Language Lea ning (LL) is a undamen al aspec o human de elopmen , enabling
indi iduals o communica e, exp ess hemsel es, and unde s and he wo ld a ound
hem [1]. I is a complex and ongoing p ocess ha begins a bi h and con inues
h oughou li e. Th ough in e ac ions wi h o he s, indi iduals acqui e linguis ic skills,
comp ehend g amma and ocabula y, and de elop he abili y o communica e e ec-
i ely [2]. Linguis ic scien is s ha e de o ed ex ensi e e o s o s udy he mecha-
nisms and pa e ns unde lying language acquisi ion, seeking o unco e he cogni i e
p ocesses in ol ed and he ac o s in luencing language lea ning ou comes [1].
2
On he o he hand, in ecen yea s, he ield o A i icial In elligence (AI) has expe-
ienced ema kable ad ancemen s, e olu ionizing a ious domains wi h i s p edic i e
and analy ical capabili ies [3]. I has ound applica ions in di e se domains, anging
om na u al language p ocessing o compu e ision and speech ecogni ion [4] [5].
Simul aneously, he ield o Machine Lea ning (ML) has wi nessed emendous
g ow h and popula i y in he ealm o A i icial In elligence (AI) [6]. ML echniques
enable he de elopmen o p edic i e and desc ip i e models ha can analyze as
amoun s o da a, ex ac pa e ns, simula e and unde s and complex sys ems and hu-
man beha iou as well as make in o med decisions [4] [5] [7].
Howe e , he black-box na u e o many ML models has aised conce ns abou hei
anspa ency and in e p e abili y [7]. As ML algo i hms become mo e powe ul and
pe asi e, he need o Explainable AI (XAI) has become pa amoun . XAI encom-
passes a se o p ocesses and me hods ha enable human use s o comp ehend and
us he esul s and ou pu s gene a ed by ML algo i hms [8]. I aims o p o ide in-
sigh s in o how an AI model ope a es, i s an icipa ed impac , and po en ial biases. By
p omo ing model accu acy, ai ness, anspa ency, and in e p e abili y, XAI con ib-
u es o mo e in o med AI-powe ed decision-making [9].
The in e wind be ween Language Lea ning and Machine Lea ning has a ac ed
signi ican a en ion om esea che s and p ac i ione s, as bo h ields sha e common
goals o unde s anding and modeling human beha io [10]. ML echniques can s ess
language lea ning p ocesses by p o iding pe sonalized lea ning expe iences, au oma -
ed language assessmen , and in elligen u o ing sys ems [8] [9].
In his pape , ou esea ch objec i e is o explo e he applica ion o XAI me hodol-
ogies in he con ex o language lea ning. Speci ically, we employ he p oposed in e -
ac i e and i e a i e Visual Analy ics amewo k [11] along wi h he wo well-known
XAI me hods, namely he Pe mu a ion Fea u e Impo ance (PFI) [12] and SHapley
Addi i e exPlana ions (SHAP) [13]. These echniques o e in e p e abili y by assign-
ing impo ance sco es o ea u es and p o iding explana ions o indi idual p edic-
ions. The goal is o classi y Spanish Te ia y Educa ion Lea ne s in e ms o hei
pe o mance elied on gene al cha ac e is ics and hei lea ning p o ile using machine
lea ning echniques and disco e knowledge and pa e ns ha a e hidden in he da a.
Mo eo e , he applica ion o XAI algo i hms o he esul s o he classi ica ion p ob-
lem aims o e alua e hem and compa e hem in e ms o which one o hem p o ides
be e insigh s o eache s and p ac i ione s.
2 Language Lea ning: P ocesses and Challenges
In ecen decades, language lea ning has ga ne ed signi ican a en ion among
linguis ic scien is s [14]. I is widely ecognized as an ac i e and con inuous p ocess
ha begins a bi h and pe sis s h oughou li e [15]. I is an ongoing endea o ha
encompasses a ious s ages o de elopmen , om ea ly childhood o adul hood [16].
Addi ionally, language lea ning plays a c ucial ole in es ablishing ela ionships wi h
amily membe s and iends, while also aiding in he comp ehension and o ganiza ion
o he wo ld [15]. Language lea ne s o all ages ac i ely engage in acqui ing new
3
ocabula y, mas e ing g amma ical s uc u es, and e ining hei communica i e skills
[16]. Linguis ic scien is s wo ldwide a e dedica ed o de i ing aluable conclusions
and co ela ions om hei esea ch in his domain [17]. Resea che s in linguis ics
s i e o un a el he complexi ies o language acquisi ion, d awing aluable insigh s
om hei in es iga ions [18].
Language lea ning p esen s se e al challenges ha a y ac oss indi iduals and
con ex s [19]. Lea ne s may encoun e di icul ies in p onuncia ion, g amma , ocab-
ula y acquisi ion, o cul u al adap a ion [20]. Fac o s such as age, mo i a ion, lea ning
en i onmen , and exposu e o he a ge language in luence he lea ning p ocess [2].
Unde s anding hese challenges and de eloping e ec i e ins uc ional s a egies a e
essen ial o acili a ing success ul language lea ning ou comes [16]. Resea ch in his
domain aims o unco e he unde lying mechanisms o language acquisi ion, iden i y
e ec i e ins uc ional me hods, and add ess he speci ic needs o di e se lea ne pop-
ula ions [16]. By examining he challenges aced by language lea ne s, esea che s
can con ibu e o he de elopmen o e idence-based pedagogical app oaches, lan-
guage assessmen ools, and in e en ions o suppo language lea ning ac oss di e -
en con ex s [16].
3 Enhancing Language Lea ning wi h Machine Lea ning
Language lea ning is a complex and dynamic p ocess ha in ol es acqui ing lin-
guis ic skills, comp ehending g amma and ocabula y, and de eloping e ec i e
communica ion abili ies [21].
One key a ea whe e ML has made signi ican con ibu ions o language lea ning is
in he de elopmen o in elligen u o ing sys ems [22]. These sys ems u ilize ML
algo i hms o unde s and he unique lea ning needs and p e e ences o indi idual
lea ne s, enabling he deli e y o pe sonalized ins uc ion and eedback [22]. By ana-
lyzing lea ne da a and pe o mance pa e ns, ML models can adap he lea ning con-
en and pace o op imize lea ning ou comes [22].
ML algo i hms also play a c ucial ole in au oma ed language assessmen , p o id-
ing objec i e and e icien e alua ion o lea ne s' language p o iciency [23]. Na u al
language p ocessing echniques enable he au oma ic sco ing and analysis o lea ne s'
w i en and spoken esponses, p o iding de ailed eedback on g amma , ocabula y
usage, and o e all language p o iciency [23]. This au oma ed assessmen p ocess
sa es ime o educa o s and allows lea ne s o ecei e immedia e eedback, enhanc-
ing he lea ning expe ience [24].
As ML models become inc easingly complex and powe ul, he need o in e p e -
abili y has gained signi ican a en ion [25]. In e p e abili y e e s o he abili y o
unde s and and explain he decisions and ou pu s o ML models [26]. I helps build
us and con idence in he p edic ions made by hese models, especially in c i ical
domains such as manu ac u ing, heal hca e and inance [25] [27] [28] [29]. Explaina-
ble AI (XAI) has eme ged as a ield ha ocuses on de eloping p ocesses and me h-
ods o enhance he in e p e abili y o ML models [25] [30]. XAI enables use s o
comp ehend he inne wo kings o ML algo i hms, unde s and he easoning behind
4
hei decisions, iden i y po en ial biases, and assess he eliabili y and ai ness o he
model's ou comes [31]. XAI aims o b idge he gap be ween he black-box na u e o
complex ML models and he need o anspa ency and accoun abili y in AI-powe ed
decision-making sys ems [25].
In conclusion, he g owing complexi y o ML models, he need o in e p e abili y
and Explainable AI has become c ucial [26]. Resea che s and p ac i ione s a e ac i e-
ly wo king owa ds de eloping me hods and echniques o enhance he anspa ency,
in e p e abili y, and us wo hiness o ML models, ul ima ely leading o mo e e-
sponsible and eliable AI sys ems [30].
4 P oposed Me hodology
In his pape , we employ and ex end he p oposed Knowledge Gene a ion Model
(KGM) (Fig. 1) o language lea ning [11] by consolida ing ad anced Machine
Lea ning echniques, o deal wi h challenges in he language lea ning domain, along
wi h XAI app oaches o p o ide mo e in e p e able and eliable indings and esul s.
Fig. 1. Knowledge gene a ion model (KGM) o language lea ning.
Speci ically, he in e ac i e and i e a i e Visual Analy ics schema os e s complex
decision-making p ocesses by le e aging wo main pipelines o p ocessing da a,
namely om aw Da a o Visualisa ion (In oVis p ocess) o Da a Mining modeling
h ough he Knowledge Disco e y in Da abases (KDD) p ocesses [32], coupling wi h
machine lea ning algo i hms (Fig. 1). In his wo k, we ocus on he p ocess and anal-
ysis o ob ained lea ning da a as well as he ou comes in e p e a ion by applying XAI
algo i hms, and pa icula ly he Pe mu a ion Fea u e Impo ance (PFI) [12] and SHap-
ley Addi i e exPlana ions (SHAP) [13]. The e o e, he impo ance o he selec ed
ea u es/a ibu es ha u ilized in he machine lea ning models will be disco e ed
p o iding use ul indings o he eache s and o he s akeholde s. Those indings,
which a e in e es ing obse a ions de i ed om he da a mining models and XAIs,
lead o u he in e ac ion be ween he human-analys (in ou case he eache s) and
he Visual Analy ics Componen (sys em) by ollowing he Explo a ion loop (Fig. 1)
[33]. Fu he mo e, he indings can be in e p e ed by expe s ( eache s) using hei
in insic p e ious knowledge in he con ex o he p oblem domain (language lea n-
5
ing) and hence, new insigh s a e eme ged, ollowed by new hypo heses ha should be
analysed and e alua ed (Ve i ica ion loop) (Fig. 1). Finally, hese i e a ion loops will
conclude o he gene a ion o new knowledge by he Knowledge Gene a ion Loop
(Fig. 1). The knowledge gene a ion conce ns he e i ica ion o hypo heses and exis -
ing assump ions based on he e ealed e idence. This e idence-based app oach pe -
mi s eache s and analys s o us hypo heses leading o gained knowledge and gene -
a e new ones [34]. O he wise, hey should disca d he hypo heses and e u n o he
explo a ion o new, undisco e ed co ela ions in he da a. Assessing he us wo hi-
ness o new knowledge depends on he collec ed e idence and equi es a c i ical e-
iew o he o e all KDD analysis p ocess s a ing om da a ga he ing. In mo e dep h
he s eps o he KDD [32] a e ollowing:
1. Selec ion - du ing his s ep a a ge da a se is c ea ed, by ocusing on a subse o
a ibu es o da a samples ha equi e u he explo a ion and analysis [35].
2. P e-p ocessing - he selec ed da ase unde goes p e-p ocessing o ob ain consis en
da a. Po en ial ac ions include handling missing alues, de ec ing ou lie s and ex-
eme alues, ea u e scaling and no maliza ion. Fo example, ex eme alues ou
o a ional in e al o he age a ibu e a e being de ec ed and elimina ed om u -
he analysis. Mo eo e , some da a mining me hods wo k well when he a ibu es
a e in he same scale. Hence, me hods o no malisa ion such as z-sco e, min-max
scaling ha e been adop ed [36].
3. T ans o ma ion - by u ilising dimension educ ion p ocesses he p e-p ocessed da-
ase is ans o med o u he analysis. Au oma ed p ocesses o ans o m he p e-
p ocessed da a in o a compa ible o ma o be u he analysed by da a mining
echniques ha e been deployed [37].
4. Modeling - in ol es he de elopmen o Machine Lea ning me hods and ech-
niques o ex ac ing o disco e ing p e iously unknown, in e es ing pa e ns o
ends in a pa icula ep esen a ional o m ha depends on he Da a Mining goals
(e.g. p edic ion o classi ica ion). In his wo k, we ha e applied machine lea ning
me hods o deal wi h he classi ica ion p oblem in language lea ning as desc ibed
in he ollowing sec ion.
5. E alua ion - his s ep is e y impo an as he gene alisa ion capabili ies o he
ained models a e assessed. The de eloped machine lea ning models es ed in
e ms o hei pe o mance agains speci ic alida ion measu es, such as accu acy,
F1-sco e e c. A e he ine uning o he pa ame e s he bes ained ML model is
selec ed [38].
6. In e p e a ion – his s ep conce ns he applica ion o he wo well-known explain-
abili y algo i hms (PFI and SHAP) and in e p e he esul s.
5 Analysis & Resul s
In his s udy, se en di e en machine lea ning me hods we e u ilized, namely Lo-
gis ic Reg ession (LR), K-Nea es Neighbo s (K-NN), Linea Disc iminan Analysis
(LDA), Decision T ee (CART), Suppo Vec o Machine (SVM), Random Fo es
(RF), and Bagging (BG) o classi y Spanish Te ia y Educa ion Lea ne s in e ms o
6
hei Placemen Tes Sco e. Speci ically, da a om he CEDEL2
1
da abase was em-
ployed.
CEDEL2 con ains da a om lea ne s o Spanish a all p o iciency le els (beginne ,
in e media e, ad anced) and di e en L1 (means lea ne s’ mo he ongue) and ‘L2’
(means lea ne s’ o eign language) backg ounds. Ini ially, con ains 3034 egis a ions
bu a e p ep ocessing he emained egis a ions dec eased o 1473 eco ds. We
selec ed six (6) ea u es om he da ase con ained in he da abase due o he ac ha
hey appea o be mos ele an o he language lea ning p oblem. These ea u es a e
Sex (Gende ), Age, Mo he s na i e language, Languages spoken a home, Yea s s ud-
ying Spanish, and Addi ional Fo eign Languages. As a ge a iable can be employed
he Placemen Tes Sco e (%) ea u e, which has classi ied in o h ee dis inc classes
which a e he ollowing: Class ‘0’ below 48% (142 egs); Class ‘1’ om 48% o
81.9% (572 egs) and inally, Class ‘2’ om 82% o 100% (759 egs).
To imp o e he accu acy o each classi ie he g id sea ch app oach was applied o
in es iga e which a e he op imal hype pa ame e se o each classi ie . We conduc -
ed a se ies o expe imen s using he a o emen ioned classi ie s o e he abo e da ase
and hei pe o mance has been assessed in e ms o he e alua ion me ics (a g. Ac-
cu acy). The K-NN and SVM classi ie s exhibi bes gene aliza ion capabili ies as
hey achie ed he highes accu acy o e he es ing se compa ed o he o he s, a ound
67.11% (Fig. 2).
Fig. 2. Pe o mance (a g. Accu acy) pe classi ie o e he aining/ es ing CEDEL2 da ase
5.1 In e p e a ion
Fea u e impo ance e e s o echniques ha assign a sco e o inpu ea u es based
on how use ul hey a e a p edic ing a a ge a iable. Addi ionally, ea u e im-
po ance sco es p o ide insigh in o he da a and he models, imp o ing he e iciency
and e ec i eness o he p edic i e models.
Pe mu a ion ea u e impo ance (PFI) is a echnique o calcula ing ela i e im-
po ance sco es ha is independen o he model used [12]. Fi s , a model is i on he
1
CEDEL2 s ands o Co pus Esc i o del Español como L2 (L2 Spanish W i en Co pus).
7
da ase , such as a model ha does no suppo na i e ea u e impo ance sco es. Then
he model is used o make p edic ions on a da ase , al hough he alues o a ea u e
(column) in he da ase a e sc ambled. This is epea ed o each ea u e in he da ase .
Then, his whole p ocess is epea ed a numbe o imes. The esul is calcula ed as a
mean o impo ance sco e o each inpu ea u e (and dis ibu ion o sco es gi en he
epea s). This app oach can be used o eg ession o classi ica ion and equi es ha a
pe o mance me ic be chosen as he basis o he impo ance sco e, such as he mean
squa ed e o o eg ession and accu acy o classi ica ion.
In e ms o explaining any black-box model, he SHapley Addi i e exPlana ions
(SHAP) me hod is, by a , he mos comp ehensi e and dominan ac oss he li e a u e
me hods o isualizing ea u e in e ac ions and ea u e impo ance [13]. The SHAP
me hods a e no only model-agnos ic, bu hey ha e been demons a ed o be applica-
ble o any ype o da a. The SHAP alues ep esen he con ibu ion o each ea u e o
he p edic ion o each indi idual ins ance. To compu e global ea u e impo ance
based on SHAP alues, he mean absolu e SHAP alue o each ea u e ac oss all
ins ances should be es ima ed.
In Fig. 3 compa isons be ween PFI and SHAP alues (in %) a e illus a ed pe
classi ie o e he CEDEL2 da ase . I should be no ed ha he a ibu es ha e been
posed in descending o de acco ding o hei SHAP alues. The indings e eal ha
he 'Yea s s udying Spanish' and 'Age' consis en ly eme ge as he mos in luen ial
ea u es ac oss Decision T ees (CART), Random Fo es , Bagging and ke nel (SVM)
classi ie s, as hei SHAP and PFI alues a e signi ican ly highe compa ed o he
alues o o he a ibu es. Fo hose classi ie s, he o de ing o impo ance o he a -
ibu es is qui e simila compa ing he SHAP and PFI alues. Howe e , he magni ude
o he con ibu ion o hese ea u es a ies depending on he model used, indica ing
he in insic di e ences be ween he machine lea ning algo i hms. Sligh di e ences
we e exhibi ed o he ca ego ies o he a ibu es ha a e ew ep esen a i es in he
da ase such as Languages spoken a home Japanese o Mo he s na i e language Po -
uguese e c.
Howe e , he beha io o XAIs algo i hms using he linea classi ie s LR and LDA
is qui e di e en . Fea u es such as 'Languages spoken a home: Japanese' and 'Mo h-
e s na i e language: Japanese' also show conside able impo ance, especially in he
Logis ic Reg ession model, sugges ing po en ial con ex -speci ic ele ance o hese
ea u es. Ne e heless, hese ea u es don' appea o be uni e sally signi ican ac oss
all models. Ano he po en ial explana ion o his could be he ac ha non-linea
na u e o he p oblem. Hence, hose classi ie s could no i adequa ely in o he da ase
and hei pe o mance is low. In conclusion, his analysis unde sco es he impo ance
o unde s anding and in e p e ing ea u e impo ance in model de elopmen and high-
ligh s he a iabili y ha can exis depending on he algo i hm and in e p e a ion
me hod u ilized.
On he o he hand, ee and ke nel-based classi ie s, such as CART, SVM, RF, e c.
can adequa ely cap u e he non-linea na u e o he da a, hence he PFI and SHAP can
be e es ima e and hie a chy o he impac o he ea u es and also exhibi a cons an
beha io . Hence, o hose models, he a ibu es ‘Yea s s udying Spanish’ and ‘Age’
play a signi ican ole in co ec ly classi ying he s uden s.
8
9
