Employing la ge language
models o emo ion de ec ion in
psycho he apy ansc ip s
Ch is ophe Lalk
1
*, Kim Ta gan
1
, Tobias S einb enne
1
,
Jana Scha a h
2
, S e en Ebe ha d
2
, B ian Schwa z
2
,
An onia Vehlen
2
, Wol gang Lu z
2
and Julian Rubel
1
1
Depa men o Psychology, Osnab ück Uni e si y, Osnab ück, Ge many,
2
Depa men o Psychology,
Uni e si y o T ie , T ie , Ge many
Pu pose: In he con ex o psycho he apy, emo ions play an impo an ole bo h
h ough hei associa ion wi h symp om se e i y, as well as hei e ec s on he
he apeu ic ela ionship. In his analysis, we aim o ain a la ge language model
(LLM) o he de ec ion o emo ions in Ge man speech. We wan o apply his
model on a co pus o psycho he apy ansc ip s o p edic symp om se e i y and
alliance aiming o iden i y he mos impo an emo ions o he p edic ion o
symp om se e i y and he apeu ic alliance.
Me hods: We employed a public labeled da ase o 28 emo ions and ansla ed
he da ase in o Ge man. A p e- ained LLM was hen fine- uned on his da ase
o emo ion classifica ion. We applied he fine- uned model o a da ase
con aining 553 psycho he apy sessions o 124 pa ien s. Using machine lea ning
(ML) and explainable a ificial in elligence (AI), we p edic ed symp om se e i y
and alliance by he de ec ed emo ions.
Resul s: Ou fine- uned model achie ed modes classifica ion pe o mance
(F1
mac o
=0.45, Accu acy=0.41, Kappa=0.42) ac oss he 28 emo ions.
Inco po a ing all emo ions, ou ML model showed sa is ying pe o mance o
he p edic ion o symp om se e i y ( = .50; 95%-CI:.42,.57) and mode a e
pe o mance o he p edic ion o alliance sco es ( = .20; 95%-CI:.06,.32). The
mos impo an emo ions o he p edic ion o symp om se e i y we e app o al,
ange , and ea . The mos impo an emo ions o he p edic ion o alliance we e
cu iosi y, con usion, and su p ise.
Conclusions: E en hough he classifica ion esul s we e only mode a e, ou
model achie ed a good pe o mance especially o p edic ion o symp om
se e i y. The esul s confi m he ole o nega i e emo ions in he p edic ion o
symp om se e i y, while hey also highligh he ole o posi i e emo ions in
os e ing a good alliance. Fu u e di ec ions en ail he imp o emen o he labeled
da ase , especially wi h ega ds o domain-specifici y and inco po a ing con ex
in o ma ion. Addi ionally, o he modali ies and Na u al Language P ocesssing
(NLP)-based alliance assessmen could be in eg a ed.
KEYWORDS
na u al language p ocessing, compu a ional psycho he apy esea ch, machine lea ning,
explainable a ificial in elligence, symp om se e i y, alliance, p ocess-ou come- esea ch
F on ie s in Psychia y on ie sin.o g01
OPEN ACCESS
EDITED BY
Ke s in Denecke,
Be n Uni e si y o Applied Sciences,
Swi ze land
REVIEWED BY
Ma k Melnykowycz,
IDUN Technologies AG, Swi ze land
Eb ahim Ghade pou ,
Sapienza Uni e si y o Rome, I aly
*CORRESPONDENCE
Ch is ophe Lalk
[email p o ec ed]
RECEIVED 30 Sep embe 2024
ACCEPTED 14 Ap il 2025
PUBLISHED 09 May 2025
CITATION
Lalk C, Ta gan K, S einb enne T, Scha a h J,
Ebe ha d S, Schwa z B, Vehlen A, Lu z W
and Rubel J (2025) Employing la ge language
models o emo ion de ec ion in
psycho he apy ansc ip s.
F on . Psychia y 16:1504306.
doi: 10.3389/ psy .2025.1504306
COPYRIGHT
© 2025 Lalk, Ta gan, S einb enne , Scha a h,
Ebe ha d , Schwa z, Vehlen, Lu z and Rubel.
This is an open-access a icle dis ibu ed unde
he e ms o he C ea i e Commons A ibu ion
License (CC BY). The use, dis ibu ion o
ep oduc ion in o he o ums is pe mi ed,
p o ided he o iginal au ho (s) and he
copy igh owne (s) a e c edi ed and ha he
o iginal publica ion in his jou nal is ci ed, in
acco dance wi h accep ed academic
p ac ice. No use, dis ibu ion o ep oduc ion
is pe mi ed which does no comply wi h
hese e ms.
TYPE O iginal Resea ch
PUBLISHED 09 May 2025
DOI 10.3389/ psy .2025.1504306
In oduc ion
Emo ions can be concep ualized as “biologically based eac ions
ha coo dina e adap i e esponding o impo an oppo uni ies
and challenges”(1, p. 152). The e a e quali a i e di e ences
be ween emo ions based on he oppo uni y o challenge ha he
emo ion has been e ol ed o add ess, as well as i s exp ession (2).
Fu he , emo ions can immedia ely impac beha io , quickly
egula ing one’s beha io based on he si ua ion (3). Fo ins ance,
ange may be an adap i e eac ion o an un ai ea men (4),
p omp ing he pe son a which i is a ge ed o change hei
beha io acco ding o he demand. Simul aneously, ange comes
wi h specific acial ea u es (5) and associa ed beha io s, such as
speaking o ce ully o e en yelling.
Since emo ions a e an essen ial pa o daily unc ioning, i is
impo an ha indi iduals a e able o egula e hem, ha is,
modula e he emo ional expe ience and i s exp ession (1). People
who lack his abili y, may be comp omised in se e al domains o
li e, which could nega i ely a ec men al heal h (1,6). The e o e, i
comes as no su p ise ha many men al heal h diso de s a e
associa ed wi h emo ion egula ion defici s (7).
Because o he impo ance o emo ion egula ion p ocesses
ac oss a ious men al heal h diso de s, emo ions play an essen ial
ole in psycho he apy (8). Mos impo an ly, hey a e associa ed
wi h symp om se e i y in mood and anxie y diso de s (9–11). E en
hough, emo ions and a ec can be di e en ia ed (e.g., a ec is
longe in du a ion, is less in en ional, ends o ha e unknown
causes, and has lowe in ensi y), he e is subs an ial o e lap
be ween bo h cons uc s (9). Fo ins ance, he co ela ion be ween
ea (emo ion) and anxie y (a ec ) was calcula ed as = .72,
amoun ing o mo e han 50% o sha ed a iance (10). The e o e,
he measu emen o someone’s emo ions can p o ide an es ima e o
hei a ec . Simul aneously, dys egula ed a ec is an impo an
ea u e in anxie y and dep ession (11), which a e cha ac e ized by
excessi e nega i e (NA) and a lack o posi i e a ec (PA). Gene ally,
a ec is sensi i e o change: In a me a-analysis, psycho he apy o
dep ession has been shown o dec ease NA and inc ease PA (12).
Acco ding o he b oaden-and-buil heo y, PA helps o
s eng hen and build esou ces by ex ending one’s hough -ac ion
epe oi e (13). Fo ins ance, PA comes wi h he u ge o play, explo e,
sa o , and connec , all o which can os e esou ces h ough he
c ea ion o new oppo uni ies and s eng hening o ela ionships.
This, in u n, aises wellbeing and PA (13). Consequen ly, i is no
su p ising ha PA can p o ec agains dep ession (14), may media e
dep ession eco e y (15), and ha lack o PA is associa ed wi h
ypical dep ession symp oms, such as sadness, loss o in e es , li le
ene gy, and apa hy (16,17). Con e sely, NA is linked wi h s ess
le els (18) and dep essi e symp oms (19). NA is a be e p edic o o
dep ession de e io a ion han PA, di e en ia es be e be ween
dep essed and heal hy indi iduals (20), and p edic s u u e
dep ession onse while PA does no (21). The e o e, NA is likely a
be e ma ke o symp om se e i y han PA.
Beyond hei associa ion wi h symp om se e i y, emo ions play
ano he c ucial ole in he con ex o psycho he apy h ough hei
impac on he he apeu ic alliance, which is one o he bes p edic o s
o success ul ea men (22). Human beings a e undamen ally social
c ea u es and emo ions a e an essen ial mechanism o he egula ion
o social ela ionships (23). Much o he adap i e s eng h o
emo ions is media ed by hei e ec s on in e pe sonal unc ioning
so ha hey can be desc ibed as in insically in e pe sonal (24):
Emo ion exp ession helps people o ecognize ou needs and
wishes, allowing hem o suppo us and s eng hening he mu ual
bond. Howe e , posi i e in e pe sonal e ec s o emo ion exp ession
a e no au oma ically gi en, bu depend on se e al c ucial ac o s
(23). In pa icula , emo ion egula ion plays an impo an ole: Fo
ins ance, indi iduals wi h high le els o NA epo mo e di ficul and
less sa is ying oman ic ela ionships (25). Though nega i e emo ions
can play an adap i e ole as well, he excessi e exp ession o NA can
be de as a ing o oman ic ela ionships, because i pe pe ua es a
ecip ocal spi al, om which i is di ficul o disengage (26). Simila ly,
he exp ession o nega i e sen imen owa ds he he apis o he
he apeu ic si ua ion is associa ed wi h lowe le els o alliance (27,
28). Dep essed mood is associa ed wi h lowe le els o emo ion
egula ion, which can lead o ange managemen di ficul ies, educed
us and o gi eness, heigh ened le els o social compa ison, as well
as social wi hd awal, all o which can impai social unc ioning and
he e o e ha m he he apeu ic ela ionship (29). In summa y, pa ien
emo ions associa ed wi h ei he wi hd awal o con on a ion and
c i icism may be pa icula ha m ul o he alliance (30). Fo example,
pa ien s wi h a lo o shame end o wi hd aw, which leads o nega i e
e ec s on he alliance (31), while ange , hos ili y, and us a ion can
impai he alliance ia con on a ion and c i icism (27,28).
Con a y, posi i e emo ions se e impo an social unc ions,
whichcanimp o e he he apeu icalliance(32–35). Mos
impo an ly, hey can inc ease in imacy and emo ional bond, as
well as enhance mo i a ion o achie e sha ed goals (35), bo h o
which a e pilla s o he wo king alliance (36). These conside a ions
a e confi med h ough longi udinal s udies, ha ha e shown
bidi ec ional e ec s be ween posi i e emo ions and alliance (32,
34). Though we did no find esul s ega ding he ela i e s eng h
o nega i e and posi i e emo ions on he he apeu ic ela ionship, i
is likely ha nega i e emo ions may ha e he g ea e impac , since
nega i e e en s end o ha e g ea e e ec s on mos a eas o li e,
including in e pe sonal ela ions (37).
Emo ions can be assessed ia many di e en means, including
ideo, audio, elec oencephalog aphy ( o an o e iew see 38),
elec omyog aphy, a ious o he physiological measu es (e.g., hea
a e, blood p essu e), o a combina ion o se e al modali ies (39–41).
Di e en me hods ha e been success ully employed depending on he
da a sou ce. Fo elec oencephalog aphy (see 38), ea u es can be
employed o machine lea ning om di e en domains, such as he
ime, equency, o bo h. Addi ionally, employing deep lea ning, he
aw da a can be used wi hou ea u e enginee ing. Fo ins ance, using
equency ea u es, good accu acy (>80%) has been achie ed bo h o
he classifica ion o alence and a ousal (42). Simila app oaches a e
possible o o he physiological da a (39). Fo ideo emo ion
classifica ion, deep lea ning models show compe i i e pe o mance
wi h a con olu ional neu al ne wo k achie ing 66% accu acy in a acial
classifica ion (43). In he oice domain, a combina ion o con olu ional
neu al ne wo ks and a ans o me a chi ec u es (wa 2 ec 2.0; 44)
Lalk e al. 10.3389/ psy .2025.1504306
F on ie s in Psychia y on ie sin.o g02
shows s a e-o - he-a pe o mance ac oss a ious asks (45). Fo ex
da a, cu en models success ully employ a ans o me a chi ec u e
(e.g., 46).
Howe e , be e esul s can be achie ed o mul imodal models.
Fo ins ance, au oma ic classifica ion o basic emo ions based on ex ,
speech, and ideo in a hidden Ma ko model achie ed good accu acy
in an expe imen al se ing (47). Rega ding a na u alis ic
psycho he apy se ing, emo ions we e mainly assessed ia
ques ionnai e measu es (48) wi h no able excep ions, whe e
emo ions we e judged by human a e s (e.g., 49,50). Howe e ,
bo h o hese app oaches ha e d awbacks: Ques ionnai e measu es
can be bu densome o pa ien s and a e unable o ack emo ions o e
he cou se o a session. While human a e s can indeed ack
emo ions o e he session, his is e y ime-consuming, so ha i is
di ficul o apply o la ge session da ase s. Wi h he eme gence o
a ificial in elligence (AI) and na u al language p ocessing (NLP),
new app oaches o he au oma ic analyses o la ge language co po a
a e a ailable (51). NLP has al eady shown p omising success in he
iden ifica ion o he apis skills (52), mo i a ional in e iew
adhe ence (53), o ele an session hemes (54,55). In he con ex
o emo ion de ec ion, M. Tanana e al. (56) ained uni-, bi-, and
ig am
1
models on he de ec ion o sen imen on 100,000 a ed
u e ances om psycho he apy ansc ip s. This wo k was la e
ex ended by he inclusion o he ans o me model Bidi ec ional
Encode Rep esen a ions and T ans o ma ions (BERT; 57)anda
model based on posi i e and nega i e a ec i e wo ds om he
Linguis ic Inqui y and Wo d Coun (LIWC; 58). In his analysis,
he BERT model showed he bes pe o mance (59). Mo e ecen ly,
Ebe ha d e al.(60) alida ed he pe o mance o ano he
ans o me model on a se o 85 ansc ip s. They ound
significan co ela ions be ween au oma ically calcula ed sen imen
and pa ien - and he apis - epo ed emo ions. Fu he , symp om
se e i y was significan ly associa ed wi h nega i e sen imen .
While hese esul s p o ide e idence o he eliabili y and
alidi y o sen imen analysis, hey a e es ic ed o he alence
dimension, classi ying all u e ance on a single dimension om
nega i e o posi i e. Though he alence dimension is highly
ele an in his con ex , a leas six basic emo ions can be
dis inguished wi h addi ional a ec s a es (2), ha can be
o ganized ac oss mul iple dimensions (e.g., alence, in ensi y,
in en ionali y, du a ion) and mul iple ca ego ies (e.g., causes,
unc ion, mimic, beha io ).
Objec i es
The e o e, we aim o fine- une a la ge language model (LLM)
o a mo e fine-g ained analysis o emo ions in he Ge man
language. We wan o show he clinical u ili y o his app oach by
applying his model o a da ase o psycho he apy sessions o
p edic symp om se e i y and alliance, employing explainable AI
o iden i y he mos impo an emo ions o he p edic ion o bo h.
Hypo heses
We expec ou fine- uned Ge man model o accu a ely cap u e
emo ions in he ansc ip , allowing o a p edic ion o pa ien
symp om se e i y. Rega ding symp om se e i y, we expec nega i e
emo ions o p edic highe symp om se e i y and posi i e emo ions
o p edic lowe symp om se e i y. Fu he , we expec nega i e
emo ions o ha e a highe impac on he p edic ion o symp om
se e i y han posi i e emo ions. Rega ding he alliance, we expec
posi i e emo ions o p edic be e alliance. Fo nega i e emo ions,
we expec lowe le els o alliance in gene al, hough pa icula ly o
emo ions associa ed wi h wi hd awal (such as emba assmen and
con usion) o con on a ion (such as ange and disapp o al). Again,
we expec ed nega i e emo ions o ha e a highe impac on he
alliance sco es.
Me hods
Pa ien s and he apis s
Ou da ase con ained 124 pa ien s (65.8% emale) who had
ecei ed ea men a an uni e si y ou pa ien clinic in T ie ,
Ge many. On a e age, pa ien s we e 38.8 yea s (SD = 12.7) old.
Rega ding he socioeconomic s a us, almos all had ei he finished a
seconda y school ce ifica e (51.6%) o hei A-le els (42.1%). Mos
had finished an app en iceship (41.3%), while 18.2% we e cu en ly
in aining o s udying and 11.6% had a uni e si y deg ee. All
pa ien s unde wen a diagnos ic in e iew employing he S uc u ed
Clinical In e iew o Axis I DSM-IV Diso de s-Pa ien Edi ion
(SCID-I; 61). They we e mos ly diagnosed wi h p ima y diagnoses
o a ec i e diso de s (n= 56), anxie y diso de s (n= 24), and
auma and adjus men diso de s (n= 16). On a e age, hey
ecei ed 2.3 (SD = 1.3, min =1,max = 5) como bid diagnoses.
The ea men was conduc ed by 47 he apis s wi h a
psychology mas e deg ee. All he apis s had a leas one yea o
p io ea men expe ience and we e ei he al eady licensed CBT
he apis s o cu en ly en olled in aining. They ecei ed
supe ision egula ly.
T ea men
The ea men consis ed o weekly CBT sessions. While he fi s
wo sessions se ed diagnos ic pu poses (ini ial assessmen in session
1 and SCID-I in e iew in session 2), he ea men began in he hi d
session. On a e age, pa ien s ecei ed 35.7 (SD = 19.7) sessions.
Ins umen s and measu es
Symp om se e i y
P io o each session, symp om se e i y was assessed ia he
Hopkins Symp om Checklis -11 (HSCL-11; 62). The HSCL-11 is an
11-i em sel - epo scale abou gene al psychological dis ess. Pa ien s
1 N-g ams e e o se s o nconsecu i e wo ds in a co pus.
Lalk e al. 10.3389/ psy .2025.1504306
F on ie s in Psychia y on ie sin.o g03
a ed a lis o 11 symp oms ( ea ulness, anxiousness, agi a ion, panic,
sleep p oblems, hopelessness, loneliness, low mood, lack o in e es ,
suicidal idea ion, and wo hlessness) on a Like - ype scale om 1
(no a all) o4(ex emely). Symp om se e i y was hen calcula ed as
he mean sco e on hese i ems. The HSCL-11 con ains a dep ession
and anxie y subscale and is highly co ela ed wi h a ious o he
anxie y and dep ession ques ionnai es (63). Fo ins ance, high
associa ions ha e been ound o he B ie Symp om In en o y
(BSI; 64; = .91) and i s subscales o anxie y ( = .82) and
dep ession ( = .91). Rega ding dep ession, high co ela ions ha e
also been ound o he Pa ien Heal h Ques ionnai e-9 (PHQ-9; 65;
= .81). Fo wo y symp oms, high co ela ions exis wi h he
Gene alized Anxie y Diso de 7 (GAD-7; 66; =.72).TheHSCL-
11 has shown good sensi i i y o change (67). Assessing he hi d
session o each pa ien in his sample, we also ound good in e nal
consis ency (w= .92) acco ding o McDonald’somega(68). Ac oss all
sessions in ou da ase , a e age symp om se e i y was a 2.2 (SD =
0.79) on a scale om 1 o 4.
Alliance
The apeu ic alliance was assessed by he pa ien ia he Session
Ra ing Scale (SRS; 69). The pa ien s filled he ques ionnai e a e each
session. The SRS con ains he concep ualiza ion by Bo din (36)o
h ee alliance componen s: 1. A ec i e bond, 2. Goal ag eemen , 3.
Task ag eemen , which a e assessed by h ee i ems. The SRS con ains
an addi ional ou h i em, which eflec s he o e all alliance. The final
sco e is calcula ed as a mean ac oss all ou i ems. The SRS has shown
gene ally sa is ying eliabili y, anging om a= .70 o.97 and a es -
e es eliabili y be ween = .54 and.70 (70). In ou sample, he SRS
showed a good in e nal consis ency in he hi d session (w=.83).
Co ela ions wi h o he alliance measu es ha e been mode a e
(HAQ-II; =.48;69;WAI; =.57–.65; 71).
T ansc ip s
The ansc ip s co pus consis ed o 553 ansc ip s o
psycho he apy sessions. On a e age, he e we e 4.5 (SD = 4.9)
ansc ip s pe pa ien , gene ally s a ing wi h session 3 and
con inuing wi h e e y fi h session (e.g., 3, 5, 10, …).
T ansc ip ion was conduc ed wi hou he use o ansc ip ion
so wa e by psychology s uden s based on he session eco dings.
Names o pe sons o places we e emo ed o educe iden ifiable
in o ma ion. The ansc ip s con ained spa se anno a ions abou
non e bal beha io o in e up ion in pa en heses. Each
ansc ip was o ganized as a able o consecu i e speech u ns
by he apis and pa ien . The ansc ip s we e no labelled o
emo ions. Fo ou analysis, we e ained only he pa ien s’speech
u ns, lea ing 104,557 speech u ns. We u he spli hese speech
u ns in o sen ences o he model in e ence and e ained all
sen ences wi h a leas h ee wo ds, lea ing 233,648 sen ences.
A e age sen ence leng h was 12.7 (SD = 10.4) wo ds. Pe session,
pa ien s spoke on a e age 422.5 (SD = 174.1) sen ences,
amoun ing o 5,362.6 (SD = 2,089.3) wo ds.
GoEmo ions da ase
We used he GoEmo ions da ase (72) as ou labeled aining
da a. The da ase consis s o 54k labeled commen s. All commen s
we e aken om Reddi , while excluding o ensi e, ulga , eligion,
and iden i y wo ds. Commen s had a leng h be ween 3 and 30
wo ds and we e balanced ac oss he sen imen , di e en emo ions,
and sub eddi s. The commen s we e anno a ed by h ee a e s
ac oss a lis o 28 emo ions (o espec i ely 27 emo ions and a
‘neu al’ca ego y). I easonable, mul iple labels could be gi en o a
single commen . In e a e ag eemen was assessed ia Cohen’s
kappa by Demszky e al. (72), anging be ween 0.331 (g ie ) and
0.468 (admi a ion). As is shown in Figu e 1, he pos s we e no well
balanced ac oss all emo ions wi h especially neu al and admi a ion
and app o al being he mos p e alen ones, while elie , p ide and
g ie we e e y a e.
XLM-RoBERTa-base language model
We employed he XLM-RoBERTa-base model (73) as ou base
LLM, which we fine- uned on he da ase . The XLM-RoBERTa-base
model was ained on a masked mul ilingual da ase o o e 2
e aby e (o which 66.6 gigaby e we e in Ge man), which allows he
model o pe o m in o e 100 languages. The model has an
embedding size o 1024 okens, co esponding o a maximal inpu
con ex leng h o abou 600–800 wo ds. The XLM-RoBERTa-base
model was p e ained wi h a Masked Language Modeling objec i e
by p edic ing 15% o andomly masked wo ds o lea n a
bidi ec ional ep esen a ion o he sen ence. Since i was only
ained on aw ex s wi hou human labeling, i is in ended o be
fine- uned on a downs eam ask, such as he classifica ion ask ha
is con ained in his pape .
Da a analy ic s a egy
The analyses we e conduc ed wi h Py hon 3.9. The comple e
wo kflow is shown in Figu e 2 and is elabo a ed below.
P e-p ocessing and la ge language model fine-
uning
We selec ed he 54k labeled commen s om he aw GoEmo ions
da ase and used au oma ic ansla ion o Ge man ia DeepL (74).
The da ase was spli in o 80% aining se , 10% alida ion se , and
10% es se . Emo ions we e classified o a commen ia one-ho
encoding
2
.We henfine- uned he XLM-RoBERTa-base model on
he da a in a mul ilabel classifica ion ask (i.e., se e al emo ions could
be classified o one sen ence) wi h a ba ch size o 16, lea ning a e o
3e–5, and weigh decay o 0.01. The comple e aining code can be
accessed ia he OSF eposi o y (75). T aining was conduc ed o 10
epochs in he da ase o ma , which allows o as e p ocessing speed
(76). The loss unc ion was Bina y C oss-En opy (BCE) wi h logi s
because o he mul ilabel implemen a ion.
Lalk e al. 10.3389/ psy .2025.1504306
F on ie s in Psychia y on ie sin.o g04
The final model showed simila me ics in he es se (F1
mac o
=
0.45, Kappa
mac o
= 0.42, Accu acy = 0.41) as he o iginal model
(F1
mac o
= 0.46) by Demszky e al. (72). Cohen’s Kappa calcula ed
be ween p edic ed emo ions and labeled emo ions anged be ween
0.15 and 0.88 wi h a mean o 0.42, indica ing mode a e ag eemen
(77). The con usion ma ix in he final es se is shown in Figu e 3.
La ge language model in e ence
A e he fine- uning was comple ed, we applied he model o
he emo ion classifica ion in ou da ase . As he labeled da a had a
leng h o 3 o 30 wo ds, we decided o conduc he classifica ion on a
sen ence le el. Each sen ence by a pa ien was hen un h ough he
model pipeline and was classified ac oss he 28 di e en emo ions
by assigning each emo ion a p obabili y be ween 0 and 1 o each
sen ence. Fo example, i a sen ence was gi en a p obabili y o 0.8
o admi a ion, his could be in e p e ed as he p edic ed
p obabili y o he p esence o admi a ion in he sen ence.
The e o e, highe alues co espond o a highe p obabili y o
he p esence o a gi en emo ion.
E alua ion s a egy o he p edic ion o ou come
Fo he p edic ion o ou come (i.e., alliance and symp om
se e i y), he au oma ically classified emo ion p obabili ies we e
agg ega ed a a session le el by calcula ing he mean o each
emo ion pe session. Since he symp om se e i y measu e
con ained a dep ession and an anxie y subscale, we conduc ed
wo sensi i i y analyses, p edic ing he espec i e subscale. The 28
agg ega ed emo ion p obabili ies we e he ea u es o he
p edic ion. Employing nes ed c oss- alida ion, se e al machine
lea ning algo i hms compe ed agains each o he in an in e nal
fi e- old c oss- alida ion, while only he bes pe o ming algo i hm
was selec ed as he algo i hm o choice o he espec i e es se in
he ex e nal en- old c oss- alida ion ia he py hon lib a y XRAI
(78). We e alua ed he model pe o mance ia co ela ion ( ; 79),
no malized oo mean squa ed e o (NRMSE; 80), and mean
absolu e e o (MAE; 81). Fo he NRMSE, no maliza ion was
conduc ed by di iding h ough he s anda d de ia ion o he
a ge a iable. Confidence in e als we e calcula ed by
boo s apping ac oss he en es olds.
Machine lea ning algo i hms
In o de o achie e good p edic ion me ics, we chose a di e se
se o ML algo i hms o accoun o ea u e in e ac ions, nonlinea
e ec s and collinea i y. The ollowing machine lea ning algo i hms
compe ed agains each o he in he in e nal c oss- alida ion: 1.
Leas Absolu e Sh inkage and Selec ion Ope a o (Lasso; 82), 2.
Elas ic ne egula iza ion and a iable selec ion (Elas ic Ne ; 83), 3.
eX eme G adien Boos ing (XGBoos ; 84), 4. Random Fo es (RF;
85), 5. Suppo Vec o Reg ession (SVR; 86), and 6. Supe Lea ne
(87). The Supe Lea ne in eg a es he ensemble o p e ious lea ne s
(i.e., Lasso, Elas ic Ne , XGBoos , RF, and SVR), using hei
p edic ed sco es as ea u es o an SVR me a-lea ne .
2 One-Ho encoding is he ans o ma ion o ca ego ial da a in o bina y
ec o s o each ca ego y. Con a y o dummy encoding, he e is no
e e ence ca ego y.
FIGURE 1
Ba plo o he emo ion equency in he ain se o GoEmo ions.
Lalk e al. 10.3389/ psy .2025.1504306
F on ie s in Psychia y on ie sin.o g05

Ou o hese six di e en algo i hms, he machine lea ning
algo i hm wi h he bes mean co ela ion in he in e nal fi e- old
c oss- alida ion was selec ed as he algo i hm o he ex e nal es
old. The e o e, i could be possible ha di e en algo i hms we e
selec ed ac oss he en es olds, e.g., fi e imes XGBoos , h ee
imes RF, and wo imes SVR.
Model explana ion
In gene al, machine-lea ning models a e no e y well
explainable due o hei complex and nonlinea modeling (88).
Howe e , one p oposed solu ion has been he use o Shapley alues
(89), which allow o an es ima ion o ea u e impac . To his end,
we employed he py hon package SHAP (SHapley Addi i e
exPlana ions; 90,91). SHAP allows o he assessmen o he
indi idual ea u e impac (i.e., how much a e he p edic ed alues
influenced by his ea u e)?, as well as he impac o g oups o
ea u es. Fu he , SHAP can be used o assess he di ec ion o a
ea u e impac (e.g., do highe ea u e alues p edic highe
ou come alues)?, as well as in e ac ions be ween ea u es.
Resul s
Desc ip i e s a is ics
The mean p obabili y o all emo ions and hei eliabili y is
p esen ed in Table 1. In addi ion, he alence o each emo ion is
indica ed (posi i e/nega i e/neu al). Al oge he , he model
classified 14 posi i e emo ions, 12 nega i e emo ions and 2
neu al emo ions. The mos p obable emo ions in he ansc ip
co pus we e (wi h he excep ion o neu al)app o al (12.98%),
disapp o al (6.11%), con usion (4.32%), and ealiza ion (4.14%).
The leas p obable emo ions we e g ie (0.11%), emo se (0.27%),
amusemen (0.73%), and elie (0.58%). Rega ding he eliabili y o
he assessmen , we p o ided he F1 me ic and Cohen’s Kappa (77)
om he es se in he GoEmo ions da ase . F1 is he ha monic
mean be ween p ecision (how o en is he model co ec when i
p edic s he emo ion)? and ecall (how o en does he model de ec
he emo ion when i ac ually occu s)?. Kappa can be in e p e ed
acco ding o (77) as ai ag eemen (>.2), mode a e ag eemen (>.4),
subs an ial ag eemen (>.6), and almos pe ec ag eemen (>.8).
Reliabili y was subs an ially high o some posi i e emo ions (e.g.,
admi a ion, amusemen , g a i ude, and lo e), while he nega i e
emo ions showed mode a e ag eemen a bes (e.g., ea , emo se,
and sadness).
To gi e some imp ession abou he labeled s a emen s om
psycho he apy ansc ip s, we compa ed some pa ien s a emen s
ha we e classified by he LLM wi h o iginal commen s om he
GoEmo ions da ase o di e en emo ions (see Table 2). Due o
confiden iali y, he psycho he apy s a emen s a e om publicly
a ailable ansc ip s. We ha e p o ided a ull lis con aining each
emo ion in ou OSF (75).
P edic ion o symp om se e i y
The machine lea ning model con aining he 28 emo ions as
ea u es showed a good pe o mance wi h ( = .50 (95%-CI:.42,.57),
NRMSE = 0.87 (95%-CI:.83,.91) and MAE = .57 (95%-CI:.55,.59).
The selec ed lea ne s o he ex e nal c oss- alida ion we e RF (6x)
and SVR (4x). The mos impo an p edic o s (see Table 3)as
calcula ed acco ding o he ela i e SHAP alue we e app o al
(9.2%), sadness (8.7%), ea (7.8%), disappoin men (6.4%), desi e
(6.3%), and sadness (6.2%). We expec ed nega i e emo ions o p edic
highe symp om se e i y and posi i e emo ions o p edic lowe
symp om se e i y. This was gene ally accu a e, hough some posi i e
emo ions we e associa ed wi h highe symp om se e i y, namely
desi e, p ide, ca ing, amusemen ,andlo e. Simul aneously, no
nega i e emo ion was associa ed wi h less symp om se e i y.
Fu he , we expec ed nega i e emo ions o ha e a s onge impac
on symp om se e i y han posi i e emo ions. Al oge he , app o al,
admi a ion, op imism, ealiza ion, exci emen , g a i ude,and elie
we e posi i e emo ions significan ly associa ed wi h lowe symp om
se e i y, pe aining o an agg ega ed ea u e impac o 27.0%, while all
nega i e emo ions (ange , ea , disappoin men , sadness, ne ousness,
disapp o al, annoyance, emba assmen , con usion, disgus , g ie , and
emo se) we e significan ly associa ed wi h highe symp om se e i y
wi h almos wice he agg ega ed impac o nega i e emo ions
(51.8%), confi ming he hypo hesis. The emaining emo ions we e
no significan ly associa ed wi h symp om se e i y.
Sensi i i y analyses o he p edic ion o
he anxie y and he dep ession subscale
Rega ding he p edic ion o he anxie y ( = .47, 95%-CI: .40,
.55) and dep ession (.53, 95%-CI: .47, .59) subscales, good me ics
we e achie ed. A lis o all associa ed emo ions and hei ela i e
FIGURE 2
Comple e wo kflow o he da a analysis (blue ep esen s da a and
yellow he LLMs). The GoEmo ions da ase is ansla ed in o Ge man
and used o fine- une he XLM-RoBERTa LLM. The LLM is hen
employed o in e he emo ions om he he apy ansc ip s o
p edic he Rou ine Ou come Moni o ing (ROM) da a.
Lalk e al. 10.3389/ psy .2025.1504306
F on ie s in Psychia y on ie sin.o g06
impac can be ob ained om Supplemen a y Tables 1,2. Fo he
anxie y subscale, he emo ions wi h he highes posi i e impac we e
ea (17.3%), sadness (8.6%), and ne ousness (7.3%), while app o al
(10.4%) and admi a ion (6.1%) had he highes nega i e impac .
Rega ding dep ession, sadness (18.6%), g ie (8.0%) and
disappoin men (7.2%) had he highes posi i e and app o al
(17.7%) and ealiza ion (4.2%) he highes nega i e impac .
P edic ion o alliance
Fo he alliance p edic ion, he model pe o mance was low o
mode a e wi h ( = .20 (95%-CI:´.06,.32), NRMSE = 0.98 (95%-
CI:.95, 1.01) and MAE = 8.76 (95%-CI: 8.34, 9.22). Rega ding he
ex e nal es se lea ne s, Elas ic Ne (4x), as well as Lasso (3x) and
RF (3x) we e selec ed. The emo ions wi h he highes impac (see
Table 4) we e cu iosi y (24.9%), con usion (16.4%), and su p ise
(5.4%). We expec ed posi i e emo ions o p edic highe alliance
and nega i e emo ions o p edic lowe alliance. Fo nega i e
emo ions, his was no he case in gene al, since annoyance,
emo se, disgus , ne ousness, sadness, and emba assmen we e
no associa ed wi h lowe alliance. Though, con usion as a ma ke
o a wi hd awal up u e and ange and disapp o al as ma ke s o
con on a ion up u es p edic ed lowe alliance. Posi i e emo ions
we e mos ly associa ed wi h highe alliance wi h he no able
excep ion o cu iosi y and app o al. Some emo ions (e.g.,
op imism, ne ousness)we eno significan ly associa ed wi h
alliance. We expec ed nega i e emo ions (con usion, ange ,
disapp o al, ea , disappoin men , g ie ; 27.5%) o ha e a g ea e
nega i e impac on alliance han he posi i e impac o posi i e
emo ions (desi e, joy, admi a ion, exci emen , ealiza ion,
amusemen , elie , lo e, ca ing; 28.95%), which was no he case.
Discussion
Ou s udy se ed h ee pu poses: Fi s , we fine- uned an LLM
o he classifica ion o 28 emo ions in Ge man. Second, we
employed his model on a da ase o 553 psycho he apy
ansc ip s o p edic symp om se e i y and alliance. Thi d, we
assessed he mos impo an emo ions o he p edic ions o
symp om se e i y and alliance. Ou esul s indica e a modes
classifica ion pe o mance o ou fine- uned model wi h F1
mac o
= .45, which is almos iden ical o he o iginal pe o mance in
English (F1
mac o
= .46), indica ing no o e all accu acy loss due o
he ansla ion. Looking a indi idual emo ions, classifica ion
accu acy a ied sligh ly, bu was mos ly simila be ween Ge man
and English ( he la ges dec ease in F1
mac o
was .11). The Kappa
alue o .42 demons a ed mode a e ag eemen , which is no
su p ising because o he inhe en limi a ion o iden i ying
emo ions only ia he ansc ip modali y while igno ing o he
modali ies such as oice in ona ion (audio; e.g., 92)o acial
FIGURE 3
Con usion ma ix in he final es se . The colo g ading is exponen ially scaled.
Lalk e al. 10.3389/ psy .2025.1504306
F on ie s in Psychia y on ie sin.o g07
exp ession ( ideo; e.g., 93). The e o e, he modes pe o mance
likely did esul om he low in e - a e - eliabili y in he
GoEmo ions da ase (Cohen’s Kappa anging om .33 o .47).
Howe e , he e was conside able a ia ion ega ding he model’s
Kappa alues, anging be ween .15 and .88. This seems logical since
some emo ions may be mo e clea ly exp essed ia language con en
(e.g., admi a ion, amusemen , ea ) hano he s(e.g.,
disappoin men , annoyance), which migh be be e cap u ed ia
oice ea u es o mimic.
Employing he fine- uned model, we success ully p edic ed
symp om se e i y ( = .50) and alliance ( =.20) om he
ansc ip s. The accu acy was compa able o a di e en app oach
based on session con en as ope a ionalized by opic modeling o
250 opics (55). Emo ions showed highe associa ions wi h
symp om se e i y han a model based on 14 cogni i e dis o ions
( = .33) o nega i e sen imen only = .08, 94). In con as ,
Ebe ha d e al. (60) ound bo h wi hin-pe son co ela ions o
posi i e sen imen ( =−.29) and la ge be ween-pe son co ela ions
o nega i e sen imen ( = .66). Howe e , hese co ela ions mus
be in e p e ed wi h cau ion as hey come om a small da ase (N=
79) and may no be s able (95).
Rega ding symp om se e i y, nega i e emo ions mainly
p edic ed highe and posi i e emo ions lowe symp om se e i y,
as expec ed. Fo nega i e emo ions, he bes p edic o s we e ange
and ea , which bo h come wi h high a ousal. Ange may be
associa ed wi h he HSCL-11 i ems agi a ion, low mood and
eeling wo hless (as i o en was sel -di ec ed) while ea may be
associa ed wi h a ious i ems o he HSCL-11, such as ea ulness,
anxiousness, agi a ion, panic, and sleep p oblems. Su p isingly, he
posi i e emo ion app o al had he highes e ec , p obably because i
was simul aneously he mos o en de ec ed emo ion (13.0%).
Though con en -wise app o al was a uzzy concep wi h low
eliabili y (Kappa = .29), i ended o be associa ed wi h
s a emen s o ag eemen , easibili y, and accep ance, which can be
p o ec i e ac o s (14,96). Desi e was associa ed wi h highe
symp om se e i y, likely because i con ained s a emen s ha
indica ed a p esen us a ion o lack o some hing (e.g., desi ing
mo e sleep, a mo e accep ing pa ne , o happiness). Rega ding he
sensi i i y analysis o anxie y, we ound he highes impac o he
nega i e emo ions ea , sadness, and ne ousness, while app o al
TABLE 1 Mean p obabili y and s anda d de ia ions, as well as F1 and
kappa alues o each emo ion.
Emo ion (pos./
neg./neu al)
M SD F1 Cohen’s
Kappa
admi a ion (pos.) 2.35% 1.41% .64 .601
amusemen (pos.) 0.73% 0.66% .78 .767
ange (neg.) 0.88% 0.67% .38 .358
annoyance (neg.) 3.17% 1.32% .27 .229
app o al (pos.) 12.98% 4.40% .34 .293
ca ing (pos.) 0.79% 0.51% .38 .365
con usion (neg.) 4.32% 2.06% .40 .378
cu iosi y (pos.) 2.10% 1.42% .51 .486
desi e (pos.) 0.58% 0.49% .39 .387
disappoin men (neg.) 2.31% 1.03% .19 .170
disapp o al (neg.) 6.11% 2.02% .32 .286
disgus (neg.) 0.65% 0.45% .41 .395
emba assmen (neg.) 0.43% 0.41% .37 .367
exci emen (pos.) 0.87% 0.55% .35 .339
ea (neg.) 1.23% 1.09% .59 .584
g a i ude (pos.) 0.40% 0.34% .89 .882
g ie (neg.) 0.11% 0.19% .31 .307
joy (pos.) 1.68% 0.97% .51 .499
lo e (pos.) 0.75% 0.69% .73 .721
ne ousness (neg.) 0.91% 0.73% .28 .276
op imism (pos.) 1.43% 0.72% .53 .512
p ide (pos.) 0.24% 0.28% .30 .299
ealiza ion (pos.) 4.14% 1.66% .17 .150
elie (pos.) 0.58% 0.36% .27 .266
emo se (neg.) 0.27% 0.25% .55 .545
sadness (neg.) 1.92% 1.18% .50 .488
su p ise (neu al) 0.50% 0.40% .53 .514
neu al (neu al) 62.59% 6.13% .60 .410
pos., posi i e; neg., nega i e;M, mean; SD, s anda d de ia ion. F1, ha monic mean be ween
p ecision ( a io be ween co ec and o al p edic ions o he emo ion) and ecall ( a io
be ween co ec p edic ions and o al occu ence o he emo ion); Cohen’s Kappa, measu e o
model- a e - eliabili y. Mean and SD alues a e epo ed om he ansc ip co pus, while F1
and Cohen’s Kappa we e calcula ed in he GoEmo ions da ase .
TABLE 2 Example ph ases om GoEmo ions and om psycho he apy
ansc ip s o di e en emo ions.
Label GoEmo ions T ansc ip
admi a ion aw, hanks! I app ecia e ha ! Yes, I mo ed hem om one
wall o he o he and i looks
eally good.
ange Ok, hen wha he ac ual ** is
you plan?
He hen comes up and ies
o open hem and I’mso
ang y I don’ wan o alk.
emba assmen Ooooo . Tha ’s eal
awkwa d, bu I mean ha
somehow s ill ended be e
han I expec ed so. Kudos ig??
I mean, bu I’m so ashamed
o i .
ea I am a aid o look, bu my
mo bid cu iosi y d aws me
o ask.
I said, ‘We’ e all sca ed.’Bu
o my b o he , I said,
‘I’m sca ed.’
ealiza ion I ’s like you didn’ e en ead
he commen you’ e
esponding o.
I eally do no ice a di e ence.
sadness so pain ul o wa ch Bu ha was ha d o hea
oo, because when I was old
ha I was losing a oo h, I
jus s a ed c ying.
Sample s a emen s we e p o ided by publicly a ailable ansc ip a alexande s ee .com.
Lalk e al. 10.3389/ psy .2025.1504306
F on ie s in Psychia y on ie sin.o g08
was he posi i e emo ion wi h he highes impac . Fo dep ession,
app o al also was he mos impo an posi i e emo ion, while
sadness, g ie and disappoin men we e he mos impo an
nega i e emo ions.
Fo alliance, con a y o ou expec a ions, only ew nega i e
emo ions p edic ed lowe alliance, while mos posi i e emo ions
p edic ed highe alliance. Howe e , as expec ed, he nega i e
emo ions wi h significan associa ions migh ha e se ed as
wi hd awal (con usion) o con on a ion ma ke s (ange ,
disapp o al).Thenega i eassocia ionwi hcu iosi y was
su p ising, especially since i explained abou a qua e o he
model impac . Looking a he sen ences ha we e classified as
cu iosi y, i seemed as i hey some imes indica ed a
misunde s anding (e.g., “So his is om my diagnosis?”),
us a ion (“And you’ e a psychologis , o wha ?”), o con usion
(“Wha will happen o me hen?”). The posi i e impac o posi i e
emo ions was highligh ed by he associa ions wi h desi e, joy,
admi a ion, and exci emen in line wi h he li e a u e (35).
TABLE 3 Symp om se e i y p edic ion: emo ions, ela i e SHAP alue,
and co ela ion wi h SHAP alue.
Emo ion Symp om Se e i y
Rela i e
SHAP alue
Co ela ion wi h SHAP
alue (95%-CI)
app o al 9.16% -.63 (-.69, -.55)
ange 8.67% .86 (.81,.91)
ea 7.77% .88 (.84,.92)
disappoin men 6.42% .79 (.73,.84)
desi e 6.32% .90 (.85,.95)
sadness 6.24% .83 (.78,.88)
admi a ion 5.96% -.71 (-.79, -.64)
ne ousness 4.72% .79 (.72,.85)
disapp o al 3.76% .82 (.78,.86)
neu al 3.46% -.01 (-.38,.37)
annoyance 3.38% .89 (.84,.93)
op imism 3.29% -.78 (-.83, -.73)
ealiza ion 3.18% -.89 (-.91, -.86)
emba assmen 2.54% .75 (.67,.83)
con usion 2.43% .81 (.73,.87)
exci emen 2.42% -.56 (-.65, -.47)
disgus 2.12% .80 (.75,.85)
p ide 2.08% .73 (.65,.81)
g ie 1.98% .55 (.44,.67)
ca ing 1.92% .29 (.04,.51)
su p ise 1.90% -.60 (-.79, -.37)
emo se 1.78% .33 (.04,.57)
joy 1.62% -.18 (-.40,.02)
g a i ude 1.60% -.47 (-.70, -.22)
amusemen 1.40% .33 (.12,.53)
elie 1.38% -.51 (-.65, -.37)
cu iosi y 1.34% .14 (-.08,.35)
lo e 1.14% .39 (.20,.55)
TABLE 4 Alliance p edic ion: emo ions, ela i e SHAP alue, and
co ela ion wi h SHAP alue.
Emo ion Alliance
Rela i e
SHAP alue
Co ela ion wi h SHAP
alue (95%-CI)
cu iosi y 24.94% -.96 (-1.00, -.92)
con usion 16.41% -.93 (-1.00, -.85)
su p ise 5.42% .77 (.55,.95)
desi e 4.72% .84 (.65,.98)
joy 4.69% .61 (.35,.85)
admi a ion 4.61% .56 (.30,.81)
exci emen 4.35% .77 (.49,.98)
ealiza ion 4.02% .65 (.36,.90)
ange 3.85% -.37 (-.69, -.04)
disapp o al 3.36% -.35 (-.65, -.05)
amusemen 3.02% .62 (.35,.86)
annoyance 2.08% .49 (.22,.78)
emo se 1.94% .50 (.15,.80)
app o al 1.62% -.23 (-.46, -.05)
elie 1.57% .48 (.20,.77)
op imism 1.43% .05 (-.11,.28)
ea 1.40% -.31 (-.55, -.08)
disappoin men 1.34% -.53 (-.77, -.25)
g ie 1.08% -.06 (-.31,.25)
neu al 1.08% .16 (.01,.37)
lo e 1.07% .44 (.15,.73)
disgus 1.06% .40 (.15,.66)
ne ousness 1.02% .01 (-.21,.25)
ca ing 0.91% .53 (.26,.80)
g a i ude 0.88% .06 (-.13,.30)
sadness 0.88% .31 (.07,.61)
p ide 0.82% .07 (-.13,.31)
emba assmen 0.42% .12 (-.03,.33)
Lalk e al. 10.3389/ psy .2025.1504306
F on ie s in Psychia y on ie sin.o g09