Tuominen, T., Jiménez Hu ado, C., Ke ola, A. (2018). Why me hods ma e : App oaching mul imodali y in ansla ion esea ch.
Linguis ica An e piensia, New Se ies: Themes in T ansla ion S udies, 17, 1–21.
Why me hods ma e : App oaching mul imodali y in ansla ion
esea ch
Tiina Tuominen
Uni e si y o Glasgow, Uni ed Kingdom
[email p o ec ed]
Ca alina Jiménez Hu ado
Uni e si y o G anada, Spain
[email p o ec ed]
Anne Ke ola
Tampe e Uni e si y, Finland
[email p o ec ed]
The s udy o mul imodal phenomena calls upon ansla ion schola s o c oss disciplina y
bounda ies and adop a ange o heo e ical and me hodological app oaches. The
di e si y o he mul imodal landscape b ings abou esea ch challenges ha mus be
ca e ully add essed o ensu e ha hese esea ch e o s yield use ul and c edible esul s.
This special issue is dedica ed o a discussion on how o engage in mul imodal ansla ion
esea ch: how adi ional esea ch me hods can be adap ed and wha kinds o no el
app oaches can be adop ed o de eloped in o de o deal wi h a di e si y o mul imodal
da a. In his in oduc ion, we i s discuss de ini ions o mode and mul imodali y and
e lec on he na u e o mul imodali y as a opic o esea ch wi hin T ansla ion S udies.
We hen explain ou a ionale o dedica ing he special issue o esea ch me hods and
in oduce h ee a eas o mul imodal ansla ion esea ch ha , in ou iew, me i
pa icula a en ion om a me hodological poin o iew. Finally, we in oduce he
a icles con ained in his special issue.
1. In oduc ion
Communica ion in oday’s socie y has become p ima ily mul imodal: we encoun e
combina ions o modes – wo ds, images, sounds and so on – whene e we look a ou
compu e s, sma phones o ele ision sc eens. Mul imodali y is also highly ele an o
ansla ion: a signi ican numbe o he ex s being ansla ed nowadays a e mul imodal,
anging om use manuals, websi es, ex books and comics o audio isual p oduc s such
as ilms and ideogames. The ansla ion and he ansla ional enqui y o such ex s needs
o include a ca e ul conside a ion o bo h he indi idual meaning-making esou ces
in ol ed and he way hey ie oge he o o m a mul imodal en i y. Accessible ansla ion,
such as audio desc ip ion (AD), is also essen ially mul imodal, because i in ol es
ansla ing in o ma ion om one mode o ano he , om isually pe cei ed s imuli o
e bal in o ma ion.
Mul imodali y o e s an in iguing special case o me hodological deba e in
T ansla ion S udies, because i equi es us o ace up o he ask o ede ining wha we a e
esea ching. The dis inguishing ea u e o mul imodal ma e ials is he p esence and
in e ac ion o he di e en modes, and i is he e o e o en necessa y o place added
Tuominen, Jiménez Hu ado & Ke ola
2
emphasis in esea ch on he non- e bal elemen s o a mul imodal message ins ead o
ocusing exclusi ely on he e bal ex . T ansla ion S udies has a long adi ion o
explo ing e bal ex s, bu when o he modes a e in oduced in o he esea ch design, we
a e aced wi h his ques ion: How do we ind he op imal me hods o deal wi h hem all
and in pa icula wi h hei in e play wi h one ano he . The e is s ill much o disco e
abou he me hods and ools ha can help us explain how ansla ion ope a es in
mul imodal con ex s.
This special issue aims o con ibu e o he dissemina ion o in o ma ion abou
di e en ways o app oach mul imodali y in ansla ion esea ch and also o he
encou agemen o inc easingly anspa en and igo ous esea ch in he ield. The
con ibu o s o his special issue acco dingly explo e how mul imodal opics ha e been
and can be app oached in T ansla ion S udies, wha he bene i s and challenges associa ed
wi h a ious me hods a e and how he me hods can be applied o di e en con ex s.
In wha ollows, we i s in oduce mul imodali y, discuss he a ious de ini ions
ha exis o he concep o “mode” and explo e he kinds o modes ha a e ypically
p esen in mul imodal ex s. Then we discuss he na u e o mul imodali y and mul imodal
ansla ion as opics o esea ch. We also explain why we chose me hods as he ocus o
his special issue and in oduce a ew a eas o mul imodal ansla ion esea ch ha a e
pa icula ly in e es ing om a me hodological pe spec i e and in he con ex o his
special issue. Finally, we in oduce he a icles con ained in his special issue.
2. Mul imodal s udies and mul imodal ansla ion s udies
2.1 Mul imodali y and modes
The concep o mul imodali y has a ac ed g owing esea ch in e es since he ea ly
2000s. Mul imodally o ien ed esea ch—o “mul imodal s udies” (O’Hallo an & Smi h,
2011, p. 1)—has i s oo s in Gun he K ess and Theo an Leeuwen’s seminal wo ks,
Reading Images: The G amma o Visual Design (1996) and Mul imodal Discou se: The
Modes and Media o Con empo a y Communica ion (2001), bo h o which buil on
M.A.K. Halliday’s (1978) no ion o language as a social–semio ic sys em and a sys emic
unc ional g amma . As i s s a ing poin , his app oach akes he h ee me a unc ions
Halliday p oposed o e bal language and ex ends hem in o de o apply hem o o he
o ms o communica ion, such as images and music (Jewi , 2014, p. 129). The sys emic
unc ional pe spec i e on mul imodali y is s ill p ominen in mul imodal s udies (see
sec ion 3.2.1).
Mul imodali y i s became a ocus o in e es in a eas o esea ch such as semio ics
and applied linguis ics, bu i has since become an inc easingly impo an pe spec i e o
esea ch in p ac ically all disciplines ha examine communica ion, including s udies in
isual a s, design, media and educa ion. As Jewi (2013) desc ibes i , mul imodal s udies
aims o unde s and “communica ion and ep esen a ion as mo e han language” (p. 250)
and, he e o e, o challenge he p e ious p edominance o language in esea ch. One could
he e o e a gue ha mul imodal s udies has eme ged om he need o unde s and how
e bal meaning-making esou ces ela e o and a e a ec ed by esou ces ha a e non-
e bal—meaning all he o he ypes o esou ces ha can be employed o con ey
meaning.
Mul imodali y is ypically de ined om one o wo pe spec i es: i can be desc ibed
ei he as he coexis ence o mul iple modes wi hin a pa icula con ex (e.g., Gibbons,
2012, p. 8) o as he p ocess o decoding he coexis ing modes om a iewe ’s o a
eade ’s s andpoin (e.g., E e e , 2015, p. 3). The la e emphasises ha coexis ing modes
do no ac ually in e ac unless hey a e being in e p e ed by someone; mul imodali y is
unde s ood as being he in e ac ion o modes in he cogni i e sys em o he iewe o he
eade .
Why me hods ma e
3
In o de o unde s and ei he o hese de ini ions, we mus e lec on wha we mean
by mode. Acco ding o Gun he K ess, a mode is a esou ce o meaning-making, one
used o ep esen a ion and communica ion (K ess, 2010, p. 82). Mul imodal s udies
ypically p oposes ha e bal language cons i u es a mode ( o a discussion o whe he
spoken language and w i en language can be unde s ood as sepa a e modes, see, o
example, Ke ola, 2018, p. 31). Ye , he ques ion o wha else cons i u es a mode is no a
s aigh o wa d one: his ypically depends on he na u e o he mul imodal ma e ial ha
is being analysed as well as on he esea ch ques ions ha a e being asked. Ins ead o
o e ing ixed classi ica ions wi h a se numbe o modes, de ini ions o mode in he
mul imodal s udies li e a u e he e o e end o o e open-ended lis s o examples o
modes. Those ha a e o en men ioned in hese lis s include images, mo ing images,
layou , ges u es, music, and sound (see, o example, Adami, 2017, p. 451; K ess, 2010,
p. 79; Page, 2010, p. 1).
In pa , his lack o de ini eness is a unc ion o he di e si y o mul imodal a e ac s
and e en s, bu i also s ems om he social–semio ic o igins and o ien a ions p e alen
in mul imodal s udies: he social–semio ic app oach o mul imodali y de ines modes om
a socially speci ic pe spec i e. In o he wo ds, wha cons i u es a mode “depends on he
social g oup who uses i and he ange o meanings ha he g oup can exp ess h ough i s
esou ces” (Adami, 2017, p. 464). K ess uses on as an example in his e lec ion o wha
cons i u es a mode: o o he s, on migh be a me e p ope y o sub-mode o w i en ex
(c . Pauwels, 2012, p. 250), bu o a communi y o g aphic designe s i is likely o be a
“ ull means o ep esen a ion” and i is he e o e a mode o ha communi y (K ess,
2010, p. 87). Adami’s (2017, p. 464) e lec ion on he opic o e s an e en mo e speci ic
example: she p oposes ha o a communi y o wine as e s, wine cons i u es a ully
a icula e mode wi h modal esou ces such as colou , a oma and as e.
In mul imodal s udies, he e m mode is some imes also used in connec ion wi h
physiological o senso y channels such as seeing ( he isual mode), hea ing
(au al/audi o y mode), ouching ( ac ile mode), smelling ( he ol ac o y mode) and as ing
(gus a o y mode). O he s e e o hese physiological channels as senso y modali ies. Fo
he eade , he un o una e o e lap be ween hese e ms can be sligh ly misleading. I is
he e o e wo h emphasising ha , o ins ance, he e m isual mode akes on di e en
de ini ions depending on he ma e ial ha is being analysed, as we can see in he a icles
con ained in his special issue: i may e e o s ill images (a icles examining illus a ed
ex s), mo ing images (a icles examining audio isual ma e ial) o in o ma ion ha is
pe cei ed wi h sigh (a icles examining AD).
The non- e bal mode ha has so a spa ked he mos esea ch in e es in
mul imodally o ien ed ansla ion esea ch is p obably he s ill image. The ole o images
in ansla ion has been examined, o ins ance, in p in ad e isemen s (e.g., To esi, 2007;
Millán-Va ela, 2004), child en’s pic u e books (e.g., Oi inen, Ke ola, & Ga a ini, 2018;
Van Mee be gen, 2010), comics (e.g., Kaindl, 2004) and illus a ed echnical ex s (e.g.,
Ke ola, 2016, 2018). Na u ally, audio isual ansla ion is also an a ea o esea ch whe e
i is ex emely in e es ing o conside he unc ions o modes and he o e all modal
con igu a ion o a mul imodal ex . In ilmic AD, o ins ance, isually pe cei ed
in o ma ion is con e ed in o an o al e bal o m. In dubbing, he ansla ion needs o i
in o s ic limi a ions posed by he o he modes o he sou ce ex (ST; o ins ance, isual
synch ony). In sub i ling, he ansla ion is p esen ed as an addi ional mode—w i en
ex —which compe es o he iewe ’s isual a en ion.
Audio isual ansla ion has, gene ally speaking, a p ominen p esence in s udies on
mul imodal ansla ion. Because i in eg a es bo h au al and isual modes, i p o ides a
good example o many o he cha ac e is ics and challenges ypical o mul imodali y. As
such, i could e en be called an “a che ypal” (Pe ego, 2012, p. 7) o m o mul imodal
ansla ion. O cou se, esea ch should no ep esen only one a ea o mul imodali y and
audio isual ansla ion alone canno answe all he ques ions ega ding mul imodali y.
Howe e , as audio isual ansla ions and hei esea ch can add ess ques ions o
Tuominen, Jiménez Hu ado & Ke ola
4
mul imodali y in an enligh ening way, we hink i is jus i iable o use hem as example
cases o ep esen mul imodali y whene e hey illumina e a ele an poin .
2.2 Mul imodal ansla ion as a clus e concep
As he mul iplici y and en a i eness o he p eceding de ini ions demons a es,
mul imodali y canno be desc ibed as a monoli hic concep . Ins ead, i co e s a wide
a ie y o gen es, o ms o communica ion, and combina ions o modes and semio ic
esou ces. This means ha mul imodali y can also be associa ed wi h many kinds o
ansla ion. I would he e o e be almos impossible o gi e a single, all-encompassing
de ini ion o wha mul imodal ansla ion means. Acco dingly, a he han labelling i a
simple, easily dis inguishable phenomenon, mul imodal ansla ion could be called a
“clus e concep ”. Ma ia Tymoczko (2007, pp. 83–90) has p oposed he idea o a clus e
concep as a way o make sense o all ansla ion in an inclusi e way. In he wo ds,
ansla ion “canno [simply be de ined in e ms o necessa y and su icien ea u es”
(Tymoczko, 2007, p. 85). T ansla ions canno be dis inguished om o he p oduc s o
phenomena by some cha ac e is ic ha all ansla ions would ha e in common, bu hey
do ha e “pa ial and o e lapping simila i ies” which allow us o use and unde s and he
concep o ansla ion (Tymoczko, 2007, p. 85). Simila ly, Luis Pé ez-González (2014a,
pp. 141–142) poin s ou ha e en he ield o audio isual ansla ion on i s own is so
di e se ha i equi es mul iple de ini ions, and he consequen ly endo ses a lexible
heo e ical unde s anding o ansla ion as a clus e concep .
The same a gumen s apply o mul imodal ansla ion: i ce ainly co e s a
mul iplici y o ac i i ies ha do no i unde a single de ini ion wi h clea bounda ies. I
would he e o e no be a p ac ical goal o y o o mula e a uni ied de ini ion o
mul imodal ansla ion. Ins ead, esea ch could aim o explo e as many con ex s and
gen es as possible in o de o de elop he mos comp ehensi e unde s anding possible o
he a ie y o ways in which his clus e concep becomes ac ualized.
Desc ibing mul imodal ansla ion as a clus e concep leads us o a undamen al
ques ion: Do we unde s and mul imodal ansla ion in he same way as any o he kind o
ansla ion o is i a di e en phenomenon? The ac ha a a ie y o modes a e in ol ed
means ha he ansla ional ac i i ies do become a ied. Fo example, in e semio ic
ansla ion, such as ansla ing isual in o ma ion in o e bal in o ma ion in AD, is a
special case o mul imodal ansla ion ha expands he de ini ion o ansla ion, because
i akes place be ween e bal and non- e bal modes. In ac , all mul imodal ansla ions
mus ake non- e bal modes in o accoun o some ex en , and hey may in oduce new
challenges o he ansla o ’s ask, such as modi ying isual in o ma ion o in oducing
shi s in he ansla ion o he e bal con en so as o emain consis en wi h he non- e bal
modes. The ield o mul imodal ansla ion he e o e inhe en ly encompasses many
special o ma ginal cases o ansla ion.
The e is no simple answe o he ques ion whe he mul imodal ansla ion can be
conside ed a dis inc phenomenon. Bu i is undeniable ha he de ini ion o ansla ion
mus be pa icula ly lexible, and he bo de s a ound his clus e concep mus be
especially hazy, in o de o accommoda e all he ac i i ies ha can be e med “mul imodal
ansla ion”.
Ano he ele an ques ion is whe he his lexible pe spec i e on ansla ion leads
o a iew o esea ch on mul imodal ansla ion as a dis inc sub ield o T ansla ion
S udies, some hing we migh call Mul imodal T ansla ion S udies. Al hough we do no
a emp o answe his ques ion conclusi ely, he b ead h o esea ch conduc ed in he
a ea in ecen yea s does sugges ha esea ch in o mul imodal ansla ion is expanding
and di e si ying. Fu he mo e, while many cu en app oaches a e indeb ed o T ansla ion
S udies and s em om a long esea ch adi ion in he discipline, some de elopmen s
owa ds de ining he objec o esea ch may sugges depa u es unique o mul imodal
Why me hods ma e
5
esea ch con ex s. Pe haps i would be mos ealis ic o see a possible subdiscipline o
Mul imodal T ansla ion S udies as a way o challenging any na ow de ini ions o
ansla ion and o b oadening he scope o T ansla ion S udies. In his in oduc ion, we
use he e m “mul imodal ansla ion s udies”, a ansla ion-speci ic de i a ion om he
e m “mul imodal s udies”, o o eg ound mul imodally o ien ed ansla ion esea ch as a
esea ch o ien a ion wi h a speci ic objec i e o add essing he p esence and in e play o
di e en modes in ansla ional con ex s. I emains o be seen whe he i de elops in o a
genuine, p ominen sub ield o T ansla ion S udies. In o de o ha o happen,
mul imodal ansla ion s udies should demons a e i s dis inc iden i y by inding ways o
add essing ques ions ha ha e been less equen ly co e ed in T ansla ion S udies,
ques ions such as he in e play o e bal and non- e bal modes in he cons uc ion o
meaning, and hus in oduce new esea ch ques ions, new me hods and new
in e disciplina y connec ions in o T ansla ion S udies.
The na u e o mul imodal ansla ion as a clus e concep will become appa en only
i , as Tymoczko (2007, pp. 86–87) ecommends, we obse e and desc ibe many examples
o i and obse e hei simila i ies and di e ences. The e o e, while much in e es ing
esea ch has al eady been p oduced in ecen yea s, he clus e concep na u e o
mul imodal ansla ion ce ainly wa an s ye u he con inuing esea ch. Fu he mo e, i
con i ms he need o es new me hods ha can unco e di e en aspec s o mul imodal
ansla ion and o m a deepe unde s anding o i s na u e. The e o e, in he ollowing
sec ion, we con inue o explo e he na u e o mul imodal ansla ion s udies by discussing
some o he me hodological challenges ela ed o he ield.
3. Me hods in mul imodal ansla ion s udies
3.1 In e disciplina i y in mul imodal ansla ion s udies
E en hough we end o sha e an unde s anding o he na u e o schola ly in es iga ion,
we do no always unde s and he e ms me hod, me hodology o e en esea ch app oach
in he same way. The e ms used o dis inguish be ween he di e en aspec s and
applica ion phases o a me hod a e some imes ambiguous. Wha some e e o as a me hod
o a me hodology is e med a me hodological design o a scien i ic app oach by o he s.
Academic knowledge o e s heo ies o explain he phenomena ha su ound us.
We equi e in o ma ion ha is sys ema ically acqui ed o desc ibe exis ing pa e ns and
disco e new ones. The e o e, in he i s place, i is o i al impo ance o adop a
me hodology o a esea ch app oach ha allows he esea che o ga he ep esen a i e
da a. In he second place, i is essen ial o employ a me hod ha desc ibes and explains
he da a and he ela ionships ha exis be ween pa s o he da a. These explana ions gi e
ise o models and heo ies ha desc ibe he phenomenon unde s udy. In his sense, a
me hod is a collec ion o ope a ions and p ocesses which a e ca ied ou so ha he
equi ed da a can be sys ema ically analysed and so ha eliable and e i iable a gumen s
can hen be d awn om i .
Saldanha and O’B ien (2013) desc ibe me hodology as “a gene al app oach o
s udying a phenomenon” whe eas me hod “is a speci ic esea ch echnique” (p. 13). The
au ho s desc ibe ools, in u n, as echnological ins umen s designed o di e en
me hods (2013), such as “a oice eco ding de ice, sc een eco ding so wa e and a
keys oke logging ool” (p. 13). In his sense, eye acking, o ins ance, is a speci ic
me hod whe eas an eye acke is a ool: he so wa e used du ing he applica ion o he
me hod.
I is use ul o keep he de ini ions o hese e ms in mind when conside ing he s a e
o esea ch in T ansla ion S udies. The ield is he e ogeneous in he way esea ch is
conduc ed, and because T ansla ion S udies is inhe en ly in e disciplina y in i s
me hodology, desc ip ions o he ools and me hods used can a y g ea ly. T ansla ion

Tuominen, Jiménez Hu ado & Ke ola
6
S udies i sel has adop ed a numbe o app oaches om o he disciplines: om he
adi ionally p e alen Li e a y C i icism and Linguis ics o mo e ecen ly es ablished
links o disciplines such as Psychology and Sociology (Saldanha & O’B ien, 2013, pp.
1–2). The e o e, as Saldanha and O’B ien (2013) poin ou , T ansla ion S udies does no
come ac oss as a discipline wi h a “single cohe en me hodology” (p. 2), e en hough
many o he me hods bo owed om o he disciplines ha e been modi ied o i he
special ci cums ances o T ansla ion S udies and many ools a e used in a speci ically
ailo ed way. This in e disciplina i y and he cons an modi ica ion o ools and me hods
explain why me hodological discussion plays an impo an ole in he discipline. I he e
is no o e a ching, es ablished me hodology, i is impo an o explain and c i ically
e alua e he b ead h o op ions o in e disciplina y c oss-o e s and hei implemen a ions
speci ic o T ansla ion S udies. Fu he mo e, i is always possible o explo e new sou ces
o me hodological inspi a ion and po en ially use ul ools. A deepe unde s anding o he
a ailable esea ch me hods and hei applica ions can acili a e a mo e disciplined
app oach o esea ch design and encou age a mo e nuanced deba e on he na u e o
ansla ion as an ac i i y and T ansla ion S udies as a discipline.
As is he case wi h T ansla ion S udies gene ally, i is impossible o delinea e a
single, cohe en me hodological amewo k o mul imodal ansla ion s udies. In o he
wo ds, bo h T ansla ion S udies and mul imodal ansla ion s udies a e also clus e
concep s in a me hodological sense. Di e en ypes o mul imodal ansla ion may o m
in e disciplina y connec ions in di e en di ec ions—audio isual ansla ion wi h ilm
s udies, echnical ansla ion wi h echnical communica ion and ideogames wi h human–
compu e in e ac ion.
The need o pe cei e mul imodal ansla ion as a me hodological clus e concep is
ein o ced by he c i icism di ec ed by some schola s a p ominen ends in mul imodal
ansla ion s udies—namely, ha he esea ch me hods adop ed o mul imodal con ex s
ha e ended o be simila o hose used wi h e bal ex s—which has led o a en ion being
ocused on language a he expense o o he modes (Kaindl, 2013, p. 257). In his ega d,
se e al schola s ha e called o mo e a ied me hods ha ake accoun o bo h he non-
e bal and he e bal aspec s o mul imodal messages (Kaindl, 2013, p. 266; Kokkola &
Ke ola, 2015, pp. 226–227; Pé ez-González, 2014a, p. 185).
Following hese ecommenda ions and de eloping a b oade me hodological oolki
would mean, i s , placing emphasis on wha is unique and speci ically challenging abou
mul imodal ansla ion s udies and, secondly, shi ing away om he e bal ocus o
T ansla ion S udies. In o de o accomplish his mo e a ied me hodological amewo k,
i would be help ul o look o models om o he disciplines and mul idisciplina y
inno a ions ha ha e no p e iously been ha nessed commonly in ansla ion esea ch.
Emb acing a combina ion o new in e disciplina y connec ions and a solid unde s anding
o he na u e o ansla ion as a a ge o esea ch could help o mo e esea ch o wa d in
inno a i e and ambi ious ways.
As we see i , he co e me hodological challenge acing mul imodal ansla ion
esea ch is in eg a ing he analysis o non- e bal modes wi h ha o e bal in o ma ion.
This challenge can ake on di e en o ms depending on he na u e o he esea ch being
ca ied ou . In sec ions 3.2.1 o 3.2.3, we in oduce h ee a eas o mul imodal ansla ion
s udies ha , in ou iew, me i pa icula e lec i e a en ion om a me hodological poin
o iew. The i s o hese is he applica ion o he me a unc ional analysis o ansla ional
enqui y. Me a unc ional analysis o e s a eady-made ool o analysing non- e bal
modes. Ye , as we discuss in he ollowing sec ion, he in eg a ion o analyses o e bal
and non- e bal in o ma ion poses pa icula challenges in his ype o esea ch.
The second a ea o esea ch, desc ibed in sec ion 3.2.2, is co pus-based me hods in
mul imodal ansla ion s udies, pa icula ly in he analysis o audio isual ex s. As
ad oca ed by Baños, B u i and Zano i (2013, p. 488), in o de o deal wi h he complex
semio ic ab ic o audio isual ex s, co pus-based audio isual ansla ion esea ch should
Why me hods ma e
7
in eg a e mul imodal analysis wi h a co pus linguis ics me hodology. Bu he p ac ical
applica ion o his idea encoun e s a numbe o concep ual and echnological challenges.
The hi d a ea o esea ch we wish o d aw a en ion o is ecep ion esea ch. The
ecep ion o ansla ed mul imodal ex s is o pa icula in e es , since he eade ’s o
iewe ’s in e p e a ion e lec s he way in which he ansla ion in e ac s wi h o he
modes. Howe e , as we w i e in sec ion 3.2.3, ou iew is ha mul imodali y has o da e
no been conside ed su icien ly o explici ly in ecep ion esea ch.
3.2 Cu en and eme ging opics in mul imodal ansla ion s udies
3.2.1 Sys emic unc ional analyses in mul imodal ansla ion s udies
The sys emic unc ional app oach o mul imodali y is linguis ically inspi ed: as i s
s a ing poin , i adop s he h ee me a unc ions Halliday (1978) p oposed o e bal
language— he idea ional, he in e pe sonal and he ex ual—and ex ends hem o apply
o all modes. In his iew o mul imodali y, “common semio ic p inciples ope a e in and
ac oss di e en modes” (K ess & an Leeuwen, 2001, p. 2). The app oach, he e o e,
ea s all modes as iden ical in hei meaning-making s a egies. In mul imodal s udies,
he me a unc ional analysis is pa icula ly popula in he analysis o illus a ed ex s, in
which such an analysis is ca ied ou on bo h he e bal and he isual pa s o he ex ,
he wo ds and he images. The me a unc ional analysis o images is ounded on he
social–semio ic accoun o how meaning is ealized isually ha was in oduced by K ess
and an Leeuwen (1996).
In he me a unc ional analysis o e bal language, he idea ional me a unc ion e e s
o he way in which ex e nal eali y is ep esen ed in he ex : he people, objec s and
places— e e ed o as “ he pa icipan s”—as well as ac ions, ci cums ances and so on
being discussed (e.g., Halliday & Ma hiessen 2004/2013, p. 29). Simila ly, image
analysis a he idea ional le el includes iden i ying bo h he pa icipan s in he image and
he ci cums ances be ween he pa icipan s (K ess & an Leeuwen, 1996, p. 42). The
in e pe sonal me a unc ion o e bal language e e s o he ela ionship ha exis s
be ween he au ho o he ex and he eade o he ex (Halliday & Ma hiessen
2004/2013, pp. 29–30). The au ho may add ess he eade in a ious ways— o ins ance,
by ins uc ing, nego ia ing, pe suading o gi ing o de s. Image analysis a he
in e pe sonal le el e e s o iden i ying he p esumed ela ionships be ween he c ea o o
he image, i s iewe and he pa icipan s in he image. This me a unc ion is said o be
ealised a h ee le els: eye con ac , size o ame (close sho , medium sho , long sho )
and pe spec i e (K ess & an Leeuwen, 1996, pp. 42–43, pp. 116–129). The ex ual
me a unc ion o e bal language e e s o he p ope y o a ex ha links indi idual wo ds
in o cohe en clauses. I in ol es aspec s such as cohesion, in o ma ion s uc u e and
hema ic s uc u e (Halliday & Ma hiessen 2004/2013, p. 30). In image analysis, his
me a unc ion is e e ed o as composi ional and i e e s o iden i ying he composi ion
o he image, he colou (s) and he ela i e sizes o he pa icipan s and hei posi ioning
(up, down, le , igh , cen e) in ela ion o he o he pa icipan s (K ess & an Leeuwen,
1996, p. 43, pp. 177–179).
Sys emic unc ional analysis has equen ly been employed in mul imodal
ansla ion s udies. Examples include analyses o sub i les (e.g., Espindola, 2012;
Mubenga, 2009, 2010), ansla ions o ad e isemen s (e.g., Feng & Espindola, 2013;
Millán-Va ela, 2004), child en’s pic u e books (Van Mee be gen, 2010) and magazines
(Chueasuai, 2013). The p ominence o he sys emic unc ional app oach o mul imodali y
in ansla ion esea ch is also emphasised in Luis Pé ez-González’s a icle Mul imodali y
in T ansla ion and In e p e ing S udies, w i en as a gene al in oduc ion o he opic o
A Companion o T ansla ion S udies (2014b). In he a icle, he au ho in oduces he
a i al and he p omise o he sys emic unc ional app oach, e e ing o i h oughou he
Tuominen, Jiménez Hu ado & Ke ola
8
a icle as “mul imodal heo y”, which could e en be in e p e ed as sugges ing ha
mul imodali y is synonymous wi h he sys emic unc ional app oach.
Sys emic unc ional analysis can be complimen ed o o e ing sys ema ic
p ocedu es o analysing mul imodal ma e ial (Ma inec & Salway, 2005, p. 341). A
possible limi a ion o he me hod, howe e , is ha , based on exis ing s udies, he
me a unc ional analyses o di e en modes, pe o med sepa a ely, canno always be
combined in a p oduc i e manne ( o de ailed e iews o me a unc ional analyses in
mul imodal ansla ion s udies, see, o example, Ke ola, 2018). I he analyses canno be
combined, wha we a e discussing is a p ocedu e o p oducing sepa a e, monomodal
analyses ins ead o a mul imodal analysis o he in e ac ion o he modes. Ano he
conce n ha has been epo ed abou he analyses o mul imodal ex s is he p oblem o
“in ini e de ail”: me a unc ional analyses o en p oduce pains akingly de ailed
desc ip ions o he componen s o mul imodal ex s bu hey a ely o mula e
gene aliza ions abou mul imodal meaning cons uc ion (Fo ce ille, 2007, p. 1236).
3.2.2 Co pus-based mul imodal ansla ion esea ch
When we e e o he quan i y and ep esen a i eness o he da a being o pa amoun
impo ance in da a collec ion, we a e e e ing o co pus-based T ansla ion S udies.
Gene ally speaking, co pus analysis is ha dly new in T ansla ion S udies, since i has a
long and ui ul his o y as a esea ch me hod (Saldanha & O’B ien, 2013). This is
e lec ed in he conside able numbe o pape s ha p esen s udies on how o sol e
di e en ansla ion p oblems o ha con i m di e en hypo heses ega ding ansla ion
as ei he a p ocess o a p oduc . Co pus analysis has been success ully applied o he
acquisi ion o ansla ion compe ence (Quinci, 2015; Rod íguez Inés, 2013), o ansla ion
uni e sals (Hal e son, 2007), o he ansla ion o s yle, and o he desc ip ion o ex
gen es and e en o specialised ex s (Jiménez C espo, 2008; López Rod íguez, 2007.
Howe e , all o hese s udies ha e a ge ed monomodal ex s o only one mode o a
mul imodal ex . When co pus analysis is applied o mul imodal ex s, howe e , he
scena io unde goes a comple e ans o ma ion.
Because o he di e en modes in mul imodal ex s, i is s ill no possible o align
an ST au oma ically wi h a a ge ex (TT), o e en o do so wi h he di e en modes in
he ex . Despi e he apid p og ess being made wi h echnology, compu e scien is s ha e
s ill no been able o c ea e a p og am capable o achie ing his ea . Indeed, a e a
somewha a duous adap a ion p ocess, ce ain so wa e applica ions, such as A las. i,
An Conc and MQDA, a e able only o ela e isual and audio uni s semi-au oma ically
wi h he TTs ha esul om he ansla ion o hese modes. In he co pus-based analysis
o mul imodal ansla ion, howe e , hese applica ions a e ye unable o acili a e he
segmen a ion o he ex in o uni s o analysis, no a e hey able o acili a e he subsequen
seman ic anno a ion o hese uni s. These limi a ions cons i u e a bo leneck in his
inno a i e ype o co pus esea ch.
In con as , adi ional co pus s udies ha e managed o mee hese challenges
success ully (Rod íguez-Inés, 2017, p. 268). This is because he segmen a ion o
monomodal ex ual co po a is based ei he on di e en le els o linguis ic s uc u e
(mo pheme, lexeme, syn agm, e c.) o on s uc u al aspec s o a ex (pa ag aphs, lines,
e c.). Un o una ely, his is no he case wi h co pus-based s udies o mul imedia o
mul imodal ex s, in which segmen a ion is conside ably mo e complex.
Wha has s ill no been s udied su icien ly in co pus-based s udies a e he
g amma s o ce ain semio ic modes and he ex s in which hey ope a e— o ins ance,
dynamic o s a ic images, ges u es and sound ypes. Acco dingly, when he esea che ’s
objec i e is o di ide a mul imodal ST in o isual, sound and/o ac ile uni s, his
segmen a ion mus be based on a de ailed analysis o i s componen s, namely, he semio ic
codes i is composed o .
Why me hods ma e
9
Fo example, in ilm s udies, he Eu opean (Case i & di Chio, 1990), and Ame ican
schools (Bo dwell & Thompson, 1986) ha e p oposed in e es ing heo ies ha a e
cu en ly being applied o he segmen a ion o audio isual co po a. In heo ies abou he
in e p e a ion o a and a is ic educa ion he e is a long adi ion o segmen ing and
decomposing s a ic wo- and h ee-dimensional images in o codes and smalle
componen s in o de o be able o in e p e hei a is ic composi ion and unc ion
(Dondis, 1973). Howe e , hese ad ances in di e en disciplines ha ocus on he
g amma ical aspec s o hei own ypes o ex ha e s ill no been ecu en ly o
sys ema ically applied in co pus-based s udies o mul imodal ansla ed ex s, al hough
some a emp s ha e been made o do so (Pé ez Payá, 2010).
Ano he ea u e o co pus s udies is he complex agging sys em ha mus be
applied o mul imodal ex s when s udying a ce ain phenomenon. This way o a aching
in o ma ion o ex segmen s o o he objec o analysis is ex emely p oduc i e in
monomodal co po a, because i is mainly au oma ed. Bu when a co pus is mul imodal,
ex agging is a ely pe o med because i is ex ao dina ily di icul o do. The e ha e
been a emp s o ag ges u es and unc ions in o he languages, howe e . A case in poin
is he agging done by he ELAN POS agge o B i ish sign language (i.e. he Digging
o Signs P ojec /BSL Co pus P ojec ). E en co pus-based in e p e ing s udies do no use
any ype o agging o he sup a-segmen al elemen s o discou se. In his sense, mos
mul imodal co po a a e o he ype desc ibed in Abuczki and Ghazaleh (2013):
Simila ly o co pus-d i en app oaches ha s udy lexical bundles (mul i-wo d sequences) (Bibe 2010:
170–172), some o he mul imodal co pus esea ches a e inspi ed by he no ion o semio ic bundles
(A za ello e al. 2005) whe e modeling language p oduc ion includes he manipula ion o esou ces as
well as ges u e and alk. Some unc ional anno a ion schemes (Allwood e al. 2007) y o code he
meaning ela ions be ween ges u es and co-occu ing speech in a sys ema ic way, and label
communica i e e en s acco ding o he alignmen o speech and ges u e. (p. 89)
In some mul imodal ex s, such as audio isual ex s o hose in which s a ic images a e
pa o he mul imodali y, no sys ema ic agging o he cha ac e is ics ha de ine di e en
ex gen es is being done ye .
Finally, because o he p oblems and limi a ions ou lined abo e, mul imodal ex
agging is gene ally pe o med manually o semi-au oma ically. Needless o say, his
p ocess is slow and complica ed. I also equi es he agging se o be p o essionally and
concep ually jus i ied, he selec ion o which p esumably depends on he esea che ’s
objec i e. Besides, hese agging se s should be based on heo ies ha ha e al eady
explained he issue unde analysis (Göp e ich, 2010). Fu he mo e, he agging p ocess
mus be subjec o a s ic ly egula ed ag eemen be ween hose who pe o m he agging
(an in e agge ag eemen ).
All o his e iden ly equi es consensus be ween di e en esea ch ini ia i es so
ha a cohe en agging sys em can be adap ed speci ically o T ansla ion S udies. Only
in his way will analyses be mo e p oduc i e. Du ing he agging p ocess, o ins ance,
agge s may consis en ly disag ee on ce ain exempla s o ypes o exempla .
Disag eemen may be conside ed o be an indica ion ha he heo e ical de ini ion o an
ins ance lacks cla i y and accu acy and he e o e equi es ede ini ion (o e en u he
heo ising) (Ho y & La id, 2010, p. 30).
S ill, he e a e c i ical oices which claim ha co pus s udies decon ex ualise he
objec o s udy. Some academics claim ha co pus s udies do no o e a eal bo om-up
app oach; mo eo e , he lack o a su icien ly la ge o ep esen a i e co pus can also
in alida e esul s. Acco ding o E ison (2010, p. 132), combining quali a i e and
quan i a i e da a is pa icula ly sui able o small domain-speci ic co po a. O he schola s
a i m ha a smalle sample pe mi s a mo e in-dep h in e p e a ion o he esul s (Sole
Gallego, 2013, p. 192); i can also pe mi a bo om-up analysis based on co pus da a ha
Tuominen, Jiménez Hu ado & Ke ola
16
4.4 In e p e ing and unde s anding ansla ed mul imodal messages
As we discussed in sec ion 3.2.3, mul imodali y p esen s an added dimension o meaning-
making o hose who a e ying o unde s and he ansla ed mul imodal message.
The e o e, he pe spec i e o ecep ion o e s impo an opics o esea ch, as we see in
he i e a icles in his sec ion. The me hods used o engage wi h he opic ange om
hose employed in Cogni i e Neu oscience o ocus g oups; hey demons a e he a ie y
o ele an esea ch ques ions and me hods ha ela e o mul imodali y.
Mul imodal communica ion is also cen al o knowledge ans e in di e en a eas
and domains o specialised knowledge, as e lec ed in he ex gen es ypical o science
and echnology. In ac , adi ional ins umen s o knowledge access, such as me apho s,
a e also being s udied om a mul imodal pe spec i e. The i s wo a icles in his sec ion
deal wi h he accessibili y o isual in o ma ion in he specialised ield o medical
communica ion.
The a icle by Ma ibel Te cedo and Alicia Casado-Valenzuela examines he ole
o mul imodal communica ion as a way o accessing medical knowledge. Mo e
speci ically, hey analyse isual images ha ac i a e a clea ly me apho ical s uc u e. The
au ho s examine how his acili a es he gene al public’s access o medical in o ma ion.
Fo his pu pose, hei assump ion is ha in mul imodal ex p oduc ion he selec ion o
sui able images is c ucial o knowledge ans e in he sphe e o specialised ansla ion.
This s udy seeks o ind answe s o wo esea ch ques ions. The i s seeks o
iden i y he ype o images wi h a me apho ical s uc u e ha a e mos equen ly used in
medical ex s o he gene al public and illus a e complex medical concep s. The second
esea ch ques ion aims o asce ain whe he me apho ical images a e p e e ed o non-
me apho ical ones and also whe he he me apho s in hese images acili a e use s’ access
o knowledge.
To add ess he i s ques ion, he au ho s base hei s udy on cogni i e heo ies
ega ding me apho ical images and demons a e hei use ulness o he ansla ion p ocess
by p o iding a ypology o hese images o en also p esen in o he cul u es. Fu he mo e,
he s udy uses a da a-d i en app oach. This was possible hanks o Va iMed, a da abase
o mul imodal ex s. The esul s show ha me apho is a key elemen in he ansmission
o expe knowledge.
The second esea ch ques ion is add essed expe imen ally. The au ho s he e o e
analyse he eal- ime knowledge access o he a ge ed ecei e s by means o isual
me apho ical elemen s. This s ep is essen ial be o e deciding which images a e mos
sui able in mul imodal communica ion (including mul ilingual con ex s). Fo his
pu pose, hey adminis e a double ques ionnai e o a se o esponden s, paying pa icula
a en ion o i s ecological alidi y.
The opic o he a icle by Juan An onio P ie o-Velasco and Vicen Mon al -
Resu ecció is he comp ehensibili y o mul imodal pa ien in o ma ion guides, which
a e o en he esul o he in alingual and in e modal ansla ion o specialised medical
ex s in o a non-specialis in o ma ional ex . Illus a ions a e ypically used o suppo
he e bal message and make i mo e comp ehensible in hese kinds o con ex s, bu we
know li le abou how isual in o ma ion may imp o e comp ehensibili y. This a icle
add esses ha gap in he esea ch. P ie o-Velasco and Mon al -Resu ecció conduc ed a
pilo s udy in which hey used a su ey and a ocus g oup o collec bo h quan i a i e and
quali a i e da a om ac ual eade s o pa ien in o ma ion guides; his hey did in o de
o ind ou wha kinds o illus a ions bes suppo pa ien s’ unde s anding o he
in o ma ion p o ided o hem. Thei indings sugges ha illus a ions do indeed play a
ole in he comp ehensibili y o hese medical ex s, and u he esea ch is needed o
unde s and his comp ehension p ocess be e .
Whe eas he i s wo a icles in his sec ion employ bo h mul imodal analysis
me hods and empi ical esea ch wi h use s o explo e he accessibili y o medical ex s,
he o he h ee a icles concen a e mo e speci ically on he p ocesses o eading,

Why me hods ma e
17
in e p e ing and unde s anding ansla ed mul imodal ex s. The i s is Olli Philippe
Lau enbache ’s heo e ical p oposal o he manne in which mul imodal ex s a e ead
and in e p e ed. The a icle in es iga es he unc ion o edundancy in mul imodal
communica ion and he way he phenomenon o edundancy can be used as a c ucial
elemen in mul imodal meaning-making. The me hodological amewo k o he a icle is
based on ele ance heo y and Kin sch’s Cons uc ion-In eg a ion Model. Lau enbache
p oposes a model o a eading p ocess o mul imodal ex s ha is based on he de ec ion
o igge s imuli ha unc ion pa ly h ough edundancy. This model has implica ions
o he ansla ion p ocess, because any changes in he edundancy s uc u e may also
in luence ecep ion.
Eliisa Pi käsalo’s a icle explo es he use o ocus-g oup discussions as a da a-
collec ion me hod o examining he mul imodal eading expe ience o comics o , mo e
pa icula ly, g aphic no els. The da a o he s udy we e collec ed om eade s om wo
cul u al backg ounds: one da a-collec ion session was held in I aly, whe e comics ha e a
long adi ion, and he o he in Hunga y, whe e comic-s yle li e a u e has no adi ionally
been popula . Pi käsalo’s s udy indica es ha he cul u al backg ound o comic eade s,
including he adi ion o eading comics, may ha e an e ec on he way eade s
unde s and he isual language o comics, which should also a ec he wo k o comic
ansla o s.
The con ibu ion o An onio Ja ie Chica Núñez connec s h ee axes a ound
which T ansla ion S udies migh e ol e in he u u e. Fi s o all, mul imodali y and i s
semio ic implica ions a ST le el may de e mine he mac o- and mic o-s uc u al
p ope ies o he TT and in luence i s seman ic ea u es. Secondly, he implica ions o
mul imodali y can be be e pe cei ed a wha we may call he “ ansla ion e en ”, namely
he ansla o ’s p oduc ion phase plus he ecep ion phase on he side o he audience.
T ansla o s need o emb ace a mo e complex and in e wined semio ic con igu a ion o
he ST. The e, a ious communica ion modes shape a mul ilaye ed discou se whose
meanings, nuances and in e ac ions need o be ully con eyed in o he TT. Finally,
Cogni i e S udies may cons i u e an indispensable heo e ical and me hodological
pa adigm acco ding o which o conduc eal and si ua ed analyses o he mul imodal
ansla ion p ocess and o accoun o i s in icacies.
In his sense, his a icle d aws on se e al con ibu ions om Social Neu oscience
and Cogni i e Neu oscience o ocus on he analysis o emo ions in AD. Those heo e ical
models allow Chica o es ablish a me hodological app oach based on psychophysics in
o de o measu e he eac ion o blind and isually impai ed subjec s o mul imodal
emo ional s imuli in AD. As a esul , his app oach may con ibu e o assessing he alidi y
and e ec i eness o ansla ion quan i a i ely, ha is, o assess he capaci y o any gi en
AD o communica e emo ions o ecipien s in cogni i e e ms. This ype o app oach
could be applied equally o o he audio isual ansla ion ypologies whe e mul imodali y
plays a decisi e ole.
The a icles in his inal sec ion demons a e jus how many open ques ions emain
abou he ways in which ansla ions and ansla o s could bes se e hei a ge
audiences, especially in si ua ions whe e hose audiences encoun e he mul i-laye ed
meaning-making o a mul imodal ansla ion. I is, o cou se, impossible o know how
indi idual iewe s, eade s o lis ene s in e p e a message, bu inno a i e heo e ical and
expe imen al app oaches a e c ucial in inc easing ou unde s anding and p o iding new
ways o explo ing wha ac ually happens when audiences p ocess ansla ions in a
mul imodal se ing.
5. Epilogue
This collec ion o a icles demons a es how esea che s a e heeding he call o igo ous
and inno a i e app oaches ha ake in o accoun all modes o mul imodal ex s and hei
Tuominen, Jiménez Hu ado & Ke ola
18
ansla ions, as well as he in e ac ion be ween modes. This includes si ua ions whe e one
mode mus ake he place o ano he in ansla ion in o de o acili a e accessibili y. A
consis en ocus on mul imodali y ensu es ha he e bal elemen s o a mul imodal
message a e no he only ones analysed and ha he special cha ac e is ics o
mul imodali y a e also aken in o accoun .
Fu he mo e, one o ou main conce ns in selec ing he a icles o his special
issue has been hei explici and e lec i e ocus on esea ch me hods. In line wi h his,
all he a icles con ain some c i ical discussion on ques ions conce ning esea ch
me hodology. Howe e , a icles on a me hodological heme can assume mul iple o ma s.
Some o he a icles a e mo e heo e ical in na u e, explo ing he backg ound and he
disciplina y unde pinnings o a speci ic me hodological app oach, whe eas o he s p o ide
examples o how a ce ain me hod can be applied in p ac ice and o e c i ical e lec ion
on he use ulness and applicabili y o his me hod in a pa icula con ex . The key elemen
in all o hem is he anspa en desc ip ion and discussion o he me hods used.
We hope ha his combina ion o longs anding and new me hods, heo e ical and
applied app oaches, and p oduc and p ocess o ien a ion will o e a well- ounded
o e iew o he me hodological mul iplici y ha cha ac e ises mul imodal ansla ion
s udies oday.
Re e ences
Abuczki, A., & Ghazaleh, E. B. (2013). An o e iew o mul imodal co po a, anno a ion ools, and schemes.
A gumen um, 9, 86–98.
Adami, E. (2017). Mul imodali y. In O. Ga cía, N. Flo es, & M. Spo i (Eds.), The Ox o d handbook o
language and socie y (pp. 451–472). Ox o d: Ox o d Uni e si y P ess.
Al es Veiga, M. (2006). Sub i ling eading p ac ices. In J. Fe ei a Dua e, A. Assis Rosa, & T. Se uya
(Eds.), T ansla ion S udies a he in e ace o disciplines (pp. 161–168). Ams e dam: John
Benjamins.
An onini, R. (2007). SAT, BLT, Spi i Biscui s, and he Thi d Amendmen : Wha I alians make o cul u al
e e ences in dubbed ex s. In Y. Gambie , M. Shlesinge , & R. S olze (Eds.), Doub s and di ec ions
in T ansla ion S udies: Selec ed con ibu ions om he EST Cong ess, Lisbon 2004 (pp. 153–167).
Ams e dam: John Benjamins.
Bai s ow, D. (2011). Audio isual p ocessing while wa ching sub i led ilms: A cogni i e app oach. In A.
Şe ban, A. Ma amala, & J.-M. La au (Eds.), Audio isual ansla ion in close-up: P ac ical and
heo e ical app oaches (pp. 205–219). Be n: Pe e Lang.
Bald y, A., & Thibaul , P.J. (2006). Mul imodal ansc ip ion and ex analysis: A mul imedia oolki and
cou sebook. London: Equinox.
Baños, R., B u i S., & Zano i, S. (2013). Co pus linguis ics and audio isual ansla ion: In sea ch o an
in eg a ed app oach. Pe spec i es, 21(4), 483–490.
Bo dwell, D., & Thompson, K. (1986). Film a . New Yo k, NY: Al ed A. Knop .
B ems, E., & Ramos-Pin o, S. (2013). Recep ion and ansla ion. In Y. Gambie & L. an Doo slae (Eds.),
Handbook o ansla ion s udies, Vol. 4, (pp. 142–147). Ams e dam: John Benjamins.
Cabezas-Cáce es, C. (2013). Audiodesc ipció i ecepció. E ec e de la eloci a de na ació l'en onació i
l'explici ació en la comp ensió ílmica. Tesis doc o al. Re ie ed om h p://hdl.handle.ne /
10803/113556.
Cania o, M. (2014). Double meaning e-nego ia ion: I alian ilms in Flande s as cul u al objec s. inTRAlinea
Special Issue: Ac oss Sc eens Ac oss Bounda ies. Re ie ed om h p://www.in alinea.o g/
specials/a icle/2075.
Case i, F., & Di Chio, F. (1990). Analisi del Film. Milan: Bompiani.
Chia o, D., & An onini, R. (2005). The quali y o dubbed ele ision p og ammes in I aly: The expe imen al
design o an empi ical s udy. In M. B. Paganelli & N. Maxwell (Eds.), C oss-cul u al encoun e s:
Linguis ic pe spec i es (pp. 33–44). Rome: O icina Edizioni.
Why me hods ma e
19
Chia o, D. (2007). The e ec o ansla ion on humou esponse: The case o dubbed comedy in I aly. In
Y. Gambie , M. Shlesinge , & R. S olze (Eds.), Doub s and di ec ions in ansla ion s udies:
Selec ed con ibu ions om he EST Cong ess, Lisbon 2004 (pp. 137–152). Ams e dam: John
Benjamins.
Chmiel, A., & Mazu , I. (2012). AD ecep ion esea ch: some me hodological conside a ions. In E. Pe ego
(Ed.), Eme ging opics in ansla ion: Audio desc ip ion (pp. 57–80). T ies e: EUT.
Chueasuai, P. (2013). T ansla ion shi s in mul imodal ex : A case o he Thai e sion o Cosmopoli an.
JoST ans, The Jou nal o Specialised T ansla ion, 20, 107–121.
C o , W., & C use, D. A. (2004). Cogni i e linguis ics. Ox o d: Ox o d Uni e si y P ess.
Desilla, L. (2014). Reading be ween he lines, seeing beyond he images: An empi ical s udy on he
comp ehension o implici ilm dialogue meaning ac oss cul u es. The T ansla o , 20(2), 194–214.
Dondis, D.A (1973). A p ime o isual li e acy. Camb idge, MA: MIT P ess.
Espindola, E. (2012). Sys emic unc ional linguis ics and audio isual ansla ion s udies: A concep ual basis
o he s udy o he language o sub i les. DELTA: Documen ação de Es udos em Lingüís ica Teó ica
e Aplicada, 28, 495–513.
E e e , Y.U. (2015). Recon igu ing My h and Na a i e in Con empo a y Ope a. Blooming on: Indiana
Uni e si y P ess.
E ison, J. (2010). Wha a e basics o analyzing a co pus? In A. O´Kee e & M. McCa hy (Eds.), The
Rou ledge handbook o co pus linguis ics (pp. 122–135). London: Rou ledge.
Feng, D., & Espindola, E. (2013). In eg a ing sys emic unc ional and cogni i e app oaches o mul imodal
discou se analysis. Ilha do Des e o, 64, 85–110.
Fo ce ille, C. (2007). Book e iew: Mul imodal T ansc ip ion and Tex Analysis: A Mul imedia Toolki
and Cou sebook. Jou nal o P agma ics, 39(6), 1235–1238.
Fuen es Luque, A. (2003). An empi ical app oach o he ecep ion o AV ansla ed humou : A case s udy
o he Ma x B o he s’ “Duck Soup”. The T ansla o , 9(2), 293–306.
Gene e, G., & Maclean, M. (1991). In oduc ion o he pa a ex . New Li e a y His o y, 22(2), 261–272.
Gibbons, A. (2012). Mul imodali y, cogni ion and expe imen al li e a u e. New Yo k, NY: Rou ledge.
Ghia, E. (2012). The impac o ansla ion s a egies on sub i le eading. In E. Pe ego (Ed.), Eye acking in
audio isual ansla ion (pp. 157–182). Rome: A acne Edi ice.
Göp e ich, S. (2010), Da a documen a ion and da a accessibili y in ansla ion p ocess esea ch. The
T ansla o , 16(1), 93–124.
Halliday, M.A.K. (1978). Language as social semio ic: The social in e p e a ion o language and meaning.
London: Edwa d A nold.
Halliday, M.A.K. (1985). An in oduc ion o unc ional g amma . London: Edwa d A nold.
Halliday, M.A.K., & C. Ma hiessen. (2004) 2013. An In oduc ion o Func ional G amma . 3 d ed. New
Yo k: Rou ledge.
Hal e son, S.L. (2007). In es iga ing g a i a ional pool in ansla ion: The case o English p og essing
cons uc ion. In R. Jääskeläinen, T. Puu inen, & H. S o esbu y (Eds.), Tex , p ocesses, and co po a:
Resea ch inspi ed by Sonja Ti kkonen-Condi (pp. 175–196). Joensuu: Joensuu Yliopis opaino.
Hand o d, M. (2010). Wha can a co pus ell us abou specialis gen es? In A. O’Kee e & M. McCa hy
(Eds.), The Rou ledge handbook o co pus linguis ics (pp. 254–269). London: Rou ledge.
Ho y, E., & La id, J. (2010). Towa ds a ‘science’ o co pus anno a ion: A new me hodological challenge
o co pus linguis ics. In e na ional Jou nal o T ansla ion, 22(1), 13–36.
Jewi , C. (2014). Mul imodal app oaches. In S. No is & C. D. Maie (Eds.), In e ac ions, images and
ex s: A eade in mul imodali y (pp. 127–136). Bos on, MA: De G uy e .
Jiménez C espo, M.A. (2012). Los o los in ansla ion: a con as i e co pus-based s udy o o iginal and
localized US websi es. JoST ans, The Jou nal o Specialised T ansla ion, 17, 126–173.
Kaindl, K. (2004). Mul imodali y in he ansla ion o humo in comics. In E. Ven ola, C. Cha les, & M.
Kal enbache (Eds.), Pe spec i es on mul imodali y (pp. 173–192). Ams e dam: John Benjamins.
Kaindl, K. (2013). Mul imodali y and ansla ion. In C. Millán & F. Ba ina (Eds.), The Rou ledge
handbook o ansla ion s udies (pp. 257–269). London: Rou ledge.
Ke ola, A. (2016). Towa ds a mul imodally o ien ed heo y o ansla ion: A cogni i e amewo k o he
ansla ion o illus a ed echnical ex s. T ansla ion S udies, 9(1), 67–81.
Tuominen, Jiménez Hu ado & Ke ola
20
Ke ola, A. (2018). Wo d–image in e ac ion in echnical ansla ion: S uden s ansla ing an illus a ed ex .
Ac a Uni e si a is Tampe ensis 2364. Tampe e: Tampe e Uni e si y P ess.
Kokkola, S., & Ke ola, A. (2015). Thinking ou side he “Me hods Box”: New a enues o esea ch in
mul imodal ansla ion. In D. Rells ab & N. Siponkoski (Eds.), Rajojen dynamiikkaa, G änse nas
dynamik, Bo de s unde nego ia ion, G enzen und ih e Dynamik. VAKKI-symposiumi XXXV 12.–
13.2.2015 (pp. 219–228). Vaasa: VAKKI Publica ions 4. Re ie ed om h p://www. akki.ne /
publica ions/2015/VAKKI2015_Kokkola&Ke ola.pd .
K ess, G. (2010). Mul imodali y: A social semio ic app oach o con empo a y communica ion. New Yo k,
NY: Rou ledge.
K ess, G., & an Leeuwen, T. (1996) 2006. Reading images: The g amma o isual design. London:
Rou ledge.
K ess, G., & an Leeuwen, T. (2001). Mul imodal discou se: The modes and media o con empo a y
communica ion. London: A nold.
K uge , J.-L., & Dohe y, S. (2018). T iangula ion o online and o line measu es o p ocessing and
ecep ion in AVT. In E. Di Gio anni & Y. Gambie (Eds.), Recep ion s udies and audio isual
ansla ion (pp. 91–109). Ams e dam: John Benjamins.
Lång, J., Mäkisalo, J., Gowases, T., & Pie inen S. (2013). Using eye acking o s udy he e ec o badly
synch onized sub i les on he gaze pa hs o ele ision iewe s. New Voices in T ansla ion S udies,
10, 72–86.
Langacke , R. W. (1987/2008). Cogni i e g amma : A basic in oduc ion. Ox o d: Ox o d Uni e si y P ess.
López Rod iguez, C. I. (2007). Unde s anding scien i ic communica ion h ough he ex ac ion o he
concep ual and he o ical in o ma ion codi ied by e bs. Te minology, 13(1), 61–84.
Mangi on, C. (2016). Recep ion o game sub i les: An empi ical s udy. The T ansla o , 22(1), 72–93.
Mangi on, C. (2018). Recep ion s udies in game localisa ion: Taking s ock. In E. Di Gio anni & Y.
Gambie (Eds.), Recep ion s udies and audio isual ansla ion (pp. 277–296). Ams e dam: John
Benjamins.
Ma inec, R., & Salway, A. (2005). A sys em o image– ex ela ions in new (and old) media. Visual
Communica ion, 4(3), 339–374.
Millán-Va ela, C. (2004). Explo ing ad e ising in a global con ex : Food o hough . The T ansla o ,
10(2), 245–267.
Mubenga, K. S. (2009). Towa ds a mul imodal p agma ic analysis o ilm discou se in audio isual
ansla ion. Me a, 54(3), 466–484.
Mubenga, K. S. (2010). In es iga ing no ms in in e lingual sub i ling: A sys emic unc ional pe spec i e.
Pe spec i es: S udies in T ansla ology, 18(4), 251–274.
O’Hagan, M. (2009). Towa ds a c oss-cul u al game design: An explo a i e s udy in unde s anding he
playe expe ience o a localised Japanese ideo game. JosT ans. The Jou nal o Specialised
T ansla ion, 11, 211–233.
O’Hallo an, K. L., & Smi h, B.A. (2011). Mul imodal s udies. In K. L. O’Hallo an & B. A. Smi h (Eds.),
Mul imodal s udies: Explo ing issues and domains (pp. 1–13). New Yo k, NY: Rou ledge.
Oi inen, R., Ke ola A., & Ga a ini, M. (2018). T ansla ing pic u ebooks: Re oicing he e bal, he isual
and he au al o a child audience. New Yo k, NY: Rou ledge.
O e o, P., Dohe y, S., K uge , J.-L., Ma amala, A., Pede sen, J., Pe ego, E., Rome o-F esco, P., Ro i a-
Es e a, S., Sole -Vilageliu, O., & Sza kowska, A. (2018). Conduc ing expe imen al esea ch in
audio isual ansla ion (AVT): A posi ion pape . JoST ans, The Jou nal o Specialised T ansla ion,
30, 105–126.
Page, R. (2010). In oduc ion. In R. Page (Ed.), New pe spec i es on na a i e and mul imodali y (pp. 1–
13). New Yo k, NY: Rou ledge.
Pauwels, L. (2012). A mul imodal amewo k o analyzing websi es as cul u al exp essions. Jou nal o
Compu e -Media ed Communica ion, 17, 247–265.
Pe ego, E., Del Missie , F., Po a, M., & Mosconi, M. (2010). The cogni i e e ec i eness o sub i le
p ocessing. Media Psychology, 13(3), 243–272.
Pe ego, E. (2012). In oduc ion. In E. Pe ego (Ed.), Eye acking in audio isual ansla ion (pp. 7–11).
Rome: A acne Edi ice.
Why me hods ma e
21
Pe ego, E., Laskowska, M., Ma amala, A., Remael, A., Robe , I. S., Sza kowska, A., Vila ó, A., &
Bo i oli, S. (2016). Is sub i ling equally e ec i e e e ywhe e?: A i s c oss-na ional s udy on he
ecep ion o in e lingually sub i led messages. Ac oss Languages and Cul u es, 17(2), 205–229.
Pé ez Payá, M. (2010). Reco es de cine audiodesc i o: El lenguaje cinema og á ico en Tagge i Imagen y
su e lejo en la audiodesc ipción. In C. Jiménez Hu ado, A. Rod íguez, & C. Seibel (Eds.), Un
co pus de cine: Teo ía y p ác ica de la audiodesc ipción (pp. 111–179). G anada: T agacan o.
Pé ez-González, L. (2014a). Audio isual ansla ion: Theo ies, me hods and issues. London: Rou ledge.
Pé ez-González, L. (2014b). Mul imodali y in ansla ion and in e p e ing s udies: Theo e ical and
me hodological pe spec i es. In S. Be mann & C. Po e (Eds.), A companion o ansla ion s udies
(pp. 119–131). Chiches e : Wiley-Blackwell.
Quinci, C. (2015). T ansla o s in he making: An empi ical longi udinal s udy o ansla ion compe ence
and i s de elopmen (Unpublished doc o al disse a ion). Uni e si á degli S udi di T ies e, T ies e.
Ramos Ca o, M. (2018). Los es udios de ecepción en T aducción Audio isual: Aspec os me odológicos.
In A. Rojo (Ed.), Diseños y mé odos de in es igación en aducción (pp. 99–124). Mad id:
An h opos.
Rod íguez-Inés, P. (2013). ¿Cómo aducen aduc o es y p o eso es de idiomas?: Es udio de co pus. Me a,
58(1), 165–190.
Rod íguez-Inés, P. (2017). Co pus-based insigh s in o cogni ion. In J. W. Schwie e & A. Fe ei a (Eds.),
The handbook o ansla ion and cogni ion (pp. 265–289). Hoboken, NJ: John Wiley & Sons.
Saldanha, G., & O’B ien, S. (2013). Resea ch me hodologies in ansla ion s udies. Manches e : S . Je ome.
Se a ini, F. (2010). Reading mul imodal ex s: Pe cep ual, s uc u al and ideological pe spec i es.
Child en’s Li e a u e in Educa ion, 41, 85–104.
Sole Gallego, S. (2013). La aducción accesible en el espacio mul imodal museog á ico (Unpublished
doc o al disse a ion). Uni e sidad de G anada, G anada.
So o Almela, J. (2013). La aducción de é minos cul u ales en el con ex o u ís ico español-inglés:
Recepción eal en usua ios angló onos. Quade ns. Re is a de T aducció, 20, 235–250.
Sza kowska, A., & Ge be -Mo ón, O. (2018). Viewe s can keep up wi h as sub i les: E idence om eye
mo emen s. PLoS ONE, 13(6), e0199331.
To esi, I. (2007). T ansla ing he isual: The impo ance o isual elemen s in he ansla ion o ad e ising
ac oss cul u es. In D. Kenny & K. Ryou (Eds.), Ac oss bounda ies: In e na ional pe spec i es on
ansla ion s udies (pp. 38–55). Newcas le: Camb idge Schola s.
Tuominen, T. (2012). The a o acciden al eading and inciden al lis ening: An empi ical s udy on he
iewing o sub i led ilms. Ac a Uni e si a is Tampe ensis 1794. Tampe e: Tampe e Uni e si y
P ess.
Tuominen, T. (2018). Mul i-me hod esea ch: Recep ion in con ex . In E. Di Gio anni & Y. Gambie (Eds.),
Recep ion s udies and audio isual ansla ion (pp. 69–89). Ams e dam: John Benjamins.
Tymoczko, M. (2007). Enla ging ansla ion, empowe ing ansla o s. Manches e : S . Je ome.
Van Mee be gen, S. (2010). Nede ländska bilde böcke bli s enska: En mul imodal ö e sä ningsanalys
[Du ch pic u ebooks become Swedish: A mul imodal ansla ion analysis]. S ockholm: Uni e si y
o S ockholm P ess.
Widle , B. (2004). A su ey among audiences o sub i led ilms in Viennese cinemas. Me a, 49(1), 98–101.
Yayoi, U.E. (2015). Recon igu ing my h and na a i e in con empo a y ope a. Blooming on, IN: Indiana
Uni e si y P ess.
d’Ydewalle, G., P ae , C., Ve aillie, K., & Van Rensbe gen, J. (1991). Wa ching sub i led ele ision:
Au oma ic eading beha io . Communica ion Resea ch, 18(5), 650–666.