Quali y o Heal h IT E alua ions
Pi kko NYKÄNEN
a,1
and Johanna KAIPIO
b
a
Uni e si y o Tampe e, School o In o ma ion Sciences, Tampe e, Finland
b
Aal o Uni e si y, School o Science, Espoo, Finland
Abs ac . Heal h IT e alua ion s udies ha e o en been ound o be o limi ed
quali y. To add ess his p oblem, se e al guidelines and amewo ks ha e been
de eloped as ools o suppo imp o emen o he quali y o e alua ion s udies. In
his con ibu ion, we e iew a ailable guidelines and hen p esen he Good
E alua ion P ac ice Guideline in Heal h In o ma ics (GEP-HI) in mo e de ail. GEP-
HI is a comp ehensi e guideline which suppo s especially planning and execu ion
o a heal h IT e alua ion s udy. The GEP-HI guideline helps o o e come he
quali y p oblems ela ed o weak s udy planning and me hodological s udy design.
We also discuss applica ion o GEP-HI on an e alua ion p ojec and discuss he
need o publish sys ema ically ollowing he ecognised publica ion guidelines.
Finally we discuss he u u e end on mul i-me hod e alua ion app oaches.
Keywo ds. E alua ion, quali y, scope, guideline, amewo k, heal h IT.
1. In oduc ion
E alua ion is he means o assess he quali y, alue, e ec s and impac s o heal h IT in
he heal h ca e en i onmen . E alua ion is de ined as he “ac o measu ing o
explo ing p ope ies o a heal h in o ma ion sys em, in planning, de elopmen ,
implemen a ion, o ope a ion, he esul o which in o ms a decision o be made
conce ning ha sys em in a speci ic con ex ” [1, p. 480]. E alua ion o e s me hods and
ools o collec e idence abou he bene i s, quali y, e ec s and impac s o heal h IT.
In heal h ca e p ac ice heal h IT applica ions o e challenging oppo uni ies o
imp o e he heal h ca e sys em’s unc ioning, e ec i eness and ou comes as well as
heal h ca e se ices quali y and deli e y, bu he e a e also p oblems and unan icipa ed
e ec s ela ed o he use o IT [2,3].
2
I is o u mos impo ance in he heal h ca e
en i onmen ha heal h IT p o ides he op imal and sa e esul s and he e o e heal h IT
applica ions need o be e alua ed wi h obus me hodologies and e alua ion esul s a e
o be epo ed ollowing s uc u ed epo ing s anda ds [4,5].
E alua ion is di icul , i deals wi h alues and no ms and a ious o ganiza ional
con ex s and s akeholde s’ in e es s, and i has o igh o unding and suppo [6].
Addi ionally, many po en ially applicable me hodologies and me hods exis and
e alua ion esul s a e o be analysed and in e p e ed in he s udy con ex [4]. A
challenge o imp o e he quali y o e alua ion s udies is o apply a sys ema ic
1
Co esponding au ho : P o . D . Pi kko Nykänen, Uni e si y o Tampe e, School o In o ma ion
Sciences, PO Box 607, FIN-33014 Uni e si y o Tampe e, Finland, Pi kko.Nykanen@u a. i.
2
See also: F. Mag abi e al., Heal h IT o pa ien sa e y and imp o ing he sa e y o heal h IT, in: E.
Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud Heal h Technol In o m 222, IOS
P ess, Ams e dam, 2016.
E idence-Based Heal h In o ma ics
E. Ammenwe h and M. Rigby (Eds.)
© 2016 The au ho s and IOS P ess.
This a icle is published online wi h Open Access by IOS P ess and dis ibu ed unde he e ms
o he C ea i e Commons A ibu ion Non-Comme cial License.
doi:10.3233/978-1-61499-635-4-291
291
app oach, plan he e alua ion s udy ca e ully and execu e he s udy ollowing
sys ema ic e alua ion guidelines [7].
In his con ibu ion we e iew he ocus and scope o he published e alua ion
s udies, discuss he quali y p oblems ela ed o hese s udies, and p esen heal h IT
e alua ion guidelines and amewo ks, namely he GEP-HI guideline.
2. The scope and quali y o e alua ion s udies
The quali y o an e alua ion s udy is dependen on many ac o s, e.g. on he objec i i y
o he s udy and on he independence o e alua o s, e e ing o hei independence on
economic in e es s, on in ellec ual in e es s and on he a ious s akeholde s’ in e es s.
An e alua ion s udy mus also be scien i ically well-es ablished on obus heo ies and
me hodologies
3
, and he s udy should be pe o med ollowing he p inciples o
scien i ic esea ch.
As he e a e many in e es ing ocuses o e alua ion, e.g. economics, e iciency,
usabili y o heal h IT, sa e y, p i acy and secu i y, compliance wi h he clinical p ocess,
unc ionali y o heal h IT, e ec s and impac s on heal h ca e ou comes, he e is also
need o use many di e en me hods sui able o measu ing he e alua ion c i e ia o
in e es . These po en ial me hods co e e.g. quali a i e and quan i a i e me hods,
s a is ics, heu is ics, e hnog aphy, human-sys em in e ac ion obse a ions, da a mining
and quali y analysis, cos -e ec i eness and cos -bene i analyses.
4
The e a e also many
po en ial pe spec i es o e alua ion, ep esen ing a ious s akeholde s’ iewpoin s, e.g.
manage ial, clinical, echnical, and hese iewpoin s may be s udied a a ious le els o
heal h ca e sys em – local, egional and na ional, o e en a EU o in e na ional le els.
5
I is o u mos impo ance, when planning an e alua ion s udy, o elabo a e and de ine
how hese issues a e ela ed o he s udy: Co e age o scien i ic obus ness, ele ance
o he cu en pu pose o he s udy, bes i o he impo an cha ac e is ics o he
speci ic cu en need and ele ance o he me hods. The quali y also comp ises he
quali y in publica ion o he s udy.
E alua ion s udies ha e been pe o med since 1960’s, howe e , mos wi h a a he
na ow scope, o en ocusing on how heal h IT sys ems a e ela ed o p o essionals’
oles, change managemen and use in ol emen [8]. La e he s udies ha e co e ed
also he success aspec s and lessons lea ned in implemen a ion and de elopmen o
heal h IT sys ems. In many cases e alua ion has been led by esea ch in e es s o
de elop me hodologies o o s udy he heal h ca e p ocesses. F om 1980’s onwa ds also
managemen issues, use accep ance and adop ion o heal h IT sys ems in heal h ca e
o ganiza ions ha e been s udied. Kaplan and Shaw made a e iew o how aspec s
ela ed o people, o ganiza ional and social issues ha e been conside ed in heal h IT
e alua ions [9]. They emphasized he need o pay mo e a en ion o hese issues du ing
3
See also: J. B ende , Theo e ical basis o heal h IT e alua ion, in: E. Ammenwe h, M. Rigby (eds.),
E idence-Based Heal h In o ma ics, S ud Heal h Technol In o m 222, IOS P ess, Ams e dam, 2016.
4
See also: D. Luzi e al., Economic e alua ion o heal h IT, in: ibid.
5
See also: L. Lee e al., Unde s anding s akeholde in e es s and pe spec i es in e alua ions o heal h
IT, in: ibid.
P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions292
heal h IT sys em design, implemen a ion and use and emphasize he need o in eg a e
mul i-me hod e alua ion o he whole li e cycle o he heal h IT sys em.
6
Van de Loo analysed e alua ion s udies published be ween 1967 and 1995 wi h
ega d o ype o he sys em, s udy design, da a collec ion, economic e alua ion and
ype o e ec measu e [10]. He ound pe o mance o use s o be a c i e ion in many
s udies, as well as ime sa ings and cos s o pa ien ca e. By con as , use sa is ac ion
was an e alua ion c i e ion only in 11% o s udies. These esul s e lec well he
si ua ion in 1990’s, usabili y and e ec s and impac s o heal h IT on heal h ca e
ou comes and quali y we e no highly alued issues ha ime.
Ammenwe h and de Keize [11] ound ha explana o y esea ch and quan i a i e
me hods ha e domina ed e alua ion s udies du ing he las 20 yea s. Howe e , he
s udies on ou come quali y and cos s o pa ien ca e, pa ien sa is ac ion and pa ien
beha iou ha e ecei ed mo e a en ion ecen ly as well as s udies on quali y o
p ocesses. he same ime, he numbe o labo a o y s udies and echnical e alua ion
s udies has declined while he numbe o s udies ocusing on he in luence o heal h IT
on quali y o ca e p ocesses o ou come o pa ien ca e has inc eased. I was ypical o
ea ly s udies, e.g. e alua ion o decision suppo sys ems and expe sys ems, ha he
ocus was on echnical issues, on ha dwa e and so wa e quali y and on sys em
pe o mance issues. Rigby [12] also no ed ha he ocus o e alua ion o a heal h IT
sys em changes du ing i s li e cycle: Du ing he implemen a ion phase e alua ion
add esses o en echnical aspec s, bu wi h a comple ed sys em ocuses on impac s on
pa ien ca e. Ammenwe h and de Keize ound also ha he numbe o in e -
o ganisa ional s udies has inc eased e lec ing he end owa ds coope a i e and sha ed
ca e. In many ecen s udies, use sa is ac ion and e iciency o pa ien ca e a e he
mos equen ly add essed e alua ion c i e ia, and he e is a slowly g owing end
owa ds e alua ion s udies co e ing mo e han one e alua ion c i e ion [11]. Also he
heal h IT de elopmen s discussed in [12,13] showed he impo ance o e alua ion, he
challenges and he p oblems in in eg a ed me hodologies and in endo sing he
impo ance o obus s udies.
These e iews show ha explana i e esea ch and quan i a i e me hods ha e
domina ed heal h IT e alua ion esea ch o a long ime, bu s udies ocusing on
p ocess quali y o ou come quali y o pa ien ca e ha e inc eased la ely. Also human,
social and o ganiza ional aspec s ha e been included in e alua ion s udies o some
ex en , and p omisingly esea ch in his aspec is g owing [9].
7
Quali a i e me hods
ha e been used in s udies ocused on use accep ance, usabili y and use ulness o on
o ganiza ional and social impac s o heal h IT. Many e alua ions ha e been ocused on
one use ’s indi idual con ex : how a heal h p o essional uses he sys em in he speci ic
con ex , how he/she accep s he echnology and how echnology i s in o his/he wo k
p ocesses, lea ing he mo e wide usage con ex s un ouched.
The e iews also e eal quali y p oblems in e alua ion s udies: he e a e p oblems
ela ed e.g. o weak s udy planning wi hou a sys ema ic, scien i ic me hodology, alse
implici assump ions made in he s udy, expe imen al e o s in he esea ch se ing,
unde - o o e -in e p e a ion o he esul s o alse conclusions, inclusion o non-
6
See also: P.J. Sco e al., Mixed me hods: a pa adigm o holis ic e alua ion o heal h in o ma ion
sys em, in: E. Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud Heal h Technol
In o m 222, IOS P ess, Ams e dam, 2016.
7
See also: B. Kaplan, E alua ion o people and o ganiza ional Issues – Socio echnical e hnog aphic
e alua ion, in: ibid.
P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions 293
neu al e alua o s, in a- o in e -o ganiza ional a iabili y in he e alua ion objec o
no el y o echnology. In some cases he e seem also o be p oblems in selec ing such
e alua ion me hods ha a e capable o measu e hose a iables and aspec s ha
desc ibe he phenomenon unde s udy. These quali y p oblems show ha he e is eal
need o imp o e he quali y o e alua ion s udies.
Since he ea ly 2000’s esea che s in he heal h in o ma ics ield ha e poin ed ou
he need o include also use pe spec i e and usabili y, in he e alua ion s udies [14,
15]. This need is mo i a ed by ecogni ion o he impo ance o good usabili y as he
heal h IT sys ems a e planned o use in heal h ca e clinical p ac ice.
8
We analyse
b ie ly also he published usabili y e alua ion s udies, on hei me hodological
app oach and quali y p oblems.
A e iew in [16] showed ha in gene al, empi ical usabili y s udies a e hea ily
a ec ed by he adi ional app oaches o e alua ion o human-compu e in e ac ion.
Usabili y s udies ha e mainly applied adi ional usabili y e alua ion me hods,
pa icula ly usabili y es ing and inspec ions. A conside able numbe o usabili y
e alua ion s udies ha e concen a ed on he la e phases o heal h IT de elopmen , and
e alua ion has been ocused on sys ems ha a e al eady in use. Recen usabili y s udies
a e cha ac e ized by a na ow ocus on use and usabili y issues, emphasis on
summa i e e alua ion a he han on design o de elopmen , isola ed sys em
de elopmen , and emphasis on in o ma ion sys ems and da a managemen [16]. Also in
many usabili y s udies he e is a lack o unde s anding o he con ex ual aspec s o
usabili y, and he cha ac e is ics o clinical wo k con ex s, hough he widely known
de ini ions o usabili y [17,18] emphasize he need o unde s and usabili y as a
con ex ual p ope y.
The e a e quali y p oblems also in he usabili y e alua ion s udies, e.g. he s udies
o en ocus on a single end-use g oup pe spec i e, use in e ace componen s, o use o
he sys em in a speci ic con ex , bu do no p o ide a comp ehensi e pic u e o
usabili y o a la ge-scale heal hca e in o ma ion sys em. Fu he , he e alua ion s udies
a ely discuss he ela ionship be ween single-sys em de elopmen and he exis ing
echnology se ing in heal hca e, o he cha ac e is ics o a ious use con ex s in which
he e alua ed sys em is used. Typically he s udies discuss summa i e usabili y esul s
on wo king sys ems; his leads o a ocus on he p oblems wi h adop ing cu en
sys ems in a gi en heal hca e en i onmen and di e s concen a ion om he design o
de elopmen o new, be e sys ems. In gene al, he quali y o usabili y e alua ion
s udies could be enhanced by applying app op ia e usabili y e alua ion me hods o he
han adi ional ones, e.g. con ex ual inqui y [19], which akes in o conside a ion he
con ex ual aspec s o heal hca e en i onmen . The planning o e alua ion s udies
should aim a p o iding e alua ion esul s bu also unde s anding o he unde lying
p oblems and suppo o he de elopmen wo k.
Usabili y e alua ion s udies ha e in mos cases been done sepa a ely om o he
e alua ion s udies, e.g. om e ec i eness, e ec s and impac s e alua ion s udies.
Howe e , i would be bene icial o in eg a e usabili y and o he e alua ion c i e ia in a
s udy. Usabili y is an impo an quali y aspec o heal h IT and usabili y is hea ily
dependen on good heal h IT sys em design, implemen a ion and adop ion [20]. Some
o he ecen usabili y s udies ha e al eady emphasized a mo e holis ic iew o
8
See also: R. Ma cilly e al., F om usabili y enginee ing o e idence-based usabili y in heal h IT, in: E.
Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud Heal h Technol In o m 222, IOS
P ess, Ams e dam, 2016.
P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions294
usabili y e alua ion and called o mul i-me hod app oaches and ocus mo e on heal h
p o essionals’ clinical collabo a ion. Among o he s, Kushni uk [21, 22] has exp essed a
conce n ha when he echnology applica ions become mo e complex, e alua ion
me hodologies will need o be con inually e ined in o de o keep pace. Mul i-me hod
app oaches a e called o as opposed o using one single me hod [23]. Kaipio
emphasizes [16] ha he use -cen ed app oach will play an essen ial ole in u u e
heal h IT design and de elopmen , because knowledge and unde s anding o he needs
o a ious use g oups, as well as clinical use con ex s, a e impo an and should be pa
o all heal h IT design and de elopmen phases.
These indings a e suppo ed in [24] by emphasizing he need o p edic i e
e alua ion me hods o accu a ely iden i y usabili y issues ha a ise om he
in e ac ion, sha ing and communica ion equi emen s o clinical wo k. Fu he mo e,
heal h IT sys ems ha e many use g oups and he use s use he sys ems o a ious
pu poses, such as clinical, nu sing, medical, adminis a i e, manage ial, s a is ical and
economical pu poses and oday, also pa ien s and ci izens use heal h IT sys ems o
hei pe sonal heal h and wellness pu poses. The e o e, i is impo an o conside he
a ie y o uses and use con ex s, and also he a ie y o e alua ion c i e ia, since he
na u e o clinical wo k, as well as he physical and o ganiza ional en i onmen s in he
wo kplace, may di e signi ican ly be ween heal hca e uni s.
This s a e o a ai s o e alua ion s udies and quali y challenges emphasizes he
need o sys ema ic app oaches and guidelines o design and o ca y ou di e en kinds
o e alua ion s udies o p o ide e idence abou he impac s and ac ual e iciency,
quali y, usabili y and sa e y o heal h IT [25].
3. Guidelines and amewo ks o heal h IT e alua ions
3.1. O e iew on a ailable guidelines and amewo ks
Many amewo ks and guidelines de eloped o e alua ion exis , aiming a suppo ing
and imp o ing s udies so ha heal h IT e alua ion is conduc ed o he highes
me hodological and scien i ic s anda ds. These amewo ks and guidelines di e in
e ms o gene ali y, speci ici y and iming ela ed o sys em de elopmen phases and
heo e ical unde pinning. In his sec ion, we will in oduce some selec ed amewo ks
and guidelines o heal h IT e alua ion.
Kaplan has sugges ed an e alua ion amewo k o 4C - Communica ion, Con ol,
Ca e and Con ex [26]. This 4C model calls o mul i-me hod longi udinal design o
o ma i e and summa i e e alua ions.
9
Shaw has in oduced he CHEATS- amewo k
[27] which iden i ies six aspec s o be impo an in e alua ion: Clinical, Human and
O ganiza ional, Educa ional, Adminis a i e, Technical and Social.
A Model o ASsessmen o Telemedicine applica ions (MAST) [28] lis s aspec s
o e alua ion wi hin se en domains o ou comes: heal h p oblem and cha ac e is ics o
he applica ion; sa e y; clinical e ec i eness; pa ien pe spec i es; economic aspec s;
o ganiza ional aspec s; and socio-cul u al, e hical and legal aspec s. MAST is planned
o be a oolki , a checklis o issues ha need o be conside ed in e alua ion. MAST is
9
See also: B. Kaplan, E alua ion o people and o ganiza ional Issues – Socio echnical e hnog aphic
e alua ion, in: E. Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud Heal h Technol
In o m 222, IOS P ess, Ams e dam, 2016.
P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions 295

based on a heal h echnology assessmen (HTA) app oach and ocuses on elemedicine
sys ems, no on heal h IT gene ally. MAST does no conside he execu ion and
managemen o an e alua ion p ojec .
10
Cusack e al. de eloped he AHRQ oolki [29] o p o ide s ep-by-s ep guidance
o de eloping e alua ion plans o heal h IT p ojec s. AHRQ assis s e alua o s in
de ining he goals o e alua ion, in iden i ying wha is impo an o s akeholde s, wha
needs o be measu ed o sa is y s akeholde s, wha is ealis ic and easible o measu e,
and how o measu e hese i ems. The AHRQ oolki is e y use ul om he
me hodological poin o iew. I can be applied wi hin o he mo e gene ic guidelines.
The oolki does no , howe e , gi e guidance on he e alua ion p ojec i sel , how o
manage i , how o ca y ou he p ojec , o how o comple e and epo he s udy.
A li e-cycle amewo k o e alua ion by Cla ke e al. [30] is ocused on how o
e alua e heal h IT in e en ions while he sys em is being designed, de eloped and
deployed. The model is o ma i e and ela es e alua ion o he phases o he sys em
de elopmen . The li e-cycle e alua ion amewo k is a aluable ool o moni o he
de elopmen p ocess and he deploymen o a new sys em.
The HOT- i e alua ion amewo k [31] conside s Human, O ganiza ion and
Technology ac o s and ecognizes in e ela ed dimensions o heal h IT success and
de e mines bo h bene i s and sa is ac ion. ‘Fi ’ in he amewo k conce ns he abili y o
heal h IT sys em, s akeholde s and clinical p ac ices o align wi h each o he .
ISO s anda ds o human-cen ed design include guidelines o usabili y e alua ion
s udies [18,32] and s a e ha he ollowing in o ma ion is needed: Desc ip ion o he
in ended goals, a desc ip ion o he componen s o he con ex o use including use s,
asks, equipmen and en i onmen s, a ge o ac ual alues o e ec i eness, e iciency,
and sa is ac ion o he in ended con ex s. E alua ion should in ol e he ollowing
asks: Alloca ing o esou ces, planning o he e alua ion, ca ying ou su icien
es ing, analysing he esul s and p io i izing issues and p oposing solu ions, and
communica ing he solu ions app op ia ely [32].
Many analyses (e.g. [33-34]) ha e emphasized ha we need o unde s and he
changes ha heal h IT sys ems b ing in o a complex heal h ca e sys em. The socio-
echnical assessmen ool, STAT-HI [35], ocuses on he socio- echnical aspec s o
sys ems and hei implemen a ion in he heal h ca e o ganiza ional en i onmen .
A ecen con ibu ion o heal h IT e alua ion is made by WHO in he con ex o
heal h IT sys ems o de eloping coun ies [36] (on his opic, see also Chap e 26:
E alua ion o Heal h IT in low-income coun ies). This comp ises o nine high-le el
p inciples, he Bellagio p inciples, which co e e.g. e idence-based cul u e, high
quali y e alua i e da a collec ion and s akeholde engagemen (see also Chap e 1: The
need o e idence in heal h in o ma ics). The p inciples emphasize he o e all
esponsibili y o ensu e ha heal h IT solu ions and policies a e subjec ed o, and
in o med by, igo ous e alua ions and ha e alua ion indings should be used and
con ibu e o e idence gene a ion, syn hesis and documen a ion, including pee -
e iewed a icles.
10
Fo de ails on MAST, see: P. Doupi, E ol ing Heal h IT Sys ems E alua ion: The Con e gence o
Heal h In o ma ics and HTA, in: E. Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud
Heal h Technol In o m 222, IOS P ess, Ams e dam, 2016.
P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions296
3.2. GEP-HI guideline o design and execu e an e alua ion s udy p ojec
The discussed amewo ks and guidelines o e alua ion aim a imp o ing he quali y
o heal h IT e alua ion s udies. They p o ide guidance and suppo on how o plan and
pe o m e alua ion s udies, bu hei applica ion scopes and con ex s a y as well as
heo e ical ounda ions and me hodological app oaches. Thus a comp ehensi e
e alua ion guideline was de eloped o suppo especially planning and execu ion o an
e alua ion s udy o o e come he quali y p oblems ela ed o weak s udy planning and
me hodological choices. The guideline was named GEP-HI, Good E alua ion P ac ice
Guideline o Heal h In o ma ics [7].
The s a ing poin o he GEP-HI guideline de elopmen was he exis ing
knowledge, expe ience and li e a u e on e alua ion s udies, me hodologies, guidelines
de elopmen , codes o e hics and good implemen a ion p ac ices. In pa icula he
ollowing e iew ma e ials and ex books p o ided he ounda ion o p epa a ion o
he guideline [1, 9, 37-39].
The GEP-HI guideline has been de eloped h ough an in o mal consensus-seeking
p ocess, wi hou ballo ing, in he communi y o heal h IT e alua ion expe s, and i has
been egula ly in open discussion h ough he HISEVAL websi e
(h p://iig.umi .a /e mi) and many con e ence wo kshops. The p ima y au ho s o GEP-
HI we e all pa icipan s o he ESF HIS-EVAL Wo kshop [4, 5] and ac i e in he EFMI
(Eu opean Fede a ion o Medical In o ma ics) and IMIA (In e na ional Associa ion o
Medical In o ma ics) wo king g oups dealing wi h e alua ion o heal h IT sys ems ( o
he de ails on hese wo king g oups, see he book appendix).
The objec i e o he GEP-HI is o gi e ad ice on how o design and ca y ou
e alua ion s udies in a ious heal h IT con ex s [7]. The guideline lis s issues o
conside , and gi es ecommenda ions on how o design e alua ion s udies, how o
make me hodological choices, how o conduc s udies and how o de ine e alua ion
c i e ia a speci ic phases o he heal h IT li e cycle.
When applied, he GEP-HI guideline has po en ial o aise he quali y o e alua ion
s udies h ough ca e ul planning, and hus con ibu e o he accumula ion o he
scien i ic e idence base. GEP-HI is complemen ed by he STARE-HI – S a emen on
Repo ing o E alua ion S udies in Heal h In o ma ics [38], which p o ides guidance
on how o epo an e alua ion s udy ( o de ails on STARE-HI, see Chap e 24:
Publishing heal h IT e alua ion s udies).
The GEP-HI guideline [7] is di ided in o pa s co esponding o he phases o an
e alua ion s udy (Figu e 1). The heo e ical backg ound o he s udy phases is
complian wi h he in o ma ion sys em de elopmen models. Implemen a ion is an
i e a i e spi al; he opics a e in gene al epea ed in dep h o b ead h o achie e
p og ess du ing all phases, and eedback loops u ge o e isi ea lie phases when new
aspec s, addi ional in o ma ion, o changes in con ex appea .
The phases o GEP-HI guideline a e:
• P elimina y ou line p esen ing he pu pose o he s udy and he i s ideas on why,
o whom, and how he e alua ion should ake place,
• S udy design cla i ying he design issues o he e alua ion s udy.
• Ope a ionaliza ion o me hods making he me hodological app oach and me hods
conc e e and complian wi h he sys em ype, he o ganiza ion and he in o ma ion
need.
• P ojec planning de eloping plans and p ocedu es o he e alua ion p ojec .
P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions 297
• Execu ion o he e alua ion s udy accomplishing he designed e alua ion s udy.
• Comple ion o he e alua ion s udy epo ing, accoun ing, a chi ing o e alua ion
s udy esul s, inaliza ion o ou s anding issues and o mal closu e o he
e alua ion s udy.
Figu e 1: Phases o a heal h IT e alua ion acco ding o he GEP-HI guideline [7].
To p og ess om one phase o he nex phase a o mal accep ance is equi ed om
he ele an s akeholde s o he planned e alua ion s udy. Fo each phase a lis o i ems
is p esen ed (Table 1) and hese should be ca e ully conside ed du ing he s udy. All
phases oge he con ain some 60 de ailed i ems, which a e p esen ed in ela ion o he
e alua ion s udy phases. When designe s and execu e s o e alua ion s udies add ess
hese i ems, he plan, s uc u e, objec i es and esul s o he s udies will become mo e
obus and consequen ly he s udies con ibu e an impo an s ep owa ds e idence-
based heal h in o ma ics.
Table 1. I ems o be conside ed a each phase o a heal h IT e alua ion acco ding o GEP-HI.
Phase
no.
Phase I ems o he phase
1
P elimina y
ou line
Ǧ Pu pose o he s udy
Ǧ P ima y audience
Ǧ Iden i ica ion o he s udy unding pa y(ies)
Ǧ Fi s iden i ica ion o s akeholde s
Ǧ Iden i ica ion o equi ed expe ise
Ǧ The o ganiza ional and use con ex o he e alua ion s udy
Ǧ Objec o e alua ion, ype o heal h IT
Ǧ Fi s explo a ion o e alua ion me hods o be used
Ǧ E hical and legal issues
Ǧ Budge
Ǧ P elimina y pe missions o publica ion
Ǧ Resul o p elimina y ou line
Ǧ Fo mal accep ance o p oceed o he nex phase
P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions298
2
S udy design
Ǧ De ailed a ionale and objec i es o he s udy
Ǧ Key e alua ion issues, ques ions, indica o s
Ǧ S akeholde analysis/Social Ne wo k analysis
Ǧ S udy me hods
Ǧ O ganiza ional con ex , he s udy se ing
Ǧ Technical se ing, he ype o heal h IT
Ǧ Pa icipan s om he o ganiza ion
Ǧ P ojec imeline
Ǧ Ma e ial and p ac ical esou ces
Ǧ Es ablishmen o he s udy eam
Ǧ Risk analysis and quali y managemen
Ǧ Budge
Ǧ E hical and legal issues
Ǧ S a egy o epo ing and dissemina ing he esul s
Ǧ Resul o s udy design
Ǧ Fo mal accep ance o p oceed o he nex phase
3
Ope a iona-
liza ion o
me hods
Ǧ S udy ype
Ǧ App oach
Ǧ Assump ions and easibili y assessmen
Ǧ F ame o e e ence
Ǧ Timing
Ǧ Jus i ica ion o he me hodological app oach
Ǧ Expe ise
Ǧ Ou come measu es
Ǧ A oiding Bias
Ǧ Quali y con ol on da a (measu es)
Ǧ Pa icipan s
Ǧ E hical and legal issues
Ǧ Resul o ope a ionaliza ion o me hods
Ǧ App o al o ope a ionaliza ion o me hods
4
P ojec
planning
Ǧ P ojec managemen
Ǧ S udy low
Ǧ E alua ion ac i i y mapping
Ǧ Quali y managemen
Ǧ Risk managemen
Ǧ Rec ui men o necessa y s a
Ǧ In o m all ele an s akeholde s
Ǧ Resul o p ojec planning
Ǧ App o al o p ojec planning
5
Execu ion o
he
e alua ion
s udy
Ǧ Unde ake he s udy, collec da a and in e p e obse a ions
Ǧ Quali y con ol o indings and obse a ion o changes
Ǧ Con inuous p ojec managemen , quali y and isk managemen
Ǧ Regula epo s
Ǧ Final esul o execu ion o he e alua ion s udy
6 Comple ion
o he
e alua ion
s udy
Ǧ Accoun ing
Ǧ A chi ing
Ǧ Repo s and publica ions
The s eng h o he GEP-HI guideline is in o cing he use o go h ough a
checklis o ele an issues ha migh o he wise only ac in o mally as aci knowledge,
o e en be o e looked. This sys ema ic app oach will inc ease he likelihood o an
e alua ion ou come wi h he desi ed le el o accu acy and p ecision and hence an
inc eased e ec i eness, and addi ionally encou age he adop ion o a scien i ically
P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions 299