Quality of health IT evaluations

Quali y o Heal h IT E alua ions Pi kko NYKÄNEN a,1 and Johanna KAIPIO b a Uni e si y o Tampe e, School o In o ma ion Sciences, Tampe e, Finland b Aal o Uni e si y, School o Science, Espoo, Finland Abs ac . Heal h IT e alua ion s udies ha e o en been ound o be o limi ed quali y. To add ess his p oblem, se e al guidelines and amewo ks ha e been de eloped as ools o suppo imp o emen o he quali y o e alua ion s udies. In his con ibu ion, we e iew a ailable guidelines and hen p esen he Good E alua ion P ac ice Guideline in Heal h In o ma ics (GEP-HI) in mo e de ail. GEP- HI is a comp ehensi e guideline which suppo s especially planning and execu ion o a heal h IT e alua ion s udy. The GEP-HI guideline helps o o e come he quali y p oblems ela ed o weak s udy planning and me hodological s udy design. We also discuss applica ion o GEP-HI on an e alua ion p ojec and discuss he need o publish sys ema ically ollowing he ecognised publica ion guidelines. Finally we discuss he u u e end on mul i-me hod e alua ion app oaches. Keywo ds. E alua ion, quali y, scope, guideline, amewo k, heal h IT. 1. In oduc ion E alua ion is he means o assess he quali y, alue, e ec s and impac s o heal h IT in he heal h ca e en i onmen . E alua ion is de ined as he “ac o measu ing o explo ing p ope ies o a heal h in o ma ion sys em, in planning, de elopmen , implemen a ion, o ope a ion, he esul o which in o ms a decision o be made conce ning ha sys em in a speci ic con ex ” [1, p. 480]. E alua ion o e s me hods and ools o collec e idence abou he bene i s, quali y, e ec s and impac s o heal h IT. In heal h ca e p ac ice heal h IT applica ions o e challenging oppo uni ies o imp o e he heal h ca e sys em’s unc ioning, e ec i eness and ou comes as well as heal h ca e se ices quali y and deli e y, bu he e a e also p oblems and unan icipa ed e ec s ela ed o he use o IT [2,3]. 2 I is o u mos impo ance in he heal h ca e en i onmen ha heal h IT p o ides he op imal and sa e esul s and he e o e heal h IT applica ions need o be e alua ed wi h obus me hodologies and e alua ion esul s a e o be epo ed ollowing s uc u ed epo ing s anda ds [4,5]. E alua ion is di icul , i deals wi h alues and no ms and a ious o ganiza ional con ex s and s akeholde s’ in e es s, and i has o igh o unding and suppo [6]. Addi ionally, many po en ially applicable me hodologies and me hods exis and e alua ion esul s a e o be analysed and in e p e ed in he s udy con ex [4]. A challenge o imp o e he quali y o e alua ion s udies is o apply a sys ema ic 1 Co esponding au ho : P o . D . Pi kko Nykänen, Uni e si y o Tampe e, School o In o ma ion Sciences, PO Box 607, FIN-33014 Uni e si y o Tampe e, Finland, Pi kko.Nykanen@u a. i. 2 See also: F. Mag abi e al., Heal h IT o pa ien sa e y and imp o ing he sa e y o heal h IT, in: E. Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud Heal h Technol In o m 222, IOS P ess, Ams e dam, 2016. E idence-Based Heal h In o ma ics E. Ammenwe h and M. Rigby (Eds.) © 2016 The au ho s and IOS P ess. This a icle is published online wi h Open Access by IOS P ess and dis ibu ed unde he e ms o he C ea i e Commons A ibu ion Non-Comme cial License. doi:10.3233/978-1-61499-635-4-291 291 app oach, plan he e alua ion s udy ca e ully and execu e he s udy ollowing sys ema ic e alua ion guidelines [7]. In his con ibu ion we e iew he ocus and scope o he published e alua ion s udies, discuss he quali y p oblems ela ed o hese s udies, and p esen heal h IT e alua ion guidelines and amewo ks, namely he GEP-HI guideline. 2. The scope and quali y o e alua ion s udies The quali y o an e alua ion s udy is dependen on many ac o s, e.g. on he objec i i y o he s udy and on he independence o e alua o s, e e ing o hei independence on economic in e es s, on in ellec ual in e es s and on he a ious s akeholde s’ in e es s. An e alua ion s udy mus also be scien i ically well-es ablished on obus heo ies and me hodologies 3 , and he s udy should be pe o med ollowing he p inciples o scien i ic esea ch. As he e a e many in e es ing ocuses o e alua ion, e.g. economics, e iciency, usabili y o heal h IT, sa e y, p i acy and secu i y, compliance wi h he clinical p ocess, unc ionali y o heal h IT, e ec s and impac s on heal h ca e ou comes, he e is also need o use many di e en me hods sui able o measu ing he e alua ion c i e ia o in e es . These po en ial me hods co e e.g. quali a i e and quan i a i e me hods, s a is ics, heu is ics, e hnog aphy, human-sys em in e ac ion obse a ions, da a mining and quali y analysis, cos -e ec i eness and cos -bene i analyses. 4 The e a e also many po en ial pe spec i es o e alua ion, ep esen ing a ious s akeholde s’ iewpoin s, e.g. manage ial, clinical, echnical, and hese iewpoin s may be s udied a a ious le els o heal h ca e sys em – local, egional and na ional, o e en a EU o in e na ional le els. 5 I is o u mos impo ance, when planning an e alua ion s udy, o elabo a e and de ine how hese issues a e ela ed o he s udy: Co e age o scien i ic obus ness, ele ance o he cu en pu pose o he s udy, bes i o he impo an cha ac e is ics o he speci ic cu en need and ele ance o he me hods. The quali y also comp ises he quali y in publica ion o he s udy. E alua ion s udies ha e been pe o med since 1960’s, howe e , mos wi h a a he na ow scope, o en ocusing on how heal h IT sys ems a e ela ed o p o essionals’ oles, change managemen and use in ol emen [8]. La e he s udies ha e co e ed also he success aspec s and lessons lea ned in implemen a ion and de elopmen o heal h IT sys ems. In many cases e alua ion has been led by esea ch in e es s o de elop me hodologies o o s udy he heal h ca e p ocesses. F om 1980’s onwa ds also managemen issues, use accep ance and adop ion o heal h IT sys ems in heal h ca e o ganiza ions ha e been s udied. Kaplan and Shaw made a e iew o how aspec s ela ed o people, o ganiza ional and social issues ha e been conside ed in heal h IT e alua ions [9]. They emphasized he need o pay mo e a en ion o hese issues du ing 3 See also: J. B ende , Theo e ical basis o heal h IT e alua ion, in: E. Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud Heal h Technol In o m 222, IOS P ess, Ams e dam, 2016. 4 See also: D. Luzi e al., Economic e alua ion o heal h IT, in: ibid. 5 See also: L. Lee e al., Unde s anding s akeholde in e es s and pe spec i es in e alua ions o heal h IT, in: ibid. P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions292 heal h IT sys em design, implemen a ion and use and emphasize he need o in eg a e mul i-me hod e alua ion o he whole li e cycle o he heal h IT sys em. 6 Van de Loo analysed e alua ion s udies published be ween 1967 and 1995 wi h ega d o ype o he sys em, s udy design, da a collec ion, economic e alua ion and ype o e ec measu e [10]. He ound pe o mance o use s o be a c i e ion in many s udies, as well as ime sa ings and cos s o pa ien ca e. By con as , use sa is ac ion was an e alua ion c i e ion only in 11% o s udies. These esul s e lec well he si ua ion in 1990’s, usabili y and e ec s and impac s o heal h IT on heal h ca e ou comes and quali y we e no highly alued issues ha ime. Ammenwe h and de Keize [11] ound ha explana o y esea ch and quan i a i e me hods ha e domina ed e alua ion s udies du ing he las 20 yea s. Howe e , he s udies on ou come quali y and cos s o pa ien ca e, pa ien sa is ac ion and pa ien beha iou ha e ecei ed mo e a en ion ecen ly as well as s udies on quali y o p ocesses. he same ime, he numbe o labo a o y s udies and echnical e alua ion s udies has declined while he numbe o s udies ocusing on he in luence o heal h IT on quali y o ca e p ocesses o ou come o pa ien ca e has inc eased. I was ypical o ea ly s udies, e.g. e alua ion o decision suppo sys ems and expe sys ems, ha he ocus was on echnical issues, on ha dwa e and so wa e quali y and on sys em pe o mance issues. Rigby [12] also no ed ha he ocus o e alua ion o a heal h IT sys em changes du ing i s li e cycle: Du ing he implemen a ion phase e alua ion add esses o en echnical aspec s, bu wi h a comple ed sys em ocuses on impac s on pa ien ca e. Ammenwe h and de Keize ound also ha he numbe o in e - o ganisa ional s udies has inc eased e lec ing he end owa ds coope a i e and sha ed ca e. In many ecen s udies, use sa is ac ion and e iciency o pa ien ca e a e he mos equen ly add essed e alua ion c i e ia, and he e is a slowly g owing end owa ds e alua ion s udies co e ing mo e han one e alua ion c i e ion [11]. Also he heal h IT de elopmen s discussed in [12,13] showed he impo ance o e alua ion, he challenges and he p oblems in in eg a ed me hodologies and in endo sing he impo ance o obus s udies. These e iews show ha explana i e esea ch and quan i a i e me hods ha e domina ed heal h IT e alua ion esea ch o a long ime, bu s udies ocusing on p ocess quali y o ou come quali y o pa ien ca e ha e inc eased la ely. Also human, social and o ganiza ional aspec s ha e been included in e alua ion s udies o some ex en , and p omisingly esea ch in his aspec is g owing [9]. 7 Quali a i e me hods ha e been used in s udies ocused on use accep ance, usabili y and use ulness o on o ganiza ional and social impac s o heal h IT. Many e alua ions ha e been ocused on one use ’s indi idual con ex : how a heal h p o essional uses he sys em in he speci ic con ex , how he/she accep s he echnology and how echnology i s in o his/he wo k p ocesses, lea ing he mo e wide usage con ex s un ouched. The e iews also e eal quali y p oblems in e alua ion s udies: he e a e p oblems ela ed e.g. o weak s udy planning wi hou a sys ema ic, scien i ic me hodology, alse implici assump ions made in he s udy, expe imen al e o s in he esea ch se ing, unde - o o e -in e p e a ion o he esul s o alse conclusions, inclusion o non- 6 See also: P.J. Sco e al., Mixed me hods: a pa adigm o holis ic e alua ion o heal h in o ma ion sys em, in: E. Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud Heal h Technol In o m 222, IOS P ess, Ams e dam, 2016. 7 See also: B. Kaplan, E alua ion o people and o ganiza ional Issues – Socio echnical e hnog aphic e alua ion, in: ibid. P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions 293 neu al e alua o s, in a- o in e -o ganiza ional a iabili y in he e alua ion objec o no el y o echnology. In some cases he e seem also o be p oblems in selec ing such e alua ion me hods ha a e capable o measu e hose a iables and aspec s ha desc ibe he phenomenon unde s udy. These quali y p oblems show ha he e is eal need o imp o e he quali y o e alua ion s udies. Since he ea ly 2000’s esea che s in he heal h in o ma ics ield ha e poin ed ou he need o include also use pe spec i e and usabili y, in he e alua ion s udies [14, 15]. This need is mo i a ed by ecogni ion o he impo ance o good usabili y as he heal h IT sys ems a e planned o use in heal h ca e clinical p ac ice. 8 We analyse b ie ly also he published usabili y e alua ion s udies, on hei me hodological app oach and quali y p oblems. A e iew in [16] showed ha in gene al, empi ical usabili y s udies a e hea ily a ec ed by he adi ional app oaches o e alua ion o human-compu e in e ac ion. Usabili y s udies ha e mainly applied adi ional usabili y e alua ion me hods, pa icula ly usabili y es ing and inspec ions. A conside able numbe o usabili y e alua ion s udies ha e concen a ed on he la e phases o heal h IT de elopmen , and e alua ion has been ocused on sys ems ha a e al eady in use. Recen usabili y s udies a e cha ac e ized by a na ow ocus on use and usabili y issues, emphasis on summa i e e alua ion a he han on design o de elopmen , isola ed sys em de elopmen , and emphasis on in o ma ion sys ems and da a managemen [16]. Also in many usabili y s udies he e is a lack o unde s anding o he con ex ual aspec s o usabili y, and he cha ac e is ics o clinical wo k con ex s, hough he widely known de ini ions o usabili y [17,18] emphasize he need o unde s and usabili y as a con ex ual p ope y. The e a e quali y p oblems also in he usabili y e alua ion s udies, e.g. he s udies o en ocus on a single end-use g oup pe spec i e, use in e ace componen s, o use o he sys em in a speci ic con ex , bu do no p o ide a comp ehensi e pic u e o usabili y o a la ge-scale heal hca e in o ma ion sys em. Fu he , he e alua ion s udies a ely discuss he ela ionship be ween single-sys em de elopmen and he exis ing echnology se ing in heal hca e, o he cha ac e is ics o a ious use con ex s in which he e alua ed sys em is used. Typically he s udies discuss summa i e usabili y esul s on wo king sys ems; his leads o a ocus on he p oblems wi h adop ing cu en sys ems in a gi en heal hca e en i onmen and di e s concen a ion om he design o de elopmen o new, be e sys ems. In gene al, he quali y o usabili y e alua ion s udies could be enhanced by applying app op ia e usabili y e alua ion me hods o he han adi ional ones, e.g. con ex ual inqui y [19], which akes in o conside a ion he con ex ual aspec s o heal hca e en i onmen . The planning o e alua ion s udies should aim a p o iding e alua ion esul s bu also unde s anding o he unde lying p oblems and suppo o he de elopmen wo k. Usabili y e alua ion s udies ha e in mos cases been done sepa a ely om o he e alua ion s udies, e.g. om e ec i eness, e ec s and impac s e alua ion s udies. Howe e , i would be bene icial o in eg a e usabili y and o he e alua ion c i e ia in a s udy. Usabili y is an impo an quali y aspec o heal h IT and usabili y is hea ily dependen on good heal h IT sys em design, implemen a ion and adop ion [20]. Some o he ecen usabili y s udies ha e al eady emphasized a mo e holis ic iew o 8 See also: R. Ma cilly e al., F om usabili y enginee ing o e idence-based usabili y in heal h IT, in: E. Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud Heal h Technol In o m 222, IOS P ess, Ams e dam, 2016. P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions294 usabili y e alua ion and called o mul i-me hod app oaches and ocus mo e on heal h p o essionals’ clinical collabo a ion. Among o he s, Kushni uk [21, 22] has exp essed a conce n ha when he echnology applica ions become mo e complex, e alua ion me hodologies will need o be con inually e ined in o de o keep pace. Mul i-me hod app oaches a e called o as opposed o using one single me hod [23]. Kaipio emphasizes [16] ha he use -cen ed app oach will play an essen ial ole in u u e heal h IT design and de elopmen , because knowledge and unde s anding o he needs o a ious use g oups, as well as clinical use con ex s, a e impo an and should be pa o all heal h IT design and de elopmen phases. These indings a e suppo ed in [24] by emphasizing he need o p edic i e e alua ion me hods o accu a ely iden i y usabili y issues ha a ise om he in e ac ion, sha ing and communica ion equi emen s o clinical wo k. Fu he mo e, heal h IT sys ems ha e many use g oups and he use s use he sys ems o a ious pu poses, such as clinical, nu sing, medical, adminis a i e, manage ial, s a is ical and economical pu poses and oday, also pa ien s and ci izens use heal h IT sys ems o hei pe sonal heal h and wellness pu poses. The e o e, i is impo an o conside he a ie y o uses and use con ex s, and also he a ie y o e alua ion c i e ia, since he na u e o clinical wo k, as well as he physical and o ganiza ional en i onmen s in he wo kplace, may di e signi ican ly be ween heal hca e uni s. This s a e o a ai s o e alua ion s udies and quali y challenges emphasizes he need o sys ema ic app oaches and guidelines o design and o ca y ou di e en kinds o e alua ion s udies o p o ide e idence abou he impac s and ac ual e iciency, quali y, usabili y and sa e y o heal h IT [25]. 3. Guidelines and amewo ks o heal h IT e alua ions 3.1. O e iew on a ailable guidelines and amewo ks Many amewo ks and guidelines de eloped o e alua ion exis , aiming a suppo ing and imp o ing s udies so ha heal h IT e alua ion is conduc ed o he highes me hodological and scien i ic s anda ds. These amewo ks and guidelines di e in e ms o gene ali y, speci ici y and iming ela ed o sys em de elopmen phases and heo e ical unde pinning. In his sec ion, we will in oduce some selec ed amewo ks and guidelines o heal h IT e alua ion. Kaplan has sugges ed an e alua ion amewo k o 4C - Communica ion, Con ol, Ca e and Con ex [26]. This 4C model calls o mul i-me hod longi udinal design o o ma i e and summa i e e alua ions. 9 Shaw has in oduced he CHEATS- amewo k [27] which iden i ies six aspec s o be impo an in e alua ion: Clinical, Human and O ganiza ional, Educa ional, Adminis a i e, Technical and Social. A Model o ASsessmen o Telemedicine applica ions (MAST) [28] lis s aspec s o e alua ion wi hin se en domains o ou comes: heal h p oblem and cha ac e is ics o he applica ion; sa e y; clinical e ec i eness; pa ien pe spec i es; economic aspec s; o ganiza ional aspec s; and socio-cul u al, e hical and legal aspec s. MAST is planned o be a oolki , a checklis o issues ha need o be conside ed in e alua ion. MAST is 9 See also: B. Kaplan, E alua ion o people and o ganiza ional Issues – Socio echnical e hnog aphic e alua ion, in: E. Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud Heal h Technol In o m 222, IOS P ess, Ams e dam, 2016. P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions 295 based on a heal h echnology assessmen (HTA) app oach and ocuses on elemedicine sys ems, no on heal h IT gene ally. MAST does no conside he execu ion and managemen o an e alua ion p ojec . 10 Cusack e al. de eloped he AHRQ oolki [29] o p o ide s ep-by-s ep guidance o de eloping e alua ion plans o heal h IT p ojec s. AHRQ assis s e alua o s in de ining he goals o e alua ion, in iden i ying wha is impo an o s akeholde s, wha needs o be measu ed o sa is y s akeholde s, wha is ealis ic and easible o measu e, and how o measu e hese i ems. The AHRQ oolki is e y use ul om he me hodological poin o iew. I can be applied wi hin o he mo e gene ic guidelines. The oolki does no , howe e , gi e guidance on he e alua ion p ojec i sel , how o manage i , how o ca y ou he p ojec , o how o comple e and epo he s udy. A li e-cycle amewo k o e alua ion by Cla ke e al. [30] is ocused on how o e alua e heal h IT in e en ions while he sys em is being designed, de eloped and deployed. The model is o ma i e and ela es e alua ion o he phases o he sys em de elopmen . The li e-cycle e alua ion amewo k is a aluable ool o moni o he de elopmen p ocess and he deploymen o a new sys em. The HOT- i e alua ion amewo k [31] conside s Human, O ganiza ion and Technology ac o s and ecognizes in e ela ed dimensions o heal h IT success and de e mines bo h bene i s and sa is ac ion. ‘Fi ’ in he amewo k conce ns he abili y o heal h IT sys em, s akeholde s and clinical p ac ices o align wi h each o he . ISO s anda ds o human-cen ed design include guidelines o usabili y e alua ion s udies [18,32] and s a e ha he ollowing in o ma ion is needed: Desc ip ion o he in ended goals, a desc ip ion o he componen s o he con ex o use including use s, asks, equipmen and en i onmen s, a ge o ac ual alues o e ec i eness, e iciency, and sa is ac ion o he in ended con ex s. E alua ion should in ol e he ollowing asks: Alloca ing o esou ces, planning o he e alua ion, ca ying ou su icien es ing, analysing he esul s and p io i izing issues and p oposing solu ions, and communica ing he solu ions app op ia ely [32]. Many analyses (e.g. [33-34]) ha e emphasized ha we need o unde s and he changes ha heal h IT sys ems b ing in o a complex heal h ca e sys em. The socio- echnical assessmen ool, STAT-HI [35], ocuses on he socio- echnical aspec s o sys ems and hei implemen a ion in he heal h ca e o ganiza ional en i onmen . A ecen con ibu ion o heal h IT e alua ion is made by WHO in he con ex o heal h IT sys ems o de eloping coun ies [36] (on his opic, see also Chap e 26: E alua ion o Heal h IT in low-income coun ies). This comp ises o nine high-le el p inciples, he Bellagio p inciples, which co e e.g. e idence-based cul u e, high quali y e alua i e da a collec ion and s akeholde engagemen (see also Chap e 1: The need o e idence in heal h in o ma ics). The p inciples emphasize he o e all esponsibili y o ensu e ha heal h IT solu ions and policies a e subjec ed o, and in o med by, igo ous e alua ions and ha e alua ion indings should be used and con ibu e o e idence gene a ion, syn hesis and documen a ion, including pee - e iewed a icles. 10 Fo de ails on MAST, see: P. Doupi, E ol ing Heal h IT Sys ems E alua ion: The Con e gence o Heal h In o ma ics and HTA, in: E. Ammenwe h, M. Rigby (eds.), E idence-Based Heal h In o ma ics, S ud Heal h Technol In o m 222, IOS P ess, Ams e dam, 2016. P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions296 3.2. GEP-HI guideline o design and execu e an e alua ion s udy p ojec The discussed amewo ks and guidelines o e alua ion aim a imp o ing he quali y o heal h IT e alua ion s udies. They p o ide guidance and suppo on how o plan and pe o m e alua ion s udies, bu hei applica ion scopes and con ex s a y as well as heo e ical ounda ions and me hodological app oaches. Thus a comp ehensi e e alua ion guideline was de eloped o suppo especially planning and execu ion o an e alua ion s udy o o e come he quali y p oblems ela ed o weak s udy planning and me hodological choices. The guideline was named GEP-HI, Good E alua ion P ac ice Guideline o Heal h In o ma ics [7]. The s a ing poin o he GEP-HI guideline de elopmen was he exis ing knowledge, expe ience and li e a u e on e alua ion s udies, me hodologies, guidelines de elopmen , codes o e hics and good implemen a ion p ac ices. In pa icula he ollowing e iew ma e ials and ex books p o ided he ounda ion o p epa a ion o he guideline [1, 9, 37-39]. The GEP-HI guideline has been de eloped h ough an in o mal consensus-seeking p ocess, wi hou ballo ing, in he communi y o heal h IT e alua ion expe s, and i has been egula ly in open discussion h ough he HISEVAL websi e (h p://iig.umi .a /e mi) and many con e ence wo kshops. The p ima y au ho s o GEP- HI we e all pa icipan s o he ESF HIS-EVAL Wo kshop [4, 5] and ac i e in he EFMI (Eu opean Fede a ion o Medical In o ma ics) and IMIA (In e na ional Associa ion o Medical In o ma ics) wo king g oups dealing wi h e alua ion o heal h IT sys ems ( o he de ails on hese wo king g oups, see he book appendix). The objec i e o he GEP-HI is o gi e ad ice on how o design and ca y ou e alua ion s udies in a ious heal h IT con ex s [7]. The guideline lis s issues o conside , and gi es ecommenda ions on how o design e alua ion s udies, how o make me hodological choices, how o conduc s udies and how o de ine e alua ion c i e ia a speci ic phases o he heal h IT li e cycle. When applied, he GEP-HI guideline has po en ial o aise he quali y o e alua ion s udies h ough ca e ul planning, and hus con ibu e o he accumula ion o he scien i ic e idence base. GEP-HI is complemen ed by he STARE-HI – S a emen on Repo ing o E alua ion S udies in Heal h In o ma ics [38], which p o ides guidance on how o epo an e alua ion s udy ( o de ails on STARE-HI, see Chap e 24: Publishing heal h IT e alua ion s udies). The GEP-HI guideline [7] is di ided in o pa s co esponding o he phases o an e alua ion s udy (Figu e 1). The heo e ical backg ound o he s udy phases is complian wi h he in o ma ion sys em de elopmen models. Implemen a ion is an i e a i e spi al; he opics a e in gene al epea ed in dep h o b ead h o achie e p og ess du ing all phases, and eedback loops u ge o e isi ea lie phases when new aspec s, addi ional in o ma ion, o changes in con ex appea . The phases o GEP-HI guideline a e: • P elimina y ou line p esen ing he pu pose o he s udy and he i s ideas on why, o whom, and how he e alua ion should ake place, • S udy design cla i ying he design issues o he e alua ion s udy. • Ope a ionaliza ion o me hods making he me hodological app oach and me hods conc e e and complian wi h he sys em ype, he o ganiza ion and he in o ma ion need. • P ojec planning de eloping plans and p ocedu es o he e alua ion p ojec . P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions 297 • Execu ion o he e alua ion s udy accomplishing he designed e alua ion s udy. • Comple ion o he e alua ion s udy epo ing, accoun ing, a chi ing o e alua ion s udy esul s, inaliza ion o ou s anding issues and o mal closu e o he e alua ion s udy. Figu e 1: Phases o a heal h IT e alua ion acco ding o he GEP-HI guideline [7]. To p og ess om one phase o he nex phase a o mal accep ance is equi ed om he ele an s akeholde s o he planned e alua ion s udy. Fo each phase a lis o i ems is p esen ed (Table 1) and hese should be ca e ully conside ed du ing he s udy. All phases oge he con ain some 60 de ailed i ems, which a e p esen ed in ela ion o he e alua ion s udy phases. When designe s and execu e s o e alua ion s udies add ess hese i ems, he plan, s uc u e, objec i es and esul s o he s udies will become mo e obus and consequen ly he s udies con ibu e an impo an s ep owa ds e idence- based heal h in o ma ics. Table 1. I ems o be conside ed a each phase o a heal h IT e alua ion acco ding o GEP-HI. Phase no. Phase I ems o he phase 1 P elimina y ou line Ǧ Pu pose o he s udy Ǧ P ima y audience Ǧ Iden i ica ion o he s udy unding pa y(ies) Ǧ Fi s iden i ica ion o s akeholde s Ǧ Iden i ica ion o equi ed expe ise Ǧ The o ganiza ional and use con ex o he e alua ion s udy Ǧ Objec o e alua ion, ype o heal h IT Ǧ Fi s explo a ion o e alua ion me hods o be used Ǧ E hical and legal issues Ǧ Budge Ǧ P elimina y pe missions o publica ion Ǧ Resul o p elimina y ou line Ǧ Fo mal accep ance o p oceed o he nex phase P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions298 2 S udy design Ǧ De ailed a ionale and objec i es o he s udy Ǧ Key e alua ion issues, ques ions, indica o s Ǧ S akeholde analysis/Social Ne wo k analysis Ǧ S udy me hods Ǧ O ganiza ional con ex , he s udy se ing Ǧ Technical se ing, he ype o heal h IT Ǧ Pa icipan s om he o ganiza ion Ǧ P ojec imeline Ǧ Ma e ial and p ac ical esou ces Ǧ Es ablishmen o he s udy eam Ǧ Risk analysis and quali y managemen Ǧ Budge Ǧ E hical and legal issues Ǧ S a egy o epo ing and dissemina ing he esul s Ǧ Resul o s udy design Ǧ Fo mal accep ance o p oceed o he nex phase 3 Ope a iona- liza ion o me hods Ǧ S udy ype Ǧ App oach Ǧ Assump ions and easibili y assessmen Ǧ F ame o e e ence Ǧ Timing Ǧ Jus i ica ion o he me hodological app oach Ǧ Expe ise Ǧ Ou come measu es Ǧ A oiding Bias Ǧ Quali y con ol on da a (measu es) Ǧ Pa icipan s Ǧ E hical and legal issues Ǧ Resul o ope a ionaliza ion o me hods Ǧ App o al o ope a ionaliza ion o me hods 4 P ojec planning Ǧ P ojec managemen Ǧ S udy low Ǧ E alua ion ac i i y mapping Ǧ Quali y managemen Ǧ Risk managemen Ǧ Rec ui men o necessa y s a Ǧ In o m all ele an s akeholde s Ǧ Resul o p ojec planning Ǧ App o al o p ojec planning 5 Execu ion o he e alua ion s udy Ǧ Unde ake he s udy, collec da a and in e p e obse a ions Ǧ Quali y con ol o indings and obse a ion o changes Ǧ Con inuous p ojec managemen , quali y and isk managemen Ǧ Regula epo s Ǧ Final esul o execu ion o he e alua ion s udy 6 Comple ion o he e alua ion s udy Ǧ Accoun ing Ǧ A chi ing Ǧ Repo s and publica ions The s eng h o he GEP-HI guideline is in o cing he use o go h ough a checklis o ele an issues ha migh o he wise only ac in o mally as aci knowledge, o e en be o e looked. This sys ema ic app oach will inc ease he likelihood o an e alua ion ou come wi h he desi ed le el o accu acy and p ecision and hence an inc eased e ec i eness, and addi ionally encou age he adop ion o a scien i ically P. Nykänen and J. Kaipio / Quali y o Heal h IT E alua ions 299