O iginal Pape
A Mobile Heal h Solu ion Complemen ing
Psychopha macology-Suppo ed Smoking Cessa ion:Randomized
Con olled T ial
Lau a Ca asco-He nandez1,2, D med; F ancisco Jóda -Sánchez3, PhD; F ancisco Núñez-Benjumea3, MSc; Jesús
Mo eno Conde3, MSc; Ma co Mesa González1, MSc; An ón Ci i -Balcells4, P o D ; San iago Ho s-F aile5, MSc;
Ca los Luis Pa a-Calde ón3, MSc; Panagio is D Bamidis6, P o D , PhD; F ancisco O ega-Ruiz1, D med, PhD
1Smoking Cessa ion Uni , Medical-Su gical Uni o Respi a o y Diseases, Vi gen del Rocío Uni e si y Hospi al, Se ille, Spain
2Cen o de In es igación Biomédica en Red de En e medades Respi a o ias, Ca los III Ins i u e o Heal h, Mad id, Spain
3Resea ch and Inno a ion G oup in Biomedical In o ma ics, Biomedical Enginee ing and Heal h Economy, Ins i u e o Biomedicine o Se ille, Vi gen
del Rocío Uni e si y Hospi al, Spanish Na ional Resea ch Council, Uni e si y o Se ille, Se ille, Spain
4Depa men o A chi ec u e and Compu e Technology, School o Compu e Enginee ing, Uni e sidad de Se illa, Se ille, Spain
5Salumedia Labs, Se ille, Spain
6Medical Physics Labo a o y, School o Medicine, A is o le Uni e si y o Thessaloniki, Thessaloniki, G eece
Co esponding Au ho :
F ancisco Jóda -Sánchez, PhD
Resea ch and Inno a ion G oup in Biomedical In o ma ics
Biomedical Enginee ing and Heal h Economy, Ins i u e o Biomedicine o Se ille
Vi gen del Rocío Uni e si y Hospi al, Spanish Na ional Resea ch Council, Uni e si y o Se ille
A enida Manuel Siu o S/N
Se ille, 41013
Spain
Phone: 34 670943651
Email: [email p o ec ed]
Abs ac
Backg ound: Smoking cessa ion is a pe sis en leading public heal h challenge. Mobile heal h (mHeal h) solu ions a e eme ging
o imp o e smoking cessa ion ea men s. P e ious app oaches ha e p oposed suppo ing cessa ion wi h ailo ed mo i a ional
messages. Some managed o p o ide sho - e m imp o emen s in smoking cessa ion. Ye , hese app oaches we e ei he s a ic in
e ms o pe sonaliza ion o human-based nonscalable solu ions. Addi ionally, long- e m e ec s we e nei he p esen ed no assessed
in combina ion wi h exis ing psychopha macological he apies.
Objec i e: This s udy aimed o analyze he long- e m e icacy o a mobile app suppo ing psychopha macological he apy o
smoking cessa ion and complemen a ily assess he in ol ed inno a i e echnology.
Me hods: A 12-mon h, andomized, open-label, pa allel-g oup ial compa ing smoking cessa ion a es was pe o med a Vi gen
del Rocío Uni e si y Hospi al in Se ille (Spain). Smoke s we e andomly alloca ed o a con ol g oup (CG) ecei ing usual ca e
(psychopha macological ea men , n=120) o an in e en ion g oup (IG) ecei ing psychopha macological ea men and using
a mobile app p o iding a i icial in elligence–gene a ed and ailo ed smoking cessa ion suppo messages (n=120). The seconda y
objec i es we e o analyze heal h- ela ed quali y o li e and moni o heal hy li es yle and physical exe cise habi s. Sa e y was
assessed acco ding o he p esence o ad e se e en s ela ed o he pha macological he apy. Pe -p o ocol and in en ion- o- ea
analyses we e pe o med. Incomple e da a and mul inomial eg ession analyses we e pe o med o assess he a iables in luencing
pa icipan cessa ion p obabili y. The echnical solu ion was assessed acco ding o he p ecision o he ailo ed mo i a ional
smoking cessa ion messages and use engagemen . Cessa ion and no cessa ion subg oups we e compa ed using es s. A olun a y
sa is ac ion ques ionnai e was adminis e ed a he end o he in e en ion o all pa icipan s who comple ed he ial.
Resul s: In he IG, abs inence was 2.75 imes highe (adjus ed OR 3.45, P=.01) in he pe -p o ocol analysis and 2.15 imes
highe (adjus ed OR 3.13, P=.002) in he in en ion- o- ea analysis. Los da a analysis and mul inomial logis ic models showed
di e en pa e ns in pa icipan s who d opped ou . Rega ding sa e y, 14 o 120 (11.7%) IG pa icipan s and 13 o 120 (10.8%)
CG pa icipan s had 19 and 23 ad e se e en s, espec i ely (P=.84). None o he clinical seconda y objec i e measu es showed
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 1h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
ele an di e ences be ween he g oups. The sys em was able o lea n and ailo messages o imp o ed e ec i eness in suppo ing
smoking cessa ion bu was unable o educe he ime be ween a message being sen and opened. In ei he case, he e was no
ele an di e ence be ween he cessa ion and no cessa ion subg oups. Howe e , a signi ican di e ence was ound in sys em
engagemen a 6 mon hs (P=.04) bu no in all subsequen mon hs. High sys em app ecia ion was epo ed a he end o he s udy.
Conclusions: The p oposed mHeal h solu ion complemen ing psychopha macological he apy showed g ea e e icacy o
achie ing 1-yea obacco abs inence as compa ed wi h psychopha macological he apy alone. I p o ides a basis o a i icial
in elligence–based u u e app oaches.
T ial Regis a ion: ClinicalT ials.go NCT03553173; h ps://clinical ials.go /c 2/show/NCT03553173
In e na ional Regis e ed Repo Iden i ie (IRRID): RR2-10.2196/12464
(JMIR Mheal h Uheal h 2020;8(4):e17530) doi: 10.2196/17530
KEYWORDS
smoking cessa ion; beha io al change; heal h ecommende sys ems; mHeal h; andomized con olled ial
In oduc ion
Tobacco use p esen s a majo p e en able public heal h p oblem;
i is he leading cause o heal h de e io a ion and p ema u e
dea h. The Wo ld Heal h O ganiza ion ecognizes smoking as
a ch onic sys emic disease among he addic ions capable o
p oducing high physical damage ela ed o mul iple diseases
[1]. Fu he , smoking kills o e 7 million people annually, wi h
global cos s es ima ed a US $1.4 illion [2].
Valida ed app oaches o acili a e smoking cessa ion include
nico ine eplacemen he apy, pha macological ea men (ie,
bup opion and a enicline), and beha io al and psychological
suppo . Combining beha io al and psychological suppo wi h
pha macological ea men is cu en ly he mos e ec i e
in e en ion o achie ing obacco abs inence [3]. Mo eo e ,
bup opion was shown o app oxima ely double he likelihood
o long- e m obacco abs inence as compa ed wi h placebo [4].
Fu he mo e, a enicline (2 mg, o al daily dose) was shown o
iple he likelihood o main aining long- e m obacco abs inence
as compa ed wi h placebo [5]. Despi e he p o en e icacy o
hese ea men s, a meaning ul numbe o smoke s ail o s op
smoking, and many ac o s in luence hei success, including
he mo i a ion o s op and con inuous suppo .
Mobile and web-based in e en ions ha e been used o acili a e
e ec i e beha io al changes in di e en heal h domains [6-9],
including smoking cessa ion. Scien i ic e idence has indica ed
ha mobile phone–based ex messages suppo ing smoking
cessa ion we e nea ly 1.7 imes mo e success ul han a con ol
app oach a 6 mon hs [10]. I has been p o en ha he beha io al
change impac is highe when use s ecei e mul iple ailo ed
heal h ecommenda ions [11,12]. Howe e , in p e ious s udies,
a adi ional compu e - ailo ing app oach was used o gene a e
such pe sonalized heal h ecommenda ions. Compu e ailo ing
in ol es he gene a ion o pa ien -speci ic ecommenda ions,
ypically in he o m o messages, by compu e s, and i is
pe o med a e pe sonal assessmen o ma ch he cha ac e is ics,
needs, and in e es s o he pa ien s [13-16]. Cupe ino e al [17]
ecen ly pilo ed a 12-week smoking cessa ion p og am in ol ing
ex messaging wi h pha maco he apy suppo , which showed
p omising abs inence a es a 3 mon hs and high pa icipan
sa is ac ion. Howe e , he e is no e idence o he long- e m
abs inence e icacy o mobile-based ailo ed in e en ions
combined wi h psychopha macological he apies.
As pa o he SmokeF eeB ain H2020 Eu opean Commission
p ojec [18], he Social-Local-Mobile (So-Lo-Mo) s udy
in es iga ed mobile and a i icial in elligence (AI) echnologies
(heal h ecommende sys em [HRS]) as a complemen a y aid
o pha macological ea men s. The expe imen al in e en ion
ocused on p o iding ubiqui ous ailo ed suppo o pa ien s
willing o s op smoking h ough a digi al he apeu ic mobile
app solu ion.
Pa icipan s ecei ing he So-Lo-Mo in e en ion we e p o ided
wi h access o a code o be in oduced in he app. The code
ac i a ed he app, connec ing i wi h he hospi al pa ien
da abase. The app au oma ically downloaded he necessa y da a
o ini ialize he app p o ile, educing he bu den o ha ing o
c ea e a use p o ile. The mobile app was connec ed wi h an AI
sys em designed o lea n om pa ien in e es s h ough hei
in e ac ions wi h he app o dynamically (1) de e mine,
pe sonalize, and send mo i a ional messages o suppo smoking
cessa ion; (2) schedule he message deli e y equency
acco ding o he ans heo e ical model o beha io al change
[19]; and (3) calcula e he mos con enien ime o send a
mo i a ional message o suppo smoking cessa ion o each
pa ien . Speci ically, he AI sys em used o ailo mo i a ional
messages is a hyb id HRS, whose design has been p esen ed
by Ho s-F aile e al [20].
The mo i a ional messages we e designed a e conduc ing
semis uc u ed in e iews in ol ing wo smoking cessa ion
expe s (a pulmonologis and a psychologis ). Based on hei
commen s, we iden i ied he ollowing i e di e en opics,
which we e he mos ele an o pa icipan s o succeed in hei
smoking cessa ion p ocess: (1) gene al mo i a ion; (2) heal hy
die ecommenda ions; (3) ecommenda ions o an ac i e
li es yle; (4) posi i e ein o cemen messages o mee ac i i y
goals; and (5) bene i s o being a nonsmoke . A o al o 150
di e en messages we e w i en o each opic by a heal h
communica ion and heal h p omo ion PhD candida e, and hey
we e alida ed by he wo smoking cessa ion expe s. This
numbe o messages was chosen o ensu e ha , e en in he wo s
case scena io whe e he AI sys em de e mines ha a use needs
o ecei e messages o a single opic, no use would ge he
same message wice du ing he in e en ion, as he maximum
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 2h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
numbe o messages ha could be ecei ed is 150 du ing he
1-yea s udy. This app oach minimizes any po en ial obo ic
and s a ic eelings.
Messages included heal h communica ion and heal h p omo ion
s a egies, such as c ea ing empa hy, adding new knowledge,
and changing exis ing misconcep ions. The messages we e sho
(less han 100 wo ds), w i en in simple Spanish, w i en using
a close and iendly one, and w i en wi h easy- o-unde s and
e ms o acili a e comp ehension by smoke s o all educa ional
le els. O he s udies ha e used s eps simila o he ones used
by us o message c ea ion [21]. Howe e , in ou design, he
message con en was no associa ed wi h any speci ic beha io al
change model. Each message was checked o be sui able o all
gende s. When a message was clea ly gende speci ic (ie,
ela ion be ween smoking and e ec ile dys unc ion in men), an
al e na e sui able message o he o he gende was also designed
(ie, isks o smoking du ing p egnancy).
HRSs in ol e sel -lea ning algo i hms ha can adap o he
cons an ly e ol ing needs and in e es s o use s and o e a high
le el o pe sonaliza ion o ecommenda ions, aking ad an age
o he so-called “collec i e in elligence.” This is a new
ecommenda ion gene a ion pa adigm ha con as s he
adi ional ailo ing app oach, which p o ides s a ic
ecommenda ions acco ding o use esponses o usually leng hy
ques ionnai es. Thus, as his no el app oach o ailo ing is s ill
in i s in ancy [22], i is o high in e es o de e mine he
ela ionship be ween he clinical ou comes o he So-Lo-Mo
s udy and his ype o echnology.
The main objec i e o his So-Lo-Mo s udy was o compa e
usual psychopha macological he apy (con ol g oup, CG) alone
and alongside he a o emen ioned digi al solu ion (in e en ion
g oup, IG) o smoking cessa ion. The seconda y objec i es
we e o analyze heal h- ela ed quali y o li e (HRQoL) and
moni o heal hy li es yle and physical exe cise habi s.
Complemen a ily, we assessed he impac o he AI-gene a ed
mo i a ional messages on smoking cessa ion ou comes in he
IG.
Me hods
S udy Design
This 12-mon h, andomized, open-label, pa allel-g oup ial
was pe o med a he Smoking Cessa ion Uni o Vi gen del
Rocío Uni e si y Hospi al in Se ille (Spain) be ween Oc obe
24, 2016, and Oc obe 24, 2018, and i complied wi h he
Decla a ion o Helsinki and Good Clinical P ac ice Guidelines.
The ec ui men pe iod closed on Oc obe 23, 2017. The local
e hics commi ee app o ed he s udy p o ocol, and w i en
in o med consen was ob ained om each pa icipan p io o
inclusion. The clinical s udy design (NCT03553173) and
echnical s udy design (NCT03206619) ha e been published
p e iously [23,24].
Randomiza ion
A echnician gene a ed a andom-g oup able (n=240) using
compu e me hods and ollowing a 1:1 a io be ween g oups.
Clinicians en olled he pa icipan s and assigned hem o he
g oup men ioned in he able acco ding o hei en olmen
sequence. Pa icipan s we e blinded o his alloca ion, as hose
in he IG we e old ha he p o ided mobile app was pa o
usual ca e. Pa icipan s in he CG we e no in o med abou he
exis ence o he app and did no ha e access o i .
S udy Popula ion
Smoke s we e ec ui ed du ing ou ine isi s o ou ou pa ien
clinic. The inclusion c i e ia we e as ollows: (1) age o e 18
yea s and desi e o s op smoking; (2) owning an And oid
sma phone (as he mobile app was only a ailable o And oid
de ices owing o ime and esou ce cons ain s in he
de elopmen phase o his s udy and And oid phones we e mo e
likely o be owned by he a ge popula ion owing o hei lowe
en y p ice as compa ed wi h iPhones); and (3) abili y o in e ac
wi h he sma phone. Sma phone li e acy was assessed by
asking he pa icipan s i hey commonly use o he ex exchange
sma phone apps, such as Mail, SMS, and Wha sApp. The only
exclusion c i e ion was any p e ious ad e se e ec ela ed o
he p esen pha macological ea men .
Powe Calcula ion and Rec ui men
A sample size o 236 was calcula ed du ing he s udy design
phase, acco ding o he ollowing pa ame e s: CI, 95%;
s a is ical powe , 80%; CG success a e, 35%; IG success a e,
55%; and expec ed d opou a e, 20%. A o al o 240 pa icipan s
we e ec ui ed and andomized o he s a i ied analysis.
In e en ion and Con ol G oups
Usual ca e in he Smoking Cessa ion Uni o Vi gen del Rocío
Uni e si y Hospi al consis s o pha macological he apy wi h
bup opion (ie, Zyn abac 150 mg; GlaxoSmi hKline, B en o d,
UK) o a enicline (ie, Champix 0.5 mg o 1 mg; P ize , New
Yo k, New Yo k, USA) plus beha io al he apy. To acili a e
ec ui men and a oid bias associa ed wi h ea men cos , he
SmokeF eeB ain p ojec inanced he d ugs o usual ca e. Thus,
all pa icipan s ecei ed hei assigned ea men s ee o cha ge.
Beha io al he apy, which was p o ided du ing ace- o- ace
ollow-up consul a ions, included a ious psychological
echniques, including mo i a ional in e iews and
cogni i e-beha io al he apy. This psychopha macological
he apy was p o ided o bo h CG and IG pa icipan s. CG
pa icipan s (n=120) ecei ed psychopha macological he apy
alone, whe eas IG pa icipan s (n=120) ecei ed
psychopha macological he apy and used he digi al he apeu ic
solu ion. Fu he in o ma ion ega ding en ollmen , alloca ion,
and he s udy analysis phases is p o ided in Figu e 1. The app
u ilized beha io al echniques by sending pe sonalized
mo i a ional messages gene a ed using AI wi h he in en o
achie e be e smoking cessa ion a es by imp o ing p og am
adhe ence and abs inence a es [20]. A e he pa icipan ead
each message, he app asked he pa icipan o a e how ele an
he message was o him o he , collec ing eedback o he AI.
Complemen ing he co e messaging ea u e, he mobile app had
o he mino sec ions. These included a use p o ile linked o
he pe o med physical ac i i y le el collec ed by Google Fi
(Google, Moun ain View, Cali o nia, USA), ou smoking
cessa ion bene i indica o s (sa ings, smoke- ee days, egained
li e hou s by no smoking, and numbe o nonsmoked ciga e es
since qui ing), ex -based in o ma ion abou smoking cessa ion,
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 3h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
and a sec ion con aining elaxing and dis ac ing elemen s
(b ea hing exe cises and minigames). Figu e 2 p esen s an o e iew o he app ea u es.
Figu e 1. CONSORT diag am o he s udy.
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 4h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
Figu e 2. S uc u e o app ea u es.
Measu emen s
In o ma ion om all pa icipan s included demog aphic (age
and sex) and socioeconomic da a (p o ession and employmen
s a us), consump ion his o y (daily ciga e es smoked, li ing
wi h smoke s, pa ne smoking s a us, smoking cessa ion
a emp s, e c), clinical in o ma ion (weigh , heigh , blood
p essu e, como bidi ies, e c), nico ine dependence measu ed
using he Fage s öm es o nico ine dependence [25], and
mo i a ion o s op smoking acco ding o he Richmond es
[26]. Sa e y was measu ed as he numbe o ad e se e en s
ela ed o pha macological he apy.
The main clinical ou come was he 1-yea smoking abs inence
a e measu ed by exhaled ca bon monoxide (CO), which was
assessed using a CO es e (Mic o+ Smoke lyze ; CoVi a, San a
Ba ba a, Cali o nia, USA) and u ine co inine es s. Pa icipan s
wi h an exhaled CO le el g ea e han 6 ppm we e conside ed
smoke s [27]. U ine co inine (SmokeSc een es ; Concep Smoke
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 5h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X

Sc een L d, Lincolnshi e, UK) is a colo ime ic es ha
measu es he main nico ine me aboli es, including co inine.
Pa icipan s wi h a co inine concen a ion g ea e han 200 ng/ml
we e conside ed smoke s [28]. Pa icipan s we e conside ed
smoke s when a leas one o he a o emen ioned condi ions
was me .
The HRQoL was assessed using he 36-i em Sho -Fo m Heal h
Su ey (SF-36) ha was alida ed in Spanish [29] and he
Eu oQol 5-dimension 5-le el (Eu oQoL-5D-5L) ques ionnai e
[30]. Physical ac i i y was measu ed using he In e na ional
Physical Ac i i y Ques ionnai e (IPAQ) [31], and a heal hy
li es yle was in e p e ed ia body mass index (BMI) a ia ions
du ing ollow-up consul a ions. A case epo o m, buil upon
he OpenClinica [32] ool, was de eloped o acili a e
in o ma ion managemen in he s udy. In o ma ion subse s we e
egis e ed acco ding o he ollowing schedule: basal and 15,
30, 60, 90, 120, 180, and 365 (±5) days a e he basal
consul a ion [23].
The HRS impac on smoking cessa ion was assessed acco ding
o he p ecision o he ecommenda ions sen by he sys em.
P ecision was calcula ed o bo h he message con en
ecommenda ions and he ime o open he messages, as he
HRS aimed o op imize bo h a iables. Fu he , we conside ed
he gene a ed engagemen o he use wi h he sys em and he
use app ecia ion o he messages as pa o he echnical
e alua ion o HRS impac . The s a ing poin o measu e he
e olu ion o hese me ics was he i s day ha a pa icipan
a ed a message (Decembe 5, 2016), and he assessmen
con inued o he ollowing 18 mon hs o e alua e he p og ess.
De ailed desc ip ions o hese me ic calcula ions a e p esen ed
below.
P ecision o he Sys em
This measu emen ocused on he sys em’s e ec i eness in
ecommending ele an messages. The mobile app equi ed he
use o a e all ead messages a leas once; o he wise, he use
could no longe in e ac wi h he app. The e we e h ee message
a ing op ions (like, dislike, and neu al). The a ings we e hen
coded as 1 o “like,” −1 o “dislike,” and 0 o “neu al.” The
p ecision me ic was calcula ed by di iding he numbe o hi s
(messages a ed as “like”) by he o al numbe o sen messages
(p ecision_p). A modi ica ion o his measu emen was
pe o med by conside ing “likes” and “neu als” as hi s
(p ecision_p_n,). We used wo a ian s o p ecision o ob ain
di e en pe spec i es on how he sys em pe o med in selec ing
ele an messages. The second app oach ocused on minimizing
he nega i ely a ed messages. Rega dless o he a ian , when
we code he hi s as 1 and misses as 0, his me ic is in ac an
a i hme ic mean (we sum he numbe o hi s and di ide hem
by he o al numbe o sen messages). The alue o his me ic
anges om 0 o 1. A highe alue is associa ed wi h mo e
sys em p ecision. This is because on sending messages a
andom, we expec 33% posi i e eedback, 33% neu al
eedback, and 33% nega i e eedback. Howe e , when he
sys em lea ns and becomes mo e “in elligen ,” he pe cen age
o nega i e eedback will dec ease o e ime, inc easing he
alue esul ing om his me ic.
Time o Open Mo i a ional Messages
This pe ained o he a i hme ic mean o he elapsed ime
be ween he message being sen and opened in a 30-day pe iod.
We assumed ha his me ic would dec ease o e ime because
he sys em was expec ed o lea n and become mo e “in elligen ”
in p edic ing he bes ime o he use o open and ead he sen
message.
Engagemen Wi h he Sys em
To measu e sys em engagemen , we included a ime s amp in
each sen message and compa ed i wi h he ime s amp sen by
he app o he se e when he use opened he gi en message.
Thus, he engagemen me ic was de e mined by he a io o
a ed messages calcula ed as he o al o all a ed messages om
one use di ided by he o al numbe o messages sen o ha
use . As we coded each message a ed wi h a “1” alue and
di ided he inding by he o al numbe o sen messages, his
a io coincides wi h he a i hme ic mean. Thus, we assume ha
a highe alue is associa ed wi h mo e use in e es in he
message and, consequen ly, highe use engagemen . As his
me ic measu ed he engagemen o he use wi h he sys em
and no he engagemen o he sys em wi h he use , we
conside ed he se da e o s op smoking o each use as he
poin o s a coun ing each mon h. A e his calcula ion, we
o se he s a da e o each use o he i s day he use a ed
he i s message. This allowed us o compa e he a e age
engagemen o each use ac oss ime. The e o e, he esul s a e
p esen ed acco ding he ela i e mon hs o he use se da e o
s op smoking (eg, M3 esul s a e he esul s o he hi d mon h
a e a pa icipan s opped smoking). Howe e , o one
pa icipan , M3 could be Ap il i he pa icipan s opped smoking
in Janua y, bu o ano he pa icipan , i could be Decembe i
he pa icipan s opped smoking in Sep embe . The measu emen
o his ollow-up was pe o med o he 12-mon h pe iod ha
each pa icipan in he IG was en olled in he s udy, as
pa icipan s in he CG did no use he mobile app.
Subjec i e Quali y o he Sys em
Sys em quali y was de e mined by he answe s in an anonymized
i e-le el Like - ype app ecia ion ques ionnai e [24], making
i impossible o link he answe s o he indi idual pa icipan s.
The ques ionnai e was comple ed by each pa icipan a he end
o he 1-yea ollow-up pe iod. The ques ions conce ned he
HRS acco ding o he Recommende Sys ems Ques ionnai e o
Use Expe ience (ResQue) model [33] and he con en o he
messages acco ding o he I-Change model [34]. O he 15
cons uc s o he ResQue model, 12 we e ep esen ed in he
ques ions, as he emaining h ee did no apply o ou HRS (ie,
pu chase in en ion). All he answe s e lec ed he le el o
ag eemen o he pa icipan wi h he p oposed opics (1
indica ing “ o ally disag ee” and 5 indica ing “ o ally ag ee”).
We assessed how pa icipan s pe cei ed he quali y o he sys em
a e 12 mon hs o usage by obse ing hei ques ionnai e
esponse dis ibu ions, means, and SDs. G oup compa isons
we e no easible owing o he comple e anonymi y o he
ques ionnai e. The aim was o measu e he deg ee o possible
imp o emen o he sys em in he ollowing a eas included in
he ques ionnai e: quali y o ecommended i ems, in e ac ion
adequacy, in e ace adequacy, pe cei ed ease o use, pe cei ed
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 6h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
use ulness, con ol/ anspa ency, a i udes, beha io al in en ions,
and message con en o in luence smoking isk pe cep ion,
con idence, social suppo , and coping ac ion suppo .
S a is ical Analysis
Desc ip i e analyses o pa icipan cha ac e is ics acco ding o
absolu e and ela i e equencies o quali a i e a iables and
mean (SD) o quan i a i e a iables we e conduc ed. Fo he
p ima y endpoin and seconda y ou comes (BMI, physical
ac i i y, and HRQoL), analyses we e conduc ed on a
pe -p o ocol basis. These analyses included all pa icipan s who
adhe ed o he schedule o eigh consul a ions (basal h ough
365 days a e he basal consul a ion).
Rega ding incomple e da a analyses, homoscedas ici y es s
we e conduc ed among g oups o cases wi h iden ical missing
da a pa e ns o e alua e whe he da a we e missing comple ely
a andom [35]. A mul inomial eg ession analysis was
pe o med o assess he a iables in luencing he p obabili y o
a pa icipan d opping ou as compa ed wi h he p obabili y o
he ea men being e ec i e. A “no e icacy” ca ego y was se
as he e e ence ca ego y. As mul inomial model e ec s a e
ela i e o he e e ence ca ego y, o assess whe he he
explica i e a iable e ec s we e di e en be ween he “d opou ”
and “e icacy” ca ego ies, he same model was adjus ed using
he “d opou ” ca ego y as a e e ence. Consequen ly, whe he
he a iable e ec s we e di e en be ween he “d opou ” and
“e icacy” ca ego ies was assessed. The ela i e isk a io (RRR)
wi h 95% CI was used o each model.
Los da a analysis and mul inomial logis ic models showed
di e en pa e ns in pa icipan s who d opped ou as compa ed
wi h he pa e ns in hose who comple ed he s udy, ega dless
o ea men e icacy. The e o e, smoking abs inence a 1 yea
was analyzed using logis ic eg ession models on a pe -p o ocol
basis and in en ion- o- ea basis, and he e ec measu es we e
he OR and 95% CI.
Fo he selec ion o logis ic eg ession and mul inomial logis ic
models in pe -p o ocol (n=94) analysis and in en ion- o- ea
(n=240) analysis, a wo-s age s a egy was adop ed. In he i s
s ep, he a iable impo ance was quan i ied using Random
Fo es [36] wi h he mean dec ease in accu acy as he sco e.
Va iables wi h a sco e abo e 0.5 we e included in an Akaike
in o ma ion c i e ion–based s epwise selec ion s a egy. The
ollowing wo es ic ions we e applied o he inal models
selec ed: (1) absence o a pa e n in model esiduals and (2)
a iables wi h a gene alized a iance in la ion ac o [37] abo e
5 we e no allowed in o de o a oid collinea i y.
To de e mine whe he he HRS me ics had an impac on he
clinical ou comes, we di ided IG pa icipan s in cessa ion and
no cessa ion subg oups a 12 mon hs o he in e en ion.
The ea e , we conduc ed es s o he 12-mon h esul s o
p ecision, ime o a e messages, and engagemen , analyzing
each o hese wo subg oups.
Resul s
Cha ac e is ics
Bo h g oups had simila baseline cha ac e is ics, excep o he
maximum abs inence ime and smoking cessa ion a emp s
(Table 1).
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 7h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
Table 1. Pa icipan baseline cha ac e is ics.
P alueCon ol g oup (n=120)In e en ion g oup (n=120)Cha ac e is ic
.0952 (43.3)65 (54.2)Female, n (%)
.0550.93 (10.85)48.38 (9.49)Age (yea s), mean (SD)
.5416.67 (3.60)16.94 (4.07)Age a smoking onse (yea s), mean (SD)
.4420.75 (9.39)21.45 (8.97)Daily ciga e es, mean (SD)
.6049 (40.8)45 (37.5)Li es wi h smoke s, n (%)
.4365 (54.2)71 (59.2)Pa ne smokes, n (%)
.0451.14 (1.07)0.88 (1.08)Smoking cessa ion a emp s, mean (SD)
.00313.68 (25.08)10.45 (25.49)Maximum abs inence ime, mean (SD)
.7327.03 (6.46)27.02 (4.91)Body mass index, mean (SD)
.4838 (31.7)33 (27.5)Unemployed, n (%)
.7944 (36.7)46 (38.3)P e ious ea men s, n (%)
.6915 (12.5)13 (10.8)Va enicline
.7016 (13.3)14 (11.7)Bup opion
.8415 (12.5)13 (10.8)Nico ine
.3111 (9.2)16 (13.3)O he s
.1259 (49.2)46 (38.3)Como bidi ies, n (%)
.061.08 (1.34)0.83 (1.25)Cha lson index, mean (SD)
.385.60 (1.99)5.89 (1.91)Fage s öm sco e, mean (SD)
.889.28 (0.88)9.32 (0.80)Richmond, mean (SD)
E icacy (Unadjus ed): 1-Yea Smoking Abs inence
Ra e
IG and CG pa icipan s who comple ed he s udy (pe -p o ocol
analysis) achie ed e icacy a es o 64.7% (CI 51.6%-77.8%)
and 40.0% (CI 25.7%-54.3%), espec i ely (P=.02; OR 2.75,
CI 1.20-6.29) and a numbe needed o ea o 4 (CI 2.0-19.0).
In he in en ion- o- ea analysis, IG and CG pa icipan s
achie ed e icacy a es o 27.5% (CI 19.5%-35.5%) and 15.0%
(CI 8.6%-21.4%), espec i ely (P=.02; OR 2.15, CI 1.13-4.08)
and a numbe needed o ea o 8 (CI 4.0-43.0). Figu e 3 shows
he e icacy e olu ion du ing he 1-yea ollow-up in each s udy
g oup acco ding o bo h analy ic app oaches.
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 8h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
Figu e 3. E icacy e olu ion du ing he 1-yea ollow-up.
Body Mass Index and Physical Ac i i y
BMI a iances we e simila o bo h g oups. In he IG and CG,
he mean BMI a iances we e 1.01 (CI 045-1.57) and 1.10 (CI
0.72-1.48), espec i ely, a 6 mon hs and we e 1.47 (CI
0.90-2.03) and 1.22 (CI 0.67-1.75), espec i ely, a 12 mon hs.
The be ween-g oup mean BMI di e ences a 6 and 12 mon hs
we e −0.09 (CI −0.77 o 0.60; P=.80) and 0.25 (CI −0.53 o
1.03; P=.52), espec i ely.
Acco ding o he obse ed IPAQ sco es, physical ac i i y
e olu ion pa e ns we e simila in bo h s udy g oups. Mo eo e ,
in he IG, 12.8% (6/47) and 25.5% (13/51) o pa icipan s
inc eased hei physical ac i i y a 6 and 12 mon hs, espec i ely,
whe eas 14.9% (7/47) and 13.7% (7/51) o pa icipan s educed
i a he co esponding poin s. On he o he hand, in he CG,
14.6% (6/41) and 24.4% (11/45) o pa icipan s inc eased hei
physical ac i i y a 6 and 12 mon hs, espec i ely, whe eas
24.4% (10/41) and 8.9% (4/45) o pa icipan s educed i a he
co esponding poin s. Table 2 summa izes he BMI and IPAQ
esul s.
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 9h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
Technical Ou comes
The complemen a y echnical esul s showed ha he HRS is
able o lea n he pa icipan s’ in e es s ega ding he suppo
message opics o smoking cessa ion. The sys em was mo e
p ecise a he end o he in e en ion han a he beginning, as
was expec ed o a ecommende sys em. Howe e , he
minimum p ecision eached was e y high (o e 96%). We
belie e ha he e may be di e en easons o his ex emely
high alue. Fo ins ance, i could be due o he implemen ed
hyb id HRS algo i hm ha mi iga es he cold s a p oblem and
accu a ely ecommends messages om he beginning acco ding
o a weigh ing o mula desc ibed in a p e ious publica ion [20].
Addi ionally, i could be due o pa icipan s belie ing ha hei
a ings would be iewed by heal h ca e p o essionals; hence,
hey modi ied hei a ings owa d a highe alue han hey
o he wise would. This po en ial Haw ho ne e ec would be a
limi a ion o digi al heal h no iden i ied in a ecen s udy [44],
as HRSs a e only s a ing o be used in heal h ca e. Ano he
eason may be ha he sys em did no allow su icien
g anula i y o he use s o o e; he pa icipan s we e only
a o ded h ee a ing op ions. A wide sp ead o a ing op ions
may ha e con ibu ed o he di e en ia ion o message ele ance
o a g ea e ex en ( o ins ance, sepa a ing ai , good, and
en i ely accu a e messages). The s a is ical analysis did no ind
a signi ican di e ence in he p ecision achie ed be ween
pa icipan s who we e success ul in smoking cessa ion and hose
who we e no success ul. Hence, he achie ed p ecision o e
ime canno be conside ed a p edic o o smoking cessa ion, as
he HRS ecommended mo i a ional messages equally well o
all pa icipan s.
The ime- o-open me ic esul s showed ha he HRS was no
able o p edic he bes ime o send a message o a dec ease
in he ime be ween he message being sen and he use opening
he message. This may be due o se e al ac o s in luencing
pa icipan beha io o he han he ime hey ecei e a message,
which we e no conside ed in he HRS. To imp o e he ime,
a iables, such as he posi ion o he phone, loca ion, las
ac i i y, and cessa ion day, should be conside ed o inclusion
as pa ame e s in he HRS.
The gene a ed engagemen by he sys em showed s a is ical
signi icance (P=.04) be ween he cessa ion and no cessa ion
subg oups a e he i s 6 mon hs o he in e en ion, a o ing
a highe engagemen o hose who managed o s op smoking.
This is in line wi h he in ui i e assump ion ha pa icipan s
who engaged wi h he sys em o a g ea e ex en ecei ed g ea e
bene i and consequen ly we e less likely o elapse. Howe e ,
hese di e ences we e no signi ican a 9, 11, and 12 mon hs
owing o a dec ease in he numbe o pa icipan s o such an
ex en a hese poin s ha he s a is ical powe was limi ed.
As no all pa icipan s in he IG comple ed he pe cei ed quali y
ques ionnai e and he ques ionnai e was anonymous, i is
impossible o ensu e ha any conclusion de i ed om he
collec ed esul s is no biased. Howe e , he esul s showed a
clea ly posi i e pe cep ion o he quali y o he sys em.
Reasonably, only hose who managed o s op smoking and
p esumably bene i ed om he sys em p o ided answe s o he
ques ionnai e, as hey would ha e been mo e mo i a ed o
espond. On he o he hand, pa icipan s who d opped ou ,
p obably due o elapse, would ha e no been in e es ed in
comple ing he ques ionnai e. Howe e , i could also be a gued
ha hose who did no manage o s op smoking would ha e
been willing o p o ide nega i e eedback on he sys em in he
anonymized ques ionnai e. Ne e heless, he posi i e end o
he answe s a o s he i s scena io. Fo simila esea ch
p ac ices in he u u e, we sugges ollowing a design ha o ces
pa icipan s o espond o he ques ionnai e bu s ill p ese es
hei anonymi y. This could be achie ed by including he
ques ionnai e in a p e ious s age o he ial o a oid an e ec
by he high d opou a e associa ed wi h digi al in e en ions
o smoking cessa ion.
The ea u es o he p esen ed AI-based digi al he apeu ic
solu ion enabled he sys em o e ec i ely suppo smoking
cessa ion by p o iding suppo and ad ice o acili a ing
abs inence, enhance mo i a ion, and clea ly show a bene i [45].
The HRS was well pe cei ed by he pa icipan s. O e ime, i
iden i ied he mos ele an mo i a ional messages o send o
each pa icipan , and hose who we e engaged wi h he sys em
o a g ea ex en managed o success ully s op smoking by he
end o he in e en ion. The e o e, his digi al he apeu ic
solu ion may alle ia e he known d awbacks o in ensi e
complemen a y beha io al in e en ions o smoking cessa ion,
which, despi e hei bene i s, equi e ex ensi e a ailabili y o
well- ained p o essionals, leading o limi ed scalabili y, poo
accessibili y, and smoke esigna ion owing o long wai ing
imes [46].
Limi a ions
In p e ious simila s udies, he d opou a e was usually e y
high, e lec ing he di icul y o smoking cessa ion and high
elapse a e [3]. In his s udy, he d opou a es we e 57.5% and
62.5% in he IG and CG, espec i ely. A high d opou a e may
bias he esul s; howe e , he d opou a e was simila be ween
he g oups, minimizing he possibili y o bias. The e icacy o
he in e en ion was e i ied by pe -p o ocol and
in en ion- o- ea analyses, p o iding a mo e ealis ic pic u e
while educing he impo ance o d opou o ull ea men
compliance. Mo eo e , an analysis o he cha ac e is ics o he
d opou and no d opou ca ego ies wi hin each g oup was ca y
ou , and he same pa e n was ob ained be ween bo h g oups
(Mul imedia Appendix 3). These indings sugges he po en ial
bene i o he in e en ion, as pa icipan s did no expe ience
abno mal ad e se e ec s and we e mo e likely o be abs inen
a e 1 yea . Ne e heless, u he explo a ion o i s e icacy is
needed o alida ion in o he cul u es, and s udies should
include a la ge sample size and eal-wo ld da a.
We ound some andom inconsis en alues in he en ies o he
egis e ed ime, including duplica e en ies and nega i e ime
alues. The p esen ed esul s co espond o il e ed da a
( emo ing such alues). Remo al o hese alues educed da a
quali y o he me ic, and he indings may no accu a ely
ep esen he en i e pa icipan g oup. These inconsis en alues
may ha e been egis e ed owing o gli ches in communica ion
be ween he sma phone app and he acking sys em on he
se e .
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 16h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X

Me ics o engagemen a he agg ega ed le el could no be
p o ided as an icipa ed in he echnical p o ocol because he
so wa e se ice used o ack use da a is no able o e ie e
in o ma ion om da a olde han 2 yea s. This in o ma ion was
no lis ed in he ea u es o he se ice when i was selec ed o
he pu poses o he s udy. We encou age u u e esea che s o
use in-house use da a acking se ices o a oid elying on
hi d-pa y so wa e.
Compa ison Wi h P io Wo k
P e ious s udies ha e explo ed he use o mobile apps
suppo ing smoking cessa ion. The Sma Qui s udy es ed he
easibili y, accep abili y, p elimina y e icacy, and mechanism
o beha io al change o an inno a i e sma phone-deli e ed
accep ance and commi men he apy app o smoking cessa ion
in 196 adul pa icipan s and epo ed a success ul cessa ion
a e o 13% a e 2 mon hs o ollow-up [47].
In ano he s udy, he au ho s assessed he e icacy o an
in e ac i e smoking cessa ion decision–aid app in 684 adul s
and epo ed a success ul cessa ion a e o 10.2% [48].
Addi ionally, an app-based mind ulness aining p og am o
smoking cessa ion was assessed in 143 pa icipan s, and a
success ul cessa ion a e o 11.1% was achie ed [49]. A pa allel,
double-blind, andomized, con olled, wo-a m ial compa ed
he e icacy o an e idence-in o med sma phone app o
smoking cessa ion (C ush he C a e) o ha o an
e idence-in o med sel -help guide (On he Road o Qui ing)
in 1599 subjec s and epo ed success ul cessa ion a es o 7.8%
and 9.2%, espec i ely [50]. In all hese s udies, he abs inence
a e was epo ed a 6 mon hs om baseline. Howe e , hey did
no include pha macological ea men as pa o he
in e en ion, which is a clea di e ence om he app oach in
ou s udy. I is ema kable ha he d opou a es o bo h he
IG and CG iden i ied in his s udy a e consis en wi h hose
p e iously epo ed in he li e a u e [50].
Resea ch has been pe o med on in e en ion e icacy o he
use o an app speci ically designed o suppo he smoking
cessa ion p ocess when added o pha macological he apy. The
p elimina y e icacy o an app designed o p omp smoke s o
engage in physical ac i i y was assessed a 6 mon hs o
ollow-up in 44 egula smoke s who ecei ed he app in
addi ion o beha io al smoking cessa ion counselling, physical
ac i i y p omo ion, and pha macological suppo , and he o e all
abs inence a es we e 36% in an in en ion- o- ea analysis and
53% in a comple e-case analysis [51]. In his p e ious s udy,
he alloca ion o pha macological ea men ( a enicline and
nico ine eplacemen he apy) was no con olled by he s udy
p o ocol, which hinde s he compa abili y o he esul s wi h
hose o ou s udy. Addi ionally, in a andomized pilo s udy,
a 12-week cou se o a enicline was p esc ibed o bo h a ms o
assess he easibili y and accep abili y o he My Mobile Ad ice
P og am (MyMAP) smoking cessa ion app and es ima e i s
e ec s on smoking cessa ion and medica ion adhe ence [52];
howe e , i s e icacy could no be e idenced owing o he small
sample size (n=33).
As s a ed in he conclusion sec ion o a ecen sys ema ic e iew
[53] ha included 26 s udies (n=33,849) in ol ing ex
messaging and app-based smoking cessa ion in e en ions, he e
is mode a e ce ain y e idence ha au oma ed ex
message–based smoking cessa ion in e en ions esul in g ea e
smoking cessa ion a es as compa ed wi h minimal smoking
cessa ion suppo and he e is mode a e ce ain y e idence o
he bene i o ex messaging in e en ions in addi ion o o he
ypes o smoking cessa ion suppo as compa ed wi h smoking
cessa ion suppo alone. The e idence o he compa ison o
sma phone apps wi h less in ensi e suppo had e y low
ce ain y, and u he andomized con olled ials a e needed
o es hese in e en ions. In his sense, he esul s o his s udy
aim o con ibu e o an inc ease in he ce ain y le el o he
a ailable e idence in his domain.
When ocusing on he assessmen o ecommende sys ems,
such as he one used in his s udy, he scien i ic communi y has
ex ensi ely concen a ed on imp o ing he pe o mance o
ecommende sys em algo i hms wi h di e en me ics [54-58],
mainly in nonheal h con ex s, such as leisu e [59] and
e-comme ce [60,61]. Among hese app oaches, common
assessmen s o p edic ion accu acy a e as ollows: (1) mean
absolu e e o and oo mean squa ed e o , as well as hei
no malized and a e aged a ian s o p edic a ings ha use s
would gi e o i ems when he ac ual ecommenda ion a ings
o he i ems a e known o he whole es se ; (2) p ecision,
ecall, and alse posi i e a e [58,62] o p edic ion o he usage
o he ecommenda ions; and (3) no malized discoun ed
cumula i e gain when he sys em p esen s a la ge lis o
elemen s (simila o a sea ch in Google), whe e we expec ha
he mos ele an elemen s a e shown a he op o he lis ; and
(4) co e age o he ecommenda ion se [63]. O he au ho s
ha e p oposed he use o su eys o assess ecommende
sys ems, such as ResQue [33].
Rega ding he e alua ion o mobile Heal h (mHeal h) apps o
beha io al change, such as he one included in his s udy, some
au ho s ha e p oposed e alua ion p ocedu es, such as he
Na ional Ins i u e o Heal h and Ca e Excellence adap a ion
[64], he Mobile App Ra ing Scale amewo k [65], and he
Applica ion Usage Fac o , which is de ined as he loga i hm o
he p oduc o he numbe o ac i e use s o a mobile app and
he median numbe o daily uses o he app [66]. Despi e hese
a emp s o se a me hodological amewo k o assess mHeal h
apps, hey s ill lack s anda diza ion and comp ehensi eness
[67]. In a ecen s udy, McKay e al iden i ied ha he e was no
a ailable ela ed bes p ac ice [68]. Ins ead, hey p oposed he
ollowing gene ic guidelines o wha hese ypes o e alua ions
should include: (1) assessmen o he quali y o heal h- ela ed
con en ; (2) e iew o he usabili y and unc ionali y o he app;
and (3) c i ique o he app po en ial wi h ega d o beha io al
change p omo ion.
Despi e he lack o consensus on speci ic e alua ion me hods,
i is well accep ed ha engagemen is a key elemen o mHeal h
solu ions o be success ul [69]. Engagemen has been conside ed
a posi i e ea ly indica o o beha io al change [70], as he
p ocess o beha io al change equi es ime. P e ious s udies
showed ha echnologies suppo ing eaching a speci ic beha io
heal h goal help people s ick o he desi ed goal [71,72]. Fu he ,
Sche e e al [73] ound a signi ican posi i e co ela ion
be ween engagemen and ewe d opou s in an mHeal h
in e en ion. A ecen app oach o p o ide an engagemen sco e
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 17h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
o a mHeal h app is he Engagemen Index p oposed by Taki
e al [74]. To calcula e he sco e, he au ho s combined he
numbe o pages in he app ha a pa icipan isi ed each day,
numbe o app accesses du ing he p og am, numbe o push
no i ica ions opened, elapsed ime be ween app accesses, and
subjec i e answe s o a ques ionnai e. Howe e , achie ing use
engagemen is di icul [75], and i s le el could be imp o ed
[76-78]. The e o e, i seems easonable o conside engagemen
as a key me ic o digi al heal h solu ions. Howe e , he
de ini ion o engagemen a ies among s udies. Fo ins ance,
Iaco iello e al [79] conside ed he numbe o imes use s
opened an app, numbe o in e ac ions hey had, and numbe o
weeks in which use s had a leas one in e ac ion wi h he
p og am as indica o s o engagemen . Owen e al [80]
de e mined engagemen by calcula ing se e al a iables, such
as he numbe o downloads and numbe o sessions (a session
being basically opening o he app). Ye , i is common o jus
use opening he app as an indica o o engagemen , as adop ed
in p e ious s udies [40,81]. All hese engagemen in e p e a ions
all unde he ca ego iza ion o “sys em usage da a” p oposed
by Sho e al [82], which is mos equen ly used. Howe e ,
o he engagemen measu emen app oaches ha e been p oposed,
such as ecological momen a y assessmen s, psychophysiological
measu emen s, and quali i e me hods.
In ou So-Lo-Mo s udy, he HRS was in ui i ely ela ed o use
engagemen , as some s udies ha e shown ha good and imely
ecommenda ions o s op smoking mo i a e use s o ead mo e
u u e ecommenda ions [74,83]. Hence, we p oposed a
combina ion o quali a i e and quan i a i e me ics, which
measu e he in ended pu pose o he sys em (quali y o he
ecommenda ions, engagemen o he pa icipan s wi h he
sys em, and pa icipan s’ pe cep ion o he sys em). To ou
knowledge, his is he i s assessmen in ol ing a HRS.
Howe e , in he case o smoking cessa ion, low engagemen
may no necessa ily esul in low impac on beha io al change,
as some s udies ha e p e iously shown he “ga eway e ec ”
[84] and “happy abandonmen ” [85]. The e a e se e al easons
o his, including he bu den o he equi ed in e ac ions as
compa ed wi h pe cei ed ou comes o he in e naliza ion o
heal h habi s deno ing ha suppo is pe cei ed as no longe
necessa y [86]. Consequen ly, we needed o ca e ully analyze
he engagemen o use s ac oss ime poin s keeping in mind all
hese ac o s o ex ac any conclusions. Thus, ou sys em’s
in ended usage [82] is expec ed o be highe du ing he i s
weeks o he smoking cessa ion p ocess and o p og essi ely
dec ease un il he e is no engagemen wi h he sys em, which
will be e lec ed in engagemen measu emen s.
Conclusions
The So-Lo-Mo in e en ion o e s a p omising s a egy o
smoking cessa ion. The use o his digi al he apeu ic solu ion
alongside pha macological ea men was much mo e e icacious
in achie ing obacco abs inence as compa ed wi h
pha macological ea men alone a he 1-yea ollow-up.
Howe e , his in e en ion did no imp o e pa icipan HRQoL
and physical ac i i y le els.
Analysis o he impac o he HRS showed ha pa icipan s
bene i ed om he ecommenda ions, and hose who engaged
wi h he sys em we e mo e likely o succeed in smoking
cessa ion. The e o e, he p oposed HRS had a posi i e impac
on he pa icipan s and o e ed hem pe sonalized and ele an
messages, al hough i could no de e mine when suppo i e
messages should be sen o minimize he ime un il hey a e
ead.
Heal h ca e p o ide s should conside inco po a ing his digi al
he apeu ic solu ion in hei usual ca e, as i can acili a e
posi i e ou comes o pa icipan s willing o s op smoking.
Acknowledgmen s
The digi al he apeu ic solu ion used in he So-Lo-Mo s udy was join ly de eloped by Salumedia Tecnologías SLU, he A is o le
Uni e si y o Thessaloniki, Se icio Andaluz de Salud, and Uni e sidad de Se illa based on he SmokeF eeB ain p ojec . We
acknowledge P o Hein de V ies and D F ancine Schneide who p o ided guidance in he da a analysis o he echnical ou comes.
This esea ch was unded by he H2020 Eu opean Commission esea ch and inno a ion p og am (g an ag eemen 681120) as
pa o he SmokeF eeB ain p ojec (www.smoke eeb ain.eu).
Con lic s o In e es
SHF is a p oduc manage a Salumedia Tecnologías SLU, he company wi h exploi a ion igh s o he digi al he apeu ic solu ion
used in he So-Lo-Mo s udy (DigiQui ). He con ibu ed o echnical da a ex ac ion o he p ecision, ime- o-open he messages,
and pe cei ed quali y me ics and hei analysis. He was no in ol ed in he clinical ial design, execu ion, o analysis o he
esul ing clinical da a. Some ins i u ions o all he o he au ho s (Uni e si y o Se ille, he A is o le Uni e si y o Thessaloniki,
and he Se icio Andaluz de Salud as a legal ep esen a i e ins i u ion o he Vi gen del Rocío Uni e si y Hospi al) signed an
exploi a ion ag eemen wi h Salumedia Tecnologías SLU o bene i om DigiQui comme cializa ion. This ag eemen was
elabo a ed and signed be o e publishing he p esen esul s bu a e he ial was inished and i s esul s we e gene a ed.
Mul imedia Appendix 1
Incomple e da a analysis.
[DOC File , 81 KB-Mul imedia Appendix 1]
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 18h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
Mul imedia Appendix 2
Heal h ecommende sys em pe o mance analysis, P ecision_p.
[DOCX File , 42 KB-Mul imedia Appendix 2]
Mul imedia Appendix 3
Cha ac e is ics o he d opou and non-d opou wi hin each g oup.
[DOC File , 126 KB-Mul imedia Appendix 3]
Mul imedia Appendix 4
CONSORT-EHEALTH checklis (V 1.6.1).
[PDF File (Adobe PDF File), 3306 KB-Mul imedia Appendix 4]
Re e ences
1. WHO. WHO Global Repo On T ends In P e alence O Tobacco Smoking, 2015. Gene a: Wo ld Heal h O ganiza ion;
2015.
2. Goodchild M, Na gis N, Tu san d'Espaigne E. Global economic cos o smoking-a ibu able diseases. Tob Con ol 2017
Jan 30;27(1):58-64. [doi: 10.1136/ obaccocon ol-2016-053305]
3. Cahill K, S e ens S, Pe e a R, Lancas e T. Pha macological in e en ions o smoking cessa ion: an o e iew and ne wo k
me a-analysis. Coch ane Da abase Sys Re 2013 May 31(5):CD009329. [doi: 10.1002/14651858.CD009329.pub2] [Medline:
23728690]
4. Reid RD, P i cha d G, Walke K, Ai ken D, Mullen K, Pipe AL. Managing smoking cessa ion. CMAJ 2016 Dec
06;188(17-18):E484-E492 [FREE Full ex ] [doi: 10.1503/cmaj.151510] [Medline: 27698200]
5. Cahill K, Lindson-Hawley N, Thomas K, Fanshawe T, Lancas e T. Nico ine ecep o pa ial agonis s o smoking cessa ion.
Coch ane Da abase Sys Re 2016 May 09(5):CD006103 [FREE Full ex ] [doi: 10.1002/14651858.CD006103.pub7]
[Medline: 27158893]
6. Condon LA, Coulson NS. Chap e 2 - Designing and Deli e ing In e en ions o Heal h Beha io Change in Adolescen s
Using Mul i echnology Sys ems: F om Iden i ica ion o Ta ge Beha io s o Implemen a ion. In: Beha io Change Resea ch
and Theo y. Cambi dge, Massachuse s: Else ie ; 2017:27-45.
7. Elbe SP, Dijks a A, Oenema A. A Mobile Phone App In e en ion Ta ge ing F ui and Vege able Consump ion: The
E icacy o Tex ual and Audi o y Tailo ed Heal h In o ma ion Tes ed in a Randomized Con olled T ial. J Med In e ne
Res 2016 Jun 10;18(6):e147 [FREE Full ex ] [doi: 10.2196/jmi .5056] [Medline: 27287823]
8. Zhao J, F eeman B, Li M. Can Mobile Phone Apps In luence People's Heal h Beha io Change? An E idence Re iew. J
Med In e ne Res 2016 Oc 31;18(11):e287 [FREE Full ex ] [doi: 10.2196/jmi .5692] [Medline: 27806926]
9. Teasdale N, Elhussein A, Bu che F, Pie nas C, Cowbu n G, Ha mann-Boyce J, e al. Sys ema ic e iew and me a-analysis
o emo ely deli e ed in e en ions using sel -moni o ing o ailo ed eedback o change die a y beha io . Am J Clin Nu
2018 Feb 01;107(2):247-256 [FREE Full ex ] [doi: 10.1093/ajcn/nqx048] [Medline: 29529158]
10. Whi ake R, McRobbie H, Bullen C, Rodge s A, Gu Y. Mobile phone-based in e en ions o smoking cessa ion. Coch ane
Da abase Sys Re 2016 Ap 10;4:CD006611 [FREE Full ex ] [doi: 10.1002/14651858.CD006611.pub4] [Medline:
27060875]
11. B ug J, Glanz K, Van Assema P, Kok G, an B eukelen GJ. The impac o compu e - ailo ed eedback and i e a i e eedback
on a , ui , and ege able in ake. Heal h Educ Beha 1998 Aug 30;25(4):517-531. [doi: 10.1177/109019819802500409]
[Medline: 9690108]
12. Dijks a A, De V ies H, Roijacke s J, an B eukelen G. Tailo ing in o ma ion o enhance qui ing in smoke s wi h low
mo i a ion o qui : Th ee basic e icacy ques ions. Heal h Psychology 1998;17(6):513-519. [doi: 10.1037/0278-6133.17.6.513]
13. Dijks a A. The Psychology o Tailo ing-Ing edien s in Compu e -Tailo ed Pe suasion. Social Pe s Psych Compass 2008
Ma ;2(2):765-784. [doi: 10.1111/j.1751-9004.2008.00081.x]
14. Enwald HP, Huo a i MA. P e en ing he obesi y epidemic by second gene a ion ailo ed heal h communica ion: an
in e disciplina y e iew. J Med In e ne Res 2010 Jun 28;12(2):e24 [FREE Full ex ] [doi: 10.2196/jmi .1409] [Medline:
20584698]
15. Kamel Ghaliba A, Naza i E, Gholian-A al M, Ta a M. Comp ehensi e o e iew o compu e -based heal h in o ma ion
ailo ing: a sys ema ic scoping e iew. BMJ Open 2019 Jan 09;9(1):e021022 [FREE Full ex ] [doi:
10.1136/bmjopen-2017-021022] [Medline: 30782671]
16. K eu e M, Fa ell D, Ole i ch L, B ennan L. Cus omizing communica ion wi h compu e echnology. In: Rou ledge ,
edi o . Tailo ed heal h messages. Mahwah, New Je sey: Law ence E lbaum Associa es, Inc; 2000.
17. Cupe ino AP, Ca ujano-Ba e a F, Ramí ez M, Rod íguez-Bolaños R, Th ashe JF, Pé ez-Rubio G, e al. A Mobile
Smoking Cessa ion In e en ion o Mexico (Vi e sin Tabaco... ¡Decíde e!): Single-A m Pilo S udy. JMIR Mheal h Uheal h
2019 Ap 25;7(4):e12482 [FREE Full ex ] [doi: 10.2196/12482] [Medline: 31021326]
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 19h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
18. Bamidis P, Pa aske opoulos E, Kons an inidis E, Spachos D, Billis A. Mul imodal e-heal h se ices o smoking cessa ion
and public heal h: he smoke eeb ain p ojec app oach. In: P oceedings o he 16 h Wo ld Cong ess on Medical and Heal h
In o ma ics. 2017 P esen ed a : MEDINFO 2017: P ecision Heal hca e h ough In o ma ics; Augus 2017; Hangzhou p.
5-9. [doi: 10.3233/978-1-61499-830-3-5]
19. P ochaska JO, Velice WF. The ans heo e ical model o heal h beha io change. Am J Heal h P omo 1997 Aug
26;12(1):38-48. [doi: 10.4278/0890-1171-12.1.38] [Medline: 10170434]
20. Ho s-F aile S, Nuñes Benjumea FJ, Ca asco He nández L, Ruiz FO, Fe nandez-Luque L. Design o wo combined heal h
ecommende sys ems o ailo ing messages in a smoking cessa ion app. In: a Xi p ep in . 2016 P esen ed a : In e na ional
Wo kshop on Engende ing Heal h wi h RecSys co-loca ed wi h ACM RecSys; 2016; Bos on, Massachuse s p. 160807192
URL: h ps://a xi .o g/abs/1608.07192
21. Ho s-F aile S, de V ies H, Malwade S, Luna-Pe ejon F, Amaya C, Ci i A, e al. Opening he Black Box: Explaining he
P ocess o Basing a Heal h Recommende Sys em on he I-Change Beha io al Change Model. IEEE Access
2019;7:176525-176540. [doi: 10.1109/access.2019.2957696]
22. Ho s-F aile S, Ri e a-Rome o O, Schneide F, Fe nandez-Luque L, Luna-Pe ejon F, Ci i -Balcells A, e al. Analyzing
ecommende sys ems o heal h p omo ion using a mul idisciplina y axonomy: A scoping e iew. In J Med In o m 2018
Jun;114:143-155. [doi: 10.1016/j.ijmedin .2017.12.018] [Medline: 29331276]
23. Jóda -Sánchez F, Ca asco He nández L, Núñez-Benjumea FJ, Mesa González MA, Mo eno Conde J, Pa a Calde ón CL,
e al. Using he Social-Local-Mobile App o Smoking Cessa ion in he SmokeF eeB ain P ojec : P o ocol o a Randomized
Con olled T ial. JMIR Res P o oc 2018 Dec 06;7(12):e12464 [FREE Full ex ] [doi: 10.2196/12464] [Medline: 30522992]
24. Ho s-F aile S, Schneide F, Fe nandez-Luque L, Luna-Pe ejon F, Ci i A, Spachos D, e al. Tailo ing mo i a ional heal h
messages o smoking cessa ion using an mHeal h ecommende sys em in eg a ed wi h an elec onic heal h eco d: a s udy
p o ocol. BMC Public Heal h 2018 Jun 05;18(1):698 [FREE Full ex ] [doi: 10.1186/s12889-018-5612-5] [Medline:
29871595]
25. Hea he on TF, Kozlowski LT, F ecke RC, Fage s öm KO. The Fage s öm Tes o Nico ine Dependence: a e ision o
he Fage s öm Tole ance Ques ionnai e. B J Addic 1991 Sep;86(9):1119-1127. [doi: 10.1111/j.1360-0443.1991. b01879.x]
[Medline: 1932883]
26. Richmond RL, Kehoe LA, Webs e IW. Mul i a ia e models o p edic ing abs en ion ollowing in e en ion o s op smoking
by gene al p ac i ione s. Addic ion 1993 Aug;88(8):1127-1135. [doi: 10.1111/j.1360-0443.1993. b02132.x] [Medline:
8401167]
27. Pea ce MS, Hayes L, Newcas le Hea P ojec , Newcas le Thousand Families S udy. Sel - epo ed smoking s a us and
exhaled ca bon monoxide: esul s om wo popula ion-based epidemiologic s udies in he No h o England. Ches 2005
Sep;128(3):1233-1238. [doi: 10.1378/ches .128.3.1233] [Medline: 16162711]
28. Pé ez-T ullén A, Ba olomé CB, Banegas JR. [New pe spec i es in he diagnosis and e olu ion o he consump ion o
obacco: ma ke s o suscep ibili y and damage]. Med Clin (Ba c) 2006 Ap 29;126(16):628-631. [doi: 10.1157/13087722]
[Medline: 16759556]
29. Vilagu G, Fe e M, Rajmil L, Rebollo P, Pe manye -Mi alda G, Quin ana JM, e al. [The Spanish e sion o he Sho
Fo m 36 Heal h Su ey: a decade o expe ience and new de elopmen s]. Gac Sani 2005 Ap ;19(2):135-150 [FREE Full
ex ] [doi: 10.1157/13074369] [Medline: 15860162]
30. Ramos-Goñi JM, C aig BM, Oppe M, Ramallo-Fa iña Y, Pin o-P ades JL, Luo N, e al. Handling Da a Quali y Issues o
Es ima e he Spanish EQ-5D-5L Value Se Using a Hyb id In e al Reg ession App oach. Value Heal h 2018
May;21(5):596-604 [FREE Full ex ] [doi: 10.1016/j.j al.2017.10.023] [Medline: 29753358]
31. Boo h M. Assessmen o physical ac i i y: an in e na ional pe spec i e. Res Q Exe c Spo 2000 Jun 13;71 Suppl
2(sup2):114-120. [doi: 10.1080/02701367.2000.11082794] [Medline: 25680021]
32. Ca elaa s M, Rousseau J, Pa layan C, de Ridde S, Ve bu g A, Ross R, e al. OpenClinica. J Clin Bioin o ma ics 2015;5(Suppl
1):S2. [doi: 10.1186/2043-9113-5-s1-s2]
33. Pu P, Chen L, Hu R, edi o s. A use -cen ic e alua ion amewo k o ecommende sys ems. : ACM Digi al Lib a y; 2011
Oc P esen ed a : RecSys '11: Fi h ACM Con e ence on Recommende Sys ems; Oc obe 2011; Chicago, Illinois p. 157-164
URL: h ps://dl.acm.o g/doi/10.1145/2043932.2043962 [doi: 10.1145/2043932.2043962]
34. de V ies H. An In eg a ed App oach o Unde s anding Heal h Beha io ; The I-Change Model as an Example. PBSIJ 2017
Ma 9;2(2). [doi: 10.19080/pbsij.2017.02.555585]
35. Jamshidian M, Jalal S. Tes s o homoscedas ici y, no mali y, and missing comple ely a andom o incomple e mul i a ia e
da a. Psychome ika 2010 Dec 3;75(4):649-674 [FREE Full ex ] [doi: 10.1007/s11336-010-9175-3] [Medline: 21720450]
36. B eiman L. Random o es s. Machine lea ning 2001;45(1):5-32 [FREE Full ex ] [doi: 10.1023/A:1010933404324]
37. Fox J, Mone e G. Gene alized Collinea i y Diagnos ics. Jou nal o he Ame ican S a is ical Associa ion 1992
Ma ;87(417):178-183. [doi: 10.1080/01621459.1992.10475190]
38. Ga ison KA, Pal P, Rojiani R, Dalle y J, O'Malley SS, B ewe JA. A andomized con olled ial o sma phone-based
mind ulness aining o smoking cessa ion: a s udy p o ocol. BMC Psychia y 2015 Ap 14;15(1):83 [FREE Full ex ]
[doi: 10.1186/s12888-015-0468-z] [Medline: 25884648]
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 20h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
39. He ne JL, Vila daga R, Me ce LD, Kien z JA, B icke JB. Fea u e-le el analysis o a no el sma phone applica ion o
smoking cessa ion. Am J D ug Alcohol Abuse 2015 Jan 14;41(1):68-73 [FREE Full ex ] [doi:
10.3109/00952990.2014.977486] [Medline: 25397860]
40. Ubhi HK, Michie S, Ko z D, Wong WC, Wes R. A mobile app o aid smoking cessa ion: p elimina y e alua ion o
SmokeF ee28. J Med In e ne Res 2015 Jan 16;17(1):e17 [FREE Full ex ] [doi: 10.2196/jmi .3479] [Medline: 25596170]
41. Ge me o h LJ, Le ine MD. Pos cessa ion weigh gain conce n as a ba ie o smoking cessa ion: Assessmen conside a ions
and u u e di ec ions. Addic Beha 2018 Jan;76:250-257 [FREE Full ex ] [doi: 10.1016/j.addbeh.2017.08.022] [Medline:
28865363]
42. Goldenbe g M, Dano i ch I, IsHak WW. Quali y o li e and smoking. Am J Addic 2014 Sep 25;23(6):540-562. [doi:
10.1111/j.1521-0391.2014.12148.x] [Medline: 25255868]
43. Fu S, Dao M, Wong CK, Yu W. SF-6D u ili y sco es o smoke s and ex-smoke s wi h o wi hou espi a o y symp oms
a ending p ima y ca e clinics. Heal h Qual Li e Ou comes 2019 Ma 15;17(1):48 [FREE Full ex ] [doi:
10.1186/s12955-019-1115-z] [Medline: 30876466]
44. Alma io CV. The E ec o Digi al Heal h Technology on Pa ien Ca e and Resea ch. Gas oen e ol Hepa ol (N Y) 2017
Jul;13(7):437-439 [FREE Full ex ] [Medline: 28867974]
45. Ka hleen FH, Young-Il K, Mei ang C, Rekha R, Ma ia P, Rajani SS, e al. Web-Based In e en ion o T ansi ioning
Smoke s F om Inpa ien o Ou pa ien Ca e: An RCT. Am J P e Med 2016 Oc ;51(4):620-629. [doi:
10.1016/j.amep e.2016.04.008] [Medline: 27647062]
46. Taylo G, Dalili M, Semwal M, Ci ljak M, Sheikh A, Ca J. In e ne -based in e en ions o smoking cessa ion. Coch ane
Da abase Sys Re 2017 Sep 04;9:CD007078 [FREE Full ex ] [doi: 10.1002/14651858.CD007078.pub5] [Medline:
28869775]
47. Bal unas L, Ludwig B, Pee S, Ricci F. Con ex ele ance assessmen and exploi a ion in mobile ecommende sys ems.
Pe s Ubiqui Compu 2011 Jun 21;16(5):507-526. [doi: 10.1007/s00779-011-0417-x]
48. BinDhim NF, McGeechan K, T e ena L. Sma phone Smoking Cessa ion Applica ion (SSC App) ial: a mul icoun y
double-blind au oma ed andomised con olled ial o a smoking cessa ion decision-aid 'app'. BMJ Open 2018 Jan
21;8(1):e017105 [FREE Full ex ] [doi: 10.1136/bmjopen-2017-017105] [Medline: 29358418]
49. Ga ison K, Pal P, O'Malley SS, Pi man B, Gueo guie a R, Rojiani R, e al. C a ing o Qui : A Randomized Con olled
T ial o Sma phone App-Based Mind ulness T aining o Smoking Cessa ion. Nico ine Tob Res 2020 Ma 16;22(3):324-331.
[doi: 10.1093/n /n y126] [Medline: 29917096]
50. Baske ille NB, S uik LL, Guindon GE, No man CD, Whi ake R, Bu ns C, e al. E ec o a Mobile Phone In e en ion
on Qui ing Smoking in a Young Adul Popula ion o Smoke s: Randomized Con olled T ial. JMIR Mheal h Uheal h 2018
Oc 23;6(10):e10893 [FREE Full ex ] [doi: 10.2196/10893] [Medline: 30355563]
51. Hassand a M, Lin unen T, Hagge MS, Heikkinen R, Vanhala M, Ke unen T. An mHeal h App o Suppo ing Qui e s o
Manage Ciga e e C a ings Wi h Sho Bou s o Physical Ac i i y: A Randomized Pilo Feasibili y and Accep abili y S udy.
JMIR Mheal h Uheal h 2017 May 26;5(5):e74 [FREE Full ex ] [doi: 10.2196/mheal h.6252] [Medline: 28550004]
52. McClu e JB, Ande son ML, B adley K, An LC, Ca z SL. E alua ing an Adap i e and In e ac i e mHeal h Smoking Cessa ion
and Medica ion Adhe ence P og am: A Randomized Pilo Feasibili y S udy. JMIR Mheal h Uheal h 2016 Aug 03;4(3):e94
[FREE Full ex ] [doi: 10.2196/mheal h.6002] [Medline: 27489247]
53. Whi ake R, McRobbie H, Bullen C, Rodge s A, Gu Y, Dobson R. Mobile phone ex messaging and app-based in e en ions
o smoking cessa ion. Coch ane Da abase Sys Re 2019 Oc 22;10:CD006611. [doi: 10.1002/14651858.CD006611.pub5]
[Medline: 31638271]
54. C emonesi P, Ko en Y, Tu in R, edi o s. Pe o mance o ecommende algo i hms on op-n ecommenda ion asks. :
RecSys '10; 2010 Sep P esen ed a : Fou h ACM Con e ence on Recommende Sys ems; 2010; Ba celona, Spain p. 39-46
URL: h ps://dl.acm.o g/doi/10.1145/1864708.1864721 [doi: 10.1145/1864708.1864721]
55. He nández del Olmo F, Gaudioso E. E alua ion o ecommende sys ems: A new app oach. Expe Sys ems wi h Applica ions
2008 Oc ;35(3):790-804. [doi: 10.1016/j.eswa.2007.07.047]
56. Ricci F, Rokach L, Shapi a B, Kan o P. In oduc ion o ecommende sys ems handbook. In: Recommende Sys ems
Handbook. Bos on, Massachuse s: Sp inge ; 2011:1-35.
57. Sch öde G, Thiele M, Lehne W. Se ing goals and choosing me ics o ecommende sys em e alua ions. In: The RecSys
2011 Wo kshops - Decisions@RecSys 2011 and UCERSTI-2. 2011 Oc 23 P esen ed a : UCERSTI2 Wo kshop a he 5 h
ACM con e ence on ecommende sys ems; 2011; Chicago p. 78-85 URL: h p://ceu -ws.o g/Vol-811/pape 12.pd
58. Shani G, Gunawa dana A. E alua ing ecommenda ion sys ems. In: Recommende sys ems handbook. Bos on, Massachuse s:
Sp inge ; 2011:257-297.
59. Benne J, Lanning S. The ne lix p ize. In: P oceedings o KDD Cup and Wo kshop 2007. 2007 Aug 12 P esen ed a : KDD
Cup and Wo kshop; 2007; San Jose, Cali o nia URL: h ps://www.cs.uic.edu/~liub/KDD-cup-2007/Ne lixP ize-desc ip ion.
pd
60. Li S, Ka ahanna E. Online Recommenda ion Sys ems in a B2C E-Comme ce Con ex : A Re iew and Fu u e Di ec ions.
JAIS 2015 Feb;16(2):72-107. [doi: 10.17705/1jais.00389]
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 21h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X

61. Scha e J, Kons an J, Riedl J. E-comme ce ecommenda ion applica ions. Da a mining and knowledge disco e y 2001
Jan;5(1-2):115-153 [FREE Full ex ] [doi: 10.1007/978-1-4615-1627-9_6]
62. He locke JL, Kons an JA, Te een LG, Riedl JT. E alua ing collabo a i e il e ing ecommende sys ems. ACM T ans.
In . Sys 2004 Jan;22(1):5-53. [doi: 10.1145/963770.963772]
63. Ge M, Delgado-Ba en eld C, Jannach D. Beyond accu acy: e alua ing ecommende sys ems by co e age and se endipi y.
In: RecSys '10: P oceedings o he ou h ACM con e ence on Recommende sys ems. 2010 Sep P esen ed a : The ou h
ACM con e ence on Recommende sys ems; 2010; Ba celona, Spain p. 257-260 URL: h ps://dl.acm.o g/doi/10.1145/
1864708.1864761 [doi: 10.1145/1864708.1864761]
64. McMillan B, Hickey E, Pa el MG, Mi chell C. Quali y assessmen o a sample o mobile app-based heal h beha io change
in e en ions using a ool based on he Na ional Ins i u e o Heal h and Ca e Excellence beha io change guidance. Pa ien
Educ Couns 2016 Ma ;99(3):429-435. [doi: 10.1016/j.pec.2015.10.023] [Medline: 26607787]
65. S oyano SR, Hides L, Ka anagh DJ, Zelenko O, Tjond onego o D, Mani M. Mobile app a ing scale: a new ool o
assessing he quali y o heal h mobile apps. JMIR Mheal h Uheal h 2015 Ma 11;3(1):e27 [FREE Full ex ] [doi:
10.2196/mheal h.3422] [Medline: 25760773]
66. Lewis TL, Wya JC. App Usage Fac o : A Simple Me ic o Compa e he Popula ion Impac o Mobile Medical Apps. J
Med In e ne Res 2015 Aug 19;17(8):e200 [FREE Full ex ] [doi: 10.2196/jmi .4284] [Medline: 26290093]
67. G undy QH, Wang Z, Be o LA. Challenges in Assessing Mobile Heal h App Quali y: A Sys ema ic Re iew o P e alen
and Inno a i e Me hods. Am J P e Med 2016 Dec;51(6):1051-1059. [doi: 10.1016/j.amep e.2016.07.009] [Medline:
27659122]
68. McKay FH, Cheng C, W igh A, Shill J, S ephens H, Uccellini M. E alua ing mobile phone applica ions o heal h beha iou
change: A sys ema ic e iew. J Telemed Teleca e 2016 Oc 18;24(1):22-30. [doi: 10.1177/1357633x16673538]
69. Singh K, D ouin K, Newma k L, Rozenblum R, Lee J, Landman A. De eloping a amewo k o e alua ing he pa ien
engagemen , quali y, and sa e y o mobile heal h applica ions. Issue B ie , Commonw Fund Pub 2016 Feb 18;5:11 [FREE
Full ex ] [doi: 10.15868/socialsec o .25066]
70. Ya dley L, Sp ing BJ, Ripe H, Mo ison LG, C ane DH, Cu is K, e al. Unde s anding and P omo ing E ec i e Engagemen
Wi h Digi al Beha io Change In e en ions. Am J P e Med 2016 No ;51(5):833-842. [doi: 10.1016/j.amep e.2016.06.015]
[Medline: 27745683]
71. Klasnja P, Consol o S, P a W. How o e alua e echnologies o heal h beha io change in HCI esea ch. In: CHI '11:
P oceedings o he SIGCHI Con e ence on Human Fac o s in Compu ing Sys ems. 2011 May P esen ed a : The SIGCHI
con e ence on human ac o s in compu ing sys ems; 2011; Vancou e , Canada p. 3063-3072 URL: h ps://dl.acm.o g/doi/
10.1145/1978942.1979396 [doi: 10.1145/1978942.1979396]
72. Smi h W, Plode e B, Wadley G, Webbe S, Bo land R. T ajec o ies o engagemen disengagemen wi h a s o y-based
smoking cessa ion app. In: CHI '17: P oceedings o he 2017 CHI Con e ence on Human Fac o s in Compu ing Sys ems.
2017 May P esen ed a : CHI '17: CHI Con e ence on Human Fac o s in Compu ing Sys ems; May, 2017; Den e , Colo ado,
USA p. 3045-3056 URL: h ps://dl.acm.o g/doi/10.1145/3025453.3026054 [doi: 10.1145/3025453.3026054]
73. Sche e EA, Ben-Zee D, Li Z, Kane JM. Analyzing mHeal h Engagemen : Join Models o In ensi ely Collec ed Use
Engagemen Da a. JMIR Mheal h Uheal h 2017 Jan 12;5(1):e1 [FREE Full ex ] [doi: 10.2196/mheal h.6474] [Medline:
28082257]
74. Taki S, Lyme S, Russell CG, Campbell K, Laws R, Ong K, e al. Assessing Use Engagemen o an mHeal h In e en ion:
De elopmen and Implemen a ion o he G owing Heal hy App Engagemen Index. JMIR Mheal h Uheal h 2017 Jun
29;5(6):e89 [FREE Full ex ] [doi: 10.2196/mheal h.7236] [Medline: 28663164]
75. O'Conno S, Hanlon P, O'Donnell CA, Ga cia S, Glan ille J, Mai FS. Unde s anding ac o s a ec ing pa ien and public
engagemen and ec ui men o digi al heal h in e en ions: a sys ema ic e iew o quali a i e s udies. BMC Med In o m
Decis Mak 2016 Sep 15;16(1):120 [FREE Full ex ] [doi: 10.1186/s12911-016-0359-3] [Medline: 27630020]
76. Eysenbach G. The law o a i ion. J Med In e ne Res 2005 Ma 31;7(1):e11 [FREE Full ex ] [doi: 10.2196/jmi .7.1.e11]
[Medline: 15829473]
77. Kohl LF, C u zen R, de V ies NK. Online p e en ion aimed a li es yle beha io s: a sys ema ic e iew o e iews. J Med
In e ne Res 2013 Jul 16;15(7):e146 [FREE Full ex ] [doi: 10.2196/jmi .2665] [Medline: 23859884]
78. Anscombe J, Baca di A. AT Kea ney Insigh s. Chicago, Illinois: AT Kea ney; 2011. Is Mobile he Cu e o he Rising
Cos s o Heal hca e? URL: h ps://www.a kea ney.com/documen s/10192/422267/
EAXIV_1_Is_Mobile_a_Cu e_ o _ he_Rising_Cos s_o _Heal hca e.pd /7337673b-bd46-4ae5-aa85-1906 83 6a05[accessed
2016-02-20] [WebCi e Cache ID 6 RLcACdh]
79. Iaco iello BM, S eine man JR, Klein DB, Sil e TL, Be ge AG, Luo SX, e al. Clicko ine, A Pe sonalized Sma phone
App o Smoking Cessa ion: Ini ial E alua ion. JMIR Mheal h Uheal h 2017 Ap 25;5(4):e56 [FREE Full ex ] [doi:
10.2196/mheal h.7226] [Medline: 28442453]
80. Owen JE, Jawo ski BK, Kuhn E, Makin-By d KN, Ramsey KM, Ho man JE. mHeal h in he Wild: Using No el Da a o
Examine he Reach, Use, and Impac o PTSD Coach. JMIR Men Heal h 2015 Ma 25;2(1):e7 [FREE Full ex ] [doi:
10.2196/men al.3935] [Medline: 26543913]
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 22h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
81. B icke JB, Copeland W, Mull KE, Zeng EY, Wa son NL, Akioka KJ, e al. Single-a m ial o he second e sion o an
accep ance & commi men he apy sma phone applica ion o smoking cessa ion. D ug Alcohol Depend 2017 Jan
01;170:37-42 [FREE Full ex ] [doi: 10.1016/j.d ugalcdep.2016.10.029] [Medline: 27870987]
82. Sho CE, DeSme A, Woods C, Williams SL, Mahe C, Middelwee d A, e al. Measu ing Engagemen in eHeal h and
mHeal h Beha io Change In e en ions: Viewpoin o Me hodologies. J Med In e ne Res 2018 No 16;20(11):e292
[FREE Full ex ] [doi: 10.2196/jmi .9397] [Medline: 30446482]
83. Okoshi T, Tsubouchi K, Taji M, Ichikawa T, Tokuda H. A en ion and engagemen -awa eness in he wild: A la ge-scale
s udy wi h adap i e no i ica ions. 2017 Ma 17 P esen ed a : 2017 IEEE In e na ional Con e ence on Pe asi e Compu ing
and Communica ions, Pe Com 2017; Ma ch, 2017; Big Island, Uni ed S a es p. 100-110 URL: h ps://ieeexplo e.ieee.o g/
documen /7917856 [doi: 10.1109/pe com.2017.7917856]
84. Schwanda V, Iba a S, Reynolds L, Cosley D. Side e ec s and ga eway ools: ad oca ing a b oade look a e alua ing
pe suasi e sys ems. In: CHI '11: P oceedings o he SIGCHI Con e ence on Human Fac o s in Compu ing Sys ems. Side
e ec sga eway ools: ad oca ing a b oade look a e alua ing pe suasi e sys ems. P oceedings o he SIGCHI Con e ence
on Human Fac o s in Compu ing Sys ems; 2011 May P esen ed a : CHI '11: CHI Con e ence on Human Fac o s in Compu ing
Sys ems; May, 2011; Vancou e , Canada p. 345-348 URL: h ps://dl.acm.o g/doi/10.1145/1978942.1978991 [doi:
10.1145/1978942.1978991]
85. Clawson J, Pa e J, Mille A, Myna E, Mamykina L. No longe wea ing: in es iga ing he abandonmen o pe sonal
heal h- acking echnologies on c aigslis . In: UbiComp '15: P oceedings o he 2015 ACM In e na ional Join Con e ence
on Pe asi e and Ubiqui ous Compu ing. 2015 Sep P esen ed a : UbiComp '15: The 2015 ACM In e na ional Join Con e ence
on Pe asi e and Ubiqui ous Compu ing; Sep embe , 2015; Osaka, Japan p. 647-658 URL: h ps://dl.acm.o g/doi/10.1145/
2750858.2807554 [doi: 10.1145/2750858.2807554]
86. Eps ein D, Ca away M, Johns on C, Ping A, Foga y J, Munson S. Beyond Abandonmen o Nex S eps: Unde s anding
and Designing o Li e a e Pe sonal In o ma ics Tool Use. In: CHI '16: P oceedings o he 2016 CHI Con e ence on
Human Fac o s in Compu ing Sys ems. 2016 May P esen ed a : CHI'16: CHI Con e ence on Human Fac o s in Compu ing
Sys ems; May 2016; San Jose, Cali o nia, USA p. 1109-1113 URL: h ps://dl.acm.o g/doi/10.1145/2858036.2858045 [doi:
10.1145/2858036.2858045]
Abb e ia ions
AI: a i icial in elligence
BMI: body mass index
CG: con ol g oup
Eu oQoL-5D-5L: Eu oQol 5-dimension 5-le el
HRS: heal h ecommende sys em
IG: in e en ion g oup
IPAQ: In e na ional Physical Ac i i y Ques ionnai e
ResQue: Recommende Sys ems Ques ionnai e o Use Expe ience
RRR: ela i e isk a io
SF-36: 36-i em Sho Fo m
Edi ed by G Eysenbach; submi ed 18.12.19; pee - e iewed by C Hao, A Vallée; commen s o au ho 13.01.20; e ised e sion ecei ed
03.03.20; accep ed 21.03.20; published 27.04.20
Please ci e as:
Ca asco-He nandez L, Jóda -Sánchez F, Núñez-Benjumea F, Mo eno Conde J, Mesa González M, Ci i -Balcells A, Ho s-F aile S,
Pa a-Calde ón CL, Bamidis PD, O ega-Ruiz F
A Mobile Heal h Solu ion Complemen ing Psychopha macology-Suppo ed Smoking Cessa ion: Randomized Con olled T ial
JMIR Mheal h Uheal h 2020;8(4):e17530
URL: h p://mheal h.jmi .o g/2020/4/e17530/
doi: 10.2196/17530
PMID:
©Lau a Ca asco-He nandez, F ancisco Jóda -Sánchez, F ancisco Núñez-Benjumea, Jesús Mo eno Conde, Ma co Mesa González,
An ón Ci i -Balcells, San iago Ho s-F aile, Ca los Luis Pa a-Calde ón, Panagio is D Bamidis, F ancisco O ega-Ruiz. O iginally
published in JMIR mHeal h and uHeal h (h p://mheal h.jmi .o g), 27.04.2020. This is an open-access a icle dis ibu ed unde
he e ms o he C ea i e Commons A ibu ion License (h ps://c ea i ecommons.o g/licenses/by/4.0/), which pe mi s un es ic ed
use, dis ibu ion, and ep oduc ion in any medium, p o ided he o iginal wo k, i s published in JMIR mHeal h and uHeal h, is
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 23h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X
p ope ly ci ed. The comple e bibliog aphic in o ma ion, a link o he o iginal publica ion on h p://mheal h.jmi .o g/, as well as
his copy igh and license in o ma ion mus be included.
JMIR Mheal h Uheal h 2020 | ol. 8 | iss. 4 | e17530 | p. 24h p://mheal h.jmi .o g/2020/4/e17530/ (page numbe no o ci a ion pu poses)
Ca asco-He nandez e alJMIR MHEALTH AND UHEALTH
XSL
•
FO
Rende X