Usability of a transdiagnostic internet-delivered protocol for anxiety and depression in community patients

Usability of a transdiagnostic Internet-based protocol for anxiety and depression in community patients Amanda Díaz-García1 [0000-0003-4398-4014], Alberto González-Robles1 [0000-0002-3749-3756], Javier Fernández-Álvarez2, Adriana Mira3, Diana Castilla3 [0000-0002-1631-1220], Juana María Bretón1 [0000-0001-6331-6078], Azucena García-Palacios1,4 [0000-0001-9250-8714], and Cristina Botella1,4 [0000-0001-8783-6959] 1 Universitat Jaume I, Castellón 2 Università Cattolica del Sacro Cuore, Milán 3 Universidad de Zaragoza 4 CIBER Fisiopatología Obesidad y Nutrición (CIBERObn), Instituto Salud Carlos III,Madrid Abstract. Internet-based psychological treatments have shown to be a promising solution to increase the accessibility to evidence-based treatments. However, the implementation of these interventions is still a challenge in health care settings. The study of the acceptability of these interventions may be a key aspect to reach successful implementation. Specifically, the study of usability may help to ensure that the interventions are well-designed and therefore increase the interest and number of people who can benefit from a psychological treatment. The present work aims to assess the usability of a transdiagnostic Internet-based treatment for emotional disorders among 87 patients who participated in it. The online program was considered well-accepted in terms of usability. This study analyzes the usability of an Internet-based treatment for emotional disorders, based on the transdiagnostic perspective and including a specific therapeutic component to address positive affect. Further research is needed in order to promote adherence and achieve the dissemination of evidence-based Internet-delivered psychological treatments. Keywords: Usability, Internet-based treatments, Emotional disorders. 1 Introduction Internet-based treatments (IBTs) have shown to be effective in the treatment of depression and anxiety disorders [1], being also considered as evidence based treatments for numerous psychological disorders [2]. Moreover, some meta-analyses reveal that these treatments are as efficacious as face-to-face traditional treatments [3]. Several advantages have been indicated in Internet interventions regarding the recruitment of patients, assessment and diagnosis, accessibility to evidence-based treatments, dissemination and comorbidity management [4]. In addition, the literature has pointed out that the use of IBTs can help to solve several mental health problems to overcome 2 common treatments barriers such as safety, geographical reach, acceptability and convenience [5]. Although Internet-based treatments seem to be a very promising solution to treat psychological disorders, it is essential to acquire more knowledge about its implementation and the acceptability of such interventions. In this regard, investigating the acceptability of the interventions delivered online may help to reach successful implementation in the routine clinical practice. Treatment acceptability refers to the degree to which users are satisfied or at ease with a service and willing to use it [6], and it has been identified as an important aspect for ethical, methodological and practical reasons in IBTs [7]. Furthermore, acceptability also refers to perceive the treatment as appropriate, fair, reasonable, and nonintrusive for a given problem [8]. Focusing on the acceptability, the literature suggests that the evaluation of the usability of these interventions is crucial in order to ensure that the system design is effective [9]. Usability testing has been described as a method for evaluating user performance and acceptance of a product during its development process [9]. Following the International Organization for Standardization guidelines, usability is measured by its effectiveness (i.e. the ability of the user to complete tasks using the system), efficiency (i.e. the resources expended in performing a task), and satisfaction (i.e. users’ subjective reactions to using the system) [10]. Moreover, different usability characteristics have been accepted as part of any software project such as learnability (i.e. to learn and interact rapidly with the system), efficiency (i.e. to achieve a high level of productivity), memorability (i.e. to retain knowledge about the system after a period of non-use), rate of errors (i.e. to have few errors while using the system), and satisfaction (i.e. to make the system pleasant to use it) [11]. The use of a well designed platform to deliver psychological treatments can have a great impact on increasing the interest and number of people who can benefit from them. However, few studies have assessed usability in Internetand Computer-based treatments [12, 13, 14, 15]. In this regard, the usability of these interventions should be further explored. The aim of this study is to evaluate the usability of a transdiagnostic Internet-based treatment for patients suffering from one or more emotional disorder (ED), including a specific therapeutic component to address positive affect. 2 Method 2.1 Participants This study is a secondary analysis of data collected as part of a clinical trial of a transdiagnostic Internet-based treatment for ED with a specific component to address positive affect [16]. Those participants interested in the study contacted via personal visits or phone calls to the Emotional Disorders University Clinic, through emails, or leaving their data on the clinic website. All participants were recruited from a community sample of patients diagnosed with one or more diagnosis of ED: major depressive disorder (MDD), dysthymic disorder (DD), (unipolar) mood disorder not otherwise specified, obsessive-compulsive disorder (OCD), and five anxiety disorders: 3 panic disorder (PD), agoraphobia (AG), generalized anxiety disorder (GAD), social anxiety disorder (SAD), anxiety disorder not otherwise specified (ADNOS) [17]. Individuals were eligible for the study if they were 18 years or older, met the DSM-IV diagnostic criteria for one or more of the aforementioned ED, had the ability to understand and read Spanish, had access to Internet at home and an email address, and provided online informed consent. Exclusion criteria were: a) suffering from Schizophrenia, bipolar disorder, or alcohol and/or substance dependence disorder; b) the presence of high risk of suicide; c) medical disease/condition that prevents the participant from carrying out the psychological treatment; d) receiving another psychological treatment during the study; or e) an increase and/or change in the pharmacological treatment during the study period (in the case of being in pharmachological treatment). All the participants included in this study participated in the transdiagnostic intervention protocol (described below). The treatment protocol from which these data were drawn was approved by the Ethics Committee of Universitat Jaume I (Castellón, Spain) and was registered at clinicaltrial.gov as NCT02578758. The sample was composed of 87 participants. Participants’ mean age was 36.75 years old (SD = 11.12, range 20-63), the majority were female (67%, 58/87), and most of them were single (54%, 47/87) and had higher studies (71%, 62/87). In addition, most participants suffered from GAD (31%, 27/87), followed by SAD (30%, 26/87) and MDD (12%, 10/87). Regarding the patterns of comorbidity in the sample, 42% of the participants had at least one comorbid diagnosis, with 11 individuals (13%) meeting criteria for two comorbid diagnoses, and 7 (8%) meeting criteria for three comorbid diagnoses. 2.2 Measures Diagnostic interview Mini International Neuropsychiatric Interview Version 5.0.0 (MINI) [18]. The MINI is a short, structured, diagnostic psychiatric interview for DSM-IV and ICD-10 diagnoses. This interview can be used by clinicians after a brief training session and has excellent inter-rater reliability (k = .88-1.00) and adequate concurrent validity with the Composite International Diagnostic Interview [18]. Usability Questionnaire System Usability Scale (SUS) [19, 20]. This scale assesses the usability of a service or product and the acceptance of technology by the people who use it. The SUS is a simple, ten-item scale that indicates the degree of agreement or disagreement with the statements on a 5-point scale (1=strongly disagree; 5=strongly agree). The final score is obtained by adding the scores on each item and multiplying the result by 2.5. Scores range from 0 to 100, where higher scores indicate better usability [21]. Following [21], the scores are replaced for adjectives and classified according to their acceptability, being not acceptable if the mean score is less than 50 and acceptable if the score is higher than 70. A score between 50 and 70 is classified as marginal accepta- 4 bility (see Fig. 1). The Usability and Acceptability Questionnaire is currently being validated by our research group, and a short-form consisting of 7 items was used in a previous study, showing a Cronbach’s Alpha of .94 [22]. Fig. 1. SUS scores by quartile ranges, acceptability ranges, and adjective ratings [19]. 2.3 Treatment protocol The treatment protocol is based on the classic transdiagnostic perspective derived from the Unified Protocol [23, 24] and some strategies from Marsha Linehan’s protocol [25]. The program includes core components, mainly designed to down-regulate negative affect (present-focused emotional awareness and acceptance, cognitive flexibility, behavioral and emotional avoidance patterns, and interoceptive and situational exposure) and a positive affect regulation component to promote psychological strengths and enhance well-being [26]. The treatment protocol also includes therapeutic components of evidence-based treatment for ED: psychoeducation, motivation for change, and relapse prevention. All these treatment components were developed through two self-applied protocol modalities with 12 (Transdiagnostic Internet-based protocol, TIBP) and 16 modules (Transdiagnostic Internet-based protocol + Positive Affect component, TIBP + PA), respectively. The description of the modules for each protocol modality have been described elsewhere [16]. Regardless of treatment modality, all participants completed the intervention through a multimedia web platform using videos, vignettes, audios, images, etc., in order to make the therapeutic content more attractive to the patients (https://www.psicologiaytecnologia.com). The program was designed to be completely self-applied via the Internet through a PC or a tablet and with a linear navigation in order to optimize the treatment structure, allowing participants with less experience in handling technologies to know how to keep moving forward at any time. 5 Fig. 2. “Screenshot” of one of the modules of the Internet-based treatment for emotional disorders 2.4 Statistical analyses Participant’s descriptive statistics of all sociodemographic characteristics and Student’s t-test for usability were examined. All statistical analyses were conducted using IBM SPSS Statistics for Windows, version 22. 3 Results 3.1 Socio-demographic data Details about participants’ sociodemographic characteristics are presented in Table 1. Table 1. Sociodemographic characteristics of participants TIBP (N=45) TIBP + PA (N=42) Total Sample Age (years) Mean (SD) Range 38.99 (12.38) 21-63 34.36 (9.15) 20-52 36.75 (11.12) 20-63 Gender, n (%) Male Female 13 (29) 32 (71) 16 (38) 26 (62) 29 (33) 58 (67) Marital Status, n (%) Single Married/Partnered 24 (53) 18 (40) 23 (55) 15 (36) 47 (54) 33 (38) 6 Divorced 3 (7) 4 (9) 7 (8) Education Level, n (%) Basic studies Medium studies Higher studies 1 (2) 11 (24) 33 (73) 4 (10) 9 (21) 29 (69) 5 (6) 20 (23) 62 (71) Principal diagnosis, n (%) MDD GAD PD/AG PD AG SAD OCD ADNOS 6 (13) 16 (36) 3 (7) 1 (2) 4 (9) 11 (24) 2 (4) 2 (4) 4 (10) 11 (26) 4 (10) 2 (5) 3 (7) 15 (35) 1 (2) 2 (5) 10 (12) 27 (31) 7 (8) 3 (3) 7 (8) 26 (30) 3 (3) 4 (5) Number of comorbid diagnoses, n (%) None 1 2 3 22 (49) 16 (35) 4 (9) 3 (7) 10 (24) 21 (50) 7 (17) 4 (9) 32 (37) 37 (42) 11 (13) 7 (8) Note: SD = Standard deviations. 3.2 Usability of the program Usability scores are shown in Table 2. According to [21], results showed that the program obtained high acceptability levels among participants in terms of usability. The overall score was 82.67/100 (SD=12.53). The Student’s t-test analysis did not reveal statistical differences between groups (t = -.60; p=.55). Table 2. System Usability Scale: Means and standard deviations TIBP (N=45) TIBP + PA (N=42) Total Sample 1. I think that I would like to use this system frequently 3.16 (.90) 3.38 (.79) 3.26 (.86) 2. I found the system unnecessarily complex 3.49 (.82) 3.14 (1.34) 3.32 (1.10) 3. I thought the system was easy to use 3.27 (1.23) 3.33 (1.28) 3.30 (1.25) 4. I think that I would need the support of a technical person to be able to use the system 3.18 (1.21) 3.19 (1.25) 3.18 (1.23) 5. I found the various functions in this system were well integrated 3.13 (1.12) 3.71 (.67) 3.41 (.97) 7 6. I thought there was too much inconsistency in this system 3.18 (1.13) 3.17 (1.21) 3.17 (1.16) 7. I would imagine that most people would learn to use this system very quickly 3.38 (.96) 3.76 (.91) 3.56 (.95) 8. I found the system very cumbersome to use 3.13 (1.46) 2.67 (1.73) 2.91 (1.60) 9. I felt very confident using the system 3.47 (.84) 3.86 (.65) 3.66 (.78) 10. I needed to learn a lot of things before I could get going with this system 3.38 (1.19) 3.19 (1.31) 3.29 (1.25) Overall score 81.89 (12.43) 83.51 (12.72) 82.67 (12.53) Note: TIBP: Transdiagnostic Internet-based protocol; TIBP + PA: Transdiagnostic Internetbased protocol + Positive Affect component. 4 Discussion The present study aimed to evaluate the usability of a transdiagnostic Internet-based treatment for patients suffering from one or more ED. The program was composed of a multimedia web platform including videos, images, vignettes, and audios, specifically designed to optimize the understanding of all the therapeutic content. Results from the SUS scale revealed that the program obtained high scores, between the third and fourth quartile, achieving the excellent rating on the Usability adjective rating scale in both treatment conditions, indicating that the program was considered very usable. In this regard, participants expressed willingness to use the system frequently, reported that the system was easy to use, and that it had functions that were well integrated. In addition, participants reported that people could learn to use the system very quickly and that they felt confident using the system. In summary, the results showed that the program was well-accepted, in terms of usability. The literature has suggested that the ease of use along with usefulness, service excellence, aesthetics, and playfulness is one the five key factors involved in the use of a system in the future [27]. Therefore, it is important to consider the study of the usability of Internet-based interventions as an important aspect in psychological treatments. Furthermore, other variables related to acceptability such as expectations, satisfaction, and treatment preference should also be considered. The present study represents an initial attempt to evaluate the acceptability of an Internet-based treatment for ED. However, this study presents some limitations that should be mentioned. First, this study only provides data about the usability of the Internet-based treatment. Information about satisfaction or treatment preferences had significantly contributed to the program’s acceptability. Second, participants in the study answered the usability scale with quantitative data but no qualitative feedback about the program was collected. Future studies should complement quantitative and 8 qualitative analyses in order to obtain more information about participant’s impressions of the program. In sum, to the best of our knowledge, the aim of this study is to analyze the usability of a transdiagnostic Internet-based treatment for ED that includes a specific therapeutic component to address positive affect. This program is presented as a wellaccepted online treatment in terms of usability. Further research is needed in this field in order to improve Internet-based programs and therefore increase the acceptance and dissemination of evidence-based Internet-delivered psychological treatments. References 1. Andersson G, Cuijpers P. Internet-Based and Other Computerized Psychological Treatments for Adult Depression: A Meta-Analysis. Cognitive Behaviour Therapy 38(4), 196–205 (2009). 2. Andersson G. Internet-Delivered Psychological Treatments. Annual Review of Clinical Psychology 12(1),157–79 (2016). 3. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer Therapy for the Anxiety and Depressive Disorders Is Effective, Acceptable and Practical Health Care: A Meta-Analysis. PLoS One 5(10):e13196 (2010). 4. Andersson G, Titov N. Advantages and limitations of Internet-based interventions for common mental disorders. World Psychiatry 13(1), 4–11 (2014). 5. Andrews G, Newby JM, Williams AD. Internet-Delivered Cognitive Behavior Therapy for Anxiety Disorders Is Here to Stay. Current Psychiatry Reports 17(1),1–5 (2014). 6. Peñate W, Fumero A. A meta-review of Internet computer-based psychological treatments for anxiety disorders. Journal of Telemedecin Telecare 22(1),3–11 (2016). 7. Kaltenthaler E, Sutcliffe P, Parry G, Beverley C, Rees A, Ferriter M. The acceptability to patients of computerized cognitive behaviour therapy for depression: A systematic review. Psychological Medicine 38(11),1521–30 (2008). 8. Kazdin AE. Acceptability of alternative treatments for deviant child behavior. Journal of Applied Behavior Analysis 13(2),259–73 (1980). 9. Kushniruk A. Evaluation in the design of health information systems: Application of approaches emerging from usability engineering. Computers in Biology and Medicine 32(3),141–9 (2002). 10. ISO 9241-11:1998 - Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability (1998). 11. Shneiderman B, Ben. Designing the user interface : strategies for effective human--computer interaction. Addison-Wesley (2010). 12. Anderson P, Zimand E, Schmertz SK, Ferrer M. Usability and Utility of a Computerized Cognitive-Behavioral Self-Help Program for Public Speaking Anxiety. Cognitive and Behavioral Practice 14(2),198–207 (2007). 13. Botella C, Mira A, Moragrega I, García-Palacios A, Bretón-López J, Castilla 9 D, et al. An Internet-based program for depression using activity and physiological sensors: Efficacy, expectations, satisfaction, and ease of use. Neuropsychiatric Disease and Treatment 12,393 (2016). 14. Castilla D, Garcia-Palacios A, Miralles I, Breton-Lopez J, Parra E, Rodriguez-Berges S, et al. Effect of Web navigation style in elderly users. Computers in Human Behavior 55,909–20 (2016). 15. Currie SL, McGrath PJ, Day V. Development and usability of an online CBT program for symptoms of moderate depression, anxiety, and stress in postsecondary students. Computers in Human Behavior 26(6),1419–26 (2010). 16. Díaz-García A, González-Robles A, Fernández-Álvarez J, García-Palacios A, Baños RM, Botella C. Efficacy of a transdiagnostic internet-based treatment for emotional disorders with a specific component to address positive affect: Study protocol for a randomized controlled trial. BMC Psychiatry 17(1),145 (2017). 17. American Psychological Association. Diagnostic and statistical manual of mental disorders (4th ed. Rev.). Washington, DC: American Psychiatric Association (2000). 18. Lecrubier Y, Sheehan D V., Weiller E, Amorim P, Bonora I, Sheehan KH, et al. The Mini International Neuropsychiatric Interview (MINI). A short diagnostic structured interview: Reliability and validity according to the CIDI. European Psychiatry 12(5),224–31 (1997). 19. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. International Journal of Human-Computer Interaction 24(6),574–94 (2008). 20. Brooke J. SUS - A quick and dirty usability scale. Usability Evaluation in Industry 189(194),4-7 (1996). 21. Bangor, Aaron, Philip Kortum, and James Miller. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies 4(3),114-123 (2009). 22. Castilla, D., Garcia-Palacios, A., Miralles, I., Breton-Lopez, J., Parra, E., Rodriguez-Berges, S., & Botella, C.. Effect of Web navigation style in elderly users. Computers in Human Behavior 55,909-920 (2016). 23. Ellard KK, Fairholme CP, Boisseau CL, Farchione TJ, Barlow DH. Unified Protocol for the Transdiagnostic Treatment of Emotional Disorders: Protocol Development and Initial Outcome Data. Cognitive and Behavioral Practice 17(1),88–101 (2010). 24. Barlow D, Allen LB, Choate ML. Toward a Unified Treatment for Emotional Disorders. Behavior Therapy 35,205–30 (2004). 25. Linehan M. Cognitive-behavioral treatment of borderline personality disorder. Guilford Press (1993). 26. Sin NL, Lyubomirsky S. Enhancing Well-Being and Alleviating Depressive Symptoms With Positive Psychology Interventions: A Practice-Friendly Meta-Analysis. Journal of Clinical Psychology 65(655),467–87 (2009). 27. Huang TL, Liao S. A model of acceptance of augmented-reality interactive technology: the moderating role of cognitive innovativeness. Electronic