Photo-realistic interactive virtual environments for neurorehabilitation in mild cognitive impairment (NeuroVRehab.PT) : a participatory design and proof-of-concept study

Journal of Clinical Medicine Article Photo-Realistic Interactive Virtual Environments for Neurorehabilitation in Mild Cognitive Impairment (NeuroVRehab.PT): A Participatory Design and Proof-of-Concept Study Filipa Ferreira-Brito 1,* , Sérgio Alves 2, Osvaldo Santos 1,3, Tiago Guerreiro 2, Cátia Caneiras 4,5 , Luís Carriço 2and Ana Verdelho 1,6,7 1Instituto de Saúde Ambiental (ISAMB), Faculdade de Medicina, Universidade de Lisboa, 1649-028 Lisboa, Portugal; [email protected] (O.S.); [email protected] (A.V.) 2LASIGE, Faculdade de Ciências Universidade de Lisboa, 1749-016 Lisboa, Portugal; [email protected] (S.A.); [email protected] (T.G.); [email protected] (L.C.) 3Unbreakable Idea Research, Lda, 2550-426 Painho, Portugal 4Laboratório de Investigação em Microbiologia na Saúde Ambiental (EnviHealthMicro Lab), Instituto de Saúde Ambiental (ISAMB), Faculdade de Medicina, Universidade de Lisboa, 1649-028 Lisboa, Portugal; [email protected] 5Healthcare Department, Nippon Gases Portugal, 2600-242 Vila Franca de Xira, Portugal 6 Instituto de Medicina Molecular, Faculdade de Medicina, Universidade de Lisboa, 1649-028 Lisboa, Portugal 7Neurology Service, Department of Neurosciences and Mental Health, Hospital de Santa Maria, Centro Hospitalar Universitário Lisboa Norte, 1649-028 Lisboa, Portugal *Correspondence: [email protected] Received: 26 October 2020; Accepted: 23 November 2020; Published: 26 November 2020   Abstract: Mild cognitive impairment (MCI) is characterized by cognitive, psychological, and functional impairments. Digital interventions typically focus on cognitive deficits, neglecting the difficulties that patients experience in instrumental activities of daily living (IADL). The global conjecture created by COVID-19 has highlighted the seminal importance of digital interventions for the provision of healthcare services. Here, we investigated the feasibility and rehabilitation potential of a new design approach for creating highly realistic interactive virtual environments for MCI patients’ neurorehabilitation. Through a participatory design protocol, a neurorehabilitation digital platform was developed using images captured from a Portuguese supermarket (NeuroVRehab.PT). NeuroVRehab.PT’s main features (e.g., medium-sized supermarket, the use of shopping lists) were established according to a shopping behavior questionnaire filled in by 110 older adults. Seven health professionals used the platform and assessed its rehabilitation potential, clinical applicability, and user experience. Interviews were conducted using the think-aloud method and semi-structured scripts, and four main themes were derived from an inductive semantic thematic analysis. Our findings support NeuroVRehab.PT as an ecologically valid instrument with clinical applicability in MCI neurorehabilitation. Our design approach, together with a comprehensive analysis of the patients’ past experiences with IADL, is a promising technique to develop effective digital interventions to promote real-world functioning. Keywords: virtual reality; cognition; transfer capacity; recovery of function; neurorehabilitation 1. Introduction Mild cognitive impairment (MCI) was initially conceptualized as a clinical entity affecting the cognitive functioning exclusively (i.e., memory capacity) [ 1 ]. However, empirical evidence has shown that J. Clin. Med. 2020,9, 3821; doi:10.3390/jcm9123821 www.mdpi.com/journal/jcm J. Clin. Med. 2020,9, 3821 2 of 18 cognitive and functional impairments co-exist from very early stages of the disease [ 2 ], and difficulties in instrumental activities of daily living (IADL) are prevalent among MCI patients [ 3 ]. Vascular mild cognitive impairment (VaMCI) is a clinical condition of vascular etiology, in which executive deficits are a prominent feature [ 4 , 5 ] and a strong predictor of functional decline and dementia [ 6 ]. Nonetheless, studies conducted in VaMCI patients are scarce, with even fewer studies focused on the development of rehabilitation instruments which target cognitive and functional impairments simultaneously. Virtual reality (i.e., computer-generated interactive environments) [ 7 ] has been recognized as a valuable resource for developing instruments that enable health professionals to accurately predict patients’ performance in everyday living activities (i.e., ecologically valid instruments) [ 8 , 9 ], especially in activities that actively engage executive functions [ 8 , 10 – 12 ]. Several virtual environments (VEs) have been developed targeting IADL, such as preparing meals [ 13 , 14 ], moving within the community [ 15 , 16 ], and cleaning and maintaining the house [17–20]. Shopping for groceries is perhaps one of the most studied IADL, with several studies showing that patients with executive deficits reveal in VEs a similar pattern of impairments to those observed in real-world tasks [ 21 – 27 ]. However, the majority of these VEs are manually designed, timeand human-resource consuming, and do not provide tasks or scenarios of sufficient realism [ 28 ]. Additionally, only a few of these VEs are designed for neurorehabilitation purposes [23] in MCI patients [22]. The feeling of “being physically present” in the virtual world, known as presence, is described as the phenomenon of users acting and experiencing emotions as if they were in the real world [ 29 ]; this is thought to promote the transfer of trained skills and behaviors from VEs to real-world contexts [ 8 , 29 ]. One design approach that has been used to improve the realism and sense of the presence is image-based rendering VEs. One example of a realistic image-based VE is Google Street View, where users can navigate within 360 ◦ photos of the surrounding environment. Previous findings have shown that the use of images of real-world scenarios results in highly visual realistic VEs and, therefore, an increased sense of presence [ 28 ]. Furthermore, image-based rendering VEs have shown promising results when applied to reminiscent therapy [ 28 ]. However, in the scarce studies that use this technique, the images are embedded as just a decorative/background element [ 30 ] or are non-interactive [ 15 ]. Other image-based VE interventions require complex technologic equipment and space availability in order to accommodate the experimental apparatus [ 28 ], which limits its widespread use in clinical and neurorehabilitation contexts. Based on the limitations mentioned above and considering the concept of function-led instruments [ 8 ], in which neuropsychological instruments should be as far as possible an accurate representation of real-world functioning, we conducted a qualitative analysis of the rehabilitation potential and clinical applicability of an image-based fully navigable and interactive virtual supermarket, NeuroVRehab.PT to promote cognition and functional capacity in VaMCI patients. This neurorehabilitation platform was developed by ISAMB (Lisbon, Portugal), LASIGE (Lisbon, Portugal), and Nippon Gases Portugal (Vila Franca de Xira, Portugal). The goals of this study are twofold: (1) to describe the design process of NeuroVRehab.PT and (2) to characterize health professionals’ (i.e., neurologists, psychologists, and neuropsychologists) perspectives about the rehabilitation potential and clinical applicability of NeuroVRehab.PT for VaMCI patients’ neurorehabilitation. 2. Experimental Section This study was conducted using a participatory design research protocol and reported according to the consolidated criteria for reporting qualitative research checklist (COREQ) guidelines [ 31 ] (see Supplementary Materials Table S1. COREQ: consolidated criteria for reporting qualitative research checklist). Two groups of stakeholders (i.e., older adults and health professionals) were invited to participate in the design of a virtual supermarket aiming to train the cognitive functions and behavior strategies recruited during a shopping activity. J. Clin. Med. 2020,9, 3821 3 of 18 The present study was performed in compliance with the Declaration of Helsinki and was approved by the following ethics committees: Comiss ã o de É tica do Centro Hospitalar de Lisboa Norte e Centro Acad é mico de Medicina de Lisboa–CAML (reference number 89/19) and Comiss ã o de É tica para Recolha e Proteç ã o de Dados de Ci ê ncias (CERPDC) (reference number CERPDC/16/2019) (see Supplementary Materials Doc S1, ethical standards, for a description of the ethical aspects considered during the execution of this project). 2.1. Participatory Design of NeuroVRehab.PT–Shopping Behaviors Questionnaire with Older Adults (Phase 1) 2.1.1. Sample and Recruitment Two senior universities of the municipality of Almada, Portugal, were contacted and agreed to participate in the study. Visits were scheduled to a group of classes identified earlier by the executive board of the two institutions. During these visits, one of the co-authors of this paper (FFB) presented the project and explained how the data gathered through the questionnaires would support the research team’s decisions regarding the main features of the virtual supermarket, such as the use (or not) of a shopping list, the number of products included in the shopping list, and the type of supermarket (grocery store vs. supermarket vs. hypermarket), among other features. To be eligible, the participants had to be more than 60 years old, be community dwelling, be responsible for grocery shopping, and give written informed consent. 2.1.2. Instruments and Procedure A questionnaire with 11 multiple-choice questions that aimed to analyze the shopping behaviors and routines of older adults was developed and reviewed by the research team (see Supplementary Materials Table S2 for the shopping behaviors questionnaire with older adults). The questionnaire was filled in individually and collected at the end of the senior universities’ sessions. Questions included items regarding the type of store they usually go to (i.e., local grocery store, supermarket, or department store), the time spent there, and the frequency they go shopping per week. Other items included the habit of using shopping lists, establishing budgets, shopping for weekly or monthly necessities, and an estimation of the amount of money spent while shopping. The questionnaire was anonymous, and besides age, gender, and professional status (active vs. retired), no other personal information was collected. 2.1.3. Data Analysis Statistical Package for the Social Sciences (IBM-SPSS, version 24.0; International Business Machines Corp., Armonk, New York, NY, USA) was used to conduct descriptive analyses of the data collected through the shopping behaviors questionnaire with older adults. For nominal variables (e.g., gender, professional status, and questionnaire responses), tables of frequencies were calculated. For continuous variables (e.g., age, time spent shopping), the mean, standard deviation, mode, minimum, and maximum were calculated. 2.1.4. Results A total of 110 participants aged between 61 and 86 years (70.92 ± 5.94 years) filled in the questionnaire. Twenty-eight were men (26.7%) and 77 women (73.3%). Fifty-three participants (53.5%) stated that they usually go to supermarkets as opposed to local grocery (n=8, 8.1%) stores or big department stores (n=38, 38.4%). Fifty-nine (56.2%) claimed that they usually make shopping lists, but only 42 (75%) use it during shopping. Before going shopping, 44 (79.4%) participants claimed that they included items in the shopping list as they remember what they need, and 10 (14.7%) organized the products according to the products’ position in the supermarket. More than half of the participants claimed they go to the supermarket less than once per week (n=56, 52.8%), alone (n=64, 60.4%), and to buy groceries for weekly necessities (n=59, 56.2%). On average, J. Clin. Med. 2020,9, 3821 4 of 18 the participants spend 59.24 ± 33.68 min on each visit to the supermarket. Sixty-one participants (58.1%) claimed that they have an estimation of how much they will spend before going to the checkout counter, and 33 (35.9%) stated that, after being informed of the value of the bill, they know precisely how much they should receive in return (see Supplementary Materials Table S2 for the shopping behaviors questionnaire in older adults descriptive data). 2.1.5. Main Implications for the Development of the NeuroVRehab.PT Supermarket was the most frequently visited type of store for food and house products shopping. A relevant percentage of the participants reported that they carry out their weekly shopping alone. The findings also suggested that shopping lists do not tend to be long, although diversified. The majority of older adults in our sample claimed that they go to the supermarket less than once per week and spend around 60 min on each visit. These data were initially collected to determine an adequate dose exposure to our digital platform. Although there are no guidelines regarding dose exposure in cognitive rehabilitation for MCI patients [ 32 , 33 ], previous studies have shown that interventions composed of few sessions with extended durations were not effective [ 34 ]. This implies that clinical trials of cognitive rehabilitation should accommodate the difficulties that VaMCI patients experience in their real-life activities (related or not to cognitive decline), provide sufficient repetition, and manage fatigue and frustration throughout the intervention length [35]. 2.2. Design and Implementation of the NeuroVRehab.PT (Phase 2) Basedon theshoppingbehaviorsurvey witholderadults, wedeveloped afirstprototype of aweb-based application called NeuroVRehab.PT. NeuroVRehab.PT allows people suffering from VaMCI to perform typical shopping tasks, such as creating shopping lists, navigating in a supermarket, adding products to the shopping basket, or sticking to a budget. The application was developed using HTML version 5.2. [ 36 ], JavaScript version ECMAScript 2019 [ 37 ], CSS version 3 [ 38 ], and PHP version 7.4.9 [ 39 ]. Thesupermarketwasconstructedusing thePhotoSphereViewerlibrary[ 40 ]andpanoramicphotographsof the interior of a typical Portuguese supermarket (captured using the Google Street View app for Android, Google LLC., Mountain View, CA, USA); see Supplementary Materials Doc S1, ethical standards, for a description of the ethical aspects considered during the image capture and editing). The platform is optimized to run on tablets. Tablets provide direct object manipulation (i.e., the person directly interacts with the target object using the fingers) and require less hand–eye coordination [ 41 ]. Older adults seem to prefer tablets to more traditional setups such as computers [ 42 ]. Previous studies showed that tablets are easier for older adults to use [ 43 ], even when they experience technological divide to some extent [44] or already present mild [20] or severe cognitive compromise (i.e., dementia) [45]. 2.2.1. Platform Description NeuroVRehab.PT is a prototype of a web-based digital neurorehabilitation platform composed of three independent game modes—supermarket, recipes, and shopping list. The supermarket is the central part of the platform and the key component of the three game modes. Supermarket Environment Description To replicate a real-world scenario, the environment of this system is composed of 49 360 ◦ panoramic photographs, together with typical supermarket noises at the background. The background noise can be turned offat any time. Users can only navigate in the supermarket by activating the full-screen mode or touching the start button (which also activates the full-screen mode). This way, users can have a clean screen to prevent distractions, as reported in previous studies [ 46 ]. Maria, a virtual shopping assistant, will guide users for the first time and regularly provide information about the game mechanics and behavioral shopping strategies that can be used in real life. Users experience the virtual supermarket from a first-person perspective (i.e., without any intermediating avatar) and can walk through the 19 supermarket sections (e.g., vegetables, fruits, bakery, dairy, frozen food) using the J. Clin. Med. 2020,9, 3821 5 of 18 arrows displayed on the screen (Figure 1). Navigation arrows are placed in the screen, at the corridors, and in the exact position where the users want to walk to. If necessary, users may zoom in and out to take a closer look at a product or use the autorotate button to locate themselves in the environment. To select a product, users touch the product and a tag with the product’s information (name, photo, category, description, and price) is displayed on the right side of the screen, together with an “add to the basket” button that allows users to add products to the shopping basket (Figure 2a). Figure 1. Interface elements of the NeuroVRehab.PT neurorehabilitation digital platform. Figure 2. ( a ) Tag with product information—name, photo, description/weight, price, and add button; ( b ) view of the screen with the levels of difficulty, three-star classification, level ID, level of difficulty; ( c ) shopping list from a difficult level (medium) with nine products (written in black) under the correspondent category (written in blue). Supermarket Game Mode The system provides fourteen game levels that ask users to go shopping with a predefined shopping list (Figure 2b). Levels have different difficulty levels—easy, medium, and hard—which differ regarding the number of products to buy, the distance between the products, the available time to complete the level, and the presence or absence of background noise. In more advanced levels, there is also a budget to be met. Furthermore, the platform is designed to enable health professionals to create custom levels. Users need to purchase all the items of the shopping list (Figure 2c) to complete the level. If the added product is listed on the shopping list, users will hear a sound of positive feedback and earn points. If users try to add a product that is not on the shopping list, a message together with a sound of negative feedback will appear, and users will keep the same points. J. Clin. Med. 2020,9, 3821 6 of 18 After selecting all the products on the shopping list, users are asked to go to the checkout counter zone and pay the groceries by choosing one of the payment methods (cash or credit card). At the end of each level, a three-star rating is attributed based on the users’ performance (time and distance walked). Recipe Game Mode The recipe game mode allows users to shop for the ingredients that are required to cook a traditional dish. Users may select among six traditional recipes—one soup, four main courses, and one dessert (Figure 3a). Each recipe is identified by a name and a photo of the dish. As an extra step of difficulty, after selecting one of the recipes the participants are asked to organize the ingredients under the correct category (e.g., apples under fruit) (Figure 3b). Correct and incorrect sorting is identified by turning the ingredients green or red, respectively. The participants need to correctly organize all the ingredients before having access to the virtual supermarket and purchasing the ingredients of the recipe. Figure 3. ( a ) View of the recipe game mode; ( b ) extra step of difficultly in the recipe game mode, in which users have to organize the ingredients (left side of the screen) under the corresponding category (right side of the screen); ( c ) view of the shopping list game mode with two products included in the list. Shopping List Game Mode In this game mode, users can create personalized shopping lists. A list ID is identified by a name customized by users, and they may add products by writing the name of the product in the search field. As they write, a drop-down list appears with products matching the search query (Figure 3c). To select an item, they have to touch it. After finishing their shopping list, the participants are asked to organize the ingredients under the correct category (similar to the activity described in the recipe game mode section) before moving on to the virtual supermarket. 2.3. Software Evaluation and User Experience (Phase 3) 2.3.1. Sample and Recruitment A purposive sample of health professionals (i.e., psychologists, neuropsychologists, and neurologists) identified through professional networking was contacted and invited to participate in the study. Health professionals were identified based on (1) their clinical and scientific experience ( ≥ 5 years) with ageing, age-related neurological disorders, and cognitive decline; (2) their knowledge of the main theoretical models of human cognition and neurorehabilitation; (3) their familiarity with computerized cognitive training or rehabilitation programs. All the participants who were contacted (face-to-face) agreed to participate, and an individual session was booked at the participants’ convenience (local and date). All the participants signed the written informed consent. 2.3.2. Instruments and Procedure Interviews took place in quiet and private rooms and lasted approximately 60 min. The participants sat in front of a desk where a Huawei MediaPad T5 tablet (Android 8) (Huawei Technologies Co. Ltd, J. Clin. Med. 2020,9, 3821 7 of 18 Shenzhen, China) was supported horizontally at 25 ◦ degrees (approximately) using a tablet stand. The platform was run in the Google Chrome application (version 79.0.3945; Google Inc., Menlo Park, CA, USA). A neuropsychologist with previous experience in qualitative research (FFB) conducted the interviews, and a second researcher member (SA, computer scientist and PhD student) was also present in five of the seven interviews and ensured that the sessions went on without any technical problems. Each session was divided into two moments. In the first part of the interview, the participants were encouraged to expresstheiropinions and thoughts verbally—the think aloud method(TA)[ 47 ]—while using the platform. The interviewer (FFB) demonstrated the TA method to participants while exploring the Gmail website. Then, the participants were encouraged to use it on the NeuroVRehab.PT platform. The participants were free to explore the platform; however, sessions were conducted so that all the participants visited the three game modes (i.e., supermarket, recipes, and shopping list) and played at least two game levels in the supermarket mode. The second part of the session consisted of a semi-structured interview focused on the participants’ experience (i.e., user experience; UX) and the perceived clinical applicability and rehabilitation potential of the platform. 2.3.3. Data Analysis The participants’ demographic data were analyzed using IBM-SPSS (version 24.0; International Business Machines Corp., Armonk, New York, United States). Interviews were transcribed for content analysis, and a list of codes was developed on the basis of two interviews (data grounded theory). This initial code system was created independently by two researchers (FFB and SA) and then discussed and merged into a comprehensive list. The remaining interview transcripts were coded independently by two researchers (FFB and SA) using the previously developed list of codes and an interrater reliability index of Cohen’s k=0.82 was obtained by calculating the mean of Cohen’s kappa indexes per interview (Cohen’s k ≥ 0.80). This kappa index was used as indicative of strong interrater reliability in healthcare research [ 48 ]. Blocks of 20 lines were used as the unity of analysis for semantic thematic analysis purposes [ 49 ]. When completing the analysis, the code system was reorganized into broad categories (major themes) and respective subcategories (minor themes). 3. Results Seven health professionals aged between 29 and 67 (47.14 ± 13.08 years) and with 17 years of professional experience (range: 5–40 years) were interviewed. Two participants were male and 5 were female, with academic backgrounds in medicine/neurology (n=2) or in neuropsychology (n=5). Two participants rated themselves as being very confident, four as being confident, and one as a little confident in using technology or new technological devices (n=2, n=2, n=3, respectively). On average, the participants spent 16 ± 7.99 h (range: 7–30 h) browsing content on the internet and 0.86 ± 1.86 h (range: 0–5 h) playing video games per week. All the participants were familiar with at least one computerized cognitive training or rehabilitation program (e.g., Cogweb ® , Rehacom ® ), and three participants (42.9%) reported that they use brain training games or computerized cognitive training programs (CCTP) in their professional practice (e.g., Fitbrain, Neuronation). Four major themes (and seven minor themes) emerged from the semantic thematic analysis: experience with NeuroVRehab.PT, rehabilitation potential, potential barriers, and opportunities (see Table 1for the complete code system, including minor themes). Due to technical problems related to audio recording, the second part of interview 2 was lost (90% of the semi-structured interview). Therefore, only the information collected during the TA phase of interview 2 was included in data analysis. Data saturation was obtained at the 5th interview, with no new codes identified in the last two interviews. J. Clin. Med. 2020,9, 3821 8 of 18 Table 1. Code system developed with the semantic analysis of the interviews. ⇒Experience with NeuroVRehab.PT •Hedonic experience and presence •Usability ⇒Rehabilitation potential •Cognitive stimulation •Transfer capacity ⇒Potential barriers ⇒Opportunities •Clinical and non-clinical contexts •Friend sourcing •Other environments 3.1. Experience with NeuroVRehab.PT 3.1.1. Hedonic Experience and Presence Overall, the participants expressed a positive attitude towards NeuroVRehab.PT and reported having fun and enjoying the platform. Different features of the platform stood out and were considered by the participants as appealing and relevant from a clinical point of view (e.g., the icon used to identify the different game levels (Figure 2b), the label with the product’s characteristics (Figure 2a), the zoom-in functionality, and the availability of a shopping list that the participants can check (Figure 1)). However, it was the high realism and visual complexity (and auditory stimuli) of the VE that captured the participants’ attention. All the participants but one explicitly mentioned this aspect during the interviews (and some of them more than once). Participant 1: “I was expecting something more, rudimentary, but not the case, I think it was . . . the products were clear and colors vivid . . . ” Participant 2: “Well, it is a very appealing image of the supermarket. It makes you want to explore it, doesn’t it? It has beautiful fruits.” Participant 4: “Ah! How cute ( . . . ) it really looks like a supermarket ( . . . ), maybe this is really a supermarket ( . . . ) The products are real, not drawings [as in 3D-modulated scenarios], I think it is good.” Participant 5: “[While performing a task in NeuroVRehab.PT] Ok, hot chocolate. Hot chocolate, now I have to find . . . . this is really realistic. In fact, it is real, I did not have this expectation.” Participant 6: “It seems very realistic, hyper-realistic.” (line 51) “The locations, the type of products, yes, yes, I think is quite realistic.” Participant 7: “The environment sound is very good; it really puts you inside of a supermarket (. . . ) we know where we are, perfectly.” 3.1.2. Usability The participants identified and played the three game modes without relevant difficulties. However, the navigation controls and the sense of orientation inside NeuroVRehab.PT were identified as two aspects that should be improved in order to provide a more smooth and pleasant experience. For instance, some participants misinterpreted the meaning of the navigation controls and interpreted them as mandatory actions to progress in the game. J. Clin. Med. 2020,9, 3821 9 of 18 Participant 2: “The arrow appears, I assume this means I need to follow the arrows.” Participant 1: “ . . . the arrow then . . . means that I need to go back to the fruit section?” Other participants considered the arrows as hints to the location of the next product in the shopping list. Participant 5: “The arrow helps to orient, doesn’t it? I did not understand if the arrow gives you a hint or if . . . Does it give you a hint?” Regarding the sense of being oriented within the supermarket, the participants felt that it “. . . could be useful, for example, before starting (the game levels) that the person visits the whole supermarket, a kind of orientation exercise, to learn (the supermarket) more or less.” (Participant 1). Still on this subject, another participant referred to the famous experiment of Willard S. Small [ 50 ] to explain the importance of having the opportunity to explore (learning) the environment before starting any specific task: “ . . . a person is in that environment; it is like a mouse when it is put on a place, like (. . . ) a maze, something like that, it (the mouse) will explore, the first thing it will do is to explore the surroundings to get a perception. We are a little bit like mice. The first thing a person wants to do is . . . in fact, surrounded by these fruits, is to see what is around and understand . . . ” (Participant 2). In this regard, the participants suggested that the presence of signs with the names of the sections and/or a map of the supermarket would improve the sense of orientation as well as the learning of the VE. 3.2. Rehabilitation Potential The health professionals considered NeuroVRehab.PT a useful and innovative instrument for cognitive stimulation that they would recommend to their MCI patients. Participant 5: “[regarding another virtual supermarket for IADL rehabilitation] ( . . . ) it was prehistoric when compared to this one.” Participant 2 “ . . . a good alternative for cognitive stimulation.” 3.2.1. Cognitive Stimulation The participants considered NeuroVRehab.PT “ . . . sufficiently appealing and demanding for MCI patients . . . ” (Participant 5) and comprehensive in terms of the cognitive functions stimulated—namely, executive functions (working memory, planning, decision making), memory, attention, spatial orientation, and math abilities. Participant 5 “ . . . working memory, of course, always pumping in my head ( . . . ) Manifestly, this also trains orientation . . . .”. [ . . . ] From an attentional point of view, it is quite demanding”. In addition, the choice for a shopping activity as the core activity of this digital intervention was considered as “ . . . meeting the necessities of this population.” (Participant 1) and “ . . . important, crucial for [patients] daily-life, if they do not have someone to do the shopping for them, they have to do it themselves . . . ” (Participant 7). The realism of the scenario was also identified as an asset that could be used to promote patients’ motivation to comply with long and emotionally demanding rehabilitation programs. Participant 7: “[regarding other CCTP] I think there is not an effort or an intent to be similar to the person’s daily life. Here I notice that effort ( . . . ) and that could be more motivating for the person who is doing the training”. J. Clin. Med. 2020,9, 3821 16 of 18 32. Bahar-Fuchs, A.; Martyr, A.; Goh, A.; Sabates, J.; Clare, L. Cognitive training for people with mild to moderate dementia. Cochrane Database Syst. Rev. 2019,2019. [CrossRef] 33. Bahar-Fuchs, A.; Clare, L.; Woods, B. Cognitive training and cognitive rehabilitation for persons with mild to moderate dementia of the Alzheimer’s or vascular type: A review. Alzheimer’s Res. Ther. 2013 ,5. [CrossRef] 34. Li, H.; Li, J.; Li, N.; Li, B.; Wang, P.; Zhou, T. Cognitive intervention for persons with Mild Cognitive Impairment: A meta-analysis. Ageing Res. Rev. 2011,10, 285–296. [CrossRef] [PubMed] 35. Sohlberg, M.M.; Mateer, C.A. Cognitive Rehabilitation: An Integrative Neuropsychological Approach; Guilford Press: New York, NY, USA, 2001; ISBN 9781572306134. 36. HTML 5.2. Available online: https://www.w3.org/TR/html52/(accessed on 24 November 2020). 37. ECMAScript ® 2019 Language Specification. Available online: https://www.ecma-international.org/ecma262/10.0/index.html (accessed on 24 November 2020). 38. Cascading Style Sheets. Available online: https://www.w3.org/Style/CSS/(accessed on 24 November 2020). 39. PHP: Hypertext Preprocessor. Available online: https://www.php.net/(accessed on 24 November 2020). 40. Photo Sphere Viewer. Available online: https://photo-sphere-viewer.js.org/(accessed on 24 November 2020). 41. Blendinger, K. Tablet-Applications for the Elderly: Specific Usability Guidelines; Ulm University: Ulm, Germany, 2015. 42. Canini, M.; Battista, P.; Della Rosa, P.A.; Catrical à , E.; Salvatore, C.; Gilardi, M.C.; Castiglioni, I. Computerized neuropsychological assessment in aging: Testing efficacy and clinical ecology of different interfaces. Comput. Math. Methods Med. 2014,2014. [CrossRef] [PubMed] 43. Shneiderman, B. Touchscreens now offer compelling uses. IEEE Softw. 1991,8, 93–94. [CrossRef] 44. Delello, J.A.; McWhorter, R.R. Reducing the digital divide: Connecting older adults to iPad technology. J. Appl. Gerontol. 2015, 1–26. [CrossRef] 45. Astell, A.J.; Joddrell, P.; Groenewoud, H.; de Lange, J.; Goumans, M.; Cordia, A.; Schikhof, Y. Does familiarity affect the enjoyment of touchscreen games for people with dementia? Int. J. Med. Inform. 2016 ,91, e1–e8. [CrossRef] 46. Alves, S.; Brito, F.; Cordeiro, A.; Carriço, L.; Guerreiro, T. Designing personalized therapy tools for people with dementia. In Proceedings of the 16th Web For All 2019 Personalization-Personalizing the Web, San Francisco, CA, USA, 13–14 May 2019. [CrossRef] 47. Someren, M.; Barnard, Y.F.; Sandberg, J. The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes; Van Someren, M.W., Barnard, Y.F., Barnard, J.A.C., Eds.; Academic Press: London, UK, 1994; ISBN 0127142703. 48. McHugh, M.L. Lessons in biostatistics interrater reliability: The kappa statistic. Biochem. Med. 2012 ,22, 276–282. [CrossRef] 49. Braun, V.; Clarke, V. Using thematic analysis in psychology. Qual. Res. Psychol. 2006,3, 77–101. [CrossRef] 50. Schultz, D.; Schultz, S.E. Rats, ants, and the animal mind. In A History of Modern Psychology; Wadsworth Cengage Learning: Boston, MA, USA, 2011; pp. 193–195. ISBN 5856420187. 51. Josman, N.; Hof, E.; Klinger, E.; Mari é , R.M.; Goldenberg, K.; Weiss, P.L.; Kizony, R. Performance within a virtual supermarket and its relationship to executive functions in post-stroke patients. In Proceedings of the 2006 International Workshop on Virtual Rehabilitation, New York, NY, USA, 29–30 August 2006; pp. 106–109. [CrossRef] 52. Werner, P.; Rabinowitz, S.; Klinger, E.; Korczyn, A.D.; Josman, N. Use of the virtual action planning supermarket for the diagnosis of Mild Cognitive Impairment. Dement. Geriatr. Cogn. Disord. 2009 ,27, 301–309. [CrossRef] 53. Chaytor, N.; Schmitter-Edgecombe, M. The ecological validity of neuropsychological tests: A review of the literature on everyday cognitive skills. Neuropsychol. Rev. 2003,13, 181–197. [CrossRef] [PubMed] 54. Spooner, D.M.; Pachana, N.A. Ecological validity in neuropsychological assessment: A case for greater consideration in research with neurologically intact populations. Arch. Clin. Neuropsychol. 2006 ,21, 327–337. [CrossRef] [PubMed] 55. Deterding, S.; Dixon, D.; Khaled, R.; Nacke, L. From Game Design Elements to Gamefulness: Defining “Gamification”. In Proceedings of the MindTrek’11, Tampere, Filand, 28–30 September 2011; pp. 28–30. 56. Deterding, S.; Dixon, D.; Khaled, R.; Nacke, L.; Sicart, M.; O’Hara, K. Gamification: Using game design elements in non-game contexts. In Proceedings of the 2011 Annual Conference Extended Abstracts on Human Factors in Computing Systems (CHI 2011), Vancouver, BC, Canada, 7–12 May 2011; pp. 2425–2428. [CrossRef] J. Clin. Med. 2020,9, 3821 17 of 18 57. Lumsden, J.; Edwards, E.A.; Lawrence, N.S.; Coyle, D.; Munaf ò , M.R. Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy. JMIR Serious Games 2016 ,4, 1–14. [CrossRef] [PubMed] 58. Sardi, L.; Idri, A.; Fern á ndez-Alem á n, J.L. A systematic review of gamification in e-Health. J. Biomed. Inform. 2017,71, 31–48. [CrossRef] 59. Ferreira-Brito, F.; Fialho, M.; Virgolino, A.; Neves, I.; Miranda, A.C.; Sousa-Santos, N.; Caneiras, C.; Carriço, L.; Verdelho, A.; Santos, O. Game-based interventions for neuropsychological assessment, training and rehabilitation: Which game-elements to use? A systematic review. J. Biomed. Inform. 2019 ,98, 103287. [CrossRef] 60. Sailer, M.; Hense, J.U.; Mayr, S.K.; Mandl, H. How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction. Comput. Hum. Behav. 2017 ,69, 371–380. [CrossRef] 61. Simons, D.J.; Boot, W.R.; Charness, N.; Gathercole, S.E.; Chabris, C.F.; Hambrick, D.Z.; Stine-Morrow, E.A.L. Do “Brain-Training” programs work? Psychol. Sci. Public Interest Suppl. 2016,17, 103–186. [CrossRef] 62. Zhang, H.; Huntley, J.; Bhome, R.; Holmes, B.; Cahill, J.; Gould, R.L.; Wang, H.; Yu, X.; Howard, R. Effect of computerised cognitive training on cognitive outcomes in Mild Cognitive Impairment: A systematic review and meta-analysis. BMJ Open 2019,9. [CrossRef] 63. Clare, L.; Woods, B. Cognitive rehabilitation and cognitive training for early-stage Alzheimer’s disease and vascular dementia. Cochrane Database Syst. Rev. 2003,4. [CrossRef] 64. De Giglio, L.; De Luca, F.; Prosperini, L.; Borriello, G.; Bianchi, V.; Pantano, P.; Pozzilli, C. A low-cost cognitive rehabilitation with a commercial video game improves sustained attention and executive functions in multiple sclerosis: A pilot study. Neurorehabil. Neural Repair 2015,29, 453–461. [CrossRef] 65. Nouchi, R.; Taki, Y.; Takeuchi, H.; Hashizume, H.; Nozawa, T.; Kambara, T.; Sekiguchi, A.; Miyauchi, C.M.; Kotozaki, Y.; Nouchi, H.; et al. Brain training game boosts executive functions, working memory and processing speed in the young adults: A randomized controlled trial. PLoS ONE 2013 ,8, e55518. [CrossRef] [PubMed] 66. L ó pez-Mart í n, O.; Fragoso, A.S.; Hern á ndez, M.R.; Terrer, I.D.; Polonio L ó pez, B. Efectividad de un programa de juego basado en realidad virtual para la mejora cognitiva en la esquizofrenia. Gac. Sanit. 2016 ,30, 133–136. [CrossRef] [PubMed] 67. Levine, B.; Robertson, I.H.; Clare, L.; Carter, G.; Hong, J.; Wilson, B.A.; Duncan, J.; Stuss, D.T. Rehabilitation of executive functioning: An experimental-clinical validation of goal management training. J. Int. Neuropsychol. Soc. 2000,6, 299–312. [CrossRef] [PubMed] 68. Londos, E.; Boschian, K.; Lind é n, A.; Persson, C.; Minthon, L.; Lexell, J. Effects of a Goal-Oriented Rehabilitation program in Mild Cognitive Impairment: A pilot study. Am. J. Alzheimer’s Dis. Other Dement. 2008,23, 177–183. [CrossRef] [PubMed] 69. Clare, L.; van Paasschen, J.; Evans, S.J.; Parkinson, C.; Woods, R.T.; Linden, D.E.J. Goal-oriented cognitive rehabilitation for an individual with Mild Cognitive Impairment: Behavioural and neuroimaging outcomes. Neurocase 2009,15, 318–331. [CrossRef] [PubMed] 70. Wilson, B.A. Memory Rehabilitation: Integrating Theory and Practice; The Guilford Prss: New York, NY, USA, 2009; Volume 25, ISBN 9781606232873. 71. De Schutter, B.; Vanden Abeele, V. Designing meaningful play within the psycho-social context of older adults. In Proceedings of the 3rd International Conference on Fun and Games, Leuven, Belgium, 15–17 September 2010; pp. 84–93. [CrossRef] 72. Barrett, M.; Chua, W.-J.; Crits-Christoph, P.; Gibbons, M.B.; Casiano, D.; Thompson, D. Early withdrawal from mental health treatment: Implications for psychoterapy pratice. Psychotherapy 2008 ,45, 247–267. [CrossRef] [PubMed] 73. Martin, D.J.; Garske, J.P.; Katherine Davis, M. Relation of the therapeutic alliance with outcome and other variables: A meta-analytic review. J. Consult. Clin. Psychol. 2000,68, 438–450. [CrossRef] 74. Roos, J.; Werbart, A. Therapist and relationship factors influencing dropout from individual psychotherapy: A literature review. Psychother. Res. 2013,23, 394–418. [CrossRef] 75. Werbart, A.; Wang, M. Predictors of not starting and dropping out from psychotherapy in Swedish public service settings. Nord. Psychol. 2012,64, 128–146. [CrossRef] J. Clin. Med. 2020,9, 3821 18 of 18 76. Martin, L.; Williams, S.L.; Haskard, K.; DiMatteo, M.R. The challenge of patient adherence. Ther. Clin. Risk Manag. 2005,1, 189–199. [CrossRef] 77. Vygotsky, L.S. Interaction between learning and development. In Mind and Society; Harvard University Press: Cambridge, MA, USA, 1978; pp. 79–91. ISBN 9780674576292. 78. Barnard, Y.; Bradley, M.D.; Hodgson, F.; Lloyd, A.D. Learning to use new technologies by older adults: Perceived difficulties, experimentation behaviour and usability. Comput. Hum. Behav. 2013 ,29, 1715–1724. [CrossRef] 79. WHO. Monitoring and Evaluating Digital Health Interventions. A Pratical Guide to Conducting Research and Assessment; WHO: Geneva, Switzerland, 2016; ISBN 9789241511766. Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).