Document [original]

11 Using Immersive Tech-

nologies to Support

Healthcare of Elderly and

Fragile People

Vona, Francesco 1

Ashrafi, Navid 1

Ćeranić, Miladin 1

Voigt-Antons, Jan-Niklas 1

1 Hamm-Lippstadt University of Applied Sciences, Immersive

Reality Lab

ORCID:

Vona, Francesco: 0000-0003-4558-4989

Ashrafi, Navid: 0009-0005-8398-415X

Ceranic, Miladin: 0009-0005-4700-1240

Voigt-Antons, Jan-Niklas: 0000-0002-2786-9262

11 Introduction

In recent years, the world of healthcare has seen a remarkable trans-

formation through the integration of immersive technologies. With an

aging global population, the demand for effective, compassionate, and

engaging healthcare solutions for the elderly has never been greater.

Immersive technologies, such as virtual reality (VR) and augmented

reality (AR), hold significant potential in the therapeutical field, where

they can be used to support older people (Vona et al., 2022b), individu-

als with cognitive disorders (Vona et al., 2022a, 2020), and other types

of conditions such as stuttering (Vona et al., 2023) or phobias (Morina

et al., 2023).

At the heart of immersive technologies lies the con-

cept of the Virtual Continuum. This concept bridges the gap between

the physical and digital realms, allowing individuals to seamlessly

transition between real-world environments and digital experienc-

es (Milgram et al., 1995). It encompasses a spectrum of technologies,

from augmented reality to virtual reality, all with the common goal of

enhancing the human experience.

Augmented Reality: AR overlays digital information and

interactive elements onto the real world (Azuma, 1997). In the context

of elderly care, AR can provide valuable support by enhancing the daily

lives of seniors. It can assist in memory and cognitive training, helping

individuals navigate their surroundings, remember important infor-

mation, and maintain mental agility. AR applications can also enable

remote consultations with healthcare professionals, offering timely

guidance and support.

Virtual Reality: VR immerses users in entirely digital

environments, creating a sense of presence and engagement (Jerald,

2016). For elderly individuals, VR can be a powerful tool for improving

physical and mental well-being. It can be used in rehabilitation thera-

py to enhance mobility and coordination, reduce pain perception, and

alleviate symptoms of conditions such as dementia and depression. VR

experiences can also serve as a form of entertainment and socializa-

tion, combating feelings of isolation among the elderly.

The potential of these immersive technologies in el-

derly care is vast. They can be applied in various contexts, including

rehabilitation centers, nursing homes, and home healthcare settings.

These technologies can help manage chronic pain, reduce anxiety, and

promote physical activity. Additionally, they facilitate cognitive stimu-

lation, reminiscence therapy, and social interaction, which are crucial

aspects of maintaining the mental and emotional well-being of elderly

individuals.

11 In this article, we are going to look at three examples of

virtual reality applications for three different types of users. The first

example will present a VR system for assessing MCI in elderly people.

The second example presents a VR system used to do public speaking

with people who stutter. The third example presents a VR system for

doing exposure therapy with patients who are afraid of dogs.

In all of the mentioned cases, virtual reality is used to

recreate a safe and secure environment that can be customized by the

therapist in order to improve the effectiveness of the experience for

the specific user. While MCI is related to age and older adults, the lat-

ter two examples of exposure therapy represent how immersive media

could support the healthcare of more fragile users, and this applica-

tion field can be transferred to the care of elderly people as well.

Example 1: VR for assessing Cognitive Func-

tions in elderly people

The first example presents an innovative VR-based tool to enhance the

ecological validity of screening tests for detecting Mild Cognitive Im-

pairments in elderly people (for a detailed description and results, see

(Vona et al., 2022b)). A MCI refers to the intermediate stage between

typical age-related cognitive decline and the early stages of dementia

(Chun et al., 2021).

The system recreates existing screening tests using a

360 virtual environment, recreating a familiar setting for the patients.

These virtual environments were captured with the Insta360 One X

camera, an omnidirectional video camera that can record spherical

photos and videos. The VR application runs on Pico devices with eye

trackers, such as the Pico Neo 3 Pro Eye. The VR experience consists

of two phases: a familiarization phase and an assessment phase.

The familiarization phase does not contribute to a

final test score but is essential for instructing users on navigation and

controller use and identifying any VR-induced side effects like motion

sickness. Users are seated on a swivel chair and start the experience in

a neutral environment, receiving auditory instructions and interacting

with the environment through a headset controller. The assessment

phase comprises two tasks to evaluate different cognitive functions:

• Visual Exploration Task: Users have to name as many objects they

see in a virtual living room within a minute. This is an adaption of

the free visual exploration task (Kaufmann et al., 2020). The system

11 uses speech recognition and eye-tracking technology to map visual

exploration and speech, generating a heatmap of visual activity (see

Figure 1).

Figure 1: The two screenshots show the Virtual Environment captured by the 360 camera

(left) and the same image with the heatmap overlapped (right)

• Target Stimulus Selection Task: Users identify and select specific

objects (yellow cups) among distractors (green cups and glasses) in

the same virtual setting. The task’s scoring is based on Signal Detec-

tion Theory metrics .

An exploratory study was conducted with 11 healthy participants (aged

21-26). The study primarily focused on system functionality and con-

tinuous data capture. Results of System Usability Scale (SUS) and the

consistency in user-system interaction were promising, even though

the sample was not the target demographic for the application. The

heatmap and keyword recognition data demonstrated a correlation

between visual behavior and speech, highlighting the system’s perfor-

mance.

Example 2: VR for Exposure Therapy for

people for who stutter

The second example presents a system that integrates VR, biosen-

sors, and speech emotion recognition to provide objective measures

of patients’ stress and emotional state during Exposure Therapy (ET)

sessions with people who stutter (for a detailed description and re-

sults see (Vona et al., 2023)). Stuttering is a speech disorder in which

the normal rhythmic flow of speech is disrupted by frequent pauses,

blocks, hesitations, and repetitions of syllables, words, and sounds

(Walkom, 2016).

The VR application runs on MetaQuest 2 devices. The

activities are realistic simulations that recreate Q&A situations and

11 were designed together with stuttering experts using a co-design

process to offer the user a Virtual Reality Environment (VRE) to train

public speaking skills. The application contains five scenarios: bar,

party with friends, doctor’s waiting room, classroom, and job inter-

view. Users receive new topics or questions in text-based or speech-

based formats, with their answers recorded for future analysis. The

experience can be customized by therapists, including the addition or

removal of visual and audio distractors.During a session, the user can

wear a biosensor (Empatica E4) for the collection of biophysical data.

Moreover the system integrates also a web dashboard for the therapist

that allows them to start new sessions, review past ones, and monitor

the user experience in real-time (see Figure 2, Figure 3). It offers tools

for post-session analysis to infer the patient’s stress state, including

the option to download audio and biological data, re-watch sessions,

and apply algorithms for data analysis. These algorithms exploit ma-

chine learning to:

• Identify the patient’s state of arousal.

• Recognize emotions from the tone of voice.

Figure 2: The screenshots in the figure show the usage of the platform during a session.

An exploratory study was conducted with 5 male participants who

stuttered. Analysis of the results showed a correlation between detect-

ed states of arousal and stuttering events. The job interview scenario

was the most stressful, triggering discomfort in users. The speech

emotion recognizer identified fear, disgust, and sadness as the most

dominant emotions in every scenario, indicating the potential of VR in

exposure therapy. Analysis of speech emotion recognition showed also

how unique and subjective is each patient’s experience, emphasizing

the importance of personalized intervention.

Figure 3: The screenshots in the figure show the usage of the platform after a session.

Example 3: VR for Exposure Therapy for

People with Dog Phobia

The third example presents a VR system for the practice of Exposure

Therapy (VRET) against Dog Phobia (Hallman, 2023). Specific phobias

(such as dog phobia) are strong fear reactions to specific objects or

situations, which would be perceived as inappropriate or illogical to

an external person. Thanks to VR the user can practice in a controlled

environment where they can face their fear by adjusting the level of

exposure to it.

The VRET consists of two parts. First, the participant

has to undergo a psychological education (PE part), where they will

hold a dialog with a virtual therapist to prepare them for the exposure

to the dog. Secondly, the participants will find themself in a scene

where they will be confronted with a dog (ES-part). The therapist can

monitor the patient during the experience thanks to a web platform

connected to the VR headset

In the PE scene, the participant is placed inside a

therapist’s room. The exposure to the dog happens on the sidewalk

of a street in a provincial town (see Figure 4). The environment has

been chosen, because with the therapy the patient should be pre-

pared for confrontations with dogs in everyday life. The environment

has not been detailed because the focus of the scene lies on the dog.

Figure 4: The screenshots show two examples of expsoure scene with the dog in two diffe-

remt positions and behaviors (left: closer and smelling ; right: far and pointing something)

The behavior of the dog should change depending ondifferent expo-

sure scenarios. The dog for example sits, lays, sleeps, walks around,

looks around, snuffles, or shakes itself. Within a scenario, the distance

between the dog and the user also changes. For the visualization of

the therapist, four virtual characters were selected, which differ from

each other in their style and in the creature they are representing (see

Figure 5). First, a realistic human has been selected, which visualizes

an adult woman. The second character is a humanoid woman, who is

visualized in cartoon style. The third character is a cartoon-like cat

and the fourth character is a cartoon-like robot. Additionally, for the

first three characters, lip synchronization has been implemented.

Figure 5: The Four selected Virtual Characters for the Study (Adult woman, Humanoid

Woman, Cartoon-like cat, Cartoon-like robot)

A study with 40 participants was conducted in order to validate the

user experience and the degree of the VRET.The UEQ-S showed good

results regarding the pragmatical, hedonic, and overall quality of the

application. The results of the cybersickness item indicate that the

application triggers no to only a little cybersickness in the users. This

may be the result of the user not moving within the application, which

is one of the main causes of cybersickness in the use of virtual reality

(Dörner et al., 2019).

11 Summary

This abstract delves into the application of immersive technologies,

notably Virtual Reality (VR) and Augmented Reality (AR), within elderly

healthcare. It illustrates this through the following examples:

Assessment of Mild Cognitive Impairment (MCI):

This study pioneers VR for MCI assessment in elderly people, en-

hancing screening test validity. Using Pico devices and 360-degree

environments, the VR experience engages users in naming objects

and identifying stimuli. Early results from an exploratory study show

promising usability and functionality, highlighting VR’s potential for

ecologically valid cognitive assessments.

Exposure therapy for People who Stutter:

VR, biosensors, and emotion recognition are combined for exposure

therapy in stuttering. Realistic VR simulations, customizable by ther-

apists, aim to improve public speaking skills. Biosensors capture data

during sessions, which machine learning analyzes to identify arousal

states and emotions from the tone of voice. A small study reveals cor-

relations between detected arousal states and stuttering, emphasizing

VR’s potential in exposure therapy.

Exposure therapy for dog phobia:

A VR system has been developed for controlled exposure therapy

against dog phobia. The treatment includes psychological education

and exposure scenarios with varied dog behaviors. Virtual therapists

and characters guide participants, with a study confirming positive

user experiences and minimal cybersickness. Results validate VR’s effi-

cacy in providing controlled exposure for specific phobias.

Integrating immersive technologies in elderly healthcare demonstrates

diverse applications, spanning cognitive assessment to exposure thera-

py. Future advancements promise more personalized interventions, fos-

tering improved well-being and cognitive function among older adults.

Persistent exploration of immersive technology in therapeutic contexts

is poised to enhance interventions for various health conditions.

11 References

Azuma, R.T., 1997. A Survey of Augmented Reality. Presence

Teleoperators Virtual Environ. 6, 355–385. https://doi.org/10.1162/

pres.1997.6.4.355

Chun, C.T., Seward, K., Patterson, A., Melton, A., MacDonald-Wicks,

L., 2021. Evaluation of Available Cognitive Tools Used to Measure Mild

Cognitive Decline: A Scoping Review. Nutrients 13, 3974. https://doi.

org/10.3390/nu13113974

Dörner, R., Broll, W., Grimm, P., Jung, B. (Eds.), 2019. Virtual und Aug-

mented Reality (VR/AR): Grundlagen und Methoden der Virtuellen und

Augmentierten Realität. Springer Berlin Heidelberg, Berlin, Heidelberg.

https://doi.org/10.1007/978-3-662-58861-1

Hallman, M., 2023. Influence of different virtual character types on the

user experience within virtual exposure therapy. Hochschule Hamm-

Lippstadt.

Jerald, J., 2016. The VR book: human-centered design for virtual reality,

ACM books. Association for computing machinery Morgan & Claypool

publishers, New York] [San Rafael, California.

Kaufmann, B.C., Cazzoli, D., Pflugshaupt, T., Bohlhalter, S., Van-

bellingen, T., Müri, R.M., Nef, T., Nyffeler, T., 2020. Eyetracking during

free visual exploration detects neglect more reliably than paper-pencil

tests. Cortex 129, 223–235. https://doi.org/10.1016/j.cortex.2020.04.021

Milgram, P., Takemura, H., Utsumi, A., Kishino, F., 1995. Augmented

reality: a class of displays on the reality-virtuality continuum, in: Das, H.

(Ed.), . Presented at the Photonics for Industrial Applications, Boston,

MA, pp. 282–292. https://doi.org/10.1117/12.197321

Morina, N., Kampmann, I., Emmelkamp, P., Barbui, C., Hoppen, T.H.,

2023. Meta-analysis of virtual reality exposure therapy for social

anxiety disorder. Psychol. Med. 53, 2176–2178. https://doi.org/10.1017/

S0033291721001690

Vona, F., Gianotti, M., Patti, A., Barbieri, J., Moskwa, L., Di Gioia, F.R.,

Garzotto, F., 2022a. Combining Virtual and Augmented Reality to

improve daily autonomy for people with Autism Spectrum Disorder, in:

Proceedings of the 2022 International Conference on Advanced Visual

Interfaces. Presented at the AVI 2022: International Conference on

Advanced Visual Interfaces, ACM, Frascati, Rome Italy, pp. 1–3.

https://doi.org/10.1145/3531073.3534499

Vona, F., Pentimalli, F., Catania, F., Patti, A., Garzotto, F., 2023. Speak in

Public: An Innovative Tool for the Treatment of Stuttering through Virtual

Reality, Biosensors, and Speech Emotion Recognition, in: Extended

Abstracts of the 2023 CHI Conference on Human Factors in Computing

Systems, CHI EA ’23. Association for Computing Machinery, New York,

NY, USA. https://doi.org/10.1145/3544549.3585612

11 Vona, F., Pieri, L., Patti, A., Tafaro, S., Saccoccio, S., Garzotto, F.,

Romano, D., 2022b. Explore 360° VR to Improve the Ecological

Validity of Screening Tests on Cognitive Functions, in: Proceedings

of the 2022 International Conference on Advanced Visual Interfaces.

Presented at the AVI 2022: International Conference on Advanced

Visual Interfaces, ACM, Frascati, Rome Italy, pp. 1–5. https://doi.

org/10.1145/3531073.3531171

Vona, F., Torelli, E., Beccaluva, E., Garzotto, F., 2020. Exploring the Po-

tential of Speech-based Virtual Assistants in Mixed Reality Applications

for People with Cognitive Disabilities, in: Proceedings of the Internation-

al Conference on Advanced Visual Interfaces. Presented at the AVI ’20:

International Conference on Advanced Visual Interfaces, ACM, Salerno

Italy, pp. 1–9. https://doi.org/10.1145/3399715.3399845

Walkom, G., 2016. Virtual Reality Exposure Therapy: To Benefit Those

Who Stutter and Treat Social Anxiety, in: 2016 International Conference

on Interactive Technologies and Games (ITAG). Presented at the 2016

International Conference on Interactive Technologies and Games (iTAG),

IEEE, Notthingham, United Kingdom, pp. 36–41. https://doi.org/10.1109/

iTAG.2016.13