Virtual reality for imbalance induction and analysis of neuromuscular postural reactivity

Universidade do Minho Escola de Engenharia José Henrique Machado Pires Virtual Reality for Imbalance Induction and Analysis of Neuromuscular Postural Reactivity Universidade do Minho Escola de Engenharia José Henrique Machado Pires Virtual Reality for Imbalance Induction and Analysis of Neuromuscular Postural Reactivity Master’s Dissertation Master’s in Electronics Engineering Work supervised by Professora Doutora Cristina P. Santos Nuno Ferrete Ribeiro ii COPYRIGHT AND TERMS OF USE OF THIS WORK BY A THIRD PARTY This is academic work that can be used by third parties as long as internationally accepted rules and good practices regarding copyright and related rights are respected. Accordingly, this work may be used under the license provided below. If the user needs permission to make use of the work under conditions not provided for in the indicated licensing, they should contact the author through the RepositóriUM of Universidade do Minho. License granted to the users of this work Creative Commons Atribuição-NãoComercial-CompartilhaIgual 4.0 Internacional CC BY-NC-SA 4.0 https://creativecommons.org/licenses/by-nc-sa/4.0/deed.pt iii STATEMENT OF INTEGRITY I hereby declare having conducted this academic work with integrity. I confirm that I have not used plagiarism or any form of undue use of information or falsification of results along the process leading to its elaboration. I further declare that I have fully acknowledged the Code of Ethical Conduct of the Universidade do Minho. Acknowledgements Na realização desta dissertação, que me permitiu crescer tanto a nível profissional como pessoal, contei com o apoio de várias pessoas. Quero expressar os meus profundos agradecimentos. Em primeiro lugar, quero agradecer à minha orientadora, professora Doutora Cristina P. Santos pela oportunidade de trabalhar neste projeto, pela confiança e acompanhamento, bem como por todos os conselhos. Quero agradecer especialmente ao Nuno Ribeiro, por me ter acompanhado do primeiro ao último momento neste desafio, partilhando conhecimento valioso e muitas vezes sacrificando o seu tempo pessoal em prol deste projeto. Deixo também um agradecimento aos colegas investigadores do BiRD Lab, por estarem sempre dispostos a ajudar. Não poderia deixar de agradecer à minha família, nomeadamente à minha mãe e avós que me proporcionaram todas as condições para a minha formação, sem nunca desistirem de mim. Por fim, o meu muito obrigado a todos, mesmo que injustamente não mencionados, que contribuíram direta ou indiretamente para a concretização deste trabalho. iv Abstract Virtual Reality for Imbalance Induction and Analysis of Neuromuscular Postural Reactivity The occurrence of falls in a projected continuously growing elderly population, together with its impact on mortality and reduced quality of life in the over-65 age group, turned the problem of falls a public health concern. It is estimated that worldwide 684,000 people die due to a fall, which occurs an average of 37.3 million times a year. The irregularity and diversity of a fall event results in the scarcity of available datasets that incorporate biomechanical and physiological data from actual falls. Due to this limitation, the researchers adopted a procedure that simulates falls in a controlled laboratory environment. However, these simulations are not representative of a real-world fall, given the multifactorial nature of a fall. This results in a gap involving the lack of adequate data for improving fall prediction and detection algorithms. Using virtual reality and drawing on its immersive characteristics, it was possible to include the concepts of place illusion and plausibility in the virtual environment. The participant can enjoy the sensation of being in the virtual environment and behave naturally, as they would if they were experiencing the real world. To achieve this, it is necessary to design a realistic virtual environment that simulate the everyday home environment. In this virtual environment, a protocol based on visual perturbations has been designed to recreate animations to induce different types of falls. Participants in the experimental protocol will be instrumented with inertial sensors that collect full-body motion kinematic data, electromyography muscle activity sensors, and electrodermal activity sensors. With this, a strategy to collect data from compensatory postural reactions similar to real-world falls induced by visual perturbations only was proposed. A protocol introducing several visual perturbations per session, which allows the collection of kinematic and physiological data representative of reaction patterns to perturbation during gait was proposed. This action resulted in a dataset construction contributing towards a practical way to solve the real-world fall data scarcity. Therefore, it provides a direct contribution to the mitigation of the occurrence of falls by providing data for fall prediction and detection algorithms. Statistical analysis of postural dynamic reactions reveals that the developed balance perturbation tool is effective and approaches results obtained in previous work with mechanical perturbations. Keywords: Virtual Reality, Motion Analysis, Balance Perturbation, Real-World Falls, Fall Prevention and Prediction. v Resumo Realidade Virtual para Indução de Desequilíbrio e Análise da Reatividade Postural Neuromuscular A ocorrência de quedas numa população idosa que se projeta em contínuo crescimento, juntamente com o seu impacto na mortalidade e redução da qualidade de vida na faixa etária acima dos 65 anos, transformaram o problema das quedas numa preocupação de saúde pública. Estima-se que mundialmente 684 mil pessoas morram devido a uma queda, que ocorre em média 37.3 milhões de vezes por ano. A irregularidade e diversidade de um evento de queda resulta na escassez de datasets disponíveis que incorporem dados biomecânicos e fisiológicos de quedas reais. Devido a esta dificuldade, os investigadores adotaram um procedimento que consiste em simular quedas em ambiente controlado de laboratório. Porém, estas simulações não são representativas da queda do mundo real. Utilizando realidade virtual e recorrendo às suas características imersivas, pode incluir-se no ambiente virtual os conceitos de ilusão de local e de plausibilidade. Assim, o participante pode usufruir da sensação de estar no ambiente virtual e comportar-se de forma natural. Criou-se um ambiente virtual realista e aproximado ao ambiente doméstico do dia-a-dia. Nesse meio virtual, concebeu-se um protocolo baseado em perturbações visuais com animações que tentam induzir todos os tipos de quedas. Os participantes serão instrumentados com um conjunto de sensores inerciais que recolhem dados cinemáticos, sensores de eletromiografia e sensores de atividade eletrodérmica. Posto isto, propôe-se uma estratégia de recolha de dados de reações posturais compensatórias semelhantes a quedas do mundo real, induzidas apenas por perturbações visuais. A estratégia engloba um protocolo de introdução de perturbações visuais que permite a recolha de dados cinemáticos e fisiológicos representativos dos padrões de reação à perturbação durante a marcha. Desta ação resultou a construção de um dataset que minimiza a escassez de dados relativos a quedas do mundo real. Logo, contribui-se diretamente para a mitigação da ocorrência de quedas, fornecendo dados para algoritmos de previsão e deteção de quedas. A análise estatística das reações dinâmicas posturais revela que a ferramenta de perturbação do equilíbrio desenvolvida é eficaz e aproxima-se de resultados obtidos em trabalhos prévios com perturbações mecânicas. Palavras-chave: Realidade Virtual, Análise de Movimento, Perturbação do Equilíbrio, Quedas do Mundo Real, Prevenção e Previsão de Queda. vi Contents List of Figures x List of Tables xiv Listings xvi Acronyms xvii 1 Introduction 1 1.1 Motivation ..................................... 1 1.2 Problem statement and scope ............................ 3 1.3 Goals and research questions ............................ 4 1.3.1 Goals ................................... 4 1.3.2 Research Questions (RQs) ......................... 5 1.4 Contribution to knowledge .............................. 6 1.5 Thesis outline .................................... 7 2 Virtual Reality for Rehabilitation 9 2.1 Introduction ..................................... 9 2.2 Non-immersive Virtual Reality ............................ 16 2.2.1 Equipments ................................ 16 2.2.2 Outcome Measures ............................. 19 2.2.3 Intervention Protocol ............................ 19 2.3 Immersive Virtual Reality .............................. 20 2.3.1 Equipments ................................ 20 2.3.2 Outcome Measures ............................. 24 2.3.3 Intervention Protocol ............................ 25 2.4 Limitations and Future Directions .......................... 25 2.4.1 Limitations ................................. 25 2.4.2 Future Directions .............................. 27 vii CONTENTS 2.5 Conclusions .................................... 27 3 Balance Perturbations and Compensatory Postural Adjustments induced by Immersive Virtual Reality 31 3.1 Introduction ..................................... 31 3.2 Materials and Methods ............................... 33 3.2.1 Search Strategy .............................. 33 3.3 Results ....................................... 33 3.3.1 Search Results ............................... 33 3.3.2 Study goals ................................ 35 3.3.3 Equipments ................................ 41 3.3.4 Virtual Reality (VR) Protocols ........................ 44 3.3.5 Measures ................................. 46 3.4 Discussion ..................................... 50 3.5 Conclusions .................................... 51 4 Virtual Environment and Project Overview 53 4.1 Introduction ..................................... 53 4.2 Virtual Environment Design ............................. 54 4.3 Visual perturbations ................................. 58 4.3.1 Visual and proprioceptive mismatch ..................... 59 4.3.2 Vertigo ................................... 61 4.3.3 ML-Axis translation ............................. 65 4.3.4 AP-Axis translation ............................. 66 4.3.5 Axis rotations ................................ 67 4.3.6 Visual field oscillations ........................... 71 4.3.7 Predefined trajectories ........................... 71 4.4 Triggers and Scripts ................................. 72 4.4.1 Scripts ................................... 74 4.5 Project Overview .................................. 83 5 Materials and Methods 85 5.1 Participants and Equipment ............................. 86 5.2 Equipment and Sensors ............................... 86 5.2.1 Virtual Reality Equipment .......................... 87 5.2.2 Sensors .................................. 88 5.3 Balance Perturbation Protocol ............................ 93 viii LIST OF TABLES 21 Dunnett t-test (2-sided) result - Right Tibialis Anterior. The color gradation means a higher value for the green-tone colors and a lower value for the red ones, i.e., in green are the entries that the corresponding visual disturbance introduced the most difference in means. . . . . 117 22 Thigh muscles (Rectus Femoris and Semitendinosus) from proeminent leg. Dunnett post hoc results. ......................................... 119 23 Dunnett t-test (2-sided) result - Pelvis Gyroscope Y-axis average. The color gradation means a higher value for the green-tone colors and a lower value for the red ones, i.e., in green are the entries that the corresponding visual disturbance introduced the most difference in means, in the given variable. .................................. 121 24 ANOVA results - CoM velocity variables. ......................... 122 25 ANOVA results - acceletometry variables (pelvis and sternum). .............. 124 26 Pelvis Segment Orientation - Quaternion. ......................... 166 27 Pelvis and L5 Segments Orientation - Euler Angles. .................... 166 28 Center of Mass Position, velocity and acceleration. .................... 167 29 Tibialis Anterior (both legs) maximum value during the perturbation of the AP forward translation. ......................................... 168 30 CoM Velocity X-Axis ................................... 169 31 CoM Velocity Y-Axis ................................... 170 32 Left Tibialis Anterior average activation during disturbances ............... 171 33 Right Gastrocnemius Medial average activation ...................... 172 34 Left Gastrocnemius Medial average activation ...................... 173 35 Right Rectus Femoris average activation ......................... 174 36 Right Semitendinosus average activation ......................... 175 37 Average accelerometry value of the pelvis on the X-axis .................. 176 38 Average accelerometry value of the pelvis on the Y-axis .................. 177 xv Listings 1 Code handling the trigger entering. ......................... 77 2 Code handling the trigger exit. ............................ 77 3 30 seconds delay script. .............................. 78 4 AP perturbations script. ............................... 78 5 Excerpt of code that places the participant in the start position, which will correspond to a distinct location for each keyboard key pressed. .................. 82 xvi Acronyms AD Alzheimer’s Disease ADL Activities of Daily Living ALS Amyotrophic Lateral Sclerosis ANOVA Analysis of Variance AP Anterior-Posterior APA Anticipatory Postural Adjustment BBS Berg Balance Scale BoS Base of Support BPPV Benign Paroxysmal Positional Vertigo BW Backward CAVE Cave Automated Virtual Environment CCI Co-contraction Index CCW Counter Clockwise CMEMS Center of MicroElectroMechanical Systems CNS Central Nervous System CoG Center of Gravity CoM Center of Mass CoP Center of Pressure CP Cerebral Paresis xvii ACRONYMS CPA Compensatory Postural Adjustment CW Clockwise DHI Dizziness Handicap Inventory DoP Direction of Progression EEG Electroencephalography EKG Electrocardiography EMG Electromyography EXTO External Oblique FR Functional Reach FW Forward GMH Gastrocnemius Medial Head GSR Galvanic Skin Response GVS Galvanic Vestibular Stimulation HMD Head-Mounted Display HS Heel-Strike IDE Integrated Development Environment IMU Intertial Measurement Unit IPQ Igroup Presence Questionnaire KPIs Key Performance Indicators LCD Liquid Crystal Display LGM Left Gastrocnemius Medial Head LoS Limits of Stability LTA Left Tibialis Anterior MANOVA Multivariate Analysis of Variance xviii ACRONYMS MDS-UPDRS-III Movement Disorder Society Unified ML Medial-Lateral MoCA Montereal Cognitive Assessment MoS Margin of Stability MS Multiple Sclerosis MSSQ Motion Sickness Susceptibility Questionnaire MVC Maximum Voluntary Contraction PAR-Q Physical Activity Readiness Questionnaire PBT Perturbation-based balance training PD Parkinson’s Disease PROMIS Patient-Reported Outcomes Measurement Information System PSD Power Spectrum Density RF Rectus Femoris RGM Right Gastrocnemius Medial Head RMS Root Mean Square ROM Range of Motion RPY Roll Pitch Yaw RTA Right Tibialis Anterior SCI Spinal Cord Injury SCM Sternocleidomastoid SD Standard Deviation SL Stride Length SOT Sensory Organization Test SSQ Simulation Sickness Questionnaire xix ACRONYMS ST Semitendinosus SV Stride Velocity SW Stride Width TA Tibialis Anterior TIS Trunk Impairment Scale TO Toe-Off TUG Timed Up and Go UPDRS Unified Parkinson’s Disease Rating Scale VE Virtual Environment vHIT video Head Impulse Test VR Virtual Reality VRR Virtual Reality Rehabilitation WBB Wii Balance Board WHO World Health Organization xx Chapter 1 Introduction This dissertation presents the research carried out in the scope of the fifth year of the Integrated Master’s in Industrial Electronics and Computer Engineering during the academic year of 2020/21. This dissertation was developed at BiRD LAB (Biomedical Robotic Devices Laboratory) of the Center of MicroElectroMechanical Systems (CMEMS), a research center of the Department of Industrial Electronics (DEI) at University of Minho, Braga, Portugal. The project is divided into four phases, namely: i) virtual environment design and implementation; ii) visual perturbation protocol elaboration for multisensory data collection; iii) data processing and assembling into a dataset; and iv) statistical analysis. The dissertation focuses on the development of a protocol for introducing visual perturbations that induce loss of balance to collect kinematic and electrophysiological data on postural reactions, which are expected to approximate a real-world fall. The data collection will be compiled into an extensive dataset. 1.1 Motivation Falls and unstable balance control are among the most challenging clinical problems faced by older adults. They are a cause of substantial rates of mortality and morbidity as well as major contributors to immobility [1] and premature nursing home placement [2]. An estimated 684 000 fatal falls occur each year, making it the second leading cause of unintentional injury death, after road traffic injuries [3]. In the United States, about three-fourths of deaths due to falls occur in the 13% of the population age ≥65, indicative of primarily a geriatric syndrome. About 40% of this age group living at home will fall at least once each year, and about 1 in 40 of them will be hospitalized. Of those admitted to hospital after a fall, only about half will be alive a year later [4]. Repeated falls and instability are very common indicators of nursing home admission [2]. The problem of falls in the elderly population is a combination of a high incidence together with a 1 CHAPTER 1. INTRODUCTION high susceptibility to injury, because of a high prevalence of clinical diseases (e.g., osteoporosis) [5,6] and age-related physiological changes (e.g., slowed protective reflexes) [7] that make even a relatively mild fall particularly dangerous. In addition, recovery from fall injury is often delayed in older persons, which in turn increases risk of subsequent falls through deconditioning. Another complication is the post-fall anxiety syndrome, in which an individual down-regulates activity in a perhaps overcautious fear of falling; this in turn further contributes to deconditioning, weakness and abnormal gait and in the long run may actually increase risk of falls [8].From a financial point of view, the fall-related injuries costs are substantial. For people aged 65 years or older, the average health system cost per fall injury in the Republic of Finland and Australia are US$ 3611 and US$ 1049, respectively. Evidence from Canada suggests the implementation of effective prevention strategies with a subsequent 20% reduction in the incidence of falls could create a net savings of over US$ 120 million each year [3]. To overcome this social and economic burden, existing systems mainly focus on detecting a fall [9] with little emphasis on fall prediction and prevention [10]. Hence, there is an urgent need for developing monitoring systems that can minimize this cost and improve the quality of life for persons who suffer from falls. Fall prediction and prevention systems are of utmost importance to accomplish this task and can help reduce the financial, physical, and emotional consequences of a fall. However, fall prediction is a challenging problem due to the combination of intrinsic and extrinsic fall risk factors that contribute to a fall [11]. A lot of research is centered around developing techniques to identify normal Activities of Daily Living (ADL) either at a basic level (e.g., walking, running, cycling) or at a higher level (e.g., preparing breakfast, washing hands). These techniques are generally applied to monitor a subject’s movements, assess physical fitness, and provide feedback. Though this research is useful, scenarios may exist where detection of abnormal activities become important, challenging, and relevant. Missing out such abnormal activities can impose health and safety risks on an individual. Falling is one of the most common type of abnormal activity and the most studied [12]. Most falls are caused by a sudden loss of balance due to an unexpected slip or trip, or loss of stability during movements such as turning, bending, or rising. The occurrence of falls is infrequent and diverse. The rarity of their occurrence lead to a lack of data to train classifiers. More than one type of fall may also occur, and their unexpectedness make it difficult to detect falls in advance. Collecting fall data can be cumbersome because it may require the person to actually undergo a real fall which may be harmful and unsafe. Alternatively, artificial fall data can be collected in controlled laboratory settings. However, that may not be the true representative of actual falls [13,14]. Analyzing artificially induced fall data can be good from the perspective of understanding and developing insights into falls as an activity but it does not simplify the difficult problem of fall prediction and prevention [15]. Moreover, the classification models built with artificial falls are more likely to suffer from over-fitting and may poorly generalize on actual falls [12]. On an average, nursing home residents incur 2.6 falls per person per year [4]. If an experiment is to be set up to collect real-world falls and assuming an activity is monitored every second by a sensor, around 2 CHAPTER 1. INTRODUCTION 31.55 million normal activities per year are gathered in comparison to only 2.6 falls. The data for real falls may be collected by running long-term experiments in nursing homes or private dwelling using wearable sensors or video cameras. However, the fall data generated from such experiments will still be skewed towards normal activities and it is difficult to develop generalizable classifiers to identify falls efficiently [16]. In addition to very few or no labelled data, the diversity and types of falls further make it difficult to model them efficiently [12]. It would also be more efficient to merge information from different wearable and environmental sensors to cover extrinsic risk factors. This merger would add to the clinical value of a fall risk assessment [17]. Some studies have already shown to be efficient using Electromyography (EMG) for gait analysis [18,19]. Despite the advantages and widespread use of inertial sensors, there is a lack of research that explores the collection of biomechanical and physiological data (e.g., galvanic skin response, blood pressure, heart beat). 1.2 Problem statement and scope Due to the rarity of the natural occurrence of falls and the inherent difficulties in collecting biomechanical and physiological data in a non-obstructive and user-friendly way in community-dwelling older adults, there is a lack of public falls datasets. Developing immersive VR environments with the concepts of place illusion and plausibility illusion [20] creates a medium in which people respond with their whole body, treating what they perceive as real. A HMD is able to present scenarios endowed with these concepts. In such device, the displays are mounted close to the eyes, and head tracking ensures that the left and right images update according to the head movements of the participant concerning the underlying VE. The separated left and right images for each eye ensure stereo vision. The participant has the illusion of moving through a surrounding, three-dimensional environment that contains static and dynamic objects. Balance control is a complex skill composed of three subsystems: the proprioceptive, vestibular, and vision system, which work together to keep us aware of our surroundings and give us the ability to react to current conditions and prepare for future changes. Immersive virtual reality completely changes our visual perception of the surrounding environment, providing a loss of balance by itself [21]. Inducing balance disturbances through audiovisual stimuli to impose changes in postural control is the core action point, attempting to mimic a realistic fall. These postural reactions induced by imposing a conflict between the visual and proprioceptive systems, are going to be recorded and analyzed to cover the gap in existing data regarding real falls. These postural reactivity patterns to visual disturbances are recorded using inertial sensors as a motion tracking system, capturing biomechanical and kinematic data, and physiological sensors such as EMG and Galvanic Skin Response (GSR) to record muscle and electrodermal activity signals. After signal acquisition from multiple sensors, a feature extraction technique is applied to retrieve 3 CHAPTER 1. INTRODUCTION meaningful information. Since data gathered from sensors contain undesired information, filtering techniques are essential. After filtering the collected data, appropriate features are selected. Since analyzing a high number of features requires a large amount of memory, finding the optimal feature set can improve the system’s performance [11]. The variables collected will build a large dataset, satisfying the problem presented initially. In addition to its extension, it has the advantage of incorporating kinematic, muscular and physiological data, collected in a virtual environment endowed with ecological validity and naturalism. These points give the data collection a very strong connection with a collection in a real-world environment. Innovatively, while the occasional use of visual perturbations speeds up the gathering of data about imbalanced circumstances, their ongoing usage encourages the training of postural reactions, contributing significantly for biomechanical studies on how to deal with falls or imbalances. From the perspective of mitigating the occurrence of falls in the elderly, this work allows a future design of a balance training tool strictly based on visual perturbations. 1.3 Goals and research questions 1.3.1 Goals The project’s overall purpose is to reduce the occurrence of falls, especially in the older community. The concrete goal of this dissertation is to prove that one can collect more realistic data of postural reactions and pre-impact falls using virtual reality rather than simulated falls in a laboratory setting, and match this data to real-world falls. To be able to prove this, several step-goals were set. Goal 1. Conduct a comprehensive literature review of publications carried out with HMD that introduce both visual or physical disturbances with the intent of disrupting balance and analyze the compensatory reactions. From this survey, understand which virtual environments and virtual reality equipment have been used the most. In addition, understand which visual disturbances are most commonly used to cause imbalances, and their parameters. This review also provides account for which sensor systems are most common and their locations, the calculated metrics and outcome measures used to assess the imbalance caused. The process that led to the achievement of this initial goal is described in Chapter 3.Key Performance Indicators (KPIs): state of the art that provides information on i) frequently used visual perturbations and ii) specifications for experimental protocols with virtual reality. Goal 2. Design and build a virtual environment in Unity software. In addition, create an automatic delivery mechanism for visual disturbances, with a temporal registry in a log from the perturbations onset and end time, in order to assist in the labeling process; Establish the connection between the motion capture system and the software, in real time, to allow the representation of an avatar of the participant’s whole body. Chapter 4defines the requirements for the virtual environment design and describes the scripts used to operate the VE with a subject. Chapter 5demonstrates the motion capture system that 4 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION whether it is a promising and profitable industry. Virtual reality continues to be a dominant force in the gaming industry. Nevertheless, it is also a tremendous asset for businesses. The increasingly sophisticated and diverse uses such as employee training, remote collaboration, testing, and prototyping, combined with creative use, have sparked industry interest, even though it is a recent technology, as seen in the Figure 2 timeline. It has also aroused the interest of big-tech and large manufacturing and software development companies, who have bet and placed themselves in this market. A market research report published in May 2022 [27] states that the global virtual reality market has been valued at $11.64 billion for the year 2021. What captivates the interest in conducting research with this technology, particularly with the contribution of knowledge to the automotive industry, healthcare, and education, among others, is the projected market size growth from 16.67 billion USD in 2022 to 227.34 billion USD in 2029. The VR market exhibits a compound annual growth rate of 45.2% over the forecast period. This report includes devices and software such as Quest 2, Google Cardboard, Unity Virtual Reality Development Software, and others. This paragraph highlights the potential for short-term investment in virtual training, engineering, and maintenance and outlines potential growth in the healthcare industry. Improved healthcare provision, patient care, and medical training place this area as a trend in the use of virtual reality. Since this dissertation fits into healthcare and utilizes virtual reality technology, it is a factor that makes the investigation relevant and with possible future profitability. It is common to separate virtual reality into two categories. The concept of immersion underlies the decision between categories. Virtual reality can be categorized into two types: immersive and non-immersive virtual reality. Table 1, taken from [28], lists the immersion categories and a few examples of the associated technology. This separation does not mean that there cannot be immersion of an individual in a virtual environment just because a projection in three dimensions is used. The concept of immersion, along with that of place and plausibility illusion will be clarified to better understand the interest of more immersive virtual reality systems in a variety of sectors and contexts. To be considered an immersive experience, the system must include a set of displays and a tracking system, at least from the head position, so that the displayed visual information varies according to the user’s movement. Such systems allow the user to move freely through the virtual environment, orient his head arbitrarily, and perceive space. It gives the sense of being in a real space called place illusion and that the presented scene is happening – plausibility illusion [20]. Immersion is an objective aspect of virtual reality environments, whereas presence is a psychological, perceptual element - the “feel of being there.” So, immersion is an essential feature of VR research because it influences a user’s VR experience and affects their sense of presence [20,29]. Table 1: Immersion categories Non-Immersive Semi-Immersive Fully Immersive Viewing Mediums Computer monitor, TV screen Panoramic TV Head Mounted Display (HMD), CAVE Cost Low Medium From low (HMD) to high (CAVE) Sense of Immersion Low Medium-High High 11 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION Due to the unique immersive characteristics of virtual reality, this technology has been used in a wide range of domains, namely in the: i) training of military practices [30], ii) education [31], iii) surgeons training [32], iv) architecture, v) high-performance athlete training [33], and vi) healthcare research, including the physical, mental and social aspects [34,35]. Within the health field, motor rehabilitation is one of the areas that is advancing rapidly and has several benefits when combined with VR systems. Once again, it is important not to divide the statement about the increasing use of virtual reality in neurorehabilitation from a social and financial perspective. Neurological disorders have a high prevalence together with short and long-term impairments and disabilities. A stroke or even Alzheimer’s Disease (AD) can be fatal. Others, such as chronic headaches or seizure disorders, cause significant disabilities. Therefore, they are an emotional, financial, and social burden to the patients, their families, and their social network. Globally, in 2016, neurological disorders were the second leading cause of death (9 million deaths). The absolute number of deaths from all neurological disorders combined increased by 39% between 1990 and 2016 [36]. Neurological disease is the number one disease category causing disability globally and ranks number three in Europe. The total cost of brain disorders was estimated at €798 billion in 2010. Direct costs constitute the majority of costs (37% direct healthcare costs and 23% direct non-medical costs), whereas the remaining 40% were indirect costs associated with patients’ production losses. On average, the estimated cost per person with a brain disorder in Europe ranged between €285 for headaches and €30,000 for neuromuscular disorders. In terms of the health economy, brain disorders likely constitute the number one economic challenge for European health care now and in the future [37]. The highlighted economic challenge puts Table 2: Neurological disease cost estimation from 2004 to 2010 [36] Estimates in 2010 Estimates in 2004 Number of subjects (million) Costs per subject (€PPC, 2010) Total costs (million €PPP, 2010) Number of subjects (million) Costs per subject (€PPC, 2004) Total costs (million €PPP, 2004) Addiction 15.5 4227 65,684 9.2 6229 57,275 Anxiety disorders 61.3 1076 65,995 41.4 999 41,372 Brain tumor 0.24 21,590 5174 0.14 33,907 4586 Dementia 6.3 16,584 105,163 4.9 11,292 55,176 Epilepsy 2.6 5221 13,800 2.7 5778 15,546 Migraine 49.9 370 18,463 40.8 662 27,002 Mood disorders 33.3 3406 113,405 20.9 5066 105,666 Multiple sclerosis 0.54 26,974 14,559 0.38 23,101 8769 Parkinson’s disease 1.2 11,153 13,993 1.2 9251 10,722 Psychotic disorders 5.0 5805 29,007 3.7 9554 35,229 Stroke 1.3 21,000 26,641 1.1 19,394 21,895 Traumatic brain injury 1.2 4209 5085 0.71 4143 2937 Total 178.5 2672 476,911 127.0 3040 386,175 the market for neurorehabilitation-related products on the rise. The neurorehabilitation devices market is expected to register a Compound Annual Growth Rate of 16.4% during the forecast period of 2019 to 2025 and was valued at 793 million USD in 2018 [38]. The demand for neurorehabilitation devices is increasing due to the rising number of neurological disorder cases, the emergence of robotic rehabilitation, a surge in the geriatric population, and the effectiveness of gaming systems in neurorehabilitation. However, strict 12 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION regulatory policies and increasing requirements of skilled professionals are expected to hamper the market growth during the forecast period. The market is expected to witness profitable growth due to the rise in neurological disorder cases such as Parkinson’s Disease (PD),AD, stroke, epilepsy, and Multiple Sclerosis (MS). Furthermore, researchers are engaged in evaluating the effectiveness and the appropriateness of adopting commercial games for neurorehabilitation. In the global neurorehabilitation market, Europe held a substantial share. This can be attributed to the rising prevalence of neurological diseases, product approvals, and the presence of developed economies. Approximately 85,000 neurologists in Europe currently provide care to these patients, which corresponds to approximately 10,000 patients per neurologist. However, there are huge disparities across European countries, ranging from 2,500 patients to 46,000 patients per neurologist [39]. For this reason it is important to develop models using new technologies that make neurorehabilitation accessible. Virtual reality, as pointed out, is a technology that has been implemented in the treatment of neurological diseases. This leads to the need to investigate the possibilities of complementing existing treatments, requiring brain research funding to flatten these social gaps of treatment accessiblilty. The following paragraph identifies a sub area of neurorehabilitation that has been receiving increasing attention. There are several reasons why training with VR is of greater interest in motor learning and has better results [40–42]. The key fundamentals of motor rehabilitation that can be improved with the contribution of virtual reality are repetitive practice, feedback (proprioceptive and exteroceptive), and motivation [43, 44]. The advantage of repetitive practice in motor learning was already known. However, these advantages, which translate into changes in the cerebral cortex, do not happen just by massive practice - there must be a designated task or an objective, which must be achieved by the patient by trial and error [44], receiving feedback on their performance [45]. Motivation and dedication are necessary for a repetitive task, two typical characteristics in a virtual reality user experience [46]. Feedback in virtual environments can be provided in real-time in a very intuitive way or known right after a training block; it was shown that it results in changes in the level of cortical plasticity and subcortical cells and synaptic connections [47,48]. However, repetition alone is not enough to induce changes in the motor cortex related to motor learning. Just increasing the frequency of use of a member does not produce significant changes. It is necessary to produce skilled limb movements [49]. Humans can learn motor skills in a virtual environment and transfer that motor learning to a real-world scenario [50]. There are several ways to classify Virtual Reality Rehabilitation (VRR). An obvious one is related to the specific patient population destined. Thus, one can distinguish musculoskeletal VRR, post-stroke VRR, and cognitive VRR. Musculoskeletal (orthopedic) patients who suffered a bone or muscle/ligament injury are younger and more numerous than other patients needing rehabilitation. The cognitive patient population groups individuals with various psychological disorders range from attention-deficit/hyperactivity to eating disorders to post-traumatic stress and phobias [51,52]. Injuries, degenerative diseases, and structural defects can impair the nervous system. Some of the conditions that may benefit from neurological rehab 13 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION may include ischemic or hemorrhagic strokes, trauma such as brain injury and Spinal Cord Injury (SCI), peripheral neuropathy, degenerative disorders such as PD,MS,Amyotrophic Lateral Sclerosis (ALS) or AD. Neurorehabilitation virtual reality interventions, with the paradigm shift of neurologic care, hopefully, is a junction that can facilitate the development of applications in the treatment of neurological patients with a significant impact on patient wellbeing [53]. Neurologic diseases interventions turned away from the primitive idea of a brain injury permanent effect on function and activity and became aware of the brain’s regenerative potential, as well as dynamic brain reorganization [54,55]. Optimal brain changes and recovery can happen, requiring controlled and intensive stimulation of affected brain networks [56]. On the other hand, human factors problems are associated with VR applications [57]. While it might seem intuitive that more immersive virtual environments would be best, they can generate cybersickness [58]. Common symptoms include nausea, ocular problems such as eye strain and blurred vision, disorientation and balance disturbances, and altered eye-hand coordination. Although the common denominator of this literature review is the use of virtual reality in neurorehabilitation, there are two main sections: non-immersive and immersive virtual reality. It is important to clarify that there is frequently an inconsistency in the terminology used in published studies [59]. The term virtual reality is used when referring to any type of computerized rehabilitation, whether it is a 3-D representation of a virtual environment or 2-D monitors. Due to the lack of clarity in the use of the term VR, Figure 3presents the taxonomy of rehabilitation systems that use virtual reality, which was taken into account when separating the chapter into two key sections. The descriptions made in the non-immersive virtual reality section 2.2 and in the immersive virtual reality section 2.3 will be concordant with this taxonomy. In this figure, semi-immersive reality represents non-immersive virtual reality in this text. 14 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION Figure 3: Taxonomy of virtual reality rehabilitation systems, based on the extent to which real and virtual information is mixed, the level of immersion and the main input device. AR = augmented reality, AVR = augmented virtual reality, IVR = immersive virtual reality, SIVR = semi-immersive virtual reality. HMD = head-mounted display. Figure reproduced from [59] Each of these sections follows the same structure: i) equipment and sensors used; ii) outcome measures evaluated and iii) intervention protocols. At the end of the chapter, limitations and future directions are presented. Finally, conclusions are presented. These two final considerations encompass both categories of virtual reality. 15 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION 2.2 Non-immersive Virtual Reality Non-immersive virtual reality, in which visual stimuli are delivered using two-dimensional representations, is usually identified in the literature as exergaming (exercise + gaming), or serious games. Non-immersive VR rehabilitation programs are applied to develop four primary outcomes: motor control, balance, gait, and strength. Improving these outcomes is an expected goal in the rehabilitation of post-stroke, patients with Cerebral Paresis (CP),SCI,PD,MS, patients with impaired coordination like ataxia, or even epileptic and dyslexic patients. In addition to these conditions, VR has been proposed and used as an assistive rehabilitation technology for individuals suffering from severe burns [60] or Guillain-Barré syndrome [61]. Stroke patients receive the most treatment with non-immersive VR, perhaps because it is the disease with the highest incidence in developed countries and ranks number 5 among all causes of death [36]. Most studies in post-stroke patients who used VR technology attempted to assess the efficiency in improving upper limb functions. Lower limb functions rehabilitation, posture rehabilitation - static and dynamic balance - cognitive and motor rehabilitation, finger fine motor movements, and sensorimotor function were also carried out [42]. There is a research increase with non-immersive VR using screen-based virtual reality and off-the-shelf gaming consoles with accompanying games related to motor rehabilitation in the elderly population. It focuses on balance training, strengthening the lower limbs, promoting mobility and quality of life, with the ultimate aim of preventing falls and treating fear of falling [62]. When discussing neurorehabilitation studies whose population is the elderly, it makes more sense to think about rehabilitating the detoriating effects that aging has on balance control. Such effects may be the consequence of some pathology [63] or simply the aging of sensory systems and neuromuscular control mechanisms [64,65]. Once it was clarified what type of systems and what level of immersion is referred to when introducing non-immersive virtual reality information, the groups of patients most intervened by this technology were identified. Qualitative information gathered in the following points is provided, with the structure previously suggested. 2.2.1 Equipments In this type of rehabilitation with VR, a feature connecting all the systems is their similar construction. They consist of control units, elements projecting a VR environment, and various peripheral devices (force platforms or movement sensors). The most straightforward visual display device is a computer monitor. Using aLiquid Crystal Display (LCD) projector and a large wall screen as the monitor will enhance the sense of depth perception and thus a sense of presence [66]. These set-ups are relatively cheap and easy to use, do not require glasses or wired headsets, and allow both therapist and patient to view the same scene. Consumer-driven forces for new ways to interact with video games have led to the development of video capture (infrared and RGB optical sensors/cameras), inertial sensing devices, and pressure sensors for 16 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION Figure 4: Patients playing with the system. The system consists of: 1) a WBB; 2) a PC; 3) Video display. Figure reproduced from [72] measuring body movement. The most widely used sensors in exergame input devices include accelerometers, gyroscopes, infrared and RGB optical sensors, cameras, and pressure sensors [67]. The following describes the main consumer videogame console systems and exergames that health researchers and clinicians have used. The Nintendo Wii game console (Nintendo; Redmond, WA, USA), which Nintendo introduced in 2006, is the most commonly used technology and a well-known commercial off-the-shelf game system that uses inertial sensors [68]. The system’s popularity is primarily due to the new approach to videogame interaction enabled through the Wii Remote™ (or Wiimote; Nintendo of America Inc. WA, USA), a wireless hand-held controller that embeds a three-axis accelerometer and a singleand dual-axis gyroscopes [69]. By fusing the sensor data from the gyroscopes and the accelerometers, the Wii Remote can measure changes in direction, speed, and acceleration with a sensitivity of ±1% [70]. The remote has an additional optical sensor on the controller that measures the position of a sensor bar mounted on the television, which emits two infrared light signals. The Wii controller can measure both rapid (using inertial sensing) and slow (using the optical sensor) movements using these sensors [71]. Figure 4exemplifies a setup from a neurorehabilitation intervention. As noted, it follows a common design. Contains a control unit, a display element and a peripheral device - a force platform. Inertial sensors have also been used in wobble boards for balance training [73,74]. These boards consist of an unstable plate, which causes the user to wobble while standing on the plate, thereby controlling the game by shifting his weight. The movements are measured using a single orientation tracker (Xsens MTx Motion Tracker, Xsens Technologies, The Netherlands), consisting of three gyroscopes. The Wii Balance Board™ (WBB; Nintendo of America Inc., WA, USA) is a peripheral device (51x31 cm) that consists of four force transducers allowing calculation of the center of pressure used for game control. Exergames such as Wii Sports™ (Nintendo of America Inc., WA, USA) require players to use the Wiimote to mimic actions performed in real-life sports. Wii Fit™ (Nintendo of America Inc., WA, USA), another exercise-based game for the Nintendo Wii, makes use of the WBB. The game contains over 40 activities designed to engage the player in physical exercises that focus on maintaining the center of balance. When 17 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION a player shifts their Center of Pressure (CoP), their onscreen avatar shifts its position on screen accordingly [67]. Comparable systems are pressure mats or panels with pressure sensors [75]. Inertial and pressure sensors hold the limitation that the user is directly contacting a controller. Alternatively, camera systems allow playing games without holding or wearing input devices, such as the gesture recognition system Eye Toy developed for Sony PlayStation II (Sony Computer Entertainment, Foster City, CA, US) [76,77]. The disadvantage of this system is that the camera system does not provide the accuracy necessary for playing faster games or taking high-resolution measurements. Commercially available webcams are also used to control exergames [78]. Kinect motion-detecting camera system from X-Box360 (Microsoft Corporation, One Microsoft Way, Redmond, WA, US) captures depth and color information and generates a point cloud of colored dots. The software is able to calculate the 3D position of the dots, thereby creating a 3D image of the environment [79]. Kinect motion detection camera has also been used in neurorehabilitation [80–82]. Some studies use custom-designed exergames using GestureTek’s Interactive Rehabilitation and Exercise System (IREX®) (GestureTek Health, Toronto, Ontario, Canada) [83], the Balance Rehabilitation Unit (BRU™, Medicaa™, Montevideo, Uruguay) [84], and different pressure mats or force platforms. Figure 5: Distribution of game consoles in exergames studies until 2015. Graphic taken from [85]. Nintendo Wii has a significant interest in the neurorehabilitation research domain. The Wii Fit software and WBB have provided an increasingly attractive way of assessing and training individual balance ability, particularly in balance control studies [68,86,87]. Undoubtedly, The use of the Nintendo Wii console and Wii Fit software in balance control rehabilitation related research has been very popular and with great acceptance by the scientific community [68]. From the graph reproduced from [85] (Figure 5), the prevalence of the Nintendo Wii console is noticeable. In 2012 the HMD Oculus was launched, which may be the cause of the decrease in studies conducted with this console. 18 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION 2.2.2 Outcome Measures There are two types of outcome measures to assess the patient’s progression through the rehabilitation program. The internal measures during the game and the external measures evaluated after the game. Intra-game, the measures are objective and measured using instrumentation devices, such as force plates, which have the advantage of providing feedback to the user while playing [88]. Outside the game, external measures can be clinical assessments to assess balance or sway variability after the intervention period. Clinical balance and mobility tests like the Berg Balance Scale (BBS) [89] and Timed Up and Go (TUG) [90] are abundantly used to quantify the effect of an exergame intervention on postural control and are considered external measures [75,84,87,91]. In addition to universal clinical tests, some disease-related specific tests are made. To assess the disability in PD, most authors use the Unified Parkinson’s Disease Rating Scale (UPDRS) [92,93]. To examine trunk motor deficit of stroke patients, use the Trunk Impairment Scale (TIS) [94]. The external balance measures based on sensor data quantify sway variability and CoP displacement in ML and AP directions during quiet stance using force plates [95]. Internal outcome measures include, for example, the percentage of missed targets, and the total movement range of CoP in ML and AP directions, all measured using pressure mats [75]. The CoP controls the game in several exergame studies but these measurements are not considered internal or external outcome measures as they do not quantify balance ability but only are used to play the game [83,96]. Regarding assessment schedule, outcome measurements were typically performed at baseline and immediately after or shortly after the intervention. Some adopt follow-up measures [67]. 2.2.3 Intervention Protocol The intervention standard protocol normalizes the structure of the virtual rehabilitation intervention, namely: i) number of sessions; ii) session content; iii) targeted population and iv) evaluation sessions. According to Burdea et al. [97], therapy in the inpatient or outpatient clinic is still preferred, as it presents a more structured environment free of home distractions. However, this forces the patient to travel to the clinical site, which is sometimes problematic. Training duration must be distinguished from session duration, including baseline assessments, equipment setup, and rest periods. Typically, the structure of a VR rehabilitation session is fixed for the first week, and then sessions get progressively longer and game harder over the remaining weeks. The games and how many times patients play can be fixed or selected during each session [97]. In Viñas-Diz and Sobrido-Prieto systematic review [98] about non-immersive VR rehabilitation on post-stroke patients, most of the studies applied the same treatment intensity in all groups, with some exceptions, in which the study group received more intensive treatment i.e., a more significant number of sessions and a longer session duration. VR rehabilitation programs analysis revealed a large discrepancy in the planned volume of exercises. The number of sessions fluctuated between 12 and 40 in stroke patients, 8 and 18 in PD, and 9 to 36 in CP. The duration of a single session ranged from 20 to 90 19 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION minutes in stoke, 20 to 50 minutes in PD, and 30 to 50 minutes in CP [99]. This review [99] concludes that a gold standard for improving rehabilitation in patients with stroke or PD or CP cannot be determined because of the immense diversity of implemented VR training in the inspected investigations. Furthermore, regarding the methodological quality of experimental protocol, which assesses whether a trial is subject to systematic errors (bias), it was found by several authors, using scales such as the PEDro score [100], that most studies had several methodological weaknesses [85,99,101]. 2.3 Immersive Virtual Reality Immersive Virtual Reality aims to completely immerse the user inside the computer-generated world, giving the impression to the user that it has stepped inside the virtual world. In Pimentel and Teixeira book [102], they inquired if for a virtual world to be considered immersive, whether it is honest enough to suspend the participant’s disbelief for a while. It means that an immersive experience does not require the virtual world to be as natural as the physical one to achieve a sense of presence. Compared with VR on a traditional computer screen or tablet, immersive virtual reality provides an enhanced sense of immersion in the virtual world. Immersive headsets, or those that incorporate motion-tracked stereoscopic HMDs, and potentially motion controllers are known to induce a stronger sense of presence and potentially a sense of realism, embodiment, memory, and spatial understanding than non-immersive devices [20,103]. A perceived embodiment can influence an individual’s sense of agency, where a greater embodiment may result in increased control. A lack of embodiment may result in the sense of decreased control or distress and lead to a distortion of capabilities [104]. The above makes immersive virtual reality especially applicable in motor rehabilitation. A recent scoping review [105] identified the clinical populations targeted in physical rehabilitation using immersive VR. Patients’ groups included stroke, SCI, disorders of the vestibular system, impaired balance, Cervical Range of Motion (ROM) impairments, PD, and congenital limb deficiency. The most common use of immersive VR rehabilitation was for upper extremity movement rehabilitation, vestibular training, and gait and balance rehabilitation. Fall prevention appears to be an area where immersive VR using HMD can significantly affect. In general terms, the patients who can benefit from immersive VR are the same ones who use non-immersive VR. The difference lies in the level of presence which may have better effects on rehabilitation at the level of motor learning [104,106]. 2.3.1 Equipments Two immersive virtual reality systems are significantly associated with neurorehabilitation investigation: head-mounted displays and the CAVE system. The Cave Automated Virtual Environment, or CAVE, is a room-based, fully immersive virtual reality system. It was invented in 1992 by researchers at the University of Illinois’s Electronic Visualization Lab [107]. The side walls are rear-projection screens, whereas the floor 20 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION of VR therapy compared with standard care practices. Scholars should increase the public availability of their VR simulations, questionnaires for measuring VR outcomes, and the datasets with accompanying code that informed their central conclusions. As mentioned in the motivation section, where the serious problem of falls is exposed, the existing systems to counteract this scourge focus mainly on fall detection, with little emphasis on predicting and preventing fall events. For this reason, it is important to mention that in addition to research efforts to promote balance training, it is also important to implement fall prevention strategies that capture the multifactorial nature of falling in order to reliably estimate the risk of falling. This strategy should be built on data acquired from various scenarios surrounding fall-related events. This information has been collected through questionnaires, fall diaries, and telephone calls. This information should be augmented with data collected from various sensors to improve the reliability and accurracy of fall detection and prediction systems [10]. The unpredictable nature of a fall puts barriers in the collection of data regarding these events. In addition to occurring infrequently, it is necessary for the subject to be instrumented at the time of the fall. The goal would be to collect data regarding real-world falls and build datasets with this information to complement fall prevention strategies. However, the strategy used to circumvent these barriers is to simulate falls in a controlled environment [136]. These simulations are not representative of a real-world fall [14]. Therefore, it is necessary to collect data from several sensors that best approximate a real-world fall [12]. 2.4.2 Future Directions Future studies could explore the use of HMD-VR applications in clinical settings and home-based training and as a component of telerehabilitation. Notably, although many of the earlier studies used expensive HMD devices that are no longer commercially available, some of the most recent studies used commercially available and low-cost devices. For example, Google Cardboard (Google, Mountain View, CA) retails online for $15 USD, or the Oculus Rift, which retails online for several hundred dollars. The accessibility of these devices should help make future studies using HMD-VR with larger populations more feasible [105]. HMDVR may enable some patients to experience greater enjoyment and motivation during therapy with few risks or side effects, suggesting that HMD-VR can be a low-risk, fun adjunct to therapy. To further substantiate VR as a useful tool in rehabilitation, one reinforces that higher-quality studies with larger sample sizes are needed. 2.5 Conclusions In conclusion, immersive HMD-VR is a commercially available, relatively inexpensive tool that can be used in rehabilitation with adult patients. It can also be used appropriately in assessment and intervention for patients with various diagnoses. A meta-analysis of virtual reality rehabilitation programs carried out 27 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION by Howard [121] answers the following question: ”Are Virtual Reality Rehabilitation programs effective?”. Comparing groups of patients that completed a VRR program with control groups receiving conventional rehabilitation therapy without VR, this author support that, in most cases, VRR is effective. Analyzing all outcomes, the patients in VRR programs improved their physical abilities 0.397 standard deviations above those who did not receive any VRR training. VRR programs that develop gait abilities demonstrated the most remarkable effects. Those programs developed to improve strength showed a significant effect beyond comparison groups. Conversely, motor control VRR programs and those designed to improve balance demonstrated the slightest effects of all. To conclude this meta-analysis, Howard reveals that VRR programs are more effective than comparable rehabilitation programs. VRR programs are apt for developing strength and gait. Similarly, VRR programs were more effective than alternatives for developing motor control and balance, despite having less pronounced effects. The scientific reasons that lead to the efficiency of these programs are essentially related to the concepts of repetition or mass practice, feedback, and motivation. The neurophysiological and functional benefits of movement observation, imagery, repetitive mass practice, and imitation therapies that facilitate voluntary production of movement can be easily incorporated in VR, allowing the clinician to use sensory stimulation targeting specific brain networks like motor areas, critical for neural and functional recovery, promoting neuroplastic changes early in the recovery phase [137]. There are several critical factors to learn or relearn motor skills: quantity, duration, and the intensity of training sessions. The repetition itself does not build motor learning. The repeated practice must be linked to incremental success at some task or goal. In the typical nervous system, this is achieved by trial-and-error practice, with feedback about performance success provided by the senses (e.g., vision, proprioception). Nevertheless, to practice movements over and over, participants must be motivated. VR provides a powerful tool endowed with all these elements – the possibility of repetitive practice, feedback about performance, and motivation to endure practice. In particular, in a virtual environment, the feedback about performance can be augmented and enhanced relative to feedback that would occur in real-world motor skills practice. Real-time feedback immediately after a trial improves the learning rate [138]. Proponents of VR believe that outcomes will be enhanced following practice in VR because of making tasks more accessible, less dangerous, more customized, more fun, and easier to learn because of the salient feedback provided during practice. Some studies examined this real-world versus VR practice issue in a controlled fashion [139–141] providing some experimental evidence that motor learning in a virtual environment may be superior. There is evidence that the proprioceptive and exteroceptive feedback associated with the execution of skilled tasks induces profound cortical and subcortical changes at the cellular and synaptic levels. Several studies provide neurophysiological evidence that motor repetition alone is not enough to induce cortical correlates of motor learning. VR also allows programming to display a virtual teacher who repeatedly performs the task, enhancing ”learning by imitation”via mirror neuron inputs. VR offers the unique capability 28 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION for real-time feedback to the participant during practice in a very intuitive and interpretable form. In contrast, many potential distracters exist in real-world situations and may slow down learning. Even if it proves to be the case that VR offers no performance advantage over real-world practice, it is still a powerful new tool that can be used to test different methods of motor training, types of feedback provided, and different practice schedules for comparative effectiveness improving motor function in patients. The technology provides a convenient mechanism for manipulating these factors, setting up automatic training schedules, training, testing, and recording participants’ motor responses [142]. Howard also answered the question, ”why are VRR programs effective?”He pointed out three main mechanisms suggested as causes of success in this type of rehabilitation program’s outcomes. The first cause is attributed to increased excitement. VR has been suggested to remove monotony and boredom from conventional therapy. Interacting with VR, whether they are exploring a new world or in a family environment, patients find it fun and exciting. Adding the competitive and challenging component of an exergame can increase the excitement. It leads to an increase in motivation to complete the tasks requested by the therapist. In addition, there is increased physical fidelity because the patient performs tasks closer to reality than abstract exercises. In gait rehabilitation, patients will walk through virtual environments instead of repeating movements with the knee or heel. The last mechanism pointed out by Howard [121] as a cause of the effectiveness of VRR programs is the increase in cognitive fidelity. In order to cognitively stimulate patients while undergoing treatment, these programs try to get as close to real-life as possible, whether by asking the patient to have a conversation while performing the exercise or solving simple math problems. This component is essential for the transfer of training to the real world to be more consistent. Thus, patients are better prepared for the distractions they face outside the clinical setting. Schulteis and Rizzo [143] suggested several additional advantages for using VR in cognitive rehabilitation, including control and consistency in the delivery of stimulation, immediate feedback through different senses, ability to intervene during practice in order to provide further instruction or guidance, the opportunity for self-training and learning in a safe environment, documentation of the patient’s performance and the ability to create customized training environments at low cost. Immersive VR allows the creation of individualized environments, while in non-immersive VR, most exergames follow the one size fits all paradigm. Using an HMD based therapy, training a task can be conducted in a simulated version of the individual’s home or vocational setting. Additionally, HMD and CAVE system allows the user to receive immediate feedback from its responses, behaviors, or physiological body reactions. This supports the notion that rehabilitation using fully immersive virtual environments can improve training tasks’ ecological validity and generalizability [143]. Several advantages that immersive VR use can offer to rehabilitation are directly relevant to rehabilitation research, assessment, and treatment. Some of these assets could exist with previously developed tools (non-immersive VR). However, VR goes beyond simply automating the past paradigms through its dynamic interactive and immersive threedimensional features. One key benefit is VR’s more naturalistic or real-life environment. This advantage is 29 CHAPTER 2. VIRTUAL REALITY FOR REHABILITATION twofold. First, the experience of being immersed within a VE allows users to forget that they are in a testing situation which may subsequently allow for assessment of behaviors under more natural conditions and provide insight into individuals’ typical behavior. These characteristic features contrast with the presentation of more artificially laden test-taking behaviors that may influence results obtained in traditional testing environments. Second, with the application of more real-life VR scenarios, it can be answered the question regarding the ecological validity of current assessments and interventions. Complete control over stimulus presentation and response measurement, according to the specifications of the clinician or researcher, is another strategic advantage of using immersive VR technology. Increased generalization of learning is another potential advantage. Better generalization of learning occurs with an increased similarity between training tasks and criterion targets. 30 C h a p t e r 3 Balance Perturbations and Compensatory Postural Adjustments induced by Immersive Virtual Reality The present review aims to find studies that use immersive VR through a HMD to introduce sensory disturbances in subjects, triggering dynamic or static anticipatory and compensatory postural behaviors. The reviewed articles must report postural adjustments with spatial-temporal parameters of gait or standing, either kinematic or electrophysiological. These parameters are indicators of the effectiveness of the disturbance inducing compensatory responses to maintain balance. This effectiveness will be the target of the systematic review, to investigate the possibility of applying sensory stimuli via HMD to provoke imbalances or real-world falls. As mentioned in Chapter 2, scientific research in the field of motor rehabilitation and the study of human gait and balance has undergone an intervention of VR. However, most studies use non-immersive virtual environments, the so-called exergames, and another large part uses systems that consider themselves immersive (e.g., CAVE system). Currently, the devices that offer greater capacity for immersion and presence in the virtual environment are HMDs. Due to their increasing economic accessibility, experiences with these devices have grown, and therapies are even being implemented in rehabilitation clinics using these devices. All existing reviews that focus on virtual reality via HMD are specific to a population of patients, as to their application. In this chapter, a synthesis of all studies that introduce visual perturbations using an HMD is made. 3.1 Introduction As age increases, so does the frailty and the fall incidents worsened mobility or ADL disability [144]. There is an age-related decline in sensory systems and reduced ability to adapt to changes in their environment 31 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY to maintain balance. Degeneration can be exacerbated by vestibular impairments and neurologic diseases [145]. Such a statement only reinforces the necessity of developing and applying effective VR balance training programs that target control of posture and balance through efficaciously integration of sensory information. For fall prevention, this integration requires rapid recalibration of visual, vestibular and somatosensory information [146]. The concept of virtual reality is built on the natural combination of two words: the virtual and the real. It refers to a range of computing technologies that present artificially generated sensory information in a form that people perceive as similar to real-world objects or events [147]. Can be defined as an approach to user-computer interface that involves real-time simulation of an environment that allows for user interaction via multiple sensory channels [148]. Similarly, Schultheis and Rizzo [143] considered virtual reality to be an advanced form of human–computer interface that allows the user to interact with and become immersed in a computer-generated environment in a naturalistic fashion [143]. Regardless of its definition specificities, the richly complex multisensorial experience offers a practice environment that can be ecologically valid [149], can elicit a substantial feeling of realness and agency if the concepts of immersion, place illusion, plausibility illusion and the fusion of these last two in the notion of a virtual body are archived. As it uses visual, sensory, and auditory feedback, there is extensive evidence that the proprioceptive and exteroceptive feedback associated with the execution of skilled tasks induces profound cortical changes associated with motor learning and that humans can transfer motor learning to a real-world environment [142]. In addition, gives an opportunity to increase the duration, intensity and mass practice needed to induce neuroplasticity, as it also strongly increases the participant’s motivation [99]. For these reasons, exploring interventions that use VR may offer unique opportunities to address areas of health need. VR is evolving at a rapid rate and presents an opportunity to enhance and support older adults’ physical and cognitive issues [129]. The variety of technologies that are considered VR range from non-immersive applications to completely immersive applications. Immersive VR can be produced by combining computers, HMDs, body tracking sensors, specialized interface devices, and real-time graphics to immerse a participant in a computer-generated simulated world that changes in a natural way with head and body motion [150]. VR can be used in several ways to re-train postural control and balance. First, VR can be used to manipulate visual feedback to produce conflicts between visual, somatosensory, and vestibular information as a way to train different sensory systems. Second, VR feedback can be systematically graded (in terms of speed and complexity) to challenge a person’s static and dynamic postural control over the course of sensorimotor training [146]. Recently, VR therapies have been introduced in the field of neurorehabilitation, especially in patients who suffered from stroke or Parkinson’s disease and in children with cerebral palsy. Previous systematic reviews have evaluated VR for balance training in patients with stroke [151,152]. There are also reviews assessing the general effectiveness of VR-based rehabilitation for Parkinson’s disease patients [92] and in patients with cerebral palsy [153,154]. Perturbation-based balance training (PBT) is a balance training intervention that incorporates exposure to repeated postural perturbations to evoke rapid balance reactions, enabling the individual to improve control of these reactions 32 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY with repeated practice [155]. The above-mentioned studies follow this paradigm, adding the perturbation on the VE. The objective of this study is to determine the evidence of visual perturbations induction of compensatory postural adjustments, and to determine VR technology viability preventing fall frequency. With this purpose, VR interventions in recent studies will be analyzed with regard to equipment’s used, features extracted from measurement systems, VR protocols comprehending the perturbation type, their amplitude or frequency and study population. Finally, a discussion section is reserved for the results found to answer two research questions: RQ1 ”Can a virtual reality headset introduce imbalances through visual perturbations? Can they cause postural reactions typical of a fall?”and RQ2 ”Which visual perturbation challenged the participants’ balance the most?”. Addressing these questions will provide a foundation to understand the impact of visual perturbations delivered by HMD-VR on balance control system. 3.2 Materials and Methods 3.2.1 Search Strategy A literature search was carried out using the boolean search strategy in the IEEEXplore, SCOPUS, Web Of Science, and PubMed databases. Combinations of the following key terms were used for each database: “virtual reality” OR “virtual environment” OR “immersive” AND perturbation. No filters were applied in each database to restrict searches. The studies were chosen under the following inclusion criteria: (1) the study was conducted using a HMD, (2) sensory disturbances were delivered to the participants (3) physiological, neuromuscular, or kinematic data were reported to study participant’s reaction to perturbations. Following this inclusion criteria, a PRISMA flowchart is shown in Figure 9. 3.3 Results 3.3.1 Search Results The initial search resulted in a total collection of 1101 publications. A selection was made according to PRISMA, detailed in the flowchart of Figure 9. From this total, 386 publications were imported from Scopus, 130 from IEEE, 386 from Web of Science, and 199 from PubMed. 144 were identified as duplicates and 855 publications were excluded by title or abstract that did not meet the inclusion criteria. Of the 102 complete articles or conference papers that remained, 51 were excluded for using VR that they consider immersive but that does not use HMDs. 42 publications underwent a full reading, a process that excluded 6 articles for not evaluating the subjects’ physiological, kinetic or kinematic parameters after disturbances. In this full reading process, 2 articles were included that fulfilled the inclusion requirements, by references. Altogether, 40 articles were included in the subsequent analysis. Figure 9presents the flowchart for the 33 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY Figure 9: PRISMA flowchart literature search process. A summary of the included studies is presented in Table 4. This summary presents the following characteristics: i) study author and year of publication; ii) study goal; iii) virtual reality system used and iv) sensors. 34 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY Table 4: Summary of the main characteristics of the included studies: reference, study objective, headset and sensors used. Reference Study goal VR-HMD Sensors [156] Age-related postural reactions Kaiser Optics ProView XL5 Force-plate; Optical motion tracking [157] HTC Vive Instrumented walkway [158,159] Nvisor Optical motion tracking [160]Glasstron LDI–100B, Sony Optical motion tracking [131,161] Patients postural reactions Oculus Rift IMU [162] Samsung Gear VR Force-plate [163] Oculus Rift Force-plate [164] Oculus Rift Force-plate [165] Visual and Proprioceptive conflict Oculus Rift Optical motion tracking [166] Oculus Rift Optical motion tracking; EMG [167] PlayStation VR Optical motion tracking [168] Balance function and threatening scenarios HTC Vive EKG; EMG; EEG [169] Sensics piSight Force-plate; EMG [170] HTC Vive IMU [171] HTC Vive Force-plate [172] iWear VR920 Force-plate; EMG; Accelerometer [173] Visual Perturbation postural reactions HTC Vive Instrumented walkway [174] Google Cardboard IMU [175] HTC Vive Optical motion tracking [176] HTC Vive Optical motion tracking [177] HTC Vive Pro Force-plate [178] HTC Vive Force-plate; Optical motion tracking; EMG [179] Oculus Rift Force-plate [180] HTC Vive Instrumented walkway [181] Oculus Rift Force-plate [182] Oculus Rift Force-plate [183] Oculus Rift DK2 Optical motion tracking; EMG; EEG [184] Oculus Rift DK2 Force-plate [185] Balance Training Oculus Rift DK2 Optical motion tracking; EMG; EEG; GSR [186] Glasstron LDI–100B Sony Instrumented treadmill; Optical motion tracking; EMG [132] Glasstron LDI–100B Sony Instrumented treadmill; Optical motion tracking [130] Glasstron LDI–100B Sony Instrumented treadmill; Optical motion tracking; EMG [187] Balance Assessment Oculus Rift DK2 Force-plate [188] Samsung Gear VR Force-plate; EMG [189] Virtual Research V8 Force-plate 3.3.2 Study goals The diagram in Figure 10 shows the broader separation of the studies’ purposes. All aim to investigate compensatory reactions induced by visual perturbations. Some channel this analysis of postural adjustments to understand differences introduced by advancing age or by diseases. Others analyze these postural reactions when visual information discordant with proprioceptive information is introduced, when placing the participant in threatening situations, or to perform an objective balance assessment, or balance training. The objectives of the studies are mostly divided into two aspects: gait analysis or standing postural analysis (Figure 11). Both can study anticipatory and compensatory reactions to visual disturbances introduced by an optical flow in the VE. More specifically, 18 studies attempt to understand changes in gait patterns, 8 35 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY Figure 10: Diagram describing the separation of study goals. of which during overground walking [157–159,167,170,173,174,180] and 10 during treadmill walking [130,132,160,165,166,175,176,183,185,186] while the remaining studies analyze postural reactions in upright stance to visual perturbation [131,156,161,163,168,169,171,172,177–179,181,182,184, 187–193], either in bipedal stance or one leg stance. Whether walking or standing upright, the analysis is carried out without exception when the participant is exposed to visual or physical disturbances of different types, namely: i) continuous and multidirectional; ii) transient; iii) discrete; iv) RPY axes rotations (single or multi-axes); and v) dissonant or concordant proprioceptive and visual information. The following divisions do not distinguish between disturbed gait analysis or standing balance reactions analysis, since it is intended to present the purposes of the studies. 3.3.2.1 Discordant proprioceptive and visual feedback Frost [165] and Drolet [166] focus on the interplay of visual feedback and proprioception. There is a crucial relationship between proprioception and visual feedback. While previous studies have separately investigated proprioceptive and visual feedback in gait, they have not shown the evoked muscle responses from visual feedback alterations. Understanding and modeling this relationship could lead to patient-specific interventions for motor rehabilitation. Frost and colleagues [165] quantify the functional role of expected vs. actual proprioceptive feedback for planning and regulation of gait measuring contralateral leg kinematics whereas Drolet and colleagues [166], through the manipulated visual and proprioceptive feedback, focus on the muscle activation patterns before, during, and after surface changes that are both visually informed 36 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY be activated during toes-up and toes-down tilt, respectively, at a latency of 80-100 ms. Vastus Lateralis and Semitendinosus were activated 110130 ms, followed by Tensor Fascia Latae and Erector Spinae approximately 30-50 ms later, while the neck muscles Sternocleidomastoideus and neck extensor were activated at 150-170 ms. Peterson and Ferris [183] introduce discrete visual perturbations on the roll, in both directions of rotation. In addition to using EEG, they recorded 8 lower-leg EMG channels. Four muscles on each leg: tibialis anterior, soleus, medial gastrocnemius, and peroneus longus. During walking, the significant intermuscular connectivity increases occurred primarily between peroneus longus and soleus muscles on either leg. Parijat et al. [130] with an eight-channel EMG telemetry Myosystem 2000 (Noraxon, USA), was used to record bilateral temporal activations from vastus lateralis, medial hamstring, tibialis anterior, and medial gastrocnemius muscles of the lower extremity. To quantify the reactive responses to the virtual slip, and to quantify training effects, the coactivity index was calculated based on the EMG activity ratio of the antagonist/agonist muscle pairs (TA/MG and VL/MH). Proactive responses were found by muscle onset of MG, TA, MH and VL activation of the slipping limb. During VR training of slips, the proactive and reactive strategies in terms of muscles, showed early activation of all slipping limb muscles by the fifth trial. In the subsequent trials, early onsets were only detected for the VL and TA. Likewise, Liu et al. in 2015 [186] use the same type of EMG sensors and the same locations. Drolet et al. in 2020 [166], through the manipulated visual and proprioceptive feedback, focuses on the muscle activation patterns before, during, and after surface changes (through a variable stiffness treadmill) that are both visually informed and uninformed. The muscle activity of both legs was obtained using surface electromyography via a wireless surface EMG system (Delsys, Trigno Wireless EMG). Electrodes were placed on the tibialis anterior, gastrocnemius and soleus muscles. These muscles were selected as they play a primary role in ankle motion and stability, in which the gastrocnemius and soleus muscles produce plantar flexion of the foot and the tibialis anterior produces dorsiflexion of the foot [202]. The results of this study provide strong evidence that anticipation of walking surface compliance changes affects both muscle activation and kinematics of the leg preparing to land on the surface of different compliance. That accelerated swing phase is also supported by increased activity on the dorsi flexor muscles (tibialis anterior) during the swing phase before the landing of the foot. Moreover, the false expectation of transitioning to the new compliant surface, i.e., being prepared to step on a compliant surface but eventually stepping on a rigid one, elicits delayed responses on the plantar flexor muscles (Soleus and Gastrocnemius). Finally, Ida et al. in 2017 [172] records surface electromyography signals using disposable surface electrodes (Red Dot, 3M, USA). The electrodes were bilaterally attached to the bellies of the tibialis anterior, medial gastrocnemius, rectus femoris, biceps femoris, external oblique, rectus abdominis, and erector spinae muscles. These leg and trunk muscles are known to be involved in the control of upright stance when dealing with symmetrical and asymmetrical perturbations [203,204]. The results supported that the obstacle avoidance task involves the activation of postural muscles in the early preparatory phase. Furthermore, the results on EMG integrals also supported that a VR setting modulates the components of postural adjustments during the 43 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY avoidance action. The findings suggest that VR potentially deteriorates natural perceptual motor reaction and postural control strategy, which may relate to a participant’s sense of presence in computer-simulated environment. 3.3.4 VR Protocols 3.3.4.1 Visual and Physical perturbations Given the restriction of the use of VR through an HMD, the most common type of disturbance induced in the subjects was visual disturbance, also called optical flow [205]. It induces modulations in gait patterns and in postural control performance [206]. Can be stabilizing when appropriate or destabilizing when conflicting. During locomotion, there is a decreased stabilization pattern due to artificial changes in the optical flow [207,208]. In contrast to mechanical disturbances, these visual disturbances do not impose inertial effects that move the position of the body in relation to the support surface. On the other hand, they create the illusion that a correction is necessary. This perception of the need for correction makes the responses to visual disturbances highly idiosyncratic, while the responses to mechanical disturbances are very consistent. This variability from subject to subject makes the introduction of visual disturbances a proper manner to assess the risk of falling, since the reaction reveals the dependence on visual feedback. Visual processing is a critical component of balance and gait stability [209]. For this reason, all studies use visual disturbances that consist of a change in the HMD display. Some studies combine optical flow with a physical disturbance [130,156,164–166,168,169,172,176,182–184,186–189]. Depending on the objective, these disturbances can be concordant or dissonant, i.e., to simulate a fall as in the real world, the subject has to visualize a rotation around the roll direction contrary to the rotation that occurs on the support surface. In most articles, visual disturbances consist of rotations or tilts on the Roll, Pitch and Yaw axes, translations on the vertical axis, and in ML and AP directions, or overlapping oscillations in the visual field, in the AP or ML axis directions. The use of moving objects in the virtual environment can also be considered a visual disturbance since it alters the sensory feedback information. For example, in Lubetzky and colleagues studies [164,184], animated objects with variable speeds were used to trigger participant’s postural changes, or virtual blocks that intend to resemble the movement of passers-by on a street. Place the participant virtually at heights can also be considered a visual disturbance because this threatening condition alters balance in a static position or during gait, increasing the weight of proprioceptive and vestibular reflexes [210]. Finally, two studies [131,161] also used a visual field perturbation to simulate a fall on stairs: instantaneous translations on the vertical axis of the VE, which transported the subject from the top of a staircase to some steps below. Some authors have introduced physical disturbances in conjunction with visual disturbances to investigate the interconnection between different sensory systems. More specifically, 2 studies [165,166] have changed the stiffness of the gait support surface, in accordance with the visual feedback or not. Another 44 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY study used a tilting platform to study the same interconnection of sensory systems [156]. To complement the threatening situation of high heights in the virtual environment, some researchers [168,169] used platforms to create a bi-directional translational AP movement or a toe-down pitch rotation to increase anxiety. In a control group, mechanical disturbances in the form of pulls to one side in the direction were applied to compare compensatory responses with similar visual disturbances [183]. The use of compliant surfaces was also found in order to hinder the process of maintaining balance, namely stability trainers and a BOSU balance trainer [164,182,184,187,188]. To induce slips, Parijat and colleagues used a slippery floor surface that moved after heel contact to make baseline measurements and to quantify training transfer effects [130,132]. 3.3.4.2 Virtual environments Most of the VE present in these studies were created with VR game engine software, mainly Unity and Unreal Engine. The latest headsets are equipped with inertial sensors that track the position and rotation of the head. Without exception, in real time, they alter the VE accordingly with the movement of the head, with a high refresh rate to avoid cyber sickness. The headsets that did not contain this functionality were equipped with an inertial sensor or used motion tracking systems information to synchronize the movements with the VE. Most of the VEs created by the researchers are realistic enough to make the experience immersive and create a sense of presence. Three of the oldest virtual environments have monochromatic representations: two simulate a simple corridor and the other only shows moving patterns. Interestingly, 2 studies [183,185] capture video in real time from a webcam mounted on the HMD and works as a VE. As a general rule, the VEs did not use the subject’s own representation in the form of an avatar. In 2 studies [169,170] focused on the relationship between high heights and balance, studied the influence of the presence of an avatar that represented the movement of the legs or hands, more precisely while performing cognitively challenging tasks. When the studies were carried out on treadmills, the VE altered at the same speed as the subject’s gait pace, whether self-chosen or imposed by the experimental protocol. Otherwise, the speed of ground walking determines the change in the VE representation. This synchronization is significant. If three-dimensional objects are presented in a dynamic, consistent, and precise manner, the VE is an ecologically valid platform for presenting dynamic stimuli in a manner that allows for both the veridical control of laboratory measures and the verisimilitude of naturalistic observation of real-life situations [112]. 3.3.4.3 Protocol specifications The diversity in the intervention duration or the exposure time to VR scenarios has already been pointed out as a limitation in several literature reviews that investigated the use of immersive VR for rehabilitation [99, 101,105]. In the studies included in this work, the same trend can be seen. In addition, as the objectives 45 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY are mainly to study compensatory adjustments triggered by visual perturbations, many of the authors do not even disclose the duration of the intervention. Some reveal the duration of exposure to the VE and whether there are pauses between trials to minimize the effects of exhaustion or lack of concentration. It is common to find pre-test periods to get the participant used to the HMD and the VE thus preventing a mask on the effects of disturbances and to assess whether there is simulation sickness. This pre-test period also increases the sense of presence. Habituation time was found in several studies, mainly in those that involved walking or interacting with the VE. In slips induction and balance training studies, follow-up measures were taken in order to quantify the transfer of skills acquired in training. These measurements were made on different days, in contrast to the majority, which collected the data in just one session, on the same day. 3.3.4.4 Population Regarding the studies populations, 22 of them used only healthy young adults for the experiments, while 5 studies used healthy older adult participants only. In 4 studies, healthy young people were assigned to the control group, while older adults participated in tests with VR. Only one conference paper does not specify the type of subjects, nor their demographic characteristics. The scarcity of control groups is evident, being common practice only in studies with pathological subjects. From this pathological population, 4 studies were found with subjects diagnosed with vestibular impairments. One of these studies with patients with vestibular loss was the only one in which the participants were appointed by their neurologists to be part of this rehabilitation and experimental assessment. Other pathological subjects with AD,PD, Mènière disease and glaucoma were subjected to trials with VR. 3.3.5 Measures The outcomes assessed in the studies are presented. The section arrangement is not chosen according to the objective of the study. However, it should be noted that the text is organized in order to place the measures from the articles that share common goals. The measurements identified are sorted by the evaluation mode of balance: static or dynamic. These metrics can be directly measured, calculated, or estimated. They can originate from a force plate or a motion tracking system, and through EMG,EEG or EKG systems. In addition, this section also presents traditional clinical tests performed and questionnaires. 3.3.5.1 Static balance measures As mentioned in the taxonomy of the studies, there is a separation regarding the way the experience is carried out: standing or walking. This subsection presents the measures performed by the studies when evaluating the balance of the subjects when standing. In the group of studies that relate the anxiety created by threatening scenarios, namely heights, in one study [168] is carried out a Sensory Organization 46 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY Test (SOT) [211] through a force plate that objectively measures the subject’s AP and ML body sway. In addition, by placing an EKG on the chest, anxiety on stance was quantified, detecting heart rate variability. Co-contraction Index (CCI) of the ankle dorsiflexion is used in this study to measure subject’s dynamic anxiety during the trials, as it has been shown in previous studies that, with the increase in fall related anxiety, co-contraction of the ankle joint agonist/antagonist muscle pairs intensifies [212]. Through the 16 front electrodes of the EEG, the Power Spectrum Density (PSD) was calculated for each of the five frequency bands, which served as a feature to classify, through supervised learning classifiers, the balance function states. In a second study [169], with a motorized platform that rises virtually to a height, postural reactions to platform accelerations in AP directions were quantified by displacements and CoP speeds, namely peak and time to peak. With the IRED markers, angular displacements in the AP and ML directions were calculated to quantify the relative movement of the arms in relation to the trunk. Here, using EMG, the CCI was also calculated for four muscle pairs in the lower leg, upper leg and trunk. The amplitudes of the muscle signals were measured by the integrals. To characterize the obstacles negotiation [172], mechanical quantities were measured based on coordinates of the rigid body such as the maximum elevation of the toes, measured vertically. Using an accelerometer, it was possible to measure the accelerations of the foot on the various coordinated axes, while overcoming an obstacle. In conjunction with a force plate, it was possible to estimate CoP displacements while EMG data was collected, for example the onset muscle activation and integrated muscle activity. Lubetzky et al. [182], with vestibular dysfunction patients and age-matched controls to quantify sensory entrainment with a force platform embedded in the floor, measured the postural sway in the AP direction. The Power Spectrum of the CoP time series was calculated at four different frequencies, each corresponding to a perturbed system. The gain and phase (modulus and argument) of the frequency response function were also calculated to, respectively, indicate the ratio of the amplitude of the response to the amplitude of the stimulus, and to express the relative timing of the response compared to the stimulus. Studies were found [187–189] that aim to validate alternative objective assessment systems to replace traditional clinical assessments of balance or dizziness, more specifically in the diagnosis of individuals with Mènière compared to the Romberg Test [213,214]; in assessing vestibular impairments compared to the Dizziness Handicap Inventory (DHI) [215]; or in the assessment of balance through objective measures to the detriment of the gold standard Equitest [216]. In the first study mentioned [189], this alternative system uses a force platform built with three force transducers arranged in an isosceles triangle to measure the vertical ground forces and the displacements in the AP and ML directions, CoP and its stabilogram - which calculates the respective ML and AP range, standard deviation and mean velocity as a function of time [217]. With these measurements, the Romberg Quotient is calculated, which is the ratio between the mean velocity with eyes closed and the mean velocity with eyes open. A feature that derives from these measurements is the Vertical ground reaction Force Power Fraction which detects slow changes and drifting, such as hanging on the safety harness. In the study that follows [188], it is intended to assess 47 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY the effectiveness of objectively measuring the parameters evaluated by DHI through a WBB, in patients suffering from vestibular impairments. The metrics calculated here are the postural sway, the ocular speed and the vestibulo-ocular reflex gain. With EMG, cervical and ocular vestibular evoked myogenic potentials are measured. Finally, in a study conducted by the same author [187], the effectiveness of the gold standard balance test Equitest is compared with the measurements made also with a WBB, namely the length path and speed of the Center of Gravity (CoG) of the body estimated from values obtained by the CoP. Another outcome of the assessment is the presence or absence of a fall during the test, and the time elapsed until the fall. In a research [179] to understand the extent to which visual disturbances induce cyber sickness in a VE, using a WBB, the postural sway was measured in the AP and ML directions, a time series was performed, and standard deviations were calculated for all participants. When determining the impact of an expected or non-expected visual disturbance [177], introduced by VR,CoP excursions were measured using a force plate, i.e., the average displacements of the CoP sway and its speed, which allowed an estimate of 95% ellipsoid area, which represents an area where changes in the CoP are such that 95% of the data is within that ellipse, and 5% outside. In an experiment to examine the contribution of vision to balance [178], laser range finders for the hip were used to calculate the Root Mean Square (RMS) of body angles to quantify body motion variability. The frequency response was analyzed with a discrete Fourier transform to determine the transfer and coherence function. These functions describe the dynamics of the body’s response to the visual stimulus. The gain of the transfer function shows the sensitivity response as a function of frequency; the transfer function phase expresses the relative timing of the response compared with the stimulus at each frequency. Bugnariu and Fung [156] analyzed the selection of strategies in the regulation of upright balance related to aging. With two force platforms, they measured the CoM displacement, the resulting force of the CoP in the horizontal plane and the moments in the AP and ML directions. From the EMG signals, the muscle latencies were registered. In another work [162], aiming to understand: i) the effect of age on balance performance; and ii) the effect of complete vestibular loss on balance performance using VR, using a WBB, instead of considering the 3-dimensional trajectory of the CoM, considering one coordinate of the projection on a horizontal plane of the CoP, determined the so-called statokinesigram. In addition to this calculation, they determined the body’s CoG trajectory as a function of time, normal limits and Limits of Stability (LoS). Santoso and Phillips [181], to investigate the optical flow effect on a subject’s postural control, measured with a force plate the ground reaction forces and moments that allowed the calculation of CoP in the AP direction, and estimated the CoP distance traveled. Similarly, most studies that attempt to understand what influence induced visual perturbations have on postural dynamics, using a force plate, it is measured the postural sway in the AP and ML directions using the CoP. Less frequently, these parameters were also calculated using motion capture systems, particularly when the objective was to identify head or trunk deviations [183,185]. 48 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY 3.3.5.2 Dynamic balance measures When the study requires participants to walk, whether on a treadmill or overground, gait parameters are measured, mainly using motion capture systems, whether they are infrared optical tracking systems or, less frequently, non-optical motion tracking systems e.g., IMUs. The most present parameters in the studies that classify gait stability were joint angles position and speeds, feet positions that allow estimating the HeelStrike (HS) and Toe-Off (TO); spatio-temporal gait parameters such as Stride Length (SL),Stride Width (SW), Stride Time (ST), Stride Velocity (SV),Direction of Progression (DoP) (the angle of the stride vector from two successive heel strikes of the same foot) and DoP deviation. From these parameters their variability, means and standard deviations were calculated, which are representative of instability in human gait [218, 219]. The CoM can also be calculated for each gait cycle, which allows estimating the Margin of Stability (MoS). Using an inertial sensor, it was also possible to measure the acceleration signals in all axes, which allows the calculation of CoM excursions. Body orientation and body displacement were also estimated, and temporal and frequency analyzes were used to monitor sudden changes due to postural adjustments [131,161]. The most considered EMG signals during locomotion were the activation times of the muscles and their integrals, to distinguish between anticipatory or compensatory muscle activations. 3.3.5.3 Traditional Clinical Tests and Questionnaires Questionnaires or traditional clinical tests were carried out in various circumstances. To assess the subject’s eligibility in the study, the Physical Activity Readiness Questionnaire (PAR-Q) and the SSQ were used. Before and after the tests, questionnaires were performed, such as weekly exercise minutes, sitting hours per day, the DHI or the Motion Sickness Susceptibility Questionnaire (MSSQ), the Patient-Reported Outcomes Measurement Information System (PROMIS): a set of person-centered measures that evaluates and monitors physical, mental, and social health in adults and children; confidence questionnaires to assess perceived anxiety, Witmer & Singer Presence Questionnaire and Slater-Usoh-Steed Questionnaire for VR presence and the Montereal Cognitive Assessment (MoCA). Other tests that were done before and after the trials were the Functional Balance and Mobility test, the Romberg test, the ability to maintain tandem stance, Time repeated Sit-to-stand, and again the MSSQ. Visual acuity tests (Snell’s chart) and cognitive deficits Mini-Mental State Examination were also conducted. In the case of patients with glaucoma, they underwent ophthalmological examination including review of medical history, visual acuity, slit-lamp bio microscopy, intraocular pressure, gonioscopy, ophthalmoscopic examination, stereoscopic optic disc photography, and Standard Automated Perimetry; history of falls with Falls Screening and Referral Algorithm. For vestibular patients, Activities-specific Balance Confidence Scale and again DHI. In the PD study, idiopathic PD patients had normal clinical postural stability measured by the retropulsion test (item 12 on Movement Disorder Society Unified (MDS-UPDRS-III)) and had a Hoehn & Yahr of 2 (OFF state). Clinical data were also collected, including years of disease duration, MDS-UPDRS-III (scored as either an in 49 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY the OFF or the ON state). A brief neuropsychological examination was performed using the Portuguese version of the MoCA with scores normalized to the Portuguese population. Additionally, self-assessment of fear and acrophobia questionnaires were seen - James Geer Fear Questionnaire, Cohen’s Acrophobia Questionnaire and International Physical Activity Questionnaire. 3.4 Discussion There has been a growing interest in balance control mechanisms investigation using VR. More specifically, virtual reality using HMDs, replacing the usual three-dimensional projections, LCD screens, and robust and difficult to access systems such as CAREN and CAVE. HMDs provide an effective means of introducing visual disturbances due to the technical qualities that allow the user to feel a high level of presence and immersion. In addition, it requires low physical space, not much expertise from researchers and a decrease in the price of these devices was observed, which makes them very attractive for this type of studies discussed in this review. As seen, these investigations focus mainly on the balance system of the human body. Postural balance is studied either during standing or gait. It would be interesting to combine the two types of static and dynamical analysis, approaching studies closer to the real conditions that a subject is confronted with on a daily basis. One can either suffer a threat to balance during walking or standing. Additionally, as mentioned in studies whose objective was to analyze anxiety and its relationship with balance, the threatening situations shown in the VE could be extended to other threatening scenarios, other than just the heights. By identifying situations and environments in which falls normally occur, these can be replicated to the VE, increasing the spectrum of physiological, neuronal, and muscular reactions beyond the fear of heights. The level of complexity and detail of the VEs created by the researchers falls short of the computational resources and tools that are currently available, which allow the creation of photorealistic scenarios. Therefore, the level of presence and immersion would substantially increase, which can trigger more realistic reactions. In addition, since most current HMDs have built-in headphones, it would increase the experience of VR if auditory stimuli were introduced along with visual stimuli. Not only to enrich the experience but also to become another stimulus that defies balance. None of the 40 studies seen added audio to their VEs. Finally, another sensory conflict that could be explored would be the haptics to simulate touching objects and collecting data from the reaction to intentional or unintentional touch. With regard to the subjects chosen for the studies, the small sample size, the lack of control groups and non-random allocation, and the failure of follow-up measures are repeatedly mentioned. This leads to the fact that these studies lack methodological quality, which can introduce bias in the results. Even so, the vast majority claim clear evidence on the effectiveness of the introduction of optical flow and the induction of compensatory behaviors, as well as on the effectiveness and reproducibility of balance assessment systems using VR. The study of these behaviors could be extended to a wide range of neurological patients as done for patients with PD,AD, and those with problems in the vestibular system, and also to those with 50 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY locomotion issues related to cardiovascular accidents or acquired brain injuries. Even more importantly, it would be interesting to test these systems in the elderly, the age group in the most fragile situation and the most exposed to falls that can cause serious damage. That said, the addition of physiological sensors (EKG and GSR) would bring additional validity to the results: a neurological (EEG) and muscular (EMG) with motion tracking systems sensor fusion. Thus, the tendency to study only the kinematics of anticipatory or compensatory adjustments would decay, and knowledge about neuromuscular changes would increase. Another interesting point in addition to the auditory stimuli inclusion, as pointed out, would be the introduction of electrical stimulation of the vestibular system with Galvanic Vestibular Stimulation (GVS), responsible for creating the illusion of movement in the head and for creating balance losses. GVS systems are simple, economical, and safe. They can contribute to an easy and controlled induction of imbalance, which can be advantageous when used in conjunction with disturbances of the visual field. Experimental protocols with VR must go through a standardization process mainly to define gold standards in the period of habituation to the device and the VE, define the duration of the exposure to avoid creating habituation to disturbances and thus mask some behaviors, define the type of disturbance or more efficient combination of disturbances to evoke imbalances, setting breaks during the trial to avoid exhaustion and cyber sickness and, finally, adopt follow-up measures in a post-intervention period to validate the reproducibility of the created system and evaluate its effectiveness. Due to space restrictions, many studies use treadmills instead of asking subjects to walk naturally on the floor. As discussed previously, the treadmill itself influence some important gait parameters. Therefore, it is advisable and more naturalistic to analyze the overground gait. Finally, the calculated metrics could be generalized throughout the studies since they all use very similar systems, mainly the force platforms and the optical tracking systems, which share even the same brand and data analysis software. This would allow generalization in the interpretation of results. These analyzes could be conducted in realtime, which required more computational capacity and synchronization between systems. However, this could enable the participant to adjust his behavior according to the visual or haptic feedback of the own compensatory movements. That is, the orientation and acceleration of the headset would no longer be the only element that closed the loop, and the movements and forces exerted on the floor would have an influence on the VE. Likewise, data processing could be done online, with automatic labeling since machine learning algorithms already allow for greater ease in this process than previous statistical methods. Another aspect to be modified is related to the public accessibility of the data generated, which could be valuable information for fall prediction classifiers. 3.5 Conclusions This review presents a broad spectrum in the application of VR perturbations to balance, as can be seen by the variability of the study goals. On the other hand, contrary to what is common in this type of review, 51 CHAPTER 3. BALANCE PERTURBATIONS AND COMPENSATORY POSTURAL ADJUSTMENTS INDUCED BY IMMERSIVE VIRTUAL REALITY it focuses only on VR delivered by HMDs, considering that only these devices configure a totally immersive experience. Instead of including articles that use VR only for motor rehabilitation, this review extends to any type of analysis that is based on postural reactions to balance disturbances, which may be visual, physical, or a combination of both. From the included studies, it can be said that the study of gait, posture, and neuromuscular reactions of individuals to external disturbances has aroused a growing interest in the scientific community, largely due to the improvement of VR systems and their accessibility. It was possible to verify that, using an HMD, it is easy and safe to introduce visual disturbances. Given its portability, and the enthusiasm it can bring to the elderly, it is estimated that it will become a widely used tool in therapy, motor rehabilitation and fall prevention training both in households and at home. The main objective of this review, which was to confirm the efficiency that the disturbances imposed on the HMD had in provoking sensory conflicts that trigger imbalance, was achieved. In addition, important information was collected for the design of an standard experimental protocol that aims to induce imbalances and near-fall situations, producing kinematic, physiological, neuronal and muscular data and to build a data set that is equipped with more realistic falls than simulated and ideally serve as a classifier for fall prediction systems, mitigating the problem of falls in the elderly community. 52 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Table 5: Visual disturbances and corresponding fall categories Visual Perturbation Fall category Roll Rotation Lateral ML Translation Object Avoidance Lateral/Forward (FW)/Backward (BW) AP Translation FW/BW FW/BW Pitch Rotation Slip Virtual/Real Object Trip Vertigo Fall from heights RPY Rotation Syncope 4.3.1 Visual and proprioceptive mismatch Figure 15: Setup and view of the participant during the virtual obstacle crossing. Partially reproduced figure from [167]. Hagio and Kouzaki [167] introduced a discordance between the visual and proprioceptive systems by showing on the HMD a virtual representation of leg height different from the real one, while the participant was crossing an obstacle, as shown in Figure 15. In other words, a discordant proprioceptive feedback was introduced. This description brought up the idea of presenting an object in the VE that the participant was instructed to cross. This object would have a virtual height smaller than a real object placed in the participant’s path, on the floor. Trying to cross the obstacle, the participant would be confident that nothing will collide with his foot. In reality, probably will hit the real object because the height of this object is greater than what the subject sees in the virtual environment. This visual disturbance would cause a trip, with a mixture of physical disturbance, since it would hit a real object. For this disruption, the edge of the sidewalk is used to instruct the participant to climb the sidewalk in a natural way (Figure 16). Frost et al. [165] and Drolet et al. [166] also introduced a mixture of physical and visual disturbance. This perturbation is visual information discordant with proprioception, by varying the stiffness of the ground via a mechanical device. 59 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 16: Sidewalk trip induction via visual and proprioceptice mismatch. Thus, participants were asked to walk in a straight line on virtual stone, randomly interspersed with pieces of sand or grass. Under certain conditions, when the participant stepped on a material they expected to be soft, a surface of stone-like hardness was introduced, creating a inconsistent sensory information, as can be understood from Figure 17 and 18. This perturbation caused muscular and kinematic changes when the subject was preparing to land with their leg on a softer surface. For the purpose of this study, this visual disturbance is of no interest since none of the authors reported loss of balance. Figure 17: The virtual reality environment consisting of the walkway, sand patch, and virtual avatar, taken from [165] 60 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 18: Figure from Drolet et al. [166] setup consisting of a treadmill of variable stiffness and virtual environment delivered by a HMD. 4.3.2 Vertigo Translations in the vertical direction are expected to place the subject in a situation of virtual high height, which can cause fear and anxiety, and has been shown to influence postural control. In this framework, four studies [168–171] placed their participants in different virtual height places, causing a sensation of vertigo and anxiety, to study the influence of these conditions on gait patterns and standing balance, as seen on Chapter 3. Figures 19 to 21 depict examples of the virtual environments created to place the 61 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW participant at virtual heights. Habibnezhad and colleagues [170] studied in these conditions of elevated heights the influence of having or not having the real-time representation of an avatar while the subject walked along a narrow path on a scaffold at the top of an unfinished building. Figure 19: During the experiment, the platform in the VE went up to 2.5 meters, 5 meters and 7.5 meters, or a pit around the participants went down to 2.5 meters, 5 meters and 7.5 meters, such that all participants experienced seven distinct height conditions [168]. 62 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 20: Avatar in view is a representation of where the individual stood, while the view for the subject was a first person view from the perspective of this avatar. Subjects were positioned at a virtual Low height (0.4 m) then a High height (3.2 m) during the experiment [169]. Figure 21: The participants were subjected to height by locating the triangular beam path on the 17th floor of an unfinished building, on the left [170]. On the right, the same author placed the participants on the edge of an unfinished building [171]. These studies have led to the development of three distinct situations of vertigo. The higher places in the virtual environment were used. The first vertigo location is the roof of one of the houses (Figure 22a). 63 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW The second is on another ceiling, in a very narrow area to walk, and that forces the participant to walk as on a beam to keep his virtual representation of the avatar from falling off (Figure 22b). The last location is between two high voltage poles (Figure 22c). This way, three distinct situations of vertigo were created: a) the participant can only fall to one side; b) can fall to both sides, and c) can fall and slide if the path in not followed. In order to induce more fear and stress in the participant, the avatar was influenced by gravitational force just like a human body. (a) (b) (c) Figure 22: Snaphots from the Vertigo places: a) Simple roof, b) Electricity Pole c) Window Roof Beam Walking Gago et al. [131] and Yelshyna et al. [161] used the vertical plane translation to simulate a fall from the top of a staircase. Figure 23 shows the virtual environment and the translation sequence. These two studies originated the idea of inducing a virtual fall in the participant. This fall should happen from an upper floor to a lower floor of the house, opening up the floor beneath the participant. This disturbance is called a Free Fall and its setup in the virtual environment can be seen in Figure 38. 64 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 23: Figures reproduced from [131,161]. Top of the staircase (left) and after a visual displacement of 1.17 meters along the vertical axis, translating to the middle of the stairs (right). 4.3.3 ML-Axis translation Visual perturbations consisting of translations in the axial plane are intended to replicate a physical perturbation that pulls or pushes the body in a left or right direction. Guzman and colleagues [194] replicated a school hallway and applied continuous oscillations in the mediolateral axis to produce foreground movement whereas Riem et al. [175] applies discrete shifts in the virtual environment with a speed of 1m/s in the mediolateral axis. The participant is standing on a bridge with a pole serving as a visual reference. Participant viewpoint and virtual position are moved to the left or right, held for 9 seconds, and returned to the original viewpoint in the middle of the bridge for one second, as depicted in Figure 24. Dennison and D’Zmura [179] created a virtual environment that represented a space station that contained long corridors. While freely navigating the corridors, approximately every 2 seconds a visual disturbance would occur for 260ms and pull the participant’s body in a left or right direction. These directions were calculated relative to the direction in which the participant was looking. This study also applies visual perturbations that pull the participant’s body forward or backward. This type of perturbation also falls into the category of AP axis translations. With this literature support, the ML plane perturbation was created, consisting of a floor moving bidirectionally, at a speed of 1m/s. The ML Translation perturbation virtual setup can be seen in Figure 37. 65 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 24: Experimental setup and virtual scene with the treadmill and red pole. Reproduced from [175]. 4.3.4 AP-Axis translation As noted in the previous point, Denison and D’Zmura’s study [179] also uses this type of translation in the sagital plane. These visual perturbations give the sensation of pulling or pushing the participant’s body, simulating a backward or forward fall. Santoso and Philips [181] used the same visual perturbation by making a hallway shift abruptly for three meters, at three different speeds, in both directions. Another study [163] uses this translation. The axis is located in the participant’s eyes, virtually located inside a tunnel. The perturbation occurs in peripheral vision and is a sum of sinusoids, meaning that the perturbation is continuous, and causes the participant to travel forward and backward on the AP axis. The two perturbations created to cover this kind of backward or forward fall were translations in a corridor. These translations occur at different locations but both have a translation speed of 1m/s and are not bidirectional: one forward, one backward. The corridor of the virtual environment that will serve as the medium for one of these perturbations is shown in Figure 25. 66 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 25: AP Translation corridor. 4.3.5 Axis rotations Euler angles were used to express the rotation direction for this visual perturbations. Euler angle notation consists of the pitch, yaw, and roll angle formed in three directions. A rotation about the x-axis, or about the frontal plane, is refered to as a perturbation in the roll. A rotation about the y-axis, which is the same as rotating the coronal or frontal plane, is a pitch disturbance. Similarly, rotation about the z-axis is a disturbance in yaw. Figure 26 describes the notations used for translations and rotations in the anatomical planes of the human body. (a) (b) Figure 26: a) Human body anatomical planes. Figure taken from [231]. Directional terminology is also displayed and will be used to denominate translations; b) RPY angles used in the notation of visual disturbances, adapted from [232]. 67 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Roll rotation is intended to imply in the human body a lateral swing from left to right or vice versa. Bugnariu and Fung [156] apply this type of visual disturbance along with an associated physical disturbance, concordant or discordant in vestibular sensory levels. The perturbation has an amplitude of 8 degrees and a speed of 36 degrees per second. The stimulus is applied in each direction, separately. Peterson and Ferris [183] also applied this kind of rotation while the subject was trying to balance by walking on a beam (Figure 28). This perturbation had an amplitude of 20 degrees, and a duration of half a second. The direction CW or CCW was random. Riem and colleagues [176] also used a perturbation on the roll in the form of pseudo random sinusoidal stimulus. The perturbation had three different intensities: 3, 6, and 11 degrees. In a similar way to these works, two directions of rotation (CW and CCW) are used, and three different amplitudes (10, 20 and 30 degrees). The rotation speed is constant. Figure 27 represents one of the perturbations in Roll. CCW 30 Figure 27: The optical flow undergoes a rotation around the axis that points in the direction the participant is facing. In this example, the virtual environment rotates through a maximum amplitude of 30 degrees in half a second, returning to its normal in one second. 68 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Decommissioning Physics Initialization Scene rendering GUI rendering Game logic End of frame Editor Pausing Input events Initialization Internal animation update Gizmo rendering Internal animation update Internal animation update yield WaitForFixedUpdate Awake OnEnable FixedUpdate Update yield WaitForSeconds yield null LateUpdate OnPreCull OnGUI yield WaitForEndOfFrame yield WWW yield StartCoroutine OnWillRenderObject OnBecameInvisible OnBecameVisible OnPreRender OnRenderObject OnPostRender OnRenderImage Reset OnTriggerXXX Internal physics update OnCollisionXXX OnDrawGizmos OnApplicationPause OnMouseXXX OnApplicationQuit OnDestroy OnDisable Reset Start State machine update OnStateMachineEnter/Exit ProcessGraph ProcessAnimation Fire animation events StateMachineBehaviour callbacks OnAnimatorMove OnAnimatorIK WriteTransform WriteProperties State machine update OnStateMachineEnter/Exit ProcessGraph ProcessAnimation Fire animation events StateMachineBehaviour callbacks OnAnimatorMove OnAnimatorIK WriteTransform WriteProperties Reset is called when the script is attached and not in playmode. The physics cycle may happen more than once per frame if the fixed time step is less than the actual frame update time. If a coroutine has yielded previously but is now due to resume then execution takes place during this part of the update. OnGUI is called multiple time per frame update. OnDisable is called only when the script was disabled during the frame. OnEnable will be called if it is enabled again. OnDrawGizmos is only called while working in the editor. OnApplicationPause is called after the frame where the pause occurs but issues another frame before actually pausing. Start is only ever called once for a given script. User callback Internal multithreaded function Internal function Legend Figure 34: Script lifecycle flowchart, partially reproduced from the Unity manual [235] With the colliders set up as Triggers to activate the animations representing the visual perturbations, the next step was to edit and set the animations. The following paragraphs briefly describe Unity’s animation system. Unity’s animation system is based on the concept of Animation Clips, which contain information about how particular objects should change their position, rotation, or other properties over time. For the current project purposes, the object to animate is be the VRCamera, a child object of the Player. The participant will see any changes made to this camera, defined as our Main Camera, as if it were his view. Animation Clips are then organized into a structured flowchart-like system called an Animator Controller. The Animator Controller acts as a state machine that keeps track of which clip should currently be playing and when the animations should change or blend. Figure 35 illustrates a simple Animator Controller. 75 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 35: Simple animator controller with two states and two bool variables to control the transitions between states. The PerturbationX and PerturbationY blocks from Figure 35 represent animation clips. The state that is unconditionally activated after pressing the run button is Default, the orange color block. From the Default state, the arrows indicate conditional transitions. For these transitions boolean variables are created such as the inTrigger_PerturbationX and inTrigger_PerturbationY variables. When one of these variables gets the value ”true”, set by script, it transitions the state and plays the animation clip. When it becomes ”false”, it returns to the Default state. To illustrate an animation clip, Figure 36 shows the example of editing a simple animation that, for the duration of one second, rotates an object by 30 degrees on the x-axis, and returns to the starting position. Figure 36: Example of an animation clip edit. When this animation is activated, it will apply a rotation on the x-axis of 30 degrees and return to the initial orientation in one second. This transformation is represented by the purple line. After, the script that detects the collision between the Player and the box collider with trigger property is 76 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW added to the gameObject with the collider component. This script has the function of activating the boolean variable that transitions state in the animator controller and thus starts playing the visual animation clip. Also, this script should save in the Unity Console Log a message with the exact time when the collision occurred and when it left the box collider, giving the perturbation onset time and its final time, along with the perturbation name. Listing 1: Code handling the trigger entering. 1[SerializeField] private Animator animator_name; 2public GameObject trigger_name; 3private bool onLog = false; 4 5private void OnTriggerStay(Collider other) 6{ 7if (other.CompareTag(”Player”)) 8 9{ 10 animator_name.SetBool(”inTrigger_PerturbationX”, true); 11 if (!onLog) 12 { 13 UnityEngine.Debug.Log(”PerturbationX onset: ” + System.DateTime.Now.ToString((”HH:mm:ss: ↩→fff”))); 14 onLog = true; 15 } 16 17 } 18 } Using the MonoBehaviour.OnTriggerStay(Collider) class, where the parameter “other” refers to the Player, the boolean variable set in the animator controller is activated. In our example listed is the variable in_Trigger_PerturbationX which causes an animator controller state transition and makes the animation start. Listing 2: Code handling the trigger exit. 1private void OnTriggerExit(Collider other) 2{ 3if (other.CompareTag(”Player”)) 4{ 5roll_animator.SetBool(”inTrigger_PerturbationX”, false); 6UnityEngine.Debug.Log(”PerturbationX final: ” + System.DateTime.Now.ToString((”HH:mm:ss: ↩→fff”))); 7StartCoroutine(TimeDelay()); 8} 9} 77 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW OnTriggerExit is called when the collider ”other”that is Player in practice, no longer has contact with the trigger. When this function is called, the boolean control variable takes the value zero, and the animation stops. The same happens for the time registration, making a calculation between the onset and the end, which gives us the duration of the disturbance. Listing 3: 30 seconds delay script. 1IEnumerator TimeDelay() 2{ 3//UnityEngine.Debug.Log(”Delay started”); 4trigger.GetComponent<Collider>().enabled = false; 5yield return new WaitForSeconds(30); 6//UnityEngine.Debug.Log(”Timer gone”); 7trigger.GetComponent<Collider>().enabled = true; 8} A 30 second delay was applied that deactivates the collider component of the gameObject so that the participant after suffering the visual disturbance can turn around and return to the starting point by passing over the trigger without it having any effect. The visual perturbations that have a different control flow are those corresponding to translations in the AP plane in both directions. The logic used does not activate a boolean variable. When the participant hits the trigger, the Player is affected of a position transformation every frame, giving the illusion that the participant is floating, at a speed of 1m/s. Below is shown an excerpt of code that controls this perturbation. The variable ”speed”is set to a value of 1 and the ”target”is the target location at the end of the corridor. This time a delay is not used. Instead, the gameObject that has the trigger is destroyed. Listing 4: AP perturbations script. 1float step = speed * Time.deltaTime; // calculate distance to move 2myPlayer.transform.position = Vector3.MoveTowards(myPlayer.transform.position, target. ↩→transform.position, step); 3if(myPlayer.transform.position == target.transform.position) 4{ 5Destroy(trigger); 6} Finally, to conclude the explanation of VE creation and the method of introducing the perturbations, a description will be given of the animations created to achieve the effect of the chosen visual disturbances from Table 6. 78 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 37: ML floor translation, continuous and bi-directional. The visual translation perturbation (VP003) in the ML plane consists of an animation applied to the gameObject representing the floor. This animation is bi-directional, continuous, and has a speed of 1m/s. Figure 37 shows the floor being animated, as well as the first-person view that the participant will experience while undergoing this balance perturbation. For free fall (VP011), an animation is also used that transforms the position of the floor, sliding under the participant’s feet and opening a void for the room below. In order for the avatar representing the participant’s body to fall to the lower floor in the virtual environment, it is necessary to make the box collider that is part of the floor move with it. In the animation edit in Figure 38 this collider control is depicted. 79 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 38: Scene view from the free fall perturbation setup. The other animation that transforms gameObjects is the animation that corresponds to Object Avoidance (VP009). The bottles that are placed on the virtual floor start flying against the participant’s head, applying only a position and rotation transformation to the bottles. Figure 39 shows the animation clip only with transformations in the position and rotation of the bottles. The target of this transformation is the avatar’s head. 80 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 39: Object avoidance clip animation setup. There is no animation governing the AP translation but with the movement of the whole Player, avoiding the camera moving and the participant having the view of his own static body while feeling himself moving in the AP plane. The Player’s motion is made by script, as has already been described. One more rotation of the VRCamera is accomplished by animation, the pitch rotation. Like the roll rotation, only in one direction, with an amplitude of 25 degrees and a speed of 60 degrees per second. 81 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW Figure 40: Animation of the pitch rotation. The example in the figure is depicted in one of the places that this disturbance can occur, in a bathroom. This animation is intended to simulate a slip. In all animations that applied a camera rotation, the HMD tracking was turned off so that the participant’s head movement would not overlap with the animation playing. The TrackedPoseDriver component applies the current pose value of a tracked device to the transform of the GameObject. In these cases, the rotation tracking type was disabled while the animation was running, leaving only the position tracking so that the participant’s movement would be detected and consequently its trigger exit. This setting is visible in Figure 40, where the value of TrackingType is set equal to 2. In order for the subject to be placed in a position to access the trigger in a few steps, the participant is placed in a position about a unit of measurement that corresponds to one meter. As several studies prove, a person wearing a HMD takes a shorter stride length than they do in a comparable real world condition [236]. On the other hand, more recent studies evidence that gait parameters such as stride length can be quantitatively equivalent in a VE and in the real world [237,238]. The script responsible for placing the Player in the defined positions consists of a series of conditions that check whether a given keyboard key has been pressed. If it was pressed, it looks for the gameObject that has the information of the position and rotation that the Player must be placed and assigns it these transformations. In the code snippet of the lines below, an example can be seen for the ’I’ key: Listing 5: Excerpt of code that places the participant in the start position, which will correspond to a distinct location for each keyboard key pressed. 1if (Input.GetKeyDown(KeyCode.I)) 2{ 3Vector3 position = this.transform.position; 4Quaternion rotation = this.transform.rotation; 5 82 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW 6position = GameObject.Find(”Position_Roll_CW20_in2”).transform.position; 7rotation = GameObject.Find(”Position_Roll_CW20_in2”).transform.rotation; 8 9this.transform.position = position; 10 this.transform.rotation = rotation; 11 } The ”this”attribute refers to the gameObject that the script is attached to, the Player. When the key is pressed, the Player becomes oriented toward the trigger, about one meter or a mean step away from entering the box collider. In the case of vertigo situations, it is placed in the high-height locations. 4.5 Project Overview Virtual Environment Unity scene development Visual perturbations Triggers and scripts Data Collection Experimental protocol Questionnaires Materials Data Processing Raw data treatment Kinematic/muscular parameters and metrics Dataset labeling Statistical Analysis Statistical test assumptions Analysis of variance Figure 41: Project Overview: essential phases. The strategy description used to approach a solution to the posed problem, it was initiated by conceptualizing and creating a virtual environment as a necessary medium to automatically introduce visual perturbations. Depending on this necessary implementation, the project comprises four main phases: i) virtual environment design and implementation; ii) visual perturbation protocol elaboration for multisensory data collection; iii) data processing and assembling into a dataset; and iv) statistical analysis. The virtual environment design has been discussed in detail. From conceptualization to construction, the software that enabled the development of the virtual environment was briefly explored. In addition, Unity allowed the development of the strategy to introduce disturbances at moments that will be dictated by the experimental protocol: execution of scripts when a trigger is detected, which starts an animation. The data collection through the experimental protocol will record kinematic, muscular and electrodermal data of the participants in response to the perturbations introduced. The explanation of this step will be provided in Chapter 5, where the systems used will be detailed. The systems are both the virtual reality setup and the sensor systems: i) motion capture system; ii) electromyography and iii) galvanic skin response. Each of these sensors will collect data in a synchronized manner. The raw data that arises and its processing will be explained. The whole process from the collected metrics to the construction of a dataset with this 83 CHAPTER 4. VIRTUAL ENVIRONMENT AND PROJECT OVERVIEW information from the three systems will be described. Additionally, the labelling of the pertrubations on the dataset is illustrated. Finally, in Chapter 6, a statistical analysis is carried out, which has to follow a set of assumptions. These assumptions are verified and statistical methods are applied to answer the research questions posed in Chapter 1, section 1.3.2. 84 CHAPTER 5. MATERIALS AND METHODS Figure 45: Sensor placement diagram, taken from [248]. 5.2.2.2 Electromyography Using kinematic measures to validate model output is an obvious and necessary step, but may be a very blunt instrument for validating neuromechanical systems. Kinematics alone is insufficient to distinguish different neural control strategies that result from different forces or patterns of muscle activation, potentially indicating different mechanisms of sensorimotor control. The measurements produced by muscle activation signals are representative, but are not equivalent to muscle tension. For this reason, one can make the mistake of trying to collect only kinematic information. Although good information of joint moments and powers is obtained, only the net moment created by all forces crossing the joint are obtained, and therefore the contribution of each muscle cannot be determined without additional information. This additional component is only accessible through EMG. Muscle activity as recorded through electromyography can provide important information as it represents amplified motor neuron pool activity and is also related to muscle force [249–251]. While it may be considered ideal to access neural signals directly, there are substantial limitations in current invasive techniques: they are intrusive, uncomfortable, painful and difficult to setup. There are two predominant forms of EMG measurement; surface and intramuscular EMG [252]. Non-invasive surface EMG is widely used for superficial, large, and easily accessible muscles. 91 CHAPTER 5. MATERIALS AND METHODS This is the case for the muscles studied, detailed later in muscles item. In most studies, the surface myographic electrical signals are collected with Ag/AgCL electrode arrays arranged in a bipolar configuration and connected by cables to signal processing apparatus that are supported on a wearable. The connection made between the signal acquisition module and the receiving module can be made by cables, fiber optics or radio-telemetry. To obtain a temporal and spatial distribution description of multiple muscle activities, it is necessary to use multichannel recording. The decision between using surface EMG or internal EMG recording in gait analysis has to do with the accessibility of the target muscle by surface electrodes. If there is no other way, it is necessary to use fine-wire probes. Muscles such as the ilio-psoas and tibialis posterior are examples of locations inaccessible by surface electrodes. Further, the interference of indwelling electrodes on the gait pattern can be substantial, with an impact on step length, cadence and gait speed [253]. The Trigno Wireless Biofeedback System is a device designed to make electromyographic and biofeedback signal detection reliable and easy. The system transmits signals from Trigno Avanti sensors to a receiving base station using a time-synchronized wireless protocol that minimizes data latency across sensors. The core architecture of the Trigno System is designed to support high fidelity EMG signals, along with complementary biofeedback signals such as movement data, force signals, contact pressure events, timing, and triggering information. Trigno Avanti sensors support a low noise, high fidelity sensing circuit for detecting electromyographic biofeedback signals from the surface of the skin when muscle contract. Sensor bandwidth is selectable between 10-850Hz and 20-450 Hz and the input range of the sensor can be selected between 22mV or 11mV depending on user needs. Trigno Avanti SensorBase Station Figure 46: Overview of the system used, with an application example. The EMG sensor and the base station are highlighted. Images reproduced from [254] 92 CHAPTER 5. MATERIALS AND METHODS After presenting the studies that used electromyography to interpret postural reactions provoked by visual or visual and physical disturbances in Chapter 3, a diagram was elaborated with the indication of the muscles used, grouped by type of disturbance. The color code corresponding to the perturbations is located at the bottom of the diagram. Since the available number of sensors is 8, the choice fell on placing sensors in those muscles that, according to the literature review, are the most comprehensive with respect to the various types of disturbance chosen. Therefore, in order to collect muscle activation data illicited by most compensatory reactions, the following muscles were chosen: i) Tibialis Anterior (both legs); ii) Gastrocnemius Medial Head (both legs); iii) Semitendinosus; iv) Rectus Femoris; v) External Oblique; and vi) Sternocleidomastoid. Tibialis Anterior [1, 4, 5, 6, 7, 8, 9, 10, 11, 12]Soleus [1, 4, 5, 9, 10] Gastrocnemius Lateral [1, 4, 5, 12] Gastrocnemius Medial [1, 4, 5, 6, 7, 8, 9, 10, 11, 12] Neck [2] Sterno-cleido Mastoideus [3, 7] Extra-ocular inferior oblique [3] Rectus Femoris [6, 9, 12, 12] Biceps Femoris [6, 9, 12] External oblique [6, 9, 12] Rectus Abdominus [6,12] Erector Spinae [6, 7, 12] Vastus Lateralis [7, 8, 11] Semitendinosus [7, 8, 11] Tensor Fascia Latae [7] L3 Level [7, 9] Middle Deltoid [9, 12] Peroneus Longus [10] Color - Visual Perturbation Vertical translation (heights) |Vertical and Horizontal | Roll Axis Tilt | Roll & Pitch Axis Tilt Pitch Tilt | Visual Motor Perturbation | Virtual and Real Obstacles Figure 47: 1 - Sun et al. 2019 [168]; 2 - Peterson et al. 2018 [185]; 3 - Chiarovano et al. 2018 [188] ; 4 - Mohebbi et al. 2020 [178]; 5 - Drolet et al. 2020 [166]; 6 - Ida et al. 2017 [172]; 7 - Bugnariu and Fung 2007 [156]; 8 - Liu et al. 2015 [186]; 9 - Cleworth et al. 2016 [169]; 10 - Peterson and Ferris 2018 [183]; 11 - Parijat et al. 2015 [130]; 12 - Porras et al. 2021 [255]. 5.3 Balance Perturbation Protocol To test the effect of visual perturbations on biomechanical variables that characterize loss of balance, or in other words, that induce compensatory postural reactions, an experimental protocol was developed to introduce the perturbations and to collect data. The design of this experimental protocol is intended to 93 CHAPTER 5. MATERIALS AND METHODS achieve a level of control that will not allow for other interpretations about its success or failure. Virtual reality is a technology that offers advantages in the control and reproducibility of the experiment. Four fundamental components in the design of the experimental protocol are brought together, namely: i) stimulus control; ii) experiment reproducibility; iii) ecological validity and iv) real-world learning transfer. Before developing the experimental protocol, a survey of articles using VR via HMD and introducing visual disturbances was conducted in Chapter 3. Information was extracted about the virtual environments and what types of visual disturbances were presented, the timing of the stimuli onset and their duration. An uncontrolled trials type study was designed. To this end, a list of tasks to be carried out with each participant was drawn up, visible in the ”Tasks”section. 5.3.1 Tasks The experimental protocol was designed on top of performing sequential tasks. First of all, it is imperative to collect the user’s demographic data: age, height, weight, gender and citizen card ID for methodological reasons and to report the study. The areas intended for the placement of the electromyography sensors were cleaned with alcohol in order to maximize the quality of the electrical signal relayed and to fix the sensors well to the skin. The participant was then asked to be instrumented with the sensors, identifying the chosen muscles: i) Tibialis Anterior; ii) Gastrocnemius Medial Head; iii) Rectus Femoris. The sensors were firmly glued to the skin. Three trials of Maximum Voluntary Contraction (MVC) were performed for each muscle for further normalisation of EMG envelope. Further, participants were equipped with the full body configuration of Xsens MVN Awinda wearable inertial system that collected data at 60 Hz, which is composed by 17 IMUs placed in the following body landmarks: i) head; ii) sternum; iii) pelvis; iv) right and left shoulders; v) right and left upper arms; vi) right and left forearms; vii) right and left hands; viii) right and left upper legs; ix) right and left lower legs; and x) right and left feet. Following the sensor placement, participants underwent the N-Pose calibration of the system. Lastly, participants also worn the Shimmer GSR device on the dominant forearm with the electrodes placed on the index and middle fingers. PPG sensor must also be placed on the extremity of the index finger. To ensure safety, the harness was placed on the participant. For safety reasons, the length of the harness rope is adjusted in order to register a minimum of 15cm between the knees and the floor. This procedure is accomplished by asking the participant to raise their feet, which will lead to the application of all the body weight into the harness system [243] Then, equip the participant with the HTC Vive Pro headset and advise the subject to ask for help whenever necessary, such as episodes of motion sickness. The subject must adjust the distance between the lenses and the distance between the face and the lenses for a proper and more comfortable usage. The subject may still have a period of habituation to the virtual reality device and the environment, performing a familiarization trial without perturbations while using the entire setup, instructing the subject to stay within the playing area restricted by a virtual blue boundary. The subject performed the following 94 CHAPTER 5. MATERIALS AND METHODS activities without perturbations: walk front, turn around and ending at the starting position (3 times). In order achieve synchronous data acquisition from all the sensor systems, Sync Lab Desktop App was used. This team-developed desktop application for Windows OS is capable of synchronously start and stop data collection from the above mentioned systems and save the collected data in the computer that runs the app. The trigger signals sent by the Desktop application are electronic or wireless pulses. The former are either sent via Syncbox, which is a team-developed hardware interface that connects to the Xsens and Delsys systems or by direct USB communication, which is used to connect to both Kinect and Optitrack cameras. The wireless communication with Shimmer GSR system is performed directly from the computer running the app. Most importantly, the subjects performed the virtual perturbations available on 6. Each perturbation from Table 6must be performed three times during the entire experimental protocol. These perturbations must be performed sequentially with a different order for different subjects. A Matlab script will generate randomly the virtual perturbations and non-perturbations introduction order. At the end, participants were asked to fill out a questionnaire about motion sickness during the experimental activity, the SSQ. Other questions will be asked to evaluate the immersion level and the effectiveness of the visual perturbations. 5.3.1.1 Questionnaires The SSQ was used to assess the occurrence of cyber sickness episodes. To try to make a subjective measurement of the sense of presence achieved by the participants and to assess the degree of immersion in the created virtual environment, Igroup Presence Questionnaire (IPQ) was used. To understand at what level the simulation experience was convincing, users have to be comfortable, engaged and experience the phenomenon of presence and place illusion. Assessing simulator sickness is widely adopted in VR research. In SSQ, each item is rated with the scale from none, slight, moderate to severe. Through some calculations, four representative scores can be found: nausea-related subscore (N), Oculomotor-related score (O), Dysorientation-related subscore (D) are the scores for the symptoms for the specific aspects. Total Score (TS) is the score representing the overall severity of cybersickness experienced by the users of virtual reality systems. The questionnaire and how the score is calculated is described in Appendix 1. The IPQ items can be presented in question form and the response is based on a scale of the degree of agreement or disagreement with the questions or statements. Questionnaire is presented in Appendix 1. 5.4 Data Processing Diagram of Figure 48 summarizes the procedure for data processing from the three separate systems until the final dataset is compiled for each participant. 95 CHAPTER 5. MATERIALS AND METHODS 05 –Dataset construction Concatenated data gathered from the three distinct systems 01 - Xsens .mvn file to .xlsx Import to Matlab Timestamps generation 03 - Shimmer Downsampling 100.21Hz to 60Hz 04 –Time Sync Add samples / cut samples to ensure time synchrony between the systems 02 - Delsys Downsampling 66.27Hz to 60 Hz Amplitude analysis (%MVC) EMG values ranging [0-100] DATA PROCESSING Figure 48: Data processing phases. To explain which data to process, the raw data collected is reminded. From Xsens, at 60Hz, positional and accelerometry data from the IMUs. This system will serve as a reference so that the data collected from the other sensors will be resampled and presented with the same timestamp as Xsens. The system responsible for the temporal synchronization of the sensors’ acquisition creates a text file called StartRecord_X with date, hours, seconds and milliseconds. The start and end time stamps serve as time boundaries for the first step: placing timestamps on the samples collected from Xsens, which will be the reference for all other data collected. After exporting the file saved in .mvn format to .xlsx, the following data is accessible. In each tab, the first column concerns the frame. Then, the orientation in quaternion and euler angles of the segments of the Pelvis, L5, L3, T12, T8, neck, head, right and left shoulder, upper arm, forearm, upper and lower leg, foot and toe. The segment position, linear and angular velocity, linear and angular acceleration in all three dimensions are also available. The Joint angles and ergonomic joint angles in reference ZXY and XZY, the center of mass, sensor free acceleration and magnetic field. Finally, sensor orientation in queternion and euler angles in three dimensions. All these kinematic features were saved in the dataset. In this initial phase, the available data was compiled from the excel file into Matlab table, adding the timestamp. This timestamp has a synchronizing role. All other systems will be time aligned with Xsens. The Delsys system collected muscle data at a sampling frequency of 1111Hz. Before downsampling to the desired frequency of 60Hz, amplitude analysis was done with scripts built into the EMGworks software. This process is a normalization of the EMG data to MVC. This post-processing method uses a maximum RMS value from a recording to normalize subsequent EMG data series. The output is displayed as a 96 CHAPTER 5. MATERIALS AND METHODS percentage of the MVC (%MVC) value, which can be used to easily establish a common ground when comparing data between subjects. This normalized data is now downsampled in a Matlab script. Additionally, the script takes negative values as zero and values above 100% as 100%. This problem stems from the MVC trials recordings. In this situation, the participant may have done less force than they actually could. Although there are three trials and the maximum value is chosen from those three values, there may be variability because of posture and breathing [256]. These values between 0 and 100 were stored in the dataset for all 8 muscles. Also, the GSR Shimmer data was downsampled from 100.21Hz to 60Hz. In case any system is delayed in starting the data record or finishing the recording, a Matlab script adds samples or cuts samples. After the necessary data processing to synchronize the data resulting from the three systems, the dataset was built concatenating the datasets of each system that were created and explained in the previous paragraphs. The Matlab script then joins the data together. It deletes the timestamps from all the systems and keeps only the Xsens one, which serves as a reference. The developed code identifies the size of the Xsens, Delsys and Shimmer samples. A state machine aligns the data if it is too much or too little. Finally, there is a concatenation of datasets, resulting in the final dataset. There is one dataset for each of the 12 subjects but at this stage, these data collections are unlabeled datasets. 5.4.1 Labelling The following paragraphs will show how the labbeling of each dataset was done. Each perturbation was associated with a number. With the exception of the trip perturbation, which has an additional number to identify the foot strike. The undisturbed walking situations also have a label, as show in Table 9 Table 9: Label encoding. Roll Indoor 1 CW10 1Roll Indoor 2 CCW20 11 AP Axis Trans - Corridor Backward 21 Roll Indoor 1 CW20 2Roll Indoor 2 CCW30 12 Pitch Indoor - Bathroom 22 Roll Indoor 1 CW30 3Roll Outdoor CW10 13 Pitch Indoor - Fridge 23 Roll Indoor 1 CCW10 4Roll Outdoor CW20 14 Window Roof Beam Walking - Vertigo 24 Roll Indoor 1 CCW20 5Roll Outdoor CW30 15 Window Roof Beam Walking - Vertigo No Avatar 25 Roll Indoor 1 CCW30 6Roll Outdoor CCW10 16 Simple Roof - Vertigo 26 Roll Indoor 2 CW10 7Roll Outdoor CCW20 17 Simple Roof - Vertigo No Avatar 27 Roll Indoor 2 CW20 8Roll Outdoor CCW30 18 Pitch Outdoor - Near Car Oil 28 Roll Indoor 2 CW30 9ML Axis Trans - Kitchen 19 Trip - Sidewalk / Trip Shock 29 / 290 Roll Indoor 2 CCW10 10 AP Axis Trans - Corridor Forward 20 Bedroom Syncope 30 Garden - Object Avoidance 31 Electricity Pole - Vertigo No Avatar 33 Stairs 35 Electricity Pole - Vertigo 32 Free Fall 34 The random sequence with the list of perturbations ordered chronologically served as a reference. For each perturbation that appeared in the sequence, one looked for the text that identifies it in the Unity log file. This Log marked the onset and end times of the disturbances that actually occurred, in the format HH:mm:ss.ms. When the string ”onset”was identified, the associated datetime was removed. This instant 97 CHAPTER 5. MATERIALS AND METHODS of time was compared with the timestamps in the dataset. At the position that found the match, it was filled with the proper label until the end of the perturbation, with a similar process. It was necessary to make some corrections and manual entries of onsets and finals of perturbations. For this, the frames associated with the timestamps were useful for visual inspection in Xsens MVN software. Additionally, a text file was used that was filled in during the execution of the experimental protocol, which pointed out failures such as errors of the Unity software not activating the visual perturbations. This meant that the perturbation was recorded but the subject did not experience the perturbation. The scripted failure detection system would count the number of perturbations that were supposed to have occurred, detect whether there was a lack of information in an onset or an ending, and give the opportunity to manually enter the correct timing of the perturbation, to correctly label the dataset. The label plot shown in Figure 49 helped to correct flaws, as it gives an overview of the perturbations that occurred throughout the experimental protocol, whether there was overlap or if there were any perturbations of unusual duration. The horizontal axis represents the samples collected and the vertical axis represents the labels. Figure 49: Ploted labels throughout the experimental protocol, for one subject. X-axis: Samples; Y-Axis: Labels. Once the automatic labeling was done and the manual corrections and additions were completed, an error control script counted the occurrence of each disturbance, now represented by numeric labels. Figure 50 shows an example of identification of the trip disturbance by visual inspection. 98 CHAPTER 5. MATERIALS AND METHODS Figure 50: Visual inspection of the onset of the trip, annotation of the shock frame and subsequent detection of the end of the disturbance. At this point, the 12 subjects datasets are fully labeled. In Figure 50 is an example of a transition from an undisturbed walking situation or neutral situation to a disturbance with label 33. 99 CHAPTER 5. MATERIALS AND METHODS Figure 51: Labelled datset. 5.5 Discussion Regarding the experimental protocol, there are two issues that should be highlighted that both serve to improve a future protocol and possible introduction of variability. First, the distances between the starting point of the gait and the trigger were fixed. An ideal situation would calculate at baseline the average stride lenght of the undisturbed gait and adjust this distance so that the participant would take about two steps before experiencing the perturbation. The second issue is related to the randomness of the introduction of the perturbations. Although a script was run that listed the perturbations randomly and did not allow for consecutive perturbations of the same type and intensity, some participants experienced the perturbations that were anticipated to be most severe at the beginning of the protocol and may have experienced a training effect for the next perturbations in the protocol. This presumption will be discussed when presenting the results of the statistical analysis. Overall, the protocol was run on schedule, generally with no failures in the introduction of the perturbations and no severe episodes of motion sickness. The data labeling processing phase deserves a brief note. Some of the visual perturbations did not write their beginnings or endings in the log. Therefore, it was necessary to manually check, frame by frame, the beginnings and ends of some visual perturbations: trip and object avoidance. A time-consuming process, requiring meticulous attention. 100 CHAPTER 6. STATISTICAL ANALYSIS Table 13: Skewness and Kurtosis absolute values and z-values for each muscular variable, with or without perturbation. Descriptives Statistic std. error z-value RTAAVG No Perturbation Skewness -0.200 0.104 -1.92462074 Kurtosis -0.277 0.207 -1.3332817 Perturbation Skewness 0.644 0.105 6.145592741 Kurtosis 1.276 0.209 6.095852283 LTAAVG No Perturbation Skewness 0.162 0.104 1.562132743 Kurtosis -0.696 0.207 -3.353548154 Perturbation Skewness 1.130 0.105 10.7821742 Kurtosis 3.313 0.209 15.8305142 RGMAVG No Perturbation Skewness 1.009 0.104 9.712540647 Kurtosis 0.725 0.207 3.497357039 Perturbation Skewness 1.578 0.105 15.05613193 Kurtosis 6.822 0.209 32.59831173 LGMAVG No Perturbation Skewness 1.815 0.104 17.46800409 Kurtosis 3.329 0.207 16.05392318 Perturbation Skewness 1.755 0.105 16.74056563 Kurtosis 3.260 0.209 15.57825029 RFAVG No Perturbation Skewness -0.239 0.104 -2.296615259 Kurtosis -0.038 0.207 -0.184753307 Perturbation Skewness 1.889 0.105 18.02081296 Kurtosis 5.004 0.209 23.91069225 STAVG No Perturbation Skewness 0.336 0.104 3.235309874 Kurtosis -0.139 0.207 -0.671756261 Perturbation Skewness 1.249 0.105 11.91366701 Kurtosis 4.057 0.209 19.38650919 EXTOAVG No Perturbation Skewness 2.015 0.104 19.39606869 Kurtosis 3.166 0.207 15.26556396 Perturbation Skewness 3.130 0.105 29.85476694 Kurtosis 17.261 0.209 82.47762381 SCMAVG No Perturbation Skewness -1.524 0.104 -14.67238138 Kurtosis 2.654 0.207 12.79916042 Perturbation Skewness -1.424 0.105 -13.58047507 Kurtosis 2.612 0.209 12.48164025 6.1.1.2 Multivariate outliers The detection of multivariate outliers relies on different methods than the detection of univariate outliers. Univariate outliers have to be detected as values too far from a robust central tendency indicator, while multivariate outliers have to be detected as values too far from a robust ellipse that includes most observations [270]. In order to detect multivariate outliers, most researchers compute the Mahalanobis distance 107 CHAPTER 6. STATISTICAL ANALYSIS Table 15: Mahalanobis distance critical values. Dependent variables Critical Value 1 10.8275662 2 13.8155106 3 16.2662362 4 18.466827 5 20.5150057 6 22.4577445 7 24.3218863 8 26.1244816 ... ... 182 246.695142 Table 16: Residuals statistics - Mahal. Distance Residuals Statistics Minimum Maximum Mean Std. Deviation N Predicted Value -0.64 1.46 0.50 0.443 1096 Std. Predicted Value -2.558 2.168 0.000 1.000 1096 Standard Error of Predicted Value 0.047 0.253 0.094 0.035 1096 Adjusted Predicted Value -5.25 328.41 0.90 10.420 1096 Residual -0.816 0.812 0.000 0.232 1096 Std. Residual -3.227 3.211 0.000 0.918 1096 Stud. Residual -3.525 3.811 0.002 1.013 1096 Deleted Residual -328.412 6.251 -0.407 10.425 1096 Stud. Deleted Residual -3.548 3.839 0.002 1.014 1096 Mahal. Distance 37.570 1093.998 172.842 156.170 1096 Cook’s Distance 0.000 9696.091 9.771 294.142 1096 Centered Leverage Value 0.034 0.999 0.158 0.143 1096 [271,272]. This method is based on the detection of values too far from the centroid shaped by the cloud of the majority of data points. To calculate the Mahalanobis distance, a linear regression was performed. This step adds a new column to the dataset with the Mahalanobis distance, called ’MAH_1’. The critical value was calculated with the formula CHISQ.INV, with the probability parameter having a value of 0.999 and the degrees of freedom corresponding to the number of dependent variables. Table 15 shows the critical values calculated for 182 dependent variables. Table 17 is the SPSS output for linear regression residuals statistics. Mahalanobis distance is described by row ”Mahal. Distance”. To calculate the Mahalanobis probability the compute variable is used with the expression p_MAH = 1-CDF.CHISQ(MAH_1,182). If p<0.001 it is a significant outlier. 175 significant outliers are identified in a total of 207144 samples, therefore, can be neglected. Outliers descriptive statistical analysis was performed by subject, by label of disturbance, and by the binary perturbation or no-perturbation distinction. Figure 55 reproducted from SPSS indicates a presence of significant outliers 108 CHAPTER 6. STATISTICAL ANALYSIS mostly in the labels corresponding to the disturbances. More specifically, 56.6 percent of the significant outliers appear in the disturbances. The AP-Backward Translation perturbation (label = 21) and Trip (Label = 290) represent 7.4% of these outliers. Figure 55: Descriptive statistics of outliers identified in perturbation. 6.1.1.3 Multicolinearity Multicollinearity occurs when one dependent variable is almost a weighted average of the others - are correlated. If the degree of correlation between variables is high enough, it can cause problems interpreting the results. The correlation can be calculated with the Pearson correlation coefficient. These numbers measure the strength and direction of the linear relationship between the two variables. The correlation coefficient can range from -1 to +1, with -1 indicating a perfect negative correlation, +1 indicating a perfect positive correlation, and 0 indicating no correlation at all. In SPSS, the calculation of Pearson’s correlation 109 CHAPTER 6. STATISTICAL ANALYSIS is done with a bivariate correlation analysis by choosing the Pearson coefficients option. Table 19 exemplifies the Pearson correlation values for the first eight variables, corresponding to the averages of muscle activations. Table 18: Pearson Correlation values for the muscle variables averages. AVG = Average; Key: RTA;LTA; RGM;LGM;RF;ST;EXTO;SCM. RTAAVG LTAAVG RGMAVG LGMAVG RFAVG STAVG EXTOAVG SCMAVG RTAAVG Pearson Correlation 10.323 0.286 0.184 0.377 0.252 0.109 -0.071 LTAAVG Pearson Correlation 0.323 10.271 -0.004 0.179 -0.048 -0.187 0.198 RGMAVG Pearson Correlation 0.286 0.271 10.243 -0.061 0.171 -0.097 0.174 LGMAVG Pearson Correlation 0.184 -0.004 0.243 1-0.164 0.123 0.697 0.314 RFAVG Pearson Correlation 0.377 0.179 -0.061 -0.164 10.115 0.039 -0.423 STAVG Pearson Correlation 0.252 -0.048 0.171 0.123 0.115 10.017 -0.49 EXTOAVG Pearson Correlation 0.109 -0.187 -0.097 0.697 0.039 0.017 10.248 SCMAVG Pearson Correlation -0.071 0.198 0.174 0.314 -0.423 -0.49 0.248 1 To interpret the values of the coefficients, positive values denote positive linear correlation, negative values denote negative linear correlation, a null value denotes no linear correlation and the closer the value is to 1 or –1, the stronger the linear correlation. Most researchers agree that a coefficient of <0.1 indicates a negligible and >0.9 a very strong relationship [273]. Strong correlations occur between variables that do not fluctuate much, so the mean value will be positively correlated with its minimum and maximum value. The accelerometry and gyroscopy values of the sternum and pelvis, whether in mean, standard deviation, or maximum and minimum values, are highly correlated, as the displacement and tilt of the trunk and hip are also positively correlated in biomechanical terms. Therefore, in the statistical model, multicollinearity does not present a problem since there is no need to reduce the independent variables. Only caution is required in interpreting the results. 6.1.1.4 Linear relationship between the dependent variables for each level of the independent variables To check this linear relationship assumption, in SPSS, produced a Scatter/Dot graph (Matrix Scatter). For simplicity, Figure 56 represents two dependent variables - RTAAVG and LTAAVG. The independent variable is the Perturbation/No Perturbation level, named BinaryLabel. 110 CHAPTER 6. STATISTICAL ANALYSIS Figure 56: Linear relationship between the dependent variables for each level of the independent variables. 6.1.1.5 Homogeneity of variance-covariance matrices Variance-covariance matrices should be compared between groups using Box’s M-test, also called Box’s Test for Equivalence of Covariance Matrices, is sensitive to non-normality. Box’s M test is a parametric test used to compare variation in multivariate samples. More specifically, it tests if two or more covariance matrices are equal or homogeneous. The null hypothesis for this test is that the observed covariance matrices for the dependent variables are equal across groups, i.e., a non-significant test result indicates that the covariance matrices are equal. The generated test statistic is called Box’s M statistic [274]. 111 CHAPTER 6. STATISTICAL ANALYSIS Figure 57: Box’s M Test. Box’s M-test p<0.001. The null hypothesis of equal (homogeneous) covariances matrices is rejected: the assumption of homogeneity of covariance is violated. However, real data in behavioral science often deviates from this assumption [275], [276]. Previous studies have found that analysis of variance is sensitive to violations of homogeneity assumptions [277–280], and several procedures have been proposed for dealing with heteroscedasticity. This suggests that heterogeneity has a greater effect on MANOVA robustness than does non-normality [278]. A desirable characteristic of a test is that while it is powerful, i.e., sensitive to changes in the specified factors under test, it is robust, i.e., insensitive to changes in extraneous factors not under test. Specifically, a test is called robust when its significance level (Type-I error probability) and power (one minus Type-II error probability) are insensitive to departures from the assumptions on which it is derived [281]. 6.1.2 Post-hoc Test If a significant F-value in an analysis of variance is obtained, it merely indicates that there are differences between the population means. This significant value is an indicator that there is an influence of visual perturbations on gait parameters. That is, visual perturbations effectively invoke imbalances. However, it does not inform in which specific perturbations the averages are significantly different. This separation will allow to infer which disturbances challenged the participant’s balance the most. At this point, after conducting the analysis of variances, differences between means individually will be inspected. The purpose is to isolate significant differences. Much of the work on multiple comparisons is based on Tukey’s work. In the case of this experimental protocol, an important comparison will be between the cases where there is no perturbation and the cases where there is visual perturbation. In this case, the undisturbed gait will serve as a control. When one group is to be compared against several, the recommended test is Dunnett’s test [282]. Dunnett’s test is performed by computing a Student’s t-statistic for each group where the statistic compares the treatment group (perturbation labels) to a single control group (no perturbation). Since each comparison has the same control in common, the procedure incorporates the dependencies between these comparisons. In particular, the t-statistics are all derived from the same estimate of the error variance which is obtained by pooling the sums of squares for error across all groups. The formal test 112 CHAPTER 6. STATISTICAL ANALYSIS statistic for Dunnett’s test is either the largest in absolute value of these t-statistics, or the most negative or most positive of the t-statistics. 6.2 Results and Discussion 6.2.1 one-way MANOVA An initial one-way MANOVA examined muscular, kinematic and electrodermical metrics as dependent variables, and binary label perturbation/no perturbation as independent variables. Skewness, kurtosis, and Lyapunov exponents of the calculated metrics were excluded from this multivariate analysis. Since the factor is the binary label perturbation/non-perturbation, no post-hoc tests are performed. Four conventional statistics for MANOVA are typically reported: Pillai’s Trace, Wilks’ Lambda, Hotelling-Lawley Trace, and Roy’s Greatest Root [283]. Figure 58 reports the four multivariate tests mentioned, as it was reported in SPSS. All multivariate tests show a significant effect of the perturbation/no-perturbation factor difference on the means of the muscular, kinematic, and electrodermal dependent variables. Pillai’s Trace = .725, F = 25.059, df = (104), p <.001. This means that there is a statistical effect of the perturbation factor on all independent variables analyzed multivariately, which has to be investigated with multiple follow-up ANOVAs. Pillai’s trace test statistic gives more robust results in the case of homogeneous and heterogeneous variances, in unbalanced samples. Regarding the one-way MANOVA results, Can Ateş et al. [284] investigate how Wilks’ lambda, Pillai’s trace, Hotelling’s trace, and Roy’s largest root test statistics can be affected when the normal and homogeneous variance assumptions of the MANOVA method are violated. Since the observations obtained without perturbation are 553 and the observations with perturbation account for 543 observations for each dependent variable, indicates an unequal sample size. This is not a mandatory assumption to proceed with variance analysis but there are two potential problems that can arise: reduced statistical power and reduced robustness to unequal variance. In practice, after detecting a statistically significant multivariate effect, researchers frequently resort to univariate ANOVAs and multiple pairwise t-tests to explain what is accounting for these group differences. Huberty and Morris [285] indicated that 96% of MANOVA follow-ups were univariate ANOVAs. In educational research, 84% of studies followed that strategy as well [286]. The popularity of the multivariate–univariate approach is not surprising as several leading textbooks on multivariate methods have suggested this strategy [287,288]. The MANOVA test functions as a gatekeeper and is used with the hopes of controlling for Type I error rates when analyzing multiple dependent variables [289]. 113 CHAPTER 6. STATISTICAL ANALYSIS Figure 58: Four conventional statistics for MANOVA reported: Pillai’s Trace, Wilks’ Lambda, Hotelling-Lawley Trace, and Roy’s Greatest Root. 6.2.2 one-way ANOVA A one-way ANOVA was performed to evaluate the effect of visual perturbations (labels) on muscle activation, kinematic parameters, CoM velocity and galvanic skin response dependent variables. One may recall the statistical variables calculated for the metrics available from the schematic in Figure 52. Again it is emphasized that only the metrics ”average - AVG”, ”standard deviation - STD”, ”minimum - MIN”and ”maximum - MAX”are in use. Overall, there are 32 variables from muscle activity, 48 variables from accelerometry and gyroscopy of the pelvis and sternum in the 3-axis, 12 variables from CoM velocity metrics, and 9 variables from galvanic skin response. A one-way ANOVA revealed that there was a statistically significant difference in the dependent variables listed in Table 20 between at least two groups. F-value and p-value are presented in Table 20 for each dependent variable. The table is arranged to show the variables with the most significant p-value first. There was no significant difference in the dependent variables showing the p-value with red text color (p>0.05). Purple corresponds to CoM velocity variables, blue to muscle variables, green to gyroscopy variables, and orange to accelerometry. (F(between groups df, within groups df) = [F-value], p = [p-value]) As an example of reporting, a one-way ANOVA was performed to compare the effect of visual perturbations (different labels) on Right Tibialis Anterior average muscle activation (RTAAVG). A one-way ANOVA revealed that there was a statistically significant difference in Right Tibialis Anterior average muscle activation between at least two groups (F(35,1060) = 11.181, p<0.001). The variables referring to the galvanic response of the skin that reflect physiological activity related to anxiety and fear of falling did not obtain significant values. Thus, were not included in the aforementioned table. 114 CHAPTER 6. STATISTICAL ANALYSIS Table 20: One-way ANOVA results: F-value and p-value. The order of this table is done in order to understand which variables had the highest F-value, that is, which were most influenced by the situations where there were visual disturbances. Additionally, variables are separated by color to get an idea of which groups of variables were most affected. Blue - muscle variables; Green - Gyroscope; Orange - Accelerometer; Purple -CoM Velocity. F-value p-value Independent Variable F-value p-value Independent Variable F-value p-value Independent Variable 44.3876 4.9562E-181 COMVelXMIN 7.398 1.67488E-31 RGMSTD 3.349156 3.9924E-10 StrGyrzAVG 30.3299 2.9712E-134 COMVelXAVG 7.397 1.68877E-31 RTASTD 3.095718 7.0099E-09 StrAccyAVG 22.269 1.3568E-102 StrGyrxSTD 7.307 5.07651E-31 RGMAVG 3.05817 1.0655E-08 StrAcczMIN 20.1081 2.34E-93 StrGyrzMAX 6.589 3.55611E-27 PelGyrzSTD 3.03915 1.3164E-08 PelAccyMIN 19.552 6.33223E-91 PelGyrzMAX 6.083 1.83851E-24 PelAccyMAX 3.017882 1.6667E-08 StrAccyMAX 18.9544 2.7722E-88 PelAccxSTD 5.544 1.40609E-21 PelAcczMIN 3.012572 1.7677E-08 StrAcczSTD 17.8613 2.22145E-83 PelAccySTD 5.367 1.2499E-20 RGMMAX 3.007972 1.8601E-08 StrAccxMIN 14.9473 7.81648E-70 PelGyrxSTD 5.22 7.57E-20 PelGyrxMAX 3.004329 1.9366E-08 StrAccyMIN 12.9882 2.50855E-60 COMVelYMAX 5.003 1.07675E-18 PelAccxMAX 3.00356 1.9532E-08 StrAcczMAX 12.8724 9.36689E-60 PelAcczSTD 4.979 1.45508E-18 LGMSTD 3.00045 2.0216E-08 StrAccxMAX 11.6453 1.26888E-53 StrGyrzMIN 4.744 2.56E-17 StrGyryMAX 2.980041 2.5328E-08 StrAccxSTD 11.18100 2.87403E-51 RTAAVG 4.724 3.25173E-17 COMVelZAVG 2.969282 2.8519E-08 StrAccxAVG 11.035 1.5833E-50 PelGyrzMIN 4.458 8.16166E-16 PelAcczMAX 2.966565 2.9386E-08 StrAccySTD 10.8063 2.33108E-49 LTAMAX 4.417 1.35E-15 StrGyryMIN 2.916839 5.07E-08 PelGyryMIN 10.5536 4.60525E-48 COMVelZSTD 4.292 6.04921E-15 PelAccxMIN 2.741791 3.38E-07 PelGyryMAX 10.4218 2.19047E-47 StrGyrySTD 3.934 4.34E-13 PelGyrxMIN 2.683527 6.2881E-07 PelAccxAVG 9.80508 3.385E-44 PelGyrySTD 3.781 2.66E-12 SCMMAX 2.680819 6.4714E-07 EXTOMAX 9.26007 2.35024E-41 RTAMAX 3.693 7.44721E-12 COMVelYSTD 2.662867 7.8266E-07 StrGyrxAVG 8.7332 1.37279E-38 StrGyrzSTD 3.678 8.9323E-12 LGMMAX 2.660458 8.0286E-07 EXTOSTD 8.64069 4.21946E-38 COMVelXSTD 3.663 1.06239E-11 COMVelYAVG 2.640012 9.9626E-07 COMVelZMAX 8.42246 5.9924E-37 LTASTD 3.619 1.77103E-11 StrGyryAVG 2.5719 2.0341E-06 PelAccyAVG 8.21407 7.60E-36 StrGyrxMAX 3.618 1.79964E-11 STMAX 2.501778 4.2048E-06 COMVelXMAX 7.4828 5.88E-32 StrGyrxMIN 3.596 2.32318E-11 COMVelYMIN 2.415235 1.0169E-05 RFMIN 7.47933 6.13649E-32 LTAAVG 3.531 4.92824E-11 RFMAX 2.400227 1.1833E-05 LGMAVG 3.467 1.03E-10 RFSTD 2.332105 2.3405E-05 STAVG 3.465 1.05448E-10 SCMSTD 2.123562 0.0001764 STSTD 3.366 3.27661E-10 PelGyrzAVG 2.063500 0.00030898 RTAMIN 2.045522 0.00036468 PelGyryAVG 2.03077 0.0004175 RFAVG 1.959229 0.00079695 RGMMIN 1.906913 0.00126532 PelAcczAVG 1.894578 0.00140912 LTAMIN 1.572945 0.01891391 StrAcczAVG 1.558554 0.02101814 PelGyrxAVG 0.955329 0.54398981 COMVelZMIN 0.836843 0.73760913 LGMMIN 0.709917 0.89614266 STMIN 0.580864 0.97617683 EXTOMIN 0.181758 0.99999998 EXTOAVG 0.14402 1SCMMIN 0.120651 1SCMAVG 6.2.3 Dunnett Post Hoc Dunnett’s t-test with ”No Pertubation”level as control group for multiple comparisons found that the mean values of independent variables in table 21 were significantly different between control group ”No Perturbation”and the perturbations described in the same table. Post hoc test p-values and confidence intervals lower and upper bound can be consulted in Appendix 3. In the same way that ANOVA results were presented, to illustrate the results obtained in the post hoc test, a muscle variable was picked - RTAAVG, which represents the average value of the percentage of activation of the Right Tibialis Anterior muscle during the occurrence of the various perturbations. Table 21 shows the results of this test. The greatest differences between the averages when comparing the control group (no disturbance) with the visual disturbances 115 CHAPTER 6. STATISTICAL ANALYSIS are colored green. The red colors indicate the opposite, meaning that there was minimal interference of the visual disturbance on the average activation of this muscle. From the mentioned table, it is apparent that the visual disturbance that most affects the contraction of this muscle is Roll Indoor 2 CCW30. For illustrative purposes, Figure 59 represents a subject during the protocol experiencing this perturbation. The labels in bold represent the other most effective perturbations. 116 CHAPTER 6. STATISTICAL ANALYSIS Table 24 presents the results of Dunnett’s test, which detects which visual perturbations, compared to the ”No Perturbation”control, caused the variable CoMVelX to show the greatest difference in the means, representing a greater influence of the perturbations that have a larger (I-J) Mean Difference value. In this case, the variable represents the average velocity of the CoM in the X-axis, which depicts faster translations of the projection of the CoM along the AP direction. As expected, the perturbations that appear in bold in Table 24 are those that have a mean difference at the 0.05 level. That is, they were strongly influenced by these perturbations. The ones with a larger value of mean difference are the perturbations in Roll and Pitch. It is natural that perturbations that cause the participant to incur a faster gait or make abrupt stops using the hip strategy, are those that most manipulate the variable CoM velocity X-axis, on average. This happens in all Roll perturbations that, despite being perturbations that try to induce a lateral drop, cause the compensatory reaction of rapid oscillations in the horizontal plane during the perturbed gait. This inference will be demonstrated via a sequential display of frames. Figure 62 shows a sequence during the experimental protocol of a subject experiencing a Roll perturbation. Figure 62: Participant experiencing a Roll perturbation. It is noticeable that the participant, trying to maintain balance, is using a hip strategy and foot positioning that causes a velocity increase on the X-axis CoM. On the other hand, from the same table (Table 24 it follows that a perturbation ML Axis Translation has no statistical significance in this variable. This is because it induces a lateral sway that is not as abrupt as Roll. From Table 24 it appears that in the Y-axis or ML plane direction, although there was a significant effect of the perturbations on the average CoM velocity, it had much lower values than in the previous analysis in the X-axis. With the exception of the maximum value. This means that the strategies employed by the participants were sufficient to react to the visual perturbation and did not enter into CoM velocities in the ML direction that could lead to a fall. However, from the maximum value, inferences can be made that there were quite strong visual perturbations that caused the CoM to reach maximum velocities in this direction. Regulation at hip level become more important in the case of stronger perturbations (hip strategy) [295]. In order to attempt to have a better understanding of the compensatory postural reactions, the 123 CHAPTER 6. STATISTICAL ANALYSIS 3-dimensional accelerometry and gyroscopy variables from the inertial sensor of the pelvis and sternum were used. Table 25 gathers the accelerometry variables. Table 25: ANOVA results - acceletometry variables (pelvis and sternum). F-value p-value Independent Variable (Acc) 18.95441747 2.7722E-88 PelAccxSTD 17.8613447 2.22145E-83 PelAccySTD 12.87239219 9.36689E-60 PelAcczSTD 6.08277026 1.83851E-24 PelAccyMAX 5.544332578 1.40609E-21 PelAcczMIN 5.003434221 1.07675E-18 PelAccxMAX 4.458436344 8.16166E-16 PelAcczMAX 4.292283176 6.04921E-15 PelAccxMIN 3.095717802 7.00993E-09 StrAccyAVG 3.058169938 1.06551E-08 StrAcczMIN 3.039150338 1.31642E-08 PelAccyMIN 3.01788209 1.66675E-08 StrAccyMAX 3.012572308 1.76773E-08 StrAcczSTD 3.00797208 1.86011E-08 StrAccxMIN 3.004329145 1.93663E-08 StrAccyMIN 3.003559771 1.95319E-08 StrAcczMAX 3.000449732 2.02156E-08 StrAccxMAX 2.980041256 2.53282E-08 StrAccxSTD 2.969281813 2.85192E-08 StrAccxAVG 2.966565049 2.93859E-08 StrAccySTD 2.683527474 6.28808E-07 PelAccxAVG 2.571899896 2.03414E-06 PelAccyAVG 1.906912998 0.001265315 PelAcczAVG 1.572945117 0.018913906 StrAcczAVG It is obvious, by inspecting the table, the first three positions to be occupied by the variables of acceleration in the IMU of the pelvis, namely those representing the standard deviation. Regarding the variables representing mean values, it is noticeable that only the acceleration in the Y-axis of the sternum occupies top positions in the table. Extra attention is required when inspecting the results for variables representing acceleration and rotation of a sensor, since sensor placements may be different. Previously, the CoM directions used the global frame, since it is a calculated variable. The axes of the inertial sensor are as shown in Figure 63. 124 CHAPTER 6. STATISTICAL ANALYSIS Figure 63: Axis orientations for Pelvis and Sternum intertial sensors. 6.2.5 The most effective visual disturbances In the muscular variables, the muscles of the lower leg stand out. Of these variables, which visual disturbances most influenced their variability are identified. To be representative, the average and maximum values will be inspected, since the analysis is dealing with muscle activations. The average value represents the amount of time that the muscle is recruited and the maximum value reached is representative of how strong the disturbance was. About the average activation of the Tibialis Anterior, the visual disturbances that had the most influence on this activation were Roll perturbations and Trip, in a general way. Of the Roll perturbations, in the muscles of both legs, the Roll CCW30, CCW20 and CW10 were the most effective. Additionally, the perturbation Window Roof Beam Walking had a high statistical significance regarding the average activation of the RTA and LTA. Regarding maximum values of Tibialis Anterior in both legs, there are impressive values in maximum muscle activation in vertigo situations. Especially in the two situations where the participant has the notion that he can fall on both sides. Regarding the average activation of the Right Medial Gastrocnemius, disturbances in the Roll in general were the most influential. Roll CCW20 and CCW30 clearly stand out. Interestingly, two pitch perturbations (Pitch Bathroom and Pitch Fridge) were also among the most effective at activating this dominant leg muscle. Trip also obtained relevant significance. Strangely, the average activation of the Left Gastrocnemius Medial of the non-dominant leg was only influenced by the Free Fall and Trip perturbations, comparable to the contact of the right leg. On the maximal values of Gastrocnemius Medial Head, the right leg muscle did not exhibit significant differences for any visual perturbations. In the left leg, an interesting result shows that both perturbations in the AP direction: forward and backward, had a great influence on the maximum contraction value of this muscle of the non-dominant left leg. Additionally, in the Bedroom 125 CHAPTER 6. STATISTICAL ANALYSIS Syncope perturbation there is also a large influence on the maximum contraction value variable of the LGM muscle. The focus of the discussion is now the visual perturbations that are most effective at introducing postural reactions into the kinematic parameters. To deduce the strength of the visual disturbance on the kinematic changes, it seems appropriate to analyze which perturbation had the greatest influence on the variables representing the CoM velocity: average and maximum value. In the mean value change of the velocity of the CoM in AP direction, it can be stated that all Roll perturbations proved to be quite homogeneously influential on this parameter, i.e., only Roll CCW30 and Roll CW10 stood out slightly in terms of significance. Apart from these, all perturbations in the Pitch and Translation AP were highly significant in introducing velocity into the CoM in the AP direction, as one would expect, since these perturbations try to induce a forward or backward drop. In relation the maximum value of the CoM velocity in the AP direction, the only perturbation that stands out is Bedroom Syncope, which indicates the effectiveness and strength of the perturbation to induce aneroposterior sway. Following the same reasoning and sequence of the previous paragraph, the present will analyze the influence of visual disturbances on the speed of CoM (average and maximum value), this time in the ML direction. The perturbations with the greatest influence on the mean value were the AP translations, the Free Fall and the Trip. On the maximum value, there was a general influence of the Roll perturbations and a main influence of the AP translation backward perturbation. All Pitch perturbations are effective at producing a change in the maximum value of the CoM velocity in the ML direction, as well as Free Fall and Trip. Finally, regarding the analysis of the velocity of the CoM in the Z-axis, the Bedroom Syncope perturbation proved to be highly influential of this parameter both in average and maximum values, showing its effectiveness and strength in inducing hip strategy, making the participant abruptly lower the center of mass to regain equilibrium, as happened with AP Translation in both directions. 126 C h a p t e r 7 Conclusion As presented in the introductory chapter, preventing the occurrence of falls and detecting them prematurely, and consequently minimizing the health and economic harms arising therefrom, is the main motivation of this work. To achieve this purpose, it is necessary to implement both fall prediction and fall detection systems and algorithms. Due to the low frequency of fall occurrence, these algorithms are often implemented with data from simulated falls in a controlled laboratory environment. There is effectively a lack of datasets containing real falls, and very few data available. The existing ones contain only experimental data from an inertial sensor. During an initial literature review, this work was contextualized in the advances that have been developed with virtual reality in the field of neurorehabilitation and with elderly subjects. A brief market analysis was done to highlight the emergence of this technology. It was possible to separate these interventions into immersive and non-immersive. Due to the increasing affordability of acquiring a fully immersive device, a headset, there has been an escalating number of studies of human balance with this apparatus. For this reason, in the third chapter a literature search is conducted, following a systematic approach, focused entirely on virtual reality delivered via head-mounted display. These studies had to introduce visual disturbances and examine the compensatory reactions of the participants. A survey was made of the most commonly used virtual reality equipment, the visual stimuli used and the experimental protocol in which they are inserted - duration of the intervention, pauses and habituation time. The most common types of participants in these interventions were analyzed, as well as the objectives of the studies, the most commonly used sensor systems, and finally, the metrics obtained and the evaluated outcomes. An additional section was included on the use of electromyography in these studies to support the choice of sensor location EMG in Chapter 5. It was found that the vast majority of studies employed a visual perturbation only and often in conjunction with mechanical perturbations. Furthermore, the use of just one 127 CHAPTER 7. CONCLUSION sensor system is a common practice. As shown, compensatory reactions are dependent on the directions and intensities of the perturbations. For this reason, there was an urgency to create multiple visual perturbations and record the compensatory reactions at various levels: physiological and kinematic. Therefore, to address the gaps identified in the literature, a proposal was presented to approach the problem. This proposal was based on the design of a virtual environment endowed with a high level of realism due to its ecological validity and presence characteristics, conferred by a home-living paradigm very close to reality. In this virtual environment, animations were created that materialize the visual perturbations. These visual disturbances are randomly presented to the subjects in an experimental protocol that will allow data collection for the construction of the dataset. This dataset intends to contain data very close to those collected in real situations. The limitations encountered in the execution of the protocol were the lack of physical space that limited the movement of the subject to a back and forth path. Furthermore, due to the nature of the triggering mechanism of the animations, it was impossible to apply all the perturbations at exactly the same gait cycle for all subjects. On the other hand, the statistical analysis covers this deficit. This statistical analysis was presented in the sixth chapter as a way to verify if the visual perturbations introduced variations in the means of the dependent variables in comparison to the no-perturbation conditions. These differences were first assessed with a multivariate analysis of variance. To understand which variables changed the most with the introduction of visual disturbances, we conducted follow-up ANOVAs. These analysis of variance showed that variables of muscle groups strongly linked to balance restoration after external perturbations were the most statistically significant. The same was true for kinematic variables that mirror the loss of balance. With these results and strong correlations with compensatory reactions resulting from physical perturbations, it is possible to claim that the visual perturbation alone was sufficient to induce the participant into imbalance. With this, RQ1 is answered affirmatively: ”Can a virtual reality headset introduce imbalances through visual perturbations? Can they cause postural reactions typical of a fall?”The results presented in Chapter 6 statistically support that it is possible and effective to induce postural reactions similar to a real-world fall by introducing visual disturbances via an HMD. The RQ2, ”Which visual perturbation challenged the participants’ balance the most?”was answered in the same chapter 6. The most effective perturbations to induce imbalances are rotations in the Roll, in the CCW direction and with amplitudes of 20 and 30 degrees. The visual disturbance that tries to simulate a syncope, called Bedroom Syncope, also excelled in the effectiveness and strength of inducing imbalance, as well as AP Translation perturbations, albeit to a lesser extent. The RQ3, ”Which virtual situation placed the participant over the most anxiety?”was not possible to be answered since the electrodermal variables did not show sufficient statistical relevance to infer in which situations the participants were under greater anxiety and stress. Regarding RQ4, ”What influence does the real-time representation of the avatar have in situations of virtual heights?”, with the statistical tests performed, a significant difference in postural reactions with and without avatar was not detectable. Whenever vertigo situations had an influence on parameters indicating loss of balance, they were with 128 CHAPTER 7. CONCLUSION similar values in situations with and without avatar. Finally, for the RQ5 ”Is it possible for a habituation phenomenon to occur to visual disturbances?”, no reduction in reactions was detected throughout the protocol. However, the answer to this research question lacks a statistic analysis that includes the exposure time as a variable. 7.1 Future Work During the execution of the project, flaws were identified that could be improved in future work that will surely be scientifically relevant. An appointment for future work would be to make labeling fully automated. In situations like Trip it is possible, detecting the shock through the IMU placed on the foot. In other situations, a simple analysis of gait parameters such as toe off and heel strike can be taken to detect the second step and there introduce the perturbation, according to the participant’s gait steps. In this way, labeling would be fully automatic and in real time. The allocation of a control group would not bring advantages in answering the research questions. In the future, this protocol could be labeled as balance training for fall prevention if the main hypothesis that visual disturbances are sufficient to cause balance loss leading to posture recovery mechanisms is proven. In that framework, the control group would make sense to evaluate the efficiency of the perturbation-based balance training protocol. Once answered affirmatively to RQ1 and specified the visual disturbances that were more effective for certain muscle groups or kinematic parameters, there are conditions to create a more specific protocol with fewer disturbances to introduce as a balance training tool. Fundamentally, the objective would be to subject elderly people to this training tool in the future and evaluate its effectiveness in preventing falls. 129 Bibliography [1] P. Tasheva, P. Vollenweider, V. Kraege, G. Roulet, O. Lamy, P. Marques-Vidal, and M. Meán. “Association Between Physical Activity Levels in the Hospital Setting and Hospital-Acquired Functional Decline in Elderly Patients.” In: JAMA network open 3.1 (Jan. 2020). issn: 2574-3805. doi: 10.1001/JAMANETWORKOPEN.2019.20185. url: https://pubmed.ncbi.nlm.nih.gov/ 32003817/. [2] J. E. Gaugler, S. Duval, K. A. Anderson, and R. L. Kane. “Predicting nursing home admission in the U.S: A meta-analysis.” In: BMC Geriatrics 7.1 (June 2007), pp. 1–14. issn: 14712318. doi: 10.1186/1471-2318-7-13/FIGURES/3. url: https://bmcgeriatr.biomedcentral. com/articles/10.1186/1471-2318-7-13. [3] Falls. url: https://www.who.int/news-room/fact-sheets/detail/falls (visited on 01/04/2021). [4] B. Moreland, R. Kakara, and A. Henry. “Trends in Nonfatal Falls and Fall-Related Injuries Among Adults Aged ≥65 Years — United States, 2012–2018.” In: MMWR. Morbidity and Mortality Weekly Report 69.27 (July 2020), pp. 875–881. issn: 0149-21951545-861X. doi: 10.15585/MMWR. MM6927A5. url: https://www.cdc.gov/mmwr/volumes/69/wr/mm6927a5.htm. [5] A. Bergland. “Fall risk factors in community-dwelling elderly people.” In: Norsk Epidemiologi 22.2 (2012), p. 1. url: www.profane.org.uk. [6] G. Bergen, M. R. Stevens, R. Kakara, and E. R. Burns. “Understanding Modifiable and Unmodifiable Older Adult Fall Risk Factors to Create Effective Prevention Strategies.” In: American journal of lifestyle medicine 15.6 (Nov. 2019), pp. 580–589. issn: 1559-8284. doi: 10 . 1177 / 1559827619880529. url: https://pubmed.ncbi.nlm.nih.gov/34916876/. [7] A. Steiner. “Effects of physical activity on postural stability.” In: Age and ageing 30 Suppl 4.SUPPL. 3 (2001), pp. 33–39. issn: 0002-0729. doi: 10.1093/AGEING/30.SUPPL_4.33. url: https: //pubmed.ncbi.nlm.nih.gov/11769787/. [8] L. Z. Rubenstein. “Falls in older people: Epidemiology, risk factors and strategies for prevention.” In: Age and Ageing 35.SUPPL.2 (2006), pp. 37–41. issn: 00020729. doi: 10.1093/ageing/ afl084. 130 BIBLIOGRAPHY [9] R. Igual, C. Medrano, and I. Plaza. “Challenges, issues and trends in fall detection systems.” In: BioMedical Engineering Online 12.1 (2013). issn: 1475925X. doi: 10.1186/1475-925X-12-66. [10] R. Rajagopalan, I. Litvan, and T. P. Jung. “Fall prediction and prevention systems: Recent trends, challenges, and future research directions.” In: Sensors (Switzerland) 17.11 (2017), pp. 1–17. issn: 14248220. doi: 10.3390/s17112509. [11] M. Hemmatpour, R. Ferrero, B. Montrucchio, and M. Rebaudengo. “A review on fall prediction and prevention system for personal devices: Evaluation and experimental results.” In: Advances in Human-Computer Interaction 2019 (2019). issn: 16875907. doi: 10.1155/2019/9610567. [12] S. S. Khan and J. Hoey. “Review of fall detection techniques: A data availability perspective.” In: Medical Engineering and Physics 39 (2017), pp. 12–22. issn: 18734030. doi: 10 . 1016 / j . medengphy.2016.10.014. arXiv: 1605.09351. url: http://dx.doi.org/10.1016/j. medengphy.2016.10.014. [13] E. Stack. “Falls are unintentional: Studying simulations is a waste of faking time.” In: Journal of Rehabilitation and Assistive Technologies Engineering 4 (2017), p. 205566831773294. issn: 2055-6683. doi: 10.1177/2055668317732945. [14] M. Kangas, I. Vikman, L. Nyberg, R. Korpelainen, J. Lindblom, and T. Jämsä. “Comparison of real-life accidental falls in older people with experimental falls in middle-aged test subjects.” In: Gait & Posture 35.3 (Mar. 2012), pp. 500–505. issn: 0966-6362. doi: 10.1016/J.GAITPOST. 2011.11.016. [15] E. Casilari, J. A. Santoyo-Ramón, and J. M. Cano-García. “Analysis of public datasets for wearable fall detection systems.” In: Sensors (Switzerland) 17.7 (2017). issn: 14248220. doi: 10.3390/ s17071513. [16] X. Yu, J. Jang, and S. Xiong. “A Large-Scale Open Motion Dataset (KFall) and Benchmark Algorithms for Detecting Pre-impact Fall of the Elderly Using Wearable Inertial Sensors.” In: Frontiers in Aging Neuroscience 13 (July 2021), p. 399. issn: 16634365. doi: 10.3389/FNAGI.2021. 692865/BIBTEX. [17] R. N. Ferreira, N. F. Ribeiro, and C. P. Santos. “Fall Risk Assessment Using Wearable Sensors: A Narrative Review.” In: Sensors 22.3 (Feb. 2022). issn: 14248220. doi: 10.3390/S22030984. [18] F. Di Nardo, A. Mengarelli, E. Maranesi, L. Burattini, and S. Fioretti. “Assessment of the ankle muscle co-contraction during normal gait: A surface electromyography study.” In: Journal of Electromyography and Kinesiology 25.2 (Apr. 2015), pp. 347–354. issn: 18735711. doi: 10.1016/ J.JELEKIN.2014.10.016. 131 BIBLIOGRAPHY [19] S. H. Roy, M. S. Cheng, S. S. Chang, J. Moore, G. De Luca, S. H. Nawab, and C. J. De Luca. “A combined sEMG and accelerometer system for monitoring functional activity in stroke.” In: IEEE Transactions on Neural Systems and Rehabilitation Engineering 17.6 (2009), pp. 585–594. issn: 15344320. doi: 10.1109/TNSRE.2009.2036615. [20] M. Slater. “Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments.” In: Philosophical Transactions of the Royal Society B: Biological Sciences 364.1535 (2009), pp. 3549–3557. issn: 14712970. doi: 10.1098/rstb.2009.0138. [21] A. M. Gonzalez and A. B. Raposo. “Fall risk analysis during VR interaction.” In: Proceedings - 19th Symposium on Virtual and Augmented Reality, SVR 2017 2017-Novem (2017), pp. 18–28. doi: 10.1109/SVR.2017.11. [22] HTC Vive Virtual Reality Headset Review | Time. url: https://time.com/4280792/htcvive-review/ (visited on 12/13/2020). [23] VIVE Pro | VIVE United States. url: https://www.vive.com/us/product/vive-pro/ (visited on 06/25/2022). [24] I. Sutherland. “The ultimate display.” In: (1965). [25] A. Yogasingam. “Virtual Boy: Nintendo’s (red and) black sheep.” In: Electronic Engineering Times 1559 (Apr. 2009), pp. 5–6. issn: 01921541. [26] Virtual Boy | Nintendo | Fandom. url: https://nintendo.fandom.com/wiki/Virtual{\_ }Boy (visited on 06/27/2022). [27] Virtual Reality Market Size and Share Forecast Report, 2029. url: https://www.fortunebusinessinsights. com/industry-reports/virtual-reality-market-101378 (visited on 06/27/2022). [28] T. S. Mujber, T. Szecsi, and M. S. Hashmi. “Virtual reality applications in manufacturing process simulation.” In: Journal of Materials Processing Technology 155-156.1-3 (Nov. 2004), pp. 1834– 1838. issn: 09240136. doi: 10.1016/j.jmatprotec.2004.04.401. [29] M. Slater and S. Wilbur. “A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments.” In: Presence: Teleoperators and Virtual Environments 6.6 (Dec. 1997), pp. 603–616. issn: 10547460. doi: 10.1162/pres.1997.6.6.603. url: https://www.mitpressjournals.org/doi/abs/10.1162/pres.1997.6.6.603. [30] K. Bhagat, W.-K. Liou, and C.-Y. Chang. “A cost-effective interactive 3D virtual reality system applied to military live firing training.” In: Virtual Reality 20.2 (2016), pp. 127–140. doi: 10.1007/ s10055-016-0284-x. 132