SUN Accessibility Pilot: Extended reality for people with serious mobility and verbal communication diseases

SUN Accessibility Pilot: Extended reality for people with serious mobility and verbal communication diseases L. Greci*. V. Croce*. L. Corsano*. F.Bosco*. *Engineering Ingegneria Informatica spa, Rome, Italy (e-mail: [email protected]). Abstract: In recent years, Extended Reality (XR) technology has emerged as a potent instrument, inducing transformative effects across various industries by providing immersive experiences and expanding accessibility for individuals encountering physical impediments. The merging of XR technology with hyper-realistic avatars and innovative interfaces exhibits significant potential in reshaping rehabilitation methodologies and elevating the quality of life for individuals afflicted with profound mobility and verbal communication diseases. Within this framework, the Social and hUman ceNtered XR (SUN) project is dedicated to investigating and advancing XR solutions that seamlessly integrate the physical and virtual words, with a specific emphasis on the human and social dimensions. The project endeavors to overcome challenges associated with hyper-realistic human avatar, constraints posed by end-user device resources, the incorporation of wearable haptic technologies to augment physical interactions and the exploration of gaze-based and gesture-based interaction modalities. This work will describe the architecture of the pilot 3 of the SUN project, XR for people with serious mobility and verbal communication diseases, focusing on the implementation of the XR Environment. Keywords: mobility and verbal communication diseases, virtual reality, sense of presence, social interaction, interaction. 1. INTRODUCTION Extended Reality (XR) technologies play a crucial role in enhancing accessibility by creating immersive digital environments tailored for individuals with physical, sensory, or cognitive impairments (Witmer & Singer, 1998). Through the integration of adjustable interfaces, voice commands, and personalized interactions, XR designs environments that cater to the diverse needs of each user (Gong et al., 2021). The focus on inclusivity empowers users to customize their digital experiences (Rivera et al., 2022), breaking down traditional accessibility barriers (Liubogoshchev et al., 2021). In communication disorders, XR serves as a catalyst for alternative communication methods, employing virtual avatars, gestures, and gaze-based interactions (Park & Kim, 2022). Avatars in XR become dynamic embodiments of selfexpression, allowing users to navigate the virtual space and convey emotions beyond the constraints of traditional communication (Nowak & Fox, 2018). Gestures and gazebased interactions amplify the communicative power of XR, offering a liberating platform for those with communication disorders to express themselves meaningfully (Ghasemi & Jeong, 2022). Moving into social interaction, XR becomes a transformative tool for individuals grappling with impairments, combating social isolation (Matthews et al., 2020) by creating dynamic virtual spaces. XR's capacity to transcend physical limitations provides a platform where users can engage, connect, and collaborate without constraints (Zweifach & Triola, 2019). These virtual environments, designed to accommodate diverse needs and preferences, empower individuals to customize avatars, interactions, and environments for a personalized and inclusive experience (Visconti et al., 2023). The immersive nature of XR social environments (Spittle et al., 2023) creates a sense of presence (Ong et al., 2021), allowing users to feel genuinely connected despite physical distances (Matthews et al., 2020). XR evolves into a social enabler, empowering individuals with impairments to actively shape and contribute to a more inclusive and connected social fabric (Ghasemi & Jeong, 2022). Moreover, the intricate XR experiences cater to individuals with sensory impairments, emphasizing a design philosophy focused on eliciting specific sensory responses. The auditory dimension is carefully crafted with soundscapes designed to convey information, atmosphere, and emotion, enriching the overall sensory journey (Santoso et al., 2022). Visual feedback considers graphic design and color elements as functional components, aiding navigation, and communication for individuals with sensory impairments (Jones et al., 2020). The tactile dimension introduces haptic feedback and interfaces, allowing users to feel and interact with the virtual environment, creating a more holistic sensory journey (Bohil et al., 2011). In essence, XR technologies redefine the landscape of accessibility, communication, and social interaction for individuals with impairments (Venkatesan et al., 2021). Beyond the creation of immersive environments, XR represents a paradigm shift in digital inclusivity, breaking down traditional barriers and fostering a more accessible and equitable digital future for all. The transformative potential of XR extends into communication disorders, combating social isolation, and catering to sensory impairments, showcasing its profound impact on creating a more inclusive and enriching digital environment (Lukava et al., 2022). 2. SUN Accessibility PILOT: XR FOR PEOPLE WITH SERIOUS MOBILITY AND VERBAL COMMUNICATION DISEASES Individuals facing various motor disabilities or recovering from strokes often encounter significant challenges in interpersonal communication and addressing essential needs (O’Connell et al., 2001). This research project aims to confront this challenge by establishing a specialized communication pathway for such individuals, introducing the capacity to interact with specific social cues and translating them into clear communication or actions. The proposed solution relies on leveraging residual abilities, assigning them communicative significance, and employing avatars within a virtual environment as necessary support. The emphasis is on creating a solution enabling the design of person-specific extended reality interactions, whether at home, in the workplace, or at school. This involves the development of innovative multi-user virtual communication and collaboration solutions that offer cohesive multisensory experiences and effectively convey pertinent social cues. Successful implementation of the pilot scenario would empower individuals with communication and motor disabilities to engage with friends and family, realistically interacting in a virtual environment. In this setup, a person with communication and motor disabilities is represented by an avatar interacting with others in a virtual environment. Concurrently, the individual perceives themselves to be present in the same environment as others and can interact with new interfaces provided by the project, known as Social and hUman ceNtered XR (SUN). The project will develop a new generation of non-invasive bidirectional body-machine interfaces (BBMIs), which will allow a smooth and very effective interaction of people with different types of sensory-motor disabilities with virtual reality and avatars. Furthermore, the solution will offer sensory feedback to users, conveying information about their movements and interactions with avatars in the virtual environment. This will be achieved with wearable devices and small actuators providing haptic and thermal feedback. This holistic approach strives to markedly improve the communication and interaction abilities of individuals grappling with both communication and motor disabilities. 3. USE CASE DESCRIPTION Francesco is 37 years old and is affected by cerebral palsy and is tetraplegic. He can’t speak or communicate and has severe limitations on arm and leg movement, making him bedridden and precluding engagement in conventional physical activities. His constrained physical state results in infrequent interactions with family and friends, leaving him confined to his bed, subject to persistent feelings of boredom and isolation. The SUN XR platform is introduced as a potential remedy for Francesco's situation. This system employs a virtual reality headset in conjunction with Surface electromyography (SEMG) sensors to translate residual arm muscles activity into corresponding virtual movements. Additionally, the platform simulates places that Francesco knows and loves, like his daughter's house or a beloved park, giving him a way to interact and explore these places. Real-life family and friends assume control of avatars within this virtual environment, fostering interactive experiences for Francesco. Furthermore, thermal feedback is incorporated into the system, augmenting sensory stimuli, and contributing to a more immersive and emotionally enriching escape from the constraints of his physical incapacitation. By utilizing alternative augmentative communication (AAC) and using BBMIs and/or gaze as input to select symbols, Francesco can communicate with his family and friends (Larco et al., 2021), forming sentences or expressing feelings. Figure 1 illustrates how users, using AAC, can form a phrase by selecting picture boards. Figure 1 An example of sentence formed using AAC. 4. PILOT ARCHITECTURE The architecture for use case of the Pilot is depicted in Figure 2. The Assistive Device, XR Environment (XRE), and 3D Assets are its main building blocks. The first block aims to develop the BBMIs and the thermal feedback device. Using arrays of printed electrodes and inertial sensors, to record muscular activity, the former will decode motor information. Every muscle contraction, detected by SEMG, is translated into a VR controller input. The latter is a device that can simulate the temperature of the digital items populating the XRE. It can be used to give the avatar's touch some warmth, which will heighten the experience's sense of presence. The data exchange between the BBMIs, the thermal feedback device and the XRE is enabled by a Python server. The core of the Pilot is the XRE, implemented using the Unity3D framework. Unity3D was chosen as the development platform for its versatility, rich content creation tools, support for virtual reality, and a strong community. The XRE comprises three modules implemented ad hoc to manage its behavior: Manager Input, Core Logic, and Manager Avatar. For deployment on the Meta Quest Head-Mounted Display (HMD) and to manage multi-users and live chat, thirdparty libraries were used, including Meta SDK, Netcode, and Vivox. Figure 2 XRE architecture The Manager Input component is designed to oversee and control the input mechanisms within the XRE. This managerial function encompasses the coordination and processing of various input sources, such as user interactions, gestures, or commands, to facilitate a seamless and responsive user experience within the virtual space. The role of the Manager Input is crucial in interpreting and translating user inputs into meaningful actions or responses, contributing to the overall functionality and interactivity of the XRE. The Core Logic governs the underlying logic and operations within the XRE. The Core Logic is responsible for managing and coordinating essential processes that contribute to the overall functionality and behavior of the virtual space. This module handles critical functions such as scene rendering, spatial tracking, user interactions, and overall system behavior. It serves as the backbone of the system, ensuring the seamless integration of various elements to create a cohesive and immersive virtual experience. The Core Logic is designed to process data, execute algorithms, and maintain the internal coherence necessary for a responsive and realistic simulation within the virtual environment. The Manager Avatar (MA) plays a pivotal role in managing the lifelike presence and actions of avatars, contributing to the overall immersion and interaction within the VR system. The MA is responsible for handling various aspects related to avatars, including their instantiation in the XRE, movement, animations interactions, and synchronization with real-world users or other virtual entities. The MA ensures that the behavior of avatars within the XRE corresponds with user inputs and commands, providing a consistent and responsive representation of users in the virtual space. The third block is a repository of 3D models used to create and populate the VE depending on the situation the user wants to live. 5. IMPLEMENTATION The XRE is primarily designed for two user categories: people with motor and communication disorders and ablebodied relatives. For the latter, the interactions with the XRE are handled by the Input Manager, using the Meta SDK. This SDK provides modular and flexible components to implement a wide range of interactions, including those involving hands and controllers. For the former, a customized interaction system is necessary due to their clinical condition. Instead of using their hands or controllers to interact with the XRE, they will utilize data coming from the BBMIs and the gaze. The exchange of data between the XRE and the BMMIs is facilitated through a client/server system utilizing socket-based data streaming. The Input Manager receives an array of bytes and decodes it into a string. Each string corresponds to a specific action to be performed within the XRE. The Core Logic is the component responsible for determining how users will interact with the XRE. While ablebodied users utilize the HMD and controllers or hand gestures to navigate, interact, and explore within the XRE, new User Interface (UI) and Interaction Mode have been designed to address limitations faced by users with restricted input capabilities. The UI (figure 3) provides four macro categories enabling the interaction with the XRE. Each category is triggered by the string (choice 1, 2, 3 and 4) generated by the Input manager whenever data from the BBMIs is received. Figure 3 UI flow allowing impaired users to interact with XRE. The functionality of the "Explore" category within the interface empowers users with the ability to navigate through the immersive environment. To each of the incoming BMMIs input is assigned a specific direction, thereby facilitating the fluid movement of the avatar within the virtual environment. The assigned directions provide a comprehensive means for users to interact and engage with the environment, fostering a rich and immersive experience within the XRE. The “Actions” category empowers users with the seamless ability to interact with objects and manipulate their virtual surroundings. This category serves as a gateway to a myriad of immersive experiences, allowing users to move beyond passive observation and actively engage with the digital environment. For instance, users can perform actions such as grabbing a book and read it, demonstrating the versatile range of interactions facilitated by this category. The "Interaction" category is pivotal for fostering dynamic engagement and social interaction amongst users. Through a diverse array of animations curated to convey a spectrum of emotions and gestures, users are afforded a rich tapestry of avenues through which to express themselves. From warm greetings and compliments that set the tone for meaningful encounters that uplift and inspire, to animations tailored to convey agreements and disagreements, users are empowered to express their perspectives and engage in lively discourse. The “Speak” category allows users to engage in conversation with others (see figure 4 where examples are reported). The formulation of coherent and meaningful sentences is facilitated through the utilization of a systematic approach via a multiple-choice menu system. This method allows for the selection of syntactical compositions for sentence construction, commencing, for instance, with the identification of a subject and subsequent linkage to verb and object. Leveraging artificial intelligence (AI), the chosen components are integrated to generate sentence characterized by natural syntax, thereby ensuring linguistic. Finally, by using a Text to Speech service the sentence is converted into naturalsounding speech. Based on the number of BBMIs signals that impaired users can provide, three interaction modes have been implemented to interact with the UI: BBMIs, Gaze + BBMIs, Gaze Only. 5.1 BBMIs This mode is designed for users capable of providing at least four inputs. Each BBMIs signal corresponds to an interaction category. Selecting a category opens a new multiple-choice panel, enabling users to execute the desired action. 5.2 Gaze + Input Mode For users able to provide at least 1 input, the interaction with the UI is point and click, the Gaze is used pointing device and the BBMIs input as click. Movement in the virtual environment is enabled by using Gaze, pointing to the desired location through a cursor. In this mode the Gaze assists users in navigating and UI selection by using confirmation input. For users capable of providing at least one input, interaction with the UI involves a point-and-click mechanism, where Gaze serves as the pointing device and BBMIs input functions as the click action. Navigation within the virtual environment is facilitated by utilizing Gaze, which directs a cursor to the desired location, by clicking the user is teleported to the destination. In this mode, Gaze aids users in navigating and selecting UI elements by providing confirmation input. 5.3 Gaze Only Mode Individuals unable to provide any input will solely rely on Gaze to explore the environment and interact with the UI. A timer is implemented to confirm actions: when the user gazes at a navigation element (e.g. the floor), after maintaining focus for 5 seconds on it teleportation to the gazed location occurs; similarly, after a gaze duration of 2 seconds on a UI interaction element, the corresponding UI action is triggered. 5.4 3D Assets Runtime Importer To dynamically populate the XRE with avatars and 3D assets such as rooms and furniture, a GLTF importer has been implemented. This format was selected as it is the sole format accepted by Unity for importing assets at runtime. Furthermore, GLTF files encapsulate geometry, textures, materials, and animations, leading to diminished data requirements for representing 3D objects compared to alternative formats. The streamlined structure of GLTF files enables swift loading during application execution and requires fewer system resources, thereby enhancing overall application performance. Leveraging the "GLTF fast" package in Unity, 3D models are asynchronously loaded and instantiated as children of a reference Game Object. 5.5 Avatars Avatars, provided by the project’s partner IGOODI, are set up properly with appropriate bones and rigging for animation and blend shapes for lip-syncing. Avatar Movements and Animations are triggered by the Input Manager and managed by the Core Logic. Depending on the input received and the state of the Avatar the proper movements and/or animation will be performed inside the XRE. Figure 4 Example of interaction with the UI to generate a sentence. 5.6 Multi-player and Voice Chat The Extended Reality Environment (XRE) represents a multi-user platform designed to facilitate interconnectedness, interaction, and communication among participants. Central to the seamless operation of this immersive space are advanced technological infrastructures, including the Netcode framework and Vivox, which are both integral components of Unity Game Services. The utilization of these cutting-edge technologies underscores the commitment to enhancing user experience through robust connectivity and real-time communication capabilities. The Netcode framework, renowned for its scalability and efficiency, serves as the backbone for facilitating synchronized interactions among multiple users within the XRE, ensuring a smooth and immersive collaborative experience. Complementing this framework is the incorporation of Vivox, a feature-rich communication service, which empowers users to engage in real-time voice chat, enhancing the sense of presence and social interaction within the virtual environment. Through the strategic integration of these technologies, the XRE not only fosters a sense of community and collaboration but also underscores its potential as a dynamic platform for immersive social experiences in the realm of extended reality. Netcode manages the synchronization of actions across different client sessions, ensuring that changes made by one player are consistently reflected to all others. This provides a seamless experience, even when users are geographically distant. Netcode also includes features for user authentication and connection security, safeguarding against unauthorized access or manipulations. Currently, the usage mode is Peer-ToPeer, but it's also possible to utilize a dedicated server for enhanced performance. Vivox offers bidirectional real-time voice communication features, allowing players to communicate vocally during sessions. It seamlessly integrates with Unity through a set of APIs and SDKs, providing high audio quality for clear communication among users, which is essential for cooperation. Vivox supports the management of vocal sessions to create separate communication channels, such as those for teams or groups of users. Additionally, it includes 3D audio functionality to enhance immersion in the virtual environment. 7. CONCLUSION AND FUTURE WORK The work encompasses a detailed exposition on the architecture and implementation strategies employed in crafting the Extended Reality Environment (XRE). Tailored with precision and innovation, the XRE is purpose-built to cater to the requirements delineated within the Accessibility Pilot of the SUN project. After thorough research and development, the XRE has emerged as a promising solution, offering potential to overcome conventional limitations. In this capacity, it aims to promote inclusivity and improve accessibility within the realm of extended reality experiences. From the conceptualization phase to the final execution, every facet of the XRE's design is scrutinized with a keen eye for optimization and effectiveness, ensuring its seamless integration into the SUN project's overarching objectives. The results of the pilot will undergo comprehensive evaluation through the utilization of focus groups, facilitating interaction analysis to meticulously assess elements such as the sense of immersion, experience sharing dynamics, and levels of engagement. Moreover, the evaluation process will extend to include assessments by professional users, ensuring a holistic appraisal of the pilot's outcomes from diverse perspectives and expertise. 8. ACKNOWLEDGEMENT This paper has received financial support by the Horizon Europe Research & Innovation Programme under Grant agreement N. 101092612 (Social and hUman ceNtered XR - SUN project). REFERENCES Bohil, C., Alicea, B., & Biocca, F. (2011). Virtual reality in neuroscience research and therapy. Nature Reviews Neuroscience, 12(12), 752-762. Gong, L., Fast-Berglund, Å., & Johansson, B. (2021). A framework for extended reality system development in manufacturing. Ieee Access, 9, 24796-24813. Ghasemi, Y. and Jeong, H. (2022). Using gaze-based interaction to alleviate situational mobility impairment in extended reality. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 66(1), 435-439. Jones, P., Somoskeöy, T., Chow-Wing-Bom, H., & Crabb, D. (2020). Seeing other perspectives: evaluating the use of virtual and augmented reality to simulate visual impairments (openvissim). NPJ Digital Medicine, 3(1). Larco, A., Peñafiel, P., Yanez, C., & Luján-Mora, S. (2021). Thinking about inclusion: designing a digital app catalog for people with motor disability. Sustainability, 13(19), Lukava, T., Ramírez, D., & Barbareschi, G. (2022). Two sides of the same coin: accessibility practices and neurodivergent users' experience of extended reality. Journal of Enabling Technologies, 16(2), 75-90. Matthews, B., See, Z., & Day, J. (2020). Crisis and extended realities: remote presence in the time of covid-19. Media International Australia, 178(1), 198-209. Nowak, K. and Fox, J. (2018). Avatars and computer-mediated communication: a review of the definitions, uses, and effects of digital representations on communication. Review of Communication Research, 6, 30-53. O’Connell, B., Hanna, B., Penney, W., Pearce, J., Owen, M., & Warelow, P. (2001). Recovery after stroke: a qualitative perspective. J Qual Clin Practice, 21(4), 120125. Ong, T., Wilczewski, H., Paige, S., Soni, H., Welch, B., & Bunnell, B. (2021). Extended reality for enhanced telehealth during and beyond covid-19: viewpoint. Jmir Serious Games, 9(3), e26520. Park, S. and Kim, Y. (2022). A metaverse: taxonomy, components, applications, and open challenges. Ieee Access, 10, 4209-4251. Rivera, F., Lamb, M., Högberg, D., & Brolin, A. (2022). The schematization of xr technologies in the context of collaborative design. Santoso, H., Wang, J., & Windasari, N. (2022). Impact of multisensory extended reality on tourism experience journey. Journal of Hospitality and Tourism Technology, 13(3), 356-385. Sigafoos, J. and Gevarter, C. (2019). Introduction to the special issue: communication intervention for individuals with complex communication needs. Behavior Modification, 43(6), 767-773. Spittle, B., Frutos-Pascual, M., Creed, C., & Williams, I. (2023). A review of interaction techniques for immersive environments. Ieee Transactions on Visualization and Computer Graphics, 29(9), 3900-3921. Venkatesan, M., Mohan, H., Ryan, J., Schürch, C., Nolan, G., Frakes, D., … & Coşkun, A. (2021). Virtual and augmented reality for biomedical applications. Cell Reports Medicine, 2(7), 100348. https://doi.org/10.1016/j.xcrm.2021.100348 Visconti, A., Calandra, D., & Lamberti, F. (2023). Comparing technologies for conveying emotions through realistic avatars in virtual reality‐based metaverse experiences. Computer Animation and Virtual Worlds, 34(3-4). Witmer, B. and Singer, M. (1998). Measuring presence in virtual environments: a presence questionnaire. Presence Virtual and Augmented Reality, 7(3), 225-240. Wolpaw, J. and Birbaumer, N. (2006). Brain–computer interfaces for communication and control., 602-614. Zweifach, S. and Triola, M. (2019). Extended reality in medical education: driving adoption through providercentered design. Digital Biomarkers, 3(1), 14-21.