A Smart Robotic Platform for Cognitive Rehabilitation Based on Multimodal User State Monitoring

A Smart Robotic Platform for Cognitive Rehabilitation Based on Multimodal User State Monitoring Cristiana Rondonia, Valeria Alessia, Francesco Scotto di Luzioa, Ylenia Crocettob, Simona Abagnaleb, Giulia Martinellib, Sara Della Bellab, Alfonso Voscarellic, Marco Pirinic, Cristina Politob, Anna Estraneob, Loredana Zolloa aResearch Unit of Advanced Robotics and Human-Centred Technologies, Universit´ a Campus Bio-Medico di Roma, Rome, Italy bIRCCS Fondazione Don Carlo Gnocchi, Florence, Italy cKhymeia Group, 35129 Padua, Italy Abstract—Robotic technologies and multimodal monitoring systems have emerged as a promising approach in rehabilitation, offering precise assistance, objective measurement, and enabling adaptive and personalized protocols addressing motor and cognitive functions. This work presents a cognitive rehabilitation platform based on TIAGo robot, operating in joystick mode for compliant human–robot interaction or in monitoring mode for user state tracking. The system combines robotic kinematics, physiological and affective features, orchestrated by a finitestate machine to ensure safe interaction. Preliminary tests with healthy volunteers demonstrated feasibility and usability. Future developments will extend validation to patients and implement real-time adaptation based on arousal and valence, paving the way for personalized and adaptive clinical protocols. Index Terms—robot-aided rehabilitation, cognitive rehabilitation, multimodal monitoring. I. INTRODUCTION Parkinson Disease (PD) is a progressive neurodegenerative disorder characterized by motor symptoms such as rigidity, resting tremor, and bradykinesia, often accompanied by Mild Cognitive Impairment (MCI) affecting attention, memory, executive function, language, visuospatial skills [1]. These impairments impact motor skills and autonomy, limiting the effectiveness of conventional rehabilitation. In this context, robotic technologies ensure precision, repeatability and safety in rehabilitation delivery, enabling clinicians to tailor therapy [2] and objectively measure motor performance. Their versatility also supports the integration of multimodal rehabilitation scenarios that integrate motor and cognitive training. Multimodal interaction enhances motivation and adherence, while offer an enriched clinical experience to the patient. Recent studies show that combining sensory, motor, and cognitive components promotes neuroplasticity and functional recovery [3]. Thus, this work introduces a smart robotic platform This work was partly supported by the European Union – NextGenerationEU under the cascade call “Age-It – Ageing Well in an Ageing Society, OPERA subproject” (PNRR Mission 4, Component 2, Investment 1.3, CUP: B83C22004800006) and the Italian Ministry of Research, under the complementary actions to the NRRP “Fit4MedRob - Fit for Medical Robotics” Grant (PNC0000007). Some of the results of this research were obtained as part of the Doctorate of National Interest in Robotics and Intelligent Machines. conceived to support cognitive rehabilitation through multimodal interaction. It leverages robotics to ensure precise motor engagement, simultaneously embedding tasks that stimulate cognitive domains, creating a unified motor and cognitive rehabilitative experience. Feasibility and usability were preliminary assessed on healthy volunteers. II. MATERIALS AND METHODS A. Experimental setup The proposed platform is composed of three main components (Figure 1): (i) TIAGo robot (PAL Robotics SL, Spain); (ii) the Virtual Reality Rehabilitation System (VRRS, Khymeia Srl, Italy); and (iii) a multimodal monitoring and data fusion architecture. Fig. 1: The experimental setup. TIAGo is the main interaction unit between participants and the rehabilitation system, operating in joystick mode (forces applied to the closed rigid gripper are converted into compliant Cartesian displacements) for motor activation and kinematic analysis of the user or in monitoring mode (RGB-D camera enabling skeletal tracking and facial expression analysis via Mediapipe [3] and Py-Feat toolboxes [4], enabling for the detection of seven categories of emotions. Moreover, the Textto-Speech engine allows the robot to provide instructions or feedback. A Finite State Machine (FSM) coordinates robot behavior across states (Figure 2: Idle, Calibration, Instruction 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629848 211 Delivery, Joystick Interaction, Monitoring, and Emergency Stop), activating the specific subset of robot functionalities, ensuring deterministic and safe operation. For example, a new task from VRRS triggers transitions from Idle to Instruction Delivery and subsequently to Joystick Interaction, while detected abnormal forces switches the FSM to Emergency Stop. Fig. 2: Compact FSM controlling robot behavior The VRRS presents standardized cognitive tasks, completed via touchscreen input (touch mode) or mediated through the robot (joystick mode). In parallel, the robot provides the main interaction, integrating cognitive challenges and physical engagement. Physiological monitoring relies on wearable sensors: BioHarness3 for measurement of heart rate (HR) and respiratory rate (RR), and Shimmer GSR unit for galvanic skin response (GSR). Data are transmitted via Bluetooth Low Energy and synchronized with robotic kinematics (100 Hz) and vision-based features (30 Hz) through ROS middleware. Arousal and valence are estimated online through fuzzy rules applied to baseline-normalized GSR, HR, RR and facial features. All affective, physiological and kinematic data are managed through a RESTful API gateway on the robot, which also manages communication with the VRRS by translating task events into ROS topics and services. B. Experimental protocol Seven healthy volunteers (3M, 4F; 27.2±4.4y.o., six righthanded) partecipated in the study. The participant was seated in front of the TIAGo, with VRRS laterally positioned for optimal accessibility and comfort. After calibrating the robotic arm to individual anthropometry, participants performed eight exercises: four in touch mode and four in joystick mode. Multimodal data (physiological, kinematic, and facial) using the sensors and the robot camera. Session began with a threeminute baseline recording without external stimuli. Afterward, participants completed the System Usability Scale (SUS) and the Technology Acceptance Model (TAM) questionnaires to assess usability and perceived acceptability. III. RESULTS AND DISCUSSIONS Multimodal data collected during the tasks showed an increased HR and RR during task execution,indicating physiological activation associated with engagement in the exercises, while GSR progressively increased independently of the activity phase. As summarized in Table I, the mean values of physiological parameters were comparable across joystick and touch modes. Statistical analysis using the Wilcoxon signedrank test confirmed that these differences were not significant, suggesting that joystick interaction did not impose additional physical or emotional load compared to the touch interface. No significant statistical differences in arousal and valence between joystick and touch (Figure 3), as assessed by the Mode HR [bpm] RR [bpm] GSR [ohm] Joystick 66.8 ±9.9 18.6 ±4.1 2.2 ±1.4 Touch 67.4 ±9.8 16.7 ±3.3 2.4 ±1.6 TABLE I: Mean and standard deviation of physiological parameters during joystick-based and touch-based tasks. Wilcoxon rank-sum test (p>0.05), suggest that joystick interaction does not carry additional emotional burden compared to the touch interface. Regarding face expression analysis, the ”neutral” was predominant, with an average activation values above 0.7in most tasks and high inter-subject variability. Usability outcomes are promising, with a mean score of 74.28 ±11.06 for SUS, exceeding the acceptability threshold of 68. Similarly, the overall TAM of 5.90 ±0.4suggests that the system is perceived positively among users, with a low standard deviations demonstrating the consistency of the responses. Fig. 3: Arousal and valence in joystick and touch mode IV. CONCLUSIONS This work introduces a robotic platform for cognitive rehabilitation enhanced by multimodal monitoring. The study demonstrated the feasibility of combining robotic interaction and psychophysiological sensing, highlighting increased activation in challenging phases and strong individual variability. The results confirm the relevance of personalization in rehabilitation and show that the robot-aided rehabilitation is perceived as comparable to conventional interaction. Future developments will exploit multimodal data to enable adaptive protocols tailored to patient needs, paving the way for its clinical validation. REFERENCES [1] Balestrino, R., and Schapira, A. H. (2020). Parkinson disease. European journal of neurology, 27(1). [2] Banyai, A. D., and Bris ,an, C. (2024). Robotics in physical rehabilitation: Systematic Review. Healthcare, 12(17). [3] Zotey, V., et. al. (2023). Adaptive neuroplasticity in brain injury recovery: strategies and insights. Cureus, 15(9). [4] Lugaresi, C., et. al. (2019). Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172. [5] Cheong, J. H., et. al. (2023). Py-feat: Python facial expression analysis toolbox. Affective Science, 4(4), 781-796. 212