Development of an Augmented Reality-based Framework for Robot-assisted Medical Contact-Rich Tasks

Development of an Augmented Reality-based Framework for Robot-assisted Medical Contact-Rich Tasks Junling Fu1, Afshin Karimi Dastjerdi1, Matteo Di Mauro1, Alessandro Albanesi1, Alberto Redaelli1, Tommaso Scquizzato2, and Elena De Momi1 Abstract— Contact-rich tasks are widely used in various medical applications, such as cardiopulmonary resuscitation (CPR), Ultrasound imaging, and palpation. To improve usability, efficiency, and intuitiveness during robot-assisted, contactrich medical tasks, a novel augmented reality (AR)-based framework has been developed. Specifically, this framework integrates robotic systems to perform the contact-rich tasks, an optical see-through head-mounted display (OST-HMD) for robot teleoperation control, and an external RGB-D camera to acquire the point cloud of the phantom or the patient. Preliminary results indicate the framework’s effectiveness, including remote robotic control, point-cloud acquisition, and force control for medical contact-rich task execution. I. INTRODUCTION Robotic technologies are increasingly adopted in medical procedures that involve frequent and delicate physical interactions, often referred to as contact-rich tasks [1], [2]. Typical examples include palpation, tissue dissection, and cardiopulmonary resuscitation, all of which demand precise force modulation and consistent motion [3]. In the context of cardiopulmonary resuscitation (CPR), the correct identification of the optimal compression site is crucial, as small deviations can significantly reduce the efficacy of chest compressions and patient survival outcomes. Robotic and augmented reality systems can help address this issue by integrating anatomical mapping with real-time feedback, thereby improving the accuracy of compression site localization. Robots offer clear advantages in these scenarios by providing tremor reduction, motion scaling, stable tool trajectories, and the ability to record and standardize procedures, thereby enhancing both safety and reproducibility. However, a major challenge in such robot-assisted interventions is the limited situational awareness available to clinicians. Conventional interfaces rely mainly on 2D interfaces and indirect haptic cues, which restrict depth perception and spatial understanding. AR can address these limitations by overlaying anatomical landmarks, force feedback, and task-specific guidance directly onto the operator’s visual field [4]. This form of context-aware, spatially aligned feedback improves decisionmaking, reduces cognitive load, and facilitates user-friendly This work was partially funded by the “Technological and patient-tailored Innovations for Maximizing Effectiveness of Cardiac Arrest Resuscitation: the TIME-CARE project”, PNRR-POC-2023-12377193. 1Junling Fu, Afshin Karimi Dastjerdi, Matteo Di Mauro, Alessandro Albanesi, Alberto Redaelli, and Elena De Momi are with the Department of Electronics, Information and Bioengineering, Politecnico di Milano, 20133, Milan, Italy. (Corresponding: [email protected]) 2Tommaso Scquizzato is with the Department of Anesthesia and Intensive Care at IRCCS Ospedale San Raffaele in Milan, 20132, Milan, Italy. and precise task execution. Specifically, this paper proposes an AR-based framework for robot-assisted medical contact-rich task execution. Section II describes the methods. Section III presents the system and experimental setup. Section IV gives the preliminary results. Finally, Section V concludes this work. II. METHODS As depicted in Fig.1, the left panel illustrates the physical setup, where an RGB-D camera is mounted on the robot end-effector, and a human torso phantom is used to simulate contact-rich medical tasks. Moreover, the right panel shows the AR visualization, including the torso point cloud, a virtual robot model, and the virtual environment that enables the operator to define precise task execution positions. Fig. 1. Overview of the proposed AR-based framework for robot-assisted medical contact-rich task execution. A. System Calibration and Point-cloud Acquisition As shown in Fig.2, the calibration is required to register the point cloud from the RGB-D camera {C}into the robot base {B}, ensuring accurate robot localization for the following contact-rich tasks execution. Specifically, based on the coordinate systems relationship, B CheTis expressed as: B CheT=B EE T∗EE CT∗C CheT(1) where B CheT∈R4×4is the transformation matrix from the robot base {B}to the checkerboard {Che}. As depicted in Fig. 2, B EE T,EE CT, and C CheTdonate the matrix between the robot base and the end-effector, the camera and the end-effector, and the checkerboard and RGB-D camera, respectively. Once calibration is performed, the point clouds captured by the RGB-D camera can be registered into the robot base frame, as depicted in the right panel of Fig. 1. 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629882 241 Fig. 2. System components for the camera and robot frames calibration. B. Augmented Reality Visualization and Robot Control The interface integrates the intuitive visualization of the remote environment, including the robot configuration, patient phantom, reconstructed 3D surfaces, and a graphical user interface (GUI) for interaction. (1) AR-based Visualization: In the Fig. 1 left panel, the AR interface built on Microsoft HoloLens 2 provides both visualization and remote control for the robotic contact-rich tasks. (2) ARbased Robot Control: The interface allows the operator to switch seamlessly between free movement and force control modalities. Specifically, the robot can be freely moved to approach the patient and then engaged in force-regulated contact with the phantom or patient. This approach can guarantee a safe contact on the patient’s body. These features make the AR interface adaptive, user-friendly, and wellsuited for the safe execution of the robotic contact-rich tasks. C. Closed-loop Force Control for Contact-Rich Tasks Specifically, the interaction force Fext ∈Rmbetween the robot and the torso phantom (patient) can be expressed as follows with a Cartesian impedance controller: Fext =M¨ ˜x+Kcar ˜x+Dcar ˙ ˜x(2) where M∈Rm×mis the mass, Kcar,Dcar ∈Rm×mare the stiffness and damping parameters. ˜x,˙ ˜x, and ¨ ˜x∈Rmare the measured position displacement, velocity, and acceleration. Furthermore, a PID controller has been added to formulate the closed-loop control, which is formulated as: ∆Fcom =KPFe+KIZt 0 Fedt+KD dFe dt (3) where Feis the force error. Fdes and Fmsr are the desired and measured forces, respectively. KP,KI, and KDare the proportional, integrative, and derivative gains. III. SYSTEMS AND EXPERIMENTAL SETUP A. System Implementation As depicted in Fig. 1, the proposed framework is composed of: (1) A Hololens 2 OST-HMD (Microsoft, WA, USA), which is adopted for robot control and patient information visualization; (2) A serial 7-DoFs robotic manipulator (LWR 4+, KUKA, Germany), which is adopted to hold the tool to perform the contact-rich tasks; (3) An RGB-D camera (RealSense D435, Intel) is attached to the end-effector of the robot to perform the point cloud acquisition of the patient. Moreover, the Ubuntu incorporating Robot Operating System (ROS) kinetic version is exploited to control the robot. B. Experimental Verification To evaluate the performance of the proposed AR-based framework, a series of experiments was conducted. Specifically, these experiments focus on three key aspects: (1) hand–eye calibration accuracy, quantified as the positional difference between the checkerboard corners detected by the robot and those detected by the RGB-D camera; (2) robot control latency when using the proposed AR-based interface, measured as the discrepancy between the commanded (desired) and actual robot positions; and (3) contact force tracking performance on a human torso phantom, assessed under three target forces, Fdes from 5 to 15 N. IV. EXPERIMENTAL RESULTS The eye–hand calibration was validated across ten distinct Cartesian positions, yielding a median error of 7.0 ± 0.3 mm. Furthermore, the translational position errors remained within 4.5 mm (excluding the z-axis, which was regulated through force control), with an observed latency of approximately 0.18 s at a movement velocity of 7 mm/s using the proposed AR-based robot control framework. In terms of force regulation, the median tracking errors corresponding to three different desired force levels, Fdes, were measured as 5.22 N, 10.23 N, and 15.20 N, respectively. V. CONCLUSION This paper presents an AR-based framework for intuitive visualization and control in robot-assisted contact-rich medical tasks. Preliminary results demonstrate the positioning accuracy, low latency, and force-tracking performance. Future work will incorporate patient-specific preoperative data to enable intuitive visualization of anatomical structures, thereby supporting precise localization in procedures such as CPR. In addition, the robotic arm will be replaced with LUCAS, a professional chest compression system, to address CPR scenarios. Deep learning-based methods will be integrated to monitor patient movement as well. REFERENCES [1] D. G. Raitt, M. Huseynov, S. Homer-Vanniasinkam, H. A. Wurdemann, and S.-A. Abad, “Soft-tipped sensor with compliance control for elasticity sensing and palpation,” IEEE Transactions on Robotics, vol. 40, pp. 2430–2441, 2024. [2] J. Fu, G. Maimone, E. Iovene, J. Zhao, A. Redaelli, G. Ferrigno, and E. De Momi, “Human-inspired active compliant and passive shared control framework for robotic contact-rich tasks in medical applications,” IEEE Transactions on Robotics, 2025. [3] R. Mazhar, A. Raza, W. Ali, and T. Hamid, “A mechatronic biomimicking simulator platform for cardio-pulmonary resuscitation,” IEEE/ASME Transactions on Mechatronics, 2024. [4] C. Gießer, J. Knode, A. Gruenewald, T. J. Eiler, V. Schmuecker, and R. Brueck, “Skillslab+-augmented reality enhanced medical training,” in 2021 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). IEEE, 2021, pp. 194–197. 242