Teleoperating the TIAGo Robot Using Body-Machine Interface: Preliminary Results

Teleoperating the TIAGo Robot Using Body-Machine Interface: Preliminary Results Mohammed Najie AL Dreay Dept. DIBRIS University of Genoa Genoa, Italy [email protected] Camilla Pierella Dept. DIBRIS University of Genoa Genoa, Italy [email protected] Matteo Moro Dept. DIBRIS University of Genoa Genoa, Italy [email protected] Maura Casadio Dept. DIBRIS University of Genoa Genoa, Italy [email protected] Abstract—We investigate a non-invasive Body–Machine Interface (BoMI) for robotic teleoperation that translates fingers motion into 2D cursor position. We propose a low-cost, webcambased approach for continuous teleoperation of the TIAGo mobile base. A two-dimensional autoencoder maps 42 hand landmarks extracted from video to a 2D cursor position, and the cursor position is what users control. A 3×3 graphical user interface then converts the cursor’s on-screen location into linear and angular velocity commands for a TIAGo mobile base. In a training–test protocol with four participants, post-training performance improved: completion times decreased for all and movement counts decreased for most, indicating more stable, efficient control on the slalom evaluation. Index Terms—Body Machine Interface, Dimensionality Reduction, Teleoperation. I. INTRODUCTION Mobile service robots can enhance independence, but their impact hinges on accessible, continuous control interfaces usable by non-expert users [1]. Body–Machine Interfaces (BoMIs) address this by mapping residual body motion to device commands; video-based BoMIs are especially attractive because they require only a webcam and commodity computing [2]. However, many existing solutions still rely on specialized sensors, which limits practical deployment [3]. Building on this idea, we have developed a novel, webcamdriven BoMI to facilitate the teleoperation of the mobile base of the TIAGo robot. This system aims to harness the movements of the hand to teleoperate the assistive robot. Our focus is on providing an intuitive and continuous control interface and a training–test protocol that evaluates learning. II. MATERIALS AND METHOD The pipeline of our implementation follows the same structure as prior work [2], but we incorporate hand landmark, which operates in real time to feed the mapping module. A. The Body-Machine Interface and TIAGo robot The BoMI uses webcam-based tracking of left-hand motion to generate continuous control signals for robot navigation. Its Python-based implementation consists of three modules, illustrated in Fig.1. This work was supported by the Italian Ministry of Research, under the complementary actions to the National Recovery and Resilience Plan (NRRP) “Fit4MedRob-Fit for Medical Robotics” Grant (# PNC0000007). 1) Data acquisition and processing: Video is captured with a webcam at 50 Hz (OpenCV). Google’s MediaPipe Hands detects [4] 21 keypoints on the left hand in real time; we use their (x, y) image coordinates, concatenated into a 42dimensional landmark. These landmarks are streamed directly to the mapping module for control. 2) Data mapping: The mapping module projects 42-D left-hand landmarks to a 2-D control space via a two code units (CUs) autoencoder (AE) trained once offline on 4,500 samples captured during a free-motion calibration sequence that lasted 90 s, which was intended to span the maximum range of possible degrees of freedom (50 Hz; 80/20 split, time-shuffled). The trained encoder is then fixed and used for all subjects, where the output of the CUs is x and y cursor coordinates. Hyperparameters were chosen to minimize reconstruction loss and balance variance across code units; the final model (5,036 parameters) uses four hidden layers of 32 units with tanh activations, linear code/output layers, Xavier weight initialization, and zero biases. Reconstruction fidelity was quantified using Variance Accounted For (VAF) [3]. Latent variance for the i-th CU (vi) was expressed as the percentage of the total code variance: vlati=vi P2 i=1 vi ∗100 (1) 3) User interface: The experimental interface was a Mobile Base Control Interface designed for navigation tasks. It implemented a nine-region graphical user interface (GUI) in the form of a 3 × 3 grid, functioning as a virtual joystick. The display was divided into nine zones, each mapped to a Fig. 1. Body-Machine Interface Architecture, which is structured into three modules: Data Acquisition and Processing (Panel A), Data Mapping (Panel B), and User Interface (Panel C). 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629880 239 specific base movement (Fig. 1.C). Participants maintained the cursor within a selected zone. Linear and angular velocities for the TIAGo base were computed from the normalized cursor offsets relative to the screen center (with a dead zone), with velocity scaling continuously according to the distance from the screen center. Linear velocity (forward/backward) was controlled by vertical cursor displacement, whereas angular velocity (left/right turns) was controlled by horizontal displacement. The interface is integrated with the ROS/Gazebo simulation: cursor-derived linear and angular velocities are published on ROS topics and applied to the TIAGo base running in Gazebo, which also provides the simulated scene view shown to participants. B. Experimental Testing 1) Participants: Four unimpaired volunteers (two female; mean age 26.25 ± 1.50 y), two right-hand dominant and two left-hand dominant, took part in the experiment. None reported musculoskeletal, postural, or neurological disorders, and all had normal hand range of motion. 2) Setup and protocol: Participants sat at a workstation with two displays: one, integrated with a webcam, provided video capture and user interface feedback; the other showed the TIAGo simulation view. The study consisted of an initial test, followed by a training block, and then a final test: •Testing phase (baseline and post-training): participants completed the same navigation test twice—once at the beginning of the experimental section to establish baseline performance and again at the end to assess learning. The test began from the start gate at the robot’s initial position, they had to follow the entire corridor route, then traverse a slalom section before exiting through the finish gate. The test concluded when the robot crossed the exit gate, with no predefined time limit for completing the path. •Training phase: between the two tests, three blocks were designed on a separate training map by shadowing a box-shaped “cart” along a predefined route that included straight sections, gradual curves, and sharp turns. A large on-screen arrow provided directional cues, indicating forward and backward segments to encourage participants to utilize the full range of maneuvers required by the nine-region graphical user interface (GUI). This process was designed to help participants practice a variety of movements, including forward and backward motion, turning, and combined maneuvers. The performance metrics were quantified by (i) completion time (s): elapsed simulation time from crossing the start gate to crossing the finish gate on the test map (no time limit), and (ii) number of movements: the count of region transitions of the on-screen cursor across the nine regions of the GUI. III. RESULTS The two latent dimensions, the nonlinear AE retained 78% of the calibration dataset variance, and its over-parameterized design produced a nearly uniform variance distribution across the two CUs (CU1 = 51.28%, CU2 = 48.72%). Fig. 2. Initial vs. final test results by subject: (A) completion time and (B) number of movements. For all subjects, training led to consistently faster navigation as evidenced by a reduction in completion time for every participant. As shown in Fig. 2.A, completion time of the test block dropped for every participant (S001: 615→410 s; S002: 865→440 s; S003: 400→240 s; S004: 545→295 s). Movement counts (Fig. 2.B) generally decreased (S001: 270→210; S002: 375→260; S003: 205→125); S004 showed a slight increase (182→190) despite a large time reduction, suggesting more efficient per-movement progress. IV. DISCUSSION AND CONCLUSION The BoMI enabled intuitive and continuous teleoperation of TIAGo’s base; after brief training, all participants completed the final test faster, and 3 out of 4 with fewer sub-movements. The 2D AE trained once generalized across users without calibration, greatly reducing setup time and supporting a compact mapping for translation/rotation. This work is a pilot study with a small cohort; in the future, we will extend it to a larger participant pool and evaluate it on a more extensive test scenario. REFERENCES [1] M. Casadio, R. Ranganathan, and F. A. Mussa-Ivaldi, “The body-machine interface: a new perspective on an old theme,” Journal of Motor behavior, vol. 44, no. 6, pp. 419–433, 2012. [2] M. Moro, F. Rizzoglio, F. Odone, and M. Casadio, “A video-based markerless body machine interface: A pilot study,” in International Conference on Pattern Recognition. Springer, 2021, pp. 233–240. [3] F. Rizzoglio, M. Giordano, F. A. Mussa-Ivaldi, and M. Casadio, “A nonlinear body machine interface for controlling assistive robotic arms,” IEEE Transactions on Biomedical Engineering, vol. 70, no. 7, pp. 2149–2159, 2023. [4] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. Yong, J. Lee et al., “Mediapipe: A framework for perceiving and processing reality,” in Third workshop on computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR), vol. 2019, 2019. 240