A Handheld Tactile Perception Device Enabling Defect Detection for Quality Inspection.

A Handheld Tactile Perception Device Enabling Defect Detection for Quality Inspection. Michele Pompilio, Gionata Salvietti, Nicole D’Aurizio, Enrico Turco and Domenico Prattichizzo Department of Information Engineering and Mathematics, University of Siena, Siena, Italy Department of Humanoids and Human Centered Mechatronics, Istituto Italiano di Tecnologia, Genova, Italy [email protected] Abstract—Defect detection in car-body manufacturing is a demanding task that traditionally relies on expert operators combining visual and tactile assessment. To advance the automation of tactile inspection, we developed a dedicated device integrating force and acceleration sensors to record the interaction dynamics arising during surface exploration. A preliminary dataset was collected with human participants and analyzed through supervised learning with standard classifiers. The results indicate that the device can reliably discriminate defective from non-defective regions. Furthermore, the collected tactile data provide a basis for training robotic systems to autonomously perform defect detection using the same sensing principles. Index Terms—Tactile Sensor, Sensor Fusion, Defect Detection I. INTRODUCTION Surface defect detection in car-body manufacturing requires perceptual and exploratory skills that human operators naturally perform through tactile sensing and fine motor control. While effective, these inspection and reworking tasks are physically demanding and repetitive, exposing workers to long-term strain. In this context, recent efforts in human–robot collaboration aim to replicate such tactile expertise in robotic systems, enabling safer and more scalable operations [1], [2]. A central step toward this goal is the design of dedicated tactile interfaces that can capture multimodal data representative of human touch. In the present study, a handheld device was developed to integrate a six-axis ATI Nano17 force/torque sensor and a triaxial ADXL335 MEMS accelerometer, coupled with a soft comb-like tip that interacts directly with the surface under inspection. This design allows simultaneous acquisition of quasi-static forces and high-frequency acceleration dynamics while the operator scans a surface (Fig. 1). When encountering small irregularities such as bumps or dents, the device provides measurable variations in both force and acceleration signals, effectively replicating the cues that humans use for defect detection [3]–[5]. By enabling operators to collect high-quality tactile data through a natural scanning motion, this device bridges human sensing strategies with robotic perception. The resulting dataset forms the foundation for supervised learning methods, allowing robotic systems to progressively acquire defect recognition skills while maintaining humans in a supervisory role. Such integration represents a step forward The research leading to these results has received funding from the European Union’s Horizon Europe programme under grant agreement No. 101120731 of the project “MAGICIAN - iMmersive leArninG for ImperfeCtion detectIon and repAir through human-robot interactioN” Fig. 1: The perception device developed for the analysis. It consists of a handle that allows the user to explore surfaces by touch, integrating an ATI Nano 17 force sensor coupled with an ADXL335 accelerometer, and a soft scallop 3D-printed in TPU. Thanks to its design, the scallop generates measurable variations in both force and acceleration whenever the user passes over a surface deformation during scanning. toward safer workplaces, consistent inspection quality, and efficient human–robot collaboration. II. DATASET COLLECTION The data collection campaign was designed to acquire synchronized, multi-modal signals during the exploration of real car body surfaces. A total of 28 panels with varying surface imperfections were scanned to ensure coverage of different defect types. Multiple users performed the exploration using the handheld tactile device instrumented with the triaxial force sensor and the triaxial accelerometer, sampled at 7 kHz and 4 kHz, respectively, providing sufficient temporal resolution to capture transient contact events and dynamic surface interactions. Positional information was recorded using a motion tracking system providing real-time six-degree-of-freedom trajectories of the device relative to the panels. Temporal alignment between tactile signals and pose data was achieved through a unified acquisition framework, which also enabled automatic labeling of defect presence by matching probe trajectories with 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629790 159 Fig. 2: Acquisition setup. During data collection, the car body is placed on a desk while the user scans its surface with the handheld device (highlighted in blue), equipped with the force and acceleration sensors. At the same time, a VICON tracking system records the device’s position with respect to the metal plate, using passive retro-reflective markers (highlighted in green). When the user passes over a defect on the metal plate (circled in red), the positional data are exploited to label the corresponding force and acceleration signals. predefined annotated regions (Fig. 2). A total of 280 acquisition sessions were collected, each lasting approximately 20 seconds. The resulting dataset contains time-aligned streams of three-axis force, three-axis acceleration, six-degree-of-freedom pose, and a binary defect presence flag generated via trajectory–annotation matching. All sessions include metadata describing defect type and experimental conditions. This dataset provides a foundation for training tactile-based classifiers and analyzing human exploration strategies using synchronized sensing and motion data. III. RESULTS AND DISCUSSION To assess the effectiveness of the proposed device for defect detection on metal surfaces, represented in this study by automotive parts, the collected dataset was subjected to a preliminary supervised learning analysis using a Random Forest classifier. The continuous sensor signals were segmented into labeled time windows and splitted into training ( 80% ) and test ( 20% ) sets. The classifier was trained in a binary setting, where the predicted label Defect/No Defect indicates respectively the presence or absence of a defect in the analyzed window. To assess the contribution of multimodal sensing, force and acceleration signals were analyzed independently and jointly to evaluate the benefits of their integration on classification performance, while positional data were excluded from this preliminary analysis since their primary role is to support the subsequent mapping of human motion onto the robotic system. After training, the classifier performance was assessed on the test set in terms of Accuracy, Precision, and Recall. Signals Accuracy Precision Recall No Def. Defect No Def. Defect Force 0.81 0.83 0.79 0.83 0.78 Acceleration 0.96 0.97 0.94 0.96 0.96 Force + Acc. 0.96 0.97 0.94 0.96 0.96 TABLE I: Results obtained on the test set using the trained Random Forest binary classifier. The classifier was first trained independently on the force and acceleration signals, and then on the combined signals. For each evaluation, the classifier’s accuracy as well as the precision and recall metrics for the No Defect and Defect labels were collected. Overall, the results indicate that the variations in force and acceleration produced by the device while scanning metallic surfaces allow accurate identification of defect presence in the explored area, yielding high precision and recall values for both No Defect and Defect classes. The results obtained for the different signal modalities are summarized in Table I. The results highlight that acceleration is the most informative sensing modality for defect detection on metallic surfaces, leading to high classification performance, whereas force signals alone provide more limited discriminative power. In the current binary classification, combining force and acceleration signals does not outperform acceleration alone. Nevertheless, force signals, while showing limited impact in this preliminary study, may play a critical role in more detailed defect type classification, where variations in force patterns could provide complementary information. Furthermore, the present analysis was conducted directly on raw sensor data. Prior studies have shown that extracting additional pre-processed features from raw time-series can significantly enhance classification outcomes [6], and future work will explore this direction. In conclusion, this study demonstrates the potential of the developed device for reliable defect detection and validates the quality of the collected dataset. These data will serve as a foundation for transferring the tactile module onto a robotic interface, where user trajectories, forces, and accelerations will be exploited to enable a robot to replicate the inspection task within a learning-by-demonstration framework. REFERENCES [1] Cherubini, Andrea, et al. ”Collaborative manufacturing with physical human–robot interaction.” Robotics and Computer-Integrated Manufacturing 40 (2016): 1-13. [2] Levine, Sergey, et al. ”Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.” The International journal of robotics research 37.4-5 (2018): 421-436. [3] Dahiya, Ravinder S., et al. ”Tactile sensing—from humans to humanoids.” IEEE transactions on robotics 26.1 (2009): 1-20. [4] Pacchierotti, Claudio, et al. ”Wearable haptic systems for the fingertip and the hand: taxonomy, review, and perspectives.” IEEE transactions on haptics 10.4 (2017): 580-600. [5] Yousef, Hanna, Mehdi Boukallel, and Kaspar Althoefer. ”Tactile sensing for dexterous in-hand manipulation in robotics—A review.” Sensors and Actuators A: physical 167.2 (2011): 171-187. [6] Strese, Matti, et al. ”Multimodal feature-based surface material classification.” IEEE transactions on haptics 10.2 (2016): 226-239. 160