The TouchBox MK3: An Open-Source Device for Finger-Based Interaction with Advanced Auditory and Vibrotactile Feedback

The TouchBox MK3: An Open-Source Device for Finger-Based Interaction with Advanced Auditory and Vibrotactile Feedback Stefano Papetti Eric Larrieux Martin Fröhlich [email protected] [email protected] [email protected] Zurich University of the Arts Zurich, Switzerland ABSTRACT The TouchBox is an open-source device aimed at fnger-based interaction, ofering advanced auditory and vibrotactile feedback. Its capabilities encompass tracking multiple fnger gestures, measuring exerted forces, and contact areas, while providing rich interactive audio and vibrotactile experiences. Previous prototypes have been utilized in psychophysical experiments and musical performance assessments. The latest MK3 version introduces novel enhancements, including a new mechanical design, onboard computation for fnger tracking using the OpenMV platform, upgraded vibrotactile actuators, and improved audio amplifers. The device is supported by an open-access repository, ofering tools and documentation for easy replication and further development. The TouchBox MK3 presents a versatile and afordable solution, well-suited for educational and research purposes, and holds potential for various multimodal human-machine interaction applications. KEYWORDS fnger-based interaction, 3D force sensing, fnger tracking, vibrotactile feedback, auditory feedback, haptic device, open-source, DIY ACM Reference Format: Stefano Papetti, Eric Larrieux, and Martin Fröhlich. 2023. The TouchBox MK3: An Open-Source Device for Finger-Based Interaction with Advanced Auditory and Vibrotactile Feedback. In INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI ’23 Companion), October 09–13, 2023, Paris, France. ACM, New York, NY, USA, 4 pages. https: //doi.org/10.1145/3610661.3616181 1 INTRODUCTION The TouchBox is an open-ended device designed to track various fnger-based gestures, precisely measure exerted three-dimensional forces, and determine fnger contact areas, while delivering rich interactive auditory and vibrotactile feedback. Its earlier iterations [ 3 ] have successfully contributed to psychophysical experiments and musical performance assessments [ 2 , 4 ]. Building upon This work is licensed under a Creative Commons Attribution International 4.0 License. ICMI ’23 Companion, October 09–13, 2023, Paris, France © 2023 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-0321-8/23/10. https://doi.org/10.1145/3610661.3616181 these achievements, the TouchBox MK3 represents a signifcant advancement with several novel features. Notably, the MK3 version incorporates a new mechanical design that includes a larger touch surface, expanding the scope of interaction possibilities. Also, fnger-tracking now relies on the OpenMV platform 1 for onboard computation, improving its responsiveness and capabilities. Moreover, the MK3 version incorporates more powerful and efcient vibrotactile actuators based on voice-coil technology, along with an upgraded audio amplifer, ofering a more realistic user experience. An extensive open-access repository is made available, ofering comprehensive documentation and software tools for easy replication and further development. 2 This aspect signifcantly contributes to the device’s suitability for educational purposes. As a low-cost yet accurate and sophisticated tool, the TouchBox MK3 ofers an ideal platform for learning the principles of vibrotactile perception, fnger-based interaction, and multimodal feedback in educational settings. Its versatility and accessibility make it a valuable asset for students, educators and researchers alike. In this paper, we present the technical details of the TouchBox MK3, elaborating on its novel features and enhancements. Additionally, we explore potential applications in education and research, highlighting its capacity as a calibrated measurement device and its signifcance in fostering multimodal human-machine interaction studies. 2 DESIGN Figure 1 presents a detailed rendering of the MK3 device, with key components labeled. Additionally, Fig. 2 showcases the fully implemented prototype of the TouchBox MK3, providing a visual glimpse of its functional components with the enclosure opened. The MK3 design leverages a combination of materials and technologies, allowing for a cost-efective yet high-performing device. The material costs, excluding manufacturing expenses, totaled approximately 350 USD, with raw materials like Plexiglas and aluminum constituting nearly 100 USD of the overall expenses. This budget-friendly approach ensures accessibility, making the TouchBox MK3 particularly well-suited for educational settings. The central component of the TouchBox MK3 is the Plexiglas top panel (1), which serves as the primary touch surface for user 1https://openmv.io/ 2https://github.com/ICST-AHMI/TouchBox 302 ICMI ’23 Companion, October 09–13, 2023, Paris, France Papeti et al. (1) (5) (4) (3) (6) (2) (7) (8) (9) Figure 1: Rendering of the TouchBox MK3, with relevant components labeled: (1) Plexiglas top panel, (2) normal force sensor (1x, at the bottom of the structure), (3) lateral force sensor (4x, one per side), (4) IR LEDs strip (2x, at opposite sides), (5) OpenMV camera, (6) vibration actuator (2x, at opposite sides), (7) Arduino microcontroller board, (8) audio amplifer, (9) MDF enclosure. Figure 2: The implemented prototype of the TouchBox MK3. interactions. The panel is connected to the sensors and actuators responsible for capturing and delivering interactive feedback. An Arduino 3 microcontroller board (7) serves as the central control unit, enabling data processing and communication between the sensors, actuators, and the host computer. To cover these intricate components securely, the TouchBox MK3 is housed in an MDF enclosure (9). This provides stability and protection, making the device suitable for use in various educational settings. Its compact and ergonomic design encourages students and educators to explore fnger-based interaction and multimodal feedback with ease. The TouchBox MK3’s thoughtful design, coupled with its affordability and accessibility, positions it as an ideal educational tool. 2.1 3D-Force Sensing Solution The TouchBox MK3 ofers precise and reliable sensing of threedimensional forces applied to its top panel. To achieve this, the 3https://www.arduino.cc/ device is equipped with fve Micro Load Cell (MLC) sensors. A CZL635 MLC (2), situated at the bottom of the structure, accurately measures the normal force applied to the top panel, with a capacity of up to 50 N. Four CZL616C MLCs (3), placed at each side of the top pane, measure lateral forces, with a maximum capacity of 7.65 N per sensor. This innovative 3D-force sensing solution efectively captures a wide range of touch interactions, enabling the user to engage with the TouchBox MK3 in a highly intuitive manner. The fve MLCs are priced at about 7 USD each, the respective op-amps are worth also 7 USD each, and the Arduino board costs 40 USD. Therefore, the overall cost of our 3D-force sensing solution is only a small fraction of the price of comparable commercial systems (usually in the 700–900 USD range). 2.2 Finger Tracking using OpenMV The TouchBox MK3’s fnger tracking capabilities are facilitated by image processing techniques implemented on an OpenMV Cam M7 board (5) running MicroPython OS. The camera system, with frmware version 4.4.2, ofers two resolution modes: low-resolution (160 × 120) and high-resolution (320 × 240). To efectively capture a large portion of the top panel despite the short distance, the device utilizes an ultra-wide-angle (fsheye) lens. The costs are respectively 65 USD for the camera and 15 USD for the lens. In conjunction with the camera setup, two PCB strips equipped with infrared (IR) LEDs (4) are strategically placed on opposite sides of the top panel. These IR LEDs emit horizontal beams that travel through the Plexiglas panel, illuminating fnger-pads touching its surface. This illumination enables efcient detection and tracking of fnger positions and contact areas through a video processing algorithm (see Sec. 3). The combined cost of the IR LEDs amounts to approximately 12 USD. 2.3 Audio-Tactile Rendering The TouchBox MK3 is complemented by a sophisticated audiotactile rendering system that enriches the user experience with dynamic and realistic vibrotactile feedback. At its core are two Lofelt L5 advanced voice-coil actuators (6), providing powerful tactile sensations based on incoming audio signals. The moving masses of these actuators displace tangentially to the top panel, optimizing the haptic experience and adding depth to the interactions [ 1 ]. To drive each actuator individually, the two channels of a Dayton Audio DTA-2 stereo amplifer (8) are utilized. The cost-efective nature of each actuator (approximately 10 USD with edu-pricing) and the amplifer (about 35 USD) ensures a high-quality audio-tactile rendering system that remains budget-friendly. The device is supplied with open-source software to synthesize auditory and vibration signals based on the incoming sensor data. This software is grounded in physical modeling, enabling a responsive and dynamic interaction, perfectly suited to the capabilities ofered by the TouchBox MK3. Through the ofered examples, users can explore a wide range of auditory and vibrotactile feedback possibilities. 3 SENSORS DATA PROCESSING Force data are uniformly sampled at a high rate of 1176 Hz with 10-bit resolution, using an Arduino Mega 2560 board connected 303 The TouchBox MK3: An Open-Source Device for Finger-Based Interaction with Advanced Auditory and Vibrotactile Feedback ICMI ’23 Companion, October 09–13, 2023, Paris, France via USB to a host computer. To achieve such synchronous data acquisition, a custom frmware was developed, utilizing low-level hardware interrupts instead of the native API. To ensure accurate measurements, a calibration procedure is available through the provided software. For the force sensors, e.g. test weights can be used to calibrate the normal force, while a force gauge dynamometer applied to the top panel can be used to calibrate the lateral forces. For the image data obtained from the OpenMV Cam M7, various preprocessing stages are applied to enhance the quality of the fnger tracking. First, the distortion introduced by the ultra-wide-angle (fsheye) lens is corrected. Finger tracking is achieved using the find_blobs function of the OpenMV Image class 4 . The algorithm searches for contiguous groups of pixels that pass a threshold test based on specifc parameters, such as minimum and maximum color channel intensity, minimum number of pixels, and minimum blob area. The number of detected blobs is currently limited to three, efectively tracking the positions of up to three fngers. The computed fnger positions are represented as distances from the center of the panel in millimeters, providing accurate spatial information. Additionally, the contact areas of the fngers with the top panel are computed in square millimeters. To achieve this, an additional calibration procedure is employed: Sample objects with known areas are placed on the panel, allowing the system to compute the efective pixel size in millimeters, which is then utilized to accurately calculate the contact areas of the fngers. In our prototype the efective pixel size is 0 . 450 mm in low-res, and 0.225 mm in hi-res mode. A software application, developed using the Max graphical programming environment 5 further processes the data from the sensors and presents them in a user-friendly graphical interface (see Fig. 3). The Max application processes the sensor data received through the device’s USB connections and presents them in a clear and intuitive manner. The GUI displays numerical information about the fngers, such as their contact areas and positions relative to the center of the top panel. Additionally, the GUI visually represents the forces applied to the panel, with the total normal force shown as a dark red disc, and the lateral forces illustrated by a red line indicating their vectorial sum. The Open Sound Control protocol 6 is also implemented, allowing easy interfacing with external systems. Overall, the supplied software enables users to interact with the TouchBox MK3, interpret and forward the sensory data in real-time, fostering a seamless and immersive user experience. 4 EDUCATIONAL SCENARIOS While the TouchBox MK3 has not yet been formally tested in educational environments, its versatile features, afordability and opensource nature make it an ideal tool for introducing students to the principles of haptic interaction and multimodal feedback. Below are several educational scenarios where the device can prove valuable: 4http://docs.openmv.io/library/omv.image.html#image.Image.fnd_blobs 5 Max is a multimedia processing environment available for Windows and Mac OS computers. 6https://ccrma.stanford.edu/osc Figure 3: Graphical User Interface (GUI) of the software application. The screenshot refects the data from two fngers pressing down with an overall normal force of 6 . 83 N, and pushing forward and left with forces of 1 . 41 N and 1 . 39 N, respectively. The leftmost section of the GUI displays numerical data for up to three fngers, including their contact areas and positions relative to the center of the top panel. In the center section, the values of forces applied to the panel are graphically represented: the total normal force is depicted as a dark red disc, while the lateral forces are illustrated by a red line, originating from the center of the disc, which shows their vectorial sum. The rightmost section of the GUI displays the areas and positions of the contacting fngers as seen by the OpenMV blob tracker. Haptic Perception Studies. The TouchBox MK3 can serve as an efective tool for conducting haptic perception studies, allowing students to explore the relationship between force exertion, fnger contact areas, and perceived vibrotactile feedback. In psychology courses, the device could be used in experiments investigating e.g. how diferent textures simulated via vibration afect human emotional responses, thereby deepening students’ understanding of sensory psychology and providing them with practical experience in experimental design. Multimodal Interaction Experiments. With its capability to provide both vibrotactile and auditory feedback, the TouchBox MK3 facilitates experiments that investigate the integration of multiple sensory modalities in human perception. Students can explore how the brain combines haptic and auditory stimuli, leading to novel insights into cross-modal perception and the development of multimodal interactive systems. This feature could be especially benefcial in special education settings, where customized educational content could be created to make learning more accessible and engaging for students with sensory impairments. Haptic Technology and Design Projects. The open-hardware nature and low-cost of the TouchBox MK3 empowers students to engage in hands-on haptic technology and design projects. They could for instance explore diferent materials, actuators, and sensors to create customized haptic interfaces for specifc applications. The experience of building and iterating upon the TouchBox MK3 304 ICMI ’23 Companion, October 09–13, 2023, Paris, France Papeti et al. to add new features or functionalities would allow students gaining real-world engineering experience in a well-defned context. Computer Science and Engineering Education. The TouchBox MK3 can be used to teach topics related to physical computing, data processing, and sensor integration. Students can gain practical experience in programming microcontrollers, implementing realtime algorithms for sensor data processing, and building interactive haptic systems. In a remote learning context, the device could substitute various lab equipment that students might not otherwise have access to, thereby breaking down barriers in their educational experience. Interactive Media and Design Studies. For students in interactive media and design, the TouchBox MK3 ofers a tangible platform for prototyping and testing novel interaction concepts. Its capability to deliver both auditory and vibrotactile feedback opens up possibilities for designing immersive and engaging interactive experiences. Students can explore novel ways to communicate information and emotions through vibrotactile and auditory cues. Accessible Education and Inclusive Design. The device’s affordability and open-source nature make it a viable option for promoting accessibility and inclusive design in education. Educators could use the TouchBox MK3 to develop educational content that integrates rich auditory and vibrotactile feedback, as well as extensive physical interaction possibilities. That would enhance the learning experience especially for students with sensory impairments or physical disabilities, ensuring that the learning experience is inclusive and engaging for all. Overall, the TouchBox MK3 is a valuable addition to several educational environments, ofering students a practical and engaging platform for exploring haptic interaction, multimodal feedback, and interactive design concepts. Its open-source nature encourages collaboration and knowledge sharing, fostering a community of educators and students who can collectively advance the feld of haptic interaction and multimodal interfaces. 5 CONCLUSIONS The TouchBox MK3 stands as an open-source audio-haptic device designed for fnger-based interaction, ofering advanced auditory and vibrotactile feedback. With its enhanced mechanical design, improved fnger tracking capabilities using the OpenMV platform, and upgraded vibrotactile actuators, the MK3 version represents a signifcant advancement over its predecessors. The device’s ability to precisely sense three-dimensional forces, track multiple fngers’ positions and contact areas, and provide interactive vibrotactile and auditory feedback opens up a wide range of applications, spanning from scientifc research to creative interactions. Moreover, the TouchBox MK3’s afordability and accessibility make it an excellent choice for educational settings. ACKNOWLEDGMENTS This research was supported by project HAPTEEV (grant n° 178972, 2018-2022), funded by the Swiss National Science Foundation. REFERENCES [1] Yuri De Pra, Stefano Papetti, Hanna Järveläinen, Matteo Bianchi, and Federico Fontana. 2023. Efects of Vibration Direction and Pressing Force on Finger Vibrotactile Perception and Force Control. IEEE Trans. Haptics 16, 1 (jan 2023), 23–32. https://doi.org/10.1109/TOH.2022.3225714 [2] Hanna Järveläinen, Stefano Papetti, Sébastien Schiesser, and Tobias Grosshauser. 2013. Audio-Tactile Feedback in Musical Gesture Primitives: Finger Pressing. In Sound Music Comput. Conf. Stockholm, Sweden, 109–114. https://doi.org/10.5281/ zenodo.850222 [3] Stefano Papetti, Martin Fröhlich, and Sébastien Schiesser. 2019. The TouchBox: an open-source audio-haptic device for fnger-based interaction. In IEEE World Haptics Conf. IEEE, Tokyo, Japan, 491–496. https://doi.org/10.1109/WHC.2019.8816172 [4] Stefano Papetti, Hanna Järveläinen, Bruno L. Giordano, Sébastien Schiesser, and Martin Fröhlich. 2017. Vibrotactile Sensitivity in Active Touch: Efect of Pressing Force. IEEE Trans. Haptics 10, 1 (jan 2017), 113–122. https://doi.org/10.1109/TOH. 2016.2582485 305