An Accessible Digital Musical Instrument for Inclusive Music Therapy

An Accessible Digital Musical Instrument for Inclusive Music Therapy Raquel Lucena1,*,Rafael Ramírez1 1Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain Abstract Accessible Digital Musical Instruments (ADMIs) have emerged as promising tools for enabling musical expression among individuals with severe motor impairments. However, many ADMIs require technical knowledge for configuration and evaluation, posing a barrier for music therapists without engineering backgrounds. In this study, we present a multimodal ADMI platform that integrates eye-tracking and touch-based interfaces, designed for individuals with mobility impairments and their non-technical caregivers. The system includes an automated music evaluation module, which extracts musical features and generates performance metrics without manual intervention. The proposed system was tested with healthy participants. Our results suggest that the system provides accessible musical experiences while equipping therapists with data-informed feedback for session planning and progress monitoring. Keywords Accessible Digital Musical Instrument, music therapy, motor impairment, chord recognition, music interface 1. Introduction Music therapy is a well-established method for enhancing cognitive, emotional and physical well-being in clinical populations [ 1 , 2 , 3 ]. However, traditional musical instruments often pose significant accessibility barriers for individuals with severe mobility limitations [ 4 ]. Accessible Digital Musical Instruments (ADMIs) aim to address this by incorporating assistive technologies such as touch, gaze and brain-computer interfaces [ 5 , 6 ]. Despite the interest in this area [ 7 ], the deployment of ADMIs in real-world therapeutic contexts remains limited. This is partly due to the technical complexity of many systems, which often require programming knowledge or manual annotation for use and evaluation. Several studies have proposed ADMIs leveraging custom hardware controllers [ 8 , 9 ], gazebased systems [ 10 , 11 , 12 ] or motion sensors [ 13 , 14 ]. Nonetheless, most systems are not designed with users who have severe impairments in mind, particularly those who can only move their eyes or hands and have speech difficulties [ 10 ]. Furthermore, many systems lack embedded evaluation tools, relying instead on qualitative annotations and questionnaires by therapists and users [15,16,17], complicating progress tracking. To fill these gaps, this paper presents a human-centric ADMI platform that prioritizes accessibility for patients and therapists. The system combines two interaction modalities (touch HCMIR25: 3rd Workshop on Human-Centric Music Information Research, September 20th, 2025, Daejeon, Korea *Corresponding author. $[email protected] (R. Lucena); [email protected] (R. Ramírez) 0009-0007-2490-4388 (R. Lucena); 0000-0002-3294-0764 (R. Ramírez) ©2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). and gaze) and incorporates a quantitative evaluation system using audio analysis and music information retrieval. This approach supports musical interaction and also enables objective assessment of progress in motor and cognitive skills through regular practice. User performance is evaluated using metrics presented in Music Information Retrieval Evaluation eXchange [ 18 ] to evaluate the audio chord estimation task [19], enabling rigorous progress tracking. Previous work [ 20 ] has emphasized that the value of an accessible musical instrument extends beyond usability and must account for the embodied relationship between the musician and the instrument. Although the design of the ADMI was guided by clinical observations and previous design guidelines [ 11 , 20 ], the evaluation was conducted only with healthy participants and testing with clinical populations is required for further validation. 2. System Design Before developing the instrument, a voluntary agreement was established with a hospital and a care center, enabling observation of 14 patients affected by mobility impairments caused by conditions such as cerebral hemorrhage, locked-in syndrome and stroke. These ethnographic observations influenced design choices. 2.1. Instrument Description The multimodal ADMI interface, developed with Python, supports interaction through a touchbased interface and an eye-tracking system. The touch-based interface implemented using MobMuPlat application [ 21 ] and Pure Data software allows users with partial upper-limb mobility to play chords by tapping on color-coded buttons displayed on tablets or smartphones. The eye-tracking system is implemented using the Tobii Eye Tracker 4C (portable and affordable [ 22 ]) and Talon software. It enables real-time gaze estimation with millisecond precision making the system accessible to individuals who are able to control only their eyes or facial movements. Users activate music elements via dwell-time activation. The ADMI includes a collection of chord samples (major and minor chords across root notes such as C, C#, D, D#, etc.) prerecorded using AmpleSound and GarageBand. The instrument is designed to be flexible, allowing new songs to be incorporated through an automated chord recognition and mapping process. However, the quality of the new songs added is constrained by the precision of the chord recognition algorithms. To promote ease of use, songs with simple chord progressions were chosen. 2.2. Design Principles The design of the ADMI was influenced by both direct observations in hospital and care center and design guidelines. First, the observations highlighted challenges such as insufficient or imprecise motor control, delayed response times and difficulty maintaining attention. Regarding the guidelines, on the one hand, we followed the principles proposed [ 11 ] for gaze-based instruments, which emphasize strategies to reduce cognitive load, minimize accidental activations and maintain attention through clear visual cues. On the other hand, we also considered the principles suggested in [ 20 ], who emphasizes that a successful instrument should foster a musician–instrument relationship based on ease of setup, intuitive and predictable response from the instrument, low entry level with opportunities for progression, enable accuracy in actions and repetition of musical ideas. Building upon these foundations, simplicity is prioritized over complexity due to cognitive and motor limitations commonly observed in hospital settings. The instrument can be configured through simple button-based menus, making setup straightforward for assistants with no technical training. Musical interaction options were limited to two to four chords per song, decreasing cognitive load and task complexity. Selecting chords as the unit of interaction was motivated by their usefulness in enabling harmonic accompaniment of popular songs, as well as their direct mapping to discrete gaze-based or touch-based controls to suit diverse motor abilities. The interface dynamically adapts the keys/buttons corresponding to the chords in the selected song. This design provides an intuitive, natural and predictable response, ensuring that players can quickly perceive a connection between their actions and the resulting sound. A central design goal was to support a low entry with high ceiling approach. Users can begin with simple tasks and advance gradually to more complex musical sequences. The repetition of a song and its evaluation using quantitative metrics facilitates is proposed as the mechanism for promoting musical and motor skill progress over time. To further support attention and timing, anticipation is implemented through visual cues, as color changes and highlights, to help users focus on the target and maintain attention. The interface also provides visual previews of upcoming chords and allow users to prepare in advance. Basic filter mechanisms, such as the minimal waiting time between consecutive sound triggers, are applied to prevent noise from triggering unintended actions. Finally, in the gaze-based instrument, due to the high sampling rate of the Tobii Eye Tracker (90Hz), quick eye movements can be registered as gaze points, which may trigger unintended notes [ 23 ]. To avoid this involuntary activation, a dwell-time activation threshold of 250 ms is used to trigger gaze-based input. Furthermore, the gaze-based interface uses a circular (pie-chart) layout with an empty central zone, minimizing accidental activation when crossing sections. 2.3. Interaction Flow The system workflow is designed to be intuitive and accessible for both therapists, caregivers and users, facilitating integration into therapeutic sessions. The interaction consists of several stages. First, in the song loading stage, users can load a new song to the system. An automated chord recognition algorithm, based on convolutional neural networks and implemented with the madmom library [ 24 ], extracts the chord progression from the selected audio and stores the sequence in a serialized file format ( .pkl ) to enable quick chord information retrieval. Next, during interface configuration, users select the input modality (touch or gaze) and choose from previously added songs through the interface. In the visual display stage, the system renders the virtual musical interface adapted to the selected modality. Each chord in the song is mapped to a unique color, enhancing visual accessibility. For the touch modality, a piano-like layout is presented on the touchscreen device, allowing user interaction (see Figure 1a). Simultaneously, a dynamic timeline bar is displayed on the computer screen: as the song progresses, colored blocks representing upcoming chords move horizontally, visually indicating the current chord, order and timing. For gaze-based interaction, a circular layout is presented (see Figure 1b). The current chord is highlighted while the opacity of other chords is diminished to minimize distraction. (a) Touch modality: the touch screen piano-style layout with the chord timeline. (b) Gaze modality: the eye-tracker device and the circular layout with color-coded chords. Figure 1: Multimodal ADMI devices for touch and gaze modalities. Anticipation is represented by the cursor, which changes its color to indicate the following chord two seconds before the transition and a progress ring represents the time remaining until the chord changes. The real-time interaction and logging stage follows, where users press keys on the tactile screen or focus their gaze on the appropriate section to play chords in synchrony with the music. Each user action is timestamped and recorded in a log files. Finally, during post-session analysis, performance metrics are extracted to assess the user’s ability to play the song and evaluate synchronization, accuracy and performance. The multimodal ADMI has a unified evaluation framework, where users or therapists can review and compare current session data with previous sessions. A report is generated that includes different visualizations to facilitate an objective assessment and track progress over time. 3. Methodology 3.1. Participants For an objective evaluation of the instrument’s accuracy and usability, five healthy participants (three women and two men) with an average age of 24.6 years ( 𝜎= 3.8 ) experimented with both methods in a controlled environment. The evaluation was conducted in sessions lasting around 20-25 minutes. At the beginning of the session, participants listened to the song a couple of times. Then, they played along with three distinct audio sequences: one with two chords, one with three chords and one with four chords. 3.2. Evaluation Metrics To quantitatively assess user performance and progress over time, metrics commonly used in audio chord estimation tasks were adapted to compare the reference chord sequence to the chords actually played by the user. The metrics used [ 18 , 19 ] provide information on the accuracy and timing precision of user’s interactions with the ADMI. Chord Symbol Recall (CSR) quantifies the proportion of time during that the user’s played input matches the reference (ground truth) chord sequence. It is computed by segmenting the chord sequence into short intervals (every 10ms) and calculating the proportion of segments where the played chords matches the reference. Formally: CSR = total duration of segments where reference equals played total duration of reference segments (1) Directional Hamming Distance (DHD) evaluates the alignment between the user’s chord sequence and the reference, penalizing both missing chords and extra chords. The directional Hamming distance is calculated by finding for each reference/played segment the maximally overlapping segment in the other sequence and then summing the differences [ 25 , 26 ]. Depending on the order of application, the directional Hamming distance measures over or under segmentation. Both directions are combined into a quality metric (Q). Formally: 𝑄= 1 − max(DHD𝑟𝑒𝑓→𝑝𝑙𝑎𝑦,DHD𝑝𝑙𝑎𝑦→𝑟𝑒𝑓 ) total duration of song (2) In addition, we compute more performance metrics for a complete view of the progress, allowing for an accurate evaluation of learning curves and improvements over time. These metrics include total number of chords played, the distribution of chord selections, the session duration and the maximum, lowest and average time intervals between chord transitions. 4. Results To evaluate the usability and performance of the ADMI, an experiment was conducted with five healthy users. Each participant interacted with the system through different modalities. During the experiment, we analyzed the metrics that allowed us to evaluate the effectiveness and performance of each mode. In Figure 2, results from the two interaction modalities evaluated across three distinct songs are presented. The green line represents the ideal value that would be achieved by perfectly playing the chord sequence. The blue and red lines represent the values obtained with the gaze-based and the touch-based systems, respectively. 5. Discussion This study investigated the performance differences between healthy participants using a multimodal ADMI. As demonstrated in Figure 2, interaction methods yielded similar results for all songs. However, the touch-based interaction shows more accurate values than the gaze-based interaction for all metrics and for most of the users attempts. Both modalities generate a similar number of chord events across participants. However, gaze-based interaction tends to produce slightly more events than the reference suggesting occasional over-triggering caused by involuntary eye movements. The average time in chord transitions is generally consistent with the reference across both modalities, but gaze-based interaction shows greater variability. The gaze-based modality exhibits larger peaks for maximum transition time and lower valleys for minimum transition time suggesting that difficulties in calibration sensitivity and rapid or unintended eye fixations can trigger undesired chord changes, respectively. The minimum transition time in touch-based is not stable either. Finally, results suggest that user performance seems to have improved with practice. Although the song Figure 2: Comparison of user performance metrics int touch-based and gaze-based ADMIs. Metrics include Number of Chords (NChords) and Average Transition Time (AvgT), Maximum (MaxT) and Minimum Transition Time (MinT), Chord Symbol Recall (CSR) and DHD-Quality Value (Q). with two chords was cognitively easier than songs with three or four chords, users achieved better results with the three and four chords songs in Q and CSR metrics, suggesting that they became more familiar with the ADMI over time. The metrics give us information about performance and accuracy of the ADMI, but are insufficient to capture the expressive and therapeutic aspects. Therefore, future hospital implementations will be complemented with questionnaires regarding the patient-instrument relationship such as ease of use and enjoyment. To provide a complete evaluation, a motor recovery assessment to quantify patients’ motor improvements and a cognitive test to assess cognitive changes should be applied. Finally, regarding the chord recognition algorithm, to avoid generating complex chord sequences some filters were applied. The most frequently occurring chords were selected, ensuring that the number of recognized chords aligned with the number expected to be played by the user. To address redundancy duplicated chords were merged within the same section and we assigned the most recently valid chord to segments where the algorithm failed to detect chords. 6. Conclusions and Future Work This paper presented a human-centered, multimodal accessible digital musical instrument where touch-based interaction modality obtained more accurate and stable results than gaze-based interaction modality. Despite unintended triggers and calibration precision issues, gaze-based input remains an alternative for users with only eye or facial mobility. By minimizing technical barriers, our system supports inclusive music therapy and expands the application of MIR metrics within healthcare contexts. A key contribution of the ADMI is the integration of quantitative performance metrixrcs that objectively assess user progress performance during sessions. These quantitative measures complement the subjective assessments and questionnaires used to evaluate most ADMIs. As a future work, we aim to conduct a long-term study with clinical population to prove its effectiveness and refine its features to meet the needs of patients. Acknowledgments This paper is part of Maria de Maeztu Units of Excellence Programme CEX2021-001195-M, funded by MICIU/AEI /10.13039/501100011033. This work is supported by IMPA project PID2023152250OB-I00 funded by MCIU/AEI/10.13039/501100011033/FEDER, UE and by Musitopia project, funded by the Fundació Barcelona Music Lab. Ethics Statement Ethics approval was approved by the Institutional Committee for Ethical Review of Projects under the reference number: X202000531s. All participants provided informed consent before participating in the study. Declaration on Generative AI During the preparation of this work, the authors used Chat-GPT-4 in order to: grammar and spelling check. After using this service, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. References [1] C. M. Weller, F. A. Baker, The role of music therapy in physical rehabilitation: a systematic literature review, Nordic Journal of Music Therapy 20 (2011) 43–61. URL: https://doi.org/ 10.1080/08098131.2010.485785. [2] J. Martín-Fernández, I. Burunat, C. Modroño, J. L. González-Mora, J. Plata-Bello, Music style not only modulates the auditory cortex, but also motor related areas, Neuroscience 457 (2021) 88–102. URL: https://www.sciencedirect.com/science/article/pii/S030645222100018X. doi:https://doi.org/10.1016/j.neuroscience.2021.01.012. [3] V. Faschi, L. A. Ludovico, F. Avanzini, E. Parravicini, M. Maestri, et al., An accessible software interface for collaborative music performance, in: Sound and Music Computing Conference (SMC), SMC, 2024, pp. 150–157. [4] OpenDot, Play and eye-tracking for child rehabilitation, https://medium.com/@OpenDot_ fablab/play-and-eye-tracking-for-child-rehabilitation-19f89683492a, 2021. [Accessed 1606-2025]. [5] S. T. Parke-Wolfe, H. Scurto, R. Fiebrink, Sound control: Supporting custom musical interface design for children with disabilities, in: M. Queiroz, A. X. Sedó (Eds.), Proceedings of the International Conference on New Interfaces for Musical Expression, UFRGS, Porto Alegre, Brazil, 2019, pp. 192–197. URL: http://www.nime.org/proceedings/2019/nime2019_ paper038.pdf. doi:10.5281/zenodo.3672920. [6] T. Colafiglio, D. Lofù, P. Sorino, A. Lombardi, F. Narducci, T. Di Noia, Neural musical instruments through brain-computer interface and biofeedback, in: Adjunct Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization, UMAP Adjunct ’25, Association for Computing Machinery, New York, NY, USA, 2025, p. 489–494. URL: https://doi.org/10.1145/3708319.3733644. [7] E. Frid, Accessible digital musical instruments—a review of musical interfaces in inclusive music practice, Multimodal Technologies and Interaction 3 (2019) 57. doi: 10.3390/ mti3030057. [8] A. D. Aziz, C. Warren, H. Bursk, S. Follmer, The flote: an instrument for people with limited mobility, in: Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility, 2008, pp. 295–296. [9] M. Lin, R. Paul, M. Abd, J. Jones, D. Dieujuste, H. Chim, E. D. Engeberg, Feeling the beat: a smart hand exoskeleton for learning to play musical instruments, Frontiers in Robotics and AI Volume 10 - 2023 (2023). URL: https://www.frontiersin.org/journals/robotics-and-ai/ articles/10.3389/frobt.2023.1212768. [10] Z. Vamvakousis, R. Ramirez, The eyeharp: A gaze-controlled digital musical instrument, Frontiers in psychology 7 (2016) 906. [11] N. Davanzo, F. Avanzini, et al., Design concepts for gaze-based digital musical instruments, in: Proceedings of the 19th Sound and Music Computing Conference, SMC network, 2022, pp. 472–478. [12] N. Davanzo, L. Valente, L. A. Ludovico, F. Avanzini, Kiroll: A gaze-based instrument for quadriplegic musicians based on the context-switching paradigm, in: Proceedings of the 18th International Audio Mostly Conference, AM ’23, Association for Computing Machinery, New York, NY, USA, 2023, p. 59–62. URL: https://doi.org/10.1145/3616195. 3616225. [13] E. Partesotti, J. Manzolli, A. Peñalba, Interactive musical technology enhances creativity: A case study with e-mocomu technology, in: INTED2017 Proceedings, 11th International Technology, Education and Development Conference, IATED, 2017, pp. 9730–9737. URL: https://doi.org/10.21125/inted.2017.2302. [14] D. Cavdir, G. Wang, Designing felt experiences with movement-based, wearable musical instruments: From inclusive practices toward participatory design, Wearable Technologies 3 (2022) e19. doi:10.1017/wtc.2022.15. [15] P. Kirk, M. Grierson, R. Bodak, N. Ward, F. Brander, K. Kelly, N. Newman, L. Stewart, Motivating stroke rehabilitation through music: A feasibility study using digital musical instruments in the home, in: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2016, pp. 1781–1785. doi:10.1145/2858036.2858376. [16] E. A. Søderberg, R. E. Odgaard, S. Bitsch, O. Høeg-Jensen, N. S. Christensen, S. D. Poulsen, S. Gelineck, Music aid: towards a collaborative experience for deaf and hearing people in creating music, in: New Interfaces for Musical Expression, 2016, p. 321–326. [17] A. Förster, N. Schnell, Designing accessible digital musical instruments for special educational needs schools—a social-ecological design framework, International Journal of Child-Computer Interaction 41 (2024) 100666. URL: https://www.sciencedirect. com/science/article/pii/S2212868924000345. doi: https://doi.org/10.1016/j.ijcci. 2024.100666. [18] J. S. Downie, The music information retrieval evaluation exchange (2005ndash;2007): A window into music information retrieval research, Acoustical Science and Technology 29 (2008) 247–255. doi:10.1250/ast.29.247. [19] C. Harte, Towards automatic extraction of harmony information from music signals., Ph.D. thesis, Queen Mary University of London, 2010. [20] A. McMillan, F. Morreale, Designing accessible musical instruments by addressing musician-instrument relationships, Frontiers in Computer Science 5 (2023) 1153232. [21] D. Iglesia, I. Intermedia, The mobility is the message: The development and uses of mobmuplat, in: Pure Data Conference (PdCon16). New York, 2016. [22] M. Cognolato, M. Atzori, H. Müller, Head-mounted eye gaze tracking devices: An overview of modern devices and recent advances, Journal of Rehabilitation and Assistive Technologies Engineering 5 (2018) 2055668318773991. URL: https://doi.org/10.1177/ 2055668318773991, pMID: 31191938. [23] A. Hornof, The prospects for eye-controlled musical performance, in: Proceedings of the International Conference on New Interfaces for Musical Expression, 2014, pp. 461–466. [24] S. Böck, F. Korzeniowski, J. Schlüter, F. Krebs, G. Widmer, madmom: A new python audio and music signal processing library, in: Proceedings of the 24th ACM International Conference on Multimedia, MM ’16, Association for Computing Machinery, New York, NY, USA, 2016, p. 1174–1178. URL: https://doi.org/10.1145/2964284.2973795. [25] S. Abdallah, K. Noland, M. Sandler, M. Casey, C. Rhodes, Theory and evaluation of a bayesian music structure extractor., in: Proceedings of the International Society for Music Information Retrieval Conference, 2005, pp. 420–425. [26] M. Mauch, Automatic chord transcription from audio using computational models of musical context, Ph.D. thesis, Queen Mary University of London, 2010.