A Trial Implementation of Augmented Reality Pitch Visualization in Ensemble Rehearsals: Insights from Wind Instrument Players

A Trial Implementation of Augmented Reality Pitch Visualization in Ensemble Rehearsals: Insights from Wind Instrument Players Yasumasa Yamaguchi1[0009000824947991] Sendai University, Japan [email protected] Abstract. Traditional electronic tuners have supported individual pitch training, yet their placement often disrupts posture and ensemble coordination. This study investigates the effectiveness of the Musical Pitch Visualization Perception (MVP) system—an augmented reality tool using Google Glass to provide real-time pitch feedback. Following earlier laboratory experiments, we deployed MVP in a rehearsal with a French horn ensemble. Results indicate that while the system did not significantly improve intonation accuracy, it supported posture and attentional coordination. Usability challenges were reported regarding visual clarity and ergonomic constraints. The findings highlight both pedagogical potential and technological limitations of AR-based support in ensemble music-making. Keywords: Augmented Reality ·Music Performance Support ·Intonation 1Introduction Musical instruments can be broadly categorized into two types: those that produce fixed pitch, such as the piano, and those whose pitch is influenced by the performer’s internal pitch sense, such as wind and string instruments. Playing a musical instrument is essentially a perceptual-motor skill, in which the performer continuously adjusts cognitive intention, motor execution, and sensory feedback [1,4]. For instruments that require fine pitch control, real-time feedback is especially critical. In recent years, the advancement of ICT has enabled the development of various feedback systems incorporating augmented reality (AR). These include systems that provide rhythmic cues within the performer’s field of vision [9], visual chord guidance for guitar learning [5], and 3D visual prompts for learning All rights remain with the authors under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Proc. of the 17th Int. Symposium on Computer Music Multidisciplinary Research, London, United Kingdom, 2025 Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 918 Y. Yamaguchi the theremin [2]. Such applications suggest the potential of AR as an intuitive tool for supporting musical skill acquisition. Despite these developments, traditional pitch training still largely relies on external reference sources such as electronic tuners or drone tones. Many musicians place tuners on their music stands to receive visual real-time pitch feedback [5]. While effective for short-term pitch accuracy, these methods have drawbacks: they divide visual attention, disrupt posture, and may hinder visual and auditory interaction with others in ensemble settings [2]. Ensemble performance demands subtle pitch adjustments, mutual listening, and visual communication with conductors—tasks that can be compromised when players fixate on tuners. Against this backdrop, we developed the Musical Pitch Visualization Perception (MVP) support system. This system uses Google Glass to provide real-time pitch feedback in the performer’s line of sight using color-coded cues (green = in tune, blue = flat, purple = sharp), allowing for posture and gaze to remain oriented toward the ensemble context [11,10]. Originally designed to support Japanese middle and high school band practice, where posture degradation from tuner use is a concern, the MVP system was shown in previous laboratory studies to outperform conventional tuners in pitch accuracy and subjective usability [11, 10]. However, those studies were conducted in controlled environments with individual players. The current study aims to assess the system’s effectiveness and usability in the more complex context of real ensemble performance, where performers must simultaneously manage pitch, timing, dynamics, and interpersonal coordination. We investigate how the introduction of AR-based pitch feedback influences attention, gaze, and auditory adaptation during ensemble play. Specifically, we deployed the MVP system in a real wind ensemble setting and collected user evaluations through questionnaires and interviews. Recent studies on AR in education have emphasized that not only technical functionality but also educational appropriateness, usability, and acceptability are crucial for successful implementation [6]. This study builds upon previous foundational research on MVP and explores its viability and reception within practical ensemble environments. 2SystemOverview The MVP (Musical Pitch Visualization Perception) support system is a realtime augmented reality feedback tool designed to support intonation control in instrumental performance. Unlike traditional tuners placed on music stands, the MVP system delivers pitch information directly into the performer’s field of view, allowing them to maintain both performance posture and ensemble engagement. Technically, the system uses Google Glass Enterprise Edition 2 as a headup display and a unidirectional condenser microphone for audio input. Audio signals are processed via the ml5.js machine learning library, which runs on a TensorFlow backend. Pitch detection is performed using the CREPE (Convolutional Representation for Pitch Estimation) algorithm, known for its high accuracy in real-time monophonic pitch estimation [3]. The detected pitch is comProc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 919 AR Pitch Visualization in Ensemble Rehearsals pared with a reference frequency and classified into one of three categories—“in tune” (green), “flat” (blue), or “sharp” (purple)—based on a ±1% tolerance. This simple three-color display is designed to minimize cognitive load and facilitate immediate adjustment during performance. The design philosophy aligns with research in ergonomics and cognitive psychology, which suggests that intuitive, low-detail feedback is more effective for real-time motor correction [7]. The system is entirely browser-based and platform-independent, allowing flexible deployment across different educational and rehearsal settings. It supports seamless integration with other ICT-based tools. In the present study, we applied this system in a real ensemble environment to assess its practical impact beyond the laboratory. 3Methodology This study recruited a French horn ensemble composed of three amateur performers ranging from undergraduate to graduate-level students. All participants could perform without wearing glasses and instead used contact lenses or performed with unaided vision. Each participant identified sections of their rehearsal where they were particularly conscious of intonation accuracy. They then performed those segments using the MVP system. After the rehearsal, they responded to a Google Forms questionnaire addressing usability and perceived effectiveness of the system. 4Results Participants evaluated the system using a 6-point Likert scale (1 = strongly disagree, 6 = strongly agree). In designing this scale, we emphasized the importance of assessing the effect of support on performance: a rating of “1” indicated “could not / unlikely to succeed,” whereas a rating of “6” indicated “could / likely to succeed.” This design ensured that both negative and positive responses could be clearly distinguished. Ratings included: –“I was able to maintain a good posture while playing.” (Mean = 4.66) –“I could sustain posture throughout the performance.” (Mean = 4.00) –“I could watch and coordinate with other performers.” (Mean = 3.66) –“I could play with accurate intonation.” (Mean = 3.33) –“I could adjust pitch while reading the score.” (Mean = 3.33) –“I could follow the ensemble’s tempo.” (Mean = 5.00) –“I could play with confidence.” (Mean = 3.66) These results suggest that while the MVP system did not significantly enhance pitch accuracy, it was beneficial in maintaining posture and synchronizing with other players. Usability was assessed on a separate 5-point scale that was designed specifically to check system functionality, and therefore included a neutral midpoint Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 920 Y. Yamaguchi option (“neither agree nor disagree”). Each item was phrased to match the nature of the evaluation: for example, “Did the system respond well to pitch input?” was rated from “1 = very poor” to “5 = very good”; “Was it easy to adjust your pitch?” from “1 = not easy at all” to “5 = very easy”; “Was the display on Google Glass easy to read?” from “1 = hardly visible” to “5 = clearly visible”; and “How closely did displayed pitch match your perception?” from “1 = largely inconsistent” to “5 = almost no difference.” Mean scores were as follows: –“Did the system respond well to pitch input?” (Mean = 3.66) –“Was it easy to adjust your pitch?” (Mean = 3.33) –“Was the display on Google Glass easy to read?” (Mean = 2.00) –“How closely did displayed pitch match your perception?” (Mean = 2.66) Although participants noted adequate system responsiveness, comments highlighted difficulties with display visibility and eye adjustment. Open-ended responses indicated that it was difficult to visually align with the Google Glass display during performance. As the sample size in this study was limited to only three participants, results are reported as mean values. 5Discussion This study examined the use of AR-based pitch feedback during ensemble performance and found that, contrary to earlier lab-based findings [11,10], the MVP system did not substantially improve pitch accuracy. Participants rated their ability to play in tune and adjust pitch only moderately, suggesting that the cognitive demands of ensemble coordination may limit the utility of visual pitch cues. On the other hand, the system showed some benefits in supporting good posture and ensemble timing. These results align with earlier reports that conventional tuners may interfere with visual coordination and body alignment [11, 10]. In ensemble settings where players must divide attention among music, conductor, and fellow musicians, even a head-up AR display may still compete with other visual and cognitive demands. Similar to findings by Pardue and McPherson [7], real-time feedback can enhance learning for some but distract others. One key issue highlighted in this study is the limitation of Google Glass itself. Participants noted difficulty in seeing the displayed feedback clearly, which corresponds to previous research documenting the narrow field of view and poor ergonomic integration of Google Glass Enterprise Edition 2 [12,8]. Studies in healthcare applications have shown that 70–80% of users experience visibility or comfort problems with this device [12]. Its display is fixed in the upper right visual field and requires an upward shift of gaze or head tilt to read, which can disrupt performance focus. Although Google Glass Enterprise Edition 2 has now been discontinued, it was selected for this study because of its comparatively lightweight design. For wind instrument performance, which relies on precise muscular control in the Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 921 AR Pitch Visualization in Ensemble Rehearsals neck and facial region, a lighter device is likely to exert less influence on performance than many currently available smartglasses. Nevertheless, given the growing availability of alternative wearable displays, future research should extend evaluation to other devices in order to validate the generalizability of the MVP system. These findings suggest that improvements in hardware—such as wider-angle displays or fully integrated see-through head-mounted devices—are critical to making AR feedback truly effective in ensemble music settings. Future research should explore wearable displays that allow intuitive, low-effort glanceability, or combine visual feedback with auditory or haptic modalities for more flexible multimodal cueing. Moreover, this study was limited by a small sample size and short exposure to the system. Longitudinal studies with a more diverse set of musicians and instruments are needed. Prior research has shown that adaptation to new feedback systems takes time, especially for novice players [7]. Finally, the MVP system, though effective in concept, requires refinement in both software display and hardware platform. With the discontinuation of Google Glass, nextgeneration smartglasses offering better resolution, contrast, and comfort could enable broader deployment in educational and performance contexts. Consideration should also be given to economic and pedagogical feasibility to ensure adoption in real-world instructional environments. Acknowledgments. This study was supported by JSPS KAKENHI Grant Number JP24K16761. Disclosure of Interests. The author declares no competing interests. References 1. Fitts, P.M.: Perceptual-motor skill learning. In: Melton, A.W. (ed.) Categories of Human Learning, pp. 243–292. Academic Press, New York (1964) 2. Johnson, D., Damian, D., Tzanetakis, G.: Evaluating the effectiveness of mixed reality music instrument learning with the theremin. Virtual Reality 24(2), 303– 317 (2020) 3. Kim, J.W., Salamon, J., Li, P., Bello, J.P.: Crepe: A convolutional representation for pitch estimation. In: Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 161–165. IEEE (2018) 4. Kreitman, E.: Teaching with an Open Heart: A Guide to Developing Conscious Musicianship for Suzuki Parents, Teachers, and Students. Western Springs School of Talent Education, Western Springs, IL (2010) 5. Martin-Gutiérrez, J., Guerra, M.S.D.R., López-Chao, V., Gastelum, R.H.S., Bojórquez, J.F.V.: Augmented reality to facilitate learning of the acoustic guitar. Applied Sciences 10(7), 2425 (2020) 6. Mei, B., Yang, S.: Chinese pre-service music teachers’ perceptions of augmented reality-assisted musical instrument learning. Frontiers in Psychology 12,609028 (2021) Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 922 Y. Yamaguchi 7. Pardue, L.S., McPherson, A.: Real-time aural and visual feedback for improving violin intonation. Frontiers in Psychology 10,627(2019) 8. The JAMA Network Journals: Google glass may partially obstruct peripheral vision, study finds. https://www.sciencedaily.com/releases/2014/11/141104163021.htm (2014) 9. Yamabe, T., Asuma, H., Kiyono, S., Nakajima, T.: Feedback design in augmented musical instruments: a case study with an ar drum kit. In: Proceedings of the 17th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA). vol. 2, pp. 126–129. IEEE (2011) 10. Yamaguchi, Y., Kawada, T., Nagahama, T., Horita, T.: A pilot study of the mvp support system using google glass. In: Proceedings of EdMedia + Innovate Learning 2022. pp. 17–28. Association for the Advancement of Computing in Education (AACE) (2022) 11. Yamaguchi, Y., Kawada, T., Nagahama, T., Horita, T.: Development and evaluation of a musical instrument performance support system using smart glasses: from the subjective evaluation of form, gaze, and performance. Japan Journal of Educational Technology 46(Suppl.), 185–188 (2023) 12. Yu, J., Ferniany, W., Guthrie, B., Parekh, S.G., Ponce, B.: Lessons learned from google glass: telemedical spark or unfulfilled promise? Surgical Innovation 23(2), 156–165 (2016) Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 923