Predicting Flutist Onset Timing in Duet Performance: A Multimodal Analysis of Gesture and Breath Cues

PREDICTING FLUTIST ONSET TIMING IN DUET PERFORMANCE: A MULTIMODAL ANALYSIS OF GESTURE AND BREATH CUES Jaeran Choi Taegyun Kwon Juhan Nam Graduate School of Culture Technology, KAIST, South Korea {jaeran.choi,ilcobo2,juhan.nam}@kaist.ac.kr ABSTRACT1 In ensemble performances, musicians use gesture and2 breath cues to synchronize their initial notes at the begin-3 ning of a piece, but the precise relationship between these4 cues and onset timing remains under-explored. This study5 investigates how flutists’ gesture and breath cues encode6 the timing information for the initial note onset. This re-7 search consists of four components: (1) Collection of a cue8 dataset containing synchronized video and audio record-9 ings of flute-piano duets, (2) Identification of cue candidate10 points through facial movement curves and breath onset-11 offset analysis, (3) Verification of predicted onset accuracy12 using linear regression on these cues compared to human13 onset asynchronies and (4) Introduction and exploration of14 a ‘trigger’ concept, defined as immediate, clearly perceiv-15 able gestures (such as stopping or raising the head) indi-16 cating the precise moment of onset. Our findings suggest17 a dual-cue system: preparatory cues broadly predict onset18 timing, while precise triggers refine the exact onset. We19 compared the time difference between the predicted and20 piano onsets with the flute–piano asynchronies and veri-21 fied the concepts of cue and trigger through expert inter-22 views. This research contributes to a deeper understanding23 of the complex phenomena of musical cues during perfor-24 mance through multimodal analysis. This paper provides25 an open-access cue dataset, which can be found on the ac-26 companying website. 1 27 1. INTRODUCTION28 Music performance is inherently multimodal, combining29 sound and motion. Although these elements primarily con-30 vey musical expression to audiences [1], they also play a31 critical role in ensemble synchronization among perform-32 ers [2]. Performers often employ specific gestures and33 breathing sounds as musical cues to synchronize their note34 onsets, particularly at the beginning or during critical mo-35 ments in a performance. The convention of cueing approx-36 1https://github.com/jaeranchoi/flutist_cue_dataset © J. Choi, T. Kwon and J. Nam.. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: J. Choi, T. Kwon and J. Nam., “Predicting Flutist Onset Timing in Duet Performance: A Multimodal Analysis of Gesture and Breath Cues”, in Proc. of the 26th Int. Society for Music Information Retrieval Conf., Daejeon, South Korea, 2025. imately one beat before the musical onset has been exten-37 sively documented in previous studies [3–5]. Head nod-38 ding gestures, in particular, are commonly used as intuitive39 musical cues, and previous studies have utilized such ges-40 tures to define cue timings, even extending their applica-41 tion to interactions with robotic musicians [5–7]. There-42 fore, understanding musical cues not only deepens syn-43 chronization knowledge but is also crucial for designing44 interactive performance systems.45 Previous studies have examined the role of gestures46 and breathing in performer synchronization. Bishop et al.47 showed that performers use visual and auditory cues to48 synchronize after silence or rests [8]. Additionally, ges-49 tures at the beginning of a piece were found to corre-50 late with tempo, particularly through falling acceleration51 curves that typically reach their midpoint approximately52 one beat before the onset [9]. However, this midpoint tim-53 ing was often not precisely one beat ahead, and exact onset54 prediction based on these curves was not explored. Vera55 et al. demonstrated increased onset asynchrony when per-56 formers restarted together after rests without visual con-57 tact, and identified relationships between breath onset-58 offset timings and rest durations, yet did not clarify how59 gestures or breathing specifically encode onset timing [10].60 Although these studies highlight the significance of ges-61 tures and breath cues in synchronization, they have not62 thoroughly investigated how combined visual and auditory63 cues precisely predict intended onset timing, especially at64 the beginning of a piece.65 One reason for the limited quantitative analyses in pre-66 vious studies is the difficulty of accurately tracking ges-67 tures. Bishop et al. utilized Kinect sensors and accelerom-68 eters to measure motion curves [9], while Timmers et al.69 employed infrared markers to measure bow velocity in a70 string quartet setting [11]. In contrast, our approach uses71 sensorless image processing techniques, applying optical72 flow methods [12] to quantitatively track facial gesture73 movements.74 This paper investigates how a flutist’s intended onset75 timing is encoded through gesture and breath cues, com-76 prising four main components: (1) Collection of synchro-77 nized video and audio data from flute-piano duet perfor-78 mances involving 20 flutists (Total 1,320 trials), (2) Ex-79 traction and annotation of gesture cue features using face80 movement curves analyzed by optical flow, alongside man-81 ual annotation of breath onset and offset timings, (3) Ver-82 ification of predictive accuracy for cue-based onset pre-83 100 Figure 1: The overall framework for the musical cue detection and onset timing prediction dictions by comparing linear regression–derived timings to84 observed human onset asynchronies and (4) Exploration of85 a trigger concept, which involves three types of immediate,86 clearly perceivable movements: quickly raising the head,87 stopping head movement after the cue or slowly raising the88 head. The first type just before fully raising the head, the89 second triggers onset immediately after stopping, and the90 third provides a less clear signal. These trigger movements91 allow performers to precisely detect the onset moment for92 accurate synchronization.93 Our analysis demonstrates clear relationships between94 the lengths of gesture and breath cues and onset timings.95 Gesture cues showed a linear relationship between po-96 sition, velocity and acceleration peaks in vertical facial97 movements and subsequent onset timing. Similarly, breath98 cues exhibited a correlation between breath duration and99 the timing interval to the note onset, with variations ob-100 served across different tempos. We further verified our hy-101 potheses through expert interviews. Additionally, we con-102 ducted a case study to address related phenomena, includ-103 ing adaptation effects—reduced discrepancies through re-104 peated rehearsal—and instances of cue execution failures,105 where significant differences between cue-based predicted106 onset and actual onset were observed. Based on these107 observations, we conclude that musicians utilize two pri-108 mary synchronization strategies: rough timing indication109 through gesture and breath cues, and precise, immediate110 signals through triggers. Furthermore, exact synchroniza-111 tion is refined through repetitive rehearsals.112 2. RELATED WORKS113 2.1 Musical Cue for Synchronization114 Musical cues, essential for performer communication,115 include visual gestures and non-musical elements like116 breathing. They are particularly valuable for precise coor-117 dination, such as in pieces with abrupt tempo changes [13],118 or synchronization after rests and tempo variations [8].119 The beginning of a musical piece is challenging for coordi-120 nation due to the absence of preceding audio cues. Bishop121 et al. examined visual cues at piece initiation, finding that122 the peak of the acceleration curve in nodding gestures indi-123 cated beat positions, while gesture duration and periodicity124 conveyed tempo information [9]. However, they did not in-125 vestigate the predictive capability of gesture cues for onset126 timing prediction.127 Most studies on synchronization between gestures and128 musical rhythm have emphasized velocity peaks rather129 than spatial positions as primary features. Su [14] demon-130 strated this using minimal laboratory setups with bouncing131 point-light and auditory stimuli. Similar findings emerged132 from string quartet studies linking bow speed to tempo133 cues [11], and conductor studies emphasizing baton veloc-134 ity and acceleration [15]. Vera et al. further highlighted135 breath cues’ significance in synchronization, particularly136 when visual contact is limited [10]. These studies under-137 line the role of gesture and breath cues in synchronization,138 suggesting velocity, acceleration, and cue length influence139 timing. However, few studies have simultaneously exam-140 ined those cues in wind instruments to assess their impact141 on timing. Our study extends this by quantitatively exam-142 ining correlations between gestures, breath cues, and onset143 timings through multimodal analysis of flute-piano duets.144 2.2 Gesture Analysis of Performance145 Motion tracking methods utilizing sensors or optical flow-146 based video tracking [16] are common for gesture analy-147 sis. Previous studies by Bishop et al. and Timmers et al.148 used attachable sensors or markers to measure movement149 and acceleration [9, 11], whereas Bochen et al. applied150 audio-visual analysis with optical flow to examine vibrato151 patterns from string players’ hand movements [17, 18].152 Maezawa et al. proposed MuEns, a multimodal score-153 following system employing optical flow to track gestures154 for automated piano accompaniment. However, in that par-155 ticular work, the authors utilized arbitrarily defined gesture156 cues instead of systematically analyzing how performers157 naturally encode onset timings [5].158 3. DATASET159 3.1 Musical Cue Dataset160 We created a multimodal cue dataset containing video and161 audio recordings of flute-piano duet performances to ana-162 lyze musical cues from gestures and breathing sounds. Fol-163 lowing previous studies [9,14], we identified peaks in posi-164 tion, velocity, and acceleration curves as potential gesture165 cue points. Breath cue points were defined by the breath166 onset and offset timings. We termed the interval between167 paired cue points as ‘cue length’ and the duration from cue168 initiation to flute onset as ‘cue-onset length’.169 3.1.1 Participants170 A total of 20 professional flutists and 3 pianists partici-171 pated in the experiment, all holding bachelor’s or master’s172 Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 101 Figure 2:Breath Cue and Onset Annotation Breath cue is defined by the breath onset and offset annotated on the mel spectrogram. Six markers are annotated per trial. Detailed information can be found in Section 3.2 degrees in performance. Each flutist-pianist pair had not173 previously performed together.174 3.1.2 Placement and Equipment175 The setup mimicked a concert stage, with flutists posi-176 tioned facing away from the pianists. The pianists could177 observe the flutists, while the flutists were instructed to178 give cues without turning or looking at the pianists. A179 camera recorded flutists at 60 fps 2, and microphones sep-180 arately captured audio from each instrument at 44.1 kHz.181 A neutral-colored screen behind flutists minimized back-182 ground interference for accurate facial movement tracking.183 3.1.3 Musical Piece184 This dataset comprises simultaneously starting flute–piano185 duets. Part 1 included a C major scale and Pachelbel’s186 Canon performed at slow (50 BPM), medium (100 BPM)187 and fast (150 BPM) tempos, each repeated twice as warm-188 up exercises (12 trials). Part 2 consisted of 18 classical189 pieces simplified for piano and arranged for simultaneous190 starts. Each piece was assigned a specific tempo (50, 100,191 or 150 BPM), and the entire set of 18 pieces was repeated192 three times, resulting in 54 trials. Each duet performed a193 total of 66 trials, resulting in 1,320 trials overall. Sheet194 music and sample audio were provided in advance.195 3.1.4 Procedure196 The recording procedure for each piece included: (1) An197 experimenter’s clap signaling start, followed by a measure198 of clicks matching the given tempo; (2) The flutist giving a199 cue after clicks ended; (3) The duet beginning in response200 to the cue.201 3.1.5 Post-session Interview202 The interviews collected the insights of the participants203 on cue strategies. Participants reported providing cues ap-204 proximately one beat (or half or two beats, depending on205 the piece) ahead, using body movements or breath. Some206 participants mentioned that in typical performance situa-207 tions, they adjust their cue timing based on the accompa-208 nying instrument and ensemble context.209 2Some videos were recorded at 30 fps due to camera overheating Figure 3:Gesture Cue Example Gesture cue is defined with the maximum(red) and minimum(black) peaks. The interval between these peaks, called ‘cue length’ (x, red line), and the duration between the maximum peak to the flute onset, called ‘cue-onset length’ (y, blue line). 3.2 Annotation and Preprocessing210 Video and audio data synchronization was achieved211 through an experimenter’s clap at the start. Using the spec-212 trogram viewer in Adobe Audition, we manually annotated213 six markers per trial on the mel spectrogram (Figure 2): the214 experimenter’s clap (Start), breath sound onset and offset215 (Breath Onset,Breath Offset), initial note onsets of flute216 and piano (Flute Onset,Piano Onset), and the flute’s sec-217 ond measure onset (2nd Measure).218 4. METHODS219 4.1 Gesture Cue Detection220 4.1.1 Motion Detection221 To detect gesture cues from flutists, we used MediaPipe’s222 face landmark detection 3[19] to reliably identify face re-223 gions, even when partially obscured by the flute. Subse-224 quently, optical flow methods [17,18] were applied to track225 facial motion. A pilot study indicated optimal face land-226 mark detection accuracy when the face occupied at least227 50% of the video frame height; videos were accordingly228 resized.229 4.1.2 Motion Feature Extraction230 We analyzed facial gestures by extracting position, veloc-231 ity, and acceleration magnitude curves from averaged y-232 axis optical flow values. Due to quantized pixel positions233 causing discrete velocity curves, we applied zero-phase fil-234 tering 4to smooth the curve while preserving peak posi-235 tions.236 4.1.3 Motion Peak Picking237 Figure 3 illustrates a typical gesture cue pattern. Within a238 one-measure window preceding the flute onset (‘cue win-239 dow’), we identified maximum and minimum peaks on240 position, velocity, and acceleration curves using the find-241 peaks 5algorithm. This approach automatically detected242 3available at: https://developers.google.com/mediapipe 4scipy.signal.filtfilt 5scipy.signal.find_peaks Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 102 Figure 4:Relationship between Cue Length and Cue-Onset Length for Gesture Position, Velocity, Acceleration, and Breath Cue Black line: overall regression; blue, green, pink: 50, 100, 150 BPM, respectively. Slopes (y=ax) and correlation (r) are shown for each curve. 848 peaks, with 394 manually annotated. Trials without243 clear peak patterns or failed tracking were excluded, re-244 sulting in 1,242 usable trials out of 1,320. Gesture curves245 from cases where peak tracking failed are also available on246 the accompanying webpage.247 4.2 Breath Cue Detection248 To detect breath cues, we annotated breath onset and offset249 within the same one-measure ‘cue window’ preceding flute250 onset, based on mel spectrograms (Figure 2). Analyses251 were conducted exclusively on the 1,242 trials that were252 verified to contain valid gesture cues.253 4.3 Onset Timing Prediction254 We examined four cues: gesture position, velocity, accel-255 eration, and breath. To explore the relationship between256 these cues and onset timing, we applied a simple linear257 regression model. Each cue length was set as the inde-258 pendent variable x, while the cue-onset length (the inter-259 val from cue start positions—maximum peak or breath on-260 set—to the actual onset) was the dependent variable y. The261 regression model, without bias, is defined by y=ax, as il-262 lustrated in Figure 3. The slope aderived from the re-263 gression indicates the ratio between cue length and onset264 timing, enabling onset prediction. Additionally, we ana-265 lyzed the trials separately by tempo to assess differences266 in cue-onset relationships, examining both the regression267 slope and Pearson correlation. A high correlation would268 confirm the cue’s validity for predicting onset timing.269 5. RESULTS AND DISCUSSION270 5.1 Patterns in Gesture Curves271 The position curve did not consistently show the same272 shape in all trials, but in most cases it remained static ini-273 tially and then displayed a clear downward-up-down mo-274 tion pattern that served as a signal (Figure 3). Even when275 the position curve deviated from the typical pattern, an276 up-down motion immediately preceding onset was consis-277 tently present (1242 out of 1320 trials, see Section 4.1.3).278 These movements were often periodic and sinusoidal, re-279 sulting in velocity and acceleration curves that mirrored280 the position curve, but phase-shifted by approximately a281 quarter cycle. Further investigation could examine how282 variations in curve characteristics, such as the degree of si-283 nusoidal shape and periodicity consistency, influence per-284 formers’ interpretation of cues and their subsequent syn-285 chronization accuracy. Moreover, analyzing deviations286 from typical sinusoidal patterns might uncover additional287 insights into performer-specific gesture strategies. Due to288 challenges in quantifying these curve characteristics pre-289 cisely, this topic remains open for future research.290 5.2 Results of the Linear Regression291 Figure 4 illustrates the relationships between cue length292 and cue-onset length for gesture and breath cues across all293 participants. Both types of cues showed linear relation-294 ships with cue-onset durations. For breath cues, eight out-295 liers were identified due to ambiguous annotations; these296 were excluded from subsequent regression analyses. Lin-297 ear regression indicated slopes of 1.53 (gesture position),298 2.28 (gesture velocity), 3.05 (gesture acceleration), and299 1.78 (breath cue). Interestingly, none of the cues yielded300 a slope close to 2, suggesting that counting the cue length301 and subsequent duration as equal units (similar to count-302 ing two beats) is not consistently applicable. The veloc-303 ity and breath cues had slopes closest to 2, but the in-304 terval from the velocity cue to onset was slightly longer305 (1.28 times cue length), while for the breath cue it was306 slightly shorter (0.78 times cue length). Gesture accelera-307 tion showed a strong correlation (0.72) with cue-onset du-308 rations, but breath cues exhibited an even stronger corre-309 lation (0.78), highlighting their superior reliability for pre-310 dicting onsets. The higher correlation with velocity and ac-311 celeration compared to position aligns with previous stud-312 ies, suggesting performers primarily perceive velocity or313 acceleration peaks as cue indicators rather than positional314 points. However, acceleration peaks closely align with pre-315 vious position peaks, indicating a potential two-peak en-316 coding pattern in position curves. Precisely quantifying317 this relationship is challenging and thus remains a topic318 for future research.319 Additionally, slopes generally decreased slightly with320 increased tempo, indicating flute onsets occurred sooner321 than expected based on proportionally shortened cue322 lengths. Furthermore, correlation coefficients for gesture323 cues decreased at higher tempos, whereas breath cues ex-324 hibited higher correlations at faster tempos, reinforcing the325 effectiveness of breath cues for synchronization.326 Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 103 Figure 5:Histogram of Note Onset Asynchrony Onset time differences between the pianist’s onset and 1. Flutist onset (Ground truth) 2-4. Predicted onset from gesture position, velocity, acceleration cue 5. Predicted onset from breath cue. ID Group Gesture Clarity Onset Timing Breath Clarity Onset Timing Async Abs Async STD late 4.5 3.3 3.9 3.4 71 71 39 A good 4.0 3.6 4.8 3.0 54 54 31 fast 3.8 2.4 4.4 2.6 -54 125 66 late 4.4 3.1 3.9 3.1 71 71 39 B good 3.1 3.4 3.3 3.4 54 54 31 fast 3.9 3.9 4.5 3.9 -54 125 66 Table 1:Result of the Expert Evaluation Expert ratings of cue clarity and execution timing (1 = late, 3 = on-time, 5 = early) on a 5-point scale, with asynchrony metrics (ms) indicating average group asynchrony. 5.3 Note Onset Asynchrony327 5.3.1 Asynchrony Comparision328 Figure 5 shows histograms of time differences between329 the pianist’s onset and five conditions: actual flutist onset330 (gray) and four predicted onsets from the linear regression331 model (Section 5.2). This error represents the discrepancy332 expected if the flutist perfectly followed our linear model333 and executed the cues precisely. A notable feature in the334 pianist-flutist asynchrony histogram is the minimal occur-335 rence of trials just before zero milliseconds. This sug-336 gests a tendency for the leader (flutist) to start slightly ear-337 lier than the follower (pianist), aligning with observations338 confirmed by expert interviews (Section 5.5.2). Addition-339 ally, we consider auditory reaction (performers starting in340 response to hearing the partner’s onset) unlikely in most341 cases, as the observed asynchronies are typically smaller342 than the average auditory reaction time (150 ms) [20].343 The ‘Pianist-Flutist asynchrony’ had the narrowest344 spread (Absolute(Abs) mean = 79ms, Standard Deviation345 (STD) = 168ms), consistent with previous research re-346 porting the first onset asynchronies slightly above 80ms347 [9]. Gesture-based predictions showed absolute mean er-348 rors between 121–168ms, with acceleration predictions ex-349 hibiting higher variability (STD = 247ms). Breath pre-350 dictions were more consistent (Abs mean = 109 ms, STD351 = 209 ms). Although the acceleration cue demonstrated352 the highest Pearson correlation, its greater variability and353 longer cue length contributed to larger errors. These find-354 ings suggest that while cue-based linear predictions are355 slightly less precise than human synchronization, breath356 cues provide more reliable predictions than gesture-based357 cues. The precision of predictions compared with actual358 flutist onsets exhibited similar patterns.359 5.3.2 Reduction of Asynchrony Through Repetition360 We also investigated whether synchronization accuracy361 improves as pianists and flutists adapt to each other’s cues.362 Figure 6: Human onset asynchrony across session repetition. As described in Section 3.1.3, we observed asynchrony363 changes through an initial warm-up session of 12 trials364 (S1) and three repeated sets of 18 pieces (T1–T3), illus-365 trated in Figure 6. Asynchrony notably decreased from366 the initial warm-up session (S1) through the second rep-367 etition session (T2) but stabilized thereafter. We inter-368 pret this as indicating adaptation effects, where performers369 quickly improved synchronization by familiarizing them-370 selves with each other’s cues. However, ongoing piece371 variation and unresolved consensus about triggers (Section372 5.4) likely prevented further reduction in asynchrony be-373 yond a certain threshold.374 5.4 Triggers375 Despite the considerable accuracy of linear predictions376 (Section 5.2), questions remained regarding precise cue377 recognition, particularly for velocity and acceleration378 peaks. Given potential perception errors in identifying379 cue points, we investigated additional factors musicians380 might use to ensure precise synchronization. We observed381 characteristic gesture patterns immediately following cue382 movements near onset timings. After the downward-383 upward-downward cue motion, flutists typically executed384 one of three distinct trigger patterns (Figure 7): (A) ini-385 tiating onset just before raising the head again, typically386 aligned with the previous upward cue movement, (B)387 briefly pausing with the head lowered and starting immedi-388 ately afterward, or (C) ambiguously initiating onset while389 slowly raising the head. Patterns (A) and (B) provided390 clear, immediate signals suitable as precise triggers. Pat-391 tern (C), however, represented ambiguous or absent trig-392 gers. Without prior agreement, these ambiguous gestures393 could cause confusion—for example, a flutist intending394 pattern (A) might be misinterpreted by the pianist as pat-395 tern (B), resulting in significant timing discrepancies. Such396 cases were indeed observed, and the validity of these trig-397 ger patterns was further confirmed through expert inter-398 views (Section 5.5). Thus, we propose that musicians ini-399 tially encode approximate timing through cues and then400 Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 104 Figure 7:Examples of three trigger types (A, B, C) Green dashed lines indicate breath onset and offset, the blue line marks flute onset, and the red line piano onset. The red shaded area represents the cue interval. achieve precise synchronization through clearly defined401 trigger gestures.402 5.5 Expert Evaluation and Interview403 After the main analysis, we conducted a two-hour inter-404 view with two expert flutists, each with over 20 years of405 ensemble and teaching experience. The session involved406 two main tasks: evaluating trials categorized as fast, good,407 or late based on onset predictions from velocity cues, and408 assessing our trigger hypothesis. Due to time constraints,409 only velocity cues were used, as they offered the most in-410 terpretable and perceptually reliable basis for cue classifi-411 cation.412 5.5.1 Evaluation of Linear Model Predictions413 Experts reviewed 8 trials from each group (fast, good, late)414 to evaluate the clarity of the cues (accuracy) and whether415 the actual onset timing matched the timing implied by the416 cue (execution timing). The experts evaluated 24 randomly417 ordered trials without being informed of the group labels.418 After agreeing with the assumption that head-nodding ges-419 tures and breathing serve as cues, experts rated cue accu-420 racy on a scale from 1 (no recognizable cue) to 5 (clearly421 recognizable cue) and execution timing from 1 (very late)422 to 5 (very early), with 3 representing precise timing. Re-423 sults are summarized in Table 1.424 Surprisingly, the experts differed notably in their eval-425 uations of breath cue execution timing, which also did not426 align clearly with actual pianist-flutist asynchrony. Expert427 A rated the ‘good’ group’s gesture cues as slightly late but428 breath cues as most accurate. Expert B rated the groups in429 descending order (late-good-fast) of perceived lateness but430 still considered the ‘late’ group relatively early (average431 3.1). Actual asynchrony partly aligned with our predic-432 tions; the ‘good’ group exhibited the smallest asynchrony,433 consistent with the cue hypothesis. However, unexpected434 discrepancies emerged, such as the ‘fast’ group showed435 later flute onsets than the piano. These inconsistencies436 likely reflect individual variations in interpreting and ex-437 ecuting cues.438 5.5.2 In-depth Interview439 After the video evaluation, we conducted detailed discus-440 sions about the linear prediction model and the trigger441 concept. Experts acknowledged the general practice of442 cueing approximately one beat ahead, noting that specific443 movements were intuitively executed rather than explic-444 itly planned. Both emphasized tempo-related influences on445 gesture and breath cues, highlighting gesture periodicity as446 crucial for clear cue delivery.447 Regarding triggers, experts initially did not consciously448 recognize different trigger types but agreed with the pro-449 posed classifications after reviewing examples. Both450 agreed type (A) was optimal, while type (B) was con-451 sidered challenging due to the flute’s physical constraints.452 Opinions on type (C) diverged: Expert A considered it in-453 herently prone to higher errors, whereas Expert B believed454 it could be viable when accompanied by precise breath455 cues and rehearsal. Experts noted that triggers may vary456 depending on musical context (phrasing, emphasis). They457 also observed potential triggers, including the flute’s end-458 point position, lip shape, and finger movements. Finally,459 both proposed that in flute-piano duets, slight delays in pi-460 ano onset might cognitively benefit synchronization, po-461 tentially explaining the scarcity of trials where piano onset462 preceded flute onset, as observed in Figure 5.463 5.6 Limitation464 Although this study contributes to understanding cue-465 based synchronization, several limitations remain. First,466 our trigger analysis was not fully quantitative; future re-467 search should develop precise methods for defining trig-468 gers from gesture curves and consider additional signals469 such as lip shape, finger movements, and horizontal flute470 actions. Second, The decision-making process for select-471 ing triggers was not investigated. Exploring how musi-472 cians choose and agree on triggers could further clarify473 synchronization strategies. Additionally, individual vari-474 ability in cue and trigger preferences was not quantitatively475 examined; future research could explore personalized syn-476 chronization strategies. Lastly, our findings apply specifi-477 cally to flute-piano duets and piece initiation. Generalizing478 these methods to other instrument combinations and ongo-479 ing musical contexts remains necessary.480 6. CONCLUSION481 This study investigated how gesture and breath cues used482 by flutists in flute-piano duets encode note onset timing at483 the initiation of musical pieces. To quantitatively analyze484 cues, we collected a cue dataset, identifying cue points via485 motion tracking and breath sounds. Our multimodal anal-486 ysis revealed linear relationships between cue lengths and487 onset timings, confirming that gesture and breath cues re-488 liably predict onset timing. In addition, we introduced the489 concept of triggers, defined as immediate gestures that in-490 dicate precise onset moments, which we validated through491 expert interviews. Future research could further develop492 the analysis of triggers to achieve a more systematic un-493 derstanding. Additionally, developing regression models494 that jointly use gesture and breath cues, or exploring so-495 phisticated predictive models, would be valuable. It could496 further extend our cue analysis methods to other instru-497 ments, ensemble configurations, and within-performance498 synchronization, providing deeper insights into the com-499 plexity of ensemble coordination.500 Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 105 7. ETHICS STATEMENT501 This study was approved by the Institutional Review Board502 (IRB), and all participants consented to video recording503 and data sharing. Each flutist’s session lasted 1.5 hours,504 pianists had 30-minute breaks between sessions, with up505 to four sessions per day. Participants received appropri-506 ate compensation. To protect privacy, the video data will507 remain confidential.508 8. ACKNOWLEDGMENTS509 This work has been supported by the National Research510 Foundation of Korea (NRF) grant funded by the Korea511 government (MSIT) under Grant RS-2023-NR077289 and512 Grant RS-2024-00358448.513 9. REFERENCES514 [1] C.-J. Tsay, “Sight over sound in the judgment of music515 performance,” Proceedings of the National Academy of516 Sciences, vol. 110, no. 36, pp. 14 580–14 585, 2013.517 [2] L. Bishop, C. Cancino-Chacón, and W. Goebl, “Mov-518 ing to communicate, moving to interact: Patterns of519 body motion in musical duo performance,” Music Per-520 ception: An Interdisciplinary Journal, vol. 37, no. 1,521 pp. 1–25, 2019.522 [3] F. K. Hukporti, Your Guide to Basic Conducting. Ac-523 cra: Noyam Publishers, 2023.524 [4] R. Page-Shipp, D. Joseph, and C. van Niekerk, “Con-525 ductorless singing group: a particular kind of self-526 managed team?” Team Performance Management: An527 International Journal, vol. 24, no. 5/6, pp. 331–346,528 2018.529 [5] A. Maezawa and K. Yamamoto, “MuEns: A multi-530 modal human-machine music ensemble for live concert531 performance,” in Proceedings of the 2017 CHI Confer-532 ence on Human Factors in Computing Systems, 2017,533 pp. 4290–4301.534 [6] X. Gao, A. Rogel, R. Sankaranarayanan, B. Dowling,535 and G. Weinberg, “Music, body, and machine: gesture-536 based synchronization in human-robot musical interac-537 tion,” Frontiers in Robotics and AI, vol. 11, 2024.538 [7] A. Lim, T. Mizumoto, L.-K. Cahier, T. Otsuka, T. Taka-539 hashi, K. Komatani, T. Ogata, and H. G. Okuno,540 “Robot musical accompaniment: integrating audio and541 visual cues for real-time synchronization with a human542 flutist,” in 2010 IEEE/RSJ International Conference on543 Intelligent Robots and Systems, 2010, pp. 1964–1969.544 [8] L. Bishop and W. Goebl, “When they listen and when545 they watch: Pianists’ use of nonverbal audio and vi-546 sual cues during duet performance,” Musicae Scien-547 tiae, vol. 19, no. 1, pp. 84–110, 2015.548 [9] ——, “Beating time: How ensemble musicians’ cueing549 gestures communicate beat position and tempo,” Psy-550 chology of Music, vol. 46, no. 1, pp. 84–106, 2018.551 [10] B. Vera, E. Chew, and P. G. Healey, “A study of ensem-552 ble synchronisation under restricted line of sight.” in553 International Society for Music Information Retrieval554 Conference (ISMIR), 2013, pp. 293–298.555 [11] R. Timmers, S. Endo, A. Bradbury, and A. M. Wing,556 “Synchronization and leadership in string quartet per-557 formance: a case study of auditory and visual cues,”558 Frontiers in Psychology, vol. 5, p. 645, 2014.559 [12] B. Li, X. Liu, K. Dinesh, Z. Duan, and G. Sharma,560 “Creating a multitrack classical music performance561 dataset for multimodal music analysis: Challenges, in-562 sights, and applications,” IEEE Transactions on Multi-563 media, vol. 21, no. 2, pp. 522–535, 2018.564 [13] S. Kawase, “Gazing behavior and coordination during565 piano duo performance,” Attention, Perception, & Psy-566 chophysics, vol. 76, pp. 527–540, 2014.567 [14] Y.-H. Su, “Audiovisual beat induction in complex au-568 ditory rhythms: Point-light figure movement as an ef-569 fective visual beat,” Acta Psychologica, vol. 151, pp.570 40–50, 2014.571 [15] G. Luck and J. A. Sloboda, “Spatio-temporal cues for572 visually mediated synchronization,” Music Perception,573 vol. 26, no. 5, pp. 465–473, 2009.574 [16] B. K. Horn and B. G. Schunck, “Determining optical575 flow,” Artificial intelligence, vol. 17, no. 1-3, pp. 185–576 203, 1981.577 [17] B. Li, K. Dinesh, Z. Duan, and G. Sharma, “See578 and listen: Score-informed association of sound tracks579 to players in chamber music performance videos,”580 in 2017 IEEE International Conference on Acoustics,581 Speech and Signal Processing (ICASSP), 2017, pp.582 2906–2910.583 [18] B. Li, C. Xu, and Z. Duan, “Audiovisual source associ-584 ation for string ensembles through multi-modal vibrato585 analysis,” Proc. Sound and Music Computing (SMC),586 pp. 159–166, 2017.587 [19] V. Bazarevsky, I. Grishchenko, K. Raveendran, T. Zhu,588 F. Zhang, and M. Grundmann, “Blazepose: On-589 device real-time body pose tracking,” arXiv preprint590 arXiv:2006.10204, 2020.591 [20] R. J. Kosinski, “A literature review on reaction time,”592 Clemson University, vol. 10, no. 1, pp. 337–344, 2008.593 Proceedings of the 26th ISMIR Conference, Daejeon, Korea, September 21-25, 2025 106