Screenless Optical Theremin with Tremolo (ScOTT)

Screenless Optical Theremin with Tremolo (ScOTT) MICHAEL GANCZ, Yale University, USA JUSTIN BERRY, Yale University, USA SHU WEI, Yale University, USA JAKE SHAKER, Yale University, USA KIMBERLY HIEFTJE, Yale University, USA ASHER MARKS, Yale University, USA Additional Key Words and Phrases: gesture, MIDI, extended reality, XR, head-mounted display, collaborative music-making, humancomputer interaction, musical mapping 1 Program Notes "Programmability is a curse." -Perry Cook, 2001 [2] Music and gesture are entangled. We describe music in terms of gesture; likewise, we must engage with gesture in order to produce musical utterance. Music draws gesture from within us, often without our conscious intervention. Likewise, gestural decisions affect the character of musical performance. When we map gesture to music, we fragment a space of infinite variety into operative metaphors: semantic, embodied, spatial, tactile, synesthetic. We classify the space of gestural possibility in terms of dialectical or combinatorial oppositions: ergonomic or effortful, strong or light, pressing or slashing [ 1 , 3 ]. Regardless, gesture remains ambiguous, endlessly programmable, despite the language that we construct around it. The present performance does not attempt to challenge existing schemes for understanding music and gesture. Instead, it explores the intersection between these two infinite spaces. A screenless XR headset that tracks movement, but does not render it visually, allows our performance to operate both in the virtual world—where gesture manifests as vectors and variables—and the physical world, where gesture manifests as a conversation with the saxophone, the guitar, and the audience. 2 Project Description 2.1 Abstract Head-mounted extended-reality (XR) interfaces provide a flexible platform for immersive and embodied musical instrument design. By combining spatial audio, ergonomic first-person gestural control, and networked interactivity, these interfaces can facilitate expressive, interesting, and emotionally resonant performances. Unlike data gloves and hyperinstruments, XR headsets can be calibrated to the physical properties of their individual users. However, these headsets tend to obscure the eyes and other parts of the face, limiting the user’s capacity to establish eye contact and transmit facial expressions. These subtle communicative elements play a crucial role in real-world collaborative musical settings where performers utilize facial cues to negotiate surface parameters such as timing, dynamics, and breath, as well as more complex qualities like atmosphere and interpretive mimesis. In this remote performance, we present the Screenless Optical Theremin with Tremolo (ScOTT), a novel gestural MIDI controller powered by a modified screenless XR headset. ScOTT focuses on the hands and arms, mapping broad gestures to coarse-grained musical parameters (e.g. pitch and velocity) and small movements to more complex musical ornaments (e.g. tremolo width and frequency). In a structured improvisation that highlights the role of ornament in musical texture, we explore the ScOTT’s capacity to balance social presence, embodied interaction, and expressivity. Authors’ Contact Information: Michael Gancz, Yale University, New Haven, CT, USA; Justin Berry, Yale University, New Haven, CT, USA; Shu Wei, Yale University, New Haven, CT, USA; Jake Shaker, Yale University, New Haven, CT, USA; Kimberly Hieftje, Yale University, New Haven, CT, USA; Asher Marks, Yale University, New Haven, CT, USA. This work is licensed under a Creative Commons Attribution 4.0 International License. NIME ’25, June 24–27, 2025, Canberra, Australia ©2025 Copyright held by the owner/author(s). NIME ’25, June 24–27, 2025, Canberra, Australia Gancz et al. Fig. 1. A collage of hand gestures used to control the ScOTT 2.2 Hardware and software specifications We run ScOTT on Unity 2022.3.23f1 using a Windows 11 Home (23H2) computer (Intel i7-13700KF, NVIDIA GeForce RTX 4080, 32 GB RAM), with a Quest Link wire connection to a modified screen-free Meta Quest 3. We employ RtMidi for Unity1and loopMIDI2to route ScOTT’s MIDI output into Reason 103or Ableton Live 124. 2.3 Gestural mapping scheme ScOTT tracks the following relationships between parts of the performer’s body: •distance between the chest and left wrist (left protrusion; 𝑝𝑙) •distance between the chest and right wrist (right protrusion; 𝑝𝑟) •distance between the left middle fingertip and wrist (left closure; 𝑐𝑙) •distance between the right middle fingertip and wrist (right closure; 𝑐𝑟) •speed of the center point between the right middle fingertip and wrist (right speed; 𝑠𝑟) In either the Unity inspector or the graphical user interface, performers may select pitch-classes in standard tuning (from 0 to 11) and minimum and maximum MIDI pitch values (from 0 to 127) in order to define an ordered list 𝑃 of possible pitches that ScOTT can trigger. They may also select minimum and maximum velocity, tremolo frequency, and tremolo width values, as well as map parameters 𝑎𝑝and 𝑎𝑣, velocity modifier 𝑚, and closure threshold 𝑡. The magnitude of map parameters 𝑎𝑝 and 𝑎𝑣 represents how ’exponentially’ the available pitch and velocity values, respectively, are distributed with respect to their relevant tracked distance. For instance, a very high 𝑎𝑝 would cluster many low notes near the performer’s chest, leaving high notes sparsely distributed as they approach arm’s reach. A low 𝑎𝑝 would distribute pitches relatively evenly in space. 5 These live-tracked and preselected values trigger MIDI notes in accordance with the following transformations: pl𝑛𝑜𝑟𝑚 =clamp(norm(pl,pl𝑚𝑖𝑛,pl𝑚𝑎𝑥 ),0,1)pr𝑛𝑜𝑟𝑚 =clamp(norm(pr,pr𝑚𝑖𝑛,pr𝑚𝑎𝑥 ),0,1)(1) 1https://github.com/keijiro/jp.keijiro.rtmidi 2https://www.tobias-erichsen.de/software/loopmidi.html 3https://www.reasonstudios.com/reason 4https://www.ableton.com/en/live/ 5𝑎𝑝and 𝑎𝑣are meaningful in the exclusive range (1, ∞) Screenless Optical Theremin with Tremolo (ScOTT) NIME ’25, June 24–27, 2025, Canberra, Australia Fig. 2. The modified screen-free Quest 3 headset used to play ScOTT. cl𝑛𝑜𝑟𝑚 =clamp(norm(cl,cl𝑚𝑖𝑛,cl𝑚𝑎𝑥 ),0,1)cr𝑛𝑜𝑟𝑚 =clamp(norm(cr,cr𝑚𝑖𝑛,cr𝑚𝑎𝑥 ),0,1)(2) pitch𝑛𝑜𝑟𝑚 =logmap𝑎𝑝(pr𝑛𝑜𝑟𝑚)pitch𝑖𝑑𝑥 =⌊scale(pitch𝑛𝑜𝑟𝑚,0,∥𝑃∥)⌋ pitch =𝑃|pitch𝑖𝑑𝑥 (3) vel𝑛𝑜𝑟𝑚 =logmap𝑎𝑣(clamp(pl𝑛𝑜𝑟𝑚 +𝑚sr𝑛𝑜𝑟𝑚,0,1)) vel =scale(vel𝑛𝑜𝑟𝑚,vel𝑚𝑖𝑛,vel𝑚𝑎𝑥 )(4) tremfreq𝑛𝑜𝑟𝑚 =(0cr𝑛𝑜𝑟𝑚 ≤t cr𝑛𝑜𝑟𝑚 cr𝑛𝑜𝑟𝑚 >ttremfreq =scale(tremfreq𝑛𝑜𝑟𝑚,tremfreq𝑚𝑖𝑛,tremfreq𝑚𝑎𝑥 )(5) tremwidth =⌊scale(cl𝑛𝑜𝑟𝑚,tremwidth𝑚𝑖𝑛,tremwidth𝑚𝑎𝑥 )⌋ (6) Here vel indicates velocity, tremfreq and tremwidth indicate tremolo frequency in Hz and width in semitones, respectively, and the logmap,scale,norm, and clamp functions are defined as follows: logmap𝑎(𝑥)=1+log𝑎(1−1 𝑎)𝑥+1 𝑎scale(𝑥, 𝑎,𝑏)=(𝑎−𝑏)𝑥+𝑎(7) norm(𝑥, 𝑎,𝑏)= 𝑥−𝑎 𝑏−𝑎clamp(𝑥, 𝑎,𝑏)=max(𝑎, min(𝑥,𝑏)) (8) ScOTT also has an "invert pitch map" button, which rewrites the logmap function as follows: invlogmap𝑎(𝑥)=−log𝑎(1−1 𝑎)𝑥+1 𝑎(9) 3 Technical Notes We will present a prerecorded 10-minute composition for ScOTT, saxophone, and guitar, alongside a score-follower animation. If the conference schedule contains a dedicated streaming block for remote performances, we prefer to be included in this block. Otherwise, we request our performance to be made available either on a dedicated screen or via a link embedded into conference materials. NIME ’25, June 24–27, 2025, Canberra, Australia Gancz et al. 4 Media Links In our supplementary materials we include a demo video showcasing our initial workshop for ScOTT, clarinet, and guitar, as well as the graphical score composed by Michael Gancz for the official performance. Our performance video will be presented as part of conference proceedings, and we will make a stable link to this performance available post-fact. 5 Ethical Standards This project is partially funded by Reality Labs Research, and the headset we use is a Quest 3 that was modified and donated by Reality Labs Research. Acknowledgments The authors would like to thank Abigail Crocker for her clarinet performance in our demo recording, and Wes Lewis for his saxophone performance in our performance recording. This work was supported by Reality Labs Research and the Yale Center for Immersive Technology in Pediatrics. References [1] Peter Bennett, Nicholas Ward, Sile O’Modhrain, and Pedro Rebelo. 2007. DAMPER: a platform for effortful interface development. In Proceedings of the 7th international conference on New interfaces for musical expression. 273–276. [2] Perry Cook. 2017. 2001: Principles for designing computer music controllers. A NIME Reader: Fifteen years of new interfaces for musical expression (2017), 1–13. [3] Nicholas Ward, Kedzie Penfield, Sile O’Modhrain, and R Benjamin Knapp. 2008. A Study of Two Thereminists: Towards Movement Informed Instrument Design.. In NIME. 117–121.