MILL: the Mobile Instrument Library and Launcher

MILL: the Mobile Instrument Library and Launcher Nick Hwang Media Arts and Game Development University of Wisconsin-Whitewater 800 W Main St Whitewater, WI 53190 [email protected] Anthony T. Marasco College of Music Michigan State University 333 West Circle Drive East Lansing, MI 48824 USA [email protected] ABSTRACT We introduce MILL, the Mobile Instrument Library and Launcher. A browser-based musical performance platform that transforms participant smartphones into networked musical instruments, MILL provides composers and performers with a library of custom mobile instruments crafted using the Web Audio API, Cycling ’74’s RNBO, and common web interface elements with unique control schemes. A tool for designing participatory performances that harnesses the ubiquity of mobile devices, MILL dissolves the boundary between performers and audience: anyone with a browser can register, receive curated instrument patches, and follow dynamic scores pushed directly to their screens. Composers and ensemble directors gain fine-grained control over ensemble texture and form, cueing instrument changes or score updates in real-time, while participants interact through intuitive gestural interfaces. By uniting multi-user synchronization, preset management, and dynamic score presentation in a platform-agnostic web environment, MILL expands possibilities for both tightly scored works and spontaneous, audience-driven performance. 1. INTRODUCTION MILL, the Mobile Instrument Library and Launcher, is a system that enables composers and sound artists to create audience-based, interactive performances using mobile phones. Designed to streamline the process of creating networked musical performance (NMP) works, MILL empowers creators to make dynamic decisions about when interactive participation will occur in a piece, which networked ensemble members are permitted to participate, and what their sonic contributions will be. Composers can use MILL to assign performance roles to participants, remotely launching digital instruments and score materials in their browsers. Furthermore, control over orchestration decisions can be ceded to the performers by providing them with the option to choose their own instruments from a collection of synthesizers, samplers, and audio processors. MILL hosts digital instruments with audio engines designed in JavaScript or WebAssembly, embracing programmers who prefer to deLicensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: owner/author(s). Web Audio Conference WAC-2025, November 19–21, 2025, Paris, France. © 2025 Copyright held by the owner/author(s). velop entirely in the browser and those who wish to import their code from other environments, such as Cycling ’74’s RNBO1. A launcher interface enables composers to orchestrate performances on the fly, determining which instruments to use and when to deploy them to the audience’s phones at a moment’s notice, as well as how to convey performance instructions such as graphical, text-based, or music notational scores on their screens. MILL streamlines participant registration, instrument preset selection, and the realtime bidirectional control data transmission between audience members and onstage performers. 2. PRIOR SYSTEMS AND HISTORICAL CONTEXT Crowd-powered, browser-based music has become a staple of the modern computer music scene over the past decade. Leveraging the power of mobile phones as real-time instruments and networkable speakers, composers have explored a litany of methods for bringing participants together to create collaborative works of art with their personal devices. As music technology ensembles became more common in the early 2000s [20], frameworks were developed that allowed digital instruments to be recalled and distributed to networked performers for multi-section compositions, along with performance instructions (multimodal scores). Examples of systems that provide this service for ensembles of laptops and single-board computers include the Laptop of Orchestra of Louisiana’s GRENDL [4], the custom server used by the Virginia Tech Raspberry Pi Orchestra [7], the HappyBrackets framework [8], and the Ensemble script used to coordinate script loading on multiple Monome Norns instruments used by the New Music Technology Ensemble at the University of Texas Rio Grande Valley [15]. In developing MILL, we aimed to bring this same rapid instrument deployment feature to the web browser. While our existing suite of mobile device-specific systems Collab-Hub [14] and MoNoDeC [13, 12] demonstrate robust bidirectional data flow and seat-based diffusion between performance leaders and audience members, both tools either ship a single webaudio instrument/patch per show or leave the choice of digital instrument up to the performers; adding or switching to a new instrument requires redeploying code or launching new software. MILL layers a library and launcher on top of the same Collab-Hub transport, enabling instrument swaps and graphic score changes to be a one-click action during performance. 1https://rnbo.cycling74.com/ Figure 1: Server-Client Model for MILL From Levin’s Dialtones (A Telesymphony) [21] to massMobile’s browser-based polls [9], the NMP repertoire contains works that explore a variety of options along the performer-audience power balance spectrum, and prior research shows that careful consideration of this balance plays a crucial role in shaping the experiences of both parties. H¨ odl et al.’s Po`eme Num´erique project showed that while smartphones can be simultaneously utilized as a design tool, a passive device for the distribution of audio throughout the concert space, and an interactive instrument in a single performance, composers insisted on retaining high-level structural control over what actions audience participants were allowed to take. At the same time, H¨ odl’s survey notes that audience members feel empowered when their actions result in audible contributions to the work [11]. To address this issue, MILL’s controller-constrained parameter ranges and engaging on-device graphical user interfaces enable composers to grant meaningful interaction to participants without ceding control over form. Websocket transport through the internet eliminates the need for a lineof-sight ultrasound protocol used by H¨ odl. At the same time, manifest-level cues allow the composer to pre-author or improvise structural changes with MILL without compromising artistic intent. MILL extends this lineage by combining instrument libraries with dynamic score control in a lightweight, browserbased system. 3. MILL MILL stores and delivers web-based digital instruments to audience members’ mobile devices in real time during a concert performance. MILL relies on either a local or remote wireless connection to provide web-based interfaces to participants and is not data-intensive. When developing instruments for MILL, composers can assign and deploy them to audience participants (either in large groups or individually), constrain particular parameters (such as pitch range), and provide musical instructions to performers in the form of text or graphical notation. MILL builds onto Collab-Hub, a server-based system that enables web-based multi-user collaboration and relies on the internet to provide web-based interfaces for both audience members and the composer/controller. 3.1 Mobile Instrument Library MILL contains a collection of premade musical instruments with different control interfaces in its library. MILL instruments can be created with the Web Audio API, Figure 2: Instrument Interfaces, Left: Without Performance instructions, Right: With Performance instructions RNBO, and Web Audio API wrappers such as Tone.js. The audio synthesis engine and user interface elements are two separate layers within the overall structure of a MILL instrument. The presentation of instrument controls and performance instructions is implemented through standard HTML, CSS, and JavaScript practices. As the library of digital instruments expands, the various interaction methods will also grow. Touchscreen interface elements such as buttons, sliders, and knobs imitate real-world physical controls. Digital instruments on mobile phones can utilize data from a device’s gyroscopes, cameras, temperature sensors, ambient light sensors, microphones, and more as input controls. Figure 2 Displays two examples of instruments in MILL’s Mobile Instrument Library that demonstrate a variety of interactive designs. On the left of Figure 2 is a four-voice chord generator. The four buttons at the bottom of the window set the quality of the chord, the horizontal slider sets the root note chromatically across an octave, and the ADSR window above sets the envelope. The chord is triggered by tapping the button at the bottom of the window. On the right side of Figure 2 is a three-voice polyphonic synthesizer with a low-pass gate that produces short pitches through taps inside a pad-shaped interface at the top of the window. Each tap cycles through one of the three voices. Position across the X-axis indicates pitch of the voice, and position on the Y-axis is voice timbre (darker at the bottom and brighter at the top). The synth runs through a delay, and the delay time and feedback are adjusted in two of the three sliders at the bottom of the window. The third slider adjusts the decay time for notes played through all voices. 3.2 MILL Server and Instrument The MILL backend, which connects to the MILL Server (built using Collab-Hub protocols), allows for registration of individual devices by creating a unique identifier for each performance. The server works to reconnect users if their connection drops and handles connection drops. The MILL backend communicates with the server to receive data on which instruments to present or instantiate, as well as performance metadata such as the current performance, compositional section, and performance instructions. 3.2.1 Instrument Layer Instrument Layer instantiates a musical instrument that includes its audio context and presents the user interface after receiving data from the performance server. We will use the term ’controller’ to describe the person designing the performance. During performance, the instrument is performed by the device holders, but certain aspects of the instrument may be constrained or controlled by the controller. For example, the controller may constrain the range of a particular instrument or parameters, such as carrier frequency within a Frequency Modulation instrument. The visual aspects of the user interface are designed with common web interface elements and CSS. 3.2.2 Performance Instruction Layer The Performance Instruction Layer presents the performance instructions on the mobile devices. These instructions may include graphical elements, images, text, or musical notation. Depending on the design, not all performances may involve instructions. When visible, the performance instructions will occupy a portion of the device’s screen, and the Instrument Layer elements will resize proportionally to accommodate them. 3.2.3 Considerations Accessibility played a role in the design choices, including the use of web-based elements for MILL rather than developing native mobile apps. The authors aimed to develop a web-based system that did not require participants to download applications or possess knowledge of networking to participate. In the development considerations made while using the Web Audio API and dynamic interfaces, including the following Web Audio protocols, user-initiated actions are required to enable audio output. Additionally, users are provided with reminders to unmute their devices to ensure audio output, and the system notifies users if the web browser they are using is not compatible with a particular instrument. 3.3 The Controller Interface The Controller Interface allows for the design of a performance using MILL. This interface is a web-based interface that connects to the MILL backend and can be accessed on mobile phones as well as larger, browser-based devices, such as laptops or tablets. MILL has three phases, which are managed by the controller: design, registration, and performance. See Figure 3 for the Design Phase layout. 3.3.1 Design Phase Design Phase allows the controller to make performance decisions for the following parameters: •which instruments are involved in which sections, •the duration of each section and overall performance, •performance instructions to be presented Figure 3: Controller Interface during Design Phase The controller inputs text-based instructions or uploads media, like image or graphic files, for performance instructions. Each design is saved on the server and can be retrieved at any time. 3.3.2 Registration Phase Registration phase occurs just before performance, once the participating audience has gathered. After an initial ’Join’ option from a landing webpage, audience members’ interfaces display the phase name and instructions to unmute their devices and enable the Web Audio AudioContext. The controller can view the number of registered and enabled devices connected to the MILL server. The controller can proceed to the Performance Phase once the audience members have completed registration. 3.3.3 Performance Phase The Performance Phase begins when the controller presses the ‘Performance’ button and starts their performance. Based on their design, a dashboard-like interface will appear, displaying performance run time, the instruments being used, the current section number, and the current and upcoming performance instructions. Additional control elements include the ability to manually change section numbers and send on-the-fly performance information, such as when the performance has ended. 4. FUTURE WORK As we prepare to introduce MILL to a broader audience, we identified several areas for future development. Enhancements to the MILL server will allow us to increase the speed of reliable data transmission across the network and host larger groups of participants in each performance. While our current Mobile Instrument Library offers users a wide range of options to integrate into their own works, we plan to continue expanding the collection with additional samplers, synthesizers, and audio effects. To help users who are new to web development, a future version of MILL will aim to simplify the GUI design process. Composers will be able to mix and match code from MILL’s Mobile Instrument Library with a provided performance control template, or they can build their own control surface out of bespoke MILL interface elements. 5. CONCLUSION MILL opens up new avenues for composers to write both tightly scored works and spontaneous, audience-driven performances with mobile devices. By combining a multi-user mobile phone networking framework and DSP launcher system, MILL realizes a long-standing goal to merge a reusable instrument library and declarative score engine into a single platform where composers can rehearse, cue, and reshape a smartphone ensemble into an orchestrated, ever-morphing choir of instruments. 6. REFERENCES [1] J. Allison and C. Dell. Aural: A mobile interactive system for geo-locative audio synthesis. In Proceedings of the International Conference on New Interfaces for Musical Expression, Ann Arbor, Michigan, 2012. University of Michigan. [2] J. J. Arango and D. M. Giraldo. The smartphone ensemble. Exploring mobile computer mediation in collaborative musical performance. In Proceedings of the International Conference on New Interfaces for Musical Expression, pages 61–64. Zenodo, June 2016. [3] L. Austin. Sound diffusion in composition and performance: An interview with Denis Smalley. Computer Music Journal, 24(2):10–21, June 2000. [4] S. D. Beck, C. Branton, and S. Maddineni. Tangible performance management of grid-based laptop orchestras. In Proceedings of the International Conference on New Interfaces for Musical Expression, pages 207–210. Zenodo, June 2011. [5] T. Blaine and S. S. Fels. Contexts of collaborative musical experiences. In Proceedings of the International Conference on New Interfaces for Musical Expression, pages 129–134, Montreal, Canada, 2003. [6] N. Bryan-Kinns and P. G. Healey. Daisyphone: Support for remote music collaboration. In Proceedings of the International Conference on New Interfaces for Musical Expression, pages 27–30, Hamamatsu, Japan, 2004. [7] I. I. Bukvic. RPi Orchestra, Jan 2019. [8] A. Fraietta, O. Bown, S. Ferguson, S. Gillespie, and L. Bray. Rapid composition for networked devices: Happybrackets. Computer Music Journal, 43(2-3):89–108, 06 2019. [9] J. Freeman, S. Xie, T. Tsuchiya, S. Weibin, Y.-L. Chen, and N. Weitzner. Using massMobile, a flexible, scalable, rapid prototyping audience participation framework, in large-scale live musical performances. Digital Creativity, 26:1–17, 07 2015. [10] J. Harrison. Sound, space, sculpture: Some thoughts on the ‘what’, ‘how’ and ‘why’ of sound diffusion. Organised Sound, 3(2):117–127, aug 1998. [11] O. H¨ odl, C. Bartmann, F. Kayali, C. L¨ ow, and P. Purgathofer. Large-scale audience participation in live music using smartphones. J. New Music Res., 49(2):192–207, feb 2020. [12] N. Hwang and A. Marasco. MoNoDeC: The mobile node controller for audience-involved sound diffusion. In Proceedings of the 49th International Computer Music Conference, pages 146–149, Seoul, South Korea, 2024. [13] N. Hwang and A. T. Marasco. A networking framework and mobile node controller for sound diffusion with audience participation. Computer Music Journal, 48(1):51–64, 03 2024. [14] N. Hwang, A. T. Marasco, and E. Sheffield. Creating telematic musical performances through Max and The Web with Collab-Hub. In L. Joglar-Ongay, X. Serra, F. Font, P. Tovstogan, A. Stolfi, A. A. Correya, A. Ramires, D. Bogdanov, A. Faraldo, and X. Favory, editors, Proceedings of the International Web Audio Conference, WAC ’21, Barcelona, Spain, July 2021. UPF. [15] A. T. Marasco. Approaching the Norns Shield as a Laptop Alternative for Democratizing Music Technology Ensembles. In NIME 2022, Jun 16 2022. https://nime.pubpub.org/pub/ahhqdi2a. [16] A. T. Marasco and J. Allison. Soundsling: A framework for using creative motion data to pan audio across a mobile device speaker array. In J. Monschke, C. Guttandin, N. Schnell, T. Jenkinson, and J. Schaedler, editors, Proceedings of the International Web Audio Conference, WAC ’18, Berlin, Germany, September 2018. TU Berlin. [17] B. Matuszewski. Soundworks - a framework for networked music systems on the web - state of affairs and new developments. In A. Xamb´o, S. R. Mart´ın, and G. Roma, editors, Proceedings of the International Web Audio Conference, WAC ’19, pages 65–70, Trondheim, Norway, December 2019. NTNU. [18] D. Poirier-Quinot, B. Matuszewski, N. Schnell, and O. Warusfel. N¨ u Soundworks: Using spectators’ smartphones as a distributed network of speakers and sensors during live performances. In F. Thalmann and S. Ewert, editors, Proceedings of the International Web Audio Conference, WAC ’17, London, United Kingdom, August 2017. Queen Mary University of London. [19] S. Robaszkiewicz and N. Schnell. Soundworks - A playground for artists and developers to create collaborative mobile web performances. In S. Goldszmidt, N. Schnell, V. Saiz, and B. Matuszewski, editors, Proceedings of the International Web Audio Conference, WAC ’15, Paris, France, January 2015. IRCAM. [20] A. Tanaka. Mobile music making. In Proceedings of the International Conference on New Interfaces for Musical Expression, pages 154–156, Hamamatsu, Japan, 2004. [21] B. Taylor. A history of the audience as a speaker array. In Proceedings of the International Conference on New Interfaces for Musical Expression, pages 481–486, Copenhagen, Denmark, 2017. Aalborg University Copenhagen.