Performance with Esteso: Interactive AI Music Duet Based on Player-Idiosyncratic Extended Double Bass Techniques

Performance with Esteso: Interactive AI Music Duet Based on Player-Idiosyncratic Extended Double Bass Techniques FILIPPO ANGELONI, Independent - Composer, Musician, Italy DOMENICO STEFANI, Dept. of Information Engineering and Computer Science, University Of Trento, Italy MATTEO TOMASETTI, Dept. of Information Engineering and Computer Science, University Of Trento, Italy Additional Key Words and Phrases: Extended Playing Techniques, Double Bass, Music AI, Interactive Music Systems ACM Reference Format: 1 PROGRAM NOTES Esteso is an interactive improvisational system for double-bass based on player-idiosyncratic extended techniques. It was created in collaboration with the contemporary double-bass player and composer Filippo Angeloni and tailored for his personal vocabulary of extended techniques. In Esteso, the musician and an AI counterpart take turns in an improvisational duet. The system’s response consists of a manipulation of the real double-bass, achieved live through a timbre-transfer neural network, granular synthesis, and reverberation. The timbre-transfer network was trained on generic double-bass recordings, resulting in a peculiar “hybrid” double-bass sound. We Machine listening is integrated in the form of a real-time classifier of extended techniques played on the double-bass, whose output controls the sound manipulation process to affect various techniques differently. The personal extended techniques chosen for Esteso are: "Brushed" Jeté,Sfregato con legno, and Percussive. Here, we propose a 10/15 minutes improvised duet performance where the double bass player interacts with Esteso, creating an action-reaction interplay between acoustic and virtual. 2 PROJECT DESCRIPTION The proposed system is affine to the “player” paradigm defined by Rowe [ 6 ], which defines an “artificial” player able to interact with human players. Esteso was implemented as a Max/MSP patch [ 5 ], and its structure can be broken down into three parts: (1) an extended technique recognition section, (2) a duet-mechanism, and (2) a sound manipulation stage. The purpose of the recognition section is to detect the use of different extended techniques from the musician and subsequently affect the system’s response. Secondly, the duet-mechanism is responsible for the action-reaction nature of the duet, enforcing simple rules that start and stop the system’s response. Finally, the sound manipulation stage is responsible for the sonic nature of the system’s response. A diagram of the system architecture is found in Figure 2. The playing technique recognition system is composed of a feature extraction stage, an onset detection stage, and a k-nearest neighbors classifier. The encoder of a RAVE model was used as a feature extractor, while a simple Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). Filippo Angeloni, Domenico Stefani, and Matteo Tomasetti. 2024. Performance with Esteso: Interactive AI Music Duet Based on Player-Idiosyncratic Extended Double Bass Techniques. 1, 1 (September 2024), 4 pages. Music Proceedings of the International Conference on New Interfaces for Musical Expression NIME’24, 4–6 September, 2024, Utrecht, The Netherlands 2 Filippo Angeloni, Domenico Stefani, and Matteo Tomasetti Fig. 1. Player interacting with Esteso Fig. 2. Architecture of Esteso peakamp object served as an onset detector. Repeated samples from the latent space of RAVE are collected upon onset detection and fed to the classifier. We focused on detecting the three aforementioned extended techniques and used the classification result to select different parameters for the sound manipulation stage. Performance with Esteso: Interactive AI Music Duet Based on Player-Idiosyncratic Extended Double Bass Techniques3 During the performance, the duet-mechanism enforces a simple rule: Esteso starts listening (therefore recording) whenever sound coming from the microphone is detected; audio is recorded to a 10-second buffer; when audio from the microphone is below a set silence threshold for 1.5 seconds, the system plays the recorded buffer, which is fed to the sound manipulation stage to produce the response. Buffer length and silence thresholds were found through experimentation with the double-bass player. Finally, the response to the musician’s phrases is obtained by manipulating the brief recordings provided by the duet mechanism. For this, we employ a granular synthesizer 1 , a RAVE timbre-transfer model [ 1 ], and a reverb effect 2 . The granular synthesizer was chosen as it can morph the temporal structure of input recordings, providing the musician with novel responses. Secondly, the timbre transfer model trained on generic double-bass recordings was found to produce peculiar sounds, which only partially resembled a real double-bass. We chose the mode as we felt its sound was interesting and found it akin to concepts as self-sabotaged instruments [ 2 ], sharing some similarities with De Souza’s class of redesigned-instruments [ 3 ]. We then used a reverb to give a sense of physical space to Esteso to differ from the dry double-bass sound. Selected parameters of the sound manipulation pipeline are affected by the results of the extended technique recognition classifier. This was mapped through a sound design process, where recordings of the musician were used to find effect parameters that would highlight each technique. Therefore, sets of parameter values were mapped to each possible output of the classifier. Apart from the performance proposed here, the system was formally evaluated through three performance sessions aimed at understanding the musician’s reaction and testing different sound manipulation mappings. NIME practices and techniques were adopted in the design of the experiments, the formal evaluation, and the extraction of themes from the musician’s comments. A scientific contribution regarding Esteso was proposed for the Call for Papers and accepted[ 7 ]. Therefore, we believe that this music performance with Esteso would support and be supported by the parallel scientific contribution and particularly fits this year’s edition of the NIME conference. 3 PERFORMANCE NOTES The performance will be available in one of 2 different modalities, depending on the musician and venue availability: (1) fixed media playback (audio+video); (2) live performance. The requirements for each respective modality are the following: 3.1 Fixed Media Playback We kindly ask the organizers to provide the following: •Projector, •Stereo loudspeaker system. 3.2 Live Performance: We kindly ask the organizers to provide the following: •Stereo loudspeaker system, 1The Max/MSP package “Petra”: https://github.com/CircuitMusicLabs/petra 2“Reverb-2” from the BEAP module package, by Matthew Davidson: https://github.com/stretta/BEAP 4 Filippo Angeloni, Domenico Stefani, and Matteo Tomasetti • USB soundcard with a minimum of 1 mono input (with +48V for condenser microphone) and 2 outputs (for speakers), •Audio cables to connect the sound card to the loudspeakers. •XLR cable, •A double-bass stool, •One table and two chairs (for laptop and soundcard, off-stage), •If possible, a double bass would be appreciated (it would solve many transport issues). Additionally, we will bring the following: •Condenser Microphone, •Laptop to run the patch, •(If it cannot be provided by organizers and can be brought with transport) a double-bass. 4 MEDIA LINKS •Short demo ( 4 mins): https://youtu.be/HEhJXAgFiXM •Full demo ( 15 mins): https://youtu.be/oncw3G4sLuM ETHICAL STANDARDS This study followed all ethical and data protection guidelines from the University of Trento. This paper complies with the NIME Conference standard [ 4 ]. The code is publicly available and referenced in the connected paper [ 7 ] and no external users were involved, as only the authors participated in the experiments. We declare no conflict of interest. REFERENCES [1] Antoine Caillon and Philippe Esling. 2021. RAVE: A variational autoencoder for fast and high-quality neural audio synthesis. CoRR abs/2111.05011 (2021). arXiv:2111.05011 https://arxiv.org/abs/2111.05011 [2] Teodoro Dannemann, Nick Bryan-Kinns, and Andrew McPherson. 2023. Self-Sabotage Workshop: a starting point to unravel sabotaging of instruments as a design practice. In Proceedings of the International Conference on New Interfaces for Musical Expression. Mexico City, Mexico, Article 9. http://www.nime.org/proceedings/2023/nime2023_9.pdf [3] Jonathan De Souza. 2017. Music at hand: Instruments, bodies, and cognition. Oxford University Press. [4] Fabio Morreale, Nicolas Gold, Cécile Chevalier, and Raul Masu. 2023. NIME Principles & Code of Practice on Ethical Research. https://doi.org/10. 5281/zenodo.7545682 [5] Miller Puckette. 1988. The Patcher. In Proceedings of the International Computer Music Conference. 420–429. [6] Robert Rowe. 1992. Interactive music systems: machine listening and composing. MIT press. [7] Domenico Stefani, Matteo Tomasetti, Filippo Angeloni, and Luca Turchet. 2024. Esteso: Interactive AI Music Duet Based on Player-Idiosyncratic Extended Double Bass Techniques. In (Accepted) New Interfaces for Musical Expression (NIME2024).