Magnetologues

Magnetologues GIACOMO LEPRI, Casa Paganini - InfoMus University of Genoa, Italy NICOLA PRIVATO, Intelligent Instruments Lab University of Iceland, Iceland Additional Key Words and Phrases: Live electronics, Neural Synthesis, Magnetic Interactions ACM Reference Format: 1 PROGRAM NOTES Stacco is a new NIME, embedding magnetic attractors that detect variations in the surrounding magnetic fields. It attracts and repels magnetic spheres displaced by the performer, allowing both fine musical control and the emergence of unpredictable interactions out of its interlaced magnetic forces. Stacco has been designed to perform with neural synthesis models, where multidimensional sonic spaces can be navigated through the exploration of tangled control parameters. It comes with two graphical scores that function as guides for the exploration of the neural synthesis model. By drawing on the surface of the interface, it is also possible to create new scores and layer them on top of each other. We propose a performance titled ”Magnetologues”, with two Staccos, one operating as a neural audio engine and the other controlling in real-time the sound in space. 2 PROJECT DESCRIPTION Neural synthesis (NN) is a new, exciting synthesis method whose distinctive features open new expressive possibilities as well as novel compositional methodologies. We propose a performance based on RAVE [ 1 ], an architecture offering (i) timbre transfer capabilities from a given audio prompt and (ii) real-time control through the direct manipulation of a multidimensional sound-space whose latent dimensions exhibit the following features: • Control parameters are deeply entangled: it is impossible to univocally map one input with one particular sonic feature. • The model’s latent dimensions are always active. Consequently, interfaces employing RAVE are ideally designed to offer continuous values rather than discrete Booleans. • The distribution of the features in the latent space is arbitrary. During training, the system autonomously extracts and maps the sound features into the latent space. These relations can be only explored empirically, a posteriori, by interacting with the trained model. Stacco (Fig. 1) is designed to deal with such affordances, allowing the user to perform with RAVE by directly manipulating its latent space via unlabeled, continuous controls. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). Giacomo Lepri and Nicola Privato. 2024. Magnetologues. 1, 1 (September 2024), 5 pages. Music Proceedings of the International Conference on New Interfaces for Musical Expression NIME’24, 4–6 September, 2024, Utrecht, The Netherlands 2 Trovato and Tobin, et al. Fig. 1. Stacco. Under the hood, Stacco features four three-axis magnetometers and as many magnetic attractors, it performs embedded synthesis or can forward its readings to a laptop for more demanding tasks. More specifically, Stacco is an inherent score [ 4 ] with embedded magnets and sensors underneath an engraved surface. The engraved board reveals the position of the four sensors and their magnetic fields through a series of graphic patterns. It consists of four miniaturised magnetic attractors combining a three-axis magnetometer with a permanent magnet each. Each sensor performs two-dimensional readings of nearby magnetic fields whilst actively attracting and repelling nearby magnets and ferromagnetic objects. The sensors are displaced in four symmetrical points underneath an oval wooden board. Overall, Stacco provides 8 continuous streams of data which relate to the x and y coordinate and strength of the magnets detected. We use Stacco to perform with a series of overlapping RAVE models. In most models, the 8 channels are mapped one-to-one to the latent dimensions of each model, with some doubling the mapping to the remaining latents. We use Stacco as the ideal controller to play with NN models such as RAVE, as it allows to finely control high-dimensional data through simple gestures, provides continuous readings, simplifies the mapping to RAVE’s latent space, and allows sketching and embedding gestural notation into the interface itself. Magnetologues 3 Fig. 2. Stacco’s graphical inscriptions 3 SCORES The performer interacts with Stacco by throwing and displacing on the board a series of magnetic spheres of variable dimensions. The sonic features of the model are then learned and manipulated through a gestural exploration of the magnetic fields which, depending on their polarity, attract or repel the magnets placed on the device. Stacco comes with two basic scores, both artistic interpretations of the interface’s magnetic fields (Fig. 2). It also allows the incorporation of tailored oval sheets with graphical inscriptions and personalised sketches using custom oval sheets of tracing paper, functioning as guides for sonic exploration. This allows the creation of personalised scores and the stacking of multiple layers on top of each other as interchangeable scenes and sketches (Fig. 4), thus aligning the notational praxis to the aforementioned, novel features of this technology (See paper submission). Since the spheres interact with the instrument through their magnetic field, the presence of the oval sheets does not change the instrument’s ergodynamics [ 2 ] whilst providing an immediate and intuitive approach to notation. Because of the connection between gesture and notation, the act of composing becomes intrinsically idiomatic to the interface [3]. 4 PERFORMANCE NOTES We propose a duo performance, duration of 12’ ca., with two Staccos, one operating a neural synth and the other controlling in real-time the sound in space via Ambisonics. A camera (e.g. high-quality webcam) will provide a close-up live video stream of the two instruments, which will be projected on a large screen behind the performers, thus allowing the audience to see in real-time the gestures performed on the two interfaces (Fig. 3). We curated a series of models - including recordings of choir performances, organ sounds and magnets catching with other ferromagnetic objects. 1 . In order to control and map multiple models, we developed various scores - i.e. oval sheets with embedded inscriptions referencing position and gestures in the models’ latent spaces. 1https://huggingface.co/Intelligent-Instruments-Lab/rave-models 4 Trovato and Tobin, et al. Fig. 3. A close-up photo of Nicola Privato performing with Stacco - Łodz (Poland) @ Act In Out - 2023 Fig. 4. An example of a custom graphical notation created for a previous Stacco performance. Magnetologues 5 Because it maps the x and y position of magnetic spheres around four attractors, Stacco is also an ideal tool for notating and performing spatialised sound. We will therefore define four sound sources (one per attractor) mapped to one or more RAVE models each, and while one performer will be playing an embodied sketch to synthesizing the sounds, the other will perform another to move it into space. 5 MEDIA LINKS A previous performance, featuring some of the synthesis models we developed, can be accessed at this link. 6 TECHNICAL RIDER • Settings: One large table for the two interfaces and two laptops; 2 power sockets; Video projector and Projection Screen. • Audio Output: As we intend to use ICST ambisonic plugin for Max, we can flexibly adapt our performance to the speakers system of the main concert hall (our own interface support a max of 8 output channels). The multichannel setup of our performance may therefore vary starting from minimum of 4 speakers (2 front and 2 rear). •Video Output: HDMI. 7 ETHICAL STATEMENT The models developed for the performance were trained with open-source datasets or/and with the informed consent of the authors/researchers that created them. ACKNOWLEDGMENTS The authors would like to thank the staff at Intelligent Instruments Lab for their generous support throughout the ideation and design of Stacco. In particular we wish to thank: Victor Shepardson, Halldor Ulfarsson and Sean Patrick O’Brien. REFERENCES [1] Antoine Caillon and Philippe Esling. 2021. RAVE: A variational autoencoder for fast and high-quality neural audio synthesis. arXiv preprint arXiv:2111.05011 (2021). [2] Thor Magnusson. 2019. Sonic Writing: Technologies of Material, Symbolic and Signal Inscriptions. London I& New York: Bloomsbury Academic. [3] Koray Tahiroglu, Michael Gurevich, and R. Benjamin Knapp. 2018. Contextualising Idiomatic Gestures in Musical Interactions with NIMEs. In Proceedings of the International Conference on New Interfaces for Musical Expression, Thomas Martin Luke Dahl, Douglas Bowman (Ed.). Virginia Tech, 126–131. [4] Enrique Tomás and Martin Kaltenbrunner. 2014. Tangible Scores: Shaping the Inherent Instrument Score. In Proceedings of the International Conference on New Interfaces for Musical Expression. Goldsmiths, University of London, London, United Kingdom, 609–614. https://doi.org/10.5281/zenodo.1178953