Absence

Absence — Performance for Audio–Visual Concatenative Synthesis DIEMO SCHWARZ, STMS Ircam–SU–CNRS, France Additional Key Words and Phrases: audio-visual multi-modal performance audio–visual performance inter-modal analogies crossmodal perception concatenative synthesis interactive visuals electro-acoustic music sound–image symbiosis ACM Reference Format: 1 PROGRAM NOTES The piece Absence evokes several notions of absence: Absence of light, absence of love, absence of humanity, when societies and all human artefacts will be destructured to gradually disappear within the materials and textures of the natural world, which, by themselves, will eventually dissolve back into primordial cosmic dust. The piece starts with the images of crepuscular cities, the buildings mere shadows, before evolving towards images of cities in ruins, warning of impeding and very possible catastrophies for humankind, to which sounds are played that evoke emptiness and desolation. These elements gradually dissolve into the basic textures of natural materials of stone, sand, water, wood, with the sounds exploring different sensations of materiality through roughness and textures. This path through different sentiments of absence is evoked in sound and images by a journey through intertwined corpora of sounds and images, linked by their cross-modal aesthetic properties of colour, texture, timbre, played via dynamic gestural control. The spatial projection of sound is an important part of the artistic research of this piece, unfolding across different levels: One layer of the electronic sound is based on field recordings, which are recontextualized, transformed, and projected live into space by gestural control. Another layer of sound consists of a pre-composed electro-acoustic soundtrack that creates the work’s basic ambience and immersion by dynamic surround sound spatialisation. The narrative of the piece reflects itself in the underlying digital fabric of the work insofar as the material nature of the physical world is represented in the form of acoustic and visual fragments (sound grains and still images) that act as a conglomerate of particles to form the shape of the whole. This symbiosis of different modes and levels of perception is nourished by the desire of subjectivity in order to emphasize the relativity of all experience. 2 PROJECT DESCRIPTION The piece Absence explores the symbiosis of audio–visual creation by installing a perceptual link between the two sensory modalities. These two nourish each other by the linked gestural navigation through spaces of audio and image characteristics. The performance thus probes human multi-modal perception by creating inter-modal associations in gestural navigation in joint audio and image corpora. It augments timbre-based sonic exploration by colour and Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). Diemo Schwarz . 2023. Absence — Performance for Audio–Visual Concatenative Synthesis. 1, 1 (May 2023), 2 pages. Music Proceedings of the International Conference on New Interfaces for Musical Expression NIME’20, July 21-25, 2020, Birmingham, United Kingdom 2 texture-based image exploration, sometimes with congruent inter-modal associations, following universal perceptive expectations, sometimes with contradictory ones. The work also proposes a new take on digital art by asking whether this performance should be seen as a digital artwork. At first sight this seems obvious — the whole system is treating digital sound, image, and movement representations algorithmically. However, looking at the content, this is not so clear cut anymore. The piece uses audio field recordings and photographic images taken in the real world for its sonic corpora. What’s more, the narrative and dynamic structure of the piece is created from gesture input, which means the physical body movment of the artist in the performance space. And finally, the output may well be projected via a digital projector and rendered from a digital audio interface, but the sights and sounds arrive at the audience through analog light and sound waves, crossing and mingling with the physical space of the performance. In the end, what is given to listen and see is a joint audio–visual recontextualisation of impressions of the real world, a symbiosis of the physical and digital. Another telling characteristic of the performance is that it is squarely opposed to all trends in modern computer graphics and sound synthesis, where everything is fluid, three-dimensional, parametric, and generative. Instead, just short sound grains and still images are used, the appearance of which is not given by parameters of a generative model, but by specifying the desired visual descriptors, modeled after the human sensory perception. And last, images are not generated by an artificial intelligence, but chosen by the human sensibility of the musician/AV performer. As such it offers an alternative approach to animation and narrative, creating evolution from instants, and movement out of stillness. 3 PERFORMANCE NOTES The performance’s duration is around 10 minutes. It requires HD video projection, ideally in an immersive setting with projection on 2 or better 3 sides surrounding the audience, and at least quadrophonic surround sound, ideally octophonic. The electronic musician should ideally be placed in the middle of the projection space. It would also be possible to project onto a planetarium dome, if such a venue is by miracle available. 4 MEDIA LINKS This video teaser documents the first performance of audio–visual concatenative synthesis with violin at the Marseille planetarium at the end of Diemo Schwarz’s IMéRA residency. This study was the basis of the piece Absence proposed here, which has been composed with a coherent audio–visual narrative and arranged for solo electronics+images. •Video: Performance for Audio–Visual Concatenative Synthesis & Violin: https://youtu.be/EFAN9fOofd0 ACKNOWLEDGMENTS This work was supported by the Arts, Sciences, Societies residency program 2021–2022 of the IMéRA Institute for Advanced Study, Aix–Marseille Université. The author would like to thank IMéRA, Observatoire de Marseille, Association Andromèede, Lionel Ruiz, Jeremie Brugidou, Laboratoire Perception, Représentations, Image, Son, Musique (PRISM), CNRS, Aix-Marseille Université, ISMM team, STMS, Ircam, Sorbonne Universités, CNRS, Ministère de la Culture.