Translating Idiosyncratic Dance Principles Into Music and Light

Translating Idiosyncratic Dance Principles Into Music and Light Daniel Bisig [email protected] Zurich University of the Arts Zurich, Switzerland [email protected] Instituto Stocos Madrid, Spain Pablo Palacio [email protected] Instituto Stocos Madrid, Spain Muriel Romero [email protected] Instituto Stocos Madrid, Spain ABSTRACT The performance work Embodied Machine employs several idiosyncratic dance principles as common foundations between choreography, music, and light. The establishment of these foundations is based on a translation of dance principles across different modalities. This process of translation involves the identification of the core aspects of each dance principle, the abstraction of these aspects into computational formalisms, and the re-mediation of the formalisms into perceivable outcomes through sound synthesis, music composition, light, and interaction design. This publication describes the selected dance principles and the process of their translation. It also provides information about the artistic and scientific context that is relevant for this work, namely the role of dance principles in contemporary dance, the inherent proximity between the moving body and music, the methods available for formalising dance principles, and examples of computational approaches for translating dance prinicples. Finally, the publication describes several scenes of the dance performance and highlights the principles that form part of each of them. CCS CONCEPTS •Applied computing → Performing arts;•Human-centered computing →Interactive systems and tools. KEYWORDS dance principles, dance and technology, sound synthesis, generative art ACM Reference Format: Daniel Bisig, Pablo Palacio, and Muriel Romero. 2024. Translating Idiosyncratic Dance Principles Into Music and Light. In Audio Mostly 2024 - Explorations in Sonic Cultures (AM ’24), September 18–20, 2024, Milan, Italy. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3678299.3678346 1 INTRODUCTION From 2020 to 2022, the authors of this paper collaborated on the realisation of a performance work entitled Embodied Machine (EM), This work is licensed under a Creative Commons Attribution International 4.0 License. AM ’24, September 18–20, 2024, Milan, Italy ©2024 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-0968-5/24/09 https://doi.org/10.1145/3678299.3678346 a production of the dance company Instituto Stocos 1 which the authors are members of. The work was premiered in 2022 as part of the extended program of ISEA international 2 .EM revolves around the notion of the stage as an embodied entity that manifests through music and light. The stage performs alongside a solo human dancer. EM represents the main artistic outcome of a Marie Curie Fellowship entitled E2-Create 3 .E2-Create combines research and artistic creation for the purpose of analysing, abstracting, and transforming principles of embodied creativity in dance into concepts and methods for creating computer-based art. The research was based on the premise that a deepened understanding of contemporary dance can foster novel creative approaches in other artistic domains and vice versa, how these other domains can extend the range of creative possibilities that are at the disposal of contemporary dance. As outcome of E2-Create,EM plays the dual role of an artistic work and a representative example of how contemporary dance and computer-based art can inform each other. More specifically, EM inquires if algorithmic and generative methods can capture aspects of contemporary dance and translate them into other modalities than the human body, and if such a translation can strenghten the conceptual, formal, and aesthetic connections between dance and generative media in a performance work. For this inquiry, a set of idiosyncratic dance principles have been selected that the third author employs in her choreographic work. The chosen target modalities are music and light. Three types of translations are experimented with: translations through which some of the charactistics of dance principles are revealed, translations through which the presence of a body is evoked, and translations that operate with different levels of autonomy. This publication describes the steps that led to the realisation of EM: the analysis of the core characteristics of the chosen dance principles, the abstraction of these characteristics into formalisms that can be represented by computer simulations and algorithmic procedures, and the translation of these representations into perceivable outcomes through sound synthesis, algorithmic composition, and light control. The publication also provides information about the artistic and scientific context that is relevant for this work. Finally, the publication describes several scenes of the dance performance and highlights the dance and translation principles that form part in each of them. 1Instituto Stocos: www.stocos.com 2ISEA 2022 extended program: isea2022.isea-international.org 3E2-Create: wp.coventry.domains/e2create 457 AM ’24, September 18–20, 2024, Milan, Italy Daniel Bisig, Pablo Palacio, and Muriel Romero 2 BACKGROUND Several artistic and scientific contexts are relevant for this publication, namely the role of dance principles in contemporary dance, the inherent connections between the human body and music, the methods available for formalising and quantifying dance principles, and examples of existing computational approaches for translating dance principles into other modalities than the human body. 2.1 Dance Principles in Contemporary Dance Contemporary Dance emerged in the 1980s as a reaction to the rigid and restrictive nature of classical dance forms, especially Ballet [ 15 ]. To distance itself from these conventions, Contemporary Dance embraces creative experimentation, improvisation, and self-expression [32]. Contemporary Dance has a distinct attitude towards the human body and its movements, which are considered an artistic art form in their own right [ 21 ]. This perspective has led to the emancipation of dance from its subservient role for storytelling, exhibiting athleticism, or supporting social conventions. The focus on body movement has led to the development of various movement vocabularies and methods for ideating movements. Some of these vocabularies and methods have become codified as techniques, the most influential of which are the Release technique [ 26 ], Graham technique [ 11 ], Cunningham technique [ 29 ], Limon Technique [ 33 ], and Gaga Technique [ 18 ]. Several techniques employ unconventional movements that are for example rapid and contrasting (Cunningham technique), unstable or temporally suspended (Limon Technique), or alternate between contraction and release (Graham technique). Often, techniques explore and push the limits of the human body to break with habitual movement patterns (Gaga technique) or exploit the body’s full movement capabilities (Release Technique). The use and refinement of the dancers’ kinesthetic awareness also plays a central role in many techniques. Examples include guided improvisation to direct bodily attention (Gaga technique), focus on the perception of the body’s center of gravity or systematic breathing (Limon technique), or work with mental imagery of kinesthetic experiences (Release technique). 2.2 Connections between Body and Music The origin of acoustic instruments is closely linked to the human body, which provides the necessary coordination and awareness to play them. Playing an acoustic instrument can be likened to a complex choreography, where the movements translate into the sound domain through sensory-motor coupling with the instrument’s bodily expressivity. The earliest examples of this connection can be traced back to the Upper Paleolithic, where there is evidence of a correspondence between paintings and resonating locations in caves [ 31 ]. These caves were not only places for playing music or working with tools, but served as resonators for movements similar to those of a violin [ 14 ]. Another historical example of the connection between dance and music dates back to 4500 BC with a technique called Quironomy, which was practiced in ancient Egypt. This technique involved hand gestures that controlled every musical aspect for instrument players and is considered the basis of Western notation. [5]. Theories of image schemas provide insights into how most musical principles emerge from an unconscious understanding of our environment in connection with our body [ 25 ]. For example, the perception of rhythms is based on a disparity of durations, intensities, and pitches that require at least two sonic events to be recognized. For example in human walking, the first event is an elevation and the second is a fall. In musical terms, this corresponds to an upbeat and a downbeat, tension and release [ 6 ]. Musical principles relating to verticality also emerge from image schemas. As Graham and Bridges explain in [ 19 ], these image schemas are closely linked to the archetypal sonic dynamic morphologies that Smalley describes in his theory of Spectromorphology [35]. Aspects of Laban Movement Analysis (see 2.3) describe the stress patterns and internal flow of body movements that can be related to the articulation of music and sound. In traditional Western music notation, these qualitative aspects are mainly concerned with how sounds are connected or stressed, which are also important in the articulation of spoken language. Here, the stress patterns and other components belong to what is defined as paralanguage and give rise to expressive aspects of speech that seem to be independent of verbal content [ 7 ]. Similarly, movement quality carries information about the dancer’s expressivity and possesses a degree of independence that allows it to be imposed on different movement trajectories and alter its perceived characteristics [20]. 2.3 Formalisation of Dance Principles The formalisation of dance principles is important in the context of this paper for two reasons. It helps to identify the aspects of dance that should be preserved in the target modality of translation, and it provides the prerequisites for developing algorithmic procedures for analysing and translating dance principles. There exists no commonly accepted taxonomy for characterising and classifying dance. Nevertheless, the Laban Movement Analysis (LMA) introduced by Rudolf Laban [ 36 ] has gained prominence. LMA is widely adoped by dance scholars and has also informed technical fields such as robotics and human computer interaction. LMA formalises different aspects of the human body and groups them into four categories: Body,Space,Shape, and Effort [ 28 ]. The Body category describes body activities, the involved body parts, and their level of coordination. The Effort category describes the dynamic and qualitative aspects of movements. The Shape category describes how the body changes shape. The Space category describes the direction and location of movement. Several computer scientists have developed computational procedures for deriving high level descriptors of bodily activites from low level physical quantities. Cammurri et al. proposed a multi-layered conceptual framework to analyse movement qualities [ 12 ]. This framework consists of four layers with layer 1 representing direct or indirect physical quantities obtained from sensor-based measurements, layer 2 consisting of low level time-varying features, layer 3 dealing with mid-level features obtained through perceptual integration, and layer 4 addressing non-verbal communication of expressive qualities. The mid-level features in this layer are amodal descriptors that can be used to design mapping strategies from the domain of bodily movement into the sonic domain. 458 Translating Idiosyncratic Dance Principles Into Music and Light AM ’24, September 18–20, 2024, Milan, Italy Larboulette and Gibet have published an extensive collection of algorithms as part of their review of computable descriptors of human motion [ 22 ]. This collection comprises low level motion descriptors that can be directly calculated from raw motion data and higher level descriptors that can be used to analyze the meaning, style, or expressiveness of motion. The computable descriptors that deal with the four factors of the LMA Effort category (Weight,Space, Time, and Flow) have been adopted for the realisation of the dance piece described in this article. Another exhaustive collection of algorithms for analyzing human behaviour has been developed as part of an attempt to classify dance movements with regards to emotion [ 4 ]. The algorithms derive from motion capture data the geometric and stylistic features of dance movements based on LMA. These features were subsequently classified as emotional states. 2.4 Translation of Dance Principles The translation of dance into other modalities offers a wide range of creative applications. This section provides an overview of exemplary projects that employ dance principles as core elements of a translation that is accomplished through computational means. Many projects experiment with the translation of elements from LMA into other modalities. Three such projects are briefly mentioned here. Larboulette and Gibet assessed if the behaviour of a nonanthropomorphic entity in the shape of a tree can express the emotional content of an originally human motion [ 23 ]. For this purpose, basic movement descriptors and Laban Effort Factors were extracted from recordings of theatrical movements executed with different emotional states. These movement properties were subsequently mapped on the parameters of a mass-spring system and a particle simulation that control the morphology and appearance of the artificial tree. Lockyer and colleagues conducted a user experience design study to evaluate how concepts from LMA can inform the creation of expressive motions for abstract objects [ 27 ]. The authors deconstruct the LMA concepts of Shape, Space, and Effort into parameters that can be expressed through established motion algorithms such as motion trajectories and flocking simulations. Fehr and Erkut employed Laban Effort Factors for designing the interaction between a visitor and an audio-visual installation [ 17 ]. In this installation, a visitor interacts through hand movements with a flocking simulation which in turn controls the creation of synthetic audio and video. The visitor’s hand movements are analysed with regards to four movement qualities derived from Laban Effort Factors: Activity,Energy,Directivity, and Consistency. These movement qualities are translated into different behavioural states that affect how flocking agents react to the visitor’s hands. Carlson and colleagues developed a system entitled Scuddle that generates incomplete movement data in the form of abstract graphics [ 13 ]. This movement data is meant to serve as catalist for a choreographer’s creativity. The system employs a genetic algorithm whose fitness function incorporates among others LMA categories. The fitness function evaluates the generated movements according to body symmetry, position, and levels and favours contralateral movements and unstable levels. Several choreographers have initiated collaborative research projects to explore means of translating core aspects of their own idiosyncratic dance principles into digital media. Synchronous Objects is a pioneering project that focuses on the cuing mechanism involved in the counterpoint dance principle by choreographer William Forsythe [ 34 ]. Based on Forsythe’s dance piece One flat thing, reproduced, the project translates this principle into digital media objects such as graphical video-overlays, abstract visualisations, or generative simulations. These translations render this choreographic principle accessible for communication and exchange across different artistic and scientific disciplines. The Double Skin / Double Mind method consists of a set of ideation techniques that have been developed by choreographer and director Emio Greco and Pieter C. Scholten to prepare dancers for the creation of new movement material. The techniques involve a set of movement principles that deal with mental imagery related to the sensation of movement, the intentional aspect of movement, and the meaning of the action that is involved in movement [ 30 ]. Alaoui and colleagues have translated several of these principles into modalities for interaction for a proof of concept installation entitled A light touch [ 1 ]. The principles Breathing,Expanding,Reducing are extracted by analysing the dynamics and energy of the visitor’s hand movements. These properties are mapped on the position and brightness of the light spot. Alaoui and colleagues have also realised an educational installation entitled Double Skin / Double Mind to familiarize dancers with several movement principles [ 2 ]. The installation attempts to recognise in the movement of the visitor the qualities Breathing,Jumping,Expanding, and Reducing. If it succeeds to do so, it mirrors the recognised quality through the behaviour of a simulated mass-spring system. Choreographer Wayne McGregor involves his dancers in the creation of new movement material by giving them tasks in the form of verbal or written instructions. This tasks are related to mental imagery which dancers explore through mental manipulation techniques. Following this approach, dancers ideate movements that are guided by their perceived and imagined surroundings [ 24 ]. The systems Choreographic Language Agent and Becoming were developed in the context of a long-term research project that aimed to understand the mental and physical processes involved in the choreography of McGregor. The Choreographic Language Agent (CLA) plays the role of an "extended interactive notebook" [ 16 ] that augments the process of movement creation. CLA interprets the natural language instructions of McGregor as abstract geometrical animations. These animations serve as inspirations for dancers to create new movements. Becoming realises a software agent that possesses a simulated body and generates unique solitions to McGregor’s tasks in a fully autonomous manner [ 24 ]. Becoming was rendered as abstract geometrical animation which was displayed on a human-scale screen during dance rehearsals. The presence of Becoming at rehearsals elicits in human dancers kinaesthetic and emotional responses. The authors of this publications have collaborated with researchers from InfoMus Lab on the translation of qualitative aspects of movement into music. Several of the movement qualities that were selected in this collaboration play an important role in the third author’s choreographic work. 459 AM ’24, September 18–20, 2024, Milan, Italy Daniel Bisig, Pablo Palacio, and Muriel Romero The research project Dance studied the representation of movement qualities in the auditory channel for the purpose of enabling blind people to perceive movement through sensory substitution. Among others, they experimented with a movement qualitity named Fluidity with they translated into audio through movement sonification [3]. In the dance piece Piano & Dancer, the third author performs alongside an electromechanical piano, whose playback she controls through full body gestures [ 9 ]. The interaction between dancer and piano is based on a real-time analysis of the movement qualities Energy,Weight,Smoothness, and Dynamic Symmetry. The detection of these qualities controls the application of compositonal algorithms that transform the score of an existing piece into a new score for the piano. 3 SELECTED DANCE PRINCIPLES Over the course of her career as dancer and choreographer, the third author has developed her own vocabulary of dance principles. These principles rely on the use of movement qualities that she employs as choreographic building blocks in her creative process. The first two authors familiarised themselves with some of these dance principles by conducting interviews with the third author and recording demonstrations of her dance principles through motion capture. Subsequently, it was decided to create a taxonomy that organises the movement qualities based on three categories: Domain,Level, and Hierarchy The category Domain specifies the aspect of movement that is foregrounded by a movement quality. The following properties are distinguished: Dynamics,Shape,Time,Body, and Space.Dynamics deals with the dynamics of movement, focusing on changes in velocity and directionality. Shape includes partial transformations of movements or poses that preserve some of their recognisability. Time addresses the temporal organisation of movement or the absence of movement. Body concerns the juxtaposition of movements and poses within and across the body. Space focuses on how movement traverses space or is located at positions within space. The category Level describes how closely related a movement quality is to basic physical or geometric attributes. A movement quality that can be described solely by such attributes is considered low level, otherwise it is considered medium or high level. The category Hierarchy represents the hierarchical complexity of a movement quality. Some movement qualities are considered elementary in the sense that they can be performed in isolation and independently of other movement qualities. Other movement qualities are more complex in that they form composites of other movement qualities. This taxonomy proved useful in chosing a subset of movement qualities for translation. The artistic criteria involved chosing movement qualities that represent a balanced distribution of domains and hierarchical complexities. The techical criteria dealt with identifing movement qualities that vary with regards to their proximity to motion quantities that can be directly derived from motion capture. The taxonomy of those movement qualities that have been chosen for translation is shown in figure 1. Video and motion capture Figure 1: Dance Principles - The dance principles that have been translated into other modalities. recordings of these movement qualities have been made publicly available as dataset 4. 4 TRANSLATION PRINCIPLES The methods that have been chosen in EM to translate dance principles into music and light can be grouped into three main approaches: translations that reveal dance principles, translations that evoke the presence of a body, translations that establish autonomy relationships between dancer and modalities. 4.1 Reveal Dance Principles through other Modalities Rendering dance principles perceivable through other modalities requires the identification and preservation of those aspects of dance that are independent of the medium through which they manifest. By doing so, the artistic potential of dance principles can be extended. Without the strong presence of a human body, dance principles can be brought to the forefront of attention. Dance principles can be shared across modalities to establish strong conceptual and aesthetic consistencies. Dance principles gain universality and can inspire new creative approaches in other artistic disciplines. 4Movement Qualities Dataset: zenodo.org 460 Translating Idiosyncratic Dance Principles Into Music and Light AM ’24, September 18–20, 2024, Milan, Italy 4.2 Representation and Abstraction of a Human Body in other Modalities In dance, the human body is the main carrier of meaning and expressivity. Human bodies are simultaneously physical, biological, social, and aesthetic entities to which audiences can relate to through a combination of empathic reactions and cultural readings. By evoking aspects of bodily presence through other modalities, these modalities become available for anthropomorphising projections. Furthermore, by treating modalities as bodies, these modalities become accessible to choreographic work. 4.3 Autonomy Relationships between Dancer and Modalities Modalities can also evoke the presence of one or several entities that exhibit autonomy and agency. In order to possess agency, an entity has to be endowed with perceptual and behavioural capabilities of its own. Depending on the level of agency, different types of relationships can be established between dancer and entity. On one extreme, the entity lacks autonomy and a dancer has full control over it. This situation establishes an instrument playing metaphor in which actions performed by a dancer influence the entity in a direct and fully predictable manner. On the other extreme, both dancer and entity possess full autonomy and behave independently of each other. On a middle ground, dancer and entity possess partial autonomy and are dependent on each other. 5 MODALITIES AND ALGORITHMS The main modalities employed in EM other than the human body are music and light. The dancer’s movements are recorded using live motion capture and subsequently analysed to derive low level physical and higher level expressive aspects. The music of EM is synthetically created and involves pre-composed parts and live movement sonification. Robotic moving lights and laser light are employed in EM. The movements of the robotic lights and the visual shapes displayed via laser light are either interactively controlled by the dancer or generated using computer simulations or algorithmic patterns. The setup of the stage is shown in figure 2. 5.1 Motion Capture and Movement Analysis The movements of the dancer are live motion captured using either a marker-based optical system (12 Qualisys MIQUS cameras and a slightly extended Animation Marker Set) or inertial measurement units (XSens Awinda system consisting of 18 sensors). From motion capture, a kinematic abstraction of the dancer is obtained in the form of joint positions and rotations. This data is processed by an analysis software 5 that has been developed by the first author to derive the following descriptors of movement: linear and angular velocity, acceleration, jerk of individual body joints, and the four Laban Effort Factors (Weight,Space,Time, and Flow) of individual body parts and the full body. The descriptors are sent to the other software components using the open sound protocol (OSC) . 5Motion Capture Analysis: github.coventry.ac.uk Figure 2: Stage Setup - The image on the left shows a photo of a performance situation in which all robotic lights emit light. The image on the right depicts a schematic representation of the stage. Different objects are represented by different colors: robotic lights in red, motion capture cameras in green, loudspeakers in violet, and laser show projectors in green. 5.2 Algorithmic Music and Sound Synthesis The sound synthesis models and algorithmic procedures have been implemented by the second author using the Supercollider programming framework. The models employ processes that are either of a choreographic nature in themselves such as random walks or that simulate a physical activity that excites a sounding object such as the airflow inside a tube. Some models employ non-standard approaches to sound synthesis while others employ physical modeling synthesis. Non standard synthesis lends itself to the translation of high level dance principles into algorithmic compositional procedures. Physical modeling synthesis can translate the physical aspects of dance gestures into physical aspects of sound gestures. The sound output of these models is spatialised using an octophonic loudspeaker array. This offers the possibility to work with and combine spatial trajectories in dance and music. 5.2.1 Interactive Sonification. Most of the music in EM employs interactive sonification to translate the dancer’s activities into the auditory domain. Different approaches have been chosen to work with motion capture data. Sometimes, the captured data is treated as a cloud of coordinated points travelling across space. In other instances, specific joints are selected and grouped to extract higher level movement aspects or relationships. The descriptors resulting from the movement analysis are combined and mapped in a manner that is specific for each synthesis model. 5.2.2 Chromatic and Microtonal Sieves, Polyrhythms, Stochastic Decisions for Timbres. In composed sections of EM, several algorithmic abstractions are used to articulate pitches, harmonies, rhythms and timbres. The pitch is treated in two ways. One method divides the octave into 48 parts which are either chromatically chosen and created using microtonal sieves. The other method treats pitch as a continuum and employs glissandi and mobile sounds. A particular approach to polyrhythms has been chosen in which multiple 461 AM ’24, September 18–20, 2024, Milan, Italy Daniel Bisig, Pablo Palacio, and Muriel Romero voices hover around each other inside measures that expand and shrink using stochastic decisions. All synthesis models are used in a manner that causes timbre to continuously envolve. 5.2.3 Simulation of Wind Instruments. The most predominant sound synthesis model is an extended digital waveguide model that employs a combination of feedback and nonlinear filtering (online audio example 6 ) ). This model can produce a wide sound palette such as wind instruments played using conventional or extended techniques or natural sounds such as breathing or air travelling through a glacier. By changing synthesis parameters, the noisy transients and stable pitch components of sounds can be individually controlled. 5.2.4 Simulation of Bowed and Plucked String Instruments. Bowed string sounds are generating using a sound synthesis model that employs several filtering methods to produce the harmonics above a fundamental (online audio example 7 ). Each harmonic’s frequency centre is stochastically perturbed by a random walk generator. This perturbation is always less than a 20% of the corresponding centre, allowing tonal fusion and the perception of all the components as a single sound with defined pitch. A physical model of a plucked instrument is employed to create two types of sounds. One type of sounds is created by a plucked monochord. The other type of sounds is created by a massive clavichord. The timing and pitch of these sounds gradually change from order to dis-order following a gaussian distribution. 5.2.5 Dynamic Stochastic Synthesis. The Dynamic Stochastic Synthesis (DSS) model is an extended version of an original model that was initially devised by Iannis Xenakis (online audio example 8 ). The model uses probability functions to generate constantly changing sound pressure curves that mimic the liveness and complexity of natural sounds. Typically, multiple models are employed to create sub-waveforms (called Gendys in the terminilogy of Xenakis) that are concatenated. Two versions of DSS are used. The first version employs five Gendys, each one corresponding to an individual body part of the dancer. The second version employs one Gendy for each joint in the motion capture recording. Both versions offers rich possibilities to translate the fragmentation and orchestration of dance principles across different body parts into sound. 5.2.6 Human Voice Synthesis. Formant Wave Function (FOF) Synthesis is employed to model vocal sounds. Here, an extended form of FOF synthesis is employed that integrates several layers of stochastic and waveshaping functions to add liveness to the sounds (online audio example 9 ). The synthesis parameters that are varied through dance include pitch (in multiple tessituras), amplitude, brightness, and vibrato. 5.3 Light Setup and Control The light setup for the performance consists of 16 robotic moving lights (Elation Lighting ACL 360I) and 2 Laser Light Show Projectors (Veneno 7W RGB). The robotic lights form two rings surrounding the performance space, with 8 lights in each ring. A ring with a 6Waveguide Synthesis Example that Models a Wind Instrument: youtube.com 7Physical Modeling Example of a Plucked Instrument: youtube.com 8Dynamic Stochastic Synthesis Example: youtube.com 9FOF Synthesis Example that Models Human Vocals: youtube.com diameter of 5 metres is on the ground and a second ring with a diameter of 3 metres is at a height of 7 metres. Two laser projectors are placed above the centre of the stage at a height of 8 metres. The light setup is complemented with two hazer machines (ROBE Haze 500 FT Pro) to render the light beams visible. The rotation, colour, and brightness of the robotic lights are controlled via DMX directly from a laptop. The size, rotation, colour, and brightness of graphical shapes rendered by the laser light are controlled by sending OSC commands to the Pangolin Beyond software. 5.4 Simulation-based Generative Systems Two simulation-based generative systems are employed to control the motion of light in real-time (i.e. the rotation of robotic lights and the transformation of laser shapes). 5.4.1 Flocking Simulation. An application to simulate flocking behaviour has been developed by the first author. This application employs a version of the ISO-Flock software library [ 8 ] that has been implemented as Addon 10 for the OpenFrameworks creative coding environment. The simulation controls the coordinated motion of agents which in turn control the rotation and/or positions of the lights on stage. The simulation models agents as point-like entities that perceive and respond to each other. The simulation extends the classical Boids system in several ways. It implements two distinct flocks whose agents are associated with different elements on stage. The simulation adapts the classical Boids behaviours (cohesion, alignment, evasion) to create coordinated movements both within and across the two flocks. The simulation also adds several custom behaviours that cause agents to limit their maximum rotational acceleration, move at a preferred speed, be attracted to or repelled from a point in space, or move along a circular trajectory. The combination of Boids-type and custom behaviours allows to design movements that mimic specific dance principles (see online animations 11 12 13 14 15 ). The agents in the first flock are associated with individual robotic lights or laser shapes. The agents in the second flock are associated with individual joints of the dancer. Only the agents in the first flock move autonomously whereas the agents in the second flock are directly controlled by the dancer (see figure 3 and online animation16 ). 5.4.2 Articulated Body Simulation. An application to simulate the movement of articulated body morphologies has been developed by the first author 17 . The application integrates the rigid body dynamics functionality of the Bullet Physics engine. Two different non-anthropomorphic body morphologies have been designed for this simulation. Both morphologies possess a non-branching body structure and the joints that connect successive body parts have one rotational degree of freedom. The smaller morphology consists of 6 joints and 7 body parts, the larger of 17 joints and 18 body parts (see figure 4). These morphologies have been chosen for two reasons: they possess very little similarity to a human body and thereby 10Flocking Simulation Addon for OpenFrameworks: github.coventry.ac.uk 11Flocking Behaviour Exhibits the Fluidity Dance Principle: vimeo.com 12Flocking Behaviour Exhibits the Levitation Dance Principle: vimeo.com 13Flocking Behaviour Exhibits the Particles Dance Principle: vimeo.com 14Flocking Behaviour Exhibits the Staccato Dance Principle: vimeo.com 15Flocking Behaviour Exhibits the Thrusting Dance Principle: vimeo.com 16Combining Flocking Simulation and Motion Capture: vimeo.com 17Simulation of Articulated Morphologies: github.coventry.ac.uk 462 Translating Idiosyncratic Dance Principles Into Music and Light AM ’24, September 18–20, 2024, Milan, Italy Figure 3: Flocking Simulation - The image on the left shows a visualisation of a flocking simulation with agents displayed as pyramids. Agents that are associated with joints of the dancer are shown in red, independent agents are shown in black. The image on the right shows a skeleton representation of the dancer in the motion capture software. draw more attention to the expressive characteristics of their movement instead of their visual appearance, their joints are arranged in a manner that mirrors the rotational degrees of freedom of a robotic light. Two different behaviours have been designed for the simulated morphologies. A behaviour named ForceBehaviour generates forces that impact externally on body parts. A behaviour named RotationBehaviour specifies target angles towards which joints rotate to. Both behaviours exert their effects either deterministically or randomly. By configuring the body properties and behaviours, movements can be generated that exhibit specific dance principles (see online animations 18 19 20 21 22 23 24 25 26 27 ). More information concerning the implementation of the articulated body simulation and its application for imitating specific movement qualitities have been published elsewhere [10]. 6 PERFORMANCE SCENES EM consists of 10 scenes that differ from each other with regards to the choreography, use of light, application of generative and sound synthesis techniques, translated dance principles, body representations, and autonomy relationships. A summary of all scenes is 18 Articulated Morphology with 6 Joints Exhibits the Fluidity Dance Principle: vimeo.com 19 Articulated Morphology with 6 Joints Exhibits the Levitation Dance Principle: vimeo.com 20 Articulated Morphology with 6 Joints Exhibits the Particles Dance Principle: vimeo.com 21 Articulated Morphology with 6 Joints Exhibits the Staccato Dance Principle: vimeo.com 22 Articulated Morphology with 6 Joints Exhibits the Thrusting Dance Principle: vimeo.com 23 Articulated Morphology with 17 Joints Exhibits the Fluidity Dance Principle: vimeo.com 24 Articulated Morphology with 17 Joints Exhibits the Levitation Dance Principle: vimeo.com 25 Articulated Morphology with 17 Joints Exhibits the Particles Dance Principle: vimeo.com 26 Articulated Morphology with 17 Joints Exhibits the Staccato Dance Principle: vimeo.com 27 Articulated Morphology with 17 Joints Exhibits the Thrusting Dance Principle: vimeo.com Figure 4: Articulated Body Simulation - The image on the left shows a schematic representation of a body architecture with 7 joints. The images on the middle and right show this morphology perform the Fluidity and Particles dance principles, respectively. Figure 5: Scene Overview - The table lists for all the scenes in Embodied Machine the methods of light control, sound synthesis, and translation of dance principles. presented in figure 5. Several scenes are described in more detail. Figure 6 provides a visual impression of these scenes. For each of these scenes, video excerpts are provided online. 463 AM ’24, September 18–20, 2024, Milan, Italy Daniel Bisig, Pablo Palacio, and Muriel Romero Figure 6: Visual Impressions of Performance Scenes - The photographs depict the following scenes, from left to right and top to bottom: Painting, Monster, Approximation, Light Forest, Mound, Labyrinth 6.1 Scene - Painting The dancer is located within the ring of robotic lights. Initially, she is stationary and performs slow and isolated movements in almost complete darkness. A flocking simulation controls the robotic lights. The lights follow the moving body parts and gradually make them visible. Later, the dancer increases her body movements and traverses space with the lights only occasionally and loosely follow her. The music is created using Waveguide Synthesis. At the beginning, the music is composed, later it is interactively controlled by the dancer. The following video excerpts are available online 28. 6.1.1 Dance Principles. The Isolation principle is performed by the dancer and lights. Initially, both employ small and isolated movements only. Later, the movements become larger and involve multiple body parts and lights. The Levitation principle is performed by those lights whose beams are pulled towards the position of one or multiple joints of the dancer. The Silence principle is alternatingly performed by the dancer and lights. For movement sonification, the dancer and sounds are related to each other via the Energy,Fluidity, and Transformation principles. The Energy and Fluidity principles performed by the dancer affects the strength and envelope of the synthetic breathing sounds. The Transformation principles performed by the dancer causes slow changes in joint rotations and positions which affect most of the other synthesis parameters (tube feedback, mouth feedback, frequency/pitch, amplitude, low pass filtering, vibrato depth, vibrato rate, tremolo depth, tremolo rate). 6.1.2 Body Representation. Each light is associated with a flocking agent and points into its direction. The lights represent the agents’ 28Scene - Painting - Online Videos: excerpt 1, excerpt 2, excerpt 3, excerpt 4 bodies and reveal their locations in space. Each joint of a dancer is associated with a synthetic voice in the music. 6.1.3 Autonomy Relationships. Music and lights change their levels of autonomy throughout the scene. These changes develop in opposite directions for light and music. The music initially consists of composed sections only. Later on, the composed sections are combined with music that is interactively controlled by the dancer. The lights are initially fully controlled by the dancer whose joints they follow. Later on, the lights becomes more autonomous, switch between joints, and follow them loosely or not at all. 6.2 Scene - Monster Inside a ring of vertical light beams the dancer moves around a thin line of laser light. As the dancer comes to a standstill, the light beams disappear and the laser line expands in size until it assumes a snake-like appearance that wraps itself around the dancer. The laser light is controlled using the articulated body simulation. The light becomes fully outstretched as it follows the dancer walking around the periphery of the floor ring. Later on, the dancer crosses the centre multiple times with the light either being pulled or pushed by the dancer or separating from her and moving on its own. The music is both pre-composed and algorithmically generated, using a combination of Waveguide Synthesis and extended Amplitude Modulation. The following video excerpts are available online 29. 6.2.1 Dance Principles. The Isolation principle is performed simultaneously by the dancer and laser light as the latter grows in size. As the movements of the light become larger, its tip performs a combination of the Particles and Levitation principle. The Levitation principle is employed to bring the top of the light close to the dancer. When the dancer traverses space, she and the light frequently employ the Starting Point principle with the dancer initiating movements and the light following them. 6.2.2 Body Representation. The robotic lights and laser light create two different bodies. The robotic lights are reminiscent of an opening and closing flower with each of the 8 lights representing one petal. The laser light draws a sequence of line segments that outline the shape of a snakelike body. Occasionally, the laser light wraps itself around the dancer, thereby overlapping its own Kinesphere with that of the dancer. The body of the robotic lights possesses a correspondence to music. Initially, the robotic lights appear as a choir of 8 glissando voices. Later, they reappear as 8 tone division of an octave that is traversed by two diverging melodic lines. 6.2.3 Autonomy Relationships. The laser light alternates in its autonomy between a passive object that is pulled or pushed by the dancer and a live-like entity that explores its surroundings. The opening and closing motions of the robotic lights are synchronised with the appearance and disappearance of the laser light. The 8 voices mirror in their glissando the movements of the robotic lights. The music is both pre-composed and algorithmically generated. The algorithmic part is synchronised with the robotic lights. 29Scene - Monster - Online Videos: excerpt 1 excerpt 2, excerpt 3, excerpt 4 464 Translating Idiosyncratic Dance Principles Into Music and Light AM ’24, September 18–20, 2024, Milan, Italy 6.3 Scene - Approximation This scene stages 8 duets, each one between the human dancer and one robotic light. The robotic lights are controlled by the articulated body simulation. In darkness, the dancer moves along the periphery of the ring of robotic lights. As she gets close to a robotic light, it emits light and performs one dance principle. The dancer responds to the robotic light by performing the same or a different dance principle. When the dancer moves away from the robotic light, it ceases to emit light and stops moving. The music is pre-composed and employs a combination of Waveguide Synthesis and Non-linear Filtering. The following video excerpts are available online 30. 6.3.1 Dance Principles. In each duet, the robotic light performs a single dance principle. The principles it choses from are: Stillness, Fluidity,Particles,Levitation, and Thrusting. Each light performs this principle either in unison or opposition with the human dancer. The music employs the Polytopia and Isolation principles. It consists of three micro-tonal layers two of which move in opposite direction while the third performs micro undulations along a horizontal line. 6.3.2 Body Representation. Each robotic light is a single body. Occasionally, the dancer treats a light beam as a tangible body by touching, hugging, or intercepting it with her arms and hands. 6.3.3 Autonomy Relationships. In each duet, the dancer and a robotic light are partially autonomous. The activation of a light is directly controlled by the dancer. The dancer is fully dependent in her visibility on the emitted light. While the light is active, it moves independently of the dancer. It is up to the dancer to decide if she moves in unison with the light or independently of it. The music is pre-composed and evolves independently. 6.4 Scene - Light Forest In this scene, the dancer moves within a metaphorical forest of light. The dancer performs extreme movements that are either very slow or very fast. These movements are mirrored by the robotic lights and the musical voices, all of them moving in perfect unison. The music is generated using Vocal Synthesis. The synthesis parameters and the rotation of the lights are directly controlled by the dancer’s joints. The parameter ranges and joint associations change during the scene. The following video excerpt is available online 31. 6.4.1 Dance Principles. The dancer employs the contrast principle by varying her movement speed between two extremes. The robotic lighs and music mirror this principle. The two dance principles Energy and Fluidity are translated into music by mapping the acceleration and jerk of the dancer’s joints to synthesis parameters. 6.4.2 Body Representation. The robotic lights and the music act as extensions of the dancer’s body. 16 joints are associated with 16 robotic lights and 34 joints are associated with 34 synthetic voices. These associations change and thereby alter how the dancer’s body is replicated through music and light. 6.4.3 Autonomy Relationships. The synthetic voices and robotic lights are directly controlled by the dancer as instruments. The music and lights possess some autonomy since their associations 30Scene - Approximation - Online Videos: excerpt 1 excerpt 2 31Scene - Light Forest - Online Videos: excerpt 1 with the dancer’s joints and the range of their control parameters changes independently of the dancer. 6.5 Scene - Mound The dancer enters a metaphorical mound that is delimited by a circle of laser light. Once inside, the mound transforms into a representation of the dancer’s kinesphere, reflecting by its size the dancer’s changing body extensions. Later, it splits into six individual ellipses that rotate and change in synchrony with the changing spatial relations between the dancer’s body parts. The music consists of interactive sonifications that are generated using physical models of bowed and plucked strings. The dancer controls via the distances between her body parts the pitch of the synthetic strings. Initially, she controls a single string. Later, she controls six strings. The following video excerpts are available online 32. 6.5.1 Dance Principles. The Relation principle plays a central role in this scene. By changing the spatial distances between her body parts, the dancer controls the pitch of the synthetic strings and the diameter of the light shapes. Later, the rotational relations among her body parts control also the rotations of the light shapes. 6.5.2 Body Representation. The music and light act as extensions of the dancer’s body and mirror the dancer’s Kinesphere. Initially, music and light represent the Kinesphere of the dancer’s entire body. Later, the dancer’s body is dissociated into multiple Kinespheres, one for each pair of body parts. This dissociation also takes place in music and light with multiple voices and light shapes representing the multiple Kinespheres. 6.5.3 Autonomy Relationships. The music and lights act as instruments that are directly controlled by the dancer. 6.6 Scene - Labyrinth The dancer moves through a labyrinth that is created by laser light. When the dancer reaches an end location, she pushes her head through a wall and triggers a transformation of the labyrinth. After each transformation, the layout of the labyrinth becomes increasingly intricate. It starts as simple straight corridor and ends as an irregular distribution of broken and intersecting walls. The music consists of algorithmically created and interactive parts and combines all the sound synthesis techniques that are used in EM. The interactive section of the music employs Dynamic Stochastic Synthesis. The following video excerpts are available online 33. 6.6.1 Dance Principles. Dance and light are related to each other via three dance principles: Spatial Recovery,Flow Effort, and Change of Plane. The labyrinth renders the association between space and dance principles visible. The walls of the labyrinth limit the movements of the dancer and enforce the bounded polarity of Flow Effort. The walls can also be interpreted as externalised planes with which the dancer aligns using the Change of Plane principle. Dance and music are related through the principles Energy and Fluidity. 6.6.2 Body Representation. The laser light represents the body of an architectural object. The walls of the labyrinth are placed at a distance that limits the dancer’s Kinesphere. In the music, the five 32Scene - Mound - Online Videos: excerpt 1 excerpt 2 33Scene - Labyrinth - Online Videos: excerpt 1 excerpt 2, excerpt 3 465