Movement Through Musical Space as a Generative Model

Movement Through Musical Space as a Generative Model Greg Corness Illinois State University, Normal IL, USA [email protected] Abstract. Generative music systems have long sought to create novel musical content, but often struggle to capture the nuanced, embodied experience of musical creation. This paper explores a generative model that conceptualizes music as movement through an abstract space, drawing parallels between physical movement and musical gesture. By examining concepts from performance theory, time metaphors, and movement analysis, the research proposes a system that models musical variation as a form of spatial navigation. The approach leverages human cognitive understanding of movement, intention, and spatial relationships to create a generative system that can produce recognizable musical patterns with complex, real-time variations. The proposed model suggests that by understanding music as a dynamic movement through temporal and abstract spaces, generative systems can more intuitively reflect human musical perception and performance. Preliminary tests indicate that this approach allows for the preservation of underlying musical sequences while generating expressive variations, potentially offering a more embodied and responsive approach to computational music generation. Keywords: Generative Music, Movement, Musical Space, Performance Systems, Time Metaphors. 1 Introduction For decades, researchers, composers, and performers have been designing and implementing systems for generating musical content. As would be expected, the design of these systems tends to reflect the situation/ space for which the system is intended, the activity, locations, and interactions. Considering that music is always experienced physically, including both time and space, the design of generative processes could be supported by acknowledging connections between the musical space and the physical space (the area, activities, time, and social expectations) in which the music will be experienced. Traditionally, interdisciplinary interactions with generative musical systems have largely focused on mapping events in a physical space to events in the musical space, leveraging the synchronization of perceptions aspiring to where the performer/player is engaged in an activity affected by the system’s music but with Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 678 2 Movement Through Musical Space, G. Corness varying levels of agency over it. Systems interacting with musicians often leverage musical syntax, through self-similarity and other algorithms, to recognize and adapt to the musician’s physical gestures associated with playing an instrument. Robotic systems incorporate body gestures leveraging social communication between performers often modeling gestures that are fundamentally conducted such as head nods and signaling motions[12, 14, 15]. In contrast, cross-disciplinary interaction cannot lean heavily on adherence to a discipline’s rules, inspiring the discussion to deeply explore the effect of other forms of communication and concepts of motion in and through space. While dance and music gestures both incorporate a rhythmic quality (a consideration of time and movement through space), the disciplines do not share an obvious common syntax, and visual cues may not always be accessible. This scenario then offers an interesting question from which to explore music as a movement through an abstract space but modeled on movement through physical space. This paper focuses on exploring concepts that can support designing generative systems that reflect the movement through physical space in the generated movement through musical space. 2 Background 2.1 Performance Theory In the context of performance, the interaction between performers develops a foundational social space for the performance. During a performance, the development of this social space is in constant negotiation by the agents involved in the sharing of the moment. As Lockford and Pelias explain: “Even when faced with the challenge to perform in an unscripted moment, performers understand that they are engaged in an ongoing communicative exchange. This exchange is a process best conceived, not as an act of information transmission or shared understanding, but as communication scholar H. L. Goodall, Jr. would have it, as an act of ‘boundary negotiation’” (Lockford and Pelias, 2004, p.433). Here the idea of boundary negotiation is an explanation of the ensemble’s development of shared social expectations over technical or artistic expectations. Studies with human performers collaborating with generative systems report developing social expectations as part of the interaction [3] [15]. We can see an example at one level of this in the context of improvisational music when the soloist and accompanist negotiate a musical conversation through harmonic extensions and rhythms that occur during a particular solo. Based on this idea, I suggest it is important that all agents can negotiate meaningful responses through a process of presenting their intentions and ‘understanding’ or acknowledging the intentions of others. 2.2 Creating System Knowledge of Space Building off the notion of social space, presence, and personal space could be described, in terms of relationship, as the gap between oneself and other objects. Understanding one’s relationship to space comes from understanding both the temporal and physical gaps between oneself and other objects. Tau Theory, presented by David Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 679 Movement Through Musical Space, G. Corness 3 Lee, provides a basis for considering the time to contact as part of the perception of space [10]. Sensing the size of the gap, the distance is not necessary for sensing the changing time to the object [10]. The ability of an animal to successfully land on an object is linked to their ability to track the changing time expectation to landing. Research has suggested that controlling the attainment of more abstract goals such as hitting a note uses a similar cognitive process [10]. Modeling this process in a computational system could support a perceived connection between the physical space and the abstract space. 2.3 Time and Gesture in Rhythm and Language The importance of physical space as a communication and expression in performance is part of the Viewpoints pedagogy. The framework presents four viewpoints of Time: • Tempo is the rate that movement occurs • Duration is how long a movement or sequence occurs or how long before a noticeable change • Kinesthetic Response is how long it takes to respond, and • Repetition is the recurrence (in the scope of the body or the space) [2] The pedagogy around rhythm in Viewpoints presents the experience of time and rhythm as beginning in our bodies. Expanding from our experience of rhythm in our bodies, it may be arguable that speech and movement in expressions such as poetry are the basis for experiencing structured time. For example, “Changes in feet in closed form poetry often indicate points of stress or a shift in thought” [1]. The focus on short and long syllables emphasizes the flow of the text. And yet, the meter of the poem still relies on a set pattern that sets up expectations that the text can push against or flow with as part of the interaction and experience. In both Viewpoints and Poetry, the perceiving of change in a pattern/sequence becomes the process of parsing gestures. A new gesture starts when there is a change in the quality of sequenced events. The development of unifying qualities then defines the ‘movement’ of the gesture. Gesture can be thought of as the awareness of movement involving a part of the body or an abstract object defined by perceivable changes in qualities with a beginning-middle-end and parsed by stress points accentuating changes in a parameter from a sequence of consistent enactment of a parameter. This understanding of time and rhythm in poetry and performance provides a possible model focused on the relationship between events, the patterns of long and short, and the stress and flow through time, the relationship implying an abstract space. This model enables an innate understanding of rhythm (space-time) that is assessable by both human performers, such as dancers or actors, physical space, and a generative system of musical space. To help structure this new concept in understanding movement gestures, we can incorporate the concept of phrasing in Laban Movement Analysis (LMA) [13]. In LMA there are 4 types of phrasings: • Impactive ---- impact or stress at the beginning • Impulsive -----stress at the end Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 680 4 Movement Through Musical Space, G. Corness • Swing -------stress in the middle • Even ----- no stress 2.4 Musical Knowledge The movement of abstract objects in musical space reflects the music analysis technique developed by Heinrich Schenker based on the notion of musical figurations prolonging pitch points by moving towards, away from, and around these posts. Schenkerian analysis suggests that any tonal musical piece can be broken down into three structural levels [7]. While each level illustrates the sequence based on tonal consonance (arpeggios, scalar sequences, etc.), each level also illustrates the figuration and paths between the consonance tones that make up the lower level. Figurations such as Neighbor Tone, Passing Tones, etc., are metaphors of a movement through a tonal space where significant events act as posts that are moved towards, around, and from. Though Schenkerian analysis focuses on the tonal space in music, a similar concept focusing on points in time can be applied to the temporal space [6]. 2.5 Time Metaphor in Music Much of the language around rhythm indicates the metaphoric expression of our experience, leveraging point and movement in space. Phrases such as ‘Pushing’ the beat or ‘Pulling’ the beat reflect an ontological metaphoric referencing our experience with physical objects [9]. Perhaps even more commonly, the terms ‘on’ the beat and ‘off’ the beat leverage the metaphor of spatial orientation. From this perspective, a beat is an object that a musician can be ON or PUSH as if it were a stool. A rhythm could be imagined as a line of stools. Leaping from stool to stool is a physicalization of being ON the beat. However, we could now imagine pulling the stool closer for a slightly shorter leap, pushing it further away for a longer leap, landing in between stools, and continuing with a run up to the stool [5]. By applying a Moving Observer metaphor, we can now see rhythm as a dance around points in time. 2.6 Music Gesture The concept of evenly dividing time into pulses, meters, and precisely recurring patterns is fundamental to the concept of rhythm in Western music. The pulse is what we tap our foot to: it is the basis of the meter, the bar, and rhythm is based on dividing and combining periods [16]. This view of rhythm suggests an object-based metaphor where periods are imagined as blocks fitting together to fill spaces of time in a rigorous organization. While this metaphor may seem to fit with the tenet of music theory, humans are not perfect timekeepers, and performers regularly vary temporal dynamics for expressive effect. This understanding implies a much more tenuous relationship between beat and flow in music and suggests that “any endogenous sense of pulse and meter is always being generated throughout the listening process” [16]. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 681 Movement Through Musical Space, G. Corness 5 2.7 Generalized Time Metaphor Other metaphors of time can help provide a less rigid model. Common references to time tend to fall back on spatial metaphors. We use language that implies a future coming towards us, such as Christmas is coming [Moving Time] or ‘I’m just trying to make it to lunch…’ [Moving Observer]. Although these are not the only ways in which we metaphorize time, these examples illustrate how we tend to conceptualize time in terms of space. They offer interesting conceptual possibilities that can be exploited to develop a model for generating events. If time can be conceived as moving through an abstract space, moments can be modeled as being locations or regions in space [9]. This idea can be used as another model to draw a connection between physical space and musical space. A stream of events passing is experienced as music. In this stream, the experiential is constantly moving towards the next point in time, like a signpost on a road. Conceiving of events in time as objects that are approached endows the interaction with the power of intention towards a goal. It acknowledges a path with both past and end. 3 Implementing a Model To model musical variation on movement through a physical space, I have worked to develop a system that produces a set of recognizable figurations around events within a recognizable musical object (pattern). These figurations are designed to imply a movement through musical space, modeling concepts borrowed from Tau of Motion [10] where the Tau gap is the process for achieving a goal, and Schenkerian analysis where musical structures are analyzed as figurations expanding on lower-level pitch events [7]. A trial implementation of the presented model has been developed that focuses on generating rhythmic figurations to imply movement into and out of repeated musical objects (set sequences), The system performs what can be referred to as structured improvisations around a precomposed theme leveraging both musical syntax and notions of movement into and out of goals. Figure 1 provides an example of a set sequence, a clave rhythm pattern. The x notes represent the goal point associated with each event in the sequence. Fig. 1. The x notes represent goal points to be achieved as the system performs a set sequence. As the system counts through the musical time/space, each goal is achieved and triggers the data associated with that event such as pitch data, likelihood of figuration, quality of figuration etc. Since the triggering of the event is associated with a goal that needs to be reached before the change can occur, it is possible to build in systems for moving the goals in response to other events within the physical space such as a performer (dancer or actor) extending their movement. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 682 6 Movement Through Musical Space, G. Corness The implications of moving a ‘sequence event goal’ becomes more apparent when the element of figurations is added. Figure 2 shows the addition of two figurations to the underlying sequence on beat 4 of the first and second bars. Fig. 2. Examples of figurations pushing into the 4th beat of each bar with a second level of figuration incorporated in the second figuration. The first figuration 1 demonstrates a short figuration pushing into the fourth beat. The second figuration again has a figuration pushing into the first beat of bar 1, but also has a figuration enhancing the second half of the fourth beat, further pushing into beat 1. Now consider if in the first figuration the ‘sequence goal’ shifted during the push to delay into the first beat of bar 2. The figuration could extend as the movement towards this goal naturally adapts to the changing space (see fig. 3). Fig. 3. examples of the system’s response if a goal point for a figuration moved during the performance of the figuration (first figuration). The base process is enacted as a single recurrent event. A point in time is chosen as a goal. This goal may be either a point selected from a sequence or generated as part of a figuration. This point can have metadata associated with it to define what event occurs when the goal is reached. This will then act as a goal, but this goal can be achieved by a simple count to it or by constructing intermediary goals as a generated sequence towards the foundational goal. This subsumptive structure can then generate complex yet responsive rhythms while implying a movement within a structured space, reflecting both the composed musical space and actions within the physical space. In this way the model affords the design of systems performing structured improvisation. The preliminary tests indicate that the design of the system does afford the playing of a sequence as composed while also allowing for complex variations to be generated in real time that accentuate the underlying sequence. The use of sequence goals as points of change seems to allow events to retain a hierarchical attention. Metadata was successfully associated with different goal points, affording the design of different variation qualities based on position within the musical space. When tested in a purely rhythmic context, rhythmic gestures were discernible and recognizable even when not played directly. The base or underlying pattern remains implied even during extended enactment of variations around the set pattern. Variations are perceived to move towards Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 683 Movement Through Musical Space, G. Corness 7 the time points set in the underlying sequence, providing a level of emphasis or perceived importance to those events without the need of duration or volume changes. 4 Conclusion and Future Work Mark Johnson lays out in his discussions on knowledge schemas that as humans in a physical and social world, our way of knowing the world starts in our bodies [8]. It is through our bodily experience in the physical world that we know each other and the objects and space around us. One way to heighten the connection between a generative music system and those who experience the generated material is to try to create perceived connections, similarities, or translations between the musical space and the experiencer’s space. Theories such as the Tau of Motion [10] lay out how humans cognitively engage with the space and objects around them through anticipation and a sense of changing relation. These theories point to an innate interaction that is based on understanding the world by understanding the intention of the objects, space, and agents around us. The perception and presentation of movement can be an integral part of this process. The proposed design model is part of an exploration into this notion of abstract/ musical motion being perceived and even associated with physical movement, not instigation movement, but a flow of movement through space towards and away from points. Though preliminary, the current results suggest that a complex connection exists between a physical and conceptual space by applying a design model based on physical movement, the physicality of a performer’s body, and of the musical structures. The general trajectory of this work focuses on the experience of system-performer or system-user communication for interacting in improvisatory sessions. This approach to designing the interaction with a music or sound generative system can be applied to areas such as interactive performance, video games or other situations where the movements/actions of a human experiencer are being mapped to changes in the sound texture and flow. The approach is a step towards facilitating the designer to design based on music and sound textures to follow a template, such as a jazz head, but not hold rigid to the precomposed structure and allow/ incorporate improvised modification that reflect the actions of the performer/player. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 684 8 Movement Through Musical Space, G. Corness References 1. Adams, S.(1997). Poetic designs: An introduction to meters, verse forms, and figures of speech. Broadview Press. 2. Bogart, A., and Landau, T.(2004). The viewpoints book: A practical guide to viewpoints and composition. Theatre Communications Group. 3. Corness, G. (2013). “Breath as an Embodied Connection for Performer-System Collabotrative Improvisation” (PhD dissertation), Simon Frasor University) 4. Corness, G., & Schiphorst, T. (2013, June). Performing with a system's intention: embodied cues in performer-system interaction. In Proceedings of the 9th ACM Conference on Creativity & Cognition (pp. 156-164). 5. Corness, G., & Carlson, K. (2020, July). Physical Time: A Model for Generating Rhythmic Gestures Based on Time Metaphors. In Proceedings of the 7th International Conference on Movement and Computing (pp. 1-6). 6. Dubiel, J. (2021). Pulling a fast more-than-one: Milton Babbitt’s time-point practice. *Contemporary Music Review, 40*(2-3), 347–357. https://doi.org/10.1080/07494467.2021.1931084 7. Forte, A. (1959). Schenker's conception of musical structure. Journal of Music Theory, 3(1), 1-30. 8. Johnson, M., (2013). Metaphor. Oxford University Press. 9. Lakoff, G., & Johnson, M. (2008). Metaphors we live by. University of Chicago press. 10. Lee, D. N. (1998). Guiding movement by coupling taus. Ecological psychology, 10(3-4), 221-250. 11. Lockford, L., & Pelias, R. J. (2004). Bodily poeticizing in theatrical improvisation: A typology of performative knowledge. Theatre Topics, 14, no. 2, 431–443. https://doi.org/10.1353/tt.2004.0026 12. Savery, R. (2021). Machine Learning Driven Musical Improvisation for Mechanomorphic Human-Robot Interaction. In Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction (HRI '21 Companion). Association for Computing Machinery, New York, NY, USA, 559– 561. https://doi.org/10.1145/3434074.3446351 13. Studd, K., & Cox, L. (2019). Everybody is a body. Dog Ear Publishing. 14. H. Wang,H., X. Zhang and F. Iida, (2024). "Human–Robot Cooperative Piano Playing With Learning-Based Real-Time Music Accompaniment," in IEEE Transactions on Robotics, vol. 40, pp. 4650-4669, 2024, doi: 10.1109/TRO.2024.3484633. 15. Weinberg, Gil, and Scott Driscoll. (2006). “Toward Robotic Musicianship.” Computer Music Journal 30, no. 4 (2006): 28–45. http://www.jstor.org/stable/4617982. 16. Weyde, T., & Armstrong, N. (2017). Generating Time: Rhythmic Perception, Prediction and Production with Recurrent Neural Networks. Journal of Creative Music Systems, 1 no. 2. Proc. of the 17th International Symposium on CMMR, London, UK, Nov. 3-7, 2025 685