Point Line Piano

Point Line Piano JAROSŁAW KAPUŚCIŃSKI, Stanford University, USA MARC DOWNIE, University of Chicago, USA PAUL KAISER, Independent Artist, USA Point Line Piano reimagines the composition, performance, and reception of piano music through interactive 3D drawing. Participants use a controller to create audiovisual gestures in 3D space, which then repeat and evolve dynamically. They can interact with the resulting worlds, altering both sound and visual qualities. Each scene the participant enters is programmed differently to guide them toward a coherent musical and visual outcome. Beyond the VR immersion, the system employs physics-based algorithms to foster a deeply embodied experience, offering a unique way to engage with abstraction. Additional Key Words and Phrases: Interactive, VR, Intermedia, Visual Music, Embodiment, Abstraction, Immersion, Piano 1 Program Notes Point Line Piano is a VR project that reimagines the composition, performance, and reception of piano music by fusing its modes of creating, playing, and listening. As you interact with it, your ears, eyes, and hands act in concert. You start by drawing lines freely in the space around you, sparking musical notes that are notched as points on the lines as you draw them. These notes quickly accumulate, forming distinct melodic phrases and rhythms, while the computer generates an intricate audiovisual dance all around you. The work enables a spatial and full-body experience of abstraction not found in any other medium. In a live concert setting it can also be used as an audiovisual instrument. Fig. 1. Screenshot from the VR experience., 2025. Authors’ Contact Information: Jarosław Kapuściński, Stanford University, Stanford, USA; Marc Downie, University of Chicago, Chicago, USA; Paul Kaiser, Independent Artist, New York, USA. This work is licensed under a Creative Commons Attribution 4.0 International License. NIME ’25, June 24–27, 2025, Canberra, Australia ©2025 Copyright held by the owner/author(s). NIME ’25, June 24–27, 2025, Canberra, Australia Kapuściński et al. 2 Project Description Point Line Piano innovates the composition, performance and reception of piano music by fusing its modes of creating, playing, and listening. It does this by means of a live graphic notation so visually compelling as to constitute the entire imagery of the piece. When you immerse yourself in Point Line Piano, you discover that your ears, eyes, and hand act in concert both with one other and with the interactive artwork itself. The start of each section begins the same way, with your hand stroking hand-drawn lines freely in the surround space of VR. As these strokes inscribe themselves in the space, the notes they spark are notched as points on the line. Rapidly accumulating, these notes form distinct melodic phrases and rhythms, which may modulate autonomously to form their own variations. In the same fashion, the computer begins to elaborate intricate geometries in response to the points and lines you’ve drawn. The controller you hold in your hand serves you in two ways: first it becomes the paintbrush by which you create your 3D notation; then it turns into the virtual lens through which to study and transform that notation. Depending on the scene, the lens may augment your experience in different ways. It may magnify those parts of the imagery you frame at the same that it amplifies or modulates the sound of the notes it frames; or it may serve as a sort of magnet by which you can pull a given note or set of notes across the musical space, thus changing not only the pitches but also the rhythmic structure and the acoustic space. As the scenes grow in complexity — there are eighteen of them — they each reach a point when you will want to step back and look at them in their entirety. By then, you’ve become a bit more familiar with this strange new realm of spatial imagery with its hints of calligraphy, of musical notation, and of architectural and scientific diagram, and now that you’ve set the whole thing in motion, it performs an astonishing audiovisual dance all around you, the artwork playing the musical notation you’ve co-created in a surprisingly coherent fashion. This ends the scene and ushers in the next. Point Line Piano enables a spatial and full-body experience of abstraction that is to be found in no other medium, opening the ears and eyes of the participant to exhilarating new aspects of perception. It will likely intrigue both the innocent child and the knowledgeable adult, each of whom may find their own satisfactions as they explore the work and then ponder their experience. 3 Technical Notes The project is made possible by OpenEndedGroup’s flexible, open-source, live-coding environment, called Field, which enables us to craft, audition and reorganize both complex geometric responses to drawing and intricate intermedia connections between music, space and line — crucially without leaving the VR environment, or even having to pause the interaction. To Field’s high performance VR-capable graphics system and a polyglot programming paradigm, we have added a custom “non-realistic” sound spatialization tool developed by Eito Murakami, using Unreal Engine. This allows us to create sound spaces that can only exist in our imagination and in VR. The project requires the following technology to run: •CPU:AMD Ryzen 9 7900X •Graphics: NVIDIA GeForce RTX4090 •RAM: 64GB DDR5-6000 •VR: HTC Vive Pro 2 or Focus 3 headset with controllers and base stations on stands •Monitor, keyboard, mouse • (Optional but recommended) Large monitor or video projection to mirror the image from the visor for the viewers who await their turn •(Optional but recommended) Sound amplification of the signal from VR (as above) 4 Media Links •Web page with a video trailer: https://www.jaroslawkapuscinski.com/works/point-line-piano/ •Video recording of a demo performance: https://vimeo.com/938043123/de174aa245?share=copy 5 Ethical Standards The project was developed with the support of Stanford University, Department of Music, Center for Computer Research in Music and Acoustics, and the European Research Council Digi-Score Project