Phantom of Utopia

Phantom of Utopia KA HEI CHENG #1, Louisiana State University #1, United States IRINA KRUCHININA #2, Louisiana State University #1, United States MATIN ESMAEILI #3, Louisiana State University #1, United States Additional Key Words and Phrases: erforming arts, human computer interaction, extended reality, gesture recognition ACM Reference Format: 1 ABSTRACT Chinese Calligraphic Dance (CCD) is introduced in this paper as an innovative artistic discipline synthesizing traditional calligraphy, dance, and interactive technology. The elements utilized in the performance derive exclusively from traditional Chinese characters, grounded in the clerical script that originated during the Han Dynasty around 200 BCE. In the historical context of Chinese calligraphic strokes (elements of Chinese words), our interdisciplinary approach transcribes these strokes into gestures, translating them into generative music and visual media through motion tracking. The present interdisciplinary exploration, influenced by somaesthetic principles, aims to deepen our awareness of bodily perception as well as provide an immersive engagement with Chinese calligraphy culture. 2 PROGRAM NOTES The interdisciplinary project “Phantom of Utopia” revolves around a classical Chinese poem, "The Peach Blossom Spring[ 1 ]," written in 421 CE by Tao Yuan-ming. Exploring music composition, movement, and extended reality as instruments to interpret text, we disassemble it into acoustic, psycho-physcal, and graphic elements and navigate their crystallization back into what the text is as a complex affinity between the imaginary (virtual) and the material (corporeal). The narrative of this poem explores a mythic, celestial realm—a utopia—set against the backdrop of societal turmoil and political instability during the Jin Dynasty. Being a Hong Kong native, the composer and creator of the project grew up surrounded by Cantonese and Chinese culture, which significantly molded her sense of identity and subsequently influenced her artistic preferences. Notably, her affinity for Chinese poetry, coupled with thoughtful considerations of musical instrumentation and orchestration, demonstrates a connection to her cultural roots. The choice of the poem was informed by the composer’s connection to her country during her symbolic journey far away from home in search of her ideal world. The music employs traditional Chinese instruments combined with the sounds from the composer’s surrounding nature, objects, and materials in Baton Rouge, Louisiana. The dancer, in this context, responds to the acoustic textures with a constant reference to the motion of writing the Chinese characters. She experiments with varying contours, speeds and spacing, to translate the traditional two-dimensional calligraphy Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). Ka Hei Cheng #1, Irina Kruchinina #2, and Matin Esmaeili #3. 2024. Phantom of Utopia. 1, 1 (September 2024), 4 pages. Music Proceedings of the International Conference on New Interfaces for Musical Expression NIME’24, 4–6 September, 2024, Utrecht, The Netherlands 2 Ka Hei Cheng #1, Irina Kruchinina #2, and Matin Esmaeili #3 into a sense of body in a three-dimensional space the way the poem suggests to experience it. The dancers’ movements are designed to incorporate somaesthetic[ 3 ] approaches, integrating bodily experiences and audio-visual sensations in extended reality. The movement proposals, in their turn, inform the visualization of the interaction between the images and sounds formed in the brain affected by words. Through live performance, the dancer further responds to the literal representations of the Chinese brush strokes by the imitative body positions on the video. This way, we show the dynamic aspect of words that physically take the mind on a journey, similar to the one experienced by the fisherman who returns to reality from the land of aspirations and attempts to bridge them together. The artistic process exhibits the creation of music informed by poetry simultaneously interpreted through the dance that provides intricate descriptions of individual strokes, words, sentences, and contextual nuances to meticulously craft each piece. The dance in its turn predicates the complex visualization of a reader’ s experience of interaction with the poem. The objective is to articulate the aesthetic allure inherent in traditional Chinese characters, concurrently fostering a heightened comprehension of the associated culture and language. 3 PROJECT DESCRIPTION We advocate the concept of treating the body as a musical instrument, capturing comprehensive data points through cameras, subsequently programming them to produce an audio-visual composition. The intricate elements of Chinese pictorial characters combine both literal and conceptual approach to reality and outgrow from simple calligraphic gestures. The project transcribes the Chinese words[ 2 ] in a poem, Peach Blossom Spring, into body gestures that are both literal/imitative and figurative/figuring out the directions of thoughts the poem gives in space. The collective bodily movements and gestures are subsequently encoded into a novel manifestation named Chinese Calligraphic Dance (CCD). This dance is sonified by means of both “categorized and uncategorized gestures/movements”, facilitated by a motion tracking system, to produce generative music and live visual effects. The motion tracking system translates the categorized and uncategorized gestures/movements into sound cues and visual effects, along with a fixed media composition. The categorized gestures are derived from the stroke elements of Chinese words. The dancer performs stroke elements with her movement and the artificial intelligence system recognizes it to trigger audio and/or visual cues and events while the uncategorized movements are inspired by the pictorial shapes of Chinese words and the textures of the music composition. After reading the poem, the dancer encodes her thinking and philosophies into dance movement according to the context of the poem. The system is designed to provide visual and sonic feedback, effectively transforming gestures into a canvas for dance movements and harnessing motion tracking data to inspire generative music composition. To enhance the narrative in the poem, we employ 3D scanning via Luma AI and pre-recorded motion tracking animation on a human body via Optitrack, resulting in the generation of OBJ and FBX files. These files serve as 3D assets within the Unreal Engine. Utilizing static and skeletal meshes, we integrate them into the Niagara animation effect through a customizable animation timeline system made with unreal engine blueprint scripting. This blueprint script facilitates the implementation of the Niagara system effect, enabling the configuration of timelines for the animation’s start, idle, and end phases. Parameters such as spawn rate, intensity, color, and size are customizable within the tool. Phantom of Utopia 3 Additionally, this tool incorporates a morphing system, offering a selection of different morphing niagara particle effects with adjustable durations. Upon the conclusion of the morphing effect, the system signals the commencement of a specified animation. The Dancer can interact with the system through a somaesthetic approach. The project explores the interaction with the text: what images it sparks in the mind, how it moves the body, how it prompts us to listen; it further prompts technological inventions to materialize poetic dimensions of meaning in space and time. The performance consists of three parts, each corresponding with the three movements in the poem. First, the dancer interacts with the text, trying to approach it literally, through imitation, by shaping the body into poses informed by the Chinese characters. The visual here shows the words as intensities, crystallizing into meaningful shapes and then bursting back into moving possibilities of meaning. The second movement, where the fisherman traveler experiences time and everyday life in the world of Utopia, allows the Dancer to recognize the self in the characters, which is amplified in the visual part that conceptually reflects her vision. The characters start converting into the poses and back into strokes on the screen as she keeps interacting with it as a reflection of her vision, and trying to translate the characters into a body language. The first and second movements mimic the creation process that started from the Dancer’s attempt to understand the language she does not know by imitating brushes as elements, and gradually becoming more familiar with them in her body as she started developing personal movements predicated by Chinese brushes. The third movement of the project we are presenting for the NIME conference brings the spectator to the end of the poem where the traveling fisherman leaves the Utopia to come back and tell about the dream land to his people, none of whom would ever be able to find the Peach Blossom Forest. The third movement reflects how the reader/dancer sees herself in each character, yet already in motion, which takes the textual interpretation process even further. As the Dancer interacts with herself on the screen/ in her vision-reading in motion, the spectator witnesses the reader’s cosmic/dreamland with a multitude of dancing self-reflections. As the Dancer keeps interacting with the animated characters, the performance exhibits the expanding sense of text as a complex movement of the mind/imagination. 4 PERFORMANCE NOTE For online performances, we offer the option of live streaming from our campus or presenting a pre-recorded video. In live settings, a projector enhances the experience by projecting onto the stage, enveloping the dancer and creating an immersive environment. The audio, available in stereo format, is adaptable to super stereo in concert halls. We are equipped with our own computer, audio interface, and projector, ensuring easy execution of the performance. The complete performance, encompassing all three movements, spans 15-20 minutes and can be presented in its entirety upon request for events in NIME. Alternatively, another version lasting 5 minutes is available for shorter performances. Please take reference to figure 1 for the event graph. 5 MEDIA LINK Third movement: https://youtu.be/NeAsumJOG2U (Please use 4K to watch the video for better quality) 4 Ka Hei Cheng #1, Irina Kruchinina #2, and Matin Esmaeili #3 Fig. 1. Event graph showing the mutual impact between music, technology and dance in the creation of live performance. ACKNOWLEDGMENTS We would like to thank our advisor and department for their support, provision of invaluable resources, and guidance, which have been instrumental in the completion of this work. ETHICAL STANDARDS The project does not involve research with people or animals, and does not involve source of funding, potential financial or non-financial conflicts of interest. REFERENCES [1] Long YL Bing. 2015. The Ethical Representations and Enlightenments of ‘Peach Blossom Spring’in Contemporary Literature. In Forum for World Literature Studies, Vol. 7. Wuhan Guoyang Union Culture & Education Company, 643. [2] Zhe Ma and Jianbo Su. 2017. Stroke reasoning for robotic Chinese calligraphy based on complete feature sets. International Journal of Social Robotics 9, 4 (2017), 525–535. [3] Richard Shusterman. 2018. Aesthetic Experience and Somaesthetics. Studies in Somaesthetics, Vol. 1. Brill.