Music( e)ali y: A Collabo a i e Imp o isa ion be ween Vi ual and Real Wo ld
YICHEN WANG, The Aus alian Na ional Uni e si y, Aus alia
CHARLES MARTIN, The Aus alian Na ional Uni e si y, Aus alia
Addi ional Key Wo ds and Ph ases: mixed eali y musical in e ace and pe o mance, head-moun ed augmen ed eali y
ACM Re e ence Fo ma :
1 PROGRAM NOTES
Music( e)ali y is a collabo a i e musical pe o mance be ween he i ual and eal wo ld. Two musicians will p esen
a musical imp o isa ion, wi h one pe o ming wi h an iPad ins umen and he o he s using a eehand augmen ed
eali y musical ins umen . While musicians a e physically loca ed in he space, he music jamming will happen ac oss
a i ual and eal en i onmen . How will he collabo a ion happen and wha is a mixed eali y musical pe o mance? Is
i h ough sonic-only eedback o pe o me s’ musical ges u es?
2 PROJECT DESCRIPTION
Music( e)ali y is a collabo a i e musical pe o mance be ween he i ual and eal-wo ld space. New ypes o mixed
eali y musical ins umen s ha e been shown unique quali ies o 3D musical in e ac ion [
1
], i is s ill ye o see how
hese ins umen s a e used wi h o he musical ins umen s in a collabo a i e con ex . Collabo a i e music-making
is well-s udied [
3
] and o en eques s some le els o communica i e cues o in o ma ion sha ing [
2
], pa icula ly in
he mixed eali y con ex whe e he musician’s iew is na u ally blocked by ha dwa e equipmen [
4
]. Wi h limi ed
in o ma ion lows be ween musicians and o he s when pe o ming wi h he mixed eali y musical ins umen , wha is
i like and how can hey each a le el o “musical ag eemen ”?
In his pe o mance, wo musicians will p esen a musical imp o isa ion, one using an iPad ins umen and he o he
using a eehand augmen ed eali y musical ins umen . While bo h musicians exis in eali y and being able o see each
o he o collabo a ion, his AR ins umen only exis s in he “augmen ed” wo ld. The AR ins umen was designed
wi h eehand in e ac ion which he musician is able o pe o m lexible musical ges u es o di e en sonic possibili ies.
The pe o me ’s in-app iew is sha ed on he s age sc een o hei collabo a o and audiences.
This musical pe o mance is no only he au ho ’s musical p ac ice o hei own ins umen , bu also a emp s o
unde s and oppo uni ies and challenges o mixed eali y musical ins umen s in music ensemble.
Licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License (CC BY 4.0). Copy igh emains wi h he au ho (s).
Yichen Wang and Cha les Ma in. 2023. Music( e)ali y: A Collabo a i e Imp o isa ion be ween Vi ual and Real Wo ld. 1, 1 (May 2023),
2 pages.
Music P oceedings o he In e na ional Con e ence on New In e aces o Musical Exp ession
NIME’23,31 May–2 June, 2023, Mexico Ci y, Mexico.
2 Wang and Ma in
3 PERFORMANCE NOTES
This a p e- eco ded pe o mance. One au ho will join he “open-jack” o li e demons a ion. The AR ins umen
i sel is open-sou ced on Gi Hub and can be deployed o mos VR and AR de ices (E.g., Oculus P o, Ques ; Mic oso
HoloLens Gen 2).
4 MEDIA LINKS
•Video: h ps://you u.be/j- -6y0zYuk
REFERENCES
[1] Flo en Be hau . 2020. 3D in e ac ion echniques o musical exp ession. Jou nal o New Music Resea ch 49, 1 (2020), 60–72.
[2]
Flo en Be hau and Luke Dahl. 2020. Adap ing & Openness: Dynamics o Collabo a ion In e aces o He e ogeneous Digi al O ches as. In New
In e aces o Musical Exp ession.
[3]
Tina Blaine and Sidney Fels. 2003. Con ex s o collabo a i e musical expe iences. In P oceedings o he 2003 con e ence on New in e aces o musical
exp ession. 129–134.
[4] Michael F Schobe . 2006. Vi ual en i onmen s o c ea i e wo k in collabo a i e music-making. Vi ual Reali y 10, 2 (2006), 85–94.
