Computer Music and Digital Media Art Through a Web-Based Collaborative Interface

MESTRADO MULTIMÉDIA - ESPECIALIZAÇÃO EM MÚSICA INTERATIVA E DESIGN DE SOM COMPUTER MUSIC AND DIGITAL MEDIA ART THROUGH A WEB-BASED COLLABORATIVE INTERFACE LUIS ARANDAS M 2019 FACULDADES PARTICIPANTES: FACULDADE DE ENGENHARIA FACULDADE DE BELAS ARTES FACULDADE DE CIÊNCIAS FACULDADE DE ECONOMIA FACULDADE DE LETRAS FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO Compu e Music and Digi al Media A h ough a Web-Based Collabo a i e In e ace Luis A andas Mes ado em Mul imedia O ien ado : P o . D . José Albe o Gomes Co-O ien ado : P o . D . Rui Penha 17 de julho de 2019 c Luis A andas, 2019 Compu e Music and Digi al Media A h ough a Web-Based Collabo a i e In e ace Luis A andas Mes ado em Mul imedia Ap o ado em p o as públicas pelo Jú i: P esiden e: P o . D . Gilbe o Be na des A guen e: P o . D . Rod igo Ca alho Vogal: P o . D . José Albe o Gomes 17 de julho de 2019 Resumo O p esen e documen o ap esen a uma in es igação ei a no con ex o do B aga Media A s, pa e cons i uin e da ede de cidades c ia i as da UNESCO (UCCN). O objec i o e e lexão un- damen al des e abalho é dado na música po compu ado e na media a e a pa i de in e aces colabo a i as. Com um oco especí ico na pe o mance a ís ica e na in e acção em sis emas digi- ais é p opos o um ambien e audio- isual, Akson. U ilizando ecnologias web, o Akson oi inicial- men e concebido como uma explo ação do que pode ia se cons uído ap o ei ando a in aes u- u a global da In e ne , bem como a ep odução musical e isual em g ande escala po á ios disposi i os. É ap esen ada uma e lexão eó ica da a e e da sua ecnologia enquan o eículo ex- ensí el da expe iência, da pessoa e do sis ema. Pala as Cha e: Música po Compu ado , Media A e, A e em Rede, Pe o mance A ís ica i ii Abs ac This documen p esen s a esea ch made in he con ex o B aga Media A s, pa o UN- ESCO’s c ea i e ci ies ne wo k (UCCN). The objec i e and undamen al e lec ion o his in es- iga ion is gi en in compu e music and media a om collabo a i e in e aces. Wi h a speci ic ocus on a is ic pe o mance and in e ac ion in digi al sys ems, we p opose an audio- isual en- i onmen , Akson. Using web echnologies, Akson was ini ially concei ed as an explo a ion o wha could be buil by le e aging global In e ne ’s in as uc u e as well as la ge-scale musical and isual ep oduc ion ac oss mul iple de ices. We p esen a heo e ical e lec ion o a and i s echnology as an ex ensible ehicle o expe ience, pe son and sys em. Keywo ds: Compu e Music, Media A , Ne wo ked A , A is ic Pe o mance iii i Lis o Figu es 2.1 Pho o named "Cage. Va ia ions" ound in A nol ini. An a icle abou 9 e enings: Thea e and Enginee ing............................... 8 2.2 Cou esy o he Au ho s. On he le a pic u e o Julien Bayle pe o ming ALPHA 1.0 in Mon éal and on he igh a pic u e o Bo is Di ide on Da a+Pe o m li e pe o mance. .................................... 10 2.3 On he le pic u e aken om Sóna 2018 o icial websi e o Ca s en Nicolai (Al a No o) pe o ming and on he igh pic u e aken om Ryoji Ikeda’s o icial websi e o he a wo k ex pa e n. ............................. 11 2.4 Pic u e o he schema ic diag am o a gene al communica ion sys em. Claude Shannon on The Ma hema ical Theo y o Communica ion. ............ 13 2.5 Cou esy o RAND Co po a ion. Ne wo k Models p oposed by Paul Ba an in 1964. Image ound in documen RM-3420-PR. . . . . . . . . . . . . . . . . . . 14 2.6 Cou esy o he Au ho s. On he le a pic u e o Telesymphony pe o mance and on he igh a pic u e o BEER pe o ming in Ne wo k Music Fes i al (2014) in Bi mingham. .................................... 14 2.7 A pic u e o The League o Au oma ic Compose s pe o ming in San F ancisco 1981 (McKinney, 2016; B own, 2002) . . . . . . . . . . . . . . . . . . . . . . . 15 2.8 Pic u e ound on a icle Long Dis ance Telephone Conce s by Scien i ic Ame - ican, Feb ua y 28, 1891, page 130. Pe o me s in New Yo k and audience in New onMassachuse s................................ 16 2.9 Cou esy o Al a o Ba bosa. A sc eensho o FMOL so wa e. A web-based ee- s uc u ed da abase. c Se giJo dà......................... 17 2.10 Kandinsky’s Yellow-Red-Blue (1925) is om his pe iod o o ally non- ep esen a ional wo ks, bu his pa icula pain ing seems e oca i e o he concep o a pe son expe iencing i id ‘synes he ic’ image y. Oil on can as, Musée Na ional d’A Mode ne, Cen e Geo ges Pompidou. Rep in ed wi h pe mission: Réunion des Musées Na ionaux/A Resou ce, NY. 2003 A is s Righ s Socie y (ARS), New Yo k/ ADAGP, Pa is. Amy Ione and Ch is ophe Tyle on Neu ohis o y and he A s.......................................... 19 2.11 Image wi h bo h he o icial logo o B aga Media A s (le ) and he o icial logo om UNESCO’s UCCN ( igh ). . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.12 Two sc eensho s o possible shapes ha Shoggo h can g aphically gene a e. These we e ound on Chad McKinney’s PhD documen (2016) . . . . . . . . . . . . . 22 2.13 Pic u e on he le is a pho og aph o a pe o mance in MIT media lab using G ou- pLoop on he igh a p in sc een o he web in e ace ound online. . . . . . . . . 23 2.14 Pic u e on he le pho og aph o he pe o mance Finge p in s in Passos Manuel (Po o) in ICLI 2018. On he igh a p in sc een o he edi o used, T oop. . . . . 23 xi xii LIST OF FIGURES 2.15 On he le pic u e o he pe o mance by And ew Blan on on Soundwa e and on he igh a pic u e o he g aphical con en displayed on he sc een h oughou he pe o mance. .................................... 24 2.16 An example o a dis ibu ed, synch onized musical ins umen (DSMI). The in- s umen is composed o a ious pa s ha exis in di e en loca ions. I uses p e- dic ion in o de o syn hesize he same audio in each loca ion a he same momen . I capi alizes on he p edic able pa h o a pe cussion malle in o de o accomplish hisgoal........................................ 25 2.17 Pic u e on he le is a pe o mance a he 2012 Ne wo k Music Fes i al in Bi m- ingham (UK) wi h Yig and on he igh is a sc eensho o he so wa e’s GUI. . . 25 2.18 On he le is a pho og aph o one o he pe o me s a he conce p esen ed a Casa da Música and on he igh is a pho og aph o he public in e ac ing ia mobile a he conce held a Be klee. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.19 Pic u e on he le is a public pe o mance o Monad wi h emo e playe s in Is an- bul, Vancou e and Be lin. On he igh a sc eensho o he so wa e’s GUI. . . . 26 2.20 Two pic u es o he Conce o Sma phones and O ches a in Geo gia Tech (2016) du ing Web Audio Con e ence (WAC). On he le And ey Bundin on elec- onics and on he igh as iolin, Alexey Ko che ko . . . . . . . . . . . . . . . . 27 3.1 P in Sc een o Akson ook by he au ho s. . . . . . . . . . . . . . . . . . . . . . 29 3.2 Cou esy o Publicações Eu opa-Amé ica. Image in O E o de Desca es by An- ónio Damásio. Diag am o a neu on wi h i ’s p incipal componen s: Cell Body, AxonandDend i es. ................................ 30 3.3 S a ing on an open sou ce code edi o A om (bo om-le ) and hen uploading i o he cloud h ough Sou ceT ee (bo om- igh ) an open-sou ce Gi GUI so wa e. When he sys em is ully upda ed we ha e he Gi Hub in e ace o he Akson eposi o y wi h all he code commi s ha by i sel in e aces di ec ly wi h He oku pipeline........................................ 31 3.4 On he le , sc eensho o he o ical ead hedocs websi e. On he middle, sc een- sho o he commi s ab in Gi Hub. On he igh a sc eensho o Akson’s bandcamp wi h music om public expe imen s. . . . . . . . . . . . . . . . . . . . . . . . . 32 3.5 Sc eensho s o he ou g aphical scenes o Akson in hei de aul s a e. . . . . . 34 3.6 Pic u e o he Mas e Dialog ( op-le ), he Sys em Dialog ( op-cen e and op- igh ) and a able wi h he a ious widge s and me hods (bo om). . . . . . . . . . 37 3.7 In his igu e we ha e 5 Akson p in -sc eens. On he i s ( op) we ha e he de aul isualiza ion o Akson. On he nex 4 we ha e he shade s used in his wo k. They a e ins ances o he E ec Compose and mul iple Pass objec s. On he middle ow we ha e he Gli chPass shade wi h wo modes o in ensi y (on he ins umen s panel called Gli ch and Noise). On he bo om ow we ha e on he le he a e - imagePass, a me hod o g aphic p ocessing ha c ea es kind o isual en ainmen some imes compa ed wi h e e b. And on he igh column we ha e he Vigne e e ec ha adds wo black abs o he main scene came a. . . . . . . . . . . . . . 39 3.8 Sc eensho o he Syn hesize panel ollowed by wo ables wi h he a ious wid- ge s unc ionsandme hods.............................. 45 3.9 Sc eensho o he Backg ound panel ollowed by wo ables wi h he a ious wid- ge s unc ionsandme hods.............................. 46 3.10 Sc eensho o he G aphics panel ollowed by one able wi h he a ious widge s unc ionsandme hods................................ 47 LIST OF FIGURES xiii 3.11 Sc eensho o he Pos -P od panel ollowed by wo ables wi h he a ious widge s unc ionsandme hods................................ 48 3.12 Sc eensho s o he Alloca e dialog ( op), he Logs dialog (middle-le ), he Sa e Se ings dialog (middle- igh ) and he Top ba wi h a panel con aining me hods o ac ion......................................... 49 3.13 Sc eensho o he modes o in e ac ion modal igge ed by he Top ba . . . . . . . 50 3.14 Keyboa d map (QWERTY) wi h a panel con aining me hods o ac ion, Keycode E en s and he unc ions ound in he code. . . . . . . . . . . . . . . . . . . . . 50 3.15 Diag am o Akson’s descen e model. ....................... 51 3.16 Diag am o Akson’s s eamed model. ....................... 52 3.17 Diag am o Akson’s alloca e model......................... 53 3.18 Diag am o Akson ile ee and olde communica ion low. S a ing on he P oc ile ( op-le ). ...................................... 54 4.1 A pho og aph ha shows pa o he space whe e he expe imen was done, se en compu e s wi h Akson unning and Joana Rod igues, a p o essional musician and pa icipan o his expe imen . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.2 Cou esy o gn a ion’s communica ion manage . Pho og aph ook on he si e o he ins alla ion du ing he mo ning o open day................... 59 4.3 A pho og aph o he pe o mance a Open ield. Cou esy o Ma celo Sousa, e- sea che andpublic.................................. 62 6.1 Synopsis made o he BMA Educa ional Se ice o be used as a sign in he space o he Akson ins alla ion on he open day. A pa ag aph is p o ided in English and in Po uguese o explain how you can in e ac wi h he wo k. . . . . . . . . . . . 69 6.2 Technical Ride de eloped o an in e ac i e ins alla ion wi h Akson. This o mal- iza ion as ins alla ion was hough o a mo e limi ed equipmen load explo ing in he same he isualiza ion o he ins umen . . . . . . . . . . . . . . . . . . . . . 70 6.3 Image shown o he pa icipan s be o e CARA’s expe imen ook place. Using his anno a ion i was possible o he pa icipan s o know how o in e ac minimally wi h he sys em. This image was a ailable o iewing h oughou he pe o mance 71 6.4 Image shown o he pa icipan s on gn a ion’s in e ac i e ins alla ion. This image was cons an ly p esen du ing he whole cou se o he ins alla ion. . . . . . . . . 72 6.5 Image shown o he pa icipan s on Open ield’s pe o mance. The image was demons a ed o he pa icipan s jus be o e he s a and was no shown a second ime.......................................... 73 6.6 Sc een sho s o Akson’s g aphical sys em, aken om he i ual came a. These can be ound in h p://www.luisa andas.o g/akson.h ml............... 74 6.7 Sc een sho s o Akson’s g aphical sys em, aken om he i ual came a. These can be ound in h p://www.luisa andas.o g/akson.h ml............... 75 7.1 Pho og aph o signage a anged a he en ance o he Akson ins alla ion in he open day. ...................................... 77 7.2 A icle abou Akson ound on gn a ion lye . . . . . . . . . . . . . . . . . . . . . 78 7.3 A icle abou Akson ound on gn a ion websi e. . . . . . . . . . . . . . . . . . . 78 7.4 Cou esy o Ilídio Ma ques, gn a ion Communica ion manage . A high- esolu ion e ical pos e image ha was ound ac oss he ci y o B aga as publici y. . . . . 79 xi LIST OF FIGURES 7.5 Cou esy o Ilídio Ma ques, gn a ion Communica ion manage . A high- esolu ion ho izon al pos e image o he e en . Was also ound ac oss he ci y o B aga as publici y. ...................................... 80 7.6 Cou esy o Ilídio Ma ques, gn a ion Communica ion manage . A pho og aph o use s in e ac ing wi h Akson du ing he open day.................. 81 7.7 Cou esy o Ma celo Sousa, esea che and public a Open ield expe imen . A pho og aph o he explana ion gi en a e he pe o mance, main aining an in e - ac ion wi h Akson using a mobile phone. . . . . . . . . . . . . . . . . . . . . . . 82 7.8 A icle abou Akson ound on h ps://www. imaseba idas.p /gn a ion-celeb a-sex o- ani e sa io-com-open-day- echeado-de-musica-e-ac i idades/. A pos by ReB TeaminFeb ua y22 h................................ 83 7.9 A pho og aph o a gn a ion lye con aining he ac i i ies o open day. ...... 84 Abb e ia ions and Symbols 3D [Th ee Dimensions] ADSR [A ack Decay Sus ain Release] AMD [Ad anced Mic o De ices] ANSI [Ame ican Na ional S anda ds Ins i u e] API [Applica ion P og amming In e ace] AV [Audio-Visual] BEAST [Bi mingham Elec oacous ic Sound Thea e] BEER [Bi mingham Ensemble o Elec oacous ic Resea ch] BMA [B aga Media A s] CARA [Cen o de Al o Rendimen o A ís ico] DAW [Digi al Audio Wo ks a ion] DDR [Double Da a Ra e] DIP [Dis ibu ed Imme si e Pe o mance] DNS [Domain Name Sys em] DOM [Documen Objec Model] DSMI [Dis ibu ed Synch onized Musical Ins umen ] DSP [Digi al Signal P ocessing] ECC [E o -Co ec ing Code Memo y] EMUI [Emo ion UI] EQ [Equaliza ion] ES [Embedded Sys ems] FFT [Fas Fou ie T ans o m] FPS [F ames pe Second] FOV [Field o View] GB [GigaBy e] GHZ [GigaHe z] GL [G aphics Language] GLSL [OpenGL Shading Language] GUI [G aphical Use In e ace] HCI [Human-Compu e In e ac ion] HD [High De ini ion] HDMI [High De ini ion Mul imedia In e ace] HTML [Hype Tex Ma kup Language] ICLI [In e na ional Con e ence on Li e In e aces] ICMC [In e na ional Compu e Music Con e ence] INESC TEC [Ins i u o de Engenha ia de Sis emas e Compu ado es, Tecnologia e Ciência] KHZ [KiloHe z] LFO [Low F equency Oscilla o ] x x i ABBREVIATIONS AND SYMBOLS MB [MegaBy e] MBPS [MegaBi s pe Second] MHZ [MegaHe z] MIDI [Musical Ins umen Digi al In e ace] MIT [Massachuse s Ins i u e o Technology] MP3 [MPEG-1/2 Audio Laye 3] MPEG [Mo ing Pic u e Expe s G oup] MS [Milliseconds] NIME [New In e aces o Musical Exp ession] NMP [Ne wo ked Music Pe o mance] OGG [Con aine Fo ma by he Xiph.O g Founda ion] OJM [O ques a de Jazz de Ma osinhos] OS [Ope a ing Sys em] OSC [Open Sound Con ol] OSX [Macin osh Ope a ing Sys em X (Apple Compu e , Inc.)] PA [Public Add ess Sys em] RAM [Random Access Memo y] TRS ["Tip, Ring, Slee e"Cable] TYPE [Yo kshi e P og amming Ensemble] UNESCO [Uni ed Na ions Educa ional, Scien i ic and Cul u al O ganiza ion] UCCN [UNESCO C ea i e Ci ies Ne wo k] VCS [Ve sion Con ol Sys em] VMM [Va iable Ma ko Models] WAC Web Audio Con e ence WAV [WAVE o m audio o ma ] WI-FI [Wi eless Fideli y] WWW [Wo ld Wide Web] XY [Coo dina es o he Ca esian Sys em] ZKM [Zen um ü Kuns und Medien echnologie] Chap e 1 In oduc ion This disse a ion ansmi s he de elopmen and expe imen a ion o a collabo a i e audio- isual (AV) en i onmen done in he con ex o B aga Media A s1, a ecen c ea i e ci y in he UNESCO’s ne wo k (UCCN)2. In 2017, B aga3was o icially designa ed as C ea i e Ci y o UNESCO in he ield o Media A s. In he ecen pas he ci y poin ed o an inc easingly echnological u u e, ein en ing i s economic ab ic ocusing on echnology companies, digi al communica ion s udios, labo a o ies and esea ch cen e s in he ield o obo ics, games, mul imedia and media a s. Simul aneously a cul u al ecosys em has gained o m and exp ession, de ining i sel in a con empo a y and ansdis- ciplina y iden i y, si ua ed in he in e sec ion be ween new media and o mal a is ic exp essions. The use o echnologies in a is ic p ac ice has led many p ojec s o anspo B aga o a dimension o in e na ional ecogni ion. This esea ch was done wi h a close ela ionship wi h i s educa ional se ice, explo ing he p ocess o echnological de elopmen and a is ic expe imen a ion o a p oposed ool. Accom- panied by his o ical and p esen e e ences on ne wo ked in e aces, his documen aims o show e e y hing ela ed o he p ojec . The digi al in e ace p esen ed was done using web echnologies explo ing ne wo ked in e - ac ion in he pe o ming a s. All so wa e and in e ace design was de eloped speci ically o his esea ch and he en i e echnical de elopmen p ocess exis s in he cloud wi h ee access. In his chap e we p esen he con ex and mo i a ion ha suppo ed his esea ch ollowed by he p oblems, hypo heses and objec i es con eyed. Finally he e is a desc ip ion o he me hod- ologies used and b ie summa y o he chap e s comp ised in he documen . 1O icial B aga Media A s websi e h p://www.b agamediaa s.com/en/ [accessed 27/06/2019] 2O icial UCCN websi e h ps://en.unesco.o g/c ea i e-ci ies/home [accessed 27/06/2019] 3B aga is a ci y and a municipali y in he no hwes o Po ugal. I is he hi d-la ges u ban cen e in Po ugal (a e Lisbon and Po o). 1 2In oduc ion 1.1 Con ex and Mo i a ion The con luence o a and con empo a y echnology has always been a signi ican pa o ou wo k, bo h as a is s and as c i ics. The conside a ion deposi ed in a and i ’s echnology has led us o o malize opinions abou human in e ac ion and aes he ics ha a e undamen al bases o he wo k p esen ed he e. The mo i a ion o his wo k is oo ed in in luences om collabo a i e compu e music o con empo a y digi al media a . Fundamen ally we a gue ha sys ems like he one p oposed a e needed as a is ic ex ension in he cloud, mainly as a way o do so. All o he people in ol ed in his esea ch (minus some audience) had clea musical, a is ic, and academic backg ounds. The sea ch o an ac ion ha co esponds bo h o he e hics o a and o he echnological de elopmen o ins umen s equally is undoub edly wha de ined he concep o he o malized en i onmen . This wo k is also p oduc o e e ences and a is ic choices o he au ho s ha hey ha e main ained o some yea s be o e his in es iga ion. 1.2 P ojec The p oposed p ojec began wi h he es ablishmen o a wo k low ha would allow a ull yea o so wa e de elopmen wi hou ail. A cons an sea ch o e e ences in he cou se o he in es iga ion was also main ained, always suppo ed by he public a is ic expe imen s. Ou aim as esea che s and a is s is o in eg a e he compu a ional en i onmen de eloped in he pe o ming a s and as a dis ibu ed digi al collabo a i e in e ace. In his way i is possible o c ea e a bond ha is consis en wi h de elopmen and ou come. We de elop and in eg a e in he cloud, an en i onmen ha can answe bo h he main e- sea ch ques ions and ou concep ual equi emen s. We a e commi ed o de eloping i wi h web echnologies c ea ing in e aces o mul iple de ices. These echnologies we e chosen o allow digi al mul imodali y bo h in he de elopmen o i as a ool and as an AV gene a ion sys em. 1.3 P oblem(s), Hypo heses and Objec i es o Resea ch Conside ing he massi e wo ldwide de elopmen o web echnologies, ne wo ked collabo a- i e in e aces and digi al AV gene a ion, we de ine he a ious pa ame e s o esea ch. We can o mula e ou p oblem conside ing he ollowing: •We ha e al eady ex ended ou senses and ou ne es by he a ious media (McLuhan, 1964) 1.3 P oblem(s), Hypo heses and Objec i es o Resea ch 3 •The in e ac i e capabili ies opened up by compu e ne wo ks allow o sha ed c ea i i y, om ex ual o elema ic eal- ime imp o isa ions o globally cons i u ed g oup p ojec s, wi h dis ance no ba ie o collabo a ion (Dixon, 2007) •New web s anda ds such as Web Audio and Web GL in oduce he capabili y o web b owse s o duplica e many o he ea u es in compu e music ools (McKinney, 2014) •By de aul , he in e ace is a sys em o de ice h ough which un ela ed en i ies can in e ac (Sá, 2010) P oblem •How can we implemen and alida e a web-based AV ool o collabo a i e in e ac ion? Hypo hesis check •Can we, h ough he use o web-based echnologies: –De elop a collabo a i e en i onmen o AV ne wo ked in e ac ion? –Use he in e ne as a ehicle o in e ac ion be ween a is s wi h po en ially global dis ances a ound he wo ld? –How can a mul i use sys em o a is ic c ea ion media e, and he eby enhance, he in e ac ion be ween agen s wi h dis inc c ea i e p o iciencies? –How can a digi al collabo a i e AV sys em con ibu e o a is ic c ea ion h ough he use o web-based echnologies? –Valida e he de eloped ool in public expe imen s using cloud ools? Objec i es The main objec i e o his esea ch is o de elop a ne wo ked AV collabo a i e en i onmen wi h web echnologies Unde s and he ole o a mul i-use digi al sys em as a media ion agen in compu e music and AV pe o mance. The p oposal and alida ion o his ool should be implemen ed in a con ex o he Educa i e Se ice o B aga Media A s. 4In oduc ion 1.4 Resea ch Me hodology In his sub-chap e we p esen he me hodology and esea ch design used. When his esea ch was in eg a ed in he con ex o B aga Media A s, he objec i es ex- plained ea lie we e e y clea , keeping he esea ch ques ions de ined du ing he wo k p ocess. We, as esea che s, ha e ca ied ou a ho ough e iew o he concep ual and echnological li e - a u e ha in u n ex ended o he end o he in es iga ion. A balance o possible ou comes and simila p ojec s/me hodologies o a c ea ion was always pe o med. The de elopmen p ocess o he p oposed en i onmen (see chap e 3) was ca ied ou wi h e- spec o p ac ice-led esea ch and esea ch-led p ac ice4(Smi h, Dean 2009; Candy, 2006) me h- ods and co e ing also he undamen al p ac ices o a is ic esea ch5(Busch, 2009; Liampu ong & Rumbold, 2008) as an academic mode o p oduc ion. A e ha ing a wo k low de ined and p ac icable in he scope o he esea ch ime scale we belie ed ha he con inuous echnical de elopmen pa allel o he a is ic/ heo e ical esea ch was he me hod o be adop ed (compu e -led p ac ice). 1.5 Disse a ion S uc u e This disse a ion is o ganized in 5 chap e s ollowed by he appendix, a achmen and e e - ences. •The i s chap e –In he i s chap e we explo e he ounda ions o his esea ch, he ype o p ojec , he mo i a ions, he esolu ion, he hypo heses and p oblems. •The second chap e –A e iew o he undamen al li e a u e is made o ins an ia e ou esea ch in his o y, always accompanied wi h con empo a y e e ences. We p esen a sub chap e wi h ecen wo ks en i led s a e o he a . In he e se e al impo an wo ks made ecen ly ha a e an impo an e e ence o his in es iga ion a e ci ed. •The hi d chap e 4A ype o esea ch ha leads p ima ily o new unde s andings abou p ac ice. Is conce ned wi h he na u e o p ac ice and leads o new knowledge ha has ope a ional signi icance o ha p ac ice (Candy, 2006) 5Fundamen ally de ined by a ha unde s ands i sel as esea ch, in ha scien i ic p ocesses o conclusions become he ins umen o a and a e used in he a wo ks. This e e s o a pa icula phenomenon in con empo a y a , in pa icula in ins i u ional-c i ique, whe eby esea ch is conside ed a pa o he a is ic p ocess and is ca ied ou by he a is he sel . In his case, a is in ac a o m o knowledge (Busch, 2009) 2.1 Li e a u e Re iew 11 2.1.4 New Media and Compu e G aphics Always con ibu ing o he d as ic change in he c ea i e p ocesses wi h echnological ad- ances (B oad and G ie son, 2016) and ha ing in mind ha isual a migh ha e a ole o play in compu e music pe o mance, we o m up a e iew on oday’s digi al compu e g aphics. In he p e ace o he book Cyclo. Id (2011) published by Ryoji Ikeda alongside wi h Ca s en Nicolai (Al a No o) we can ead in he p e ace ha he a is s decla e ha hey ha e “abandoned he idea ha he image ac s only as a unc ional accompanis o sound and ins ead subo dina ed he audio elemen o ou desi e o he image” (Ikeda and No o 2011) (Ca alho, 2018). On his posi ion be o e a is ic p ac ice we can see he g aphical image ha ing a di e en ole o play inside he AV pe o mance, no speci ically an income o ou come o music bu he o iginal com- posi ional ma e . Figu e 2.3: On he le pic u e aken om Sóna 2018 o icial websi e o Ca s en Nicolai (Al a No o) pe o ming and on he igh pic u e aken om Ryoji Ikeda’s o icial websi e o he a wo k ex pa e n. In e es ing wo k can also be ound in he ield o synaes hesia7and AV ela ionships. The wo k o Kandinsky on colo /music (Kandinsky, 1947) and he Quayola’s sonic isualize s (Quay- ola, 2011a) a e a big e e ence o ake in accoun . Rega ding he possibili ies a is s ha e on con en gene a ion, jus like music he e a e se e al ways o making ideo o digi al g aphics wi h o wi hou eal- ime in e ac ion in a is ic pe o - mance. En i onmen s like P ocessing (F y & Reas, 2001), OpenF amewo ks (Liebe man, Wa son, & Cas o, 2005) (Gomes, 2014) a p ojec s a ed a MIT Media Lab concei ed in 2001, o Cinde (Gómez, Colub i & Cha alambos, 2016; Bell e al., 2012) a so wa e ha also has a good en i- onmen o he a is o p og am digi al g aphics and sound wi h he a ailabili y o use in hei p ojec s o any use (comme cial o o he wise). 7Synaes hesia - Neu ological Condi ion on Wikipedia h ps://en.wikipedia.o g/wiki/Synes hesia [accessed 27/06/2019] 12 Con empo a y and His o ical P ac ice The massi e de elopmen happening wo ldwide on he ield o in e ac i e g aphics is as on- ishing and has g ea possibili ies. Also, he de elopmen o isual compu ing nowadays has a big cohe ence wi h he ype o ha dwa e and API’s people ha e access o, he widesp ead a ailabili y o p og ammable g aphics p ocessing uni s (GPU’s) and shading languages such as GLSL (Ros e al., 2009) makes isual a easie o de elop. Some media a is s also use and exploi gaming p og amming sys ems, such as Uni y8, Un eal Engine9o Godo 10. 2.1.5 Ne wo ked Sys ems and Pe o mance As humans, we c ea e s uc u es and ne wo ks o almos e e y ype o collabo a i e beha io , and NMP has bene i ed om o e h ee decades o de elopmen (McKinney, 2016). Examples o ha can be he expe imen s o he San F ancisco Bay A ea ne wo k band pionee s and o cou se he de elopmen o he in e ne . In oduc ion o p o ocols like Open Sound Con ol (OSC) (W igh , 2002) o mul i-pu pose communica ion p esen good ways o ex ending hose ne wo ks. The e is also ele an wo k on he ield o collabo a ion and ne wo king ha p esen s ax- onomies and ex ended esea ch ha help us s uc u e a be e ela ionship wi h he pas . Such examples a e Displaced Soundscapes - Compu e Suppo ed Coope a i e Wo k o Musical Ap- plica ions (Ba bosa, 2003) and In e ne Music: An In oduc ion (Hugill, 2005). And o don’ lose de ini ion, he e m ne wo k music can quickly become opaque and wi hou meaning. Fo ha I would like o b ing he book Ne wo king Founda ions (Cicca elly & Faulkne , 2004) whe e a ne - wo k is de ined as a sys em ha allows communica ion o occu be ween wo people o machines (McKinney, 2016). When we alk abou ne wo ked digi al sys ems we a e also h own o he big in as uc u e o hose sys ems, and he e is a p oblem aised by B idle (2018) ha s udies he opaci y o ha compu a ion. As digi al sys ems ake place inside he machine, hidden inside buildings and big s uc u es ha p opaga e hemsel es as long as wi es and WI-FI enables, he e is a ba ie om he use . And e en when we dis up ha ba ie , i emains beyond he comp ehension o mos . The agg ega ion o complex sys ems in con empo a y ne wo ked applica ions means ha no single pe son e e sees he whole pic u e (B idle, 2018). Tha is a p oblem o human in e pe sonal beha iou and b ings us mul iple sociological pa adigms (e.g. au oma ion bias) (Pa asu aman & Manzey 2010), bu migh no be a p oblem in pe o ma ic in e ac ion. The ou come o he piece is mul iple imes he mos impo an hing, and i he e is a clea ade o in o ma ion in he main pa h o communica ion (pa h de ined as sende - ecei e link) wi h a con olled amoun o noise (Shannon, 1948) ha migh be a p oblem sol ed (bo h o he public and pe o me s). To he communica ion and o he embodimen . The 8Uni y main websi e h ps://uni y.com/ [accessed 27/06/2019] 9Un eal Engine main websi e h ps://www.un ealengine.com/ [accessed 27/06/2019] 10Godo main websi e h ps://godo engine.o g/ [accessed 27/06/2019] 2.1 Li e a u e Re iew 13 poin ha de ines con olled communica ion as a scale is based on he model de ined by Claude Shannon (1948). Figu e 2.4: Pic u e o he schema ic diag am o a gene al communica ion sys em. Claude Shannon on The Ma hema ical Theo y o Communica ion. Nowadays, om he simples social media use o he mos complex se ious in o ma ion s uc- u e, we ely on an inc edibly big communica ion ield. Tha ield by de aul , ies o e ase his noise sou ce and lea es us space o de elop c ea i e ideas and ways o expe imen ing i in new ways. These new ways o exp essing can some imes ake place in an in e ac i e dis ibu ed com- pu e sys em o eal- ime music made by mul iple use s (B own, 2007). To di e en ia e he models o collabo a i e beha iou as ypes, we ha e o spli sys ems by da a communica ion a chi ec u es. These ne wo ked models o collabo a i e ac ions migh be di ided mainly in Cen alized Ne wo k Models,Decen alized Ne wo k Models, and Dis ibu ed Ne wo k Models (He ns , Gu le, & Pe i , 2000; Pulkka, 1995). These di e en ways o building so wa e and lows o in o ma ion also b ing di e en ways o explo ing i , as medium o he a i sel (Blan on, 2019; McLuhan, 1967). And as he e a e new ways o explo ing i , new ideas and sys ems come o li e. A good example o musical pe o mance is he iconic wo k Dial ones (A Telesymphony) (Béda , 2002), a conce whose sounds a e wholly p oduced h ough he ca e ully cho eog aphed inging o he audience’s own mobile phones, p esen ed a A s Elec onica Fes i al (2001) and a he Swiss Na ional Exposi ion (2002). In con as o ha model o in e ac ion we ha e he The Bi mingham Ensemble o Elec oacous ic Resea ch (BEER) pe o ming a he Ne wo k Music Fes i al in 2014 wi h he h ee membe s si ing a a able. Su ounded by he audience con as ing 14 Con empo a y and His o ical P ac ice Figu e 2.5: Cou esy o RAND Co po a ion. Ne wo k Models p oposed by Paul Ba an in 1964. Image ound in documen RM-3420-PR. he e o o show ha he music is played in eal- ime, his pe o mance explo es a minimalis in e ac ion wi h he compu e (Akke mann, 2016). Figu e 2.6: Cou esy o he Au ho s. On he le a pic u e o Telesymphony pe o mance and on he igh a pic u e o BEER pe o ming in Ne wo k Music Fes i al (2014) in Bi mingham. Complex ne wo king schemes allow o he sha ing o pe o mance da a be ween musicians, howe e , he connec ion be ween he music and he lis ene s – bo h ensemble pa icipan s and audience membe s can be di use and un ocused (Diakopoulos, 2012), ha ’s why I b ing wo di e en examples o pe o ma i e in e ac ion. I ’s also impo an o no ice echniques like li e-coding (Colins e al., 2004) ha a e com- monly used nowadays in mee ings such as algo a es (Collins & McLean, 2014; Collins e al., 2003). In he e, we can see an e olu ion om bands like The Hub wi h s a e o he a li e-coding. The Hub is a band om 1980’s ha used San F ancisco bay a ea as a e ile g ound o compo- si ion expe imen ing wi h mic ocompu e s as a musical ins umen au oma a (B own, 2002), also 2.1 Li e a u e Re iew 15 g owing om he League o Au oma ic Compose s (Bou well, 2009). A good e e ence can be Alexand a Ca denas, cu en ly based in Be lin and also pa o he li e-coding scene in Mexico Ci y (Alex, 2013). Figu e 2.7: A pic u e o The League o Au oma ic Compose s pe o ming in San F ancisco 1981 (McKinney, 2016; B own, 2002) The li e coding pe o mances equi e lexible ools o eal- ime musical s uc u ing and he ool i sel ( he coding en i onmen ) mus no be a hind ance while pe o ming. Jus like a classi- cal ins umen , he a is needs o ha e clea con ol o e he engine. Seeking an agile and quick deploymen o he con en , he a is needs o use amewo ks ha allow ha . No only o li e- coding, bu also o apid p o o yping. The e a e examples like ChucK11 (Wang & Cook, 2003), TidalCycles12 (McLean & Wiggins, 2010), Sonic Pi13 (Aa on & Blackwell 2013), Imp omp u14 (So ensen, 2005) and Fluxus15 (Magnusson, 2014). These allow he as gene a ion o audio and/o g aphics. Some o hese amewo ks, ools, and a wo ks exis hanks o he needs o he a is ha c ea ed hem, which is some hing ha happens a lo in digi al sys ems. And some imes, when making a e iew o exis ing music and audio- isual p og ams we can obse e ha some aspec s o 11ChucK o icial websi e h ps://chuck.cs.p ince on.edu [accessed 27/06/2019] 12TidalCycles o icial websi e h ps:// idalcycles.o g/index.php/Welcome [accessed 27/06/2019] 13Sonic Pi o icial websi e h ps://sonic-pi.ne / [accessed 27/06/2019] 14Imp omp u o icial websi e h p://imp omp u.moso.com.au [accessed 27/06/2019] 15Fluxus o icial websi e h ps://www.paw al.o g/ luxus/ [accessed 27/06/2019] 16 Con empo a y and His o ical P ac ice a is ic exp ession especially on elec o-acous ic composi ion a e no add essed by s anda d music p og ams (Thiebau e al., 2008). Tha migh be, o no , a p oblem. 2.1.6 Ins umen s using Web Technologies By le e aging he in e ne as a li e so wa e ecosys em, his chap e will examine how such echnology can bes each a is s, and li e up o i s po en ial o undamen ally change c ea i e p ac ice in he ield. As a s a ing poin compa ed o he days o oday we ha e he long dis ance conce s by elephone. Figu e 2.8: Pic u e ound on a icle Long Dis ance Telephone Conce s by Scien i ic Ame ican, Feb ua y 28, 1891, page 130. Pe o me s in New Yo k and audience in New on Massachuse s. C ea ing pla o ms whe e he newes echniques can be used by a is s as pa o hei day- o- day job is a poin o inno a ion ha happens all o e he wo ld. Using b owse -based ools is one o hem. The e a e some simila i ies on he esea ch happening now and he i s expe imen s o musical pe o mance using in e ne connec ions. Fo ins ance, audio s eaming o collabo a ion was a big opic o esea ch in Uni e si y o Sou he n Cali o nia (1993) (Schoole & Touch, 1993), nowadays wi h big ad ances in b oadband speed. A couple o yea s la e , in June 13, 2002 he Uni e si y o McGill and S an o d help he i s anscon inen al jam session u ilizing high quali y audio and ideo which la e led he Uni e si y o Sou he n Cali o nia de elop hei own echnologies o wha hey called Dis ibu ed Imme si e Pe o mance (DIP) (Chew e al., 2004). 2.1 Li e a u e Re iew 17 App oaching composi ion as plu al beha iou , we ha e an impo an miles one in collec i e c ea ion sys ems wi h he piece o so wa e FMOL. In 1997 he Ca alan hea e company La Fu a dels Baus commissioned he expe imen al a is Se gi Jo dà o de elop an online collec i e music c ea ion sys em (Ba bosa, 2006; Ba bosa, 2003). The a is could connec o his p ojec and ind a web-based da abase which allow he use o upload pieces o he se e o la e modi ica ion. This p ojec la e included he play F@us 3.0, inspi ed on Goe he’s wo k. Figu e 2.9: Cou esy o Al a o Ba bosa. A sc eensho o FMOL so wa e. A web-based ee-s uc u ed da abase. c Se gi Jo dà Nowadays, we ha e in e aces embedded in he b owse s ha allow us o con ol all kinds o compu a ional asks h ough many di e en API’s in o ou o he WWW. Recen de elopmen s also allow use s o make hei own lib a ies and de elopmen in powe ul languages like Ja aSc ip (JS) (Flanagan, 2002) and i is impo an o audio- isual a is s ha wo k on b owse -based ech- nologies o unde s and he s uc u e o bo h he Web Audio API (Fa o y & Se a 2018; Smus, 2013) o audio de elopmen and he WebGL API (Pa isi, 2012) o g aphics and in e ac i i y. // Jus as he In e ne o e s huge po en ial as a sha ed musical pe o mance space (Jin, 2015), i also has o he g ea applica ions o he a s. Tha can be ound in examples like music p oduc ion on cloud ools (Mo andi, 2017), au oma ic audio mas e ing (Najduchowski, 2018) and collabo- a i e code edi ing (Fiala, Yee-Kind & G ie son, 2016). Which is he example o Sound ap16, LandR17 and CodePen18. Some o hese ools ound ac oss he in e ne may no be done speci i- cally o and o a is s, and no a all co ela ed wi h a is ic pe o mance. Bu migh be in e es ing o s udy as a po en ial use case. 16Sound ap main websi e h ps://www.sound ap.com/ [accessed 27/06/2019] 17LandR main websi e www.land .com/ [accessed 27/06/2019] 18CodePen main websi e h ps://codepen.io/ [accessed 27/06/2019] 18 Con empo a y and His o ical P ac ice 2.1.7 In e media and Digi al In e aces The e is a lo o in e es ing wo k om a is s all o e he wo ld ying o me ge echnologies oge he , and he e is also a g ea desi e o ex end he human senses in a . A is s o cou se also explo e ha ield, and one way o doing ha is o me ge echnologies oge he and messing a ound wi h di e en eelings. An example o ha migh be he wo k o Neil Ha bisson (2013), audio isual a is and p esiden o he Cybo g Founda ion19 who implan ed a senso in his b ain o pe cei e colo s using hei equency ange and ansmi ing i o his b ain (B yan , 2013), his allows he a is o "hea ” colo s. This in e en ion in his body was con- side ed one o he mos shocking pos -human ans o ma ions o all imes alongside he ex eme pe o mances o Yoko Ono and Ma ina Ab amo i´ c (Jones, 2013). The a is S ela c, pseudonym o S elios A cadiou, migh also be wo h men ioning as one o he mos impo an a is s ying o push he bounda ies o human pe cep ion and capabili ies, known o his ans o ma ion o he ea in his a m, pe o mances like he “Thi d Hand” in his body (Paolo & Ki k, 1995) and o ex s like “F om Zombie o Cybo g Bodies - Ex a Ea , Exoskele on and A a a s”. These ex ensions a e jus like he mul imodali y o an ins umen and i ’s p opaga ion, some- imes h ough di e en media. I ’s a ba ie ha is designed by each medium b oken when neces- sa y, om echnological me hods o he human body. Using he iconic ph ase by McLuhan (1964) "We ha e al eady ex ended ou senses and ou ne es by he a ious media" we all unde s and he in en ions o hese analogies. Coming back o he audio isual domain, senso y associa ions be ween colo and music a e ound ac oss many cul u es as well. One o he ea lies is he Py hago ean ques o assign a pa icula colo o each musical no e, bu in mo e ecen imes we ha e mo e people in ol ed in synaes he ic expe iences like Cha les Baudelai e, Wassily Kandinsky, Se gei Eisens ein, and Richa d Feynman (Ione & Tyle 2003). We know o Kandinsky’s in e es in "colo ed hea ing”, and also Nina Kandinsky, his wi e, w o e ha he passiona ely lo ed colo s om ea ly childhood and claimed o dis inguish a pa icula smell and musical sound o each colo , a common synes he ic ai (Kandinsky, 1947, p.9). These analogies a e also a b idge o he wo d media a and new media a , wha e e ha means. The s uc u es ha a e buil ake ad an age o one media as ex ension o ano he . The e ms ansmedia20 and c ossmedia a e impo an on his ma e bu in ega ding his pa icula example in e media migh be mo e accu a e. We ha e o ealize he impo ance o he e m in e media, b ough o us by he compose and poe Dick Higgins, also pa o he g oup Fluxus (Smi h, 1998). The exp ession says ha when wo o mo e disc e e media use concep ually, hey become in e media, hey hen di e ence om mixed media, being insepa able om he essence o he wo k o a (Longhi, 2002; Higgins, 1984:138). 19Cybo g Founda ion main websi e h ps://www.cybo g ounda ion.com [accessed 27/06/2019] 20 2.1 Li e a u e Re iew 19 Figu e 2.10: Kandinsky’s Yellow-Red-Blue (1925) is om his pe iod o o ally non- ep esen a ional wo ks, bu his pa icula pain ing seems e oca i e o he concep o a pe son expe iencing i id ‘synes he ic’ image y. Oil on can as, Musée Na ional d’A Mode ne, Cen e Geo ges Pompidou. Rep in ed wi h pe mission: Réunion des Musées Na ionaux/A Resou ce, NY. 2003 A is s Righ s Socie y (ARS), New Yo k/ ADAGP, Pa is. Amy Ione and Ch is ophe Tyle on Neu ohis o y and he A s. 2.1.8 Con empo a y Media A As quo ed in he i s chap e , his esea ch is ca ied ou in he con ex o B aga Media A s, ha ing a e y close ela ion wi h media a p ac ice. C ea ed in 2004, he UNESCO C ea i e Ci ies Ne wo k (UCCN) p omo es coope a ion wi h and among ci ies ha ha e iden i ied c ea i i y as a s a egic ac o o sus ainable u ban de elop- men . Wi h he g ow h o his ne wo k, UNESCO aims o place c ea i i y and cul u al indus ies a he hea o he ci ies de elopmen plans and o p omo e an ac i e coope a ion be ween hem21. Since i s ounda ion, B aga has been in he a an -ga de o i s di e en con empo a y scena - ios, we he i was unde he oman empi e o du ing he sump uous ba oque pe iod. Wi h each challenge, he ci y has become iche in di e si y and mo e cu ious abou nea u u e.22. 21Ci a ion o he UCCN page on B aga Media A s Websi e h p://www.b agamediaa s.com/en/uccn/ [accessed 27/06/2019] 22Ci a ion om B aga Media A s o icial Websi e h p://www.b agamediaa s.com/en [accessed 27/06/2019] 20 Con empo a y and His o ical P ac ice 2.1.8.1 Media A s De ini ion The e a e many de ini ions o Media A s. Fo B aga Media A s we s a wi h he concep o Media A s as an in eg a ion o new echnologies in o a is ic and c ea i e p ac ice, as well as in social exchange and inno a ion. This includes disciplines such as ideo games, digi al anima- ion, digi al and in e ac i e a , sound a , ilm and digi al ideo, digi al ele ision and he use o echnology in he pe o ming a s23. 2.1.8.2 Educa ional Depa men Goals New echnologies ha e e olu ionized he wo ld o a is ic c ea ion, ex ending i o new a is s and, abo e all, new a s. In some cases democ a izing while in o he s isola ing. The B aga Me- dia A s Educa ional Se ice akes his space be ween c ea ion, Media A s and he communi y. Relying on new echnologies, i encou ages he consump ion o a , pe o mance, c ea ion and de elopmen o ools and knowledge. The goals should be achie ed h ough a ans e sal p o- g amming ha se es he communi y using media as he basis o de elopmen 24. As a connec ion link o o he media a ci ies a ound he wo ld, B aga Media A s alongside gn a ion25 galle y has hos ed a is s wo ldwide e e enced like Ryoichi Ku okawa26, Lucas Pa is27 and Nono ak28. Figu e 2.11: Image wi h bo h he o icial logo o B aga Media A s (le ) and he o icial logo om UNESCO’s UCCN ( igh ). 2.1.8.3 Towa ds Media A s P ac ice Wunschmaschine und wel e indung which alludes o wo speci ic aspec s o echnology is one example o his ma e . I e lec s in aspec s o echnology, mo e speci ically o bo h i s ision- a y po en ial as an objec on o which o p ojec an asies and desi es, ha ing he powe o in en he wo ld (Büsche e al., 1999). Also, in 1997, Felde e (Eine Geschich e de Technik isionen) s a es ha echnology-based isions a e o be seen as mnemo echnical s uc u es ha allow o hink he unimaginable, o p ojec an image o he ne e -be o e-seen... “Technology as con en 23BMA public de ini ion on Media A p ac ice. 24Public objec i es o BMA Educa ional Se ice as a di use o a is ic ac i i y. 25gn a ion main websi e h p://www.gn a ion.p [accessed 27/06/2019] 26Ryoichi Ku okawa main websi e h p://www. yoichiku okawa.com [accessed 27/06/2019] 27Lucas Pa is main websi e h p://lucaspa is.ca [accessed 27/06/2019] 28Nono ak main websi e h ps://www.nono ak.com accessed 27/06/2019] 2.2 S a e o he A 27 de ices o ganized, ep oducing di e en noises, samples, and syn hesized sounds om andom loca ions in he hall. This pe o mance is a solid example o wha can be accomplished wi h a node.js se e , ins an ia ing a ious sound gene a o s and p opaga ing in he ec o o connec ed use s using mobile phones as playback machines. This pe o mance was also p esen ed a he He zen Uni e si y and C oss A es i al (2015), bo h in S . Pe e sbu g, Russia. This a wo k has clea ela ionships wi h Telesymphony by Golan Le in and his mode o pe o ma i e in e ac ion. The cellphones o he public we e also emo ely con olled by And ey Bundin. Figu e 2.20: Two pic u es o he Conce o Sma phones and O ches a in Geo gia Tech (2016) du ing Web Audio Con e ence (WAC). On he le And ey Bundin on elec onics and on he igh as iolin, Alexey Ko che ko . As said ea lie , he in e ne is a place whe e people end o wo k on a lo o di e en ideas collabo a i ely (some imes singula ly), and we can ind inc edible examples o ins umen s de el- opmen some imes ne e done in he academic con ex ha ac ually can be a e e ence. Tha is he example o wo ks like Simulac a34, he Ch ome Expe imen s Lab35, he Mul i-playe Piano36, and Blockdus 37. The e a e also p ojec s ha i in his sec ion bu don’ pa icula ly use he web no web echnnologies. P ojec s like Tempes (Vig oux & Schmi , 2012); STM Duali y (Sanz & Ba andun, 2015) and Imposi ion (Schwa z & Edisonnoside 2012) (Ca alho, 2018). To use echnology as a new mean o a is ic exp ession is e y impo an , e olu ion o he indi idual by he indi idual. (S ela c, 2011). 34O icial eposi o y o he au ho h ps://gi hub.com/ChadMcKinney/Yig [accessed 27/06/2019] 35O icial websi e o Ch ome Expe imen s lab h ps://expe imen s.wi hgoogle.com/collec ion/ch ome [accessed 27/06/2019] 36O icial websi e o he Mul i-playe Piano h p://www.mul iplaye piano.com [accessed 27/06/2019] 37O icial websi e o Blockdus h ps://blokdus .com [accessed 27/06/2019] 28 Con empo a y and His o ical P ac ice 2.3 Conclusion On he basis o his ex ensi e ex , and wi h his g oup o e e ences aised, some imes di e en in many aspec s, a p ac ical pa will be pe o med using some o he cha ac e is ics ound ac oss all o hem. S uc u ing ela ionships on collabo a i e ne wo ked sys ems and AV in e ac ion, we p esen Akson, a digi al en i onmen as p oduc o his esea ch. We will documen and explain he whole p ocess o concep ual and echnical de elopmen on he nex chap e o his documen . Wi h an app oach ollowing he guidelines o p ac ice-led esea ch (Liampu ong & Rumbold, 2008), we c ea e a s ong bond wi h a is ic pe o mance, making expe imen s du ing he esea ch pe iod. The ou come o all he e e ences aised in his documen as li e a u e and as con empo a y p ac ice on he ield a e wha we conside o be he ounda ions o Akson. Chap e 3 Akson En i onmen Akson (’æk,son) is he digi al en i onmen p esen ed in his esea ch, and his chap e p esen s i ’s concep ual and echnical de elopmen in de ail. The ollowing sec ions a e made in o de o demons a e he en i e p ocess o e olu ion wi h an in ensi e explana ion o he po en iali ies, hei e e ences and he in e ace. Figu e 3.1: P in Sc een o Akson ook by he au ho s. 3.1 O e iew To s a o explain he whole exis ence o Akson, i migh be a good p ac ice o de ine one o he bigges concep ual easons i has, as an en i onmen . Akson is a b idge, a linking cu en be ween people, be ween de ices and be ween audio/g aphics. Akson is also called so due o he ce eb al o ganiza ion o he Axons. I we look a he neu al a chi ec u e o a human b ain and 29 30 Akson En i onmen y o unde s and he s uc u e ha neu ons c ea e, we ealize ha he Axon is he medium ha a neu on uses o es ablish con ac wi h ano he neu on (A bib & Aleksande , 1995). The ne ous issue (o neu al) is composed o ne e cells (neu ons), suppo ed by cells in he glia. Neu ons a e he cells essen ial o b ain ac i i y. In ou b ains he e a e billions o hese neu ons o ganized in local ci cui s, which in u n cons i u e subco ical egions. Neu ons ha e h ee impo an componen s: a Cell Body, Dend i es, and he Axon. Neu ons a e in e connec ed in ci cui s whe e he e is he equi alen o conduc i e wi es (axon ibe s o neu ons) and connec o s (synapses, he poin s a which axons con ac he dend i es o o he neu ons) (Damásio, 1994). The way he Akson s uc u e is o iginally hough , is a bi like b ain ac i i y, and no jus an analogy o he Axons as connec ion links. The bundles o axons a e se he b ain subs ance in he whi e mass, connec ing di e en egions o he ce eb al co ex. Some egions a e local, be ween egions o he co ex sepa a ed by small millime e s, while o he connec ions bind egions ha a e a apa , such as, o example, co ical egions om one o he ce eb al hemisphe es o co ical egions o he o he (Pu es e al, 2004; Damásio, 1994). This is undamen ally how we should look a Akson in e ms o scale and ne wo k s uc u e. The neu ons exis in he ne wo k o he b ain, ha by i sel exis s in he ne wo k o he body, ha exis s in he ne wo k o he socie y. Akson also exis s as a ne wo k, in he ne wo k o he Web ha by i sel in e aces wi h o he ne wo ks ac oss he wo ld. I also c ea es smalle s uc u es wi h people ha a e cu en ly in he same in e ac ion model (see sec ion 3.7), be ween de ices and be ween audio/g aphics. Figu e 3.2: Cou esy o Publicações Eu opa-Amé ica. Image in O E o de Desca es by An ónio Damásio. Diag am o a neu on wi h i ’s p incipal componen s: Cell Body, Axon and Dend i es. This analogy o he human body and he s uc u es in which i is subme ged is undoub edly one o he mos impo an concep s ha de ines Akson. I is also a possible way o in e p e ing i as an in e ace and/o o o he ypes o in e sec ion be ween di e en media, spaces and people. 3.2 Akson and Web Technologies 31 3.2 Akson and Web Technologies Akson echnologies ely on he Web and i ’s p og amming in e aces, allowing e e y de ice ha can connec o he in e ne o use he sys em. I has se e -side p og amming o de ine di - e en modes o in e ac ion (in e pe sonal and singula ), and on -end o Audio and G aphics gene a ion/playback. This way i ’s possible o ha e gene al con ol capaci y be ween use s (a is s o audience) and es ablish in e ac ion me hods inside he ne wo k. In he beginning o his esea ch he code s a ed o be alloca ed in he He oku1in as uc u e, a cloud applica ion pla o m ha has simila p ope ies o he Amazon AWS2cloud compu ing se ices. And he e sion con ol o he code was done h ough Gi Hub3, a common so wa e de elopmen pla o m. These in e aces ha e c oss in eg a ion o make i easy o deploy digi al sys ems. On he nex igu e we ha e a diag am ha shows he wo k- low adop ed o he code de elopmen in his esea ch. Figu e 3.3: S a ing on an open sou ce code edi o A om (bo om-le ) and hen uploading i o he cloud h ough Sou ceT ee (bo om- igh ) an open-sou ce Gi GUI so wa e. When he sys em is ully upda ed we ha e he Gi Hub in e ace o he Akson eposi o y wi h all he code commi s ha by i sel in e aces di ec ly wi h He oku pipeline. 1h ps://www.he oku.com/home [accessed 27/06/2019] 2h ps://aws.amazon.com [accessed 27/06/2019] 3h ps://gi hub.com [accessed 27/06/2019] 32 Akson En i onmen Cloud se ices ha o e Unix machines o people o use, no mally in e ace wi h se e al p og amming languages and/o lib a ies. In his esea ch, we use node.js4(Teixei a, 2012; Tilko & Vinoski, 2010) o backend p og amming. This is a JS un ime buil on Ch ome’s V8 JS engine. A Google’s open sou ce high-pe o mance JS and WebAssembly engine, w i en in C++ (G ay, 2009). node.js lib a ies we e managed h ough npm5(Wi e n, Su e & Rajagopalan, 2016), s a ic ile se ing h ough exp ess6(Bush & Linden, 2016; Holmes, 2015) and he connec ions be ween use s we e managed h ough socke .io7(Cadenhead, 2015; Rai, 2013; McCaw, 2011), a lib a y o eal- ime web applica ions ha enables wo-way communica ion be ween web clien s and se e s. Wi h he se e handling he connec ions and he beha iou o clien pages, i was hen possible o s a de eloping he AV sys em on op o he Web Audio API (Robe s, Wake ield & W igh , 2013; Wyse & Sub amanian, 2013) and he WebGL API (Pa isi, 2012; Ma ion & Jomie , 2012). A be e desc ip ion o he AV engine will be done in he nex poin s o he chap e . The e a e online esou ces ha suppo he assessmen and unde s anding o his esea ch. The main code (wi h he p og ess placed in ime) can be ound on h ps://gi hub.com/luisA andas/akson. Documen a ion ega ding he code can be ound on h ps://akson. ead hedocs.io/en/la es /index.h ml (an open-sou ce and ee documen a ion pla o m). And sound acks om he public expe imen s done wi h Akson (see chap e 4) can be ound on h ps://luisa andas.bandcamp.com/album/akson- en i onmen . Figu e 3.4: On he le , sc eensho o he o ical ead hedocs websi e. On he middle, sc eensho o he commi s ab in Gi Hub. On he igh a sc eensho o Akson’s bandcamp wi h music om public expe imen s. 3.3 Audio Engine As p e iously men ioned (sec ion 3.2), he sound sys em is buil om he Web Audio API. This is a high-le el Web API o p ocessing and syn hesizing audio in web applica ions. I in- ol es handling audio ope a ions wi hin an audio con ex , and has modula ou ing. I is designed ca e ully o allow a powe ul and e sa ile way o con olling audio on he Web (Kleimola & 4h ps://nodejs.o g/en/ [accessed 27/06/2019] 5h ps://www.npmjs.com [accessed 27/06/2019] 6h ps://exp essjs.com [accessed 27/06/2019] 7h ps://socke .io [accessed 27/06/2019] 3.4 G aphics Engine 33 La kin, 2015; Smus, 2013). Technically, he Akson sound sys em is made using na i e JS (Flana- gan, 2002) and he lib a y one.js (Mann, 2015). In hese lib a ies a e cons uc ed he wo main di isions in he sound sys em. The Syn hesize (Lead) and an Au o-Fil e ed Noise gene a o (Backg ound). These ha e indi idual con ols on hei me hods/ unc ions as abs ac ed classes o he sys em. The e is also he possibili y o de ine he no es he cu en Syn hesize is going o play, and de ailed explana ion on he in e ace will be made in sec ion 3.6. 3.4 G aphics Engine The g aphics sys em is buil om WebGL (sec ion 3.2), an API used o c ea e 3D g aphics in a Web b owse . Based on OpenGL ES 2.0, WebGL uses he OpenGL shading language, GLSL, and o e s he amilia i y o he s anda d OpenGL API (Angel & Sh eine , 2011). Because i uns in he HTML5 Can as elemen , WebGL has ull in eg a ion wi h all DOM in e aces (Anyu u, 2012). Akson has ou echnically a anged scenes as objec s. They ha e mul iple AV p ope ies unique o each o he , allowing in e ac ion wi h he ins umen ’s me hods wi hou equi ing a GUI. As he use uses he capabili ies o he in e ace, hey a e alloca ed dynamically in he space o he web page. The g aphics sys em uses na i e JS and he h ee.js (Di ksen, 2013) lib a y. A p ojec ha c ea es a ligh weigh 3D g aphics lib a y wi h a de aul WebGL ende e . 3.5 Ins umen s o Akson The Akson is o iginally designed ha ing ou main di isions as a g oup o ins umen al po- en ials. As ci ed in sec ion 3.3 he e a e wo main di isions in he sound sys em (Lead and Backg ound), one called G aphics ( o con ol mainly me hods in he a ious g aphic scenes) and ano he called Pos -P od which has gene al AV con ols in he en i onmen . The a ious dis- ibu ed me hods belonging o each pa can be accessed by he Ins umen s panel ound in sec ion 3.6.3.2. The di ision o Akson’s expe ience in o hese ou pa s is also explo ed in he a ious modes o in e ac ion (see sec ion 3.7). 3.6 In e ace To expe ience Akson, he use does no need o deal wi h code o unde s and how he echnolo- gies desc ibed in his documen wo k in he i s place. One can access he cu en hos by yping in a b owse www.akson.xyz o download he las s andalone e sion om Gi Hub ab eleases. The ci ed DNS se es he cu en se e used in He oku. I is ecommended o use Ch omium- based b owse s, he main a ge o his p ojec . Ch omium8is an open-sou ce web b owse p ojec 8Ch omium main websi e h ps://www.ch omium.o g [accessed 27/06/2019] 34 Akson En i onmen in which Google Ch ome bases i s sou ce code. As men ioned in sec ion 3.4, Akson is di ided in o ou scenes ( ig. 3.5). These allow in e ac- ion by he compu e mouse and/o ouch sc een, explo ing di e en sound p ope ies/me hods o he sys em. Sonically, he i s and ou h scenes (coun ing in igu e 3.5 om le o igh ) explo e me hods ela i e mainly o Lead, and he second and hi d explo e mainly o Backg ound. Figu e 3.5: Sc eensho s o he ou g aphical scenes o Akson in hei de aul s a e. In he ollowing sec ions he ela ionship be ween g aphical in e ac ion and sonic eedback will be explo ed in dep h. These ela ionships a e also b oken down by a cus om GUI ha can be ound when he use connec s by Desk op (sec ion 3.6.1). Also, WebGL is a DOM API, which means ha i can be used om any DOM-compa ible language: JS, Ja a, o —i you embed WebKi in o an applica ion on a Mac — Objec i e C (Thiemann, 2005). •In e ac ion on he Fi s Scene –He e he use is able o play no es on he main Syn hesize (see sec ion 3.6.3.2) by clicking he ins an ia ed geome ies. When Akson begins, objec s a e ins an ia ed and i ’s possible o o malize in e ac ion h ough ay acing9. This allows AV eedback om a cus om pe spec i e. The ins ancia ed geome ies change colo on click ( o all use s). •In e ac ion on he Second Scene –The second scene le s you con ol Backg ound p ope ies in he o m o o a ion o he con ained geome y. This scene c ea es a cus om dodecahed on wi h e ices sca e ed ac oss space ha by i sel con ols mul iple ins ances o he Backg ound domain o Akson. A mo e de ailed explana ion on he me hods can be ound in ig. 3.9. •In e ac ion on he Thi d Scene –The hi d scene c ea es a new ins ance o geome ies ha occupy he whole ield o he web page. These geome ies can be con olled by he XY posi ion o he mouse/ ouchpad and also con ol me hods o he Backg ound domain. By mo ing h ough he web page he use can ha e a dynamic iew o he objec . •In e ac ion on he Fou h Scene 9 h ee.js Ray acing/Raycas e documen a ion h ps:// h eejs.o g/docs/api/en/co e/Raycas e [accessed 27/06/2019] 3.6 In e ace 35 –The ou h scene has a un look simila o he i s bu wi h a di e en aes he ic and con olling di e en me hods o he Lead. He e he mas e syn hesize is played by playing no es when he e is human in e ac ion. The ins an ia ed geome ies may be in wi e ame mode, and he colo s may also be di e en . These isual aes he ic choices hen ha e epe cussions on he ype o sound played. Rega ding AV ela ionships. As he beha io o hese sys ems may be ied o di e en ac- o s, a pe spec i e on in e ac ion is no solely ocused on ac ion- eac ion pa e ns, bu on he o e all a iable beha io o he wo k, in each occu ence and in esponse o in e ac ion (Ribas, 2013). These ela ionships a e also explo ed collabo a i ely among use s who sha e Akson, a mo e de ailed explana ion in modes o in e ac ion can be ound in he sec ion 3.7. 3.6.1 Desk op Akson is an audio- isual ins umen ha allows in e ac ion on he b owse sc een di ec ly wi h he g aphics gene a ed, bu also has a mo e con en ional GUI composed o dials, slide s and bu ons (sec ion 3.6.3). This GUI is ound on compu e s (bo h Desk op and Lap op) and can be enabled/disabled om he space ba on he keyboa d (keyboa d unc ions in de ail can be ound in sec ion 3.6.3.5). Wa an ies a e no gua an eed o wo k on able s (bo h iOS and And oid) due o he ola ili y o machine ypes (in size and specs) bu i is always possible o call he i ual keyboa d. In he nex sec ions we explo e he a ious ypes o in e aces c ea ed o Akson, how hey can be used and wha hey con ain. The e is a main ba a he op o he window, he e a e in e aces in Dialog10 o ma and Modal11 o ma . They a e buil using he Nexus open-sou ce p ojec (Allison, 2011) ha has a se o nexusUI Ja aSc ip objec s (Allison, Oh & Taylo , 2013) demons a ing many dynamic unc ions ha can be done wi hin he can as objec inhe i ing om HTML5 DOM. Akson has 64 slide s, 109 bu ons, 8 dials and bidi ec ional socke connec ions o all o hem. 3.6.2 Mobile On mobile phones, Akson exis s in a speci ic way and a bi di e en om compu e s. I e- mo es all GUI, allowing only he in e ac ion wi h he scenes h ough he ouch sc eens. Whene e he se e ecognizes an inpu , i ques ions wha kind o in e ace i is. I i s ays on he ype o in e ace ecognized as mobile phone, hen Akson will un his way. De ices o his kind also allow you o in e ac wi h scenes and play con en . The e is also an adop ed cha ac e is ic made o allow a la ge-scale con olled expe ience. I he e is a de ice in he 10Dialog de ini ion on Compu e Hope h ps://www.compu e hope.com/ja gon/d/dialogbo.h m [accessed 27/06/2019] 11Modal de ini ion on Compu e Hope h ps://www.compu e hope.com/ja gon/m/modal-window.h m [accessed 27/06/2019] 36 Akson En i onmen Desk op/Lap op ca ego y (see sec ion 3.6.1) changing he scene, he mobile phones also change. In his way, i is possible o in e ac emo ely (wi hou GUI and wi hou AV con en s eaming) wi h la ge quan i ies o mobile phones wi h po en ially la ge dis ances and o ep oduce con en in a cohe en way. I also allows us o hink o hem as esonan machines in space (Blan on, 2019; Hen iques, 2015) as a dis ibu ed a ay o speake s. The igu e on he le is a pho og aph o a Huawei P10- Li e mobile phone connec ed o Akson. I is an And oid sma phone wi h a sc een o 5.2 inches and a esolu ion o 1920x1080 pixels. Wi h ha dwa e om 8 Co e o 1.9 GHZ and unning he EMUI 8.0 sys em. The igu e shows he de aul s a e o Akson on mobile de ices. 3.6.3 G aphical Use In e ace Apa om all he au oma ically gene a ed g aphical/sonic con en and he po en iali ies in he in e ac ion o he scenes, he g aphic in e ace will be ully explained in his sec ion. I is di ided by sec o s and hey a e illus a ed wi h images, ables and ex sepa a ed by subsec ions. 3.6.3.1 Mas e Dialog The Mas e Dialog. I has e ec on he main ou o he b owse sound/g aphics sys em. Con- nec ed o he AudioDes ina ionNode12, i p o ides use ul con eniences such as he abili y o se he olume and mu e/s op he en i e applica ion. This dialog has 8 dials, 3 bu ons and an oscillo- scope. On he igu e 3.6 we can see (on he le ) a p in sc een o a Mas e Dialog and (on he middle and igh ) he Sys em dialog (opened by he Abou bu on). This b ings up a window wi h in o - ma ion abou he sys em whe e Akson is unning, and explains he a ious models o in e ac ion in he ne wo k (sec ion 3.7). The in o ma ion we can ge om he sys em a e unc ions om he na iga o and window JS objec s and we can compa e ne wo k in o ma ion and AV p ope ies o he machine cu en ly playing. The a ious in e ac ion models a e explained wi h e e ences o Todd Winkle ’s book Composing In e ac i e Music (1998). On he bo om o he igu e we ha e he a ious widge s lis ed and hei ac ions. The dialog windows o Akson a e minimizable, closable and a e d aggable, o gi e eedom o he use . The 12h ps://de elope .mozilla.o g/en-US/docs/Web/API/AudioDes ina ionNode [accessed 27/06/2019] 3.8 Code, Cons uc ion and Implemen a ion 43 g oups o nego ia e hese challenges and s uc u e hei collabo a ion." (F eeman & Van T oye , 2011) 3.8 Code, Cons uc ion and Implemen a ion The whole p ocess o Akson’s echnological e olu ion since he beginning o he esea ch was documen ed om he ou se . All he s eps aken (commi s and uploads o he se e s) a e eco ded so ha e e yone can see how i e ol ed in ime. I is possible o c ea e compa isons wi h he decisions aken and o ela e hem o he expe imen s h oughou he hesis p ocess. Gi Hub has ea u es as VCS ha allow you o e e iles back o he p e ious s a e (any ime on he e- sea ch pe iod). On he main Akson eposi o y websi e h ps://gi hub.com/luisA andas/akson you can na iga e o he Code ab h ps://gi hub.com/luisA andas/akson/commi s/mas e and na iga e he eposi o y in ime o compa e he new Akson s a es. O cou se in compu e p og amming some s eps a e aken jus o sol e logical p oblems ha a ise when we se a goal. These s eps we e aken in o de o achie e a bigge challenge in he o e all s uc u e o he sys em, o en building he ea u es ha oday make Akson. On ig. 3.16 we show a diag am o he Akson eposi o y ile ee and he olde communi- ca ion low. S a ing on he P oc ile ( op igu e), he Unix se e is old o execu e node se e .js, and hen he se e is s a ed. F om he e a public olde is ins an ia ed wi h he index.js ile ha will ead he emaining iles and b idge dependencies. In he lowe le co ne we ha e he layou o he olde s and iles seen om he oo di ec o y and o he igh he a ious ile p o isions in each olde (docs,public,ex e nal and node modules). Wi h his se , we in eg a e Akson’s Gi Hub h ough Webhooks (Mo gan, 2018) o dynamically change he se e in he Uni ed S a es. 3.9 Conclusion In his chap e i ’s done an ex ensi e jou ney o he concep , de elopmen and echniques used in he p oposed en i onmen , Akson. He e we demons a e i ’s ea u es, how hey wo k and how hey we e done. I p esen s a inished AV en i onmen and i s e olu iona y p ocess also ela ed o he public expe imen s o which i was a ge ed (chap e 4). The e is a de ailed desc ip ion o he in e ace and he ac ions i has as a ool, wi h g ea emphasis on he e e ences ha in luence he way o hinking ha led Akson o be done. The design o a new ool is a i ualiza ion o many ac ions (Ba don & Malmbo g, 1997). When someone designs a ool, ins ead o ocusing on he ac ion s/he ocuses on some hing much mo e gene al. Wha Akson ep esen s and means is subjec o an explana ion a he beginning o he chap e . I ’s demons a ed ha he p oblem o be sol ed is g ea e han simply he de elopmen 44 Akson En i onmen o a collabo a i e sys em, o a ne wo ked sys em, and e en g ea e han jus building his sys em. I is a c ea o o ne wo ks. A ne wo ked en i onmen wi hin ne wo ks, c ea ing ne wo ks. As an ins umen and as a ansmu able en i onmen . The analogy o he o ganiza ional pa e ns in he human b ain and he ex ension o he human ges u e in he digi al wo ld is men ioned as main concep o i s exis ence. I ’s also done a ull e iew o he unde lying echnologies used in his p ocess. The explo- a ion o oppo uni ies p o ided by he cloud and how ha was wha o igina ed he easy esolu ion o he in e ac ion me hods used. All he ea u es o Akson, amewo ks and lib a ies used a e also explained. Mos o all, his chap e se es o unde s and he a ious ways o using Akson, how i wo ks and wha i ep esen s. I di ides he cen al componen s o he sys em in he a ious sec ions and subsec ions, c ea ing poin e s be ween hem. 3.9 Conclusion 45 Figu e 3.8: Sc eensho o he Syn hesize panel ollowed by wo ables wi h he a ious widge s unc ions and me hods. 46 Akson En i onmen Figu e 3.9: Sc eensho o he Backg ound panel ollowed by wo ables wi h he a ious widge s unc ions and me hods. 3.9 Conclusion 47 Figu e 3.10: Sc eensho o he G aphics panel ollowed by one able wi h he a ious widge s unc ions and me hods. 48 Akson En i onmen Figu e 3.11: Sc eensho o he Pos -P od panel ollowed by wo ables wi h he a ious widge s unc ions and me hods. 3.9 Conclusion 49 Figu e 3.12: Sc eensho s o he Alloca e dialog ( op), he Logs dialog (middle-le ), he Sa e Se ings dialog (middle- igh ) and he Top ba wi h a panel con aining me hods o ac ion. 50 Akson En i onmen Figu e 3.13: Sc eensho o he modes o in e ac ion modal igge ed by he Top ba . Figu e 3.14: Keyboa d map (QWERTY) wi h a panel con aining me hods o ac ion, Keycode E en s and he unc ions ound in he code. 3.9 Conclusion 51 Figu e 3.15: Diag am o Akson’s descen e model. 52 Akson En i onmen Figu e 3.16: Diag am o Akson’s s eamed model. 4.2 Expe imen a gn a ion 59 •Mos people used he equipmen p o ided by he s a ion and no hei own mobile de ices o in e ac wi h he sys em h oughou he expe ience. •Apa om musicians and a is s ha asked ques ions abou he sys em, who spen mo e ime in e ac ing we e child en and young people up o 30 yea s old. •O e all, while in e ac ing wi h Akson, people we e mo e ocused on he g aphics un il hey disco e ed he capabili ies o he GUI. E en i he sys em had a dedica ed sc een o show a endees an image wi h key con ols o in e ac wi h he keyboa d (see ig. 6.4), help was always gi en o anyone en e ing he ins alla ion space. Mos people also in e ac ed e bally when hey came ac oss he se up and some pa ici- pan s also used hei de ices, he mos success ul ones we e Ch omium-based b owse s (cu en ly de aul on And oid is Ch ome). Tha helped o see he pe o mance on all kinds o de ices (Sa a i wasn’ always success ul and a "No MIDI Suppo " ale was common). Figu e 4.2: Cou esy o gn a ion’s communica ion manage . Pho og aph ook on he si e o he ins alla ion du ing he mo ning o open day. 60 Akson Public Expe imen s 4.3 Expe imen a Open ield The hi d expe imen was done on May 6 h in Open ield8C ea i elab, loca ed in he cen e o Po o, home o a Media A Collec i e. I was pe o med wi h equipmen kindly p o ided by he A elie . 4.3.1 The Pe o mance The hi d public expe imen o Akson as a esea ch con ex happened as a pe o mance. I was done wi h he help o he musician F ancisca Gonçal es (also membe o he Open ield collec i e) and ook place in he labo a o y. The na u e o his expe imen was based on a mo e con en ional lap op pe o mance (see sec ion 2.1.5). The wo pe o me s played wi hou any p e ious ehea sal. The equipmen was kindly p o ided by he A elie , apa om he pe o me s pe sonal ma e- ial (lap ops and audio de ices). The sys em was composed o one compu e o each pe o me , a sound sys em and a p ojec o . This expe imen ook 10 minu es and s a ed a 10:15 in he mo ning wi h 6 esea che s as public. One compu e was a MacBook P o (la e 2013) wi h a p ocesso o 2.7GHZ In el Co e i5, 13.3-inch (2560x1600), RAM memo y o 8GB 1867MHZ DDR3 and a G aphics ca d In el I is G aphics 6100/1536MB unning OSX Moja e 10.14. The second compu e was o model MacBook P o (la e 2015) wi h a p ocesso o 2.2GHZ In el Co e i7, RAM memo y o 16GB 1600MHZ DDR3 and a G aphics Ca d In el I is P o 1536MB unning OSX Sie a 10.12.6. The i s compu e was connec ed o a Sca le 2i4 (2nd Gen), an audio in e ace o 4 way ou s up o 192KHZ wi h 24- Bi . The second compu e was connec ed o a Roland Quad-Cap u e, an audio in e ace wi h 2x2 analog ou s up o 192KHZ a 24-Bi . These wo in e aces connec ed o he main mixe o model Al o L-12, a wel e channel legacy mixe wi h in eg a ed DSP o 24-bi ha connec ed o 2 Tannoy Re eal 502. The second compu e connec ed by HDMI o a p ojec o EPSON EB-S05 o 1080p (1920x1080) esolu ion wi h 3100 ANSI Lumens. Bo h he equipmen con igu a ion and he pe o mance in e ac ion mode we e esol ed be o e i happened. I was de ined ha he wo pe o me s would ha e wo dis inc unc ions du ing hei du a ion, one ocusing mainly on he syn hesize (Lead) and ano he on he il e -noise (Back- g ound). In his way hey could explo e a ious esolu ions o he wo k wi h p ede ined ules. This is he hi d and las mode o in e ac ion o be in eg a ed in o Akson. As explained in chap e 3.7.3, i di ides he cen al con ol (dashboa d and bidi ec ional connec ions) in o ou pa s so ha i can be di ided be ween pe o me s. The es s we e done in he WI-FI ne wo k one hou be o e he pe o mance and we e sa is ac o y enough so we didn’ ha e o se up ou own ne wo k o speed imp o emen . We an speed es s ha demons a ed a la ency o <13MS and 8O icial websi e Open ield h p://open ield-c ea i elab.com 4.3 Expe imen a Open ield 61 <4MS o ji e . The download speed was o <45.00MBPS and upload speed o <53.00MBPS. We also had he po 5000 opened o se e Akson. 4.3.2 The Pa icipan s Pa icipan s in ol ed in his pe o mance as a public had all expe ience wi h pe o ming a s and we e in ol ed wi h he academy as esea che s. They did no know wha me hod o in e ac- ion he wo pe o me s ook, no was i explained how he pe o mance would ake place, hey we e only old ha i hey wished hey could connec o ou sys em ia he Akson o icial link, p ojec ing a igu e on he wall wi h he add ess ( ig. 6.5). When he pa icipan s sha ed hei iews ega ding he expe imen he e was a clea ag eemen on he a ious opics men ioned. •Pa icipan s liked o know mo e abou Akson and how i wo ks as a sys em, no jus s aying o he con en i c ea ed. •They wan ed o know exac ly wha he pe o me s we e doing, sepa a ely. •They ag eed ha , gi en he aes he ics o he sound and he g aphic load, he pe o mance should ake mo e han 10 minu es. Compa ed wi h he o he expe imen s, he audio- isual ela ions we e mo e accu a e. The e was clea g aphical eedback when some kind o sound was gene a ed. Howe e , his kind o pe o mance always b ings a g ea abs ac ion o embodied pe sonal in e ac ion. When he pe o mance was o e , imp essions we e exchanged on how Akson wo ked, also o ma ch he pa icipan s i s o e all imp ession. The e we e no ex e nal connec ions o he wo pe o me s 62 Akson Public Expe imen s Figu e 4.3: A pho og aph o he pe o mance a Open ield. Cou esy o Ma celo Sousa, esea che and public. 4.4 Pe o mance in xCoAx 2019 As he nex public alida ion, a he beginning o July, an al eady accep ed ne wo ked pe o - mance will ake place on xCoAx92019. An in e na ional con e ence on A and Compu a ion in Milan, I aly. He e, alongside musician Gilbe o Be na des, will p esen wo di e en in e ac ion models based on he a chi ec u es o Akson. We will explo e The Conduc o Model, he pa adigm o he Symphony O ches a whe e he conduc o s a e he mas e con olle s shaping he dynamics and di ec ing ime lows. And he F ee Imp o isa ion me hod, based on he ee jazz mo emen o he six ies ha p oduced pe o mances ha we e highly in e ac i e, spon aneous, exp essi e and unp edic able (Winkle , 1998). Dynamic con ol will be done in he wo modes o in e ac ion ha a e es ablished wi h he audience o he enue o wi h anyone who can join Akson globally. I is p esen ed in he sec ion 5.2 a u u e de elopmen o a midi lea ning sys em using Web- MIDI so he a is s can connec di e en de ices o Akson and map he con ols o he GUI. This 9O icial xCoAx websi e h p://xcoax.o g [accessed 27/06/2019] 4.5 Conclusions 63 can be achie ed using public open-sou ce API’s. Akson al eady accep s MIDI, bu is speci ic o he au ho s pe iphe als and he e is no way o in oducing di e en ones wi hou ha d coding. 4.5 Conclusions In his chap e he public expe imen s o Akson we e p esen ed. As a collabo a i e digi al in- e ace he h ee modes o in e ac ion (see sec ion 3.7) we e explo ed wi h i ’s p ocess documen ed. These expe imen s, abo e all, se ed o de elop he inal e sion o Akson p esen ed in his documen . Much o he eedback ob ained was analyzed as e lec ion ma e ial. The g ea decisions made h oughou Akson’s de elopmen ha e always been based on his p ocess as a c ea i e loop (some hing ha can also be seen on he o icial eposi o y). An example o ha a e he in e ac ion modes chosen. No only based on e e ences, hey we e he esul o common opinions ega ding in e pe sonal collabo a ion a e he expe imen s. The p oposed in e ac ion me hods we e all publicly es ed wi hin he ime p oposed as e- sea ch (one school yea ), and a e echnically inished allowing hem o be es ed and demons a ed o he wo ld. 64 Akson Public Expe imen s Chap e 5 Conclusion In his chap e we p esen he gene al conclusions o he esea ch. A lis o achie ed objec i es is made and a se o u u e wo k is p oposed. The chap e s p esen ed be ween he in oduc ion and he p esen one a e he desc ip ion o he pa h his esea ch has aken and he unde lying echnologies used. A wo k done in he ligh o he BMA g oup in AV collabo a i e web in e aces. We app oach he p oposed p oblem and esea ch ques ion in a p ac ical way p esen ing a in- ished, open-ended, ee and c oss-pla o m solu ion o he wo ld. The Akson en i onmen . He e, also a ge o se e al u u e wo k p oposals. This esea ch had a g ea deal o so wa e de elopmen which su e ed some ad e si ies. Akson is a sys em wi h se e al ea u es in di e en ields and we ied o ma ch he same le el o complexi y, aes he ics and unc ionali y in all o hem. All hese issues we e esol ed by keeping a eco d in ime and wi h he help om a is ic expe imen s. The objec i es o which we ha e p oposed he esea ch we e ca ied ou , and we main ained a e y ca e ul ela ionship wi h he possible con ibu ions so ha he esul s could be achie ed in he p oposed ime scale. 5.1 Con ibu ions The e a e se e al con ibu ions in his esea ch ha can gene a e conclusions. •A web-based collabo a i e AV en i onmen ha can se e as s uc u al ab ic in collabo a- i e a is ic pe o mance and/o in e ac i e ins alla ions. •A public cou se in so wa e de elopmen ha can se e as map o u u e esea ch. Bo h on -end and back-end. •A collabo a i e digi al space ha o e s h ee di e en modes o in e ac ion be ween pe - o me s/a is s/audiences, publicly es ed and ee o use. •Duly de ailed public expe imen s ha can se e as a is ic o scien i ic inspi a ion/explo a ion in u u e p ojec s. 65 66 Conclusion •The pe o mances and he ins alla ion done inc eased in e es and a achmen o he local a is ic communi y. •A is ic expe imen s ha ma e ialize a cus om aes he ic pa e n, chosen by he au ho s. •An AV en i onmen ha can ex end he a is ic ges u e h ough he cloud wi h po en ially la ge dis ances ac oss he globe. •An AV gene a ion s eam ha can be used as con en dissemina o by la ge numbe s o de ices as esonan and isual machines in space. •An online space o ee access ha can be used by anyone in he wo ld who has access o he in e ne . The a ious con ibu ions lis ed can always ha e ami ica ions be ween hem. They can poin o a ious ypes o u u e wo k and in a ious a eas. In he nex sec ion we will lis se e al possibili ies as u u e esea ch. 5.2 Fu u e Wo k A e he wo k in oduced, documen ed and comple ed come up some op ions ha can be aken as u u e in es iga i e wo k. We, as a is s and esea che s, ha e come o a ious conclusions ega ding he pa h ha his p ojec can ake. Based on e lec ions aken in ela ion o he wo k done, we p opose ami ica ions in a ious a eas. Some mo e linked o scien i ic de elopmen and o he s mo e ela ed o a is ic expe imen a ion. We belie e ha Akson has he po en ial o be in eg a ed in o mo e domains apa om wha his disse a ion explo es and ha can be ex ended o o he a eas. He e we will lis he a ious a eas ha we conside ele an wo k. •A good explo a o y con inua ion o his esea ch would be he s udy o mo e modes o in e ac ion and hei in eg a ion in o Akson. Winkle ’s p oposed me hods (1998) can be one g ea e e ence. •A b anch o wo k mo e ocused on he AV ela ionship and wi h a mo e de ailed heo e ical de elopmen in his o ical e en s. Fo example in colo o gans and syne gies be ween sound and image also ound in he wo k o Rod igo Ca alho (2013). •A de elopmen o wo k no only explo ing he in e ne and i s in as uc u e as medium bu also as a in i sel . The undamen al s uc u e ha de ines Ne A . A g ea e e ence on ha can be he book The A Happens He e: Ne A An hology (Conno , 2019). •The de elopmen o Akson as an au oma ic con en gene a ion sys em o explo a ion o la ge-scale sound sculp u es aking ad an age o he in e ne as a medium. 5.2 Fu u e Wo k 67 •Syn hesize no e gene a ion h ough s ochas ic modelling such as a iable ma ko models (VMM’s), allowing he abili y o c ea e musical ph ases be ween people wi hou de e min- is ic induc ion (sec ion 3.6.3.2). •The de elopmen o a pe iphe al con ol lea ning sys em using WebMIDI (Wyse & Sub a- manian, 2013). •The possibili y o s eaming sound and g aphics h ough WebRTC echnology (Johns on & Bu ne , 2012). •Explo a ions in a pe o ma i e con ex in which he Akson is used as an ins umen in con- junc ion wi h o he classical ins umen s (e.g. iolin, cello). •The s udy o dis ibu ed elec oacous ic ep oduc ion sys ems wi h complex a ays o ma- chines (e.g. compu e and/o mobile phones). •We unde line ha one o he g ea u u e possibili ies o Akson may in ol e i s in eg a ion in o a is ic p ac ice ou side he academic and scien i ic con ex . •The use o Akson as a sys em o analysis and composi ion is also some hing we conside impo an . All o hese u u e wo k possibili ies ci ed he e we e speci ically designed o he Akson en- i onmen . These a e p ojec s ha , based on ou e lec ion a e he de elopmen o his esea ch, we conside po en ially in e es ing o u u e wo k. These may gi e ise o a ied con ibu ions, in di e en ways, in di e en a eas. 68 Conclusion 6.6 Akson’s Came a Sc een sho s 75 Figu e 6.7: Sc een sho s o Akson’s g aphical sys em, aken om he i ual came a. These can be ound in h p://www.luisa andas.o g/akson.h ml. 76 Appendix Chap e 7 A achmen 7.1 Akson signage in gn a ion Figu e 7.1: Pho og aph o signage a anged a he en ance o he Akson ins alla ion in he open day. 77 78 A achmen 7.2 A icles abou Akson by gn a ion Figu e 7.2: A icle abou Akson ound on gn a ion lye . Figu e 7.3: A icle abou Akson ound on gn a ion websi e. 7.3 Ve ical lye o open day 79 7.3 Ve ical lye o open day gn a ion www.gn a ion.p Sex o ani e sá io celeb ado com um conjun o de a i idades g a ui as pa a odas as idades. á ios locais · 10:00-04:00 g a ui o (sujei o à lo ação dos espaços) música · ins alação · a i idades educa i as 27 ab il 2019 www.s udiodob a.com JOÃO PAIS FILIPE ESCOLA DO ROCK GRAHAM DUNNING SENSIBLE SOCCERS QUADRA AMMAR 808 ALGOBABEZ DJ K-SETS WAVELENGHTS OF LIGHT POR MAOTIK TOTAL ECLIPSE POR DIANA POLICARPO MACHINE MESSAGE POR BERRU ESTAÇÃO DE EXPERIMENTAÇÃO: AKSON MINI MAPA SONORO MÚSICA ATIVIDADES EDUCATIVAS INSTALAÇÃO Figu e 7.4: Cou esy o Ilídio Ma ques, gn a ion Communica ion manage . A high- esolu ion e ical pos e image ha was ound ac oss he ci y o B aga as publici y. 80 A achmen 7.4 Ho izon al lye o open day Figu e 7.5: Cou esy o Ilídio Ma ques, gn a ion Communica ion manage . A high- esolu ion ho izon al pos e image o he e en . Was also ound ac oss he ci y o B aga as publici y. 7.5 Pic u e o Akson’s Ins alla ion 81 7.5 Pic u e o Akson’s Ins alla ion A pho og aph kindly p o ided by he gn a ion o ganiza ion. I was aken on Ap il 27 and shows pa o he explo a o y s a ion. Cou esy o Ilídio Ma ques, gn a ion communica ion man- age . Figu e 7.6: Cou esy o Ilídio Ma ques, gn a ion Communica ion manage . A pho og aph o use s in e ac ing wi h Akson du ing he open day. 82 A achmen 7.6 Pic u e o Open ield pe o mance Figu e 7.7: Cou esy o Ma celo Sousa, esea che and public a Open ield expe imen . A pho og aph o he explana ion gi en a e he pe o mance, main aining an in e ac ion wi h Akson using a mobile phone. 7.7 A icle abou Akson on Rimas e Ba idas 83 7.7 A icle abou Akson on Rimas e Ba idas Figu e 7.8: A icle abou Akson ound on h ps://www. imaseba idas.p /gn a ion-celeb a-sex o- ani e sa io-com-open-day- echeado-de-musica-e-ac i idades/. A pos by ReB Team in Feb ua y 22 h. 84 A achmen 7.8 Pho og aph o gn a ion’s lineup Figu e 7.9: A pho og aph o a gn a ion lye con aining he ac i i ies o open day. Re e ences 91 (Pennycook, 1985) Pennycook, B uce W. "Compu e -music in e aces: a su ey." ACM Compu - ing Su eys (CSUR) 17.2 (1985): 267-289. (Pucke e, 2007) Pucke e, Mille . The heo y and echnique o elec onic music. Wo ld Scien i ic Publishing Company, 2007. (Pulkka, 2005) Pulkka, A. Spa ial Culling o In e pe sonal Communica ion wi hin La ge-Scale Mul iUse Vi ual En i onmen s - Mas e o Science Disse a ion. 1995. Human In e ace Technology Labo a o y - Uni e si y o Washing on. (Pu es & Williams, 2004) Pu es, D. & Williams, M. (2004). Neu oscience 3 d Edi ion. Sinaue Associa es Inc (July 1, 2004) (Qua an a, 2013) Qua an a, Domenico. Beyond New Media A . Lulu P ess, Inc, 2013. (Rai, 2013) Rai, Rohi . Socke . IO Real- ime Web Applica ion De elopmen . Pack Publishing L d, 2013. (Ramsay & Pa adiso, 2015) Ramsay, Da id B., and Joseph A. Pa adiso. "G ouploop: a collabo a- i e, ne wo k-enabled audio eedback ins umen ." NIME. 2015. (Reilly, 2013) Reilly, Ka a. Thea e, pe o mance and analogue echnology: his o ical in e aces and in e mediali ies. Sp inge , 2013. (Ribas, 2013) Ribas, Luísa. "Audio isual Dynamics: An app oach o Sound and Image Rela ions in Digi al In e ac i e Sys ems." xCoAx : P oceedings o he i s con e ence on Compu a ion, Communica ion, Aes he ics and X, Be gamo, I aly 2013 (Robe s, Wake ield & W igh , 2013) Robe s, Cha les, G aham Wake ield, and Ma hew W igh . "The Web B owse As Syn hesize And In e ace." NIME. 2013. (Ros e al., 2009) Ros , Randi J., e al. OpenGL shading language. Pea son Educa ion, 2009. (Sá, 2010) Sá, C is ina Fe nandes Al es de. "O que é o in e ace? Da en i icação á iden i icação do in e ace enquan o complexo mediado ." (2010). (Sá, 2016) Sa, Ad iana de Almeida Po ela Viana. A Pe cep ual App oach o Audio-Visual Ins u- men Design, Composi ion and Pe o mance. Diss. Goldsmi hs, Uni e si y o London, 2016. (Sca one, 2002) Sca one, G. P. (2002). R Audio: A C oss-Pla o m C++ Class o Real ime Audio Inpu /Ou pu . In P oceedings o he In e na ional Compu e Music Con e ence, Go bo g, Sweden, pp. 196-199. ICMC. (Scien i ic Ame ican, 1891) Scien i ic Ame ican (1891a). A long dis ance elephone conce . [accessed 27/06/2019] (Schoole & Touch, 1993) Schoole , E e M., and J. Touch. "Dis ibu ed music: a o ay in o ne wo ked pe o mance." In e na ional Ne wo k Music Fes i al, San a Monica, CA (1993). (Senna & A oza 2017) Senna, Guille mo, and Juan Na a A oza. "Da ia: A New F amewo k o Composing, Rehea sing and Pe o ming Mixed Media Music." 92 Re e ences (Shannon, 1948) Shannon, Claude Elwood. "A ma hema ical heo y o communica ion." Bell sys em echnical jou nal 27.3 (1948): 379-423. (Smi h, 1998) Smi h, Owen F. Fluxus: The his o y o an a i ude. San Diego: San Diego S a e Uni e si y P ess, 1998. (Smi h, 2009) Smi h, Hazel, ed. P ac ice-led esea ch, esea ch-led p ac ice in he c ea i e a s. Edinbu gh Uni e si y P ess, 2009. (Smus, 2013) Smus, Bo is. Web Audio API: Ad anced Sound o Games and In e ac i e Apps. " O’Reilly Media, Inc.", 2013. (So ensen, 2005) So ensen, And ew. "Imp omp u: An in e ac i e p og amming en i onmen o composi ion and pe o mance." P oceedings o he Aus alasian Compu e Music Con e ence 2009. 2005. (S allab ass, 2003) S allab ass, Julian. "The aes he ics o ne . a ." Qui Pa le 14.1 (2003): 49-72. (S ela c, 1995) S ela c In e iewed by Paolo A zo i and Ki k Wool o d o Academy o Media A s, Cologne, Ge many, 9/6/1995. [accessed 27/06/2019] (S ela c, 2011) S ela c. (2011). A , Design, Fu u e o Humani y - In e iewed by Adam Fo d [accessed 27/06/2019] (S ela c, 2014) S ela c. (2014). Zombies, Cybo gs & Chime as: A Talk by Pe o mance A is , P o S ela c - Cu in Uni e si y [accessed 27/06/2019] (Teixei a, 2012) Teixei a, Ped o. P o essional Node. js: Building Ja asc ip based scalable so - wa e. John Wiley Sons, 2012. (Tilko & Vinoski 2010) Tilko , S e an, and S e e Vinoski. "Node. js: Using Ja aSc ip o build high-pe o mance ne wo k p og ams." IEEE In e ne Compu ing 14.6 (2010): 80-83. (Thiebau e al 2008) Thiebau , Jean-Bap is e, Pa ick GT Healey, and Nick B yan-Kinns. "D aw- ing Elec oacous ic Music." ICMC. 2008. (Thiemann, 2005) Thiemann, Pe e . "A ype sa e DOM api." In e na ional Wo kshop on Da abase P og amming Languages. Sp inge , Be lin, Heidelbe g, 2005. (Wang & Cook 2003) Wang, Ge, and Pe y R. Cook. "ChucK: A Concu en , On- he-Fly, Audio P og amming Language." ICMC. 2003. (Winkle , 1998) Winkle , Todd. Composing in e ac i e music: echniques and ideas using Max. MIT p ess, 2001. (Wi e n, Su e & Rajagopalan, 2016) Wi e n, E ik, Philippe Su e , and Sh i am Rajagopalan. "A look a he dynamics o he Ja aSc ip package ecosys em." 2016 IEEE/ACM 13 h Wo king Con e ence on Mining So wa e Reposi o ies (MSR). IEEE, 2016. (Wyse & Sub amanian, 2013) Wyse, Lonce, and S ikuma Sub amanian. "The iabili y o he web b owse as a compu e music pla o m." Compu e Music Jou nal 37.4 (2013): 10-23. Re e ences 93 (Zicca elli, 2002) Zica elli, Da id. "How I lea ned o lo e a p og am ha does no hing." Compu e Music Jou nal 26.4 (2002): 44-51.