Syntactic Analysis of Digital Artworks

Syntactic Analysis of Digital Artworks Dr. Rocio Garcia-Robles ASTERISM Research Group University of Seville [email protected] Dr. Aurea Muñoz-del-Amo Graphics and Digital Creation Research group University of Seville [email protected] ABSTRACT Digital artworks have specific features to be analyzed compared to others. In current paper, a methodology for analyzing digital artworks is presented. Firstly, some features proposed by relevant academics are reviewed. Secondly, an integrated methodology is suggested. Finally, four digital installations are further explored based on the afore-mentioned methodology. CCS CONCEPTS •Arts and Humanities;•Media Arts;•Fine Arts; KEYWORDS Digital art, Digital aesthetics, Interactive installation, Interdisciplinar ACM Reference Format: Dr. Rocio garcia-robles and Dr. Aurea muñoz-del-amo. 2021. Syntactic Analysis of Digital Artworks. In 10th International Conference on Digital and Interactive Arts (ARTECH 2021), October 13–15, 2021, Aveiro, Portugal, Portugal. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3483529.3483670 1 DIGITAL ARTWORK RELEVANT CHARACTERISTICS On one hand, when we think about a digital media, a large part of digital artistic production takes the ’installation’ format. The concept of ‘installation’ in the artistic context is complex and diffuse, at the same time, since it has very different meanings. Among others, there is an interesting definition given by an expert on the field, Claudia Giannetti. She proposes certain features that tend to characterize, to a greater or lesser extent, the different installation artistic practices. According to her criteria [ 1 ]: Installation, in the field of art, is the method of generating new forms of expression as processes that seek the following objectives: •Interdisciplinary and congenital hybrid practice. •Break with the closed shape of the object. •Emphasis on the ideas of site-specificity and intervention. •Research on the relationship between: • Context (space, architecture, environment, surroundings, ...). •Time (duration). •Component parts of the work. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. ARTECH 2021, October 13–15, 2021, Aveiro, Portugal, Portugal • Multiplicity or interrelation of elements or materials (expanded collage, expanded assemblage). •Concern about the role of the receiver. • Protagonist of the notion of process (vs. concept of a single, permanent and finished work). •Understanding of the work as a social and public space. • Enhancement of the multi-sensorial dimensions of the works. Therefore, on one hand, "they are works that try to overcome the limits of the object, insofar as they investigate the integration and expansion of the work in space, or transform the space itself into a work of art" [1]. On the other hand, it is noteworthy how this definition of installation given by Giannetti is not restricted to digital media, but is applicable to previous art productions, and to both: analog and digital environments. In this sense, it would be convenient to review the historical predecessors of the installation (such as: ’assembly’ (assemblage), ’environment’ (environment), ’environment’ (ambient), etc.). This installation concept evolution could be summarized as follows: “The idea that one could provide a history of the facility is perhaps curious in view of its relative youth. However, it is possible to try to write such a history, although it must be more than a simple recounting of those occasions during the 20th century in which similar forms have made their appearance. Of course, a list can easily be made including: Futurism; the Cubist collage; Duchamp’s ’ready-mades’; the Dada and the constructions of Schwitters and Baader; El Lissitzky and his Constructivist approaches to space; again Duchamp and his contributions to the Surrealist exhibitions of 1938 and 1942; Fontana’s ’spatialism’; the ’assemblage’; the ’Happenings’; Klein and Manzini; the Pop paintings by Kienholz, Oldenburg, Segal and Thek; ‘Fluxus’, Minimalism; environmental art (‘Land Art’); ‘Arte Povera’; ’Process Art’; Conceptualism; . . . but more or less this is nothing but a history of modern art. What is needed above all is the trace of certain ideas of this story, in particular the notion that space and time (that is, in relation to actual duration rather than the abstract notion of time) in themselves constitute the raw material for art. We must also point out the trend, observable throughout modernism, of art merging with life” [2]. Rocio Garcia-Robles and Aurea Muñoz-Del Amo Therefore, the reflections by Oxley et al. allow us to verify how installation practices have been present throughout the 20th century, although they have received various names throughout the history of art. On the other hand, there are multiple possible ontological classifications that might be useful to classify a digital work, as well as to define the degree and type of interactivity in digital systems. All these parameters allow broadening the criteria that can be used when analyzing a digital artistic production. Thus, from a medial point of view, a possible classification of digital media is the next one [ 3 ]: digital image, digital sculpture, digital installation and virtual reality, performing arts (performance, musical and sound art), digital animation and video, software art, game art, database art, net art. However, this classification is still somewhat rigid, given the intermedial that usually characterizes contemporary digital artistic production. But, even so, this taxonomy is useful when evaluating a production contextually, since each media allows the development of a type of language and carries a certain aesthetic. Anyway, we are currently in a post-medial phase in the history of contemporary art, so there are other intrinsic features more relevant when analyzing digital artworks, such as ‘interactivity’. Regarding the degree and type of interactivity in digital systems, the following experts’ contributions are especially relevant: • [ 1 ] Among the models of interactive systems and according to the degree of human-machine interactivity Giannetti suggests three types of interactivities mediated by images, representations, sounds, robotic systems, etc: • Mediator system: punctual, simple, usually binary reaction to a given program. • Reactive system: interference in a program through the structuring of its development within the scope of given possibilities. It is a selection interactivity, which implies the possibility of multidirectional access to audiovisual information for the execution of operations predetermined by the system, and therefore limited to these. • Interactive system: independent structuring of a program that occurs when a receiver can also act as a transmitter. It is a content interactivity, in which the interactor has a greater degree of possibility to intervene and manipulate audiovisual or other information (such as robotics) or, in more complex systems, generate new information. •[1][4] Edmond Couchot suggests a further differentiation: • External interaction consists of the human-machine interface, as well as the forms offered by the environment, whose data is processed by computer through different interfaces. • The internal interaction corresponds, on the contrary, to the communicative behavior between the virtual objects themselves (the constituent elements of Virtual Reality, Artificial Life, ...), which can generate behavioral models for the animation of the so-called synthesis actors. • [ 1 ][5][8]From the point of view of technical devices, Roger Malina suggests a list of five essential criteria or characteristics of interactive media has been drawn up: • The possibility of carrying out an interaction that changes the internal status of the computer. • The viability of the computer to integrate learning possibilities, so that the internal status of the computer can be changed when the interaction occurs. • The possibility of connecting several physically remote computers through telecommunication networks. • The ability to assimilate and process signals that are not accessible to the human senses in various ways, and to connect these signals in a synesthetic way. • The ability to store a large amount of information that is easily accessible. To these we should add the feasibility of self-generating original meaningful information (not preprogrammed, as in Artificial intelligence systems) and the ability to simulate behaviors as if they were living organisms (such as intelligent agents or Artificial Life beings). • [ 1 ][6] In this technical context it is important to also distinguish between the different types of non-trivial machines. Heinz von Foerster determines the differentiation between trivial and non-trivial machines according to the field in which they operate. Trivials are describable in their causal mode and, as predictable machines, are only possible in nonphysical realms, such as mathematics; the machines that exist in physical space are always non-trivial, since it is subject to entropic processes. It can be differentiated between two types of non-trivial machines: those that try to approximate their behavior to trivial machines; and those that without dissimulation behave in a non-trivial way. The former are functional machines that perform specific tasks; the latter are machines potentially conducive to interactivity. • [ 1 ][7] From another perspective that has behavior and consciousness as a point of reference, Peter Weibel distinguishes between three levels of interaction: • Synesthetic interaction, which consists of the interaction between materials and elements, such as image and sound, color and music. • Synergistic interaction, which occurs between energy states, as in works that react to change in the environment; and • Communicative interaction or kinetic interaction between people and between people and objects. In any case, the environment or context of the work is decisive for its implementation. The introduction of the context element in the interaction process means the insertion of its condition as an influencing factor in the communication process. Consequently, the limit or breadth of the interaction is proportional to the "informational" limit or breadth of the interactive context itself. 2 SYNTACTIC ANALYSIS OF DIGITAL ARTWORKS According to previous discussion, a possible integrated methodology for analyzing digital artifacts could be as follows: Of course, in addition to the taxonomic analysis that has been exposed that alludes to the characteristics of digital artistic production, the conceptual analysis is especially relevant. This document does not explain how to analyze this dimension, although it should be considered, because it constitutes an important part of the discourse that the author (and the viewer) confers on the work. Syntactic Analysis of Digital Artworks Table 1: Integrated methodology Integrated methodology for analyzing digital artifacts Giannetti - Mediator system - Reactive system - Interactive system Couchot - External interaction - Internal interaction Malina a) The possibility of carrying out an interaction that changes the internal status of the computer b) The viability of the computer to integrate learning possibilities, so that the internal status of the computer can be changed when the interaction occurs c) The possibility of connecting several physically remote computers through telecommunication networks d) The ability to assimilate and process signals that are not accessible to the human senses in various ways, and to connect these signals synesthetically e) The ability to store a large amount of information that is easily accessible. To these we should add the feasibility of self-generating original meaningful information (not pre-programmed, as in Artificial intelligence systems) and the ability to simulate behaviors as if they were living organisms (such as intelligent agents or Artificial Life beings) Heinz von Foerster - Trivial machine - Non-trivial machine Peter Weibel - Synesthetic interaction - Synergistic interaction - Communicative interaction or kinetic interaction Therefore, in addition to the afore-mentioned parameters (medial, types and degrees of interaction), it is advisable to approach the conceptual analysis in order to have a global and at the same time detailed vision of the work. 3 ILLUSTRATING SYNTACTIC ANALYSIS OF FOUR DIGITAL ARTWORKS In order to illustrate the proposed analysis, we have selected four artworks: “The Book of Air”, by Doris K. Vila; “Mountain, one hundred and fourteen thousand polygons”, by Joanie Lemercier; “Someone” by Lauren McCarthy and "Uncanny Mirror", by Mario Klingemann: “The Book of Air” by Doris k. Vila (1993). It is an interactive installation composed of five large holograms; as the viewers walk across the room, their shadows activate computer-controlled sound, video and lights, altering the holographic images as well. Let’s read the description of the dynamics of the work by the author: “It was a responsive environment that incorporates five large format holograms (60x110cm). Five banks of 16 computer-controlled lights, interactive digital video, and sound reconstructed the hologram. When viewers walked on an interactive floor filled with sensors, their infrared shadows selectively actuated the lights. These lights showed various colors and images. Paula Chagas, a Brazilian composer, composed the sound. There were two video projections illuminating two of the holograms. One projection came from an infrared video camera mounted over the heads of the spectators, as they walked on the interactive track for the first time. In this way, the spectators who watched saw themselves in real time as they would be seen from the sky, in an allusion to the arbitrariness of the Cartesian point of view. The second projection was interactive digital video. Walking on certain parts of the ground, the spectators activated a selection of 20 video sequences, 15 processed images and 20 lines of text. The entire room became an instrument played by the spectators. It took on life when they entered the 100 m2 space. The viewers were more than a part of the work, they determined what it looked like and how it sounded and what they saw. The non-linear narratives of holograms refer to the trans-cultural beliefs and stories that exist in the air. When we inhale them, we write them deep into our bodies."[9] “Mountain, one hundred and fourteen thousand polygons”, by Joanie Lemercier (2016 – 2018). This visual installation combines digital print on paper and a 5 minute video projection. According to the explanation that accompanies the display of this piece, exhibited between February and July 2021 at the Fundación Telefónica, in Madrid: “This piece depicts a large valley surrounded by mountain peaks. In fact, what we really see is just a grid mesh distorted by an algorithm. The procedural landscape questions the connection between nature and code: what if our reality could be recreated and simulated with mathematical functions? A layer of projected light creates an illusion of depth and blurs our sense of distance. A cycle of day, night, and different seasons bring life to the wallpaper landscape, distorting our perception of time and space. Conceived in July 2016 for a group show in Brussels, Lemercier got invited to present Montagne, cent quatorze mille polygones in Tokyo in February 2018. The piece was exhibited in a skyscraper on the same floor as the Tokyo Observatory, which offers a 360-degree view of the city with Mount Fuji 100 kilometres away. Once on site, Lemercier got inspired by the harsh Japanese winter and decided to add the snowstorm as part of the piece. Some of his artworks would evolve in time, not as pure in situ creations but as an attempt to reflect a specific moment, mood or atmosphere.”[10] Rocio Garcia-Robles and Aurea Muñoz-Del Amo "Someone", by Lauren McCarthy (2019). The artwork is a human version of Amazon Alexa. During two months, four participants’ homes around the US were installed with custom-designed smart devices (including cameras, microphones, lights, and other appliances). At the same time, at the 205 Hudson Gallery in New York, a command center was installed, so visitors could observe the four houses via laptops, watch inside and remotely control their devices. Visitors would hear home occupants call out for “Someone” so the visitors could step in as their home automation assistant and respond to their needs. “Uncanny Mirror”, by Mario Klingemann (2018). The artist employs neuronal networks and GAN (Generative Adversarial Networks) algorithms to experiment with creating new images. He trains his AI system to interpret visitor’s faces as data and simultaneously generate new portraits made of everybody else who look at the mirror before. So viewers find themselves on the screen just as the AI has seen them. Considering the definition of ‘installation’ reviewed in section 1, those four works could be considered as interdisciplinary practices and multisensory. In all of them, there is a rupture of the closed form of the work, firstly fostered by the underlying concept of process (the work articulates non-linear narratives, that responds to a real-time processing of different kinds of information, such as spatial location, speech recognition, etc.). Secondly, by the concern for the role played by the receiver (viewers who determine the audiovisual aspect and response depending on their interaction with the work). Next, the syntactic analysis is applied as meta-information about those art productions, as explained in table 2 As shown in the syntactic analysis (table 2), the four selected artworks can be surveyed and compared in a methodical way. This could be used as a first approach before regarding with the further semantical analysis. Therefore, the proposed procedure may help to systematize the digital artworks analysis. However, it would be interesting to test that syntactic methodology with a greater number of works, in order to validate the usefulness of the proposed include items. 4 CONCLUSIONS The complexity of digital artworks makes the analysis process even more difficult than in more traditional art productions. In this framework, a syntactic methodology analysis has been suggested for systematic review. It is an integrated methodology that could be enlarged. There are other interesting experts’ taxonomies (e.g. [ 10 ]), although only some of them have been selected and analyzed in the current paper. Moreover, it supports comparative analysis among several artworks. Of course, the suggested methodology should be complemented by a semantic approach, but it is out of the bounds of the current paper to accomplish both, as well as other perspectives (postmodern features, globalization, etc.). REFERENCES [1] [Claudia Giannetti, 2002. Estética Digital: Sintopía del arte, la ciencia y la tecnología. Associació de Cultura Contemporània L’Angelot, Barcelona. [2] Nicola Oxley, Nicolas de Oliveira and Michael Petra, 1998. Installation Art. Thames and Hudson, London. [3] Bruce Wands, 2006. Art of the Digital Age, Thames and Hudson, London. [4] Edmond Couchot, 1995. Synthèse et simulation. In Poissant, Louise (ed), Esthétique des Arts Médiatiques, Vol II, 275-290. [5] Roger F. Malina, 1990. Der Beginn einer neuen Kunstform. In Prix Ars Electronica. Internationaler Wett-bewerb für Computerkünste. Brücknerhaus Linz, Linz 152- 160. [6] Peter Krieg, 1993.Versuch über Interaktion und Medien. In Kunstiche Spiele. Klaus Boer Verlag, Múnich, 180-181. [7] Peter Weibel, Der Ausstieg aus der Kunst ais höchste Form der Kunst. In Kustforum International. Ruppichteroth, B. 103 (Sept/Oct, 1989), 225-231. [8] Roger Malina, 1995. La recontre de l’art et de la science. In Esthétique des Arts Médiatiques, Vol II, 39-48. [9] Doris K. Vila, 1994. Holo-dynamics: Linking holography to interactive electronic media. In Proceedings of 5th International Symposium on Display Holography SPIE. Vol. 2333. https://doi.org/10.1117/12.201898 [10] Fundación BIACS, 2008. Youniverse: Sevilla, Granada / [3 ª Bienal Internacional de Arte Contemporáneo de Sevilla], BIACS 3, Sevilla. [11] Katja Kwastek 2013. Aesthetics of Interaction in Digital Art. The MIT Press, Massachusetts. Syntactic Analysis of Digital Artworks Table 2: Analysis of four digital artifacts Integrated methodology for analyzing digital artifacts The Book of Air Mountain Someone Unncany Mirror Giannetti - Mediator system X - Reactive system X X - Interactive system X Couchot - External interaction X X X - Internal interaction X Malina a) The possibility of carrying out an interaction that changes the internal status of the computer X b) The viability of the computer to integrate learning possibilities, so that the internal status of the computer can be changed when the interaction occurs X c) The possibility of connecting several physically remote computers through telecommunication networks X d) The ability to assimilate and process signals that are not accessible to the human senses in various ways, and to connect these signals synesthetically e) The ability to store a large amount of information that is easily accessible. To these we should add the feasibility of self-generating original meaningful information (not pre-programmed, as in Artificial intelligence systems) and the ability to simulate behaviors as if they were living organisms (such as intelligent agents or Artificial Life beings) X Heinz von Foerster - Trivial machine X - Non-trivial machine X X X Peter Weibel - Synesthetic interaction X - Synergistic interaction X - Communicative interaction or kinetic interaction X X