Postcards and Supasigns : Extending Integrationist Theory Through the Creation of Interactive Digital Artworks

An Interdisciplinary Journal on Humans in ICT Environments ISSN: 1795-6889 www.humantechnology.jyu.fi Volume 3 (1), February 2007, 54–67 54 POSTCARDS AND SUPASIGNS: EXTENDING INTEGRATIONIST THEORY THROUGH THE CREATION OF INTERACTIVE DIGITAL ARTWORKS Abstract: Integrationism is a post-structuralist theory of language and communication. The theory has been applied to a groundbreaking analysis of writing as a form of communication where writing is teased apart from speech and realigned with spatial configurations in general. Although it has many practical applications, this view can be extremely difficult to comprehend when expressed as a very specific form of writing, that is, as written words on paper. A solution to this problem is offered by the creative interaction design of two digital artworks, Postcard From Tunis and Postcards From Writing. The works are interactive multimedia pieces that creatively express the integrationist theory of writing and extend it into the transformations of writing that are possible in the human-computer interface. More generally, the unique rollover-based interfaces of these works both express the integrationist theory of communication and suggest that it is necessary in order to explain the creation of communicative signs that they demonstrate are possible. Keywords: writing, integrationism, human-computer interface, rollover, interactive multimedia, digital art. INTRODUCTION The focus in this article is a challenging theory of language and communication called integrationism and its analysis of writing as a form of communication. The intention of this paper is not to argue the validity of integrationist theory, nor to contextualize it within communication in general and semiotics in particular. Rather, the intention here is to briefly outline integrationism’s approach to writing and the creative expression of this approach in two interactive multimedia artworks by the author, Postcard From Tunis1 (1997) and Postcards From Writing2 (2004). As will be seen below, post-structuralist theories of language and communication can be extremely difficult to express as written words on paper. The problem is even more obvious © 2007 Sally Pryor and the Agora Center, University of Jyväskylä URN:NBN:fi:jyu-200770 Sally Pryor Faculty of Arts University of Melbourne, Australia Postcards and Supasigns 55 when applied to an analysis of writing as a form of communication. This paper describes how the two artworks employ creative interaction design to offer new ways to understand difficult theoretical ideas. The interfaces that will be described involve standard human-computer interaction elements: screens, speakers, and mouse movements. Their distinctive characteristic is the creative and highly developed involvement of rollover activities. This paper is thus located at a point of intersection between the creative arts, the humanities, and interaction design. ARTISTIC BACKGROUND Tunis is the capital of the North African country of Tunisia (Figure 1). It is well known that the artist Paul Klee was tremendously influenced by a visit to Tunisia. The “light and tonalities” he discovered transformed the way he perceived color, leading him to famously declare in Kairouan (Tunisia) that “Colour and I will always be as one. I am a painter” (Klee, 1914, as cited in Naubert-Riser, 1990, p. 49). In a more modest way, the time I spent living in Tunis in 1992 transformed my own perspective as an artist. In my case, I became powerfully aware of communication, language, and writing. As a result of the hospitality and generosity that I encountered, I began to learn to speak and read Arabic informally. This also enabled me to think about communication, language and writing in new ways, and I wanted to express this artistically. I realized that the emerging art form of interactive multimedia, combined with my ability to program its human-computer interface, offered me a way to express these experiences. In Tunis I was particularly intrigued by the concept of writing and I began to see it in a new light. This was in part because of my exposure to everyday written Arabic. It was also as a result of the richness of Tunisia’s 3,000 years of writing and the traces of ancient scripts and symbols. It seemed to me that there were strong relationships between writing and pictures and important functional differences between writing and speech. I began to search for answers to the apparently simple question, What is writing? Fortuitously, my research in Figure 1. A still from Postcard From Tunis with a star indicating the location of Tunis. Pryor 56 Tunis led me to the work of Roy Harris (Harris, 1986), a theorist of writing who offered a groundbreaking explanation of my observations. WRITING AS A FORM OF COMMUNICATION Writing about writing is a rather reflexive activity, although this is rarely mentioned. Actually, it is very difficult to think clearly about writing as a form of communication because we live in cultures where written words are usually rather important. In these environments, the traditional view is that real writing represents speech (Pryor, 2003). This view is so widespread that it is considered to be common sense and is rarely stated. For example, at a trAce3 New Media Writing seminar that I attended in 2004, the discussion shifted from writing to written words. However, the term writing includes mathematical and musical notation, so it cannot be simply equated with written words. Nevertheless, nobody at the seminar appeared to notice that the topic had changed and therefore no mention of the distinction was made. Why is it so difficult to think about writing as a form of communication without returning somehow to the idea that writing represents speech? Roy Harris (1995) has pointed out a number of reasons. First, most accounts of writing focus on the forms of writing that are linked to speech and marginalize those other forms, such as musical or mathematical writing. In fact it is very hard to think about writing at all and not be influenced by the enormous social, cultural, and political importance of alphabetic writing. However, as Harris has noted, social importance is not the same thing as theoretical importance. In other words, just because one type of writing is culturally dominant does not mean that it is theoretically privileged or that it should be used as the paradigm case. Second, alphabetic writing is usually considered to be the end product of the development of increasingly sophisticated writing systems, moving from pictures, through picture writing to word writing, and ending with the triumph of the alphabet. However, this idea that the alphabet is the most advanced of all forms of writing is a rather ethnocentric view; that is, it reflects the idea that the culture of one ethnic group, variously labeled Western or European, is superior to the cultures of other groups. Third, the simplifying assumptions that are used to teach the alphabet in the West encourage us to think that writing represents speech. However, correlational patterns between letters and sounds are not the same as representational relationships. And lastly, when we think of writing, we are powerfully influenced by the writing space of the printed book. We tend to think that this is the paradigm case of writing when it is actually a very specific form of writing. The weakness of the “traditional” theory that real writing represents speech becomes most obvious in its analysis of writing that does not represent speech. In these cases, the traditional theory necessitates a search for what these forms of writing do represent. The most widely used terms to describe this kind of writing are: • logograms; • pictograms (or pictographs); and • ideograms (or ideographs). Postcards and Supasigns 57 The first term, logogram, refers to word-writing, that is, to the representation of a word which will be voiced differently in different languages. An example is the logogram 9, which can be voiced as nine, neuf, and so on. The latter two terms, pictogram and ideogram, have a variety of definitions (assumed or explicit) that generally link them to pictures and not to speech at all. The most clear-cut definitions are that a pictogram is a simplified picture of the thing represented, and an ideogram represents an idea in general. However these distinctions quickly break down when actually applied. To study them in practice, we can try to select one of them to analyze the graphic sign at the beginning of the line in Figure 2. 99543221 Figure 2. An example of communication involving an image that can be classified in various ways: as a logogram, a pictogram, or an ideogram. Does represent • a word: telephone (in English), téléphone (in French), (in Arabic), and so on, thus classifying it as a logogram? • a simplified picture of the thing represented (a somewhat old-fashioned telephone), thus classifying it as a pictogram? • the idea of telephoning in general, thus classifying it as an ideogram? There is no satisfactory way to decide whether is a pictogram, an ideogram, or a logogram because we cannot clearly decide what it represents. However, we do understand what it means and that its proximity to the integers that follow changes the way we interpret them. We know that they do not indicate the number ninety-nine million, five hundred and forty three thousand, two hundred and twenty one, but in fact a sequence of telephone keys to press. Could we understand writing better by abandoning the idea that writing must represent something? This does not seem immediately useful; however the integer 0 provides a practical example. Zero literally represents nothing. However, the difference between the numbers 21 and 201 makes it clear that zero can certainly mean something. In the second number, zero’s proximity to the integer to its left, that is, 2, has changed the way that we interpret that integer. Clearly it is possible to understand at least some forms of writing in terms of spatial relationships. To proceed further, we need to change the focus from a view that writing must represent something to an understanding of how writing means something. We need to begin the analysis with a more general theory of language and human communication. This is the approach taken by Roy Harris, where he argues that all forms of writing involve spatial relationships. THE INTEGRATIONIST VIEW OF WRITING Harris bases his theory of writing on integrationism, a general theory of human communication in all its forms, both linguistic and non-linguistic. Integrationism challenges Pryor 58 existing terminologies and assumptions and proposes a new set of concepts to explain the difference of its approach. This theory of language and communication was originally developed by a group of linguists at the University of Oxford during the 1980s, and the discussion has continued internationally since then. Harris is one of its leading theorists. Integrationism opposes the segregationist theory that “communication systems (codes) exist autonomously as social facts, independently of their users” (Harris, personal communication, October 20, 2000). Thus according to integrationists, an act of communication cannot presuppose languages (codes) to be already present and available for use; in fact, the opposite is true (Harris, 1998a, p. 5). Language must presuppose communication itself: There can be no language without communication. In the integrationist view, human communication is designed to integrate past, present, and future activities, with time being the primary axis along which these activities are integrated. Human communication is understood, then, as the contextualized integration of activities by means of signs. More generally for integrationists is the position that although speech is culturally very important, it is not central to any theoretical understanding of how human communication takes place. From an integrationist perspective, human communication has been confused with transport, and language has been confused with the use of tools. Integrationists view human beings as language makers, not language users. There are no abstract meanings of language that exist outside of actual contexts, and language is not based on a fixed code that communication participants are sharing. Therefore, a linguistic form is not considered to be an abstract code: It is not an entity with an independent meaning and existence, like a spoon or fork sitting in a virtual cutlery drawer, waiting to be brought out and used. For integrationists, context is extremely important, not simply in the sense of a setting or backdrop. In contrast, each of us contextualizes in our own way, which is reflected in the common observation that although we may all hear the same words at a particular event, they may mean something different to each of us. Integrationism is not by any means the sole post-structuralist theory of communication and has not, of course, issued the only challenge to the ways that conventional linguistic theories explain meaning and interpretation as being contained in words or symbols. However, a discussion of the various theoretical approaches in this area is well beyond the scope of this paper and the reader is referred to Harris (1996, 1998a). Nevertheless, integrationism does involve a major paradigm shift that can make it extremely hard to understand. This difficulty is focused on the term sign, which has a very specific meaning for integrationists. Rather than representing something, the integrationist sign integrates activities in a specific context. The meaning of the sign is this integration of activities, rather than being something else that is conveyed or represented in addition to the activities integrated. For Harris, “the meaning of a sign is its integrational function—not its capacity to represent anything else” (Harris, 2000a, p. 57) and “a sign cannot exist except in some temporally circumscribed context. That contextualization is an indispensable condition of its very occurrence” (Harris, 1998b, p. 12). An integrationist sign, therefore, cannot be separated into the form of the sign and its content. The sign is a multidimensional construct and it has no meaning separate from an episode of communication. When Harris applies integrationist theory to an analysis of writing as a form of communication, the cultural importance of written words does not prevent him questioning the Postcards and Supasigns 59 centrality of speech to any understanding of what writing is. And because the integrationist sign integrates activities rather than representing something in addition to the activities, it is also possible to bypass the problem posed by the idea that writing must represent something. From this new perspective, Harris points out that the conventional view of writing “confuses the function of the written sign with just one of its possible uses” (Harris, 1995, p. 7). For Harris, writing is a form of communication that utilizes nonkinetic spatial configurations to integrate the biomechanically diverse activities of reading and writing and this “contextualized integration relies in the great majority of cases on a visual framework and visual analogies” (Harris, 2000b, p. 83). Harris argues that a fundamental characteristic of the written sign is that while its formation is kinetic, that is, it involves movement, the written sign itself is static and hence it can be reprocessed, that is, it can be read again and again. In contrast, a spoken sign is a kinetic sign: To hear it again (without using recording technology) we must rely on memory. Thus, in this view, writing actually has much more in common with pictures than it does with speech because the fundamental nature of writing is based on spatial configurations and relationships. As a result, there are actually no fixed boundaries between writing and pictures at all. Distinguishing between writing and drawing involves studying the macrosocial and biomechanical factors of the activities that are integrated (Harris, 1995, p. 48). Harris does not “seek to arbitrate” the use of the term writing as he is more interested in “studying the semiological mechanisms of certain forms of communication” (Harris, 1995, p. 71). However, in distinguishing writing from drawing he has noted that “what characterizes writing is that you have to process the signs in a specific order, not at random” (Harris, 1998a, p. 122). Harris makes a distinction between the written sign and the written form. The former is not the same as the latter because “different activities of interpretation may confer different significations on the same set of marks” (Harris, 1995, p. 68). This idea is hard to apply to alphabetic writing because our early education encouraged us to think that the question “What does B represent?” has a very simple answer. However the examples in Figure 3 suggest that this B has no abstract invariant meaning that is the same from situation to situation. More generally, while some forms of writing may well be integrated with speech communication, for Harris these forms do not represent speech because we “misconstrue a complex of pedagogically inculcated practices as evidence of a representational relationship between speech and writing” (Harris, 1986, p. 108). This is not to say that writing and speech cannot be closely linked. Harris points out that writing in the Western culture has become specialized over the years to integrate speech communication. Thus a symbiotic relationship has developed between the two: a strong influence both of speech on writing and of writing on speech, and this interrelationship is reflected in changes in both (Harris, 2000b, p. 77). However, this specialized kind of writing must not be made the paradigm case for writing in general because the deployment of graphic forms on a surface can create signs that are unique to writing (Harris, 1995, p. 118). The written sign is not the same kind of sign as the spoken sign and writing is not restricted to the continuum of sound. Pryor 60 B refers to . . . (assuming macrosocial understandings) Spoken name of English alphabetic letter #33B1FF Hexadecimal (base 16) number (the equivalent of decimal 11) BATH Pronunciation guide 1 800 BUY TV Telephone key to press B2O3 The element Boron in the Periodic Table B. My second point is Numbering system Picture (in this context) Figure 3. These examples show that the graphic form B has no abstract invariant meaning. THE CONTRIBUTION OF POSTCARD FROM TUNIS Harris’ view of writing involving nonkinetic spatial configurations integrating biomechanically diverse activities is more apparent when encountering a previously unknown form of writing. My interactive multimedia artwork Postcard From Tunis (1997) offers this experience to users who are not Arabic-literate. The work is a personal portrait of Tunis, a city and culture that I love, within which are eight ordinary Arabic words that reflect the themes of the portrait. I programmed the interface so that the work offers users an informal experience of learning to read these Arabic words. Through interaction with the work, a user is offered an experience of the idea that there are no fixed boundaries between writing and pictures (Pryor, 2003). This experience takes a variety of forms. First, in a number of screens there is a moving cursor that is not controlled by the user. This cursor continually traces written Arabic words Postcards and Supasigns 61 from right to left, starting from the far right-hand side, as shown in a static form in Figure 4 (see also Figure 8). As discussed above, Harris points out that once a written sign has been formed, it becomes static and it gives no indication of the kinetic process of its formation. Thus, simply looking at a written Arabic sign will not tell a non-Arabic-literate reader the order of its formation. In this program, however, the moving screen cursor gives a clue: In integrationist terms, it traces (and exposes) the order of formation of the static written sign. Thus, what is created through the combination of the kinetic cursor and the static written sign is a new kinetic written sign in which the formation can be reprocessed. This may seem a small point, but it is significant because a fundamental aspect of the written sign has been transformed. This new written sign tells the reader how to start processing it, that is, where to start scanning and in what direction, and it does this without using words. Postcard From Tunis is multidimensional, combining graphics, photographs, animation, spoken and written words, sound recordings, and music. It is interactive; there are multiple hyperlinked pathways through the material. However, its particular quality is the extensive use of rollovers. A rollover is the activity that occurs when the user moves the mouse (without clicking it) over a programmed area of the screen, resulting in the on-screen movement of the user cursor and a variety of audiovisual responses. Rollovers are rarely mentioned in works on human-computer interaction and are usually overlooked in favor of the hyperlink. However rollovers have a powerful communicational potential. The rollover design in this work enables a gestural and immersive experience for users. As they explore the artwork, they create real-time collages (i.e., layers) and montages (i.e., juxtapositions in time) of sounds, images, and texts. Within this audiovisual experience, the eight Arabic words are interwoven as various combinations of visual and auditory forms. In the new communication space of Postcard From Tunis, the user’s integrated activities (looking, listening, and moving and clicking the mouse) create many kinds of signs. The artwork contains a number of active sites such that, when a user rolls over one of them, the following responses are integrated: • the graphic (image or text) changes visually in some way; • audio plays, for instance, a spoken Arabic word; • the background sound track level drops; and • the cursor changes to indicate whether this location is also clickable. This rollover functionality is very powerful. Neumark (2000, p. 4) notes that “when sound and image suddenly meet at the moment of the user’s interaction, users can experience an intimate engagement and pleasure distinctive to CD-ROM.” As each screen has its own background sound composition, a user’s rollover movements generate a customized soundtrack made up of these rollover responses montaged and collaged together over the background composition. Figure 4. Direction of movement of the moving screen cursor in Postcard From Tunis Pryor 62 In this work, speech is decentered from its usual dominant position: An auditory sign plays only when the user rolls over a picture or script. For example, rollover on both forms in Figure 5 would transform them visually and trigger the sound [felooka; based on my Australian English phonetics], which also creates a meaningful link between them. The generic rollover routine varies so that the four components (graphics, audio, background audio adjustment, and cursor changes) are sometimes joined by other responses. In certain screens, additional graphic forms also appear in response to rollovers (see Figure 6) and thus create dynamically reflexive written signs that indicate in writing, but not in words, how the user is to read them. As an example, upon entering certain screens, an entire written word highlights as the related spoken word plays. Then, one at a time and moving right to left, individual alphabetic letters (or combinations made up of a consonant joined with a long vowel) are visually highlighted (and hence separated from the written word) and the integrated pronunciation plays. At the same time, any vowel marks are displayed and the equivalent individual alphabetic letter(s) appear(s) above the written word. After this sequence, a similar set of activities is integrated whenever a user rolls over any part of the written word, thus creating a dynamically reflexive written sign that indicates how to read it. In the example illustrated in Figure 6, rollover activity on the far right side of the written Arabic word is integrated with seeing that portion of the word highlight and hearing a sound [fff, my phonetics] begin to play. At the same time, a graphic sign appears immediately above it, which is the corresponding alphabetic letter, Faa. Subsequent rollover activity on this alphabetic letter is integrated with the spoken name of the letter and a mouse click would take the user to a postcard containing an interactive Arabic alphabet Figure 5. An example of graphic forms in Postcard From Tunis that trigger the same sound through rollover interaction. Figure 6. A still from Postcard From Tunis showing a screen containing dynamically reflexive written signs.