Repositorio Institucional de Documentos

Miguel Aguilera Lizarraga Supervisor: Manuel González Bedia Co-supervisor: Francisco Serón Arbeloa Master's Thesis in Systems and Computer Engineering ********* September 2011 3 1 2 4 Copyleft c 2011 Miguel Aguilera Lizarraga <[email protected]> Coordination Dynamics in the Sensorimotor Loop. You are free to copy, modify and distribute this work, provided that this notice is preserved, under the terms of the Creative Commons Attribution-ShareAlike 3.0 Unported License. The full license can be found at: http://creativecommons.org/licenses/by-sa/3.0/ Abstract The last two decades have witnessed radical changes of perspective about the nature of intelligence and cognition, leaving behind some of the assumptions of computational functionalism. From the myriad of approaches seeking to substitute the old rule-based symbolic perception of mind, we are especially interested in two of them. The first is Embodied and Situated Cognition, where the advances in modeling complex adaptive systems through computer simulations have reconfigured the way in which mechanistic, embodied and interactive explanations can conceptualize the mind. We are particularly interested in the concept of sensorimotor loop, which brings a new perspective about what is needed for a meaningful interaction with the environment, emphasizing the role of the coordination of effector and sensor activities while performing a concrete task. The second one is the framework of Coordination Dynamics, which has been developed as a result of the increasing focus of neuroscience on self-organized oscillatory brain dynamics. It provides formal tools to study the mechanisms through which complex biological systems stabilize coordination states under conditions in which they would otherwise become unstable. We will merge both approaches and define coordination in the sensorimotor loop as the main phenomena behind the emergence of cognitive behavior. At the same time, we will provide methodological tools and concepts to address this hypothesis. Finally, we will present two case studies based on the proposed approach: 1. We will study the phenomenon known as “intermittent behavior”, which is observed in organisms at different levels (from microorganisms to higher animals). We will propose a model that understands intermittent behavior as a general strategy of biological organization when an organism has to adapt to complex changing environments, and would allow to establish effective sensorimotor loops even in situations of instable engagement with the world. 2. We will perform a simulation of a phonotaxis task performed by an agent with an oscillator network as neural controller. The objective will be to characterize robust adaptive coupling between perceptive activity and the environmental dynamics just through phase information processing. We will observe how the robustness of the coupling crucially depends of how the sensorimotor loop structures and constrains both the emergent neural and behavioral patterns. We will hypothesize that this structuration of the sensorimotor space, in which only meaningful behavioral patterns can be stabilized, is a key ingredient for the emergence of higher cognitive abilities. Resumen Durante las dos últimas décadas hemos sido testigos de cambios radicales de perspectiva acerca de la naturaleza de la inteligencia y la cognición, dejando atrás algunas de las asunciones del funcionalismo computacional. Del gran número de propuestas candidatas a sustituir el viejo paradigma basado en una percepción de la mente construida sobre la manipulación simbólica de representationes abstractas, estamos especialmente interesados en dos de ellas. La primera es la de la Cognición Situada y Corporeizada, donde los avances en el modelado de sistemas adaptativos complejos mediante simulaciones por ordenador han reconfigurado la forma en la que explicaciones mecanicistas, corporeizadas e interactivas conceptualizan la mente. Estamos particularmente interesados en el concepto de bucle sensorimotor, que ofrece una nueva perspectiva sobre qué es necesario para una interacción significativa con el entorno, enfatizando el papel de la coordinación entre actividades sensorimotoras al desarrollar una tarea concreta. La segunda es el marco de la Coordinación Dinámica, el cual ha sido desarrollado como resultado del creciente interés en neurociencia por la auto-organización de las dinámicas oscilatorias en el cerebro. Este marco proporciona herramientas formales para estudiar los mecanismos a través de los cuales sistemas biológicos concretos pueden estabilizar estados coordinados en situaciones en las que de otro modo se volverían inestables. En este trabajo, uniremos ambas perspectivas para definir la coordinación en el bucle sensorimotor como el fenómeno principal que subyace la emergencia del comportamiento cognitivo. Al mismo tiempo, presentaremos herramientas y conceptos metodológicos para enfrentarnos al problema planteado. Finalmente, presentaremos dos casos de estudios basados en las propuestas anteriores.: 1. Estudiaremos el fenómeno conocido como “comportamiento intermitente”, que se puede observar en organismos a diferentes niveles (desde microorganismos hasta animales superiores). Desarrollaremos un modelo que entiende el comportamiento intermitente como una estrategia general de organización biológica frente a situaciones en las que un organismo tiene que adaptarse a entornos cambiantes complejos, que al mismo tiempo permitiría establecer bucles sensorimotores efectivos incluso en situaciones de interacción inestable con el mundo. 2. Llevaremos a cabo una simulación de una tarea de fonotaxis realizada por un agente controlado por una red de osciladores como controlador neuronal. El objetivo será caracterizar el acoplamiento robusto y adaptativo entre la actividad perceptiva y la dinámica del entorno tan sólo a través del procesamiento de información de fase. Observaremos como la robustez del acoplamiento dependerá de cómo el bucle sensorimotor estructura y limita los patrones emergentes tanto a nivel neuronal como del comportamiento del agente. Plantearemos que esta estructuración del espacio sensorimotor, en la que sólo son estabilizados patrones de comportamiento significativos, es un elemento clave para la emergencia de habilidades cognitivas de nivel superior. Publications During the development of this thesis, the author has contributed to the following related publications: Conference Publications •Aguilera, M., Bedia, M. G., Barandiaran X. E. and Serón, F. (2011). The adjustmentdeployment dilemma in organism’s behaviour: theoretical characterization and minimal model. Proceedings of the IEEE Symposium on Artificial Life, pp. 116- 123. Paris, April 11-15, 2011. •Castillo, L. F., Bedia, M. G. , Aguilera, M. and Uribe, A. L. (2011). A proposal for improving retrieval processes in case-based reasoning systems able to act in dynamic environments. Proceedings of the International Symposium on Distributed Computing and Artificial Intelligence, 1(1), 1-4. Salamanca, April 6-8, 2011. •Castillo, L. F., Bedia, M. G., Aguilera, M., Uribe A. L., Manrique, M. and Isaza, G. (2011). Case-based reasoning and real-time systems: exploiting successfully poorer solutions. Proceedings of the 6th Colombian Computing Congress. Posters •Fernández, M. and Aguilera, M. (2011). Exploring the limits of situated and dynamical cognition: embedded vs. extended cognition. Poster presented at the Workshop on Embodied, Distributed and Extended Cognition: Philosophical Perspectives, Department of Philosophy, Universistat Autònoma de Barcelona. Barcelona, March 24-25, 2011. nity. Neuroscientific models have exploited dynamical properties like auto-criticality or metastability for finding the mechanisms that allow brains to coordinate huge numbers of cells into coherent cognitive functions. In chapter 2 we are going to introduce the mathematical framework of coordination dynamics and how it applies to the brain operations. Stressing the role of mechanisms of synchronization and metastability leading to the emergence of cognitive behavior. Also, chapter 3 will review some of the tools that are presented as candidates to provide new understanding about the emergence of cognition in this dynamical framework. In the second part of the thesis, chapters 4 and 5 will present two case studies that explore some of the implications of the ideas presented in the previous part, as well we try to give some insights about how dynamical processes describing low-level biological mechanisms can be connected to higher level cognitive phenomenon, making a link between neural dynamics and the concept of sensorimotor loop. Concretely, the first model will study the phenomenon known as “intermittent behavior”, which is observed at different biological levels (from animals to microorganisms and neurons), trying to find a model that captures the universality and robustness of this mechanism of adaptation to uncertain environments. The second model will implement an agent solving a phonotaxis task using a network of coupled oscillators as a neural controller. There we will analyze how phase information allows robust adaptive couplings between perceptive activity and environmental dynamics; allowing the emergence of the described phenomena of synchronization and metastability within the sensorimotor loop. Finally, chapter 6 will integrate the conclusions of the different parts of the thesis, trying to build a conceptual and methodological framework that takes the emergence of dynamically coordinated sensorimotor loops as the cornerstone of cognition. It is worth to note that this work pretends to be a first approach towards a more ambitious research work. So, during its composition we have looked for an equilibrium between the gathering of information about the issues we are interested in and exploration and experimentation within this issues. The separation between these two complementary activities corresponds to parts I and II of the thesis respectively. ix Part I Disentangling Sensorimotor Coupling 1 Chapter 1 Cognition and the Sensorimotor Loop Since its foundation, artificial intelligence (AI) has been composed by two overlapping and complementary strands. The first one is the engineering approach to AI, which mainly aims to produce new kinds of intelligent systems, understanding intelligence as the ability of a system to perceive its environment and take actions that maximize its chances of success (Russell & Norvig, 2003). However, many key ideas of artificial intelligence have been developed from a different perspective. This is the scientific approach to AI, which is mainly concerned with understanding what is and isn’t possible in natural and artificial intelligent systems. Thanks to this interest in the understanding of what constitutes intelligent behavior and how can it be generated, the major technical and conceptual advances in cognitive science have always been connected directly or indirectly to artificial intelligence. However, the engineering approach have traditionally ruled over the vast majority of the AI community, frequently dismissing or ignoring some of the critiques related not with particular solutions for specific problems but the very foundations of the research program in AI. These critiques point to what has been called an ontological blindness in AI (Sloman & Chrisley, 2005). With this term, it is referred to the fact that the preconceptions about a particular approach to a problem can prevent (e.g., a group of researchers) to identify what kind of entities, properties, relations, and processes need to be explained or modeled; and therefore constraining possible research directions that could point to possible alternative solutions for the problem. This chapter tries to explain what is our position about the classical approach to AI, presenting some critiques concerned with the nature of intelligent behavior. Also, we introduce what we consider as a powerful conceptual tool for overcoming some of the “blindnesses” that affect the study of intelligence and is the cornerstone of some alternative approaches to cognitive science: the sensorimotor loop. 1.1 Fleeing from Cognitivism. The Dynamical Approach to Cognitive Science Since the times of the development of formal logic, the theory of computation and the expansion of computers led to the consolidation of the “computer metaphor of mind’ ’ for explaining intelligence. The central intuition about this metaphor is that intelligence (including human intelligence) is similar to a computer, thus cognition can be defined as the computation of symbolic representations according to some set of rules. According to this view, a cognitive system would act correctly as long as the symbolic representations 3 Chapter 1: Cognition and the Sensorimotor Loop it processes are actually an accurate representation of the real world; and the processing of such information leads to a successful solution of the problem the system is facing. This approach to understand intelligence, named the computationalism or cognitivism, has been the base for traditional artificial intelligence. This approach takes as its starting point the concepts of representation and computation, boiling down cognitive processing to the computational manipulation of representational inner states. The brain is considered to be a piece of biological hardware, and the mind is the software running on top. Cognition consist in rule-governed manipulation of symbols, which can be performed in a Turing machine (Newell & Simon, 1972). The only necessary condition to reproduce any kind of intelligence would be to have a rich enough repertoire of symbols and a detailed enough collection of rules for manipulating these symbols. However, as the attempts of classical AI for reproducing a somehow human-like intelligence showed its limitations, the computationalist approach started to be questioned. Concretely, the first critiques started in the ‘80s with the resurgence of connectionism (Rumelhart et al. , 1986), which criticized the idea of a linear processing of symbols according to rules and proposed in turn models inspired in the distributed structure of neural networks. The new connectionist approach emphasized the advantages of distributed systems with massive parallel processing of information, allowing to reproduce cognitive phenomena which have been ignored or let aside before, as pattern recognition, associative memory, preservation of global effectiveness despite of local structural damage, etc. Even when the connectionist approach solved some of the problems of the computationalist models, it still had some inherited limitations from the old computationalist perspective. In this way, connectionism has been defined as an “unfinished revolution” (Clark, 1997), since it stills maintains the core ideas of computationalism: i.e., cognition is fundamentally a computational and representational process disembodied and dissociated from its biological roots. If classical computationalism reduced cognition to some form of symbol crunching according to algebraic rules, connectionism attempts to explain cognition in terms of the computational manipulation of subsymbols, according to statistical rules. That is, if classical cognitivism assigns symbolic content to the sort of physical entities that get stored in von Neumann architectures, connectionist cognitivism assigns subsymbolic content to the sort of physical entities that are fully distributed and superposed on the neural network’s weight matrix. Independently if we consider the mind as a sequential (computationalism) or distributed (connectionism) machine, intelligence is still being defined in terms of abstract manipulation of symbols with sematic value (Gomila & Calvo, 2008). 1.1.1 The dynamical systems approach and post-cognitivism Despite the limitations of connectionism, it brought a change of perspective with deep implications artificial intelligence. Particularly, the introduction of recurrent connectionist models allowed the emergence of a new approach: the Dynamical Hypothesis (van Gelder, 1995). With the slogan “cognitive agents are dynamical systems”, this view represents an important extension of the connectionist approach. Recovering ideas brought by the first cybernetists like the recurrent, dynamical nature of the brain (Ashby, 1952), the importance of feedback and stability in complex evolving systems (Wiener, 1948) or the study of systems completely embodied and embedded in the real world (Walter, 1950), the Dynamical Hypothesis insists on the importance of analyzing the evolution of internal patterns of activations over time (French & Thomas, 2001). Dinamicism is not necessarily a complete refutation of cognitivism, but provides a methodological turn that questions some of its basic assumptions. Its breaking point is that it attacks the principal postulate of the computational approach. With the em- 4 Chapter 1: Cognition and the Sensorimotor Loop phasis on the fact that cognition is a phenomenon that evolves over time, symbols and rules are no longer sufficient to explain cognition. Where computationalist said that the underlying mechanisms that gave rise to symbols and rules manipulating them were not important, dynamicists stress the importance of the “subcognitive” mechanisms underlaying the cognitive phenomena. While computationalists were explicitly uninterested in linking the neural level to the higher symbolic level, dynamicists seek to understand how higher levels of cognitive behavior might arise from sequences of lower level actions. After leaving behind the discussion about cognition being a centralized or distributed process, different post-cognitive views arose, stressing the importance of viewing cognition not as abstract computation but as interactive, embodied an embedded. The difference between both approaches was quite graphically described by Hurley (1998) when she warns against the “cognitive sandwich” metaphor represented by cognitivist approaches. According to them, cognition might fill the space in between a perception-action bun. In this view, either in computationalist or connectionist approaches, cognition would play the same role between a perception and an action layer. In contrast, post-cognitivist approaches conceive cognition and behavior in terms of the dynamical interaction (coupling) of an embodied system that is embedded into its surrounding environment. Thus, it is necessary not understand just the brain/mind in itself, but the whole coupled system, not leaving apart the interaction of perceptual and motor apparatus with their environment. In the last decades, different scientific programs have explored different approaches to a post-cognitive study of mind. However, in this work, we are going to center our interest in two conceptual issues that we consider to be central for understanding cognitive systems: •Sensorimotor coupling: cognitive systems cannot be understood without the context or environment in which it moves, evolves, develops. Interactivism and dynamicism explain how robust but flexible coupling with the environment emerges, breaking apart the “cognitive sandwich”. This dynamic interaction brings sensorimotor aspects to the center of the study of cognition. Also bringing both the importance of the body, as the way to ground behavior in the environment, and active perception, whose role is guiding action rather that constructing representations of the world. •Coordination dynamics: the mind is a complex, distributed and fluid adaptive process. The real-time interaction of neural, bodily and environmental factors engaged in sensorimotor loops depends crucially on the time course of this process. Neural activity and bodily interactions show non-linear, time dependent and continuous behavior, so dynamical systems theory seems an appropriate framework for studying it. In addition, cognition is an emergent, self-organizing phenomenon, arising out of the local activity of distributed units, with no single location in the system acting as a central controller. Coordination dynamics are fundamental to understand how cognition arises built upon the versatile and flexible mechanisms within the brain. We will dedicate the rest of this chapter for introducing sensorimotor coupling, while coordination dynamics will be address in chapter 2. 1.2 The Sensorimotor Loop Cognitive processes emerge from the dynamical interplay between the neural system of an agent, its body, and the environment, creating what we call a sensorimotor loop. The 5 Chapter 1: Cognition and the Sensorimotor Loop coupled nature of that interaction makes quite difficult, if not impossible, to separate and understand the components of a cognitive system in isolation from the rest of the system. Sensorimotor loops merge both actions and dispositions into a dynamical structure that cuts across the brain-body-environment continuum. As a result, sensorimotor loops stand in a privileged theoretical position to study the relationships between concrete neural mechanisms and behavior. They do not privilege either perception or action, or any other modularist assumption, and cover a wide range of behavior, from simple reflexes to different scales of learning and development. The concept of sensorimotor loop arises from the ideas brought by different approaches to cognitive science that stressed the role played by the body and the agent’s coupling with the environment. These approaches can be synthesized in the concepts of situatedness and embodiment, and understanding these terms would be the first step towards understanding the nature of sensorimotor loops. 1.2.1 Situated Cognition The idea of situatedness means that the agent is embedded in a world, in opposition to classical AI models that are just given some information to solve a problem in an uncoupled way, not participating in the world as would agents in the usual sense. When agents are situated, their whole interaction with the environment is controlled by the agent itself. Information is perceived through sensors (e.g. photodetectors, microphones, collision sensors) and behavior of the agent has to be deployed by means of its own actuators (e.g. legs, wheels, arms). Thus, agents do not deal with abstract descriptions of the world, but they use its own perception of the world, the here and now, to generate its behavior. In this sense, being situated in an environment is going to give sense to the agent’s actions. We could synthesize the role of situatedness in the following ideas (Beer, in press), which have been traditionally neglected in AI and cognitive science: 1. Concrete action. Taking action in the world is more fundamental than making abstract descriptions of it. While conscious deliberation clearly has its role, the ultimate job of an intelligent agent is to do something, to take some concrete action with external consequences. 2. Situatedness. An agent’s immediate environment plays a central role in its behavior. This environment is not only a rich source of constraints and opportunities for the agent, but also a context that gives meaning to the agent’s actions. 3. Interactionism. An agent’s relationship with its environment is one of ongoing interaction. The environment does not serve merely as a source of isolated problems for the agent to solve, but rather a partner with which the agent is fully engaged in moment-to-moment improvisation. A classic example of situated cognition are Braitenberg vehicles, which develop complex behavior based in quite simple mechanism that allow different degrees of coupling with the environment, generating complex and unpredictable behaviors (Braitenberg, 1984). For example, a vehicle with two light sensors directly connected to a pair of motors is able to perform a phototaxis task (figure 1.1). We can see how the agent does not perform any kind of computation, but the problem is solved through a continuous interaction with the environment and a sensorimotor coupling between the motor actions and the sensory signals. Even when the robot is very simple, it can perform an interesting behavior. And, when the complexity of the vehicles is slightly increased, the behaviors shown by them are surprisingly interesting. 6 Chapter 1: Cognition and the Sensorimotor Loop Figure 1.1: Braitenberg phototactic vehicle. The agent approaches a light source with just two connections between the right sensor and the left motor and viceversa. Situated interaction with the environment allows simple control systems to display complex behaviors. Figure adapted from Braitenberg (1984). 1.2.2 Embodied Cognition As we have explained, one of the greatest potentialities of the dynamicist approach is its capability to take account of how abstract cognitive abilities emerge from mechanisms at the biological level. This fact leads directly to the notion of embodied cognition, which claims that it is impossible to understand cognition without the bodies in which cognitive processes take place. According to this view, the body allows an agent to experience the world directly, since its actions are dynamically engaged with the world and have an immediate feedback on the agent’s own sensations. Embodiment constitutes a physical grounding of the agent, which is forced to face the kind real-world issues organisms have to deal with. As result, material properties and morphology of the body often play a key role in the generation of behavior. Thus, the body not only imposes constraints in the behavior of an agent, but it is going to allow an agent to exploit the world by sensorimotor coupling with the environment. According to Beer (in press), there are at least three distinct ways in which embodiment affect cognition: 1. Physical embodiment. Physical aspects of an agent’s body are crucial to its behavior, including its material properties, the capabilities for action provided by the layout and characteristics of its degrees of freedom and actuators, the unique perspective provided by the particular layout and characteristics of its sensors, and the modes of sensorimotor interaction that the sensors and actuators collectively support. 2. Biological embodiment. Not only are the physical characteristics of bodies important, but the specifically biological facts of an organism’s existence must also be taken into account, including the relevant neuroscience, physiology, development and evolution. 3. Conceptual embodiment. Even when engaged in pure ratiocination, our most ab- 7 Chapter 1: Cognition and the Sensorimotor Loop stract concepts are still ultimately grounded in our bodily experiences and bodyoriented metaphors. Figure 1.2: Passive dynamic walkers exploit its morphology to walk down a slight incline without external power. Figure adapted from McGeer (1990). A good example of how embodiment can be exploited for generating behavior are passive dynamic walkers (McGeer, 1990, figure 1.2), which are capable of walking down a ramp without any sensing, actuation, or control. Nonetheless, the behavior of the robot emerges from the exploitation of its own dynamics - i.e., how gravity, friction, and the forces generated by the swinging of the legs and arms act on it. As a result their walking behavior is very energy efficient and looks surprisingly natural. This presents a big contrast compared with classical AI walking robots, which normally are hard to design and show clumsy and unnatural movements. In dynamic walkers, the processing normally required for controlling walking is taken over by the proper morphology and the right materials, as well as the coupling of the legs movements with gravity in a pendulumlike way. As well, animals also exploit this embodied dynamical mechanisms for walking, exploiting the complexity of a system constituted by our bones, joints, tendons and so on. 1.2.3 Brain-Body-Environment Systems So far we have explained how the dynamical approach to cognitive sciences emphasizes the temporal dimension of behavior, while situatedness concerns the role played by the ongoing interactions of an agent with its immediate environment and embodiment stress the role of the physical properties of the agent’s body in its behavior. Even when all these ideas have been historically developed in different contexts, they work much better when they are understood as a unit. The combination of the three of them leads to the notion of a brain-body-environment system, composed of an agent’s nervous system, its body, and its environment; conceptualizing all of them as coupled dynamical systems which are in continuous interaction (Beer, 1995a, figure 1.3). The idea of a brain-body- environment system has fundamental implications in cognitive and brain sciences and raises many empirical and theoretical challenges. A review of the accomplishments of the research in this direction was developed by Beer (2008). The traditional view in Artificial Intelligence and Cognitive Science decomposes the sensorimotor loop into an open perceive-think-act loop, understanding that the purpose of perception is to create an internal model of the environment to decide what is the right action to perform. This view assumes the existence of a central stage, between 8 Chapter 1: Cognition and the Sensorimotor Loop sensing and acting, where the contents of the environment are represented. On the other hand, the brain-body-environment system approach remarks that the role of perception is not reconstructing the environment, but transform the sensory signal to produce some motor output. Figure 1.3: An agent and its environment as coupled dynamical systems in a sensorimotor loop. The agent is composed of coupled nervous systems and body dynamical systems. A deeper critique to the traditional perceive-think-act view points its tendency to ignore the closed-loop nature of the perceptual process (Pfeifer & Scheier, 1999). According to the brain-body-environment perspective, behavior can be only understood when the sensorimotor loop is considered as a whole, and we cannot understand it by studying its parts in isolation. Indeed, the very nature of the problem to be solved can change when the sensorimotor loop is established. This is the case for active vision, where gaze control simplify the processing of visual perception (Ballard, 1991), auditory perception of distance, which is improved when movement is involved (Rosenblum, 1993) or tactile perception, where object recognition is greatly enhanced with active manipulation (Lederman & Klatzky, 1996). This view suggests that perception is an ongoing activity of exploration rather than an abstraction of perceptual experience into a final interpreted percept. 1.3 Redrawing the cognitive phenomenon The approach presented here, which considers the sensorimotor loops to be at the center of the emergence of cognition, can be summarized in the three ‘radical embodiment’ propositions presented by Clark (1999): 1. Understanding the complex interplay of brain, body and environment requires the tools and methods of dynamical systems theory. 2. Traditional notions of internal representation and computation are inadequate and unnecessary. 3. The typical decomposition of the cognitive system into a variety of functional subsystems or modules is often misleading, and blinds us to the possibility of alternative, and more explanatory decompositions into dynamical systems that cut across the traditional brain-body-environment divisions. Within this framework, the power of dynamical tools is to model the embodied agent-environment interactions and the biological mechanisms that do not stand for any 9 Chapter 2: Synchronization and Metastability in the Brain because convergence of connectivity is no longer the main variable of feature extraction; rather, it is the temporal coherence of neurons, representing the various attributes of objects, that matters. The main advantage of this approach, is that it offers an unlimited coding capacity for feature combinations, as well as the possibility of mapping crossmodality representations onto each other without altering the their coding formats. It has been proposed that this coding may be achieved by means of phase information (Singer, 2007), due to the ability of neurons to code aptitude values into a temporal code of spike timing. Even when binding operation can be accomplished by fixed anatomical connections, binding must be a versatile mechanism since features processed in parallel by different areas of the brain need to be bound selectively and transiently. Such versatility can be obtained by exploiting the temporal dimension as coding space. Here, oscillatory networks offer the option to use phase relations of spike timing for the selection, gating and routing of signals exploiting by adjusting oscillation frequencies, phase relations of oscillations and exploiting a variable spectrum of conduction velocities. 2.3.3 Theta Rhythms: Representation by Phase Information Hippocampal cells studies in rats discovered the existence of the so called “place cells”, which encode information about the position of a rat in its cage, exhibiting a high firing rate when the animal is in a specific location in an environment corresponding to the “place field” of the cell. It was also discovered that the spike phase of place cells shifted systematically in relation of the ongoing theta oscillations (O’Keefe & Recce, 1993). This phenomenon, called phase precession, is the same hypothesized for phase coding in binding mechanisms. When recording the activity of a place cell when the rat crosses a linear track corresponding with the place field of the cell, the phase of the place cell spikes shifts monotonically as a function of the rat’s position. The relationship between position and spike phase is independent of the firing rate or the speed of the animal, and depends only on the size of the place field. Ideally, the slope is a line between the beginning and the end of the place field, spanning 360 degrees, so that information about successive metric distances is reflected in the precise temporal sequences within cycles. Furthermore, place fields overlap, and place cells establish stronger associations with cells that are activated near each other in time, tying together sequential places. This allows to link representation of the current positions with representation of the past and the expected future. This way, the temporal compression mechanism of the theta oscillation objectively defines spatiotemporal context. Coding for ordered locations and distances is analogous to learning an episode of sequentially presented or visited items. The difference lies in the nature of the inputs rather than the nature of hippocampal computations. The same principles used in navigation in a one-dimensional track can be used to explain episodic memory. The situation changes when we turn to trying to explain navigation in two-dimensional maps and semantic memory. While cells in one-dimensional travel had unidirectional place fields (determined by the position in the track), in two-dimensional environments, exploration leads to crossing the same positions from different directions. Now, place cells are going to be tied to different routes, becoming omnidirectional. Their activation is no longer going to depend on a temporal context. They explicitly define positions. Similarly, multiple episodes involving a common item can free this item from its spatiotemporal context. Neurons which are members of an omnidirectional assembly collectively define or symbolize the semantic “meaning” of an item (Buzsáki, 2006). 16 Chapter 2: Synchronization and Metastability in the Brain 2.4 Coordination Dynamics Observations of brain modes of behavior can be fascinating and give us important insights about self-organization in the brain. However, it is hard to achieve a deeper understanding of brain activity if we cannot build models representing how they work. A promising formal counterpart for the previous observations can be found in the framework of Coordination Dynamics, which describe the mathematical formulae and paradigms governing the coupling of environmental stimuli to their effectors. Kelso (1995) proposed coordination as a fundamental feature of life, allowing neurons, brain and mind to give rise to complex patterns of behavior. The proposed mechanisms for achieving coordination are self-organized interactions, leading to the spontaneous formation of patterns and pattern change in open non-equilibrium systems. According to this, mental activity would not constitute a programmable, static, timeless entity outside the brain, but would be sustained by a constantly shifting dynamic system of pattern formation. Coordination dynamics has presented some surprisingly simple and elegant models as the Haken-Kelso-Bunz (HKB) model, which describes the dynamic of the relative phase between two non-linearly coupled oscillators (Haken et al. , 1985); Also, this approach introduces a important difference with other dynamical systems approaches: “in contrast to classical dynamics that deals with fundamental quantities such as mass, length, and time and their relation, coordination dynamics deals with informational quantities of a relational kind” (Kelso, 1995, p. 95). The proposed relational quantities are the relative phases of different non-linearly coupled oscillators, which characterize the relation among different parts of the system, independently of the nature of interactions. In this way, relative phases would constitute the order parameter of neural systems. Coordination Dynamics is based on nonlinear coupling of the components of the system, constituting an emergent coordinative (higher) level of description of the system behavior. The nonlinearity of the coupling is a fundamental characteristic allowing the coupling to be nonspecific with respect to the patterns of coordination that emerge. That is, different coupling functions can give rise to the same coordination patterns, and changes in coordination can be brought about in a variety of ways. These invariance of function and multifunctionality are intrinsic properties of the coordination dynamics approach, hypothesized to be one fundamental characteristic of living organisms. Being the lack of a one-to-one relationship between self-organized coordination patterns a basic property of life. 2.4.1 The HKB Model The HKB model is the driving example for Coordination Dynamics, describing the behavior of two non-linearly coupled oscillators. Initially, the model described the relative phase dynamics of bimannual coordination, but the model has been proved to represent the coordination dynamics of different behavioral (Kelso, 1995), neural (Jirsa et al. , 1998) and social (Kelso et al. , 2009) phenomena. In the basic version of the HKB model, the relative phase derivative between the two oscillators follows the equation: ˙ φ=δω −a∙sin(φ)−2b∙sin(2φ) + pQ∙ε(2.1) where φis the relative phase between the two interacting oscillators; aand bare the coupling coefficients; δω is the difference between the intrinsic frequencies of the pair of oscillators; and εrepresents the presence of noise fluctuations of strength Q. In (Kelso, 1995) it is found a more detailed description of this equation. 17 Chapter 2: Synchronization and Metastability in the Brain Figure 2.2: A simple theoretical model of brain coordination dynamics exhibits multistability, adaptive phase shifts, critical phase transitions and metastability. Taken from Tognoli & Kelso (2009) We can consider the HKB model as a self-organized system, where φis the order parameter, and a,band δω are control parameters. For fixed values of aand b, and shifting the value of δω (although a different selection of control parameters could show the same results), we found three different possible situations for the model (figure 2.2): 1. When δω ∈(0,1), there is a situation of multistabiliy, where there coexist two attractors at φ= 0 yφ=π(i.e., the two oscillators being synchronized in-phase and anti-phase). 2. When the control parameter shifts to δω ∈(1,2), we find a nonequilibrium phase transition. That is, when the control parameter passes through a critical point, a qualitative change in the attractors takes place. In this case we will only have one attractor at φ= 0, and the attractor at φ=πdisappears. If we reverse the situation, decreasing δω again, the system will stay always in the φ= 0 attractor, thereby exhibiting hysteresis. 3. When the control parameter approaches a given threshold, the system attractors move and eventually disappear. Over this threshold, when δω ∈(2,4), the relative phase have no longer fixed points but engages in what Kelso calls a metastable dynamic. In this state, relative phase tends to stay near one of the previous attractor points, φ= 0 and φ=π, but eventually escapes and falls into the other attractor. It is important to note that, even when the attractors do not exist anymore in the metastable zone, there is still attraction. When the relative phase system is in a point where it used to be an attractor (near , φ= 0 and φ=π), the system tends to stay in that point, even when eventually it is going to be forced to leave. Anyway, due to the periodic nature of relative phases, the system is always going to be temporarily trapped again around the metastable quasi-attractor. This HKB models reflects some important properties of biological dynamics, as multistability, hysteresis and metastability. In a nutshell, coordination dynamics provides two useful mechanisms to understand the behavior of neural dynamics: 18 Chapter 2: Synchronization and Metastability in the Brain Figure 2.3: Dynamics of a metastable behavior where φis the relative phase of the system at instant tand f(φ)is the relative phase at instant t+ 1. It is show how the system is temporally trapped near a quasi-attractor. Taken from Kelso (1995). •It allows to represent a situation where the system is able to switch between two states qualitatively different in an open nonequilibrium system. Classical Artifitial Intelligence implements switches or regulators to model changes of behavior. In contrast, Coordination Dynamics is able to explain switching without any switches at all. •In the dynamical approach to cognitive science there are tendencies to model different behaviors just adding attractors to the system (see for example Dynamic Field Theory implementations (e.g., Bicho & Schöner, 1997), or the attractors of an artificial neural network). Coordination Dynamics is able to model attraction without the need of attractors (figure 2.3). This mechanism is hypothesized to allow the brain to rapidly change between behavioral states, preventing either the brain dynamics to get stuck in a fixed point or flying apart due to the lack of attracting forces. The thesis of Coordination Dynamics is that the human brain would be a device that, rather than compute information, would hop quickly among different metastable states. In the human brain, attraction and repulsion influences coexist in a finely balanced way, allowing the emergence of two key properties: one, the ability to enter and exit coherent spatiotemporal patterns of neural activity; and two, the ability to engage and disengage participating subsystems in a flexible way (Kelso, 1995). 2.5 Metastability and Cognitive Dynamics Computationalism views the brain as a passive, stimulus driven device, simply reacting to sensory inputs according to ‘bottom-up’ processing in hierarchically organized neural architectures. However, this view has been challenged by new neurophysiological data indicating that the brain should be regarded as a much more active and adaptive system, in which ‘top-down’ mechanisms continuously create predictions about forthcoming stimuli and constantly match expectations against signals from the environment (Engel et al. , 2001), therefore defining cognition as a fundamentally action-oriented phenomenon. However, despite the great amount of mathematical and simulation modeling in current large-scale neurodynamics, there is still little systematic exploration of the coupling between brain, body and environmental dynamics. Most approaches focus on how oscillatory coordination carry information within the brain, letting apart ideas about sit- 19 Chapter 2: Synchronization and Metastability in the Brain uatedness and embodiment of the cognitive phenomenon (see chapter 1), which assume that cognition does not build on context-invariant models but is subject to constrains imposed by and ever-changing brain-body-environment relations that have to be coped with in an adaptive and context-dependent manner. This section aims to gather some of the insights that could be useful to integrate recent discoveries in large-scale neuroscience about oscillatory coordination with the emphasis of the traditional dinamicist approach to cognitive science on sensorimotor loop coupling. 2.5.1 Temporal coding mechanisms for building ‘top-down’ loops It was proposed by Singer (1999) that internal coordination of spike timing in cortical neural networks was a relevant factor representing context dependent dynamic interactions. As seen in section 2.3.2, dynamic binding strategies would depend on the dynamically adjustable configuration of the response of distributed neurons rather than a hierarchical structure of neurons. Also, temporal cues seem to be exploited for perceptual coupling (Lee & Blacke, 1999). As well, psychophysical and physiological evidence indicates that neural networks are highly sensitive to temporal relations among discharges in input connections, being particularly receptive to synchronous inputs (Singer, 1999). Thus, coding in neural networks is proposed to be achieved by means of oscillatory modulation (Singer, 2007). Timing of neuronal spikes relative to the phase of the oscillation cycle depends on the strength of the excitatory drive. When the drive is stronger, discharges will take place earlier. In consequence, the amplitude of excitatory drive can be converted into spike timing, being the phase precession of discharges a direct measure of input intensity. This relation makes it possible to convert rate coded amplitude values into a temporal code of spike timing. These neurophysiological observations link directly with the ideas of Kelso about relative phases being the main coordination parameter encoding valuable information in organisms, leading to the emergence of higher system levels channeling individual neuron responses. In this way, oscillations allow to encode information and to define relations between the activity of spatially distributed neuron groups. When neuronal groups become entrained in synchronous oscillations, they will tend to emit spikes in synchrony and this enhances the impact that these output signals will have on target cells. Synchronization can thus be used to select signals for further joint processing and to accelerate the propagation of the signals across distributed networks. The above leads to a discussion about the structure of information flows in neural networks (Engel et al. , 2001). According to the cognitivist view, the ideas of ‘top-down’ and ‘bottom-up’ were referred to the distinction between expectation-driven processing and stimulus-driven processing. In this way, behavior could be controlled largely by a sensory stimulus or dominated by intrinsic factors such as attention, memory or expectation of forthcoming sensory events. On the other hand, complex systems ideas introduce a new view of ‘top-down’ where the whole is able to determine the perception of the parts. However, coordinated temporal encoding of information lead us to another variant of a ‘top-down’ approach, in which large-scale dynamics can have a predominant influence on local neuronal behavior by ‘enslaving’ local processing elements. This idea of top-down would not require a processing hierarchy, but the dynamic ’capture’ of neurons into a larger assembly could occur between areas at the same processing level or within an area. This kind of organization works by creating top-down dynamical loops between different cognitive levels. Also, as we are going to see below, these loops are not only limited to internal activity of the brain, but across the whole brain-body-environment system, allowing the brain to anticipate change in a fluctuating world. 20 Chapter 2: Synchronization and Metastability in the Brain Extending the loop: anticipation in brain oscillations If the brain is considered as an active and adaptive system, comparison of sensor input with existing knowledge is essential for perception. Experimental data suggest that this is achieved by top-down modulation of sensory information processing, mediated by context-depending modification of the temporal patterning of neural responses, particularly by influencing their coherence. According to this view (Engel et al. , 2001), top-down factors would lead to states of ‘expectancy’ or ‘anticipation’, expressed in the temporal structure of activity patterns in the brain. These process would be carried by large-scale assemblies where the entraining effects of neural assemblies carrying high-level representations over assemblies involved in the processing of new information would allow a continuous prediction of environmental inputs in a dynamical top-down process. Local patterns would be constantly subject to modulation by specific synchronizing and desynchronizing influences carrying predictions about specific feature constellations. Top-down dynamical loops are not limited to neural assemblies but, in order to build anticipatory responses, have to cut across the brain-body-environment system. These top-down modulatory effects on neuronal activity have been seen to be played by attention, working memory and behavioral context. These kinds of processes would be the link between the self-generated temporal dynamics in neuronal networks with the view of the brain as an active and adaptive device. 2.5.2 Cognitive functions of cell assemblies In previous sections, cell aggregation and disaggregation in the brain by means of synchronization has been presented as a fundamental mechanism for building cognitive states. These transient distributed subsets of neurons with strong reciprocal connections have been called cell assemblies or neuronal ensembles. A cell assembly can be activated or ignited from any of its smaller subsets, due to the assumed strong interconnections, and they must “hold” after its activation during an determined lapse of time. Varela (2006) proposes that a singular and specific cell assembly underlies the emergence and operation of every cognitive act (perception-action, memory, motivation, etc.). Here, cell assemblies would emerge through fast, transient phase locking of activated neurons, understood as some kind of temporal “glue” that allows neural coherence. Cell assemblies would incorporate or discard external and internal information into its coherent activity, while different assemblies are evaluated until one is transiently stabilized and expressed behaviorally. The entire process takes the form of a bifurcation from a noisy background to the emergence of a transiently stable, distributed structure bound by synchrony. The interpretation of transient coherency-generating process generated by the nervous system would constitute mental-cognitive states. As both external and internal influences are fundamental in the generation of cell assemblies, the sensorimotor loop must be taken as a whole. Since mental states must have a immediate neural consequences at the level of behavior and perception, they will have subsequently a direct effect on neural events (in the form of a downward causation). As well, mental states are always bound to the body, embedded in a particular field of sensation. Metastability and Consciousness Other theories about metastability in the brain stress its importance for explaining consciousness. A central role in these theories is given to the so-called Dynamic Core, a 21 Chapter 2: Synchronization and Metastability in the Brain parallel and distributed dynamic process constituted by a large but distinct sets of distributed neuronal groups believed to be the integration center of consciousness (Werner & Jirsa, 2006). The Dynamic Core Hypothesis associates cognitive events with the formation of distributed clusters of neurons that, at the same time, are quite differentiated from the rest of the system, being able to detect features of particular signal inputs, and are able to functionally integrate, by temporal correlation and synchrony, these features into particular objects and background. A crucial point of the Dynamic Core Hypothesis is that, instead of thinking about integration and non-integration as binary and conflicting concepts with nothing in-between, the metastable nature of the Dynamic Core allows a continuum of integration. Another inspiration for theories of consciousness and metastability is the Global Workspace Hypothesis proposed by Baars (1997). The theory postulates the existence of a multitude of small and unconscious processes that gain access to a limited capacity “global workspace”, which is able to integrate competing and cooperating processes and is the gateway to consciousness. This global workspace would allow widespread interactions between otherwise independent brain functions, recruiting different unconscious processes when needed for solving problems or dealing with new situations. Different works have address the issue of supplying dynamics to the Global Workspace Hypothesis, allowing mechanisms by which sub-networks with synchronized activity can be transiently formed. This transient formation of functional neural complexes would be driven by phase transitions in metastable dynamics, being responsible for the existence of different cognitive events. Anyway, despite of the importance of neural dynamics, it is necessary to describe conscious situated agents in terms of how neural dynamics are embedded in the somatic and environmental context of the animal’s life (Thompson & Varela, 2001). In higher primates three kind of cycles are distinguished: •The organismic regulation of the body, which is the base of emotional states (a primal consciousness or sentience, the feeling of being alive). This cycle would include biochemical interaction between brain and body, linking sensors and effectors to neural processes. •Sensorimotor coupling, which links what the organism senses as a function of how it moves and how it moves as a function of what it senses. Here, neural assemblies would mediate the coordination of sensory and motor functional areas, and sensorimotor coupling with the environment would constrain and modulate this neural dynamics, allowing the organism to be a situated agent. •Intersubjective interaction, which includes signaling of affective states and sensorimotor coupling to create different forms of social cognition. The former will be based in the mechanisms which make us able to perceive our own or others emotional states, while the latter would include mechanisms as the so-called ‘mirror neurons’, which generate sensorimotor loops necessary to recognize gesture of others. 2.6 Coordination for the engagement of Sensorimotor Loops In conclusion, despite the advances of neuroscience describing the kind of dynamics that allow the emergence of cognitive behavior, the study of these dynamics in isolation seems not to be enough. As the dynamical approach to cognitive science have insisted during the last two decades, cognitive dynamics have to be understood within the different loops coupling internal neural dynamics with bodily and environmental dynamics. 22 Chapter 2: Synchronization and Metastability in the Brain So far, the main proposal is to study relative phase information as the order parameter in self-organized neural dynamics coordination. Two different behavior are hypothesized to take place in the human brain: •Non-equilibrium phase transitions in synchronized systems, which establish different phase relations between the elements of the system, giving place to coding, binding and representation of information in the brain. •Metastable dynamics, which allows the brain to rapidly switch between behavioral states, being able to create and vanish coherent patterns of neural activity, as well as flexibly engage and disengage subsystem of the neural substrate. This two mechanisms would be responsible of generating top-down loops of dynamically capture of neurons in larger assemblies and, eventually, trespass the neural system to form part of sensorimotor loops. Thus, coordination dynamics would be responsible for the emergence of precisely regulated attraction and repulsion forces to form transient forms synchronization and metastability, allowing thus the engagement of stable or metastable sensorimotor loops. The following chapter will address the great difficulties both in the experimental and theoretical levels for the study of how sensorimotor loops could arise based on these coordination mechanisms. We will analyze both the methodology and the tools necessary for studying the role of coordination in the behavior of embodied and situated agents. 23 Chapter 3 Tools for Designing Emergent Cognitive Behavior In previous chapters, we have presented a conceptual framework to study cognition, based on the features of self-organized coordination in dynamical systems. These coordination mechanisms, based on different forms of synchronous and metastable entrainment of oscillating systems, would be the base for the emergence of sensorimotor loops. This view of cognition assumes that cognitive behavior is an extremely complex process emerging from tightly coupled physical, genetic, neural and environmental factors. Therefore, divide-and-analyze methodologies will not lead to a correct understanding of it. In this chapter, we will introduce a methodology for understanding the emergence of cognitive behavior from sensorimotor loops: the synthesis of artificial agents that exhibit emergent adaptive behaviors, with the objective of understanding the processes of self-organization that lead to the engagement of sensorimotor loops in cognitive tasks. Due to the limitations of analytical observation, we will go the other way around, trying out to discover the self-organized mechanisms that allow synthetic agents to display emergent behaviors, and observe how they are related with the natural phenomena we are interested in. We will propose to use evolutive techniques in minimal models to find the coordination mechanisms that allow agents to exploit sensorimotor couplings in ways that are not obvious for a designer. Additionally, we will propose two different tools for studying coordination within neural systems: continuous-time recurrent neural networks and Kuramoto oscillatory networks. 3.1 The Synthetic Approach: Understanding by Building The approach we are proposing is the so-called constructive or synthetic approach. This approach is presented as a methodology used by the sciences of the artificial (e.g. artificial intelligence, robotics, or cognitive modeling) to contribute to the scientific research in life and cognition. These sciences increasingly claim to go beyond a mere engineering approach, and provide a purely scientific approach to crucial topics of natural sciences (like biology, psychology and neuroscience). Their proposal would conform a new methodology for these sciences to find the hidden mechanisms in natural systems. This methodology, whose objective could be summarized in the idea of “understanding by building”, presents a shift in the usual order of behavior analysis and model building. It requires the researcher to embed first basic hypothesis about life and cognition within a working model, and only then to examine the behaviors they produce. The objective behind this is to provide simple scientific explanations of complex natural systems (Damiano & Cañamero, 2010). 25 Chapter 3: Tools for Designing Emergent Cognitive Behavior (˙ θi=ωi+PN j=1 kij sin(θj−θi) ˙ kij =(αcos(θi−θj)−kij)(3.4) representing that the coupling coefficient grows fastest for two oscillators that are in phase and decays fastest for out-of-phase oscillators, as would correspond for the Hebbian rule. The speed of the coupling coefficient dynamics is going to depend proportionally on , while αis going to determine how effectively the connection between two oscillators is strengthen by their degree of synchronization. The result of this rule is a multistable behavior, in which a group of oscillators can either be stably synchronized or nonsynchronized depending on the initial conditions of their coupling coefficient. Another interesting property of this learning rule is that, if we consider a small value of , the coupling coefficients are either non-stationary and small, for oscillators operating at different frequencies, or fixed for two oscillators that are synchronized. So, as we approach to the limit →0, we can neglect all kij for nonsynchronized oscillators, and define kij =αcos(θi−θj)for synchronized oscillators. Substituting this into Eq. 4.4 we obtain that oscillators within the same synchronized cluster follow the equation: ˙ θi=ωi+ N X j=1 α 2sin(2(θj−θi)) (3.5) while for the oscillators that do not belong to a cluster ˙ θi=ωi, which would be equivalent to the Kuramoto model for the double phases. 3.4.2 Information Measures in Oscillatory Networks When studying complex systems, due to the inherent nonlinear nature of local interactions, the whole system must be simultaneously considered. In consequence, generally it is hard to know much about the relative importance of individual nodes within a system. This is the case for synchronization in oscillator networks, which usually are analyzed as a whole, being the process of synchronization somewhat like a ‘black box’, with the initial state of the system as an input and synchrony as an output. Ceguerra et al. (2011) proposed to use an informationally-based approach to analyze local dynamical process occurring in oscillator networks during the synchronization process at an individual level. Concretely, they proposed an emerging technique in complex system science: local information dynamics (Lizier, 2010). Information dynamics refers to the storage, transfer and modification of information by the elements of a system performing a distributed computation, and it particularly focuses on how these operations vary in time and space. Information storage refers to the amount of information in the past of a variable that is relevant to predicting its future. The local active information storage for a variable X is the local (un-averaged) mutual information between its semi-infinite past x(k) n= {xn−k+1, ..., xn−1, xn}(as k→ ∞) and its next state xn+1 at time step n+ 1: aX(n+ 1) = lim k→∞ log2 p(xn+1|x(k) n) p(xn+1)(3.6) Finite-k estimates are represented as aX(n, k). And the active information storage is the average over time: AX=haX(n)in. The meaning of the local active information storage aX(n+ 1) is the stored information that is currently in use by variable Xin computing its next state xn+1 at time n+ 1. Information transfer is the information provided by a source about a destination’s next state that was not contained in the past of the destination. The local transfer 32 Chapter 3: Tools for Designing Emergent Cognitive Behavior entropy from a source Yto a destination Xis the mutual information between the previous state of the source ynand the next state of the destination xn+1, conditioned on the semi-infinite past of the destination x(k) n(as k→ ∞): tY→X(n+ 1) = lim k→∞ log2 p(xn+1|x(k) n, yn) p(xn+1|x(k) n)(3.7) Again, tY→X(n+ 1, k)represents the finite-k estimates, and the transfer entropy is the time average TY→X=htY→X(n)i. The transfer entropy measures directed, dynamic flows of information, removing any stored information from being considered as transfer. Ceguerra et al. apply these measures on the time-series of differentials ˙ θi, computing how much of the change in phase each time step for a given node can be predicted from its own past (AX) and how much can be predicted from each of its neighbors that was not in its past (TY→X). Since the values of the model are continuous-valued, they computed the measures using kernel estimation (Schreiber, 2000), finding that values of k= 2 were enough for accurately computing the measures. Their results show that, during the phase transition from incoherence to synchronized behavior, information storage falls while transfer increases; reflecting decreased unilateral behavior and increased coordination between nodes. Once a synchronized state is reached, information transfer vanishes between nodes, because the nodes’ behavior is then predictable from its own past. Also, information measures show that information transfer stops much earlier than the moment when the order parameter shows that a coherent state has been reached, i.e., the distributed computation by the system which makes synchrony possible is complete much earlier than it would appear synchrony is actually achieved. Meaning that the process of synchronization just points the system in the correct direction and lets it go to run its course. 3.4.3 Kuramoto Networks as Neural Controllers Evolutionary robotics have been recognized as a useful tool in investigating biological hypotheses capturing essential elements of the brain-body-environment interactions that underlie the generation of behavior, in a way that studies of disembodied neuronal dynamics cannot achieve (Beer, 2003). The interesting point of evolutionary robotics is that, even when there are insufficient details to fully specify a system in advance, it allows the exploration of classes of mechanisms and the automatic creation of working models (Harvey et al. , 2005). There has been plenty of work on coupled oscillator networks as controllers of complex motor behaviors, particularly in the field of locomotion (Ijspeert et al. , 2005). However, to date there has been very little research on the wider issues of neuronal synchronization and phase information in the generation of embodied cognitive behaviors. One of the pioneer steps in this direction is the work of Moioli et al. (2010) in which they used a network of oscillators as the nervous systems of simulated robotic agents engaged in some minimally cognitive tasks. Their main contribution to the Kuramoto model, is to suggest how oscillatory networks can be provided with input and output mechanisms, so they are able to engage in sensorimotor brain-body-environment loops. Thus, they propose to modify the original Kuramoto equation in a way that the frequency of each node is the result of the sum of its natural frequency of oscillation with the scaled value of the sensory input related to that node: ˙ θi= (ωi+ziIi(t)) + N X j=1 kij sin(θj−θi)(3.8) 33 Chapter 3: Tools for Designing Emergent Cognitive Behavior where Ii(t)is the value of the different sensory inputs at instant tand ziis the scaling value. As well, the outputs of the system can be defined as the sine of the phase differences of two particular oscillators (Santos et al. , 2011) or as the linear combination of the sine of ensembles of phase differences (Moioli et al. , 2010). Santos et al. (2011) also showed how measures used in neuroscience for the detection on neural assemblies (especially those working in the binding problem, seen in section 2.3.2), like the measure of the moments of (quasi) phase-locking, by filtering out moments of phase-scattering, could be useful for understanding the nature of sensorimotor loops. 3.5 Synthetic Minimal Cognitive Agents for Understanding the Emergence of Sensorimotor Loops As we have claimed at the beginning of this chapter, our objective is to analyze the emergence and function of sensorimotor loops in cognitive behavior. However, due to the nonlinear and counterintuitive properties of self-organized emergent systems this is not an easy task. That is why our research approach involves to analyze the behavior of minimal models, solving simple tasks in dynamical environments, as our contribution will show. At first glance, it could seem that we are building nothing but ‘toy models’ that have little to do with real-world problems. However, as the reader will observe, even in quite simple systems can perform surprising behaviors when we design our agents as dynamical systems. Even in these simple models it is not intuitive to understand how the agent’s behavior arises, especially when we generate our models with genetic algorithms. And it is by analyzing the behavior of our agents how we are going to gain insights about the role of their different dynamical couplings. We deliberately want our models to be simple to get rid of some of the constraints we have to face in more complex environments (like robotics), and go straight to analyze the mathematical models that are behind the properties we want to study. We are not interested in practical implementations but in the general abstract mechanisms that are behind the emergence of intelligent behavior. In this sense, these simple models will be a first test to prove the consistence of the proposals we have made about the role of coordination dynamics and sensorimotor loops in the generation of cognitive behavior. We have hypothesized that cognition is built upon sensorimotor coupling generated by synchronous and metastable entrainment of neural patterns. In the next part of the thesis, these ideas will be embedded in two different synthetic models. We will present two case studies for testing the claims made in this first part. Firstly, we will explore the implications of metastable dynamics in the organization of behavior at different levels of the biological dominion, through the resolution and implementation of a model related to different instances of “intermittent behavior” in diverse living beings. In a second place, we will built an embodied and situated agent facing a simple phonotaxis task, with the objective of analyzing the role of coordination mechanisms for building a sensorimotor loop. 34 Part II Metastability and Synchronization in the Sensorimotor Loop 35 Chapter 4 Intermittency and Metastability in Organism’s Behavior The brain makes use of metastable dynamics in which neural patterns continuously fall outside their natural equilibrium state, rapidly switching from one state to another. This principle is necessary for the brain’s ability to make sense out of seemingly random environmental cues. In this chapter, we compare these intermittent mechanisms observed in the brain with the behavioral patterns of organisms that display intermittent behavior. These organisms continuously alternate behavioral patterns when they have to adapt to complex situations, similarly to metastable brain dynamics. Thinking that these phenomena are based in a common principle, we formulate an abstract model trying to capture the essence of both of them. In this way, intermittency could be a general strategy broadly used in biological organization (from neural to behavioral and cognitive levels) for adapting to unknown changing environments. This model shows how a system that maximizes its interactions with its environment and displays suboptimal solutions to a problem is more suitable for adapting to a changing environment than steady models that try to maximize their current fitness at a given time. Finally, we analyze the minimal mechanisms necessary for implementing a system that follows an intermittent strategy and we search for the conditions to embed them in the sensorimotor loop. 4.1 Intermittent Strategies As we have seen in chapter 2, the brain is a device of great complexity whose dynamics are built upon highly flexible metastable states. By living between stable (attracting) and unstable (repelling) influences, our mind can switch gracefully between distributed neural states. Rather than using an active force to destabilize and switch from one stable state to another, mental states consist in intermittent, short-lived, transient events that vanish to leave room for new ones. In this way, we could say that neural coordination dynamics does not possess any stable states at all, since they are constantly flowing from one state from another. The way the brain handles the complex task of adapting to unpredictable and changing environments is becoming intermittent. By the transient integration of numerous, distributed, constantly interacting parts of the brain, neural subsystems are flexibly engaged and disengaged to adapt to changing situations. Accordingly, phase synchronization and desynchronization allows the brain to rapidly switch among suitable mental states. However, neural dynamics is not the only domain where we can find this kind of intermittency between metastable states. We see in nature how different organisms hap- 37 Chapter 4: Intermittency and Metastability in Organism’s Behavior pen to act alternating different short-lived states at different levels of behavior. Indeed, many organisms’ behaviour is intermittent: they move, pause, and move again (Kramer & McLaughlin, 2001). We could wonder whether intermittent behavior is 1) just a epiphenomenon, i.e., a consequence derived from the intermittent nature of neural organization; 2) a consequence of physical or dynamical constrains (e.g. the muscles needing to rest after some time of activation and therefore acting intermittently); or 3) a strategy developed by organisms to face the challenge of dynamically adjusting their behaviour to changing environments. If the latter is true, we could ask ourselves if the alternation of intermittent modes of behavior could be a general strategy developed at different levels of biological organization to adapt to complex, ever-changing environments the same way as the brain does. Trying to offer an answer to this question, in this chapter we depict intermittent behavior view from the perpective of coordination dynamics. We have worked recently in what we called the adjustment-deployment dilemma (Aguilera et al. , 2011) in which an agent has to find an equilibrium between two complementary behaviors. This dilemma captures the difficult compromise between the time spent in adjusting a response and the time used to deploy it: the adjustment process improves fitness with time but it is also assumed that such fitness decays with time (e.g. environmental conditions change), if you spend very little time adjusting the fitness of the action is poor, but if you spent too much time before deployment the result is no longer valid. We review the mathematical model of the dilemma, with the objective of (1) analyzing whether intermittency could be an universal mechanism to deal with adaptation to complex situations and (2) gaining insights about how metastable regimes could be responsible for the intermittent solutions that different organisms find for adaptation to their environments. 4.2 Introduction: Intermittent Behavior and the Adjustment- Deployment Dilemma Most models of biological behavior are based on steady state assumptions, considering that actions occur at constant speeds. However, many organisms’ behavior (ranging from protozoans to mammals) is intermittent: they move, pause briefly, and move again. These pauses last from milliseconds to minutes, being part of a dynamical system by which organisms adjust their behavior to changing environments (Kramer & McLaughlin, 2001). Intermittent behavior is a widespread biological pattern. Despite the energetic costs of acceleration and deceleration, a variety of benefits arise when pauses are alternated with action. Intermittent bounding and undulating flight modes in birds (which alternate periods of flapping with pauses where wings are either extended to permit gliding or held close to the body) save mechanical power compared to continuous flight over a broad range of speeds (Rayner et al. , 2001). A similar effect takes place in fishes ‘burst-coast’ swimming (Videler & Weihs, 1982). Many species, when chasing a prey, alternate pauses and moves to stabilize their sensory field. Thus, while moves tend to be straight, both pursuits of a prey and changes of direction are initiated after pauses (Lock & Collett, 1979; Evans & O’Brien, 1988; Tye, 1989). ‘Saltatory search’ in foraging animals (from insects and lizards to mammals) minimizes the search time by alternating phases of fast motion and phases of intensive search (Anderson et al. , 1997; Bénichou et al. , 2005). Additionally, intermittent behavior has benefits that are related not that much with external physical behavior but also the dynamics of mental processes like attention to the visual field. For example, when examining the visual field, eye movement is not smooth but alternates rapid eye movements (saccades) in which perception is suppressed, 38 Chapter 4: Intermittency and Metastability in Organism’s Behavior interspersed with stable intervals (fixations) in which the brain is able to process and recognize central objects of the scene and locate peripheral objects which will be target in the next saccade (Schall & Thompson, 1999). Other examples in that sense are primates pausing briefly while moving between trees in the canopy, being the pauses related to the requirement to identify a route for the next movement sequence (Cannon & Leighton, 1994) All these examples, all along the biological spectrum, follow a common underlying pattern that combines two mutually-exclusive stages: •Adjustment would be a behavior that improves the position of an organism or increases its possibilities of making the most of its situation. This could be done by increasing potential energy during flapping, augmenting perception in a pursuit to localize the prey, moving to non-explored areas in searches, perform exploratory eye movements, or pause to process the different possible routes through the trees. •Deployment would be a behavior that takes advantage of the possibilities generated in the previous phase. Here is where our examples can keep flying without further energetic costs, moving towards the chased prey, scanning the new area, fixate the eye position for allowing mental processing of the image, or keep moving through the forest. Interestingly, the intermittency between adjustment and deployment is not a mere sequencing of complete or autonomous behavioral patterns, but poses a problem of functional coordination dynamics: How long do I have to spend gliding before I flap again? How much time do I need to spend focusing and pointing before I shoot? What is the best ratio between stopping for orientation and walking in a changing environment? A correct dynamic equilibrium between adjustment and deployment is crucial in most cases and might change under different circumstances. We have coined the term adjustmentdeployment dilemma to name a generic characterization of this problem. To our knowledge, no explicit theoretical, mathematical or simulation approach has yet explicitly addressed it. Despite the ubiquity of this intermittency between adjustment and deployment, most computational and theoretical models typically operate on two broad categories of modeling frameworks: a) continuous and situated steady behavior (e.g. the agent approaches a light source but does not stop to rest, orient or propel itself) or b) some kind of action selection or decision making procedure that operates over a perceived situation and then triggers a behavioral response (without much consideration of the temporal dimensions of the interaction). In both frameworks the temporal structure of the adjustmentdeployment dilemma is either absent (due to abstraction and simplification assumptions or due to the constrained scope of the modeled behavior) or is hidden to explicit analysis (since the focus typically remains on global task performance or specific mechanisms and procedures). 4.3 Formalization of the Adjustment-Deployment Dilemma In order to explore the adjustment-deployment dilemma we have simplified the problem to its minimal form. In general terms we have an organism adjusting its behavior (or solution to a problem) and then executing or deploying it. We can take for example the case of a toad chasing a prey, having to alternate movement with pauses for stabilizing its visual field (Lock & Collett, 1979). In the absence of obstacles the toad is going to move towards the position where the prey was just the instant before the toad started to move. Prey velocity has no influence on the direction of the toad’s movement. Also, 39 Chapter 4: Intermittency and Metastability in Organism’s Behavior when the toad is moving it is not going to correct its course if the prey is changing its position. The distance the toad walks in a single bout is going to depend on the initial separation between the toad and prey, and it is not altered if the prey vanishes or moves during the toad’s approach. Both the distance moved and the direction of the toad are not corrected by visual feedback until the toad stops its movement (figure 4.1). Figure 4.1: Representation of a toad moving intermittently while chasing a prey. The toad only can change its direction when it stops, due to the visual blurring while it is moving. In terms of our adjustment-deployment dilemma, the toad has to alternate between a ‘move’ state, where it can approach the prey, and a ‘stop’ state, in which it can stabilize the image it perceives and update the information about the prey’s position. Thus, the toad has to find an equilibrium between adjusting its orientation and deploying a pursuit behavior. We also can see how the relative amount of time expended in either state is going to depend on the dynamics of the problem. When the prey move slowly or when it is far away, the toad has less necessity of adjusting its behavior, and can move during longer amounts of time, while when the prey moves fast or it is too close, the toad has to stop and adjust its orientation more frequently, having less time for effectively moving towards the prey. Table 4.1: Minimal intermittent behavioral model: concepts Concept Notation Behavior Description Suitability f(t)Adjustment: f(t) = (1 −e−t/τ ) Deployment: f(t)=(e−t/ε) Mean ability of an organism of maximizing the achievement of its goals. Choice γ(t)Adjustment: γ(t) = γ0 Deployment: γ(t) = γ1 Binary exclusive choice of an organism between adjustment and deployment. Performance p(t)p(T) = 1 TRT 0γ(t)∙f(t)dt Mean results obtained during deployment. Optimal solution fopt(t)fopt(t) = arg max f(t) p(t)Behavior that maximizes performance. More explicitly, we have expressed the model in a series of mathematical terms, which are seen in Table 4.1. We introduce them below. 40 Chapter 4: Intermittency and Metastability in Organism’s Behavior Suitability (Fitness) It represents the mean ability of an organism of maximizing the chances of achieving its goals, i.e., obtaining a successful solution for a given problem or situation. The suitability (or the quality) of a solution in an instant tis denoted by a fitness function f(t)∈[0,1]. We will assume that: 1. The organism has an adjustment mechanism for improving its behavior over the environment. It is known the functional relation between the quality of a solution and time during adjustment. Generally, it is a nonlinear function (the effort in obtaining better results grows in relative terms with time), and we assume it to be exponential, f(t) = K(1 −e−t/τ ), where τis the adjustment speed. 2. We assume that the solution degrades throughout time as the environment changes. Also being exponential the functional dependency between quality of a solution and time, i.e., f(t) = K(e−t/ε), where stands for the degradation rate. In the case of the toad, the fitness will correspond to the difference between the prey location and the toad’s orientation. When the toad is pointing the prey fitness is 1, and it starts decreasing when the prey moves. Choice The resolution structure of the dilemma can be captured with a single variable denoted by γ(t)∈ {γ0, γ1}, that is, as the binary exclusive choice of the system over time, γ0 representing adjustment and γ1deployment. Now, the following equations to describe the behavior of the system result from the previous formalization: •Adjustment: f(t)=1−e−t/τ , γ(t) = γ0 •Deployment: f(t) = e−t/ε, γ(t) = γ1 The structure of the dilemma can thus be reduced to finding the strategy (i.e. the value of γ(t)) that obtain the better results. Performance In order to compute the quality of the obtained results by a specific choice function γ(t), we will define the evolution of the fitness over time: ˙ f(t) = 1 τ(1 −f(t)), γ(t) = γ0 −1 εf(t), γ(t) = γ1 (4.1) The agent performance will be obtained just integrating the fitness of the system during the deployment periods (the ones in which the agent is obtaining a benefit from the world, so we will take γ0= 0, and γ1= 1). Both previous functions can be combined, obtaining the global behavior equation: ˙ f(t) = −γ(t)∙1 εf(t) + (1 −γ(t)) ∙1 τ(1 −f(t)) (4.2) And the quality of the obtained results will be defined by the performance of the agent, p(T), evaluated in an interval (0, T): p(T) = 1 TZT 0 γ(t)∙f(t)dt (4.3) 41 Chapter 4: Intermittency and Metastability in Organism’s Behavior -0.5 0 0.5 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 r a Figure 4.6: Value of rfor different values of awith δω = 1,b= 0.3and Q= 0. The values are arbitrary and similar results are got for other combinations, including the situations where fluctuations are present. between state 1 and 2 (i.e., determining the value of r), while, in region B, the task is much harder. This result contrasts with the results we found both in natural data and the adjustment-deployment dilemma, where most of animals display strategies that spend much more time in one state than another, since they find more situations where the best strategy is to favor one state of behavior over another. Marginally stable system The previous situation change when we allow one or both of the previous metastable states to reach a quasi-stable or marginally stable state (figure 4.5.b). That is, we define a synchronized system whose stability will be threatened by the presence of internal fluctuations: ˙ φ=δω −asin(φ)−2b sin(2φ) + √Q ε δω, a, b 3˙ φ > ˙ φ0∀φ, ˙ φ0<0(4.8) where ˙ φ0is a arbitrary value which is relatively small compared to the noise. In this way, we put them in a situation where there is stability but any small perturbation makes the system to loss its stability (i.e., metastability is not going to be intrinsic to the system but caused by external perturbations). In this case, we found that transitions are not periodic anymore, but noise-driven. a) b) -0.2 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 r a -0.1 -0.05 0 0.05 0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 r a Figure 4.7: a) Value of rfor different values of awith δω = 1,b= 0,45 and ε= 0.4. b) Value of rfor different values of aand δω = 1,b= 0.5and ε= 0.4. 48 Chapter 4: Intermittency and Metastability in Organism’s Behavior As we increase the value of bwe see that both metastable states approach the region of stability (and eventually one or both of them will fall into it if we use the correct value of a). Our strategy will be the following: we will try increasing values of bwith its corresponding span of values of a, computing the shape of the corresponding rfunction. Along the process, we will analyze how the appearance of the rfunction changes. The results reveals that rchanges its form from the tangential function we had before to a sigmoid function, with an almost linear function for intermediate values of b. In this new situations, we can see that the properties of region Aand region B change drastically (figure 4.7). Now, the graphs can show the same kind of sigmoidlike functions that we observed in the solution of the adjustment-deployment dilemma (figure 4.7.b). Therefore, the proposed intermittency behavior based in a marginally stable mode seems to account for the results of the adjustment-deployment dilemma, being it a more suitable choice than pure metastable modes for modeling intermittent behavior. 4.7.2 Sensorimotor controller Once we have characterized a switching behavior with enough precision, we need to model a controller to trigger alternation of behavioral patterns by driving the HKB model to the desired modes of behavior. However, we will simplify our model to a one-output network which decides whether the agent has to deploy or adjust. We leave apart the task of converting these choices into parameters of the HKB model. We used continuous-time recurrent neural networks (CTRNNs, see chapter 3) in order to implement a dynamical system capable of developing the optimal strategy in any possible situation. CTRNNs are a good choice for the proposed task because (1) they are the simplest nonlinear, continuous dynamical neural network model; (2) despite their simplicity, they are universal dynamics approximators in the sense that, for any finite interval of time, CTRNNs can approximate the trajectories of any smooth dynamical system (Beer, 1995a). The general form of a CTRNN with Nneurons is: ˙yi=1 τi (yi+ N X j=1 wijσ(gj(yj+θj)) + Ii)(4.9) where i= 1,2, ..., N,yis the state of each neuron, τis its time constant (τ > 0), wij is the strength of the connection from the jth to the ith neuron, θis a bias term, gis a gain term, σ(x)=1/(1 + e−x) is the standard activation function, and Irepresents a constant external input. In this case, the only knowledge the network is coding about the world is the current quality of the solution being implemented, i.e. I=Kf∙f(t), where Kfis a gain term. One of the neurons (e.g. i=N) was considered as the output of the system. This output will determine the values of γ(t), and therefore the following f(t). γ(t) = 0, yN(t)≤0 1, yN(t)>0(4.10) Once the neural networks were defined, by using a genetic algorithm we find a network that develops an optimal behavior, tending to select with more probability the networks that achieve a higher value of p(T). Adaptation Without Learning The objective is to obtain a system which is able to display a suitable behavior for a given dynamics. In the first place, we define the τ(t)and (t)functions, which represent 49 Chapter 4: Intermittency and Metastability in Organism’s Behavior the dynamics of the world at each moment. These functions will determine the value of f∗(t). For the training, we tried a situation with constant rates: τ(t)=1, (t)=1, f∗(t)=0.5was selected. Given these dynamics, the genetic algorithm was executed for various sizes of neural networks. The result showed that even for CTRNNs with N= 1 (one single neuron), the network was able to obtain the optimal results for the given dynamics (Figure 4.8). 0123456789 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 f(t) t Figure 4.8: Fitness function f(t)for f∗(t)=0.5(red dashed line). Response of a singleneuron network. From now on, it was taken the single-neuron network adapted to a constant situation of f∗(t)=0.5without any change. The following objective was to observe the response of this neuron to environments with dynamics which were different than the one of the training. The experiment consisted in defining f∗(t)as a) a ramp function, b) a step function, c) a triangle wave and d) f∗(t)as result of defining τ(t)and (t)as sinusoidal functions. We obtained that the neuron was able to adapt to any of these changing dynamics obtaining an optimal performance (i.e., with f(t)≈f∗(t)) as seen in figure 4.9. Therefore, the mechanism implemented for a single neuron for adapting to a particular world dynamic (f∗(t)=0.5) was able to adapt to any other smoothly changing dynamics without any further training. The same results were observed when the genetic algorithm obtained in the first place a neuron adapted to any other situation different to f∗(t)=0.5. The resulting neuron was always able to adapt to the new changing forms of f∗(t). System Behavior The robustness of this result was achieved because of the resulting structure of the neuron and its consequent behavior. Its, behavior is based on the coupling between the external (the environment) dynamics and the internal (the neuron’s) dynamics of the system. The system external dynamics were represented by the variations of the fitness function f(t), that is, ˙ f(t)(representing the effect of the agent behavior on its own situation in the world). Similarly, we took ˙y(t)for representing the system internal dynamics, determined by the variations of the internal state y(t). With the purpose of seeing intuitively the effects of the different dynamics ( ˙ f(t)and ˙y(t)are quite spiky functions), the systems dynamics were represented by the variables ˙ fm(t)and ˙ym(t), being the filtered moving averages of ˙ f(t)and ˙y(t). According to these parameters the neuron behavior could be explained at different levels: 50 Chapter 4: Intermittency and Metastability in Organism’s Behavior a) b) 0 5 10 15 20 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 f(t) t 0 5 10 15 20 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 f(t) t c) d) 0 5 10 15 20 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 f(t) t 0 5 10 15 20 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 f(t) t Figure 4.9: Fitness f(t)of the resulting neuron, without any further learning, tried for different world dynamics: a) f∗(t)as a ramp function, b) f∗(t)as a step function, c) f∗(t)as a triangle waveform and d) fopt(t)as results of defining τ(t)and (t)as sinusoidal functions. a) b) c) -1 -0.5 0 0.5 1 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 ˙y(t) y(t) -1 -0.5 0 0.5 1 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 ˙y(t) y(t) -1 -0.5 0 0.5 1 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 ˙y(t) y(t) 2 4 6 8 10 12 14 16 18 20 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 y(t) t 2 4 6 8 10 12 14 16 18 20 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 y(t) t 2 4 6 8 10 12 14 16 18 20 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 y(t) t Figure 4.10: Values of ˙y(t)and y(t)when a) f(t)=0.5, b) f(t)=0.25 and c) f(t)=0.75. y(t)≷0determines whether the neuron generates or executes a solution (notice that in each case rdep =f(t)). Therefore the functions describe a case where the neuron a) generates solutions as much times as it executes them, b) generates solutions more times than it executes them and c) generates solutions less times than it executes them. 51 Chapter 4: Intermittency and Metastability in Organism’s Behavior a) b) 0123456789 0 0.5 1 f(t) t 0123456789 -5 0 5 10 15 ˙ fm(t) t 0 5 10 15 20 -2 0 2 4 6 ˙ym(t) t 0 5 10 15 20 0 0.5 1 f(t) t 0 5 10 15 20 -5 0 5 10 ˙ fm(t) t 0 5 10 15 20 -2 0 2 4 6 ˙ym(t) t Figure 4.11: Values of the system fitness f(t), the system external dynamics ˙ fm(t) and the system internal dynamics ˙ym(t)for different situations. The different situation dynamics are defined by the value of f∗(t), represented by the red dashed line. 1. When f(t)≃f∗(t)(figure 4.10), the neuron feedback loop is able to compensate the output deviations. It makes the neuron to behave like a nonlinear oscillator around f∗(t). 2. If f∗(t)shows a constant value, but fitness is not at this optimal value, i.e., f(t)≷f∗(t)(figures 4.11.a), then the system tends to f(t) = f∗(t). Internal and external dynamics ( ˙y(t)and ˙ f(t)) act together in order to adapt fitness to its optimal value. 3. In the last case, when f∗(t)is changing throughout time (figure 4.11.b), the following happens. If f∗(t)changes, that means that the world dynamics (i.e. the adaptation and degradation rates) are changing, therefore ˙ fm(t)changes and f(t) is no longer around f∗(t). Nevertheless, the system dynamics ˙y(t)changes in reaction to the changes in ˙ fm(t), counteracting them. This will recover the equilibrium of the system in a new point, which will be f(t) = f∗(t). As seen, the system is able to act in two different time levels. On the first one the agent can respond to transient changes of f(t), keeping the fitness at its optimal value, alternating adaptation and deployment. On the second time level (slower than the first one) the agent can adapt its average fitness value to f∗(t), expanding the adaptive opportunities of the agent. Nevertheless, these results are preliminary, and more consistent proof of this hypothesis should be obtained from simulation in which both the intermittency mechanisms and the sensorimotor controller should be engaged in embodied and situated agents performing task in an intermittent way. 52 Chapter 4: Intermittency and Metastability in Organism’s Behavior 4.8 Metastable Mechanisms for Intermittent Behavior In this chapter we have depicted an essential aspect of intermittent behavior, namely the adjustment-deployment dilemma, the dynamic interplay between the time spent on adjusting a solution to the environment or bodily circumstances before deploying it, and the execution time taken by the deployment of the solution. Our hypothesis is that intermittent behavior is closely connected with the metastable dynamics observed in the brain, being both part of a more general framework in which intermittency is the mechanism used to couple a system’s dynamics with the dynamics of a complex and ever-changing environment. Despite its ubiquity in biological behavior, to our knowledge, this is the first characterization, formalization and modeling approach to the adjustment-deployment problem. We have formalized mathematically the structure of this dilemma and numerically computed its optimal solution for different values of the problem-structuring parameter which turns out to be the ratio between speed of adjustment and speed of the adjusted solution decay while deployment takes place. The optimal solution to the adjustment-deployment dilemma, for fixed ratio between increasing quality of adjustment and decay-rate while deploying, turns out to require a compromise with non-maximal quality and a high rate of alternation between adjustment and deployment. The distribution of optimal strategies over the range of parametric values takes a sigmoidal shape, meaning that, overall distribution of solution should show many instances of biological behavior where adjustment is very fast and longer periods of deployment are present or the contrary; i.e. long periods of adjustment followed by quick deployment. The distribution of intermittency in animal behavior seems to match our model’s optimal solution distribution. But what are the mechanisms capable to achieve the optimal solution under changing conditions? A CTRNN composed of a single node was shown to be capable of achieving this optimal solution, being its input an indicator of the success of its deployment. The results suggest that optimal solutions to the adjustment-deployment dilemma could, in principle, be instantiated on very simple mechanisms, if the appropriate conditions occur, and should therefore be accessible even to unicellular systems. We also present some results that show how switching between complementary patterns can be modeled by an HKB model operating in a marginally stable state, better than pure metastable functioning. Thus, systems posed in between the border of stable synchronized and metastable states would be in an optimal situation to adapt to changing environments. Needless to say the present model is still in need of further development. Some of the underlying assumptions should be relaxed and the model complexified. For instance, many crucial temporal aspects of the adjustment-deployment dilemma were left aside in this study and many of them might provide avenues for future research. The inclusion of forced perceptual delays, evaluation delays (the organisms need to take some time to taste a food source, or to evaluate the outcome of its interaction), the possibility of overlap between adjustment and deployment, constraints on deployment duration, etc. should be included in further development. The measurement of fitness and quality of solution could also be enriched by including additional cost function to deployment (energy expenditure), adjustment (risk of being detected/hunted) or associated with the switching between the both of them, and a variety of spatial and embodiment constraints. Future development should also include reference-to and modeling-of specific examples of animal behavior that face the adjustment-deployment dilemma, compare the model to existing data and include the necessary adjustments on parameters and, most probably, add more dimensions to the problem. 53 Chapter 4: Intermittency and Metastability in Organism’s Behavior 4.9 Metastable Adaptive Behavior We have shown that, when faced with the adjustment-deployment dilemma, we gain more by assuming a compromise with a suboptimal quality solution and maximizing our interactions with the environment. Recurrent testing of our solution into the environment renders betters results than indefinitely adjusting our models of the solution. These results show how metastable dynamics can be used as a general adaptive tool when facing uncertain environments. Due to its generality, the presented abstract model could be used at any level of biological organization: from neural dynamics to behavior and higher cognitive functions. We have also shown how very simple mechanisms can find solutions to relatively complex problems, illustrating how a wide range of living systems could successfully cope with domain invariant adaptive problems like the adjustment-deployment dilemma. Since intermittent behavior is found in organisms such as protozoans which do not possess a neural system, we hypothesize that metastable adaption could be a principle of minimal cognition. The generality of the model and the robustness of our results support this idea (van Duijn et al. , 2008). Our model also brings forth the necessity to include the temporal dimension of cognitive processes into our theoretical framework. Speed, intermittency, decay-rates, and deployment duration crucially matters when it comes to real-world problem solving. Including these temporal features into the sensorimotor loop, we obtain a highly robust adaptive behavior. However, how is this temporal dimension successfully embedded in the sensorimotor loop? How can organisms exploit the coupling between temporal patterns of both internal and external dynamics? The next chapter implements a minimal model of an agent performing a phonotaxis task, exploring how robust behavior can result for a sensorimotor loop which embeds critical temporal features of the interaction with the world. 54 Chapter 5 Robust Coordination in the Brain-Body-Environment System In previous chapters, we have claimed that coordination dynamics is responsible for the emergence of behavioral patterns which create sensorimotor loops allowing robust and flexible interactions with the world. This leads us to wonder about how dynamical coupling in the brain-body-environment system may face the influence of internal and external perturbations on the actions of an agent. Some other works in theoretical neuroscience and systems biology consider that robustness is just internally generated (i.e. a property of the internal structure of organisms). However, the dynamical approach we are proposing in this work opens us the door for exploring other perspectives about robustness, which intend to analyze it as a dynamical phenomenon of coupling in the sensorimotor loop cutting across the brain-body environment system (Fernandez-Leon, 2010). From this point of view, in this chapter we analyze robustness in a simple taxis task, in which an agent has to perform a behavioral response to a directional stimulus or gradient of stimulus intensity. Analyzing previous models of oscillator networks performing similar tasks, we hypothesize that the necessary condition for robust sensorimotor loops to emerge is that both the neural system and the processed stimulus share the same metrics in order to create a structured behavioral space. 5.1 Emergence of Robustness in the Sensorimotor Loop When we decide what type of neural controller we use for modeling cognitive agents, it could make a difference what kind of predictions our system is generating. Concretely, feed-forward and feedback networks are good at predicting what is going to happen next, while oscillator networks are a good choice for predicting when things happen (Buzsáki & Draguhn, 2004). That is, we can think that, while feed-forward and feedback networks are good at processing input information expressed in terms of amplitudes of input and output parameters, oscillator networks should easily deal with relative phase information. We can see this clearly in the work of Santos et al. (2011), where they evolve a physical agent controlled by a Kuramoto network to perform a phototaxis task, where the input of the network is the the light intensity received by two sensors; using the model seen in section 3.4.3. We can see how they achieve the phototaxis behavior, but the agent movement is somehow awkward and it breaks (becomes unstable) when the agent is too close to the light source. This does not happen when agents are controlled by other type of neural models, as CTRNNs. Also, when the sensorimotor loop is perturbed by temporal delays, it easily dissolves making impossible the phototactic movement. 55 Chapter 5: Robust Coordination in the Brain-Body-Environment System This leads us to think that robustness may be enhanced when the perceived stimulus and the neural controllers share the same metrics and process the same type of information. Kuramoto networks may not be the best option to compute absolute amplitude information or amplitude gradients from the environment, because their metrics are different from the ones of oscillator phase differences. Maybe they could play a more interesting role if we use them to solve phototactic-like behavior just using phase, or phase-like, information. These ideas link directly with the fact that human comprehension of the different signals (voice, image, etc.) that we receive depends more in the phase information of the signal than the amplitude information (Oppenheim, 1981), since phase represents locations of different elements in a signal (either in space or time), leaving apart their amplitude. Since the brain is conformed by multiple oscillating processes, it make sense to think that we are especially prepared to easily compute phase information from the environment. For the following sections we are going to design a phonotaxis task, where an agent has to approach an object just perceiving information about the phase of a wave which is sent from the agent and echoed in the object. We will see how the processing of relative phase information allows the formation of beautifully simple and robust sensorimotor loops with Kuramoto oscillator networks. 5.2 Oscillatory networks as a neural controller in a minimal situated phonotaxis task In the first place, we designed an scenario with a minimal task to start analyzing sensorimotor loops in oscillatory networks. The model consists or a two dimension simulated environment, an agent and an object the agent must reach. The agent’s movement is controlled by a network of four coupled oscillators, being one of them connected to a sensor. The sensor sends an acoustic wave to the environment, which is going to be reflected in the goal object coming back to be detected by the sensor. The phase of the oscillator connected to the sensor is going to be the same than the phase of the travelling wave that is reflected from the object to the agent and received by the sensor. Therefore, we define the oscillator network as: ˙ θi=     ωi+P4 j=1 kij sin(θj−θi), i = 1,2,3 ωi−2π λ2˙ d, i = 4 (5.1) where θiis the phase of the ith oscillator, ωiis the oscillator’s intrinsic frequency, Nis the number of oscillators and kij is the coupling factor from the jth to the ith oscillator. Also, dis the distance between the agent and the goal. Thus, the phase of the 4th oscillator is going to be the phase of the acoustic wave the reflected acoustic wave travelling from the goal to the agent. The agent (figure 5.1) is modeled as a solid circular body radius R= 0.1with two diametrically opposed motors, which can move forward or backwards with different velocities. The agent’s mass is quite small (there is no inertial resistance) so the motor output is the tangential velocity at the point of the body where the motor is located. The translational movement can be calculated using the velocity of its center of mass (the vectorial average of the motor velocities), and the rotational movement by calculating the angular speed (difference of the tangential velocities divided by the body diameter). The activation of the motors is going to be defined as Rsin(φ12)for the right motor and Rsin(φ13)for the left motor, being φij =θj−θi; that is, the sine of the relative phase 56 Chapter 5: Robust Coordination in the Brain-Body-Environment System Figure 5.1: Schema of the two wheeled physical agent controlled by a Kuramoto network of coupled oscillators connected to an acoustic sensor. between oscillators 1and 2activates the right motor and the same for the relative phase of oscillators 1and 3and the left motor. Rwill be a gain factor An elitist genetic algorithm was used to obtain the parameter values that allow the agent to successfully perform the phonotaxis task. The algorithm used 14 genes of 5 bits each, defining the values of ωi(range [0,5]), kij (range [0,5]) and 2π λ(range [0,15]). Moreover, it was used a population size of 80, with a trial length of 400, selecting the 20 fittest individuals and recombining the rest with a mutation rate of 0.01. The fitness function was defined as the combination of two factors F=FD+FP, where FDrepresents the reduction of the distance to the goal between the initial and final position of the agent FD= 1 −df di, being FD= 0 if df> diand FPrepresents the amount of time the agent at a distance less than 10 body lengths to the goal (i.e., a distance of 1). The position of the goal is generated randomly. In 8generations the agent successfully performed the phonotaxis task (figure 5.2), obtaining fitness values of FD= 0.96 and FP= 0.72. -1 0 1 2 3 4 5 -1 0 1 2 3 4 5 Figure 5.2: Trajectory of the agent reaching an object in a phonotaxis task. The agent does not follow a straight line but follows a slight detour. 5.2.1 Dynamical Behavior Analysis We see that the problem was solved performing what apparently seems as a quite simple behavior. However, if we look more carefully what is going on it is not trivial. On a first approach for solving a phonotaxis task, one may think that, at least, two sensors located at different positions are needed to correctly determine the position of an object, as used in previous phonotaxis models (Webb & Harrison, 2000), so the relative phase of the received waves reflects the direction of the sound source. If we have only one sensor, the only way to gain some information about the environment is to move, thus modifying the frequency of the perceived wave because of the Doppler effect. This gives the agent information about its relative speed respect to the goal. However, moving towards the goal is does not give the agent any information about its relative position respect to the 57 Chapter 5: Robust Coordination in the Brain-Body-Environment System with the intermittent cycle, the sensorimotor loop is going to emerge again. Also, this effect could be used as a band-pass filter, because it only takes the sensory information that is coupled within the sensorimotor loop. 012345 -2 -1 0 1 2 3 0 50 100 150 200 250 300 350 400 0 1 2 3 4 5 6 d t Figure 5.9: Trajectory of the agent chasing a moving goal distance dto the goal over time. When the agent is moving, it loses sight of the goal, so it has to adjust its behavior by turning. Adjustment-deployment dilemma: chasing a moving goal We have identified that the agent implements a metastable behavior when it cannot maintain its internal stability. We have linked this fact with the intermittency in the adjustment-deployment model we studied in chapter 4. Is it the same phenomenon? Or it is just a coincidence that both models show an intermittent behavior? For answering to that question we have modeled a new scenario for the same agent we have been using so far. The new situation is that the agent has to reach a moving goal. The problem the agent has to solve is that, when it is chasing the goal and the goal is moving in the opposite direction, the relative speed of both get slower than when the object was still, making the agent not to perceive correctly the received wave and lose sight of the goal. However, the agent is going to be able to adjust its situation by turning and regaining the sight of the agent (figure 5.9). As in the previous cases, the deployment of the approaching movement is interspersed with turning behavior to adapt to the distortion in the received signal. -1 0 1 2 3 4 5 6 -5 -4 -3 -2 -1 0 1 2 Figure 5.10: Trajectory of the agent chasing a moving goal which moves with two different velocities. At the middle of the simulation the goal velocity is reduced to the 30% of the original velocity, resulting in a change of the movement of the agent. 64 Chapter 5: Robust Coordination in the Brain-Body-Environment System Is this really an adjustment-deployment strategy? If so, the adjustment-deployment intermittency should be adapted to the dynamics of the environment. We prepared a new simulation, in which the speed of the goal changes at some point. Concretely, at t= 250, the speed of the goal is reduced to the 30% of the original speed. In this case, we see in figure 5.10 how the agent modifies its behavior with the change of velocity. After the velocity change, the distortion of the signal perceived by the agent is smaller, thus allowing the agent to deploy the ‘approaching the goal’ behavior for more time until it has to adjust its situation by turning. The behavior displayed shows a typical characteristic of the adjustment-deployment dilemma: when the environment changes slowly, the agent spends more time in ‘deployment’ than when the world changes faster and more adjusted solutions to the problem are needed. 5.4 Codification of Distance Information in the Relative Phase We have designed an agent that is able to perform a phonotaxis task by building a sensorimotor loop that allows the agent to know the position of the goal. If this is true, the agent has to somehow code the information about the goal within its oscillatory neural controller, giving rise to the emergence of a representation. As it has stressed before (O’Keefe & Recce, 1993; Singer, 2007, see section 2.3) representation in oscillatory networks might be coded in the phase relations of the elements of the networks with a precision in the millisecond range. Figure 5.11: Representation of the variables d, which represents the distance to the goal, and α, which represents the angle between the goal and the agent’s orientation. If the agent is able to build a sensorimotor loop gaining information about the location of its goal, we ask ourselves if the information is encoded in the agent’s internal neural dynamics. For answering that, we are going to analyze how the relative position of the agent affect the phase of the oscillator network. We analyze the variables dand α(figure 5.11), being dthe distance from the agent to the goal and αthe normalized angle from the heading of the agent to the goal, and how they are related with the relative phases of the system. For the following analysis, we take the variable φ12, the relative phase between the 1st and the 2nd oscillator, but any other choice of relative phase would give us similar results, since they are all phase-locked in the default situation. The first experiment we have run tests if the phase representation always codifies the same situation in the same way. For that, we run different simulations with different starting points for the agent. If we represent the trajectories of the relative phase respect 65 Chapter 5: Robust Coordination in the Brain-Body-Environment System 012345 0 0.2 0.4 0.6 0.8 1 φ12 d 1 2 3 4 5 6 0 0.5 1 φ12 α Figure 5.12: Plot of the relative phase φ12 for 10 different trajectories of the agent. It is seen how all trajectories converge to a unique one. Also, it is seen how the same distance information is codified in a different way depending on the state of the neural oscillator controller. to the values of dand α(figure 5.12), we can see how all the trajectories converge into a unique path, which uniquely represents the distance and angle information for a concrete state. We also see that information is codified in different ways depending on the state of the agent. In this sense, if we analyze the tridimensional plot of the relative phase respect to dand α(figure 5.13), we can distinguish a first stage when the agent is far from the goal and it is just orienting itself and turning to face the goal; a second stage where the agent starts approaching the goal, and a third stage where the agent start turning to reach the goal. In the first stage, the distance to the goal does not change much, but we see that the orientation of the agent changes as it turns, while the relative phase also changes. Concretely, as the agent approaches to the goal (α≈6), the relative phase increases to φ12 ≈1. In the second stage, it is the orientation which is kept constant, while the distance decreases. Phase representation is kept also constant while the agent is far from the goal (d > 2), but when the agent is closer to the goal (d < 2) the phase information starts changing, while a slight change in the orientation αtakes place. These changes become bigger the closer the agent is to the goal. So, in this short range representation not only the orientation angle but also distance to the goal is codified in phase information. Finally, in the third stage, the agent shows an undulating trajectory until it stabilizes in a point where it is continually turning around the goal at a close distance. We have seen how relative phase is able to represent the orientation information of the agent in the different stages of behavior. Additionally, phase information can codify distance information in the short range. That makes sense if we consider how the agent is perceiving information from the world. The agent can detect the acoustic waves coming from the goal. And, given the size of the agent, when it is far enough we can consider that the agent is perceiving a linear wavefront, so it has no way to know what the distance to the goal is, and it can just know if it is approaching or not. However, when the agent is closer to the goal, the linear wavefront approximation do not hold anymore, and it can perceive not only ‘where’ but ‘how far’ the goal is (figure 5.14). 66 Chapter 5: Robust Coordination in the Brain-Body-Environment System 0 2 4 6 0 2 4 6 8 0 0.5 1 1.5 d α φ12 0 1 2 3 4 5 0.2 0.4 0.6 0.8 1 1.2 φ12 d 1 2 3 4 5 6 7 0 0.2 0.4 0.6 0.8 1 1.2 1.4 φ12 α Figure 5.13: Plot of the relative phase φ12 for the trajectory of an agent performing phonotaxis. Relative phase codifies aspects of both distance (d) and orientation angle (α) Figure 5.14: Schema of how making a detour allows the agent to embed knowledge of how close is the goal. When the agent is far away, it perceives ‘plain’ wavefronts. Waveforms start curving as the agent starts getting closer to the goal. 67 Chapter 5: Robust Coordination in the Brain-Body-Environment System Intermitency and information coding If we perform the same analysis than before when the agent behaves intermittently, the result is not too different. Again the orientation of the agent is codified in the relative phase as seen in figure 5.15. Nevertheless, since the agent is deploying a cycloidal behavior, it cannot process information about its relative distance to the goal because the instability of the ‘approaching to the goal’ behavior that allowed to compute it. Thus, relative phase can only code information about the agent’s orientation. It is worth to note that the coding of orientation information is performed in the same way than in the default unperturbed case. 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 φ12 d 0 1 2 3 4 5 6 7 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 φ12 α Figure 5.15: Plot of the relative phase φ12 for a cycloidal trajectory of an agent performing phonotaxis. Relative phase codifies aspects of orientation angle (α). Distance (d) is no longer codified. 5.5 Phase-Locking and Metastability in the Sensorimotor Loop We have proposed that dynamic coupling between the parts of the brain, and between brain, body and environment allow the emergence of robust behavioral patterns, being this coupling modeled by non-linearly coupled oscillators. In chapter 2 we have defined two different mechanisms to model interaction of brain parts (Tognoli & Kelso, 2009). 1. The first mechanism is synchronization of brain dynamics, which is defined as the formation and dissolution of attractors that bind local oscillations into phase locked states. These regimes are hypothesized to correspond to transiently synchronized neural cell assemblies (Varela, 2006). 2. The second involves forms of coordination different that phase-locking, defined as metastability in brain dynamics, represent the binding of areas with different intrinsic oscillatory properties which are just partially coordinated. Here, oscillatory patterns show dwelling tendencies in a quasi-phase-lock intermittent mode. This mode of behavior is more robust and allows flexible and versatile behavior (Kelso, 1995). We have built a simulation model that shows both kind of dynamics for solving a phonotaxis task, making us to think about which is the role of synchronization and metastability in cognitive behavior and how these two are related. We start with synchronization. In the first agent that is obtained from the genetic algorithm, when performing the original unperturbed phonotaxis task, we saw that the 68 Chapter 5: Robust Coordination in the Brain-Body-Environment System neural controllers of the agent showed a behavior of full phase-locking with different phase relationships. Typically, neurophysiologist have centered their attention in zerolag synchronization, although some studies have addressed other phase relationships (Womelsdorf et al. , 2007; Palva & Palva, 2009). Tognoli & Kelso (2009), in addition to inphase and antiphase synchronization, consider near inphase or near antiphase entrainment in brain synchronization dynamics as important neural coordination states. In the model simulated here, we can see (figure 5.3) how phase-lockings other than inphase or antiphase can be important for the processing of the information in the sensorimotor loop, being able to code different states of the brain-body-environment system. As well, having a continuous space of phase relationships will be useful for coding environmental information. Let’s continue with metastability. Other result show in our model is that, in the sensorimotor loop, not only the neural dynamics determine behavior, but perception shapes the form of neural dynamics. The coupled sensorimotor loop arises as an order parameter that channels the interactions of the elements of the system, constraining its degrees of freedom. For the unperturbed situation, the sensorimotor loop allows a stable behavior of the agent. Anyway, the unperturbed sensorimotor loop is situated at the border of stability. When the stability of the loop is broken, metastable behaviors arise. These metastable behaviors allow the agent to explore different strategies to solve the problem, showing a more robust behavior than the stable initial one, having all the properties seen in chapter 4. Thus, stable synchronization and metastable dynamics appear closely linked. We see in our model how the most efficient and robust behavior is the one that lives between stability and metastability, which can show a synchronized predictable behavior when the environmental conditions allow it, and can deploy a metastable intermittent behavior when it is needed a more flexible adaptation mechanism. 5.6 Robustness and Behavioral Pattern Structuring In sensorimotor loops, while neural dynamics determine behavior, perception is going to shape the form of neural dynamics. This is one of the hallmarks of sensorimotor loops, which allows the emergence of robust behavioral patterns. As we see in our model, the received acoustic wave stabilizes the system in a phase-lock state. Also, the behavior of the agent is going to stabilize the Doppler effect of the received signal in the neural controller. And, if we analyze the metastable modes of behavior, we see that behavior and the signal are intermittently trapped in quasi-phase-locking modes. There could be four different modes of behavior for our agent: a) attraction to the goal (phonotaxis), b) orbiting at a constant distant to the goal (the behavior we see when the agent reach the goal), c) repulsion from the goal (the opposite to phonotaxis), or d) motion independent to the goal. Just in the latter case, which is also the one that allows a larger variety of behaviors, the agent would not be engaged in any sensorimotor loop, and there will not be any phase-locking. Thus, if we oblige the system to be stable while having an input from the goal, it is always going to engage in any sensorimotor loop. Indeed, if we modify the natural frequencies of the three oscillators of our system (all of them except the sensor oscillator) and we keep high values for its coupling factors, we can see how the behavior of the agent always lies in the the ‘a)’, ‘b)’ or ‘c)’ modes of behavior (where ‘b)’ is just a transition in the continuum between ‘a)’ and ‘c)’), either showing synchronous or metastable behaviors (figure 5.16). In this case, when the cases which still maintain high values of motor activation, we show how the agents perform surprising spiral movements towards or away from the goal (figure 5.17.a). If, on the 69 Chapter 5: Robust Coordination in the Brain-Body-Environment System Figure 5.16: Table of possible behaviors when the agent is engaged in a sensorimotor loop with random natural frequencies for the oscillator network and strong coupling factors. The movement of the agent can be either stable or metastable, and approaching or moving away from the goal a different speeds. The possible behaviors also can show different degrees of curvature. other hand, we cut the connections from the sensor oscillator, we see how the movement is independent from the goal, and the system becomes unstable (figure 5.17.b). We see how sensorimotor loops not only allow adaptive behavior to arise, but determine what behaviors are actually possible (or not) for the agent. We can easily see how coordination dynamics in the sensorimotor loop is a great advantage for adaptive behavior: they force the agent to implement, from all the set of possible patterns of behavior, only those behaviors that are meaningful respect to its environment and allow a mutual stabilization of the perceived stimulus and neural dynamics. If our behavior is constrained to the set of behaviors seen in figure 5.16, it is much easier to find the parameters of the neural controller that perform the desired behavior just by a random search than in the case where we have no behavioral constrains. In conclusion, any type of coupling with the environment is not enough for guaranteeing a robust behavior. Instead, robustness emerge when the nature of coordination in the sensorimotor loop organizes the behavioral options of the agent into those that maintain the internal stability of the system. As we have seen, in such structured sensorimotor loops, the agent is able to display a highly robust behavior just by assuring its internal stability. Thus, synchronization and metastable dynamics can be seen as mechanisms to impose the necessary internal stability in structured sensorimotor loops to perform robust structured behavioral patterns. 70 Chapter 5: Robust Coordination in the Brain-Body-Environment System a) b) -30 -20 -10 0 10 20 30 -30 -20 -10 0 10 20 30 -2 -1 0 1 2 3 4 5 -2 -1 0 1 2 3 4 5 0 500 1000 1500 2000 2500 3000 3500 4000 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 r t 0 50 100 150 200 250 300 350 400 0 0.2 0.4 0.6 0.8 1 r t Figure 5.17: a) Agent with modified oscillator natural frequencies moving away from the goal and keeping its internal state stable. b) Agent with removed sensor oscillator connections, moving independently of the goal and having an unstable internal state. The stability of the sensorimotor loops structures what kinds of behaviors are possible to perform for the agent. 71 Chapter 6 Sensorimotor Coordination for Adaptive Behavior In this last chapter, we recap what we have seen throughout this work. First, we started refusing some assumptions of the classical artificial intelligence approach, which overrelies on world-modeling and planning capacities that often are completely uncoupled to the agent’s interactions with the world. Consequently, we proposed the ‘dynamical approach’ approach to cognitive science as a more realistic perspective to analyze and explain the emergence of cognitive behavior, by defining cognitive agents as dynamical brain-body-environment coupled systems. This approach is based on how the deployment along time of coordinated neural patterns sustains the emergence of cognitive functions. Assemblies of neurons entrain themselves in coherent transient states, integrating information between different functional sectors by rapidly engaging and disengaging parts of the neural system, at the same time they interlock in coupled interactions with the environment. In the approach we have presented, we have pointed out what we considered as two of the main hallmarks of cognitive emergence: coordination dynamics and sensorimotor loops. Coordination dynamics inspires on the organization of pattern activity in the brain to define a mathematical framework for analyzing the formation and coordination of patterns in complex self-organized systems. It tries to understand the role of oscillatory networks in the brain to maintain its autonomy and coherence while preserving the versatility of self-organized criticality. We have introduced two types of dynamics that the human brain uses to coordinate the oscillating functional parts that compose it: •Synchronization. It enables the brain to create (transiently) stable states which establish phase relations that are going to define functional cell assemblies. These assemblies would be responsible for the emergence of cognitive functions. •Metastability. When a stable coordination of different nodes of the brain is not stable or cannot cope with the requirements form the environment, the synchronized regime breaks apart and metastability arises. However, some attraction will remain at the points of the phase space where attractors where placed, giving place to relative coordination between oscillating nodes. This dynamics allows rapid switching between cell assemblies formation and flexible engagement and disengagement between functional parts. However, despite the great advances brought by neurophysiological studies about the dynamics of the brain, we consider that this research line has showed some limitations 73 Bibliography Braitenberg, V. 1984. Vehicles: Experiments in synthetic psychology. Cambridge, MA: MIT Press. Breakspear, M., Heitmann, S., & Daffertshofer, A. 2010. Generative models of cortical oscillations: neurobiological implications of the kuramoto model. Frontiers in human neuroscience,4. Buzsáki, G. 2006. Rhythms of the Brain. Oxford University Press. Buzsáki, G., & Draguhn, A. 2004. Neuronal Oscillations in Cortical Networks. Science, 304, 1926–1929. Bénichou, O., Coppeya, M., Moreaua, M., Sueta, P. H., & Voituriezb, R. 2005. A stochastic theory for the intermittent behaviour of foraging animals. Physica A,356(151). Cannon, C. H., & Leighton, M. 1994. Comparative locomotor ecology of gibbons and macaques: Selection of canopy elements for crossing gaps. American Journal of Physical Anthropology,93(4), 505524. Ceguerra, R., Lizier, J., & Zomaya, A. 2011. Information storage and transfer in the synchronization process in locally-connected networks. Proceedings of the IEEE Symposium Series on Computational Intelligence 2011. Clark, A. 1997. Being There: Putting Brain, Body, and World Together Again. MIT Press, Cambridge, MA. Clark, A. 1999. An embodied cognitive science? Trends in Cognitive Sciences,3(9), 345–351. Damiano, L., & Cañamero, L. 2010. Constructing Emotions. Epistemological groundings and applications in robotics for a synthetic approach to emotions. AIIB Symposium 2010. Di Paolo, E. A. 2000. Homeostatic adaptation to inversion of the visual field and other sensorimotor disruptions. From Animals to Animals, Proceedings of the Sixth International Conference on the Simulation of Adaptive Behavior, 440 – 449. Di Paolo, E. A. 2010. Robotics inspired in the organism. Intellectica,53-54, 129–162. Di Paolo, E. A., Noble, J., & Bullock, S. 2000. Simulation models as opaque thought experiments. Page 497506 of: Artificial Life VII: The Seventh International Conference on the Simulation and Synthesis of Living Systems. Eichenbaum, H. 2002. The Cognitive Neuroscience of Memory: An Introduction. Oxford University Press. Engel, A. K., Fries, P., & Singer, W. 2001. Dynamic predictions: oscillations and synchrony in top-down processing. Nature Reviews Neuroscience,2(10), 704–716. Evans, B. I., & O’Brien, W. J. 1988. A reevaluation of the search cycle of planktivorous arctic grayling, Thymallus arcticus. Canadian Journal of Fisheries and Aquatic Sciences,45, 187–192. Fernandez-Leon, J. A. 2010. Behavioural robustness and the distributed mechanisms hypothesis. Ph.D. thesis, University of Sussex, UK. 80 Bibliography Freeman, W., Rogers, L., Holmes, M., & Silbergeld, DL. 2000. Spatial Spectral Analysis of Human Electrocorticograms Including the Alpha and Gamma Bands. Journal of Neuroscience Methods,93(2), 1111–1121. French, R. M., & Thomas, E. 2001. The Dynamical Hypothesis in Cognitive Science: A review essay of Mind As Motion. Minds and Machines,11(1), 101–111. Funahashi, K. I., & Nakamura, Y. 1993. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks,6, 801–806. Gomila, A., & Calvo, F. J. 2008. Directions for an embodied cognitive science: towards an integrated approach. Pages 1–25 of: Handbook of Cognitive Science: An Embodied Approach. Elsevier Publishers Limited. Haken, H., Kelso, J. A. S., & Bunz, H. 1985. A theoretical model of phase transitions in human hand movements. Biological Cybernetics,51(5), 347356. Harvey, I., Paolo, E. Di, Wood, R., Quinn, M., & Tuci, E. 2005. Evolutionary robotics: a new scientific tool for studying cognition. Artificial Life,11(1-2), 79–98. Hebb, D. O. 1949. The Organization of Behavior. Wiley, New York. Held, R., & Hein, A. 1963. Movement-produced stimulation in the development of visually-guided behavior. Journal of Comparative and Physiological Psychology,56(5), 872–876. Hopfield, J. J. 1984. Neurons with graded response properties have collective computational properties like those of two-state neurons. In: Proceedings of The National Academy of Sciences - PNAS. Hurley, S. L. 1998. Consciousness in Action. Cambridge, MA: Harvard University Press. Ijspeert, A., Crespi, A., Ryczko, D., & Cabelguen, J. M. 2005. From swimming to walking with a salamander robot driven by a spinal cord model. Science,315(5817), 1416–1420. Izhikevich, E. M. 2000. Neural excitability, spiking and bursting. International Journal of Bifurcation and Chaos,10(6), 1171–1266. Izquierdo, E. 2008. The dynamics of learning behaviour: A situated, embodied, and dynamical systems approach. Ph.D. thesis, University of Sussex, UK. Jirsa, V. K., Fuchs, A., & Kelso, J. A. S. 1998. Connecting cortical and behavioral dynamics: bimanual coordination. Neural Computation,10(8), 2019–2045. Kelso, J. A. S. 1995. Dynamic Patterns: The Self-Organization of Brain and Behavior (Complex Adaptive Systems). The MIT Press. Kelso, J. A. S., de Guzman, G. C., Reveley, C., & Tognoli, E. 2009. Virtual partner interaction (VPI): exploring novel behaviors via coordination dynamics. PLoS ONE, 4(6), e5749. Kramer, D., & McLaughlin, R. 2001. The behavioural ecology of intermittent locomotion. American Zoologist,41, 137–153. Kuramoto, Y. 1984. Chemical oscillations, waves, and turbulence. Springer, New York. 81 Bibliography Lederman, S., & Klatzky, R. 1996. Haptic aspects of motor control. In: Handbook of Neuropsychology. Volume 11: Action and Cognition. Amsterdam: Elsevier Science Publishers. Lee, S. H., & Blacke, R. 1999. Visual form created solely from temporal structure. Science,284, 1165–1168. Leopold, D. A., & Logothetis, N. K. 1999. Multistable phenomena: Changing views in perception. Trends in Cognitive Science,3, 254–264. Lizier, J. T. 2010. The local information dynamics of distributed computation in complex systems. Ph.D. thesis, School of IT, University of Sydney. Lock, A., & Collett, T. 1979. A toad’s devious approach to its prey: A study of some complex uses of depth vision. Journal Of Comparative Physiology. A Neuroethology Sensory Neural And Behavioral Physiology,131, 179–189. McGeer, T. 1990. Passive walking with knees. Proceedings of the 1990 IEEE International Conference on Robotics and Automation,3, 1640–1645. Moioli, R. C., Vargas, P. A., & Husbands, P. 2010. Exploring the Kuramoto model of coupled oscillators in minimally cognitive evolutionary robotics tasks. In: IEEE Congress on Evolutionary Computation. Moreno, A., Ruiz-Mirazo, K., & Barandiaran, X. 2010. The impact of the paradigm of complexity on the foundational frameworks of biology and cognitive science. In: Philosophy of Complex Systems. Elsevier Handbook of The Philosophy of Science series. Newell, A., & Simon, H. A. 1972. Human Problem Solving. Englewood Clilffs, NJ: Prentice Hall. Nolfi, S., & Floreano, D. 2000. Evolutionary Robotics. MIT Press. Cambridge, MA. O’Brien, W. J., Evans, B. I., & Browman, H. I. 1989. Flexible search tactics and efficient foraging in saltatory searching animals. Oecologia,80(1), 100–110. O’Keefe, J., & Recce, M. L. 1993. Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocampus,3(3), 317–330. Oppenheim, A. V. 1981. The importance of Phase in Signals. Proceedings of the IEEE, 69(5). Palva, S., & Palva, J. M. 2009. New vistas for α-frequency band oscillations. Trends in Neurosciences,30, 150–158. Pearlmutter, B. A. 1990. Dynamic Recurrent Neural Networks. Tech. rept. School of Computer Science, Carnegie Mellon University. Penttonen, M., & Buzsáki, G. 2003. Natural logarithmic relationship between brain oscillators. Thalamus Related Systems,2(2). Pfeifer, R., & Scheier, C. 1999. Understanding Intelligence. Cambridge, MA: MIT Press. Philips, W.A., & Singer, W. 1997. In search of common foundations for cortical computation. Behavioral and Brain Sciences, 657–683. 82 Bibliography Rayner, J. M. V., Viscardi, P. W., Ward, S., & Speakman, J. R. 2001. Aerodynamics and energetics of intermittent flight in birds. American Zoologist,41, 188–204. Revonsuo, A., & Newman, J. 1999. Binding and Consciousness. Consciousness and Cognition, 123–127. Rosenblum, L. 1993. Acoustical information for controlled collisions. In: Contributions to Psychological Acoustics. Oldenburg, Germany: Bibliotheksund Informationssystem der Carl von Ossietzky Universitaet Oldenburg. Rumelhart, D. E., McClelland, J. L., & Group, CORPORATE PDP Research. 1986. Parallel distributed processing: explorations in the microstructure, vol. 2: psychological and biological models. MIT Press Cambridge, MA, USA. Russell, S. J., & Norvig, P. 2003. Artificial Intelligence: A Modern Approach (2nd ed.). Upper Saddle River, New Jersey: Prentice Hall. Santos, B., Barandiaran, X. E., & Husbands, P. 2011. Metastable dynamical regimes in oscillatory network modulated by an agents sensorimotor loop. Proceedings of the IEEE Symposium Series on Computational Intelligence. Schall, J. D., & Thompson, K. G. 1999. Neural selection and control of visually guided eye movements. Annual Review of Neuroscience,22, 241–259. Schreiber, T. 2000. Measuring information transfer. Physical Review Letters,85(2), 461. Seliger, P., Young, S. C., & Tsimring, L. S. 2002. Plasticity and learning in a network of coupled phase oscillators. Physical Review E,65. Singer, W. 1999. Neuronal synchrony: a versatile code for the definition of relations? Neuron,24(1). Singer, W. 2007. Binding by synchrony. Scholarpedia,2(12), 1657. Sloman, A., & Chrisley, R. L. 2005. More things than are dreamt of in your biology: Information-processing in biologically-inspired robots. Cognitive Systems Research, 6(2), 145–174. Sonerud, G. A. 1992. Search tactics of a pause-travel predator: adaptive adjustments of perching times and move distances by hawk owls (Surnia ulula). Behavioral Ecology and Sociobiology,30(3), 207–217. Strogatz, S. 2000. From Kuramoto to Crawford: exploring the onset of synchronization in populations of coupled oscillators. Physica D: Nonlinear Phenomena,143(1-4), 1–20. Thompson, E., & Varela, F. J. 2001. Radical embodiment: neural dynamics and consciousness. Trends in Cognitive Sciences,5(10). Tognoli, E., & Kelso, J. A. S. 2009. Brain coordination dynamics: True and false faces of phase synchrony and metastability. Progress in Neurobiology,87, 31–40. Tye, A. 1989. A model of search behaviour for the northern wheatear Oenanthe oenanthe (Aves, Turdidae) and other pause-travel predators. Ethology,83, 1–18. van Duijn, M., Keijzer, F.A., & Franken, D. 2008. Principles of Minimal Cognition. Casting Cognition as Sensorimotor Coordination. Adaptive Behavior,14(2), 157–170. 83 Bibliography van Gelder, T. 1995. What might cognition be, if not computation? Journal of Philosophy,92, 345381. Varela, F. J. 2006. Neuronal Synchrony and Cognitive Functions. Pages 95–108 of: Self-Organization and Emergence in Life Sciences. Springer. Videler, J. J., & Weihs, D. 1982. Energetic advantages of burst-and-coast swimming of fish at high speeds. Journal of Experimental Biology,97, 169–178. Wagner, U., Gais, S., Haider, H., Verleger, R., & Born, J. 2004. Sleep Inspires Insight. Nature,427, 352–355. Walter, W. G. 1950. An imitation of life. Scientific American,182, 4245. Webb, B., & Harrison, R. R. 2000. Integrating Sensorimotor Systems in a Robot Model of Cricket Behavior. Proceedings of the Society of Photo-Optical Instrumentation Engineers,4196, 113–124. Werner, A. G., & Jirsa, V. K. 2006. Metastability, criticality and phase transitions in brain and its models. Biosystems,90(2), 496–508. Wiener, N. 1948. Cybernetics or Control and Communication in the Animal and the Machine. MIT Technology Press and Wiley, New York. Womelsdorf, T., Schoffelen, J. M., Oostenveld, R., Singer, W., Desimone, R., Engel, A. K., & Fries, P. 2007. Modulation of Neuronal Interactions Through Neuronal Synchronization. Science,316, 1609–1612. 84