A Neuromorphic Vision and Feedback Sensor Fusion Based on Spiking Neural Networks for Real-Time Robot Adaption

A Neuromorphic Vision and Feedback Sensor Fusion Based on Spiking Neural Networks for Real-Time Robot Adaption Pablo López-Osorio,* Juan Pedro Domínguez-Morales, and Fernando Perez-Pe˜na 1. Introduction It has long been known that biological locomotion is produced by a neural structure called central pattern generator (CPG). [1–5] This structure is made of neural circuits that are specialized in generating rhythmic patterns of neural activity even without requiring external sensory inputs. [6,7] They are located in the spinal cord of vertebrates or in the nervous system of invertebrates and are responsible for controlling and coordinating repetitive movements and sequences of activity, such as walking in mammals, [8] the rhythm of swimming in fish, [9] the control of the heartbeat, [10] or the basic rhythm of breathing. [11] Usually, these neural structures are made of, at least, two neuronal populations that alternate their cycles of activity, producing oscillations in the output without the need to receive oscillating signals as stimulus. The research field called neuromorphic engineering aims to implement these neural structures in electronic devices, emulating the way living beings have solved complex problems using biological circuits. [12] Neuromorphic robotics combines both the neuromorphic engineering community and roboticists, leading to potential applications that bring together expertise from both fields. [13,14] The robotics field borrows concepts from biological CPGs to create locomotion in robotic platforms. To implement these neural structures, there are several options available, including using coupled oscillators, artificial neural networks (ANNs), or spiking neural networks (SNNs). Among these options, SNNs stand out as the closest to biology in terms of performance and behavior. This is primarily due to its temporal information encoding, which enhances its resilience against noise when compared to ANNs. [15] Furthermore, SNNs stand out in terms of biological synaptic plasticity and energy efficiency, [16] as they only consume energy when neurons emit spikes (action potentials), closely resembling the functioning of the brain. Consequently, when applied in robotics, the use of SNNs results in reduced resource requirements, lower energy consumption, and simplified algorithms, in contrast to traditional approaches that rely on ANNs or coupled oscillators. The use of SNNs for locomotion generation gave rise to spiking central pattern generator (sCPG). Using spiking neurons, a rhythmic output pattern can be generated, which can be used to generate motion on robotic platforms. In addition, one of the advantages of SNNs is that they can generate very stable patterns P. López-Osorio, F. Perez-Pe˜na School of Engineering Universidad de Cádiz 11519 Puerto Real, Spain E-mail: [email protected] J. P. Domínguez-Morales Robotics and Technology of Computers Lab. ETSII-EPS Universidad de Sevilla Sevilla 41012, Spain The ORCID identification number(s) for the author(s) of this article can be found under https://doi.org/10.1002/aisy.202300646. © 2024 The Authors. Advanced Intelligent Systems published by Wiley- VCH GmbH. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. DOI: 10.1002/aisy.202300646 For some years now, the locomotion mechanisms used by vertebrate animals have been a major inspiration for the improvement of robotic systems. These mechanisms range from adapting their movements to move through the environment to the ability to chase prey, all thanks to senses such as sight, hearing, and touch. Neuromorphic engineering is inspired by brain problem-solving techniques with the goal of implementing models that take advantage of the characteristics of biological neural systems. While this is a well-defined and explored area in this field, there is no previous work that fuses analog and neuromorphic sensors to control and modify robotic behavior in real time. Herein, a system is presented based on spiking neural networks implemented on the SpiNNaker hardware platform that receives information from both analog (force-sensing resistor) and digital (neuromorphic retina) sensors and is able to adapt the speed and orientation of a hexapod robot depending on the stability of the terrain where it is located and the position of the target. These sensors are used to modify the behavior of different spiking central pattern generators, which in turn will adapt the speed and orientation of the robotic platform, all in real time. In particular, experiments show that the network is capable of correctly adapting to the stimuli received from the sensors, modifying the speed and heading of the robotic platform. RESEARCH ARTICLE www.advintellsyst.com Adv. Intell. Syst. 2024,6, 2300646 2300646 (1 of 15) © 2024 The Authors. Advanced Intelligent Systems published by Wiley-VCH GmbH without the need to incorporate any sensory or brain information to regulate their operation or oscillation frequency. [14,17,18] Previous works that explore the implementation of CPGs as signal generators for robotic actuator control are: ref. [19] where two different topologies of CPGs with coupled oscillators are presented to provide different motion patterns to a quadruped robot; however, these patterns are not interchangeable on the fly; others like ref. [20] or ref. [14] use a set of CPGs to produce various motion patterns in hexapod robots; however, the pattern cannot be modified autonomously. The same occurs in ref. [19]. There are some previous works that use a sCPG similar to the one proposed in this article but they include an open-loop network that does not conceive the use of sensory inputs for realtime modification of the system. For example, in refs. [14,20,21], three sCPGs designs capable of implementing different gait patterns on robotic platforms are presented, using the SpiNNaker neuromorphic platform [22] together with a field programmable gate array (FPGA) to process the signals received from the neuromorphic chip. In ref. [21], in addition, a dynamic vision sensor (DVS) is used that influences the selection of the hexapod’s walking rhythm, but a constant connection to a computer is required. In ref. [23], the authors propose another sCPG architecture with the ability to modify its amplitude, phase, and frequency without requiring any sensory information. Other neuromorphic platforms such as Loihi [24] have also been used to generate locomotion in a robotic lamprey that can select up to two different gait patterns but does not use a control loop over the generated [25] motion. These previous works demonstrate that the use of sCPGs provides significant energy savings compared to conventional actuation and control methods. Furthermore, these works highlight the fact that it would be interesting to incorporate external sensory feedback to modify the behavior of the neural network and, therefore, the behavior of the robotic platform. There are some previous works that include sensory feedback. [26–28] In such cases, conventional oscillators have been used to model the behavior of the CPG, or Hopf oscillators, as in ref. [29] where it is used to generate the rhythmic pattern in the CPG. In another recent work, [30] they used different inertial measurement unit sensors and distance sensors for a robotic fish to modify in real time a series of coupled oscillators, autonomously selecting the most optimal swimming routine depending on the surroundings. In ref. [31], 12 spiking neurons modulated by sensory feedback are used to shape the sCPG and the modification of the motion pattern of the robotic structure is achieved by adding or subtracting the neural structures, whereas in our work, it is done by an adaptive mechanism of the SNN. Furthermore, the authors used the Izhikevich neural model, [32] which involves a more complex computational model than the one proposed in this article: the leaky integrate-and-fire (LIF) model. Conversely, in ref. [33], a neuromorphic auditory sensor (NAS) is used as an input sensor to decide the pattern to be followed by the sCPG in SpiNNaker, depending on the audio processed by a FPGA and recognized by the SNN. In ref. [34], a closed-loop system is presented that incorporates a time difference encoder in SpiNNaker together with a FPGA to process the intermediate signals, which analyzes the information received from a NAS and a sound source localization system to modify the behavior of a robotic arm. In our work, we use a similar approach: a fusion of the force-sensitive resistors (FSRs) and the DVS is used to modify the behavior of the neural network implemented in SpiNNaker in real time. A recent work proposes the combination of firing and nonfiring neurons to simulate a neural network that regulates amplitude, phase, and frequency (each independently) in a closed-loop system. [35] On the one hand, the nonfiring neurons used in Strohmer’s study are LIF neurons, with a threshold high enough to avoid neuronal firing when the analog input varies. On the other hand, the firing neurons employ the adaptative exponential integrate-and-fire (AdEx) model, a considerably more complex neuronal model than that used in the work presented in this article, with a complex implementation in neuromorphic hardware. The primary focus of the study is the search for an optimal combination of spiking and nonspiking neurons to create a sensorimotor neuron network capable of adjusting the network’s output (spiking) based on analog parameters at the network’s input (nonspiking). However, locomotion is not the primary objective of this article, although it is briefly mentioned. As mentioned earlier, the authors propose compatibility of their network with neuromorphic hardware, although this compatibility is limited only to the nonspiking neurons and a cloud-based simulation service called CloudBrain. The proposal made in this article differs in several aspects: compared to ref. [35], a different spiking neuron model is used and all neurons are firing neurons. The second aspect that sets us apart is the use of information obtained from an FSR and a DVS that allows us to modify the behavior of the neural network controlling the hexapod robot. This enables us to adjust its speed and direction based on terrain characteristics and activity detected within the field of view of the DVS. The third and most important aspect to emphasize is the validation of our model on neuromorphic hardware, specifically on the SpiNNaker platform, where we utilize high-level software tools that enhance the reproducibility of the network for the scientific community. The main goal of this work is to design and implement a SNN composed of a sCPG and a Winner Take All (WTA) network that change their behavior depending on external variables in the environment of a hexapod robot. On the one hand, the FSR sensors will increase or decrease the activity of the sCPG depending on the stiffness of the terrain, resulting in a higher or lower speed of the hexapod. On the other hand, the DVS sensor will detect the predominant direction of motion, generating spikes associated with this direction and sending them to the WTA network. The WTA network will then select the winner and transmit the spikes to the FPGA to modify the behavior of the robotic platform, implementing a prey–predator model. Furthermore, the SNN network presented in this work is able to adapt to these input stimuli and modify the behavior of the neural network only with the external sensory information provided. Although FSR and DVS sensors are used, the adaptation step presented in this work can be used with any type of sensor to introduce the external stimulus. The main contributions of this work are: 1) To the authors’ knowledge, this work is the first to combine neuromorphic and analog sensors as input to the SpiNNaker platform for closed-loop control of a hexapod using sCPGs; 2) The experiments have not only been simulated but also implemented in hardware, in this case using an FPGA and a spinn-3 platform; www.advancedsciencenews.com www.advintellsyst.com Adv. Intell. Syst. 2024,6, 2300646 2300646 (2 of 15) © 2024 The Authors. Advanced Intelligent Systems published by Wiley-VCH GmbH 26404567, 2024, 5, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/aisy.202300646 by Universidad De Sevilla, Wiley Online Library on [25/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License and 3) The implemented workflow is fully functional in real time and with closed-loop online terrain adaptation and pattern change. The rest of the article is structured as follows: Section 2 is divided into two subsections: first, the materials (Section 2.1) used in this work are introduced, including the neuromorphic platform and the hardware used. The implemented methods are described in Section 2.2, along with the SNN models. The results obtained are presented in Section 3 and discussed in Section 4 and, finally, the conclusions of this work are briefly discussed in Section 5. 2. Experimental Section 2.1. Materials This section describes both the software and hardware used to perform the experiments proposed in this work. 2.1.1. Spiking Neural Network Architecture (SpiNNaker) SpiNNaker [22,36,37] stands for “Spiking Neural Network Architecture”and is a research project and hardware platform designed to simulate large-scale biological neural networks. It was developed at the University of Manchester in the UK as part of the Human Brain Project (HBP) [38] and is primarily used in the fields of computational neuroscience and biologically inspired artificial intelligence. The idea behind SpiNNaker is to use specialized hardware design to simulate how neural networks work in the human brain. SpiNNaker models the communication between neurons in a neural network using a large number of low-cost and effective processors coupled in a certain way instead of relying on conventional central processing units. [39] These processors can perform computations at the neuronal level and connect to each other in a low-latency, high-speed network. This configuration enables an asynchronous communication infrastructure for transmitting short packets, representing individual neuron firings, [40] identified using address event representation (AER). [41] Various SpiNNaker machines were manufactured and made commercially available, including SpiNN-3 and SpiNN-5, which house 4 and 48 SpiNNaker chips, respectively. Additionally, they incorporate spiNNlinks, [42] which facilitates real-time input/output interfacing with neuromorphic sensors and other neuromorphic platforms such as FPGAs. [43–45] To design and implement SNNs on these machines, researchers can employ a PyNN-based software package [46] named sPyNNaker. [47] Notably, the Advanced Processor Technologies group from the School of Computer Science at the University of Manchester recently constructed a million-core machine, accessible through the HBP portal. In this study, a local SpiNN-3 machine was utilized to implement the robotic platform. 2.1.2. FPGA An FPGA Nexys 4 DDR [48] development board was used for communication between the sensors and the neuromorphic platform. This board is equipped with an FPGA Xilinx Artix-7 XC7A100T-1CSG324C, which offers a programmable logic capacity of 100 000 cells and digital signal processors slices, providing sufficient processing power for this work. 2.1.3. DVS The AER DVS128 retina chip (silicon retina) [49] is an array of autonomous photoreceptors (pixels) that respond to logarithmic changes in brightness intensity in real time by placing the address of that specific pixel on an asynchronous bus. Only pixels that are stimulated by a change in illumination generate spikes, saving storage and bandwidth over conventional vision sensors. The pixel addresses use of the AER protocol and contain the X and Ycoordinates of the photoreceptor that detected the change in brightness. In this work, a retina DVS sensor [50,51] was used, containing a 128 128 pixels matrix, whose generated output is used as input to a WTA network to decide whether the robotic platform should continue forward or whether it should turn left or right, to target the direction that generates the highest spiking activity. The addresses associated with each pixel are composed of 7 bits associated with the Ycoordinate, 7 bits associated with the Xcoordinate, and a polarity bit representing the change in contrast, where a logic 1 means an increase in brightness and a logic 0 means a decrease. This sensor is connected to a pAER interface that allows real-time visualization of the retinal output, while simultaneously sending the events to the FPGA platform through the CAVIAR connector. [52] 2.1.4. FSRs To modify the running speed of the robotic platform, different analog sensors have been used. These sensors will generate a voltage output depending on the pressure that is exerted on them. Since the weight of the robotic platform is stable and equally distributed on each leg, the only variable that will modify the value provided by these sensors will be the terrain on which they are located: If the robot is on firm, flat ground (such as concrete), the forces will be distributed evenly over the surface of the sensors, generating a higher pressure and therefore a higher voltage. If, in contrast, the platform is on more uneven ground (such as sand), the forces generated by the weight of the robot will not be evenly distributed over the entire surface of the sensors, thus generating less pressure on them and therefore a lower voltage. 2.1.5. Dynamixels AX-12A Actuators Eighteen Dynamixel AX-12A motors will be used to move the robotic platform. [53] These actuators have the ability to control their speed, temperature, position, voltage, and load supported. Each of these motors incorporates an ARM Cortex-M3 microcontroller that manages the internal logic of the motor, allows them to be easily controlled, and enables bidirectional communication via the Dynamixel network protocol. In addition, multiple motors can be daisy-chained on a single communication bus, allowing www.advancedsciencenews.com www.advintellsyst.com Adv. Intell. Syst. 2024,6, 2300646 2300646 (3 of 15) © 2024 The Authors. Advanced Intelligent Systems published by Wiley-VCH GmbH 26404567, 2024, 5, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/aisy.202300646 by Universidad De Sevilla, Wiley Online Library on [25/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License multiple motors to be controlled with a single cable and a single interface. 2.1.6. Robotic Platform A hexapod has been designed and manufactured using 3D printing, which allows all the necessary electronics to be housed on its surface. The robotic platform performs the movement thanks to the 18 Dynamixels motors that it incorporates, which are controlled by the FPGA. 2.1.7. Minimum Required Configuration Figure 1 illustrates the minimum configuration required to accurately replicate the tests conducted in this study. This setup includes a SpiNN-3 board, a DVS, an FSR, an FPGA unit, and two Dynamixel AX-12A motors. One of the Dynamixel motors is used to regulate speed in response to signals from the FSR, while the other adjusts orientation based on the information provided by the DVS. 2.2. Methodology 2.2.1. Neuron Models The neural model used to implement the neurons of the network is the LIF model. This model is defined according to Equation (1) and (2) dV dt¼ðVVrÞþRIðtÞ τm (1) if VðtÞ¼Vth then lim δ!0;δ>0VðtþδÞ¼Vr(2) where Vis the membrane potential of the neuron, Rrepresents the resistance presented by this membrane, τ m the time constant of the neuron, V r the resting potential, V th the threshold required to trigger the firing of the neuron, and I(t) is the stimulus (in the form of current) entering the neuron. CPG Populations: The CPG is the main neural structure of the network and is composed of the populations Aand Bof 100 neurons each, which form the CPG AB , represented in Figure 2. The number of neurons per population and their parameters have been set prioritizing the reduction of the total number of neurons in the network: using fewer neurons per population causes erratic network performance, such as loss of rhythmicity or network outputs that do not correspond to the expected result. These populations, on the one hand, are self-excited and self-inhibited with a probability of 25% and 75% and weights of 5 and 1.5 nA, respectively. These probabilistic parameters have been established since the number of excitatory synapses received by the adapting network populations from the FPGA-SpiNNaker input populations are more numerous than the inhibitory connections produced internally in the adapting network itself. Furthermore, the excitatory synaptic weight values are considerably higher than those of the inhibitory synaptic weights, maintaining a balance between the probability of connection and the strength with Figure 1. Minimum required configuration to replicate the experiment. Figure 2. CPG AB network topology. www.advancedsciencenews.com www.advintellsyst.com Adv. Intell. Syst. 2024,6, 2300646 2300646 (4 of 15) © 2024 The Authors. Advanced Intelligent Systems published by Wiley-VCH GmbH 26404567, 2024, 5, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/aisy.202300646 by Universidad De Sevilla, Wiley Online Library on [25/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License which the connection is established. The probability of inhibition between one population and the other is also 75%, with a weight of 0.25 nA. Due to this configuration, when one of the populations has activity, the other will be inhibited accordingly, causing the oscillation in the network. It is important to note that these populations are designed to work as tonic neurons, so they are injected with a very low but constant external current to ensure that they are constantly generating activity. Therefore, in these two populations the neuronal model would be as follows dV dt¼VrVþRðIexc Iinh þIstÞ τm (3) where I st is the current injected into neurons in populations A and B. This value is set to 0.1 nA, which is sufficient to maintain the desired minimum oscillation. The neuronal parameters governing the behavior of these populations are presented in Table 1. Adaptive Populations: In this part of the network, we can distinguish three populations: POP REF is composed of 50 neurons that directly receive the input stimuli linked to the FSR, and its neuronal parameters are those presented in Table 1. POP E and POP I are populations of 100 neurons each, and their weights are configured in such a way that, depending on the activity of POP REF , the activity of these neurons increases or decreases. In turn, these populations have inhibitory connections between them, achieving an oscillatory phenomenon between these two populations similar to that of CPG AB . These populations are based on the neuronal model LIF, but their neuronal parameters are adapted in such a way that the input–output ratio of these populations is 1:1, thus ensuring that their behavior is closely related to the input provided by the FSR. Their parameters are shown in Table 2, and their connection scheme is illustrated in Figure 3. By connecting CPG AB to the adaptive network via excitatory and inhibitory synaptic connections, it is possible to modify its activity depending on the input received from the FSR sensors. The complete network is shown in Figure 4 and its operation can be summarized in two possible cases: 1) If the activity of POP REF is greater than the activity of CPG, POP Exc will be strongly excited by POP REF and weakly inhibited by POP A , which will increase its activity, in turn causing an increase in the activity of CPG. Furthermore, POP Exc will strongly inhibit POP Inh , preventing the latter from inhibiting CPG; and 2) If the activity of POP REF is lower than the activity of CPG, POP Inh will be strongly excited by CPG and weakly inhibited by POP REF , which will increase its activity, which in turn will cause a decrease in the activity of CPG. In addition, POP Inh will strongly inhibit POP Exc , preventing the latter from exciting CPG. The weights of this entire network have been tuned using a grid search-based weight search algorithm, and are configured to ensure stable oscillations of the CPG over the entire input range of the FSRs. This input will oscillate between 10 and Table 1. Neuron parameters for the proposed CPG in SpiNNaker hardware platform. Parameter Value c m 0.25 nF τ m 12.5 ms τ r 0.001 ms u reset –70.0 mV u rest –65.0 mV u th –50.0 mV τsyne7.0 ms τsyni17.1 ms I bias 0.1 mA Table 2. Proposed neural parameters for the adaptive populations on the SpiNNaker hardware platform. Parameter Value c m 0.1 nF τ m 0.1 ms τ r 0.0 ms u reset –65.0 mV u rest –65.0 mV u th –64.91 mV τsyne0.1 ms τsyni0.1 ms I bias 0.0 mA Figure 3. Topology of the CPG EI network. Figure 4. FSR adaptive network topology. www.advancedsciencenews.com www.advintellsyst.com Adv. Intell. Syst. 2024,6, 2300646 2300646 (5 of 15) © 2024 The Authors. Advanced Intelligent Systems published by Wiley-VCH GmbH 26404567, 2024, 5, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/aisy.202300646 by Universidad De Sevilla, Wiley Online Library on [25/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 171 Hz (for minimum and maximum pressure values, respectively). Thanks to this configuration, the CPG will provide stable oscillations but with variable activity, which will be used to select the gait pattern to be adopted by the robotic platform. WTA Network: The neurons of the WTA network receive the activity generated by the DVS retina and their output will be used to select the direction to be followed by the robotic platform. These populations incorporate the same parameters as those of CPG AB , except for the injected current (Table 3). For this reason, the activity or inactivity of these neurons is due solely to the input activity received by these populations from the FPGA. The topology of this network and its interaction with the DVS can be seen in Figure 5. Since the retina is a 128 128 pixel matrix, to select the direction of the robotic platform, it was decided to divide the Xcoordinate into three windows: the first 39 pixel columns are linked to the right direction, the next 50 to the center, and the last 39 to the left. In this way, when a pixel detects changes in brightness, the direction AER associated with that pixel will be sent to SpiNNaker. Depending on its Xcoordinate, this address will be used to excite one of the three populations in SpiNNaker. The weights of the synaptic connections are configured so that the population that generates the most activity is the winner and strongly inhibits the previous two; thanks to this, only the AER events of a single winning population will be transmitted to the robotic platform at a time, allowing it to decide which direction to follow. 2.2.2. Complete SNN Topology The complete topology of the neural network that allows controlling the speed and orientation of the robotic platform is shown in Figure 6 and its distribution of populations and neurons is shown in Table 4. 3. Experiments and Results This section describes the experiments conducted with respect to the proposed spiking architecture. Four experiments are carried out without the retina to check the performance of the network considering the external sensor changing value. Another four Table 3. Neuron parameters for the proposed WTA network in SpiNNaker hardware platform. Parameter Value c m 0.25 nF τ m 12.5 ms τ r 0.001 ms u reset 70.0 mV u rest 65.0 mV u th 50.0 mV τsyne7.0 ms τsyni17.1 ms I bias 0.0 mA Figure 5. DVS adaptive network topology. Figure 6. Complete topology of the adaptive network with a total of 9 populations and 579 neurons. www.advancedsciencenews.com www.advintellsyst.com Adv. Intell. Syst. 2024,6, 2300646 2300646 (6 of 15) © 2024 The Authors. Advanced Intelligent Systems published by Wiley-VCH GmbH 26404567, 2024, 5, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/aisy.202300646 by Universidad De Sevilla, Wiley Online Library on [25/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License experiments include the retina to show the network decision considering the spiking input stimulus. The first experiment includes a 10 Hz input stimulus from the population bound to the FSR (minimum value provided by the FSRs). The results are shown in Figure 7. It can be seen that the activity of POP EXC and POP FSR is very similar. Due to the interaction of POP EXC and POP INH with CPG AB , the average oscillation frequency of the obtained POP CPG is 7 Hz. It is important to note that the frequency of the CPG does not linearly adapt to the input frequency of the FSR. It rather moves in a narrow frequency range (between 5 and 20 Hz) which is enough to be able to select and modify the gait pattern of the robotic platform. The second experiment includes, as stimulus, intermediate frequencies of 85 Hz from the population bound to the FSR. The results are shown in Figure 8. In this experiment, it can be observed how the activity of the POP EXC population is greatly increased by the excitation received by POP FSR . It is interesting to highlight how, although increasing the frequency of POP FSR should decrease the frequency of POP INH , the opposite effect occurs due to the increase in the oscillation frequency of POP CPG , which excites, albeit weakly, POP INH . The average oscillation frequency of the CPG is 13 Hz. As a third experiment, the CPG network was subjected to constant inputs of 171 Hz (maximum values provided by the FSRs) from the population bound to them. The results are shown in Figure 9. In this case, it can be observed how POP EXC is strongly excited by POP FSR , while the activity of POP INH is considerably reduced. In this scenario, not only does the oscillation frequency of POP CPG increase, which is around 20 Hz, but also the number of shots at a given time instant in each population of POP CPG increases. Finally, a last experiment has been performed in which the input linked to the FSR is oscillatory, and it can be observed how the CPG adapts to the input by increasing or decreasing its total frequency (the frequency obtained by considering both populations of the CPG) and its individual frequency (the frequency of each neuron within each population at individual time points when its population is firing). The results are shown in Figure 10. As mentioned previously, the WTA network will receive all AER addresses associated with retinal pixels that have detected changes in luminosity. It is possible to receive up to 20 000 addresses per second on SpiNNaker, each representing one of the 128 possible directions linked to the X-coordinate of the retina. Various experiments have been conducted to subject the network to different stimuli, using a pen to create motion in front of the retina and activate the pixels. It is important to note that these scenarios propose situations that will not occur in a real and Table 4. Number of neurons per population in the complete network. Population Number of neurons A 100 B 100 Exc 100 Inh 100 Ref 50 Center 50 Right 39 Left 39 FSR 1 Figure 7. Behavior of the CPG network at 10 Hz inputs. Thanks to the regulating action of the POP EXC and POP INH populations, the output frequency of the CPG AB remains stable around 7 Hz. www.advancedsciencenews.com www.advintellsyst.com Adv. Intell. Syst. 2024,6, 2300646 2300646 (7 of 15) © 2024 The Authors. Advanced Intelligent Systems published by Wiley-VCH GmbH 26404567, 2024, 5, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/aisy.202300646 by Universidad De Sevilla, Wiley Online Library on [25/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License controlled environment of the robotic platform, such as receiving similar changes in luminosity in more than one window simultaneously or introducing noise into one of the windows to assess the network’s behavior in response to these phenomena. First, a higher activity was generated in the left region of the DVS, where it can be observed that during the entire simulation the stimulus was generated intermittently in the POP LEFT region intermittently, generating peaks of minimum activity (e.g., Figure 8. Behavior of the CPG network at 85 Hz inputs. Intermediate frequencies cause a generalized increase in oscillations in all populations, while POP INH and POP EXC keep the CPG stable throughout the duration of the experiment. Figure 9. Behavior of the CPG network in response to 171 Hz inputs. It is interesting to observe how the significant excitation caused by POP EXC on the CPG not only increases its frequency to 20 Hz but also leads to an increase in the number of neurons firing in each population of the CPG during the moments when they are active. www.advancedsciencenews.com www.advintellsyst.com Adv. Intell. Syst. 2024,6, 2300646 2300646 (8 of 15) © 2024 The Authors. Advanced Intelligent Systems published by Wiley-VCH GmbH 26404567, 2024, 5, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/aisy.202300646 by Universidad De Sevilla, Wiley Online Library on [25/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License t=1000, t=2000, and t=4000 ms) in some instants due to the stimulus coming out of the visible range of the retina. The results are shown in Figure 11. Another experiment was also carried out in which more activity was generated in the central region of the DVS. The results can be seen in Figure 12, and it is possible to check that the detection of events in all the windows does not imply activity in all of them at the output of the WTA network. The activity will be maintained only in the population that has generated the highest number of events in a given time. The same experiment was also performed generating increased activity in the right region of the DVS. In this experiment, it is again verified that, although the central population has produced some events in the instants t=800 and t=3000 ms, the output of the WTA network only contemplates the events of the winning population. The results can be seen in Figure 13. Finally, an experiment has also been performed in which the WTA network is subjected to random activity over the entire visual field of the DVS. The result of this experiment can be seen in Figure 14. It is interesting to see how the network shows tolerance to instantaneous noise. For example, although at t=1200 ms, the winning population at that instant is POP Left , at the output of the WTA POP Center remains the winner because POP Left has not generated enough activity for long enough to exceed the activity previously generated by POP Center . 4. Discussion The set of experiments conducted demonstrates: 1) the ability of the sCPG model to adapt to variable inputs from the FSRs by changing its oscillation frequency to increase or decrease the speed of the robotic platform, respectively; and 2) the ability of the WTA network model to select the spatial region with the highest spiking activity from a DVS input, thus orienting the robotic platform toward that direction. Thanks to these two features, course and speed adaptation, it is possible to implement a fully functional “prey–predator”model where analog and neuromorphic sensor inputs are fused. This information can be used by the network implemented in SpiNNaker to make a decision about the behavior of the robotic platform. From a bioinspired point of view, the biological features of the CPG (described in Section 1) show the functions that these neuronal structures can perform in living beings, as well as the characteristics of the neurons that form this type of neural structure. The CPG design includes biologically plausible structures. In the biological model, the neurons that make up the CPG can be either tonic or phasic. In the former case, tonic neurons emit the neurotransmitter at a constant but low rate, whereas phasic neurons act almost instantly, emitting a rapid pulse of activity to bring postsynaptic neurons into action quickly. In this work, both types of neurons have been implemented: on the one hand, CPG AB is made up entirely of tonic neurons. They include a very low (0.1 nA) but constant current injection, which ensures that even if no stimulus was received from the FSRs, the oscillation of the CPGs could be maintained over time, fulfilling the intrinsic rhythm generation that biological CPGs show. On the other hand, the POP EXC and POP INH populations act as phasic neurons, since at a minimum excitation signal received, they are able to reach the threshold to produce a neuronal firing: exciting or inhibiting the CPG AB almost instantly. With this feature, the centralized control to which the CPG is subjected is modeled. Although CPGs can function autonomously, they are often under Figure 10. Behavior of the CPG network in response to oscillatory inputs. The CPG adjusts its frequency based on the input associated with the FSRs, demonstrating its adaptation to external stimuli. www.advancedsciencenews.com www.advintellsyst.com Adv. Intell. Syst. 2024,6, 2300646 2300646 (9 of 15) © 2024 The Authors. Advanced Intelligent Systems published by Wiley-VCH GmbH 26404567, 2024, 5, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/aisy.202300646 by Universidad De Sevilla, Wiley Online Library on [25/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License