Active Perception with Dynamic Vision Sensors. Minimum Saccades with Optimum Recognition

IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEM, VOL. XX, NO. XX, XXXX 1 Active Perception with Dynamic Vision Sensors. Minimum Saccades with Optimum Recognition Amirreza Yousefzadeh1, Garrick Orchard2, Teresa Serrano-Gotarredona1, and Bernab´ e Linares-Barranco1 1Instituto de Microelectr´ onica de Sevilla IMSE-CNM (CSIC and Univ. de Sevilla), Sevilla, Spain ({reza,terese,bernabe}@imse-cnm.csic.es) 2Temasek Laboratories and Singapore Inst. for Neurotechnology (SINAPSE) at National Univ. of Singapore (garrickorchar[email protected]) Abstract—Vision processing with Dynamic Vision Sensors (DVS) is becoming increasingly popular. This type of bio-inspired vision sensor does not record static scenes. DVS pixel activity relies on changes in light intensity. In this paper, we introduce a platform for object recognition with a DVS in which the sensor is installed on a moving pan-tilt unit in closed-loop with a recognition neural network. This neural network is trained to recognize objects observed by a DVS while the pan-tilt unit is moved to emulate micro-saccades. We show that performing more saccades in different directions can result in having more information about the object and therefore more accurate object recognition is possible. However, in high performance and low latency platforms, performing additional saccades adds additional latency and power consumption. Here we show that the number of saccades can be reduced while keeping the same recognition accuracy by performing intelligent saccadic movements, in a closed action-perception smart loop. We propose an algorithm for smart saccadic movement decisions that can reduce the number of necessary saccades to half, on average, for a predefined accuracy on the N-MNIST dataset. Additionally, we show that by replacing this control algorithm with an Artificial Neural Network that learns to control the saccades, we can also reduce to half the average number of saccades needed for N-MNIST recognition. I. INTRODUCTION Interest in Dynamic Vision Sensors (DVSs) has grown rapidly in recent years, while at the same time this innovative technology is becoming more accessible to researchers, highlighting such sensors’ potential to enable low-latency sensing at low computational costs. However, DVSs operate differently than traditional frame-based sensors, and therefore the processing of event data requires a different approach than frame-based data. Frame-based vision systems process frames (static images), or sequences of frames, captured by a traditional camera. Frame capture and frame processing are usually two separate steps, with the quality of the captured frame affecting the quality of the processing outcome. Acquiring high-quality frames usually involves minimizing the relative motion between the frame subject(s) and the sensor during frame exposure to eliminate motion artifacts. Conventional machine vision algorithms are designed to process each frame individually, regardless of the amount of information contained in it. This is only efficient if the rate of change in the scene is similar to the frame rate, whereas in practical applications there are sometimes moments without any movement and moments with fast motion. DVSs are event-driven temporal contrast sensors that operate on a different principle, detecting the time and pixel location of light intensity changes in the scene. For the subject to be visible to the sensor, there has to be relative motion between sensor and subject, because without motion - and under constant lighting - there would be no pixel intensity changes for the sensor to detect. To ensure relative motion between sensor and subject, either the sensor or the subject (or both) need to be moving. In a typical sensing scenario the motion of the subject to be sensed cannot be controlled, therefore moving the sensor is a more convenient approach. This, however, poses the question of how to move the sensor. For actuated frame-based vision systems, sensor motion typically involves simply pointing the sensor towards the subject using a pan-tilt unit or by mounting it on a moving platform (such as a mobile robot). However, in biological vision - by which event-driven vision sensors are loosely inspired - there is growing evidence that the motion of the vision sensors (eyes) plays a vital role in perception, and that such movement is both well controlled (albeit subconsciously) and task-dependent [1]. Examples from nature include the jumping spider, which actively moves its retina [2], the praying mantis, which executes a peering type motion for depth perception, or pigeons, which move their heads back and forth to perceive depth. Even in humans, there is growing evidence that micro-saccades during fixation play a key role in perception [3], rather than just correcting erroneous ocular drift, as was previously believed. Intuitively, in an action-perception loop in natural animals, it is obvious that perception influences action and that action influences perception. However, most works and benchmark datasets focus on how best to perceive in order to influence action, since with pre-recorded data it is not possible to influence recordings through actions1. This paper looks instead at how best to act to influence perception. More specifically, this study uses the DVS, an event-driven temporal contrast sensor, to address the well-known MNIST [4] recognition task. It investigates how such a sensor should move to aid recognition in a closed action-perception loop, where the system decides what action to take next (if any) 1However, in this work we used pre-recorded data to compare our results with other works, but from a different perspective. We tried to keep the performance while using less information from the dataset, as will be explained later. This file is not the final version. This article is accepted to be published in IEEE TBCAS 2018 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEM, VOL. XX, NO. XX, XXXX 2 based on the sensory data it has received beforehand. We also look at whether knowledge of the action taken can be used to improve accuracy in the recognition task. For this, we mounted a DVS camera on a pan-tilt unit looking at static images. In a real-world application, for a example a moving robotic platform, the DVS should have saccadic movements that are much faster than the motion of the platform. The MNIST dataset is well known in computer vision as a relatively easy means of testing recognition models. It contains 70,000 labeled pictures of handwritten digits, 60,000 for training and 10,000 for testing. We presented static MNIST samples in front of a DVS which was mounted on a pantilt unit and recognized the handwritten digit by analyzing the output events of the DVS after each saccade. When we used saccades to imitate biological eye movements and object recognition in the proposed network, an interesting question arose: how is the recognition task affected by saccade direction and how many saccades are needed to recognize an object? As expected, we noticed that each saccade can contain unique information about the object related to the direction and speed of the saccadic motion. Since performing each saccade needs a mechanical movement plus event processing, it is desirable to reduce the number of saccades while keeping the same recognition performance. After several experiments, we designed an algorithm that can suggest the direction of the next saccade based on the current information about the object. Our results show that smartly chosen saccades can reduce the average number of saccades to half in comparison to random saccades with similar recognition accuracy. Interestingly, we noticed that a neural network can perform this task and intelligently suggest saccades with almost the same performance as an analytically developed algorithm. Section II explains how a DVS works and previous approaches to use the MNIST dataset with these type of sensors (or simulated sensors). In Section III we explain our proposed approach for eventdriven processing and object recognition by using a DVS, the proposed algorithm for prediction of an efficient subsequent saccadic direction for better object recognition, and how a neural network can be trained to intelligently suggest a next saccade direction. The results of the experiments are given in Section IV. In this Section, we introduce our hardware-software platform for real-time object recognition with DVS saccades to demonstrate the performance of the proposed algorithm in practice. Finally, brief conclusions are provided. II. BACKGROUND This Section provides background information on DVSs and previous experiences with event-driven MNIST datasets. A. Dynamic Vision Sensors DVSs output data in the Address Event Representation (AER) format [8], where each event consists of a pixel address and a single bit. The single bit indicates whether the intensity change was positive or negative. The concept is illustrated (a) (b) (c) Fig. 1: (a) Event generation for a DVS pixel when changing in light intensity passes a pre-defined threshold ✓. For a positive/negative change, a positive/negative event will be generated. (b) Propeller rotating in front of a DVS [5]; the USB-AERmini2 board [6] time-stamps events from the DVS and sends them to a computer through USB. (c) Reconstructed frame from DVS output with jAER software [7]. It contains 864 events collected during 624µs (1.3Meps - million events per second) IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEM, VOL. XX, NO. XX, XXXX 3 in Fig. 1. As shown in Fig. 1(a), when logarithmic change of light intensity passes a threshold ✓, an event will be generated. Fig. 1(c) shows the reconstruction of DVS events for time interval during which a propeller is rotating in front of the DVS. Reconstruction is done with the jAER software [7] where black dots indicate events with negative polarity and white dots indicate events with positive polarity. In a DVS each pixel spikes asynchronously as soon as it detects a given change in light log-intensity. These sensors can outperform frame-based vision sensors in terms of data compression, dynamic range, temporal resolution and power efficiency. Several different event-driven temporal contrast vision sensors exist [5], [9], [10], [11], [12], [13], [14], [15], [16]. For example, Samsung [17] recently presented a 640 ⇥480 pixel DVS which consumes a total of 27mW at a data rate of 100keps and 50mW at 300Meps, has better temporal resolution than a 2000fps camera and a dynamic range of more than 80dB. Gou et al. recently presented the highest resolution DVS (with on-demand image acquisition) with 768⇥640 pixels [18]. When something moves in front of a DVS, multiple pixels almost simultaneously generate events that evoke synchronybased neural coding [3]. In this kind of coding, information is not spike rate coded or spike rank/order coded [19]. Although neurons spike asynchronously, information is coded not in the exact time of each individual spike but in the simultaneous firing of a group of spikes together. In this work have employed two different event-driven sensors. First, to compare accuracy with other published studies, a pre-recorded dataset (N-MNIST) was used [20] which was recorded using the Asynchronous Time-based Image Sensor (ATIS) [12]. Secondly, the IMSE-DVS [5] was used to demonstrate the approach in a closed loop system in real-time. B. Previous Approaches to Event-Driven MNIST MNIST is arguably the most popular dataset used thus far for event-driven vision, and some different models have been applied to different event-driven variants of the original MNIST dataset. Three main event-driven versions of the MNIST dataset have been reported. The first is a recording of a subset of MNIST with different moving digits of different sizes presented to a DVS [21]. The second approach is to convert frames to spikes by means of intensity to delay conversion [22] or Poisson distributions [23], [24]. The most recent approach is a full conversion of the MNIST dataset at the original pixel scale, generated by moving the sensor with fast saccades while viewing static digits [20] (dubbed N-MNIST). This dataset is captured by mounting the ATIS sensor [12] on a motorized pan-tilt unit and having the sensor move while it views the MNIST samples on an LCD monitor. New techniques for efficient event processing are gradually being introduced. HOTS [25] is a new hierarchical machine learning technique that extracts visual features from events. HFirst [26] is a hierarchical Spiking Neural Network (SNN) for object recognition which uses a simple feed forward learning mechanism. This network has been implemented in low power parallel platforms such as FPGAs and SpiNNaker [27] and achieved 71.15% accuracy on the N-MNIST dataset [20] without optimization. Synaptic Kernel Inverse Method (SKIM) [28], a new learning method for synthesizing SNNs, achieved 92.87% accuracy on the N-MNIST dataset. Spike Time Dependent Plasticity (STDP) is an unsupervised bio-inspired learning method for SNNs. Kheradpishe et al. [22] developed a multi-layer SNN equipped with STDP, achieving 98.4% accuracy on the MNIST dataset by converting all the MNIST frames to events through intensity to delay conversion. J.H. Lee et al. [24] developed a new method to adapt the famous error backpropagation technique for SNNs, achieving 98.66% accuracy on the N- MNIST dataset. N-MNIST contains three saccade recordings for each MNIST sample. Based on our knowledge, no research has been done to improve the performance of recognition by considering each of these saccades as an individual source of information which is coupled to the direction of the saccade. We have used the N-MNIST dataset to benchmark performance of our proposed method but in an entirely different perspective. While it makes sense to use all the three saccades from each sample to improve recognition accuracy, in real-time robotic applications each additional saccade comes with a cost in power consumption and recognition latency. Therefore when we used N-MNIST, we considered the cost of each saccade along with the recognition accuracy and tried to use fewer saccades to recognize the handwritten digits. III. EVENT-DRIVEN RECOGNITION WITH SACCADES In this Section, first, we explain the methods that we used for processing saccadic events in a neural network. This neural network is an efficient feed forward network that receives DVS events and processes them to recognize handwritten digits. The output of this network is a prediction vector that contains ten values (one for each digit). Later on, in Section III-B, we integrate this network into a closed-loop system along with a block to control the direction of saccadic motion. This block, which we call the “Next Saccade Prediction” (NSP) block tries to suggest an optimal direction for the subsequent saccade based on the current output of the feed forward handwritten digit recognition network. The NSP block executes an analytical algorithm in software to suggest the next saccade direction. In Section III-C we show how the NSP block can be replaced by a neural network. In this case, both the feed forward and feedback processing will be done by using neural networks. This allows to easily implement the system fully in hardware. A. Feed forward Symbol Recognition Neural Network As mentioned earlier, a DVS is power efficient and has considerably less latency than conventional frame-based sensors. However, it is generally harder to extract information from the DVS events by using conventional image processing methods. To extract information efficiently, we propose processing groups of events that are generated close together in time rather than processing individual events. This is not a new idea and has been used in efficient hardware implementations IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEM, VOL. XX, NO. XX, XXXX 4 Fig. 2: Average event rate of a saccade (including all 28⇥28 pixels) in the N-MNIST dataset per millisecond. A frame is constructed by integrating the events in the time span of +/- 5ms (between 45ms to 55ms) around the peak average event rate at 50ms. [29], [30]. There is evidence that this kind of processing also takes place in biological cortex [31]. A block which is called “frame-maker” was therefore designed to group events occurring close together in time into a packet (equivalent to a frame in conventional image processing). 2By using such a “frame-maker” after DVS sensing, it was possible to create an automatic adaptive frame-rate camera for our system input. To build a frame from each saccade, one approach is to put a fixed number of events in a frame [30]. This method may result in multiple frames for a saccade. Also, each stimulus may need a variable number of events to construct a precise frame. For example, digit ‘8’ is bulkier than digit ‘1’ and will require more events to have a clear frame. In the N-MNIST dataset [20], each saccade takes about 100ms. A saccadic movement has the highest velocity in the middle of the saccade. Therefore output event rate is maximum around 50ms after the start of a saccade. Fig. 2 shows the average event rate of one saccade in the N-MNIST dataset. Our experiments show that collecting events during a short time when the event rate is high will result in a sharp frame. As it is illustrated in Fig. 2, a frame for each N-MNIST saccade can be created by collecting all the events which are generated in a time span of 10ms around the center of a saccade. The events outside this time will not be processed3. Fig. 3 shows three frames that are generated by three saccades of a sample in the N-MNIST dataset. Directions of these three saccades are shown in Fig. 4, and are labeled SAC H, SAC DU, SAC DD. Throughout the paper we will use the same labels to refer to different “concepts”: for example, in Fig. 3 use them for labeling “frames”, each corresponding to one of the three 2We report elsewhere [32] that the delay of processing such frames in hardware is in the order of 20µs. Consequently, the latency of processing the frame once the events are available is negligible with respect to the time of collecting them (10ms). 3This optimum 10ms interval is dependent on the mechanical saccadic movement used for these experiments. Speed and displacement have been kept fixed, only changing the angle. For different speeds and displacements, one may need to change this optimum 10ms slice. Fig. 3: Three saccades captured from sample ‘80’ of test set in N-MNIST dataset. The colors show the polarity of the events. Blue is for negative events and purple is for positive events. Dark blue indicates places where both positive and negative events occurred Fig. 4: Direction of saccades in the N-MNIST dataset, SAC DD (Diagonal Down Saccade), SAC DU (Diagonal Up Saccade) and SAC H (Horizontal Saccade). IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEM, VOL. XX, NO. XX, XXXX 5 Fig. 5: Block diagram of proposed feed forward system for symbol recognition with saccades, using a DVS. The DVS is connected to a moving pan-tilt unit. The “frame-maker” block assembles a frame after each saccade and then stores that frame in its corresponding memory (SAC DD, SAC DU or SAC H). Here we limit the direction of movements to three to remain compatible with the N-MNIST dataset. The Control Block is responsible for controlling the direction and speed of the pan-tilt unit movements based on the user input. Fig. 6: “Symbol Recognition Neural Network” (SRNN) with one, two and three saccades for a sample of N-MNIST dataset. Each column represents a hyperframe with three frames, each for one saccade direction. When a saccade direction is not available, its frame is left blank. saccades; later on we will use them to label three “memories” or three “flag bits”, always each corresponding to one of the three saccades. Fig. 5 shows the block diagram of the system that has been used to perform symbol recognition with a DVS through saccadic movements. The output of the “frame-maker” block is a 28⇥28 pixel binary frame,4if polarity information is not used; otherwise the output frame is 28 ⇥28 ⇥2bits. After the “frame-maker” block, a conventional Artificial Neural Network (ANN) is implemented to process the frames and recognize the handwritten digits. This ANN is called “Symbol Recognition Neural Network” (SRNN) and receives a hyperframe as its input. The hyper-frame is a frame with 28 ⇥84 (28 ⇥28 ⇥3) pixels (assuming polarity bit is ignored) and can contain all the frames made by the three saccades of each sample. This network should be able to work with one, two or three saccades of each input sample, as illustrated in Fig. 6. Therefore, during the training phase, SRNN was trained to accommodate all possibilities. This means that the SRNN can use one, two or all three saccades of a sample to predict the 4There is only one bit for each pixel in this frame, so multiple events with the same address will not carry additional information. To save power, it is recommended to adjust the DVS parameters (like threshold, refractory period, ...) and pan-tilt unit parameters (like the velocity and range of movement) in a way that each pixel generates maximum one event for each frame. presented digit5. We used a blank input in the position of the non-available frames to construct the 28⇥84 pixels of a hyperframe. The SRNN can have an arbitrary number of layers with various architectures (convolutional, fully connected, etc.). In this work, we tried a few small but accurate enough neural networks to perform our experiments. For a given input sample i, we interpret the SRNN output ˆ Yi=(ˆyi0,ˆyi2,...ˆyij,...,ˆyi9)as the vector of prediction probabilities ˆyij for each class j. The recognized digit is the one with the highest prediction value. The sum of all the values in a prediction vector is normalized to one. Therefore, each value can be interpreted as a probability. The quality of the prediction vector can be measured by calculating the following “prediction loss function” Li=|| ˆ YiYi|| 2(1) where Liis the loss for sample i,ˆ Yiis called the prediction vector for sample iand Yiis the ground truth label for sample irepresented using one hot encoding (yij =1if the class is j, and zero otherwise). B. Closed Loop Recognition with Analytical Algorithm In biological vision, it is the movement of the retina (saccades and micro-saccades) that enables one to see clearly [3]. In our study, we used a standard pan-tilt platform to move the DVS while the object in front of it was fixed. The information obtained from saccades is determined by the movement parameters. A movement can be described in terms of its velocity, distance, and direction. The movement velocity can affect the rate at which events are generated. For a clear saccade to be captured, the movement has to be sufficiently fast over a short distance. The recognition task can also be affected by the direction of the movement. For example, horizontal saccades intensify vertical edges but suppress horizontal ones. This also influences the relative positions of positive and negative events, leading to different perceptions of the same object (see Fig. 3). Intuitively, two strong enough saccades of a DVS with different directions should be sufficient to retrieve all the information from a two-dimensional picture. For objects without any prominent edges parallel to the direction of the saccade, just one saccade could be enough for recognition. An extra saccade increases power consumption and delay in recognition, but it may also provide additional information about the object. In real applications, a robot can choose to perform an extra saccade or not, depending on its current knowledge of the object. The same decision can be made regarding the direction of the saccade. As shown later in Section IV, we noticed that more than 94% of the test samples in the N-MNIST dataset could be recognized correctly with only one saccade. Therefore, performing an extra saccade for those samples represented a waste of time and power. In this Section, we look at how we can predict the need for an additional saccade and the best 5Recognition accuracy increases (on average) when more saccades are provided. IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEM, VOL. XX, NO. XX, XXXX 6 Fig. 7: Block diagram of the closed-loop recognition system. The NSP block calculates the best direction for the next saccade, if it is needed. direction for it. We added another block to Fig. 5, called NSP. This block closes the loop of our system, as shown in Fig. 7. Fig. 7 shows the inputs and output of the NSP block. One of the inputs is ˆ Yiwhich is generated by processing one or more saccades. The other three inputs to the NSP block indicate which saccades have contributed to recognition. This information is important to avoid suggesting an already performed saccade again. An ‘OR’ logic can determine which memory block contains non-zero pixels. Memory blocks which correspond to the saccades that have not been executed contain all zero values. The output of the NSP block determines the next saccade direction. The next saccade direction can be one of these four possibilities (see Fig. 7): 1) No extra saccade 2) SAC DD 3) SAC DU 4) SAC H The NSP block is implemented in a closed loop with the SRNN to perform the following tasks in the order shown: 1) Receive events from the first saccade and make the first guess about the object 2) Predict the best direction for the next saccade, if necessary 3) Command the pan-tilt unit to perform the next saccade 4) Combine information from all the saccades performed so far to improve recognition accuracy 5) Continue from step 2) Our experimental results showed that if the SRNN is not sure about the recognition results, the NSP block should request extra saccades. To quantify the amount of uncertainty in a prediction vector, we used the definition of entropy in information theory Hi=X j ˆyijlog2(ˆyij)(2) Fig. 8 illustrates the relationship between entropy and prediction loss (see eq. (1)) for the test samples in the N-MNIST dataset for a specific SRNN. It shows that having a small entropy cannot guarantee a correct recognition. Sometimes, it is possible that the SRNN can be very confident but the answer is wrong. Our experiments show that in these cases, performing extra saccades cannot help to find the correct Fig. 8: Relationship between entropy and prediction loss for the 10,000 test samples of the N-MNIST dataset after presentation of all the three saccades. A two layer SRNN has been used. A sample is classified as correctly recognized when the position of the maximum value in its prediction vector correctly shows the class of the presented digit. For each 0.1 interval in the x-axis, we indicate the percent of test samples within this interval (see boxes on the top part of the figure). answer and the only solution is to improve the SRNN6. High entropy in a prediction vector means less certainty about the results and this is the time when performing extra saccades might be helpful7. The NSP block can decide to ask for an extra saccade when the entropy is higher than a predefined threshold ✓H. This ✓His a user-defined parameter and depends on the cost of an additional saccade. Obviously, choosing a smaller ✓Hwill result in performing more saccades on average and increasing the average recognition accuracy. In Section IV the relationship between ✓H, the average number of saccades and the recognition accuracy for our experiments will be explained in detail. If the NSP block asks for an extra saccade, another mechanism will also be needed to define the best choice among different saccade directions. For this purpose, we decided to extract some statistics from the training set of the N-MNIST dataset. Before performing any saccade, our system does not have any information about the presented object. In this case, the NSP block chooses the saccade that shows the best average performance among all the three saccades during training. For each of the possible next saccade directions, we have defined a vector which is called “Confidence Coefficient Vector” (CCV). To explain how to calculate the CCVs, suppose that the first saccade is SAC1. The possible next saccades are SAC2 and SAC3. The CCV for SAC2 and SAC3 is a 10-element vector 6In this work we do not intend to improve the capability of the SRNN with novel techniques, rather we would like to use the available network as efficiently as possible. 7For example, when the prediction vector is [1,0,0,0,0,0,0,0,0,0] entropy is zero, while it is maximum (3.32) when the prediction vector is [0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1]. In the first case, the network is confident that the presented sample is digit ‘0’ while in the second case, the network is not sure about any of the classes. IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEM, VOL. XX, NO. XX, XXXX 7 which is computed using the following equations: A={i|Li(SAC1,SAC2,Blank)<Li(SAC1,Blank,SAC3)} B={i|Li(SAC1,Blank,SAC3) <Li(SAC1,SAC2,Blank)} CCV (SAC1!SAC2) = mean(ˆ Yi(SAC1)) for all i2A CCV (SAC1!SAC3) = mean(ˆ Yi(SAC1)) for all i2B where: 1) iis the sample number 2) ˆ Yi(SAC1) is the prediction vector for sample iafter presentation of SAC1 3) CCV (SAC1!SAC2) is the Confidence Coefficient Vector when SAC2 is performed after SAC1 4) CCV (SAC1!SAC3) is the Confidence Coefficient Vector when SAC3 is performed after SAC1 5) Li(SAC1, SAC2, SAC3) is the prediction loss function in eq. (1) when SAC1, SAC2 and SAC3 is performed. If one of the inputs is Blank, it means the prediction vector from which the Liis calculated, did not have information about that specific saccade. In other words, the CCV is the average of the prediction vectors for the training set samples with the minimum prediction loss for a pre-defined next saccade. CCVs are calculated once, after training the SRNN. This method can easily be extended for more than three saccades. The NSP block chooses the next saccade which has the most similar CCV to the current prediction vector. In this way, we hope that the upcoming saccade will be in a way that is best for the existing guess of the SRNN. The similarity of two vectors can be calculated by measuring their Euclidean distance. This method, even though it is not a very precise method for choosing the next saccade direction, is nevertheless quite simple. The algorithm for the NSP block can be summarized as follows: 1) The first saccade is always the one that shows the best average results for the training samples. 2) Once the entropy (see eq. (2)) of the prediction vector of the first saccade has been calculated, it is compared with ✓Hto see whether an extra saccade is needed or not. 3) If an extra saccade is needed, the best saccade can be found using the CCVs 4) Calculate the entropy again and compare it with ✓Hto see whether an extra third saccade is needed or not. 5) Continue from step 3) C. Closed-Loop Recognition with a Neural Network Next, we investigated the use of an ANN to predict which saccade should be performed next. In this case, both feed forward and feedback paths will be equipped with neural networks which reduce the complexity of the system. Therefore we replaced the proposed algorithm by a neural network inside Fig. 9: Inputs and outputs of “Next Saccade Prediction Network” (NSPN) the NSP block in Fig. 7. Note that using a Neural Network to predict the next saccade simplifies the implementation complexity (in hardware) and scales well as datasets become more complex. An algorithmic implementation is not so obvious to implement in hardware and may need significant changes when complicating the dataset. Fig. 9 shows the inputs and outputs of the “Next Saccade Prediction Network” (NSPN). Inputs and outputs of this network are highly compatible with the previously presented NSP block. The “Symbol prediction vector” is the output of the SRNN. The other four inputs indicate the currently executed and available saccades. Previously, in the NSP block we only used three inputs for providing this information, which was enough. However, during training, we noticed that the neural network can learn better if the inputs are normalized. We wanted to train the network to predict the best saccade direction for the initial movement as well, so we decided to add an active input for this case which is called “NO-Saccade”. The “NO-Saccade” input is equal to the ‘NOR’ of the other three inputs (SAC DD, SAC DU, SAC H). The NSPN outputs are four values which represent a “cost” for each action. In the current implementation, the pan-tilt unit will move in the direction with minimum predicted cost (Hard- Threshold)8. For example, when the value of output “Cost of No extra saccade” is the minimum, it means the NSPN is suggesting not to do any further saccade, because the current information about the object might be enough. To train this network, we calculated a “cost” for each action for all the possible combinations of inputs and used it for supervised learning. To calculate this “cost”, first, we determined the power and latency cost of an additional saccade. This value, which we call “saccade cost” (or “mechanical cost”) SC, is equivalent to the ✓Hin the previously introduced NSP block. When this value is high, the NSPN is more likely to suggest no extra saccade for the next action. Here, we used the same SCfor all three saccades. Another critical parameter to determine the cost of each action is how much this action will help to reduce the “prediction loss” (see eq. (1)). We define the “cost” of each action as the sum of the “prediction loss” value (which is calculated after performing the action) and the SC Costi=SC+Li(3) For the “No extra saccade”, SCvalue is zero. Otherwise, it is a predefined constant value. From the four possible next 8Another suggestion is not to restrict the DVS to move in the direction of one of these three saccades, but to let it move in a mixture of directions based on the cost of each saccade (Soft-Threshold). IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEM, VOL. XX, NO. XX, XXXX 8 Fig. 10: Different experimental configurations for testing whether feeding direction information can be helpful for learning. actions, the system picks the one whose output predicts the minimum cost. Our goal is that the system learns to reduce the total number of saccades required to achieve the same recognition performance. IV. EXPERIMENTAL SETUPS AND IMPLEMENTATION RESULTS Section IV-A describes the results of our experiments for saccade-based recognition with the DVS moving in a predefined direction. Later on, we describe the results for closed loop recognition, when DVS movement was controlled by the NSP block (Section IV-B) and the NSPN (Section IV-C). For all Neural Network blocks we used Tensorflow python library on GPUs for off-line training. The systems were then implemented as real-time interactive platforms with the DVS on a pan-tilt unit, while using MATLABTM for all the processing. A. Feed forward Recognition with Saccades This Section reports only the test results for the open loop system in Fig. 5 when the DVS moves in a predefined direction (i.e., no saccade prediction). For these experiments we used the pre-recorded N-MNIST dataset [20]. Our goal here is to study under what condition it is relevant to provide specific information about direction of provided TABLE I: Accuracy of experiments in Fig. 10 for different network sizes, using only positive events, after ‘3’ epochs. 1-Layer 2-Layer 3-Layer Experiment 1 91.9% 95.5% 97.7% Experiment 2 70.5% 83.9% 89.2% Experiment 3 95.0% 96.9% 97.2% Experiment 4 82.1% 95.0% 97.3% saccades to improve recognition. To do so, we designed the experiments shown in Fig. 10. In experiments ‘1’ and ‘3’, input hyper-frames are built by allocating saccades in the same positions, while for experiments ‘2’ and ‘4’, saccade positions are intentionally shuffled. Let us call these saccade positions “channels”. To see the effect of network size, three different network sizes for the SRNN were implemented: a 3-Layer network (3C5x5-128FC-10FC), a 2-Layer network (3C5x5-10FC) and a 1-layer network (10FC) 9. The spikes’ polarity bits reveal information about the movement direction. To find out the effect of using spikes’ polarity, we have done all the experiments with and without using spikes’ polarity. Each experiment was carried out once using only the positive polarity events and once using both polarity events. When only the positive polarity was used, input size was 28 ⇥28 ⇥3, while when using both polarities input size was 28⇥28⇥3⇥2. In the second case, the network size was therefore larger. By comparing experiments ‘1’ and ‘2’ in Fig. 10 we wanted to find out if it helps or not to feed all saccades through the same channel. To have a fair comparison, we always used the same input size, but the hyper-frame arrangements are different. In this case, accuracy is calculated by averaging the prediction vectors of all three saccades. By comparing experiments ‘3’ and ‘4’ in Fig. 10 we wanted to find out the effect of feeding explicit information about direction when all three saccades are available. In experiment ‘3’ a hyper-frame contains only one arrangement of saccades (SAC DD/SAC DU/SAC H) while in experiment ‘4’ all six possible shufflings are provided. We also tested the speed of learning. We report the accuracy of the networks after 3 training epochs and also after 50. It should be noted that the number of training samples in each epoch in the different experiments was not equal. While there are 60,000 training samples in the N-MNIST dataset, in experiments ‘1’ and ‘2’ we had 180,000 (60,000⇥3), in experiment ‘3’ we had 60,000 and in experiment ‘4’ we had 360,000 (60,000⇥6) hyper-frames in each epoch. Tables I, II, III and IV show the results obtained in all the Fig. 10 experiments. Based on the results we concluded the following: 1) Feeding explicit direction information accelerates learning. This can be seen by comparing the accuracy of the networks after 3 and 50 epochs. For experiments ‘1’ and ‘2’ which had the same number of hyper-frames in each epoch, 9FC indicates a Fully Connected layer while C indicates a Convolutional layer. For example, 3C5x5 means a convolutional layer with three feature maps and kernel size of 5⇥5and 10FC means a fully connected layer of 10 neurons. IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEM, VOL. XX, NO. XX, XXXX 9 TABLE II: Accuracy of experiments in Fig. 10 for different network sizes, using only positive events, after ‘50’ epochs. 1-Layer 2-Layer 3-Layer Experiment 1 92.6% 97.3% 98.3% Experiment 2 85.2% 89.4% 97.9% Experiment 3 95.4% 97.2% 98.6% Experiment 4 89.4% 96.4% 98.1% TABLE III: Accuracy of experiments in Fig. 10 for different network sizes, using both positive and negative events, after ‘3’ epochs. 1-Layer 2-Layer 3-Layer Experiment 1 94.8% 97.4% 97.2% Experiment 2 74.7% 84.2% 95.1% Experiment 3 96.3% 95.8% 97.8% Experiment 4 80.1% 96.6% 97.6% Table V shows the difference between accuracies after 3 and 50 epochs. It can be seen that by using explicit direction information (experiment ‘1’), the training process converges faster. 2) When network size is smaller, feeding explicit direction information (experiments ‘1’ and ‘3’) improves recognition accuracy. This indicates that larger networks with more learning capacity can extract saccade directions from each frame themselves without the need of explicit information10. Table VI shows the difference between accuracies of experiments ‘1’ and ‘2’ and between accuracies of experiments ‘3’ and ‘4’. It shows that when network size is larger, the difference between accuracies drops. 3) Using polarity of spikes improves the recognition accuracy. This can be because of multiple reasons. First, the network size is larger in this case (input size is twice, which results in more synaptic connections for the input layer). Second, there is additional information about the object in the negative polarity spikes, as the edges producing positive events are not identical to the edges producing negative events. TABLE IV: Accuracy of experiments in Fig. 10 for different network sizes, using both positive and negative events, after ‘50’ epochs. 1-Layer 2-Layer 3-Layer Experiment 1 95.4% 97.8% 98.6% Experiment 2 89.0% 95.2% 98.6% Experiment 3 96.5% 98.0% 98.8% Experiment 4 89.8% 97.0% 98.1% TABLE V: Accuracy difference between ‘50’ and ‘3’ training epochs, for experiments ‘1’ and ‘2’ in Fig. 10, when using only positive events. 1-Layer 2-Layer 3-Layer Experiment 1 0.7% 1.8% 0.6% Experiment 2 14.7% 5.5% 8.7% 10To investigate more about this fact, we have done another experiment. In this test, we trained three neurons (each corresponds to one saccade direction) to predict the direction of a saccade. The input of each neuron was a 28 ⇥28 pixel frame. We found out that this network could predict the saccade direction (which the input frame is made of) with more than 98% accuracy without using spikes polarity. TABLE VI: Difference of accuracies between experiments ‘1’ and ‘2’ and between experiments ‘3’ and ‘4’ in Fig. 10 after 50 epochs when using both polarities. 1-Layer 2-Layer 3-Layer Experiment (1)-(2) 6.4% 2.6% 0.0% Experiment (3)-(4) 6.7% 1.0% 0.70% Fig. 11: Hardware setup for moving the DVS with a pan-tilt unit The third reason is that using both polarities at the same time contains information about the movement direction. 4) In experiments ‘1’ and ‘2’, the final prediction vector for each sample is the average of the prediction vectors for all three saccades. In experiments ‘3’ and ‘4’, the neural network receives all the saccades and mixes them. Experimental results show that, in general, averaging the prediction vectors for all three saccades is not always the best strategy and a neural network itself may find a more optimized way to mix the prediction vectors. B. Using Closed-Loop Next Saccade Prediction Algorithmic Block Fig. 11 shows the hardware setup with the DVS mounted on a mechanical pan-tilt unit so that it can be moved in a desired direction. We have used this setup for a real-time demo [33], however, for reporting the next results, we have used the N- MNIST dataset, so that interested readers can reproduce them. For the closed loop recognition experiments, we selected a 2-layer SRNN (5C5x5-10FC) with the configuration of experiment ‘3’ in Fig. 10 and we used only positive polarity events. Table VII shows the average accuracies of the SRNN TABLE VII: Accuracy of two-layer(5C5x5-10FC) SRNN for the different combinations of input saccades with N-MNIST dataset (only positive polarity events are used) Input saccades Training Accuracy Testing Accuracy SAC DD 95.8% 94.4% SAC DU 95.5% 93.6% SAC H 96.6% 94.5% SAC DD and SAC DU 98.4% 96.9% SAC DD and SAC H 98.7% 97.1% SAC DU and SAC H 98.7% 97.0% All Saccades 99.3% 97.7%