Feedforward Categorization on AER Motion Events Using Cortex-Like Features in a Spiking Neural Network

1 Feedforward Categorization on AER Motion Events Using Cortex-like Features in a Spiking Neural Network Bo Zhao, Member, IEEE, Ruoxi Ding, Student Member, IEEE, Shoushun Chen, Senior Member, IEEE, Bernabe Linares-Barranco, Fellow, IEEE and Huajin Tang, Member, IEEE Abstract—This paper introduces an event-driven feedforward categorization system, which takes data from a temporal contrast Address Event Representation (AER) sensor. The proposed system extracts bio-inspired cortex-like features and discriminates different patterns using an AER based tempotron classifier (a network of Leaky Integrate-and-Fire spiking neurons). One of the system’s most appealing characteristics is its event-driven processing, with both input and features taking the form of address events (spikes). The system was evaluated on an AER posture dataset and compared to two recently developed bio-inspired models. Experimental results have shown that it consumes much less simulation time while still maintaining comparable performance. In addition, experiments on the MNIST image dataset have demonstrated that the proposed system can work not only on raw AER data but also on images (with a preprocessing step to convert images into AER events) and that it can maintain competitive accuracy even when noise is added. The system was further evaluated on the MNIST-DVS dataset (in which data is recorded using an AER dynamic vision sensor), with testing accuracy of 88.14%. Index Terms—Feedforward categorization, AER, Spiking Neural Network, event-driven, MNIST. I. INTRODUCTION NEUROMORPHIC engineering is a growing branch of engineering that takes inspiration from biological neural systems in order to optimize engineered systems. By incorporating novel knowledge from neuroscience, researchers in neuromorphic engineering aim to build electronic systems that have the same efficiency as biological computation [1], [2]. Recent years have witnessed increasing efforts in event-driven neuromorphic systems [3]–[7]. One desire behind these efforts is to emulate the biological usage of the asynchronous sparse event-driven signaling as a core aspect of the computational architecture. Address Event Representation (AER) sensors naturally provide a way to incorporate demand-based computation. AER sensors have an output-by-demand nature. They remove a lot of data redundancy in the scene, and only output the relevant information (i.e. “features”) as an asynchronous stream of digital events, which makes the following processing B. Zhao and H. Tang are with the Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR), Singapore 138632 (e-mail: {zhaob,htang}@i2r.a-star.edu.sg). R. Ding and S. Chen are with School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798 (e-mail: {ding0075,eechenss}@ntu.edu.sg). B. Linares-Barranco is with the Institute of Microelectronics of Seville (IMSE-CNM-CSIC), Spain (e-mail: [email protected]). Correspondence: Shoushun Chen. e-mail: [email protected] systems able to be designed as fully event-driven. In particular, AER vision sensors enable pixel-parallel image processing at the focal plane. Each pixel in the sensor can individually monitor the relative change of light intensity, and it will request to output an event if the change is greater than a user-defined threshold. There are cases wherein multiple pixels request to output events at the same time and, therefore, we need asynchronous row and column arbitration tree circuits to process the pixel requests and arrange the output sequence in a fairly random manner [8]. Only one pixel request is granted at a time. Once the arbitration process is completed, the pixel address is sent out, and the pixel will restart its operation. The output of AER vision sensors is a stream of address events. These sensors are often categorized as temporal contrast AER silicon retinas [9], [10]. Despite many institutions using the AER protocol, interfacing hardware remains difficult and it requires a deep understanding of all the components used. One drawback of AER silicon retinas is the high cost of the silicon area per pixel. Limited feature extraction can be carried out at the pixel level [11], [12] and hence the output events are barely enough for the direct input of classification algorithms. Additional preprocessing, such as segmentation, resizing, repositioning, and even more complicated high level feature extraction, is still needed. However, most existing algorithms are based on conventional frame-driven image sensors. In order to adopt these algorithms, a common practice (jAERViewer [13], for example) is to divide events into fixed time slices (20 ms, for example) and accumulate them into pseudo-pictures. Each incoming event is associated with an address, which is used to “light” the corresponding pixel in the picture. Fig. 1 shows a space-time scatter plot of one piece of address events captured by an AER vision sensor [14]. A person is performing standup and sit-down actions in this recording. The lower part of this figure shows several selected frames reconstructed from the address events. By inspecting these reconstructed frames, we can easily find that the human silhouette in some frames is incomplete or totally missing. The main difficulty arrives from the asynchronous nature of “motion” with respect to the time “slice”. A motion may fall into two time slices and neither of the two pseudo pictures tells the right story. In fact, this is a common problem when using frame-driven image sensors for motion processing. In order to fully utilize the power of AER sensors, the concept of event-driven processing should be applied to every 2 00.5 11.5 22.5 33.5 4 0 10 20 30 40 0 10 20 30 40 x addr y xt (s) Input events some reconstructed frames 20ms Fig. 1. Example of one piece of address events captured by an AER vision sensor. The lower part shows some reconstructed frames by dividing events into time slices and applying accumulation in each slice. We can find that the human silhouette in some frames is incomplete or totally missing. This is a drawback of frame-driven processing on motion events. signal processing stage. For example, event-driven object tracking is studied in [15], [16]. In [15], an embedded vision system is designed and combined with an AER vision sensor to achieve real-time object tracking using an efficient eventdriven clustering algorithm. Delbruck and Lichtsteiner [16] adopt a similar algorithm for tracking, and they further used the tracking results to control a servo motor goalkeeper. Running on a laptop computer, the system can track and block balls with low latency (approximately 2.8 ms). In addition, the event-driven convolution for feature extraction has been exploited in [17]. AER 2D convolution chips for neuromorphic spike-based cortical processing have been designed to accelerate the convolutions of programmable kernels over the AER visual input. These convolution chips have been combined with other AER processing blocks to build larger neuromorphic systems [18], [19]. The “Convolution AER Vision Architecture for Real-Time” (CAVIAR) project [18] is such a massive neuromorphic vision system that it performs sensory, processing, learning, and actuating in a row under the AER hardware framework. The system senses the motion of objects with a temporal contrast silicon retina. It performs feature extraction through convolutional processing and winner-take- all. Learning chips based on spiking neurons are also included in CAVIAR for spatial-temporal pattern classification. The system can recognize and track a rotating dot of a certain size. Although this application is simple, CAVIAR demonstrates the power and potential of the promising AER technology. Moreover, the event-driven convolution is also applied to Convolutional Networks (ConvNets) in [19] to generate a frame-free event-driven ConvNet for feature extraction and categorization on AER visual events. The event-driven ConvNets have a similar architecture with the conventional framedriven ConvNets [20], where convolution and subsampling modules interlace. Due to the frame-free processing, eventdriven ConvNets have a significant improvement in terms of input-output latency. However, the learning of the event-driven ConvNets is based on mapping from frame-driven ConvNets but not naturally spike-based learning. This work seeks to adopt event-driven processing at every signal processing stage. We introduce an event-driven feedforward categorization system, which takes events from a temporal contrast AER vision sensor. The sensor is equipped with direct difference hardware in the pixel and outputs an event if a threshold is reached [14]. The output data is a stream of address events. Each event has an address and a time stamp; the address indicates which pixel the event is from, and the time stamp represents the event’s time of occurrence. Each address event is sent in parallel to a battery of orientation filters based on the Gabor functions [21], and the convolution operation is performed on the fly to generate a batch of feature maps (see Fig.2 and section II for more details). The feature extraction unit is inspired by a recent hierarchical model of object categorization in the primate visual cortex [22], [23]. Each neuron competes with other neurons located within its receptive field, and it can only survive and reach the higher layer if it wins a MAX-like operation [24], [25]. In addition, we proposed an asynchronous Motion Symbol Detector to activate another stage of spike generation. The dynamics of the aforementioned survival neurons, which represent the strength of “features”, go through a small set of neurons that work in Time-to-First Spike (TFS) mode. The generated spikes, again in the form of AER, are fed to a spiking neural network namely tempotron [26] for classification. Developing integrated spiking network models that include both encoding and learning stages for rapid and efficient pattern recognition has attracted increasing interests [27], [28] recently. The full tempotron network is very large. However, due to the MAX operation and the AER nature of the feature spikes, we can achieve the same results as the full network using a very small network that has only 100 inputs. This will tremendously reduce the hardware cost. Our major contribution resides in two areas: 1) an asynchronous Motion Symbol Detector to capture “motion symbols” and then trigger the classification; 2) a virtually fully connected tempotron network that could greatly reduce the hardware cost. The ultimate aim of this work is to develop a real-time human posture categorization system using an AER temporal contrast vision sensor, which does not produce intensity images but rather a train of spikes. In the application of assisted living, due to privacy concerns, the elderly may be reluctant to be monitored by conventional image sensors. The AER vision sensor solves this problem since its event output protects the privacy of the person being monitored. The rest of this paper is organized as follows. Section II describes the system architecture and Sections III - V illustrate its building modules. The experimental results are reported in Section VI. Some discussions are provided in Section VII and conclusions are drawn in Section VIII. 3 II. SYSTEM ARCHITECTURE Fig. 2 shows the architecture of the proposed system. One appealing characteristic of our system is its fully event-driven processing. Similar to most categorization systems, it can be divided into two parts, namely feature extraction and classification. The classifier that we use is a spiking neural network constructed with tempotron neurons, which can efficiently learn and discriminate spatiotemporal spike patterns. The flow of information processing is as follows. 1) Feature Map Construction and Neuron Competition Each address event from the AER vision sensor will be projected onto a group of simple S1filters. Each filter models a neuron cell that has a certain size of receptive field and responds best to a basic feature of a certain orientation. The response of each S1neuron is changing dynamically due to the event-driven convolution as well as a forgetting mechanism. Leakage is introduced to eliminate the impact of very old motion events on current response. Each S1neuron competes with other neurons that are located within its receptive field. It can only survive and reach the higher layer C1if it wins the MAX operation. The survived C1neurons represent some salient bar features [24], [25]. 2) Motion Symbol Detection and Feature Spike Generation Note that S1and C1maps are updated for each incoming AER motion event. In order to avoid carrying out classification on the feature maps all the time, a “Motion Symbol Detector” module is introduced in our system. This module consists of a leaky integration neuron and a peak detection unit. Each input event initiates a postsynaptic potential to this neuron. The total potential is continuously monitored by the peak detection unit. When a peak is detected, a pulse will be triggered to turn ON the switches in Fig. 2. At that moment, C1feature maps are fed to a set of “TFS neurons”, where C1responses are converted into spikes. 3) Categorization by a Spiking Neuron Network The classifier that we use is a network of tempotron neurons. In principle, we need all the C1responses for classification. The number of inputs of the tempotron network is the same as the number of C1responses. Thanks to the MAX operation and the AER nature of the feature spikes, we can achieve a virtually fully connected system by physically activating only a very small subset of the network. Only a small portion of neurons survive in C1 feature maps after the MAX operation and, therefore, we only need to build a few “TFS neurons” for the response- to-spike conversion. Each feature spike is associated with an address, which can be used to access a lookup table (LUT) and fetch the corresponding weight. Note that, in our system, spikes are used in all the processing stages. This is driven by a few design criteria: 1) to avoid falling back to frame-driven processing; 2) to avoid processing the dynamic responses all the time; 3) to reduce the resource requirements for hardware implementation. III. FEATURE MAP CONSTRUCTION AND NEURON COMPETITION A. Related Works Primates’ vision is extremely accurate and efficient in object categorization. This is ascribed to the ventral pathway processing in the visual cortex. It has been a hot topic for decades to model the feature representations in the visual cortex and design systems that mimic the cortical information processing. Until today, our understanding of the visual cortex has been boosted by massive research works in neurobiology and neurophysiology. The current theory of the cortical mechanism responsible for “rapid categorization” has been pointing to a hierarchical and mainly feedforward organization [29], [30]. This organization can provide hierarchical features of increasing complexity and invariance to size and position, making object categorization a multi-layered and tractable problem [31], [32]. Among many neurophysiologically plausible models of information processing in the visual cortex, HMAX, proposed by Riesenhuber and Poggio [22], is one of the most popular feedforward theories. HMAX extends the Hubel and Wiesel classical models of complex cells built from simple cells [33]. It summarizes the basic facts about the ventral visual stream (V1-V2-V4-IT ). HMAX consists of a hierarchy of “S” layers and “C” layers (“S” and “C” follow the notation of Fukushima [34]). The “S” layer cells increase feature complexity by using linear weighted summation of the inputs; while “C” layer cells increase invariance through the nonlinear MAX operation. The HMAX model was further extended by Serre et al. [23]. The whole feedforward architecture remained (S1-C1- S2-C2). The S1and C1layer correspond to the simple and complex cells in primary visual cortex V1, while S2and C2are roughly related to V2and V4, respectively. The first two layers of the Serre model are mostly consistent with the original HMAX (differences exist in the adoption of Gabor filters [21] rather than difference of Gaussians [35]). The last two layers (S2and C2) are where Serre et al. have made significant modifications. Learning is introduced at stage S2. A number of patches are randomly extracted from the C1maps of the training images. Then for each image, the Gaussian radial basis function [36] is applied to the distance between C1 maps and patches, followed by a MAX operation to generate the shift- and scaleinvariant C2features. Promising results comparable to state-of-the-art computer vision systems have been achieved in object recognition tasks on natural images [23]. B. Proposed Cortex-like Feature Extraction Inspired by the aforementioned feedforward models of the cortical information processing (HMAX and Serre model), we propose a convolution-based network to extract features from motion events. For the purpose of simplicity, we only adopt a hierarchy of two layers (S1and C1). Note that, in our model, we use a different MAX operation in C1layer. The eventdriven convolution with a forgetting mechanism is introduced in the S1layer for continuously event-driven processing. The 4 AER Motion Sensor time addr ... Feature Spike Conversion ... time Convolution: 4 scales 4 orientations S1 C1 MAX Competition ... 1 2 N Leaky Integrate neuron Motion Symbol Detector S1 map C1 map bar feature Tempotron Classifier M 2 1 TFS neurons (addr, volt) addr 1 addr 2 addr M M x N weights Weights LookupTable Peak detection Switches Fig. 2. The architecture of the proposed categorization system. The system consists of several building blocks, namely Convolution and Competition, Feature Spike Conversion, Motion Symbol Detector and Tempotron [26] classifier. Each input event is projected onto a group of dynamic S1feature maps through event-driven convolution with a forgetting mechanism. S1neurons compete with local neighbors through the MAX operation to strive for survival in C1 layer. Survived C1neurons represent some salient bar features. The “Motion Symbol Detector” can detect a burst of events in a short time period and then take a snapshot of the dynamic C1feature maps. Survived C1neurons at that snapshot go through a small set of “TFS neurons” to be converted into spikes, which are further fed to a network of tempotron neurons for classification. We use the address of each feature spike to fetch its corresponding weight from the “Weights Lookup Table”. The final categorization decision is made according to the output of tempotron neurons. The lower right part of the figure illustrates the concept of our feature extraction. One S1map is shown at the bottom. The corresponding C1map in the middle has only one survived neuron due to the MAX competition. The neuron’s position is the same as that of the S1peak. The survived C1neuron represents a bar feature of a certain size and orientation at that position. overall data flow can be summarized as motion events → S1maps →C1maps. Simple cells (S1) are used to build feature selectivity. This is performed by convolving the input event with a network of Gabor filters [21]. Each filter models a neuron cell that has a certain size of receptive field and responds best to a basic feature of a certain orientation. Considering both the coverage of various sizes and orientations and the complexity of implementing the algorithm into hardware, we trade-off the network to 4 scales (ranging from 3 to 9, with a step length of 2) and 4 orientations (0o, 45o, 90oand 135o). The function of Gabor filter can be described as: G(x, y) = exp (−X2+γ2Y2 2σ2)×cos (2π λX)(1) where X=xcos θ+ysin θand Y=−xsin θ+ycos θ. The filter parameters (orientation θ, aspect ratio γ, effective width σand wavelength λ) have been well tuned in pioneering work [23], [37], and here we adopt a similar set of these parameters. The filter parameters are listed in Table I. The event-driven convolution is illustrated in Fig. 3. When an input address event comes in, the convolution kernel is overlaid onto the response map at the position specified by the input event’s address. Each element of the convolution kernel is then added to the corresponding original response. TABLE I PARAMETERS OF GABOR FILTERS filter sizes 3 5 7 9 effective width σ1.2 2.0 2.8 3.6 wavelength λ1.5 2.5 3.5 4.6 aspect ratio γ0.3 orientations θ0; π 4;π 2;3π 4 The response map is thereby updated. In addition, in order to eliminate the impact of very old events on the current response map, a forgetting mechanism is adopted. Each pixel in the response map will decrease (or increase) towards the resting potential (usually set as 0) as time goes by. For implementation simplicity, we use a constant linear leakage. It is in this way that we get 16 S1convolution maps. For a certain feature (say a bar), each neuron in the 16 maps gives a response. C1cells are obtained by performing the MAX- like operation over simple S1units. The MAX operation is performed across the local neighborhood to find the center of the feature. As illustrated in Fig. 4, the neurons located in different-scale S1maps have different receptive fields, such as 3×3, 5×5, 7×7 and 9×9. Each neuron competes with all the other neurons located within its receptive field. It can only survive and reach the C1layer if it is the MAX in this area. After the MAX operation, each survival neuron in C1maps represents a feature, i.e., a line segment with a certain size and 5 Event from AER vision sensor Add kernel to Response map Response map after sum Response map due to leakage Address event Conv kernel (a) (b) (c) (d) Fig. 3. Event-driven convolution with a forgetting mechanism. (a) The input event comes in. (b) The convolution kernel is overlaid onto the response map at the position specified by the event address. (c) shows the updated response map after adding the convolution kernel to the map. (d) shows the decayed response map after a while. Scale S1 maps Compete with neighbors Orientation 3 5 7 9 0o 45o 90o 135o B A Fig. 4. MAX over local neighborhood. Neurons located in different-scale S1 maps have different receptive fields, such as 3×3, 5×5, 7×7 and 9×9. Each neuron competes with all the other neurons located within its receptive field. It can survive in C1layer only when it is the MAX in this area. The right 3D figure shows an example of one S1map, in which neuron A will survive in C1layer but neuron B will not. orientation (see the lower right part of Fig .2). Note that in the proposed system, S1 and C1 are updated together for each input event. This process is illustrated in Fig. 5 using a 3×3receptive field as an example. Each input address event from the sensor triggers the event-driven convolution and MAX operation. The input event’s address specifies the operational window (the 3×3red dots in Fig. 5). The convolution involves updating the leakage for these 3×3S1 neurons and then adding the kernel to the S1 map. After convolution, each S1 neuron in the operational window (i.e. each red dot) competes with its 3×3local neighbors (more exactly, 3×3−1neighbors), and will only be fed (written) to the C1 map if it is the maximum among its neighbors. Note that the blue neurons in the figure need to be refreshed to make their values up to date (i.e. to update their leakage) before the MAX operation. This is because the lateral competition/inhibition has to be applied to the responses of the same timing. It does not make sense if a neuron compares its current response to another neuron’s previous response. IV. MOTION SYMBOL DETECTION AND FEATURE SPIKE GENERATION As mentioned above, in frame-driven sensors, a motion may be wrongly segmented into different frames due to the asynchronous nature of “motion” with respect to the time Address event from the sensor Convolution (adding kernel to S1 map) MAX operation (S1 neurons compete with neighbors) Fig. 5. Event-driven convolution and MAX operation using a 3×3receptive field as an example. The input event’s address specifies the operational window (the 3×3red dots), where the convolution and the MAX operation are performed. The blue neurons in the figure need to be refreshed before the MAX operation. “slice”. On the other hand, in the event-driven system, the C1feature maps are updated for each input event. Then when is a good time for classification? Note that the time interval between two consecutive events from the AER motion sensor can be very small (100 nanoseconds or less depending on the handshaking speed of the sensor). To avoid carrying out classification all the time, we propose a time domain clustering algorithm and introduce a “Motion Symbol Detector” module to the system. The word “symbol” is borrowed from the terms used in speech recognition. The AER motion sensor only outputs a few noise events when capturing a static scene, whereas it generates a burst of output events when presented with moving objects. Here, we use the word “symbol” to denote one slice from such a burst of output events. The “Motion Symbol Detector” module consists of a leaky integration neuron and a peak detection unit. As illustrated in Fig. 6(a), each input event contributes a postsynaptic potential (PSP) to the neuron. For an input event received at time ti, the normalized PSP kernel Kis defined as: K(t−ti) = V0×(exp(−(t−ti) τm )−exp(−(t−ti) τs )) (2) where τmand τsdenote the two decay time constants of membrane integration and synaptic currents, respectively. For simplicity, τsis set to be τm/4.V0normalizes PSP so that the maximum value of the kernel is 1. The neuron’s total potential is then obtained by superposition. V(t) = ∑ ti K(t−ti) + Vrest (3) where Vrest is the resting potential of the neuron, which is typically set as 0. A peak detection unit is thereafter applied on the neuron’s total potential to locate temporal peaks. The principle of peak detection is as follows. For a certain timing t0, the potential at that timing is considered as a peak if the following criterion is met: V(t0)≥V(t); ∀t∈[t0−tSR/2, t0+tSR/2] (4) where tSR denotes the time span of the search range. This means the potential at time t0compares itself with all the potentials within its search range [t0−tSR/2, t0+tSR/2]. If 6 time Input event PSP of each event Total potential time time superposition Input event reconstruction Line feature reconstruction At nonpeakAt peak (a) Total potential time t 1 t 1 -t SR /2t 1 -t SR Total potential time t 2 t 2 -t SR /2t 2 -t SR peak not peak search range search range (b) Fig. 6. Motion Symbol Detector. (a) Each input event generates a postsynaptic potential. The integration neuron’s total potential can then be obtained by superposition. (b) Peak detection on the total potential. The potential at a certain time compares with other potentials in its temporal search range. If it is the maximum in the search range, it is considered as a peak; otherwise, it is not. its potential is the maximum, it is then considered as a peak. If we denote t0+tSR/2as tc(current timing), then the potential at timing t0=tc−tSR/2is considered as a peak if V(tc−tSR/2) ≥V(t); ∀t∈[tc−tSR, tc](5) Fig. 6(b) illustrates two examples of peak detection. The upper one shows that V(t1)is not a peak since it is not the maximum among its search range [t1−tSR, t1]; while the lower one V(t2)is considered as a peak since (5) is met. When a peak is identified, a pulse will be triggered to turn ON the switches in Fig. 2. At that particular moment, C1feature maps are fed to the following processing stages. Note that, in order to avoid the detection of very small peaks caused by background noise events, a threshold should also be applied. In addition, we can also add a refractory time to limit the frequency of the output pulse, i.e., the Motion Symbol Detector will remain halted for a while after a pulse has been generated. C1feature maps at a certain moment selected by the Motion Symbol Detector will be fed forward to a set of TFS neurons. Each TFS neuron is in charge of the conversion of one response. All TFS neurons work in parallel and should be triggered simultaneously. As stated in its name, each TFS neuron generates only one spike. The higher the response, the shorter the time to first spike. Let m×ndenote the spatial resolution of the input AER motion events. After the convolution and MAX operation, each C1feature map has the same size as the input resolution, and the number of all the responses in the C1layer is thus 4×4×m×n(we use 4×4filters). A fully parallel response-to-spike conversion would require 4×4×m×nTFS neurons, and thereby lead to huge hardware resource usage. Fortunately, due to the MAX surviving operation, only a small amount (refer to Section VI-A for the detail analysis) of neurons survive in the C1layer (i.e., most C1responses equal to zero). Instead of using all C1responses, we only forward the survival neurons’ responses (nonzero ones) together with their unique addresses (positions within 16 C1maps). After conversion, the addresses of the original responses should be preserved and fed forward together with the corresponding spikes. In this way, the features are encoded back to AER spikes (also called spatiotemporal spikes). Each spike has a time stamp and an address. The time stamp is inversely proportional to the strength of the C1response, and the address indicates the C1neuron’s position. Thereafter, we can use a bioinspired spiking neural network named tempotron to make the categorization decision. V. CATEGORIZATION BY A SPIKING NEURAL NETWORK In this section, we will illustrate how we perform classification on extracted feature spikes using a network of spiking neurons. Various models have been proposed in the literature to describe the dynamics of a single spiking neuron, such as the Leaky Integrate-and-Fire (LIF) model [38], Hodgkin- Huxley model [39], and Izhikevich model [40]. Among these models, LIF has the simplest structure and thus has been widely used. By combining multiple spiking neurons and storing weight information in synapses, we can construct a spiking neural network to learn and discriminate spatiotemporal spike patterns. Experimental studies in neuroscience have revealed a phenomenon namely spike-timing-dependent plasticity (STDP). The synaptic strength will be regulated by the relative timing of presynaptic and postsynaptic spike. Researchers have observed a long-term potentiation of synaptic strength (when a presynaptic neuron fires shortly before a postsynaptic neuron) and a long term depression (when a presynaptic neuron fires shortly after a postsynaptic neuron) [41]. STDP-based rules have been studied in [41]–[43] for the unsupervised learning of spike patterns. In addition to unsupervised STDP rules, supervised learning schemes, such as tempotron [26] and ReSuMe [44], have also been widely exploited. Compared to ReSuMe, which specifies a desired firing time, the tempotron learning rule only needs to label the status of firing or not, and thus it is more suitable for our real-world stimuli categorization tasks. Tempotron is a model of supervised temporal learning that allows a spiking neuron to efficiently discriminate spatiotemporal spike patterns. It utilizes spike timing information and integrates postsynaptic potentials from afferent spikes with different addresses. These properties make tempotron by nature a perfect match for our extracted AER feature spikes. 7 2 1 3 4 5 Afferentpotential V thr V rest Weight change 0 1 -1 (b) Pattern 1 (c) Pattern 2 0T Time 0T Time (a) PSP kernel t max t max 1 0 0510 m τ × m τ × K(t) Fig. 7. The dynamics and the learning rule of the tempotron neuron. (a) shows the PSP kernel. (b) and (c) illustrate the operations of tempotron using two spatiotemporal patterns. The vertical thick bars stand for spikes, and the dash curve beside each bar denotes the PSP kernel generated by the corresponding spike. For pattern1in (b), the total potential crosses the threshold, which means the neuron would fire for this input. If this is an error (the neuron should not fire for this input), then we find all the spikes before tmax and decrease the weights of corresponding afferent synapses. P attern2in (c) does not make the neuron fire, if this is an error, the weights of those afferent synapses which have spikes before tmax will be increased. Note that the curve of weight change is just the mirror of the PSP kernel. A. Tempotron Learning Rule Tempotron uses the LIF neuron model. Each input spike initiates a PSP, which has a fast-rising and slow-decaying shape, as can be seen in Fig. 7 (a). The neuron’s membrane potential is the weighted summation of the PSPs from all the input spikes: V(t) = ∑ i ωi∑ ti K(t−ti) + Vrest (6) where ωiand tiare the synaptic efficacy (weight) and the firing time of the ith afferent synapse, respectively. Vrest is the resting potential of the neuron. Kdenotes the normalized PSP kernel as defined in (2). If the neuron’s potential is higher than a specified threshold, the neuron will fire an output spike and then reset its potential to the resting level. Fig. 7 (b) and (c) illustrate the dynamics and the learning rule of the tempotron neuron using two sample spike patterns. In Fig. 7 (b) the neuron fires since the membrane potential caused by pattern1exceeds the threshold. After firing, the neuron shunts all the following input spikes and the potential gradually decreases to the resting level. In other words, the spikes arriving after the firing time have no impact on the postsynaptic potential any more. In Fig. 7 (c), the neuron does not fire since the membrane potential fails to cross the threshold. The tempotron learning rule aims to train the weights so that the output neuron can fire or not according to its class label. If the neuron is supposed to fire (or not fire, on the other hand) but it actually fails to do so (or does fire, vice versa), then the weights should be modified in the following way. First, we find the peak potential during the effective period and label the corresponding time stamp as tmax. Second, we update the weights using the equation: ∆ωi=       λ∑ ti<tmax K(tmax −ti),if fail to fire −λ∑ ti<tmax K(tmax −ti),if fire wrongly 0,otherwise (7) where λdenotes the learning rate. For example, in Fig. 7 (c), the neuron fails to fire. If this is an error, we need to increase the weights of those afferents that have spikes arriving before tmax. B. Virtually Connected Tempotron Network In principle, we need all the C1responses for classification. In an N-class categorization task, we need Ntempotron neurons, with one for each category. Therefore, the tempotron network has Noutputs and 4×4×m×ninputs, where m×ndenotes the resolution of each C1map and 4×4 represents the number of C1maps. The total number of weights (synapses) is 4×4×m×n×N. The size of the tempotron network is quite large. However, thanks to the beautiful nature of the spatiotemporal AER spikes and the MAX operation where only very few neurons survive after competition, we can achieve a virtually fully connected system by physically activating only a very small subset of the network. We use a LUT to store all the weights (refer to Fig. 2). Each feature spike is associated with an address, which can be used to access the LUT and fetch the corresponding weight. During the training process, for an N-class categorization task, we label the Ntempotron neurons using the one-hot coding scheme. If a pattern belongs to the first class, then the first tempotron neuron’s output is labeled 1(which means it should fire), and all the other neurons’ outputs are labeled 0(not fire). During testing, the decision making for each input pattern is easy: just to check which neuron fires. In order to further improve the performance, we can use multiple neurons for each category [28]. Since the initial weights are set randomly, these neurons will have different weights after training. We then use the majority voting scheme to make the final decision: to check which category has the largest number of firing neurons. Note that throughout our experiments the timings of all feature spikes fed to the tempotron network were normalized into the range of [0,1]. In other words, the total time window of tempotron was set as T= 1. There was no time unit in our tempotron network. The membrane time constant τmin the tempotron was set as 0.1, the learning rate was set as λ= 0.1 and the number of tempotron neurons for each category was set as 10. These parameters are summarized in Table II. VI. EXPERIMENTAL RESULTS A. On AER Posture Dataset We have evaluated the performance of the proposed algorithm on real AER motion events captured from our dynamic 8 TABLE II PARAMETERS FOR THE TEMPOTRON NETWORK Total time window Tof tempotron 1 Membrane time constant τmof tempotron 0.1 Learning rate λof tempotron 0.1 Number of tempotron neurons per category 10 vision sensor. Our AER vision sensor uses logarithmic response pixel circuits, in which the output voltage is a logarithmic function of the amount of light striking a pixel. In addition, the circuits need a threshold to generate temporal difference motion events. The threshold is set to be 100 mv. We captured three human actions, namely bending to pick something up (BEND), sitting down and standing up (SIT ST AND), and walking back and forth (WALK). Fig. 8 shows a few reconstructed sample images. Each row corresponds to one action; images are reconstructed from the AER motion events, using the aforementioned fixed time slice approach with a frame interval of 20 ms. Fig. 8. Some reconstructed frames from our posture dataset. There are three kinds of human actions, each row shows an action. Note that in the proposed system, we only focus on the detection and recognition of abrupt action transitions. We do not focus on the movements that happen at a constant speed since they can be inferred from the last action transition. The system performs recognition only when abrupt changes of body movement occur, such as suddenly bending down, sitting down, and suddenly changing the walking direction. Compared to constant movements, abrupt changes tend to generate more events in the sensor output, causing a burst effect. In our system, we use a Motion Symbol Detector to detect such a burst of events (i.e. a motion symbol) generated by abrupt changes, and then trigger the classification at those moments. 1) Event-driven Centroid Computation: The human’s position may vary in the field of view, especially for the WALK action. In this case, position invariance is necessary for the algorithm. This can be achieved by aligning the human posture silhouette to the center of the scene using the centroid information. The alignment process is simple. We can simply offset the address of each incoming motion event before it is fed to the S1maps. An alternative way is to align the address of C1feature spikes. The latter method involves less computation since the number of survived C1neurons is very small. Fig. 9 illustrates the event-driven centroid calculation. Similar to feature extraction, a map of leaky integration neurons are built. Each incoming event initiates a PSP kernel in the neuron specified by the event’s address. The PSP kernel is the same as equation (2). Let kidenote the PSP kernel of the neuron with address xi, and let ndenote the number of neurons. Using PSP potential as the weight of each address, we can easily calculate AER vision sensor … Address time k1 k2 k3 kn on-the-fly centroid 1 1 n i i i cn i i x k x k = = = ∑ ∑ x1 x2 xn Fig. 9. Event-driven centroid calculation. Each incoming event initiates a PSP kernel for the corresponding neuron. The vertical bars represent events, and the fast-rising and slow-decaying curves depict PSP kernels. By using PSP potential as the weight of each address, the centroid can be easily calculated using the equation shown. 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 0 200 400 600 800 1000 integ potential 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 0 10 20 30 40 t (s) centroid addr row col Fig. 10. Simulation results of event-driven centroid calculation on a stream of address events. A human is first walking to the left and then back to the right side. From bottom to top, the three rows respectively show the centroid curves, the potential curve of the integration neuron in “Motion Symbol Detector”, and some images reconstructed at selected timing points. The green dot depicts the calculated centroid and the arrow means the moving direction. We can see that the centroid curves match the human action well. the centroid address xcusing the following equation: xc= n ∑ i=1 xiki/n ∑ i=1 ki(8) This process can be visualized in Fig. 10. A person is walking to the left and then back to the right side. The middle and lower figures illustrate the potential curve of the Motion Symbol Detector neuron and the event-driven centroid calculation results, respectively. The upper row shows the images reconstructed at several selected timings. The green dot highlights the centroid and the arrow indicates the moving direction. We can see that the calculated centroid follows the human’s action quite well. Note that in this experiment, we assume there is only one person in the scene. If more people exist, we could resort to event-driven clustering algorithms (e.g. methods presented in [15], [16]) to obtain the position of each cluster (person). 2) Parameters Selection: Although very few neurons survive after MAX operation in C1, the number varies for different input scenarios. For future hardware implementation 9 consideration, we need to fix the number. We define a rule as follows: we first get the statistics of the survived neurons in C1layer (e.g. mean µ, standard deviation σ), and then the number of feature spikes (as well the number of “TFS neurons”) is determined by: M≥µ+ 3 ×σ(9) The statistics of the three posture groups are shown in Table III. TABLE III NUMBER OF SURVIVED C1NEURONS Three Groups of Postures BEND SITSTAND WALK mean (µ) 40 45 65 std (σ) 10 7 9 We can see that the number of survived neurons in C1layer is small. According to (9), M≥65+3×9 = 92 and, therefore, using 100 TFS neurons is enough. There are several parameters in our algorithm that need to be tuned according to specific applications. When using our AER vision sensor to observe walking humans, a minimum time of approximately 10-20 ms is needed to reconstruct a humanlike silhouette. Therefore, we set the membrane time constant τmin the Motion Symbol Detector as 20ms. In addition, transition actions like bending and sitting-down last no more than one second during our data collection, thus the search range parameter in the Motion Symbol Detector is set to be tSR = 1s. The leakage rate in the event-driven convolution is set to be 1/τm= 50s−1. These parameters are summarized in Table IV. We also provide the MATLAB codes of the proposed algorithm, which can be accessed from our lab website [45]. TABLE IV PARAMETERS FOR POSTURE DATASET Time constant τmin Motion Symbol Detector 20 ms Search range tSR in Motion Symbol Detector 1 s Leakage rate in event-driven convolution 50 s−1 3) Performance: The posture dataset consists of 191 BEND, 175 SITSTAND and 118 W ALK actions. We randomly pick out 80%of these actions for training and the others for testing. By repeating this evaluation process ten times, we get the average performance. For the training set, we obtain a correct rate of 100%; while for the testing set, the correct rate is 99.48%on average, with a standard deviation of 0.35%. We then ran the algorithm on a continuous event stream which is combined from all the testing actions. The result is shown in Fig. 11. The blue line represents the ground truth of classification, and the red circles denote the decisions made by our algorithm. We can see that the decisions match very well with the ground truth. The proposed system was compared with two popular biologically inspired algorithms: the original HMAX scheme [22] and the model proposed by Serre et al. [23]. The MATLAB codes of these two models can be downloaded from the Web. For the original HMAX scheme, there are 256 C2 features. For the Serre model, 1000 patches are randomly extracted from the C1 layer of the training images and then used for template matching in layers S2 and C2. This leads to 1000 C2 t (s) BEND SITSTAND WALK 0 50 100 150 200 250 300 350 400 ground truth decision made Fig. 11. The performance of the proposed algorithm on our posture dataset. All testing actions are connected one by one into a continuous event stream and then fed to the system for evaluation. We can see that the decisions made by our algorithm (red circles) match very well with ground truth (blue line). features. Some of the patches extracted are blank due to sparse input data, but the ratio is very small (about 1.6%) and will not have a substantial impact on the results. Both the HMAX and the Serre model use the linear Support Vector Machine (SVM) for classification. To perform multi-class categorization on the three-class posture dataset, we implemented the One- Versus-All (OVA) SVM scheme using the LIBSVM library [46]. Since both the HMAX and the Serre model are designed to recognize 2D frames/images instead of events, our AER posture data cannot be used directly. We use the Motion Symbol Detector in our system to select motion symbols. Each motion symbol is a piece of events that took place before a peak timing (that is found by the Motion Symbol Detector). We reconstruct each motion symbol into an image. The reconstructed images are then fed to the HMAX and the Serre model for performance comparison. We randomly pick out 80%of these images for training and the others for testing. The testing results of these two models (averaged from 10 runs) are shown in Table V, where they are compared to the performance of the proposed algorithm. In this table we also report the simulation time taken by per motion symbol (or per image) running on a workstation with two Xeon E5 2.4GHz CPUs and 32GB RAM. The original HMAX scheme has the worst performance and the shortest simulation time due to its relatively simple computation. The proposed system has a performance that is comparable to the Serre model, but its simulation time is approximately 50%less than that of the Serre model. TABLE V PERFORMANCE COMPARISON ON THE AER POSTURE DATASET Models Accuracy on Testing Set Simulation time (mean ±standard deviation) per symbol/image HMAX [22]+SVM 78.65% ±3.37% 0.0285 s Serre [23]+SVM 99.84% ±0.25% 1.0071 s this work 99.48% ±0.35% 0.4991 s B. On MNIST Image Dataset We have further evaluated our algorithm on a standard handwritten digit dataset MNIST [47], which has ten digits (0-9) and 70,000 images in total. Fig. 12 shows some sample images of this dataset. Our algorithm works on AER events instead of images and, therefore, we have to convert these pictures into events. We use a basic thresholding method to convert grey level MNIST images into binary images. The black pixels stand for the background and the white ones for the foreground. Address