Pipeline AER arbitration with event aging

Pipeline AER arbitration with event aging Juan A. Le˜ nero-Bardallo and Fernando P´ erez-Pe˜ na Departamento de Ingenier´ ıa en Autom´ atica, Electr´ onica, Arquitectura y Redes de Computadores Universidad de C´ adiz, Avda. Universidad de C´ adiz 10, 11519 Campus Universitario de Puerto Real, Spain Emails: {juanantonio.lenero,fernandoperez.pena}@uca.es R. Carmona-Gal´ an and ´ A. Rodr´ ıguez-V´ azquez Instituto de Microelectr´ onica de Sevilla (IMSE-CNM), Consejo Superior de Investigaciones Cient´ ıficas y Universidad de Sevilla, C/ Am´ erico Vespucio s/n, 41092, Seville, Spain Emails: {rcarmona, angel}@imse-cnm.csic.es Abstract—We present a simple circuit to handle communication between cells of neuromorphic arrays. It allows cells to operate continuously without waiting for acknowledgement signals back from the AER (Address Event Representation) arbitration circuitry. The module also implements aging of cell petitions i.e., old petitions to access to the AER bus are automatically discarded to give priority to the more recent ones and alleviate the bus congestion. The new arbitration scheme has been implemented and tested. A particular application scenario with an image sensor with spiking pixels that sense light continuously is explained. The sensing errors per event due to discontinued pixel operation can be minimized a factor 8.1. Experimental data obtained with real visual scenes are provided. Keywords: AER, event communication, arbitration, collision discarding, pipeline, fair arbitration, greedy arbiters. I. INTRODUCTION The most extended way of modelling the point to point connectivity between different neurons of the nervous system is the Address Event Representation (AER) protocol [1]–[3]. Neurons share a common bus to communicate between them. When an action potential is produced by one neuron, it places its address on a digital bus via an encoder. Synapses that are supposed to receive that action potential are connected to the same bus via a decoder and get stimulated when their address occurs. AER communication exploits the fast speed of digital buses to amend the limited connectivity that can be achieved with physical wires. Neurons send a request signal to the AER bus indicating that are ready to transmit information. If the bus is not busy, they are allowed to place their address on it. Then, they wait until they receive back an acknowledgement signal to continue their normal activity. The common bus is shared. Hence, when two or more neurons elicit a spike simultaneously, there are collisions that have to be resolved accordingly. Arbiters are usually employed to handle the simultaneous petitions of different neurons. Depending on how collisions are handled, different arbitration schemes can be proposed. Particularly speaking, among the most extended ones, when implementing bio-inspired spiking vision sensors are greedy arbiters [4]. These blocks always give the same priority to one of its inputs when all of them are simultaneously active. As a result, if we arbiter pixels in a vision sensor array with high activity, the same rows will always be able to send information out when there is too much activity. Others, will not be able to send information out of chip. Thus, the spatial/temporal pixel information will be corrupted. Such limitation is imposed by the number of pixels or neurons and the amount of time required to complete the handshaking cycle. The development of bio-inspired vision sensors [5]–[7] has increased the size of pixels arrays and focused the interest of the AER community on developing more efficient and faster arbitration schemes. Fair arbitration approaches give the same priority to all the pixels when they try to access the AER bus [8]. This approach has also some inherent limitations and is not a paradigm of an ideal arbitration scheme. If pixels are frozen for a long amount of time until they are acknowledged, they will not sense changes of the visual scene during that time. Therefore, either greedy or fair arbitration systems can corrupt the temporal information when a sensor is generating event activity close to the maximum event rate that the AER arbitration system can tolerate. To overcome this limitation, some authors proposed more elaborated arbitration techniques with petition aging, i.e. pixels that have been waiting for a long time to transmit information, can be released and continue their normal activity [9]. Unfortunately, the circuitry required in [9] to implement this functionality is more elaborated and the arbitration speed is slow. In this paper, we propose simple AER circuitry to implement fast asynchronous communication with aging. If events cannot be sent after a certain amount of time, pixel petitions will be discarded automatically. Another advantage of this approach is that pixels can continue their normal activity thereafter a spike is elicit. Statically, pixel operation is faster than the handshaking cycle. Death times do not occur because pixels do not need to wait for an acknowledgement signal to continue operating. This requirement is specially demanded by Dynamic Vision Sensors (DVS) sensors with low latency and good temporal resolution. 978-1-4673-6853-7/17/$31.00 ©2017 IEEE _req_y _req_x _ack_y Mn3 Mn5 M n4 C1Mn2 reset_x reset_y Reset Mn1 Mp1 Pixel Circuitry with Continuous Operation Self-Reset SPIKE New AER Circuitry to Implement Aging & Pipeline Operation spike_on Fig. 1. Implementation of the proposed AER logic in a spiking pixel that requires continuous operation. On the left, pixel sensing circuitry. In the middle, proposed logic for the AER communication/arbitration and interconnection with the rest of pixel blocks. Right: extra logic to handle the AER communication signals with the arbitration scheme of Hafliger [10]. Transistors sizes are: (W/L, μm/μm): Mp1=0.5/1, Mn1=Mn2=0.5/0.7, Mn3=1/0.7, Mn4=0.5/0.7, Mn5=0.5/0.7, C1=40fF. The digital buffer was implemented with minimum size transistors. Fig. 2. 100 Monte Carlo simulations showing the transient signals involved in the AER communication and the aging mechanism. Top: In blue trace, transient voltage at the capacitor C1of Fig. 3. In red trace, transient voltage of signal spike on that initiates the AER communication. Bottom: Spike model employed for the simulation. It is active during 25ns. It initiates the AER communication. II. CIRCUITRY FOR PIPELINE PIXEL OPERATION WITH EVENT AGING Fig. 1 displays the new AER circuitry that has to be added in-pixel to implement the proposed arbitration scheme. The new block is compatible with different implementations of the peripheral AER circuitry. For instance, the communication signals plotted in the diagram correspond to the arbitration scheme proposed by Hafliger [10]. It can reach event rates of 10Meps, that is enough to arbitrate low/medium pixel arrays. When the pixel fires, the capacitor C1, that is equivalent to an analog memory, is charged and the req ysignal is active to #10 6 1.1248 1.1248 1.1249 1.1249 1.125 1.1251 1.1251 Voltage (V) 0 2 4 C1 Voltage SPIKE Time (ns) #10 6 1.1248 1.1248 1.1249 1.1249 1.125 1.1251 1.1251 Voltage (V) 0 2 4 reset_x Pipeline Operation Cycle=167 ns Handshaking Cycle=180 ns Fig. 3. Simulation showing several signals involved in the handshaking cycle. It is started when a pixel fires (SPIKE). It finished when the reset signal is received (reset x). It is possible pixel continuous operation because pixels can continue operating after its spike signal is released. initiate the handshaking cycle for the entire pixel row. Then the ack y and reset y signals are received and the req y signal is sent to arbitrate the column petitions. Finally, the reset xsignal is received and the voltage at C1is reset. Usually the handshaking cycle last a few hundred of nanoseconds [4], [10]. The pixel firing frequency is much lower than the average duration of the handshaking cycle. Thus, pixels can operate continuously in time, not being interrupted by the AER handshaking cycle whether there is no AER bus congestion. Fig. 2 displays 100 Monte Carlo simulations of the analog memory when a pixel fires an activates it. Initially, the memory is discharged (see top of Fig. 2). Then, the transient voltage at the capacitor C1reaches a steady state that depends on the transistors leakage currents. Transistors Mn1and Mn2 leakage current are dominant over transistor Mp1. Whenever a pixel fires, the memory voltage is reset. Then the voltage decreases until it reaches the steady state. The digital signal spike on enables and activates the AER communication during the amount of time that it is high. Such period depends on the capacitor’s time constant. In the simulation, the average time that was active was 6.4ms, with a standard deviation of σ=8.95%. The designer has to decide the average time that can keep pixel petitions waiting until they are attended. The spike was modelled with a pulse active during 25ns (see Fig. 2). Note that the signal spike on is never active if the pixel does not fire. Since the aging time is much higher than the handshaking cycle and the pixels firing frequencies, mismatch between different pixel time constant will not affect the array performance. Under high pixel activity, the handshaking cycle last more due to the multiple simultaneous petitions of different pixels. In this case, we can distinguish three possible situations. If the pixel fires once and waits for a long time without being acknowledged, the spike will be discarded because the signal spike on will be released when the voltage at C1reaches a certain value. If two or more bus accesses are required by one pixel without being previously acknowledged, the capacitor will be reset again and the pixel petition will remain active. Only the last event will be sent out of the chip. In the unlikely case that the AER bus resets C1at the same time that the pixel is firing, this spike will be lost. Statically, this method has the advantage of pixel continuous operation. Under normal activity, the handshaking cycle last a few hundred of nanoseconds and pixel pipeline operation is possible. In cases with high AER bus activity, there is temporal loss of the oldest petitions. The maximum temporal error of an event is known beforehand by the user and it is imposed by the time constant of capacitor C1. Moderate event loss still allows an interpretation of the visual scene. It is preferable to the presence of inactive death pixels that appear with classic arbitration schemes when there is heavy congestion [4], [10]. Fig. 3 shows a transient simulation with a pixel array to illustrate the transient voltages when one pixel fires. For the sake of simplicity, we just show the signal that initiates the handshaking cycle, SPIKE; the signal that concludes it, reset x; and the transient voltage at capacitor C1. The duration of the complete handshaking cycle is about 180ns, just taking into account the arbitration logic delays. However, once the pixel disables the signal SPIKE, it can continue operating and sensing changes within the visual scene. Pixel request lasts a few seconds. The pipeline operation lasts 167ns. Thus, idle time gaps due to AER handshaking procedure are avoided with this new approach. This is interesting for DVS sensors that can detect very fast transient changes of the visual scene [5]–[7]. If there is moderate AER bus congestion, the new scheme is even more beneficial because the handshaking cycle will last more. In situations of AER bus saturation and high handshaking delay, spikes will be discarded if petitions are not attended during a time interval equivalent the capacitor time Tint event #1 event #2 event #(n-1) event #n Vint V reset Time Voltage V bot 'V T +T eff reset T eff (a) Tint event #1 event #2 event #(n-1) event #n Vint V reset Time Voltage V bot 'V T +T eff arb T eff (b) T +T +T eff arb reset Fig. 4. (a) HDR pixel transient voltage with the proposed AER logic. Operation error is only introduced by the self-reset time (Treset). (b) Pixel transient voltage with a conventional arbitration scheme. The dead time introduced in the pixel operation is equal to the sum of Treset plus the duration of the handshaking cycle, Tarb. (a) (b) (c) Fig. 5. (a) Snapshot with a commercial camera of a visual scene with dynamic range of 126dB. (b) Illumination levels measured with the HDR sensor. (c) Representation of the HDR sensor outputs using tone mapping compression. constant. III. EXPERIMENTAL RESULTS AND VALIDATION The proposed arbitration scheme was validated with the implementation of a HDR image sensor [11]. Its pixels perform a light to frequency conversion and send events asynchronously. For design requirements, pixels must sense light continuously avoiding temporal gaps due to the AER handshaking cycle. An array of 96×128 pixels was implemented. The sensor pixel operation is depicted in Fig. 4. Pixels integrate charge at a integration capacitor during the integration time (Tint). Every time that the pixel voltage at the integration capacitance reaches a voltage threshold, pixels self-reset and transmit an event off-chip. The illumination level is computed digitizing the voltage Vint at the end of Tint and taking into account the number of events (if any) associated to each pixel. The less representative bits correspond to the value of Vint. The more significant bits are associated to the number of events. Obviously, pixel continuous operation is mandatory to minimize the error (maximize Teff in Fig. 4). With the proposed AER logic to operate continuously, the only error is due to the amount of time required by the pixel to reset itself (Treset). With a conventional arbitration approach, the error would be equal to duration of the handshaking cycle (Tarb) plus the pixel self-reset (Treset). The value of the reset time is Treset ≈25ns is always the same. However, the value of Tarb is undetermined. In situations with low congestion in the bus, Tarb ≈180ns in our implementation. Under situations of very high event activity, Tarb is higher. Thus, the first approach will introduce less error in the measurement. As long as, Tarb <T eff +Treset, there will not be event loss. We can define the error per event as the ratio between the amount of time that the pixel is not sensing light and the amount of time required to fire one event. In the situation of Fig. 4.(a), it will be given by: a=Treset Teff +Treset (1) And in the situation described in Fig. 4.(b): b=Treset +Tarb Teff +Tarb +Treset (2) According Equations (1) and (2), the maximum error will be obtained when Teff reaches its minimum value, i.e. the pixel is exposed to the maximum level of illumination. It is not easy to estimate the minimum value of Teff in a real scenario. Let us considerer a visual scene like the are depicted in Fig. 5.(a) with a large intra-scene dynamic range of 126dB. There is a very bright light source that is part of a visual scene. A few pixels are exposed to very high levels of illumination and fire at high frequency without saturating the AER bus. In Fig. 5.(b) the illumination values are plotted with a color scale. In Fig. 5.(c) a tone mapping image, closer to our perception of the visual scene, was rendered with the sensed illumination values. The integration time was set to Tint =110ms and the number of events associated to the highly illuminated pixels were #events=7,003. Therefore, the value of Teff is approximately given by: Teff =Tint #events −Treset ≈15.6μs(3) Substituting Teff in Equations (1) and (2), the errors per event are a=0.16% , b=1.30% and b/a=8.10. Thus, sensing errors due to discontinued pixel operation can be minimized with this approach. Furthermore, under situations of heavy congestion in the bus, old pixel petitions will be discarded after a certain amount of time. In this particular implementation, we used external greedy arbiters in the AER periphery [10]. The approach is compatible with fair external arbiters [8] or different pixel architectures too. IV. CONCLUSIONS We have described simple AER circuitry to allow pipeline operation of cells in neuromorphic arrays. A particular example with pixels of a event-based sensor has been displayed. Pixels can continue integrating charge thereafter they fire requesting access to the AER bus. Thus, they can operate at high speed remaining unaltered by death times due to the handshaking cycle. Furthermore, petitions that are unattended during a long period of time, are automatically discarded, releasing the amount of load of the AER periphery in situations of heavy congestion. Error estimations due to arbitration delays based on experimental measurements with pixels exposed to high activity have been reported. The proposed approach reduces the error significantly. V. ACKNOWLEDGEMENTS This work was supported in part by the University of C´ adiz under Grant PR2016-072; in part by the Spanish Ministry of Economy and Competitiveness under Grant TEC2015-66878- C3-1-R, Co-Funded by ERDF-FEDER; in part by Junta de Andaluc´ ıa CEICE under Grant TIC 2012-2338 (SMARTCIS- 3D); and in part by ONR under Grant N000141410355 (HCELLVIS). REFERENCES [1] M. Silvilotti, “Wiring considerations in analog VLSI systems with application to field-programmable networks,” Ph.D. dissertation, Cal. Inst. of Tech., Pasadena, California, 1991. [2] M. Mahowald, An Analog VLSI System for Stereoscopic Vision. Kluwer, 1994. [3] A. Mortara and E. A. Vittoz, “A communication architecture tailored for analog VLSI artificial neural networks: intrinsic performance and limitations,” IEEE Transactions on Neural Networks, vol. 5, no. 3, pp. 459–466, May 1994. [4] K. A. Boahen, “Point-to-point connectivity between neuromorphic chips using address events,” IEEE Trans. Circuits Syst. II, vol. 47, no. 5, pp. 416–434, 2000. [5] P. Lichtsteiner, C. Posch, and T. Delbruck, “A 128 ×128 120dB 15μs latency asynchronous temporal contrast vision sensor,” IEEE Journal of Solid-State Circuits, vol. 43, no. 2, pp. 566–576, February 2008. [6] C. Posch, D. Matolin, and R. Wohlgenannt, “A QVGA 143dB dynamic range asynchronous address-event PWM dynamic image sensor with lossless pixel-level video compression,” IEEE Journal of Solid State Circuits, vol. 46, no. 1, pp. 259–275, January 2010. [7] J. A. Le˜ nero-Bardallo, T. Serrano-Gotarredona, and B. Linares-Barranco, “A 3.6μs latency asynchronous frame-free event-driven dynamic-vision- sensor,” IEEE Journal of Solid-State Circuits, vol. 46, no. 6, pp. 1443– 1455, June 2011. [8] A. M. T. Linn, D. A. Tuan, C. Shoushun, and Y. K. Seng, “Adaptive priority toggle asynchronous tree arbiter for AER-based image sensor,” in 2011 IEEE/IFIP 19th International Conference on VLSI and Systemon-Chip, Oct 2011, pp. 66–71. [9] J. T. Marienborg and T. S. Lande, “Analog state transmission with digital hardware,” in NORCHIP 1998, 1998, p. 249256. [10] P. H¨ afliger, “A spike based learning rule and its implementation in analog hardware,” Ph.D. dissertation, ETH Z¨ urich, Switzerland, 2000, http://www.ifi.uio.no/ hafliger. [11] J. A. Le˜ nero-Bardallo, R. Carmona-Gal´ an, and A. Rodr´ ıguez-V´ azquez, “A high dynamic range image sensor with linear response based on asynchronous event detection,” in 22nd European conference on circuit theory and design, ECCTD 2015, August 2015, pp. 1–4.