Frequency Analysis of a 64x64 Pixel Retinomorphic System with AER Output to Estimate the Limits to Apply onto Specific Mechanical Environment

Frequency Analysis of a 64x64 Pixel Retinomorphic System with AER Output to Estimate the Limits to Apply onto Specific Mechanical Environment Fernando Perez-Peña1, Arturo Morgado-Estevez1, Alejandro Linares-Barranco2, Gabriel Jimenez-Moreno2, Jose Maria Rodriguez-Corral1, and Rafael J. Montero-Gonzalez1 1 Applied Robotics Research Lab, University of Cadiz, C/Chile 1, 11002-Cadiz, Spain {fernandoperez.pena,arturo.morgado,rafaeljesus.montero, josemaria.rodriguez}@uca.es 2 Robotic and Technology of Computers Lab, University of Seville, Av. Reina Mercedes s/n, 41012-Seville, Spain {alinares,gaji}@atc.us.es Abstract. The rods and cones of a human retina are constantly sensing and transmitting the light in the form of spikes to the cortex of the brain in order to reproduce an image in the brain. Delbruck’s lab has designed and manufactured several generations of spike based image sensors that mimic the human retina. In this paper we present an exhaustive timing analysis of the Address-Event- Representation (AER) output of a 64x64 pixels silicon retinomorphic system. Two different scenarios are presented in order to achieve the maximum frequency of light changes for a pixel sensor and the maximum frequency of requested directions on the output AER. Results obtained are 100 Hz and 1.66 MHz in each case respectively. We have tested the upper spin limit and found it to be approximately 6000rpm (revolutions per minute) and in some cases with high light contrast lost events do not exist. Keywords: Bio-inspired, Spike, Retinomorphic Systems, Address Event Representation. 1 Introduction The human retina is made up of several layers. The first one is based on rods and cones that capture light. The following two additional layers of neurons are composed of different types of cells [1]. Next one layer is composed of bipolar, horizontal and amacrine cells. Each cell has specific skills; the horizontal cells implement a previous filter that has an inhibitory affect on the photoreceptors when light is shone onto them, the bipolar cells are responsible for the graded potentials generation. There are two different types of bipolar cells, ON cells and OFF cells that produce on and off graded potentials. The last type of cells of this layer are the amacrines, they connect * This work was supported by the Spanish grant VULCANO (TEC2009-10639- C04-02). distance bipolar cells with ganglion cells. The last layer of the retina is composed by ganglion cells. They are responsible for the action potentials or spikes generation. Information travels along the optic nerve that is composed by the axons of ganglion cells. Delbruck’s silicon retina [2] consists of a set of 64x64 temporal differential light sensors that mimic the behavior of bipolar cells. There are several researchers from different labs and countries working on this type of bio-inspired and spike-based systems. They are called neuromorphic engineers. Telluride Neuromorphic Engineering Workshop and CapoCaccia Cognition Neuromorphic Engineering Workshops [3][4] are events where these researchers present and interchange ideas and results. This type of systems is called retinomorphic and they have a pixels structure. Each pixel should copy the behavior of one bipolar cell plus the sensing light. This type of system was firstly proposed at 1988 by Mead and Mahowald [5] with an analog model of a pixel. But it was in 1996 when Kwabena Boahen presented two works [6][7] that established the basis for the silicon retinas and their communication protocol. After them, Culurciello [8] described a gray level retina with 80x60 pixels and a high level of response with AER output. The most important fact in all these works is the design of the spikes generator. Other way to design retinas, apart from AER, is with visual microprocessors based on the cellular neural network universal machine. A review of this type of designs could be find at [9]. In this paper we have used the 64x64 Delbruck retina developed under the EU project CAVIAR. This retina uses an Address Event Representation (AER) communication strategy (Fig. 1). If any pixel of the retina needs to communicate a spike, an encoder assigns a unique address to it and then this address will be put onto the bus using a handshake protocol. AER was proposed by Mead lab in 1991 [10] as an asynchronous communication protocol for inter neuromorphic chips transmissions. Two handshake lines of request (REQ) and acknowledge (ACK) are managing the communication. Fig. 1. Spiking pixels using AER communication between neuromorphic chips 2 64x64 AER Retina Chip We used a silicon bio-inspired retina of 64x64 pixels (Fig. 3) designed by P. Lichtsteiner and T. Delbruck at Neuroinformatics Institute at Zurich [2]. This retina generates events corresponding to changes in log intensity, so static scenes do not produce any output. For this reason, each pixel has two outputs, ON and OFF events or two directions if we look through AER. The type of the event depends on the sign of the derivative of the light evolution respect to the time; if there is a positive change of light intensity within a configurable period of time a positive event should be transmitted. Consequently if it is a negative change of light intensity, then a negative event appears on the output AER bus. Due to the sign of the events, the address space used by this retina goes from 0 to 8191; although only 4096 addresses could be spiking. The frequency of output events is proportional to the light amplitude changes. The bigger the light intensity change, the more output events are produced. This frequency of output events can be adjusted through available bias, but in that case the activity of idle pixels is also increased. We have configured the retina in order to reduce the AER traffic of those pixels with no intensity changes, which could imply a decrease of the output frequency of spikes for a particular pixel whose intensity is changing. At Delbruck’s paper [2] there are several tests to characterize the retina but we need to know the behavior at the worst condition in order to use the retina at any industrial application (the aim of Spanish Project VULCANO). That kind of environment typically requires detecting and producing a decision taking and an action to really fast moving or rotating objects. It is very important to know exactly the maximum detected change of pixel light in the AER retina in order to determine the maximum frequency of rotation for a particular object. It is also important to know if there is any lost event at those frequencies. With those data it is possible to determine the best rpm observed for any kind of industrial machinery. At CAVIAR project [11] a standard for the AER protocol was defined by Häfliger. This standard defines a 4-step asynchronous handshake protocol (Fig. 2 and Table 1). Next section presents the experimental methodology to calculate exactly the times expressed at Häfliger standard. With those times and the number of pixels spiking at the same time, it is possible to determine the bandwidth limit of produced events and, if present, the percent of lost events. Fig. 2. Timing for a valid AER transmission with a 4 phase handshake protocol. Single sender and receiver [11]. Table 1. Timing requirements of a 4-step asynchronous handshake protocol (Fig.2) Times min max Avg t1 0s ∞ t2 0s ∞ ≤ 700 ns t3 0s ∞ t4 0s 100 ns t50s 100 ns t6 0s ∞ 3 Experimental Methodology In this section we present and describe two different methods in order to extract the bandwidth limit and the percent of lost events. We have used several AER-tools for these experiments. We have used the jAER viewer and Matlab functions, available at the jAER wiki [12]. Furthermore, a logic analyzer from manufacturer Digiview (Model DVS3100) (Fig. 3) has been used. 3.1 Environment In order to make the tests two different scenarios have been used. The first one described in Fig. 3 is mainly composed of a mechanical drill. The reason to use this type of mechanical tool is because they provided a huge margin of spin frequency. This fact allows us to compare the spin frequency and the maximum frequency of one pixel, which is our first goal. Fig. 3. Picture of the scenario prepared for the first test. Components are 1. Mechanical Drill, 2. Logic Analyzer, 3. Sequencer Monitor AER and 4. 64x64 pixels retina. For the second test, we have taken advantage of the problem from the typical environment of a laboratory. The fluorescent tube makes all the objects at the room change their luminosity with the power network frequency (50 Hz at Spain). For us, that frequency is not visible, but the retina notice it, so it is possible to achieve that all the pixel spiking focus the retina on the tubes. With this scenario, the logic analyzer will show the proper times of each spike and the Häfliger times could be resolved. The Sequencer Monitor AER board called USB2AER, has been used in both experiments. This board consists of two main components, the Cypress FX2LP USB2.0 transceiver and a Xilinx Coolrunner 2 CPLD with 256 macroblocks of digital logic (XC2C256). It has three 16 bit word-parallel AER ports following a 4 phase handshake protocol (2 additional REQ and ACK signals). The power consumption is 60 mA while monitoring and sequencing and the sustained bandwidth of the board is 5 Meps (Mega Events per second) [13]. 3.2 Maximum Spike Frequency For this test we have assembled the retina to an USB2AER Monitor Sequencer Interface [13] that connects an AER bus to the computer sending packets of USB composed of sequences of Address-Events and timestamps that indicate the time instant when the event was coming from the AER retina. In order to determine the frequency it is necessary to focus on a few pixels of the retina. To obtain this response of the retina we have stimulated it with a high range of frequency of the mechanical drill. The range goes from 0Hz to 100Hz (which is equivalent to a mechanical range from 0rpm to 6000rpm). The assembly appears at Fig. 3. The retina has been placed so the drill is stimulating just a few pixels of the retina. These pixels are producing output AER traffic which frequency depends on the drill spin frequency. Using the Java application associated to the USB2AER monitor available at Sourceforge, called jAER viewer, we have captured a sequence of AER. Under Matlab we have extracted the most repetitive addresses and we have processed them in order to know which pixels are spiking. Additionally, all directions were ordered and the most repetitive direction was obtained. Also, we have looked for that direction inside the information from logic analyzer and studied the sampled frequency for that pixel for each spin frequency of the drill. Note that, for high frequency luminosity changes, between two consecutive events of one pixel, all the other active pixel events should fit. 3.3 Maximum Frequency of Requested Directions For the final aim of this test we cannot use the AER monitor board because its USB interface will limit the bandwidth peak of events to the size of the buffer and clock speed. To determine this maximum frequency on the output AER bus of the retina it is necessary to illuminate all pixels in order to saturate the arbiter inside the retina that is managing the writing operation of events on the AER bus. So firstly we need to connect the retina to the AER monitor in order to check that the whole retina is illuminated and, therefore, all the pixels are producing events. Then the AER monitor is disconnected and the retina is connected to the logic analyzer with a jumper connecting the request and acknowledge signals. The aim is to calculate the time expressed by the standard of Häfliger. Captured data by logic analyzer has been processed with Matlab in order to determine the minimum, medium and maximum inter-spikes-interval times. 4 Results and Discussions The results obtained for the first testing scenario are shown in Fig. 4. It shows the evolution of spike frequency for the most repetitive direction calculated in front of the spin frequency of the drill expressed in rpm. The graph shows that when the drill is quiet, there is no output frequency of events. So it can be said that the minimum frequency is 0 Hz. With increments at the spin frequency the spiking frequency level increases up to 100Hz which is the saturation level. Using Matlab, we have fitted the values obtained within a linear polynomial regression. The coefficient of determination r-square of the regression is 0.7684. It shows a quite reliable approximation of the trend. The regression line is represented as a discontinuous line at the graph. The nonlinear and non systematic behavior could be explained from two points of view: on the one hand is the fact that at those kind of industrial machinery with high rpm there is a process to stabilized the head and that could provoke some wrong values; in the other hand is that we have just choose the most repetitive pixel instead of an average of a few of them. Fig. 4. Maximun spike frequency evolution for the spike frequency in front of the spin of the drill expressed at rpm We have raised, step by step, the spin of the drill from 6000 rpm to 7000 rpm in order to test the retina. In this range, the target began to disappear from the retina view. This is the empiric limit for this retina. Another result of this analysis should be highlighted: if the maximum frequency is 100 Hz, it is necessary to fit the 4096 addresses within this 10 ms (Fig. 5) in order to aim no miss events. In both trials, the times by Häfliger standard have been obtained as it is shown in table 2: Table 2. Comparative timing table from the obtained at trials and defined by Häfliger standard Times Häfliger Laser Trial Fluorescent Trial t1 (0 - ∞) 10 ns 200 ns t2 (0 - ∞) 60 ns 30 ns t3 (0 - ∞) - - t4 (0 – 100 ns) 990 ns 60 ns t5 (0 – 100 ns) 20 ns 60 ns t6 (0 - ∞) 370,16 µs 470 ns Note that t3 is included in t4 and it is impossible to be measured because the valid address is still on the bus until the next one arrives. At the fluorescent trial we were looking for the maximum frequency of any requested address. That is the inverse of consecutive request times and the same as the sum of t2, t4, t5 and t6 that result 1.66 MHz (Fig.5). If we join together the 10 ms obtained at the drill scenario between two consecutive events of the same pixel, that could be called time frame, and t2+t4+t5+t6 obtained on the tubes scenario between any two consecutive events, a maximum number of events could appear within these 10ms, as shown in Fig. 5. Fig. 5. Time Frame in the worst condition of luminosity change for the retinomorphic system composed of a 64x64 pixels retina Note that, with these timings, up to 16666 addresses could be placed within the time frame. If we had considered an address space of 4096 pixels, it would have confirmed no lost events. This situation has appeared in high luminosity change conditions, which shows the excellent behavior for an industrial application. 5 Conclusions In this paper we have presented a study for the timing and limits of a retinomorphic system composed of a 64x64 pixels retina in an industrial application. Two scenarios were assumed to test the retina; one to determine the maximum spike frequency and the other one for the maximum request frequency. With those results we have checked the upper limit of spin drill to be approximately 6000 rpm. Also, the results shown in Fig. 5 reveal that in the worst condition of luminosity change for our retina there will be no lost events. Therefore, this AER retina can be used on industrial applications that do not require pixels changing at frequencies higher than 100Hz and that do not produce AER bandwidths higher than 1.66Meps for the present analog bias configuration of the AER retina. References 1. Linsenmeier, R.A.: Retinal Bioengineering. In: He, B. (ed.) Neural Engineering. Bioelectric Engineering, vol. 3, pp. 421–484. Springer, US (2005) 2. Lichtsteiner, P., Delbruck, T.: A 64x64 AER logarithmic temporal derivative silicon retina. Research in Microelectronics and Electronics 2, 202–205 (2005) 3. The 2011 Cognitive Neuromorphic Engineering Workshop, http://capocaccia.ethz.ch/capo/wiki/2011/ 4. Cohen, A., et al.: Report to the National Science Foundation: Workshop on Neuromorphic Engineering, Telluride, Colorado, USA (June-July 2004), http://www.ini.unizh.ch/telluride 5. Mead, C.A., Mahowald, M.A.: A Silicon Model of Early Visual Processing. Neural Netw. 1, 91–97 (1988) 6. Boahen, K.A.: A Retinomorphic Vision System. IEEE Micro. 16, 30–39 (1996) 7. Boahen, K.A.: Retinomorphic Vision Systems. I. Pixel Design. In: IEEE International Symposium on Circuits and Systems, Suppl., Piscataway, NJ, United States, pp. 9–13 (1996) 8. Culurciello, E., Etienne-Cummings, R., Boahen, K.A.: A Biomorphic Digital Image Sensor. IEEE Journal of Solid-State Circuits 38, 281–294 (2003) 9. Roska, T., Rodriguez-Vazquez, A.: Toward Visual Microprocessors. Proceedings of the IEEE 90, 1244–1257 (2002) 10. Sivilotti, M.: Wiring Considerations in Analog VLSI Systems with Application to Field- Programmable Networks. Ph.D. Thesis, California Institute of Technology, Pasadena CA (1991) 11. Serrano-Gotarredona, R., Oster, M., Lichtsteiner, P., et al.: CAVIAR: A 45k Neuron, 5M Synapse, 12G connects/s AER Hardware Sensory-Processing-Learning-Actuating System for High-Speed Visual Object Recognition and Tracking. IEEE Trans. Neural Networks 20, 1141–1438 (2009) 12. jAER Open Source Project, http://jaer.wiki.sourceforge.net/ 13. Berner, R., Delbruck, T., Civit-Balcells, A., et al.: A 5 Meps $100 USB2.0 Address-Event Monitor-Sequencer Interface. In: IEEE International Symposium on Circuits and Systems, ISCAS, New Orleans, LA, pp. 2451–2454 (2007)