A spatial calibrated AER contrast retina with adjustable contrast threshold

A Spatial Calibrated AER Contrast Retina with Adjustable Contrast Threshold J. A. Leñero-Bardallo, T. Serrano-Gotarredona, and B. Linares-Barranco Instituto de Microelectrónica de Sevilla (IMSE-CNM-CSIC). C/ Américo Vespucio s/n, 41092 Sevilla , Spain. E-mail: [email protected] Abstract Address Event Representation (AER) is an emergent technology for assembling modular multi-blocks bio-inspired sensory and processing systems. Visual sensors (retinae) are among the first AER modules to be reported since the introduction of the technology. Spatial contrast AER retinae are of special interest since they provide highly compressed data flow without reducing the relevant information required for performing recognition. Reported AER contrast retinae perform a contrast computation based on the ratio between a pixel’s local light intensity and a spatially weighted average of its neighbourhood. This results in compact circuits, but with the penalty of all pixels generating output signals even if they sensed no contrast. In this paper we present a spatial contrast retina with bipolar output: contrast is computed as the relative difference between a pixel’s local light and its weighted spatial average. As a result, contrast includes a sign and the output will be zero if there is no contrast. Furthermore, an adjustable thresholding mechanism has been included, such that pixels remain silent until they sense an absolute contrast above the adjustable threshold. The pixel contrast computation circuit is based on Boahen’s Biharmonic operator contrast circuit, which has been improved to include mismatch calibration and adaptive current based biasing. As a result, the contrast computation circuit shows much less mismatch, is almost insensitive to ambient light illumination, and biasing is much less critical than in the original voltage biasing scheme. A full AER retina version has been fabricated. In the present paper we provide simulation and preliminary experimental results. I. Introduction AER is a spike based representation hardware technique for communicating spikes between layers of neurons in different chips. AER was first proposed in 1991 in one of the Caltech research labs [1], and has been used since then by a wide community of neuromorphic hardware engineers. Spatial contrast AER retina sensors are of special interest. Computing contrast on the focal plane reduces data flow significantly, although relevant information for shape and object recognition is conserved. In a conventional luminance sensor (a commercial camera) all pixels are sampled with a fixed period and its light intensity (integrated over this period) is communicated out of the sensor to the next stage. In an AER sensor pixels are not sampled. On the contrary, the pixels are who initiate an asynchronous communication cycle, called “event”, when a given condition is satisfied. For example, a spatial contrast retina pixel would send an event whenever the computed local contrast exceeds a given threshold. As a consequence, AER systems are frame-less: there is no sequence of still frames as in conventional video. Rather, the information flows in a continuous manner between AER modules as events are generated at the pixel levels, either in the sensors or at later processing stages. Previously reported spatial contrast retinae [2],[3] compute contrast as the ratio between a pixel’s locally sensed light intensity and a spatially weighted average of its surrounding neighborhood computed with some kind of diffusive network (1) Since this is always positive, let us call it “unipolar” contrast computation. This yields circuits where no subtraction operation is required. This is crucial to maintain mismatch (and precision) at reasonable levels. Note that for computing and circuits have to handle directly photo currents, which can be as low as pico-amperes or less. Performing a simple mirroring operation introduces mismatches with errors in the order of 100% [4]. This can be overcome by increasing transistor area, but then leakage currents may become comparable to the available photo currents. Consequently. while handling photo currents, it is desirable to keep complexity at a minimum. Therefore, from a circuit point of view, the way of computing contrast as in eq. (1) was very convenient. However, this presents an important drawback: when there is no contrast ( ) then . In an AER circuit this means that a pixel sensing no contrast will be sending out information (events) and consuming communication bandwidth on the AER channels. This is contrary to the advantages of AER (where it is expected that only information relevant events will be transmitted) and contrary to the advantages of computing contrast at the focal plane (so that only contrast relevant pixels need to send information). In prior work [3], although spatial contrast was computed by eq. (1) in the retina, a post-processing with AER (convolution) modules was added to effectively compute the contrast as the signed (or bipolar) quantity (2) This reduced significantly the data flow (from about 400keps1to about 10keps), but also at the expense of reducing the speed response of a pixel by a factor of about 10. In the present paper we present a new spatial contrast retina design, where the contrast computation follows eq. (2). The design is based on the original contrast computation circuit by Boahen [2], which has been 1. keps stands for “kilo events per second”. Iph xy,() Iavg xy,() Icont xy,()Iref Iph xy,() Iavg xy,() ------------------------ = Iavg Icont Iavg Iph =Icont 0≠ Icont xy,()Iref Iph xy,() Iavg xy,() ------------------------ 1–   = improved to overcome its inherent limitations on mismatch, ambient light dependence, and critical controllability. Section II summarizes a prior AER mismatch-calibrated contrast retina pixel that followed eq. (1), Section III summarizes briefly Boahen’s spatial contrast computation circuit, Section IV summarizes a more compact calibration circuit than the one used in [3] and which has been used in the present design, and Section V introduces the new pixel design. Finally, Sections VI and VII provides simulation and experimental results respectively. II. Prior AER Mismatch-Calibrated Unipolar Spatial Contrast AER Retina Fig. 1 shows the basic schematic of the contrast computation circuit used in a previous unipolar spatial contrast retina [3]. A p+/nwell photo diode sensed current is replicated twice using a sub-pico-ampere current mirror [5]. The first replica is used in a cascoded diffusive network [6], which implements the discrete approximation of the 2D Laplacian equation (3) This equation provides a good spatial average of over neighboring pixels, such that closer pixels contribute more to this average than distant pixels. The second replica of the photo current is fed together with to a translinear circuit computing the ratio between both, scaled by reference current . The resulting current is thus proportional to a unipolar contrast (as in eq. (1)) and is fed to an integrate-and-fire neuron generating periodic spikes with a frequency proportional to . Scaling current is made locally trimmable for each pixel in order to compensate for all mismatch contributions from the photo diode, current mirror, diffusive network, translinear circuit, and integrate-and-fire neuron. As a result, inter-pixel mismatch contrast computation could be reduced from about to using 5-bit pixel registers to control . Pixel complexity was kept relatively simple (104 transistors + 1 capacitor) thanks to the unipolar nature of the contrast computation, and the whole pixel could be fit into an area of in a CMOS process. The main drawback is that pixels with no contrast would generate output events at a constant rate proportional to . To overcome this, the 4-AER-module system shown in Fig. 2 was assembled to compute effectively a bipolar contrast as in eq. (2). A uniform image AER flow with negative sign bit was merged with the retina AER flow and fed to an AER convolution chip [7] configured to operate as an array of signed (bipolar) integrators. As a result, the background DC component in eq. (1) was removed, yielding a computation equivalent to that in eq. (2). However, as a backside effect, the effective firing rate of a pixel at the output channel was reduced by a factor of around 10, thus diminishing its speed response. In the design presented in this paper, this is solved by performing all the bipolar contrast computation at the sensor chip using an improved version of Boahen’s original biharmonic contrast computation circuit. III. Boahen’s Biharmonic Contrast Circuit A trivial solution to subtract the DC component of the circuit in Fig. 1 is to subtract another DC current from . However, this second DC current also needs to be trimmable to compensate for its own mismatch. Note that the available trimming current will not compensate it together with the rest of mismatch sources [3]. However, the same result can be achieved by using Boahen’s compact biharmonic spatial contrast computation circuit [2]. Thus, by using a slightly more complex diffuser network, we will not require any more the sub-pico-ampere current mirror nor the translinear circuit. The original circuit, in its all-PMOS version, is shown in Fig. 3. The schematic only shows two neighboring pixels of a 1-D retina version. In practice, the horizontal transistors form a 2D mesh. The continuous approximation of this circuit solves approximately the following equations [6] (4) Solving for results in the biharmonic equation used in computer vision to find an optimally smooth interpolating function of the stimulus . Consequently, the output is the second order spatial derivative of the Fig. 1: Block diagram of pixel in prior unipolar contrast retina Iph xy,() Iph xy,() 1λxx2 2 ∂ ∂λyy2 2 ∂ ∂ ––    Iavg xy,()= Iph Iavg xy,() Iref Icont xy,() Icont xy,() Iref σ60%≈σ6%≈ Iref xy,() 58µm56µm× 0.35µm Iref Fig. 2: Setup used to convert the unsigned AER retina output with DC level to a signed AER stream with no DC level Iref 2xy,() Icont xy,() Iref xy,() Fig. 3: Boahen original contrast computation circuit Ihxy,()Iph xy,()a∇2Icxy,()+= Icxy,()Iub∇2Ihxy,()–= Ih Iph Icxy,() interpolation according to the bottom eq. (4). Since the interpolation is a spatially integrated version of the stimulus, can be interpreted as a version of a first order derivative of the stimulus, therefore, spatial contrast.This can also be understood with the help of Fig. 4. The top trace shows a step stimulus and its spatial average ( or ). The center trace shows the contrast computation as , and the bottom trace shows the contrast computation as the second order spatial derivative of . Both are equivalent. According to the bottom eq. (4), includes a DC term . The original circuit implementation of this model suffered from a series of drawbacks. First, mismatch was comparable to output signal. Second, output signal would degrade for the same contrast stimulus when changing lighting conditions. Third, bias voltages and in Fig. 3 had very narrow and critical tuning range. All three drawbacks have been improved with the present implementation. IV. New Compact Calibration Circuit We reduce mismatch by introducing calibration. One dominant source of mismatch is the DC component in the bottom of eq. (4). Since this current is set constant, independent of lighting conditions, we can directly subtract it with a trimmable current source. The output current will thus be directly the bipolar contrast current we were looking for. To implement the trimmable current source, we follow the recently reported very compact circuit based on series transistors association [8]. Fig. 5 shows the basic principle behind this circuit. Each switched MOS operates as a segment of an effective longer MOS whose length is controlled digitally by switching individual segments from ohmic to saturation, and vice versa. The key consists in making each segment to contribute approximately as a power of to the total length. As a result, the effective length is digitally controlled as in a digital-to-analog conversion. Fig. 6 shows the circuitry used to subtract the DC component of the contrast current. Transistors to the left of the dashed line are shared by all pixels, while those to the right are replicated for each pixel. Transistors form a translinear loop [6], thus . Current is a mirrored version of by transistors and . is made of the digitally controlled length MOS of Fig. 5. Consequently, is proportional to this length. is added to the base bias current , thus compensating mismatch originated at and in the pixel. V. The improved spatial bipolar contrast pixel Fig. 7 shows how the pixel has been modified to include a current biasing scheme for controlling the original voltages and in Fig. 3. This way, gate voltages and tend to follow voltage excursions at nodes ‘C’ and ‘H’. The first advantage of this is that biasing will adapt to ambient light conditions. For example, if all photodiode currents are scaled up/down by the same factor, the voltage at all nodes ‘H’ will follow it logarithmically. Also, since is constant, the voltage at node ‘C’ will also tend to follow the same shift. Since bias currents and are kept constant, the gate voltages of transistors and will thus follow also this same global voltage shift, adapting to the global light change. Another beneficial effect of this current biasing scheme is that it attenuates mismatch. After doing careful mismatch analysis and identifying the main sources of mismatch for this circuit, one can find out that transistor and current are the dominant sources of mismatch. The effect of will be compensated by calibration, and the effect of will be attenuated by the current biasing scheme. Note that mismatch in all transistors will introduce random voltage variations at nodes ‘H’ and ‘C’. These variations will be transformed into random lateral currents through transistors and . The random currents through will be collected by output current and can be compensated by calibration. However, random currents through transistors operate as if they were generated by the photodiodes. Thanks to the current biasing scheme, an increase in ‘C’ will increase the gate voltage of the new bottom NMOS transistor, increasing its source voltage, thus increasing the gate voltage of , which will reduce Ih Ic Fig. 4: Interpretation of spatial contrast computations Iph Iavg IhIavg Iph ⁄ Ih IcIu Vcc Vhh Fig. 5: Digitally controlled length MOS used for calibration 2 Fi g . 6: Translinear tunin g circuit M14– Iun=I1I2I3n ⁄ I3nI3Mta Mtb Mtb Iun Ical Iub Iub Fig. 7: Boahen improved circuit with current biasing Vcc Vhh Vcc Vhh Iu Ihh Icc MhMc Ma IuIu Ma Ma MhMcMh Ic Mc Mc the lateral random current. A similar effect will be happening for transistors . Finally, the third advantage is a more robust means for biasing the lateral transistors. In the original scheme, voltages and suffered from a very narrow and critical tuning range (about 100mV or less). Now, bias currents and can be tuned over several decades, while still perceiving their effect. Another modification introduced is thresholding. This is done at the integrate-and-fire neuron, as is shown in Fig. 8. Since contrast current can now be positive or negative, we need two comparators to detect whether the net contrast integration has reached an upper threshold (generating a positive event at pulse+) or a lower one (generating a negative event at pulse-). In either case, the capacitor will be reset to a central voltage . For introducing thresholding a third comparator detects whether capacitor voltage is above or below and turns on either a positive or negative threshold current, which needs to exceed for producing an event. The thresholding transistors in Fig. 8 use large area (2/30) to reduce mismatch. VI. Simulation Results To verify the expected performance of the pixel, Monte Carlo simulations have been performed on a one dimensional array of 100 pixels. The top trace of Fig. 9 shows the output contrast current obtained when applying a uniform optical stimulus (no contrast) before performing calibration. The output contrast current range is . Before calibration . This is equivalent to a precision of 2.8 bits. The bottom trace shows the same, but after calibration, with . This is equivalent to a precision of 6.1 bits. After calibration at uniform 50pA photo current, we apply a factor 2 step stimulus at the photo detectors for different values: from 50 to 100pA, from 10 to 20pA, and from 5 to 10pA, while bias current was always set to 250pA. The response of the contrast computation circuit is shown in Fig. 10. As can be seen, although photo current changes by a factor of 10, output response remains at the same level. VII. Experimental Results A full 32 x32 pixel AER bipolar spatial contrast retina has been designed and fabricated in a 0.35µm CMOS 4-metal process. Fig. 11 shows the layout of a four-pixel cell. The area of an individual pixel is . Pixel layout is quad-mirror-symetric. This way, digital crosstalk to analog circuit is minimized. The n-well photo diodes (P.D.) are placed in the center of each pixel. Their area is . All of the chip, except the photo diodes, is covered with top metal. Fig. 12 shows a microphotograph of the fabricated 3.6x3.5mm2retina indicating the main circuit blocks. The AER retina is currently under test. The scheme used to test the circuit is shown in Fig. 13. The retina outputs are sent to an AER bus. This bus is connected to the board USB2AER [10] which sends the AER data from the bus to a USB computer port. This information is processed by a Java program [9] that converts AER data into a sequence of frames for real-time visualization. The USB2AER board also sends the AER data to another AER bus which is connected to Datalogger. This board has an internal memory that can store up to 524.000 sequencial events and download them to the computer hard disk. Finally, four different boards are used to send instructions from different USB ports of the PC to set up the retina internal registers. To generate programable stimuli, a monitor that can be placed in the retina visual field is used. Stimuli can be generated with Matlab routines. To improve the quality of the images and reduce mismatch effect, the retina should be calibrated before operating with it. The calibration procedure is as follows: • The circuit is illuminated with uniform white light. Ideally, none of the pixels should spike. However, there is mismatch and some pixels spike with a non-zero frequency. • Using the Datalogger, each pixel frequency is measured and saved. Mh Vcc Vhh Icc Ihh Fig. 8: Integrate and fire neuron with thresholding Icont Vhigh Vlow Vref Vref Icont Fig. 9: Output contrast current under uniform stimulus. Top trace is before calibration. Bottom trace is after calibration. Icont Iu ±250 pA±=∆Icont = Icont max Icont min – = 70 pA ∆Icont 7.3 p A = Iu 80 80µm2 × 10x10µm2 • The values of the currents , and are set to get the calibration stairs following the method described in [8]. • Finally, the optimum calibration words, , to minimize output frequency are computed for each pixel and uploaded to their internal registers. • Thresholding can be introduced with the circuit of Fig. 8 to remove post-calibration residual mismatch. Fig. 14 shows an input stimulus and a snapshot captured with the new retina. Grey color indicates no contrast (zero output). Pixels with negative contrast have been plotted within a scale ranging from grey to white and pixels with positive contrast have been plotted within a scale ranging from grey to black. VIII. Conclusions A new AER bipolar contrast retina has been presented. It uses an improved and calibrated version of Boahen’s contrast circuit. Simulation and preliminary experimental results are provided. IX. Acknowledgements This work was supported by EU grant 216777 (NABAB), Spanish grant TEC2006-11730-C03-01 (SAMANTA2) and Andalucian grant P06TIC01417 (Brain Systems). JA. Leñero-Bardallo was supported by a JAE scholarship. Fig. 10: Spatial contrast computation after calibration for a factor 2 step photo current stimulus at different values. Fig. 11: Layout of 4-pixel basic cell. DIGITAL ANALOG P.D. 80um I1I2I3 wcal PIXEL ARRAY BIAS GENERATORS AER ARBITERS AND DECODERS AER ARBITERS AND DECODERS DIGITAL REGISTERS Fig. 12: Microphotgraph of the 3.6x3.5mm2 chip. X. References [1] M. Sivilotti, Wiring Considerations in Analog VLSI Systems with Application to Field-Programmable Networks, Ph.D. Thesis, California Institute of Technology, Pasadena CA, 1991. [2] K. Boahen and A. Andreou, “A contrast-sensitive retina with reciprocal synapses,” in J. E. Moody (Ed.), Advances in neural information processing, vol. 4, pp. 764--772, San Mateo CA, 1992. Morgan Kaufman. [3] J. Costas-Santos, T. Serrano-Gotarredona, R. Serrano-Gotarredona and B. Linares-Barranco, "A Spatial Contrast Retina with On-chip Calibration for Neuromorphic Spike-Based AER Vision Systems," IEEE Trans. Circuits and Systems, Part-I: Regular Papers, vol. 54, No. 7, pp. 1444-1458, July 2007. [4] T. Serrano-Gotarredona and B. Linares-Barranco, "CMOS Mismatch Model valid from Weak to Strong Inversion", Proc. of the 2003 European Solid State Circuits Conference, (ESSCIRC’03), pp. 627-630, September 2003. [5] B. Linares-Barranco and T. Serrano-Gotarredona , "On the Design and Characterization of Femtoampere Current-Mode Circuits," IEEE Journal of Solid-State Circuits, vol. 38, No. 8, pp. 1353-1363, August 2003. [6] A. G. Andreou and K. Boahen, “Translinear Circuits in Subthreshold CMOS,” Analog Integrated Circuits and Signal Processing, Kluwer, no. 9, pp. 141–166, Apr. 1996. [7] R. Serrano-Gotarredona, T. Serrano-Gotarredona, A. Acosta-Jiménez, and B. Linares-Barranco, “A Neuromorphic Cortical Layer Microchip for Spike Based Event Processing Vision Systems,” IEEE Trans. on Circuits and Systems, Part-I, vol. 53, No. 12, pp. 2548-2566, Dec. 2006. [8] J. A. Leñero-Bardallo, T. Serrano-Gotarredona, and B. Linares-Barranco, "A Calibration Technique for Very Low Current and Compact Tunable Neuromorphic Cells. Application to 5-bit 20nA DACs," IEEE Trans. Circuits and Systems, Part-II: Brief Papers, vol. 55, No. 6, pp. 522-526, June 2008. [9] jAER Open Source Project 2007 [Online]. Available: http://jaer.wiki.sourceforge.net. [10]R. Berner, T. Delbruck, A. Civit-Balcells and A. Linares-Barranco, “A 5 Meps $100 USB2.0 Address-Event Monitor-Sequencer Interface”, IEE International Symposium on Circuits and Systems, 2007, ISCAS 2007. Retina Board USB2AER Datalogger Tx Tx Tx Tx Copy Reset I-pots Registers AER BUS AER BUS usb usb usb usb usb STIMULUS usb Fig. 13: Scheme used to test the AER retina Fig. 14: Input stimulus and snapshot captured with the retina.