Programmable 2D image filter for AER vision processing

Programmable 2D Image Filter for AER Vision Processing Teresa Serrano-Gotarredona’, Andreas G. Andreou2, and BernabC Linares-Barranco’ ‘Instituto de Microelectr6nica de Sevilla (IMSE), Centro Nacional de Microelectr6nica (CNM), Ed. CICA, AV. Reina Mercedes s/n 41012 Sevilla, SPAIN. Phone: 34-5-4239923, Fax: 34-5-423 1832, E-mail: terese @imse.cnm.es 2Dept. of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA Abstract A VLSI architecture is proposed for the realization of real-time 2D image filtering in an Address-Event- Representation (AER) vision system. The architecture is capable of implementing any convolutional kernel F(x,y) as long as it is decomposable into x-axis and y-axis components, i.e. F(x,y)=H(x)V(y), for some rotated coordinate system {x,y}, and if this product can be approximated safely by a signed minimum operation. The proposed architecture is intended to be used in a complete vision system, known as the Boundary-Contour-System and Feature-Contour-System (BCS-FCS) Vision Model. I. Introduction Human beings have the capability of recognizing objects, figures, and shapes even if they appear embedded within noise, are partially occluded or look distorted. To achieve this, the human vision processing system is structured into a number of massively interconnected neural layers with feedforward and feedback connections among them. Neurons communicate by means of electrical streams of pulses. Each neuron broadcasts its output to a large number of other neurons, which can be inside the same or at different layers, and the way this is done is through physical connections called synapses [ 11. One big problem encountered by engineers when it comes to implement bio-inspired (vision) processing systems is to overcome the massive interconnections. The Address Even Representation (AER) [2]-[5] approach is a possible solution. Fig. 1 shows a schematic figure outlining the essence behind AER. Suppose we have an “emitter” chip containing a large number of neurons or cells 01, 02, 03, ... whose activity changes in time with a “relatively slow” time constant. For example, if Chip 1 is a retina chip and each neuron’s activity represents the illumination sensed by a pixel, the time constant with which this activity changes can be equivalent to Frame-Rate (i.e., 25-30 changes per second or a time constant of about 30-40ms)’. The purpose of an AER based communication scheme is to be able to reproduce the time evolution of each neuron’s activity inside a second or “receiver” chip, using a fast digital bus with a small number of pins. In the “emitter” chip the activity of each pixel has to be transformed into a pulse stream signal such that pulse width is minimum and the spacing between pulses is reasonably high to time multiplex the activity of a relatively large number of neurons. Every time a neuron produces a pulse its address or code should be written on the bus. For the case more than one pulses are produced simultaneously by several neurons, a classical arbitration tree can be introduced [2]-[4], or one based in Winner-Takes-All (WTA) row-wise competitions [6], or simply by making no neuron accessing the bus in case of a “collision” [7]. Whatever method is used the result will be the presence of a sequence of addresses or codes on the digital bus that one or more receiver chips can read. Each receiver chip must contain a decoding circuitry so that a pulse reaches the neuron (or neurons) specified by the address read on the bus. If each neuron integrates the sequence of pulses properly, the original activity of the neurons in the emitter chip will be reproduced. AER allows easily to add more complicated processing. For example, input images can be translated or rotated by remapping the addresses while they travel from one chip to the next. By properly programming an EEPROM as a look-up table any address remapping can be implemented, by simply inserting the EEPROM between the two chips. Furthermore, many EEPROMs can be connected in parallel each performing, for example, a rotation at a specific angle, and each delivering the remapped addresses to a set of specialized processing chips. In the architecture proposed in this paper, we implement a synaptically weighted projection field for each address read on the bus. This can be done by either having a hard-wired kernel in the filtering chip [8], or by implementing a programmable one, as proposed in this paper. 11. The Programmable Filter The programmable filter described in this paper is intended to be used in a vision model system, known as the Boundary- Contour-System (BCS) and Feature-Contour-System (FCS) [9]. Such vision model consists of a set of layers computing convolutions. The convolutional kernels used in most of these layers F(x,y) are decomposable into x- and y-axis components, F (x, y) = H (x) V (y) , for some rotated coordinate system {x,y} . Using AER allows us to implement a filtering chip only for the coordinate system {x, y} for which F(x,y) is decomposable. To do the filtering for another coordinate system { x’, y’} , rotated with respect to {x, y> an arbitrary angle a , we can use the same chip but providing addresses which have been rotated -a previously. In the filtering chip, the convolutional kernel is implemented as follows. Let us call To (p, q) the sequence of pulses the AER bus receives for address (p,q). Every time a pulse for address (p, q) is received, pulses are sent to all pixels in its vicinity. This way, a lossy integrator at pixel (x,y) of the receiver chip will integrate the sequence of pulses 1. In this paper we consider “Real-Time” a processing that is performed at frame rate (30-40ms) or faster. 0-7803-547 1 -0/99/$10.000 1999 IEEE IV- 159 p+L q+L p-L q-L PI(X,Y) = c F(x-p,y-q)P0(p,q) (1) which are all pulses coming in from its vicinity, weighted by the convolutional kernel F (x, y) . The weighting is performed by modulating the width of each incoming pulse. Thus, every time a pulse is received for pixel (p, q) , a pulse of width F(x-p,y-q) is sent to pixel (x,y) in its vicinity Pulse width modulation is done as follows. When a pulse for coordinate (p, q) is received, all columns x in the vicinity of column p receive a pulse of width IH (x - p) I , and all rows y in the vicinity of rows q receive a pulse of width IV(y - q) I . The values of H (.) and V(.) are stored in a small on-chip RAM. The integrator at coordinate (x. y) receives a pulse of width equal to the minimum of IH (x -p) I and IV(y - q) I . Consequently, the convolutional kernel the system will implement is an approximation to F (x, y) = H (x) V (y) , which is F,,, (x, Y) = sign [ff(x) I sign [V(Y) I min (Iff(x) I, IV(y)l) (2) the signed minimum of the vertical and horizontal components. 111. Circuit Description The address bus provides the coordinates (xo, yo) of the neuron (or pixel) around which the convolutional kernel should be applied. Pulses will be applied to all rows with ycoordinate in the interval [yo - L, yo + L] , and all columns with x-coordinate in the interval [xo - L, xo + L] , where 2L + 1 is the width considered for the kernel. Pulses will be modulated in width according to function IV(y) I for the rows, and function IH (x) I for the columns. At each pixel there is an AND gate which provides a pulse of width equal to the minimum of I V(y) I and IH (x) I . This pulse will generate a fixed magnitude current pulse of the same width which will be integrated on a capacitor. Each pixel contains two integrators. One of them, called the “positive integrator”, integrates the pulses of width min (IH (x) I, IV(y) I) when sgn (H (x) ) sgn (V(y) ) > 0 ; while the other, called the “negative integrator”, integrates the pulses when sgn(H(x))sgn(V(y)) <O. The values of V(n) and H (n) (n = -L, . . .O.. ., L ) are stored digitally on chip in a small RAM. Fig. 2 shows the block diagram of the system. In RAM X and RAM Y digital words of n+ 1 bits are stored (Px-~, ..., Rx,, ..., RxL and Ry-,, ..., Ry,, ..., Ry, ). The first bit Sx, (or Sy,) indicates the sign of the function H (x) (or V(y) ). The following n bits indicate the absolute value IH(x) 1 (or IV(y) 1 ). These n bits linearly control the length of the pulse triggered by monostables Mx, (or Mys ). The monostables achieve this by charging with a constant current a programmable capacitor controlled by the n bits in Rx, or Ry,. The pulses generated by the monostables are sent through lines Tx, (or Ty,) and are triggered whenever an external pulse amves to the system. When an external pulse arrives, the input decoders activate lines xi and y. corresponding to the address of the arriving pulse. Thi selection cells controlled by xi (cells Cxi-, , in Fig. 2, 1 = -L, . . .O.. .L ) connect the pulse in line Tx, ’to line if the sign bit Sx, is ‘1’. If the sign bit Sx, is ‘0’ line Tx, IS connected to the negative line Pni-,. This way, pulses Tx, (or Ty,) are sent through lines Pxi’_, or Pxi-, (PYT-~ or (XE {p-L, ... ) p+L} ,YE {q-L, ..., q+L} 1. Py: ) depending on the sign of the weight stored in Rx, (or R< .’ Each neuron c.. has two integrators. The positive integrator accumulates charge when pulses are simultaneously arriving through horizontal and vertical lines of the same sign. That is, it integrates a pulse when lines Px; and Py+ (or lines Pxi and Py: ) are simultaneously high, or equivalently it performs the operation (Pxt n PyJ’) U (Px; n Pyj) . Similarly, the negative integrator accumulates charge when pulses arriving through horizontal and vertical lines of opposite sign Px: and Py; (or Px; and PyT) are simultaneously high, that is, it performs the operation ( Pxt n Pyj) U (Px; n PyJ’) . Fig. 3(a) depicts the schematic of the selection cell Cxi - , , (or CY^-,^) used to select the neighborhood of cells wh&e the monostable pulses have to be sent. It consists of two NAND gates controlling the PMOS switches MP+ and MP- , and two NMOS pull down transistors MN+ and MN- . Each selection cell Cxi-, , has two control signals (the decoder output xi and the sigh bit Sx, from RAM X), one input signal (the monostable output Txl) and two outputs (PX:-~ and Pxi-,). When a pulse arrives with address (xi,y.) , it I activates the decoders output xi and y.. respectively. The decoder output xi controls all the selectidn cells Cxi- , with 1 E [-L, ..., L] . When xi is high, if the sign bit Sx,’is ‘l’, the selection cell Cxi - ,, , connects the monostable output line Tx, to the positive line Pxt-,. If the sign bit Sx, is ‘O’, line Tx, is connected to the negative line Px;-,. The same is valid for the Y coordinate selection cells. Each synaptic cell ci. has two integrators, the positive and the negative. Fig. 3(bj shows the circuit diagram for the positive integrator. The negative is identical, except for labelling. The integrator is based on the Capacitor-Diode integrator concept for subthreshold MOS operation [4]. This integrator has some interesting properties with respect to a conventional linear RC-integrator: Steady state current is proportional to pulse stream frequency Steady state current is proportional to pulse width Steady state current ripple is independent of current level Steady state is reached, for a given precision, after a constant number of pulses In Fig. 3(b), the two AND and the NOR gates provide a pulse of width equal to the minimum of the pulse width coming in horizontally and vertically. This pulse turns ON current source M, providing a current pulse of ampqtude Iw+(controlled by bias voltage V,). Since transistors M, and M, are biyed in subthreshold, the integrator input and output currents, Ii, and I;, are related by [4] ZJ J J (3) where A = exp ( ( Vdd- VA) hT) , QT = CV~K, vT is thermal voltage and K is a charactenstic subthreshold dimensionless technology parameter whose value may range from 0.60 to 0.98 [ 101. When a train of pulses of width Th and frequency 1/T (T>> Th) is applied to this integrator, the steady state output current and ripple are[4] (4) where z = QdI,,, >> Th. Equation (4) expresses that the relative resolution in the integrator output is constant, IV- 160 independent of the signal level, and that each integrator outputs a current which is proportional to the frequency 1 /T and width Th of the input pulses. If the AER input image pixel intensity is linearly encoded with the frequency of the arriving pulses, and the convolutional kernel is encoded as the pulses width, the output current of the integrators would be the input image filtered by the convolutional kernel. Fig. 4(a) shows an Hspice transient simulation for one of the integrator cells in Fig. 3(b). Transistor sizes are W = 12pm and L = 12pm, integrating capacitor is C = O.lpF, pulse amplitude is Iw = 13.SnA, pulse width is Th = loons, frequency of pulse stream is 1IT = 80KHz , Vdd = SV, and voltage VA was set to 4.67y,-(which yields a current gain from transistor M;‘- to M, of around 2000). Similar simulations were performed by sweeping the frequency of the input pulse stream 1IT and the width of the pulses Th . The results are shown in Fig. S. Fig. S(a) shows the steady-state current level as a function of frequency, while maintaining Th =, lOns . Fig. S(b) shows the steady-state current level as a function of pulse width, while maintaining the frequency constant at 4KHz. Sometimes, in 2D image filtering processing, a rectification operation has to be performed. This is the case, for instance, when doing orientation extraction with Gabor-like kernel filters. The output of the filter is rectified for each pixel [9]. Because of this, the chip scan-out circuitry, which brings out of the chip the state of a cij cell, has been designed to be able to add a rectification operation. The random access scanning circuitry can read the rectified output current of any cell selected by the Random Scan Bus of Fig. 2. The output decoder X (see Fig. 2) selects a column i through pin Sai. When a column is not selected, the output currents I$ and I; of all c.. cells in that column flow to a line of constant voltage VREF i/see Fig. 3(b)). If column i is selected, currents I+ and I:. of all c.. cells in these columns flow to lines 1% and I:, ‘J IJ J respectively, of the scan out cell Scanj shown in F:g. 6. Each scan out cell Scani receives two input currents I,?, I; and provides an output current \I,? - I;\ . Current I: is mirrored through a PMOS current mirror and subtracted from current I+. The PMOS current mirror has an active input clamped to a v’oltage VREF. This maintains a constant voltage VREF at output nodes I$‘- of cells c.. when they are selected, thus speeding up the read out of currents. Current IT - I; enters the current comparator composed of transistors M, , M, and OPAMP, [ll], whose input node (and output 1:. of all selected c.. cells) is clamped to voltage VREF . If current I: - If is positive transistor M, will sink this current. hanihtor M, shares its gate with M, and its source is connected to a voltage reference of value ViEF, thus transistor M, mirrors the current passing through M, , ‘J IJ ‘J If current I+- I: is positive transistor M, sources this current, whic’ is mirrored by transistor M3 because its source is clamped to VREF by the current comparator composed by transistors MS, M6 and OPAMP2. Therefore, the current through M3 and M5 is, J This current is again reflected by the PMOS transistor pair M, ,M, . At the output node, the currents through transistors M, and M, are added together to get the rectified current I~ = 11; -.z~I . S!nce transistors MI - ~7 operate in weak inversion, increasing the source voltage of transistors M4 and M7 with respect to VREF will make the current mirrors M2, M4 and MS, M7 to have a gain higher than one (actually the gain will be exponentially controlled by this voltage difference). This allows to have a current gain such that the output current is of the order of hundreds of VAmps or even some mili -Amps, making it possible to drive this current directly off-chip at high speeds. Fig. 4(b) shows an Hspice simulation of the DC characteristic of a scancell. In this simulation, current I+ was set to 80nA and current I: was swept from OnA to 160nA . Two traces are shown in Fig. 4(b). The dotted line shows the current [$ - 71 + flowing through transistor M, . The solid line corresponds to current [I; - I;] + flowing through transistor M, . Using the analytical solution of eq. (3) and the Hspice simulation results, behavioral system level simulations were performed on the arquitecture of Fig. 2 for a 128 x 128 array, doing a Gabor filtering for vertical lines extraction. Fig. 7(b) shows the output obtain for the input image in Fig. 7(a). J J IV. Conclusions An architecture that implements a programmable 2D image filter has been presented. The architecture allows to implement any 2D filter F(p,q) decomposable into x-axis and y-axis components F(p, q) = H (p) V(q) such that the product can be approximated by a signed minimum. Positive and negative values of H (p) and V (q) can be programmed. The architecture requires an AER input. This allows to rotate the 2D convolution kernel any angle. Circuit simulation results of critical components were given. System-level behavioral simulations of a 128x128 array have been included which validate the proposed approach. V. References Gordon M. Shepherd, The Synaptic Organization of the Brain, Oxford University Press, 3rd Edition, 1990. J. Lazzaro, 3. Wawrzynek, M. Mahowald, M. Sivilotti, and D. Gillespie, “Silicon Auditory Processors as Computer Peripherals,” IEEE Transactions on Neural Networks, May, 1993. M. Mahowald, An Analog VLSI System for Stereoscopic Vision, Kluwer Academic Publishers, 1994. Kwabena Boahen, “Retinomorphic Vision Systems,” Microneuro’96: Fifth Int. Con$ on Neural Networks and Fuzzy Systems, Laousanne, Switzerland, February 1996. A. G. Andreou, R. C. Meitzler, K. Strohbehn, and K. A. Boahen, “Analog VLSI Neuromorphic Image Acquisition and Pre- Processing Systems,” Neural Networks, vol. 8, No. 7/8, pp. 1323- 1347, 1995. 2. Kalayjian, J. Waskiewicz, D. Yochelson, and A. G. Andreou, “Asynchronous Sampling of 2D Arrays using Winner-Takes-All Arbitration,” Proceedings of the 1996 IEEE Int. Symp. on Circuits and Systems (1SCAS’96), Atlanta, vol. 3, pp. 393-396, 1996. A. Mortara, E. A. Vittoz, and P. Venier, “A Communication Scheme for Analog VLSI Perceptive Systems,” IEEE Journul of Solid-State Circuits, vol. 30, No. 6, pp. 660-669, June 1995. P. Venier, A. Mortara, X. Arreguit, and E. A. Vittoz, “An lntegrated Cortical Layer for Orientation Enhancement,” IEEE Journal of Solid-State Circuits, vol. 32, No. 2, pp. 177-186, February 1997. S. Grossberg, E. Mingolla, and J. Williamson, “Synthetic Aperture Radar Processing by a Multiple Scale Neural System for Boundary and Surfwe Representation,” Neural Nefworky, vol. 8, No. 7/8, pp. 1005-1028. 1995. [10]A. G. Andreou and K. A. Boahen, “Translinear Circuits in Subthreshold MOS,” Analog Integrated Circuits and Signal Processing, vol. 9, pp. 141-166, 1996. [l I] A. Rodriguez-Vbquez, R. Dominguez-Castro, E Medeiro, and M. Delgado-Restituto, “High Resolution CMOS Current Comparators: Design and Applications to Current-Mode Function tieneration,” Analog Integrated Circuits and Signal Processing, vol. 7, pp. 149.165, 1995. IV-161 Fig. 1: AER Interchip Communication Scheme (b) I I (e) 4 Fig. 3: Schematic of (a) neighborhood selection cell and (b) D".," one half of core cell &ode-capacitor integrator I Fig. 2: Floorplan of Complete 2D Filterink System (a) l/r x 10' (b) '* 1 lo-' Fig. 5 : Hspice Simulation Results of Core Cell Integrator, (a) Steady-State Current vs. frequency of Input Pulse Strqam for Th=lons, (b) Steady-State Current vs. Th , for l/T=4KHz Fig. 4 Hspice Simulation of (a) Integrator Cell Transient and (b) DC characteristics of Scan-Out Cell Fig. 6: Schematic of a cell to scan out the absolute value of the difference of two currents Fig. 7: System-Level Behavioral Simulations of a 128x128 Array. (a) Input Image, (b) Output Image of Pseudo-qabor Filter extracting Vertical Edges IV- 162