Toward visual microprocessors

Toward Visual Microprocessors TAMÁS ROSKA, FELLOW, IEEE, AND ÁNGEL RODRÍGUEZ-VÁZQUEZ, FELLOW, IEEE Invited Paper This paper outlines motivations and models underlying the design of visual microprocessors based on the cellular neural network universal machine. We also overview the state of the art regarding the realization of these microprocessors in the form of very large-scale integration chips. Examples corresponding to measurements realized on these chips are enclosed for illustration purposes. Keywords—Analogic cellular supercomputing, cellular neural networks, CNN technology, visual microprocessors. I. INTRODUCTION For more than 100 years, the living visual system of mammals has been intensively studied by neuroscientists and biophysicists alike. Recently, computer engineers have been active creating machine vision systems. Still, although many ideashavebeenproposedandimplementedinsilicon[1]–[3], including resistive grid “silicon retinas,” programmable cellular neural/nonlinear network (CNN)1models of the visual pathway, as well as many “smart optical sensors,” no complete neuromorphic model of the topographic parts of the visual pathway has been made available. The reason is simple: the lack of understanding of the detailed operation of many key components located atthe front-end of the visual system, notably, the retina and the lateral geniculate nucleus (LGN). Hence, the representation of the visual scene from the input to the higher layers has been unknown. Of the many exciting Manuscript received May 31, 2001; revised February 15, 2002. This work was supported by grants from the Hungarian Academy of Sciences, the Spanish MCyT (Project TIC1999-0826), the National Research Fund of Hungary (OTKA), the CEE (Project IST-1999-19007), and the Office of Naval Research (Projects N00014-00-C-0295, N68171 97-C- 9038 and N68171 98-C-9004). T. Roska is with the Analogic and Neural Computing Laboratory, MTA- SzTaki(HungarianAcademyofScience)andPázmányUniversity,Budapest H-1111, Hungary (e-mail: [email protected]). Á. Rodríguez-Vázquez is with the Department of Analog and Mixed-Signal Circuit Design, IMSE/CNM, 41012 Sevilla, Spain (e-mail: [email protected]). Publisher Item Identifier 10.1109/JPROC.2002.801453. 1Cellular neural/nonlinear network (CNN) models were introduced by Chua and Yang in 1988 [5], and then generalized and used as a model for bionic eyes by Chua, Roska, and Werblin [6]–[8]. Their principles and applications for visual processing are covered in [9]. partialresultsrelated tothe visualpathway,somerecent findings (see, for instance, [4]) suggest a few sound principles. • Sensing and processing are interactive processes, and the processing is mainly analog, combined with masks of binary (yes/no) maps. • The basic structure is composed of several stacks of layers of neurons connected by local receptive field organizations with different spatial distributions and time constants. • The processing strategy is a kind of “multiscreen theater”; namely, from a given visual scene, several parallel maps are generated and then further processed. This is true even in the mammalian retina [4] where about a dozen parallel channels are organized. To implement neuromorphic visual models on silicon, we have two ways: • Pick up a specific task and its model and implement it on silicon. This is the usual way, leading to very useful, task-specific smart sensors. • Make mixed-signal2visual microprocessors. That is, processors which combine optical sensing with analog cellular spatial-temporal dynamics and some form of logic (they are called analogic processors because they combineanalogandlogicprocessingstructures),which have receptive fields like elementary instructions, and the possibility of storing and executing user-selectable sequences of instructions (programs). Clearly, the second approach is more demanding in terms of architecture, very large-scale integration (VLSI) chip design, and computational infrastructure, leading to a new type of hardware/software system design. This paper focuses on the second approach. Namely, we will briefly review the analogic cellular computer architecture,someCMOSprototypechipsrelatedtothatarchitecture, and the accompanying computational infrastructure. Some examples measured from the so-called ACE4K chip [10] and the CACE1K chip [11] are included for illustration purposes. The former has a one-layer architecture, while the latter has a three-layer architecture inspired by the CNN model of the 2Mixed-signal means that analog and digital signal representations are combined, and hence analog and digital signal processing. 0018-9219/02$17.00 © 2002 IEEE 1244 PROCEEDINGS OF THE IEEE, VOL. 90, NO. 7, JULY 2002 Authorized licensed use limited to: Universidad de Sevilla. Downloaded on April 15,2020 at 14:17:43 UTC from IEEE Xplore. Restrictions apply. mammalian retina proposed in [12] based on the discoveries about the functionality of the inner part of this retina as reported in [4]. II. CNN-BASED VISUAL MICROPROCESSORS Back in the 1960s, the building blocks for logic design had been the various logic circuits (micromodules) implementing different “smart” logic tasks. These had also been used to make digital computers. The digital computer has a key attribute due to J. Von Neumann, namely stored programmability. It means that the same core architecture, via algorithms coded in software, can be used for a myriad of tasks. Or, to put it in another way, the architecture is open to the human intellect for millions of algorithmic innovations. This is the functional secret behind the success of the digital microprocessor, first made in the early 70s. Visual microprocessors aim to mimic this functional secret. However, they aremixed-signaldeviceswhichrealizeanalog-and-logicspa- tial/temporal processing tasks (wave processing), and hence require quite different building blocks [3]. The front-end “devices” encountered in natural vision systemsarecapableofacquiringandprocessingimagesinacon- current manner. The retina contains photoreceptors and dynamicallycoupledprocessingcellsofdifferenttypes.Among many other tasks, the early processing realized at the retina servestoextractimportantfeaturesfromtherawsensorydata and, thus, to reduce the amount of information transmitted for subsequent processing. In contrast to that, image acquisition and processing are usually separated in conventional artificial vision systems. One key aspect of visual microprocessorsistheintegrationofsensingandstoredprogrammable processing (SPP) at the analog signal array level—the integrated SPP principle. Among many other things, this allows us to tune the sensors dynamically, pixel by pixel, depending on the content and even on the context of the changing scene. Some of the key architectural aspects have been discussed in [13]. Some features which make the visual microprocessors addressed in this paper different from other topographic smart sensors [1], [2] include the following. • They use a core analog processing array (a CNN [5]–[7]) with tunable interaction weight patterns and embedded pixel-wise data memories. • This programmable and reconfigurable array is embedded in a computer architecture resulting in the so-called CNN univesal machine (CNN-UM). • The CNN-UM is stored programmable and capable of implementing analogic spatial–temporal algorithms through the smart synergy of hardware and software. All the signal variables are continuous, except for the discreteness in space (pixels orvoxels). At the same time, visual microprocessors retain the extraordinary strength of digital computers, their unconstrained variability via programming or software. Obviously, such software and related algorithms are different from conventional ones. Below we summarize the main architectural and algorithmic ideas underlying CNN-based visual microprocessors. It is worth mentioning that although most of their present-day applications are related to vision, many other Fig. 1. A typical simple CNN structure. Fig. 2. The standard output nonlinearity. topographic problems (tactile and auditory), including topographic optimization, are among the emerging applications. A. CNN Dynamics CNNs can be either single-layer or multilayer. Consider first a single layer consisting of a two-dimensional (2-D), regular grid of cells , where and are the row and column coordinates. The topography of such a structure is shown in Fig. 1. Assume each cell hosts a processor with its real-valued input, state(s), and output signals, , and , respectively. In such a 2-D layer, each cell processor is connected to its neighbors (in a 3 3or5 5, etc., neighborhood or sphere of influence), denoted by . The simplest first-order cell state dynamics is given by3 (1) where is called the threshold of the cell and are called the feedback and feedforward synaptic operators or templates; in case of a 3 3 neighborhood of radius 1, they are 3 3 matrices. The state and the output signals of each cell are typically related through the following nonlinear output equation: (2) depicted in Fig. 2. However, the nonlinearity could be of several types and it could also be included in a simpler dynamic equation form. Namely, the standard nonlinearity 3The time is scaled in the relative time unit  which is the time constant of the simple first-order cell dyanmics. ROSKA AND RODRÍGUEZ-VÁZQUEZ: TOWARDS VISUAL MICROPROCESSORS 1245 Authorized licensed use limited to: Universidad de Sevilla. Downloaded on April 15,2020 at 14:17:43 UTC from IEEE Xplore. Restrictions apply. Fig. 3. The initial picture and the diffused picture using a diffusion template defined by gene G . in (2) and the cell-state dynamics represented by (1), the so-called Chua–Yang model, could be replaced by the full-range model which means that , and that the first term in (1) is replaced by a nonlinear function whose shape is the inverse of that used for the standard nonlinearity [14]. Once the cell dynamics is fixed, the interaction patterns and and the offset value define the functionality of the CNN layer. Given an input signal array for , defined as a picture with pixel values , the set of values determines the outcome of the CNN dynamic process. This set is called a cloning template or a gene. In the space-invariant case, the templates are 3 3 (or 5 5or7 7) matrices. This means that a CNN array can be defined by the cell dynamics and the 19 (or 51 or 99) numbers of the templates and the offset . The input image could be either static or dynamic; hence, a CNN layer plays the role of an image processor. The peculiar property of controlling the functionality of a whole array of interconnected cells by means of just a few interconnection weights (e.g., 19 numbers) is very familiar to neurobiologists. Indeed, the cloning template is no more than a receptive field organization in the retinotopic part of the visual pathway [8]. On the other hand, the CNN paradigm is well suited for representing many topographic sensory modalities via their receptive field organizations. The first attempts [15] have been followed by many other useful results. In a nontrivial case, the CNN dynamics is a wave acting for a finite time . For example, for a diffusion template or gene we have (3) Fig. 3 shows the initial state and the output image (at elapsed time). There exists a very wide catalog of templates covering a myriad of applications. Also, because these templatesareprogrammablebydefinition,learning canbeincorporated to adapt the templates either globally, for example, using a genetic algorithm [16], or locally. Thus, not only associative memories can be constructed, e.g., [17], but the plasticity of the brain might be directly modeled [13]. Fig. 4. The extended cell of the CNN-UM. B. The CNN-Universal Machine (CNN-UM) [7] If we furnish each CNN cell processor with local memories [local analog memory (LAM) and local logic memory (LLM)] and a local communication and control unit (LCCU) to send/receive information to/from the global analogic programmingunit(GAPU), wegetthe extendedCNNcell ofthe CNN-UM architecture. For practical reasons, in each cell we add a local logic unit (LLU) and a local analog output unit (LAOU) which take inputs and send outputs from/to their local memories, LLM and LAM, respectively. Fig. 4 shows the extended cell schematically. The GAPU is the conductor of the extended cell array, communicating with each cell via the LCCUs of each cell. The GAPU contains three registers and a global analogic control unit (GACU), the latter of which is the host of the stored program and controls the whole array computer. The three registers store the cloning templates [analog programming-instruction register (APR)], the local logic instructions [logic program-instruction register (LPR)], and the switch configuration codes [switch configuration register (SCR)], respectively. The CNN-UM can be viewed as an array computer defined on flows [18]. Algorithms can be constructed where the elementary instruction is the solution of a partial differential equation (PDE). This correspondence was highlighted already in the seminal paper [5] for the heat equation; also, in [19], a mechanical system was modeled by a CNN. Later, systematic methods have been devised to convert PDEs defined in continuous space into CNN dynamics [20]. Recent advances in compleximage processing show that PDE-based techniques seem to be superior in many respects (e.g., [21]). The drawback is their high computational complexity when implemented in digital processors. Here, using a CNN, solution of a nonlinear PDE is the basic task. The next example shows a complex analogic spatial/temporal algorithm used for the calculation of the inner boundaries of the left ventricle in an echo-cardiogram [22]. Active waves [23] are used as algorithmic steps. For reference, we 1246 PROCEEDINGS OF THE IEEE, VOL. 90, NO. 7, JULY 2002 Authorized licensed use limited to: Universidad de Sevilla. Downloaded on April 15,2020 at 14:17:43 UTC from IEEE Xplore. Restrictions apply. Fig. 5. The bold arrows represent different cloning templates. Some of them are performing the solution of complex nonlinear PDEs as elementary instructions; these are written on the left-hand side of the figure with their execution times on the right-hand side. In addition, several simpler instructions and templates are used, for instance, local logic operations. also show the execution times of the algorithmic steps on the so-called ACE4k chip [10]. C. Example 1 A flow diagram is depicted in Fig. 5 of the analogic CNN algorithm with some typical intermediate results. Observe that it can be interpreted as a combination of three image flows merging and branching during the processing stage of a single frame. Here the third flow stands for the information calculated from the current frame, the second one for the intermediate results obtained from the previous frame, while the first one represents the binary masks generated from the previous result. The core of the three main processing stages of the algorithm can also be described by PDEs (left): 1) image filtering and reconstruction derived from nonlinear diffusion PDEs; 2) motion estimation derived from optical flow PDEs; and 3) trigger wave-type active contour-based boundary tracking derived from reaction-diffusion nonlinear PDEs. These PDE approximations, executed on the ACE4K chip, can be completed within a millisecond, allowing the processing system to reach its peak performance around four thousand frame/sec (right). D. Multilayer and Complex Cell CNN-UM The multilayer CNN structure was already introduced in [5]. It is used when several 2-D CNN layers are necessary Fig. 6. Fig. 3 shows the initial state and the output image (at T =2 elapsed time). There exists a very wide catalog of templates covering a myriad of applications. Also, because these templates are programmable by definition, learning can be incorporated to adapt the templates. Either globally, for example, using a genetic algorithm [16], or locally. Thus, not only associative memories can be constructed, e.g., [17], but the plasticity of the brain might be directly modeled [13]. to describe the spatial-temporal dynamics. In many cases, the layers are just cascaded, and the consecutive instructions of the CNN-UM are adequate to model the same process. However, in those cases where interlayer feedback does exist, we need the multilayer CNN structure. Such a multilayer CNN is useful for modeling the vertebrate retina [12]. Fig. 6 shows the conceptual architecture of a second-order dynamics, three-layer cell which has been prototyped in the ROSKA AND RODRÍGUEZ-VÁZQUEZ: TOWARDS VISUAL MICROPROCESSORS 1247 Authorized licensed use limited to: Universidad de Sevilla. Downloaded on April 15,2020 at 14:17:43 UTC from IEEE Xplore. Restrictions apply. Fig. 7. Using the CACE1K chip, programming the layer time constants and the A -templates on the two dynamic layers, a double wave propagation can be programmed. The resulting sequence of snaphots shows the different speed and the different types of waves on the two layers. chip called CACE1K [11]. The dynamic operation is given according to the following expressions: (4) where represents the built-in difference arithmetic. The operation of this prototype is hence controlled by the 23 parameters involved in (4), given as (5) plus the relative values of the time constants of Layers 1 and 2, totaling 25 different parameters. Many types of nonlinear waves (trigger-, traveling-, auto-, and spiral-waves) can be obtained by properly controlling these parameters [23]. E. Example 2 This example illustrates the generation of double-wave propagation using the CACE1K chip [11]. The template element values for this operation are (6) and the ratio between the time constants of the two layers is . Using the same chip, very recently we have been able to implement some of the key inner retinal effects, impossible to realize on first-order layers. More detailed results are reported elsewhere [24]. Our quest to make a programmable prototype spatial-tem- poral computer which could also serve as a visual microprocessor could be justified in two ways. On the one hand, we haveprovenearlierthattheCNN-UMis universal.Ina sense, it is equivalent to the Turing machine. The proof was realized by implementing the game of life. On the other hand, in each cell, with not more than four layers, we can implement any nonlinear multi-input single-output operator with fading memory. This is only one side of the story. On the other side, which is similar to the digital computers or Turing machines in which the -recursive functions are the formal descriptions of the algorithms with proven capabilities, we have also determined the equivalent formal notion of algorithms as the -recursive functions with similar properties [18]. Hence, we have all the theoretical background to establish our new type of computer for topographic operations, in particular for vision. Moreover, it has turned out that the neuromorphic constructs for most of the topographic senses with accompanying processing are quite similar to those of CNN models [9]. III. ANALOGIC VISUAL MICROPROCESSOR IN SILICON CNN-based analogic visual microprocessors have similarities with the so-called single instruction multiple data (SIMD) systems [25], although they work directly on analog signal representations obtained through embedded optical sensors and hence do need neither a front-end sensory plane nor analog-to-digital converters. The architecture of these visual microprocessors is illustrated in Fig. 8 through two prototype chips, namely, ACE4K [10] and ACE16K [26]. In both cases, as in other related chips [11], [27]–[29], the architecture includes a core array of interconnected elementary processing units, surrounded by a global circuitry. This latter circuitry is intended for: • control and timing; • adressing and buffering of the core cells; • input/output; • storage of user-selectable instructions (programs) to control the sequence of operations of the processing core; • storage of user-selectable analogic programming parameter configurations (templates). 1248 PROCEEDINGS OF THE IEEE, VOL. 90, NO. 7, JULY 2002 Authorized licensed use limited to: Universidad de Sevilla. Downloaded on April 15,2020 at 14:17:43 UTC from IEEE Xplore. Restrictions apply. Fig. 8. Architectures of analogic visual microprocessor chips: (a) ACE4K [10] and (b) ACE16K [26]. On the other hand, the core of interconnected processing units embeds different functions on a common silicon substrate (see Fig. 9 for illustration purposes), namely: • 2-D sensing; • 2-Danalog/digitalarrayprocessingconcurrentwiththe signal sensing; • 2-D spatio-temporal processing determined by local, receptive-field-like programmable interconnections; • 2-D memory banks for concurrent online uploading and downloading of short-term analog and digital data. Several analogic visual microprocessor chips in different CMOS technologies have been reported during the last few years. Particularly, [10], [11], and [26]–[29] report those implementations with at least 20 20 pixels. Table 1 presents a summary of some of their most relevant data. Some columns ROSKA AND RODRÍGUEZ-VÁZQUEZ: TOWARDS VISUAL MICROPROCESSORS 1249 Authorized licensed use limited to: Universidad de Sevilla. Downloaded on April 15,2020 at 14:17:43 UTC from IEEE Xplore. Restrictions apply. Fig. 9. llustrating the embedding of different functional features at the core processing array of visual microprocessors. (a) Microphotograph of the ACE4K chip (left) and conceptual representation of the distributed functions embedded in the core array (right). (b) Layout of a processing unit of the ACE16K showing the areas occupied by the different functions realized concurrently by the core array. correspond to chips intended for black and white input images,while othersare forchips whichaccept gray-scaleinput images. As with any other analog processing circuit, figures of merit about performance must contemplate accuracy and area occupation in addition to speed and power consumption. The speed measure here is proportional to the number of cells, the inverse of the time constant, and a weighted number of multipliers per cell. Any comparison must refer to the number of operations per second and to the accuracy. The data in the table highlights the following. • There is a tradeoff between area occupation (cell density)andaccuracy,ontheonehand,andspeedofoperationandpowerconsumption,onthe other. Thistradeoff is typical of analog integrated circuits [33]. • Theevolutiontowardscaled-downtechnologiesreports advantages in terms of speed and cell density. Actually, the ACE16K chip has 128 128 resolution and is capable of realizing sequences of 64 instructions; using up to 32 different templates (each template consisting of 24 8-bit-coded analog programming values) during a sequence; loading and downloading full-size gray-scale images to and from the cache memory, and havingalwayseightfull-sizeimagesavailableforusage during the flow; with an internal processing time of 160 ns, and providing digitally coded output images (obtained with a battery of internal converters) with a downloading time of 0.128 ms. The capability to design cells with maximum density, speed and accuracy, and minimum area and power consumption relies basically on the exploitation of all functional features offered by the MOS transistor. This is very different from digital design, in which only the switching capability of the MOS transistor is exploited. The design of the entities which interconnect the cells (synapses) defines one of the 1250 PROCEEDINGS OF THE IEEE, VOL. 90, NO. 7, JULY 2002 Authorized licensed use limited to: Universidad de Sevilla. Downloaded on April 15,2020 at 14:17:43 UTC from IEEE Xplore. Restrictions apply. Table 1 Summary and Comparison of Chip Implementations major issues. In order to do this, different possibilities may be chosen a priori, as illustrated in Fig. 10. In all cases, electrical controllability is provided by default. However, the different strategies exhibit quite a different performance in the presence of systematic and random error sources, as well as a different incidence of the global signal transmission errors. Hence, careful analysis and optimization is needed to select the best approach. Such analysis and optimization are needed to achieve the cell density and accuracy levels featured by last generation chips. The background for such procedures can be found in [3], [10], [11], [26], and [28]. IV. ABOUT SCALING DOWN It is expected that the performance figures featured for these chips can be further enhanced as technology scales ROSKA AND RODRÍGUEZ-VÁZQUEZ: TOWARDS VISUAL MICROPROCESSORS 1251 Authorized licensed use limited to: Universidad de Sevilla. Downloaded on April 15,2020 at 14:17:43 UTC from IEEE Xplore. Restrictions apply. Fig. 10. Using a single NMOST for voltage-to-current transformation. Only first-order terms are included in the displayed behavioral equations. down. However, one problem arises due to the necessity of maintaining analog accuracy, and hence the quality of the analog design, as transistor sizes decrease. Below we first identify mismatch as the main limit for the analog accuracy and then explore different tradeoffs associated with the analog design in the presence of mismatch. A. Mismatch Versus Noise as a Limiting Factor Mismatch makes two nominally identical devices behave differently when they are used in a real integrated circuit. Basedon theformulationofmismatchas afunctionofdevice geometries in [30], the variance of the large-signal transconductance parameter , the threshold voltage , and the slope factor4as function of the device area and aspect ratio can be represented as (7) where is the transistor channel area and is the transistor aspect ratio. Another accuracy limiting factor is noise. The equivalent noise current for an MOS transistor can be expressed as [31] (8) where and vary between 1 and 2, within the ohmic region and of this quantity in saturation, and is the small-signal transconductance parameter. 4In the original model, the variance was formulated for the body effect factor  1  ( n ) can be obtained as a function of  ( V ) and  (  ) . Let us consider that the only significant mismatch error is that of the large-signal transconductance parameter —as it actually happens in many practical circuits used for establishing interconnections in analog array processors [32], [33]. In terms of the transistor area and aspect , this error is expressed as (9) Under similar assumptions, the noise contribution can be approximated by (10) Using typical parameters for CMOS 0.5- m technologies (V, V, V, cm V s , m , V F) and considering a bandwidth of 1–5 MHz, we conclude that, for devices with channel areas of about 50 m , the matching level sets an accuracy slightly above 8 b while for this same area and a channel aspect ratio of 0.1 the noise poses a limit in the resolution of 10.48 bit, far beyond from that posed by mismatching phenomena. B. The Effect of the Scaling Process Let us assume that lateral dimensions scale as (11) Thus, the gate oxide thickness, which approximately evolves in current technologies as , scales as (12) 1252 PROCEEDINGS OF THE IEEE, VOL. 90, NO. 7, JULY 2002 Authorized licensed use limited to: Universidad de Sevilla. Downloaded on April 15,2020 at 14:17:43 UTC from IEEE Xplore. Restrictions apply.