An ART1 microchip and its use in multi-ART1 systems

1184 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 5, SEPTEMBER 1997 An ART1 Microchip and Its Use in Multi-ART1 Systems Teresa Serrano-Gotarredona and Bernab´ e Linares-Barranco Abstract—Recently, a real-time clustering microchip neural engine based on the ART1 architecture has been reported. Such chip is able to cluster 100-b patterns into up to 18 categories at a speed of 1.8  s per pattern. However, that chip rendered an extremely high silicon area consumption of 1 cm 2 , and consequently an extremely low yield of 6%. Redundant circuit techniques can be introduced to improve yield performance at the cost of further increasing chip size. In this paper we present an improved ART1 chip prototype based on a different approach to implement the most area consuming circuit elements of the first prototype: an array of several thousand current sources which have to match within a precision of around 1%. Such achievement was possible after a careful transistor mismatch characterization of the fabrication process (ES2-1.0  m CMOS). A new prototype chip has been fabricated which can cluster 50-b input patterns into up to ten categories. The chip has 15 times less area, shows a yield performance of 98%, and presents the same precision and speed than the previous prototype. Due to its higher robustness multichip systems are easily assembled. As a demonstration we show results of a two-chip ART1 system, and of an ARTMAP system made of two ART1 chips and an extra interfacing chip. Index Terms—Adaptive resonance theory, analog circuits, analog conputers, analog integrated circuits, analog processing circuits, analog systems, ART neural networks, circuits, clustering methods, CMOS integrated circuits, CMOS memory integrated circuits, integrated circuit design, large-scale integration, learning systems, neural-network hardware, nonlinear circuits, real-time systems. I. INTRODUCTION SINCE the invention of the ART1 architecture in 1987 [2] many high-level neural processing systems have been developed [3] which are based on the ART1 or more evolved but similar architectures [4]–[8]. These high-level neural systems have internal complex structures, but many times they are based on a small number of ART-like building blocks. When these high-level neural systems have to be used in real-world applications, portable equipments, robots, industrial control applications, etc., it is not always possible to rely on software programs running on expensive workstations. In such cases it is mandatory to build a piece of hardware that realizes physically the neural processing system. The availability of ART-like modular chips would significantly boost the proliferation of ART-based neural hardware systems. Due to the inherent internal hierarchy of ART-based neural systems their hardware realization would be significantly simplified if Manuscript received June 25, 1996; revised April 2, 1997. The authors are with National Microelectronics Center (CNM), 41012 Sevilla, Spain. Publisher Item Identifier S 1045-9227(97)05251-X. (a) (b) Fig. 1. (a) ART1 architecture diagram. (b) Algorithmic operation description of VLSI-friendly fast-learning ART1 system. robust and low-cost ART-like chip modules would be readily available. Although some preliminary work was done to build ART- based hardware prototypes [9], [10], it is not until recently that a fully functional reasonable size real-time clustering microchip neural engine based on the ART1 architecture has been reported [1]. It is based on a slightly modified version of the ART1 algorithm which was shown to preserve all its 1045–9227/97$10.00 1997 IEEE SERRANO-GOTARREDONA AND LINARES-BARRANCO: ART1 MICROCHIP AND ITS USE 1185 (a) (b) (c) Fig. 2. (a) Circuit diagram of current-mode ART1 chip. (b) Detail of synapse S ij . (c) Detail of controlled current source C i . original computational properties [11], but has a more VLSI- friendly algorithmic structure. The reported ART1 chip was able to cluster binary input patterns of up to 100 pixels into up to 18 different categories. The chip was able to classify an input pattern and learn its relevant characteristics by updating its internal knowledge, all in less than 1.8 s. The chip internal circuit architecture also allowed modular expansion of the clustering system. Assembling an array of these chips would result in ART1 systems able to cluster 100 pixel input patterns into up to 18 categories. Unfortunately, the resulting area consumption (and cost) of the chip was extremely high (1 cm ), and consequently its yield1 performance was extremely low (6%). Nevertheless, due to the fault-tolerant nature of the algorithm, most of the faulty chips still were able to perform satisfactorily [1]. A straightforward solution to the yield problem is to include extra redundant circuitry in the chip together with some selftesting subsystems that would identify and disconnect faulty subcells. This method is used intensively in large-area highdensity commercial DRAM chips. However, this redundancybased yield enhancement technique increases silicon area, requires more processing circuitry and increases design effort and cost [13]. In this paper a new ART1 chip is presented which solves the yield problem using a different approach: area 1Percentage of fault-free chips over total number of fabricated chips. reduction. After careful MOS transistor electrical parameter mismatch characterization of the technological process to be used, we were able to identify the maximum chip area for which the parameter variations would remain within the necessary limits to preserve the required system operation precision. We concluded that for the ES2-1.0 m CMOS process, for transistors of size m spread over a die area of the order of 2.5 2.5 mm, and for current levels around 10 A, the standard deviation of transistor current mismatch is of the order of %. Taking this into account we were able to design and fabricate an ART1 chip capable of clustering 50-b input patterns into up to ten categories, with a yield performance of 98%, and whose area is 15 times less than that of the first prototype. The chip showed a very robust behavior which enabled us to implement some multichip ART1 systems. As an illustration we will show results of a two-chip ART1 system and of a three-chip ARTMAP system. This paper is structured as follows. In the next section the VLSI-friendly ART1 algorithm employed is reviewed as well as the circuit design that maps it into hardware. In Section III we show why the first prototype has a very high area consumption, how we performed a careful technology current mismatch characterization, and how we modified the circuit to drastically reduce its area, while maintaining system precision and speed 1186 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 5, SEPTEMBER 1997 Fig. 3. Tree-like current-mirror structure for generating a large number of matched current sources. Fig. 4. Simplified diagram of mismatch characterization chip and experimental setup. performance. In Section IV we will provide measured experimental results of the new ART1 chip and of a two-chip ART1 system. Section V describes how to assemble an ARTMAP system and provides measured experimental results as well. Finally, we conclude in Section VI. II. VLSI-FRIENDLY ART1 ALGORITHM AND CIRCUIT IMPLEMENTATION An ART1 system is a self-organizing neural associative memory capable of generating in an unsupervised way stable recognition codes in response to a series of arbitrarily many, arbitrarily ordered, and arbitrarily complex binary input patterns. As shown in Fig. 1(a) the ART1 architecture consists of two layers. The bottom layer has nodes each of which receives the th binary pixel of the external input pattern . The top layer has nodes, each of which represents a learned category or cluster of input patterns . Each layer node connects to all layer nodes through binary weights which can be either “0” or “1.” Each layer category is characterized by the set of weights that connects to it. Every time an input pattern is presented to the input layer an internal search process starts which, when finished, results in activating a single layer category. This category is the one that best represents the input pattern according to the value of a vigilance parameter which can be tuned within the interval [0,1]. For small values many patterns will be clustered into the same category, while for high values only very similar patterns will be considered to belong to the same category. In the original ART1 paper by Carpenter and Grossberg [2] the operation of the system (a) (b) Fig. 5. Measured current for an array of MOS transistors with the same V GS and V DS voltages (for a nominal current of 10  A), spread over a die area of 2.5 2 2.5 mm. (a) Array of NMOS transistors and (b) array of PMOS transistors. was described by sets of nonlinear differential equations. It was also mentioned that the operation of the system could be described by an algorithmic flow diagram which basically describes the steady state of the differential equations. This algorithmic description was named as the fast-Learning mode of operation. Fig. 1(b) shows a modified version of the original fast-Learning ART1 operation which has a higher potential for VLSI circuit implementations. It has been shown that this algorithm preserves all the original computational properties of an ART1 system [11]. The operations to be performed are the following. 1) Reset all binary weights . 2) Read a binary input vector I. 3) Compute a set of analog “choice functions” or distances (1) or in vector notation2 (2) 2Given a vector a  ( a 1 ; 111 ;a N ) , the notation j a j represents its ` 1 norm j a j = N i =1 j a i j , and the intersection operator between two vectors represents the component-wise logical AND operation. SERRANO-GOTARREDONA AND LINARES-BARRANCO: ART1 MICROCHIP AND ITS USE 1187 Fig. 6. Measured currents of the L B array for the new ART1 chip prototype. 4) Select the maximum among all .If is this maximum then the th node is set to while all others are set to . Hence, layer acts as a winner-take-all (WTA). 5) Check the vigilance criterion: if the criterion is not satisfied. In such case, force and return to Step 4). Otherwise, the criterion is satisfied and the weights must be updated to incorporate the characteristics of pattern into category (3) or in vector notation (4) The way this algorithm can be implemented in a parallel analog current-mode processing circuit is depicted in Fig. 2(a). It consists of a 10 50 array of synapses ,a1 50 array of controlled current sources , two 1 10 arrays of unity-gain current mirrors CMA , CMB ,a1 10 array of current comparators CC , a ten-input WTA circuit, two unitygain current mirrors CMM and CMC, and an adjustable-gain (0 ) current mirror. Registers and the NOR gate are optional. The circuit diagram of a synapse is shown in Fig. 2(b). It contains three current sources, a latch, and a set of NMOS and PMOS transistors acting as switches. The state of the latch is set to “1” by activating the RESET signal prior to circuit operation, or is set to “0” during circuit operation if , and simultaneously. The synapse generates two currents, one of value which is drained from node in Fig. 2(a), and another of value drained from node . Nodes and are shared by all synapses in the same row. Consequently, the total input current to mirror CMA and injected to input of the WTA is (5) Note that current is provided by current mirror CMM to all nodes. Similarly, the total input current to each mirror CMB is (6) Fig. 2(c) shows the circuitry for each cell . This cell drains a current from node . The total input current for the -gain mirror is thus . This current, amplified by a factor , is replicated by mirror CMC and compared against each current at each CC current comparator. If (7) comparator CC deactivates the WTA input current ,by making the WTA control input . This way, current will not compete in the WTA. Consequently, only the currents that meet the vigilance criterion (7) will compete. The maximum among these currents, let us call it , will make while the rest become . Once a single winner is active the signal can be activated making (8) An uncommitted node is one that has not yet been selected as a winner. Such nodes have their initial weight values . Consequently, their corresponding row of synapses will generate the same current (9) 1188 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 5, SEPTEMBER 1997 Fig. 7. Training sequence for a one-chip ART1 system with  =0 : 3 and  =1 : 1 . The function of the shift register is to enable only one uncommitted cell to compete for the winner. Every time an uncommitted cell wins, the shift register content is shifted one position and the next uncommitted node is enabled for WTA competition. The NOR gate signals that all nodes are already committed. III. YIELD AND AREA OPTIMIZATION BY PROCESS MISMATCH CHARACTERIZATION From a system precision point of view it is important to make all and current sources to match within the required precision. When we designed our first ART1 prototype [1] we had no information concerning the long distance matching behavior of large arrays of current sources for the technology we were using. Therefore, we decided to use a mirror tree-like structure to generate all current sources from two external and current references. This approach is shown in Fig. 3. Each multiple-output current mirror had at the most ten outputs. Each current mirror was laid out using common centroid techniques, thus minimizing gradientinduced mismatch at the expense of increasing die area. If each current mirror of stage introduces a mismatch error characterized by a standard deviation , the total error of a Fig. 8. Training sequence for a two-chip ART1 system with  =0 : 5 and  =2 . -stages cascade is given by (10) The last stage is the most numerous and will occupy most of the area. Pure random mismatch is inversely proportional to transistor area and current level [12]. If we want to keep around 1%, each stage must have smaller errors. In our first ART1 prototype chip [1] most of the die area was spent by the th stage of common centroid low-mismatch multiple-outputs current mirrors. The resulting ART1 chip had a die area of 1cm while having a 100-node layer and an 18-node layer. The yield performance of a microchip has the following approximate dependence on die area : (11) where, for this technology the estimated average defect density is cm . For cm yield results to be around 6%.3Although most of the faulty chips rendered satisfactory clustering behavior [1] we decided to increase yield by reducing die area. In order to keep the system precision around 1% without using a large-area-consuming tree-like mirror structure a careful long distance mismatch characterization of the technological process to be used was necessary. A special purpose chip was designed in the ES2-1.0 m CMOS technology to estimate the matching behavior of large transistor arrays, for different transistor sizes. The chip contains a matrix of cells, each of which has several NMOS and PMOS transistors of several sizes, plus a transistor selection circuitry. Fig. 4 shows schematically the chip together with an experimental set-up to measure all transistors. In the chip all 3The chip pad ring area is not included in the yield computation. SERRANO-GOTARREDONA AND LINARES-BARRANCO: ART1 MICROCHIP AND ITS USE 1189 Fig. 9. ARTMAP architecture. NMOS and PMOS transistors have their sources connected to pin , all NMOS transistors have their drains connected to pin , all PMOS transistors have their drains connected to pin , all transistors have their gates short-circuited to their sources, except for one pair of NMOS and PMOS transistors. This pair has their gates connected to the external pin . A digital bus and internal decoding circuitry selects one pair among all. By connecting a curve tracing instrument (in our case, the HP4145) to pins and the selected NMOS transistor can be accessed and characterized, while by using pins and the selected PMOS transistor can be measured. This technique has been used to characterize the mismatch behavior of several technological processes [14]. For transistors of size 10 m 10 m spread over a chip area of 2.5 mm 2.5 mm, biased by the same gate-to-source and drain-to-source voltages so that their nominal current was around 10 A, we measured the current spreads depicted in Fig. 5. Fig. 5(a) shows, as a function of transistor position, the current measured for each transistor of an NMOS array. Fig. 5(b) shows the same for a PMOS array. As can be seen, the surfaces present a long distance gradient component and a short distance noise component. Let us call the measured currents surface . For this surface we can compute the best fit plane . Then, for each point we can define (12) By computing the standard deviation of , we are extracting the noise component of surface . The gradient component is defined by plane . The maximum deviation due to the gradient component is given by (13) On the other hand, for the noise component, 98% of the points remain within the interval. Consequently, let us define the maximum deviation due to the noise component TABLE I CURRENT MISMATCH COMPONENTS FOR TRANSISTOR ARRAYS WITH 10  A NOMINAL CURRENT, 10  m 2 10  mTRANSISTOR SIZE, AND 2.5 2 2.5 mm DIE AREA FOR THE ES2-1.0  m CMOS PROCESS as . Let us now define (14) as the ratio between noise component and gradient component contributions. Table I shows these ratios measured for NMOS and PMOS transistors of size 10 m 10 m, driving nominal currents of A and for different chips. Also shown in Table I are the standard deviations of the noise component , the maximum deviation of the gradient component , and the total standard deviation of transistor currents , computed as (15) The current mirror tree-like structure of Fig. 3 was intended to suppress the gradient component of a 1 cm chip. The noise component can only be reduced by increasing transistor area [12]. Table I reveals that for die areas of 2.5 mm 2.5 mm, transistor sizes of 10 m 10 m, and nominal currents of 10 A, the contribution of noise component is equal or higher than the gradient component, while the standard deviation of current mismatch is kept below 1%. Consequently, for these dimensions we can avoid the use of high area consuming 1190 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 5, SEPTEMBER 1997 Fig. 10. Flow diagram of ARTMAP training mode operation. circuit structures (like common centroid mirrors arranged in a tree-like fashion) to eliminate the gradient component, and directly implement a single current mirror with all the outputs needed. This is the approach we used in the present ART1 chip prototype. This chip has a die area of 2.5 mm 2.2 mm, and contains an array of 50 10 synapses, each synapse with two and one current sources. The current sources transistors are of size 10 m 10 m and drive a nominal current of 10 A. Fig. 6 shows the measured currents of the array. Table II shows the measured values of the mismatch components of the and current sources arrays for all fabricated chips. Note that the total current mismatch standard deviation is less than 1% for all chips. Due to the much smaller chip area its fabrication cost is much less and its yield performance is significantly higher: 98% by applying (11). In the next sections single chip operation experimental results are described as well as results for systems assembled with several ART1 chips. IV. EXPERIMENTAL RESULTS OF ART1 SYSTEMS All ten fabricated chip samples were fully operational and for none of them we were able to detect any fault in its subcircuits. All system components could be isolated and independently characterized. The circuit performances of the different subcircuits were similar to those of the first prototype [1], and consequently their characteristics will not be repeated in this paper. Here we will only provide some illustrative examples on system level behavior. Although the chip is analog in nature, its inputs and outputs are digital. Therefore, it is possible to test its system level behavior using a digital test equipment (in our case, the HP82000). This equipment applies digital input vectors ( ), SERRANO-GOTARREDONA AND LINARES-BARRANCO: ART1 MICROCHIP AND ITS USE 1191 TABLE II MEASURED MISMATCH COMPONENTS FOR THE FABRICATED ART1 CHIP PROTOTYPES reads digital output vectors ( ), and reads the internal weights () at each processing step. Three external reference currents need to be supplied to the chip: and . Current [see Fig. 2(a)] is needed to assure that all currents reaching the WTA are positive. The ART1 system behavior is controlled by two externally adjustable parameters, and . is the gain of a current mirror and is adjusted through a digital word applied externally [1], while is controlled by appropriately setting currents and . To test the system behavior it was trained with a set of ten -b input patterns. Each pattern represents each of the ten digits from “0” to “9.” The last input pixel was always set to zero and it is not shown in the figures. The classification of the set of input patterns was repeated for different values of the vigilance parameter and several values of parameter . Fig. 7 shows the training sequence for and . The first column represents the input pattern applied to the system. The remaining ten columns correspond to the weights stored in each category when the input pattern has been classified and learned. The boxed category is the winning category after the WTA competition. In this case, learning self-stabilizes after two input pattern presentations. That is, no modification of the winning category or the stored weights take place in subsequent presentations of the input pattern sequence. As shown in Fig. 7, the system has clustered all ten input patterns into four categories. A two-chip ART1 system was assembled. In this case, the input patterns had binary pixels. Fig. 8 depicts a training sequence performed on this system. The system classifies the ten input patterns into eight categories after a single presentation of the input pattern set. The sequence of Fig. 8 was obtained for a vigilance parameter of , and A, A). V. ASSEMBLING AN ARTMAP SYSTEM USING ART1 CHIP MODULES An ARTMAP system [7] consists of two ART1 subsystems connected through an Inter-ART module, as depicted in Fig. 9. Let be an -dimensional input vector to the first ART1 subsystem ART1 , and an -dimensional one for the second ART1 subsystem. An Fig. 11. Flow diagram of the prediction ARTMAP operation. ARTMAP system is a supervised learning neural network that learns the correspondence between two simultaneous input patterns and . Two modes of operation can be distinguished: •Training Mode, during which pairs of input patterns () are provided, and the ARTMAP system learns their correspondence. •Prediction Mode, during which only patterns are provided to the first ART1 subsystem, and ARTMAP predicts the corresponding ART1 cluster. Fig. 10 illustrates the algorithmic description of ARTMAP operation in training mode [7]. After reading two input vectors and each ART1 module selects an winning node (for ART1 and for ART1 ) that meet their vigilance criteria. The inter-ART module, which is simply an array of binary weights initially set to “1,” learns the correspondence between the ART1 winning category and the ART1 one by making if otherwise (16) or, in vector notation (17) However, if ART1 category and ART1 category become simultaneously active and the Inter-ART weight has already been set to “0,” this means that ART1 category has already been assigned to a different ART1 category. In this case ART1 vigilance parameter is increased until deactivates and a different ART1 category is selected. During the prediction mode of operation subsystem ART1 does not receive any inputs. Only subsystem ART1 receives external input patterns and selects a winning category .ART1 outputs, which are the outputs of the complete 1192 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 5, SEPTEMBER 1997 (a) (b) (c) Fig. 12. (a) ARTMAP hardware assembly. (b) Diagram of Inter-ART chip. (c) Detail of Inter-ART chip cell. ARTMAP system, are activated by the Inter-ART module (18) or equivalently in matrix notation (19) where Wis the weight matrix of the Inter-ART module. According to the way the Inter-ART weights are set, for each ART1 active category only one ART1 category will be chosen, but an ART1 category can be activated by more than one ART1 cluster.4Fig. 11 shows the algorithmic flow diagram of the ARTMAP prediction mode operation. An ARTMAP hardware system can be assembled using two ART1 chips and an extra chip for the Inter-ART module, as is shown in Fig. 12(a). The Inter-ART chip, shown in Fig. 12(b), is simply an array of cells whose simplified schematic is depicted in Fig. 12(c). Each cell has a latch which is set initially to “1” and changes to “0” if , and the LEARN signal is high. Extra transistors, 4This is true unless the input pattern a activates an uncommitted ART1 a F 2 node (pattern ais not recognized as belonging to any ART1 a category). In this case, w jk =1 8 k , and all ART1 b F 2 nodes would be activated, implying that the applied input pattern is not recognized as belonging to any of the learned categories. not shown in Fig. 12(c), are also included to read out the weight values. During training mode the value of weight is used to control a digital counter that increments the value of .If the counter will increase its value until the ART1 winning category changes and becomes “1.” At this moment the counter stops and its content represents the appropriate value for . The system level operation of the ARTMAP hardware system has also been tested using the HP82000 digital test equipment. Fig. 13 shows a system training sequence. The first column, named , represents the input patterns applied to the ART1 chip. The column named represents the input patterns applied to the ART1 chip. The columns named and represent the stored weights in the ART1 and ART1 modules after the classification and learning of each input pattern pair. The boxed categories are the ones that remain active after the search process has finished, and these are the only ones that are updated with learning. Below each ART1 winning category the final value of the vigilance parameter needed in the search process to choose this category is indicated ( was increased in steps of ). The last column shows the stored weights in the inter-ART module which represent the learned correspondence between the ART1 and ART1 categories (index is coded vertically from top to bottom, while index is coded horizontally from