Gaussian Pyramid: Comparative Analysis of Hardware Architectures

1 Gaussian Pyramid: Comparative Analysis of Hardware Architectures Fernanda D. V. R. Oliveira, Jos´ e Gabriel R. C. Gomes, Jorge Fern´ andez-Berni, Ricardo Carmona-Gal´ an, Roc´ ıo del R´ ıo, ´ Angel Rodr´ ıguez-V´ azquez Universidade Federal do Rio de Janeiro – COPPE – PEE – Rio de Janeiro, RJ 21941-972 Instituto de Microelectr´ onica de Sevilla E-mails: [email protected], [email protected], b[email protected], [email protected], [email protected], [email protected] Abstract—The paper addresses a comparison of architectures for hardware implementation of Gaussian image pyramids. Main differences between architectural choices are in the sensor frontend. One side is for architectures consisting of a conventional sensor that delivers digital images and which is followed by digital processors. The other side is for architectures employing a non-conventional sensor with per-pixel embedded pre-processing structures for Gaussian spatial filtering. This later choice belongs to the general category of “artificial retina” sensors which have been for long claimed as potentially advantageous for enhancing throughput and reducing energy consumption of vision systems. These advantages are very important in the internet of things context, where imaging systems are constantly exchanging information. The paper attempts to quantify these potential advantages within a design space in which the degrees of freedom are the number and type of ADCs (single-slope, SAR, cyclic, Σ∆ and pipeline), and the number of digital processors. Results show that speed and energy advantages of pre-processing sensors are not granted by default and are only realized through proper architectural design. The methodology presented for the comparison between focal-plane and digital approaches is a useful tool for imager design, allowing for the assessment of focal-plane processing advantages. I. INTRODUCTION Images and Vision, i.e. the extraction of meaningful spatialtemporal information from the visual stimulus, are crucial for the interaction of “things” with the environment. Indeed, these days image and vision sensors are flooding all application territories, their usage and markets are increasing at exponential pace and they are expected to play important roles within Internet of Things domains [1], [2]. One major obstacle faced by image and vision sensor architects is the huge amount of data required for image coding. These data stress intermediate storage resources and communication channels. Also, their handling requires large energy budget particularly if on-line reaction is targeted. Different paths are being explored in the quest of overcoming these difficulties, covering from innovative sensor front-ends to enhanced multi-core back-end processors. Dynamic vision sensors [3] and computational image sensors [4]-[6] are relevant examples of advances regarding front-ends. In both cases sensors are meant to extract information, instead of just data, from the scene. Thus, reduced sets of abstract data, as opposed to raw data, are downloaded from the sensor for processing, hence de-stressing the system. This paper deals with information-centric computational image sensors. Particularly a 6-T active pixel is proposed to expedite the calculation of the Gaussian Pyramid (GP) [7]. This is relevant because image pyramids, and in particular the GP, constitute the first stage of many computer vision processing pipelines [8]-[11]. Also, their computation mobilises significant computational resources and results into large delay and energy consumption. However, the functional primitive underlying GP calculation is rather simple — just diffusions across the scene plane are needed. Diffusions can be implemented by embedding simple mixed-signal circuits at pixel level, as done for instance in [12], [13]. Actually, results in [13] demonstrate orders of magnitude of improvement in throughput and energy consumption when compared to architectures using conventional, data-centric, sensors. But these advantages have the counterpart of much larger pixel pitch. The sensor hereby described aims at overcoming this drawback by using only two extra transistors per pixel; i.e. by employing a 6-T APS instead of the standard 4-T APS used in conventional image sensors [14]. The main asset of the 6-T GP pixel proposed in this paper comes from the parallel implementation of diffusions. However, data interchange requirements are still significant. It means that potential advantages of the non-conventional architecture are not granted by default. Exploring the conditions under which these advantages really occur, and benchmarking them, is the main purpose of this paper. In other words, we perform comparative throughput and energy analyses of a non-conventional architecture based on a 6-T GP pixel, on the one hand, and a conventional architecture, on the other hand. In this later architecture all processing for the GP takes place in the digital domain — no pre-processing at all is performed in the sensor. We consider different kinds of data readout and various architectural choices regarding the number of analog-to-digital converters (ADCs) embedded in the sensor readout channel, the type of ADC, and the number of processors in the digital back-end. Results presented in this paper show that potential advantages of the non-conventional architecture are largely dependent on the choice of these highlevel, architectural degrees of freedom. The paper is organized as follows: Sec. II defines the concept of Gaussian Pyramid and sets the context of its hardware realization; Sec. III addresses a pixel implementation where 2 only 6 transistors are required to perform GP processing; time and energy numerical comparisons are presented in Secs. IV and V, respectively; a case study is described in Sec. VI in order to validate the proposed implementation; finally some concluding remarks are presented in Sec. VII. II. GAUSSIAN PYRAMID IN COMPUTER VISION Object detection is the starting point for most computer vision pipelines. Once a particular object of interest is detected, it can be segmented, tracked, recognized etc. A major challenge for the implementation of this early vision task is that the scale of targeted objects is not known a priori. Objects can enter the surveyed scene at different distances from the image sensor. Those appearing at distant locations will require higher resolution to be detected than close-up objects for which most of the pixels will contain redundant information. The concept of pyramid representation [7] thus arises as a multi-resolution scene representation where each frame making up an image flow is progressively filtered and subsampled in order to efficiently deal with the search of objects at different scales. An example of pyramid is shown in Fig. 1(a). The images with no subsampling are depicted in Fig. 1(b) for better visualization of the applied filtering. Formally, filtering followed by subsampling is defined by the reduce operation given by: fl(i, j) = X mX n K(m, n)·fl−1(2i+m, 2j+n),(1) where Kis the filtering kernel and flis the image of the pyramid at level l[7]. The canonical way to construct a pyramid representation is based on Gaussian filtering [10]. This filter ensures that no artifacts are generated when going from finer to coarser scales. Indeed, Gaussian Image Pyramid is one of the predefined vision functions included in the industrial standard OpenVX [15]. We make use of this standard definition in our analysis. Concerning hardware realization, a conventional approach to generate a GP is that of Fig. 2(a). The image sensed by an M×N pixel array is converted into digital and stored in memory. A prescribed number of Processing Elements (PEs) then access memory in order to process the image just captured and generate the corresponding pyramid. PEs can operate in parallel. This approach will constitute our reference realization for comparison. Due to the significance of the GP as a fundamental processing primitive in computer vision, numerous non-conventional approaches aiming at boosting its hardware performance have also been reported [12], [13], [16]-[20]. Among them, mixedsignal focal-plane sensing-processing [12], [13], [16], [17] stands out as the best approach in terms of parallelization and energy efficiency. Additional circuitry is incorporated per pixel, usually connected to its counterpart at neighboring pixels, in order to concurrently process the image sensed by photo-sensitive circuit elements. Unfortunately, this approach typically suffers from large pixel pitch, thereby having a negative impact on key parameters of image sensing like sensitivity, resolution, noise etc. Fig. 1: (a) Example of pyramid representation; (b) Pyramid in (a) with no subsampling in order to highlight the effect of the applied filtering. M×N A/D Memory Output (Image pyramid starting with images of resolution M×N) PE PE PE Digital processor (a) 2M ×2N A/D Output (Image pyramid starting with images of resolution M×N) (b) Fig. 2: Block diagram of two approaches for hardware realization of the Gaussian Pyramid: (a) conventional digital approach, where PE stands for Processing Element. PEs can operate in parallel; (b) focal-plane sensing-processing approach. III. PROPOSED PIXEL IMPLEMENTATION In order to address this drawback of focal-plane sensingprocessing realizations, we thoroughly analyze a focal-plane realization of Gaussian filtering requiring only two extra transistors per pixel. A basic block diagram of this realization is depicted in Fig. 2(b). The proposed circuit implementation is shown in Fig. 3. An n-channel transistor can be used as a switch, resulting in a pixel with six transistors [21]. Pixel operation starts by resetting the floating diffusion nodes. After the integration time, the charge accumulated at the photodiode cathode is transferred (according to TX) to the floating 3 4T pixel Proposed 6T pixel s1 s2 TX Reset Rs M1 M3 M4 M2 Fig. 3: 2×2 section of a matrix with the proposed pixel for focal-plane Gaussian filtering computation. Two transistors acting as switches (s1and s2) are included inside each pixel to connect the floating diffusion nodes of neighboring pixels. diffusion node. When the switches close, charge redistribution is performed among parasitic capacitors at the corresponding floating diffusion nodes. The average voltage after charge redistribution represents the mean luminance in the sub-matrix where the pixels were connected. This operation is lossy. Once pixels are interconnected in a sub-matrix, all parasitic capacitors end up holding the same voltage level. This loss of the original information does not prevent the GP generation. As an example, consider the 8×8 matrix in Fig. 4(a), where the pixel values encode an original image. This first step consists in connecting the pixels into 2×2 blocks, to perform an average operation inside each block, as shown in Fig. 4(b). If we sample one pixel inside each block, then the resulting image has half the number of rows and half the number of columns of the original image. The first step is necessary to perform convolution in the proposed way, but it reduces the resolution of the image. Sub-sampled image pixel positions are written in pi,j format in the middle of each block in Fig. 4(b). All subsequent steps perform Gaussian filtering on this sub-sampled matrix. In the first subsequent step, we change the grid and, again, group the pixels into 2×2 blocks. This grid change and the result of the new charge redistribution step is shown in Fig. 4(c). After the charge redistribution we have that p′ i−1,j−1= (pi−1,j−1+pi−1,j +pi,j−1+pi,j)/4, p′ i−1,j = (pi−1,j +pi−1,j+1 +pi,j +pi,j+1)/4,p′ i,j−1= (pi,j−1+pi,j +pi+1,j−1+pi+1,j)/4and p′ i,j = (pi,j +pi,j+1 + pi+1,j +pi+1,j+1)/4, where p′represents the pixel values after the second charge redistribution. The result is equivalent to filtering the sub-sampled image from Fig. 4(b) with the 2×2 binomial filter: K= [1 1; 1 1]/4. If we change the grid again, back to the first grid, as shown in Fig. 4(d), we perform the same filtering for a second 2 1 20 100 00012 7 6 13 15 14 34 6 7 9 3 2040 23 20 13 44 5 151 1 80 50 31 14 14104 11160 4 301210 18 8 13 7 0 17 107 71 11 12 10 60 19 1 07 10 9 221 24 4 4 4 4 4 4 4 4 12 12 12 12 12 12 12 12 12 12 12 12 4 4 4 4 44 44 44 44 12 12 12 12 4 4 4 4 36 36 36 36 36 36 36 36 28 28 28 28 36 36 36 36 28 28 28 28 12 12 12 12 20 20 20 20 pi;j pi;j−1pi;j+1 pi;j+2 pi−1;j pi−1;j−1pi−1;j+1 pi−1;j+2 pi+1;j pi+1;j−1pi+1;j+1 pi+1;j+2 pi+2;j pi+2;j−1pi+2;j+1 pi+2;j+2 (a) (b) 4 12 12 4 36 20 8 8 8 8 4 4 16 16 16 16 16 16 16 16 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 16 16 16 16 16 16 24 24 24 24 24 24 24 24 24 24 24 24 28 28 28 28 32 32 32 32 8 8 12 12 12 12 14 14 14 14 14 14 14 14 8 8 8 8 13 13 13 13 21 21 21 21 21 21 21 21 20 20 20 20 20 20 20 20 24 24 24 24 22 22 22 22 23 23 23 23 19 19 19 19 18 18 18 18 25 25 25 25 30 30 30 30 (c) (d) Fig. 4: Gaussian filtering example. time. Changing the grid back to the first one corresponds to filtering the sub-sampled image from Fig. 4(c) with the 2×2 binomial, or equivalently, to filtering the sub-sampled image from Fig. 4(b) twice with the 2×2 binomial, or once with the 3×3 kernel: K= [1 2 1; 2 4 2; 1 2 1]/16. According to the example in Fig. 4 we conclude that, for every grid change, the sub-sampled image is filtered with the 2×2 binomial kernel. The size of the targeted kernel determines the number of times that the grid must be shifted and charge redistribution enabled. The possible kernels that can be implemented with the proposed hardware are 2×2 binomial kernel cascade associations. Figure 5 presents the steps required for the generation of a three-level pyramid according to the definition of GP of the standard OpenVX. Step (2) from Fig. 5, is required for changing the image resolution. To generate Level 0, which is the GP starting level, we sample one pixel inside each 2×2 block of the image generated after this charge redistribution. This image is then filtered through steps (4) to (7), resulting in the image that is subsampled to generate Level 1. To compute Level 2, we connect the pixels into 4×4 blocks, with the same goal of step (2), thus reducing the resolution. As in the calculation of Level 1, four charge redistribution operations are performed to filter the image, which is done in steps (10) to (13). By the end of these operations the result is subsampled, generating Level 2. To create a pyramid with four levels, the pixels are connected into 8×8 pixel blocks. The maximum number of levels that can be generated by the proposed hardware mainly depends on the fabrication technology leakage current and the floating diffusion node capacitance. 4 2N 2M (1) Capture the image (2) Connnect the pixels into 2 ×2 blocks Grid 1 Grid 2 Grid 1 Grid 1 (4)Change the grid Grid 2 (5)Change the grid (7)Change the grid (9) Connnect the pixels into 4×4 blocks (11)Change the grid Grid 4 Grid 3 Grid 4Grid 3 (12)Change the grid (13)Change the grid (3) Sample M×N pixels to generate the Level 0 of the image pyramid (8) Sample M/2×N/2 pixels to generate the Level 1 of the image pyramid Grid 3 (14) Sample M/4×N/4 pixels to generate the Level 2 of the image pyramid (6)Change the grid (10)Change the grid Fig. 5: Example of Gaussian Pyramid generation at the focal plane. IV. TIME ANALYSIS COMPARISON The main goal of this paper is to compare our reference digital implementation — depicted in Fig. 2(a) — to the focalplane approach just described — sketched in Fig. 2(b). Note that for the focal-plane realization the resolution of the Level- 0 filtered image (M×N) is a quarter of the resolution of the captured image (2M×2N). In the digital processor, the convolution is based on sliding a binomial kernel across the image. At every location, the image pixels inside the kernel window are multiplied by the kernel elements, and the multiplication results are summed. For efficiency, the digital processor has a multiply and accumulate (MAC) unit, formed by one or more PE. The binomial kernel only requires addition and division by four, so the MAC unit is realized by simple digital circuitry (logic adders and shift registers) placed outside the pixel array. Filtering with a 2×2 kernel requires four pixel values for each kernel window, but two of these values are kept from the previous window operation, requiring only two memory-read accesses per window. Likewise, one MAC operation per window can be spared if we consider a partial result from the previous window. After each window computation the memory is accessed for writing the result. For a numerical comparison, the flows of both architectures are broken into tasks, which are analyzed considering processing time and energy consumption. In the time analysis, each task is related to a variable τthat represents the time to perform a given task once. We then compute the number of times the task is executed. Overall time is equal to τmultiplied by the number of executions of that task. After finding the processing time expressions for both approaches as functions of τ, each τis associated with the clock period, τClk, which leads to expressions with a single global variable. The time needed for image capture is approximately the same for both approaches, so it is not considered in the time comparison. The same idea applies to the data output transmission. A. Focal-Plane Approach Time Analysis The focal-plane approach steps are inferred from Figs. 5 and 2(a). Aside from capture and transmission, there are two main steps: 1) Gaussian Pyramid generation: the time it takes to generate the GP depends on the number of charge redistribution operations multiplied by the time it takes for a single charge redistribution. Image size does not affect the GP generation time, because this operation runs concurrently across the matrix. Kernel size determines the number of charge redistributions per level. We need nk−1charge redistribution operations to implement an nk×nkkernel. From Fig. 5 we see that this operation is repeated at every level, except the last one. Finally, we sum the charge redistribution operations that take place when the pyramid level changes. The overall number of charge redistribution operations is NCR = (NLev −1) ·(nk−1) + (NLev −1) = nk(NLev −1), where NLev is the number of pyramid levels. Multiplying NCR by the time required for performing one charge redistribution, τCR, we have the overall processing time τF PP roc =nk(NLev −1) ·τCR. 5 2) Analog-to-digital conversion: after each computation at the focal plane, pixel values are read out and sent to an analog-to-digital conversion stage, which comprises one or more ADCs. The time required for performing one sample conversion by one ADC is τADC . Overall data conversion time depends on the number of ADCs, NADC, and on the amount of data converted, Nconv. To compute Nconv, we note that for every pyramid level the image size is reduced by a factor of 4: Nconv = MN +MN/4 + ...+MN/22(NLev −1). Overall conversion time is thus: τADCT otal = NLev X n=1 M·N 22(n−1) · τADC NADC .(2) Overall focal-plane processing time is obtained by adding up τF PP roc and τADCT otal : τF PT otal =nk(NLev −1) τCR + NLev X n=1 M·N 22(n−1) · τADC NADC (3) B. Digital Implementation Time Analysis The digital approach requires more steps than the focalplane approach, as it can be seen in Fig. 2: 1) Analog-to-digital conversion: the captured image is immediately converted to digital. This is the only data conversion required by this approach. The size of the converted data is equal to the pixel array size. Thus, τADCT otal =M·N·τADC /NADC . 2) Memory storage: the resulting M×N digital values are stored into an internal memory. Time taken by this step is M·N·τMem, where τMem is the time required for accessing a single memory position. To consider simultaneous memory access, we introduce a new variable, NbusMem, that represents the number of possible parallel accesses. The total time required by this step is τmatrixMemW rite =M·N·τMem/NbusMem. 3) Gaussian Pyramid generation: the digital processor reads input values for the current pyramid level from a memory, performs multiply and accumulate operations and writes the result back into the memory. The number of times this operation is performed depends on image size and on the number of times the image is filtered by the binomial kernel inside each level. Image size changes at every level according to a series similar to the one given for the number of conversions, Nconv, except for the fact that we do not perform convolutions at the highest level. The number of operations is equal to: Nop = (nk−1) · NLev−1 X n=1 M·N 22(n−1) .(4) At least two pixel values are necessary in every computation of the 2×2 binomial kernel convolution, so we define τmemRead = 2τMem. At least three multiply and accumulate operations are used in the 2×2 kernel. The time required for performing these operations by one MAC unit is τconvolutionW indow = 3τop, where τop is the time required by a single MAC operation. The resulting value is written in the memory through a single access, and so τmemW rite =τMem. The time needed by a single PE to perform the convolution is obtained by multiplying the number of operations by the sum (τmemRead +τconvolutionW indow + τmemW rite). Assuming that more than one PE is available, and that NbusMem simultaneous memory accesses are allowed, parallel convolution operations are carried out. The overall time required for performing the convolution operations is, then, τconvolution =Nop(2τMem/NbusMem + 3τop/NP E + τMem/NbusMem). If NP E > NbusMem, then memory access collisions occur. To simplify the analysis, we ignore this issue by assuming that every PE may access the memory at any moment, with no additional hardware complexity. Then, in the τconvolution equation, NbusMem is substituted by NP E:τconvolution = Nop(2τMem + 3τop +τMem)/NP E . By adding τADCT otal ,τMatrixMemW rite, and τconvolution, we have the digital approach overall time: τdigitalT otal =M·N· τADC NADC +M·N· τMem NbusMem + (nk−1) · NLev−1 X n=1 M·N 22(n−1) 2τMem + 3τop +τMem NP E (5) C. ADC Architectures Comparison Before using the above equations to compare focal-plane and digital approaches, it is important to remember that it is common to work with the ADC at a clock period different from the one used for the other parts of the circuit. In our case, we define τClk as the period of the clock signal that controls the pixel array, memory, and digital circuitry. The ADC clock period, on the other hand, is KADC ·τClk, where KADC depends on ADC type. We consider five ADCs commonly used in CMOS image sensors: ramp, successive approximation register (SAR), sigma-delta (Σ∆), cyclic and pipeline [22]. To compare ADC types and find the appropriate clock period in each case, we use reported imagers in which the performance figures of the embedded ADCs are included [23]-[58]. ADCs have already been compared by different authors [22], [59]. The present comparison focuses exclusively on ADCs designed for image sensors, in the context of comparative time and energy analysis, including recently published works. The ramp converter, a linear approximation converter with simple architecture requiring low area and low power consumption [60], is probably the most used converter in image sensor applications [36]-[46]. It is suitable for working with high clock frequencies. We thus use it as a reference for other converter types: the ramp ADC clock period is equal to the global clock, τClkRamp =KRamp ·τClk =τClk, so 6 KRamp = 1. The data converters in the comparison were designed for different resolutions. For a fair comparison, we normalize the conversion rates and energies for the same number of bits, which is set as Nbits = 8. Although imagers with higher number of bits are common, eight bits per pixel is more typical [61]. The conversion rate normalization depends on the number of clock cycles per bit each converter architecture requires. A single slope ramp converter, for example, requires 2Nbits ·τClkRamp (maximum) for a conversion. The normalized conversion rate considering eight bits is f′ s= 2Nbits ·fs/28, where fsand Nbits are the reported conversion rate and resolution. For the SAR and cyclic converters, the conversion time is Nbits ·τClkSAR,Cyclic , so the normalization is f′ s=Nbits ·fs/8. The Σ∆ conversion time depends on the oversampling rate (OSR). For second-order incremental Σ∆ converters, the number of bits is Nbits = log2[OSR ·(OSR + 1)] −1, where OSR is the reported oversampling rate. We consider an oversampling rate equal to 25, which yields resolution equal to 8.3 bits. The normalization is f′ s=OSR ·fs/25. The pipeline converter conversion time is one τClkP ipeline , with some latency, which does not depend on the number of bits, i.e. normalization is not required. Pipeline converters are not as common in image sensors as the other converter types (simulation results have been reported, as well as experimental results from ADC chips working together with imaging chips), but they are included in the comparison because of their improved speed. To normalize energy figures, we assume that the power consumption doubles for every bit added [59]: E= 28· P/(f′ s·2Nbits ). Walden’s figure of merit for ADCs [62] uses the effective number of bits (ENOB) instead of the resolution. The normalized energy values in Fig. 6 are based on the resolution because some of the references do not report ENOB. Figure 6 shows the normalized energy versus normalized conversion rate for the five ADC types considered. The median conversion rate and energy (black markers in the figure) are chosen as representative values for each converter type. The median values suggest that, for eightbit resolution, cyclic and SAR converters are approximately two times faster than ramp converters. The conversion times are related according to τADCRamp = 2 ·τADCSAR,Cyclic and τADCRamp = 28·τClkRamp ,τADCSAR,Cyclic = 8 · τClkSAR,Cyclic . So, the cyclic or SAR converters run at a clock which is approximately 16 times slower than the ramp converter clock. For the focal-plane and digital approaches comparison, we thus assume KSAR,Cyclic = 16, where KSAR,Cyclic is the constant that multiplies the global clock period τClk to yield τClkSAR,Cyclic . The Σ∆ conversion time is 1.3 times smaller than the ramp ADC conversion time, so τClkΣ∆ = 28·τClkRamp /(1.3·25) ≈8τClkRamp . The multiplying constant is KΣ∆ = 8. For the pipeline converter, τClkP ipeline = [28/(τADCRamp /τADCP ipeline )]τClkRamp and τADCRamp /τADCP ipeline ≈130,KP ipeline = 2. Summarizing, we defined Kramp = 1, since this converter is used as reference, and, using reported figures, found KSAR,Cyclic = 16,KΣ∆ = 8 and KP ipeline = 2. These constants define the ratio between the ADC clock period and the clock period τClk, used for the other stages of the circuit. Conversion rate (kSa/s) 101102103104105 Energy per sample (pJ/Sa) 10−3 10−2 10−1 100 101 102 103 Cyclic Pipeline SAR Σ∆ Ramp Fig. 6: Eight-bit normalized conversion rate versus energy per sample of five types of ADC. The median values for each type of ADC are plotted using black unfilled markers of the corresponding shape. D. Time Comparison Results We now establish some default values for the parameters in Eqs. (3) and (5), and associate the overall times to a global clock period. As explained in Sec. IV-C, τClk is the period of the clock signal that controls the pixel array, memory, and digital circuitry and KADC ·τClk is the ADC clock period. Assuming that charge redistribution is practically instantaneous, it is clear from Eq. (3) that the bottleneck of the focal-plane approach is at the ADC, because of the amount of data to be converted. The digital approach bottleneck, on the other hand, is either at the ADC or at the processing stage, which depends on ADC type. For both approaches, we explore different ADC types and NADC values. For the digital approach, we explore several NP E. We thus do not define default values for τADC ,NADC , and NP E . The maximum NADC value is set to the number of columns at pyramid Level 0, since image sensors with one ADC per column are commonly found [63]. Although stacking technologies allow for the integration of one ADC per pixel [23], this is still an upcoming technology with high fabrication costs. We use VGA (video graphics array, 640×480 pixels) standard for the pyramid Level 0 image size. Consequently, the pixel array size in the focal-plane approach is 1280×960. The time analysis does not change significantly if the resolution increases, but the bandwidth for the transmission of the generated data increases. Increasing the resolution and using one ADC per column also increases power consumption. The pyramid size can not be too large, because computation accuracy is limited by leakage currents. The operations can be performed as long as the capacitance voltages are not affected by these currents. We set NLev = 4. To achieve a 7 TABLE I: Time analysis equations parameters. Parameter Value Pyramid Level 0 size (M×N) 640×480 Maximum number of ADCs (NADCMax ) 640 Equivalent kernel size (nk) 5 Number of bits (Nbits) 8 Number of levels (NLev) 4 Number of memory accesses (NbusMem) 4 Time to perform charge redistribution (τCR)1τClk Time to access the memory (τMem)2τClk Time to perform a MAC operation (τop)2τClk reasonable compromise between the circuit complexity and speed, we set NbusMem = 4. Choosing NbusMem = 1 would impair digital circuit performance, but increasing the number of simultaneous memory accesses increases digital circuit size and complexity. Charge redistribution, memory access and MAC operation times (τCR,τmem and τop) are written as functions of the clock period τClk. Charge redistribution itself is practically instantaneous, but the time it takes to drive the charge redistribution switches is considered, so τCR = 1τClk. The time to access the memory, τmem, was defined as 2τClk considering that one clock period is necessary to define the position of the memory access and another to actually access that position. The time to perform a MAC operation, τop, was also defined as 2τClk, since two clock cycles are necessary to perform the division by four operation and that the sum is performed with combinational logic, which does not depend on the clock. Table I summarizes the established parameter values. Applying the parameter values in Eqs. (3) and (5) yields: τF P = 15τClk +640 ·480 ·1.33τADC NADCF P ,and (6) τdigital = 640 ·480 τADC NADCDig +τClk 2+63τClk NP E .(7) Equations (6) and (7) allow different NADC values for focalplane and digital approaches. Charge redistribution time is not taken into account, because of its negligible contribution to Eq. (6). The ratio between the expressions in Eqs. (7) and (6) is: τdigital τF P =τADC /τClk NADCDig +1 2+63 NP E  1.33τADC /τClk NADCF P .(8) Using the KADC constants defined in Sec. IV-C, we replace τADC in Eq. (8) by an appropriate function of τClk, which depends on the converter architecture. For the ramp converter we have τADC = 28·KRamp ·τClk = 256τClk. Considering that both the focal-plane and digital approaches use the ramp converter, the maximum advantage that the focal-plane approach achieves occurs when NADCDig = 1, NP E = 1 and NADCF P =NADCMax = 640. The focal-plane approach is then 600 times faster than the digital approach. If NADCDig =NADCF P =NADCMax = 640, the focal-plane approach is 120 times faster. For ramp converters, the effect of increasing the number of PEs is shown in Fig. 7, in dashdotted line, where the ratio between digital and focal-plane total operation times is plotted. With only four PEs, the focalplane approach is 31 times faster, so for ramp ADCs the focal plane advantage is modest. For the SAR or cyclic converters, we have τADC = Nbits ·KSAR,Cylic ·τClk = 128 ·τClk. These converters require fewer clock cycles to perform one conversion, but their operation frequency is limited, hence resulting in performance comparable to that of the ramp ADC. The maximum advantage the focal plane achieves with SAR or cyclic converters corresponds to 700 times faster. The dashed line in Fig. 7 shows the evaluation of Eq. (8) for the SAR converter when NADCDig =NADCF P =NADCMax = 640. To reduce the advantage of the focal plane to less than two orders of magnitude, three PEs are necessary. With ten PEs, the focalplane approach is 28 times faster. The Σ∆ conversion time depends on the OSR, which is equal to 25, as explained in Sec. IV-C: τADC =OSR ·KΣ∆ ·τClk = 200 ·τClk. The dotted line in Fig. 7 shows the comparison between focalplane and digital approaches when the Σ∆ converter is used. The result is in between the ramp converter and the SAR converters: only two PEs are necessary to reduce the advantage of the focal plane to less than two orders of magnitude. For the pipeline converter analysis, we assume that it is not possible to integrate 640 converters inside the chip, because an imager with one pipeline converter per column has not been reported, to the best of our knowledge. For this converter, τADC =KP ipeline ·τClk = 2 ·τClk. The solid lines in Fig. 7 correspond to results considering different numbers of pipeline ADCs. The focal-plane approach is highly advantageous when the number of ADCs is higher than 64. In this case, 18 PEs are necessary to drop the focal plane advantage to less than two orders of magnitude. The speed of the digital processor may be increased by using double data rate (DDR), which allows for memory access and shift operation (division by four) to be carried out in a single clock period. In order to perform timing comparisons between the focal-plane approach and generic digital circuits not having additional power or area requirements, we do not take the DDR into account in the analysis. Nevertheless, if τmem =τop = τClk, the processing time ratios presented in Fig. 7 halve. While focal-plane processing is being performed it is not possible to capture a new frame, which limits the frame rate. Even though, we can guarantee that the frame is always way above 30 frames/sec for the VGA resolution. If we consider a 100 MHz global clock, and one ramp converter per column, then approximately 1600 µs are necessary for generating the GP. Assuming that the image capture requires an additional 400 µs, then 2000 µs are necessary for image capture and GP generation, which yields frame rate around 500 fps. If the image resolution is increased to 6400×4800 (a factor of 100), it is still possible to achieve 60 fps by keeping the same conditions, which are namely one ramp converter per column and a global clock frequency of 100 MHz. V. ENERGY ANALYSIS COMPARISON The energy analysis is more complicated because it is highly dependent on the architecture, the technology parameters are 8 NPE 5 10 15 20 25 30 35 40 0 100 200 300 400 500 600 1 3 5 7 9 11 13 15 0 50 100 150 200 250 300 Ramp, NADC = 640 SAR, NADC = 640 Σ∆,NADC = 640 Pipeline, NADC = 1 Pipeline, NADC = 4 Pipeline, NADC = 16 Pipeline, NADC = 64 Pipeline, NADC = 160 τdigital/τF P Fig. 7: Ratio between digital and focal-plane processing times as a function of the number of PEs. Ramp, SAR, Σ∆, and pipeline ADCs are shown, respectively, in dash-dotted, dashed, dotted, and solid lines. For better visualization, a zoom of the curves is presented in the top right of the figure. also of major importance and there is no global parameter (as the clock period was global in the time analysis). Also, aside from the stages necessary for the GP generation in each approach, both architectures must comprise the controlling circuits outside the pixel matrix, which are responsible for the interface between each stage shown in Fig. 2. Although these circuits play an important part on the energy consumption, a proper energy analysis of the controlling circuitry requires a careful design of this stage, which is not under the scope of this paper, so these circuits are not considered. For the ADC stage, the energy consumption depends on the type of converter and architecture. A general empirical analysis on the energy efficiency of ADC architectures can be found in [64]. This paper defines a lower boundary for energy consumption per sample equal to 22(ENOB−9), and states that lowering the resolution below nine bits results in minor advantages. The minimum energy per sample in our case, eight bits, would be thus equal to 1 pJ/Sa. Although it is important to have this lower boundary limit it is also interesting to consider converters that have been used for image sensors. As mentioned in Sec. IV-C, several references were used for finding representative values of conversion rate and energy consumption for each ADC architecture. The median energy consumption per sample for each type of ADC, which can be seen in Fig. 6, is used in this section. Aside from the ADC, the other sources of energy consumption can be divided in: DC consumption, EDC , when there is a constant current flowing, usually for biasing circuits; dynamic consumption, EDynamic, as a result of the circuit activity, which requires charging and discharging capacitive nodes of the circuit; static consumption, EStatic, which is the energy WL WL BL BLWrite Wbit Vbias M1 M2 Mbias Fig. 8: One-bit SRAM memory cell, inside the dashed box, and memory write control circuit. that the transistor consumes even when it is off, depending on the leakage current Ileak; and short-circuit consumption, EShortcircuit, which is another source of dynamic energy and happens when switching the inputs of a logic gate, in a moment when both n-channel and p-channel transistors are on, thus allowing for a short-circuit current to flow. The short-circuit current can be minimized by matching the rise/fall times of the input and output signals, reaching a maximum of 15% of the total dynamic consumption [65]. EShortcircuit is computed as a portion of the dynamic energy: EShortcircuit = 15(EDynamic +EShortcircuit)/100 → EShortcircuit = 15EDynamic/85. In the following equations, Cnis the node capacitance, VddM is the pixel matrix voltage supply and Vdd is the voltage supply outside the pixel matrix. The dynamic power consumed by a digital circuit can be estimated by Pdynamic =Nd·Cn·V2 dd ·f0→1where Ndis the number of nodes and f0→1is the switching frequency of the nodes from 0 to 1 [65]. This equation is found considering that every node in the digital circuit is capacitive and that the energy necessary to charge a capacitive node is equal to Cn·V2 dd. The switching frequency can be written as a function of the clock frequency: f0→1=αfclk =α/τClk, where αis called switching activity factor and represents the probability of a node switching from 0 to 1, resulting in Pdynamic =α· Nd·Cn·V2 dd/τClk. The energy is given by Pdynamic multiplied by the time during which the circuit operates: Edynamic = α·Nd·Cn·V2 ddτtotal/τClk. In our case, τtotal can be computed according to the time analysis presented in Sec. IV. The SRAM memory is considered for the energy analysis of the digital circuit. The schematic diagram of a one-bit cell of this memory is shown in Fig. 8. The memory has the same size of the Level 0 image in the pyramid, M×N, and each pixel is represented with Nbits. In order to read a value from the memory, we need to select the memory row using the switch WL and read the result in the BL bus. Writing requires selecting a memory cell through the WL switches and setting Write to zero, which closes transistor M1or M2, depending on the bit that is being written, Wbit. If Wbit is logical zero, transistor M2closes and the bias current generated by Vbias discharges the bitline BL. If Wbit is logical one, transistor M1 closes and the bias current discharges the bitline BL and thus charges BL. 9 A. Focal plane Except for the A/D conversion stage, which was explained in the beginning of the section, the steps that were considered for the energy consumption estimation are described next. As opposed to the time analysis computation, here we have to consider the image capture and readout steps because the pixel matrix size has an influence in the consumption. 1) Image capture: this operation involves, for each pixel, charging the floating diffusion node and operating the Reset and TX switches, shown in Fig. 3. Dynamic: the energy for capturing a single pixel can be estimated as the one necessary for charging three capacitances, EpixCapture = (CF D ·V2 ddM ) + (CRst ·V2 ddM ) + (CT X · V2 ddM ). Since this operation happens for every pixel of the matrix, Ecapture = 2M·2N·EpixCapture. The capacitances CF D,CRst and CT X can be replaced by the node capacitance Cn, thus Ecapture = 2M·2N·(3· Cn·V2 ddM ).Static: transistors M1and M2from Fig. 3 are off for most of the operation and contribute with static energy consumption, EmatrixStatic = 2(2M·2N·VddM · Ileak ·τF PT otal ), where τF PT otal is given by Eq. (3). 2) Charge redistribution: this operation is passive, but energy is necessary to close the switches that connect the floating diffusion nodes. Dynamic: the energy that is needed to control two switches per pixel, 2Cn·V2 ddM , must be multiplied by the number of times the charge redistribution is performed (from Sec. IV-A) and by the size of the pixel matrix, since the operation is performed throughout the entire matrix, ECR = (NLev −1)nk·2M· 2N·(2Cn·V2 ddM ), where nkis the size of the filter. 3) Image readout: reading a pixel requires closing the row select switch and enabling the current source that biases the source follower. This current flows for the time necessary to charge the pixel matrix column capacitance. Dynamic: the gate of transistor M4, from Fig. 3, is connected to a bus with every other select transistor of the same row of the matrix, the equivalent capacitance is estimated as 2M·Cn. The pixel matrix column capacitance, on the other hand, depends on the number of rows and is estimated as 2N·Cn. The dynamic energy is thus EpixelReadDynamic = (2M+2N)·Cn·V2 ddM . The pixel matrix columns capacitances are charged whenever a pixel is read. The number of times a pixel is read is equal to Nconv, defined in Sec. IV-A. The row select switch is activated every time the image is being read, once for each row, thus Nconv/M times. The total energy is EreadT otal = [(Nconv/M)·2M+Nconv ·2N]·Cn·V2 ddM . B. Digital For the digital approach, we have the following steps: 1) Image capture: following the same analysis as in the focal-plane case, but changing the image size, yields Ecapture =M·N·(3 ·Cn·V2 ddM )and EmatrixStatic = 2(M·N·VddM ·Ileak ·τDigital). 2) Image readout: also very similar to the focal plane, but the bus capacitance changes and the image is read only once, EreadT otal = (N·M+M·N·N)·Cn·V2 ddM . 3) MAC operation: the digital processor that is considered is a MAC unit formed by a logic adder and a shift register. Dynamic: the energy consumed by a digital circuit was explained in the beginning of this section. In the case of the MAC operation, the time during which the circuit operates is Nop·3τop (according to Sec. IV-B), so EMACdynamic =α·Nd·Cn·V2 dd(Nop ·3τop)/τClk. Static: depends on the overall number of transistors inside the digital ports. Half of the transistors inside a common logic gate are off, so EMACstatic =NOff ·Vdd· Ileak ·τDigital.Short-circuit: as explained in the beginning of the section, EMACshort = 15EMACdynamic/85. 4) Memory read: reading requires charging the WL bus capacitance CW L, two switches per bit, and the BL or BL bus capacitance, represented by CBL.Dynamic: EreadDyn = (α·CBL +CW L)·V2 dd ·Nop ·2τmem/τClk, where Nop ·2is the number of times the memory is accessed for reading, according to Sec. IV-B. The activity factor αis only necessary for the BL bus and represents the cases where the bus voltage does not change when closing WL. The WL switch remains closed while the reading is performed and opens right after, so there is no activity factor in this case. Static: from Fig. 8, inside a one-bit memory cell, each inverter has one n-channel transistor and one p-channel transistor. Regardless of the state of the memory there is one p-channel transistor off and one n-channel transistor off. Besides, the WL switches can be formed by one nchannel transistor each, which are off most of the time. Thus, EreadStatic = 4·Vdd ·Ileak ·τDigital.Short-circuit: EreadShortcircuit = 15(EreadDyn)/85. 5) Memory write: writing a single value in the memory requires more energy than reading a single position of the memory because the bias current is activated, and the writing controlling circuits are used. Dynamic: EwriteDyn = (α·CBL +CW L +α·CWbit +CWrite + Cn)·V2 dd ·NMemW rite ·τmem/τClk, where CWbit is the capacitance of the input Wbit of the controlling circuit, CWrite is the capacitance of the node Write and Cnis the gate capacitance of either M1or M2, which are complementary nodes, so only one capacitance is considered. The number of times the memory is accessed for writing is NMemW rite =M·N+Nop, from Sec. IV-B. Static: the static power consumption is only due to the contribution of the write control circuit, because the cell circuit contribution was provided in item (4). Transistors M1and M2are on only when a bit is written, so we assume that they contribute with the static consumption during the entire operation. These transistors are necessary for every column of the memory matrix, so it must be multiplied by Nbits ·M. Furthermore, inside the NOR gates there is always two transistors off. We can consider that this circuit is repeated for each bit and for, at least, each NbusMem, resulting in EwriteStatic = (M·2·Cn+NbusMem ·4·Cn)·Nbits ·Vdd ·Ileak ·τDigital. Short-circuit: EwriteShortcircuit = 15(EwriteDyn)/85. DC: the bias current, that is activated whenever we need to swap a bit in the desired writing position, flows only