Diseño CMOS de un sistema de visión “on-chip” para aplicaciones de muy alta velocidad

UNIVERSIDAD DE SEVILLA DEPARTAMENTO DE ELECTRÓNICA Y ELECTROMAGNETISMO DISEÑO CMOS DE UN SISTEMA DE VISIÓN “ON-CHIP” PARA APLICACIONES DE MUY ALTA VELOCIDAD Memoria presentada por: FRANCISCO JOSÉ JIMÉNEZ GARRIDO Para optar al grado de Doctor Sevilla, 2015 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 2 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 3 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad Memoria presentada por: FRANCISCO JOSÉ JIMÉNEZ GARRIDO para optar al grado de Doctor Sevilla, Octubre de 2015 Dirigida por: Ángel Rodríguez Vázquez Rafael Domínguez Castro Tutora: Belén Pérez Berdú Departamento de Electrónica y Electromagnetismo Universidad de Sevilla Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 4 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 5 CONTENTS Resumen De La Tesis ................................................................................................................. 9 Thesis Summary ........................................................................................................................ 11 Resumen Extendido, Motivaciones y Conclusiones ............................................................. 13 Chapter 1 - Introduction and Motivations ............................................................................... 27 1.1. Introduction .................................................................................................................. 27 1.2. Outline of the Thesis.................................................................................................... 31 Chapter 2 – Basic Concepts, Definitions and Terms ............................................................. 37 2.1. From Light Radiation to Picture Elements ................................................................... 37 2.1.1. Imagers: Concept and Basic Terms .................................................................... 37 2.1.2. Basic Radiometric Concepts ............................................................................... 41 2.1.3. Lenses ................................................................................................................. 43 2.2. Basic Photosensors ..................................................................................................... 45 2.2.1. Photon Absorption ............................................................................................... 45 2.2.2. Accumulation Mode Photo-Sensors .................................................................... 47 2.2.3. Other CMOS Photo-sensors ............................................................................... 50 2.3. Pixels ........................................................................................................................... 51 2.3.1. Passive Pixels ..................................................................................................... 51 2.3.2. In-pixel readout electronics (APS) ....................................................................... 53 2.3.3. In-pixel processing electronics ............................................................................ 58 2.4. Digital Imagers, Camera Systems, Vision Systems .................................................... 61 2.4.1. Concept of Digital Imagers and Camera Systems .............................................. 61 2.4.2. Concept of Vision Systems and Basic Vision Operations ................................... 64 2.4.3. Hardware-software co-design ............................................................................. 66 2.4.4. Design of Hardware/Software systems ............................................................... 67 2.4.5. Vision Systems Architectures .............................................................................. 68 2.5. Typical specs for camera and vision systems ............................................................. 72 2.5.1. Application requirements, system specifications ................................................. 72 2.5.2. Camera specifications ......................................................................................... 75 2.5.3. Optics................................................................................................................... 76 2.5.4. Camera Functionality ........................................................................................... 77 2.5.5. Camera Image Quality......................................................................................... 78 2.6. Predecessors of the Q-eye .......................................................................................... 87 2.6.1. The CNN paradigm .............................................................................................. 87 2.6.2. Leon O. Chua and Lin Yang model ..................................................................... 89 2.6.3. Full Signal Range (FSR) model ........................................................................... 90 2.6.4. Image processing based on CNN template ........................................................ 91 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 6 2.6.5. CNN templates design......................................................................................... 93 2.6.6. The ACE and CACE chips .................................................................................. 93 2.6.7. Limitations of the ACE and CACE chips ............................................................. 99 2.6.8. Other types of computational sensors ............................................................... 101 2.7. Challenges for cameras and vision systems. Overview of the state-of-the-art ......... 103 2.7.1. Cameras-on-chip ............................................................................................... 105 2.7.2. Vision systems-on-chip...................................................................................... 106 Chapter 3 - The Q-Eye pixel .................................................................................................... 109 3.1. Q-Eye cell architecture .............................................................................................. 109 3.1.1. Block diagram and embedded operators .......................................................... 109 3.1.2. Dataflow, signal formats and control of the Q-Eye cell ..................................... 112 3.1.3. Rationale for selection of the Q-Eye operators ................................................. 115 3.1.4. Illustrating data reduction through the Q-Eye .................................................... 116 3.2. Circuit implementation of the Q-Eye cell operators ................................................... 118 3.2.1. Global Analogue References ............................................................................ 118 3.2.2. Data storage and transfer .................................................................................. 119 3.2.3. Image shifting .................................................................................................... 123 3.2.4. Optical sensor .................................................................................................... 125 3.2.5. Multiplier-Accumulator Circuit (MAC) ................................................................ 128 3.2.6. Logic and morphological operators ................................................................... 131 3.2.7. Convolutions ...................................................................................................... 134 3.2.8. Diffusion grid ...................................................................................................... 136 3.3. Characterization and modeling of errors ................................................................... 141 3.3.1. Analogue memory errors ................................................................................... 141 3.3.2. MAC errors ........................................................................................................ 154 3.3.3. Resistive grid errors ........................................................................................... 165 3.3.4. Errors in The Optical Sensor ............................................................................. 166 3.4. Exemplary results of degradation caused by errors .................................................. 173 3.4.1. time-invariant errors ........................................................................................... 174 3.4.2. time-variant errors ............................................................................................. 177 3.4.3. Practical recommendations for processing in the presence of errors ............... 180 Chapter 4 - The Q-Eye chip .................................................................................................... 183 4.1. The Q-Eye chip block diagram and architecture ....................................................... 183 4.1.1. External interfaces of the Q-Eye ....................................................................... 185 4.1.2. Control strategies: Configuration registers and Programmable Control Unit .... 192 4.1.3. Miscellaneous blocks ......................................................................................... 195 4.2. The Q-Eye read-out ................................................................................................... 204 4.2.1. About the readout channel architecture ............................................................ 205 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 7 4.3. Address-event read-out ............................................................................................. 221 4.3.1. Asynchronous downloading of images. Address-event reading ....................... 221 4.3.2. Random access to load data in array. Address-event writing ........................... 223 4.3.3. Global mean of images...................................................................................... 224 4.4. Supply and biasing in the Q-Eye ............................................................................... 225 4.4.1. Power supply and packaging ............................................................................ 225 4.4.2. Substrate noise .................................................................................................. 228 4.4.3. Power and reference distribution ....................................................................... 231 Chapter 5 - The Eye-RIS Vision System ................................................................................ 237 5.1. Eye-RIS systems architectures ................................................................................. 237 5.1.1. Generalities about the processing chain of vision ............................................. 237 5.1.2. Progressive processing, CVIS-based architectural concept ............................. 240 5.1.3. Eye-RIS architecture outlines ............................................................................ 242 5.2. Illustrating progressive, distributed processing: part-finding application ................... 243 5.3. The Eye-RIS_v1 ........................................................................................................ 249 5.3.1. NIOS-II Processor ............................................................................................. 250 5.3.2. Memory Map ...................................................................................................... 250 5.3.3. Digital Image Processor .................................................................................... 251 5.3.4. Digital Input/Output ports ................................................................................... 252 5.3.5. Q-eye controller ................................................................................................. 255 5.3.6. NIOS-Q-Eye controller communication ............................................................. 257 5.3.7. Controller memory ............................................................................................. 258 5.3.8. Controller microprocessor ................................................................................ 259 5.4. The Eye-RIS_v2 ........................................................................................................ 259 5.4.1. Architecture and control overview of task scheduling ....................................... 260 5.4.2. Pre-processing and post-processing implementation issues ............................ 264 5.4.3. External interfaces ............................................................................................. 265 5.5. The Q-eye and the Eye-RIS in operation .................................................................. 269 5.5.1. HDR signal acquisition ...................................................................................... 269 5.5.2. Spatial filtering ................................................................................................... 271 5.5.3. Extraction of images features and calculation of intermediate image representations .................................................................................................................. 275 5.5.4. Decision making ................................................................................................ 283 Appendices .............................................................................................................................. 287 Appendix A – Settling errors in gregorian’s s&h ................................................................. 287 Appendix B – DC errors in MAC operation ........................................................................... 291 References ............................................................................................................................... 299 Abbreviations and constants ................................................................................................. 314 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 8 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 9 RESUMEN DE LA TESIS Esta Tesis presenta arquitecturas, circuitos y chips para el diseño de SENSORES DE VISIÓN CMOS con procesamiento paralelo embebido. La Tesis reporta dos chips, en concreto:  El chip Q-Eye  El chip Eye-RIS_VSoC y dos sistemas de visión construidos con estos chips y otros sistemas “off-chip” adicionales, como FPGAs, en concreto:  El sistema Eye-RIS_v1  El sistema Eye-RIS_v2 Estos chips y sistemas están concebidos para ejecutar tareas de visión a muy alta velocidad y con consumos de potencia moderados. Los sistemas resultantes son, además, compactos y por lo tanto ventajosos en términos del factor SWaP cuando se los compara con arquitecturas convencionales formadas por sensores de imágenes convencionales seguidos de procesadores digitales. La clave de estas ventajas en términos de SWaP y velocidad radica en el uso de sensores-procesadores, en lugar de meros sensores, en la interface de los sistemas de visión. Estos sensores-procesadores embeben procesadores programables de señal-mixta dentro del pixel y son capaces por tanto de adquirir imágenes como de pre-procesarlas para extraer características, eliminar información redundante y reducir el número de datos que se transmiten fuera del sensor para su procesamiento ulterior. El núcleo de la tesis es el sensor-procesador Q-Eye, que se usa como interface en los sistemas Eye-RIS. Este sensor-procesador embebe una arquitectura de procesamiento formada por procesadores de señal-mixta distribuidos por pixel. Sus píxeles son por tanto estructuras multifuncionales complejas. De hecho, son programables, incorporan memorias e interactúan con sus vecinos para realizar una variedad de operaciones, tales como:  Convoluciones lineales con máscaras programables.  Difusiones controladas por tiempo y nivel de señal, a través de un “grid” resistivo embebido en el plano focal.  Aritmética de imágenes.  Flujo de programación dependiente de la señal.  Conversión entre los dominios de datos: imagen en escala de grises e imagen binaria  Operaciones lógicas en imágenes binarias.  Operaciones morfológicas en imágenes binarias.  etc. Con respecto a otros píxeles multi-función y sensores-procesadore anteriores, el Q-Eye reporta entre otras las siguientes ventajas:  mayor calidad de la imagen y mejores prestaciones de las funcionalidades embebidas en el chip;  mayor velocidad de operación y mejor gestión de la energía disponible;  mayor versatilidad para integración en sistemas de visión industrial. De hecho, los sistemas Eye-RIS son los primeros sistemas de visión industriales dotados de las siguientes características:  Procesamiento paralelo distribuido y progresivo.  Procesadores de señal-mixta fiables, robustos y con errores controlados.  Programabilidad distribuida. La Tesis incluye descripciones detalladas de la arquitectura y los circuitos usados en el pixel del Q-Eye, del propio chip Q-Eye y de los sistemas de visión construidos en base a este chip. Se incluyen también ejemplos de los distintos chips en operación. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 16 años, con una previsión total de mercado próxima a los 17 billones para 2018 [Yole14a-b]. La Figura 3 muestra los volúmenes de producción para sistemas CIS en diferentes campos de aplicación. Las principales ventajas de las tecnologías CMOS pueden resumirse en los siguientes puntos:  Coste,  Tamaño,  Consumo de potencia,  Velocidad,  Flexibilidad funcional. Esta Tesis se centra en el estudio de los sistemas de vision basados en tecnología CMOS y no pretende analizar ni cuantificar las ventajas y desventajas de los sistemas CMOS frente a los sistemas CCD. Los lectores interesados en sensores de imágenes basados en tecnologías CCD pueden consultar, entre otras muchas referencias, a [Theu95]. FIGURA 3. PREVISIÓN DE VENTAS DE SENSORES CIS POR CAMPOS DE APLICACIÓN Predicción realizada por Yole Development Los avances tecnológicos en los sistemas de visión CMOS han sido impulsados principalmente por las aplicaciones de consumo. Estas condicionan también los desafíos técnicos de industrias y universidades en relación con los sistemas CIS. Como ya se mencionó, el mercado de consumo demanda píxeles con un “pitch” cada vez menor y con resolución espacial (número de píxeles) cada vez mayor. Además de este aumento en la resolución, otros retos tecnológicos para los sensores de imágenes CMOS son: a) mejora de la calidad de imagen a través de la mejora de los circuitos analógicos de lectura; acondicionamiento de la señal, conversión AD (Analógico/Digital) y procesamiento digital de la señal; b) aumento de la velocidad de transmisión de datos, mejorando los circuitos de comunicación; c) reducción del área, potencia y coste a través de la integración de circuitos embebidos en un único chip (“system on chip”) [ISW13] [Elmg14] [IS14]. Los avances recientes incluyen sensores de varios millones de píxeles con “pixel-pitch” del orden de 1µm, velocidad de datos superior a los 10Gpx/s [Taka13], arquitecturas de lectura y conversión AD reconfigurables [Leñe14] [Kawa08], corrección de imagen [Chen12], gestión térmica y energética, etc. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 17 Todos los desafíos tecnológicos citados anteriormente se centran en la adquisición de matrices de intensidad de luz que definen imágenes 2-D. Durante los últimos años también se han desarrollado técnicas y arquitecturas con el objetivo de incluir en los sistemas de visión la estimación de profundidad, es decir, información 3-D. Uno de los principales impulsores de estas actividades es el desarrollo de interfaces hombre-máquina para la industria del entretenimiento [Payb14], pero el potencial de estas tecnologías es enorme en campos de como vigilancia, inspección industrial, automoción y medicina. Además de las técnicas basadas en la estereoscopía, la triangulación y similares, se están haciendo esfuerzos significativos en el desarrollo de píxeles CMOS que sean capaces de capturar la información de profundidad a través de técnicas de tiempo de vuelo (Time of Flight). Entre las diferentes aproximaciones, los diodos de avalancha, Single Photon Avalanche Diodes (SPADs), están siendo objeto de un creciente interés [Seit11]. Aparte de tecnologías de pixel específicas para ToF (tales como SPADs, doble puerta de transferencia, etc.), las medidas de tiempo de vuelo requieren complejos esquemas de iluminación [Nicl11]— todavía lejos de una solución integrada en un solo chip para capturar imágenes en 3-D. Por otra parte, los sensores 3-D incorporan a fecha de hoy muchas menos capacidades de procesamiento “on-chip” que los actuales sensores de visión 2-D. En esta Tesis, no se abordan problemáticas relacionadas con la adquisición de imágenes 3-D. Retomando el hilo de las ventajas del uso de tecnologías CMOS, tal vez la más relevante está conferida por la capacidad para diseñar sistemas de visión completos en un único chip (“system-on-chip”) en lugar de sólo sensores de imagen. El concepto de cámara integrada en un único chip (“camera-on-chip”) fue introducido en [Foss97]. Con referencia al diagrama de bloques de la Figura 2, este concepto implica la incorporación en un único substrato semiconductor del máximo número posible de los bloques funcionales necesarios para implementar una cámara. Por tanto, los sistemas de visión CMOS integrados en un único chip pertenecen a la categoría de sensores inteligentes — “smart sensors”; por ejemplo, sensores con inteligencia embebida [Meij01]. El concepto SENSORES INTELIGENTES es genérico y ambiguo. En general, la característica ‘inteligente’ hace referencia a cualquier tipo de circuito empleado para procesar las señales adquiridas por los sensores, ya sea en funciones como la lectura de datos y la corrección de errores, o para la extracción de información a partir de las imágenes. En la práctica pueden considerarse diferentes niveles de inteligencia. El más bajo implica básicamente funciones relacionadas con la lectura de las imágenes, corrección de errores y comunicaciones, que son las más utilizadas actualmente en la industria. En el mercado encontramos sensores de imágenes inteligentes con microprocesadores embebidos que generan imágenes de alta calidad completamente corregidas a velocidades del orden de miles de imágenes por segundo (Frames-per-Second, FpS) y con niveles de ruido equivalentes inferiores a un electrón [ANAFOCUS]. Niveles de inteligencia superiores, como los requeridos para extraer e interpretar la información contenida en las imágenes, y generar las decisiones y acciones correspondientes, han sido estudiados y desarrollados en el ámbito académico durante años [Eklu96] [Rodr04] [Abbo08] [Fern11] [Lopi11], y es solo ahora cuando empiezan a implementarse en sistemas industriales [ANAFOCUS]. Entre las razones de este retraso podemos citar la falta de estandarización en las soluciones arquitecturales, en su programación y en los formatos de salida [Khro13]. Sin embargo, los problemas arquitecturales pueden abordarse mediante el uso de arquitecturas celulares y, en particular, a través de la utilización del paradigma de redes celulares no lineales (Cellular Nonlinear Network, CNN. Este paradigma se adapta a las características específicas del problema, en conreto:  gran cantidad de datos con una estructura matricial regular que contiene propiedades e información distribuida en la globalidad, y  que pueden ser extraídas a través de interacciones y operaciones locales,  basadas en la propagación y procesamiento de señales en el espacio, a lo largo y ancho de cada imagen, y en el tiempo, a lo largo de la secuencia de imágenes [Rosk01] [Chua02]. Dependiendo del nivel de inteligengia integrada, distinguiremos dos tipos de sensoresprocesadores CMOS:  Sensores de imágenes CMOS inteligentes (Smart CMOS Image Sensors) denominados SCIS.  Sensores de visión CMOS (CMOS VIsion Sensors) denominados CVIS. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 18 La Figura 4 ilustra las diferencias entre SCIS, por un lado, y CVIS, por otro lado. Aunque las entradas de ambos circuitos integrados son imágenes capturadas por foto-sensores localizados en el plano focal, sus salidas son de naturaleza diferente. En el caso de los sistemas SCIS, los resultados básicos de la cadena de procesamiento son imágenes; bien en escala de grises o en color, en formato analógico o digital, sin o con correcciones, etc. Por el contrario, los resultados de los sistemas CVIS pueden no ser imágenes sino características 2 o incluso decisiones basadas en el análisis espacio-temporal de la información contenida en la escena. FIGURA 4. ILUSTRACIÓN DE LAS DIFERENCIAS ENTRE LOS SISTEMAS SCIS (ARRIBA) Y CVIS (ABAJO) Pese a las diferencias funcionales, los sistemas SCISs y CVISs son estructuralmente similares en que ambos combinan dos tipos diferentes de sub-sistemas integrados en un substrato de silicio común:  Los usados para la adquisición de imágenes: por ejemplo, la captura de la potencia lumínica y su transducción en tensión o corriente. Estos consisten esencialmente en uniones pn polarizadas en inversa (foto-diodos) y en circuitos de señal-mixta que controlan la operación del sensor.  Los que se utilizan para la lectura de los resultados del proceso de transducción de la luz, acondicionamiento de la señal y conversión AD, los empleados en la corrección y procesamiento, en el control global del sistema y las comunicaciones. Estos subsistemas están constituidos por circuitos analógicos, de señal-mixta y circuitos puramente digitales. Estas similitudes pueden ser explotadas durante la fase de diseño ya que muchos conceptos, arquitecturas y circuitos son compartidos por ambos tipos de sistemas y pueden ser reutilizados por los diseñadores. Este hecho queda reflejado a lo largo de la Tesis. 2. RESUMEN DE LA TESIS Como se ha mencionado anteriormente, el mercado de sistemas de visión, incluyendo sectores como vigilancia, visión industrial, sistemas de transporte inteligentes, electrónica de consumo, guiado de vehículos, redes inalámbricas de sensores, etc., demanda:  reducción de los costes de producción a nivel de sistema;  aumento del nivel de integración, con implementaciones físicas compactas, de tamaños reducidos y con el menor número posible de componente; 2 Características de una imagen pueden ser interpretadas como manifestaciones de la información contenida en una imagen; por ejemplo, el número de objetos presentes en una imagen, localización de objetos, etc. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 19  mayor calidad de la imagen y mejores prestaciones de las funcionalidades embebidas en los chips;  mayor velocidad de operación;  mejor uso posible del presupuesto de potencia disponible.  etc. Esta Tesis describe sensores de visión CMOS (CVISs), y sistemas basados en dichos chips, para alcanzar los objetivos anteriores mediante:  arquitecturas concebidas para el procesamiento paralelo de las imágenes;  el uso de procesadores de señal-mixta, en lugar de meramente digitales, distribuidos por pixel. Las razones que subyacen a estas soluciones se detallan a continuación. El PROCESAMIENTO PARALELO es muy adecuado para manejar la gran cantidad de datos presentes en las primeras etapas de la cadena de procesamiento de imágenes. Tal como se ilustra en la Figura 5, en las etapas iniciales de la cadena de procesamiento de la visión, el número de datos puede ser muy elevado pues es proporcional a la resolución espacial y al número de bits por dato de píxel. Además, muchos de los datos son redundantes y por lo tanto inútiles para los fines de extracción de información y la realización de tareas de visión. Sin embargo, el número de datos decrece a medida que se ejecutan los flujos de información y detección de características a través de la cadena de procesamiento. En realidad, en la toma de decisión final del sistema de visión están implicados muy pocos datos. Por ejemplo, en aplicaciones de vigilancia, las decisiones van a depender principalmente de si la escena contiene algún objeto sospechoso o no. FIGURA 5.- CADENA DE PROCESAMIENTO EN ALGORITMOS DE VISIÓN. Empezando arriba con las imágenes captadas por el sensor (datos en crudo), la cantidad de datos decrece y el nivel de abstracción aumenta. N representa el número de filas, M el número de columnas y B el número de bits usado por dato de pixel: n<N; m>M and p<(n,m). Las arquitecturas propuestas en esta Tesis mapeas esta estructura de datos en capas mediante el uso de estrategias de procesamiento específicas para cada paso de la cadena. En las arquitecturas de sistemas de visión convencionales la frontera entre sensores y procesadores está localizada en un punto done la cantidad de datos a transmitir es grande (ver la linea discontinua en la parte superior de la Figura 5). De hecho, los sistemas convencionales Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 20 descargan imágenes completas de los sensores para su posterior procesamiento, por lo que requieren una cantidad significativa de memoria y tienen una demanda computacional elevada. Por ejemplo, para una imagen con formato de alta definición para TV, con una resolución espacial 1920x1080 y una longitud de palabra de 10 bits por pixel, el número de datos asociados a una escena (“frame”) es 20.74Mbits, equivalente a 2.6Mbytes. Por tanto, la velocidad del flujo de datos asociada a la transmisión de esta imagen en un sistema operando según el estandar de video (30FpS) es 622Mbits por segundo. Para sistemas operando a altas velocidades, por ejemplo las aplicaciones de inspección industrial pueden requerir 1000FpS, el tratamiento de esta gran cantidad de datos puede llegar a convertirse en un obstáculo insuperable. Sin embargo, usando sistemas CVISs como los propuestos en esta Tesis, la frontera o interfaz entre el sensor 3 y los procesadores digitales está localizada en un punto donde la cantidad de datos a transmitir es pequeña, pues se transmiten características presentes en la escena en lugar de los datos de pixeles completos de la imagen (ver la linea discontinua en la parte inferior de la Figura 5). Supongamos que nuestra aplicación objetivo consiste en seguir objetos que se mueven a una velocidad de 40m/s en la escena. Esto requiere capturar y analizar imágenes a una razón de 2000Fps. En el resultado del proceso de captura y análisis el único dato pertinente es la posición predicha de los objetos. Esta es en realidad la única información transmitida al procesador digital. Pero para extraer dicha información se deben realizar antes las siguientes tareas:  Adquisición de imágenes,  filtros paso de baja,  detección de actividad,  estimación de movimiento,  seguimiento del objeto,  lazo de control y  predicción de posición. Los sistema CVISs propuestos en esta Tesis implementan estas y otras tareas directamente en el plano focal, evitando así los cuellos de bolleta en la comunicación de los datos y reduciendo los requerimiento de potencia de cálculo memorias asociados a las aqruitecturas convencionales. Consideremos por ejemplo el caso de una operación de convolución para un filtrado espacial linear. Esta operación implica 9 multiplicaciones y ocho sumas por pixel. Con lo cual, para el ejemplo previo del estándar de alta definición para TV (resolución espacial 1920x1080 a 30FpS) la velocidad de procesamiento requerida sería 1.06GOpS (Giga Operations per Second). Y el filtrado es solo una de las muchas tareas que deben ser llevadas a cabo en las primeras etapas del procesamiento de imágenes. En resumen, además de los problemas en la comunicación y almacenamiento de datos, los sistemas de visión convencionales neecistan potencias de cálculo elevadas. Es obvio que el problema es abordable con las tecnologías digitales actuales siempre que no existan limitaciones en el consumo de potencia y en el tamaño del sistema. Pero bajo fuertes restricciones de consumo de potencia, tamaño y velocidad de operación, es conveniente considerar arquitecturas alternativas como las porpuestas en esta Tesis. Además del procesamiento paralelo, otra característica distintiva de las soluciones propuestas en esta Tesis, es el uso de CIRCUITOS DE SEÑAL-MIXTA para procesamiento. Esto circuitos operan a mayor velocidad y con mayor eficiencia energética que sus homólogos digitales, bajo el supuesto de que los requisitos de precisión sean moderados. Además, procesar en paralelo con circuitos analógicos es ventajoso, frente a la alternativa digital, en términos de la cantidad de operaciones realizables para un consumo de potencia y una velocidad de operación dadas [Carm13]. En resumen, las arquitecturas de sistemas de visión propuestas en esta disertación superan a las soluciones convencionales en términos de la figura de mérito SWaP así como en términos de la relación entre la potencia-de-cálculo y el consumo-de-energía. Estas características, junto con el coste, resultan ideales para reducir la brecha que existe entre los detectores binarios y los detectores de área en aplicaciones de inspección y visión artificial — 3 Los sistemas CVISs no solo adquieren imágenes sino que las procesan. Por ello, un término más conveniente para ellos es el de sensor-procesador en lugar del de sensor. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 21 ver Figura 6. Es decir, los sistemas propuestos en esta Tesis pueden, por coste, tamaño y velocidad, reemplazar a los sensores uni-dimensionales que se usan actualmente en muchas cadenas de producción, capacidad de procesamiento superiores en varios órdenes de magnitud. FIGURA 6.- POSICIÓN DEL SENSOR INTELIGENTE TOSHIBA-TELI SPS [TOSH14] EN EL PLANO DE MERCADO OBJETIVO COSTE FRENTE A FUNCIONALIDAD. ESTA CÁMARA DE TOSHIBA-TELI EMPLEA ARQUITECTURAS CVIS CONCEBIDAS EN ESTA TESIS. Imagen tomada de http://www.toshiba-teli.co.jp/en/products/industrial/sps/sps.htm El sistema CVIS básico desarrollado en esta Tesis se llama Q-Eye. El píxel implementado en el sensor Q-Eye ha sido dotado de un alto nivel de capacidad de procesamiento y programación con el fin de implementar el mayor número posible de las funciones de procesamiento requeridas en las aplicaciones de visión temprana [Toma06]. Esta capacidad de programación del Q-Eye permite el desarrollo de cámaras inteligentes y sistemas de visión embebidos con una funcionalidad versátil. Es decir, el Q-Eye no está diseñado para una tarea específica sino que puede implementar una amplia variedad de algoritmos de visión mediante programación. Los píxeles del sistema Q-Eye intercambian información con sus vecinos para realizar distintas operaciones, tales como:  Convoluciones lineales con máscaras programables.  Difusiones controladas por tiempo y nivel de sañal, a través de un grid resistivo embebido en el plano focal.  Aritmética de imágenes.  Flujo de programación dependiente de la señal.  Conversión entre los dominios de datos: imagen en escala de grises e imagen binaria  Operaciones lógicas en imágenes binarias.  Operaciones morfológicas en imágenes binarias.  etc. El CAPÍTULO 3 proporciona una descripción detallada de la arquitectura del píxel del Q-Eye. Esto incluye esquemas para todos los circuitos integrados dentro del píxel, la descripción del comportamiento nominal de los circuitos, el análisis de los errores principales de los circuitos, y ejemplos del impacto de estos errores en la operación del sistema. El SISTEMA DE VISIÓN CMOS Q-EYE es descrito en el CAPÍTULO 4, donde se explica en detalle los puertos de comunicación, la programabilidad del sistema, las arquitecturas de los canales de lectura y los bloques misceláneos. Este Capítulo 4 introduce también los aspectos a nivel de sistema que deben considerarse durante la fase de diseño. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 22 La Tesis también describe el desarrollo de sistemas de visión basados en el sensor CVIS Q- Eye. Las diferencias entre estos sistemas y sistemas de visión convencionales han sido presentadas anteriormente haciendo referencia a la localización de la interfaz de datos entre la interfaz física de entrada sensor-procesador y los procesadores digitales estándar dentro de la cadena de procesamiento – ver Figura 5. La Figura 7 y la Figura 8 complementan estas explicaciones. Por un lado, la Figura 7 corresponde a un sistema de visión convecional. En este sistema, la interface de entrada captura la luz asociada a las imágenes de entrada y trasmite los datos de “frames” completos, representado por el símbolo F, para su posterior procesamiento. Como se mencionó anteriormente, la enorme cantidad de datos asociados a una imagen crea cuellos de botella en comunicaciones, memorias y potencia de cómputo. RAM RAM ADCADC F A/D Conversion Image Acquisition F Post-processing Decision making Pre-processing Tasks Conventional Vision System Image Sensor Converter Digital Processing FIGURA 7.- FLUJO DE DATO EN UN SISTEMA DE VISIÓN CONVENCIONAL Por otro lado, la Figura 8 corresponde a los sistemas de visión considerados en esta Tesis. En este caso, la interfaz de entrada no sólo captura la imagen sino que además procesa la información contenida en la misma. Por tanto, la cantidad de información a transmitir, representada por el símbolo f << F, a los posteriores bloques de procesamiento es mucho menor. En realidad, en la arquitectura de la Figura 8, el procesamiento se lleva a cabo de forma progresiva, distribuyendose las tareas de procesamiento entre la interfaz de entrada y el núcleo de procesadores digitales. La interfaz de entrada sensor-procesador realiza un pre-procesado de los datos ejecutando los algoritmos de procesamiento en paralelo directamente en la matriz de píxeles. Estos algoritmos deben tratar una gran cantidad de datos y son muy demandantes en términos de potencia y velocidad, por lo que se benefician del uso de circuitos de señalmixta. Respecto al pos-procesamiento, éste trata una menor cantidad de datos pero requiere lógica compleja con numerosos puntos de decisión, diferentes caminos de ejecución, etc. El pos-procesamiento consiste en complejos algoritmos los cuales requiren programabilidad y reconfigurabilidad. La carga computacional no es alta porque la cantidad de datos a procesar es mucho menor, relajándose las especificaciones de los procesadores digitales incluido en el sistema de visión. En definitiva, las diferentes secciones de la arquitectura propuesta se adaptan a la naturaleza de los datos a tratar, y el sistema final es mucho más eficiente que uno convencional. RAM RAM CVIS f, f << F Post-processing Decision making Pre-processing Tasks Image Acquisition Post-Processing FIGURA 8.- FLUJO DE DATOS EN UN SISTEMA BASADO EN EL CVIS Q-EYE. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 23 En el CAPÍTULO 5, se describen dos sistemas de visión desarrollados sobre el Q-Eye, en base al concepto representado en la Figura 8. Dichos sistemas son:  Eye-RIS v1 y  Eye-RIS v2 (Vision System on Chip), Estos sistemas sirven como demostradores de la viabilidad técnica y las ventajas de incorporar CVIS al diseño de sistemas de visión embebidos. Las diferencias entre ambos sistemas están en el nivel de integración. El segundo sistema (Eye-RIS_v2) integra sobre el mismo silicio el sensor-procesador Q-Eye y el microprocesador RISC, mientras que el primero (Eye-RIS_v1) integra el microprocesador RISC en una FPGA fuera del silicio. El Capítulo 5 incluye también secciones que muestran estos sistemas en operación. Esta Tesis es completada con un CAPÍTULO 2 dedicado a introducir conceptos básicos relacionados con la adquisición y procesamiento de imágenes y a explicar los principios de las aproximaciones arquitecturales empleadas a lo largo de esta Tesis. 3. CONCLUSIONES Y CONTRIBUCIONES Con esta Tesis se demuestra la viabilidad técnica del desarrollo e implementación de sensores CVIS (CMOS Vision Image Sensors) con procesadores paralelos de señal-mixta “per-pixel” y de sistemas de visión basados en dichos sensores para aplicaciones industriales. Las ventajas principales de estos sistemas de visión respecto a las arquitecturas convencionales de sistemas de visión son:  aumento del nivel de integración de las implementaciones físicas permitiendo en última instancia la realización de sistemas de visión compactos, en un solo chip;  mayor velocidad de operación;  mejor uso posible de la potencia y la energía disponibles. Las contribuciones de la Tesis producen desde la base del diseño e implementación de una arquitectura específica de sistema CVIS – el sensor-procesador Q-Eye. El estudio realizado para el diseño del Q-Eye ha considerado los diferentes niveles de jerarquía y campos de conocimiento que lleva asociado el desarrollo de un sistema de estas características. Dichos niveles van desde la definición de su arquitectura y especificaciones en base a las necesidades determinadas por las aplicaciones objetivo, hasta el análisis y diseño de los bloques de circuito que implementan sus funciones básicas usando técnicas de circuitos integrados. Partiendo del Q-Eye se han concebido e implementado dos sistemas de visión que usan a dicho CVIS como interface. Estos sistemas son:  Eye-RIS_v1 y  Eye-RIS_v2. Este último es un claro exponente del nivel de integración que puede alcanzarse con las técnicas propuestas en la Tesis pues s trata básicamente de un sistema de visión implementado íntegramente en un único chip (Vision System on Chip). A continuación se resumen las aportaciones principales de la Tesis:  Definición de un conjunto de operaciones básicas que pueden implementarse con circuitos a nivel de pixel y que permiten la implementación de las funciones de bajo nivel usualmente utilizadas en las primeras fases del procesamiento de imágenes. Poseer este conjunto de operaciones básicas dota al sensor-procesador de la versatilidad y programabilidad adecuados para desarrollar un elevado número de algoritmos en una amplia gama de aplicaciones. En concreto, el pixel del Q-Eye tiene mayor nivel de programabilidad que la mayoría de sus predecesores, muchos de los cuales incluyen circuitería para desarrollar tareas específicas y no son adecuados por lo tanto para sistemas industriales. Los CVISs ACE y CACE tienen niveles de programabilidad semejantes pero sus implementaciones son mucho más costosa en términos de área y potencia. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 24  Implementación mediante circuitos de cada una de las operaciones básicas incluidas al nivel de pixel, garantizando un nivel de precisión moderado (8-bits), y optimizando consumos de potencia y área del pixel para conseguir la mayor resolución espacial y el mejor “fill factor” posible. El resultado de ello ha sido que el pixel del Q-Eye tiene uno de los mayores niveles de densidad de dispositivos por área entre todos los pixeles con procesamiento implementados hasta la actualidad. Además, tanto la operación básica como los circuitos que la implementan se han desarrollado para garantizar robustez en la ejecución de los algoritmos de procesamiento. Una operación robusta del sensor significa que los resultados de los algoritmos, y las configuraciones y secuencia de instrucciones asociadas a los mismos, se mantienen tanto de chip a chip, como para condiciones ambientales y de entorno cambiantes dentro de los límites especificados. De hecho, una demostración de esto, es que los sistemas de visión Eye-RIS basados en el sensor inteligente Q-Eye son los primeros sistemas de visión industriales con procesamiento masivamente paralelo de señal-mixta en el plano focal que son fiables, robustos y con errores controlados susceptibles de calibración. Esto ha propiciado su comercialización.  Con objeto de garantizar la precisión de 8 bits, se ha llevado a cabo un análisis exhaustivo de los errores y no-idealidades asociadas a las implementaciones de las operaciones básicas. También se ha desarrollado una estrategia para implementar las funciones de procesamiento de bajo nivel de manera que se obtenga la menor acumulación posible de errores. Para ello se han generado modelos de las operaciones básicas a nivel de pixel y sus no-idealidades, para posteriormente estudiar el efecto de estos errores en los algoritmos de procesamiento. Esto se traduce en una gran precisión y reproducibilidad de la operación del Q-Eye “en campo“.  En base al pixel desarrollado, se ha diseñado el sensor CVSI Q-Eye en una tecnología 180nm estándar CMOS de 1.8V. Las funcionalidades más relevantes implementadas en el mismo se resumen a continuación: o Este sensor-procesador adquiere y procesa imágenes de forma concurrente, extrayendo características a muy alta velocidad a partir de la información espaciotemporal contenida en la secuencia de imágenes. La alta velocidad de operación viene conferida por el uso de procesamiento masivamente paralelo. o El Q-Eye se ha implementado en una arquitectura de procesador tipo Single Instruction Multiple Data (SIMD) que saca el máximo rendimiento de la capacidad de procesamiento de la matriz de píxeles multi-función. o Las instrucciones a ejecutar por dicha matriz se aplican a través de una interfaz de programación y bloque de control. Estos últimos contienen memorias para almacenar micro-instrucciones, las cuales definen la secuencia de operaciones a desarrollar por la matriz de pixeles con procesamiento embebido. El contenido de dichas memorias puede ser actualizado por un “host” externo. Esta arquitectura permite que el micro-procesador del sistema de visión, en el cual se integra el sensor inteligente diseñado, pueda definir los algoritmos de procesamiento de imágenes de bajo nivel a desarrollar por el Q-Eye, el cual actúa como “hardware” específico de procesamiento o co-procesador. o Las imágenes resultantes de este procesamiento son descargadas a través de la interface de imágenes. Esta interface se ha diseñado con un elevado grado de configurabilidad, pudiéndose cargar y descargar tres tipos de imágenes:  Imágenes binarias que representan la información comprimida resultante del procesamiento de bajo nivel y que contienen las características necesarias para el procesamiento de alto nivel.  Coordenadas de eventos o píxeles activos. Este protocolo de descarga es óptimo en velocidad para imágenes con puntos activos dispersos.  Imágenes en niveles de gris con 8 bits de resolución. Esta interfaz se ha definido principalmente con propósito de test. Pero puede usarse como interfaz de entrada salida al utilizar al Q-eye sólo como co-procesador de imágenes en niveles de gris. La capacidad de cargar y descargar imágenes permite utilizar al sistema Q-Eye sólo como co-procesador de imágenes (binaria y en niveles de gris) Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 25 o Para la implementación física de la interface de imágenes se han diseñado bloques de señal mixta tales como:  Convertidores AD Flash de 8 bits de resolución.  Convertidores DA de 8 bits de resolución basados en decodificadores.  Comparadores.  Circuitos “Sample & Hold”. Como resultado del diseño se han descrito los aspectos más importantes a tener en cuenta para el diseño e la implementación de todos los sub-bloques y estructuras funcionales, sus no-idealidades y los efectos de las mismas sobre la calidad de imagen.  La correcta operación de los circuitos de señal-mixta incluidos en la matriz de procesamiento e interfaz de imágenes requieren tensiones de referencias y corrientes/tensiones de polarización que han sido implementadas internamente a partir de un band-gap de alta precisión, convertidores DA y buffers. En relación con la polarización de los bloques de señal-mixta se ha desarrollado una estrategia de polarización y distribución de las referencias que garantiza un comportamiento robusto de los circuitos.  Con el objetivo de dotar al sistema Q-Eye con funcionalidades adicionales útiles para la integración en un sistema de visión se ha incluido en el Q-Eye bloques como: o Sensor de temperatura o Convertidor AD de aproximaciones sucesivas de propósito general conectado a distintos nudos internos que pueden se interesantes para un control automático del tiempo de exposición, procesos de calibración, etc. o “Power on Reset” para la aplicación de un “reset” inicial durante el encendido del sistema Q-Eye, garantizando un estado inicial conocido. o Unidad de generación de los relojes internos. Estos bloques, junto con la generación interna de referencias, se integran en el Q-Eye (“on-chip”) para reducir el número de componentes externos (“off-chip”) requeridos para la operación del sensor consiguiendo el mayor nivel de compacidad posible en las cámaras y sistemas de visión que incorporen el Q-Eye.  Se ha definido una arquitectura para un sistema de visión basado en el Q-Eye adecuada para su integración completa en un único chip, considerando la problemática que conlleva el diseño de sistemas embebidos. Esta arquitectura ha tenido en cuenta las características del sensor-procesador relativas a su configuración/programación y potencia de cálculo para realizar un co-diseño “hardware-software” óptimo. En una primera aproximación al problema se ha desarrollado el sistema de visión Eye-RIS_v1 para verificar la arquitectura del sistema de visión propuesta. Este sistema de visión está compuesto en líneas generales por el sensor inteligente Q-Eye, un controlador que actúa de interface ente el sensor-procesador y el micro-procesador del sistema, circuitería digital de procesamiento específico, memorias de programa y datos e interfaces de comunicación. En este caso, tanto el micro-procesador como la circuitería de procesamiento específica y el controlador van implementados en una FPGA.  Una vez verificada la arquitectura propuesta para el sistema de visión, se ha llevado a cabo el diseño e implementación física de una versión de dicho sistema en un único chip, utilizándose la misma tecnología 180nm estándar CMOS de 1.8V. Este sistema de visión “on-chip” se denomina Eye-RIS_Vsoc. Durante la fase de desarrollo del mismo se han identificado y estudiado todos los aspectos críticos desde un punto de vista del diseño e implementación, confirmándose que son similares a los presentes en el desarrollo de cualquier sistema de visión embebido: o Área: esta limitación implica una restricción en el número de recursos disponibles como son las memorias de programas y datos, condiciona la resolución espacial y la precisión del procesamiento, limita el número de “pads” para los interfaces de comunicación y alimentaciones, acota el número y complejidad de los procesadores, etc. o Restricciones temporales: El sistema de visión tiene un conjunto de procesadores y controladores que deben realizar un determinado algoritmo en un tiempo determinado por la velocidad de operación requerida en la aplicación. Por ello, los Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 32  increased integration level in physical implementations (compact vision systems);  largest possible image quality combined with per-chip embedded functional performance;  largest possible operation speed;  best possible usage of the available power budget. This monograph reports CVISs, and vision systems based on them, to meeting these challenges through:  parallel processing architectures with,  per-pixel distributed, mixed-signal processors. Rationale underlying these architectural strategies are described in the paragraphs below. PARALLEL PROCESSING is very well suited to handle the large amount of data present at early stages of the vision processing chain. As Figure 5 illustrates, the number of data available at initial steps of the vision processing chain may be huge for sensors with large spatial resolution and large number of bits per pixel data. Also, many of the data are redundant and hence useless to the purposes of extracting information and performing vision tasks. However, as information flows across the processing chain and features are extracted from the incoming images, the number of data decreases. Actually, very few data are typically involved in final decision-making; for instance, in surveillance applications decisions can be made depending on whether scenes include some suspicious object or not. Figure 5. PROCESSING CHAIN OF VISION. The amount of data decreases and the abstraction level increases as data evolve from the sensor interface (raw data); N represents the number of rows, M the number of columns and B the number of bits used per pixel data: n < N; m < M and p < (n, m). The a architecture proposed in this Thesis maps this layered data structure by using processing strategies fitted to each step in the chain. In conventional vision system architectures the border between sensors and core digital processors is placed at a point where the amount of data is large. This border is marked by the dashed line drawn at the top in Figure 5. It means that conventional vision system architectures download complete image frames from sensors, thereby posing significant memory and computational demands on the processing side. For instance, for an image with high definition TV format, with a spatial resolution 1920x1080 and a word length of 10bits per pixel, the number of data associated to a frame amounts to 20.74Mbits, equivalently 2.6Mbytes, and the throughput associated to the transmission of this image in a system operating at stadard video Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 33 rate of 30FpS is 622Mbits per second. If frame rates in the range of 1000FpS are required, as it happens for instance in some inspection applications, handling the corresponding huge amount of data may become an unsourmountable obstacle. However, using the herein proposed CVISs shifts the border between front-end sensors 5 and core digital processors to a point where features instead of frames are transmitted. The number of data at this border, marked by the dashed line at the bottom in Figure 5, is much smaller than at the top one. Assume for illustration purposes that we target to track objects moving at 40m/sec within a scene. It requires capturing and analysing images at 2000F/s rate. At the outcome of the capture/analysis process the only pertinent data are the predicted positions of the objects. This is actually the only information required by the core digital processors. But to extract this information the following tasks must be completed:  Image acquisition,  low-pass filtering,  activity detection,  motion estimation,  object tracking,  loop control, and  position prediction. CVISs in this Thesis performs these tasks directly at the sensory front-end, thereby precluding data communication bottlenecks and reducing the computational payload of digital processors. A simple exmaple will help us gaining insignt in this payload: Consider the case of a simple convolution operation required for a linear spatial filtering. It involves 9 multiplications and 8 additions per pixel. Thus, for the the previous example of high definition TV images at 30FpS a processing speed equal to 1.06GOpS (Giga Operations per Second). And, this is just one of the many tasks that must be completed at early vision stages, Hence, besides communication and memory bottlenecks, conventional vision system architectures also create processing bottlenecks. Of course, if there is no power size constraints and system size is neither a problem, you may scale conventional architectures to face these bottlenects. But, under speed and SWaP constraints it may be advisable seeking for alternative architectures. MIXED-SIGNAL CIRCUITS. Besides parallelism and distributed processing, the usage of mixedsignal processing circuits is another main attribute of the architectures proposed in this Thesis. Mixed-signal circuits are intrinsically faster and have larger power efficiency than digital counterparts under the assumption that accuracy requirements are moderate. Also, parallel analog processing reports advantages in terms of the number of operations attainable for given power consumption [Carm13]. All-in-all, it means that the vision system architectures proposed herein outperforms conventional solutions in terms of SWaP figures as well as in terms of operations/power values. These features, together with their cost, make them ideally suited to bridge the gap between point detectors and area detectors for machine vision and inspection applications – see Figure 6. The core proposal of this Thesis is a CVIS called Q-Eye. The Q-Eye pixel has been endowed with a high level of programmability in order to implement the largest possible number of early processing functions required by vision applications [Toma06]. This programming capability of Q-Eye permits the development of smart cameras and embedded vision systems with a versatile functionality, not designed to perform a specific task. They can be programmed to carry out several image processing algorithms. Actually, Q-Eye pixels interchange information with their neighbors to realize a variety of operations such as:  Linear convolutions with programmable masks.  Time- and signal-controlled diffusions (by means of an embedded resistive grid).  Image arithmetics.  Signal-dependent data scheduling.  Gray-scale to binary transformation. 5 CVSIs do not only sense but also process images. Hence, the correct word to refer them should be front-end sensorprocessors instead of just front-end sensors. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 34  Logic operation on binary images.  Mathematical morphology on binary images.  etc Figure 6. Positioning of the Toshiba-Teli SPS at the Cost-vs-Function target market plane. This camera from Toshiba-Teli employs the CVISs and vision architectures devised in this Thesis [Toshiba]. Image taken from http://www.toshibateli.co.jp/en/products/industrial/sps/sps.htm CHAPTER 3 provides a detailed architectural description of Q-EYE PIXEL. This includes schematics for all circuits embedded within the pixel including circuit details for all function embedded within the pixel, analysis of the main circuit errors and illustrations of their impact on ssytem operation. The Q-EYE CVIS CHIP is described in CHAPTER 4, where communication ports, system programmability, readout architecture and miscellaneous block are explained. Also, the issues at system level to be considered during the design phase are introduced. This Thesis also reports VISION SYSTEMS built using the Q-Eye CVIS at the front-end. Differences between these systems and conventional vision systems have been outlined above with reference to the location of the border between front-end sensor-processors and cores processors within the processing chain of vision – see Figure 5. These differences are further highlighted via Figure 7 and Figure 8. On the one hand, Figure 7 corresponds to a conventional vision system, where the front-end just senses input images and full frames, represented by the symbol F, are downloaded from this front-end for processing. As already mentioned, the huge amount of data associated to full frames create communications, memory and computational bottlenecks. RAM RAM ADCADC F A/D Conversion Image Acquisition F Post-processing Decision making Pre-processing Tasks Conventional Vision System Image Sensor Converter Digital Processing Figure 7. DATA FLOW IN CONVENTIONAL VISION SYSTEM. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 35 On the other hand, Figure 8 corresponds to the vision systems considered herein. Notice that the front-end does not only sense but also pre-process the information. Hence, it sends much smaller amount of data, represented by the symbol f << F, for ulterior processing. Indeed, in the architecture of Figure 8, processing is made progressivley by distributing processing tasks between the front-end and the core porcessor sections. The front-end already realizes a preprocessing of the data by executing parallel-processing algorithms directly in the pixel array. These algorithms involve many data and are very demanding in terms of power and speed thereby benefiting from the usage of mixed-signal techniques. Regarding post-processing, this stage handles less data but require complex logic with numerous decision points, alternate paths of execution, etc. The post-processing stage consists of complex image processing algorithms which require programmability and reconfigurability. The computational payload is not so high because the amount of data to process is less, therby relaxing the specifications of processors included in the vision system. All-in-all, the diiferent sections of the proposed architecture are fitted to the nature of the data they handle, and the whole system is much more efficient than the conventional one. RAM RAM CVIS f, f << F Post-processing Decision making Pre-processing Tasks Image Acquisition Post-Processing Figure 8. DATA FLOW IN VISION SYSTEM BASED ON THE Q-EYE CVIS. CHAPTER 5 is devoted to present a couple of vision system built according to the concept of Figure 8 and using the Q-Eye CVIS as sensor-processor front-end. These visions systems are respectively called:  Eye-RIS_v1 and  Eye-RIS_v2. Differences between these systems rely in the degree of embedding. Namely the second one embeds the Q-Eye and a digital RISC microprocessor in a single silicon substrate, while the first employs an off-chip RIS microprocessor implemented of FPGA. This chapter also shows the diiferent chips and systems in operation. The Thesis is completed with a Chapter, namely CHAPTER 2, devoted to introduce basic concepts related to image acquisition and processing. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 36 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 37 2. CHAPTER 2 – BASIC CONCEPTS, DEFINITIONS AND TERMS CSISs, SCISs and CVISs are complex systems whose design embraces quite different disciplines, including: solid-state photo-sensors, analog signal conditioning, readout and analog- to-digital conversion, digital image correction, digital microcontrollers, memories, etc. Therefore, the design of this type of advanced systems-on-chip requires the concourse of multidisciplinary teams. Also, because correct operation calls for close interaction among the different parts, global knowledge is mandatory for successful design and correct operation. This is particularly pertinent for CIS embedding smartness (SCIS and CVSI) and for those employed at high-end applications and/or requiring high-speed or low-noise image capture, on-chip image correction, scene interpretation, high dynamic range capture, etc. All these features demand architectural and circuital innovations and pose significant challenges to designers. These challenges are even more demanding when, as it happens in this Thesis, sensing is combined with image processing. Actually, pixels, chips and systems proposed throughout the monograph belong to general category of advanced sensory-processing imagers. Hence, besides the multidisciplinary challenges involved in pure imagers, others including image processing, mixedsignal processing circuits, digital processing circuits, hardware-software co-design, etc. must be considered as well. Core chapters of this Thesis describe respectively the architecture and design of a sensoryprocessing CVIS, called Q-Eye, and the architecture and implementation of two vision systems, called Eye-RIS, based on this CVIS. Thus Chapters 3 and 4 cover respectively the Q-Eye pixel and the Q-Eye chip and Chapter 5 covers the design and applications of two versions of the Eye-RIS systems. Previous to the presentation of the architectures, the circuits and the methods backing the design and usage of Q-Eye and Eye-RIS chips, this chapter is intended to present basic concepts related to imagers and vision systems. 2.1. FROM LIGHT RADIATION TO PICTURE ELEMENTS 2.1.1. IMAGERS: CONCEPT AND BASIC TERMS IMAGE SENSORS are aimed to convert optical scenes into electrical images. The concept is illustrated in Figure 1 where the light coming from a scene element gets focused by an imaging system into a sensing area, called image plane, consisting of an array of photo-detectors. Figure 1. CONCEPT OF IMAGE SENSING. Figure from [R. C. Gonzalez and R. E. Woods, Digital Image Processing, Prentice Hall, 1992] Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 38 Each site in the array (usually called pixel) embeds a photo-sensing device to obtain a spatial sample of the image when excited by the photons that impinge on the area corresponding to the detector. Samples are usually called picture elements. While sensitive element units may also be called picture elements, we prefer using the term pixel to refer to the physical entities and the term picture element to refer to the values provided by these entities. At the outcome of the photo-sensing and transduction processes the input optical scene gets sampled in space and encoded into a two-dimensional electrical image described by a twodimensional mathematical function,     1 ,, 1 ij iN I i j t x t for jM     Eq. 1 where N indicates the number of rows in the image (the image height), M indicates the number of columns in the image (the image width), and t is the time variable. Obviously, practical image sensors must include sub-systems to allow electrical images be read by external systems – READ-OUT CIRCUITRY. Two basic subsystems involved on that are:  Pixel addressing circuits,  Read-out circuits. Figure 2 shows a conceptual architecture of a CMOS image sensor including these two entities where electrical images are read as analog signals. The pixel array defines the sensing area and is a main agent for image quality. Vertical and horizontal access circuits are used to reaching pixels and reading pixel signal values. Usually, shift registers are employed to control addressing. A decoder can be used to access pixels randomly. The readout circuit in Figure 2 consists of a one-dimensional array of switches and a Sample and Hold (S&H) circuit [Ohta88] [Seit00]. Noise cancellation circuits, such as Correlated Double Sampling (CDS) [Enz96], are typically employed there. Pixel Array Vertical Addressing Column Amplifiers Analogue Mux Horizontal Addressing Output Amplifier Analogue output Figure 2. CONCEPTUAL BLOCK DIAGRAM OF AN ANALOGUE READ-OUT CMOS IMAGE SENSOR. Most modern CIS employ Analogue-to-Digital Converters (ADC) in the readout circuitry and employ sophisticated error correction circuits in the read-out channels, usually in per-column basis [Rodr15]. Further details about these digital CIS will be presented in section 2.5.1. MONOCHROMATIC AND COLOUR IMAGES. In the case of monochromatic (gray-scale) images each sample consists of a value which represents the light intensity captured by photo-detector placed in the corresponding array site (see Figure 3-left). These values are carried either by analog voltages or by Digital Numbers (DN) obtained through analog-to-digital conversion of corresponding voltage levels. In some cases, currents may be employed to carry picture element values [Berm04]. In the case of color images, the mathematical representation requires several bi-dimensional functions instead of just one. Typically, three functions are employed each encoding one out of three main color channels: red, blue and green – illustrated in Figure 3-right. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 39 Figure 3. (LEFT) MONOCHROMATIC IMAGE: 2D FUNCTION U(X, Y) WHOSE AMPLITUDE AT COORDINATES (X, Y) IS A POSITIVE SCALAR QUANTITY AND WHOSE MEANING IS DETERMINED BY THE IMAGE SOURCE; (RIGHT) POLYCHROMATIC (COLOR) IMAGE: COLLECTION OF FUNCTIONS UK(X, Y) DEFINED ON A 2D SPACE, AND CORRESPONDING TO DIFFERENT SPECTRAL COMPONENTS Electrical images do contain information which is encoded in the variations of the picture element values across space, on the one hand, and over time, on the other hand [Gonz92] [Seit00] – see Figure 4. These latter variations are typically synchronous with a clock and hence happen at regular intervals of this clock period. Moreover, picture element values are commonly obtained through processes where electrical charges get accumulated into capacitors over the clock period interval. Although some applications employ asynchronous, continuous-time readout, integration and discrete-time operation are typically employed because the instantaneous values of picture elements are extremely weak unless the illumination levels are very high. Figure 4. ILLUSTRATING VARIATIONS OF PICTURE ELEMENTS OVER TIME Let us compile a few useful DEFINITIONS before closing this section:  IMAGE SIZE, also referred to as image resolution or spatial resolution. This is expressed as the number of pixels obtained by multiplying the width (number of columns: M) by the height (numbers of rows: N). Different formats used in practical applications involve different number of pixels (M x N value) and form factors (M-to-N ratio). For instance: CIF is 352 x 288, VGA is 640 x 480, QSXGA is 2560 x 2048, etc. Figure 5 shows a chart of spatial resolution formats taken from Wikepedia Commons. pixel (1,1) pixel (i,j) pixel (N,M) Photo-detector array (1,1) (1,2) (1, ) (2,1) (2,2) (2, ) ( ,1) ( ,2) ( , ) i i i M i i i M I i N i N i N M       Mathematical representation Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 40 Figure 5. CHART OF SPATIAL RESOLUTION FORMATS. Chart from Wikipedia Commons  FRAME is somewhat equivalent to electrical image as it denotes the set of picture element matrices obtained after each period of the clock controlling the operation; i.e., after each snapshot.  FRAME RATE is the inverse of the clock period which determines the temporal updating of the picture elements; i.e. it is the frequency at which electrical images are updated in time. Most imaging systems operate on a frame-by-frame basis which involves obtaining and downloading complete frames. However, smart imagers may operate in asynchronous way by detecting events, thus precluding the downloading of complete frames [Delb06].  PIXEL PITCH denotes the physical dimensions of the pixels. Pixels employed used for last generation consumer electronic imagers are in the 1m range [Font11] [ISW13], while pixels of industrial CVISs devised for factory automation may have tens-of-m pitch; 35m in the case of the Eye-RIS system [Rodr10a] [Rodr10b]. This larger pitch is required to perform parallel processing on pixel and reports significant speed advantages in the analysis of image flows by reducing the amount of data of interest  similar to what animal retinas do [Rosk01] [Zara11].  FILL FACTOR, is the percentage of the pixel area which is occupied by sensitive devices. The fill factor value depends on the amount of no-sensitive circuitry embedded on-pixel as well as on the technology employed. For instance, sensing chips can either be irradiated at the front (frontside illumination) or at the back (backside illumination) [Font11]; 3-D chips consisting of vertically integrated physical layers can be used to separate the sensing devices from the processing ones [Garr08]; sensing materials can be deposed on top of CMOS wafers [ISORG]; etc.  REGIONS OF INTEREST. It denotes a sub-set of the total picture elements set.  IMAGE SCALES are downscaled, subsampled versions of the electrical images that preserve the features of the original ones. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 41 2.1.2. BASIC RADIOMETRIC CONCEPTS As already mentionned, electrical images are formed in a process that involves the irradiation of photo-sensor devices with an ElectroMagnetic Radiation (EMR) and the subsequent transformation of irradiated energy patterns into picture element values. This transformation happens point-wise across a spatial distribution of sensitive elements placed at the focal plane of the incoming EMR flow. The figure at the inset illustrates the concept for a monochromatic image. 2.1.2.1. PATTERNED RADIANCE MAPS AND SCENE FORMATION SCISs, CVISs are excited by optical scenes and meant to sense them. Optical scenes consist of modulated EMR power patterns. The formation of optical scenes and electrical images involves different disciplines, such as radiometry, materials science  particularly light-matter interactions, optics and, of course electronics. Object features are encoded within optical scenes in the form of spatial-temporal variations of the EMR radiant flux ; where the radiant flux, measured in Watts, represents the rate of change (power) of the EMR energy. When the photo-sensor array gets the radiation ,characterized by a patterned flux, electrical images are generated whose picture element values capture the spatial-temporal variations of the incoming scene. This generation involves the absorption of the radiant flux by the semiconductor and the subsequent transformation of the flux photons into electrical charges. Figure 6, adapted from [Fiet12], illustrates the generation of a scene power flux under outdoor lighting conditions  similar concepts apply for indoor condition. The illustration shows a chain of processes: i) an EMR source (the sun in this illustration) radiates energy; ii) part of this energy is lost and other part irradiates an object of interest; iii) this object produces a reflected radiant flux; iv) part of this flux reaches the camera lens and finally the sensors placed at the camera focal plane. Figure 6. ILLUSTRATING THE GENERATION OF A SCENE POWER FLUX UNDER OUTDOOR LIGHTING CONDITIONS. This figure has been adapted from R.D. Fiete, Modeling the Imaging Chain of Digital Cameras. SPIE Press 2012. Let us start with the EMR source. Its irradiation rate is characterized by the so-called radiant exitance (or emittance) which, in the case of the sun, can be approximated by using a temperature T=5,800ºK in the expression derived by Max Planck´s for the blackbody radiation: Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 48 The SENSITIVITY is defined as the amount of photocurrent ph I produced when a unit of power o P is incident on the material: ph ph o I RP  Eq. 18 The QUANTUM EFFICIENCY is defined as the ratio of the number of generated photo-carriers to the number of input photons. The number of input photons per time unit is: o P h   The number of generated carriers per time unit is: ph I e being e the electron charge. Thus, the quantum efficiency is determined by the equation:   / / Lph o Ie h QE R P h e        Eq. 19 Incident photons penetrate into the semiconductor according to their energy. Photons with smaller energy or longer wavelength penetrate into the semiconductor, while photons with larger energy or shorter wavelength are absorbed near the surface. The photons absorbed in the depletion region are swept by the electric field and accumulated in the potential well. By using the definition of absorption coefficient given by Eq.13, the light intensity depend on the position within the silicon:       z o dP z Pe dz       Eq. 20 Considering that each absorbed photon in the depletion region generates an electron-hole pair, the quantum efficiency is equivalent to:               0 1 p nn zz o zWz z o P e dz QE e e P e dz                       Eq. 21 Where W is the width of depletion region and n z is the distance from the surface to the edge of depletion region, see Figure 11. The width of depletion region is given by the equation:     2Si d a bi bias ad N N V V We N N        Eq. 22 Si  is the dielectric constant of silicon, a N and b N are the impurity concentrations, bias V the biasing voltage of pn-junction and bi V the built-in potential of pn-junction given by: 2 ln da bi B i NN V k T n        Eq. 23 Where i n is the intrinsic carrier concentration for silicon. The limits of depletion region are determined by the equations: Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 49 d a n aN N zW N  Eq. 24 d p ad N zW NN   Eq. 25 The quantum efficiency curve is dependent on the impurity profile of the n-type and p-type regions as well as position of pn-junction j z . Only carriers photo-generated in the depletion region are accounted for; some portion of the photo-generated carriers outside the depletion region diffuses and reaches the depletion region, but in the calculation these diffusion carriers are not account for. The red and Infra-Red (IR) light penetrates most deeply and reaches the p-substrate layer, where it produces minority carrier electrons. In p-substrate region, there is a weaker electric field, and the mechanism which moves the photo-generated carriers is the diffusion. Part of these photo-generated carriers are recombined and do not contribute to the signal charge, while others arrive at the edge of the depletion region and accumulate in the potential well, contributing to the signal charge. The extent of the contribution depends on the diffusion length of electrons produced in the p-substrate. The surface-interface states produce deep levels in the middle of the band-gap, carriers near this states (photo-charges generated too close to the surface in the n-diffusion) are easily trapped in these levels. The lifetime in these states is generally long and trapped carriers are finally recombined there. Such trapped carriers do not contribute to the signal charge. Blue light suffers from this effect because the absorption length corresponding to these wavelengths is lesser (Figure 10), and thus has smaller quantum efficiency than longer wavelengths. The surface of semiconductor is source of other important parameter, the dark current. Dark current is constituted by the carriers not photo-generated. This means that they are generated under dark conditions. The effect of this dark contribution will be studied in sections 3.3.4.1 and 3.4 in chapter 3. The dark current in photodiode has several sources: the diffusion current ( s I ), the surface leak current, impact ionization current, tunnel current, generation-recombination current, etc. In order to decrease the degradation of quantum efficiency for shorter wavelengths and reduce the dark current, the pinned photodiode (buried photodiode) has been developed. The structure of pinned photodiode is depicted in Figure 13. n p-sub out sel Pinned photodiode rst VDD FD TG n p+ Figure 13. STRUCTURE OF PINNED PHOTODIODE. The topmost surface of the photodiode has a thin p+ layer being buried the photodiode under the surface. In the p+-n-p- structure both p layers are on substrate potential (GND). As the voltage applied to the n-layer is increased, the depletion regions of both pn-junctions grow toward each other. At a certain voltage, the pinned voltage Vp, the depletion regions meet and no more majority carriers can be extracted from the device. The device is fully depleted [Fox98]. The potential then remains fixed inside the device and cannot be increased any further, the voltage is pinned. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 50 The potential profile at the surface is strongly bent so that the accumulation region is separated from the surface where the trapped states are located. Eventually, the photo-generated carriers at shorter wavelengths are quickly swept to the accumulation region by the bent potential profile near the surface and contribute to the signal charge, improving the quantum efficiency for shorter wavelengths and reducing the dark current contribution associated to the surface, because the charge collection region is separated from the silicon surface through the p+ layer. Other benefit of this structure lies in the complete charge transfer from a large accumulation area into the small Floating Diffusion capacitance (FD) through the transfer gate, allowing the implementation of a high electron-voltage conversion factor. In standard photodiode of 3T pixel architecture (Figure 12b), the increment of photodiode area to increase the number of collected electrons involves an increment of capacitance associated to the photodiode. Therefore, the electron-voltage conversion factor decreases. The total number of collected photons increased, but the conversion factor decrease; therefore, a trade-off between collecting area and conversion factor exists. For pinned photodiode, the collecting area and conversion capacitance are independent, allowing the increment of collecting area without increment of conversion capacitance. For light sensing operation, the pinned photodiode is initially fully depleted. During the integration phase, photo-generated majority carriers are stored in the depletion region (accumulation mode), decreasing the potential of pinned photodiode below pinned voltage Vp. For readout process, the floating diffusion (FD) is first reset to VDD. This reset voltage may now first be read out for true CDS (removing the KTC or reset noise). Next, the transfer gate (TG) is turned on and the complete photo-charge is transfer to conversion capacitance (FD). The complete transfer takes place if the voltage on FD remains above the pinning voltage while the pinned photodiode operates below this voltage. 2.2.3. OTHER CMOS PHOTO-SENSORS In addition to the accumulation mode, the reverse biased photodiode can operate connected to a MOS transistor working in sub-threshold region, which converts the photocurrent to voltage. This topology, depicted schematically in Figure 14, is called logarithmic pixel. High dynamic range images are achieved via logarithmic compression during conversion to voltage, using the exponential I-V characteristic of MOS transistor in sub-threshold. n p-sub out sel PhotoDiode VDD VDD Figure 14. PHOTODIODE PIXEL OPERATING IN LOGARITHMIC MODE. The structure of photo-gate, depicted in Figure 15, is similar to the MOS capacitor structure. Photo-generated carriers are accumulated in the potential well generated under the gate when a bias voltage is applying on it. For readout, the complete charge can be transferred via the transmission gate TG onto the floating diffusion FD after that had been reset. Thus, most of the photosensitive area does not contribute to the charge-voltage conversion capacitance unlike to the photodiode structure described in the previous section. The resulting high conversion coefficient (denoted by conversion gain) allows for a high sensitivity. However, the sensitivity is mostly lost by the low transparency of the overlying gate material. As an additional advantage, this reset-transfer structure permits true CDS operation including subtraction of reset noise. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 51 p p-sub out sel PhotoGate rst VDD FD TG PG + + + + + + + + + + + + + + + + + + n Figure 15. PHOTO-GATE PIXEL. 2.3. PIXELS Pixels are central to the design of imaging devices, including SCISs and CVISs. Pixels include photo-sensitive devices along with devices used for addressing, readout, etc. In same cases, particularly for CVISs, pixels may include a significant percentage of no-sensitive devices. When this happens it is common to use the term cells, instead of pixels, for each of the elements in the front-end array. Most of the silicon area of SCISs and CVISs chips is occupied by pixels. For instance, pixels take around 80% of the area of last generation, high-speed SCISs devised for machine vision, and approximately 50% of last generation CVISs devised for inspection and factory automation. Note in the illustrative cases of Figure 2 that all pixels are identical and that they are arranged into an uniform, rectangular array. Although this is the most common pixel arrangement, in some special cases the grid may not have a rectangular shape and/or the elements can be unequal. A particularly relevant exception follows a foveated distribution which resembles that of animal retinas. These distributions shows a central region, called fovea, composed of densely packed, small pixels and an increase of the pixel size (correspondingly a decrease of the pixel packing) towards the periphery. Figure 16 shows practical examples of foveated pixel distributions [Pard96]. Figure 16. EXAMPLES OF FOVEATED PIXEL ARRANGEMENTS. 2.3.1. PASSIVE PIXELS CMOS pixels include non-sensitive devices (for instance, MOS transistors) in addition to the photo-diode (sensitive area). These non-sensitive devices are included in relation to addressing and readout functions. According to the introduction of active elements at pixel level, CMOS pixels can be divided into two main groups:  passive pixel sensors (PPS), [Weck67], and  active pixel sensors (APS), [Nobl68]. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 52 The passive pixel (PP) consists of a photodiode and just one switching transistor (selection transistor). This transistor is used as a charge gate, transferring the content of the pixel to a Charge Integration Amplifier (CIA), see Figure 17. Figure 17 describes the PPS image sensor including the addressing circuitry and readout path. Most modern PPS architectures use a CIA per column in the array. Y-Addressing CIA X- Addressing Readout Circuitry Vref Cint Reset Out sel<i-1> sel<i> sel<i+1> col<i-1> col<i> col<i+1> Vcol Output Buffer CIA Vcol Vref CIA Vcol Vref CIA Vcol Vref Vref Figure 17. PPS ARCHITECTURE. The voltage Vref is used to reset the photo-diode into the reverse bias stage. After reset operation, the selection transistor is turned off during the integration time, where the photodiode collects charges proportionally to the amount of incident illumination. When the selection transistor is turned on again, the photo-generated charge is transferred from the photodiode to the readout circuit. The total charge that flows to the data column is equal that charge collected during the integration period. This charge is integrated on the capacitance associated to data column and output as a voltage through the CIA circuitry. After reading the signal voltage, during the reset phase where the selection transistor is turned on, the column bus and photodiode voltages return to Vref via the charge amplifier. At the end of reset process the selection transistor is turned off, the charge stored in Cint is removed by the reset transistor in the readout circuit, and the integration process start again. Figure 18 shows the control diagram for a PPS. The out voltage is given by the expression: int photodiode out ref Q VV C  Eq. 26 Reset Sel<i> Out Exposure Photodiode reset Photodiode reset Vref Vsignal Vref Read signal Figure 18. PPS CONTROL DIAGRAM. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 53 The passive pixel has the highest fill factor. But, its structure has major problems due to its large capacitive loads at the input of CIA corresponding to the data column. Previous result in Eq.26 has not considered the finite gain of amplifier in the readout circuit. Since the large bus is directly connected to each pixel during readout, the finite gain of amplifier implies that the load capacitance associated to the large bus affects to the charge integrated in Cint reducing the conversion factor. This large capacitance of column bus means that the RC time constant is very high and the readout speed is slow. In addition, passive pixel readout noise is typically high. The thermal noise generated during the reset process of photodiode has associated a noise power expressed in charge given by the expression: 2 eB n k T C   Eq. 27 Where B k is the Boltzmann constant, T the absolute temperature and C the sampling capacitance. A PPS has a large sampling capacitance (data column capacitance) and hence large noise cannot be avoided. Because of all these factors, PPS does not scale well to larger array sizes or faster pixel readout rates. In order to overcome the limitations presented in the preceding paragraphs an active buffer is introduced at pixel level giving rise to the active pixel concept, [Foss92]. 2.3.2. IN-PIXEL READOUT ELECTRONICS (APS) An active pixel (AP) includes a local active buffer [Ando97] which is usually a source follower transistor together with the reset and selection transistors. Figure 19 describes the general structure of an active pixel. Photodetector Reset Select Local buffer Column bus Photodetector Reset Select Column bus Transfer A) Photodetector is a photodiode B) Photodetector is a photogate or pinned photodiode Local buffer FD FD Selection Transistor Selection Transistor Reset Transistor Reset Transistor Figure 19. ACTIVE PIXEL SCHEME. Active pixels typically have a fill factor (relation between the sensing area and the total pixel area) of only 50-70%. This fact implies a reduction in the effective sensitivity area; and therefore, a reduction in the photo-generated charge. However, APS topologies present advantages also. For instance, the reduced capacitance in each pixel leads to lower read noise for the array, what means an increment in the dynamic range and signal to noise ratio. The active pixels can be divided into three types, depending on the photo-sensitive structure implemented: photodiodes, photo-gates and pinned photodiodes. The active pixel employs a photosensitive device and a local readout circuit of three transistors: one reset transistor (Reset), one row select transistor (Select) and one source-follower transistor. Usually, the photodiode AP has a structure called 3T pixel (consist of three transistors) and the pinned photodiode or photo-gate has a structure called 4T transistor, where the transfer gate is considered a transistor, see Figure 19. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 54 The charge-to-voltage conversion occurs at the capacitance of integration node (Floating Diffusion, FD), which comprises the input source-follower transistor and all other parasitic capacitances connected to that node. In the case of photodiode pixel, this capacitance consists of photodiode capacitance, diffusion capacitance of reset transistor and input capacitance of Source Follower. But, in the case of pinned photodiode pixel, the capacitance associated to the photodiode does not contribute to the FD capacitance. Furthermore, the photodiode capacitance is null while photodiode stay pinned (the depletion regions are met). The pinned photodiode pixel has a significant advantage over the photodiode pixel. This advantage of pinned-photodiode pixel is an integration node capacitance independent from the photo-sensitive area. In case of photodiode pixel, the integration node capacitance includes the capacitance associated to the photodiode, which depends on area and perimeter of latter. Then, larger photodiodes means more sensitive area but also an increment in the integration node capacitance, which implies a reduction in the conversion gain. Therefore, the responsivity is supposedly independent of detector size. On the contrary, in a pinned photodiode pixel, the photo-sensitive area can be increased to collect a larger number of electrons and the electron-voltage conversion capacitance remains fixed because photo-detection and photo-conversion regions are separated and the pinned photodiode has associated a null capacitance while stays in pinned state (voltage of photodiode remains above the pinned level). In both pixels, the Source Follower (SF) transistor acts as a buffer to isolate the integration node and the output data column; the load of this buffer (the active-current-source load) is located on each column, which is common for all pixels of column, in order to keep the fill factor high. The reset transistor cleans the photodiode and initializes the sensing node (FD) before charge-to- voltage conversion. It is usually implemented with a NMOS transistor; this allows a higher fill factor. However, an NMOS transistor with VDD on both gate and drain can only reach a voltage of VDD-VT, because the NMOS transistor operates in sub-threshold at this level, and thereby decreasing the dynamic of the pixel. Generally, 3T pixel operation can be divided into two main stages: the reset phase (consisting of photodiode reset and integration node initialization) and phototransduction phase. For 3T pixel architecture, the reset transistor controls the integration time cleaning the photodiode and initializing the integration node. During the reset phase, the reset transistor is turned on and carriers of photodiode flow through the reset transistors towards the pixel power supply, cleaning the photodiode. Moreover, the conversion capacitance (CFD) is set to a reset voltage. After that, the reset transistor is turned off and the photo-generated carriers are collected by the photo-diode, the phototransduction phase begins at this moment in parallel with the charge-to-voltage conversion. During phototransduction phase of 3T pixel, the capacitance associated to the Floating Diffusion node is discharged during the integration time by the photo-generated current, which is proportional to the incident light power. In the case of 3T pixel array, the operation of pixels is performed applying a rolling control technique, described in Figure 20. All pixels in each row are reset and read out in parallel, but the different rows are processed sequentially. This operation mode is denoted by rolling shutter. After phototransduction phase, the signal value is read out to one Sampled & Hold (S&H) circuit of the Correlated Double Sampling block (CDS). Then, the pixel is reset, and the reset value is read out to a second S&H of the CDS circuit. The CDS circuit, which can be located at the bottom of each column, subtracts the photo-generated signal value from the reset value to obtain the output signal. Its main purpose is to eliminate Fixed Pattern Noise caused by random variations in the threshold voltage of reset and pixel amplifier transistors (Source Follower). In addition, it should reduce the 1/f noise in the circuit and eliminate the reset noise in a true CDS operation. However, for 3T pixel, the CDS circuit subtracts the signal pixel value from the reset value of the next frame. In this case, the CDS operation is not truly correlated, the read noise is limited by the reset noise on the photodiode. In order to implement a true CDS, a memory is required to store the correlated reset voltage. Indeed, the fact that reset phase entails the photodiode reset operation and the initialization of sense node in a 3T pixel makes impossible a true CDS operation. The operation associated to 4T pixel architecture can be divided into four main stages: the photodiode reset phase, the accumulation or phototransduction phase, the FD reset phase and Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 55 the charge-to-voltage conversion or transfer phase. The pixel operation is described in next paragraph. reset<i-1> select<i-1> reset<i> select<i> reset<i+1> select<i+1> Tframe Readout Exposure Reset Reset Phototransduction Figure 20. CONTROL SIGNALS FOR 3T PIXEL IN ROLLING SHUTTER MODE. First, the photodiode is cleaned turning on the transfer and reset transistors. The photogenerated carriers flow to the pixel power supply during the photodiode reset phase. When the transfer transistor is turned off, the photo-generated carriers are accumulated in the photodiode (accumulation phase). Just before transferring the photo-generated charge, the integration node FD is reset turning on the reset transistor. The reset value is read out for CDS operation turning on the selection transistor. After the reset readout is finished, the signal charge accumulated in the photodiode is transferred to the integration node through the transfer transistor, which is turned on. Following, the signal voltage resulting from charge-to-conversion process is read out. reset<i-1> transfer<i-1> Tframe select<i-1> reset<i> transfer<i> select<i> reset<i+1> transfer<i+1> select<i+1> Reset Photodiode Reset FD Charge Transfer Row Readout Exposure Photodiode Reset FD Reset Transfer Accumulation or phototransduction Figure 21. CONTROL SIGNALS FOR 4T PIXEL IN ROLLING SHUTTER MODE. The pinned photodiode pixel can operate following a rolling technique described in Figure 21. A true CDS operation can be implemented when the 4T pixel works in rolling shutter mode. Since, before transferring the charge from the photodiode to the FD node, this FD node is initialized to the reset voltage and then this value is read out. After that, the transfer is carried out and signal voltage resulting from the integration in FD capacitance is read. In this case, the thermal noise of reset transistor is completely correlated in two readout phases and it can be eliminated. The 4T-APS achieves low noise operation and thus its noise performance is comparable to CCDs. The rolling shutter mode has an important disadvantage in high speed applications where there are scenes with objects moving at high speed, see Figure 22. In this case, the objects appear distorted in the sensed images because the start and end of the light collection for each row is slightly delayed from the previous row. These objects will be perfectly captured in a global shutter operation. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 56 Figure 22. ROLLING SHUTTER MODE FOR HIGH SPEED APPLICATIONS Figures from Wikipedia Common The introduction of transfer gate in pinned photodiode pixels makes possible to carry out a global operation in the pixel array. This means that all pixels can operate in parallel. The reset phase of photodiode, exposure and charge transfer phase are carried out in parallel by all pixels of the array, at the same time. The readout process of array follows a rolling technique, see Figure 24. This operation mode is denoted by global shutter. The transfer gate makes possible to implement an electronic global shutter method, [Yadi91]. This technique uses an analogue memory element inside each pixel. In the case of pinned photodiode pixel, the FD node is the analogue memory. For a photodiode pixel, an additional transistor is required in order to implement a 4T structure as it is described in Figure 23. In this case, the shutter transistor operates in a similar way than transfer transistor in pinned photodiode pixel. Photodetector Reset Select Local buffer Column bus FD Shutter Reset transistor Transistor selection Figure 23. SNAPSHOT STRUCTURE FOR PHOTODIODE PIXEL. 4T structure allows all pixels of array to expose at the same time. After the integration time the signal charge is stored in the analogue pixel memory until readout. An important issue that should be addressed in the global shutter technique is the shutter efficiency. The charge stored in the internal pixel memory is affected by:  the incident light due to the parasitic sensitivity of diodes associated to the diffusions of FD node  leakage current of these diffusion. The effects due to this phenomenon are appreciable in applications with high level of luminosity, short exposure times and low speed in the readout operation. Under these conditions, the value of parasitic exposure corresponding to the floating diffusion could not be negligible respect to the exposure associated to the photodiode operation. In this case, gradients in the output image can be detected. Exposure parameter is given by the product of light power by the integration time. In order to reduce the impact of FD parasitic sensitivity, layout techniques in the implementation of pixel are considered to cover at maximum level the FD node and minimize the leakage currents. For instance, in the case of pixel described in Figure 23, shutter efficiency can be increased using a PMOS transistor as a shutter, if it is well separated from the photodiode. Unfortunately, a PMOS shutter decreases the fill factor and, due to increased parasitic capacitances, also decreases the conversion gain. The global shutter mode does not allow the true CDS operation. Therefore, the reset noise cannot be eliminated. In order to carry out a true CDS, two analogue memories in pixel are called for, one memory to store the reset value before integration and another to store the photo-generated signal. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 57 Regarding the concurrency of acquisition operation and readout process which is demanded in high speed applications, the readout process cannot be carried out in parallel with the photodiode reset phase in the case of pinned photodiode pixels operating in global shutter mode, see Figure 24. This means that the photodiode operation is not completely concurrent with the readout process. In order to achieve the completely concurrence, an additional transfer gate is included in the pixel to clean the photodiode without transferring charge to the FD node, see Figure 25. reset<i-1> transfer<i-1> Tframe select<i-1> reset<i> transfer<i> select<i> reset<i+1> transfer<i+1> select<i+1> Reset Photodiode Reset FD Charge Transfer Array Readout Exposure Figure 24. CONTROL SIGNALS FOR 4T PIXEL IN GLOBAL SHUTTER MODE. Photodetector FD Reset Select Local buffer Column bus FD Transfer Reset transistor Transistor selection PD Reset Figure 25. 5-T PINNED PHOTODIODE PIXEL STRUCTURE In addition, this fifth transistor or second transfer gate is used to avoid a known blooming effect in the photodiode, Figure 26 shows this phenomenon. For this reason, this second transfer gate is denoted by anti-blooming transistor also. Figure 26. BLOOMING EFFECT Figures from Wikipedia Common Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 64 standard cameras and camcorders. Indeed, smart cameras are particular types of embedded vision systems intended for decision making in vision-enabled automatic control system [Belb09]. To that purpose smart cameras include sub-systems to directly interact with automation equipment via its discrete I/Os and built-in industrial protocols, like EtherNet. Image Sensor Image Enhancement & Restoration Feature extraction Detection / Segmentation Representation & Description Recognition / Classification STANDARD CAMERA SMART CAMERA Data-intensive tasks Data-intensive tasks Data-intensive tasks Math-intensive tasks Math-intensive tasks Figure 32. SMART CAMERA CONCEPT. Figure 32 illustrates differences between smart and normal cameras. The data processing block of smart cameras completes early-processing and high-level image processing. Main components of a smart camera are typically the same than those of a normal camera. However, the former embeds powerful signal processing units to perform image feature extraction and/or pattern analysis based on application-specific requirements. Also a smart camera has a output generator which produces a coded representation of image features and/or results from the pattern matching, or in some cases, control signals for other devices (e.g. alarm triggering signal) or actions (e.g. sending a picture of the number of a car which is speeding to police). There are two major differences between a smart camera and a standard or normal camera. The first is at architectural level. A smart camera usually has a special image processing unit containing one or more high performance processors to run ASIP algorithms seeking to extract information from images. This processing hardware is usually more powerful than that employed for normal cameras where only improving quality matters. The other main difference is at data interface level. A smart camera outputs either the features extracted from the captured images or a high-level description of the scene, which is transmitted into an automated control system. But, for normal cameras the output is the processed version of the captured image for human consumption. For this reason, normal video cameras have large output bandwidth requirements (in direct proportion to the resolution of image and frame rate), while smart camera can have very low data bandwidth requirements at the output. 2.4.2. CONCEPT OF VISION SYSTEMS AND BASIC VISION OPERATIONS The concept of vision system was brought to scene in this chapter as part of the description of smart cameras. Vision systems are targeted to analyze images and to extract useful information from them in order to solve a determined task. Application areas are large, including: industrial robots, autonomous vehicle, surveillance, people counting, machine vision, computer-human interaction, artificial intelligence, intelligent transportation systems, etc. Processing algorithms employed in vision systems are based on COMPUTER VISION science [Shap91] [Davi12][Szel11]. Computer vision covers a wide range of topics which are often related to other disciplines. For instance, artificial intelligence is defined as the study and design of intelligent agents, where intelligent agent is a system that perceives its environment and takes actions that maximize its chances of success. The information about the environment could be provided by a vision system, acting as a vision sensor and providing high-level information about the environment. Others fields that are closely related are signal processing, mathematics (statistics, optimization, geometry), image processing, image analysis, etc. Many Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 65 methods in computer vision are based on these fields. Neurobiology plays an important role, specifically the study of biological vision system. There are extensive studies of eyes, neurons and the brain structures devoted to processing of visual stimuli. Results of these studies have been a coarse and complicated description of how biological vision systems operate in order to solve certain vision tasks. Based on these descriptions, a subfield within computer vision has been developed, where artificial vision systems are designed to mimic the processing and behavior of biological systems. Also, some of the learning-based methods developed within computer vision have their background in biology. Some main functions carried out by general vision system architecture are listed in the bullet points below:  IMAGE ACQUISITION. Digital images are produced by one or several cameras. Generally, depending on the type of sensor, the resulting image data are either ordinary 2D images, or 3D volumes, or image sequences. The pixel values can represent several physical measures such as: light intensity in one or several spectral bands (gray images or color images), depth, absorption or reflectance of sonic or electromagnetic waves, nuclear magnetic resonance, etc.  PRE-PROCESSING, including image enhancement and restoration. In general terms, they are similar in that both pursue to deliver images that are better suited for given application than original provided by the sensors. More specifically, restoration attempts also to reconstruct or recover an image that has been degraded by using some a priori knowledge of the degradation phenomenon.  FEATURE EXTRACTION. Image features of various complexity levels can be extracted: edges, ridges, localized interest points, corners, blobs, etc. A key problem here is choosing those features that will achieve significant data reduction while preserving the information that will allow subsequent algorithms to distinguish items of interest in images.  DETECTION/SEGMENTATION. Identification of relevant points and regions of the image for further processing, subdividing the image into its constituent parts or objects. Segmentation is one of the most important elements in automated image analysis because it is at this step that objects or entities of interest are extracted from the image for subsequent high-level processing, such as description and recognition.  REPRESENTATION AND DESCRIPTION. After an image has been segmented into regions, the resulting aggregate of segmented pixels must be represented in a form suitable for further computer processing. Basically, there are two methods for representing a region: o Representation based on its external characteristics o Representation in terms of its internal characteristics Generally, an external representation is chosen when the primary focus is on shape characteristics (morphological characteristics), while an internal representation is selected when application is based on reflectivity properties (color and texture). Choosing a representation scheme, however, is only one part of the task of making the data useful for high-level processing algorithms. The next task is to describe the region based on the chosen representation. For example, in an external representation based on boundary of regions, the boundary can be described by features such as its length, the orientation of the straight line joining the extreme points, the number of concavities in the boundary, etc.  RECOGNITION (classification and estimation). Pattern recognition is the assignment of some sort of output value or label to a given input value or instance, according to some specific algorithm. Classification is one of tasks included in the pattern recognition field, which attempts to assign each input value to one of a given set of classes. Pattern recognition assumes that the image may contain one or more objects and that each object belongs to one of several predetermined types or classes.  SEMANTIC LEVEL. The main tasks at this level include possible joint analysis of recognition phase results from one or several cameras, other sensory and data base inputs, data fusion, event description and control signal generation. The algorithms interpret the object classification outputs and make decisions. As already mentioned in Chapter 1 (see Figure 5 and its associated wording), vision involves a chain of processing functions through which get progressively reduced. On the one hand, Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 66 representation, description, recognition and semantic level may be roughly grouped under the label of HIGH-LEVEL PROCESSING functions. These correspond to math-intensive tasks involving typically small data sets. On the other hand, image acquisition, pre-processing, feature extraction and detection/segmentation are roughly grouped as LOW-LEVEL PROCESSING. These are data-intensive tasks where the amount of data associated to the images is very large requiring a high data throughput in the communications and high computational power for applications demanding high resolution and high frame rate. The relevance of each processing function within the vision processing chain is largely application-dependent; and applications may range from relatively simple targets, such as counting bottles in a production line, to building robots that are to able comprehend the world around them based on the combination of vision and other senses. This applicationdependence has an obvious impact on implementation, both at the hardware and the software level. 2.4.3. HARDWARE-SOFTWARE CO-DESIGN Vision systems consist of hardware and software components. The value of system can be measured by some objectives that are specific to its application domain (performance design, manufacturing cost, ease of programmability) and it depends on both the hardware and the software components. Hardware/software co-design pursues meeting system-level objectives through the synergistic combination and concurrent design of hardware and software. Hardware and software must be designed together to guarantee proper operation under demanding power, cost and reliability constraints. Hardware/software system designers faces the same problems as ASIC designers: characterizing the components, understanding the operation of component networks and choosing a network topology based on the requirements. While a great deal of research has addressed methods for pure hardware designs and pure software designs, the increasing number of applications requiring embedding calls for new hardware/software design methodologies capable of meeting performance goals under SWaP and cost constraints. In general, different co-design problems are found and design approaches are developed depending on the classes of digital systems. To characterize these systems, several general criteria are considered, such as domain of application, degree of programmability and implementation features. DOMAIN OF APPLICATION. Digital systems can be classified according to their principal domain of applications. A digital system can provide a service as a self-contained unit, or as a part of a embedded system. Examples of self-contained digital systems are information processing systems, ranging from laptop computers to super-computers. Applications of embedded systems are present in the manufacturing industry (plant and robot control), consumer products (intelligent home devices), in vehicles (control and maintenance of cars, planes ships), in telecommunication applications, and in territorial and environmental defense systems. Digital systems can be geographically distributed (telephone network), locally distributed (aircraft control with different processing units on a local area network), or lumped (workstations). DEGREE OF PROGRAMMABILITY. Most digital systems contain hardware which is programmed by some software to perform the desired functions. Hence the abstraction level used for programming models is the means of interaction between hardware and software. There are two key issues related to programmability level:  Levels of programming: Digital systems can be programmed at different levels: application, instruction and hardware levels. The highest abstraction level is the application level, where the system runs dedicated software programs that allow the user to specify determined functionality options using a specialized language. Common components in digital systems are microprocessors, microcontrollers and programmable DSPs, which have an instruction set architecture. The instruction set defines the boundary between hardware and software by providing a programming model of the hardware. Instruction-level programming is achieved by executing on the Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 67 hardware the instructions supported by the architecture. In some applications domains, such as data processing for telecommunications and image processing, it has been demonstrated that replacing standard processors by ASIPs is practical. ASIP are instruction-level programmable processors with an architecture tuned to a specific application [Goos97]. In an ASIP design, the instruction set and hardware structure are chosen to support efficiently the instruction mix of the embedded software for specific application. Unfortunately, this special instruction set implies developing applicationspecific compilers. Hardware-level programming means configuring the hardware, after manufacturing, in a desired way. An example is microprogramming, where the behaviour of the control unit is determined by a micro-program, which is stored in a memory. Today microprogramming is common for DSPs, but not for general purpose microprocessors using RISC architectures, mainly due to performance reasons [Henn90]. Reconfigurable circuits, whose main representative are FPGAs, are the most significant example of hardware-level programming.  Access to programming: This item defines who has access to what level of programmability. The end-user programming is often limited to application-level programming. An application developer relies on the programming language tools, operating system and the high-level programming environment for application development. A system integrator uses the system programmability in order to ensure compatibility of system components with market standards.  Implementation features: System implementation deals with circuit design style, manufacturing technology and integration level. The circuit design style relates to the selection of circuit primitives, clocking strategy and circuit operation mode. A system may have components with different scale of integration and different fabrication technologies. The choice of hardware technology for the system components affects the overall performance and cost. System-level field programmability can be achieved by storing programs in memories and exploiting programmable interconnection. The architectural and performance characteristics associated to the components comprising a system determined the strategy to develop in the Hardware/Software codesign. 2.4.4. DESIGN OF HARDWARE/SOFTWARE SYSTEMS The design of Hardware/software systems involves modeling, validation and implementation. Modeling is the process of conceptualizing and refining the specifications, producing a hardware and software model. Validation is the process of achieving a reasonable level of confidence. And implementation is the physical realization of hardware through synthesis and of executable software through compilation. The modeling style can be homogeneous or heterogeneous. In the first case, a modeling language (C programming language) or a graphical formalism is used to represent both the hardware and software portions. In the second case, the hardware/software partition is often outlined by the model itself, because hardware and software components can be described in the corresponding languages. Hardware/software system design is composed by four main tasks [Wolf92]:  Partitioning the function to be implemented into smaller, interacting pieces. The partition of a system into hardware and software is critical because it has a first order impact on the cost-performance characteristics of the final design. Given a specific architecture, partitioning of a system-level functional description results in a labelling of its tasks as hardware or software operations. Partitioning can be decided by the designer, with a successive refinement and annotation of the initial model, or determined by CAD tools. Several approaches to partitioning task have been developed [Barr94] [Agra95] [Bend96] [Vahi94] to improve system performance while executing specific algorithms Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 68  Allocating those partitions to microprocessors or other hardware units, where the function may be implemented directly in hardware or in software running on a microprocessor.  Scheduling the times at which functions are executed. Scheduling can be defined as assigning an execution start time to each task in a set, where tasks are linked by some relations (dependencies, priorities, etc.). The tasks can be elementary which are denoted by operations (hardware operations or computer instructions) or can be an ensemble of elementary operations which is denoted by process (software programs). Task execution may be subject to real time constraint or not. Tasks execution requires the use of resources, which can be limited in number, thus causing the serialization of some task execution. The scheduling tasks are present at different abstraction levels during the design phase: operation scheduling in hardware level [Knap96], instruction scheduling in compilers [Aho88] and process scheduling in operating systems [Rama94].  Mapping a generic functional description into an implementation on a particular set of components, either as software suitable for a given microprocessor or logic which can be implemented from given hardware libraries. The design goals in each task depend on the application (performance, manufacturing cost, testability, etc.). The solutions to these problems clearly interact, the available choices for scheduling are controlled by how the design was partitioned, so on. Furthermore, not only can each of these steps be applied to the software and hardware components separately, but also to the division into hardware and software components itself, and the design decisions made for the hardware and software components separately interact with the co-design problem. 2.4.5. VISION SYSTEMS ARCHITECTURES The organization or architecture of a vision system is highly application-dependent. Some are stand-alone systems which solve a specific measurement or detection problem, while others are sub-system of a larger entity that include other sub-systems for control of mechanical actuators, planning, information databases, man-machine interfaces, etc. The architecture of a vision system also depends on whether its functionality is pre-defined or it can be modified during operation. In general, a standard vision system includes a camera or several cameras, a data processor responsible for carrying out the image processing algorithms, a system controller responsible for the configuration and control of system, memories to store images and programs, communication ports, a display unit and libraries of image processing routines included in a software environment conceived for writing, compiling, executing and debugging Vision Applications. Figure 33 is a schematic representation of a vision system. Processing library Escene Camera PC Display Memories Communication ports Operator console Image Processing Data-Intensive Processing Math-Intensive Processing Data bus Control bus Controller (µP) Application Development Environment Figure 33. VISION SYSTEM DIAGRAM. Physical vision system architectures can be roughly classified in the following groups:  PC-based vision systems  Embedded vision systems  Network based vision systems Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 69  Hybrid vision systems PC-BASED VISION SYSTEMS are straightforward, flexible and robust, although bulky and slow unless high-performance processing are employed. This type of system relies on the PC’s CPU to perform image analysis, feature extraction and pattern recognition tasks. The availability of various vision processing libraries for PC platforms makes this approach very popular and very well suited for fast application prototyping [MATROX]. PCs also provide a more flexible environment for building user interfaces. Processing library Escene Camera PC Display Operator console Frame Grabber Application Development Environment Figure 34. PC-BASED VISION SYSTEM. PC-based vision systems include usually a camera connected to a PC through either a frame grabber or a communication port such as USB, Firewire, CameraLink or Ethernet, and usually an Application Development Environment (ADE), see Figure 34. Historically, frame grabbers have been the predominant interface. But, trends have changed in recent years; direct camera connections via USB, CameraLink or Ethernet interfaces have became practical and will be prevalent in the near future in order to reduce the system cost. Sometimes a camera is connected to a PCI (Peripheral Component Interconnect) processing board within a PC. In this case, the PCI board may perform most of the ASIP and output generation, while the PC provides a flexible operator interface or additional processing power. This kind of system is a special case of a compact vision system and a PC-based system. Compact vision systems are usually composed by a camera connected to an external embedded processing unit in a separate box to provide ASIP and communication/networking functionality. An EMBEDDED VISION SYSTEM integrates image capture, all processing required by the task to perform by the vision system and application specific output generation into a single device casing – Figure 35 depicts the architecture of embedded vision system. While data-intensive tasks require high speed hardware to deal with high resolution and high frame rate, mathintensive tasks often require high performance processors to deal with issues such as pipelining and floating-point arithmetic. For demanding applications, hardware architecture of embedded vision systems can be based on a multiple-processor platform, with one or more processors capable of implementing parallel processing (FPGA) performing data-intensive tasks, and a DSP or RISC (Reduced Instruction Set Computer) performing math-intensive tasks. Figure 35 depicts the architecture of embedded vision system. Smart cameras are actually a particular type of embedded vision systems. Smart cameras offer the benefit of interacting directly with automation equipment via its discrete I/Os and built-in industrial protocols, like EtherNet. Unlike a PC-based vision system, which requires a separate discrete I/O card and the integration of third-party software to support these industrial protocols, a smart camera can support these devices. This smart camera concept as an embedded vision system has advantage over PC- based architecture, more compact volume and achieves lower cost and power consumption. This last parameter is essential for stand-alone applications. But, smart cameras have strengths and weaknesses like PC-based vision systems. Historically, a smart camera has been limited in terms of its processing capabilities. It does not offer the computing power of a traditional PC- based vision system because it cannot dissipate the amount of heat that is generated by high performance but power-hungry CPUs. Usually, a smart camera offers a software environment that does not require traditional programming or coding. Typically, the smart camera is designed to perform a specific task. Because the majority of smart cameras are configured, they are wellsuited to users with little or no traditional programming experience. The disadvantage to this is that a user cannot work outside the mold; his application must match what the smart camera supports. However, applications that require complex logic in terms of numerous decision Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 70 points, alternate path of execution and fallback mechanism, as well as very specialized algorithms, are better suited to traditional programming, which is the realm of PC-based systems but also available with some smart cameras. Escene Camera Memories Communication ports Image Processing Data-Intensive Processing Math-Intensive Processing Data bus Control bus Controller (µP) VISION SYSTEM Figure 35. EMBEDDED VISION SYSTEM. Compared with PC-based systems, embedded vision systems are subject to hard constrains from the point of view of design, implementation and production. Examples of typical restrictions are low power consumption, limited resources, real-time processing and low cost. These constrains proceed from the fact that embedded systems are usually considered for the implementation of products with a high level of compactness. Currently, embedded vision systems have progressed fueled by the availability of: powerful processors and memories with large compactness level and high operation speed, real-time operating systems, development of algorithms more suitable for computational implementation and the improvement of system development software and tools, among other factors. Traditionally, embedded vision systems are designed for custom tasks. Hence, their hardware can be tailored to implement only those indispensable algorithms with minimum power and cost. The downside to this approach is that the user application must match what the embedded system support. However, by incorporating smart dose of programmability and configurability, for instance, by using a micro-processor for system control and including configurability in the operation of ASIP hardware, users can adapt the system for a variety of applications using an application development environment similar to PC-based system. But obviously, there is a trade-off between compactness-power consumption-cost and versatility. In general, vision systems are usually embedded into larger systems executing monitoringcontrol functions and performing information-processing functions. In many practical applications embedded control systems regulate mechanical components based on image analysis. Hence, in the case of real-time applications, image analysis must be completed within prescribed time windows for proper operation [Shin94]. Since vision systems can be visualized as embedded systems working in real-time, their design methodology is similar to that followed in the development of embedded systems used in other application fields [Wolf92]. Indeed, many embedded systems are implemented as distributed systems, where hardware/software partition represents a physical partition of system functionality into application-specific hardware and software executed on one or more processors with inter-processor communication links between them. In the case of vision systems, the application-specific hardware performs the data-intensive tasks belonging to the early-processing stage. This specific hardware is implemented by custom circuits (ASICs), reconfigurable circuits (FPGA) or DSP’s. The system control and high-level image processing will be performed by one or several processors (microcontrollers and microprocessors). For example, in the Eye-RIS system proposed in this Thesis, the application-specific hardware is composed by the Q-Eye and a digital image processor, both controlled by a dedicated controller. And the software is executed by the microprocessor NIOSII. While detailed explanations are provided in Chapter 5, at this point it is convenient to copy the Eye-RIS block diagram for easier reading – see Figure 36. These embedded systems can be characterized by a co-processing architecture, where a micro-processor works in conjunction with dedicated hardware to deliver a specific application. The particular implementation of a co-processing architecture depends on the level of parallelism supported between hardware and software components. The co-processing Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 71 hardware can operate under direct control of the microprocessor, which stalls while the dedicated hardware is operational. On the other hand, the co-processing hardware can operate concurrently with software execution. Image Acquisition In-pixel Image Processing In-pixel Image Memory SIS Q-Eye Sensorprocessor Image post-processing, application control and decision making On-chip data and program memory Nios II RISC Microprocessor Eye-RIS VSoC On-chip image memory I/O and Communications Eye-RIS v2.1 Vision System External image memory External program and data memory Serial Flash Memory I/O Interface Q-eye Controller DIP Figure 36. BLOCK DIAGRAM OF EYE-RIS_V2. These embedded systems can be characterized by a co-processing architecture, where a micro-processor works in conjunction with dedicated hardware to deliver a specific application. The particular implementation of a co-processing architecture depends on the level of parallelism supported between hardware and software components. The co-processing hardware can operate under direct control of the microprocessor, which stalls while the dedicated hardware is operational. On the other hand, the co-processing hardware can operate concurrently with software execution. Other factors which affects the partitioning formulation is the number of processors considered for the target architecture and the hardware/software interface. According to scheduling tasks, timing constraints are common for hardware circuits and software applications in embedded control systems in which the vision system can be classified. During concurrent operation of hardware and software portions, there is an important issue related to scheduling, the synchronization of several operations and processes. Synchronization is needed when some delay is unknown in the model. Relative scheduling is an extended scheduling method to work with operations with unbounded delays [Ku92]. When considering compilation for general-purpose microprocessors, instruction selection and register allocation are often achieved by dynamic programming algorithms [Aho88]. But in embedded systems, the dedicated hardware is usually implemented by an ASIP, thus resulting a more complex compiler for this special processor because of irregular structures such as inhomogeneous registers sets and connections. For instance, the vision system Eye-RIS has required the development of a particular compiler associated to the microprocessor included in the Q-Eye controller in order to translate the specific programming language associated to the Focal Plane Processor (FPP) Q-Eye to the micro-instructions that Q-Eye controller will transmit to the Q-Eye system. The Q-Eye system together its controller and the digital image processor constitute the application-specific hardware included in the vision system Eye-RIS. In this case, the specific hardware thanks to the programming capability of Q-Eye has associated a set of instructions that give rise to the CFPP code, which is the language used for programming the CVIS Q-Eye. Once introduced the embedded vision systems, another type of vision systems will be presented, which belongs to a higher hierarchy level from an architectural point of view, the NETWORK BASED VISION SYSTEMS. They are composed by a network of cameras (usually smart USB or Firewire cameras) that acquire and processing images locally to be subsequently sent to a central computer (server) or processing hardware in order to process the received information and take a decision. Currently, this type of vision systems is widely implemented in Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 72 surveillance applications, see Figure 37. A complex video surveillance system with built-in local intelligent features extraction can be loosely considered as a network of smart cameras. Commonly, the video surveillance system is composed of several layers: camera layer, network layer, central computer layer and take-decision layer which consists of trained security personnel. Intelligent video surveillance systems carry out a re-distribution of ASIP tasks among the four layers, notably shifting processing load from security personnel to central computers and distributed smart cameras. The use of smart cameras reduces significantly the bandwidth problem caused by increment of cameras present in the system and their accuracy specs. Clearly, the transmission of raw pixels over the network is less efficient than sending the intermediate analysis results developed by the smart cameras. Camera layer Network layer Central computer layer (servers) Network layer Takedecision layer Figure 37. VIDEO SURVEILLANCE SYSTEM Finally, it is very common that some of the previous architectures have to be implemented jointly during the development of a real application, giving rise to the HYBRID VISION SYSTEM concept. 2.5. TYPICAL SPECS FOR CAMERA AND VISION SYSTEMS Cameras and vision systems constitute complex systems where several disciplines of engineering and science are involved in their development, as verified in previous sections. Regarding the development of products, real applications determine the specifications for the main parameters associated to the different sub-blocks which constitute the vision system or camera and the inputs for the different disciplines involved in the product design phase. Conceptually, a camera is a sub-block in vision systems. Thus, the camera specifications are included and can be derived from the vision system specs. For this reason, this section starts introducing the system level parameters specified by the applications which will be performed by the vision system and finishes with the camera and image sensor parameters to be considered during the design phase. 2.5.1. APPLICATION REQUIREMENTS, SYSTEM SPECIFICATIONS In order to define the product specifications, a detail study about the tasks which will be performed by the system and conditions under application runs identifies the key parameters to consider during the design phase and determines the requirements, fixing the product characteristics [Hombe]. These characteristics represent the specifications for the vision system sub-blocks. In general, the vision system characteristics or parameters that play an important role in vision applications can be classified in different types depending on disciplines involved. MECHANICAL characteristics: In embedded vision systems such as smart cameras, mechanical parameters that must be considered in relation to the case are size, weight, manufacturing materials, lens/image sensor alignment, etc. In general, the vision system is usually composed by several blocks (cameras and processors) depending on application. Each block has associated determined mechanical parameters indicated previously. But also not only the subblocks dimensions are important, there are vision systems where it is critical location of component such as cameras and illumination. As for the location of these components, the adjustment is important for installation, operation and maintenance. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 73 SPEED OR TIMING PERFORMANCES: The tasks developed by the vision system have to be finished within a specified time. The timing requirements determine the data throughput associated to the data path of vision system (including the data interface of camera) and the maximum operation frequency for digital processors and memory blocks, defining the maximum frame rate of vision system which is expressed in frame per seconds. Therefore, these requirements regarding processing time will influence the choice of the hardware platform and will eventually limit the possibility of using certain algorithms. Others important timing parameters associated to the image sensor operation are the minimum and maximum exposure times. These specifications are related with the lighting conditions. ACCURACY CHARACTERISTICS: There are four parameters that must be considered in relation with this matter: effective spatial resolution, measurement accuracy, camera resolution and optical performance of camera. The first two parameters are related with the accuracy required in real scenes and the other two corresponds to specs of imager. The effective spatial resolution or object resolution results from a direct mapping of real-world objects to the image sensor. It can be measured in millimeters per pixel. This spatial resolution depends on the camera resolution and field of view; the mapping is done by the lens. The measurement accuracy can be considered as the overall accuracy of the system. It is the smallest feature in a captured image that can be measured. Depending on the processing algorithm, the required measurement accuracy can determine different camera specs (spatial resolution and optical performances). This means that both contrast of features as processing algorithms decide when features are measurable. For instance, if the contrast of small defects is poor, the algorithm might not be able to detect a single pixel defect; then four or five pixels might be necessary. Certainly, the contrast of the feature is determined by the image quality (temporal and spatial noise associated to the image sensor operation). The required image quality defines the optical performances associated to the camera. Thus, the effective spatial resolution that is necessary in order to achieve determined measurement accuracy depends on the feature contrast (image quality) and the processing algorithms. SOFTWARE CHARACTERISTICS: The vision system has always several parameters and functionalities (exposure time, gain, processing tasks, control tasks, etc.) associated to the different sub-blocks of a vision system, which can be configured by the user through a PC in a general case. This means that a User or Programming Interface must be defined in order to configure and control the vision system. This Programming Interface depends on:  System architecture type (PC-based system, embedded system, network based systems, etc).  Parameters to be configured.  Communication interfaces considered for the vision system, which include a control port (UART, RS232, SPI, etc.) and an image interface (CameraLink, USB, GigE, CoaXpress, etc.).  Tasks to be performed by different blocks that comprise the system.  A generic Programming Interface will be used or a custom one will be designed. When vision systems allow users to define operations and processing algorithms which consist in a set of tasks to be performed by the hardware, the definition of libraries including basic operations and functions is very useful for the applications engineers. This fact provides a significant added value to the product. The user of vision systems can design new algorithms based on these libraries in a easy manner and short time periods. In order to facilitate the creation of new algorithms and applications an Application Development Environment must be designed over the Programming Interface. This ADE software has sense in general purpose vision system which has not designed to perform a specific application or task, where the user cannot modify the operations and processing algorithms implemented by the system. FUNCTIONAL CHARACTERISTICS: The applications to be performed by the vision system define the functions and operations required in each processing level (acquisition, pre-processing, feature extraction, segmentation, description, classification and semantic) in order to implement the corresponding algorithms. Functions related with the control (operation flow) and configuration Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 80 designed such that its non-linear characteristic will be negligible respect to the pixel SF nonlinearity. For monochrome images, the effects generated by a non-linear responsivity curve are not directly observable in capture images, when this non-linearity is an Integral Non Linearity (INL) type, as shown in Figure 39 where a quadratic compressive non-linearity error has been considered. In the case that the non-linearity error is a Differential Non Linearity (DNL), artifacts can be observed in sensed images, mainly in regions of an image where a gradual gradient in gray levels is present, because some gray levels disappear due to the Differential Non Linearity, as described in Figure 40, where digital codes have been lost due to the AD conversion process. Presence of DNL errors in the ADC converters of readout channels may be detected by analyzing the histogram of images similar to image shown in Figure 41. For instance, the error described in Figure 40 provokes missing codes in the histogram. 0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 0.016 0.018 0.02 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1Responsivity Characteristic uJ/cm2 Voltage non-linear linear Figure 39. INTEGRAL NON LINEARITY IN RESPONSIVITY CURVES. Figure 41 represents the artifact generated by the DNL error depicted in Figure 40. In a real scene consisting of a continuous degradation of light intensity, the DNL error generates an artificial edge. This issue is unacceptable for further processing levels in vision applications. 0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 0.016 0.018 0.02 0 500 1000 1500 2000 2500 3000 3500 4000 4500 Responsivity Characteristic Digital Number uJ/cm2 non-linear linear Figure 40. DIFFERENTIAL NON LINEARITY IN RESPONSIVITY CURVES. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 81 Linear Characteristic 500 1000 1500 2000 0.5 1 1.5 Non-Linear Characteristic 500 1000 1500 2000 0.5 1 1.5 Figure 41. ARTIFACT DUE TO DNL ERROR. For color images, both types of non-linearity generate artifacts. In this case, any type of nonlinearity in the responsivity curve affects to the color balance results. The color filter array deposited over the pixel array to implement the color image sensor affects to the responsivity associated to each type of color pixel (Red, Green or Blue). Each color filter has a different transmittance in function of wave length. This fact implies that each responsivity curve associated to each color has a different overall system gain. In case of a linear responsivity, the color balance has to apply different factors to obtain the real color for a particular pixel. In the presence of non-linearity errors, the correct color value cannot be obtained by multiplicative color constant factors. In this case, the multiplicative color factors depend on intensity level due to the non-linearity errors, complicating the implementation of color balance algorithm. As mentioned in previous paragraphs, the overall system gain has components associated to the pixel and components associated to the readout channel circuitry. Any deviation respect to the nominal value of these components due to the mismatch of physical devices provokes spatial noise. In this case, the spatial noise is proportional to the signal (Photo Response Non Uniformity). This spatial noise is a parameter related with the image quality, which will be explained in next section 2.5.5.2. In a similar way, the mismatch of non-linearity error of responsivity curve generates spatial noise in sensed images. The spatial noise is not proportional to the photo-generated signal in this case. Thus, the calibration-correction algorithm will be more complex. Usually, a linearization process is required for this correction. 2.5.5.2. IMAGE QUALITY PARAMETERS In section 3.3.4 of Chapter 3, a detailed description of non-idealities associated to the behavior of pixels is performed. In this case, the study is focused on the particular pixel of Q-eye system, but their conclusions can be extrapolated to the standard behavior of photosensors without loss of generality. The previous mentioned study described the sources of different errors or deviations respect to the ideal behavior, mainly the errors associated to the pixel (photo-diode, reset transistor and output Source Follower). In this section, the subject of study will be focused on their effects in captured image at camera system level and will present the parameters which characterize these noises. From the image quality point of view, the noise of a captured image has two components: TEMPORAL NOISE, which is composed by the contributions of pixel and readout channel. It can be lumped into three additive components:  Shot noise due to the photo-current ph i and dark-current dc i generated by the photodiode, whose Power Spectral Density (PSD) ,expressed in e2, is given by the equation:   2int 1 shot ph dc ph dc i i T n n q        Eq. 33 Where int T is the integration or exposure time and ph n , dc n are the electrons photogenerated and electrons generated in dark conditions respectively. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 82  Reset noise associated to the thermal noise of reset transistor included in the pixel. Its PSD is denoted by 2 reset  (expressed in e2).  The readout noise which is composed by the electronic noise of pixel source follower and readout channel. The corresponding PSD is denoted by 2 read  (expressed in e2). The total temporal noise power results from the addition of powers associated to each contribution: 2 2 2 2 temp shot reset read        Eq. 34 The temporal noise in dark conditions represents the smallest detectable input signal: 2 2 2 temp dark reset read     Eq. 35 Where the shot noise contribution of dark-current has been considered negligible. This temporal noise in dark conditions together with the effective full well capacity (nsat), which defines the largest non-saturating input signal, determines the Dynamic Range (DR) parameter. The dynamic range quantifies the ability of an image sensor to adequately represent both high lights and dark shadows in a scene. Mathematically it is defined by the equation: 10 20 log sat temp dark n DR    Eq. 36 Another important parameter to determine the image sensor quality is the Signal to Noise Ratio (SNR), which is defined as the ratio of the input signal power to the average input referred noise power:   2 10 2 2 2 10 log ph shot reset read n SNR      Eq. 37 The SNR parameter is a measurement of quality. As explained in section 2.5.1, the measurement accuracy required by applications defines the maximum noise level which allows developing correctly the processing algorithm without errors. This system specification determines the SNR necessary in order to implement a particular application. The SNR increases with the input signal ( ph n ), first at 20dB per decade for small input since read noise dominates, then at 10dB per decade when shot noise due to the photo-detector current dominates. The maximum SNR achievable by the image sensor is determined by the effective full well capacity ( sat n ): 2 max 10 10 22 10 log 10 log () sat sat sat dc reset read n SNR n nn         Eq. 38 Therefore, the effective full well capacity ( sat n ) is involved in the definition of dynamic range and the maximum quality achievable by the sensor. For a given output range of readout channel, there is a tradeoff between responsivity and effective full well capacity, what means that there is a tradeoff between responsivity and achievable image quality ( DR and max SNR ). Regarding this trade-off between responsivity and image quality, the pixel conversion gain is other parameter that contributes to this trade-off. The PSD of reset noise (read section 3.3.4.2 of Chapter 3) is given by: 22 () reset FD kT V C    Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 83   22 2 reset FD kTCe q    Eq. 39 Then, considering the power spectral density of reset noise expressed in electrons in the floating diffusion (FD) node, the reset noise is proportional to the capacitance associated with this FD node. In order to reduce the noise contribution associated to the reset transistor, the FD capacitance should be incremented. But, this increment means a larger conversion gain (Eq.30). An increment in the conversion gain implies a reduction in the full well capacity for a given output range of readout channel. Depending on the readout channel architecture implemented in the camera system, the temporal noise and settling errors associated to the analogue references used in the readout process could generate artifacts in capture images such as line noise or flickering between frames in a video sequence. For instance, in a column ADC approach for Digital Image Sensor, the noise in a global reference common to all converters can generate a line noise as shown in Figure 42. 2-dimensional temporal noise 200 400 600 800 1000 200 400 600 800 1000 2-dimensional temporal noise and line noise 200 400 600 800 1000 200 400 600 800 1000 Figure 42. TEMPORAL LINE NOISE EFFECT. If this temporal line noise is an offset, then it can be compensated when Optical Black columns are implemented in the pixel array. The corresponding offset error is calculated averaging the Optical Black columns in order to subtract the mean value from the corresponding active pixels. During the design phase the PSD of this line noise must be considered and compared with the expected two-dimensional temporal noise. The DR and SNR features can be improved implementing a true Correlated Double Sampling. During this operation, the output of pixel is sampled twice: once after reset and the second time with the photo-generated signal. The sample resulting from the first sampling phase is given by: '' 1reset read n n n Eq. 40 The sample obtained in the second sampling phase is: 2ph dc shot reset read n n n n n n     Eq. 41 Thus, the difference of samples results:     '' 21 ph dc shot reset reset read read n n n n n n n n n        Eq. 42 In the case of true CDS ( ' reset reset nn ), the PSD associated to the operation is given by the equation: 21 2 2 2 2 n n shot read       Eq. 43 When true CDS operation cannot be carried out, for example, a 4T pixel topology operating in global shutter, the resulting temporal noise is: Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 84 21 2 2 2 2 22 n n shot reset read          Eq. 44 The spatial noise has not been included in the definition of DR and SNR because the spatial noise is unlikely that it is corrected. So, the fundamental limits for DR and SNR are given by the temporal noise. In the case that the spatial noise or Fixed Pattern Noise is considered, the DR and SNR will be given by the equations: 10 20 log sat temp dark FPN n DR     Eq. 45   2 10 2 2 2 10 log ph shot reset read FPN n SNR         Eq. 46 FIXED PATTERN NOISE (FPN) characterizes the spatial variation in pixel outputs under uniform illumination conditions due to device and interconnect mismatches, which provokes different transfer functions or responsivities for each pixel. The errors associated to the static characteristic (responsivity) have a global component equal for all pixels and readout channels and a random component due to this mismatch of device behavior. The last one generates FPN on the sensed image, because under uniform illumination the resulting image presents spatial variations. This spatial noise remains superimposed on the image in a video sequence. Mainly there are two FPN components:  Offset (Dark Signal Non Uniformity, DSNU): Generated by the random component of photodiode dark current, the different offset contributions associated to the readout channels when more than one channel is implemented in the image sensor for download process, mismatch of settling errors during the pixel reading, etc. Under dark conditions, the captured image presents spatial noise due to this mismatch offset associated to the pixel operation and readout channel. The offset error resulting from pixel operation is distributed randomly over the image, and this is the component which is denoted by DSNU. Usually, the offset error associated to the readout channels generates structures or artefacts in the image, which are unacceptable for further processing. These structures depend on the readout architecture. For instance, for only one readout channel, the error offset of circuitry is a global contribution to the image. But, when two or more channels are used to download an image Vertical Fixed Patter Noise (VFPN) appears due to the error offset associated to each channel. This is a typical parameter for column ADC approach in Digital Image Sensors. There are readout architectures which comprises several channels per column to download several rows in parallel. This is a usual topology for high resolution and high frame rate image sensors. In this case, a Horizontal Fixed Pattern Noise (HFPN) could be observed in capture images. Therefore, the spatial noise in dark conditions is given by the equation: 2 2 2 2 _ _ _spatial offset DSNU VFPN offset HFPN offset        Eq. 47 The VFPN and HFPN are unacceptable in vision applications, in case that the image sensor has this type of spatial noise calibration/correction procedures must be implemented at camera level. The correction coefficients will be additive values to compensate channels offsets. Under dark conditions, if a unique readout channel is considered, only the random offset contribution associated to the pixel array operation can be observed in captured images, Figure 43 shows an example. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 85 Image in dark conditons 200 400 600 800 1000 200 400 600 800 1000 -100 0 100 200 300 0 1 2 3 4 5x 104Histogram Figure 43. DSNU EFFECT. Calculating the histogram, the captured image has a global offset due to the systematic component of dark current and readout circuitry offset. And the random distribution is provoked by the mismatch of dark current among pixels. Assuming a column ADC approach for readout architecture, the offset contribution of column circuitry introduced VFPN in addition to the random offset associated to the pixel array, see Figure 44. DSNU & VFPN 100 200 300 400 500 600 700 800 900 1000 100 200 300 400 500 600 700 800 900 1000 Figure 44. VERTICAL FPN EFFECT. Depending on the VFPN magnitude, the vertical line can be detected or not. In Figure 45, several PSDs for VFPN are shown considering always the same 2-dimensional random spatial noise (DSNU). stdvfpn=stddsnu 200 400 600 800 1000 200 400 600 800 1000 stdvfpn=stddsnu/2 200 400 600 800 1000 200 400 600 800 1000 stdvfpn=stddsnu/4 200 400 600 800 1000 200 400 600 800 1000 stdvfpn=stddsnu/8 200 400 600 800 1000 200 400 600 800 1000 Figure 45. PSD OF VFPN VERSUS PSD OF DSNU. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 86  Gain (Photo Response Non Uniformity, PRNU): Generated by the mismatch of pixel conversion gain, which generates a 2-dimensional random spatial noise, the mismatch of analogue gain applied by the analogue circuits of readout channels, when two o more channels are implemented, the settling errors during the reading process, etc. Usually, the random noise introduced by the pixel array is denoted by PRNU. The gain error introduced by the readout channels has a systematic component which introduces a global effect and a random component due to the device mismatch which generates VFPN and HFPN in a similar way than offset component. But in this case, the magnitude of gain error depends on input signal, being null under dark conditions and maximum when maximum signal is reached. For a particular signal level the spatial noise due to the gain errors is given by the expression: 2 2 2 2 _ _ _spatial gain DSNU VFPN gain HFPN gain        Eq. 48 Therefore, the total spatial noise power is composed by the two error contributions: 2 2 2 __FPN spatial offset spatial gain     Eq. 49 The gain VFPN and HFPN are unacceptable for vision applications too. Then, calibration/correction algorithms must be developed when the image sensor presents this type of spatial errors. Considering a linear transfer function of image sensor, the correction coefficients consist in multiplicative factors. In addition to the temporal and spatial noises, there are other important parameters which determined the image quality required by a particular application. One of these parameters is the Modulation Transfer Function. The MTF parameter depends on several factors related with the pixel, such as pixel size, metallization, presence or absence of micro-lens, etc. The meaning of this parameter has been introduced in section 2.5.3 [Naka08]. Other important characteristic associated to the image sensor is the parameter denoted by SHUTTER EFFICIENCY, mainly when the image sensor operates in Global Shutter. This parameter is related with the fact that the diffusions associated to the integration node (Floating Diffusion, FD) constitute a parasitic photo-detector. In Global Shutter, the photo-generated carriers are stored in the FD capacitance during the frame reading process. In a row by row download scheme, this means that the charge stored in the last read rows is more exposed to the operation of parasitic photodiode than the first downloaded rows. Figure 46 depicts the degradation suffered by a uniform image due to this phenomenon. The degradation is caused both by the dark current associated with this parasitic photodiode as its sensing operation collecting photo-generated charge into the FD capacitance. This latter component is predominant. Thus, the Shutter Efficiency parameter is defined as the ratio between the photodiode responsivity and the Floating Diffusion responsivity. With the aim of increasing the shutter efficiency, several strategies are considered at architectural level (introduction of analogue memories in pixel) and physical level (doping profiles, metallization, micro-lens, etc.). 200 400 600 800 1000 100 200 300 400 500 600 700 800 900 1000 2000 2020 2040 2060 2080 2100 2120 2140 2160 2180 Figure 46. PARASITIC SENSING BY FD DURING READING PROCESS IN GLOBAL SHUTTER. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 87 Finally, there is a phenomenon which degrades image quality of pictures acquired in consecutive captures. It is called image lag. Lag occurs when residual charges remains in the pixel during the next consecutive exposed frame and may corrupt the image (especially images of bright objects in low-light scenes). Residual charges may come due an incomplete transfer in the photodiode or dynamic errors during the reset phase of integration node [Naka08]. To conclude this section, Table.1 summarizes the main parameters to consider when the quality of a camera or image sensor must be determined. Parameters Units Array resolution pixel x pixel Pixel size µm Conversion bits (word length) Responsivity Full Well Capacity e Pixel Conversion Gain µV/e Readout Analogue Gain Fill Factor x Quantum Efficiency % Responsivity V/(µJ/cm2) Non-Linearity % Full Scale Noise Dark conditions DR dB Temporal line noise %FS DSNU %FS VFPNoffset %FS HFPNoffset %FS With illumination SNR dB PRNU %FS VFPNgain %FS HFPNgain %FS Dark Current e/sec Modulation Transfer Function Shutter Efficiency Image Lag Table.1. Camera image quality. 2.6. PREDECESSORS OF THE Q-EYE In this section, several Analog Parallel Array Processors (APAP) will be introduced which have been designed in order to carry out image processing efficiently with outstanding computing power figures when compared to conventional serial digital signal processors. The neural networks (NN) are parallel processing architectures. These neural networks are composed by a set of processing units interconnected, called neuron. Each unit processes the weighted inputs which come from the other neurons belong to the network. Also, the output unit is transmitted to the rest of network being the input to others neurons. Different types of neural networks can be defined depending on type of processing units implemented in the neural network and the interconnection among units to generate the processing network: feed-forward neural network, recurrent neural network, fully-connected neural network, partially connected neural network, etc. An interesting recurrent neural network (RNN) from an image processing point of view is the Cellular Neural Network (CNN) defined by professors L.O. Chua and L. Yang in 1988 [Chua88]. The CNN is a new class of analogue dynamic array processors, where the elementary processing units are locally connected. This means that they interact directly only with those processing units located within a finite local neighborhood. The interest of CNNs is based on the fact that many functions related to the processing of multi-dimensional analogue signals, such as images, can be formulated as operations between each individual (one-dimensional) signal value (pixel), and the values placed within a local receptive field (neighborhood), directly mappable onto CNNs. 2.6.1. THE CNN PARADIGM The Cellular Neural Network [Chua93] is an array of non-linear dynamic processing units, called cells, which occupy the nodes of a n-dimensional grid. These cells satisfy two properties: Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 88 interactions between them are restricted to a local scope and state variables are continuousvalued signals. Due to the images are two-dimensional variables, from now, a two-dimensional array composed by MxN processing cells will be considered, without loss of generality because definitions and results are easily extended to higher dimensional networks. A cell of processing array will be denoted by   ,C i j , where   1,iM represents the row and   1,jN the column. The   ,C i j neighborhood of r order is defined by the following equation:         , , /max , ,1 1 r N i j C k l k i l j r k M l N        Eq. 50 where r is a positive integer. Figure 47 represents a first order neighborhood, where each cell of the array interacts directly only to those cells of the grid located within a finite radius 1r . C(1,1) C(1,2) C(1,3) C(2,1) C(2,2) C(2,3) C(3,1) C(3,2) C(3,3) Figure 47. FIRST ORDER NEIGHBORHOOD. The topology of CNN becomes univocally defined by the grid shape (rectangular, hexagonal, etc.), the neighborhood order and a set of boundary conditions which are defined by the set of cells situated less than r cells away from the border of the network. Without loss of generality, the shape of grid considered in the study has been rectangular. Each cell has associated three variables:  The state variable, denoted by   ij xt , represents the cell state and has information about the cell energy.  The output variable, denoted by   ij yt , is the cell output. It is obtained from the state variable through a non-linear transformation f :     ij ij y t f x t   Eq. 51 Figure 48 describes several output functions.  The input variable, denoted by   ij ut , represents the external excitation. f(xij) xij 1 1 -1 -1 f(xij) xij 1 1 -1 -1 f(xij) xij 1 1 -1 -1 Figure 48. OUTPUT FUNCTION. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 89 For a given initial state of cells represented by a vector of initial state     00 ij xx , an external excitation described by an input vector   ij uu and particular boundary conditions, the processing array evolves following a determined dynamic and the outcome is given by an output vector   ij yy . These vectors for a two-dimensional processing array adopt the form of matrix. The dynamic of CNN cell is described mathematically by the ordinary differential equation in continuous time:                                           ,, ,, ,, ,, , , ,, rr ij ij ij ij kl i k j l k r l r ij kl ij kl i k j l i k j l ij kl i k j l i k j l i k j l dx t g x t z A y dt B u C x D u x y                                       Eq. 52  is the time constant associated to the processing array, g is the dissipation function (Figure 49) represents several examples of dissipation functions), ij z is an offset term for each processing cell and ,ij kl A , ,ij kl B , ,ij kl C and ,ij kl D are the interaction operators. These interaction operators are non-linear in the more general case, but many processing tasks require only linear interaction among cells, thus receiving the name of interconnection weights. Moreover, ,ij kl C and ,ij kl D operators are null for many applications. In most practical cases, the nonlinearity appears only at the scaling factor of an otherwise linear transformation. g(xij) xij g(xij) xij g(xij) xij (a) (b) (c) Figure 49. DISIPATION FUNCTIONS. The general model of CNN has been presented. Several particular cases based on the general model have been developed in order to implement many image processing algorithms. 2.6.2. LEON O. CHUA AND LIN YANG MODEL The original model of CNN presented by Chua and Yang has a constant neighborhood order (neighborhood radius r is constant) and the interconnection weights do not depend on cell position in the array. These characteristics provide the property of translational invariance. The local interconnection, regularity and translational invariance are features very convenient for a VLSI implementation. The dissipation function considered in the Chua-Yang model is linear, option (a) in Figure 49. The interconnection operators ,ij kl A and ,ij kl B are linear, while the operators ,ij kl C and ,ij kl D are nulls. Under these considerations the differential equation which describes the behavior of processing cell is given by: Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 96 Each processing block of cell belongs to one of the two different CNN layers of the network. These CNN cores are coupled by some inter-layer weights and their outputs are combined by a third additional layer embodying analog arithmetic. This scheme constitutes a 2nd-order 3-layer CNN, whose conceptual diagram is represented in Figure 57. The evolution law of each cell, ( , )C i j , following this 2nd-order 3-layer CNN model is given by these two coupled differential equation:               11 11 11 11 1, 1 1, 11, 11,00 1, 12 2, 1, 1, 2, 2 2, 22, 22,00 2, 21 1, 2, 2, rr ij ij kl ij ij ij i k j l k r l r rr ij ij kl ij ij ij i k j l k r l r dx t g x t a y b u a y z dt dx t g x t a y b u a y z dt                                   Eq. 63 Where the non-linear losses term and the output function in each layer are those of the FSR CNN model, described in section 2.6.3 . Figure 58 depicts the circuital diagram of 2nd order CNN model described by Eq.63. Programming different dynamics in this CNN is possible by adjusting the template elements and the time constants associated to each layer. CNN Layer 1 CNN Layer 2 a12·y1a21·y2 Arithmetic Layer 3 y1y2 b1·u1b2·u2 z1 z2 a11·y1 a22·y2 w1·y1+w2·y2 output Figure 57. CONCEPTUAL DIAGRAM OF THE 2ND ORDER CNN. + z1,ji u1,ij From neighbours 1 1  ʃ g( ) f( ) b11,00 a11,00 a11,(-1,-1) a11,(-1,0) a11,(-1,0) a21 + z2,ji u2,ij From neighbours 2 1  ʃ g( ) f( ) b22,00 a22,00 a22,(-1,-1) a22,(-1,0) a22,(-1,0) a12 LAYER 1 LAYER 2 x1,ij y1,ij x2,ij y2,ij Figure 58. CIRCUITAL DIAGRAM OF CACE PROCESSING BLOCKS. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 97 2.6.6.3. PROCESSING CIRCUITRY: CNN LAYER AND SYNAPSES In this section, the physical implementation of CNN processing layer used in ACE and CACE chips is presented. This processing circuitry is composed of four main building blocks: an input block based on a class II current conveyor which collects the current contributions from neighboring cells, a current memory used to store and subtract an offset term generated by the chosen synapse structure (based on S3I regulated cascade current memory), a non-linear block which processes the input current (based on voltage comparators and switches) and finally the synapse which is a four quadrant analog multiplier [Espe94b][Domi98]. Figure 59 depicts a block diagram of this processing core. One of most important blocks in CNN cell is the synaptic block, because the majority of the circuitry structures employed in the cell are strongly dependent on its implementation [Carm01]. The ideal characteristic associated to the synapse is described by the following equation: o w x I k V V   Eq. 64 Where w V and x V represent the weight and cell state signals, o I is the output current and k is a constant with dimensions [A/V2]. If the cell state and the weight signals are represented by voltages, they can be easily conveyed to any high-impedance node by a simple wire. On the other side, the output of the multiplier should be represented by a current. In this way, the contributions of all neighbors can be summed at the input of the processing core wiring all these contributions concurrently to a low-impedance node. Vz,(i,j) From neighbours Current Memory (S3I) Vw,z Vw,(0,0) Vw,(-1,-1) Vw,(-1,0) Vw,(-1,0) Vx,(i,j) Current Conveyor (CCII) x yz Non-linear Block Vwo C Cin To neighbours     , ,, rr o w z z w k l x k l k r l r I k V V k V V          o I Figure 59. ANALOGUE PROCESSING CORE. There are several CMOS circuits for implementing a multiplier [Espe94b] [Gilb68] [Socl85]. Among them, there is one approach which is based on a single MOS transistor operating in the ohmic region. Using one transistor reduces the area requirements and a better relation between bias power and signal power is obtained (in the saturation region the information is carried by a small fraction of the actual currents flowing through the devices). Figure 60 describes a PMOS transistor operating in ohmic region. A p-type channel has been considered because its higher channel resistance allows smaller current, and so power consumption, for the same transistor length. + -VG + -VA + -VL G A L Figure 60. SINGLE MOS MULTIPLIER. Considering: G xo x A wo w L wo V V V V V V VV    Eq. 65 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 98 Being 0x V and 0w V bias references to guarantee that the transistor operates in ohmic region, the output current is given by the equations:   ' 2 w o w x w xo T wo w x offset w V I V V V V V V V V I V                      Eq. 66 ' o ox W CL     Eq. 67       0 ' 0 T p B DD wo w B w wo T T p B DD wo B w wo V V V V V V VV V V V V                       Eq. 68 The correct operation of single-MOS multiplier requires a virtual reference at node L ( L V ), this low-impedance node is implemented by the current conveyor, whose circuital realization is depicted in Figure 61. + -A Ibias Vw0 Iin Io=Ibias+Iin Figure 61. CURRENT CONVEYOR CIRCUIT. The offset term   offset w IV is stored in the current memory to be subtracted later. In order to achieve the necessary accuracy levels, a S3I current memory is employed [Toum93]. Its corresponding circuit is shown in Figure 62. C1C2C3 Φ1Φ2Φ3 Iin I1I2I3 Timing diagram V t Φ1 Φ2 Φ3 Figure 62. S3I CURRENT MEMORY CIRCUIT. Therefore, the CNN operation requires an offset calibration step. For this purpose, before CNN operation, but after new weights has been uploaded, all synapses are reset to 0 x V ( 0Gx VV ). Then the resulting current, which is the sum of the offset currents of all synapses concurrently connected to the same node, is memorized. This value will be subtracted on-line from the input current when the CNN loop is closed to run CNN dynamic. Associated to the non-linear block, there are two operation modes: soft limiting mode and hard limiting mode. In soft limiting mode, the total input current, after flowing through the current conveyor, is directed towards an electrical node in which a non-linear resistor is in charge of keeping the Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 99 voltage value within some limits. Except for this resistor, the node impedance is purely capacitive. The ideal characteristic of this non-linear resistor is given by:   0 x xL x xL x xH x xH VV g V V V V VV           Eq. 69 The total output current flowing into the capacitive node with the non-linear resistor produces the CNN dynamic process when the loop is closed, which is described by the equation:       , ,, rr xx w z z w k l x k l k r l r dV C g V k V V k V V dt             Eq. 70 The soft limiting mode is intended to be used in applications with gray-scale images. On the other hand, the hard limiting mode is intended for those applications providing a binary result. In this case, the total output current is sent to the capacitive input of a voltage comparator. The comparator operation is described by: min 0 0 max 0 0 () 0 x in x x in x x in x V V V V g V V V V V        Eq. 71 The dynamic evolution is modeled by the equation:       0, ,, rr in x in w z z w k l x k l k r l r d V V C k V V k V V dt            Eq. 72 In conclusion, running spatial-temporal dynamics in these networks requires following several initialization and calibration steps. First of all, acquisition of the input image and auxiliary masks and patterns. After that, the analog instruction constituted by the set of synaptic weights needed for a specific operation, is selected and transmitted to all cells in the array. Then, the offsets of critical Operational amplifiers are calibrated. After this, the time-invariant offsets of the synaptic blocks are computed and stored in the current memories. At this state, network is almost ready to operate. Then, the state capacitors and the feedforward synapses are initialized by means of the appropriate switch configuration, and the network evolution is run by closing the feedback Loop in each processing element (according to the current freezing mask). Finally, before stopping the network, the final state is stored in a LAM for further operations. 2.6.7. LIMITATIONS OF THE ACE AND CACE CHIPS As mentioned in previous sections, in-pixel processing approach there is a tradeoff between accuracy and pixel area, which affects directly to the achievable fill factor. For a given pixel pitch, the area devoted to processing circuitry is removed from sensing area. On the other hand, the integrated circuit fabrication process is exposed to a number of factors that can provoke the presence of defects on the chip. This phenomenon reduces the production yield and is characterized by the density of defects. This defect density increases with the die area, increasing the risk of suffering from fatal defects in fabrication, and thus, obtaining a smaller production yield. Therefore, yield limits the spatial resolution of processing array that can be integrated in a chip. From an implementation point of view, another factor that limits the resolution of processing array is the power consumption per cell. For instance, in the case of ACE and CACE chips, at least one synapse is associated to each contribution of neighboring pixels, plus the contribution of the pixel itself and the bias term. Therefore, each processing cell contains ten synapses at least. Each synapse generates a current contribution which is summed and integrated by the processing circuitry. This means that additional circuitry is required for the correct operation of multipliers and processing of signal currents. In this case, this additional circuitry is comprised by a current conveyor, current memory and limiter, whose operation implies determined power consumption. In CACE chip, the sum of all contributions ranges from 18µA to 46µ [Carm01]. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 100 This total current implies a power consumption of 0.15mW in the worst case, considering a power supply of 3.3V. Then, the power consumption associated to the synapse operation, without considering the other processing circuitry, for an array of 32x32 resolution is around 0.5W. In the case of quarter CIF resolution (176x144), the power consumption increases to 3.8W. This power consumption produces an increment of chip temperature, being very important to consider the thermal characteristics of package in the calculation of this increment. An usual thermal coefficient for packages is around 10º-20ºC/W, this means that the die temperature increases between 10º-20ºC per watt. Thus, the temperature increment for a processing array with resolution 176x144 ranges from 65ºC to 103ºC, where a room temperature of 27ºC has been considered. This range of temperature is high for the photodiode operation, the dark current increase exponentially with the temperature degrading the quality image significantly. Another problem or design issue related with the power consumption and multiplier implementation is the distribution of weight voltages ( w V ), which are associated to a diffusion terminal of MOS transistors (Figure 60). The metal lines that distribute the weight references drain current, and thus, the voltage drop provoked by line resistance must be considered. This drop in the weight voltages mean that processing cells in the array have different weights generating gradients in the processed image. For example, the mean filter is a low pass filter characterized by a template whose coefficients are equals: 111 999 111 999 111 999 A          All pixels contribute with the same weight, being the output image: 1 9 rr ij kl k r l r yx    Eq. 73 For a uniform input image ( ij x ), the output image ( ij y ) should be the same. But in the case that weight contributions change along the array, the resulting image presents a gradient. For instance, a linear horizontal drop in the weight distribution means that the convolution template depends on column coordinate:                   1, 1 1,0 1,1 0, 1 0,0 0,1 1, 1 1,0 1,1 1 1 1 1 1 1 9 9 9 1 1 1 ( ) 1 1 1 9 9 9 1 1 1 1 1 1 9 9 9 j j j A j j j j j j j                                                     Considering negligible the mismatch of drop error among contributions, the result of mean convolution depends on column coordinate ( j ), generating a horizontal gradient:     11 9 rr ij ij k r l r y j j x         The drop associated to the power supplies distribution, which is provoked by the current demand of processing cells, has a similar effect on the CNN operation. Therefore, depending on error drop allowed by the processing circuitry in order to achieve the required accuracy in the algorithm implementations, there is a tradeoff between current consumption per cell and width of distribution lines (for weight references and power supplies). This width is determined by the current that flows through the line and the length of this one. Then, the resolution is determined by current consumption per cell and processing array area. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 101 There is an additional issue related with the errors introduced by the references and power supplies distribution, the dependence of these errors with the technological parameter variations and temperature. Although this errors due to gradients in the weight references and power supplies could be compensated by an optimization of coefficients and inputs associated with the CNN operation or by any other calibration procedure, these correction factors may change with temperature or may not be valid for different chips. This is a critical problem for the production and introduction into the market of a particular vision system. 2.6.8. OTHER TYPES OF COMPUTATIONAL SENSORS As read in sections 2.3.3 and 2.6, the CNN approach is one of the different architectures that can be considered for the implementation of an Analog Parallel Array Processor (APAP). The CNN approach processes in parallel all contributions, which represents the maximum degree of parallelism to implement. But in this case, parameters such as power consumption and processing array resolution become limiting. That is, there is a tradeoff between the level of parallelism in processing and power consumption-resolution figures. In order to optimize the cell area and reduce the power consumption per cell, which would allow increasing the resolution of processing array at the expense of a reduction in computing power, the different contributions could be processed serially. This means that instead of nine multipliers working in parallel, the cell operation requires only one multiplier developing the operations serially, with the corresponding reduction in area and power consumption. This strategy increases the cell processing time, which implies reducing the computational power, but also allows a sensingprocessing array with higher resolutions. Note that this computational level achieved by having one operator (one multiplier) per pixel does not compromise the speed requirements demanded by the vast majority of applications. Most of the implemented APAPs include the minimum circuitry necessary to perform a specific image processing task at pixel level to optimize power and area; several examples have been reported in section 2.3.3. They are application specific information processors at pixel level. Therefore, this approach implies to develop vision systems for a specific processing algorithm with a low capability of programmability. There is a tradeoff between the flexibility or programmability level and implementation efficiency (power consumption and area). Ideally, from a functional point of view, a high level of programmability to carry out many processing algorithms is a desirable characteristic of a vision system with the objective of engage the maximum number of applications. In order to include the greatest possible number of pixel functions ensuring a level of efficiency that makes viable the system, a set of basic operations required by different image processing tasks belonging to the early-processing stage has to be defined. And these basic operations can be implemented efficiently by different type of circuits at pixel level. For instance, the spatial low pass filters can be implemented by a resistive or diffusion grid (section 3.2.8 of Chapter 3), the arithmetic operations can be developed by switchedcapacitors circuits (section 3.2.5 of Chapter 3). Spatial high pass filters can be obtained based on these arithmetic operations and low-pass filtering. Generally, in case that a generic spatial filter has to be implemented, the spatial-domain functions performed in the early-stages of digital image processing in image enhancement algorithms can be expressed as:     , ( , )y i j T x i j Eq. 74 Where   ,x i j is the input image, ( , )y i j the processed image and T the operator. One of the principal approaches in this formulation is based on the use of so-called templates or convolution masks. Basically, a template is a two-dimensional matrix whose coefficients are chosen to detect a given property in the image. Usual dimensions of this template is 3x3, Figure 63 shows the coefficients and corresponding image pixel locations. The convolution operation for a 3x3 template is described by the following equation: Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 102     1 2 3 4 5 6 7 8 9 , ( , ) ( 1, 1) ( 1, ) ( 1, 1) ( , 1) ( , ) ( , 1) ( 1, 1) ( 1, ) ( 1, 1) y i j T x i j w x i j w x i j w x i j w x i j w x i j w x i j w x i j w x i j w x i j                                  Eq. 75 w1 (i-1,j-1) w2 (i-1,j) w3 (i-1,j+1) w4 (i,j-1) w5 (i,j) w6 (i,j+1) w7 (i+1,j-1) w8 (i+1,j) w9 (i+1,j+1) Figure 63. 3X3 CONVOLUTION MASK. In this generic case, the basic operator for image processing can be implemented in the analog domain at pixel level by a switched-capacitor circuit (multiplier-accumulator, section 3.2.5 and section 3.2.7 of Chapter 3), where the different coefficients of template are defined by relations between capacitors. Each contribution associated to a determined coefficient is calculated and added or subtracted serially to obtain the processed pixel value. The main characteristic required by pixels to perform convolutions is the interconnectivity among neighbors. In this physical implementation, the distribution of analogue references which are necessary to perform the circuit operation and the signal voltages are distributed by purely capacitance nodes. Therefore, the drop associated to the current flow is not present in these references as in CNN approaches. Only the power supplies drop must be considered during the design phase. But, in this case, this drop does not affect directly to the operation, being a second order error given by the non-ideal behavior of analogue circuit, which can be compensated by design. So far, only an APAP (Analogue Parallel Array Processor) architecture at the focal plane (processing-in-pixel) has been considered for the implementation of smart sensors. But, there is other possible interesting architecture which reduces the power consumption respect to the previous one and solves the pixel fill-factor problem, obtaining higher spatial resolution and accuracy, since the processing circuitry has available a greater area budget. This new architecture is based on a column processing approach, similar to the column ADC approach implemented in the Digital Image Sensors. This new approach for the smart sensor consists of a conventional pixel array and associated with each column there is an elementary processor, which has the ability to connect with the neighboring columns. Figure 64 depicts the proposed architecture. Column Processor Column Processor Column Processor Column Processor Column Processor I/O Interface Vertical Addressing Sensing Array Control Interface Sensor Control Figure 64. COLUMN PROCESSING APPROACH. In general, each elementary column processor can be composed by analogue processing circuitry, analogue memories, A/D conversion and threshold operation as interface with digital domain, digital memories and elementary digital processors. Figure 65 presents a general block Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 103 diagram of an elementary digital processor. Usually, the elementary digital processor is a custom digital circuit designed and optimized to performed specific tasks. Analog Processor Analog Memories A/D conversion & thresholding Digital Memories Elementary Digital Processor Figure 65. ELEMENTARY COLUMN PROCESSOR BLOCK DIAGRAM. A particular case is the column processor composed by an A/D converter plus digital memories and elementary digital processor. This case can be interpreted as the first step towards parallel processing architecture on-chip in order to perform the tasks of early-processing algorithms. It is an approach similar to that followed by conventional digital cameras. But in this case, due to the implementation of low-level processing circuitry on chip, the system gets a higher integration level and avoids the speed limitations associated with the data interface. This last feature together with a higher parallelism in the processing implies an increment in the computing power of system. The introduction of a microprocessor into the smart sensor with access to digital memories of elementary column processors allows the implementation of high-level processing algorithms, given rise to the concept of vision system on chip. 2.7. CHALLENGES FOR CAMERAS AND VISION SYSTEMS. OVERVIEW OF THE STATE-OF-THE-ART Challenges are determined by market demands and society requirements. For instance, in surveillance applications, current Closed Circuit TV systems (CCTV) are mostly not automated systems and rely strongly on trained security personnel to perform image analysis, object tracking and identification. The increasing number of cameras makes difficult the analysis in real-time by security personnel. Network bandwidth is another important issue affecting realtime processing needed for crime prevention. The intelligent video surveillance systems (IVSS) will try to provide solutions to these problems. Smart cameras will be one of the fundamental building blocks of the IVSS, making it possible to build and deploy automated, distributed and intelligent multi-sensory surveillance systems capable of tracking humans and suspected objects, analyzing human behaviors, etc. (Read section 2.4.5). Another important application area for vision systems and cameras is the industry Machine Vision. The demand for compact vision systems has been increasing over the years and is primarily driven by the increasing demand for better production efficiency and quality control in industries such as robotics, semiconductor, electronics, pharmaceutical, manufacturing, food, plastics, printing, etc. The tasks of these compact systems perform bar-code reading, part inspection, flaw detection, surface inspection, dimensional measurement, assembly verification, print verification, object sorting, optical character recognition, etc. Other important application areas for embedded vision systems, where their demand is increasing too, are Intelligent Transport Systems (ITS), Human Computer Interface (HCI), games, toys, automobiles (collision avoidance, passenger monitoring), unmanaged vehicles, etc. All previously submitted applications require either high-spatial-resolution or high-frame rate or both. This fact means a high throughput for the data interface of image sensor and camera. Current digital cameras on the market that contain a digital CMOS image sensor can achieve very high transmission frequencies above 78 Gbits per second, which corresponds to a sensor with a resolution of 1280x1024, including 12 bits AD conversion and running at 5000 frames per Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 104 second. These throughputs imply currently that a custom interface must be designed to interconnect these high frame rate cameras with others blocks in a vision system, which are responsible for image processing. This custom interface increases the production cost of vision system. Standard interfaces on the market such us GigE, USB, Firewire, Camera Link or CoaXpress do not achieve these throughputs. The fastest interface is CoaXpress whose maximum throughput is 6.25 Gbits per second. Therefore, the standard communication interfaces used currently in vision systems are a bottle neck and limit the operation speed of vision systems. In this context, embedded vision systems (as smart cameras) incorporating parallel processing in the focal plane may be an alternative; this approach is able to avoid this bottle neck in the image transmission, because only a compress version of real images with useful information is transmitted. Technically, high frame rates are achieved in the capture operation by CMOS image sensors. Thus, CIS technology allows designing high speed cameras which continue to shrink in size, reduce in cost and gain in performance (Moore’s law) thanks to its ability to integrate into a single chip; all the building blocks that comprise the image sensor: pixel array, readout circuitry (including the AD converters), sensor control block, the digital data interface, etc. (see section2.4.1). For instance, the column ADC approach is commonly used in high speed digital image sensors. But this high speed operation in digital CMOS image sensors is attained losing image quality. Therefore, there is a trade-off between speed-parallel processing level and quality. One of the main lines of research and development in CIS technology is improving image quality in order to reach levels comparable to those obtained in CCD technology, which is a mature technology with excellent image quality; high sensitivity and long exposure time, thanks to extremely low noise, high quantum efficiency and very high fill factor. Regarding to the temporal read noise, current readout circuitry implemented in image sensors chip has noise levels similar to that obtained in CCD image sensors (1-8 e), operating both sensors at same frame rates. The trend in digital image sensors is the design of new readout circuitry for high-speed applications extending this low noise level to the maximum possible operating frequency. Other important parameter to obtain a good image quality is the number of photo-generated electrons for a given irradiance, take into account that the SNR parameter is defined by the maximum number of photo-generated electrons and the noise level, read section 2.5.5.2. This parameter is determined by the quantum efficiency and fill factor together with the pixel area. Improvements related to QE require optimization at the process level: optimization of photodetector doping profile to improve photo-charge collection and top layers to enhance optical transmission. Thanks to new high volume markets, silicon founders are investigating to develop photodiodes with higher QE and lower Dark Current. To compensate the loss of fill factor, micro-lenses are widely used in CMOS image sensors and backside thinning is currently being developing. In addition, several attempts are on the way to get near 100% fill factor and efficient charge collection. They are based on 3D integrated circuits where photodiodes are made in an optimized detection layer on top of a CMOS readout/processing circuitry layer, see Figure 66. Figure 66. 3D IC TECHNOLOGY BASED ON THROUGH SILICON VIAS (TSV) INTERCONNECTION Figure obtained from Wikipedia Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 105 In summary, current applications demand vision systems with more operation speed and accuracy (high spatial resolution and low noise). High speed implies the following challenges:  Higher responsivity in sensing operation in order to reduce the exposure time. This requirement demands increments in quantum efficiency and fill factor.  Higher frame rate and computational power: This means to develop shorter conversion times during readout process and parallel architectures for data processing. Processing at pixel level limits the fill factor and spatial resolution.  Limited bandwidth of data transmission channels: This limitation is overcome by the introduction of processing at sensor and camera levels. And higher accuracy involves the challenges:  Higher spatial resolution: This requirement implies smaller pixels, which decrease the responsivity per pixel. Also, an increment in the number of pixels carries a higher data throughput.  Higher precision in the readout and processing circuitry: This objective represents an increase of area. For the case of processing at pixel level, this increment of area affects to the fill factor and spatial resolution. 3D integrated circuit technology is a promising approach to address the previous challenges. This technology allows implementing a massively parallel analog processing without jeopardizing resolution and responsivity, because the readout-processing circuitry is located in a different layer, see Figure 66. In addition, the accuracy achievable by the processing circuitry will be higher since the increment of area required for that may not affect to the pixel area and fill factor. Regarding the accuracy and signal integrity, locating the digital circuitry (memories and micro-processor) in a third different layer reduces the impact digital activity on the effective precision reached by the mixed-signal processors, since sensing devices, mixed-signal circuitry and digital circuitry are implemented in independent substrates. From a practical point of view, when a camera or vision system achieved the specifications for speed and accuracy to develop a specific task successfully, come into play parameters such as cost and dimensions. Both directly related with the integration level that can be achieved during the system implementation, 2.7.1. CAMERAS-ON-CHIP As deduced from examples presented at the beginning of section 2.7, the market is demanding cameras more compact. There are applications where the number of cameras required is high (for example, surveillance applications), or the space available for camera location is very limited (for example, machine vision or automobile applications), etc. At this juncture, characteristics such as size and cost must be minimized. Currently, a digital camera whose architecture is described in Figure 67 can be implemented into a single chip in CMOS technologies. This approach has several advantages demanded by the market: reduction cost, power consumption and size of the camera. Lens Camera on Chip Data Processing Memory bank µProceso r Control Interface Data Interface clk_ext ADC Column Amplifiers Pixel Array Sensor Control PLL Analogue References Figure 67. CAMERA-ON-CHIP ARCHITECTURE. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 112 (logic ‘1’). Therefore, active pixel downloading becomes especially effective when the readout images have few white (or black) pixels. The same operation can be used in the opposite way; isolated pixel in a binary image can be activated by providing its (x,y) coordinates from outside. BINARY TO ANALOGUE CONVERSION (BAC). This function is used to generate an analogue image from a binary one. The input must be the binary image stored in one LDM and the output the LAM where the resulting analogue image will be stored. The generated image has the maximum analogue value for pixels that are 1 in the binary input image, and has the minimum analogue value for pixels that are 0 in the input image. The implementation of this conversion function requires using the basic operations of some others functional blocks described above. In particular, the global references block and analogue memory block. In this last one, the writing mask function is used taking into account the binary image to be converted. ANALOGUE TO BINARY CONVERSION (ABC). This function is carried out by the MAC block operating as a comparator in order to develop a threshold operation. The result of this operation will be stored in a binary Local Digital Memory (LDM). IMAGE I/O. This block is the interface for image uploading/downloading in the complete system and will be described in Chapter 3. 3.1.2. DATAFLOW, SIGNAL FORMATS AND CONTROL OF THE Q-EYE CELL Optical images are the primary Q-Eye inputs. Each spatial image sample is generated by a number of photons impinging into a corresponding active area of a photodiode. Photons are first transformed into electrical charges and then into voltages; these voltages are stored in a local analogue memory for ulterior processing  see Figure 2. Optical Sensor Iph (photons) Conversion Gain CG (µV/e) Ne (electrons, e)   maxmin ,VVV NCGV im eim   Local Analogue Memories   maxmin ,VVVim  Figure 2. PROCESS OF LIGHT SENSING IN THE Q-EYE CELL. The intensity value sensed by the cell photodiode is mapped onto an unsigned output voltage   yxVim , comprised within the range   maxmin ,VV , namely     minmax ,, VVyxVim  . During processing the unsigned signals stored in the LAMs are handled and interpreted as signed signals by making the transformation:     zeroim VyxVyxI  ,, where 2 minmax VV Vzero   is at the middle of the dynamic range   RR VV , of the signals being handled during processing. Figure 3 illustrates the signal modalities, corresponding ranges and signal transformations within the Q-Eye cell. It is worth mentioning that the sign of the signals employed for analogue processing carries itself information which may be relevant for processing, such as the direction of the movement of an object present in the scene, the direction of zero crossings, etc. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 113 When an analog image is downloaded from the Q-Eye, it is converted to digital such that the sensing range   maxmin ,VV and the processing range   RR VV , are both linearly mapped into the positive interval   255,1 by means of Analogue to Digital Converters (ADC) embedded in the external image interface. Likewise, when a digital gray-level image is loaded to the Q-Eye, the interval   255,1 is linearly mapped into the sensing range   maxmin ,VV by means of Digital to Analogue Converters (DAC) embedded in the I/O Image Interface of system, see section 4.1 and section 4.2 of Chapter 4. im V Local Analogue Memories   maxmin ,VVVim  im V Binary to Analogue Conversion Analogue to Binary Conversion im D im D Local Digital Memories   1,0 im D   zero V zero V - + + + im V im V out I in I Analogue processing circuitry   RR VVI ,      max min 1 0 VVD VVD imim imim      1 0 imzeroim imzeroim DVV DVV   inout IfI  out D in D Digital processing circuitry   inout DgD    1,0 im D Figure 3. SIGNAL MODALITIES AND RANGES IN THE Q-EYE CELL. As already mentioned, the Q-Eye handles both analog and binary images. Note in Figure 3 that the analogue-to-binary and binary-to-analogue conversions implement a correspondence between the “0” binary level and the reference voltages R V or min V and the “1” binary level and analogue reference R V or max V . During loading and downloading of binary images, the external image interface of Q-Eye writes directly the binary values “0” and “1” in LDMs. Image Acquisition Analogue Memories Analogue processing min V max V zero V min V max V zero V R V R V 0 Analogue to binary image conversion 0 1 128 255 1 128 255 Grey-level storing mode 0 1 Binary storing mode Eye-RIS Image Memory Image Load/Download 1 Binary processing 0 1 Binary Memories CVIS Q-Eye Figure 4. DATA FLOW AND IMAGE CODIFICATION IN THE Q-EYE. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 114 Figure 4 summarizes the data flow in the Q-Eye cell and the data codification used in the main processes involved in the operation of the cell, namely: image acquisition, analogue storing, analogue processing, binary storing and binary processing. Proper control is needed to exploit the parallel processing capabilities of the Q-Eye. Control of cell array has been designed in the Q-Eye system following the principles of Single Instruction Multiple Data architecture (SIMD) [Zara11], [Dunc90]. The cell array processes in parallel one image applying the same operation to each pixel. The operation is determined by the control unit of system. Several control signals arrive to each functional block in the Q-Eye cell for configuring and controlling the operation of the corresponding mixed-signal circuit, see Figure 5 and Figure 6. LLU HitAndMiss LDMs Image I/O Active Pixel I/O Optical Sensor ladn Resistive Grid LAMs MAC Image shifter Analog-to-binary converter Binary-to-analog converter lddn OS_control_bus RG_control_bus LAM_control_bus MAC_control_bus Shifter_control_bus LLU_control_bus H&M_control_bus LDM_control_bus IO_control_bus AP_control_bus PCU Programmable Control Unit PCU_bus<59:0> Global References Ref_control_bus Q-eye pixel Figure 5. Q-EYE CELL CONTROL. The control of system is managed by the Programmable Control Unit digital block named PCU. This block controls the processing array applying a control signal vector or instruction which is stored in a register called PBCTRL included in PCU block. The PBCTRL register has a width of 60 bits. Each bit is associated to a specific control signal of the Q-Eye cell. Processing Array PCU PCU_bus<59:0> Figure 6. SIMD ARCHITECTURE. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 115 Instructions are applied simultaneously to all cells in the array. The implementation of one basic operation in the processing array (for instance, read one analog memory, see Table 2) implies the application of a sequence of these instructions. Further details are given in Chapter 4, section 4.1.2. 3.1.3. RATIONALE FOR SELECTION OF THE Q-EYE OPERATORS Basic early vision tasks include image convolutions, spatial-temporal filtering with programmable bandwidth, extraction of image features such as centroids, mean values, pixel coordinates… [Gonz92]. Early vision is aimed to reduce the amount of data associated to images before sending these data for further digital processing [Toma06] [Shi05]. The rationale is that images do contain much more data than information. In other words the same information can be retrieved with a reduced set of the data captured by the optical sensors and defining image frames. Reducing the data is far from trivial because it must be done without degrading the underlying information. In the more general case, linear, non-linear, static, dynamics and statistical operations are required to cover “all” possible early vision task. However, in practice a tradeoff between the in-pixel circuitry and the pixel pitch must be made [Rodr03]. In the case of the Q- Eye this tradeoff is addressed on the basis on the experience accumulated with previous generations of ACE chips [Espe94a, b] [Rodr93] [Carm01] [Liña02a, b] [Carm03] [Rodr04] [Liña04] and the request posed by practical applications [Ohta08] [Deva08]. The available operations can be classified within three categories, namely:  Point-wise operations: o Storing (gray-level and binary images) o Addition o Subtraction o Multiplication by a constant o Threshold operation o Combinational operations (binary images)  Local operators: o Low-pass Gaussian filters, implemented by diffusion using a resistive grid. o Hit-And-Miss binary operations considering 3x3 neighbourhood. o Spatial convolutions, combination of arithmetic operations (addition, subtraction and multiplication by a constant) and shifting operation.  Shifting operation (gray-level and binary images)  Inter-frame processing: o Addition o Subtraction o Combinational operations Point-wise operators do only depend on the value of the pixels to which these operators are applied. Regarding local operators, they require as well the values of the pixels located within a certain neighborhood of the processed pixel. Point-wise and local operators are used for intraframe processing. Memory elements are used to store several consecutive frames and thus implement inter-frame operations. For instance, in motion detection applications it is common to compute a difference image by subtracting the pixel value of a previous frame from the value in the current frame. The early-processing stage comprises image enhancement and restoration followed by processing to extract image features such as object locations, shapes, edges, etc. During image enhancement and restoration, the input images are transformed in some predefine sense to simplify the extraction of features to be used during the post-processing stage. Typical functions used in the enhancement and restoration process include:  the gray-level transformations to improve the image contrast,  smoothing or averaging spatial filters to reduce the spatial noise, Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 116  sharpening spatial filters to highlight fine image details,  temporal averaging to reduce the temporal noise,  non-linear median filters to eliminate the impulse noise, etc. The gray-level transformations can be implemented by using point-wise operations. Linear filters can be implemented through the resistive grid and arithmetic operations or spatial convolutions. The Q-Eye cell includes a non-linear operation, the threshold operation, which permits to implement non-linear filters in the processing array, for example Rank-values filters, like the median filter, or adaptive filters. The extraction of image features comprises many complementary tasks. Image segmentation is a typical processing which subdivides an image into its constituent regions or objects. The level to which the subdivision is carried depends on the problem being solved. The segmentation should stop when the objects of interest in an application have been isolated. Segmentation algorithms generally are based on one of two basic properties of intensity values:  discontinuity and  similarity. In the first category, the approach is to partition an image based on abrupt changes in intensity. The detection of intensity discontinuities (points, lines, edges) is developed applying high-pass filters. The principal approaches in the second category are based on partitioning an image in regions that are similar according to a set of predefined criteria. Thresholding, region growing, and region splitting and merging are examples of tasks in this category. Several morphological operators can be implemented with Hit-And-Miss operations and combinational logic. The discontinuities detection functions and threshold function together with morphological operators constitute the early-processing level in the segmentation algorithms and extract the features interesting for a determined application. These features are described in a binary image which contains a reduced set of data compared with the amount of data contained in the original gray-level image. This compression of data implies a faster communication between the Q-Eye and the microprocessor which develop the high-level functions, for example, the postprocessing segmentation and classification algorithms. This is the reason why the Q-Eye includes a fast interface for loading and downloading sparse binary images. Therefore, characteristic algorithms of early-processing level such as filtering, threshold, edge detection, morphological transformations, binary operations, arithmetic operations, etc. are implemented by applying a group of simple operators on every pixel of the image. The earlyprocessing algorithms are characterized by being simples and applied to a vast amount of data. In this case, the computation requirements are very high being optimal a parallel processing architecture consists of a high number of elementary and simple processors, similar to the approach implemented in the Q-Eye. The architecture of elementary processor must be the simpler simple as possible in order to optimize power consumption and area. There is a trade-off between the number of operators implemented in the elementary processor and their power consumption and area. For instance, the Q-Eye cell pitch limits the maximum number of control signals which can be driven to each processor. 3.1.4. ILLUSTRATING DATA REDUCTION THROUGH THE Q-EYE The Q-Eye plays the fundamental role of reducing the amount of data at the early stages of the vision processing chain, illustrated in Figure 7. This figure describes the overall architectural target with reference to a conceptual representation of a vision processing chain [Russ92]. It includes several processing steps and shows that the amount of data decreases as information travels along the chain, namely:  At initial steps the number of data is huge and many of the data are redundant and hence useless to the purposes of reaction prompting.  As information flows across the processing chain and abstract features are extracted from the incoming images, the number of data decreases. In conventional vision architectures the border between sensors and processors is placed at a point where the amount of data is large. However, in the Eye-RIS architecture this border is located at a point where the amount of data is small. Assume for illustration purposes that we Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 117 target to tracking objects moving at 40m/sec into a scene. It requires capturing and analyzing images at 2000F/s rate. At the outcome of the capture/analyze process the only pertinent data is the predicted position of the objects. This is actually the only information driven to the digital processor. The Q-Eye plays the fundamental role of extracting this information by executing the following tasks within the sensor: i) Image acquisition; ii) low-pass filtering; iii) activity detection; iv) motion estimation; v) object tracking; vi) loop control, and vii) position prediction. and thus shifting the border between sensors and processors to a point where only a few data are left – see Figure 7 and Figure 8. Figure 7. PROCESSING CHAIN OF VISION. AS THE DATA EVOLVE FROM THE SENSOR (RAW DATA), THE AMOUNT OF DATA DECREASES AND THE ABSTRACTION LEVEL INCREASES. M REPRESENTS THE NUMBER OF COLUMNS, N THE NUMBER OF ROWS AND B THE NUMBER OF BITS USED PER PIXEL DATA: N < N; M < M AND P < (N, M). Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 118 Figure 8. ILLUSTRATION OF THE PROGRESSIVE DATA REDUCTION OF DATA ALONG THE VISION PROCESSING CHAIN AS IT ACTUALLY HAPPENS IN THE EYE-RIS. ALL STEPS OF THE PROCESSING CHAIN ABOVE BUT THE LAST ONE ARE COMPLETED IN THE Q-EYE SENSORY-PROCESSING FRONT-END. THUS, THE DATA SET DELIVERED FOR PROCESSING BY THE HOST DIGITAL PROCESSOR IS QUITE SMALL. Figure 8 shows major steps of the processing chain implemented within the Eye-RIS_v2.1 to find defective parts in a production line. Eye-RIS_v2.1 identifies defective parts based on feature analysis instead of brute force pattern matching. It enables speed improvements (number of pieces per second) of several orders of magnitude as compared to conventional systems [COGNEX]. The figure illustrates the progressive reduction of data along the processing chain. Out of the some 26kbyte raw data acquired by the sensors at the Q-Eye front-end, only some 100byte remain after pre-processing and are actually coded in digital domain and sent to the post-processing stage. Such reduction, together with the intensive parallel processing performed at the front-end is instrumental to achieve this data reduction and hence the overall efficiency enhancement in the completion of the part finding task. 3.2. CIRCUIT IMPLEMENTATION OF THE Q-EYE CELL OPERATORS This section describes the circuitry employed for the different blocks in the Q-Eye cell and presents the equations that support the ideal behavior of such circuitry. Electrical analyses included in this section are first-order for all the cases. The impact of circuit errors will be discussed in Section 3.3. 3.2.1. GLOBAL ANALOGUE REFERENCES Correct operation of the Q-Eye mixed-signal blocks require proper biasing. Also reconfigurability considerations call for a large variety of biasing conditions. Table 1 describes the main reference voltages indicating the functional block which uses the references. REFERENCE DESCRIPTION FUNCTIONAL BLOCK Vpch Precharge voltage of pixel Optical sensor Vhpch Control signal of reset transistor in pixel Optical sensor Vhold Reference used during hold configuration in analogue memories LAM block, Global Reference block Voffset Reference to define the offset in MAC operation. MAC block, Global Reference block Vmisc General purpose reference Global Reference block Table 1. Analogue references of processing cell. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 119 The Global Reference block consists of an analogue multiplexer that connects the references Vhold, Voffset and Vmisc to the local analogue node ladn in function to the control signals. Figure 9 depicts the architecture. Global references have been represented in red and the control signals in blue. This notation will be maintained throughout the chapter. The general purpose reference Vmisc is a useful programmable reference which allows loading in parallel way a gray level uniform image to the processing array. Each analogue reference described in Table 1 is generated on-chip in the Q-Eye system by a programmable DAC through a configuration register accessible through the external programming interface. Vhold Voffset Vmisc ladn s_hold s_offset s_misc Figure 9. GLOBAL ANALOGUE REFERENCE BLOCK. 3.2.2. DATA STORAGE AND TRANSFER As already mentioned, rach Q-Eye pixel embeds 9 LAMs and 9 LDMs. The topology of these local memories will be described in following sections. 3.2.2.1. ANALOGUE MEMORIES Figure 10 depicts the architecture of the LAM block. This block consists of two banks of analogue memories. One bank has seven (7) memories and it is called LAMs bank. The other bank has two (2) analogue memories and it is named Auxiliary LAMs bank. Each memory bank consists of a S&H (Sample and Hold) circuit containing N (either seven or two) capacitors associated to a common, shared, sensing amplifier. The S&H topology has been chosen to yield:  non-destructive recovery of the stored signal;  reduced harmonic distortion caused by signal-dependent switching errors; and  insensitiveness to the sensing amplifier offset [Greg94]. The basic operations associated to a memory bank are writing, reading and holding respectively. During the writing phase, the capacitor m C is charged by the voltage   Nqdata VV  :       nVVCnQ NqdatamCm Eq. 79 In the reading phase, the charge of capacitor m C is:         111  nVnVCnQ NqNoutmCm Eq. 80 Because the bottom plate of m C is isolated during the reading phase, the charge in the capacitor does not change,     1 nQnQ mm CC , and hence the output voltage in the Nout node is given by:         nVnVVnV NqNqdataNout  11 Eq. 81 Considering a sensing amplifier with infinite gain obtains     nVnV NqNq 1 , meaning that the data can be recovered during reading:   dataNout VnV 1 Eq. 82 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 120 To neighbor multiplexer input ladn slm c_mux w_lm r_lm hold Nq Nout Vhold d_m_lm mask sm<0> sm<1> sm<2> sm<3> sm<4> sm<5> sm<6> w_lm_a r_lm_a hold Nq_a Nout_a Vhold sm<7> sm<8> slm_a c_mux_a v_lcl LAMs bank Auxiliary LAMs bank Figure 10. LAM BLOCK ARCHITECTURE. In order to optimize power consumption and area, the sensing amplifier has been implemented by a telescope cascode inverter [Malo01] and the capacitors have been developed with MOS transistors (MOS capacitors)  see Figure 11. Vbiasp Vcasp Vcasn Vin Vout M1 M2 M3 M4 Vtop Vbottom Figure 11. TELESCOPE CASCODE INVERTER. Note from Figure 10 that writing and reading operation can be made conditional through the usage of the writing-reading mask. The configuration signal d_m_lm enables the mask option. In this case, the writing or reading operation is masked depending of a binary image stored in a particular local digital memory. To optimize the Q-Eye pixel area, the binary mask used in the Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 121 LAMs bank is defined by the same local digital memory which stores the mask for the resistive grid. The instruction sequence to develop the operations related with the memory bank is presented in Figure 12 and Table 2. The control of the Auxiliary LAMs bank is similar to the LAMs bank control. slm w_lm r_lm sm<i> hold ladn Writing phase Reading phaseHolding phase Vdata Vdata Figure 12. INSTRUCTIONS FOR ANALOGUE MEMORY OPERATION. Nº Instruction slm w_lm r_lm sm<i> hold ladn Operation 1 0 0 0 0 0 - - 2 0 1 0 0 0 Vdata Writing 3 1 1 0 1 0 Vdata Writing 4 1 0 0 1 0 Vdata Writing 5 0 0 0 0 0 - - 6 0 1 0 0 1 - Holding 7 0 0 0 0 0 - - 8 0 0 1 1 0 - Reading 9 1 0 1 1 0 Vdata Reading 10 0 0 1 1 0 11 0 0 0 0 0 Table 2. Analogue memory instructions. During the reading phase, the memory banks can drive the stored value in the local data node ladn or send the data to the analogue multiplexer, activating the slm, slm_a control signals or c_mux, c_mux_a signals respectively - see Figure 10. In the holding phase, the analogue memory bank is configured to reduce the effect of leakage currents in the MOS capacitors controlling the voltage of critical nodes. 3.2.2.2. DIGITAL MEMORIES Figure 13 depicts the architecture of the LDM block. It is composed by 9 latches with the schematic of Figure 14. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 128   luxesIph Saturation level   VVim 1 VVV Treset  2 VVV Treset  3 VVV Treset  4 VVV Treset  1_ph I 2_ph I 3_ph I 4_ph I linear ph Imax_ HDR ph Imax_ min ph I sat V Figure 23. HDR SENSING MODE CHARACTERISTIC. The Dynamic Range (DR) in linear mode is given by:          min max_ log20 ph linear ph linear I I DR Eq. 90 Where linear ph Imax_ is the maximum illuminance in linear mode for which the image data reaches the saturation value. In HDR mode, the dynamic range is improved. The increment of decibels is:          linear ph HDR ph I I DR max_ max_ log20 Eq. 91 the increment depending on the programmed compression law. 3.2.5. MULTIPLIER-ACCUMULATOR CIRCUIT (MAC) 3.2.5.1. ARITHMETIC OPERATIONS The MAC implements the following operation: )( 210 IIkIIout  Eq. 92 where:       maxminmaxmin00 ,,, VVVVVVVVIVVI zerooffsetRRzerooffset      maxmin1111 ,, VVVVVIVVI imRRzeroim      maxmin2222 ,, VVVVVIVVI imRRzeroim  1im V and 2im V are the voltages associated to the corresponding input images, zero V is the reference voltage which fixes the zero level in the processing domain. offset V is an analogue internal programmable reference to define the offset contribution. k is a programmable Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 129 constant whose value can be 2, 1 or 1/2. Addition, subtraction and scaling operations can be developed using this basic MAC operation. Figure 24 shows the electrical implementation of the MAC block by using Switched-Capacitor (SC) circuits with control and configuration signals marked in blue. Figure 25 shows the sequence of control signals required to implement a basic operation. Cin=f(Cu,cai) Cu Cu Cu Cu Cout=f(Cu,cao) ladn ladn cai cao s_mac_w s_mac_r w_mac r_mac zero Nq Nout Voffset Figure 24. ELECTRICAL IMPLEMENTATION OF MAC. Three phases can be defined in the MAC operation.  During phase 1, the charge stored in in C and out C is given by the equations:     Nqiminin VVCphQ 1 1 Eq. 93     Nqoffsetoutout VVCphQ1 Eq. 94 In this phase 1, the MAC reads the operand 1im V from the internal local node ladn and stores it in the in C capacitor. The out C capacitor is reset to the reference offset V .  During phase 2, the charge stored in in C and out C is:     Nqiminin VVCphQ 2 2 Eq. 95     NqNoutoutout VVCphQ2 Eq. 96 In phase 2, the MAC reads the operands 2im V and stores it in in C . The capacitor out C is connected to implement the feedback loop storing the charge difference in capacitor in C between phase 1 and phase 2. The output voltage is readily calculated into first-order approach by applying charge conservation principle to high impedance node Nq:   21 imim out in offsetNout VV C C VV  Eq. 97  During phase 3, the MAC writes in the analogue local node ladn the result given by the arithmetic operation. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 130 s_mac_w w_mac zero r_mac s_mac_r ladn phase1 phase2 phase3 1im V 2im V out V Figure 25. SEQUENCE OF INSTRUCTIONS FOR A MAC OPERATION. The programmable gain factor k is configured through the signal cai and cao. Table 4 describes the gain factor which can be programmed: cao cai Cin/Cout k 0 0 Cu/Cu 1 0 1 2Cu/Cu 2 1 0 Cu/2Cu ½ 1 1 2Cu/2Cu 1 Table 4. Configuration of gain factor. The electrical implementation of the inverter and the capacitor within the MAC is the simplest possible to preclude penalizing the pixel pitch  see Figure 11. Errors caused by these simple circuits, and particularly the non-linearity of the MOS capacitors must be taken into account during the design phase and will be addressed in section 3.3.2. 3.2.5.2. THRESHOLD OPERATION The MAC circuit can also be used to implement the analogue-to-binary conversion or threshold operation. The sequence of control signals required is represented in Figure 26. Since the threshold operation is described by the equation:          0 0 21min 21max imim imim out VVifV VVifV V Eq. 98 The MAC SC circuit behaves as an auto-zero comparator with this control sequence. s_mac_w w_mac zero r_mac s_mac_r ladn phase1 phase2 phase3 1im V 2im V out V Figure 26. SEQUENCE OF INSTRUCTIONS FOR A THRESHOLD OPERATION. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 131 During phase 3, the result of comparison operation is put by the MAC circuit in the local node ladn to pass the analogue data to a digital memory, interconnecting the internal cell nodes ladn and lddn. For this threshold operation, the possibility of connecting a programmable analogue global reference to each analogue local node ladn in parallel is very useful. This function permits to define different threshold levels. The threshold level can be defined by an image, in this case, several threshold levels are considered locally. 3.2.6. LOGIC AND MORPHOLOGICAL OPERATORS 3.2.6.1. LOCAL LOGIC UNIT (LLU) This programmable block implements a large catalog of logic operation through programming. It consists of a programmable multiplexor controlled by the logic operands Op1, Op0 and whose inputs are for configuration signals, namely: ht00, ht01, ht10, ht11. These configuration signals define the truth table of combinatory operation. See Figure 27 and Table 5. Op0 Op1 Out 0 0 ht00 0 1 ht01 1 0 ht10 1 1 ht11 Table 5. Definition of binary operation. Op0 Op1 ht00 ht01 ht10 ht11 0 1 2 3 Out Figure 27. LLU IMPLEMENTATION. The operands are provided by two digital memories. Different combinatorial operations (NOR, NAND, XOR, AND, OR, etc.) can be implemented by properly setting ht00, ht01, ht10 and ht11. For instance:  NOR operation requires ht00=1, ht01=0, ht10=0 and ht11=0,  NXOR operation requires ht00=1, ht01=0, ht10=0 and ht11=1. 3.2.6.2. MORPHOLOGICAL OPERATORS The basic operation of binary morphology, known like Hit-and-Miss operation [Russ92] is developed by the HitAndMiss functional block whose architecture is described in Figure 28. It checks whether the 3x3 neighborhood of a cell matches a specified pattern or not. The inputs to this block are the image to be analyzed (that must be stored in a specific LDM) and the pattern to match. The 3x3 local input image is driven by the bus bv<8:0> in Figure 28 and the pattern is defined by the configuration signals pm<8:0> in the same figure. Through the interconnection bus bv<8:0>, the neighbor pixel values are considered in the morphological operation following the scheme depicted below Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 132 bv<8> bv<0> bv<1> bv<2> bv<3> bv<4>bv<5> bv<6> bv<7> Neighbour interconnection bus pm<8> pm<0> pm<1> pm<2> pm<3> pm<4>pm<5> pm<6> pm<7> Matching pattern definition The result is an image with white pixels at points whose 3x3 neighbourhood matches the input pattern.       0:80:800:8 0:80:810:8 pmbvbvf pmbvbvf morph morph Eq. 99 Three states can be defined as the value of a pixel for the pattern matrix, namely:  1 (white),  0 (black) and,  DNC (Do Not Care), meaning that the value of that pixel is irrelevant to the matching. The DO Not Care pattern is defined through the configuration bus pm<8:0> in a similar manner like the Matching pattern. In the DNC pattern, 1 means that the corresponding operands do not affect to the morphological operation. As a very simple example, the following pattern looks for pixels that are south-western corners inside an image.           000 110 10 DNC This Matching pattern is defined in the HitAndMiss blocks in two steps. Into first step, the DNC pattern defined through the configuration bus pm<8:0> is stored in the dynamic digital memories (DNC memories). The DNC pattern for this example is:           000 000 100 Into second step, the rest of coefficients in the Matching matrix are defined through the bus pm<8:0>. This matching matrix in this case is:           000 110 010 The operation of HitAndMiss block is controlled by the signals low, high, dnnc, dpnc, s_morph, and pm<8:0>. The sequence of instructions to develop the morphological operation is described in Figure 29 and Table 6. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 133 Binary comparison element Do Not Care memory bv<i> pm<i> Comparison Element Comparison Element Comparison Element Comparison Element Comparison Element Comparison Element Comparison Element Comparison Element lddn bv<0> pm<0> bv<1> pm<1> bv<2> pm<2> bv<3> pm<3> bv<4> pm<4> bv<5> pm<5> bv<6> pm<6> bv<7> pm<7> Comparison Element bv<8> pm<8> FUNCTIONAL DESCRIPTION PHYSICAL IMPLEMENTATION Comparison Element bv<8> pm<8> bv<7> pm<7> bv<6> pm<6> bv<5> pm<5> bv<4> pm<4> bv<3> pm<3> bv<2> pm<2> bv<1> pm<1> bv<0> pm<0> high Vhigh Vhigh low dpnc dnnc s_morph lddn Figure 28. HITANDMISS FUNCTIONAL BLOCK. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 134 s_morph dpnc dnnc low high s_pm<8:0> DNC pattern Matching pattern Phase 1 Phase 2 Figure 29. SEQUENCE OF INSTRUCTIONS FOR HITANDMISS OPERATION. Nº Instruction s_morph dnnc dpnc low high pm<8:0> 1 0 0 0 0 0 0 2 1 1 1 1 0 dnc_pattern 3 1 0 1 1 0 dnc_pattern 4 1 0 0 1 0 dnc_pattern 5 1 0 0 1 0 match_pattern 6 1 0 0 0 0 match_pattern 7 1 0 0 0 1 match_pattern 8 1 0 0 0 0 match_pattern 9 0 0 0 0 0 match_pattern Table 6. Sequence of instructions for HitAndMiss operation. During phase 1, the DNC pattern is loaded in the dynamic digital memories and the local digital data node lddn and HitAndMiss block are pre-charged to a low reference voltage (Vlow) -- see Figure 28 This reference corresponds to a 0 digital value. At the end of this phase, the matching pattern is fixed in the HitAndMiss block when the pre-charge process is taking place. During phase 2, a high reference voltage (Vhigh) is applied to the HitAndMiss block as Figure 28 shows. If the matching pattern and pixel neighbourhood coincide, the output of morphological block and local node lddn are charged to the high reference. This reference corresponds to a 1 digital value. The result of morphological operation will be stored through the data node lddn in a local digital memory. The reference voltage fixed in the lddn will be regenerated to digital values by the local digital memory. 3.2.7. CONVOLUTIONS The MAC block together with the analogue shifting block permits the implementation of spatial filters defined by 3x3 templates. In the implementation of spatial filters, each contribution associated to a neighbor requires a multiplication-accumulation operation defined by the equation:   zeroi out in VV C C Eq. 100 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 135 i V is the signal voltage associated to the corresponding neighbor. The weight of filter is defined by the programmable factor out in C C . The sign of the neighbor contribution is determined by the phase of MAC operation in which signal voltage is read, see Figure 25 and Eq.97. For example, equation Eq.100 represents a positive contribution. The operation with opposite contribution sign to Eq. 100 is:   izero out in VV C C Eq. 101 Due to the circuit topology used for the implementation of MAC block, the weight or scaled factor used during the consecutive multiplication-accumulation process must remain constant. Under these conditions, the convolution operation can be implemented as the sum of intermediate values resulting from the multiplication-accumulation processes associated to each group of neighbors with the same contribution weight. Figure 30 and Table 7 show the sequence of instructions required to carry out three consecutive multiplication-accumulation operations in the MAC block, described mathematically by:         zerozerozero out in out VVVVVV C C V 321 Eq. 102 s_mac_w w_mac zero r_mac s_mac_r ladn phase1 phase2 phase3 1 V zero V out V 2 V phase1 zero V zero V phase1 phase2 3 V phase2 Figure 30. CONSECUTIVE MULTIPLICATION-ACCUMULATION OPERATION. Nº Instruction r_mac w_mac zero s_mac_w s_mac_r ladn 1 0 0 0 0 0 V1 2 0 1 1 1 0 V1 3 0 1 0 0 0 V1 4 0 0 0 0 0 V1 5 1 0 0 0 0 Vzero 6 1 0 0 1 0 Vzero 7 1 0 0 0 0 Vzero 8 0 0 0 0 0 V2 9 0 1 0 1 0 V2 10 0 1 0 0 0 V2 11 0 0 0 0 0 Vzero 12 1 0 0 0 0 Vzero 13 1 0 0 1 0 Vzero Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 136 14 1 0 0 0 0 Vzero 15 0 0 0 0 0 Vzero 16 0 1 0 1 0 Vzero 17 0 1 0 0 0 Vzero 18 0 0 0 0 0 V3 19 1 0 0 0 0 V3 20 1 0 0 1 0 V3 21 1 0 0 0 0 V3 22 1 0 0 0 1 Vout 23 1 0 0 0 0 Vout 24 0 0 0 0 0 Vout Table 7. Instructions for multiplication-accumulation operation. The signal voltages belonging to the pixel neighbors are connected to the local node ladn through the interconnection bus bv<7:0> which is input of the analogue multiplexer. The analogue multiplexer is controlled by the configuration bus pm<7:0>. The interconnection scheme between the pixel and the neighborhood is shown in Figure 17 The reference voltage zero V is applied to ladn node by the Generic reference block. 3.2.8. DIFFUSION GRID 3.2.8.1. RESISTIVE DIFFUSION GRID A resistive grid connecting pixel is included within the Q-Eye to emulate the implementation of Gaussian filters [Fern12] [Kana07]. Assuming a 2-Dimensional symmetric Gaussian function   yxG D, 2 with variance  , described by the equation:   2 22 2 2 22 1 ,        yx DeyxG Eq. 103 the convolution mask required to implement it is calculated from the impulse response which results into a matrix of parameters whose values depend on  . The larger the variance, the larger the size of the mask needed to accurately implement the filter. In practice the mask size gets truncated to a limited neighborhood. The matrix below displays the matrix values within a radius-2 neighborhood for 1  : 0.0030 0.0030 0.00300.0030 0.0133 0.0133 0.0133 0.0133 0.0219 0.0219 0.0133 0.0133 0.0133 0.0133 0.1621 0.0219 0.0219 0.0596 0.0596 0.05960.0596 0.0983 0.09830.0983 0.0983 Outside this radius, the coefficients are considered negligible. The resistive grid implements Gaussian filters as the outcome of a diffusion process. It is extremely efficient in both speed and power consumption [Fern12]. Assuming the grid driven by a constant input image equal to 0 whose central pixel is 1 (normalized input), the diffusion length diff n is defined as the radius of the circumference, expressed in number of pixels, where the value associated to the pixels are 2 1 below (3dB fall) of the final value for the central pixel. On the one hand, this length depends on the time constant of the RC network implementing the grid. On the other hand, this diffusion length can be shown to be proportional to the variance parameter  :   kndiff Eq. 104 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 137 Hence, the variance and the associated convolution mask can be synthesized by properly setting the RC time constant. Let us consider for simplicity the case of 1-dimensional diffusion network  see Figure 31. Cell R/2 Cmem R/2 R/2 R/2 Cmem Cmem R/2 R/2 R/2 Cmem R/2 n0n1n2ni Figure 31. 1-DIMENSIONAL DIFFUSION NETWORK MODEL. Assume the network is finite, although large, and that a unitary step is applied to one cell, located almost infinitely far away from the end of the network. The time constant for a network of n resistors can be calculated considering the open-circuit time constant approach [Sedr04] 8. In this case, the time constant associated to each node of network from the input node 0 n is given by the equation:   2 1 1     ii CRnCR mem i n memi  Eq. 105 The temporal evolution of a particular node with time constant i  is:   i t out eiV   1 Eq. 106 Consequently, the time required for each node to reach the 2 1 factor of input step is:   2 1 23.123.1 2 1 1ln           ii CRt memiii  Eq. 107 Therefore, the following relation between diffusion length and diffusion time is obtained:     0.615 1 1 diffusion mem diff diff diff diff t R C n n n n             Eq. 108 where  is the constant time of the diffusion network and  is a constant of proportionality. The diffusion grid is composed by a CMOS resistance, and has several configuration and control signals as it is depicted in Figure 32. The control signal s_rg connects the local grid node Nrg to the data node ladn. The control signal rg_me launches the diffusion process; and the configuration signal e_m_rg enables the mask operation during the diffusion process. The resistive grid operation involves three phases:  The first phase is a reset phase of the diffusion grid where all internal nodes of network are fixed to a specific global reference, for instance zero V .  During second phase, the data node ladn is initialized to the input image which will be filtered. The input operand is driven by a local analogue memory.  During third phase, the internal capacitance of local grid node Nrg is connected to the ladn node remaining the local analogue memory node connected in writing configuration to the data node ladn and local grid node. And simultaneously, the diffusion process is launched. When the diffusion process finishes, the resulting filtered image is loaded in this analogue memory. Therefore, the capacitance associated to a particular node of network during the diffusion process is the capacitance of the analogue memory. 8 The dominant high-frequency time constant is estimated as the sum of the individual time constants due to each of the capacitance when all other capacitances are set to zero. To find the individual time constant for a given capacitance, independent voltage sources are replaced with ground (independent current sources are opened), and the resistance seen by the capacitor is determined. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 144 + - Vout Vbottom Cmem Vtop rsw3 gm*vgs vgs vs Co ro + - Vref Vref Vout rsw3 Cmem Vtop Vbottom Cin Cin Figure 41. CIRCUITAL MODEL FOR THE READING PHASE. Let us make the following assumptions:  The ON resistance of SW1 ( 1sw r ) can be neglected 9  2 ,swmo ggg  ( 2 2 1 sw sw r g )  msw gg  2 Then, the dynamic of S&H circuit in the writing phase is described by the following first order transfer function, read Appendix A:   s s g CCC sH wr m inmemo writing       1 1 1 1 Eq. 117 And the acquisition error associated to the temporal response is described by the expression Eq. 118, when a step function starting at 0t and size wr V is applied.   wr t wrwr eVtv     Eq. 118 Regarding the reading phase and assuming that the switch SW3 is designed with impedance such that its effect on the dynamic of circuit is negligible, we obtain the following transfer function: 1 1 1 1 _                    rd mem in m rdeq mem in reading s C C g C s C C H  Eq. 119 The settling error during the reading phase can be calculated in the same way that the acquisition error during the sampling phase is obtained from the transfer function. Considering an input step function starting at 0t and size rd V :   rd t rdrd eVtv     Eq. 120 9 The ON resistance of SW1 can be neglected provided this switch is wide enough. This is actually limited by the injection of its channel charge into the capacitor. However, this charge is basically precluded by turning SW1 OFF only after SW2 is fully OFF. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 145 3.3.1.2. CHARGE INJECTION AND CLOCK FEEDTHROUGH When the switch SW2 in Figure 37 is turned OFF at the end of sampling phase, the voltage stored at the capacitor changes due to two different phenomena:  Channel charge injection; i.e. part of the charge which builds the MOS channel is routed to the sampling capacitor.  Clock feedthrough, i.e. the coupling of the clock control signals through parasitic capacitances into the storing node – see Figure 42. VSS Vref Cmem Vcontrol Vi VSS VDD ON OFF Cgd Figure 42. NMOS SWITCH. CHARGE INJECTION. The amount of charge injected into the memory capacitor mem C depends on the speed and shapes of the clock signal edges and the relative impedances seen at the access transistor terminals. Its exact calculation is extremely complicated or even unfeasible [Wegm87]. It can be approximated by using the following expression for the channel charge of a NMOS transistor while in the ON state:     i SSiTiDDoxch VVVVVLWCQ  Eq. 121 Where   SSiT VVV  models the body-effect:     BSBBTSBTVVVV   0 Eq. 122 0T V is the threshold voltage for 0 SB V ,  is the body-effect constant, B  is the surface potential in strong inversion and SB V is the source-bulk voltage drop. From here, the change in the stored voltage happens to be:     SSiTiDD mem ox ch VVVVV C LWC V    Eq. 123 where  represents the undetermined fraction of the channel charge that is injected into the memory capacitor   10   . For design purposes, the worst case 1  can be considered. This means that 100% of the channel charge gets injected into the memory capacitor. CLOCK FEEDTHROUGH. For the second phenomenon, let us consider the capacitive divider formed by the gate-to-drain overlap parasitic, gd C , and the memory capacitor mem C . When the input signal, i V , is no longer driving the storage node, because the switch have been turned OFF, the control signal, control V may cause some extra voltage degradation by being fed through capacitive coupling to the sampling capacitor [Weiz02]. Once the control signal reaches the level indicated in the following equation:   SSiTicontrol VVVVV  Eq. 124 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 146 The switch turns OFF and the voltage at the memory capacitor does not follow the input signal i V . But the control signal still goes down to the power supply SS V , affecting to the stored voltage through the parasitic capacitance between the gate and the source terminals. This results in an error in the stored voltage that can be expressed by:     SSSSiTi gdmem gd feed VVVVV CC C V   Eq. 125 Both error contributions are signal dependent, and therefore, produce harmonic distortion in the more general case. However, in the actual memory cell used for the Q-Eye, the signal dependence is removed and only offset errors remains thanks to the control applied during the writing phase. The injected charge is basically generated when SW2 turns OFF. SW1 is turned OFF after that SW2 is fully OFF. The voltages at terminals of SW2 transistor remain constant always during writing phase. Therefore, the error introduced by SW2 is signal-independent introducing an offset component. Still these offset errors have to be considered in the design phase because they produce spatial noise in the images (FPN) due to mismatch. In order to reduce this error large capacitance values must be considered. But large capacity values compromises analog circuit speed. Consequently, there is a tradeoff between feedthrough/injection charge error (attainable accuracy) and speed requirements. In addition, large areas associated to the devices of processing cell imply a degradation of resolution and fill factor parameters; therefore there is a tradeoff between area (fill factor, resolution) and accuracy. 3.3.1.3. OFFSET AND FINITE GAIN OF AMPLIFIERS Offset insensitiveness of Figure 37 is valid only into first order approach. Besides the residual offset created by non-ideal switching, the finite OTA DC gain yields incomplete offset voltage cancellation. It can be readily calculated with the help of Figure 43 and yields:   osinout VVV      1 1 Eq. 126 where:         mem in oC C A1 1  Eq. 127 Considering 1  , we obtain:   osinout VVV   1 Eq. 128 Showing that the errors associated to the finite gain and offset of the operational amplifier appear as a gain-error and an offset in the read data. These non-idealities provoke fixed pattern noise in the image stored locally in the memories of cell array, see Section 3.4. + - Vin Vout Vbottom r_lm w_lm slm Cmem Vtop SW1 SW2 SW3 + - Vos Ao Cin Figure 43. NON IDEALITIES OF OPERATIONAL AMPLIFIER. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 147 3.3.1.4. NOISE LIMITATIONS IN THE MEMORIES The electronic noise of passive and active components produces random contributions to the errors in the stored voltages and hence impacts the behavior of the Q-Eye processing cell. Figure 44 illustrates the static models used for the evaluation of each noise source during the writing and reading phases. The ON-resistances of switches SW1, SW2 and SW3 generate a thermal, white noise described respectively by the following Power Spectral Densities (PSD):   1 21_1_ 4SWBSWnSWnTRkvfS  Eq. 129   2 22_2_ 4SWBSWnSWnTRkvfS  Eq. 130   3 23_3_ 4SWBSWnSWnTRkvfS  Eq. 131 Where B k is the Boltzmann’s constant   123 1038.1  JK , T is the absolute temperature (Kelvin) and 1SW R , 2SW R and 3SW R are the on-resistance of switches. + - Vn_sw1 Vout Vbottom Cmem Vtop Rsw1 RSW2 + -vn_amp Cin + - + - Vn_sw2 Cout + -Vout Vbottom Cmem Vtop RSW3 + -vn_amp Cin + - Vn_sw3 Cout Writing phase Reading phase Figure 44. NOISE SOURCES IN ANALOGUE MEMORY PHASES. Regarding the amplifier, its equivalent input noise is modeled by a voltage source ampn v_ at the positive input terminal. This is mostly due to the MOSFETs input transistors and has a thermal and a flicker component. The input-referred thermal noise of a MOSFET transistor in saturation is approximately:    f g TkfS m BMOSwhite 0 1 3 8 _ Eq. 132 In addition, the flicker (or 1/f) noise, which is nearly independent of the bias condition, is approximately given by:    f fWLC K fS ox MOSf 0 1 _/1 Eq. 133 Where K is a process and temperature dependent parameter. The thermal noise PSD function referred to the amplifier input in CMOS technology can be generally described by: Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 148     amp m B ampwhiteampwhite g Tk vfS   1 3 8 2 2__ Eq. 134 m g is the amplifier transconductance and amp  is a factor depending on the particular topology. Both noises (thermal and flicker) are uncorrelated in the same device, so the total noise contribution is given by the direct addition of their PSDs. The total PSD have the generic shape shown in Figure 45. f (log)   fS ampn _ ampflic Sker_ ampwhite S_ k f Figure 45. PSD OF THE AMPLIFIER NOISE. The frequency at which flicker noise is equal to the thermal noise is called corner frequency k f . Below k f , flicker noise dominates while thermal noise dominates beyond k f [Raza01]. Taking this into account, the PSD of amplifier noise, referred to its input, can be written as:   ampfampwhite k ampwhiteampnampn SS f f SvfS _/1__ 2__ 1         Eq. 135 f fS Skampwhite ampf  _ _/1 Eq. 136 Considering the different noise contributions presented above, the error of charge trapped into the memory capacitor at the end of the writing phase is given by:       wr ampampnosSWninmemmem HwrvVvvCwrQ _1_ Eq. 137 Where the wr amp H is the transfer function corresponding to the amplifier in a unit-gain configuration: s H wr wr amp    1 1 Eq. 138 And we have assumed a single-pole model with infinite DC-gain for the amplifier:     0 0 1A s A sHamp  Eq. 139 The noise source 2_ swn v does not affect to the charge stored in the memory capacitor because the Vbottom node is not influenced by this source during writing phase due to the feedback loop configuration. Furthermore, the charge stored in the memory capacitor during reading phase is: Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 149       rdvVvvCrdQampnosSWnoutmemmem _3_  Eq. 140 Taking into account the charge conservation law between the writing and reading phases, the noise at the output of analogue memory during reading phase is described by following equations:     rdQwrQmemmem  Eq. 141       rdv s wrvvvvrdvampn wr ampnSWnSWninout __3_1_ 1 1    Eq. 142 Denoting:     s wrvrdvv wr ampnampnAZn   1 1 ___ Eq. 143 and assuming that all noise sources are no-correlated, the PSD of noise at the output of analogue memory in reading phase is:   2_ 23_ 21_ 2__ AZnSWnSWnoutnoutnvvvvfS  Eq. 144 where, 2_AZn v is the PSD resulting from the auto-zeroing operation. The analogue memory topology implements a basic technique used to reduce the offset associated to the amplifier, the Correlated Double Sampling (CDS) technique. This can be described as an auto-zeroing (AZ) operation followed by a S&H process [Enz96]. The basic idea of AZ process is sampling the unwanted offset and noise and then subtracting it from the instantaneous value of the contaminated signal. AZn v_ represents the resulting noise of AZ operation. The AZ technique [Wieg89] has a baseband transfer function which imposes a zero at the origin of frequency that cancels any offset and strongly reduces the 1/f noise. In addition to this basic high-pass filtering process, if the AZ is a sampling process, the wideband noise of amplifier is aliased down to the baseband, increasing the resulting in-band power spectral density. The resulting PSD   fS AZn_ of the AZ operation can be decomposed into two components: one composed by the baseband noise (which is reduced by the AZ process) and the other by the fold-over components introduced by aliasing associated to the periodic sampling of noise at the input of amplifier to compensate the input offset:         fSfSfHvfS fold sampled ampnAZnAZn _ 2 0 2__ Eq. 145 sampled ampn S_ is the first-order low-pass filtered PSD of input-referred noise of amplifier, when the noise is sampled with the amplifier in a unity-gain configuration:         wr wrc wrc ampn wr ampampn sampled ampn f f f fS fHfSfS              2 1 1 _ 2 _ _ 2 __ Eq. 146 Where wrc f_ is the 3-dB noise bandwidth for the writing configuration. The fold-over component results from the replicas of the original spectrum of noise at the input of amplifier shifted by the integer multiples of sampling frequency of amplifier offset:                s sampled ampn n n nnfold T n fSfHfS _ 0 2 Eq. 147 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 150 Where s T is the offset sampling period, and the square magnitude of the corresponding band transfer function is expressed by:                                   2 2 2 2 2 2cos1 2 2cos1 2 2sin 2 2sin s s s s ndfT dfT dn dn dfT dfT dn dn dfH         Eq. 148 hAZs s hTTT T T d Eq. 149 AZ T is the time interval to sample the amplifier offset and h T is the hold time. The baseband transfer function   2 0fH is given by:                             2 2 2 2 02 2cos1 2 2sin 1 h h h h fT fT fT fT dfH     Eq. 150   fH0 shows a high-pass characteristic. This means that the baseband transfer function imposes a zero at the origin of frequency which cancels the amplifier offset and reduces the 1/f noise. The CDS operation can be described as an AZ operation follow by a sample and hold operation. Therefore, the signal at the output of a circuit using CDS is sampled and held. Despite this, the effect of CDS on the amplifier offset and noise is very similar to that of the AZ operation. The baseband transfer function implements a high-pass filtering operation in the same way the AZ does. On the other hand, although the band transfer functions for 0n are different from those obtained for AZ process, the fold-over component provoked by aliasing is comparable because the wideband noise has already been sampled once. Taking into account that the input-referred noise PSD of amplifier contains both a white and a 1/f noise component,       sampled ampf sampled ampwhiteampampfampwhite sampled ampn SSfHSSfS _/1_ 2 _/1__  Eq. 151 The PSD of resulting voltage from AZ process can be expressed by:       fSfSfS fAZn white AZnAZn/1___  Eq. 152     whitefold sampled ampwhite white AZnSSfHfS __ 2 0_  Eq. 153     ffold sampled ampf fAZnSSfHfS /1__/1 2 0 /1_ Eq. 154 If the amplifier’s broadband white noise sampled ampwhite S_ during the writing phase is considered as an ideally low-pass filtered white noise having an equivalent noise bandwidth wr ampeq BW _ , the aliasing effect introduced by the sampling process implies that the original noise power spectrum is shifted by multiples of sampling frequency and summed, resulting:       sampwhites wrampewhitefold TfcSTBWfS   2 ___ sin12 Eq. 155 The equivalent noise bandwidth associated to the white component of amplifier’s noise sampled ampwhite S_ is defined by: Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 151   wr wrc wrc sampleampwhite ampwhite wr ampeq fdf f f dffS S BW                 4 1 2 1 11 _ 02 _ 2 0 _ 2_ _ Eq. 156 A similar analysis can be developed for the first-order low-pass filtered 1/f noise component sampled ampf S_/1 . The fold-over component for the 1/f noise can be approximated in the Nyquist range by:     sswrcskampwhiteffold TfcTfTfSfS              2 __/1_ sin 3 2 ln12 Eq. 157 The folded-back component of the 1/f noise will be generally submerged by the aliased white noise [Enz96] [Fisc82], [Gobe83], so that it can be neglected for noise computation in most practical cases. Therefore, the PSD of resulting voltage from AZ process is given by:         fSfSfHfS whitefold sampled ampnAZn__ 2 0_  Eq. 158 If the under-sampling factor sampeq TBW  _ 2 is much larger than unit, since the baseband term   2 0fH is bounded by 1.6, the autozero white noise white AZn S_ is dominated by the aliased broadband noise component:       sampwhitesampeqwhitefoldAZnTfcSTBWfSfS   2 ____ sin2 Eq. 159 Considering the effect of CDS operation over the amplifier’s noise and the periodic operation of analogue memory (periodic sampling of offset), the PSD of noise at the output of analogue memory in reading phase is:         fSfSfSfS whitefoldSWnSWnoutn_3_1__  Eq. 160 Under these conditions, the total power of noise at the output of analogue memory is:     dffSBWSBWSdffSP AZnSWeqSWnSWeqSWnoutnout    0 _3_3_1_1_ 0 _ Eq. 161 Taking into account that:   ss sTT dffTc       2 1 2 1 sin 0 2    Eq. 162 The following equation is obtained to describe the total power of noise: wr ampeqampwhiteSWeqSWnSWeqSWnout BWSBWSBWSP __3_3_1_1_  Eq. 163 In general, the particular transfer function of each noise source to the output of circuit has a great complexity which depends on the considered resistive and capacitive parasitic. Moreover, the equivalent bandwidth for each noisy source differs from each other. This complex behavior can be approximated suppressing the impact of the switch on resistance in the determination of the transfer functions. Under this conditions, the system behaves as a single pole transfer function with a common equivalent bandwidth for all the noisy sources. In the case of the analogue memory, the transfer function has a different pole in the writing phase and reading phase:   rd ampeqSWn wr ampeqampwhiteSWnout BWSBWSSP _3___1_  Eq. 164 The equivalent bandwidths for the writing and reading configurations are defined as: Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 152     wreq m wr wr wr amp wr ampeq C g df f dffHBW _ 02 2 0 _44 1 21 1           Eq. 165     rdeq m rd rd rd amp rd ampeq C g df f dffHBW _ 02 2 0 _44 1 21 1           Eq. 166 Consulting the equations Eq. 117 and Eq. 119 the equivalent capacitances are: meminowreq CCCC  _ Eq. 167         mem in oinrdeq C C CCC 1 _ Eq. 168 Therefore, the total power of noise at the output of analogue memory is:   rdeq m SWB wreq m m B ampSWBout C g RTk C g g Tk RTkP _ 3 _ 14 4 43 8 124               Eq. 169 The S&H circuits are design so that their settling is constrained by the amplifier bandwidth and not by the switch resistances. Under this condition, the following approximation can be done:   wreq B ampout C Tk P _ 3 4 1   Eq. 170 This power of noise at the output can be reduced by using a larger sampling capacitor. This is exactly the opposite of what can be made for improving the acquisition time. Therefore, there is a tradeoff between accuracy and operation speed. The random electronic noise is a fundamental parameter in Vision Systems, Cameras and Image sensors because determines the maximum Signal to Noise Ratio (SNR) and Dynamic Range (DR) attainable by the system. In general, like we can observe in the study of random noise in the analogue memory, a tradeoff between power consumption and random noise exits in the design of the mixed-signal blocks which composes the Q-Eye processing cell. In a similar way, there is a tradeoff between area and random noise figure. 3.3.1.5. LEAKAGES IN HOLD MODE The circuit of Figure 37 assumes one amplifier per memory data. Bear in mind, however that in the LAM employed at the Q-Eye the opamp is shared by all the memories in the LAM – see Figure 10. It means that reading does not happen immediately after writing, but the data remain stored until they are read upon demand of the processing algorithm. Figure 46 depicts the hold phase configuration considering only one memory element for simplicity. During the hold period, leakage currents of the switches SW2 and SW4 may degrade the stored voltage. However, the former is not relevant because to bottom node voltage is set by the opamp feedback action. Actually, this switch remains ON to preclude crosstalk between data stored in the analog memories happening due to diffusion leakage associated to the bottom plates of capacitors. Regarding the leakage due to the SW4, it has two components, namely:  The OFF-current due to sub-threshold conduction ( sub I ):          Tk Vq Tk Vq DSsub B GS B GS eeIII 1 0 Eq. 171 It is minimized by applying a reference voltage Vhold to control the voltage drop between the drain and the source terminals. The SW4 switch is implemented by a low threshold voltage transistor in order to optimize the signal range of the processing cell. Therefore its Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 153 sub-threshold conduction is not-negligible. This sub-threshold current depends on drainsource voltage (VDS) too. Then, with the reference voltage Vhold the sub-threshold current is control through the drain-source voltage of SW4 transistor.  The reverse-current ( pn I ) of the reverse-biased junction formed by the n+ diffusion of SW4 and the p-type substrate:         n pn p np pn L nD L pD AqI 00 Eq. 172 Where, q is the charge of an electron ( C 19 10602.1   ), A is the pn-junction area, p D and n D are the diffusion coefficients for holes and electrons, p L and n L their diffusion lengths and 0n p and 0p n are the minority-carrier concentrations in each side of junction. + - Vhold Vout Vbottom r_lm slm sm0 Cmem Vtop SW1 SW2 SW3 Ao SW4 Vin w_lm Figure 46. HOLD PHASE CONFIGURATION OF ANALOGUE MEMORY. The injected charge is given by: offleakleak tIQ  Eq. 173 Where leak I is the leakage current and off t is the time interval during which the current is integrated. This interval is the time during which the selection switch is OFF. The voltage error of the output data during the reading phase generated by the injected charge is: mem offleak mem leak leak C tI C Q V    Eq. 174 During the reading phase, the reverse current generated by the diffusion diode associated to the bottom of the MOS capacitor also impacts on the stored charge. The bottom node is a pure capacitive node which is discharged by this reverse diffusion-diode current of capacitor during the reading configuration, Figure 47. In this case, the integration time of leakage current is the reading time. Since this reading time is small (around 400 nanoseconds), this leakage component is small and negligible. It must be stressed that the most significant impact of leakage is not due to voltage deviations themselves (which if uniform can be corrected through a global offset signal) but by their mismatches, which lead to FPN (spatial noise, see read Section 3.4) - very complicated of compensate. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 256 Registers 3, 4, 5, and 6 (INST_CODE, PAR_1, PAR_2, END) active the state machines which control the load/download processes through the Image Interface of Q-Eye system. The implemented functions which control the communication protocols with Q-Eye system are loadgrey, download-grey, load-bin, download-bin, ae-direct and ae-inverse. These functions control the input and output of images in the corresponding formats: grey images, binary images and coordinates. ADD_WR_PB[11:0] ADD_RD_PB[11:0] 01234567891011 DAT_PB_OUT[9:0] N.U. INST_CODE[2:0] START BIT1BIT2BIT3 N.U. PAR1[11:0] PAR2[11:0] END N.U. NUM_POINTS_AED[11:0] N.U. ROW0_AED[7:0] COL0_AED[7:0] ROW1_AED[7:0] COL1_AED[7:0] N.U. N.U. N.U. ROWINIT[7:0] ROWEND[7:0] N.U. ROWUDCOLLR N.U. DAT_PERIPHERAL[11:0] R/nW ADD_PERIPHERAL[11:0] N.U. DAT_PB_IN[9:0] Figure 12. INTERFACE REGISTERS STRUCTURE. In register 3, bits INST_CODE<2:0> select the communication function to execute, Table.4 describes the codification. BIT1 and BIT2 are bits to configure some parameters of data communication functions together with the registers 4 and 5 (PAR_1, PAR_2). The START bit is a control bit. When the START bit is set to 1 the corresponding control machine of selected function is launched. When the control machine has finished the load/download process, the bit END of register 6 is set to 1 by the controller to indicate the end of process. COMMUNICATION FUNCTION INST_CODE[2:0] LOAD GREY 000 LOAD BIN 001 DOWNLOAD GREY 010 DOWNLOAD BIN 011 ADDRESS EVENT DIRECT 100 ADDRESS EVENT INVERSE 101 Table.4. Communication functions code. Parameters of Q-Eye Image Interface are stored in registers 12 and 13. Through registers 14 and 15 (DAT_PERIPHERAL, ADD_PERIPHERAL), the Q-Eye controller can access to external peripherals. The Image Communication Functions will be described briefly in the following subsections. 5.3.5.1. LOAD/DOWNLOAD GREY These functions developed the load and download processes of grey level images. Table.5 presents the configuration parameters. PARAMETER DESCRIPTION PAR_1[11:0] Number of memory page where the image will be stored or taken BIT1 External or Internal memory is used in load/download process Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 257 BIT2 Inversion of image Table.5. Parameters of grey image Load/Download processes. 5.3.5.2. LOAD/DOWNLOAD BIN These functions developed the load and download processes of binary images. Table.6 describes the configuration parameters. PARAMETER DESCRIPTION PAR_1[11:0] Number of memory page where the image will be stored or taken BIT1 External or Internal memory is used in load/download process BIT2 Inversion of image Table.6. Parameters of binary image Load/Download processes. 5.3.5.3. ADDRESS EVENT DIRECT/INVERSE These functions control the load/download processes of coordinates associated to active points in the Q-Eye processing array. Table.7 defines the configuration parameters. PARAMETER DESCRIPTION PAR_1[11:0] Number of memory page where the image will be stored or taken PAR_2[11:0] Number of points to load or download (zero means process all points given by the Q-Eye system) BIT1 External or Internal memory is used in load/download process BIT2 Set to 1 actives a special download mode Table.7. Parameters of Address Event processes. The data structure in the memory is described in Figure 13. Each coordinate is composed by two numbers (row and column). Two points are stored in a word memory of 32 bits (POINT0 is processed first and POINT1 is second). Figure 13. STRUCTURE OF ADDRESS EVENT DATA. In the special download mode activated by configuration bit BIT2, only two points are downloaded in the registers 8, 9, 10 and 11 (ROW0_AED, COL0_AED, ROW1_AED, COL1_AED). 5.3.6. NIOS-Q-EYE CONTROLLER COMMUNICATION During the execution of a vision application by the Eye-RIS system, a program is executed in the NIOS microprocessor. This program makes call to the Q-Eye controller in order to execute FPP microinstructions for Q-Eye system. Therefore, a communication channel between both blocks, NIOS and Q-Eye controller, is necessary, for control tasks and data exchange (programs and images) see Figure 14. The NIOS-II microprocessor can access in writing and reading mode to several areas of Q-Eye controller related with the execution of FPP microinstruction through the called Configuration Interface. The address map of NIOS has parts associated to the Q-Eye controller: 0 7 15 31 8 COL0[7:0] 16 ROW0[7:0] 23 24 ROW1[7:0] COL1[7:0] POINT 1 POINT 0 Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 258  Internal registers of Q-Eye system  Q-Eye controller registers  Internal memory of Q-Eye controller  Communication registers Programming Interface EN_PB RW_PB ADD_PB<11:0> DAT_PB<9:0> Programming Interface Q-eye system RW_IM START_IM DAT_IM<31:0> END_IM Programming Control Data Interface Control Interface Registers Q-eye controller Microprocessor NIOS Microprocessor Memory Communication Interface Figure 14. CONTROL FLOW OF EYE-RIS SYSTEM. The NIOS microprocessor can reset the Q-Eye controller through one of its registers with Input/Output Ports (PIOs). The communication registers allows the NIOS microprocessor to send instructions to the Q-Eye controller and to receive information. Through these registers, the NIOS microprocessor can send a start instruction for the execution of a micro-program, send the internal address memory of controller from the micro-program will start, receive information about the status of micro-program execution, send control signals to the Q-Eye system (ENG, CLK_CTRL, RESET), receive information related with controller FIFOs, etc. 5.3.7. CONTROLLER MEMORY 5.3.7.1. DATA MEMORY The Q-Eye controller works with an internal data memory consists of 4K words of 12 bits. This data memory contents 32 general purpose registers and the rest of memory corresponds to generic data (global variables, tables, stack, etc.). The internal memory is used by the controller instructions to take parameters or give a resulting data. This memory can be written and read by the NIOS microprocessor during the time intervals when the controller does not execute programs. The data memory of controller can be increased using an external memory which is accessed by special instructions that transfer the data between both memories. The operation is carried out using the internal memory. 5.3.7.2. PROGRAM MEMORY The instructions to execute by the controller are stored in the program memory. This memory can have a variable size, until 4096 pages of 4096 instructions each page. The instructions consists of 24 bits words. The program memory is read by the controller and written and read by the NIOS when the Q- Eye controller does not execute any micro-program. The Q-Eye controller works with this program memory through a register of page selection (PROG_PAGE special register in data memory) and a counter program of 12 bits which indexes the selected page. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 259 5.3.8. CONTROLLER MICROPROCESSOR The microprocessor of Q-Eye controller is responsible for the micro-program execution. This micro-program is composed by the FPP (Focal Plane Processor) instructions which codified the operations of Processing Array. Table.8 describes the main instructions of microprocessor contained in the Q-Eye controller. The Q-Eye controller has 12 generic lines of interruption. Each interruption has associated an interruption vector which indicates the start address of interruption routine, a state bit, enable bit and priority code. Through the registers 14 and 15 (DAT_PERIPHERAL, ADD_PERIPHERAL) contained in the Interface registers the controller configures and controls six timers  see Figure 12. INSTRUCTION DESCRIPTION NOP No operation RETI Interruption routine return JUMP Conditional and unconditional jump CALL Sub-routine call END Program end RET Sub-routine return PUSH Introduction of parameter into the stack POP Extraction of last data in the stack and storing in data memory NOT Not operation SET Introduction of value into the memory data INC Increment operation DEC Decrement operation COMP Comparison operation AND AND operation OR OR operation ADDC Addition operation SUBC Subtraction operation SHIFTR Right shifting operation SHIFTL Left shifting operation DELAY Wait operation LOAD Transfer data from external memory to the internal memory STORE Transfer data from internal memory to the external memory WR_INTREG Write a data into an Interface register RD_INTREG Read a data from an Interface register Table.8. Microprocessor instructions of Q-eye controller. 5.4. THE EYE-RIS_V2 Figure 15 is a copy of the block diagram of Figure 3-bottom that has been also inserted here for easier reading. It shows a block diagram for the system Eye-RIS_v2. As shown in the figure, the Eye-RIS_v2 consists of the so-called Eye-RIS_VSoC and some external memories. The Eye-RIS_VSoC embeds the Q-Eye, the DIP and the Q-Eye controller on a single silicon substrate. The Eye-RIS_VSoC is hence a much more powerful CVIS than the Q-Eye because it is capable of performing both pre-processing and post-processing tasks. Figure 16 shows the placement of the Eye-RIS_VSoC sub-blocks on a silicon substrate corresponding to a standard CMOS 0.18m technology with one poly and six metals. Compactness level achieved by the Eye-RIS_v2 system is larger than the corresponding level of the Eye-RIS_v1. Indeed, the implementation of an external serial flash interface in the Eye- RIS_VSoC allows the Eye-RIS v2 to work in stand-alone mode without requiring external hosts. Obviously, the fact that the RISC microprocessor is implemented using the same thick-line technology that the photo-sensors limits the operating frequency. However, this is not particularly critical since post-processing is less data intensive than pre-processing. Also, operation at lower frequencies means smaller substrate noise. Finally, design is easier because transistor lengths are larger than in main-stream VLSI digital technologies. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 260 Image Acquisition In-pixel Image Processing In-pixel Image Memory CVIS Q-Eye Sensorprocessor Image post-processing, application control and decision making On-chip data and program memory Nios II RISC Microprocessor Eye-RIS VSoC On-chip image memory I/O and Communications Eye-RIS v2 Vision System External image memory External program and data memory Serial Flash Memory I/O Interface Q-eye Controller DIP Figure 15. EYE-RIS_V2 BLOCK DIAGRAM. CMOS imager with per-pixel processors and memory 32-bit RISC microprocessor (NIOS-II) Memory Memory Control unit CVIS Q-eye Figure 16. EYE-RIS VISION SYSTEM ON CHIP LAYOUT. 5.4.1. ARCHITECTURE AND CONTROL OVERVIEW OF TASK SCHEDULING Figure 17 provides larger details of the Eye-RIS_VSoC architecture [Anaf2v1b]. Its design and implementation face similar issues than other embedded system. For example, restrictions regarding available silicon area, pose limitations on the resources that will be implemented in hardware and will be available for application developments. Data and program memories are Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 261 one of these resources, whose dimension is crucial in embedded vision systems. The Eye-RIS VSoC contains on-chip RAMs composed by two sections. One section of 256 Kbytes is devoted to store the program code and data. The other section of 128 Kbytes is used for storing images. Also, the memory map of VSoC is composed by a reserved section where internal registers are mapped and peripheral section dedicated to PIOs used for reading and/or writing the peripherals available in Eye-RIS_v2. Data bus Input Image Flow (F) control Data bus NIOS-II Microprocessor Pre-processed Image Flow (f) f << F Master SPI USB 2.0 DEBUG / JTAG Eye-RIS Vision System on Chip UART serial 2xPWM Slave SPI IRQs Ext 16 x User Definable I/O CLOCK & RESET control clk & reset clk & reset clk SDRAM I/O EPCS I/O clk & reset Q-Eye Controller 128K Program RAM 64K Data RAM control Data bus DIP Q-eye Figure 17. EYE-RIS_VSOC ARCHITECTURE. Eye-RIS_v2 includes memories outside the VSoC, namely SDRAM memories composed by two sections of 16 Mbytes, one dedicated to store the application code and data and other is the image memory. In addition, a Serial Flash memory is included. So, there is a section in the memory map which maps the interface used for controlling this serial flash memory. The Flash memory is non-volatile, and it is mainly used for storing program that will run immediately after power-up. The memory map is described in Figure 18 the size of on-chip memories can be compared with the size of outside chip memory, being confirmed that the amount of memory inside chip has to be considerably less due to physical size constraints. The Eye-RIS_VSoC is characterized by a co-processing architecture, where the microprocessor works in conjunction with the dedicated hardware (Q-Eye and DIP) to deliver a specific application. In this case, the application-specific hardware can operate concurrently with software execution performed by NIOS-II microprocessor. In this context, critical points to be treated in the definition of task scheduling during the design phase of a vision system and application developments are:  The concurrent operation of dedicated processors (Q-Eye and DIP) and the generalpurpose microprocessor (NIOS-II).  Timing constraints which are common in high-speed applications.  Task execution with limited resources. As explained in Chapter 4, the Q-Eye is a programmable sensor-processor capable to execute FPP microinstructions. This set of micro instructions give arise to the programming language of Q-Eye, denoted by CFPP code [Anaf2v1a]. From a software point of view, this language has involved the development of a specific compiler. CFPP code is structured according to the two concepts described below:  Functions. A function in CFPP code is the same as a function in any other kind of programming language. It is a set of statements of CFPP code that implements a certain algorithm and can optionally receive arguments and/or return a value as a result of its execution.  Sections. A section in CFPP code is similar to a function. But the main difference is that it can be called by an external host (in this case, the NIOS-II processor). This makes sections the entry points from the host. They can be seen as the points through which the NIOS-II can access the Q-Eye services. Sections cannot call other sections and cannot Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 262 be called from CFPP code. Therefore, sections are only called by the NIOS-II microprocessor. The execution flow in Eye-RIS systems is controlled by the NIOS-II microprocessor. This means that the Q-Eye and DIP operate under direct control of the NIOS-II microprocessor through the Q-Eye controller. The execution flow is defined by the code executed within the NIOS-II microprocessor. It decides which section is being executed at all times, although inside a section the flow is determined by its CFPP code. Eye-RIS External Image memory Nios II External Program and Data memory 0x02000000 0x03000000 0x03FFFFFF Peripherals SDRAM Program (16 Mbytes) Serial Flash (8 Mbytes) SDRAM Images (16 Mbytes) Nios II stand alone code Internal Use Binary Images Grey Images User Data 0x00000000 0x00100000 0x00200000 0x00700000 0x00790000 Serial Flash Controller 0x00000000 0x00000800 0x01280000 Reserved Eye-RIS On-chip Image memory Nios II On-chip Prog and Data memory 0x01400000 0x0141FFFF 0x01500000 0x0153FFFF Reserved Reserved Figure 18. EYE-RIS_V2 MEMORY MAP. The NIOS-II microprocessor does not have to stall while the Q-Eye is operational in order to achieve the maximum degree of parallelism in the processing for high speed-applications. When the microprocessor does not stall, some mechanism of synchronization is necessary. This synchronization can be implemented through proper interruption procedures. The use of interruptions between NIOS and Q-Eye controller allows synchronizing both processors while working in parallel. For example, a special function called nioscall has been developed for FPP code. With this function in the FPP code, the Q-Eye controller can send an interruption to the NIOS-II processor, which interrupts the task in progress to develop a new task required by the Q-Eye controller. During the time required for NIOS to perform this new task, the FPP and its controller remain waiting until NIOS automatically returns the task result to Q-Eye controller and allows the FPP to continue its work. At this point the NIOS processor goes on with the preplanned tasks. Figure 19 depicts the procedure described previously. In case interruptions are not employed, parallelism is achieved in different way. Namely, the NIOS launches the execution of a particular task in the FPP Q-Eye using an asynchronous section which allows the NIOS to continue execution of its code. Later, when the NIOS-II has completed its task or estimates that FPP has finished the algorithm, the NIOS takes the result of FPP algorithm. Accordingly, to ensure proper synchronization in this procedure the application developer must have a thorough knowledge of the execution time of the FPP task, in order to know when it has finished, collect the results and launch new tasks. It is very difficult for the application developer to optimize the parallelism level in co-processing with this method. From user point of view, one of the most important characteristics of a vision system is the application development software. The software architecture is closely related to the architecture of the associated hardware. The application development environment is a dedicated piece of software conceived for writing, compiling and debugging programs in order to facilitate the vision application development. Usually, the development environment has a set Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 263 of libraries including functions which avoid that the user has to work at instruction level with the system processors. NIOS II microprocessor FPP+DIP (Q-eye Cotroller) Start execution NIOS program FPP section 1 NIOS program Start execution Section 1 NioscallInterruption FPP task Wait NIOS program Send result Re-start Section 1 Figure 19. PARALLEL OPERATION BASED ON INTERRUPTIONS MANAGEMENT. Extended Image Processing Library Adaptive and HDR acquisition, Blob Management, classification, affine transformations, ... Image Processing Library Image subtraction and addition, Filters, Logical and Morphological operations, ... Eye-RIS Basic Library SIS Control, Display and Save images, Timers Management, ... Standard Library Timers and interrupt management Nios II Processor CVIS Q-Eye Nios II C/C++ CFPP Code Eye-RIS Vision Application Figure 20. SOFTWARE ARCHITECTURE OF EYE-RIS SYSTEM. In the case of Eye-RIS system, there are two processors working in parallel. Therefore, the system is programmed combining two programming languages: C/C++ code for the NIOS microprocessor and CFPP code for Q-Eye. Each operation or task belonging to earlyprocessing has associated a set of Q-Eye microinstructions which constitute a CFPP function. Among the CFPP functions there are some functions or processing operations (reading/writing memories, arithmetic operation, diffusion, combinational operation, etc.) from which image processing functions (filters, morphological operations, adaptive thresholding, etc.) can be developed. Likewise, the post-processing tasks are C/C++ functions which will be developed by the microprocessor. Also, there are low level tasks (instruction level) for the NIOS microprocessor such as execution control, data transmission from PC, IRQ management, timing functions, communications ports operation, etc. and for the Q-Eye controller microprocessor (IRQ management and timing functions) which are composed by a set of instructions. The set of instructions associated to Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 264 each low level task has been implemented by a specific function, simplifying the programming for the user. All these functions that define the image processing tasks are organized into libraries and facilitate the programming of vision system. Although, The Eye-RIS system can be programmed using plain CFPP and C++ codes instead of using this library functions. The library organization is hierarchical, following the abstraction level associated to each task. Figure 20 depicts the programming model of Eye-RIS systems. 5.4.2. PRE-PROCESSING AND POST-PROCESSING IMPLEMENTATION ISSUES As explained in different parts of this monograph, the architecture of the Q-Eye is specifically conceived to execute pre-processing vision tasks involving regular computational flows consisting of a set of operations that are performed repeatedly. However, the correspondence between dedicated hardware and pre-processing algorithms is not so directly if the algorithm must adapt to the application conditions. Examples of these adaptive algorithms are found in the last section of this chapter. Adaptive algorithms are characterized by irregular flow involving data-dependent decisions which are unpredictable. Handling this unpredictability calls for flexibility and programmability of the processor. However, both the Q-Eye and the DIP are pure co-processors that perform only mathematical operations and that cannot make decisions themselves. Therefore, a microprocessor or CPU must be involved in the implementation of such algorithms. In the case of Eye-RIS systems, the NIOS microprocessor is involved in these adaptive algorithms. Seeking for maximum parallelism in pre-processing and post-processing tasks, which minimizes the processing time for high speed applications, a microprocessor has been introduced in the Q-Eye controller (read corresponding sections of the Eye-RIS_v1 description in this Chapter). This way, the application-specific hardware composed by the Q-Eye, the DIP and the Q-Eye controller can operate in parallel with the NIOS, thus optimizing the processing time. For very high speed applications, not only the parallelism between pre-processing and postprocessing is important, but to get parallelism or concurrency between the sensing process and processing is critical. A pipeline approach, where the optical sensor works in parallel with the circuitry responsible for early-processing and the microprocessor in charge of post-processing tasks, achieves maximum frame rate during vision system operation. Figure 21describes graphically this concept. Sensing #1 Sensing #2 Sensing #3 Sensing #4 Pre-processing #1 Pre-processing #2 Pre-processing #3 Post-processing #1 Post-processing #2 Figure 21. PIPELINE ARCHITECTURE. In a pipeline structure where several processors are working in parallel with limited resources, the task scheduling becomes a critical topic. Regarding this issue, the IRQ (Interruption ReQuest) management gets to play an important role in synchronizing different process running in parallel. An interesting example of this fact is the parallelization in the smart sensor (Q-Eye) of sensing and pre-processing performed by the circuitry included in the pixel. This functionality is achieved thanks to the IRQ management performed by the microcontroller included in the Q- Eye controller. Several functions in CFPP code have been developed for the Q-Eye controller in order to implement interrupt-based acquisition of images. This set of functions allows parallelizing the acquisition of images with other processing in the Q-Eye. Figure 22 describes the interrupt-based acquisition. Once the interrupt-based acquisition mode is enabled by a CFPP function, the exposition is configured and launched by another one. At this instant, the sensor starts the acquisition and immediately after that, the execution flow continues with the next instruction. In this way, acquisition and processing can be performed in parallel. Once the specified exposure time has elapsed, the acquisition of image is finish and other CFPP function synchronizes the end of this exposition and the processing operations that are being carried out in parallel. The latency required to attend an interrupt and save the acquired image slightly depends on the CFPP code being executed. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 265 Enable IRQ-based acquisition Configure exposition Start exposition Processing Sensing Synchronization end Figure 22. INTERRUPTION-BASED ACQUISITION. 5.4.3. EXTERNAL INTERFACES Like any embedded vision system, communication interfaces are important system characteristics that application developers must consider.The Eye-RIS_VSoC includes the following Input/Output Interfaces:  A JTAG Debug Module to load and execute programs on NIOS-II microprocessor, pause them, analyze trace data (internal registers and memories), etc.  An USB 2.0 port mainly used for fast exchange of images between the Eye-RIS vision system and PC.  A 32MB SDRAM interface to expand the memory map.  An external Serial Flash interface.  A set of I/O peripherals: o 1 UART o 1 SPI (Serial Port Interface) Master o 1 SPI slave o 2 PWMs o 1 Rising Edge Interrupt input pin o 1 Falling Edge Interrupt input pin o GPIO[15:0], 16 software-programmable general purpose digital I/O lines. Due to the limited number of pins available on the package in order to reduce cost, the UART, SPI, PWM and interrupt ports are multiplexed with the GPIO. Those interfaces that have not been presented before will be described in following subsections. 5.4.3.1. USB 2.0 A USB 2.0 Interface is connected to the PC through a QuickUSB module [USBa-c]. QuickUSB is based on Cypress EZ-USB FX2LP microcontroller [Anaf2v1b]. The interface implemented in the Eye-RIS_VSoC in order to communicate with Cypress EZ-USB FX2LP microcontroller is described in Figure 23. An additional function of USB interface is to feed the system when connected to a host that provides power (for example a PC). 5.4.3.2. EXTERNAL SDRAM INTERFACE The Eye-RIS_VSoC has a dedicated interface to connect a 32Mbytes external SDRAM. This memory is dedicated to store the application code running on NIOS-II and images. The lower 16Mbytes are reserved to store the NIOS-II code whereas the upper 16Mbytes are available to store up to 440 grey images and 512 binary images, see Figure 24. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 272 In order to guarantee minimum accuracy level, the elements of template must belong to the interval   8,8 . Thus, a global pre-scale is applied to the input image to preclude saturations during the operation development. In this case the allowed coefficients are given by:     1,1 , 1,2,4,8 ppq q       Eq. 372 However, this does not limit the variety of filters that can be implemented in the Q-Eye. Same examples are listed in the next paragraphs. SHARPEN FILTER. This filter is commonly used for image enhancement and produces a result equivalent to that obtained by applying a Laplacian filter to the input image and adding the latter to the result. Usual convolution masks which for this filter are: 12 0 1 0 1 1 1 1 5 1 1 9 1 0 1 0 1 1 1 TT                                   Eq. 373 where the template 2 T can be expressed as a combination of two templates: 2 1 1 1 0 0 0 1 8 1 0 1 0 1 1 1 0 0 0 T                           Eq. 374 Figure 29 shows the results obtain by the Q-Eye implementing the convolution mask 2 T . Input image Filtered image Figure 29. SHARPEN FILTER DEVELOPED BY THE Q-EYE. LAPLACE FILTER. This function performs a high-pass filter over the input image. It can be implemented by the convolution masks: 12 0 1 0 1 1 1 1 4 1 1 8 1 0 1 0 1 1 1 TT                                   Eq. 375 This filter is applied for detecting edges. It is a discrete formulation of the second-derivative, isotropic Laplacian operator [Gonz15]. Figure 30 represents the results obtained by the convolution mask 2 T implemented by the Q-Eye. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 273 Input image Filtered image Figure 30. LAPLACIAN FILTERED IMPLEMENTED IN THE Q-EYE SYSTEM. SOBEL FILTER. This other high-pass filter is applied to detect directional edges. It is a discrete formulation of the first-derivative operator, gradient [Gonz15]. Convolution masks to detect edges in different directions are defined by the following templates: 12 1 0 1 1 2 1 2 0 2 0 0 0 1 0 1 1 2 1 TT                                   Eq. 376 34 0 1 2 2 1 0 1 0 1 1 0 1 2 1 0 0 1 2 TT                                   Eq. 377 Templates 1 T and 2 T detect edges in the horizontal and vertical directions, while templates 3 T and 4 T find edges in diagonal directions. Figure 31 shows the results obtained by the Q-Eye. Input image Filtered image Figure 31. SOBEL FILTER IMPLEMENTED IN THE Q-EYE SYSTEM. PREWITT FILTER. It is another approximation to the first-derivative, applied to find edges. Convolutions masks that define this filter are represented by 1 T and 2 T . 12 1 1 1 1 0 1 0 0 0 1 0 1 1 1 1 1 0 1 TT                             Eq. 378 Figure 32 presents the results obtained by the implementation of a vertical Prewitt filter in the Q- Eye processor. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 274 Input image Filtered image Figure 32. PREWITT FILTER CARRIED OUT BY THE Q-EYE. LOW-PASS FILTERING THROUGH DIFFUSIONS. The processing array of Q-Eye includes a diffusion grid which allows implementing Gaussian filters with the sigma parameter defined by the diffusion time. Figure 33 shows the result obtained by the Q-Eye system when a diffusion process with 0.4   is carried out. A remarkable characteristic of this diffusion operation is that can be masked, only pixels point out by a binary image defined as a mask are updated. Figure 34 presents an example carried out by the Q-Eye where a diffusion mask is defined. Input image Filtered image Figure 33. GAUSSIAN FILTER IMPLEMENTED BY Q-EYE. Input image Diffusion mask Processed image Figure 34. MASKED DIFFUSION PROCESS. HIGH-PASS FILTERING THROUGH DIFFUSIONS. Diffusive processes implement low-pass filtering functions. High-pass and band-pass filters can be implemented by combining low-pass filtering and arithmetic operations. When the accuracy of output image is not guaranteed due to this image results from the application of convolution mask with a high number of non-zero coefficients (which provokes an accumulation of errors associated to the sequence of analog operations), the use of Gaussian filters in combination with arithmetic operations lead to very efficient filter implementation in terms of speed, accuracy and power consumption. Figure 35 shows the combined use of arithmetic operations and Gaussian filters to implement a high-pass filter in the Q-Eye. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 275 Input image Low-pass filter High-pass filter Figure 35. HIGH-PASS FILTER BASE ON LOW-PASS FILTER AND SUBSTRACTION. The high-pass filtered image is the result of subtraction of the input image and low-pass filtered version. 5.5.3. EXTRACTION OF IMAGES FEATURES AND CALCULATION OF INTERMEDIATE IMAGE REPRESENTATIONS In the vision processing chain, feature detection is typically followed by segmentation. An important task for segmentation is thresholding. The outcome of thresholding operation is a binary image from which the useful information can be extracted by applying combinational and morphological functions. There are several thresholding operations available in the Q-Eye – described in subsequent paragraphs. 5.5.3.1. THRESHOLDING GLOBAL THRESHOLD: This function is mathematically described by:     1, ( , ) 0, in out in im i j th im i j im i j th       Eq. 379 This threshold operation is global, all pixels are compared with the same threshold value ( th ). Figure 36 presents a global thresholding developed by the Q-Eye. Input image Thresholded image Figure 36. GLOBAL THRESHOLD IMPLEMENTED BY THE Q-EYE. ADAPTIVE GLOBAL THRESHOLD: In this case, the global operation employs a threshold value calculated at the outcome of an adaptive algorithm with the following steps:  An initial estimate for threshold reference ( th ) is made. Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 276  The image is segmented using th. Two sets of pixels are created as a result, 1 S representing the pixels whose grey level is above th and 2 S representing the pixels whose grey level is below th .  The mean value of the two sets of pixels, 1 M for 1 S and 2 M for 2 S , are calculated.  A new threshold is calculated:   12 1 2 th M M   Eq. 380  Successive iterations are made until the value of th is smaller than a specific tolerance. Figure 37 shows the results of applying this adaptive threshold operation. Input image Thresholded image Figure 37. ADAPTIVE GLOBAL THRESHOLD IN THE Q-EYE. LOCAL THRESHOLD: This threshold operation is represented by:         1 , , ( , ) 0 , , in out in im i j th i j im i j im i j th i j       Eq. 381 In this case, the threshold value is not a global constant for all pixels. Instead, each pixel is compared to a different value. This operation is an extremely efficient operation as it is performed simultaneously in all pixels. Figure 38 presents the binary image obtained in the Q-Eye system when the threshold image is obtained by applying a Gaussian filter to the input image. This Gaussian filter has to be associated a high  parameter in order to obtain the information contained in the low frequency domain to calculate the local threshold. Input image Thresholded image Threshold image Figure 38. LOCAL THRESHOLD IMPLEMENTED BY Q-EYE. ADAPTIVE LOCAL THRESHOLD: The threshold image considered to carry out the local threshold is calculated by statistically examining the intensity values of a local neighborhood around each pixel. In this case, a spatial low-pass filter of the local intensity distribution is performed over the input image. The size of the neighborhood (or, in other words, the bandwidth of the low-pass Diseño CMOS de un Sistema de Visión “on-Chip” para Aplicaciones de muy Alta Velocidad On Chip CMOS Vision Systems for High-Speed Applications 277 filter) has to be large enough to cover sufficient foreground and background pixels; otherwise a poor threshold will be obtained. Figure 39 shows an example of adaptive local threshold performed by Q-Eye. 5.5.3.2. BINARY OPERATIONS The binary images obtained after the thresholding operation are processed by binary functions with the main objective of selecting the objects of interest in a scene depending on their topological features. Binary operations are more robust than MAC and other analogue operations, because the result is always regenerated. Thus, a large number of operations can be concatenated without compromising accuracy. Input image Low-pass filtered image Thresholded image Figure 39. ADAPTIVE LOCAL THRESHOLD. The Q-Eye pixel includes circuitry for parallel implementation of the following binary functions:  Logic operations: not, and, or, xor, nand, nor, nxor, etc.  Generic hit and miss operation  Set of binary morphological operations  Set of flooding functions allowing a basic management of blobs and holes. HIT-AND-MISS OPERATIONS. They look for a given pattern in a binary image. The pattern is specified by a 3 x 3 matrix where each element can be 1, or equivalently ‘W’ (white), DNC (Do Not Care) or 0, ‘B’ (black). For illustration purposes, Figure 40 describes the patterns required to look for bottom-right corners in an image. Figure 40. BOTTOM-RIGHT CORNER PATTERNS. These patterns are represented by the matrix: DNC W B W W B B B B      Eq. 382 Figure 41 shows the outcome of applying the hit-and-miss operation described by this pattern to the image shown at the left. Note that white pixels in the processed image determine the points where a corner meeting the specified pattern has been found. [Document text truncated for crawler view.]