Proposal of Architecture and Circuits for Dynamic Range Enhancement of Vision Systems on Chip designed in Deep Submicron Technologies

Propuesta de Arquitectura y Circuitos para la Mejora del Rango Din´ amico de Sistemas de Visi´ on en un Chip dise˜ nados en Tecnolog´ ıas CMOS profundamente Submicrom´ etricas Proposal of Architecture and Circuits for Dynamic Range Enhancement of Vision Systems on Chip designed in Deep Submicron Technologies Sonia Mar´ ıa Vargas Sierra Directors: PhD. Gustavo Li˜ n´ an Cembrano, PhD. Elisenda Roca Moreno Departamento de Electr´ onica y Electromagnetismo Universidad de Sevilla Instituto de Microelectr´ onica de Sevilla (IMSE-CNM) Consejo Superior de Investigaciones Cient´ ıficas (CSIC) A thesis submitted for the degree of Doctor of Philosophy 2012 ii To David and my family Acknowledgements First, I would like to thank both my thesis directors, PhD. Gustavo Li˜ n´ an Cembrano and PhD. Elisenda Roca Moreno, for the help, encourage and comprehension during difficulties. I want to express my love and gratitude to my boyfriend David for believing in me much more that I will ever do, and his understanding of how important this work has been for me. I am so grateful to my beloved family, whose unconditional support, understanding and love have always help me to seek my dreams. I want to show my appreciation to Prof. Bedrich J. Hosticka and his group of the Fraunhofer Institute for Microelectronic Circuits and Systems (IMS), for accepting me to stay with them during three months, and for their nice treatment during that term. I am thankful for all the words of encouragement and emotional support from colleagues and friends that I have received. Especially, I would like to thank all the colleagues, that during lunches and breakfasts, have accompanied me during all these years. Summarizing, I would like to thank all of those who contributed to the success of this work in one way or another. Resumen Una c´ amara es esencialmente un dispositivo para capturar escenas. La informaci´ on sobre estas escenas se mide mediante la captura de luz, tanto emitida como reflejada por los objetos. Tradicionalmente, la fotograf´ ıa ha intentado capturar representaciones de las escenas tal y como lo har´ ıa el ojo humano. Esto significa capturar los fotones en la banda del llamado espectro visible (t´ ıpicamente comprendido entre longitudes de onda de 380 a 780 nan´ ometros). El m´ etodo empleado para medir la luz ha evolucionado desde los procesos qu´ ımicos usados al comienzo de la fotograf´ ıa, hasta los procesos puramente electr´ onicos utilizados hoy en d´ ıa, gracias a la aparici´ on de los sensores electr´ onicos de imagen. Desde su invenci´ on en la d´ ecada de los 60 en el siglo pasado, y especialmente en la primera d´ ecada de ´ este, las prestaciones de los sensores electr´ onicos de imagen han evolucionado de una forma vertiginosa. Esto ha sido motivado, especialmente en los ´ ultimos a˜ nos, por un mercado que crece de manera exponencial. No obstante, las c´ amaras de hoy en d´ ıa a´ un sufren de algunas limitaciones. Una c´ amara ideal ser´ ıa aquella en que las im´ agenes captadas tengan muy poco ruido, un rango din´ amico excepcional (o al menos el m´ aximo rango din´ amico encontrado en escenas reales), gran n´ umero de p´ ıxeles, capturadas con un sensor de alta sensibilidad, en un sistema de muy bajo consumo de potencia y todo ello mientras se mantiene un coste de fabricaci´ on bajo. Aunque en los ´ ultimos a˜ nos se han realizado grandes esfuerzos hacia la mejora de muchas caracter´ ısticas, como por ejemplo, el incremento del n´ umero de p´ ıxeles (sensores de muchos megap´ ıxeles [1]), la reducci´ on del coste, la minimizaci´ on del consumo de potencia y disminuci´ on del ruido (con ruidos de lectura por debajo de un electr´ on rms [2] en sensores de ´ ultima generaci´ on), otras, como es el caso del rango din´ amico, han experimentado un desarrollo m´ as lento. La tendencia evolutiva dictada por los planes de marketing de la mayor´ ıa de compa˜ n´ ıas es generalmente incrementar el n´ umero de p´ ıxeles mientras se mantiene el tama˜ no f´ ısico del chip. Inevitablemente, esto lleva a una reducci´ on del tama˜ no del p´ ıxel (en muchos casos por debajo del l´ ımite de difracci´ on, llamado Disco de Airy). En consecuencia, esto provoca una reducci´ on casi continua del m´ aximo n´ umero de cargas fotogeneradas que pueden ser detectadas y almacenadas a nivel de p´ ıxel (reducci´ on de la llamada Full Well Capacity), lo que produce una reducci´ on, en la mayor´ ıa de los casos, del rango din´ amico alcanzable. En la actualidad, existen dos alternativas tecnol´ ogicas en el dise˜ no de sensores electr´ onicos de imagen para sensar luz en el espectro visible, los denominados Dispositivos de Carga Acoplada (CCD) y los circuitos Complementary Metal Oxide Semiconductor (CMOS), los cuales han coexistido durante aproximadamente 50 a˜ nos. En un principio, los sensores de imagen CCD fueron la opci´ on principal para la mayor´ ıa de las aplicaciones de imagen debido a sus superiores caracter´ ısticas de ruido, sensibilidad y tambi´ en eficiencia en la transferencia de carga. Sin embargo, los sensores de imagen CMOS han experimentado un gran impulso desde finales del siglo pasado. Adem´ as, se han desarrollado tecnolog´ ıas CMOS especialmente dedicadas para la detecci´ on de luz visible, lo que hace que tengan una calidad de imagen comparable a la de los dispositivos CCD. Estas tecnolog´ ıas, usualmente conocidas como tecnolog´ ıas CMOS de sensores de imagen o tecnolog´ ıas CIS, han aumentado el n´ umero de potenciales aplicaciones para los sensores de imagen CMOS. Hoy en d´ ıa, los sensores de imagen CMOS han copado la mayor´ ıa del mercado, b´ asicamente debido a la posibilidad de a˜ nadir etapas de acondicionamiento de se˜ nal (o incluso procesamiento de se˜ nal) en el mismo chip, permitiendo dimensiones muy reducidas para ser insertadas, por ejemplo, en un tel´ efono m´ ovil, as´ ı como debido al reducido consumo de potencia en comparaci´ on con los dispositivos CCD. El trabajo presentado en esta tesis trata de proponer nuevas t´ ecnicas para la expansi´ on del rango din´ amico en sensores electr´ onicos de imagen. En este caso, hemos dirigido nuestros estudios hacia la posibilidad de proveer dicha funcionalidad en un solo chip. Esto es, sin necesitar ning´ un soporte externo de hardware o software, formando un tipo de sistema denominado Sistema de Visi´ on en un Chip (VSoC). El rango din´ amico de los sensores electr´ onicos de imagen se define como el cociente entre la m´ axima y la m´ ınima iluminaci´ on medible. Para mejorar este factor surgen dos opciones. La primera, reducir la m´ ınima luz medible mediante la disminuci´ on del ruido en el sensor de imagen. La segunda, incrementar la m´ axima luz medible mediante la extensi´ on del l´ ımite de saturaci´ on del sensor. Cronol´ ogicamente, nuestra primera opci´ on para mejorar el rango din´ amico se bas´ oen reducir el ruido. Varias opciones se pueden tomar para mejorar la figura de m´ erito de ruido del sistema: reducir el ruido usando una tecnolog´ ıa CIS o usar circuitos dedicados, tales como calibraci´ on o auto cero. Sin embargo, el uso de t´ ecnicas de circuitos implica limitaciones, las cuales s´ olo pueden ser resueltas mediante el uso de tecnolog´ ıas no est´ andar que est´ an especialmente dise˜ nadas para este prop´ osito. La tecnolog´ ıa CIS utilizada est´ a dirigida a la mejora de la calidad y las posibilidades del proceso de fotosensado, tales como sensibilidad, ruido, permitir imagen a color, etc´ etera. Para estudiar las caracter´ ısticas de la tecnolog´ ıa en m´ as detalle, se dise˜ n´ o un chip de test, lo cual permite extraer las mejores opciones para futuros p´ ıxeles. No obstante, a pesar de un satisfactorio comportamiento general, las medidas referentes al rango din´ amico indicaron que la mejora de ´ este mediante s´ olo tecnolog´ ıa CIS es muy limitada. Es decir, la mejora de la corriente oscura del sensor no es suficiente para nuestro prop´ osito. Para una mayor mejora del rango din´ amico se deben incluir circuitos dentro del p´ ıxel. No obstante, las tecnolog´ ıas CIS usualmente no permiten nada m´ as que transistores NMOS al lado del fotosensor, lo cual implica una seria restricci´ on en el circuito a usar. Como resultado, el dise˜ no de un sensor de imagen con mejora del rango din´ amico en tecnolog´ ıas CIS fue desestimado en favor del uso de una tecnolog´ ıa est´ andar, la cual da m´ as flexibilidad al dise˜ no del p´ ıxel. En tecnolog´ ıas est´ andar, es posible introducir una alta funcionalidad usando circuitos dentro del p´ ıxel, lo cual permite t´ ecnicas avanzadas para extender el l´ ımite de saturaci´ on de los sensores de imagen. Para este objetivo surgen dos opciones: adquisici´ on lineal o compresiva. Si se realiza una adquisici´ on lineal, se generar´ an una gran cantidad de datos por cada p´ ıxel. Como ejemplo, si el rango din´ amico de la escena es de 120dB al menos se necesitar´ ıan 20-bits/p´ ıxel, log2(10120/20)=19.93, para la representaci´ on binaria de este rango din´ amico. Esto necesitar´ ıa de amplios recursos para procesar esta gran cantidad de datos, y un gran ancho de banda para moverlos al circuito de procesamiento. Para evitar estos problemas, los sensores de imagen de alto rango din´ amico usualmente optan por utilizar una adquisici´ on compresiva de la luz. Por lo tanto, esto implica dos tareas a realizar: la captura y la compresi´ on de la imagen. La captura de la imagen se realiza a nivel de p´ ıxel, en el dispositivo fotosensor, mientras que la compresi´ on de la imagen puede ser realizada a nivel de p´ ıxel, de sistema, o mediante postprocesado externo. Usando el postprocesado, existe un campo de investigaci´ on que estudia la compresi´ on de escenas de alto rango din´ amico mientras se mantienen los detalles, produciendo un resultado apropiado para la percepci´ on humana en monitores convencionales de bajo rango din´ amico. Esto se denomina Mapeo de Tonos (Tone Mapping) y usualmente emplea solo 8-bits/p´ ıxel para la representaciones de im´ agenes, ya que ´ este es el est´ andar para las im´ agenes de bajo rango din´ amico. Los p´ ıxeles de adquisici´ on compresiva, por su parte, realizan una compresi´ on que no es dependiente de la escena de alto rango din´ amico a capturar, lo cual implica una baja compresi´ on o perdida de detalles y contraste. Para evitar estas desventajas, en este trabajo, se presenta un p´ ıxel de adquisici´ on compresiva que aplica una t´ ecnica de mapeo de tonos que permite la captura de im´ agenes ya comprimidas de una forma optimizada para mantener los detalles y el contraste, produciendo una cantidad muy reducida de datos. Las t´ ecnicas de mapeo de tonos ejecutan normalmente postprocesamiento mediante software en un ordenador sobre im´ agenes capturadas sin compresi´ on, las cuales contienen una gran cantidad de datos. Estas t´ ecnicas han pertenecido tradicionalmente al campo de los gr´ aficos por ordenador debido a la gran cantidad de esfuerzo computacional que requieren. Sin embargo, hemos desarrollado un nuevo algoritmo de mapeo de tonos especialmente adaptado para aprovechar los circuitos dentro del p´ ıxel y que requiere un reducido esfuerzo de computaci´ on fuera de la matriz de p´ ıxeles, lo cual permite el desarrollo de un sistema de visi´ on en un solo chip. El nuevo algoritmo de mapeo de tonos, el cual es un concepto matem´ atico que puede ser simulado mediante software, se ha implementado tambi´ en en un chip. Sin embargo, para esta implementaci´ on hardware en un chip son necesarias algunas adaptaciones y t´ ecnicas avanzadas de dise˜ no, que constituyen en s´ ı mismas otra de las contribuciones de este trabajo. M´ as a´ un, debido a la nueva funcionalidad, se han desarrollado modificaciones de los t´ ıpicos m´ etodos a usar para la caracterizaci´ on y captura de im´ agenes. Esta memoria de tesis doctoral est´ a organizada en cinco cap´ ıtulos. El cap´ ıtulo 1 describe los principales conceptos referentes a los sensores de imagen enfocados al alto rango din´ amico. El cap´ ıtulo 2 presenta el estudio de una tecnolog´ ıa CIS para su consideraci´ on para t´ ecnicas de mejora del rango din´ amico. El cap´ ıtulo 3 describe el nuevo algoritmo de mapeo de tonos desarrollado para optimizar la compresi´ on de imagenes capturadas desde escenas de alto rango din´ amico. El cap´ ıtulo 4 presenta el chip que se ha dise˜ nado y fabricado, el cual incorpora un sensor de imagen que utiliza el nuevo algoritmo de mapeo de tonos. El cap´ ıtulo 5 describe los resultados experimentales y el comportamiento del chip. Finalmente, se detallan las conclusiones y el trabajo futuro. [1] Productos de la compa˜ n´ ıa Samsung: Sensor de Imagen CMOS S5K2P1, 16-Megap´ ıxel y tecnolog´ ıa con p´ ıxel de iluminaci´ on posterior de 1.34 μs. [2] Publicaciones de la compa˜ n´ ıa Caeleste: “A 0.5 noise electrons CMOS pixel”, por Bart Dierickx, Nayera Ahmed, y Benoit Dupont. Conclusiones El objetivo de esta tesis doctoral ha sido desarrollar nuevas arquitecturas, algoritmos, y t´ ecnicas de dise˜ no de circuitos, para la mejora del rango din´ amico de la parte sensora en Procesadores de Plano Focal. El trabajo presentado en esta tesis constituye un avance en el estado del arte de sensores de alto rango din´ amico, proponi´ endose una soluci´ on innovadora para mejorar el rango din´ amico en sensores de imagen CMOS mediante la introducci´ on de una t´ ecnica de mapeo de tonos. Esta t´ ecnica no s´ olo incrementa considerablemente el rango din´ amico, sino que tambi´ en optimiza la representaci´ on final de la imagen mediante una reducida cantidad de bits (siete) por p´ ıxel. Esto se consigue generando informaci´ on acerca de la probabilidad de distribuci´ on de potencia de la luz incidente en el anterior fotograma, y usando esta informaci´ on para ajustar como se comprime y guarda el fotograma actual. Las aportaciones realizadas en esta tesis pueden agruparse en tres campos diferentes: (1) exploraci´ on de tecnolog´ ıas CIS dedicadas, (2) dise˜ no de algoritmos para el sensado de alto rango din´ amico basado en el mapeo de tonos, y (3) dise˜ no de un sensor QCIF de alto rango din´ amico, en una tecnolog´ ıa est´ andar, el cual sirve como demostrador de lo que puede conseguirse con las t´ ecnicas desarrolladas. Las principales conclusiones para cada una de estas l´ ıneas de investigaci´ on se resumen a continuaci´ on: Respecto al trabajo en tecnolog´ ıas CIS: El uso de tecnolog´ ıas CIS para la mejora del rango din´ amico de los sensores indica avances significativos en t´ erminos de reducci´ on de corriente oscura (factor 29) e incremento de sensibilidad (factor 10) cuando se compara con el sensor equivalente en una tecnolog´ ıa CMOS est´ andar. Adem´ as, la posibilidad de a˜ nadir microlentes encima de los sensores ofrece una mejora a´ un mayor de la sensibilidad (con un m´ aximo del 80 % en los p´ ıxeles de menor tama˜ no) y mejora el crosstalk, sin impacto en el tama˜ no del p´ ıxel o en el ruido el´ ectrico a nivel del sensor. Las limitaciones tecnol´ ogicas asociadas a las tecnolog´ ıas CIS excluyen la incorporaci´ on de transistores PMOS y contactos de substrato en el ´ area del sensor, reduciendo fuertemente, por lo tanto, la cantidad de “inteligencia” que puede ser incorporada a niveldep ´ ıxel. Estas limitaciones producen dos importantes inconvenientes para esta alternativa. Primero, esto casi descarta1este tipo de tecnolog´ ıas cuando se dise˜ nan Procesadores 1Se puede utilizar procesamiento basado ´ unicamente en transistores NMOS, aunque su posible funcionalidad es muy limitada. CONTENTS 1.4. Basic Concepts in CMOS Image Sensors ......................... 18 1.4.1. Radiometry vs. Photometry ........................... 19 1.4.2. Spectral Response ................................ 19 1.4.3. Sensitivity .................................... 19 1.4.4. Quantum Efficiency ............................... 19 1.4.5. Full Well Capacity ................................ 19 1.4.6. Microlenses .................................... 19 1.4.7. Color Filter Array ................................ 20 1.4.8. Airy Disk ..................................... 21 1.4.9. Modulation Transfer Function .......................... 22 1.4.10. Electronic Shutter Types ............................. 23 1.4.10.1. Rolling Shutter ............................. 23 1.4.10.2. Global Shutter ............................. 24 1.4.11. Methods for Noise Reduction .......................... 24 1.4.11.1. Correlated Double Sampling ..................... 24 1.4.11.2. Double Delta Sampling ........................ 24 1.4.12. Frame Rate .................................... 24 1.4.13. Crosstalk ..................................... 25 1.4.14. Shading ...................................... 25 1.4.15. Signal-to-Noise Ratio .............................. 26 1.4.16. Dynamic Range .................................. 26 1.5. High Dynamic Range Imagers .............................. 26 1.5.1. Companding Mode Sensors ........................... 28 1.5.2. Multimode Sensors ................................ 29 1.5.3. Well Capacity Adjusting Sensors ......................... 29 1.5.4. Pulse Modulation Sensors ............................ 30 1.5.5. Multiple Sampling Sensors ............................ 31 1.5.6. Sensors with In-pixel Exposition Control .................... 32 1.6. Conclusions ........................................ 32 2. SCU: A Test Chip in CMOS Image Sensor Technology 35 2.1. Introduction ........................................ 35 2.2. Selected CIS Technology ................................. 35 2.3. Evaluation Chip ...................................... 37 2.3.1. Sensors Array ................................... 39 2.3.2. Pixels Circuitry .................................. 41 2.3.3. Variations in Pixels Parameters .......................... 43 2.3.3.1. Pixel Size ............................... 43 2.3.3.2. Layout of Active Diffusion ...................... 43 2.3.3.3. Threshold Voltage of Source Follower Transistor ........... 45 xiv CONTENTS 2.3.3.4. Use of Microlenses .......................... 46 2.3.4. Analog Output Buffer .............................. 47 2.4. Control of Operation ................................... 47 2.4.1. Image Capture Operation ............................. 47 2.4.2. Operation Modes ................................. 51 2.4.3. Timing Specifications .............................. 52 2.5. Measurements ....................................... 53 2.5.1. Experimental Setup ................................ 53 2.5.2. Spectral Sensitivity ................................ 57 2.5.3. Dark Current ................................... 59 2.5.4. Dynamic Range .................................. 61 2.5.5. Performance Comparison ............................. 62 2.5.6. Microlenses Effect ................................ 63 2.5.7. Crosstalk ..................................... 64 2.6. Conclusions ........................................ 64 3. Tone Mapping Algorithm 67 3.1. Introduction ........................................ 67 3.2. Background ........................................ 68 3.2.1. Human Vision .................................. 68 3.2.1.1. Dynamic Range Adaptation ...................... 68 3.2.1.2. Tone Mapping Techniques: Basic Concepts ............. 70 3.2.2. Dynamic Range of Reproduction Devices .................... 71 3.3. Tone Mapping Techniques ................................ 73 3.3.1. Global Operators ................................. 73 3.3.1.1. Miller’s Operator ........................... 74 3.3.1.2. Tumblin-Rushmeier’s Operator .................... 74 3.3.1.3. Ward’s Scale Factor .......................... 74 3.3.1.4. Ferwerda Visual Adaptation Operator ................. 74 3.3.1.5. Logarithmic and Exponential Operators ............... 74 3.3.1.6. Drago Logarithmic Operator ..................... 75 3.3.1.7. Reinhard-Devlin Photoreceptor Operator ............... 75 3.3.1.8. Ward Histogram Adjustment ..................... 75 3.3.1.9. Schlick’s Rational Operator ...................... 75 3.3.2. Local Operators .................................. 76 3.3.2.1. Chiu’s Operator ............................ 76 3.3.2.2. Rahman Retinex Operator ....................... 77 3.3.2.3. Pattanaik Multiscale Observer Model ................. 77 3.3.2.4. Ashikhmin’s Operator ......................... 78 3.3.2.5. Reinhard et al. Photographic Tone Reproduction . . ......... 78 xv CONTENTS 3.3.3. Frequency Domain Operators .......................... 78 3.3.4. Gradient Domain Operators ........................... 79 3.4. Proposed Algorithm .................................... 79 3.4.1. Analog Compression ............................... 80 3.4.2. Digital Compression ............................... 82 3.4.2.1. Time Stamp Code ........................... 82 3.4.2.2. Tone Mapping Code .......................... 84 3.4.3. Some Alternatives for Level per Bin Assignment ................ 90 3.5. Simulation Results .................................... 91 3.5.1. Composition of HDR Image using Multiple LDR frames ............ 91 3.5.2. Mathematical Simulations ............................ 92 3.6. Algorithm Comparison .................................. 98 3.7. Conclusions ........................................ 101 4. TVHC: A HDR Tone Mapping Imager in Standard CMOS Technology 105 4.1. Introduction ........................................ 105 4.2. Fabrication Process .................................... 106 4.3. Architecture of the TVHC Chip ............................. 108 4.4. Pixels ........................................... 109 4.4.1. Pixel Digital Circuitry .............................. 112 4.4.2. Auto-Zeroing Reset Technique .......................... 115 4.4.3. Pixels Physical Characteristics .......................... 118 4.5. Digital to Analog Converter ................................ 119 4.5.1. Dark Signal Contribution Attenuation ...................... 120 4.6. Vref Distribution Scheme ................................. 122 4.7. Code Generator ...................................... 131 4.8. Pixels Control and I/O Interface ............................. 131 4.9. Attainable Accuracy ................................... 133 4.10. Control of the Operation ................................. 138 4.10.1. Image Capture Operation ............................. 138 4.10.2. Timing Requirements ............................... 141 4.11. TVHC Photograph and Layout .............................. 141 4.12. Other Tone Mapping Hardware Implementations .................... 141 4.13. Conclusions ........................................ 144 5. Experimental Results 147 5.1. Introduction ........................................ 147 5.2. Experimental Setup .................................... 147 5.2.1. TVHC PCB Host ................................. 147 5.2.2. Optical Setup ................................... 149 xvi CONTENTS 5.2.3. Lens and Optical System ............................. 151 5.2.4. Operation Modes ................................. 152 5.2.4.1. Tonemapping Operation Modes .................... 153 5.3. Measurements ....................................... 153 5.3.1. Spectral Response ................................ 153 5.3.2. Dark Discharge .................................. 154 5.3.3. Sensitivity .................................... 156 5.3.4. Dynamic Range Measurements ......................... 156 5.3.5. Noise Measurements ............................... 159 5.3.6. Shading Effects due to Heterogeneous Pixels Layout .............. 162 5.4. Captured Tonemapped Images .............................. 164 5.5. Summary of TVHC Characteristics ............................ 172 5.6. Conclusions ........................................ 172 6. Conclusions and Future Work 175 6.1. Conclusions ........................................ 175 6.2. Future Work ........................................ 177 A. TVHC Time Requirements 181 References 185 xvii CONTENTS xviii List of Figures 1.1. Passive Pixel Sensor. ................................... 3 1.2. Active Pixels Sensors schematics. ............................ 4 1.3. Digital Pixel Sensor schematic. .............................. 5 1.4. Pixel shrinking tendency. ................................. 6 1.5. Frontside (with and without light guides) vs. Backside Illumination. .......... 7 1.6. Pixel Isolations. ...................................... 8 1.7. Illustration of 3D integration idea. ............................ 11 1.8. PN Photodiode layers. .................................. 12 1.9. Pinned Photodiode layers. ................................ 13 1.10. Microlens intended performance. ............................. 20 1.11. Bayer CFA structure. ................................... 21 1.12. Irradiation pattern in observation plane. ......................... 22 1.13. Sampling of an ideal point of light source. ........................ 23 1.14. Logarithmic pixel schematic. ............................... 28 1.15. Linear-Logarithmic pixel schematic. ........................... 29 1.16. LOFIC pixel schematic. .................................. 30 1.17. Pulse Modulation pixel schematics. ........................... 31 2.1. UMC CIS Technologies for Broad Applications. ..................... 36 2.2. Color Filter and Microlenses characteristics. ....................... 37 2.3. Block Diagram of SCU. .................................. 39 2.4. Layout of SCU Chip. ................................... 40 2.5. Pixel schematic. ...................................... 41 2.6. Pixel operation phases. .................................. 42 2.7. Pixels layout of performance analysis arrays. ...................... 44 2.8. Pixels layout of crosstalk analysis arrays. ........................ 45 2.9. Microlenses effect on light. ................................ 46 2.10. Microlenses 3D ilustration. ................................ 47 2.11. Schematic of the Analog Output Buffer. ......................... 48 2.12. Control timing illustrating data capture in performance arrays. ............. 50 xix LIST OF FIGURES 2.13. Control timing illustrating data capture in crosstalk arrays. ............... 50 2.14. Electrical simulation of pixels data retrieval. ....................... 51 2.15. Timing illustration. .................................... 52 2.16. SCU PCB test board blocks. ............................... 54 2.17. SCU PCB test board schematic. ............................. 55 2.18. SCU PCB test board. ................................... 56 2.19. SCU measurement setup. ................................. 56 2.20. Newport 6334 lamp spectral irrandiance. ......................... 58 2.21. Spectral Response. .................................... 59 2.22. Conversion gain curves from VDDPIX to out pad for all the arrays. .......... 60 2.23. Discharge in darkness. .................................. 60 2.24. Signal to Noise Ratio. ................................... 62 2.25. SNRxSensitivity comparison ............................... 63 2.26. Relative sensitivity gain by microlenses. ......................... 64 2.27. Crosstlak influence comparison. ............................. 65 3.1. Organization of the retina. ................................ 69 3.2. Light levels effect on photoreceptors visual function. .................. 70 3.3. Intersection of discharge signals with ideal analog reference. .............. 80 3.4. Assignation of digital codes at intersection time. ..................... 83 3.5. Example TMC curve. ................................... 86 3.6. Example of the distribution of TMC codes all in one bin. ................ 86 3.7. Example of TMC codes when levels per bins are not submultiple of total subdivisions. 87 3.8. Look Up Table of code decrements for TSC representation of 3-bits. .......... 88 3.9. Look Up Table of code decrements for TSC representation of 7-bits. .......... 88 3.10. TMC code distribution with zero levels assigned to first bin. .............. 89 3.11. TMC code distribution with zero levels assigned to a middle bin. ............ 90 3.12. LDR frames used for HDR composition. ......................... 93 3.13. Photocurrent image. .................................... 94 3.14. Photocurrent image histogram. .............................. 94 3.15. Time Stamp Image. .................................... 95 3.16. TSI Histogram of photocurrent image. .......................... 95 3.17. Tone Mapping Curve vs. evaluation subdivisions for photocurrent image. . . ..... 96 3.18. Final Tone Mapped Image. ................................ 97 3.19. Histogram of the Tone Mapped Image. .......................... 97 3.20. Results from Luminance HDR. .............................. 99 3.21. Results after the application of CLAHE. ......................... 100 3.22. Results of applying the different levels per bin assignation modes (a-i), and (j) their corresponding tone-mapping curves. ........................... 102 3.23. Results of CLAHE in comparison with mode 6. ..................... 103 xx LIST OF FIGURES 4.1. Photodiode layer arrangement. .............................. 107 4.2. TVHC: High-Level Block Diagram. ........................... 108 4.3. Block Diagrams of the Basic and Time Stamp Pixels. .................. 110 4.4. 5T Amplifier in the Pixels. ................................ 111 4.5. 5T Amplifier Gain Plots. ................................. 111 4.6. 5T Amplifier Comparator Simulations. .......................... 112 4.7. Pixel SRAM Cell. ..................................... 113 4.8. Pixels Digital Control. .................................. 114 4.9. Offset of the combination of the buffer, comparator and digital circuitry. ........ 114 4.10. Simulation of auto-zero improvement. .......................... 118 4.11. Pixels Group. ....................................... 118 4.12. Pixel Area Organization. ................................. 119 4.13. Voltages generated by DAC block. ............................ 120 4.14. DAC Characteristics. ................................... 121 4.15. Voltage range for Dark Signal elimination. ........................ 122 4.16. Charge Injection Amplifier Cell Schematic. ....................... 126 4.17. Charge Injection Amplifier Operation. .......................... 127 4.18. Schematic of the Amplifier in the row-wise Vre f distribution scheme .......... 128 4.19. Buffer Gain Plots. ..................................... 129 4.20. Folded Cascode Buffer Errors. .............................. 129 4.21. Effect of buffers offset in the distributed Vref signal for Tstep =1.2μs........... 130 4.22. Operation of the Vref distribution scheme for Tstep =1.2μs................ 130 4.23. Settling Error in the Vref distribution scheme for Tstep =1.2μs.............. 131 4.24. Sense Amplifier 1-bit Cell. ................................ 133 4.25. Exemplary Temporal Configuration for 40ms Exposure ................. 136 4.26. Standard deviation of intersection time (Tcross) for Vref =0.7V.............. 137 4.27. Reset voltage versus photogenerated current. ....................... 138 4.28. TVHC Operation. ..................................... 139 4.29. TVHC Layout. ...................................... 142 4.30. TVHC Photograph. .................................... 142 5.1. TVHC PCB general scheme. ............................... 148 5.2. TVHC PCB schematic. .................................. 150 5.3. TVHC board photograph. ................................. 151 5.4. Spectral Response. .................................... 154 5.5. Dark discharge. ...................................... 155 5.6. Dark discharge after calibration. ............................. 155 5.7. Calibration SNR comparative. .............................. 156 5.8. Response to white light. .................................. 157 5.9. Dynamic range measurements. .............................. 158 xxi LIST OF FIGURES 5.10. Photon Transfer Curve. .................................. 161 5.11. Illustration of nearest metals distribution in horizontal axis. ............... 163 5.12. Metal shadows image capture. .............................. 163 5.13. Pixel vertical lines zoom micrograph. .......................... 164 5.14. Tungsten lamp. ...................................... 166 5.15. Halogen lamp. ....................................... 167 5.16. Fluorescent lamp. ..................................... 168 5.17. Led torch. ......................................... 169 5.18. Ceiling Lamp. ....................................... 170 5.19. Natural light through window. .............................. 171 6.1. 3D-IC layers. ....................................... 178 6.2. 3D-IC pixel schematic. .................................. 179 A.1. Pixel memories erase operation. ............................. 181 A.2. Reset and exposition operation. .............................. 182 A.3. Image download operation →1row............................ 182 xxii List of Tables 1.1. Ambient luminance levels for some common lighting environments. .............. 27 2.1. SCU Chip Main Characteristics ............................... 38 2.2. Summary of the arrays included in the chip. ......................... 40 2.3. Analog and Digital Control SCU Pins. ............................ 48 2.4. Row and column addressing of the arrays. .......................... 49 2.5. Timing description. ..................................... 52 2.6. SCU PCB board devices. .................................. 57 2.7. Sensitivities at 550nm. ................................... 59 2.8. Discharge slope in darkness. ................................. 61 3.1. Example of bins distribution. ................................ 84 3.2. Levels per bin calculus. ................................... 85 3.3. Subdivisions with decrement in the case of 7 levels per bin in 8 levels. ............. 87 3.4. Levels per bin calculus for photocurrent image. ........................ 96 4.1. Some typical values for ρkσ................................. 137 5.1. TVHC Chip Main Characteristics .............................. 173 A.1. Control signals time requirements .............................. 183 xxiii 1. BACKGROUND for imaging applications like microlenses (which focus the light directly into the detector area of the pixel) and color filters (that allow color imaging), which will be explained in following sections. During the last few years, different CIS technologies have been developed that include different optimized sensors. A first process variation consists of modifying the doping profiles of the photodiode to improve sensitivity and noise in darkness (dark current), keeping the traditional 3T-APS architecture. Another modification introduces the use of pinned photodiodes in a 4T-APS architecture with transfer gate. It is important to remark that in these technologies, the circuitry surrounding the pixels is limited. Consequently, usually PMOS devices are not allowed because the additional well will affect the proper performance of the photodetector. Special doping profiles, including different doping levels for the substrate in the pixel area will also affect the analog circuitry in the pixels. Therefore, the development of Digital Pixel Sensor schemes is very limited in these technologies. 1.1.3. Trends on Image Sensor Technologies As it is the case in standard CMOS technologies, where technology scaling is leading the technology evolution, pixel shrinking [5] is the common trend also in image sensor technologies. Figure 1.4 shows the general tendency of smallest pixels during recent years. Pixels below 2μm usually share circuitry, such as the select transistor and a source follower transistor but with different TG signals. This shrinking tendency is slowing down due to increasing technological difficulties at so small dimensions, which is noticeable observing the pixels below 1.4 microns. Besides the modifications of doping profiles and the improvement of photodetectors, several additional steps are being included in order to improve the optical performance of CIS technologies, most of them already used in the past in CCD technology. These innovations in CIS technologies are explained in the following subsections. 2005 2006 2007 2008 2009 2010 2011 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 Year Pixel Pitch (microns) Figure 1.4: Pixel shrinking tendency. 6 1.1.3.1. Backside Illumination vs. Frontside Illumination Traditionally, CMOS Image Sensors have been fabricated/designed for Front Side Illumination (FSI) operation [6], where the light arrives to the detector after crossing through different structures. The photons travel through layers (such as microlenses, color filters, passivation layers, oxide layers, antireflective coatings) between the aperture of the metal lines of the pixels, finally creating photogenerated charges in the silicon. Figure 1.5 shows the crosssection of two pixels, where (a) shows a typical organization of a FSI scheme, where the light path can be clearly observed and photogenerated charges are indicated as red dots (e-): Microlenses: They focus the light in the area of the photodetector. Planarization: Layer added in the process step to create a smooth flat surface on top via chemicalmechanical methods. Color Filters: They filter the wavelength of the light to the required bandwidth of the spectrum. They are usually red, green and blue allowing for color CMOS imagers. Passivation: It protects the surface against mechanical damages and contamination by losing its chemical reactivity, avoiding effects such as oxidation due to the reaction with surrounding air. Oxide Layers: The light travels through these layers in the areas where no metal lines are present, although it can be reflected and refracted by them. Camera manufacturers have aimed their efforts toward the increment of spatial resolution. This is commonly known as the race toward megapixels, which is mainly based in the reduction of pixel size. If the shrinking pixel tendency [7] continues, the main drawback of this scheme will increase: if the ratio between the pixel stack height and the dimension of the aperture is too low, the optical performance is Antireflective Coating (a) FSI Antireflective Coating (b) FSI with light guides Antireflective Coating (c) BSI Figure 1.5: Frontside (with and without light guides) vs. Backside Illumination. 7 1. BACKGROUND ”poor”. In order to solve the loss of optical performance due to the low amount of light arriving to the photodetector two schemes have been developed. One variation preserves the FSI scheme but includes a light enclosing path from the passivation layer to the photodetector. Thus, it eliminates the light lost in oxide layers and scattering between metals. An illustration of this method is shown in figure 1.5(b) [8]. The second alternative is the development of CMOS imagers under a scheme inherited from CCD [9] called Back Side Illumination (BSI). In this design, the wafer is inverted and the light does not arrive to the photodetector through the top layers but through the substrate. Now, the metal lines are under the photodetector in the light path making unnecessary to keep an aperture of metal lines. Figure 1.5(c) shows the position of the typical layers in BSI scheme. Therefore, the light arrives to the photodetector through only microlenses, planarization, color filters, and passivation. FSI has the advantage of lower cost and the maturity of the technology. Moreover, light guides achieve low crosstalk that is an issue in BSI due to the device structure. However, BSI achieves higher efficiency in the capture of incident light and the optical path can be optimized independently to the electrical one. Therefore, FSI is the dominant technology due to its lower cost but the use of BSI is increasing in applications with very small pixels (1.4 microns pitch pixel and beyond). 1.1.3.2. Deep Trench Isolation (DTI) Lately, due to the low pixel dimensions, the unwanted effects of the neighbor pixels are larger. These effects are denominated crosstalk, whose cause will be explained in section 1.4.13. In order to reduce crosstalk, several technological schemes have been developed apart from the typical Shallow Trench Isolation (STI). Shallow Trench Isolation (STI) is the mainly used isolation technology for CMOS technologies under 0.5μm. Regarding CMOS image sensor, STI prevent crosstalk between neighbor pixels via diffusion. A shallow trench is etched in the surface of the wafer and then filled with a dielectric serving as the Field Oxide (FOX) positioned as shown in figure 1.6 (a). The first modification to STI added P+implants creating an isolation wall in the side of the STI (a) Shallow Trench Isolation (b) Dedicated P+implant Isolation (c) Deep Trench Isolation Figure 1.6: Pixel Isolations. 8 structure as shown in figure 1.6(b). Other solution is the use of adapted pinned deep photodiode on n-substrates also with isolation wall and STI. However, the recent trend is to substitute the STI for Deep Trench Isolation (DTI) structures [10] depicted in figure 1.6 (c). In this method, the Si-SiO2interface is extended as a deep wall in the substrate with high aspect ratio such as 1:25, achieving higher depth than in the other solutions. DTI is done during the same processing step of the STI formation for the external area of the pixel array. However, it is needed only one extra mask. A passivation step is done in order to avoid degradation on dark current and other failures in pixels due to the increment of defects in the larger Si-SiO2interface. DTI high aspect ratio deep isolation1provokes a high charge collection volume, due to the created barriers that avoid losing some carriers, and therefore the improvement of collection performance. Moreover, the structure performs a sort of light guide structure due to the different refractive index between Si and SiO2. Therefore, it improves the performance for off-axis illumination, which arrives with an inclination. 1.1.3.3. 3D Integration Recently, the dimensions of the smallest pixels have arrived to the physical shrinking limits of imaging lenses, as will be explained in section 1.4.8. In order to solve this issue, two different options can be considered. First, the surrounding circuitry can be increased taking advantage of the shrinking of standard CMOS Image Sensors processes. Second, instead of increasing the amount of circuitry horizontally, it can increase vertically. Microelectronic systems have typically consisted of multiple integrated circuits (ICs). The connection of these systems has been traditionally done by 2D wiring in a Printed Circuit Board (PCB), where every chip has its package. To improve performance, several chips can be enclosed in a single package side by side, named Multi-Chip Package (MCP). The natural evolution of this scheme must take advantage of the vertical dimension to achieve a smaller footprint. This leads to 3D integration, where vertically stacked chips will connect. 3D integration approaches can be divided in these categories [11][12]: Packaging-based: it is applied over completed chips by means of the pads [13]: •Package-on-Package (PoP): multiple packaged chips are vertically stacked [14]. The chips can be wire-bond [15] or flip-chip [16] type [17]. An advantage is the possibility of IC packages of being fully tested with high yield prior to stacking, whereas bare die stacking suffer from yield issues [11]. •System-in-Package (SiP): multiple naked dies are vertically stacked in a single IC package. It is usually fabricated connecting thinned chips and their connection is achieved via the wire bonding of the enclosing package and large dimension Ball Grid Array (BGA), which are an array of solder balls. 1Large height and narrow width. 9 1. BACKGROUND Assembling-based: different levels of functionalities are created with normal integration processes, each with a different substrate, and connected to each other. The main representative techniques are Microbumps and Through Silicon Vias (TSV), which is a passing through vertical electrical connection in a wafer or die with copper or tungsten. The way in which the chips are connected varies depending on the technology: •Die-to-Die (D2D): individual dies are glued together to form the 3D-IC. It is made with die-to-die bonding (Microbumps) and Through Silicon Vias (TSV). TSV are performed by laser drill or deep reactive-ion-etching and later filled with copper. The interconnect pitch is large because it is necessary to compensate for misalignment of dies. Fabrication involves several extra manufacturing steps, such as alignment, resulting in potential yield reduction. •Die-to-Wafer (D2W): individual dies are stacked on top of dies that are still not cut from a wafer. First, each individual die must be aligned to the base wafer. Interchip electrical connection can be formed by post-bond via formation or die bonding process. The alignment of each die takes about as long as the alignment of an entire wafer. The alignment process is typically the most expensive equipment used in the 3D assembly process. The advantages of these processes are that the wafers on which the different layers of the 3D stack are produced can be of different size. Another advantage is that the individual dies can be tested before the stacking process and only “Known-Good-Dies” (KGD) can be used, thereby increasing the yield of the 3D-IC. •Wafer-to-Wafer (W2W): it is realized with full semiconductor wafers, which are aligned, bonded, and diced. Therefore, they must be of same size and similar material for match of the thermal coefficient of expansion. The vertical interconnection is usually achieved using TSV. Wafer-level handling and processing allows hundreds or thousands of devices to be created at once in comparison with dies handling. Consequently, it results in a more cost effective approach. Monolithic: this approach is the vertical extension of normal standard integration technologies. Each layer is processed sequentially starting from the bottom layer. Therefore, devices are built on a substrate wafer by mainstream process technology. After proper isolation, a second device layer is formed and devices are processed by conventional means on the second layer. This sequence of isolation, layer formation, and device processing is repeated to build a multi-layer structure. This option allows for lower power consumption and faster speed than using separate circuits. Thermal cycling can degrade underlying devices. Due to the sequential processing of layers, the manufacturing throughput is low. 3D integration results in a chip that is a combination of multiple tiers (wafer layers) of thinned 2D integrated circuits. The tiers are stacked, bonded, and vertically connected, as illustrated in figure 1.7. It is possible to choose independently the best technology for each layer based on the functionality, performing a heterogeneous 3D integration. It may results in a reduction of costs and an increase in performance. First tier can be implemented using a technology aimed for image sensors. Second tier 10 Figure 1.7: Illustration of 3D integration idea. could be developed in a technology for high analog performance. Finally the last tier, for example including the digital processing, can take advantage of a very deep sub micron technology [18]. The more mature 3D technologies are related to packaging. However, due to the high possibilities of 3D integration, these technologies are under high research nowadays. The main advantages of 3D approaches are: more suitable form factor, cost, integration of heterogeneous technologies, performance, improvement of interconnection time delay (shorter connections), possibility of high speed and low power consumption. The main disadvantage is the thermal management and the technical difficulties in the process development. 1.2. Photodetection in CMOS Technologies Incident light on a semiconductor generates electron-hole pairs or photogenerated carriers, due to the photons of energy greater than the bandgap energy. In order to sense these charges, the most used structure in CMOS image sensors are photodiodes [19][20]. A photodiode is based on a junction of P and N semiconductor material. In the surroundings of the junction, the area is depleted of free charge carriers, forming a depletion zone with an electric field. When photons of energies greater than the bandgap energy are absorbed in or close to a PN junction create electron/hole pairs, which are pulled by the electric field of the contact potential across the PN junction. Inserted in a pixel, the photodiodes are reverse biased creating a larger depletion zone in order to sense more photogenerated carriers. Mainly, two types of photodiodes are used in CMOS imagers: PN photodiodes and pinned photodiodes, which will be explained in the following subsections. 1.2.1. PN Photodiode PN photodiode [21] has been the most used photodetector in early generations of CMOS image sensors. They are already present in standard CMOS process technologies, thus enabling image sensor design within a general-purpose IC design environment. Therefore, the PN photodiode pixel is a cost effective solution. PN photodiodes use the electric field of reverse-biased PN junction to separate the photogenerated 11 1. BACKGROUND electron-holes pairs. These photogenerated carriers are created by the incident light absorbed in the depletion region. The electric field moves the carriers towards the edges of the depletion region, generating a reverse photogenerated current Iph, which is (approximately) directly proportional to the incident light. The typical P and N layers used in order to create PN photodiodes are: n+diffusion/p-well (n+/PW). n-well/p-type substrate (NW/PSUB). n+diffusion/p-type substrate (n+/PSUB)1. In an n+/PW photodiode, a highly doped n+region is formed in a PW region. The increased doping concentration and reduced depth of the PW region in recent highly integrated CMOS technologies reduces the thickness and depth of the depletion layer and affects sensitivity. In terms of realistic sensitivity, n+/PW photodiode is suitable for above 0.5-0.8μm rule CMOS technology [20]. NW/PSUB photodiodes have a larger and deeper depletion zone compared to n+/PSUB photodetectors. Therefore, a larger photoconversion volume can be obtained, even with deep submicron processes. Therefore, NW/PSUB photodiodes usually implies better sensitivity. However, due to layout rules, well formation usually implies the need of larger pixels than the n+/PSUB photodiode option. 1.2.2. Pinned Photodiode The layer scheme of an n+/PSUB photodiode with a 3T APS pixel circuitry is shown in figure 1.8. The dotted line indicates the boundaries of the depletion zone. The junction capacitor is charged to p-epi RESET VDD VDD ROW SELECT OUT p+ n+ Figure 1.8: PN Photodiode layers. 1This photodiode is obtained in CMOS twin-well technologies when in the region under the photodiode (in this case, p-type substrate) a p-well doping profile is not applied. 12 VDD through the RESET transistor, resulting in a wider depletion zone. The reset transistor is turn OFF and the photogenerated current discharges the sensing node which is connected to a source follower amplifier, whose voltage will be measured when the transistor ROW SELECT is ON. This output is connected to the column selection circuitry, which adds the polarization of the source follower of the pixel. The main drawback of PN photodiodes are the noise due to dark current, explained in section 1.3.4, due to surface generation and thermal noise associated with the photodiode reset (initial charge). In order to solve the main drawbacks of the PN photodiode, it was developed the pinned photodiode [22]. This structure is somehow a photodiode structure that has been buried in order to isolate the behavior from the noise created by the surface. Figure 1.9 shows an illustration of the typical structure of a 4T APS pixel circuitry. The pixel consists of a pinned photodiode and four transistors that include a TRANSFER GATE, a RESET transistor, an amplifier transistor (source follower), and a selection transistor (ROW SELECT). Regarding the photodiode, the top layer is a thin p+layer under which are stacked an n layer and the p-substrate (P+NP structure). Using appropriate doping profiles, both P+N and NP junction depletion region can be joined forming a depletion zone dragged away from the surface. The p+layer has the same potential as the p-substrate and the accumulation region is separated from the surface, where the trapped states are located producing dark current levels. At the beginning, the floating diffusion n+is reset by the RES ET transistor. Then the pixel output is read as a reset reference signal, which includes the pixel offset of the source follower and the reset noise. Then, the photogenerated charge is integrated in the pinned photodiode potential-well and transferred via the TRANS FER GATE to the floating diffusion n+to be sensed. The conversion charge to voltage is realized by the capacitance of the floating node n+, which is connected to the source follower. The output of the pixel is read and subtracted by the reset reference in order to obtain the final measure with compensated noise. TRANSFER GATE RESET VDD VDD ROW SELECT OUT Figure 1.9: Pinned Photodiode layers. 13 1. BACKGROUND There are several advantages in this structure. The photodetection and the photoconversion area are separated. The pinned photodiode has less dark current because the p+layer masks the traps from the surface, which are one of the main sources of dark current. Therefore, the noise performance of PN photodiode is commonly worse than that of the pinned photodiode. They achieve high sensitivity and good image quality. Therefore, they can compete with CCD imagers. Compared to the PN photodiode, it offers a smaller full well capacity, which is the maximum quantity of charge that can be storage in a pixel (see subsection 1.4.5), and higher sensitivity. In PN photodiodes, the full-well capacity can be increased through the photodiode capacitance, but in a pinned photodiode it is limited to the pinned potential of a photodiode. If the pixel is working in charge mode, this feature enhances the dynamic range. However, the main drawback of this structure is the cost and difficult design. A proper depletion zone requires not only well-formed thin p+layer in the surface but also an elaborate design of the potential profile by precise fabrication process control. Incomplete charge transfer from the photodiode to the floating node node causes excess noise, image lag, and a nonlinear response. Therefore, the pinned photodiode and the TG should be optimized to avoid this issue; this must be performed also in CCDs but they operate in high voltage, which simplify the problem. However, in pinned photodiodes (due to the low voltage operation) very accurate dopant control is required in fabrication. High transfer efficiency requires optimizing the region between the photodiode and the transfer gate while optimizing photodiode depth. Furthermore, forming an L-shaped transfer gate will improve transfer efficiency due to a three-dimensional potential effect [20]. Additionally, the reset transistor still causes some noise. They also have lower fill factor than PN photodiode, due to the inclusion of an additional transistor (the transfer gate) and the charge capacity is limited by the pinned potential [20]. 1.3. Noise Noise in image sensors can be defined as any sensed variation from the ideal scene to be captured [20] and therefore it deteriorates imaging performance. The imperfections found in sampled images are originated in the optics, sensor and post-processing operation. However, electronic noise is related to the last two path of the signal while optical imperfections are named optical distortion and aberrations, which will be described in section 5.2.3. The noise can be presented as fixed offsets, random noise, etc. with statistical variation with time, location, temperature, signal level, position, and/or the design of the system. At low illumination, the dominant source of noise is reset and readout transistors, which is related to the electronics. At high illumination, the dominant source of noise is the photodiode shot noise and PRNU, which is caused by the quantized nature of light. However, there are several noise components in CMOS image sensors [23]: Thermal Noise: It is caused by the effect of temperature in transistors. Photon Shot Noise: It is provoked by the quantized nature of light. 14 Reset Noise: It originates at the reset switch of the pixel. Dark Current: It is the electric current that flows through the photodetector in the absence of light. Flicker Noise: It is caused by the defects in the lattice structure. Fixed Pattern Noise: It is provoked by the variations caused by technological process. It is fixed in position. •Photo Response Non Uniformity: It is caused by the pixels different responses to light. •Dark Signal Non Uniformity: It is caused by the pixels different dark signal. Quantization Noise: It is caused by the Analog-to-Digital conversion of the pixel signal. Partition Noise: It is caused by the reset operation. Random Telegraph Signal Noise: It is caused by the trap and emission of carriers in the Si-SiO2 interface of the channel and discrete fluctuation of the dark current. 1.3.1. Thermal Noise Thermal noise [24] is caused by the effect of temperature in the carriers, i.e by random thermally induced motion of carriers in resistive regions. It is also named Johnson noise because J.B. Johnson discovered it in 1928. This noise exists in any semiconductor. There is a constant thermal motion of carriers in any direction of semiconductor. They frequently collide with the material molecules and change the direction of motion. Therefore, in a long interval, it has a zero mean value, but in a short interval, these random fluctuations form a noise current. Noise will increased with temperature due to its effect over the mean square speed of current carriers. Thermal noise has constant power over all frequencies, i.e. white noise. Therefore, the bandwidth of circuit determines the noise value. The expression of thermal noise voltage is [25]: V2 n=4kTRΔf(1.1) where kis Boltzmann constant, Tis absolute temperature, Ris equivalent resistance and Δfis bandwidth of measuring amplifier. As thermal noise is often associated with the actual circuit bandwidth, the design of the input stage of amplifier is particularly important. 1.3.2. Photon Shot Noise Photon Shot Noise (PSN) [26] is related to the quantum nature of light. There is an uncertainty in the number of photons collected in the phodectector. The number of photons incident in a photodetector illuminated by a perfectly uniform source of light follows a Poisson distribution. The magnitude of PSN is the square root of the mean number of electrons stored in the photodetector. Therefore, the root mean square noise voltage due to this noise is: 15 1. BACKGROUND (a) Surface representation. −15 −10 −5 0 5 10 15 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 k·a·sin(θ) I(θ) / I(0) (b) Cross section. Figure 1.12: Irradiation pattern in observation plane. Airy Disk. The radius of the first dark ring is given by the equation: r1=1.22Rλ 2a≃1.22 λf D=1.22 λf/# (1.7) where fis the focal length of the lens which acts as aperture, assuming R≃f, and Dis the diameter of the aperture. The figure f/# is typically used in the description of lenses, which correspond with the focal length divided by the diameter of the aperture. It must be noticed that, the dimension of the first dark ring will determine the maximum resolution achievable in pixels sampling this image. As an example, if we consider r1for green light (555nm) with a typical f-number such as 2.8, the radius of the airy disk is 1.9μm. Therefore, if we sample the image (in the focal plane) with pixels with dimensions below this 1.9μm, the captured image will be affected by the diffraction pattern causing an error. Figure 1.13(a) shows the position of too small pixels in the pattern, where an error is induced as the image should contain a single pixel receiving light. Figure 1.13(b) shows the effect of pixels with dimensions that enclose the airy disk, causing a proper sampling of the projected image. Therefore, nowadays pixel sizes below 2μm are around the physical limits of pixels shrinking. However, as explained in section 1.4.7, the color pixels are really captured by four pixels due to color channels, which performs a spatial and color oversampling of the airy disk that still maintain good performance for pixels below the boundary. In the future, approaches shrinking much further the pixels will not be necessary, as the oversampling will be insufficient, and therefore the circuitry surrounding the photosensor can experiment a boost inside the pixels. 1.4.9. Modulation Transfer Function Modulation Transfer Function (MTF) [43] indicates the performance of the system in capturing information of an object as a function of the spatial frequencies. Therefore, it indicates the image sharpness and resolution. The MTF is influenced by, not only the image sensor, but also the optics. 22 (a) Too small pixels. (b) Properly sized pixels. Figure 1.13: Sampling of an ideal point of light source. It shows the number of stripes of light-dark pattern that can be photographed per millimeter in the image plane. The MTF indicates that the higher the spatial frequency of details, the greater the decrement of performance. However, the shape of the decrement indicates the MTF of the sensor. 1.4.10. Electronic Shutter Types In photography, a shutter is a mechanism to control the exposition to the light for a period of time, which allows capturing an instantaneous scene. Traditionally, mechanical shutters have been used, which were placed on top of the image sensor or lenses. This mechanical shutter will allow passing the light through an aperture, and after an amount of time, it will completely block it by physical means. However, it is also possible to control the exposition time by means of electronic circuitry, which is named electronic shutters. CMOS cameras usually use global or rolling shutter type. 1.4.10.1. Rolling Shutter In pixels without in-pixel memory, the time between the reset operation and the readout operation determines the exposure time. Therefore, since read-out is sequential, a gradient in the output voltages can be observed due to the differences in the effective exposure time. The integration time for pixels that are read later is augmented by the read-out time of previous pixels. This is specially noticeable when the readout time is high and/or arrays are large [44]. In the rolling shutter mechanism [45], the start and end of the light collection for each row is slightly delayed from the previous row. This minimizes the gradient, which will be present when applying simultaneous reset but sequential read out. However, it leads to image distortion when capturing scenes in motion. This will be further explained in section 2.4.2. 23 1. BACKGROUND 1.4.10.2. Global Shutter The global shutter technique [46] uses a memory element inside each pixel to provide the functionality of a mechanical shutter. This allows a simultaneous reset, exposure and store (in the in-pixel memory) of the whole array. Therefore, the data is not a affected by the delay between the readouts of the pixels. 1.4.11. Methods for Noise Reduction 1.4.11.1. Correlated Double Sampling Correlated Double Sampling (CDS) [47] is a method that suppresses kTC noise from pixel reset, 1/f noise from the in-pixel source follower and Fixed Patter Noise (FPN) (mainly originated by pixel- to-pixel variation in source follower threshold voltage). In this method, two samples from the pixel are taken, one with the data at the chosen exposition time and other during or just after reset. The reset signal is then subtracted from the signal at exposition time (read signal), resulting in a pixel signal with noise reduction. It must be remarked that, the CDS method is effective only if the pixel signal is captured linearly and the mismatch has mainly an offset component. These assumptions are not always true and more elaborate CDS circuits, which can compensate for gain mismatch and nonlinearity in the circuits, are required. 1.4.11.2. Double Delta Sampling Although the CDS reduces noise to a large extent, it also includes a component of FPN due to mismatch in the CDS circuits. These circuits are usually shared by columns of pixels and therefore introduce column-FPN. Double Delta Sampling (DDS) method is used to reduce column-FPN in the CDS method [48][49]. This method includes a column circuitry to remove the offsets due to column drivers and hence reduce column-FPN. The DDS circuit calculates the difference between the voltages from two consecutive read-out operations per column channel. During the read operations, the actual voltage of the CDS data (reset and read signals) are read out and stored in analog memories (capacitors). Then, the capacitors storing these two signals are short circuited. The two outputs of the DDS circuit are the difference of the reset and read signals minus the short-circuited value. The two output signals are sent out of the chip and subtracted (read-reset) by the subtraction circuit in the off-chip data acquisition system. Thus, the complete operation of the double delta sampling circuit is performed. 1.4.12. Frame Rate The output of the majority of image sensors is an image with the value of the output of each pixel, i.e. a frame. The time of delay between the generation of a new frame determines the frame rate [50], which is expressed in frames per second. 24 There are several applications, such as automotive or surveillance, which needs a high frame rate to respond as fast as possible. However, the processing system should have sufficient computational power to be fast enough to utilize the high amount of data produced by high frame rate sensors. Moreover, it will need more resources in image sensors with large bit count representation. 1.4.13. Crosstalk The influence that neighbor pixels have on a pixel is named Crosstalk [51]. It implies an error in the output of the pixel as it includes an amount of signal that does not belong to the pixel, since light was not directly impinging on it. Therefore, it degrades the spatial resolution, reduces overall sensitivity and induces poor color separation. The crosstalk can originate by three main causes: Spectral: It is due to the effect of the color filters. It is caused by the non-zero transmission of the filters in wavelengths, which they should be blocking. Optical: Part of the light coming with a certain angles reaches a neighbor pixel. It is caused by the photons coming from neighbors before electron/hole generation. It is caused by reflection, refraction, or diffraction in top layers or even substrate. This type of crosstalk can be minimized by the use of microlenses. Electrical: It is generated by photogenerated carriers in a photodectector, which moves to a neighbor pixel. These carriers are generated deep in the photoconversion area by long wavelength light and they may diffuse. It happens in the silicon due to electrical mechanisms such as diffusion and drift. This type of crosstalk depends on the pixel design. 1.4.14. Shading Shading [20] is a slowly varying or low spatial frequency output variation in an image. Usually, the center of the image appears brighter while a gradient is present toward darkening the periphery. The causes of shading can be: Dark Current: Local heat sources that exist introduce a thermal distribution, producing dark current gradients. Microlens: The efficiency of a microlens is diminished at the periphery of an imaging array due to the higher angle of incident in light. Therefore, the output of pixels at the periphery decreases. Electrical: Non-uniform biasing and grounding may cause shading. Optical: Even in the absence of microlenses, the angle of the incident light is higher in the periphery. Due to the metals stack height, the periphery pixels will loss performance because they receive light with higher angles. The aperture in the optics can increase the difference of angles between the center and the periphery. Optical shading can be minimized by proper design of the 25 1. BACKGROUND microlenses taking into account the difference in angles and therefore with non-heterogeneous array of microlenses. 1.4.15. Signal-to-Noise Ratio The Signal-to-Noise Ratio (SNR) [20] is the ratio between the signal and the noise in the sensor. This determines the quality of a certain image and it will depend on the condition of the capture, such as the exposition time. Noise in image, such as dark current discharge, will increase with exposition time. Therefore, in linear image sensors higher exposition time decreases the SNR of the image. 1.4.16. Dynamic Range The Dynamic Range (DR) [20] is the ratio between the largest and the smallest signal level that can be measured. The smallest signal to be measured is the noise floor. In other words, the noise equivalent illumination is the illumination that will generate a signal equal to the signal generated by the noise. The largest signal to be measured in simple APS pixels will be determined by the full well capacity as it will determine the saturation point. DR(dB)=20 log10 Lightmax Lightmin (1.8) Most sensors have a DR around 60-70dB, which is mainly determined by the integration node capacitance. However, for some applications, this DR is insufficient. Applications, such as automotive, require 100dB or more of dynamic range. 1.5. High Dynamic Range Imagers Real word scenarios contain a large range of illuminations. The dynamic range of the natural world is high where the difference in contrast is over 100,000,000:1 as indicated by table 1.1 [52][53]. We have to distinguish two kind of High Dynamic Range (HDR) situations: (1) inter-frame HDR, when the HDR illumination does not take place simultaneously (scenarios depicted in table 1.1), (2) intra-frame HDR, when the high difference in illuminations takes place in the same frame or scene. For example, a large intra-frame HDR scene can occur if it includes both an outdoor area illuminated by direct sunlight and an indoor area illuminated by interior light or even in shadows. Inter-frame HDR captures can be performed controlling the exposure time. However, intra-frame HDR scenarios imply more complex mechanisms in other to be properly captured. Therefore, typical cameras are limited in intra-frame dynamic range and they are not able to properly capture high dynamic range scenes. Some applications need cameras that are prepared to capture images in HDR scene environments in order to work properly. Therefore, DR is a very important figure of merit in CMOS Image Sensors. 26 Condition Illuminance(lux) Clear night sky 0.001 Quarter moon 0.01 Full moon 0.1 Late twilight 1 Twilight 10 Heavy overcast 100 Overcast sky 1,000 Full daylight 10,000 Direct sunlight 100,000 Table 1.1: Ambient luminance levels for some common lighting environments. The linear response of APS image sensors is very limited in dynamic range and therefore variations in image sensor designs are introduced to create High Dynamic Range image sensors. In order to improve this characteristic, there exist two options: (1) reduce the noise floor or (2) extend the saturation point toward higher light intensities. Leaving aside solutions consisting on cooling the sensor, as it is commonly done in astronomy applications, the reduction of the noise floor level has almost unavoidably led to introducing great modifications in the silicon fabrication process, and so related to the use of CIS technologies. Regarding the extension of saturation point, two main options arrive immediately: linear or compressive acquisition. If the pixel is designed with linear acquisition, the binary image representation will need at least 17 bits/pixel for a DR greater than 100dB, log2(10100 20 )=16.6. This high amount of data necessary for linear acquisition defines two main tasks in HDR imaging, understood as the overall task of producing an optimum HDR scene representation: Image capture: to capture the scene information by means of sensing light. Unavoidably, this task must be performed at pixel level. Compression: to reduce the amount of data necessary in the HDR image representation. It can be performed in-pixel (non-linear light acquisition), system level, or post-processing by software in a computer. This fact proposes a classification of pixels depending on a linear or compressive acquisition. Regarding possibilities not related to technology, and so devoted to the extension of the saturation point, different mechanisms have been proposed in the literature to improve the DR of image sensors, which will be described in the following subsections [54]: Companding Mode Sensors: They perform an analog compression, usually due to logarithmic behavior of a load. Multimode Sensors: They have several modes of operation. Usually, a linear response is applied to low illumination levels and a logarithmic response is applied to high light levels. 27 1. BACKGROUND Well Adjusting (Clipping) Sensors: The capacity of the well is adapted to extend the DR. Pulse Modulation Sensors: •Pulse Width Modulation Sensors: They measure the time to reach the saturation signal expressing the pixel signal by a pulse width. •Pulse Frequency Modulation Sensors: The pixel signal is converted to frequency. Multi-sampling Imagers: Several images are taken with different exposition times and later composed in an HDR image. Sensors with local exposition control: Every pixel decides its exposition time in order to improve DR. 1.5.1. Companding Mode Sensors Companding sensors [55] compress the pixel signal instead of having a linear representation. The compression increases toward the higher illumination levels, and it is usually done by a logarithmic transformation. The logarithmic conversion from the photogenerated current to a voltage with logarithmic compression is realized via a non-linear load. The load is a MOS transistor that operates in the sub-threshold region, which shows a logarithmic I-V characteristic. For typical photocurrents in the pA to nA range, which by nature will not be much higher, the load transistor will operate in sub-threshold. Figure 1.14 shows a typical pixel schematic. These pixels work in continuous time operation mode (non-integrating). The main drawbacks are slow response time for low light levels and small voltage swings. Their main drawback is the large FPN. Several noise reduction techniques, pixel or chip level, can be used in order to achieve a noise performance similar to integration mode pixels. Figure 1.14: Logarithmic pixel schematic. 28 Figure 1.15: Linear-Logarithmic pixel schematic. 1.5.2. Multimode Sensors Multimode sensors have multiple modes of operation and combine them in order to improve DR. An example of such sensor is linear-logarithmic sensors [56]. The schematic is shown in figure 1.15. At low illumination, they operate as conventional linear pixels and, at high illumination, logarithmic compression is applied. Usually, the criterion for choosing one operating mode is derived from the characteristic of the output signal at that illumination. Therefore, the logarithmic compression is used in the higher illumination in order to extend the point of saturation. 1.5.3. Well Capacity Adjusting Sensors The well capacity adjusting (clipping) sensors adapt the maximum charge that can be accumulated during integration, in order to overcome the limitation of the capacitance of the integration node. Usually, the charges that excess the capacity of the integration node is derived or drained to another node achieving an extended non-linear photoresponse curve. A typical approach of these sensors is the use of a Lateral Overflow Integration Capacitor (LOFIC) [57]. Figure 1.16 shows an example of this type of pixels [58]. The scheme is realized by a pinned photodiode with transfer gate TG and a floating diffusion FD used to convert charge to voltage. The switch Sis used to drain the overflow charge into the capacitance CS. First, the floating diffusion FD and the capacitor CSare reset through the transistors N5and N2. During integration time, if the photodiode is saturated the additional charge will overflow through the transistor N4. Consequently, this charge will be collected by the capacitor CFD and CS, which are connected through transistor N5. After integration, switch N5is turned off. Then, the signal in the photodiode is read activating the switch N4. Finally, N5is again connected to distribute the charges stored in both capacitors depending on the ratio of CFD and CS. The voltage at FD node minus the source follower threshold voltage of N3is read out. The advantage of this approach is the possibility to minimize the capacitance in the FD node and 29 1. BACKGROUND Pixel Figure 1.16: LOFIC pixel schematic. therefore it achieves higher sensitivity. However, the loss of full well capacity is compensated by the overflow capacitance for high saturation. Summarizing, it achieves high SNR, high sensitivity, high Full Well Capacity, and low dark current. The drawback of this sensor is the decrement of fill factor due to the capacitor and extra switch. 1.5.4. Pulse Modulation Sensors Pulse Modulation sensors [59] use the time that takes the integration node to reach a certain reference analog voltage. Therefore, instead of express the light intensity as a value of charge, voltage, or current, it is expressed as time. If reference voltage is not fixed, it will perform an on the flight compression. It also can perform an in-pixel conversion and therefore this type of sensors is usually used to perform DPS pixels. These imagers can be divided in two main categories: Pulse Width Modulation (PWM): The time between the reset and crossing the reference voltage is measured. The width of the pulse is inversely proportional to the photogenerated current. A typical pixel is shown in figure 1.17(a). Pulse Frequency Modulation (PFM): After reset, when the integration node crosses the reference level an event is triggered, which will be counted usually in pixel-level, and performs a new reset. This operation is continuously repeated performing a photocurrent controlled oscillator, where the number of pulses in a period of time is a linear function of the incident illumination. The Dynamic Range depends on the number of bits of the counter. A typical pixel is shown in figure 1.17(b). The readout schemes of these sensors can be perform in a frame-based or event-based way. The frame-based structure reads out the content of the pixels row by row in a sequential manner at the end of the exposition or several times in order to determine the pulse characteristics. The most typical eventbased readout is the Address Event Representation readout. In this asynchronous readout type, once the data is ready, the pixel requests for the output bus. Once the acknowledgment takes place, a signal is 30 ph Pixel 1 ref (a) PWM ph Pixel 1 ref (b) PFM Figure 1.17: Pulse Modulation pixel schematics. sent back by the chip-level AER control to start the transmission of data. When multiple requests occur, an arbiter decides the sequence of readout. This scheme provides an efficient use of the output bus. A PWM pixel with global reset is usually denominated “Time-To-First-Spike” (TTFS), where typically event-based readout is used. Dynamic Range is no longer limited by the power-supply rails and therefore by the voltage swing of the pixel. Consequently, in these pulse-based (time-based) schemes, the dynamic range is limited by which range of integration time can be measured. The disadvantage of these methods is the high decrement of the fill factor. 1.5.5. Multiple Sampling Sensors These sensors performs several frame captures but with different exposition times [60]. In every individual capture, the whole array of pixels will take the same integration time. This method produces 31 2. SCU: A TEST CHIP IN CMOS IMAGE SENSOR TECHNOLOGY it [vignetting], etc.), we decided to design a test chip in order to measure what this CIS 0.18μm technology could offer us. The objective was, then, to analyze the main characteristics of the photosensors and to evaluate possible strategies for dynamic range improvement methodologies [54], bearing in mind the desired compatibility with Focal Plane Processors typical circuitry. In our study, we have analyzed sensitivity, dark current and crosstalk, and how these factors are affected by photodiode’s size, shape, use of microlenses, and use of Low-Voltage transistors in the 3T-APS circuitry. UMC 0.18μm technology is offered through Europractice in two different choices: (1) the Conventional Photodiode, which is an enhanced PN photodiode, or (2) the Ultra Photodiode, which is a pinned device. Since neither a set of macros (predefined sensor layout) nor other information about the performance of the Ultra PD is provided by the foundry1, and we cannot afford the cost of an iterative tunning (several runs) through design-fabrication-measurements of the pinned sensor, we opted for the conservative option of using the conventional photodiode, which is less complex as it contains a lower amount of modifications from an ordinary CMOS technology photodiode, making this option more secure for first silicon implementation success. It is important to remark that N-Well and P-implant regions are only allowed outside the pixels area. Consequently, neither PMOS devices nor substrate contacts are permitted inside the pixels. The lack of substrate contacts is compensated by a very low resistivity substrate in pixels area (described low resistive epitaxial layer). However, NMOS only circuitry inside the pixels implies a serious constraint for focal plane processing architectures. In order to characterize the performance of the conventional photodiodes in this CIS technology, the test chip (internally named as Sensors Chip from UMC or SCU) includes 12 arrays of pixels based in the 3T Active Pixels Sensors (3T-APS) structure, which is described in section 2.3.2. These arrays are addressed using an unified row and column bus (as parts of an unique sensor array whose dimension is 256x139). The chip incorporates a single output buffer to avoid column-wise fixed-pattern-noise (FPN) effects. The block diagram of the chip is shown in figure 2.3 and its main characteristics are summarized in table 2.1. Technology UMC 0.18μm4M/2P CMOS Image Sensor (CIS) Technology, Conventional Photodiode Package 84 pins Ceramic Leadless Chip Carrier (CLCC84) Pixels Arrays 8 Arrays of 64x64 pixels 4 Arrays of 11x11 pixels Pixel Pitch 3 types: 3.5, 5, 7μm Fill factor 3.5μm pitch: 46 % 5μm pitch: 60 % 7μm pitch: 70 % Number of transistors 139110 Power supply dual supply: 1.8V and 3.3V Die Size 2358x2358mm2 Table 2.1: SCU Chip Main Characteristics 1The Ultra PD is mainly devoted to company’s own developments. 38 ROW Decoder RST Decoder ROW <7:0> SELECT<7:0> RST <7:0> ROW Enable ROW Enable Crosstalk RST Enable RST Enable Crosstalk SELECT Enable Figure 2.3: Block Diagram of SCU. The chip employs a single row decoder with separated enable signals depending if the arrays are intended for either performance or crosstalk analysis (see section 2.3.1), whereas both types of arrays share the same addressing bus ROW<7:0>. Similarly, columns are addressed using a single multiplexer controlled by SELECT<7:0>and an enable signal. APS reset operation is executed row-wise (through a reset decoder) in order to allow for the Rolling Shutter operation mode (see section 2.4.2). 2.3.1. Sensors Array The SCU test chip contains 12 arrays of 3-T APS pixels. These arrays can be categorized into two clusters according to the measurement that each array is intended for: Performance analysis arrays: 8 arrays of 64x64 pixels where shape, size, source follower threshold voltage, and usage of microlenses are modified in order to study how these variations affect the performance of the photosensors. Crosstalk analysis arrays: consisting on 4 arrays of 11x11 pixels, where only the central pixel [5,5] is supposed to receive light. To that purpose, all pixels but the central one are covered by metal structures, which serve for light blocking purposes. This arrangement allows studying how the light received by one pixel affects the signal in its near neighbors. The layout of both types of arrays will be shown in the figures of subsection 2.3.3.2. Table 2.2 summarizes the variations on the different arrays in the chip (array number corresponds to that assigned in figure 2.4), where the layout of the chip is shown. 39 2. SCU: A TEST CHIP IN CMOS IMAGE SENSOR TECHNOLOGY Figure 2.4: Layout of SCU Chip. Array Characteristics Array Size Pixel Pitch (μm) Shape Low Vth Microlenses 1 64x64 3.5 Round-like -  2 64x64 3.5 Round-like - 3 64x64 3.5 Octagonal - - 4 64x64 3.5 Round-like - - 5 64x64 7 Octagonal - - 6 64x64 7 Round-like - - 7 64x64 7 Round-like - 8 64x64 7 Round-like -  9 11x11 3.5 Round-like -  10 11x11 3.5 Round-like - - 11 11x11 5 Round-like - - 12 11x11 7 Round-like - - Table 2.2: Summary of the arrays included in the chip. 40 2.3.2. Pixels Circuitry All the pixels in the chip are 3T-APS. This topology reduces read-out time and allows a nondestructive retrieval of pixel data. Just for illustration purposes, figure 2.5 shows the schematic of the pixels (using 3.3V transistors). Pixel voltage is defined as the voltage drop across the photodiode, which is read out by using a basic source follower. Indeed, only the active transistor of the source follower is included within the pixel (M1in figure 2.5). The required biasing current source (a single transistor usually) is either column-wise distributed or shared among all the pixels in the array (our case). The operation of the 3T-Active Pixel Sensor is very well known in the CIS literature and it has already been briefly described in chapter 1. Therefore, we will only provide here some highlights of how this configuration works. The typical operation of the APS sensor involves three phases, as shown in figure 2.6: 1. Reset phase: During this phase, the photodiode’s parasitic capacitor (more precisely, all capacitors connected to photodiode’s sensing node, whose aggregation is usually denoted as Cph)is initialized to the so-called reset voltage Vphrst by switching on S1. Once this process settles down, the reset switch is turned off. It is straightforward to demonstrate that the starting voltage for the Pixel 0.24 0.34 0.24 0.5 0.24 0.34 Figure 2.5: Pixel schematic. 41 2. SCU: A TEST CHIP IN CMOS IMAGE SENSOR TECHNOLOGY next phase is simply given by: Vphrst =VDDPIX −Vth −Vft ±Vn(2.1) where Vth is the threshold voltage of the reset transistor S11,Vft is the feedthrough error introduced when switching offthe reset switch [67], and Vndenotes the reset noise. 2. Exposition phase: When the reset signal is released, the photogenerated current discharges the integration node Vph. This process, which is inherently non-linear due to the non-linearities of integrating capacitors, and also due to the dependency of the photogenerated current Iph with the volume of the collection region, which is also a function of the voltage drop across the photodiode, is usually simplified to a linear process where the pixel voltage evolves as 2: Vph =Vphrst −Iph Cph Δt(2.2) 3. Readout phase: At the end of the exposition phase, whose duration is commonly referred as the exposition time or exposure, the row access transistor (S2) is enabled and the pixel is connected to the readout circuitry, producing an output that can be expressed as: Vout =G[Vph(Texp)] ·Vph(Texp)−Vshi f t[Vph(Texp)] ±Vrn (2.3) where Vph(Texp) corresponds to equation 2.2 evaluated for Δt=Texp,Gis the gain of the source follower (usually simplified to one, but which is actually a function of the pixel voltage), Vshi f t is the voltage shift created by the source follower due to the threshold voltage of transistor M1(that is also a function of the pixel voltage), and Vrn is the so-called readout noise. Figure 2.6: Pixel operation phases. 1Due to the body effect this threshold voltage also depends on Vphrst if the reset switch is an NMOS transistor. This can be eliminated if we use a PMOS switch as reset device [66], however this is not allowed in this CIS technology and, on the other hand, penalizes fill factor heavily in small pitch pixels. 2The effect of dark current and other non-idealities are not considered here for simplification. 42 2.3.3. Variations in Pixels Parameters Using the previously described APS architecture, we have introduced variations in the physical parameters of the pixels in different arrays in order to evaluate how these parameters influence the performance of the sensors. Regarding the layout of the active diffusion, we have modified shape and size [68]. Additionally, to get information about the sensitivity gained by the use of microlenses, we have included versions (arrays) of the same pixels with a without microlenses, and the same has been done for the use of low threshold voltage transistors in the source follower. Briefly, the design parameters under variations are: Pixel Size (Pitch) Active Diffusion Layout Threshold Voltage of Source Follower Transistor Use of Microlenses 2.3.3.1. Pixel Size Regarding the performance analysis arrays, two different pixel sizes are considered in the study: 3.5x3.5μm2and 7x7μm2. Their layouts are shown in figure 2.7. The 3.5μm pitch was selected because it contains the minimum photodiode area that preserves the correct angles and spacing in the connection between the active diffusion and the rest of the circuitry (as defined by UMC layout design rules). Therefore, this 3.5μm option is the minimum achievable pixel pitch for this study and the 7μm pitch pixels decrease the spatial resolution by a factor 4. The latter is expected to provide better fill-factor and sensitivity per area unit. On the other hand, the crosstalk arrays contain 11x11 pixels and selected pitches are 3.5μm, 5μm and 7μm. 2.3.3.2. Layout of Active Diffusion Studies recommend [69] to avoid abrupt angles in the layout of the active diffusion because they result in stress and malformation that lead to dark current and noise increment. Therefore, inner and outer angles are limited in the sensing diffusion. Two alternative shapes have been considered in our study: Octagonal: which results in active diffusions with 8 abrupt corners (135◦degrees) (plus 4 corresponding to the region that connects to the reset transistor, see figure 2.7). Round-like: which results in active diffusions with several “softer” corners. The shapes of the different active diffusions of the so-called “performance evaluation pixels” are shown in figure 2.7, whereas the shapes for the “crosstalk evaluation pixels” are shown in figure 2.8 which only adds the variation of having a 5μm round-like pixel. Pixels having octagonal-shaped photosensors only contain 135◦(90◦+45◦) angles in the active diffusion and they are specially intended to 43 2. SCU: A TEST CHIP IN CMOS IMAGE SENSOR TECHNOLOGY (a) 3.5μm Octagonal (b) 3.5μm Round-like (c) 3.5μm Round-like Low Vth (d) 3.5μm Round-like Microlenses (e) 7μm Octagonal (f) 7μm Round-like (g) 7μm Round-like Low Vth (h) 7μm Round-like Microlenses Figure 2.7: Pixels layout of performance analysis arrays. 44 (a) 3.5μm Round-like (b) 3.5μm Round-like Microlenses (c) 5μm Round-like (d) 7μm Round-like Figure 2.8: Pixels layout of crosstalk analysis arrays. test the dark current of sensors having the minimum amount of corners (90◦angles are not recommended by design rules). 2.3.3.3. Threshold Voltage of Source Follower Transistor According to their threshold voltage, this technology offers three types of transistors, namely: Zero, Low and Regular Vth. The use of low Vth transistors in the source follower improves the dynamic range of linear APS, since it directly increases the attainable output swing. Pixels with identical layout have been included with low or regular Vth source follower transistors. This variation is implemented in order to determine whether there is any counterpart in the use of the low Vth transistors within the pixels. Zero threshold voltage transistors have not been used since they are forbidden within the sensor array (UMC design rule). The layouts in figure 2.7 (c) and (g), differ from (b) and (f) respectively, only in an added layer that defines the source follower transistor as a low threshold device. 45 2. SCU: A TEST CHIP IN CMOS IMAGE SENSOR TECHNOLOGY 2.3.3.4. Use of Microlenses Microlenses increase sensitivity and minimize crosstalk [37]. In our design, microlenses are centered in the middle of the sensing diffusion and they focus the light to this point. It allows rays that otherwise would not hit the sensor area to arrive to the sensor surface and be detected. Clearly, this is especially important under no perpendicular incidence of light situations. The overall effect is that microlenses increase the amount of light power at the sensor surface (as compared to the same situation without microlenses) and, consequently, the number of photogenerated carriers. In order to study the performance of microlenses in this technology, pixels with the same layout with and without microlenses are included in the chip. The main purpose is to characterize how they improve sensitivity and crosstalk. Figure 2.9 shows a simplified illustration of the effect of the microlenses over vertical and lateral incident rays of light. Figure 2.10 illustrates, in a 3D View, the position of the microlens over the metal aperture of the pixel. Dimensions in this figure illustrate microlenses effect in small pitch pixels, where the size of the aperture is comparable to the height of different metal layers and passivations. The layouts of the pixels in figure 2.7 (d) and (h) are identical to (b) and (f), respectively, with the exception of the microlens layer definition, which appears as a purple crossed square over the photodiode diffusion. The microlens center has been located in the center of the photodiode diffusion, because the microlens will focus the light around this location. As the separation between microlenses is fixed, they are not completely enclosed by the pixel layout. In consequence, the pixel layout shows part of the microlenses of other pixels. The layout of the crosstalk pixel with microlens in shown in figure 2.8 (b). (a) Lateral incident light (b) Vertical incident light Figure 2.9: Microlenses effect on light. 46 Figure 2.10: Microlenses 3D ilustration. 2.3.4. Analog Output Buffer This block includes the biasing circuitry of the pixel source follower plus an additional PMOS source follower. Figure 2.11 shows the schematic of the buffer. Transistor P5partially compensates the level shifting introduced by the pixel NMOS source follower and connects to the output pad. The NMOS transistors N1and N2complete the bias current circuitry of the pixel source follower connected through the input of the output buffer. The biasing is performed through an NMOS mirror with a ratio 93 to 1. An externally applied current of 111.34μA is scaled down approximately to 1.2μA. The biasing of the PMOS section is accomplished by means of a PMOS current mirror, composed by transistors P1and P2. In this case, a cascode arrangement is added between the mirror and the source follower transistor in order to increase the output resistance of the mirror (transistors P3and P4) [70]. The bias current of the PMOS section is 44.14μA. These biasing currents can be approximately obtained by connecting a 10kΩresistor between 1.8V and IBIASN through and a 50kΩresistor between IBIASP and ground. 2.4. Control of Operation The realization of any measurement on the SCU chip requires applying control signals in the proper order. Since the SCU chip does not include any internal control unit nor sequencer, everything is left to the user, which needs to provide the signals, timing, and sequences correctly. Table 2.3 summarizes the digital and analog control signals of the chip. Table 2.4 shows the addresses and enable signals of the different arrays. 2.4.1. Image Capture Operation According to the typical 3T APS operation mode, the first operation is to reset the photodiode so that its associated capacitor is charged to a reference voltage Vphrst , as defined in equation 2.1. VDDPIX is 47 2. SCU: A TEST CHIP IN CMOS IMAGE SENSOR TECHNOLOGY The controlled stable light source is generated by a 250 Watt 24V DC Quartz Tungsten Halogen (QTH) Newport 6334 lamp. It is a visible and near infrared source with smooth spectral curve and stable output. Figure 2.20 shows its spectral irradiance. It produces an approximate flux of 10000 Lumens at 3400K. It contains a doped tungsten filament inside a quartz envelope filled with rare gas and a small amount of halogen. This lamp is mounted inside of a 66884 Research QTH Newport Lamp Housing. The housing includes a F/1 Two Element Fused Silica condenser for a 33 mm diameter collimated output beam. An included rear reflector collects the lamp’s back radiation. A power regulated fan cools the lamp and housing. This lamp housing is supplied by a stable power supply, a 69931 Newport Radiometric Power Supply. This is a highly regulated source of constant current or constant power, which includes an RS-232 interface to be controlled by a PC. Depending on the measurement to be performed, the white stable light generated by the source needs to be altered in a very accurate way in either wavelength or power. In order to produce light of a narrow wavelength, bandpass filters are applied in light path. This is used to measure the spectral response [72] of the SCU. The filters are Andover Corporation Standard Bandpass Filters. These filters have Central Wavelength (CWL) from 400 to 750nm (50nm steps) and Full Width at Half Maximum (FWHM) of 10nm in a 50mm diameter, (400FS1050 - 750FS1050). The light power received in the SCU chip is measured by combined use of a calibrated photodetector and an optical power meter. The detector is a Newport 918-SL Low-Power Silicon Photodetector, which is calibrated in the range [400 - 1100]nm. Calibration setup is electronically stored inside the detector’s head for use when connected to the used 1930-C Power Meter. Power Supply 5V-9V VDC 1.8V Regulator 3.3V Regulator 5V Regulator FGPA XC2S200E SPARTAN-IIE XILINX System Clock Configuration jumpers Push buttons PROM Flash XILINX XCF02S RAM 4M x 16 bits RS-232 JTAG USB DA Converter AD Converter ASIC SCU Biasing Configuration switches Configuration jumpers Figure 2.16: SCU PCB test board blocks. 54 5 5 4 4 3 3 2 2 1 1 D D C C B B A A USB_RXF# USB_WR USB_TXE# USB_RD# USB_PWREN# USB_D3 USB_D0 USB_D2 USB_D1 USB_D7 USB_D4 USB_D5 USB_D6 USB_D0 5V_DIG USB_D2 USB_WR USB_PWREN# USB_RD# USB_TXE# USB_D5 USB_D6 USB_D1 USB_RXF# USB_D3 USB_D4 USB_D7 5V_DIG 5V_DIG 3.3V_DIG DIN TMS 5V_DIG 3.3V_DIG TDI USB_D[0:7] CCLK TCK INIT 1.8V_DIG 3.3V_DIG 3.3V_DIG 3.3V_DIG RTS_RS232 CTS_RS232 RXD_RS232 TXD_RS232 DSR_RS232 DSR_FPGA CTS_FPGA RTS_FPGA RXD_FPGA TXD_FPGA DSR_FPGA TXD_FPGA CTS_FPGA RXD_FPGA RTS_FPGA A0 A1 A2 A3 A4 A5 A7 A6 A8 A9 A11 A10 A[11:0] USB_D[0:7] USB_D3 USB_D0 USB_D1 USB_D2 USB_D7 USB_D6 USB_D4 USB_D5 USB_WR USB_RXF# USB_PWREN# USB_TXE# USB_RD# 3.3V_DIG A1 A3 A5 A7 A9 BA0 A11A10 A2 A8 A6 BA1 A0 A4 DQ[15:0] LDQM 3.3V_DIG 3.3V_DIG 3.3V_DIG 3.3V_DIG RST_0 ROW_0 SELECT_0 RST_1 RST_2 RST_3 RST_4 RST_5 RST_6 RST_7 ROW_1 ROW_2 ROW_3 ROW_4 ROW_5 ROW_6 ROW_7 SELECT_1 SELECT_2 SELECT_3 SELECT_4 SELECT_5 SELECT_6 SELECT_7 RST_0 ROW_0 SELECT_0 RST_1 RST_2 RST_3 RST_4 RST_5 RST_6 RST_7 ROW_1 ROW_2 ROW_3 ROW_5 ROW_6 ROW_7 SELECT_1 SELECT_2 SELECT_3 SELECT_4 SELECT_5 SELECT_6 SELECT_7BA1 BA0 DQ7 DQ6 DQ5 DQ4 DQ3 DQ2 DQ1 DQ0 DQ8 DQ9 DQ10 DQ11 DQ12 DQ13 DQ14 DQ15 A3 A4 A5 A7 A6 A8 A9 A11 A10 A[11:0] DQ14 DQ15 BA1 DQ0 DQ1 DQ2 DQ3 DQ4 DQ5 DQ6 DQ[15:0] DQ7 DQ8 DQ9 DQ10 DQ11 DQ12 DQ13 A0 A1 A2 BA0 UDQM LDQM UDQM WEn CLK CASn CKE RASn CSn DQ7 DQ6 DQ5 DQ4 DQ3 DQ2 DQ1 DQ0 DQ8 DQ9 DQ10 DQ11 DQ12 DQ13 DQ14 DQ15 CKECLK CSn 3.3V_DIG 3.3V_DIG 1.8V_DIG 3.3V_DIG ADC_DO ADC_D1 ADC_D2 ADC_D3 ADC_D5 ADC_D4 ADC_D6 ADC_D7 ADC_D8 ADC_D9 ADC_D10 ADC_D11 DAC_DB0 DAC_DB1 DAC_DB2 DAC_DB3 DAC_DB4 DAC_DB5 DAC_DB6 ADC_DO ADC_D2 ADC_D3 ADC_D4 ADC_D6 ADC_D5 ADC_D7 ADC_D8 ADC_D9 ADC_D10 ADC_D11 DAC_DB1 DAC_DB2 DAC_DB3 DAC_DB4 DAC_DB0 DAC_DB7 DAC_DB6 DAC_DB5 CLK_ADC 3.3V_DIG 3.3V_ANA 3.3V_ANA 3.3V_DIG OE PDOWN CLK_ADC GC GC OTR DAC_DB7 OTR HBEN nCS nWR nCLR nLDAC nPD BUF GAIN 3.3V_DIG 3.3V_ANA 3.3V_DIG HBEN nWR nCLR nLDAC ROW_EN_CROSS SELECT_EN RST_EN ROW_EN RST_EN_CROSS ROW_EN_CROSS SELECT_EN RST_EN ROW_EN RST_EN_CROSS 3.3V_ANA 5V_ANA 5V_DIG 5V_ANA RASn 3.3V_DIG 3.3V_DIG 3.3V_DIG 3.3V_DIG 3.3V_DIG WEn TDO 3.3V_DIG 3.3V_DIG 3.3V_DIG3.3V_DIG 3.3V_DIG 3.3V_DIG DONE CASn RASn CKE CASn CSn WEn CLK ROW_4 nPD BUF 3.3V_DIG 3.3V_DIG OE PDOWN 3.3V_ANA 1.8V_DIG 1.8V_DIG 3.3V_ANA 1.8V_DIG 1.8V_DIG 1.8V_DIG 1.8V_DIG 3.3V_ANA 3.3V_ANA 3.3V_ANA 3.3V_ANA 3.3V_ANA 3.3V_ANA GAIN nCS ADC_D1 3.3V_ANA Title Size Document Number Rev Date: Sheet of <Doc> 1.0 TEST BOARD DIVISA-SCU D 11Friday, December 14, 2007 Title Size Document Number Rev Date: Sheet of <Doc> 1.0 TEST BOARD DIVISA-SCU D 11Friday, December 14, 2007 Title Size Document Number Rev Date: Sheet of <Doc> 1.0 TEST BOARD DIVISA-SCU D 11Friday, December 14, 2007 DECOUPLING CAPS FPGADECOUPLING CAPS FPGA DECOUPLING CAPS FPGA DECOUPLING CAPS USB USB MODE2MODE1MODE0 SRAM DECOUPLING CAPS ASIC GROUND PINS AUXILIARY PADS DECOUPLING CAPS ASIC DECOUPLING CAPS ASIC 5V-9V ¿? 1 1 COLOCAR ATRAS R51 ¿? R51 ¿? SW1 nPROGRAM SW1 nPROGRAM C31 100n C31 100n R2 10k R2 10k C81 100n C81 100n L2 680nHL2 680nH C17 100n C17 100n C57 100n C57 100n R33 4.7k R33 4.7k R6 1k R6 1k C93 100n C93 100n TORN3TORN3 R29 4.7K R29 4.7K JP4JP4 C53 100n C53 100n C12 100n C12 100n D2 LED D2 LED J7J7 12 C67 C>47uF C67 C>47uF R45 45k R45 45k C73 100n C73 100n R30 100 R30 100 C28 100n C28 100n R16 51(1206) R16 51(1206) C41 100n C41 100n R22 10k R22 10k C107 100n C107 100n SW3 SW DIP-4/SM SW3 SW DIP-4/SM P1 CONNECTOR DB9 P1 CONNECTOR DB9 5 9 4 8 3 7 2 6 1 C100 10uF C100 10uF R15 1k5R15 1k5 R36 390 R36 390 C43 22pC43 22p C5 100n C5 100n R46 1 R46 1 TP3TP3 DAC U17 AD5341 DAC U17 AD5341 HBEN 1 BUF 2 VREF 3 VOUT 4 GND 5 CS 6 WR 7 GAIN 8 CLR 9 LDAC 10 PD 11 VDD 12 DB0 13 DB1 14 DB2 15 DB3 16 DB4 17 DB5 18 DB6 19 DB7 20 C48 100n C48 100n TORN2TORN2 R21 2k2 R21 2k2 R10 1k R10 1k TP5TP5 C20 100n C20 100n C33 33nF C33 33nF C24 100n C24 100n C104 100n C104 100n C38 10uF C38 10uF TP1TP1 R12 no montar R12 no montar C82 100n C82 100n J8J8 12 C58 100n C58 100n A B J1 A B J1 1 2 3 R8 10k R8 10k C86 100n C86 100n C3 100n C3 100n C94 100n C94 100n U11 XCF02S U11 XCF02S DO 1 (DNC) 2 CLK 3 TDI 4 TMS 5 TCK 6 CF_N 7OE/RESET_N 8 (DNC) 9 CE_N 10 GND 11 (DNC) 12 CEO_N 13 (DNC) 14 (DNC) 15 (DNC) 16 TDO 17 VCCINT 18 VCCO 19 VCCJ 20 C101 10uF C101 10uF C74 100n C74 100n C47 100n C47 100n C89 100n C89 100n C54 100n C54 100n C37 100n C37 100n C69 100n C69 100n TORN4TORN4 C14 100n C14 100n R28 100R28 100 C7 100n C7 100n C92 100n C92 100n R32 4.7k R32 4.7k R37 390 R37 390 D3 LED D3 LED C42 100n C42 100n C66 10uF C66 10uF R47 no montar R47 no montar J5 CONN RECT 16X2 J5 CONN RECT 16X2 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 C79 10uF C79 10uF C62 100n C62 100n R9 1k R9 1k U18 CON_JACKU18 CON_JACK Vdc 1 GND 2 GND 3 U16 LM1117MP-3.3 U16 LM1117MP-3.3 VIN 3 GND 1 VOUT 2 VOUT 4 CON2 JTAG PROG CON2 JTAG PROG 1 2 3 4 5 6 C44 22pC44 22p JP1JP1 C9 100n C9 100n J3 CONN RECT 4x2 J3 CONN RECT 4x2 2 4 6 8 1 3 5 7 XTAL OSC 50MHz XT1 CFPS-39 XTAL OSC 50MHz XT1 CFPS-39 ENABLE 1 GND 2 OUTPUT 3 VCC 4 TORN5TORN5 D1 LED D1 LED SCU ASIC U7 ZOCALO_DIVISA SCU ASIC U7 ZOCALO_DIVISA RST_0 69 RST_1 68 RST_2 67 RST_3 66 RST_4 61 RST_5 60 RST_6 59 RST_7 58 ROW_0 77 ROW_1 82 ROW_2 83 ROW_3 84 ROW_4 1 ROW_5 6 ROW_6 7 ROW_7 8 SELECT_0 25 SELECT_1 24 SELECT_2 23 SELECT_3 22 SELECT_4 17 SELECT_5 16 SELECT_6 15 SELECT_7 14 RST_EN 74 ROW_EN 75 RST_EN_CROSS 76 ROW_EN_CROSS 9 SELECT_EN 30 VCASP 37 IBIASP 38 IBIASN 40 OUT 39 VDDPIX 36 V3IO 10 V3IO 18 V3IO 26 V3IO 31 V3IO 54 V3IO 62 V3IO 70 V3IO 78 V3IO 2 V0IO 27 V0IO 32 V0IO 55 V0IO 63 V0IO 71 V0IO 19 V0IO 79 V0IO 3 V0IO 11 VDD 4 VDD 12 VDD 20 VDD 28 VDD 56 VDD 64 VDD 72 VDD 80 VSS 5 VSS 13 VSS 21 VSS 29 VSS 57 VSS 65 VSS 73 VSS 81 VDD33_DIG 47 VDD33_DIG 48 VDD33_DIG 49 VDD33_DIG 50 VAD 33 VADC 52 VAD 41 VAD 44 VADC 42 VAS 35 VAS 46 VAS 43 VAS 53 VASC 34 VASC 45 VAD 51 R42 10k R42 10k C36 100n C36 100n R3 10k R3 10k C39 100n C39 100n J10J10 12 C40 100n C40 100n R25 ¿? R25 ¿? C26 100n C26 100n R14 27RR14 27R R41 390 R41 390 U2 XC2S200E/PQ208 U2 XC2S200E/PQ208 VCCO 13 VCCO 26 VCCINT 37 VCCINT 28 VCCO 38 VCCO 53 VCCO 66 VCCINT 67 I/O 3 I/O 4 I/O 5 I/O LP49 6 I/O L49N 7 I/O 8 I/O 9 I/O L48P 10 I/O L48N 11 I/O L47N_YY 16 VCCINT 14 I/O L46P_YY 17 I/O L46N_YY 18 I/O L45P 20 I/O L45N 21 I/O 22 I/O L44P_YY 23 I/O L44N_YY 24 I/O 27 I/O 29 I/O L43P 30 I/O L43N 31 I/O L42P_YY 33 I/O L42N_YY 34 I/O L41P_YY 35 I/O L41N_YY 36 I/O L40P 40 I/O L40N 41 I/O 42 I/O 43 I/O 44 I/O L39P 45 I/O L39N 46 I/O 47 I/O L38P_YY 48 I/O L38N_YY 49 I/O L37P_YY 56 I/O 57 I/O 58 I/O L36P_YY 60 I/O L35N 61 I/O L35P 62 I/O L34N 63 I/O L34P 64 I/O L33N 68 I/O L33P 69 I/O 70 I/O L32N 71 I/O L32P 73 I/O 74 I/O L31N 75 I/O L31P 81 I/O 82 I/O L30N 83 I/O L30P 84 I/O L29N 86 I/O L29P 87 I/O L28N 88 I/O L28P 89 M1 50 M2 54 GND 12 GND 19 GND 25 GND 32 GND 39 GND 51 GND 65 GND 72 DONE 104 I/O L1P 198 I/O 201 I/O L0P_YY 202 I/O L0N_YY 203 I/O 204 I/O L5N 187 I/O L4P 188 I/O L4N 189 I/O L3P 191 I/O L2N 194 I/O L1N 199 I/O 200 I/O L8N 174 I/O L7P 175 I/O L7N 176 I/O L6P 178 I/O 180 I/O L5P 181 DIN 153 I/O L12P_YY 154 I/O 162 I/O 163 I/O L10P_YY 164 I/O 167 I/O L9P 168 I/O L9N 169 I/O L15N 140 I/O L15P 141 I/O L14N 145 I/O L14P 146 I/O 147 I/O L13N 150 I/O L13P 151 I/O 152 I/O 127 I/O 129 I/O 134 I/O L17N 135 I/O L17P 136 I/O L18P_YY 133 I/O L16N 138 I/O L16P 139 I/O L21N_YY 120 I/O L21P_YY 121 I/O L20N_YY 122 I/O L19N 125 I/O L19P 126 I/O L25N_YY 101 I/O L25P_YY 102 I/O 110 I/O L23N 111 I/O L23P 112 I/O 113 I/O L22N 115 I/O L22P 116 I/O L27N 93 I/O L27P 94 I/O 95 I/O 96 I/O L26N_YY 97 I/O L26P_YY 98 I/O 99 I/O 100 GCK0 80 GCK1 77 TMS 2 I/O 206 I/O 205 M0 52 GND 1 I/O L47P_YY 15 I/O L37N_YY 55 I/O L36N_YY 59 VCCINT 76 VCCINT 90 VCCO 78 GND 79 GND 85 GND 92 GND 103 GND 117 GND 124 VCCO 91 VCCO 105 VCCO 118 VCCINT 119 PROGRAM 106 INIT 107 I/O L24P_YY 108 I/O 109 I/O L20P_YY 123 VCCINT 128 VCCO 130 VCCINT 142 VCCO 143 GND 131 GND 137 GND 144 GND 158 GND 170 GND 177 GND 183 I/O L18N_YY 132 I/O 148 I/O 149 VCCO 156 VCCO 171 VCCINT 172 VCCO 184 VCCINT 186 VCCINT 195 TDO 157 TDI 159 I/O L11P_YY 160 I/O L11N_YY 161 I/O 166 I/O L10N_YY 165 I/O L8P 173 I/O L6N 179 GCK2 182 GCK3 185 GND 190 GND 197 I/O L3N 192 I/O L2P 193 VCCO 196 TCK 207 VCCO 208 CCLK 155 I/O 114 C105 0 ohm C105 0 ohm J2 SMB J2 SMB C83 100n C83 100n C59 100n C59 100n U14 IS42S16400U14 IS42S16400 A0 23 A1 24 A2 25 A3 26 A4 29 DQ0 2 DQ1 4 DQ2 5 DQ3 7 DQ4 8 DQ5 10 DQ6 11 DQ7 13 DQ8 42 DQ9 44 DQ10 45 DQ11 47 DQ12 48 DQ13 50 DQ14 51 DQ15 53 VDD 1 VDDQ 43 GND 28 CS 19 WE 16 RAS 18 CAS 17 LDQM 15 UDQM 39 NC 36 A5 30 A6 31 A7 32 A8 33 A9 34 A11 35 BA0 20 BA1 21 VDD 14 VDDQ 3 VDDQ 9 VDDQ 49 VDD 27 GNDQ 6 GNDQ 12 GND 41 GND 54 GNDQ 46 GNDQ 52 CLK 38 CKE 37 NC 40 A10 22 U20 L4940 U20 L4940 VIN 1 GND 4 VOUT 3 C95 100n C95 100n C77 100n C77 100n R23 10k R23 10k R26 100R26 100 C49 100n C49 100n C60 100n C60 100n C65 100n C65 100n C16 100n C16 100n C55 100n C55 100n TORN7TORN7 C61 100n C61 100n C34 100n C34 100n C71 10uF C71 10uF C10 100n C10 100n R4 1k R4 1k U21 ADR366 U21 ADR366 NC 1 GND 2 VIN 3VOUT 4 TRIM 6 C91 100n C91 100n C22 100n C22 100n J_GND_DIGJ_GND_DIG 1 2 C51 100n C51 100n SW2 nRESET SW2 nRESET C19 100n C19 100n U5 93C46/56/66 U5 93C46/56/66 CS 1 SK 2 DIN 3 DOUT 4 VCC 8 NC 7 NC 6 GND 5 R48no montar R48no montar SW4 SW DIP-4/SM SW4 SW DIP-4/SM U1 FT245BL U1 FT245BL D0 25 D1 24 D2 23 D3 22 D4 21 D5 20 D6 19 RSTOUT# 5 XTOUT 28 RESET# 4 EECS 32 EESK 1 EEDATA 2 AGND 29 GND 9 D7 18 GND 17 TEST 31 TXE# 14 WR 15 RD# 16 RXF# 12 PWREN# 10 VCC-IO 13 3V3OUT 6 AVCC 30 VCC 26 VCC 3 USBDM 8 USBDP 7 XTIN 27 SI/WU 11 J11J11 12 RN1 10k RN1 10k 1 2 3 4 5 6 7 8 R11 1.3k R11 1.3k C4 100n C4 100n L1 680nHL1 680nH C97 100n C97 100n C76 100n C76 100n C102 100n C102 100n C25 100n C25 100n R19 0 ohm R19 0 ohm C45 100n C45 100n C70 10uF C70 10uF JP2JP2 C87 100n C87 100n L4 680nHL4 680nH C35 10uF C35 10uF R35 ¿? R35 ¿? C90 10uF C90 10uF C75 100n C75 100n TORN6TORN6 C13 100n C13 100n C18 100n C18 100n R1 470R R1 470R R43 10k R43 10k R13 27RR13 27R C21 100n C21 100n R18 10k R18 10k C84 100n C84 100n C63 100n C63 100n R7 10K R7 10K J_GND_ANAJ_GND_ANA 1 2 C96 100n C96 100n R44 5k R44 5k C15 100n C15 100n R50 ¿? R50 ¿? R31 4.7K R31 4.7K C30 100n C30 100n C80 100n C80 100n ADC U15 AD9237 ADC U15 AD9237 D0(LSB) 7 D1 8 D2 9 D3 10 D4 11 D5 12 D6 13 D7 14 D8 17 D9 18 D10 19 D11(MSB) 20 VIN+ 29 VIN- 30 OTR 21 MODE2 1 SENSE 23 VREF 24 REFB 25 REFT 26 AVDD 27 AVDD 32 AGND 28 AGND 31 CLK 2 PDWN 4 DRVDD 16 DGND 15 OE 3 GC 5 NC 6 MODE 22 EX-PAD 33 C56 100n C56 100n R5 1k R5 1k C106 100n C106 100n L3 680nHL3 680nH C8 100n C8 100n C50 100n C50 100n C72 100n C72 100n TORN8TORN8 R39 R R39 R C27 100n C27 100n R34 100 R34 100 C52 100n C52 100n R49 25 R49 25 JP3JP3 C1 100n C1 100n RS-232 TTL/CMOS U10 MAX3241ECAI / SSOP RS-232 TTL/CMOS U10 MAX3241ECAI / SSOP R2OUTB 20 R1OUTB 21 GND 25 VCC 26 C1+ 28 C1- 24 C2+ 1 C2- 2 V+ 27 V- 3EN 23 SHDN 22 T1IN 14 T2IN 13 T3IN 12 R1IN 4 R2IN 5 R3IN 6 R4IN 7 R5IN 8 T1OUT 9 T2OUT 10 T3OUT 11 R1OUT 19 R2OUT 18 R3OUT 17 R4OUT 16 R5OUT 15 C85 10uF C85 10uF TP2TP2 C103 100n C103 100n TP4TP4 J4J4 12 R20 0 omh R20 0 omh TORN1TORN1 C6 100n C6 100n C23 100n C23 100n U12 LM1117MP-1.8 U12 LM1117MP-1.8 VIN 3 GND 1 VOUT 2 VOUT 4 C32 100n C32 100n R40 1 R40 1 C11 100n C11 100n C68 no montar C68 no montar C46 100n C46 100n XT2 6MHz XTAL XT2 6MHz XTAL R17 4.7k R17 4.7k C2 100n C2 100n R27 100R27 100 J9 CON6 J9 CON6 1 2 3 4 5 6 RN2 10k RN2 10k 1 2 3 4 5 6 7 8 J6 CONN RECT 7x2 J6 CONN RECT 7x2 2 4 6 8 10 12 14 1 3 5 7 9 11 13 CON1 CN-USB CON1 CN-USB 1 2 3 4 5 6 Figure 2.17: SCU PCB test board schematic. 55 2. SCU: A TEST CHIP IN CMOS IMAGE SENSOR TECHNOLOGY Figure 2.18: SCU PCB test board. Lamp Color Filters SCU Sample Beam Splitter Light Power Meter Figure 2.19: SCU measurement setup. 56 Block Components Function CMOS Image Sensor Chip SCU ASIC Chip in CIS 0.18μm4M/2P technology Sensor under test FPGA Xilinx Spartan II-E XC2S200E package PQ208 Central Control System PROM Flash memory Xilinx XCF02S 2 Mbit Store FPGA program RAM Integrated Silicon Solution Synchronous Dynamic RAM IS42S16400D 1Mx16bitsx4 banks Store images from SCU Digital to Analog Converter Analog Devices AD5341B Generate VDDPIX Voltage Analog to Digital Converter Analog Devices AD9327BCPZ-65 Digitize pixel data JTAG port 6 pins JTAG connector Port to program the FPGA from PC RS232 port MAXIM MAX3241 Serial communication with PC USB port FTDI FT245BL Serial communication with PC ASIC biasing circuit Potentiometers and resistors Generate VCAS P Voltage, IBIASP and IBIASN currents Clock generator IQD Frequency Products CFPS-39 crystal 50Mhz oscillator System clock Voltage supply generators National Semiconductor LM1117MP-3.3 and LM1117MP-1.8, ST Microelectronics L4940D2T5 5V Generate analog and digital supply voltages Configuration jumpers 3 jumpers for FPGA, 5 jumpers for ADC, 2 jumpers for clock Configure FPGA mode of operation, ADC and clock Configuration switches 2 switches for ADC, 4 switches for DAC Configure DAC and ADC operation Manual user controls 2 push buttons PROGRAM and RESET FPGA buttons State leds 2 leds Inform of active power supply and successful FPGA programming Table 2.6: SCU PCB board devices. 2.5.2. Spectral Sensitivity Spectral sensitivity has been measured for the range from 400nm to 900nm, in 50nm steps. Results are shown in figure 2.21 where the corresponding dark current contribution has been subtracted1. Results are the average values of the array of pixels taking a central region of interest (ROI), which discards 5 rows and columns on each side, since those might be affected by the surrounding circuitry. The discharging rate is estimated from a least square linear polynomial fit of 20 measurement points equally distributed in time. The first important highlight is that it is clearly noticeable that the round-like diffusion layout has better sensitivity for both 3.5 and 7μm pixel pitches. Although the aim of using round-like pixels was to diminish dark current due to the less stressing angles, we also find that it increases sensitivity. The reason for this seems to be based on the fact that round and octagonal pixels have the same aperture in metals but the round-like diffusions have less area than octagonal ones, therefore their associated parasitic capacitance at the sensing node is smaller. A smaller sensing area produces a smaller volume of the collection region underneath the diffusion. However, it seems that lateral collection and carrier diffusion into the collection region is significant. Consequently, in this case, the smaller capacitance, 1The measurements are the average data of the interpolated slope of discharge minus the dark current contribution and divided by the light power applied at the different wavelengths. 57 2. SCU: A TEST CHIP IN CMOS IMAGE SENSOR TECHNOLOGY Figure 2.20: Newport 6334 lamp spectral irrandiance. which is direct consequence of the smaller area, produces a significant sensitivity improvement. We can also observe that pixels with low threshold voltage transistors in the source follower have a slightly better sensitivity than the same pixels with regular transistors, something that adds to the positive effect of the expansion of the output voltage swing in about 0.4V. This effect is simply explained by the fact that low-Vth transistors have smaller gate capacitance than regular ones. Therefore, as it happened in the previous case, the overall capacitance at the integration node is smaller for the same collection region, thus producing a gain in the sensitivity of the sensor. The expansion of the output swing due to the low threshold voltage source follower can be seen in the static conversion gain of the signal path from VDDPIX to the output pad, which is shown in figure 2.22. Finally, we have also compared the results at 550nm, shown in table 2.7, with the results of a similar set of measurements over a test chip previously designed by our group, which employed the standard corresponding technology (UMC CMOS 0.18μm, but using 1.8V transistors[73]). Measurements in the standard technology resulted in a sensitivity of 0.1655 V lux·sfor 5μm pitch pixels and 0.0958 V lux·s for 3μm pitch pixels. Hence, using the CIS technology increases sensitivity by about ten times at this wavelength. 58 400 450 500 550 600 650 700 750 800 850 900 0 200 400 600 800 1000 1200 1400 1600 1800 2000 Wavelength(nm) Sensitivity (V/s / W/m2) 3.5μm octagonal 3.5μm round 3.5μm round lvt 3.5μm round μlenses 400 450 500 550 600 650 700 750 800 850 900 0 200 400 600 800 1000 1200 1400 1600 1800 2000 Wavelength(nm) Sensitivity (V/s / W/m2) 3.5μm octagonal 3.5μm round 3.5μm round lvt 3.5μm round μlenses 7μm octagonal 7μm round 7μm round lvt 7μm round μlenses Figure 2.21: Spectral Response. Array Sensitivity( V lux·s) 3.5 round μlens 1.37 3.5 round lvt 0.918 3.5 octagonal 0.837 3.5 round 0.858 7 round octagonal 1.84 7 round 2.08 7 round lvt 2.23 7 round μlens 2.42 Table 2.7: Sensitivities at 550nm. 2.5.3. Dark Current A very important topic in image sensors is the dark current effect, as this effect determines the minimum amount of detectable light power [74]. Dark current measurements have been taken in complete darkness at 20 different exposition times, ranging from 300ms to 5s. Repeated measurements are the average values of every array of pixels, using the same Region of Interest (ROI) as in the sensitivity measurements. Figure 2.23 shows the evolution of the pixel voltage (average) in total darkness, whereas table 2.8 summarizes the results for all the arrays in the chip. First of all, it can be observed that, the average dark signal, which is mainly produced by the effect of dark current and integrating capacitor, expressed as Idark Cpix , is about 42.6mV sfor the 3.5μm pitch pixels and around 16.9mV sfor the 7μm pitch pixels. This certainly expresses a substantial enhancement considering the previous measurements made in the standard CMOS 0.18μm UMC technology, producing results of 340mV sfor 5μm pitch pixels and 1.22V sfor 3μm [73]. 59 2. SCU: A TEST CHIP IN CMOS IMAGE SENSOR TECHNOLOGY 0.5 1 1.5 2 2.5 3 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 Vddpix(V) Output Voltage(V) 3.5μm octagonal 7μm octagonal 3.5μm round 7μm round 3.5μm round lvt 7μm round lvt 3.5μm round μlenses 7μm round μlenses Figure 2.22: Conversion gain curves from VDDPIX to out pad for all the arrays. 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 Exposition Time(ms) Pixel Voltage(V) 3.5μm round μlenses 3.5μm octagonal 7μm octagonal 7μm round lvt 3.5μm round lvt 3.5μm round 7μm round 7μm round μlenses Figure 2.23: Discharge in darkness. 60 Array Discharging rate in darkness( mV s) 3.5 round μlens 40.358 3.5 round lvt 52.271 3.5 octagonal 35.28 3.5 round 42.408 7 round octagonal 15.018 7 round 17.654 7 round lvt 17.206 7 round μlens 17.654 Table 2.8: Discharge slope in darkness. Secondly, it is noticeable that the octagonal pixels, in both sizes, feature the lower dark signal. Although this seems to be contradictory to our assumptions about dark current and abrupt angles, we cannot conclude the opposite at this point, since we need to bear in mind that what we are measuring here is the combined effect of dark current and parasitic capacitance, and we need to check SNR measurements (combining sensitivity and dark signal) in order to extract a meaningful conclusion. Thirdly, concerning the low threshold voltage in the source follower, we have to consider that these pixels exhibit a higher leakage due to the special doping profile in the channel and the thinner gate oxide [75]. Observing the result for the 3.5μm pitch pixels, we easily identify that it low-Vth pixels show the highest dark signal for this pitch. However, if we examine the 7μm pitch pixels, the relative difference is only 2%, a possible reason for that is the fact that in the 3.5μm pitch pixels, the smaller gate capacitance of the source follower reduces significantly the global capacitance at the integration node (as confirmed by sensitivity measurements) whereas this is not so important in the 7μm pitch pixels, where the photodiode’s parasitic capacitance is largely dominant. 2.5.4. Dynamic Range Figure 2.24 shows the Signal to Noise Ratio (here noise includes both readout noise and dark current effects). Obviously, dark current discharge will be the most remarkable contribution over the time (as readout noise power is not expected to increase with the exposition time). For the 7μm pitch pixels, it is observed that the low Vth exhibits the best SNR, as predictions forecast. The main reason for this is the 0.4V extended output swing due to the low-Vth source follower. Second position is occupied by the octagonal pixels, due to their lower dark signal, with the rest of options exhibiting very similar figures. The 3.5μm pitch pixels set exhibits a slightly different behavior. In this case, the octagonal option shows the best performance, due to its lowest dark contribution with the low-Vth getting the second position in this figure of merit. In this case, the expansion of the output swing is not sufficient to counterpart the leakage of the low-Vth transistor. As in the 7μm pitch pixels, the rest of options exhibit very similar SNR. 61 2. SCU: A TEST CHIP IN CMOS IMAGE SENSOR TECHNOLOGY 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 20 25 30 35 40 45 50 55 Exposition Time(ms) Signal to Noise Ratio(dB) 3.5μm octagonal 7μm octagonal 3.5μm round 7μm round 3.5μm round lvt 7μm round lvt 3.5μm round μlenses 7μm round μlenses Figure 2.24: Signal to Noise Ratio. 2.5.5. Performance Comparison We have discussed in the two previous sections how the octagonal pixels exhibit lower dark signal than the round-shaped pixels, and how this seems to be contradictory to the generally accepted assumption. Indeed our design does not allow us to measure dark current directly but its effect in the discharging rate of the photodiode’s associated capacitance in absolute darkness. Since the integration capacitance in round and octagonal pixels are different, measurements about dark signal are not absolutely representative of the real performance of the two shapes. On the other hand, we already described how the smaller associated capacitor of round-shaped pixels produced higher sensitivity than in octagonal pixels. Since our design does not allow us (this has certainly become a lack of the design) to extract the photodiode’s parasitic capacitances and simulation models available through Europractice are not precise at all, we have created a Figure Of Merit (FOM), which combines sensitivity and SNR. Indeed, if we consider that sensitivity increases as the parasitic capacitor shrinks down, whereas SNR increases as this capacitance grows, defining a FOM as: FOM =SNR×S ensitivity (2.6) may allow us to decide which of the many sensors in the SCU performs better, and to run comparisons considering pitch, shape, etc. Figure 2.25 shows the results for the FOM when the sensitivity is evaluated at 550nm. Results are clearer in this case, with round pixels with microlenses featuring the best FOM for the two pitches, followed by the round pixels including Low-Vth source follower transistor. Finally, when 62 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 2 3 4 5 6 7 8 x 104 Exposition Time(s) Signal to Noise Ratio(dB) x Sensitivity(V/s/W/m2@550nm) 3.5μm octagonal 7μm octagonal 3.5μm round 7μm round 3.5μm round lvt 7μm round lvt 3.5μm round μlenses 7μm round μlenses Figure 2.25: SNRxSensitivity comparison comparing the performance of the round and octagonal shapes without any additional extra feature, we see that for the 7μm pitch pixels, the round diffusions obtain a better FOM whereas for the 3.5μm pitch case, both shapes behave very similarly, as in this case, the gate capacitance of the source follower is an important component of the total integration capacitance. 2.5.6. Microlenses Effect In order to observe the effect of the microlenses, we can simply consider the sensitivity results (figure 2.21) for the arrays including microlenses. For a simpler visual evaluation, figure 2.26 shows the ratios in the sensitivity of the same pixel with and without microlenses for the two pitches employed in this design. First of all, it is noticeable that the use of microlenses significantly improves the response of the 3.5μm pitch pixels, and it affects positively in the 7μm pitch pixels but at a lower ratio. Obviously, this improvement difference is caused by the relationship between pixel pitch, microlenses radius and thickness of the metals and passivations on top of the sensor. In the case of the 3.5μm pitch pixels, lateral dimensions are comparable to vertical dimensions and, therefore, the amount of light with large angle of incidence redirected into the sensor area is comparatively higher for this pixel than for the larger 7μm pitch pixels. 63 3. TONE MAPPING ALGORITHM Good colour vision Good acuity Visual function Absolute rod threshold Cone threshold Rod saturation begins Damage possible No colour vision Poor acuity SCOTOPIC MESOPIC PHOTOPIC Typical ambient light levels Starlight Indoor lighting Moonlight Sunlight Figure 3.2: Light levels effect on photoreceptors visual function. becomes incapable of discriminating details. Finally, the superior boundary for cone system implies a physical damage (Damage possible) to the photoreceptors, such as stare directly to the sun. Regarding light wavelength, rods acquire illumination in a monochromatic way while cones perceive colors. In order to perform color perception, there are three types of cones: L, M and S as they perceive mainly red, green and blue, respectively. Combining these mechanisms, the human eye is sensitive to light between approximately 380 to 830 nanometers. 3.2.1.2. Tone Mapping Techniques: Basic Concepts The perception of the illumination in terms of brightness is usually used in tone mapping algorithms. The perception of brightness is expressed as Luminance ( cd m2). It constitutes an approximate measure of how bright a surface appears to a human person. Luminance is in fact the most relevant photometric unit to HDR imaging [79]. It is generated weighting the measured radiance to the human perception characteristic in wavelength. The perception of brightness is not linear in humans but it is compressed toward higher values. Moreover, it approximately follows a logarithmic behavior over a broad range of illuminations. This is proposed by the Webber-Fechner law. Weber’s law was first described in 1834 by the German physiologist Weber [80], and was later formulated quantitatively in 1860 by the experimental psychologist Fechner [81], founder of the modern psychophysics. In psychophysical studies, human visual adaptation is evaluated by measuring the minimum amount of incremental light by which an observer distinguishes a test object from the background light. This minimum increment is called a visual threshold or Just- Noticeable Difference (JND). Weber stated that the change in stimulus intensity that produces a just noticeable difference in a human sensation is a constant fraction of the starting intensity of a stimulus. In other words, the minimum amount by which a stimulus magnitude must be changed in order to produce a noticeable variation in sensory experience is a constant fraction of the level of stimulus intensity. So, Weber’s Law, which has wide generality across different sensory magnitudes and modalities, is ex- 70 pressed in equation 3.1 where cis a constant and Iis the level of stimulus intensity. The constant nature of this fraction suggests that visual adaptation acts as a normalizer, scaling scene intensities to preserve our ability to sense contrasts within scenes. ΔI I=c(3.1) Fechner based his work on Weber’s Law. He proposed that as the stimulus intensity increases, it takes greater and greater changes in intensity to change the perceived magnitude by some constant amount. Therefore, the perceived magnitude is a logarithmic function of stimulus intensity multiplied by a specific constant. Fechner’s Law relationship is expressed in equation 3.2, where Sis the perceived magnitude, kis a constant and Iis the stimulus intensity. S=k·log(I) (3.2) These fundamental laws are usually valid for general sensory phenomena and can account for many properties of sensory neurons. However, the human visual perception is not necessarily logarithmic in the entirety of the dynamic range and the logarithmic law is not the only psychophysical law to explain brightness perception. The Stevens’ Power Law was introduced in 1957 by Stanley Smith Stevens [82]. It proposed that perception follows a power law as expressed in equation 3.3, S=k·Iγ(3.3) where Sis the perceived magnitude, kis a constant, Iis the stimulus intensity and γis an exponent constant that depends on the type of stimulation. In practice, the Weber law is a good approximation for bright illumination conditions and for lower adaptation. Luminance levels power functions fit better to the experimental data. However, it is indubitable that, apart from the accuracy of the perception function itself, the theories propose the increasing compression toward higher stimulus intensities. Summarizing, some concepts that must be considered in the general description of tonemapping algorithms have been presented: Luminance, Webber-Fechner Law, Just-Noticeable Difference (JND) and Stevens’ Power Law. 3.2.2. Dynamic Range of Reproduction Devices Image visualization devices fall into two major categories: hardcopy and softcopy devices. Hardcopy devices are static and passive. Some hardcopy devices produce transparencies filtering the light, but most produce the reflection of ambient illumination. In general, they are many methods for permanent reproduction, such as: traditional ink presses dye-sublimation thermal, laser and ink-jet printers 71 3. TONE MAPPING ALGORITHM Regarding DR, hardcopy media will never become HDR because this would entail the invention of methods such as light-emitting paper. Hardcopy are usually limited to a contrast ratio of 200:1, and so it is usually insufficient even for 8-bit LDR representations. Softcopy devices are dynamic and active. Most of these display devices include an integrated light source. Displays are commonly not able to generate pure black emission because there is leakage of light from other pixels or reflections from ambient light. The limitations of the most typical softcopy devices are: Cathode-ray tubes (CRTs): The fundamental constraint for dynamic range in CRTs is their maximum brightness, which is limited by the amount of energy we can safely deposit on a phosphorescent pixel without damaging it or generating unsafe quantities of X-ray radiation. The CRT displays have a very high dynamic range, but it is not useful because the range is mainly at the low end, where we cannot see it under normal viewing conditions. A good CRT monitor will be close to displaying a pure black, and therefore offering a contrast that exceeds 1000:1. LCD displays: There is no fundamental limit to the amount of light one can pass through an LCD screen, as it only imply changing the backlight source. Regarding the backlight, LCD displays can be divided in two main categories: Light Emitting Diode (LED) and Cold Cathode Fluorescent Lamp (CCFL) backlight. LED backlight can achieve better brightness and an individual control over pixel backlight (white or RGB) improving the dynamic range. LCD display operation is not able to completely turning on and offthe backlight, and therefore they achieve a typical static contrast ratio of only 400-600:1. Plasma displays: Despite not being able to display the purest black, Plasma Panel exhibit the best contrast, offering a 1000-3000:1 ratio, due to higher brightness capabilities. Moreover, LCD panel suffers from angle view limitation in comparison with CRT and Plasma displays. If the viewer is situated in a certain angle regarding the center of the display, the person will experience a loss of brightness, contrast, and color. In certain cases, angled view will imply the perception of image details in the darkest or brightest areas, which are lost in direct viewing due to contrast limitations. CRT displays were in previous years the common monitor for computer. However, they have been surpassed by LCD panels because they are thinner and lighter for same display size. Regarding softcopy DR, HDR displays are in development and will become available in the future. However, typical monitor displays nowadays have a contrast ratio capability of only 100-1,000:1. Summarizing, typical hardcopy methods of representation are limited in DR even for the representation of 8-bits photography. Regarding softcopy, the reader must consider the depicted dynamic range and angle of view limitations of the used display. 72 3.3. Tone Mapping Techniques Tone mapping techniques are intended to adapt HDR images to LDR representation. However, since the early development of these techniques, the final objective was the presentation of images to human users in conventional displays. Since early stages in the research of tone mapping, it is noticeable that, direct compression of the intensity level and the contrast range (to fit them into the display limits) is not sufficient to reproduce the accurate visual appearance of the scene. In the beginning, researchers studied the problem and suggested the use of models of the human visual system. Therefore, tone-mapping algorithms usually incorporate visual models in their operation. Due to the complexity of this task, tone mapping have been classically performed in the field of the computer graphics. Therefore, image sensor architectures toward tone mapping are not common. Tone mapping techniques are usually divided in four categories: Global operators: The same function is applied to all the array of pixels. The function depends on global variables, such as mean luminance. Local operators: They apply a different function depending on the pixel. The function usually depends on local variables, such as the value of the pixel and its neighbors. Frequency Domain operators: The function depends on the spatial frequency of the image. Gradient Domain operators: The applied function depends on the derivative of the image; it is to say, the direction and value of the change between pixels. Usually, they attenuate the image gradients with large magnitude while magnifying the small ones. The most used schemes are global and local operators. However, given the nature of the problem (infinite kinds of scenarios), it is not possible to have an all-in-one method. Depending on the user or application requirements and the HDR scene in particular, a method or even a combination of them will be better suited. 3.3.1. Global Operators Global tone mapping algorithms apply the same function to all pixels of the image. The applied function can be a power function, a logarithm, a sigmoid, or a function that depends on a global descriptor, such as the histogram. Global operators have the advantage of being computationally efficient. In general, global operators are the fastest to execute because normally only two or three passes over the image are required. In each pass, very simple computations are performed. Many of them may be executed in “real time” in computers. Moreover, the performance of global operators usually depends only on the size of the image. Global operators are usually much faster than the other operators. Consequently, applications that require speed or limited resources should consider global operators over all others. Global tone mapping operators are the simplest to implement, but they tend to lose details. Global operators functions usually increase monotonically. Otherwise, visually unpleasant artifacts will be 73 3. TONE MAPPING ALGORITHM introduced because the produced image seems rare or artificial to the human visual system. In fact, the higher the dynamic range of a scene the more values must be mapped to the usual 256 levels of display by a monotonically increasing function. Hence, their disadvantage is their lack of visibility and contrast preservation in very large dynamic range scenes. Additionally, some operators are not able to handle all kinds of HDR scenes. In order to have a general idea of the algorithms that can be performed globally to the HDR scenes, some typical global tone mapping operators are briefly summarized in the following subsections. 3.3.1.1. Miller’s Operator Miller et al. developed in 1984 a tone-reproduction operator that tries to preserve the sensation of brightness of the image before and after dynamic range reduction [83]. Brightness is a complex function of both luminance and spatial configuration. Miller et al. proposed a model that keeps the brightness ratios constant before and after dynamic range compression. Stevens’ psychophysical data is used for luminance calculation. 3.3.1.2. Tumblin-Rushmeier’s Operator The Tumblin-Rushmeier operator is based on the same psychophysical data as Miller’s operator (Stevens’ studies), but the brightness function is stated slightly differently [84]. Tumblin-Rushmeier attempts to preserve the brightness values themselves, in opposition to Miller et al. who proposed to preserve brightness ratios. 3.3.1.3. Ward’s Scale Factor This technique tries to preserve contrast instead of brightness [85]. It proposes that the smallest perceptible difference (Just Noticeable Difference [JND]) in a real scene corresponds to the smallest perceptible difference in the image. The psychophysical used data belongs to contrast sensitivity model of Blackwell’s studies [86]. 3.3.1.4. Ferwerda Visual Adaptation Operator The concept of matching JNDs of real world and displays was extended by Ferwerda et al [87]. However, they extended the complexity of the psychophysical data. Ferwerda et al. added a scotopic vision component while Ward’s contrast-based scale factor only takes care of photopic vision. The perception data also modeled the loss of visual acuity (the ability to perceive details) under scotopic lighting and the process of illumination adaptation. 3.3.1.5. Logarithmic and Exponential Operators These techniques apply a simple global tone mapping curve following a logarithmic or exponential behavior [79]. These compressions are consistent with the logarithmic like behavior of brightness 74 perception stated by Fechner’s law. Logarithmic and exponential operators are the most uncomplicated tone mapping operators. Therefore, they provide a baseline for comparison of resources effort. This last usually determined by execution time in personal computers. For medium dynamic range scene, these simple techniques provide a final image quality comparable with operators that are more complex. 3.3.1.6. Drago Logarithmic Operator Drago et al. also propose a logarithmic function application, but it is extended to manage a wider dynamic range than the simple logarithmic operators [88]. This operator implies a logarithmic relationship of the tone mapping technique following Stockham’s work [89]. He recommended that the displayed luminance Ldis derived from the ratio of world luminance Lwand maximum luminance in the scene Lmax. This tone mapping operator realizes the adaptive adjustment of the logarithmic base depending on pixel radiance. The base is varied between 2 and 10, allowing contrast and detail preservation in dark-medium regions and large compression in bright regions. 3.3.1.7. Reinhard-Devlin Photoreceptor Operator Reinhard and Devlin [90] proposed in this operator a modified version of the photoreceptor model which was developed by Hoods et al. [91]. This operator is applied to each of the red, green, and blue color channels separately. An important contribution of their work is the control of color saturation in the way of von Kries model of 1902 [92], which enunciated that chromatic adaptation occurs independently in the LMS cones. 3.3.1.8. Ward Histogram Adjustment This operator proposed by Ward Larson et al. is inspired in the typical histogram enhancement techniques, which are applied to LDR images to improve contrast [93]. This operator computes a histogram of a subsampled density image to obtain the distribution of pixels in luminance. The density image will be a logarithmic representation of the pixels in an approximation of brightness from luminance values. The result of the cumulative histogram can be directly used to map luminance values to display values. This technique tries to keep the contrast rather than maximizing it. Therefore, post processing steps are applied to the model to simulate glare, color sensitivity and spatial acuity to maintain subjective correspondence and visibility. 3.3.1.9. Schlick’s Rational Operator This technique was introduced by Christophe Schlick [94] in 1994. It is intended to improve computational efficiency and simplify parameters rather than the subjective correspondence achieved by previous methods. This operator applied a first-degree rational polynomial mapping function as an alternative to previous linear, exponential or logarithmic mappings. The application of rational functions is efficient to compute and easy to tune due to only two user parameters. This operator works prop- 75 3. TONE MAPPING ALGORITHM erly when applied to high-contrast scenes but it is especially well suited for scenes containing strong highlights. 3.3.2. Local Operators Local operators generally perform a local adaptation depending on the value of the pixel of interest and a set of pixel in its neighborhood. Therefore, a set of the pixel neighborhood determines how this pixel is compressed. Hence, a bright pixel in a dark neighborhood is different from a bright pixel in a bright neighborhood and the same for dark pixels. Local operators do not provide a monotonically increasing function. Therefore, two pixels with equal luminance values can be mapped into two different LDR values. Regarding the computation of local operators, several parameters are widely implied: amount of neighbor pixels to be included in the computations weight of each neighbor pixel in relation to the pixel of interest once the adaptation reference levels are determined, how are they finally applied In general, local operators provide much better perception results than global ones. This is explained by the fact that the human vision system is more sensitive to local contrast than to global contrast. This means that the human vision does not adapt to the scene as a whole but to smaller regions instead. Hence, local operators are frequently inspired by biological features of the human visual system. However, this good performance has the disadvantage of an increased complexity, as these operators are much more computationally intensive and harder to tune than global operators. The tuning difficulty is due to several parameters in the algorithms that have to be set empirically. Moreover, local operators may have the disadvantage of halo or ring artifacts. These artifacts are usually present in the surrounding of sources of light due to the presence of high gradients in illumination. The local operators to be applied in tone mapping techniques are not as straightforward as global ones. Following sections will present the main characteristics of the most typical local operators. 3.3.2.1. Chiu’s Operator The first local operator was developed by Chiu et al. in 1993 [95]. This operator was inspired by artistic techniques that frequently use spatially varying techniques to trick the eye into perceiving a wider dynamic range than the real one. Particularly, the areas around bright features may be dimmed somewhat to accentuate them. They stated that the eye is more sensitive to reflectance than luminance, and hence slow spatial variation in luminance may not be greatly perceptible. This operator multiplies the pixel luminance by a local average producing a low-pass filtered version of the image. Consequently, the pixels in this image are a weighted local average. Finally, these values are inverted and then multiplied by the original pixel luminance scaled by a constant factor. This operation suffers of one of the typical problems of local operator: halos around bright features. In order to solve this issue, Chiu’s operator first approach included a smoothing stage, which iterates at 76 least 1,000 times over the image with a small filter kernel. This would reduce the effect of halos but it makes this operator to have an unpractical amount of computation. 3.3.2.2. Rahman Retinex Operator Rahman et al. [96] developed their operator over a reinterpretation of the Land’s retinex theory [97] of 1971. There are two version of this operator: single-scale and multiscale. The multiscale version was introduced to reduce halo effects. In the single-scale version, this operator applied a low-pass Gaussian filter over the image to find local multiplying factors. They divide the pixel value by its low-pass filtered value. Finally, the logarithm of the result composes a reduced contrast “single-scale” version. In the multiscale version, the filter is applied to the image with kernel sizes doubling every step. This produces a group of images with increasing blurring. Finally, the final image is realized by the weighted sum of this stack of images with scaling and offset constants. It is remarkable that, RGB channels are computed independently rather than unified on a single luminance channel. Therefore, it implies that the Gaussian-blurring convolution must to be repeated at least three times per image. Even more computation will be required in the multiscale version, where it would make three Gaussian filters for every image of the stack. 3.3.2.3. Pattanaik Multiscale Observer Model An observer viewing a real scene would perform a chromatic adaptation. If the same scene were represented in a display, the observer would adapt differently to the image in the display device. Consequently, the displayed image is likely to be perceived differently than the real scene. Chromatic adaptation should be considered in HDR imaging and specially in tone mapping. However, tone mapping operators usually ignore this characteristic for simplicity. Pattanaik et al. proposed a multiscale observer model, which includes one of the most complete color appearance models and consists of several steps of operation [98]. The appearance of color to humans are described by a group of properties named ”color appearance correlates” [99]. These variables can be computed from the color’s tristimulus (RGB) values as well as the description of the environment. Typically useful appearance correlates from a visual stimulus are: Brightness: Visual stimulus appears to emit more or less light. Lightness: The area where visual stimulus takes place appears to emit more or less light in proportion to a similarly illuminated area that is perceived as a white stimulus. Hue: attribute of color. Perfect grey, where all color (RGB) components are the same, is an achromatic color. Chroma: difference between a visual stimulus and an achromatic stimulus of the same brightness. 77 3. TONE MAPPING ALGORITHM Saturation: difference between a visual stimulus and an achromatic stimulus judge regardless of their brightnesses. Appearance correlates are not computed directly but it made necessary the use of color spaces [100]. This multiscale color appearance model simulates luminance, pattern and color processing of the human visual system to accurately predict the color appearance attributes of spectral stimuli in complex surroundings under a wide range of illumination and viewing conditions. Therefore, the computation of this model is complex due to a large amount of steps in the process. Therefore, this model must only be used for images with an extreme dynamic range. In lower dynamic range scene, simpler models will be best suited. 3.3.2.4. Ashikhmin’s Operator The multiscale observer model try to include in the model all the steps of human visual processing that are currently well understood. However, for the limited present purpose, this complexity is not necessary. Ashikhmin’s operator attempts to model human visual perception in useful characteristics toward dynamic range compression [101]. Therefore, this operator tries to obtain results similar to Pattanaik’s operator but in a significantly simpler computational model. 3.3.2.5. Reinhard et al. Photographic Tone Reproduction The Reinhard et al. photographic tone reproduction operator [102] is inspired by the Zone System described by the photographer Ansel Adams in 1940s [103]. A zone is defined as a Roman numeral associated with an approximate luminance range in a scene, as well as an approximate reflectance of a print. There are eleven print zones, ranging from pure black (zone 0) to pure white (zone X), each doubling in intensity, and a potentially much larger number of scene zones. The photographer uses measured information in the field to improve the chances of producing a good final print. 3.3.3. Frequency Domain Operators Frequency domain operators rely on the fact that low frequency content in an image tends to be high dynamic range and high frequency content tends to be low dynamic range. Attenuating the low frequencies in the Fourier domain, HDR scenes could be compressed while the high frequencies (the low dynamic range details) are preserved. Frequency domain operators are computationally based on the Fast Fourier Transform (FFT) to obtain a frequency domain representation. In these operators the performance of the FFT transform determine the computation time. The execution time of an FFT depends on the size of the image. However, this dependency is not linear. Images that are powers of two will be faster computed, and images of prime numbers size will be the slowest to compute. Three typical frequency domain operators are: Oppenheim Frequency-based Operator [104], Durand Bilateral Filtering[105] and Choudhury Trilateral Filtering [106]. 78 3.3.4. Gradient Domain Operators Considering the gradients in the image, it is possible to partially distinguish between illuminance (incident light, slow gradients with arbitrary HDR) and reflectance (reflected light, high frequency, and LDR). The differentiation transforms images from the luminance to the contrast domain. The final integration of the thresholded derivatives leads to the reconstruction of image reflectance. Working with diffusely reflecting scenes, the separation of illuminance and reflectance may be reasonably successful. Contrast domain offers several advantages over luminance domain: Contrast can be modified depending on its magnitude, hence taking advantage of contrast perception characteristics. Extremely sharp contrast can be achieved without introducing halo artifacts. The pyramid of contrast values, which approximates the localized contrast perception, can take into account both local and global contrast relations. Without this, undesired effects can take place in the areas where the input luminance is the same. Although it has advantages, the gradient domain HDR compression consumes significant computational time. Two well-known gradient domain operators are: Horn Lightness Computation [107] and Fattal Gradient Domain Compression [108]. 3.4. Proposed Algorithm The present tone mapping algorithm is intended to capture images from HDR real world scenes and compress them into a low dynamic range representation. At the same time, this algorithm must preserve the details of the objects in the scene minimizing the computational complexity taking advantage of the parallelism of the focal plane circuitry. Under these assumptions, our algorithm performs a global tone mapping operator in the HDR scene. Once the compression variables are set, it will adapt the compression curve to the image to be processed. This adaptation will take a previously captured subsampled non-linear histogram as an indicator of the probability of illuminations. This method is capable of producing visually satisfactory results for a large set of examples. Moreover, as the computation of the tone mapping curve can be performed with little processing effort, this algorithm is suitable not only for tone-mapping over still images but also for HDR videos. However, for still images, it implies two image captures for every image due to the necessity of a previous image as an estimator of illumination probabilities. Our algorithm performs two kind of compressions derived from the type of circuitry used in the approach: analog and digital compression. The analog compression is performed due to the mathematical intersection of the discharge signal in the integration pixel with a variable analog voltage reference. The discharge signal of a pixel is linearly dependent on the illumination of the scene. The digital compression is performed by assigning a digital code of the tone mapping curve of compression to each 79 3. TONE MAPPING ALGORITHM 7 6 5 4 3 2 1 0 10 30 40 50 6020 Time (µs) TMC<2:0> (a) TMC<2:0>vs. time graphically. (b) TMC<2:0>vs. subdivisions. 7 3 65 3210 210 4 (c) Digital Reference vs. time as digital bus representation. Figure 3.5: Example TMC curve. the duration of every bin by the number of possible levels to be assigned (2bits o f T MC ). This calculation comes from the possibility of assigning all the levels to a single bin, possibility which is shown in figure 3.6 that occurs when the whole population of the image intersects in a single bin. In this case, the example of table 3.2 is simple; the number of levels per bin for the different bins are a submultiple of the number of subdivisions for evaluation. Generation of the TMC reference can simply be performed by decreasing one code every Number o f subdivisions per bin Number o f levels per bin subdivisions. In this case, the number of subdivisions is 8, so the change will take place every 8 Number o f levels per bin on a bin depending on its assigned levels per bin, resulting in a change of code every 8, 4, 2 and 8 subdivisions inside the bins for bins 3, 2, 1 and 0, respectively. However, when the number of levels per bin is not a submultiple of the number of subdivisions for evaluation (3, 5, 6 and 7 levels per bin), the generation of TMC reference is more complex. Figure 3.7 shows the example for 3 and 5 levels per bin in bins 2 and 1 respectively. The assignment of 3 654 210 7321 0 Figure 3.6: Example of the distribution of TMC codes all in one bin. 86 3 720 210 13456 Figure 3.7: Example of TMC codes when levels per bins are not submultiple of total subdivisions. codes cannot be exactly linear inside the bins. In order to solve this issue, the ideal rate must contain decimals. Therefore, the boundaries of subdivision where the code decrement must take place is found accumulating 8 Number o f levels per bin with no rounding and round in the final value. The case for 7 bins per levels is shown in table 3.3 where the ratio of change is 8 7=1.1429. Now, the problem is how many decimal are needed in order to avoid wrong results. This is important because it will not be possible to have infinite accuracy in the physical implementation of this algorithm. As it can be noticed in table 3.3, if only 1 decimal is used in the ratio, it provokes a mistake which is indicated with bold fonts. In the case of the calculation with all decimals, the subdivision where no decrement of TMC takes place is 4 but using only 1 decimal it is subdivision 5. In this case, the number of bits used for TMC is low. If the number of bits used is higher, as an example 7 bits, the worst ratio for accuracy will be 128 127 levels per bin =1.0079 where all decimal must be used in order to avoid an error. If this number is represented in binary, it will be necessary 14 bits (log2(10079) =13.2991) plus the 7-bits necessary for the integer part of the accumulation, resulting in 21-bits. If this method is used, it will be also necessary to calculate the ratio in every level per bin with an accuracy of 14 bits, or having a memory of 128 words of 14 bits that contains the previously calculated values. Both options could imply a delay in the calculations between bins. In order to avoid this, another option have been developed, the subdivision with a decrement is already stored in a Look Up Table (LUT). This LUT contains the positions of TMC one code decrement in the subdivision boundaries of a bin depending on the assigned levels per bin. The row address of the LUT corresponds with number of assigned levels, which have been obtained in the previous calculations. An example of the LUT for TMC of 3 bits is shown in figure 3.8, where 1 indicates the position of one code decrement and 0 Level Level ·1.1429 Round (Level ·1.1429) Level ·1.1 Round (Level ·1.1) 1 1.1429 1 1.1 1 2 2.2858 2 2.2 2 3 3.4287 3 3.3 3 4 4.5716 54.4 4 5 5.7145 6 5.5 6 6 6.8574 7 6.6 7 7 8.0003 8 7.7 8 Table 3.3: Subdivisions with decrement in the case of 7 levels per bin in 8 levels. 87 3. TONE MAPPING ALGORITHM Figure 3.8: Look Up Table of code decrements for TSC representation of 3-bits. indicates no change in TMC code. Every change of bin must imply a decrement, in order to not share levels between bins (except in the 0 levels per bin case1). Hence, the first bit of a row of this LUT always starts with a jump (1) but in the first row, belonging to 0 levels per bin data. Having the levels per bin and this fixed LUT for single code decrements, the tone-mapping curve can be generated in capture time just retrieving the corresponding 1-bit word for a subdivision for evaluation. Figure 3.9 shows the composition of the LUT for a TMC of 7-bits, where green pixels indicate the position of one code decrement and blue pixel indicate no change. The rows indicate the number of levels assigned to a bin. The calculation of this LUT has been performed distributing geometrically the code decrements. The positions of bits 1 will take place in positions which are rounded multiple of 128 levels per bin . Then for n levels per bin the positions of bits 1 are rounded 1·128 n,2·128 n, ... n·128 n, which implies always a bit 1 at Subdivision Level per bin 20 40 60 80 100 120 20 40 60 80 100 120 Figure 3.9: Look Up Table of code decrements for TSC representation of 7-bits. 1If a change takes place the LUT row will be equivalent to the 1 levels per bin case. 88 position 128 ( n·128  n=128 independently of n). In order to differentiate code between bins, it is preferred to perform a code decrement in boundaries between bins. Therefore, it is required always a bit 1 (TMC one code decrement) in position 1 of every word; the solution followed in this work is to perform a flip left to right of the LUT. For example, the positions of bits 1 of the look up table for 3 levels per bin will be 3. Unflipped positions 3 will be 128 3=42.6667, 128·2 3=85.3334, 128·3 3=128. This numbers are rounded to be 43, 85 and 128. Finally, the results of flipping left to right will be (129-128)=1, (129-85)=44 and (129-43)=86 1. Additionally to the LUT information, every first code change is dismissed. In the example of figure 3.5(c), bin with TSC<1:0>=3 is assigned 1 level per bin. Then, we go to the second row of the LUT, which correspond to 1 level per bin, it indicates that the row is ”10000000” that corresponds to activation or not of a decrement for the subdivision indicated by its position in the row, subdivision, as indicated in the x-axis of figure 3.8. This row of the LUT indicates a single decrement at the beginning (subdivision 1), and no decrement in the next evaluation subdivisions of that bin. As in figure 3.5(c), the code starts already at maximum code, this first decrement is not necessary. However, in the figure 3.10, bin TSC<1:0>=0 is assigned with 2 levels per bin, the LUT row will be the third row of the LUT ”10001000” but here the first decrement of a bin is necessary to change from TMC code 2 to 1. Therefore, the first code indicated by the corresponding row of the LUT for a bin will be dismissed only if it is the first decrement that will take place in the TMC reference. In the case of a bin with 0 levels per bin assigned, no code change takes place, it is shown in figure 3.10 (0 levels in bin 3) and figure 3.11 (0 levels in bin 2). Summarizing, the necessary steps to generate the TMC curve are: 1. Accumulate TSC data to obtain a non-linear subsampled histogram. 2. Divide the number of pixels in the bins by the quotient of dividing the total number of pixels in the subsampled histogram by the total amount of levels at disposal, 2bits o f T MC . 3. Floor round the result of step 2. 4. Distribute lost levels in the bins with highest remainders in step 2. 5. Compose TMC signal in time with the positions of the decrements in the subdivisions indicated in the LUT, depending on the values of levels per bin obtained in step 4. 3 76 4320 210 5 1 Figure 3.10: TMC code distribution with zero levels assigned to first bin. 1It must be noticed that the LUT of TMC with 3-bits in figure 3.8 does not correspond to the calculations in table 3.3 because of the flip left to right. 89 3. TONE MAPPING ALGORITHM 7 3 432 210 56 01 Figure 3.11: TMC code distribution with zero levels assigned to a middle bin. 3.4.3. Some Alternatives for Level per Bin Assignment The method used in the distribution of levels per bin depending on the histogram of the previous TSI will determine the tonemapping operation mode. The distribution of levels per bin can be applied by a simple statistical calculation; i.e. depending on the weight of the bins in the histogram. However, variations over this method can be useful in some circumstances. Some possible modes of operations by simply changing the mechanism in the calculus of the levels per bin are: 1. Equal distribution: The same number of levels per bin is assigned in all active bins. It is used as the mode for the first shot, when we do not have TSI information. 2. Weighted: The levels are distributed depending on the weight of the bins in the histogram of TSI. The number of levels per bins is obtained by dividing the values of the histogram by number o f pixels in TS I number o f levels and floor rounding. The non-assigned levels are distributed between the bins with the higher remainders of the division. This option has been already described in previous sections. 3. Bin threshold: A bin is active when the histogram bin value is higher than a threshold. Levels are equally distributed between the active bins. 4. Avoid concentration in one bin: This mode operates as the weighted mode but it avoids a high concentration of too many levels in a single bin. If the number of levels in a bin exceeds a configurable limit, saturation is applied and the rest of levels are distributed between the bins with higher assigned levels. 5. Weighted with bright priority: If the most populated bin is a bright one, then this bin and the near lower neighbors are amplified by a fixed number of levels if they exceed a threshold. The rest of levels are assigned proportional to its weight in the TSI histogram. 6. Low populated priority: Low populated non-empty bins are amplified by assigning a fixed number of levels. The remaining levels are distributed as in the weighted mode among all the bins. 7. Weighted with minimum threshold: All non-empty bins are pre-assigned with a fixed number of levels. The rest of levels are distributed as in the weighted mode. 8. Non-linear levels adjustment: A non-linear function is applied to the TSI histogram, then the levels are weighted distributed. Here, the applied function was y=x1 1.75 . 90 9. Weighted with minimum low light priority: A minimum amount of levels is applied to non-empty bins. Then, if the most populated bin corresponds to one bin receiving high light (short crossing time), the levels per bin of bins receiving low light (long crossing time) are amplified by assigning an addition of several levels. Finally, the remaining levels are distributed as in the weighted mode. 3.5. Simulation Results In the present section, an example over a real HDR scene will be illustrated. First, it is explained how to obtain an HDR scene mathematical representation. Second, the obtained TSI is presented as well as the performed calculation, resulting in the levels per bin. Third, the tone mapping curve (TMC codes) is obtain from the levels per bin. Finally, the tone mapping curve is applied to the original HDR image obtaining the result image of the algorithm. 3.5.1. Composition of HDR Image using Multiple LDR frames In other to have a mathematical representation of a HDR scene, the combination of several Low Dynamic Range (LDR) frames is performed. This technique obtains a HDR image combining multiple images of the same scene that differ only in exposition time. These images are supposed to be linearly captured, and therefore they are not compressed or tone mapped. In our case, the photographs have been taken with different exposition times by the technique of exposure bracketing, where several shots of the same subject are captured using different camera settings. This has been performed varying the camera options manually: the aperture has been fixed to f/16 (low aperture to have all the objects in focus), the focal length is 55mm and the shutter speed has been varied. There have been obtained 18 frames with shutter speeds of 1 4000 ,1 2000 ,1 1000 ,1 500 ,1 250 ,1 125 ,1 60 ,1 30 ,1 15 ,1 8,1 4,1 2, 1, 2, 4, 8, 15 and 30 seconds. Exposure Value (EV) [109] in fotography indicates all combinations of a camera’s shutter speed and relative aperture that give the same light exposure. The EV definition is shown in equation 3.11, where fis the f-number of the relative aperture and t is the shutter speed in seconds. EV =log2 f2 t(3.11) In the exposition compensation, which take as a base a calculated optimum EV value, EV=0isthe considered correct general exposure, positive values are overexposed and negative values are underexposed. An increment of 1 EV of compensation denotes a double exposure from the previous step. The EV calculated by AdobeR PhotoshopR CS5 in the 18 frames with the variations are -8.09, -7.06, -6.06, -5.06, -4.06, -3.06, -2, -1, 0, +0.91, +1.91, 2.91, 3.91, +4.91, +5.91, +6.91, +7.81 and +8.81 respectively. The images are shown in figure 3.12. These pictures or photographs have been acquired with a commercial camera Nikon D90 [110][111], which includes a CMOS sensor [112]. The photographs have been taken using a tripod and an infrared remote control in order to avoid vibrations, which will cause differences in the scene between different frames. In order to reduce noise, as it will be greatly amplified in this kind of compositions, the frames have been captured with the lowest possible ISO 91 3. TONE MAPPING ALGORITHM sensitivity: ISO 100 equivalent (named by Nikon L1.0). Moreover, a dark frame has been subtracted in the frames with exposition times 15” and 30” by the large exposition noise reduction option of the camera. The images has been taken with 12-bits AD conversion in RAW format (NEF in Nikon) that does not alter the information taken by the camera by post-processing (tonal adjustments, compression, etc.), which occurs in formats such as JPG. Proper HDR composition of a set of images is not an easy task as, in commercial cameras, the exposition time is difficult to tune. Therefore, the obtained images have been introduced in the AdobeR  PhotoshopR CS5 HDR merging engine [113]. This engine uses the metadata in the image files to determine the exposure values of the images and the tonal response curve of the camera that took the images. It will perform an auto blend of parts of the scenario captured by each photograph. The engine also performs automatic alignment of the LDR frames and ghost removal in the final HDR image (caused by scene changes between photographs). The engine combines the images in an HDR representation of 32 bits known as the Radiance picture format (.hdr) [79], which is composed by three color channels of 8-bits (Red, Green, Blue) plus a common exponent of 8-bits. 3.5.2. Mathematical Simulations The original HDR image is cropped and subsampled to the size we are aiming to implement in the hardware version, which will be a low resolution frame, a QCIF image (144x176 pixels). The final radiance picture format (.hdr) is then imported by the MatlabR Image Processing Toolbox engine in double data type. Then, it is converted to monochrome resulting in pixels values between 391·10−6 and 29.36. This data imply a dynamic range of about 20·log10(29.36 391·10−6)=97.5dB. In order to study the algorithm in higher dynamic range, the data are linearly ranged to fit in a set of data between 10−9 (for 29.36) and 10−15 (for 391·10−6). Therefore, it will result in the same relative distribution between photocurrents but extended to the DR of choice (120dB). These artificial photocurrents values, which are scaled pixel data, can be considered as a representation of the pixels photocurrents because the multiexposure LDR frames has been linearly captured. Representing images using a gray colormap is usual for monochrome images. However, this will lead to loss of details in the several possible representation of this work combined with the vision system of the reader. Therefore, figure 3.13 shows the photocurrents image in gray colormap and in jet colormap for the improvement of details perception. The jet colormap performs a rainbow set, which goes from blue assigned to lower values to red in the higher values passing through the colors green, yellow, and orange. From now on, the images will be shown in both colormaps with a right side bar indicating the range of the colors (colorbar). Figure 3.14(a) shows the histogram of the photocurrent images, however little can be distinguished in the linear representation of the image and its histogram due to the high differences between values. In order to achieve a proper illustration of the distribution of illumination zones, figure 3.14(b) shows the histogram of the logarithm of base 2 of the photocurrents, which is a typical representation in HDR histograms as it corresponds with the distribution of EV. It is noticeable that there exists 3 zones, the left zone corresponds to the background, the middle zone is the lamp holder and surroundings that reflects high amounts of light, and the right zone is the light bulb that emits the 92 (a) EV +8.81 (b) EV +7.81 (c) EV +6.91 (d) EV +5.91 (e) EV +4.91 (f) EV +3.91 (g) EV +2.91 (h) EV +1.91 (i) EV +0.91 (j) EV 0 (k) EV -1 (l) EV -2 (m) EV -3.06 (n) EV -4.06 (˜ n) EV -5.06 (o) EV -6.06 (p) EV -7.06 (q) EV -8.09 Figure 3.12: LDR frames used for HDR composition. 93 3. TONE MAPPING ALGORITHM 20 40 60 80 100 120 140 160 20 40 60 80 100 120 140 1 2 3 4 5 6 7 8 9 x 10−10 (a) Gray colormap 20 40 60 80 100 120 140 160 20 40 60 80 100 120 140 1 2 3 4 5 6 7 8 9 x 10−10 (b) Jet colormap Figure 3.13: Photocurrent image. light. In order to apply the algorithm to this image, the discharge signals of the pixels are simulated taking the values of the HDR image (with values from 10−9to 10−15) as photocurrents. The discharge is evaluated via equation 3.4. The capacitor in the photodiode (Cph) node is assumed to be 30fF and Vrst is 3.3V. The fixed reference voltage during the time bins is 1V. Therefore, during the last bin a voltage ramp from 1V to 3.3V has been simulated. The TSC curve has 4-bits for representation, achieving 16 bins, and TMC curve has 8-bits resulting in a final tonemapped image of 256 gray levels. Once, the image containing the 4-bits TSC data is obtained, the histogram of TSI is calculated to obtain how many pixels cross in every bin. The TSI is shown in figure 3.15 and its histogram in figure 3.16, where the commented 3 zones are noticeable. 1 2 3 4 5 6 7 8 9 x 10−10 0 0.5 1 1.5 2 x 104 Frequency (Number of Pixels) Bins (A) (a) Histogram of Iph −45 −40 −35 −30 0 500 1000 1500 2000 2500 3000 Frequency (Number of Pixels) Bins (log2(A)) (b) Histogram of log2(Iph) Figure 3.14: Photocurrent image histogram. 94 10 20 30 40 50 60 70 80 10 20 30 40 50 60 70 2 4 6 8 10 12 14 16 (a) Gray colormap 10 20 30 40 50 60 70 80 10 20 30 40 50 60 70 2 4 6 8 10 12 14 16 (b) Jet colormap Figure 3.15: Time Stamp Image. 2 4 6 8 10 12 14 16 0 500 1000 1500 2000 2500 3000 Figure 3.16: TSI Histogram of photocurrent image. Next, the levels per bin must be calculated. This calculation is shown in table 3.4. We have divided the numbers of pixels in the bin by the constant =Number o f pixels in T S I 2bits o f T MC . As the histogram image is subsampled by four in a QCIF image, the amount of pixels in the TSI is 144·176 4=6336. Then, this quantity is divided by the number of total levels considering a final representation of 8 bits, 6336 256 =24.75. Now, the result of the division of the number of pixels in the bin by 24.75 is floor rounded. However, the remainder of that division is kept to be sorted. The sum of the assigned levels will lead to only 250 levels. The lost levels will be 256 −250 =6. These levels will be equally distributed in the bins with 6 highest remainders. The increment of levels in table 3.4 is highlighted by the use of bold letters. Now, we have a total of 256 levels assigned. Tone Mapping Code composition over time is now achieved by the retrieval of the LUT data, which contains the evaluation times when the TMC decrements occurs. The LUT, in this case, have 257 (256 plus 0) positions of 256 bits. The information of the LUT is retrieved in every bin depending on the 95 3. TONE MAPPING ALGORITHM (a) Mode 1 (b) Mode 2 (c) Mode 3 (d) Mode 4 (e) Mode 5 (f) Mode 6 (g) Mode 7 (h) Mode 8 (i) Mode 9 500 1000 1500 2000 2500 3000 3500 4000 0 50 100 150 200 250 Subdivision for evaluation TMC<7:0> Mode 1 Mode 2 Mode 3 Mode 4 Mode 5 Mode 6 Mode 7 Mode 8 Mode 9 (j) Applied tone mapping curves. Figure 3.22: Results of applying the different levels per bin assignation modes (a-i), and (j) their corresponding tone-mapping curves. 102 (a) Mantiuk’06 (b) Mantiuk’08 (c) Fattal (d) Drago (e) Durand (f) Reinhard’02 (g) Reinhard’05 (h) Ashikmin (i) Pattanaik (j) New hardware aimed-algorithm Figure 3.23: Results of CLAHE in comparison with mode 6. 103 3. TONE MAPPING ALGORITHM function and a pseudo-linear function. Moreover, the result of the cumulative histogram is also used to calculate the global tone mapping curve of compression. This tone mapping algorithm is suitable for on-chip implementation taking advantage of the reduced amount of computation that it is required. Since only comparison with an analog signal and storing capabilities are required inside the pixel, it is possible to implement part of it as focal plane circuitry. The parallelism of operation, which is allowed by focal-plane circuitry, makes this algorithm especially well-suited for focal plane processors or 3D integration technologies in future approaches. Therefore, the algorithm minimizes the amount of data in the representation of HDR scenes for later use in postprocessing, and therefore cutting down the necessary resources for these future evolutions. The algorithm itself does not have a limited dynamic range to handle or an established number of bits for pixel representation. However, a low compression will take place in low dynamic range scenes. Similarly, low compression will occur if a high number of bits is chosen for image representation. The amount of bits in the image representation will be defined by a trade-off. This will depend on the hardware implementation, which implies physical limitations, and the application, which determine the limits of the target representation. The physical limitations are derived from characteristics such as, noise, area of the pixels, transistors mismatch, etc. Compared with computer graphics tonemapping, our algorithm compresses the steps necessary in a normal tone mapping operation, as the pixel data is captured already tone mapped. Whereas in computer graphics tonemapping, the necessary steps will be capturing several images with a camera, storing, downloading to PC or system, and the application of the algorithm in the computer. In our case, we do not need to move unnecessary data, as the information is already captured and stored at pixel level. 104 Chapter 4 TVHC: A HDR Tone Mapping Imager in Standard CMOS Technology 4.1. Introduction This chapter describes the so-called Time Voltage Histogram Camera chip, or simply TVHC, a QCIF resolution HDR imager, which implements the tone-mapping algorithm for DR improvement, which has been described in chapter 3. Clearly, the differences between a simulation environment like MatlabR and the mixed-signal on-chip implementation of our algorithm, will force some adaptations in the real hardware implementation. Up to now, the algorithm has been designed and simulated by the infrastructure provided by MatlabR , with floating point (nearly unlimited to our purposes) accuracy, huge memorization resources, and not paying too much attention to the attainable frame rate (as we basically dealt with still images). Unfortunately, the limitations in the software implementation are loose, compared with the available resources to map this algorithm onto a chip. However, the fully parallel Single Instruction Multiple Data (SIMD) architecture of focal plane processors also offers some advantages. Thus, for instance, the chip will benefit from the parallel processing capabilities in the comparison with Vref . In this case, instead of continuously downloading the image and comparing every read pixel with the current value of Vref , we will get these NxM comparisons on-pixel and in continuous-time, without the need for a complete frame readout. This prevents the large evaluation errors to appear in the highest photocurrents, which might arise from different readout times between pixels (similar to what happens when we described the rolling shutter option in chapter 1). On the other hand, the kind of computation that can be performed within a pixel is constrained by the maximum allowable power consumption (directly proportional to the number of pixels), pitch (that, in general, increases as more resources are included on pixel), and accuracy (that, if limited by mismatch, requires using larger devices or calibration blocks, with area penalties in both cases). In addition to that, we must bear in mind that there is a need to provide an aperture in the metal layers to capture as much light as possible in order to maximize sensitivity. This creates further limitations for possible routing among 105 4. TVHC: A HDR TONE MAPPING IMAGER IN STANDARD CMOS TECHNOLOGY blocks inside the pixel. The TVHC has been conceived as a complete Vision-System-on-Chip (VSoC). However, we have to remark that this chip is a proof of concept prototype (i.e. the first prototype with this tone mapping algorithm). Therefore, for versatility and security purposes, some non-critical parts of the algorithm have been implemented externally (in an FPGA), as it is shown in chapter 5. This includes the calculation of the levels per bin and the LUT. Thanks to this flexible implementation, the whole TVHC system (TVHC camera +host board) can operate with several LUTs, if necessary, and can change the method for the assignment of levels per bin on the fly. The FPGA’s architecture and program, along with the rest of the support circuitry will be explained in chapter 5, where the whole test infrastructure will be described. The system within the FPGA has been coded in Verilog and it includes limited use of internal FPGA resources (a memory and a divider). This may produce a non-optimized code for the FPGA synthesis but guarantees that the same code can be synthesized [120] with reduced modifications in the selected integration technology in future improved versions of the chip. The chapter is organized as follows: First, the selected integration technology with some optical sensor enhanced features is introduced. Second, the general architecture of the chip is explained along with the functionalities of its blocks. Third, a description of the operation of image capture is included. Finally, there is a general description of tone mapping systems in comparison with our approach. 4.2. Fabrication Process The present chip has been designed using the Austriamicrosystems (AMS) CMOS 4M/2P 0.35μm OPTO technology. This technology is based in the standard AMS 0.35μm (C35B4C3) technology, and includes some opto-flavored modifications to provide enhanced light sensing characteristics: Inorganic Anti-Reflecting-Coating (ARC) layer 14μm EPI substrate Conversely to the usual approach of depositing the ARC layer just on top of the photosensor, this technology adds the inorganic ARC on the top of the passivation layers [121]. In any case, the use of ARC layers results in a higher and smoother spectral response of the sensor. On the other hand, the EPI layer diminishes the dark current. These variations in combination with the base technology make the process to feature these characteristics [122][123]: High Sensitivity ⇒Responsivity of Photodiode1at 550nm =290mA/W. Low Dark Current ⇒Dark Current of Photodiode <45pA/cm2(at 27◦C). Low cost ⇒Only 1 additional process step (Inorganic ARC Layer). Capacitor per area for Photodiode: 0.08fF/μm. 1For N-well to P-substrate structures. 106 psubstrate n-p- pn+p+ FOX p+ FOX Anti-Reflective Coating Contact Via 1 Via2 Via3 Metal 4 Metal 3 Metal 2 Metal1 ILDFOX IMD1 IMD2 IMD3 PROT1 PROT2 Figure 4.1: Photodiode layer arrangement. Minimum Pixel Size: 6μm×6μm. Poly-Insulator-Poly (PIP) and Metal Insulator Metal (MIM) Capacitors. 4 metal layers. Supply Voltage: CMOS 3.3V with optional 5V IOs. High driving capabilities and ESD reliable peripheral cells available. Process compatibility with mixed base technology, and therefore possible reuse of designs. Although this technology offers limited modifications for enhanced optical sensors comparing with the UMC 18μm CIS technology used in the SCU chip, it does not limit the kind of circuitry that can be allocated next to the photosensors. The recommended physical sensor of this technology is an Nwell- Psubstrate photodiode. This photodiode has a wider depletion region due to the lower doping and a deeper junction than the previously employed (SCU chip) Ndiffusion-Psubstrate structure. However, Nwell layers usually require higher spacing from surrounding circuitry than conventional N+(Ndiffusion) implants. Therefore, this kind of photodiodes will always result in a larger pitch (for the same technological node). The cross section of the photodiode with the ARC layer is shown in figure 4.1, where FOX is the field oxide, ILDFOX is the oxide between the polysilicon layer and Metal 1, IMD are the Inter-Metal Oxides and PROT indicates passivation layers. 107 4. TVHC: A HDR TONE MAPPING IMAGER IN STANDARD CMOS TECHNOLOGY Array of Pixels 148x180 pixels Control Signals Buffers DAC Charge Injection Amplifiers Sense Amplifiers Read Buffer Bias Generator IbiasP_Vref IbiasN_pixel Vdac DAC_CLK Code Generator TSP BP BP BP TSP BP BP BP TSP BP BP BP TSP BP BP BP TSP BP BP BP TSP BP BP BP IbiasN_Vref DAC_CLR DAC_MODE<1:0> Vrst Vtop Vbot Vbias Vref<147:0> POWER_ON NOVOFF PCH_START PCH_CLK READ_C READ_H WRITE READ_CLK READ_CLR ROW_CLK ROW_ENABLE ROW_CLR nRST nEVAL nREAD TMC<6:0> CLEAR_CODE_HIST CODE_CLK HIST_CLK C1<6:0> C2<6:0> C3<6:0> C4<6:0> H1<3:0> H2<3:0> TSC<3:0> nRST<147:0> nEVAL<147:0> nREAD<147:0> nROW<147:0> ARRAY_BUS<3239:0> DATA_OUT<1619:0> Figure 4.2: TVHC: High-Level Block Diagram. 4.3. Architecture of the TVHC Chip As in any other imager, the core of the TVHC prototype is an array of pixels whose functionality is supported by many peripheral blocks. These blocks, among many other things, are responsible for biasing, generation of analog references and digital controls, and high-speed image transference. The chip contains all the necessary circuitry to acquire the Tone Mapped Image (TMI) and the Time Stamp Image (TSI) as well as downloading them, already digitized, through a high-speed 36-bits bus (nearly 343Mbits/s1). The high-level block diagram of the chip is shown in figure 4.2. The main blocks included in this diagram are: Array of Pixels: an array of pixels with QCIF (144×176) resolution. The array contains a border of 2 dummy pixels (i.e. it has 4 additional rows and columns, two on each side), which provide identical surroundings to all pixels in the array. DAC: A Digital to Analog Converter generates the analog voltage reference Vref for the pixels. Charge Injection Amplifiers: This block distributes the signal generated by the DAC. Control Signals Buffers: They drive the digital controls signals of the pixels row by row using a Clock-Tree generated distribution of digital buffers, which guarantee minimum skew (specified at 136bits 100ns ·1 1024·1024 =343.3Mbits/s 108 a maximum of 300ps when using the automatic Place and Route [P&R] tool) and precise control timing. Sense Amplifiers: to acquire TSI and TMI outputs from the pixels. Since the digital codes of the Tone Mapping Image (TMI) and the Time Stamp Image (TSI) are stored in static 6T memories [124] within the pixels, a direct readout of these pixels could produce data flipping on the SRAM (and consequently, a loss of information) due to the high capacitive load of the vertical metal lines that are used to access the pixels. The sense amplifiers are responsible for precharging the column buses at a proper voltage level, retrieve and store the information from the pixels. All the pixels in a row are read at the same time. Read Buffer: This is a one row buffer memory, which allows fast image retrieval. We have employed a double buffer topology, such that when we are receiving a row in the Sense Amplifiers block, we are downloading the previous row from the Read Buffer into the FPGA in the TVHC board. Code Generator: It generates the digital signals TMC<6:0>and TSC<3:0>to the pixels. These buses use gray coding in order to diminish switching at the pixel level to only one bit per write cycle (thus saving a considerable amount of power). Bias Generator: This block generates the required biasing for all the analog circuits of the chip. It uses a Proportional To Absolute Temperature (PTAP) current source as reference for a bank of current mirrors, which generates 3 different scaled copies of this reference current (IbiasP Vref, IbiasN Vref, IbiasN Pixel). This block also provides an output voltage Vbias to the host board. This voltage can be employed to monitor the temperature of the chip (something that is very useful when adapting the levels of the A-to-D conversion for the attenuation of the effects of the dark current in the output image). Furthermore, this Vbias node can be used as startup node for the bandgap reference [125]. The reference current is produced by a bandgap circuit [30], which employs 2 parasitic vertical bipolar transistors. The operation of this subsystem is very well known and documented in the literature and its description is not considered necessary here. 4.4. Pixels In order to realize the tone mapping algorithm, the pixel must contain an integration node Vph, which is discharged by the photogenerated current. This Vph signal must be compared with an analog reference Vref . When the intersection of the signals Vph and Vre f takes place, the values of the Tone Mapping Code (TMC) (7-bits bus) and the Time Stamp Code (TSC) (4-bits bus) must be stored. Two different pixels have been designed: Basic Pixels (BP): acquire only the Tone Mapped Image (TMI) →TMC<6:0>. Time Stamp Pixels (TSP): acquire Tone Mapped Image (TMI) and Time Stamp Image (TSI) → TMC<6:0>and TSC<3:0>, respectively. 109 4. TVHC: A HDR TONE MAPPING IMAGER IN STANDARD CMOS TECHNOLOGY The block diagrams of the BP and TSP are shown in figure 4.3(a) and (b), respectively. Note that the general structure of the pixels is the same but the TSP contains four more static RAM (SRAM) cells to store TSC<3:0>. The integration node Vph is discharged by the current, which is generated by a 3x3μm2Nwell/Psub photodiode. The photodiode is connected to the analog comparator via a buffer in order to isolate the comparator’s kickback noise [126], which is mainly produced by the multiple changes in the voltage reference Vref . The reset of the photodiode is performed by the feedback loop stablished by the buffer, the comparator and the reset switch P1. Thus, the operation of the circuit is offset-free in a first approach (see Section 4.4.2 for details). The buffer and the input stage of the comparator (its output signal is amplified by subsequent digital drivers to create a rail-to-rail high gain output) share a common design of a 5T NMOS-input Single Stage Vph Vbuf nRST Vref Digital Control nEVAL nROW nREAD Vcomp SRAM 7bits S HOLD P1 TMC<6:0> (a) Basic Pixel Schematic. Vph Vbuf nRST Vref Digital Control nEVAL nROW nREAD Vcomp SRAM 7bits+4bits S HOLD P1 TMC<6:0> TSC<3:0> (b) Time Stamp Pixel Schematic. Figure 4.3: Block Diagrams of the Basic and Time Stamp Pixels. 110 in in out Figure 4.4: 5T Amplifier in the Pixels. Differential Amplifier. The schematic of this circuit is shown in figure 4.4. Notice that the transistors are relatively large as compared to the minimum transistor that can be fabricated in this technology, 0.4 0.35 μm, as it is dictated by accuracy requirements (see section 4.4.2). Simulations show that the available input range of the buffer and comparator combined is 0.7-2.6V. Figure 4.5(a) shows the simulated DC Gain of the buffer and figure 4.5(b) shows the small signal gain at Vin=1.65V. The output of the input stage of the comparator for several Vref values is shown in figure 4.6(a) and a Monte Carlo simulation of the input-referred offset voltage of the 5T Amplifier for Vref =1.65V is shown in figure 4.6(b). From the simulation results, we obtain that this offset voltage has a mean value of 320μV and a standard deviation of nearly 3.40mV. 0 0.5 1 1.5 2 2.5 3 0 0.2 0.4 0.6 0.8 1 Vin(V) Gain (a) DC Gain. 100102104106 −100 −90 −80 −70 −60 −50 −40 −30 −20 −10 0 10 Frequency (Hz) Gain (dB) (b) AC [email protected]. Figure 4.5: 5T Amplifier Gain Plots. 111 4. TVHC: A HDR TONE MAPPING IMAGER IN STANDARD CMOS TECHNOLOGY 1.43 1.435 1.44 1.445 1.45 1.455 1.46 1.465 1.47 0 5 10 15 20 25 30 35 40 45 Vref (V) Frequency μ=1.4482V, sd=4.46548mV, N=100 (a) Voltage at the crossing point: Auto-zeroing disabled. 1.43 1.435 1.44 1.445 1.45 1.455 1.46 1.465 1.47 0 5 10 15 20 25 30 Vref(V) Frequency μ=1.44848V, sd=737.272μV, N=100 (b) Voltage at the crossing point: Auto-zeroing enabled. Figure 4.10: Simulation of auto-zero improvement. 4.4.3. Pixels Physical Characteristics Pixels are arranged in groups of 2x2 elements containing 3 Basic Pixels and 1 Time Stamp Pixel. This allows us to implement the required 1 4subsampling for TSI generation in a very simple manner. Figure 4.11(a) shows the physical distribution of Basic and Time Stamp Pixels in the 2x2 arrangement, which occupies an area of 66 ×66μm2(33 ×33μm2per pixel). Notice that, conceptually, each TS Pixel takes some area from its 3 Basic Pixel neighbors in order to allocate its extra SRAM resources. However, in practice, what happens is that both BP and TSP contain 8-SRAM (7TMC+1TSC) modules with the TS related registers being only controlled by the logic in the TS pixel. (a) Pixels Group Organization. (b) Pixels Group Layout. Figure 4.11: Pixels Group. 118 The layout of the group is displayed in figure 4.11(b). Observe that the SRAM modules have been grouped in the central vertical region, sharing digital power and ground lines. In the pixel array, other groups will be placed in every side (left, right, top and bottom). Therefore, the PMOS analog transistors and the digital control circuitry are also grouped side-by-side between pixels, equally sharing their power lines. This allows to vertically route the I/O buses compactly on top of center and left-right sides (metal 2 and metal 4 on top for the data bus and its negated version), which improves the attainable pitch. Figure 4.12 shows the position and relative areas occupied by the different functional blocks on the pixel (left top pixel of the group). Due to the layout rules, a large separation must be provided between any floating (“hot”) Nwell, such as the photodiode, and the rest of circuitry. In order to take some advantage of this a priori wasted space, the metal aperture over this area has been extended as much as possible. Carriers created within this area can contribute to the pixel photocurrent by reaching the photodiode collection volume through diffusion. The metal opening in the layout is 9.75 ×7.3μm2whereas the photodiode is 3 ×3μm2. 4.5. Digital to Analog Converter A digital to analog converter (DAC, or D-to-A) generates the analog reference Vref . The temporal evolution of Vref is sketched in figure 4.13. First, it takes the value of the reset voltage Vrst during photodiodes initialization, then it goes down to Vbot during the first 15 temporal windows (usually most of the exposition time), and finally it ramps up from Vbot to Vtop at the end of the exposition time in order to provide the A-to-D conversion of the low illuminated pixels. The large initial voltage drop, from Vrst to Vbot, must be as fast as possible in order to expand the illumination capturing range of highly illuminated pixels. Indeed, if a pixel is so bright that it discharges the integration node before Vref has reached Vbot, the SRAM on this pixel will keep its reset value (127). This fact is indeed the limiting factor for capturing high illuminations. There are also some limitations regarding the rate of change of Vref during the A-to-D conversion in the last temporal window. On the one hand, the time between steps (A-to-D clocking) must be long enough to allow proper settling of the Vref signal through the long (almost 6mm) horizontal wires carrying this signal to the pixels. On (a) Functional Blocks Positions. 1.18% 6.54% 21.28% 21.17% 15.18% 34.64% 1-Reset Switch 2-Photodiode + Metal Opening 3-Comparator 4-Buffer 5-Digital Control 6-Memories (b) Area percentage. Figure 4.12: Pixel Area Organization. 119 4. TVHC: A HDR TONE MAPPING IMAGER IN STANDARD CMOS TECHNOLOGY Vtop Vbot Vrst Figure 4.13: Voltages generated by DAC block. the other hand, since the both Vtop and Vbot are programmable (in order to adapt to the real linear range of the in-pixel amplifiers, and to diminish the effects of dark current on dark pixels), the accuracy of the settling process must be adapted accordingly in order to keep the equivalent number of bits above the 7-bit limit. In other words, if the ADC operating voltage swing is reduced, then the allowable settling error must be lowered, and so the frequency of the A-to-D clock. The rate of change of Vre f is controlled by external signals. In order to save IO pins, the ADC uses a counter (with signals DAC CLK and DAC CLR as controls) instead of having a 7-bit input to program the ADC output voltage. Besides, a 2-bit control bus DAC MODE selects whether the DAC has to provide Vrst,Vbot, or the voltage ramp: DAC MODE =00 ⇒Vrst DAC MODE =10 ⇒Vbot DAC MODE =01 ⇒Counting Mode (output voltage ramps up) The DAC is built using a classical resistor ladder architecture [129] whose basic element contains a 260.87Ω1poly2 resistor and a complementary CMOS switch. Since this is quite a conventional design, we will not add any further details here. Just for verification purposes, figure 4.14 shows the simulation results for Differential and Integral Nonlinearity Errors [130]. 4.5.1. Dark Signal Contribution Attenuation We have already mentioned how we can attenuate most of the offset contributions from the analog circuitry in the pixel by using an auto-zeroing technique to establish the reset voltage. The ultimate purpose of this calibration process is to improve the uniformity of the captured image. In this section, we will present another possibility provided by the chip to enhance the uniformity of the final images, specially in its darker zones, where noise is usually more noticeable. In this case, the main goal is to reduce the visual effects of the dark current on the darker pixels. To that purpose, we will employ a programmable definition of the voltage levels for A-to-D conversion. 1Result provided by the extraction tool in Cadence. 120 20 40 60 80 100 120 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 x 10−6 Input Code LSBs (a) DNL 20 40 60 80 100 120 −4 −3 −2 −1 0 1 2 3 x 10−5 Input Code LSBs (b) INL Figure 4.14: DAC Characteristics. It is easy to understand that the effects of the dark current will be more noticeable in those pixels having a photogenerated current that is similar to the level of the dark current. In such cases, these pixels may look very noisy and provide little extra details to the image (except in some remarkable situations, like automotive applications), while they do worsen the overall quality of the image, which is appreciated by an independent observer. When correctly operated in the HDR mode, our chip will codify these dark pixels within the last bin, and will digitize them using a single-slope A-to-D conversion. The current Ipix, which discharges the integration node Vph in the pixels, can be basically considered as created by the photogenerated current Iph plus the dark current Idark. Figure 4.15 illustrates how we can attenuate the visual effects of dark current over the darker zones of the image. Assume that we can measure (as we will explain in chapter 5) the dark current contribution (basically its average value Idark and standard deviation σ[Idark]) and that we can also measure, at the same time, the temperature of the chip1, so that we can create a table containing temperature, Idark,σ(Idark), and exposition time. Now, if we lower the top voltage of the A-to-D converter such that a pixel whose total discharge current (Iph +Idark)isIdark +3σ(Idark)2produces a voltage drop below Vrst −Vtopnew , then these pixels will be in the saturation zone of the A-to-D conversion curve, and consequently will be assigned to the value 0. On the one hand, by doing this, we are clearly neglecting all those pixels whose pixel current is below this limit, and on the second hand, we are reducing the full signal scale of the A-to-D converter as well, or in other words, making its LSB smaller. Thus, we need to guarantee that, for this new voltage swing: The ADC keeps its 7-bit accuracy (including cumulative settling errors), see next section. The analog circuitry in-pixel keeps its 7-bit accuracy (remember that, accordingly to simulations we can vary the operating voltage range from a maximum of 1.9V to just 0.57V). 1Remember that we have the Vbias output which comes from a PTAP. 2For the current temperature and exposure configuration. 121 4. TVHC: A HDR TONE MAPPING IMAGER IN STANDARD CMOS TECHNOLOGY top bot rst p dark+3 pix Figure 4.15: Voltage range for Dark Signal elimination. 4.6. Vref Distribution Scheme Leaving aside the design of the HDR algorithm and the implementation of the in-pixel circuitry, the scheme to precisely distribute the Vre f (t) signal to a set of 148 ×180 (26640) nodes, in an array which is nearly 5mm ×6mm, has been by far the hardest design problem in our work. Needless to mention, the accurate distribution of Vre f (t) is cornerstone for the correct operation of the whole chip. Indeed, differences in Vref (t) across the pixels (due to either offset or gain errors from the buffers driving this signal to different rows, settling differences due to the different distances, as this signal has to travel to reach every pixel, etc.) directly add to the residual offset of the in-pixel circuitry. Therefore, it can, eventually, ruin the operation of the imager, if they are significant enough. If one seeks for a precise mismatch-free distribution of the Vre f signal, the most straightforward (and logic) approach is to employ a single DAC followed by a powerful output buffer to drive the array. This solution has some important limitations though. First, driving the huge array of Vre fij nodes from a single point implies that one cannot operate faster than the intrinsic time constant of the whole array. Also, since there is only one buffer providing the power, there is a large concentration of power consumption in a very reduced area. This unavoidably leads to the creation of a hot spot around this place, which affects the performance of the pixels nearby (due to the increased temperature in the zone1). Let us talk about temporal specifications at this moment. Obviously, we cannot talk properly about temporal specifications without knowing the type of load we are driving. Since the array of pixels was already designed when we started the design of the Vref distribution scheme, we were able to extract its parasitic capacitances and resistances, and compound a parasitic simulation model. Then, this complex model (it contains more than 26000 nodes) was simplified as follows in order to obtain useful design equations and meaningful constraints: We extract a row of pixels and excite the Vref node with a constant 1A input current (this is a convenient value which simplifies calculations, and since we are considering linear parasitics it does not create any problem). In this simulation, we measure the slope of the Vref signal in all nodes in the row (to check for discrepancies due to the internal time constant of the pixels) and 1Observe that dark current has an exponential dependence with temperature [20]. 122 extract the associated capacitance of the row simply as 1/slope. Simulation results show that Crow =10.82pF. Now, we excite with a voltage pulse and measure how it propagates through the 180 pixels in this row. We measure the time constant at the farthest point, and define it as the row time constant. Simulation results show that τrow =4.54ns. We adjust the Rrow parameter such that a first order RC system with Crow =10.82pF has the same time constant. The result is that Rrow needs to be configured to be Rrow =419.59Ω. To check the correct operation of the 1st order model, we compare the dynamic evolution of the simplified model with that at the farthest point in the array. The only noticeable differences happen at the very first 200ps, however, the dynamic evolution is identical (to the 4th decimal number) from this time on, and therefore, we consider this model valid. The same procedure is applied for the whole array (which in addition to the 148 rows of 180 pixels also contains the parasitics associated to the vertical line that interconnects all Vref rows). In this case, we decide to use as input port the point in the middle of the array (that is the optimum if we are distributing the signal from a single point). From simulations, we obtain τarray =787ns, with Rarray =483.11Ω,Carray =1.629nF. Having the load modeled, we need to talk about settling error specifications. Clearly, during the last temporal window, the 7-bit single slope A-to-D conversion establishes a maximum accumulated error that must remain below 1 2LSB. In our case, the accumulated error comes not only from the settling error in the DAC, but also from the noise, DAC’s non-linearities, offsets (will see later), and the already mentioned residual error in the operation of the auto-zeroing employed to establish the reset. Considering that for the maximum allowable voltage swing (2.6V-0.7V) the 7-bit accuracy LSB corresponds to some 14.84mV, and that the accumulated errors are estimated (from simulations) to be around 3·737.272μs1=2.21mV plus other effects, so we overestimate the total in 3mV, the maximum settling error (Emax) must not be larger than some 14.84mV 2−3mV =4.42mV, which giving an additional margin we choose about 4mV. These results in two different situations depending on whether we are within the last temporal window (doing the A-to-D conversion) or moving from Vrst to Vbot just after releasing the reset signal. When executing the A-to-D conversion in the last step, the Vre f signal to be transmitted to the array moves in steps of 1LSB from Vbot to Vbot +27LSB. Assuming that each step has a duration Tstep, the 1σin auto-zero enabled simulations. 123 4. TVHC: A HDR TONE MAPPING IMAGER IN STANDARD CMOS TECHNOLOGY settling error Ekat the kth step can be evaluated using1: V(k·Tstep)=Vbot +k·LS B −Ek Ek=LS B j=k  j=1 e−jα=LS B 1−e−(k+1)α 1−e−α−1 α=Tstep τ (4.14) Since the settling error accumulates step by step, the worst case (obviously) is committed at the last conversion step, k=27. Hence, our constraint becomes: Emax ⩾LS B 1−e−(k+1)α 1−e−α−1⇒1−e−(k+1)α 1−e−α⩽1+Emax LS B (4.15) For (k+1)αvalues larger than 4, or, in other words, we apply TSTEP times that are not shorter than τ/32, this equation converges to: Tstep ⩾τ·ln 1+LS B Emax (4.16) which results in Tstep(τarray)⩾1.22μs, and Tstep(τrow)⩾7ns. Therefore, it does not matter how powerful the output buffer of the DAC is, we simply cannot transmit Vref to the array with the necessary accuracy in less than 1.22μs, if we want to keep our specifications within the last step of the analog staircase signal. This also has an important implication regarding the duration of the last temporal window, since it sets a minimum of Tramp(τarray)⩾156μsor Tramp(τrow)⩾0.9μs. A second constraint is derived from the first transition of the Vref signal. Just after reset, this signal is changed from Vrst to Vbot. The time required to establish this change with the required accuracy (Emax) defines the maximum photogenerated current that can be assigned a value different from 0 (that is the maximum detectable photocurrent). Thus, if we need to produce a change from Vrst to Vbot, the minimum time for producing this change with the required accuracy is: Tmin ⩾τ·ln Vrst −Vbot Emax (4.17) A priori, we can assume that Vrst Vtop, hence, this limit becomes: Tmin ⩾τ·ln 2N·LS B Emax (4.18) which evaluates to Tmin(τarray)⩾4.85μs, and Tmin(τrow)⩾27.98ns. Thus, we can say that the maximum theoretical ratio between largest detectable photocurrents, if we drive Vref to the array from a single point, and if we use one buffer per row is: 1Here, we use that the Sum of Geometric Progression is j=k−1 j=0rj=1−rk 1−r[131]. 124 R=τarray τrow =168.99 (4.19) Assuming that the minimum detectable photocurrent is limited by noise factors, driving Vref to the array row by row results in a maximum Dynamic Range improvement of nearly 20 ·log10(168.99) = 44.56dB (simply because most of our algorithm relies on making temporal measurements), when compared to the alternative solution of using a single buffer for the whole array. Needless to say, this is a theoretical limit, which assumes that we can incorporate a buffer per row whose response time is significantly faster than τrow (4.54ns), something which is not trivial at all in a 0.35μm technology. Clearly, if the time to load a row is limited by the internal dynamics of the buffer (τbu f ) rather than for the time constant of the row, the ratio in equation 4.19 becomes: R=τarray τbu f (4.20) hence, as long as we can design a buffer having the pitch of a row1, which can drive a row much faster than τarray, we would always have a Dynamic Range improvement when using a distributed topology to transmit Vref to the pixels as compared to the single buffer option. However, distributing Vre f using one buffer per row has an important drawback, the offset voltages of the buffers add to the signal being transmitted. Therefore, the specifications may become unreachable not because of the speed of the system, but because the final point is simply incorrect. Consider that, the error must remain below Emax =4mV for the typical operating range [0.7, 2.6] ( and it may need to be lowered when using a lower range). If we maintain our 3σapproach, it means that the standard variation of the offset voltage of the buffer should not exceed some 4mV 3=1.3mV in the typical case. As a first solution, one may consider using any of the typical analog techniques for offset correction in amplifiers (including using very large transistors in the differential pair, common centroid configurations, auto-zeroing, etc.). Our system has a particularity though; the buffer must have the same pitch as the pixel. This, together with the need for a long lasting calibration of the offset voltages (because of the long exposures employed in low-illuminated scenes), made us finally opt for a hybrid solution, which combines, sequentially, a distributed buffer topology, and a single point driving for a precise establishment of the final voltage. Our solution combines the use of one amplifier per row during the first moments of Vre f changes, and driving the array from a single point (and rely on the charge redistribution principle) at the final part of the transient to guarantee that all Vrefij nodes reach the same final voltage, or more precisely, that all Vre f nodes are established to the same voltage with the desired accuracy level. Therefore, we will not meet our maximum attainable speed (limited by τrow) but neither we shall be limited2by the time constant of the whole array (τarray). The solution, which is given the name of Charge Injection Amplifiers Block, is inserted between the output of the DAC and the horizontal metal lines that distribute Vref to the pixels row by row. Figure 4.16 shows the schematic of the cell, which is included in every row. Here, Vdac 1And sufficiently small in the horizontal dimension to not affect significantly the final size of the chip. 2Only partially limited. 125 4. TVHC: A HDR TONE MAPPING IMAGER IN STANDARD CMOS TECHNOLOGY NOVOFF nNOVOFF NOVOFF nNOVOFF Vdac Vrefi IbiasP IbiasN nPOWER_ON POWER_ON POWER_ON nPOWER_ON NOVOFF nNOVOFF Figure 4.16: Charge Injection Amplifier Cell Schematic. is the output of the DAC block and Vre fiis the Vref metal line of the ith row. Clearly, two signal paths can be enabled, depending on the state of two complementary transfer gates. One path allows a direct connection between Vdac and Vre fi(NOVOFF=1), while the alternative path inserts a buffer between these two nodes (NOVOFF=0). The operation of the charge injection amplifier block is illustrated in figure 4.17. At the very end of the reset phase, when the Vref signal must drop very fast from Vrst to Vbot, the amplifiers are connected allowing this change to happen faster than if we were using a single point distribution scheme. However, the steady state voltages are incorrect due to the introduced offsets. Once Vre finodes have stabilized, the amplifiers are disconnected, and the direct path is engaged short-circuiting all Vrefirow lines to the output of Vdac. Now, charge redistribution occurs and all nodes will reach the same final voltage once the transient settles down. The key idea here is that we only need to evolve following the time constant of the array the time required to lower the average settling error (because we have many amplifiers) of the first phase (evolving with the time constant of the buffer) below our accuracy limit. Let us illustrate how are we gaining operation speed thanks to this configuration. Assume that all Vrefilines reach their steady state value Vrefi(Trst)=Vrst +Vof f i(where Vof f iis the offset voltage of the buffer driving the ith row), and that now we switch from Vrst to Vbot keeping the buffers connected during a time T1. In this case, every Vre filine reaches a voltage value: Vrefi(T1)=(Vbot +Vof f i)+(Vrst −Vbot)e−T1/τbu fi(4.21) Now, we disconnect the buffers and engage the direct path for a time T2. In this case, the evolution is dictated by τarray, such that the final voltage at the farthest point in the array reaches a value1: 1Here we assume (validated through Monte Carlo simulations) that: (1)i=Nrows i=1 Vof f i Nrows ≈0 due to the large number of rows. (2) Redistribution of charge at the beginning of the second phase of the transient is only significant between the very close neighbors of a row and, hence, it is not limited by whole array dynamics. Or, in other words, if we select a row, the charge stored in a very reduced numbers of rows (up and down) around this row is very close to that of the offset free case. 126 ref dac rst bot Figure 4.17: Charge Injection Amplifier Operation. Vref (T2)=Vbot +  Vbot +(Vrst −Vbot)e−T1/τbu f −  Vbote−T2/τarray (4.22) therefore, the settling error at t=T1+T2is given by: Eset =(Vrst −Vbot)e−T1/τbu f −T2/τarray (4.23) whereas if we were driving the array from a single point for the same time, we would have: E∗ set =(Vrst −Vbot)e−(T1+T2)/τarray (4.24) and the question is, under what conditions (Eset <E∗ set)?, which results in the following inequality: e−T1/τbu f −T2/τarray <e−(T1+T2)/τarray (4.25) which can we rewritten in the form: −T1 τbu f −  T2 τarray <−T1 τarray −  T2 τarray (4.26) The solution to this type of inequality is: τarray >τ bu f (4.27) The inequality results in the logic condition that the buffers response time must be faster than that of the array. 127