Repositorio Institucional de Documentos

Tesis Fin de M´aster M´aster en Ingenier´ıa de Sistemas e Inform´atica Curso 2009-2010 Reverse Tone Mapping for Suboptimal Exposure Conditions Bel´en Masi´a Corcoy Septiembre de 2010 Director: Diego Guti´errez P´erez Departamento de Inform´atica e Ingenier´ıa de Sistemas Centro Polit´ecnico Superior Universidad de Zaragoza Gracias... ...a mis padres y hermana, por aguantarme todo, por su eterno e incondicional apoyo, y por todo lo que he aprendido de y gracias a ellos. ...a Diego, por su ayuda, por su infinita paciencia y sus infinitos ´animos, y por tant´ısimas cosas que he aprendido de ´el. ...a Jos´e Mar´ıa Mart´ınez Montiel, por darme la oportunidad que me dio y por su contribuci´on a que yo est´e en el mundo de la investigaci´on. ...a la gente del lab, porque trabajar con personas as´ı hace que todo sea mucho m´as f´acil. ...a ´ Oscar, porque sin su ayuda yo no estar´ıa ahora escribiendo esto, y por supuesto por los buenos ratos. ...a mis amigos, por las risas cuando estaba de baj´on (y cuando no lo estaba :) ), por las juergas para desconectar, por las horas escuch´andome hablar (y lamentarme) de funciones sigmoidales, linealizaciones o experimentos psicof´ısicos, y de lo duro que es esto de la investigaci´on, por los innumerables caf´es, por las cervezas a las ocho de la tarde, y sobre todo, por estar ah´ı siempre. RESUMEN La mayor parte de las imágenes y videos existentes son de bajo rango dinámico (generalmente denominado LDR por las siglas del término en inglés, low dynamic range). Se denominan así porque, al utilizar sólo 8 bits por canal (R,G,B) para almacenarlas, sólo son capaces de reproducir dos órdenes de magnitud en luminancia (mientras que el sistema visual humano puede percibir hasta cinco órdenes de magnitud simultáneamente). En los últimos años hemos asistido al nacimiento y expansión de las tecnologías de alto rango dinámico (HDR por sus siglas en inglés), que utilizan hasta 32 bits/canal, permitiendo representar más fielmente el mundo que nos rodea. Paulatinamente el HDR se va haciendo más presente en los pipelines de adquisición, procesamiento y visualización de imágenes, y como con el advenimiento de cualquier nueva tecnología que sustituye a una anterior, surgen ciertos problemas de compatibilidad. En particular, el presente trabajo se centra en el problema denominado reverse tone mapping: dado un monitor de alto rango dinámico, cuál es la forma óptima de visualizar en él todo el material ya existente en bajo rango dinámico (imágenes, vídeos...). Lo que hace un operador de reverse tone mapping (rTMO) es tomar la imagen LDR como entrada y ajustar el contraste de forma inteligente para dar una imagen de salida que reproduzca lo más fielmente posible la escena original. Dado que hay información de la escena original que se ha perdido irreversiblemente al tomar la fotografía en LDR, el problema es intrínsecamente ill-posed o mal condicionado. En este trabajo, en primer lugar, se ha realizado una serie de experimentos psicofísicos utilizando un monitor HDR Brightside para evaluar el funcionamiento de los operadores de reverse tone mapping existentes. Los resultados obtenidos muestran que los actuales operadores fallan -o no ofrecen resultados convincentescuando las imágenes de entrada no están expuestas correctamente. Los rTMO existentes funcionan bien con imágenes bien expuestas o subexpuestas, pero la calidad percibida se degrada sustancialmente con la sobreexposición, hasta el punto de que en algunos casos los sujetos prefieren las imágenes originales en LDR a imágenes que han sido procesadas con rTMOs. Teniendo esto en cuenta, el segundo paso ha sido diseñar un rTMO para esos casos en los que los algoritmos existentes fallan. Para imágenes de entrada sobreexpuestas, proponemos un rTMO simple basado en una expansión gamma que evita los errores introducidos por otros métodos, así como un método para fijar automáticamente un valor de gamma para cada imagen basado en el key de la imagen y en datos empíricos. En tercer lugar se ha hecho la validación de los resultados, tanto mediante experimentos psicofísicos como utilizando una métrica objetiva de reciente publicación. Por otro lado, se ha realizado también otra serie de experimentos con el monitor HDR que sugieren que los artefactos espaciales introducidos por los operadores de reverse tone mapping son más determinantes de cara a la calidad final percibida por los sujetos que imprecisiones en las intensidades expandidas. Adicionalmente, como subproyecto menor, se ha explorado la posibilidad de abordar el problema desde un enfoque de más alto nivel, incluyendo información semántica y de saliencia. La mayor parte de este trabajo ha sido publicada en un artículo publicado en la revista Transactions on Graphics (índice JCR 2009 2/93 en la categoría de Computer Science, Software Engineering, con un índice de impacto a 5 años de 5.012, el más alto de su categoría). Además, el Transactions on Graphics está considerado como la mejor revista en el campo de informática gráfica. Otra publicación que cubre parte de este trabajo ha sido aceptada en el Congreso Español de Informática Gráfica 2010. Como medida adicional de la relevancia del trabajo aquí presentado, los dos libros existentes hasta la fecha (hasta donde sabemos) escritos por expertos en el campo de HDR dedican varias páginas a tratar el trabajo aquí expuesto (ver [2, 3]). Esta investigación ha sido realizada en colaboración con Roland Fleming, del Max Planck Institute for Biological Cybernetics, y Olga Sorkine, de New York University. Contents 1 Prologue 1 2 Introduction 1 2.1 High Dynamic Range Imaging: background . . . . . . . . . . . . . . . . . . . . . 1 2.2 Reverse Tone Mapping: the problem . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.3 Approach and contributions of this work . . . . . . . . . . . . . . . . . . . . . . . 3 3 Previous Work 5 3.1 Reversetonemapping ................................. 5 3.2 Userstudies....................................... 6 4 Experiment One: rTMO Evaluation 7 4.1 Introduction....................................... 7 4.2 Stimuliandsubjects .................................. 7 4.3 Procedure........................................ 9 4.4 ResultsandDiscussion................................. 9 5 Experiment Two: HDR vs. LDR Monitor 12 5.1 Introduction....................................... 12 5.2 ResultsandDiscussion................................. 12 6 Expanding Over-exposed Content 14 6.1 Introduction....................................... 14 6.2 Gammaexpansion ................................... 14 6.3 Validation........................................ 16 7 Selective Reverse Tone Mapping 18 7.1 Introduction....................................... 18 7.2 Using the Zone System for rTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 7.3 Content-awarerTM .................................. 21 7.3.1 Detecting salient features . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 7.3.2 Expanding the dynamic range . . . . . . . . . . . . . . . . . . . . . . . . . 22 7.4 ResultsandDiscussion................................. 22 8 Conclusions and future work 24 8.1 Conclusions....................................... 24 8.2 Futurework....................................... 25 ii References 27 A Complete set of stimuli 31 B Objective validation: complete results 33 C Subjective validation for the dark images series 36 D Saliency Detection 37 D.1 Learning-based saliency detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 D.2 SaliencyCuts...................................... 38 E Publications 40 iii List of Figures 2.1 The problem of traditional LDR imaging . . . . . . . . . . . . . . . . . . . . . . . 2 2.2 The multibracketing technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.3 The reverse tone mapping problem . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4.1 Representative samples of the stimuli used in our tests . . . . . . . . . . . . . . . 8 4.2 Sample complete bracketed sequences . . . . . . . . . . . . . . . . . . . . . . . . 8 4.3 Results of our rTMO evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.4 Distribution of outlier indices for all four rTMOs . . . . . . . . . . . . . . . . . . 11 5.1 LDRvs.HDR ..................................... 13 6.1 Expansion following a gamma curve . . . . . . . . . . . . . . . . . . . . . . . . . 15 6.2 Objectivevalidation .................................. 17 7.1 Examples of images containing large saturated areas . . . . . . . . . . . . . . . . 18 7.2 Division of luminance in zones according to Ansel Adams’s System . . . . . . . . 19 7.3 Luminance decomposition for zone-based reverse tone mapping . . . . . . . . . . 20 7.4 Zone-based reverse tone mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 7.5 Saliency detection methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 7.6 Complete pipeline using our interactive rTM approach . . . . . . . . . . . . . . . 23 7.7 Reverse tone mapping using different zone-based expansion functions for the salient object and the background . . . . . . . . . . . . . . . . . . . . . . . . . . 23 A.1 Complete set of stimuli used in our experiments. . . . . . . . . . . . . . . . . . . 32 B.1 Objective validation: Building scene . . . . . . . . . . . . . . . . . . . . . . . . . 33 B.2 Objective validation: Sunset scene . . . . . . . . . . . . . . . . . . . . . . . . . . 34 B.3 Objective validation: Graffiti scene . . . . . . . . . . . . . . . . . . . . . . . . . . 34 B.4 Objective validation: Strawberries scene . . . . . . . . . . . . . . . . . . . . . . . 35 B.5 Objective validation: Lake scene . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 C.1 Results for the dark images series . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 iv 1. Prologue When I entered this project, a collaboration between Roland Fleming from the Max Planck Institute in T¨ubingen, Olga Sorkine from New York University, and Diego Gutierrez from Universidad de Zaragoza had already been established, and the whole of the project has been carried out within this collaboration. The aim was to work on the field of reverse tone mapping, and the starting point, the intuition that current algorithms would fail if the image was not correctly exposed. The first steps were taken with a previous publication, by Miguel Martin and the three already cited authors, where image exposure, and its possible correlation with different image statistics, was analyzed [1]. I entered the project at that point, and we had the double objective of: a) formally evaluating the current algorithms, and b) if there was room for improvement, devising a new reverse tone mapping operator. Two publications have spawned from this work, the initial objectives being widely covered. The first of them, Evaluation of Reverse Tone Mapping Through Varying Exposure Conditions, which covers most of the contents of this master thesis, was presented at SIGGRAPH Asia 2009 in Yokohama, Japan. A total of 70 out of 275 papers were accepted for the conference, the acceptance rate being 25%. Articles accepted for this conference are also published in the journal Transactions on Graphics, whose JCR 2009 index is 2/93 in the category of Computer Science, Software Engineering (5-year impact factor 5.012, highest of its category). Transactions on Graphics is regarded as the top journal in the field of Computer Graphics. The second publication, Selective Reverse Tone Mapping, a smaller sub-project which covers the work described in Chapter 7, has been accepted for publication in Congreso Espa˜nol de Inform´atica Gr´afica 2010. As another representative measure of the impact of the work here presented, the only two technical/scientific books (i.e. in academia, written by researchers in the field; there are many divulgative books on HDR for photographers) which, to our knowledge, have been published on high dynamic range imaging, dedicate several pages to the explanation of our work and findings on reverse tone mapping. These books are High Dynamic Range Imaging, Second Edition: Acquisition, Display, and Image-Based Lighting [2] and Advanced High Dynamic Range Imaging Theory and Practice [3]. The present master thesis contains essentially the same information as both papers, adding an explanation on the problem and its background and certain data and information which did not appear in the articles, and merging them into one single work. My interest in what the emergent high dynamic range technologies would bring lead me to begin research in this field. With this master thesis I put an end to my work on reverse tone 1 1. Prologue mapping, but my interest on image capture, processing and editing has continued to increase, and I will be carrying on with research in the field of computational photography, hopefully in successful collaboration with the Camera Culture Group from the MIT Media Lab (lead by Ramesh Raskar). 2 2. Introduction 2.1 High Dynamic Range Imaging: background The dynamic range of a scene is defined as the difference between the highest and the lowest luminance values of the scene. The human visual system (HVS) can cope with lighting conditions that range over up to 10 orders of magnitude in luminance. As an example, a dark night with no moon and the stars as the only light would typically have luminance values of around 10−3cd/m2, while sunlight at noon on a clear bright day is around 105cd/m2. This is achieved through adaptation, that is, the HVS can allocate its dynamic range at will. Adaptation is the reason for well known phenomena like the fact that when we come out of the cinema or out of a dark place into sunlight it takes us a few seconds to see properly. In a certain instant, within the same scene, we can perceive up to five orders of magnitude in luminance. However, our HVS does not adapt equally well to all conditions: in dark scenarios we perceive luminance differences well, and color badly, while the opposite holds for bright scenarios. Even though we can perceive up to ten orders of magnitude in luminance (five within the same scene), traditional capture and display systems can deal with up to two log-units of luminance, since they use 8 bits per channel. As a consequence, many times traditional images fail to faithfully recreate the real world. An example of this is shown in Figure 2.1. None of the images succeed in reproducing the real scene as a human observer would see it. Through the adjusting of exposure, range can be allocated either to the foreground (left) or to the background (right), but our HVS would be able to perceive both foreground and background simultaneously. As a solution to this, Debevec and Malik proposed their multibracketing technique [4]. An image of high dynamic range (HDR) can be obtained by combining several single low dynamic range (LDR) photographs taken with different exposures (see Figure 2.2). The resulting images, which are HDR, use 16 or 32 bits per channel. After this pioneer work, and during the last decade, we have assisted to a great expansion of HDR technologies, not only for capturing images, but also for displaying, storing and processing them. It is now common believe that HDR technology will be commonplace in the near future. However, as with any other big change in technology, there is a transition time during which both technologies, LDR and HDR, coexist. The first issue that arised was how to display HDR images in the LDR off-the-shelf displays that everyone had at home. This gave origin to the problem of tone mapping, and dozens of algorithms have been developed in the last decade which try to solve this problem in the best possible way. The opposite problem is called reverse tone mapping: once HDR displays become common, what do we do with all legacy material, 1 4. Experiment One: rTMO Evaluation Figure 4.1: Representative samples of the stimuli used in our tests. Top: bright images (Building, Lake, Graffiti, Strawberries, Sunset), each showing a certain degree of over-exposure. Bottom: dark images (Car, Flowers, Crayons, Pencils), with varying degrees of under-exposure. Figure 4.2: The complete bracketed sequence for the Building and Flowers scenes. and over-exposure. From each exposure in the bracketed sequence, we obtained three candidate renditions for display on the HDR monitor using a representative subset of reverse tone mapping algorithms: ldr2hdr [17], Banterle’s operator [13] and linear contrast scaling [8]. Except for the straightforward linear scaling (in Yxy color space, and thus performed on linearized values) we obtained the images from the authors of the original algorithms, in order to ensure accuracy in the implementation. For the ldr2hdr algorithm the parameters used were 150 pixels for the standard deviation of the large Gaussian blur applied to the mask, a brightness amplification factor α= 4 and a gradient image baseline width for divided differences of 5 pixels, plus a 9×9-pixel kernel for the antialiasing blur and a 4-pixel radius for the open operator used to clean up the final edge stopping function (please refer to the original paper for a detailed explanation of these parameters). In the case of Banterle’s operator, when generating the expand-map, the parameters of the density estimation were a radius ranging from 16 to 42 pixels (smaller radius for lower exposures) and a threshold of 1 to 4 light sources (lower threshold for higher exposures), being 2048 the number of generated light sources for Median Cut sampling. In both cases, Banterle’s operator and ldr2hdr, images were linearized using gamma correction (γ= 2.2). We also added a fourth LDR rendition in which the original images are presented within a luminance range matched to a typical desktop TFT monitor. The goal of this fourth image is to study whether the established assumption that visual preference is given to HDR holds over a range of exposures. A gender-balanced set of twelve subjects with normal or corrected-to-normal acuity and normal color vision were recruited to participate in the experiment. All subjects were unaware of the purpose of the study, and were unfamiliar with HDR imaging. 8 4. Experiment One: rTMO Evaluation 4.3 Procedure Participants viewed the stimuli on the Brightside HDR display in a dark room. On each trial, subjects were presented with all four renditions of a given exposure of a given scene in a 2×2 array (a stimulus quadruple). The positions of the four renditions within the array were random across trials, and the order of the trials was random with the constraint that consecutive trials did not present the same scene. The subjects’ task was to rate the quality of the four renditions on a scale from 1 to 7, according to how accurately the images depicted how the scene would appear to the subject if they were actually present in the scene. Thus the key criterion for comparison was the subjective fidelity of the renditions. Subjects were given unlimited time for each trial and could modify their rating of any of the renditions on a given trial before proceeding to the next trial. Additionally, they were free to assign the same values to all four renditions on a given trial, although they were instructed to try to use as much of the 1-7 scale as possible within the experiment as a whole. To aid them in setting their scale, and to accustom them to the experimental procedure, the subjects were presented with a number of practice trials before the start of the experiment. 4.4 Results and Discussion Several conclusions can be drawn from this test. First, for our images, there was a clear difference in perceived quality between the bright and the dark series: subjects clearly preferred the reverse tone mapped depictions of darker images over brighter ones. This can be seen in Figure 4.3: not only is the overall mean value significantly higher in the former case, but it is relatively stable across exposure as well. In contrast, for the bright images, there is a general downward trend in ratings across the four exposure levels. Note that this gradual decrease in performance does not correlate with the subjective perception of quality of the original LDR image: in a previous pilot study, users picked different exposures for each series as the subjective best, not necessarily the same as the objective best (defined as the one with the smallest proportion of under- and over-exposed pixels [8]). The trend instead correlates with the proportion of over-exposed pixels and the mean luminance, which do increase with exposure. Secondly, we can observe systematic differences between the rTMOs. On average, subjects rated the ldr2hdr and the Linear rTMOs best (the difference between the two failed to reach statistical significance), followed by the LDR images, and finally the output of the Banterle’s rTMO (see Figure 4.3). Pairwise Wilcoxon rank sum tests (similar to a non-parametric version of the t-test) reveal that these differences were significant to p < 0.05, except for ldr2hdr vs. Linear in the bright series and Banterle’s operator vs. the LDR depiction in the dark series (see Table 4.1 for the complete results). It is important to note, however, that this ordering does not hold for all conditions. For instance, the LDR depiction was systematically ranked lower than two of the rTMOs, suggesting that indeed HDR visualization is still preferred over LDR, even for under- and over-exposed images. Surprisingly, though, it ranked higher on average than Banterle’s rTMO for bright images. The poor overall performance of Banterle’s rTMO with this data set is probably due 9 4. Experiment One: rTMO Evaluation l2h Ban Lin LDR 1 2 3 4 5 6 7 Exposure 1 Mean Rating l2h Ban Lin LDR 1 2 3 4 5 6 7 Exposure 2 Mean Rating l2h Ban Lin LDR 1 2 3 4 5 6 7 Exposure 3 Mean Rating l2h Ban Lin LDR 1 2 3 4 5 6 7 Exposure 4 Mean Rating l2h Ban Lin LDR 1 2 3 4 5 6 7 Exposure 1 Mean Rating l2h Ban Lin LDR 1 2 3 4 5 6 7 Exposure 2 Mean Rating l2h Ban Lin LDR 1 2 3 4 5 6 7 Exposure 3 Mean Rating l2h Ban Lin LDR 1 2 3 4 5 6 7 Exposure 4 Mean Rating 1 2 3 4 1 2 3 4 5 6 7 Ratings as a function of Exposure Mean Rating Exposure ldr2hdr Banterle Linear LDR Gamma 1 2 3 4 1 2 3 4 5 6 7 Ratings as a function of Exposure Mean Rating Exposure ldr2hdr Banterle Linear LDR Figure 4.3: Top: bright images series. The blue bars represent the mean ratings across subjects for the four rTMOs (ldr2hdr, Banterle’s, Linear and LDR) with increasing exposure levels. The last chart clearly shows the downward trend in perceived image quality. Error bars represent standard errors on the mean. The red line in the first four charts represents the mean ratings for our proposed γ-curve expansion (see Chapter 6). It can be seen that it rates generally higher and is more stable. Bottom: same information for the dark images series, showing higher overall means and a more stable perceived quality across exposures. (i-j) pb(i, j)pd(i, j) ldr2hdr - Banterle’s 2.0532e-21 2.8633e-7 ldr2hdr - Linear 0.5734 0.0283 ldr2hdr - LDR 1.7762e-6 1.4976e-11 Banterle’s - Linear 1.1739e-22 0,0013 Banterle’s - LDR 4.4489e-11 0.1938 Linear - LDR 1.4697e-7 2.0538e-6 Table 4.1: Results of the Wilcoxon rank sum tests for the bright and dark series (denoted by subindices b and d respectively). Values of p < 0.05 are considered to indicate statistically significant differences between rTMOs. Thus, all differences were significant except for ldr2hdr vs. Linear in the bright series and Banterle vs. LDR in the dark series. to the fact that it often exaggerates the errors in poorly exposed images, resulting in intrusive artifacts. This becomes clear when we measure the extent to which each rTMO yields outlier rating values for each image. We calculate the median rating for each image across rTMOs. We then obtain the outlier index as the difference in rating for each rTMO relative to this median value: (Outlier index)rT MOi= (Rating)rT MOi−(Median rating across rTMOs) (4.1) When an rTMO is neutral, simply reflecting the overall quality of the exposure of the image, then the outlier index tends to be close to zero. However, when an rTMO stands out relative to the others (for example due to the introduction of artifacts), then the outlier index tends to deviate from zero. In Figure 4.4, we plot the histogram of the outlier index values for the three rTMOs and the LDR depiction. It is notable that for ldr2hdr, Linear and LDR, the distribution tends to be relatively tightly tuned, while for Banterle’s the spread is much broader. 10 4. Experiment One: rTMO Evaluation −4 −2 0 2 4 0 20 40 60 80 100 ldr2hdr −4 −2 0 2 4 0 20 40 60 80 100 Banterle −4 −2 0 2 4 0 20 40 60 80 100 Linear −4 −2 0 2 4 0 20 40 60 80 100 LDR −4 −2 0 2 4 0 20 40 60 80 100 ldr2hdr −4 −2 0 2 4 0 20 40 60 80 100 Banterle −4 −2 0 2 4 0 20 40 60 80 100 Linear −4 −2 0 2 4 0 20 40 60 80 100 LDR Figure 4.4: Distribution of outlier indices for all four rTMOs. Top: bright series. Bottom: dark series. This means that on the one hand, when it performs well, it tends to equal or exceed the others. However, it sometimes introduces substantial artifacts that cause the images to look worse than if they were not reverse tone mapped at all. Although this seems to contradict a recent study where Banterle’s operator actually outperformed other rTMOs [9], it is important to note that the experiments carried out in both cases differ significantly: first of all, in the work by Banterle et al. [9] the LDR source images were again well exposed, which is the regime within which Banterle’s rTMO performs well, as we also found. However, when the source material is less flattering, we found that the algorithm sometimes produces clearly visible artifacts, which leads to lower ratings. Second, in [9] the authors used a 2AFC paradigm with direct ground truth comparison, whereas we propose a rating approach, which allows users to report their relative subjective preferences. Both tasks are valid ways of assessing fidelity. However, ours has the advantage that it is closer to the real usage scenario: in general the ground truth is unknown and is not presented for comparison. 11 5. Experiment Two: HDR vs. LDR Monitor 5.1 Introduction We noticed that artifacts produced by ldr2hdr and Banterle’s rTMOs are typically visible in low dynamic range renditions of the images. This is because they generally have a spatial component: they are not simply due to inappropriate intensity levels for certain features, but they also include fringes, visibly boosted noise and other artifacts. To analyze this, we performed a second experiment with seven new subjects, which was identical to the first experiment, except that on each trial, the 2×2 stimulus array was tone mapped using histogram adjustment1[23]. The array was then presented on a standard TFT monitor (note that this means that the LDR control condition now appears much darker than on a normal TFT). 5.2 Results and Discussion In Figure 5.1, we plot the average ratings for each image in the LDR control condition against the average ratings in the HDR condition. As can be seen from the scatter plot, the ratings in the LDR control condition correlated extremely strongly with the ratings in the original experiment on the HDR monitor (r2= 0.9018). We found no significant difference between bright and dark images. This result does not imply that the images look the same in LDR as in HDR: the subjects were not asked to compare these conditions directly, and previous studies have confirmed that HDR depictions are preferred over LDR [8]. Indeed, none of the subjects saw both renditions. However, it does demonstrate that the pattern of preferences is extremely well conserved. In other words, the images that were less preferred on the HDR monitor were also less preferred when tone mapped back down to LDR. This has two important implications. First, the strong correlation found suggests that a reasonably predictive evaluation of a rTMO could be made without directly testing on an HDR monitor. Second, as noted, the subjective ratings of HDR images that have been generated from LDR images seem to depend more on the presence or absence of disturbing spatial artifacts than on the exact intensities of different features. A similar observation (confirmed by our test) was made by Aydin et al. [24]: they noted that the key issue in image reproduction is to accurately maintain the important features while preserving overall structure, whereas achieving an optical match becomes relatively less important. This becomes 1We have used the pcond program in Radiance to tone map the stimuli. 12 5. Experiment Two: HDR vs. LDR Monitor 1 2 3 4 5 6 7 1 2 3 4 5 6 7 Ratings on LDR monitor Ratings on HDR monitor Figure 5.1: Scatter plot showing a strong correlation between ratings on an HDR monitor and ratings when the images were tone mapped back down to LDR and presented on a standard TFT monitor. even more salient given that the dark-adaptation state of the observer is typically unknown, making absolute intensities meaningless to the user. The design philosophy that emerges from these considerations is that it is generally better to apply simpler, less-aggressive rTMO schemes if the original image is imperfect. Failing to fully recreate the HDR experience is less disturbing to users than unintended artifacts that can occur when poorly-exposed images are adjusted too aggressively. In the following section we present a simple and robust approach to boosting the dynamic range of over-exposed images, and show that it is less prone to artifacts than other rTMOs. 13 6. Expanding Over-exposed Content 6.1 Introduction Our experiments have shown that the danger with computationally sophisticated reverse tone mapping schemes is the potential to make the image appear worse than before processing, through the introduction of objectionable artifacts. However, the goal of a rTMO is to make the image content look better in general and avoid, under any circumstances, making it look worse. Simple global reverse tone mappers, such as linear scaling and gamma boosting, never cause polarity reversals, ringing artifacts or spuriously boost regions well beyond their context. Our first experiment clearly indicates that there is room for improvement in devising an rTMO for bright input images with large saturated areas, whilst darker images turn out much better. We thus focus on the former in this section. 6.2 Gamma expansion Examining the bright sequence in Figure 4.2 we observe that as exposure increases, more detail is lost as pixel values become saturated, and colors fade to white. It thus seems reasonable to attempt to depict the image in a way that the remaining details become more prominent, as opposed to boosting saturated areas as existing rTMOs do. Note that we do not aim to recover information lost to over-exposure, for which existing hallucination techniques may work [22], but rather to increase perceived quality. We make the following key observations, which have been confirmed by previous studies on reverse tone mapping: on the one hand, darker HDR depictions are usually preferred for bright input LDR images [19]; on the other hand, in many cases contrast enhancements improve perceived image quality [17]. These suggest expansion of the linearized luminance values following a simple γcurve, which has the desired effect of darkening the overall appearance of the images while increasing contrast. Figure 6.1 shows how the expansion is performed, how the final HDR luminance values relate to the input LDR luminances. Linearization of the luminance values prior to the dynamic range expansion was done with a gamma curve (γ= 2.2), following the findings by Rempel et al. [17] which note that simple gamma correction can be used for linearization instead of the inverse of the camera response without producing visible artifacts. To avoid amplifying noise, a bilateral filter [25] can be used prior to expansion [17]. Gamma expansion may potentially boost noise; however, over-exposed 14 6. Expanding Over-exposed Content images tend to be significantly less noisy than under-exposed ones. Our psychophysical tests confirmed that noise amplification did not affect the final perceived quality. Figure 6.1: Expansion following a gamma curve. The x-axis shows luminance values in low dynamic range, while the y-axis shows normalized high dynamic range luminance values. The curves show different gamma expansion functions, with the value of γranging from 1.0 (straight line) to 6.0, at intervals of 0.25. After the luminance expansion chromaticities are recovered according to the following equations [2]: Rhdr =Lhdr(Rldr Lldr )(1/s)(6.1) Ghdr =Lhdr(Gldr Lldr )(1/s)(6.2) Bhdr =Lhdr(Bldr Lldr )(1/s)(6.3) where Laccounts for luminance, and R, G, B for intensities in each of the channels. sis a factor which accounts for saturation and can be manually adjusted (it ranges between 0 and 1) to obtain the best depiction. Obviously, the problem with the proposed expansion lies in automatically obtaining an imagedependent suitable γvalue, to avoid the cumbersome manual readjustment of the display settings for each individual image to be shown. For this, we first obtain a measure of image brightness, for which we compute its key value; this key acts as an indicator of whether the scene is subjectively dark or light. Since overall brightness can be approximated with log-luminance [26, 15], we estimate the key of an image as [27]: k=log Lavg −log Lm log LM−log Lm (6.4) where log Lavg = (Px,y log(L(x, y) + δ))/n.Lmand LMare the minimum and maximum image luminances respectively, nis the number of pixels and L(x, y) is the pixel luminance. The small 15 6. Expanding Over-exposed Content offset δprevents singularities when L(x, y) = 0. We exclude 1% of the highest and lowest pixel values following the suggestion in [27], to make the estimation less sensitive to outliers. We asked users in a pilot study to manually adjust the value of γin a set of images, and fitted empirical data with a linear regression γ=a·k+b(with a= 10.44 and b=−6.282), which relates γas a function of the image key (r2= 0.82). We have used this expression in this work to compute the reverse tone mapped results in this paper. Table 6.1 shows the key and γvalues used for all the stimuli. 1234 Building 0.697 / 1.22 0.762 / 1.5 0.816 / 1.75 0.845 / 2.6 Lake 0.7714 / 1.1 0.7453 / 1.2 0.7487 / 1.5 0.7830 / 2.25 Graffiti 0.7666 / 1.2 0.8193 / 1.35 0.8738 / 1.5 0.9184 / 1.75 Strawberries 0.6696 / 1.22 0.7218 / 1.35 0.7218 / 1.55 0.8479 / 1.9 Sunset 0.7022 / 1.1 0.8103 / 1.35 0.8016 / 1.4 0.8713 / 1.75 Table 6.1: Key and γvalues for the five scenes and the four exposure levels. 6.3 Validation To provide a subjective evaluation of the performance of this strategy, we repeated Experiment One (Section 4), substituting the LDR depiction with our γ-expanded versions in order to maintain the 2×2 stimulus array. The red line in Figure 4.3 shows the results. Experiment One provides useful information about the subjective perception of image quality. However, we are also interested in evaluating our approach from an objective point of view. The problem is the fact that the intended comparison needs to be performed between an LDR and an HDR image. Recently, Aydin and colleagues [24] have presented a novel image quality metric which identifies visible distortions between two images, independently of their respective dynamic ranges. The metric uses a model of the human visual system, and classifies visible changes between a reference and a test image. The authors identify three types of structural changes: loss of visible contrast (when contrast visible in the reference image becomes invisible in the second one), amplification of invisible contrast (when invisible contrast in the reference image becomes visible in the second one), and reversal of visible contrast (when contrast polarity is reversed in the second image with respect to the reference). It is important to remember that, as Rempel and colleagues noted [17], contrast enhancement tends to increase perceived quality, and therefore is a desired outcome of the rTMO. Figure 6.2 shows the results of this metric1comparing two of the original LDR images (reference images) with the corresponding outputs using linear expansion, ldr2hdr, Banterle’s operator and our proposed γcurve. Our method reveals more detail, shows no loss of contrast and minimizes gradient reversals. Note that while our approach may fail to utilize the dynamic range to its full extent in some cases, it has the important and experimentally validated advantage of avoiding objectionable and unpredictable artifacts. 1We have used the online implementation provided by the original authors of the paper: http://drim.mpiinf.mpg.de/generator.php 16 6. Expanding Over-exposed Content LDR image Linear expansion ldr2hdr Banterle’s operator our γcurve Figure 6.2: Objective validation. Comparing the results of several rTMOs with the image quality metric from Aydin et al.[24]. The reference LDR images are Lake (top) and Building (bottom) as depicted in Figure 4.1 (which correspond to the third and second exposure levels in the series. Please refer to Appendix B for all the exposures in all the scenes). Green, blue and red identify loss of visible contrast, amplification of invisible contrast and contrast reversal respectively. Our γexpansion does not lose any contrast, while minimizing gradient reversals. More importantly, it reveals more detail in the most significant areas of the images (trees, grass, bushes and buildings in the images shown). 17 8. Conclusions and future work 8.1 Conclusions Previous works on the perception of HDR images and rTM design have assumed that the input images were, in general, correctly exposed. While these provide valuable knowledge that could guide the development of both HDR display hardware and reverse tone mapping algorithms, existing LDR legacy content actually covers a wide range of exposures, including material that suffers from bad exposure. As currently designed, existing rTMOs tend to boost over-exposed areas more than the rest of the image. The strategy works well for small areas such as light sources or highlights if the rest of the image is correctly exposed, but no performance evaluation on generally over-exposed imagery had been performed. Experiment One shows that performance of rTMOs decreases for input images containing a large number of over-exposed pixels, while they seem to perform significantly better for darker images. This suggests that for bright images the consensual approach of boosting bright areas could be improved. We have shown that a simple rTMO based on γexpansion, without the need for explicitly detecting saturated areas, outperforms existing rTMOs in these cases, and propose an empirical expression to automatically find a suitable γas a function of the image’s key, without user interaction. This rTMO has the desired properties of boosting contrast and detail in non-saturated areas of the image, visually compensating for the lack of information in the saturated ones. We have performed two validation studies, both subjective and objective. The first one has confirmed that our approach increases the perceived image quality for these kind of images. Pairwise Wilcoxon rank sum tests revealed that the differences in rating were statistically significant with respect to all other rTMOs tested. Given that it produces darker overall images with increased contrast, this result is in accordance with previous suggestions [19, 17]. The second evaluation uses a recently published image quality metric which operates with arbitrary dynamic ranges [24]. The metric concludes that our method reveals more detail in non-saturated areas, does not reduce contrast and shows less gradient reversals than the other rTMOs tested. Thus, the artists’ original intentions are better preserved. In both experiments we used typical numbers of subjects for a within-subject design in psychophysics, and the results were highly coherent across subjects. In Experiment One the reported results are statistically significant to the p < 0.05 level, meaning that the chances that the outcome of the pairwise comparisons would change after running more subjects from the same population is less than 5%. Indeed, for many of the results, the probability is many orders 24 8. Conclusions and future work of magnitude lower than this, which implies that the qualitative pattern of the results is well conserved across subjects. Likewise, data from Experiment Two exhibit a correlation coefficient of 0.9018, notably conclusive in statistical terms. Our findings seem to indicate that superior rTMOs should take into account global statistics about the image, and not just individual pixel values. We have derived a simple strategy based on the key value of the images, but more sophisticated strategies could also be devised, possibly including high-level semantics. We also ran the same expansion on the images from the dark series: as expected, we found no significant improvements over the tested rTMOs, given that our expansion is designed for bright images (Figure C.1 in Appendix C shows the results of this evaluation). The results from our second experiment confirm that spatial artifacts are more disturbing than inaccuracy in reproduced intensity levels [24]. We found a very strong correlation in the pattern of preferences when viewing images on HDR and LDR displays. This does not mean that the images looked the same, but it does suggest that the artifacts that emerge with poorly-exposed input images are spatial in nature and severe enough that HDR evaluation is not necessary: they can also be clearly seen in LDR. Our results complement those in the work by Aky¨uz et al. [8], where the authors show that, for correctly exposed imagery, a simple linear expansion works well and suggest that sophisticated treatment of LDR data may not be necessary. In fact, our work is consistent with that of Aky¨uz et al. [8] in the sense that our proposed γcurves approach linear scaling when the image is approximately correctly exposed. In a second part of the work, we have presented an interactive approach to reverse tone mapping which can be useful for a wide variety of images, especially those containing large saturated areas. The basis of our method is inspired by photographer Ansel Adams’s wellknown Zone System, which allows us to divide the luminance range of the image into zones. With the aid of this division in zones, and in an interactive process, a piece-wise linear function to expand the LDR image can be provided by the user. Furthermore, our technique includes the possibility of using higher-level information as a guide for the expansion, segmenting the image in the object of interest and the background and using different expansion functions for each. This interactive approach offers a tool to expand the dynamic range of a scene with significant yet intuitive control over the final result. Besides, being able to freely adjust the luminance ranges of the zones makes it possible to obtain very different HDR depictions of the same input image, potentially providing an artistic tool for photographers and artists in general. 8.2 Future work The conclusions drawn aim to be valuable for further development of HDR display technology, HDR imaging in general and the development of future LDR expansion algorithms in particular. However, further tests on LDR expansion are desirable. As the community investigates this issue further, this and similar studies will surely be extended and updated. Future reverse tone mapping strategies could involve the design of a contrast-based rTMO, following the findings of the work by Mantiuk et al. [44], which shows promising results in the field of contrast processing 25 8. Conclusions and future work of HDR images, working in visual response space. Similarly, reverse tone mapping for video content is a key challenge in this field. In order to develop operators that gracefully handle changes in exposure over time, it is crucial to first understand how they fail in the static case, for which we hope this work stimulates future research. Regarding future work, adding a fitting step of the piece-wise linear rTM functions proposed to smoother ones would be desirable. In the same sense, when dealing with content-aware rTM, taking care of the luminance transitions in the boundary between the objects of interest and the backgrounds would be necessary, either by somehow smoothing the binary mask or by placing constraints to the relationship between both -the object’s and the background’sexpansion functions. Besides, thorough comparison between the proposed rTM technique and existing reverse tone mapping operators by means of psychophysical experiments would certainly be interesting for the field. Additionally, salient object detection is an open field of research, and our approach would definitely benefit from future advances in this field. 26 Bibliography [1] Miguel Martin, Roland Fleming, Olga Sorkine, and Diego Gutierrez. Understanding exposure for reverse tone mapping. In Congreso Espa˜nol de Inform´atica Gr´afica, pages 189–198, 2008. [2] E. Reinhard, W. Heidrich, P. Debevec, S. Pattanaik, G. Ward, and K. Myszkowski. High Dynamic Range Imaging, Second Edition: Acquisition, Display and Image-Based Lighting. Morgan Kaufmann Publishers, 2010. [3] F. Banterle, A. Artusi, K. Debattista, and A. Chalmers. Advanced High Dynamic Range Imaging Theory and Practice. A.K. Peters Ltd., 2010. [4] Paul E. Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. Computer Graphics, 31(Annual Conference Series):369–378, 1997. [5] H. Seetzen, W. Heidrich, W. Stuerzlinger, G. Ward, L. Whitehead, M. Trentacoste, A. Ghosh, and A. Vorozcovs. High dynamic range display systems. ACM Trans. Graph., 23(3):760–768, 2004. [6] Akiko Yoshida, Rafal Mantiuk, Karol Myszkowski, and Hans-Peter Seidel. Analysis of reproducing real-world appearance on displays of varying dynamic range. Computer Graphics Forum, 25(3):415–426, 2006. [7] H. Seetzen, H. Li, L. Ye, G. Ward, L. Whitehead, and W. Heidrich. Guidelines for contrast, brightness, and amplitude resolution of displays. In Society for Information Display (SID) Digest, pages 1229–1233, 2006. [8] Ahmet Oˇguz Aky¨uz, Roland Fleming, Bernhard E. Riecke, Erik Reinhard, and Heinrich H. B¨ulthoff. Do HDR displays support LDR content?: a psychophysical evaluation. ACM Trans. Graph., 26(3):38, 2007. [9] Francesco Banterle, Patrick Ledda, Kurt Debattista, Marina Bloj, Alessandro Artusi, and Alan Chalmers. A psychophysical evaluation of inverse tone mapping techniques. Computer Graphics Forum, 28(1):13–25, 2009. [10] Scott Daly and Xiaofan Feng. Bit-depth extension using spatiotemporal microdither based on models of the equivalent input noise of the visual system. In Proc. of Color Imaging VIII: Processing, Hardcopy, and Applications, volume 5008, 455. SPIE, June 2003. 27 [11] Scott Daly and Xiaofan Feng. Decontouring: prevention and removal of false contour artifacts. In Proc. of Human Vision and Electronic Imaging IX, volume 5292, 130. SPIE, June 2004. [12] Francesco Banterle, Kurt Debattista, Alessandro Artusi, Sumanta Pattanaik, Karol Myszkowski, Patrick Ledda, Marina Bloj, and Alan Chalmers. High dynamic range imaging and LDR expansion for generating HDR content. Annex Eurographics 2009, April 2009. [13] Francesco Banterle, Patrick Ledda, Kurt Debattista, and Alan Chalmers. Inverse tone mapping. In GRAPHITE ’06, pages 349–356, New York, NY, USA, 2006. ACM. [14] Francesco Banterle, Patrick Ledda, Kurt Debattista, Alan Chalmers, and Marina Bloj. A framework for inverse tone mapping. Vis. Comput., 23(7):467–478, 2007. [15] Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. Photographic tone reproduction for digital images. ACM Trans. Graph., 21(3):267–276, 2002. [16] Francesco Banterle, Patrick Ledda, Kurt Debattista, and Alan Chalmers. Expanding low dynamic range videos for high dynamic range applications. In Proc. of the Spring Conference on Computer Graphics, New York, NY, USA, 2008. ACM. [17] Allan G. Rempel, Matthew Trentacoste, Helge Seetzen, H. David Young, Wolfgang Heidrich, Lorne Whitehead, and Greg Ward. Ldr2Hdr: on-the-fly reverse tone mapping of legacy video and photographs. ACM Trans. Graph., 26(3):39, 2007. [18] Rafael Pacheco Kovaleski and Manuel M. Oliveira. High-quality brightness enhancement functions for real-time reverse tone mapping. The Visual Computer, 25(5-7):539–547, April 2009. [19] Laurence Meylan, Scott Daly, and Sabine S¨usstrunk. The reproduction of specular highlights on high dynamic range displays. In IS&T/SID 14th Color Imaging Conference, 2006. [20] Laurence Meylan, Scott Daly, and Sabine S¨usstrunk. Tone mapping for high dynamic range displays. In Proc. IS&T/SPIE Electronic Imaging: Human Vision and Electronic Imaging XII, volume 6492, 2007. [21] Piotr Didyk, Rafal Mantiuk, Matthias Hein, and Hans-Peter Seidel. Enhancement of bright video features for HDR displays. Computer Graphics Forum, 27(4):1265–1274, 2008. [22] Lvdi Wang, Li-Yi Wei, Kun Zhou, Baining Guo, and Heung-Yeung Shum. High dynamic range image hallucination. In Eurographics Symposium on Rendering, pages 321–326, 2007. [23] G. Ward, H. Rushmeier, and C. Piatko. A visibility matching tone reproduction operator for high dynamic range scenes. IEEE Trans. on Visualization and Computer Graphics, 3(4):291–306, 1997. [24] Tun¸c Ozan Aydin, Rafal Mantiuk, Karol Myszkowski, and Hans-Peter Seidel. Dynamic range independent image quality assessment. ACM Trans. Graph, 27(3):69, 2008. [25] C. Tomasi and R. Manduchi. Bilateral filtering for gray and color images. In ICCV, pages 839–846, 1998. 28 [26] Jack Tumblin and Holly E. Rushmeier. Tone reproduction for realistic images. IEEE Computer Graphics and Applications, 13(6):42–48, 1993. [27] Ahmet Oˇguz Aky¨uz and Erik Reinhard. Color appearance in high dynamic range imaging. SPIE Journal of Electronic Imaging, 15(3), 2006. [28] Ansel Adams. The Print. The Ansel Adams Photography series. Little, Brown and Company, 1983. [29] Peter J. Burt and Edward H. Adelson. The Laplacian pyramid as a compact image code. IEEE Transactions on Communications, 31(4):532–540, 1983. [30] Anat Levin, Dani Lischinski, and Yair Weiss. Colorization using optimization. ACM Trans. Graph., 23(3):689–694, August 2004. [31] Soonmin Bae, Sylvain Paris, and Fr´edo Durand. Two-scale tone management for photographic look. ACM Trans. Graph., 25(3):637–645, 2006. [32] Dani Lischinski, Zeev Farbman, Matt Uyttendaele, and Richard Szeliski. Interactive local adjustment of tonal values. ACM Trans. Graph., 25(3):646–653, 2006. [33] Zeev Farbman, Raanan Fattal, Dani Lischinski, and Richard Szeliski. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph., 27(3):67, 2008. [34] Chris Johnson. The practical zone system. Focal Press, 3rd edition, 1999. [35] Carson Graves. The zone system for 35mm photographers. Focal Press, second edition, 1997. [36] Norman Koren. A simplified zone system. www.normankoren.com/zonesystem.html. [37] L. Itty, C. Koch, and E. Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell., 20(11):1254–1259, 1998. [38] T. Liu, J. Sun, N.-N. Zheng, X. Tang, and H.-Y. Shum. Learning to detect a salient object. In IEEE Computer Vision and Pattern Recognition, 2007. [39] Yu Fu, Jian Cheng, Zhenglong Li, and Hanqing Lu. Saliency cuts: An automatic approach to object segmentation. In International Conference on Pattern Recognition (ICPR), 2008. [40] Y. Y. Boykov and M. P. Jolly. Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. In ICCV, pages I: 105–112, 2001. [41] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. ”grabcut”: interactive foreground extraction using iterated graph cuts. ACM Trans. Graph, 23(3):309–314, 2004. [42] X. D. Hou and L. Q. Zhang. Saliency detection: A spectral residual approach. In Computer Vision and Pattern Recognition, 2007. [43] Belen Masia, Sandra Agustin, Roland W. Fleming, Olga Sorkine, and Diego Gutierrez. Evaluation of reverse tone mapping through varying exposure conditions. ACM Transactions on Graphics (Proc. of SIGGRAPH Asia), 28(5):160:1–160:8, 2009. 29 [44] Rafal Mantiuk, Karol Myszkowski, and Hans-Peter Seidel. A perceptual framework for contrast processing of high dynamic range images. ACM Trans. Graph., 3(3):286–308, 2006. [45] Y. Y. Boykov and V. Kolmogorov. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(9):1124–1137, 2004. 30