Consensual Pupillometry for robotic emotional recognition

Recent Innovations in Mechatronics (RIiM) Vol. 2. (2015). No. 1-2. DOI: 10.17667/riim.2015.1-2/4. 1 Consensual Pupillometry for robotic emotional recognition Dr. Fekete Róbert Tamás PhD Budapest University of Technology and Economics, Faculty of mechanical engineering Department of Mechatronics, Optics and Information Engineering Budapest, Hungary [email protected]me.hu Raj Levente MSc Budapest University of Technology and Economics, Faculty of mechanical engineering Department of Mechatronics, Optics and Information Engineering Budapest, Hungary [email protected]me.hu Tamas Neumer BSc Budapest University of Technology and Economics, Faculty of mechanical engineering Department of Mechatronics, Optics and Information Engineering Budapest, Hungary [email protected] Abstract – The main contribution of this research is to develop of a new measuring device for measuring directand consensual light reflex of human pupil. In this case the pupillary light reflex means changing of diameter of pupil due to constant light stimulus as a function of time. Researches certify that the pupil diameter and its change reflects the person’s mental or physical state. There are several “eye tracking” devices available on the market, however, none of these are able to perform a proper consensual measurement. The basic principle of the consensual pupil measurement is that while only one of the eyes is stimulated, reactions of both eyes are being recorded and analyzed. The aim of this project was to develop a software, applicable to an existing prototype device in order to carry out “consensual pupillary reflex” measurements. Keywords – human eye, pupil, consensual pupillary light reflex, image processing, eye tracking, robotics, emotional recognition I. INTRODUCTION There are several researches discussing the pupillary light reflex. Since the human pupil is controlled by the automatic nervous system, it is adequate to objectively measure different physical and mental states of the person. [1][3] In 1965 Eckhard H. Hess investigated correlation between the test persons’ mental state and their pupil size. Pictures of different topics were shown to the test persons while their pupil reaction was recorded and measured. Hess found that men and women showed different reactions to a given picture. For example, when the topic of the picture was a baby, the reactions of women showed significantly greater interest than that of men’s. Hess concluded that there was a correlation between people’s emotional state and their pupil size in fact. [13] Correlation can be found not only between mental states and the pupil size but between physical variables and the pupil size, too. In 2003 Lucas RJ examined if there was correlation between mice’s the genome and their pupillary reaction to light. [10] As discussed above, many physical and mental variables can be bound to the pupillary reflex, however, most of these were determined as a result of direct pupillary measurement. Even the most outstanding researchers in this area use the Tobii eye tracking device for their work, a device that only allows direct measurement and eye-tracking. In the few cases when consensual measurement was carried out, the device was non-professional. Thus we can conclude that there seems to be no professional device on the market performing consensual measurement right now. Since found no such on the market, the final goal would be to develop a low-cost, but professional device that performs both direct and consensual measurement. [1] Figure 1 – Images from our instruments Recent Innovations in Mechatronics (RIiM) Vol. 2. (2015). No. 1-2. DOI: 10.17667/riim.2015.1-2/4. 2 II. HARDWARE INTEFACE Two Logitech 200 web camera chips record the images that will be analyzed as the last step of the measurement. An “NI USB 6008” input/output device, manufactured by National Instruments, controls a set of infra and normal LEDs. The normal LEDs are turned on and off during the measurement, changing the amount of light that reaches the test person’s eyes, thus forcing the pupils to dilate or contract. The infrared LEDs enable to take pictures even in dark environment, while the normal LEDs are turned off. Figure 2 – Consensual pupillary measuring instrument hardware III. SOFTWARE INTEFACE The software was written in C# language, the Microsoft .Net environment was used as development platform. Although there are several frameworks for the image processing and I/O handling external libraries such as the Aforge.net, Microsoft DirectShow, and National Instrument DAQmx were used in our project in order to lower the costs. [3] The user interface provides options to set camera parameters and to customize the measurement itself. In a typical measurement, first the left, then the right eye is being stimulated. The cameras take images and later the software calculates the pupil di-ammeter on these images. The results are stored in an Excel file, enabling further processing and analysis. The software determines the pupil diameter based on video streams captured during the measurement. To make it more efficient, it doesn’t calculate the pupil size in real time, but it saves the frames at a user specified location. Figure 3 - Start/Stop cameras, Set Image Location Throughout the measurement the pupil size is measured first in darkness, while it is stimulated by the light emitted by LEDs. In order to get results which are comparable with each other, the algorithm calculating the pupil diameter has to be the same in both light and dark circumstances. Usually web cameras have a so called “Auto Balance” function that sets the exposure property of the camera in order to get an image with the desired luminance. However, this functionality results in drastic frame per second (FPS) drop. To avoid that, the “Auto Balance” function of the cameras had to be turned off and the parameter adjustment was achieved manually by applying different gain parameters for the light and dark environments. After having the device placed in measuring positions, the program calculates the gain values that will be used to get images of the same luminance throughout the whole measurement. Figure 4 - Auto-Adjust Properties To fully understand the Auto-Adjust functions, first the image processing algorithm is discussed. The function takes the image of the eye and applies a threshold filter of a certain value so that the irrelevant details of the image are filtered out. Then it creates the negative of the image and by a “filling” algorithm it fills up the spots representing the mirrored LEDs. As the last step, the shape recognition algorithm is called to detect the circle, which is the outer perimeter of the pupil. Recent Innovations in Mechatronics (RIiM) Vol. 2. (2015). No. 1-2. DOI: 10.17667/riim.2015.1-2/4. 3 Figure 5 - Steps detection algorithm In conclusion, the “Auto-Adjust” function has got three objectives: find a gain value that can be applied in darkness, find a threshold value that returns the pupil, and find a gain value that makes images of the same luminance, as it was in darkness. In a measurement the program takes a gain value, and applies it to the camera. The pupil diameter is calculated with different threshold values on the picture shot. When the pupil diameter has been calculated on a given image with the whole set of threshold values, the program takes the next gain value, applies it to the camera and repeats the process again. In the end the program ends up with a table like the following: Figure 6 - Table of the pupil diameters at different gain and threshold pairs (The columns are the different gain values, while the rows represent the different threshold values) The algorithm processes each column and determines the gain value the pupil was detected with the most time. (Column marked with yellow on Figure 6). The middle value of the column is taken and the corresponding threshold value will be used later. (Red cell on Figure 6) While looking for the gain value in light, the program first takes an image in dark and calculates its luminance – using later as a reference. Then it shoots pictures with different gain values and compares the luminance of these images with the luminance of the reference. The gain value to the smallest deviation corresponding will be applied in light. The different LED events are set in the fourth step. The operator defines as an event sequence which LEDs shall be switched on and at which time. It is also possible to save, and load a given event sequence. Figure 7 - Set events The measurement starts only after the amount of time specified in the numeric up-down has elapsed, so that the operator has time to place on the device. Figure 8 - Start Capturing To process the images made by a measurement run the image-containing folder is opened. If the “Measurement Correction” is checked an algorithm corrects the measurement errors. Figure 9 - Analyze images The program loops through the images, calculates the pupil diameter on the image and writes the value in an excel table. When all images have been processed, the correction algorithm processes the table, and corrects the errors: it checks if the first and last pupil diameters are 0. If so, it replaces them with the first or the last non 0 values. (79.5 and 79 on Figure 10) Then it looks for sequences, when the pupil couldn’t be detected. (Series of 0s.) If found, then the program applies the linear regression model and replaces the missing values. Recent Innovations in Mechatronics (RIiM) Vol. 2. (2015). No. 1-2. DOI: 10.17667/riim.2015.1-2/4. 4 Figure 10 - Example of the correction algorithm When the camera parameters found by the program were correct, there is no significant luminance difference between pictures taken in darkness, or LED light. Figure 11 - Images of same luminance during a LED flash IV. MEASUREMENT RESULTS As a result, the program calculates the pupil diameters on the images, and fills and excel table with the values. Figure 12 - Measurement results for the left eye Figure 13 - Measurement results for the right eye As seen on Figure 12 and Figure 13 the first (orange) and the second (gray) measurements are located farther from the other 3. Interestingly, these measurements were made with only 20 FPS, while the other three with 30 FPS. The underlying reason for this might be that some background processes of the computer were using much resources and it was not able to process 30 but only 20 frames per second. If the first two measurements are ignored, the results are more convincing. Figure 14 - Measurement results for the left eye Figure 15 - Measurement results for the right eye Recent Innovations in Mechatronics (RIiM) Vol. 2. (2015). No. 1-2. DOI: 10.17667/riim.2015.1-2/4. 5 At another test, the results of 10 measurements were taken into account. Here the results have higher standard deviation. This may be a result of the fact that all the 10 measurements were taken after each other that might have resulted in mental or physical differentiations. Figure 16 - Measurement results for the left eye Figure 17 - Measurement results for the right eye There are several ways to optimize the device in order to get better results. On the hardware side, the two “commercial” web cameras could be replaced by more professional ones. These cameras could deliver pictures of higher resolution and Frame per Second rate. As seen on Figure 12, the reflection of the LEDs are present on the pupil’s image as tiny white dots. However when the pupil contracts, these white dots often intersect with the edge of the pupil, thus making it hard for the shape recognition software to find a correct circle. In order to avoid that the LED lights could be repositioned, or the whole concept could be replaced by the “Pupil Centre Corneal Reflection”, since here the pupil is shiny and it’s easier to detect on the picture. On the software side there could have been used other object recognition algorithms as well such as OpenCV or “C”. A better correction algorithm could also enhance and smoothen the results. [6] REFERENCES [1] Ana Belén Roig, Julián Espinosa, Jorge Pérez and David Mas (2014): High-speed video-oculography applied to assess pupil light reflex. VII European / I World Meeting in Visual and Physiological Optics [2] Ana Laura A. Moura (2013): The Pupil Light Reflex in Leber’s Hereditary Optic Neuropathy, Invest Ophthalmol Vis Sci. 2013 Jul 2; 54(7):4471-7. [3] Jianxin Wu, Nini Liu, Christopher Geyer, James M. Rehg (2013): AReal-time Object Detection Framework – IEEE Transactions on Image Pro-cessing, doi: 10.1109/TIP.2013.2270111 [4] Malin Forne (2012): Physiology as a tool for UX and Usability testing, Stockholm, Master of Science Thesis [5] Somlai Juidt, Kovács Tibor (2012): Neuro-ophthalmologia, NOSZA Alapítvány, Budapest, 2012 [6] Malin Forne (2012): Physiology as a tool for UX and Usability testing, Stockholm, 2012 [7] Márkus Attila (2012): Neurológia. Akadémiai Kiadó [8] Bharti Vidyapeet, Bhatati Vidyapeeth (2011): Advances in image processing for detection of plant diseases - Journal of Advanced Bioinformat-ics Applications and Research, ISSN 0976-2604, pp 135141 [9] Ying Gao, Armando Barreto, Malek Adjouadi (2009): Detection of Sympa-thetic Activation through Measurement and Adaptive Processing of the Pupil Diameter for Affective Assessment of Computer Users - American Journal of Biomedical Sciences, ISSN: 1937-9080 doi: 10.5099/aj090400283 [10] Lu Yin, Robert G. Smith, Peter Sterling, and David H. Brainard (2009): Physiology and Morphology of Color-Opponent Ganglion Cells in a Reti-na Expressing a Dual Gradient of S and M Opsins Responses Follow a Du-al Gradient in Cone Opsin Expression - The Journal of Neuroscience doi: 10.1523/JNEUROSCI.5471-08.2009 [11] Lucas RJ, Hattar S, Takao M, Berson DM, Foster RG, Yau KW (2003): Di-minished pupillary light reflex at high irradiances in melanopsinknockout mice – Science, PMID: 12522249 [12] Csépe Valéria, Általános pszichológia I., OSIRIS Kiadó, Budapest, 2007 [13] Lu Yin, Robert G Smith, Peter Sterling, David H. Brainard (2006): Chro-matic Properties of Horizontal and Ganglion Cell Responses Follow a Du-al Gradient in Cone Opsin Expression – The Journalof Neuroscience, doi: 10.1523/JNEUROSCI.1071-06.2006 [14] Eckhard H. Hess (1965): Attitude and Pupil Size, Scientific American no 212, 1965, pp. 46-54.