Drowsiness detection by thoracic effort signal snalysis with professional drivers in real environments

Drowsiness Detection by Thoracic Effort Signal Analysis 1 Drowsiness Detection by Thoracic Effort Signal Analysis with Professional Drivers in Real Environments. Noelia Rodríguez Ibáñez a , Miguel Ángel García González b , Juan Ramos Castro b , Mireya Fernández-Chimeno b a Research Department, FicoMirrors Pol. Ind. Can Magarola, C/ Rec de Dalt,08100 Mollet del Vallès (Barcelona), SPAIN Tel.: +34 93 561 00 00; E-mail: noeli[email protected] b Grup d’instrumentació Electrònica I Biomèdica. Departament d’Enginyeria Electrònica Universitat Politècnica de Catalunya Abstract— Objective: The aim of this work is to develop a new index to assess the alertness state of drivers based on the respiratory dynamics derived from an inductive band. Background: Detection of drowsiness while driving is a leading objective in advanced driver assistance systems. A biomedical variable like thoracic effort, which is related to autonomic nervous system, provides direct information of the driver physiological state, instead of indirect indicia of the participant's behavior. Therefore, they may be especially useful to collect detailed information of the drowsiness state and anticipate risky situations while driving. Method: The experiment described in this paper allows us to study how drivers react in strenuous conditions, in which they tend to get fatigued and drowsy, but struggle against falling asleep. The respiration data used in this study was recordered by doing 72 hours of simulation driving test and 100 hours of real vehicle tests in real environment. Results: The results demonstrate the viability of drowsiness detection in real vehicle using thoracic effort signal. The proposed method has a sensitivity of 93.7% and specificity of 86.3% in detecting full awake drivers while it has a sensitivity of 83.1% and specificity of 95.3% in detecting drowsy drivers. Conclusions: The proposed index may be promising to assess the alertness state of real drivers. Applications: The potential applications of the algorithm are to detect drowsiness states while driving and give an alarm in commercial fleets or professional drivers to assure the integrity of the driver. Key Words— Alertness state, Advanced Driver Assistance Systems, Fatigue, Fleet, and Inattention. I. I NTRODUCTION RIVER drowsiness is one of the main causes of vehicle accidents. A recent study showed that 20% of crashes and 12% of near-crashes were caused by drowsy drivers (NHTSA VSR technical report, 2009). The morbidity and mortality associated with drowsy-driving crashes are high, perhaps because of the higher speeds involved combined with delayed reaction time (Faber, 2004). Driver behavior monitoring, and the reliable detection of drowsiness and fatigue is one of the leading objectives in the development of new Advanced Driver Assistance Systems (ADAS). Nowadays, most systems of drowsiness detection in the market are based on measurements of driving performance, which evaluate variations of the control of velocity, steering wheel D Drowsiness Detection by Thoracic Effort Signal Analysis 2 angle and other variables recorded by the CAN (Controlled Area Network) bus. Some research groups have also advanced methods based on the movement of eyes and head (Wright, Stone, Horberry & Reed, 2007). There are also approaches based on biomedical signals, like cerebral, muscular and cardiovascular activity, although most of them are yet far from being effectively introduced in the market. Biomedical variables related to the autonomic nervous system could provide direct information of the driver physiological state. Therefore, they may be especially useful to collect detailed information of the drowsiness cycle and anticipate risky situations while driving (Reyner & Horne, 1998). The aim of this work is to detect drowsiness in drivers by the analysis of the variability of the respiratory signal measured with a thoracic band in real driving conditions. II. M ATERIALS AND M ETHODS A. Tests overview This paper presents two different tests, one in a driving simulator involving 36 subjects and the other one in real vehicle involving 13 subjects, with different objectives: The main objectives of the simulator tests are to gather a database of biomedical signals, plus driving performance parameters from drivers in both wakeful and drowsy conditions that may be successfully used to study the measurable changes related to inattention and drowsiness; and to find patterns in those signals that allow distinguishing the periods of drowsiness, contrasted with medical judgments based on Electroencephalogram (EEG) and current methods used to detect the fall of attention as external observer and percentage of eye closure (PERCLOS) assessment. The subjects in the driving simulator test were 17 male and 19 female in different conditions: sleep deprived (6 male and 8 female), fatigued (4 male and 5 female) and fully awake (7 male and 6 female). 72 hours of driving simulation test , have been recorded with the objective to adjust measurement procedures in the real vehicle tests. The objective of the real vehicle tests is to check the biomedical parameters that we choose as indicative of somnolence, due to previous analysis with simulator tests signals, could be useful to detect drowsiness in real driver performance using a fleet of professional drivers. The subjects in the real vehicle test were professional drivers (11 male, 2 female) with ages between 26 and 56 years (35.5 ± 8.9 years (mean ± standard deviation)) and no clinical conditions. The tests were carried out in two different routes: highway and mountain . . The objective of this test is to analyze the driver behavior with different concentration levels while driving using the variables selected as significant of the simulator tests . One hundred hours of real vehicle tests in real environment have been recorded. B. Simulator test setup The number of participants was limited by the strict conditions of the test, which required different degrees of sleep-deprivation, and to pass a test to exclude people with physical problems or propensity to suffer simulator-sickness. A group of 36 volunteers between 25 and 45 years was selected to participate in this experiment: 18 of them performed the test in the afternoon, having slept normally the night before, and the other 18 performed it in the early morning deprived of sleep, after their workday and having remained awake during the 24 previous hours. All subjects signed a consent form, were informed of the purpose of the experiment, and were paid for their participation. Drowsiness Detection by Thoracic Effort Signal Analysis 3 The simulator consisted in a bench with a driver seat, a seatbelt to fasten the subject, and a driving simulator software (figure 1). A biomedical monitor (Bitmed eXim Pro, BitMed) was used to record main biomedical signals such as electroencephalogram, electrooculogram, electrocardiogram and thoracic effort. The Electroencephalogram (EEG) signal was recorded with a full 10-20 system device (figure 2) to record the alpha and vertex waves that are indicative of the onset of somnolence. The Electrooculogram (EOG) signal was measured with four spoon electrodes: two were located in the outer cantus of each eye in the case of the horizontal EOG setup, and two more electrodes located in the upper part and in the lower part of the right eye (figure 3) Leads I and II of the Electrocardiogram (ECG) were measured with one use stick electrodes attached directly to the chest skin as we show in figure 4. The respiratory signal was measured in all cases using an inductive band located at the middle trunk above the diaphragm and the signal was low pass filtered at 5 Hz. Video signal was recorded with a WebCam to generate the external observer variable and with an infrared high resolution camera to analyze the recording with PERCLOS algorithms. The external observer signal that is a subjective evaluation of the state of the driver is obtained by at least four medical experts that evaluates minute per minute variables like shape of the face, eye blink, driver movements, etc All signals were sampled at 250 Hz and synchronized by a trigger button on the steering wheel or the internal clock of the computers. All the computers of the instrumentation setup were placed behind the simulator bench, which was surrounded by panels to create a closed environment. To further prevent distraction and the “white coat effect”, the experiment was monitored from a control room out of the sight, so that the subject could not be aware of being accompanied or observed. The temperature at 24ºC, the faint light and the monotonous sound of an engine causes an increment of fatigue and therefore somnolence. The measurement protocol was: i) 5 minutes of measurement to calibrate the devices. ii) 5 minutes of resting time to recording the basal activity of the subject. iii) 1 hour 45 minutes of data measurement while driving along a highway with low traffic and smooth curves in a night simulation. iv) 15 minutes of relax with the lights turn off and silence. C. Real vehicle tests The objective of the real vehicle tests (Lee, Choi, Kim, Kim, Baek, Ryu, Sohn & Park, 2007) is to prove that the behavioral and biomedical parameters that we choose as indicative of somnolence in simulator tests are useful to detect drowsiness in real driver performance using a fleet of professional drivers. The participants in the test were professional drivers (11 male, 2 female) with ages between 26 and 56 years (35.5 ± 8.9 years (mean ± standard deviation)) and no clinical conditions. The tests were carried out in two different routes: highway and mountain (Åkerstedt, Peters, Anund & Kecklund, 2005), to analyze the driver behavior with different concentration levels (Oron-Gilad, Ronen & Shinar, 2008). To perform these tests, a real vehicle was equipped with a WebCam a Bitmed eXim Pro biomedical monitor (same as in simulator tests) to record EEG, EOG and thoracic effort signals and an infrared high resolution camera. The main differences between this setup and the simulator setup are that the sampling frequency was 100 Hz and the EEG signal was measured with a composition of four spoon electrodes located on the vertex zone of the cranium and Drowsiness Detection by Thoracic Effort Signal Analysis 4 attached to the head surface with colloid. Once the subjects are seated and connected to the acquisition systems, they were asked to drive for around 8 hours on a real highway or a mountain route stopping during at least 10 minutes every two hours of continuous driving or every time they felt drowsy (Ting, Hwang, Doongh & Jeng, 2008). Figure 6 shows an image of the measurement environment. D. Drowsiness state classification In order to classify the state of the driver, a reference signal was needed. The Gold standard (GS) combines the results of the analysis of EEG, external observer evaluation of video recording and PERCLOS to assign states of ‘awake’, ‘fatigue’ and ‘drowsiness’ and updates every minute. EEG and PERCLOS thresholds used to evaluate the awake, fatigue or drowsy sates were defined according to the personal basal state. The GS defined three phases: phase 0 or attentive corresponds to a fully awake driver, phase 1 or fatigue to a fatigued driver and phase 2 or drowsy to a drowsy driver (De Rosario, Solaz, Rodríguez & Bergasa, 2010). The EEG parameter used in the analysis was the ratio of vertex waves per minute (Yeo, Li, Shen & Wilder-Smith, 2009). This measurement was adjusted by the judgement of a team of medical experts (Clinica Dexeus, Barcelona, Spain), who visually interpreted the set of EEG + EOG signals, to discard ‘false’ waves caused by eye movements or other artefacts. PERCLOS was calculated with a monocular computer vision system. This system was tested in driving simulators and demo-cars driving in real conditions, and it was found to be robust to head turns, partial occlusions and illumination changes in both day and night scenarios. The PERCLOS measure indicated accumulative eye closure duration over time, excluding the time spent on normal eye blinks. The degree of eye opening was characterised by pupil shape. As eyes closed, pupils became occluded by the eyelids and their shapes became more elliptical. Therefore we could use the ratio of the pupil ellipse axes to characterise the degree of eye opening. We considered that eye closure occurred when that ratio was over 80% of its nominal size. Then, the measurement of eye closure duration was calculated as the time that the eyes remained in that state (De Rosario, Solaz, Rodríguez & Bergasa, 2010). The behaviour of users was based on the subjective assessment of their body and face movements. Body and face movements were annotated by three external observers that analyzed the video and classified every minute of the test. The Gold Standar (GS) was classified as ‘attentive’, ‘fatigued’ or ‘drowsy’, according to the criteria given in Table I. The levels of EEG and PERCLOS associated with changes from ‘attentive’ to ‘fatigued’, and from ‘fatigued’ to ‘drowsy’, were determined in each test, and a confidence interval based on the whole set of data was defined for these thresholds, as represented in Table I. These thresholds were used to define the GS, as a combination of the EEG, external observer and PERCLOS variables. The algorithm that defined this control signal considered that a high power of EEG vertex waves (and a few alpha waves) was a reliable indicator of drowsiness, but that incipient fatigue could appear before this pattern occurred; besides, frequent blinks and high eye closure appeared early, although eyelid movement patterns vary a lot. Both signals were combined with the external observer evaluation of the state. E. Thoracic effort signal Previous to the development of the algorithm several patterns of every state of atection (awake, fatigue or drowsy) were characterized visually with the simulator data. Drowsiness Detection by Thoracic Effort Signal Analysis 5 As can be seen in the figure 7, during the simulator test, the thoracic effort signal becomes irregular due to the transition of awake state to fatigue and more sharply between awake state and drowsiness. The first line of the figure shows a very regular thoracic effort signal in amplitude and in frequency. In spite of the presence of a yawn in the middel of the line, this part of the data is representative of an awake or basal sate (figure 8). When the presence of yawns and sights becomes higher (line 2 of the data) that means that the subject is going in to a fatigue state (figure 9). Also the amplitude and the frequency of the signal become lower than the basal state (first line of the data). The next line (line 3) shows the pass between fatigue and drowiness. As we can see in the figure 7 several bursts of irregular signal appears indicating that the subject is entering in a drowsiness state. The line 4 correspond to a drowsiness state where the subject fall sleep during the simulator test (Figure 10) To automate the analysis of this variability of the thoracic effort signal the Thoracic Effort Derived Drowsiness index (TEDD) was developed. F. Thoracic effort derived drowsiness index In order to classify the state of the driver from the thoracic band, we have used a new index based in the comparison between the characteristics of the respiratory signal when a subject is awake (it is supposed that in this state the respiration is stable and more or less periodic), with the characteristics of the signal along the driving period. The Thoracic Effort Derived Drowsiness index (TEDD) is computed as follows: The thoracic effort signal (Resp) is first filtered using a second order Butterworth lowpass filter with a cutoff frequency of 1 Hz. The first three minutes of acquisition are devoted to the search of a 40 second reference window where the respiratory signal is maximally stable. In order to identify so, a sliding window with a delay of one sample is displaced along the first three minutes and the following estimator is computed for each sample: ( ) ( ) 2 1 2 1 1 ( ) 1 ( ) n M i Wi M n i Resp i n RCX M M Resp i = = =       = −       ∑ ∑∑ (1) M is the number of samples inside the window. The estimator quantifies the stability of the variance (Inclan & Tiao, 1994) so the maximum of stability corresponds to the window that has the minimum value. Once the interval with maximum stability is set, the 70% percentile of Resp inside the interval (Th) is obtained. The respiratory period is determined breath by breath by identifying the crossings of Th with positive slope across the whole recording, and computing the time interval between consecutive crossings. The BB time series is defined by these time intervals. Next, the smoothed BB time series (sBB) is obtained by applying a moving average filter of 4 respiratory periods. The breathing variability signal is obtained as ( ) ( ) ( 1) VBB n sBB n sBB n = − − (2) This variability is smoothed again with another moving average filter of 10 respiratory periods obtaining the sVBB time series and finally, the TEDD index for the respiratory period n is defined as: Drowsiness Detection by Thoracic Effort Signal Analysis 6 ( ) ( ) ref sVBB n TEDD n VBB = (3) being VBBref the mean value of VBB inside the reference window. TEDD is, then, an index of the stability of the breathing frequency. To classify the state of the driver and compare with the GS, two empirical thresholds have been obtained from the studies in driving simulators (De Rosario, Solaz, Rodríguez & Bergasa, 2010). The mean of TEDD (mTEDD) has been computed for each minute of the recording. For each minute, if mTEDD • is below 3, a phase 0 is decided. • is between 3 and 6, a phase 1 is decided. • is above 6, a phase 2 is decided. G. Statistical analysis For each minute of recording, the phases obtained by the thoracic effort signal and the GS were compared in order to estimate the sensitivity and specificity of TEDD. Table II shows a symbolic assignment to interpret equations (4) to (9). In the equations, BC should be interpreted as the number of times in all recordings that TEDD classified the minute as a phase 1 while GS classified it as phase 2. According to table II, sensitivity (Sens) and specificity (Spec) for each phase is defined as: 0 AA Sens AA BA CA =+ + (4) 0 BB BC CB CC Spec BB BC CB CC AB AC + + + =+ + + + + (5) 1 BB Sens BB AB CB =+ + (6) 1 AA CC AC CA Spec AA CC AC CA BA BC + + + =+ + + + + (7) 2 CC Sens CC AC BC =+ + (8) 2 AA BB AB BA Spec AA BB AB BA CA CB + + + =+ + + + + (9) III. R ESULTS Figure 11 shows an example of the performance of TEDD for a subject that it’s always alert and a subject with occasional drowsiness. The driver’s stops along his/her route are easily recognizable because the thoracic effort signal is zero (denoting disconnection of the inductive band). The drowsy subject is specially fatigued at the start of the recording and improves the performance after the first stop. Drowsiness Detection by Thoracic Effort Signal Analysis 7 The second driver performed pretty well during the whole recording being always alert. Table III shows the results of sensitivity and specificity for all subjects. IV. D ISCUSSION AND CONCLUSIONS The results confirmed the viability of drowsiness detection while driving using the thoracic effort signal. The Phase 1 state shows a lower sensitivity because it is a transition zone. Some misdetections of the algorithm may be due to the inter-subject variability of the thoracic effort signal. In order to check so, future work will test if the thresholds on mTEDD can be adapted with the body mass index of the subject under measurement. TEDD avoids many pitfalls of PERCLOS that is highly influenced by sunlight or the wearing of sunglasses. The recordings in real vehicles analyzed in this paper did not show any adverse effect in the results due to vibrations and movement due to driving. The thoracic band is a robust sensor that is sensitive to changes in the entire thoracic contour, while other sensors that measure displacement of the thoracic wall (ie. radar) will be more sensitive to vibrations and body motion. Further work will focus on unobtrusive measurement of the respiratory signal using bioimpedance techniques in order to avoid the use of the inductive band. The results show that TEDD may be a promising index to assess the alertness state of real drivers. V. AKNOWLEDGMENTS Manuscript received March 28, 2011. This work was supported by CIDEM project RD09-1- 0030 and cofinanced by FEDER and FICOMIRRORS R EFERENCES Åkerstedt, T, Peters, B, Anund, A, Kecklund, G, “Impaired alertness and performance driving home from the night shift: a driving simulator study”, Journal of Sleep Research vol. 13, no. 1, pp. 17–20. De Rosario, H; Solaz, J.S.; Rodríguez, N.; and Bergasa, L.M. (2010). Controlled inducement and measurement of drowsiness in a driving simulator IET Intell. Transp. Syst. 4(4): 280-288. DOI:10.1049/iet-its.2009.0110 Faber, J, “Detection of different levels of vigilance by EEG pseudospectra”, Neural Network World vol. 14, no. 3-4, 2004, pp. 285–290 Inclan C, Tiao GC. Use of cumulative sums of squares for retrospective detection of changes of variance. Journal of the American Statistical Association, vol. 89, 1994, pp 913-923 (ISSN: 0162-1459 Lee, HB, Choi, JM, Kim, JS, Kim, YS, Baek, HJ, Ryu, MS, Sohn, RY, Park, KS, “Nonintrusive Biosignal Measurement System in a Vehicle”, Proceedings of the 29th Annual International Conference of the IEEE EMBS Cité Internationale, 23-26 August 2007, Lyon, France NHTSA VSR "Chapter 3: objective 2, what are the environmental conditions associated with driver choice of engagement in secondary tasks or driving while drowsy? What are the relative risks of a crash or near-crash when engaging in driving inattention while encountering these environmental conditions? Oron-Gilad, T, Ronen, A and Shinar, D, “Alertness maintaining tasks (AMTs) while driving”, Accident Analysis & Prevention vol. 40, no. 3, May 2008, pp. 851-860. Drowsiness Detection by Thoracic Effort Signal Analysis 8 Reyner, LA, Horne, JA, “Falling asleep whilst driving: are drivers aware of prior sleepiness?”, International Journal of Legal Medicine vol. 111, 1998, pp. 120–123. Ting, PH, Hwang, JR, Doongh, JL, Jeng, MC, “Driver fatigue and highway driving: A simulator study”, Physiology & Behavior vol. 94, no. 3, 2008, pp. 448–453. Wright, NA, Stone, BM, Horberry, TJ, Reed, N, “A review of in-vehicle sleepiness detection devices”, TRL Limited, Published Project Report PPR157, 2007 Yeo, MVM, Li, X, Shen, K, Wilder-Smith, EPV, “Can SVM be used for automatic EEG detection of drowsiness during car driving?”, Safety Science vol. 47, no. 1, 2009, pp. 115–124. Drowsiness Detection by Thoracic Effort Signal Analysis 9 VI. T ABLE CAPTIONS Table I. Classification criteria to obtain gold standard signal Table II. Symbolic assignment for sensitivity and specificity definition Table III. Sensitivity and specificity of proposed index while real driving Drowsiness Detection by Thoracic Effort Signal Analysis 16 Figure 3 Drowsiness Detection by Thoracic Effort Signal Analysis 17 Figure 4 Drowsiness Detection by Thoracic Effort Signal Analysis 18 Figure 5 Drowsiness Detection by Thoracic Effort Signal Analysis 19 F igure 6 Drowsiness Detection by Thoracic Effort Signal Analysis 20 Figure 7 Drowsiness Detection by Thoracic Effort Signal Analysis 21 Figure 8 Drowsiness Detection by Thoracic Effort Signal Analysis 22 Figure 9 Drowsiness Detection by Thoracic Effort Signal Analysis 23 Figure 10 Drowsiness Detection by Thoracic Effort Signal Analysis 24 0 50 100 150 200 250 300 350 400 - 5000 0 5000 Time (min) Thoracic effort Subject with drowsiness and fatigue 0 50 100 150 200 250 300 350 400 0 5 10 15 20 Time (min) TEDD 0 50 100 150 200 250 300 350 400 - 5000 0 5000 Time (min) Thoracic effort Always Phase 0 0 50 100 150 200 250 300 350 400 0 5 10 15 20 Time (min) TEDD Figure. 11 Drowsiness Detection by Thoracic Effort Signal Analysis 25 VIII. BIOGRAPHIES Noelia Rodriguez-Ibañez obtained her degree in Biology in 2006 by the Universidad de Barcelona (UB), Spain, and Master in Biomedical Engineering in 2007 by the Universidad Politècnica de Cataluña (UPC), Spain, and a Master in Nanosciences and Nanotechnology in the Universidad de Barcelona (UB), Spain, (2010). Currently she works in the R+D department of FicoMirrors giving biomedical support to different projects related with detection of drowsiness with biomedical signals and test design. She is currently working in a PhD in Biomedical Engineering on the field of the detection of drowsiness with thoracic effort signal. Miguel A. García-González received the Ingeniero de Telecomunicaci´on degree, in 1993, and the Doctor Ingeniero Electrónico degree, in 1998, both from the Universitat Politècnica de Catalunya, Barcelona, Spain. He is currently an Assistant Professor of Electronic Engineering, Universitat Polit`ecnica de Catalunya. He teaches courses in several areas of medical and electronic instrumentation. He is engaged in research on instrumentation methods and ECG, arterial blood pressure, and EMG measurements. His current research interests include time series signal processing by time-domain, frequency-domain, time-frequency spectra, and nonlinear dynamic techniques, and noninvasive measurement of physiological signals. Mireya Fernández-Chimeno (M’90) received the Ingeniero de Telecomunicación and Doctor Ingeniero de Telecomunicación degrees from the Universitat Politècnica de Catalunya, Barcelona, Spain, in 1990 and 1996, respectively. She has been a Vice-Dean of the Telecomunication Engineering School (ETSETB) from 1996 to 2000. She is currently an Associate Professor of Electronic Engineering at the Universitat Politécnica de Catalunya, Barcelona, Spain. She is also a Quality Manager of the Electromagnetic Compatibility Group (GCEM), Technical University of Catalonia. GCEM is one of the centers of the Tecnological Innovation Network of Generalitat de Catalunya (autonomical govern of Catalonia). She teaches courses of electronic instrumentation, acquisition systems, and electrical safety. She is the coauthor of Electronic Circuits and Devices (Edicions UPC, 1999), and Automatic Test Systems (Edicions UPC, 1999) both published in Spanish or Catalan. Her current research interests include biopotential measurements (high-Resolution ECG, beat-to-beat ECG monitoring, and heart rate variability, etc.) and electromagnetic compatibility, mainly oriented to medical devices and hospital environments. Juan Ramos-Castro (M’94) received the Telecomunication Engineering and Eng.D degrees from the Universidad Politècnica de Catalunya (UPC) Barcelona, Spain, in 1992 and 1997, respectively. In 1992, he joined the Department of Electronic Engineering as a Lecturer, and since 1997, he has been an Associate Professor, teaching courses in several areas of electronic instrumentation. He is a Member of the Biomedical Research Center at the UPC. His current research interests include biomedical and electronic instrumentation.