Assessment of Biosignals for Managing a Virtual Keyboard

Assessment of Biosignals for Managing a Virtual Keyboard Manuel Merino1, Isabel Gómez1, Alberto J. Molina1, and Kevin Guzman2 1 Electronic Technology Department, University of Seville Avd. Reina Mercedes s/n, 41012, Seville, Spain [email protected] {igomez,almolina}@us.es http://matrix.dte.us.es/grupotais/ 2 Guadaltel S.A. Pastor y Landero, 19, 41001, Seville, Spain [email protected] Abstract. In this paper we propose an assessment of biosignals for handling an application based on virtual keyboard and automatic scanning. The aim of this work is to measure the effect of using such application, through different interfaces based on electromyography and electrooculography, on cardiac and electrodermal activities. Five people without disabilities have been tested. Each subject wrote twice the same text using an electromyography interface in first test and electrooculography in the second one. Each test was divided into four parts: instruction, initial relax, writing and final relax. The results of the tests show important differences in the electrocardiogram and electrodermal activity among the parts of tests. Keywords: affective interfaces, biosignals, control system, disability. 1 Introduction According to the Eurostat [1], the total population in Europe was 362 million in 1996 which 14.8% of the population between 6 and 64 years old have physical, psychological or sensory disabilities. Therefore, assistive and adapted systems have to be developed. The aim of these ones is to help users in their diary tasks, letting them to live in a more comfortable way. Focus on software, in general, it can be claimed that these applications have a high degree of flexibility and customizable, but their throughput strongly depends on the adapted device or interface the user needs to interact with it. The aforementioned throughput might also be dramatically reduced for people who suffer from severe disability. Another lack of these software programs is that they cannot be automatically adapted when they are running and self-adjusting to user requirements. The goal of developing a method for matching parameters that could be modified in an adaptative row-column scanning to each individual user is discussed in [2], but, in that paper contextual information of the application is considered instead of physiological signals. Nowadays there are several researches focus on in incorporate the feature of self-adaptation in different systems or applications. So, based on this concept there are papers whose aim is modify the environment to adapt to the user needs/preferences (Ambient Intelligence) or use biosignals to adjust an application automatically (Physiological Computation [3]). This paper describes how the use of virtual keyboard (VK) affects different physiological signals and how these ones could be used to incorporate automatic adaptation of functioning parameters in this VK application. 2 Related Work Disable people need interfaces to be adapted to their physical and cognitive skills. Whatever the user’s type of disability, the interface needs to detect user’s will in order to select the highlighted row or column on the virtual keyboard. In most cases, user is able to move some muscles and press a special switch, or control a mouse pointer on computer screen through an accelerometer [4]. But sometimes user’s ability is constrained to move the eyes; therefore an eye-tracker is required. Electric biosignals can be used instead of those systems. For instance, an electromyography (EMG) system can record muscle activity and substitute the switch, and also electrodes placed around the eyes can be used to detect gaze [5, 6]. Moreover, interfaces based on Brain Computer Interface (BCI) don’t require user movements [7] because this one is based on electroencephalogram (EEG). Biosignals can be also used as a source of information when user emotional state must be determined. Electrodermal activity (EDA) [8] and electrocardiogram (ECG) are two of the biosignals most used in this area. EDA has been used in several researches about stress and cognitive load [8-10]. It has been shown EDA is a good method to detect different emotional states. A typical experiment to measure the stress [9] is to make a subject calculate several arithmetic operations of different level of difficult in 4 minutes and 3 phases. From phase to phase the level of stress goes up by increasing the operation difficulty. Significance changes in EDA among phases have been reported, such that the cognitive load was evaluated correctly in 82.8% of studied cases. It has also reported in several papers that the cardiac activity is also affected by the kind of task. In [10] the authors have used biosignals as objective indicators of state of users, where EDA, ECG, EMG and breath frequency were recorded while the subjects played a game. 3 Methodology 3.1 Application The tests were developed using a customizable VK [6] shown in figure 1. Among other things the VK allows: keys personalization, using different control interfaces based on events and generating a list of predicted words. An extended VK distributed by rows of six characters each is selected. High frequency characters in Spanish language are placed on first rows. A row/column scanning method was used to select a letter or a word on a prediction list. The minimum number of selected characters used to turn on prediction engine was set to 3. The scan time or dwell time was established in 1.6s for EMG control interface and 1.8s for EOG control interface based on 0.65 rule [11]. Fig. 1. Extended Letter VK 3.2 Signal Acquisition Three different biosignals were simultaneously recorded during the tests. One of them was used to manage a VK application (EMG or EOG), and the others were registered to determine the stress/fatigue of the user induced by the control interface (ECG, EDA) [8-10]. The assembly used to record the different biosignal is shown in the figure 2. A monopolar assembly was employed to record the ECG signal (figure 2.a). The EDA assembly was placed on the wrist (figure 2.b) [12]. Three electrodes on the arm were used for the EMG signal (figure 2.c). Finally, to pick EOG signal up two electrodes by direction (figure 2.d) where placed around the eye [13] and the ground and reference sensors were placed respectively in the right and left ears. Fig. 2. Electrode positions. a) ECG. b) EDA. c) EMG. d) EOG. Ag/AgCl electrodes with self-adhesive/conductive gel were used to record the biosignals. Two bioamplifiers were employed: g.USBamp and g.MOBILab of gTec. The first of them registered the control biosignals with a sample frequency of 512Hz for EMG and 128Hz for EOG. A notch filter in (48, 52)Hz was implemented to eliminate the power electric noisy, and a bandpass filter in (5, 200)Hz for EMG and (0.1, 30)Hz for EOG. Also, the version 2.0 of the BCI2000 software was employed [7] to send the event to the VK through a socket. The other bioamplifier recorded the ECG signal with a frequency of 256Hz, a notch filter in (48, 52)Hz and a bandpass filter in (0.5, 100)Hz. The EDA signal employed a different system: QSensor [12]. This one is a wearable system that wears in the wrist and uses Ag/AgCl electrodes. Different sample frequency can be employed and the data are stored in the internal memory. 32Hz of sample frequency was setting. The version 7.6.0.324 of Matlab application was utilized to offline analysis of the recorded data. 3.3 Procedure Trials are focused on text input function of the VK. EMG and EOG signal are considered like control interfaces. Five users without disabilities between 26 and 45 years old participated in these trials (mean 31.2, sd. 7.79); only one of them had used the application before. The temporal line of the experiment is splitted into four parts: 1. Task explanation and electrode setting. Users are instructed about the task they are going to do. Then electrodes to record EDA, ECG, EOG and EMG signal are placed. 2. Ten minutes of relax. During this time user is reading a magazine. 3. Main activity. Users input a predefined text using virtual keyboard. Command to control the application is based on biosignals. Duration of the trial is user dependent, this is, there is no time to finish it. 4. Rest time. 10 minutes to recover, reading a magazine. The main task consists in writing a text of 43 words (209 characters). The application is operated by a single event which, in turn, is generated in two different ways: Muscle activity by EMG signal processing or Horizontal Eye Movement by EOG analysis. A wrong character had to be corrected. However, the mistakes made by selecting an incorrect word from the prediction list had to be ignored, so the user had to write the next word. The ECG and EDA signals are recorded in all phases, excluding phase 1 where the users are instructed. By measuring these signals, it is possible to acquire objective information about the user emotional state. Two State-Trait Anxiety Inventory (STAI) questionnaires were also provided to subjects. They filled them in at the beginning of the relax phases (phases 2 and 4). So, a subjective measure of stress is also achieved. 4 Work and Results Different parameters of affective signals have been researched. The heart rate variability (HRV) has been the main extracted feature from ECG. Each HRV vector element shows the time between two neighbor heartbeats. The extracted data are classified in two groups: information based on temporal analysis or based on frequency analysis. The standard deviation of beat-to-beat or NN intervals (SDNN), the square root of the mean squared difference of successive NN intervals (RMSSD), the proportion of NN intervals that differ by more than 50 ms (pNN50), the width of the minimum square difference triangular interpolation of the highest peak of the histogram of all NN intervals (TINN), and the entropy of HRV are the parameters of the first group. The other group is based on calculate the power spectrum density (PSD) of the HRV. The PSD is divided in 4 parts: frequency from DC to 0.003Hz (Ultra Low Frequency - ULF), from 0.003 to 0.04Hz (Very Low Frequency - VLF), from 0.04 to 0.15Hz (Low Frequency) and from 0.15 to 4Hz (High Frequency - HF). One additional parameter is the ratio between HF and LF (HF/LF). The influence of each band about the PSD was calculated as the sum of PSD of the band divided by overall PSD sum. In all parameters, a slide window of 5 minutes of width with a shift of 1 heartbeat was applied. Five parameters were extracted from EDA signal: tendency, average amplitude, mean of derivative, average increase in periods of rising, and time ratio increase. A slide window of 5 minutes of width with a shift of 1 sample was applied. The extracted information was analyzed through an analysis of variance (ANOVA) for each single subject with a threshold of 1% (alpha parameter - p < 0.01). Four analyses for each subject were realized: relax phase 1 vs. main task, main task vs. relax phase 2, relax phase 1 vs. relax phase 2, and relax phase 1 vs. main task vs. relax phase 2. Difference between using EMG or EOG interfaces was the goal of the next ANOVA analysis: relax phase 1 of EMG control vs. relax phase 1 of EOG, main task of EMG vs. main task of EOG, and relax phase 2 of EMG vs. relax phase 2 of EOG. The expected results were the relax phases weren’t affected (p > 0.01) and the main activities showed significative difference between EMG and EOG (p < 0.01). However, the ANOVA results of majorities of subjects in relax phases showed an important influence of unknown variable (p < 0.01). So, variations in ECG and EDA signals due to doing main task with EMG or EOG interfaces are not conclude, in despite of the expected results for main tasks were obtained. Table 1. STAI test results for control interfaces Phase User 1 User 2 User 3 User 4 User 5 EMG EOG EMG EOG EMG EOG EMG EOG EMG EOG Relax 1 40 46 43 38 43 47 52 50 43 44 Relax 2 44 45 44 35 41 36 46 45 38 37 The EMG and EOG control interfaces showed the kind of phase affects directly all EDA parameters (p < 0.00045). Also, SDNN, RMSSD, pNN50 (p < 0.0007) and the bands ULF and VLF (p < 0.0085) of ECG parameters showed significative changes when using the EMG interface, while SDNN, pNN50 (p < 0.0004) and all frequency bands (p = 0) were affected for EOG control . On the other hand, the table 1 with the STAI result showed that the different phases affected to stress state of the subjects, such that the subjective stress level of two of them was higher in the end of the main task than in the begging of test for EMG. The others were a stress level higher in the begging of the test than in the end of main task. Also, the variance (var.) of the modulus of the difference between phases in STAI test was bigger in EOG (var. 8) than EMG (var. 3.44). 5 Conclusion and Planned Lines of Research This paper presents how the biosignals can be used to manage an application from two points of view: the voluntary control actions and self-adaptation based on the measure of emotion. EMG and EOG are used to control and EDA and ECG to measure emotion. To sum up, it have done EMG and EOG test and it has been determined by ANOVA analysis what are the parameters of EDA and ECG signals that are affected in different phases of trial. Significant changes were observed in these parameters. So, important differences were found between relax phases and main task. However, it was not possible determinate ECG and EDA changes between EMG or EOG interface, because an unknown variable affected the relaxed periods. In the future, the aim will be develop an intelligent agent to incorporate to the VK application. This one will adapt automatically the software depending on emotional state of the user to do easier the use of system. So, it is necessary to determine exactly what the causes are which produce the changes in the biosignals, tired or stressed, and how these states affect to the measured parameters. Acknowledgements. This project has been carried out within the framework of a research program: (p08-TIC-3631) – Multimodal Wireless interface funded by the Regional Government of Andalusia. References 1. Eurostat. Heath Statistics. Luxembrourg: Office for Official Publications of the European communuties (2002), http://epp.eurostat.ec.europa.eu ISBN: 92-894-3730-8 (retrieved on January 2012) 2. Simpson, R.C., Koester, H.H.: Adaptive One-Switch Row-Column Scanning. IEEE Transactions on Rehabilitation Engineering 7(4), 464–473 (1999) 3. Fairclough, S.H.: Fundamentals of Physiological Computing. Interacting with computers 21(1-2), 133–145 (2009) 4. Gómez, I., Anaya, P., Cabrera, R., Molina, A.J., Rivera, O., Merino, M.: Augmented and Alternative Communication System Based on Dasher Application and an Accelerometer. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A. (eds.) ICCHP 2010. LNCS, vol. 6180, pp. 98–103. Springer, Heidelberg (2010) 5. Dhillon, H.S., Singla, R., Rekhi, N.S., Rameshwar, J.: EOG and EMG Based Virtual Keyboard: A Brain-Computer Interface. In: 2nd IEEE International Conference on Computer Science and Information Technology (2009) 6. Merino, M., Gómez, I., Rivera, O., Molina, A.J.: Customizable Software Interface for Monitoring Applications. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A. (eds.) ICCHP 2010, Part I. LNCS, vol. 6179, pp. 147–153. Springer, Heidelberg (2010) 7. Schalk, G., McFarland, D.J., Hinterberger, T., Birbaumer, N., Wolpaw, J.R.: BCI 2000: A General-Purpose Brain-Computer Interface (BCI) System. IEEE Transactions on Biomedical Engineering 51(6), 1034–1043 (2004) 8. Boucsein, W.: Electrodermal Activity, 1st edn. Springer (1992) 9. Setz, C., Arnrich, B., Schumm, J., La Marca, R.: Discriminating Stress from Cognitive Load Using a Wareable EDA Device. IEEE Transactions on Information Technology in Biomedicine 14(2), 410–417 (2010) 10. Mandryk, R.L., Inkpen, K.M., Calvert, T.W.: Using Psychophysiological Technique to Measure User Experience with Entertainment Technologies. Behavior & Information Technology 25(2), 141–158 (2006) 11. Simpson, R., Koester, H.H., Lopresti, E.F.: Selecting an appropriate scan rate: the rule. 65. Assistive Technology 19(2), 51–58 (2007) 12. Poh, M.-Z., Swenson, N.C., Picard, R.W.: A Wearable Sensor for Unobtrusive, Long- Term Assessment of Electrodermal Activity. IEEE Transactions on Biomedical Engineering 57(5), 1243–1251 13. Merino, M., Rivera, O., Gómez, I., Molina, A., Dorronzoro, E.: A Method of EOG Signal Processing to Detect the Direction of Eye Movements. In: SensorDevices 2010, pp. 100–105 (2010)