Brain-computer interface based on generation of visual images

Brain-Computer Interface Based on Generation of Visual Images Pavel Bobrov 1,2 , Alexander Frolov 1 , Charles Cantor 3,4 , Irina Fedulova 5 , Mikhail Bakhnyan 6 , Alexander Zhavoronkov 7 * 1Institute of Higher Nervous Activity and Neurophysiology of Russian Academy of Sciences, Moscow, Russia, 2Technical University of Ostrava, Ostrava Poruba, Czech Republic, 3Department of Biomedical Engineering, Boston University, Boston, Massachusetts, United States of America, 4Department of Physiology and Biophysics, University of California Irvine, Irvine, California, United States of America, 5Moscow State University, Department of Computational Mathematics and Cybernetics, Moscow, Russia, 6Moscow State University, Department of Physics, Moscow, Russia, 7The Russian State Medical University, Moscow, Russia Abstract This paper examines the task of recognizing EEG patterns that correspond to performing three mental tasks: relaxation and imagining of two types of pictures: faces and houses. The experiments were performed using two EEG headsets: BrainProducts ActiCap and Emotiv EPOC. The Emotiv headset becomes widely used in consumer BCI application allowing for conducting large-scale EEG experiments in the future. Since classification accuracy significantly exceeded the level of random classification during the first three days of the experiment with EPOC headset, a control experiment was performed on the fourth day using ActiCap. The control experiment has shown that utilization of high-quality research equipment can enhance classification accuracy (up to 68% in some subjects) and that the accuracy is independent of the presence of EEG artifacts related to blinking and eye movement. This study also shows that computationally-inexpensive Bayesian classifier based on covariance matrix analysis yields similar classification accuracy in this problem as a more sophisticated Multi-class Common Spatial Patterns (MCSP) classifier. Citation: Bobrov P, Frolov A, Cantor C, Fedulova I, Bakhnyan M, et al. (2011) Brain-Computer Interface Based on Generation of Visual Images. PLoS ONE 6(6): e20674. doi:10.1371/journal.pone.0020674 Editor: Simon Rogers, University of Glasgow, United Kingdom Received November 16, 2010; Accepted May 10, 2011; Published June 10, 2011 Copyright: ß2011 Bobrov et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: The work was partly supported by program of the Presidium of RAS ‘‘Basic research for medicine.’’ The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing Interests: The authors have declared that no competing interests exist. * E-mail: [email protected] Introduction A brain-computer interface (BCI) establishes a direct functional interaction between a human or animal brain and an external device. There are numerous recent advances in BCI development and implementation driven by scientific and technological achievements, as well as social and commercial demands. Basic research has revealed correlations between brain signals and mental states [1,2,3,4,5]). This provides a variety of brain signals which might be used for BCI design [1]. Recent technological advances allow real-time on-line processing of multi-channel EEG data using low-cost commercial EEG devices (e.g. Emotiv EPOC EEG headset [6]). The proliferation of these devices into the consumer market has been accelerated by the ability to utilize BCI to partially restore function in various disabilities (see [2,7]) and by a growing interest in using BCI for gaming and other consumer applications [8,9,10]. Figure 1 depicts a general scheme of an EEG-based BCI. The interface consists of an EEG acquisition system, data processing software for feature extraction and pattern classification, and a system to transfer commands to an external device and, thus, providing feedback to an operator. One approach for BCI design is based on the discrimination of EEG patterns related to different mental states [4,11,12]. In this approach the subject is requested to perform different mental tasks. The classifier is trained to distinguish between EEG patterns related to these tasks. Execution of each task causes a certain command being sent to an external device, allowing the operator to control it by voluntarily switching between different mental tasks. If commands sent to the external device trigger different movements, then psychologically compatible mental states are imaginary movements of different extremities. For example, when a subject controls a vehicle or a wheelchair, he can easily associate right hand movement with a right turn of the device. Moreover, mental states related to imaginary movements of extremities are clearly identified by corresponding EEG patterns (synchronization and desynchronization reactions of the mu rhythm, [13,14]), as demonstrated in successful BCI projects such as Graz [2,5,15] and Berlin [16] BCI. Potential applications of BCI extend beyond motion control, including controlling home appliances, selecting contacts in a phone address book or web search engine manipulation. Such tasks are more naturally accomplished by controlling the BCI with voluntary generation of corresponding visual images. Recent work by Cerf et al. [17] demonstrates human ability to voluntarily regulate the activity of neurons responsible for generation of visual images, however, their experiments were based on invasive recordings. As with motion imagination [18,19], functional MRI data suggest that various spatial brain activation patterns correlate with specific types of imagined and perceived visual images [11,20]. According to this data, generation of visual images PLoS ONE | www.plosone.org 1 June 2011 | Volume 6 | Issue 6 | e20674 activates nearly the same brain centers as does the perception of the actual image [21]. It has also been demonstrated that brain activity patterns vary not only by the type of the visual images, but also among images of the same type, and that analysis of such patterns allows to identify the image viewed by the subject, not only its type [22,23]. These findings provide the rationale to hypothesize that brain activity patterns corresponding to specific generated visual images can be identified using EEG. Main goal of this study is to evaluate this hypothesis. In particular, it evaluates the opportunity to classify EEG patterns related to imagination of faces and houses. These types of generated visual images were shown to have different brain activity patterns in functional MRI studies [21,24,25]. One crucial part of a BCI system is the EEG pattern classifier, which identifies patterns corresponding to the subject’s various mental states. There are many approaches for design of such classifiers [26]. The Common Spatial Patterns (CSP) method [27], allowing classification of states of two classes, and its multi-class generalization, the Multiple-class Common Spatial Patterns (MCSP) method [28,29,30] are widely used and considered to be quite efficient. This study compares the Bayesian and MCSP classifiers both based on EEG covariance matrix analysis. However, the Bayesian classifier has lower computational complexity than MCSP which makes it real-time adaptable. An additional objective of this study is to evaluate the significance of EEG artifacts caused by blinking and eye movements, which generate patterns that can differ significantly in various mental states. Recognition of these artifacts can substantially improve EEG-based classification of these mental states. Patterns of involuntary eye movement may differ significantly, especially when different images are being imagined. Therefore, identification and removal of these artifacts is essential to ensure that BCI performance is based on classification of patterns of brain activity itself, and not based on eye-movement patterns. This study was performed using two types of encephalographic caps: an easy to use readily available 16-channel EPOC (Emotiv Systems Inc., San Francisco, USA) and a 32-channel ActiCap (Brain Products, Munich, Germany). Emotiv EPOC is one of the most accurate consumer EEG headsets with the largest user community. This device can be potentially used to build a larger brain activity profile database by building a system for conducting remote experiments via web and opening it to the large user community. The experimental results were validated using ActiCap research EEG device since the research community is not yet actively using EPOC with few publications referencing the use of the device [10]. Method Subjects Seven male subjects aged from 23 to 30 participated in the study. All subjects were right-handed and had normal vision. The experimental procedure was approved by the Board of Ethics at the Institute for Higher Nervous Activity and Neurophysiology of the Russian Academy of Sciences. All participants signed the informed consent forms before participating in the experiments. All experiments involved non-invasive safe procedures and resembled a computer survey while using the non-invasive commercially-available EEG devices. The procedures were also described in the recruitment phase, where students and staff of several academic institutions were offered to participate in experiments involving EEG BCI. Experimental Design The experimental protocol is schematically illustrated in Figure 2. The experiment was conducted on 4 consecutive days, the one series per day. Each series of the first three days consisted of two sessions (Figure 2A). The first, training, session was designed to train BCI classifier. The second, test, session was designed to provide subjects with the output of the BCI classifier in real time to enhance their efforts to imagine pictures. At the fourth experimental day the training and test sessions were preceded by auxiliary session, which was designed to obtain supplementary data Figure 1. General scheme of an EEG-based BCI. EEG is recorded by electrodes placed on the scalp and digitized by an ADC. Computer processing extracts features most suitable for identifying the subject’s intensions. When intension is classified, a certain command is sent to an external device (e.g., a display). Feedback provides the subject with results of his actions thus allowing him to adapt to the system behavior. doi:10.1371/journal.pone.0020674.g001 BCI Based on Generation of Visual Images PLoS ONE | www.plosone.org 2 June 2011 | Volume 6 | Issue 6 | e20674 for estimating the influence of EOG artifacts on the BCI performance. After the experiment was completed, the efficiencies of Bayesian and MCSP classifiers were compared offline. The influence of EOG artifacts on BCI performance was evaluated by comparing classification accuracy before and after the artifacts removal from the data of the fourth day. At the beginning of the study each subject was presented with two types of pictures: faces (10 pictures from the Yale Face Database B [31]) and houses (10 pictures from the Microsoft Research Cambridge Object Recognition Data Base, version 1 [32], adjusted to black-and-white). Subject selected one face and one house as their preferred samples to imagine during the experiment. Subject was sitting in a comfortable chair, one meter from a 170 monitor. Subject was instructed to fix his gaze on a motionless circle (1 cm in diameter) in the middle of the screen, located at eye level. Three grey markers were placed around the circle as displayed in Figure 1. Green color of a particular marker indicated which mental task has to be performed. Left or right marker indicated that the subject should imagine a house or face. The top marker indicated relaxation. Each command to imagine a picture was displayed for 15 seconds and was preceded by a relaxation period of 7 seconds. Each clue was preceded by a 3-second warning (corresponding marker turned blue). Each day experimental series had fixed correspondence of the marker and the picture. In the first series the left marker indicated face and the right marker indicated house. This relation was reversed in each sequential series to prevent classification based on steady marker position. Each series contained two sessions of three blocks (Figure 2(A)). The sessions were separated by a 5-minute interval. Within each block commands to imagine face or house were placed in random order, and each command was presented twice in a block (Figure 2(B)). Thus, each block displayed command to imagine a picture during 30 seconds and relaxation command during 28 seconds. In total, each picture has been imagined during 90 sec and subject has been relaxing for 84 sec in each of training and test sessions. The entire session took approximately 4.5 minutes. Before each session the subject had a chance to view and remember selected pictures. Each day during the training session, the BCI classifier was trained to recognize three states: imagining the face, imagining the house, and relaxation. During the test session, the classifier was both trained and tested. The subject was provided with visual feedback: central circle turned green, if the classifier recognized the target state, otherwise it turned red. Each day the classifier was trained from scratch. During the auxiliary session, the subject was asked to blink (approximately once per second) and to move gaze from the center of the screen in the indicated direction, and back, fixing gaze for 0.5 sec in each position. The directions were up, right, down, and left. Each eye movement condition took 40 seconds, and the blinking condition took 30 seconds (Figure 2(C)). Data recording During the first three days of the study, EEG was recorded using the Emotiv Systems Inc. (San Francisco, USA) EPOC 16-electrode cap (Figure 3(A)). The electrodes were located at the positions AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4 according to the International 10–20 system. Two electrodes located just above the subject’s ears (P3, P4) were used as reference. The data were digitized using the embedded 16-bit ADC with 128 Hz sampling frequency per channel and sent to the computer via Bluetooth. The data were band-pass filtered in 5– 30 Hz range. The impedance of the electrode contact to the scalp was visually monitored using Emotiv Control Panel software. On the 4 th day of the experiment, EEG and EOG (electrooculogram) were recorded using the Brain Products, (Munich, Germany) ActiCap (Figure 3(B)): 24 electrodes (Fz, F3, F4, Fcz, Fc3, Fc4, FT7, FT8, Cz, C3, C4, Cpz, Cp3, Cp4, P3, P4, Poz, Figure 2. Schematic illustration of experiment protocol and each session timing. Sequence of sessions (A), structure of each training (test) session block (B), and structure of auxiliary session (C) are presented. Warnings are marked by blue and instructions to execute each task are marked by green. Instruction durations are given in seconds. Within each block, the instructions to imagine the face or the house are placed in random order, and each instruction is presented twice in a block. doi:10.1371/journal.pone.0020674.g002 BCI Based on Generation of Visual Images PLoS ONE | www.plosone.org 3 June 2011 | Volume 6 | Issue 6 | e20674 Po3, Po4, Po7, Po8, Oz, O1, O2) were used to record EEG and 6 (SO1, IO1, LO1, SO2, IO2, LO2) electrodes, placed around the eyes, were used to record EOG. Central frontal electrode (Afz) was used as reference. The level of electrode impedances was evaluated by means provided by cap producers. The signals were displayed in real time on the computer screen that allowed for controlling their quality visually. The signal provided by EPOC seemed to be noisier than by ActiCap. This observation was confirmed quantitatively during the deleting of EEG artifacts. The data were acquired with 200 Hz sampling frequency and band-pass filtered in 1–30 Hz range by a computer encephalograph (NBL640, NeuroBioLab, Russia) and additionally filtered in 5–30 Hz range by a software FIR filter using MATLAB Filter Design toolbox. All other data processing was also carried out with MATLAB (the Mathworks Inc., Natick, MA, USA). EEG pattern classification The algorithms used for mental state classification are described in the following sections. Bayesian approach. Suppose that there are Ldifferent classes of mental states and for each mental state the EEG data distribution is approximately Gaussian with zero mean. Assume that Ci, a covariance matrix of the data corresponding to the i-th mental state, is nonsingular. Then, the probability to obtain signal Xunder the condition that it corresponds to the i-th mental state is P(Xji)*exp ({Vi=2), where Vi~XT:C{1 i:Xzln(det(Ci)). Following the Bayesian approach, the maximum value of P(Xji), i~1,:::,L, determines the class to which Xbelongs. Hence, the signal Xis considered to correspond to the k-th mental state as soon as k~arg min (Vi). The equality XTC{1 iX~trace(XXTC{1 i)implies that Vi~trace(XXTC{1 i)zln(det(Ci)) ð1Þ Because all Viare rather variable, it is more beneficial to compute the mean values SViTfor sequential EEG epochs using (1) SViT~trace(CC{1 i)zln (det(Ci)) ð2Þ where Cdenotes an epoch data covariance matrix computed as SXXTT. Therefore, to perform the classifier learning it was sufficient to compute the covariance matrices corresponding to each mental state. The classifier was tested by approximating the covariance matrix for each 1-second EEG epoch and computing SViT according to (2). In addition, during the test session the classifier was adjusted after the end of each block. For each mental state the covariance matrix Cb iwas computed based on the block data and the covariance matrix Ciwas replaced with ((1{c)CizcCb i), where the parameter cis 0.01. MCSP method. This approach is based on covariance tensor analysis [30]. In case where tensors are second order (i.e., they are covariance matrices), the MCSP method can be described as follows. ThecovariancematricesCi,i~1,:::,Lare obtained based on multidimensional EEG data recorded during the classifier learning. Then matrices Miare sought to meet the following requirements MiCiMT i~Dið3Þ MiCSMT i~Ið4Þ where CS~C1zC2z:::zCL,Iis the identity matrix, and Diis a diagonal matrix. The problem of obtaining the matrices Mihas an explicit solution. Indeed, if CS~UDUTis the singular value decomposition (SVD) of the CSmatrix, with Ua unitary matrix and Da diagonal matrix, then it is easy to prove that Mi~UT iD{1=2UT,whereUiis the unitary matrix obtained from the SVD decomposition D{1=2UTCiUD{1=2~UT iDiUT i.The matrices Miare used to project both the training and the test data onto feature space and, therefore, to obtain a set of training and test feature vectors for each state. The signal corresponding to a certain state is segmented into epochs and for each epoch vectors vi~diag MiSXXTTMT i  ,i~1,:::,L, are computed by estimating variances of all components of vectors ji~MiXbased on the epoch data X.ThenXis mapped onto a feature vector j~log (v),wherev is concatenation of all vectors vi,andlog ( )is a component-wise logtransform. After that, classification of the test feature vectors is performed. We used the SVM algorithm described in [33] for feature vector classification. When two mental states are classified, results obtained by MCSP and CSP [27] are identical. In this case M1~M2~Mand Figure 3. Electrode locations for EEG headsets used in this study. EPOC (A), ActiCap (B). doi:10.1371/journal.pone.0020674.g003 BCI Based on Generation of Visual Images PLoS ONE | www.plosone.org 4 June 2011 | Volume 6 | Issue 6 | e20674 SjjTTis a diagonal matrix. If a particular component of j1has low variance for a certain state, then this component of j2has high variance in this state and vice versa, since D1zD2~I. Then feature vectors corresponding to different states can be easily separated. Similarly, when MCSP is applied to classify more than two states, then for each state ithere exist components of jiwith variance significantly lower than variances of the respective components of jj,j=i.Thiscan explain the efficacy of MCSP. Evaluation of classifier quality To compare Bayesian and MCSP classifiers, offline analysis of both training and test session data was performed. The comparison was offline for several reasons. At first, the training session data could not be analyzed online, because the classifier had not been trained yet. Secondly, it was reasonable to use only one classifier (Bayesian in our experiments) for online feedback control of the mental states during the test session. To evaluate classifier efficiency, EEG records corresponding to different mental states were split into epochs of 1 second length. Then the artifact data were identified according to the 3srule. All epochs with more than 7% of samples marked as artifact were excluded from the analysis. After the exclusion of the artifact epochs, repeated random sub-sampling validation was performed. The epochs were split into training and test sets with 90% of epochs being used for classifier training and 10% epochs being used for testing. Each learning set contained about 70 epochs and about 7 epochs were used for classifier testing in each of two sessions for EPOC and, respectively, about 75 and 8 for ActiCap. As a result of averaging over 100 splits, a confusion matrix P~(pij)was obtained. Here pij is an estimate of probability to recognize the i-th mental state in case the j-th mental state is to be produced. Note that in case of the good state recognition diagonal elements of matrixPare significantly greater than non-diagonal ones; and if there is no classification error, then Pis identity matrix. Mean of the confusion matrix diagonal elements p~X i pij=Lð5Þ was chosen as an index of the classification quality. It is easy to see, that p~1when the states are recognized perfectly, and p~1=Lif classification is independent of the mental states produced. The classifier performance was also measured by computing the mutual information between the commands to produce mental states and the states classified: g~{ X ij pijp0jlog2(pij=pi0)ð6Þ In equation (6) pi0~Pjpijp0jis probability of the i-th state to be recognized and p0jis probability of the j-th command to be presented. Notice that if probabilities to display each command are assumed to be equal, then p0j~1=L. In this case g~log2Las soon as there is no classification error. Also notice that g~0when the state recognition is independent of the commands. Consider a special case when probabilities of correct recognition of different mental states equal to each other, i.e. pii~pfor all i, and probabilities of incorrect recognition also equal each other, i.e. pij~(1{p)=(L{1) for all j=i. Then the mutual information between the displayed commands and the states classified can be obtained as follows: g~log2Lzplog2pz(1{p)log2((1{p)=(L{1)) ð7Þ Based on [34], equation (7) is often used to estimate BCI efficacy ([35,36,37]). But if the corresponding assumptions are not true, the value of g, calculated according to (7), is lower than the actual mutual information. In this study we used the general formula (6). EOG artifact removal To evaluate the influence of EOG artifacts on BCI performance we compared BCI efficiencies before and after artifact removal from recordings of the fourth day. To remove the artifacts from recordings of the training and test sessions we concatenated the recordings with those of the auxiliary session. These data were not filtered in 5–30 Hz range in order to avoid artifact attenuation that could impair their detection. To identify the artifacts we used implementation of the Infomax Independent Component Analysis (ICA) algorithm (EEGLAB RUNICA, [38]). As a result of ICA, multidimensional signal Xcontaining both EEG and EOG data, is represented as X(t)~Wj(t) where Wis a matrix of weights where columns specify contribution of the corresponding independent component into each EEG or EOG channel and vector j(t)specifies intensities of the independent components. Since NEEG~24 electrodes were used to record EEG and NEOG~6electrodes were used to record EOG, Wis a (NEEGzNEOG)|(NEEGzNEOG)matrix. The obtained independent components were sorted according to their contribution to the total variance of the EOG signals. The first NAcomponents constituting 97% of the variance were treated as artifact components. Artifact removal was performed by setting intensities of artifact components to zero. This is equivalent to removing the first NAcolumns of W. Thus, refined EEG signal is represented as XEEG(t)~WEEGjEEG(t) where WEEG was a NEEG|(NEEGzNEOG{NA)matrix of weights and jEEG was (NEEGzNEOG{NA)-dimensional vector of nonartifact source intensities. The WEEG matrix was obtained from the Wmatrix by removing NAcolumns corresponding to the artifact components and NEOG rows corresponding to the EOG channels. Results The first part of this section demonstrates the results of the offline classifier comparison. We also show that the EOG input into EEG is sufficient for BCI control. This emphasizes importance of EOG artifact removal to obtain BCI based on brain activity only, and not based on eye movement and blinking. Afterwards, results of classification of the data with EOG artifacts removed are presented. BCI efficiency Table 1 shows the confusion matrix obtained for one subject by offline Bayesian classification of the data recorded during the 4 th day training session. It can be seen that matrix is diagonally dominant, which means that correct recognition is prevalent. In this case p, mean of the matrix diagonal elements, equals to 0.54 and g, mutual information, equals to 0.14. In contrast, p~0:33 and g~0if three states were classified randomly. Figures 4 and 5 show pand gvalues for all 4 experimental days for all subjects. Left panes (A and C) of each figure represents BCI Based on Generation of Visual Images PLoS ONE | www.plosone.org 5 June 2011 | Volume 6 | Issue 6 | e20674 classification quality for Bayesian approach and right panes (B and D) represent quality for MCSP. Respectively, upper panes (A and B) represent quality for training session and lower panes (C and D) represent quality for test session. For every subject and every session of each day values of pexceed the value of 0.33 which corresponds to random classification. Two-way ANOVA test for EPOC data did not reveal dependence of pand gindices on the day of experiment (Pw0:15 for both sessions, both classifiers, and both indices). However this test revealed significant increase of pand gwhen comparing MCSP classification of training and test sessions (P~0:013 for pand P~0:023 for g). At the same time, the increase was not significant for Bayesian classification (P~0:20 for pand P~0:08 for g). Two-sample t-test for ActiCap data revealed that pindex was significantly higher for MSCP classification of the test session than for training session (P~0:03). Observed trend of quality improvement in the test session might be explained by subjects’ additional training during the preceding training session as well as the increase in subjects’ focusing on the task when they were provided with visual feedback. Table 1. Confusion matrix obtained with the Bayesian classifier for one subject based on the data of the 4 th day learning session. Commands presented Relax Imagine the house Imagine the face Recognized states Relaxation 0.53 0.15 0.18 Imagining the house 0.20 0.52 0.24 Imagining the face 0.27 0.33 0.58 doi:10.1371/journal.pone.0020674.t001 Figure 4. Classification quality for all subjects during training and test sessions, as measured by value p.Classification quality during training is displayed on left panes A and C, quality during test is presented on panes B and D. The first row of columns corresponding to the 4 th day (4a) represents pvalues computed from data for 16 EEG electrodes, and the second one (4b) represents the values computed from data for all 24 EEG electrodes. Notice that each column exceeds the level p= 0.33 related to random classifying. doi:10.1371/journal.pone.0020674.g004 BCI Based on Generation of Visual Images PLoS ONE | www.plosone.org 6 June 2011 | Volume 6 | Issue 6 | e20674 Also ActiCap data were classified significantly better than EPOC data (two-sample t-test, Pv0:015 for both training and test sessions and both classifiers). For the ActiCap data the maximum value of pover all subjects equals to 0.66 (0.68) for the Bayesian (MCSP) classifier and the maximum value of gover all subjects equals to 0.39 (0.48) for the Bayesian (MCSP) classifier. Average values of pand gover all subjects and sessions equal 0.52 (0.56) and 0.15 (0.20) for Bayesian (MCSP) classifier. For the EPOC data maximum values over all subjects and experiment days equals to 0.52 (0.63) for pand 0.18 (0.40) for g. Average values of pand g over all subjects, sessions, and days equals to 0.45 (0.48) and 0.07 (0.11) correspondingly. It can be seen that on average MCSP classifier based on covariance matrix analysis performed slightly better than the Bayesian one. The difference in classification Figure 5. Quality of classification measured by index g.Data representation is the same as in Figure 4. doi:10.1371/journal.pone.0020674.g005 Table 2. Confusion matrix obtained as a result of Bayesian classification of EEG patterns, corresponding to various eye movements and blinking, prior to EOG artifact removal. Commands presented Blink Move gaze upwards Move gaze right Move gaze downwards Move gaze left Recognized states Blinking 0.73 0.06 0.00 0.05 0.00 Upward eye movements 0.04 0.66 0.02 0.10 0.01 Rightward eye movements 0.00 0.04 0.81 0.05 0.27 Downward eye movements 0.22 0.21 0.01 0.78 0.02 Leftward eye movements 0.01 0.03 0.16 0.02 0.70 doi:10.1371/journal.pone.0020674.t002 BCI Based on Generation of Visual Images PLoS ONE | www.plosone.org 7 June 2011 | Volume 6 | Issue 6 | e20674 quality was small but significant for EPOC data (two-sample t-test, Pv0:01 for pand gover all subjects, experimental days, and sessions) although not significant for ActiCap data (two-sample ttest, Pw0:05 for pand gover all subjects and sessions). Influence of EOG artifacts It could be possible that the relatively high BCI quality observed is related to eye movement. During imagining selected pictures subjects might be making involuntary eye movements in specific patterns, detailing the imagined pictures. Below we show that classification between states is actually based on differences in brain activity measured by EEG, and not on patterns of eye blinking and movements. To investigate this we demonstrated that patterns resulting from eye movements and blinking on EEG could be easily discriminated by means of the proposed classifiers. On the fourth experimental day auxiliary session data were processed as described in Evaluation of classifier quality section above. Recall that the EEG patterns recognized were induced by blinking and eye movements of four types (up-center, right-center, down-center and left-center). Table 2 shows the confusion matrix coefficients (pij) for one subject, obtained as a result of Bayesian classification. The diagonal coefficients of the matrix presented are observably dominant indicating high classification quality. Mean values of p among the subjects were 0.6360.04 and 0.6460.04 for the Bayesian and the MCSP classifiers respectively, and corresponding Figure 6. Decrease of total variance of signals after sequential removal of the independent components for all subjects. Left pane (A) represents signals recorded by EOG electrodes, and right pane (B) corresponds to EEG electrodes. Removed artifact components are marked by red points. doi:10.1371/journal.pone.0020674.g006 Figure 7. Signals from EOG electrodes for one of the subjects and corresponding independent components, identified as EOGrelated for this subject. EOG signals are presented in blue (6 lower curves), corresponding independent components are displayed in red (5 upper curves). From left to right, signals correspond to: blinking (B), moving eyes upwards (U), to the right (R), downwards (D) and to the left (L). Each curve is normalized by the standard deviation. doi:10.1371/journal.pone.0020674.g007 BCI Based on Generation of Visual Images PLoS ONE | www.plosone.org 8 June 2011 | Volume 6 | Issue 6 | e20674 mean values of gwere 0.8660.14 and 0.8560.13. Note that p~0:2when five states are classified randomly. The mutual information gfor the EEG patterns mentioned above is an order of magnitude greater than the quality achieved in recognition of patterns that correspond to imagining the pictures. Next, EOG artifacts were removed from the auxiliary session data using the method described in section EOG artifact removal. Briefly, the ICA decomposition of the signal was obtained and the components comprising the 97% of total variance of signal from EOG channels were eliminated. Figure 6(A) shows residual values of EOG signal variance during the sequential removal of components rated according to their contribution. Results are shown for all 7 subjects. Margin of 3% is also shown in Figure 6(A). Note that the number of the artifact components never exceeded the number of EOG channels so the covariance matrices computed based on the refined data were never singular. Figure 6(B) shows decrease of the total variance of signals from EEG channels during the sequential removal of the EOG-related components. Contribution of the components into EEG signal is quite substantial, but their removal suppresses EEG signal less significantly than it does for EOG signal. Total EEG variance, averaged over all subjects, remains at 30% after artifacts are removed. Figure 7 shows the signals from 6 EOG electrodes for one subject and 5 independent components, identified as EOG-related for this subject. There is an evident correspondence between the components and types of EOG artifacts. Figure 8 demonstrates the result of EOG artifact removal. The data previously contaminated with artifacts became indistinguishable from the data initially containing no artifacts. Figure 9 presents distributions of individual ICA components related to EOG artifacts over the head. They agree with distributions for modeled blinking and eye movement artifacts, described in [39]. The confusion matrix for data obtained in auxiliary session of the fourth day after EOG artifact removal is presented in Table 3 for the same subject as in Table 2. As shown, recognition quality is substantially reduced due to artifact removal. Mean values of p among the subjects dropped to 0.5060.03 and 0.4260.03 for the Bayesian and MCSP classifiers respectively, and corresponding mean values of gdropped to 0.4760.10 and 0.2960.06. The difference of quality measures computed before and after artifact removal is significant (two-sample t-test, Pv0:01 for both classifiers and both measures). It is remarkable that even with nearly complete exclusion of EOG artifacts the quality of EEG pattern classification remained quite high and significantly exceeded random level (onesample t-test performed for pindex, Pv10{4). The second step to investigate possible EOG artifact effect on classification of EEG patterns corresponding to imagining the pictures was to evaluate the quality after EOG artifact removal. Data obtained on the fourth experimental day were used since EOG was recorded only during this day. During processing training and test sessions, each session data was filtered in 1–30 Hz range and concatenated with the auxiliary session data. The concatenated records were decomposed using ICA to identify EOG-related components and eliminate them. The artifact Figure 8. Result of blinking artifact suppression for one of the EOG channels and one of EEG channels. Blue and red curves represent signals before and after artifact components removal respectively. doi:10.1371/journal.pone.0020674.g008 Figure 9. Spatial distributions of individual ICA components related to EOG artifacts. Graphs correspond to blinking (B), moving eyes upwards (U), to the right (R), downwards (D) and to the left (L). doi:10.1371/journal.pone.0020674.g009 BCI Based on Generation of Visual Images PLoS ONE | www.plosone.org 9 June 2011 | Volume 6 | Issue 6 | e20674