Repositorio Institucional de Documentos

Trabajo Fin de M´aster Ingenier´ıa Biom´edica Curso 2010-2011 Acquisition, characterization and classification of Feedback Event-Related Potentials during a time-estimation task Adquisici´on, caracterizaci´on y clasificaci´on de Feedback Event-Related Potentials durante una tarea de estimaci´on de tiempo Eduardo L´opez Larraz Junio de 2011 Director: Javier M´ınguez Zafra Codirector: Luis Montesano del Campo Departamento de Inform´atica e Ingenier´ıa de Sistemas Centro Polit´ecnico Superior Universidad de Zaragoza A todos los que me han ayudado a lo largo de este camino. iii Resumen Las se˜nales de feedback son componentes fundamentales dentro de los interfaces cerebro-ordenador (brain-computer interfaces o BCI), ya que suministran informaci´on para guiar la tarea ejecutada en cada momento. Se ha demostrado que la presentaci´on de este tipo de est´ımulos produce cierta actividad en el cerebro que puede ser medida y clasificada. Dado que estos est´ımulos pueden darse mediante distintas modalidades sensoriales, es importante conocer los efectos que cada tipo de feedback produce en las se˜nales cerebrales, as´ı como cu´al es el impacto que tiene en la clasificaci´on de estos potenciales. El objetivo de este trabajo fin de m´aster es la realizaci´on de un estudio sobre los potenciales elicitados en el cerebro tras la presentaci´on de se˜nales de feedback, tanto positivo como negativo, mediante tres v´ıas sensoriales: visual, auditiva y t´actil. Se pretende desarrollar una BCI que permita adquirir potenciales evocados por distintos est´ımulos de feedback para su posterior caracterizaci´on y clasificaci´on. La estructura del presente trabajo se divide en cinco bloques principales. El primero de ellos consisti´o en la b´usqueda y estudio de bibliograf´ıa relacionada, lo cual permiti´o al autor crear la base de conocimiento necesaria para realizar el resto del trabajo. En segundo lugar se procedi´o a dise˜nar una BCI con un protocolo de experimentaci´on que permitiese adquirir los potenciales cerebrales elicitados por feedback, mediante el registro de se˜nal electroencefalogr´afica (EEG). Una vez ideado el protocolo, se procedi´o a la ejecuci´on de una serie de sesiones de experimentaci´on con 15 personas. De ellas, 5 realizaron los experimentos recibiendo la modalidad de feedback visual, 5 recibieron la modalidad auditiva y 5 t´actil. Por tanto, la parte pr´actica de este trabajo se ha basado en la realizaci´on de 30 sesiones de experimentaci´on (2 con cada uno de los sujetos), de alrededor de una hora de duraci´on cada una. Cada sesi´on de experimentaci´on consisti´o en realizar un montaje de electroencefalograma con 32 electrodos, ejecuci´on y supervisi´on de la brain-computer interface, y finalmente retirada de todo el equipo de EEG y limpieza del mismo. Las sesiones de experimentos de 5 de los sujetos se realizaron en un laboratorio acondicionado para tal efecto en la Universidad de Zaragoza, las de los restantes 10 sujetos fueron realizadas en Bit&Brain Technologies, empresa spin-off de la Universidad de Zaragoza que se dedica a tareas de I+D utilizando tecnolog´ıa BCI. Tras la obtenci´on de la actividad EEG de las 15 personas, el siguiente paso consisti´o en realizar una caracterizaci´on de los potenciales adquiridos. Esta caracterizaci´on fue llevada a cabo desde el punto de vista de se˜nal (Grand Averages) y de localizaci´on de fuentes, estudiando los focos de activaci´on cerebral que generan el EEG medido. v En ´ultimo lugar, se procedi´o a la evaluaci´on de varias estrategias de clasificaci´on basadas en Support Vector Machines. Mediante la exploraci´on de distintas estrategias se trat´o de evaluar el porcentaje de clasificaci´on que se obtiene cuando se entrena el sistema con datos del propio sujeto que se va a clasificar y cuando se entrena con datos de sujetos distintos, tanto si sus se˜nales han sido generadas por la misma modalidad de feedback como si han sido generadas por alguna otra. De forma adicional al trabajo inicialmente descrito en la propuesta de este trabajo fin de m´aster y, partiendo de los buenos resultados obtenidos, se quiso ir m´as all´a, dando una aplicaci´on pr´actica a las herramientas desarrolladas. Dado que el reconocimiento de potenciales elicitados por feedback tiene un gran potencial en algunas terapias de rehabilitacion, se utilizaron datos de un entrenamiento de neurofeedback para mejoras cognitivas, llevado a cabo en la empresa Bit&Brain Technologies con sujetos sanos. Durante este entrenamiento se adquirieron potenciales de feedback de 5 sujetos, que fueron estudiados y clasificados del mismo modo que los adquiridos con el protocolo incialmente dise˜nado. vi Abstract Feedback stimuli are fundamental components in Brain-Computer Interfaces (BCI), since they provide key information to guide the executed task in each moment. It is known that the presentation of feedback stimuli elicits certain brain potentials that can be measured and classified. As stimuli can be given through different sensory modalities, it is important to understand the effects of different types of feedback on brain responses and their impact on classification. The objective of this master thesis is to perform a deep study about the brain potentials elicited after presentation of positive and negative feedback signals, through three sensory inputs: visual, auditive and tactile. To that end, a BCI that allows to acquire brain potentials elicited by different feedback stimuli has to be developed. Once the potentials are recorded, characterization and online classification can be carried out. This master thesis has been divided into five main blocks. First one was the search and study of related bibliography, what allowed the author to acquire the required knowledge to perform the remaining steps of the work. Second step was the design of a BCI with an experimentation protocol that allowed to record the brain potentials evoked by feedback, through the use of electroencephalography (EEG). Once the protocol was designed and implemented within the BCI, the third step was the execution of a series of experimentation sessions with 15 subjects. Five of the subjects performed the experiments receiving visual feedback, five received auditive modality, and five received tactile feedback. Therefore, the practical part of this master thesis consisted of the execution of 30 experimentation sessions (2 with each subject), that lasted about one hour each. Each experimentation session was composed of three steps. First step was the set up of the electroencephalogram, that required to place 32 electrodes on the scalp. Second step was the execution and supervision of the brain-computer interface, while last one was the retreat and cleaning of the EEG equipment. The experimentation sessions of 5 of the subjects were performed in a laboratory of Universidad de Zaragoza fully equipped to that end. Experiments of the remaining 10 subjects ware carried out at Bit&Brain Technologies, spin-off company of Universidad de Zaragoza. This company is involved in R&D labors using BCI technology. After recording the EEG activity of the 15 people, the next step consisted of performing a characterization process of the brain signals. That characterization was accomplished from a point of view of signal processing (Grand Averages) and of source localization, studying the brain foci of activation that generated the measured EEG. Fifth step was the evaluation of several classification strategies based on Support Vector Machines. Exploring different classification strategies allowed to test the accuracies that could vii be achieved when training the system with data of the subject that is going to be classified, or when training with data of different subjects that received feedback of the same or of a different modality. As an addition to the work initially described in the master thesis’ initial proposal, and taking into account the good results obtained, there was a proposition to go further, performing a new practical approach with the designed tools. Since recognition of feedback evoked brain responses has an important potential in some rehabilitation therapies, the author also analyzed data of a neurofeedback training for cognitive enhancement carried out at Bit&Brain Technologies with healthy subjects. During that training feedback potentials of 5 subjects were recorded, and subsequently they were studied and classified in a similar way to the ones acquired with the initially designed protocol. viii Index 1 Introduction 1 1.1 Motivation ....................................... 1 1.2 Relatedwork ...................................... 2 1.3 Scopeofthework.................................... 3 2 Acquisition, Characterization and Classification of Visual, Auditive and Vibratory Feedback Potentials 5 2.1 Methods......................................... 5 2.1.1 Experimentation Paradigm and Protocol . . . . . . . . . . . . . . . . . . . 5 2.1.2 EEGRecording................................. 7 2.2 Results.......................................... 7 2.2.1 Characterization of Multimodal Feedback Potentials . . . . . . . . . . . . 9 2.2.2 Feature Extraction and Classification . . . . . . . . . . . . . . . . . . . . 11 3 A New Practical Approach: Recognition of Feedback Potentials within Neurofeedback Training 15 3.1 Methods......................................... 15 3.1.1 Neurofeedback Training Procedure . . . . . . . . . . . . . . . . . . . . . . 15 3.1.2 EEGRecordings ................................ 16 3.2 Results.......................................... 17 3.2.1 Characterization of Feedback Potentials . . . . . . . . . . . . . . . . . . . 17 3.2.2 Feature Analysis and Extraction . . . . . . . . . . . . . . . . . . . . . . . 18 3.2.3 Calibration and Classification . . . . . . . . . . . . . . . . . . . . . . . . . 19 ix 1. Introduction 1.2 Related work of compliance of the subject to the neurofeedback training. 1.2 Related work Brain-computer interfaces (BCI) are systems that allow to translate in real-time the electrical activity of the brain in commands to control devices. They do not require muscular activity and can therefore provide communication and control for people with devastating neuromuscular disorders such as cerebral stroke, spinal cord injury, among others. The most significant characteristic of these systems is the use of invasive or non-invasive methods to record brain activity. Invasive techniques require a clinic intervention to implant the electrodes on the cortex, while non-invasive methods place the sensors outside the skull (e.g., Electroencephalogram EEG) [8]. Research with invasive BCIs has shown accurate results, for instance, when controlling a robot device. However, these settings involve technical and clinical difficulties for humans, such as the maintenance of the electrodes in the cortex, infection risks, and damage to the brain [9]. On the other hand, much research in BCI has focused on non-invasive recording methods to be accessible for a wide range of users, and currently, electroencephalography (EEG) is the most widely used technology. Electroencephalography consists of the non-invasive recording of voltage differences over the scalp. Applying this technique in BCIs two types of electrical brain activity are mainly recorded: phase-locked and non-phase-locked activity. Phase-locked activity mainly comprises Event-Related Potentials (ERPs), which are synchronous changes in electrical activity produced as a response to an external event. Non-phase-locked activity comprises spontaneous brain activity and certain activity elicited by stimuli. However, that activity does not appear synchronously, but as an asynchronous variation in power at certain power band frequencies. Consequently to analyze ERPs it is useful to employ averaging techniques, as they reduce the signal-to-noise ratio of the potentials, while for asynchronous activity averaging techniques lead to an attenuation of the information [10]. Several types of ERPs have been described in the literature, and one broad category are the error-related potentials (ErrPs) [11]. Some of these potentials have been characterized when the human realizes that an error has been committed by himself in a choice reaction task [12], by observation of another user committing an error [13], or by observation of a computer [11] or a simulated robot [14] in interaction or operation tasks. Several of these potentials have also been successfully recognized online in the context of BCI [11, 15]. Another type of error-related potentials is produced when a subject is informed that he has committed an error (feedback ErrPs). Studies have shown that the typology of positive and negative feedbackelicited responses is different. Namely, an error-related negativity (ERN) occurs with higher amplitude (in absolute value) in the second case [1, 4]. Studies about error responses are always linked to a characterization or classification of the differences in mental processes elicited by correct and incorrect contexts. Classification of event-related potentials in different contexts with BCI is a wide field of research in neuro-engineering. Those systems try to calibrate a classifier with examples of one subject and recognize new examples of the same subject online [11, 14, 15]. Therefore an interesting new approach is to obtain a generic database of potentials of different subjects to train a system able to classify examples of a new person. Additionally, one emerging field of BCIs where recognition of feedback potentials is becoming 2 1. Introduction 1.3 Scope of the work of wide interest is neurofeedback. Neurofeedback therapies seek to produce improvements in the control of specific brain patterns through a learning process via feedback modulation. EEG Neurofeedback has been successfully applied in two main areas: (i) cognitive enhancement (e.g., in the improvement of the attention and/or working memory [7] or cognitive tasks [16]); and (ii) neurotherapy (e.g., treatment of neurological and psychological disorders, such as attention deficit/hyperactivity disorder [6], and epilepsy [17], among others). 1.3 Scope of the work The present master thesis comprises a deep study of brain signals elicited by presentation of feedback stimuli. To achieve this goal, the work has been divided into certain steps that comprise the sub-objectives that had to been attained:  Search and study of related bibliography that allows the author to acquire the required knowledge to begin the development of the work. To succeed in this research, it is necessary to acquire solid skills on brain-computer interfacing, biological signal processing, and machine learning tools.  Design of an experimentation protocol based on previous research, and Execution of several experimentation sessions in order to register the brain potentials evoked by the presentation of different modalities of feedback.  Development and application of signal processing tools that allow to work with the recorded EEG activity and characterize the brain potentials from a point of view of signal typology, and of source localization.  Evaluation of classification strategies that explore the mean performances that can be achieved using different approaches, such as training and testing with data of one subject alone, or different configurations of systems trained with data of various subjects that are tested with data of a different one. The leading plan of the present work was to study feedback event-related brain potentials elicited after stimulation via visual, auditive or tactile sensory inputs. Therefore a protocol that could provide each kind of those potentials had to be designed. Additionally, although it was not included in the master thesis’ initial proposal, there was a proposition to go further, performing a new practical approach with the designed tools, given the good results initially obtained. The author could access to data of a neurofeedback training protocol executed at Bit&Brain Technologies with healthy subjects, with the objective of enhance cognitive skills. Since recognition of feedback evoked brain responses has an important potential in certain neurorehabilitation therapies (e.g., in neurofeedback trainings), it was a good opportunity to work in a rehabilitation context and not only with a task-controlled protocol. Hence, a sixth step was included in the present work: analysis of brain potentials elicited by feedback stimuli given through a neurofeedback training. During that training, feedback potentials of 5 subjects were recorded, and subsequently they were studied and classified in a similar way to the ones acquired with the initially designed protocol. 3 1. Introduction 1.3 Scope of the work The present document is structured as follows: chapter 2 describes the study carried out about the potentials elicited by the multimodal feedbacks. Chapter 3 presents the expansion of the study, that was executed by analyzing feedback potentials elicited during a neurofeedback training. Finally, in chapter 4 are presented the conclusions and future lines of the work. 4 2. Acquisition, Characterization and Classification of Visual, Auditive and Vibratory Feedback Potentials In this chapter are presented the procedures followed for the acquisition, the characterization and the classification of feedback event-related potentials elicited by different sensory stimuli. This has been done through the design and employment of a BCI based on the paradigm proposed in [4]. This paradigm is a time-estimation task, where the user has to estimate an interval of 1 second, receiving a positive or a negative response depending on his/her accuracy. The BCI was configurable, giving the possibility to choose between three sensory feedback modalities: visual, auditive and vibrotactile. The two following sections correspond to the methods applied and the results obtained. 2.1 Methods In this section is shown the experimentation protocol employed to acquire the brain potentials, as well as the software and hardware used to record the EEG signals. 2.1.1 Experimentation Paradigm and Protocol The experimental protocol followed in this work is based on the proposed by Miltner in [4]. It consisted of a time-estimation task, where a subject received positive or negative feedback depending on his/her accuracy when trying to delimit a time interval of one second. The setting of the experiment was a person comfortably sat, observing a computer screen while the EEG was recorded. Each trial started with a visual cue to indicate that the subject had to press a button a given time later (1 second) and then, depending on the proximity to this time, a positive/negative feedback was given 0.6 seconds later. Three types of feedback modalities were integrated: visual, auditive, and vibrotactile. Visual feedback was given as a green tick (positive) or a red cross (negative). Auditive feedback stimuli were given through two speakers as an harmonious jingle (positive) or a low tone (negative). Vibrotactile feedback was given through five low-power vibrator devices assembled as a customized 5 2. Acquisition, Characterization and Classification of Visual, Auditive and Vibratory Feedback Potentials 2.1 Methods Figure 2.1: Diagram of the experimentation protocol used to acquire the multimodal feedback potentials. gadget, and controlled through an Arduino programmable board1. The gadget was placed on the left forearm of the subjects, fastened with an elastic band. It vibrated with a low intensity for positive feedback (one vibrator moving at half-power), or with high intensity for negative feedback (five vibrators moving at the same time). Note that vibratory feedback signals were not painful, and the intensities were clearly perceptible and distinguishable by all subjects. Five different subjects performed the experiments with each feedback modality (i.e., totaling 15 subjects). The participants were duly informed about the protocol. For each participant, the experiment was carried out in two sessions, where each session consisted of 30 blocks of 10 trials (Figure 2.1). The two sessions were performed in different days. In order to balance the number of potentials corresponding to positive and negative responses, a time window was computed dynamically every 10 trials taking into account all the previous results (the window was decreased as the subject’s time-estimation performance improved and increased as the performance deteriorated). The time window represented the precision required: the higher the time window was, the lower precision was necessary to receive positive feedback. With this strategy, approximately 150 positive and 150 negative feedback potentials were obtained for each participant and session. The experiments corresponding to the visual modality (five subjects) were performed at Universidad de Zaragoza, in a laboratory fully equipped to that end. These experiments served as a test of the BCI designed. The remaining experiments (ten subjects), corresponding to auditive an vibratory modalities, were performed at the installations of Bit&Brain Technologies, spin-off company of Universidad de Zaragoza. 1http://www.arduino.cc/ 6 2. Acquisition, Characterization and Classification of Visual, Auditive and Vibratory Feedback Potentials 2.2 Results Figure 2.2: Location of the electrodes selected (in green) over the 10/10 system. 2.1.2 EEG Recording The EEG was recorded using a commercial gTec system, consisting of 32 active EEG electrodes, connected via USB to a computer. The electrodes were placed at FP1, FP2, F7, F8, F3, F4, T7, T8, C3, C4, P7, P8, P3, P4, O1, O2, AF3, AF4, FC5, FC6, FC1, FC2, CP5, CP6, CP1, CP2, Fz, FCz, Cz, CPz, Pz and Oz, according to the international 10/10 system (Figure 2.2). The ground and reference electrodes were placed on FPz and on the left earlobe, respectively. The EEG was digitized at a sampling frequency of 256Hz, power-line notch-filtered to remove the 50Hz line interference, and bandpass-filtered between 0.5 and 10Hz. A Common Average Reference (CAR) filter was applied to remove any background activity detected on the signal. The signal recording and processing, the visual application, and the synchronization between the feedback stimuli and the EEG were developed within the BCI2000 platform [18]. 2.2 Results This section describes a characterization of the feedback potentials with the different modalities, and the results of the different classification strategies. The characterization has been made from three points of view: (i) grand averages, to assess the signal morphologies; (ii) source localization, to study the neural sources of the EEG; and (iii) psychology, to explore how the different modalities affected the subject’s perception. Different strategies were tested to evaluate the effects of intra-subject and inter-subject classifier calibration using the same or distinct feedback modalities. 7 2. Acquisition, Characterization and Classification of Visual, Auditive and Vibratory Feedback Potentials 2.2 Results (a) (b) (c) Figure 2.3: Grand average signals for positive, negative, and difference potentials. The X-axis indicates the time in milliseconds with respect to feedback presentation. Left, center and right columns correspond respectively to visual, auditive, and vibrotactile modalities. 8 2. Acquisition, Characterization and Classification of Visual, Auditive and Vibratory Feedback Potentials 2.2 Results 2.2.1 Characterization of Multimodal Feedback Potentials The first analysis carried out was the exploration of the grand averages of the cerebral responses. The grand average is computed as the mean of a set of EEG signals corresponding to one channel, time-locked at the moment the stimulus has been presented. Biological studies [19] suggest that, in order to obtain differences that can be compared, around one hundred of examples of each type of stimuli (in our case, positive and negative feedback signals for each of the three sensory modalities) are needed to sufficiently increase the signal-to-noise ratio, which is very low in these methods. The grand averages of channel FCz were computed for both types of responses (to positive and negative feedbacks) and for all participants, separately for each feedback modality (visual, auditive, and vibratory). Channel FCz was chosen since it is broadly used in ERP studies [1]. The results are displayed in Figures 2.3a-c along with the difference between negative and positive potentials (referred as difference potentials). The difference potentials are broadly used in neurophysiological studies, since they provide information about the cerebral error processing, and give a measure about differentiability between EEG signals corresponding to erroneous and non-erroneous processing [12]. Difference potentials evoked by visual and auditive feedbacks are similar in terms of peaks or components (the negative components in the auditive case are more pronounced). The difference potential of vibratory feedback shows a significantly different behavior, presenting a notably lower amplitude, and a more oscillating morphology. A deeper analysis of vibratory-evoked responses revealed that the grand averages of each subject presented considerable different typologies. Hence, the total average suffered a great reduction in amplitude. That was not the case in the other two modalities, whose morphologies were substantially similar between subjects. However, the negative component at approximately 500 ms after the stimulus remained similar between the three modalities. Another important information to be studied when analyzing event-related potentials is the identification of the brain sources that origin the acquired EEG. The brain cortex is divided into 52 regions, according to its cytoarchitecture, called Brodmann Areas (BA) [20]. Previous studies have shown that erroneous mental processing (measured as the difference potential, or the differential activity between negative and positive responses) activates the Anterior Cingulate Cortex (ACC, Brodmann Areas 24, 32) [1]. In order to estimate the neuronal sources responsible for the generation of the EEG potentials recorded, the EEG Source Localization problem, also called the inverse problem, has to be resolved. In this work source localization tool sLoreta [21] was used. Figures 2.4a-c display the results obtained with this tool. For each feedback modality, the Anterior Cingulate Cortex was activated at some points in time of the difference potentials. In particular, visual and auditive modalities, analyzed on its positive peak (approximately 400 ms after feedback) showed a clear focus of activity on ACC, with best matches being Brodmann Areas 24 and 32 (visual), and 32 and 6 (auditive). Vibratory modality required extensive analysis. When using sLoreta with the average of all subjects, no focus of activity was found in the areas related to error processing. However, when exploring the individual averages of the participants, ACC was activated for some grand average peaks (see Fig. 2.4c that shows activation at a negative peak for participant 3). A detailed 9 2. Acquisition, Characterization and Classification of Visual, Auditive and Vibratory Feedback Potentials 2.2 Results inspection of the EEG measurements revealed that potentials elicited by vibrotactile feedback had great variations in time latencies, which had an impact on the grand averages. This could be due to the fact that this modality was not obviously associated to what a subject understands as a erroneous/correct stimulus and to differences in reaction time. These dependencies could have an impact on the cognitive process involved in the potentials and thus in the event-related response obtained in the EEG. (a) (b) (c) Figure 2.4: Source localization main activities found for the average of the subjects in visual and auditive modalities in the main positive peak, and for participant 3 in vibratory modality in its main positive peak. Left, center and right columns correspond respectively to visual, auditive, and vibrotactile modalities. Additionally, all the participants filled out a Likert scale questionnaire to further analyze these effects from a psychological point of view. The results indicated that the visual feedback group showed a higher satisfaction level than the other groups, being vibrotactile feedback the worst valued. Subjects belonging to the vibrotactile group reported that the two vibration intensities did not evoke directly the associated positive or negative information. 10 2. Acquisition, Characterization and Classification of Visual, Auditive and Vibratory Feedback Potentials 2.2 Results 2.2.2 Feature Extraction and Classification An r2analysis was carried out to find the spatiotemporal areas with the most significant differences between positive/negative conditions. Figures 2.5a-c display the average r2coefficients for the three feedback modalities. This analysis indicated that the most representative information to differentiate between the two conditions was found in the fronto-central channels (FC1, FC2, CP1, CP2, Fz, FCz, Cz and CPz), at time instants between 200 and 600 milliseconds after feedback presentation. The raw signal for these channels at the time interval selected was downsampled to 64 Hz, and normalized to the range [0-1]. Eight feature vectors (one per selected channel) were concatenated and the total feature vector was composed of 208 features. Note that r2coefficients were lower for vibrotactile feedback, suggesting that potentials in this modality were not very discriminative. Two types of classification results are reported. The first type is an offline classification validated by cross-validation to study the generalization properties. The second type is constituted of four strategies to achieve an online classification more adapted to the online usage of this technique in real settings. The selected classifier was a Support Vector Machine (SVM), as it has been used previously to recognize event-related potentials [22]. The SVM was used with a radial basis function kernel and a bandwidth dependent on the number of features. The offline classification performance is first discussed as a performance benchmark between subjects and modalities. Results were obtained using a 10-fold cross-validation strategy for each subject, using the datasets from the two sessions together. Results are presented in Table 2.1. The results showed similar accuracies between modalities and subjects. The mean performance for all types of potentials was over 70% in all modalities, and the average accuracies of the three modalities were higher than 75%. Also, all participants achieved similar classification rates, suggesting that the recognition of potentials is stable across participants. Table 2.1: Classification rates Visual Auditive Vibratory Pos.F. Neg.F. Avg. Pos.F. Neg.F. Avg. Pos.F. Neg.F. Avg. P.1 72.61 79.97 76.38 89.38 90.40 89.32 70.99 61.97 66.69 P.2 74.21 78.32 76.32 76.70 68.43 73.34 78.80 68.75 73.97 P.3 80.26 83.91 82.01 91.75 81.10 85.95 80.74 73.77 77.02 P.4 87.27 76.06 81.66 88.45 87.10 87.62 86.02 79.41 82.67 P.5 73.75 80.66 76.99 81.33 74.23 78.33 89.54 78.66 84.33 Average 77.62 79.78 78.67 85.52 80.25 82.91 81.22 72.51 76.94 The online classification of feedback potentials is now analyzed. Four different strategies were evaluated, differing in the data used to train the classifier (i.e. to calibrate the BCI system). This calibration was refined incrementally after a given number of examples had been classified, by adding them to the training data. In the experiments, recalibration was accomplished using 10% of the remaining data (30 trials for the first strategy and 60 for the remaining). The first strategy corresponded to a user-specific calibration session (the first experimentation session) and studied the online classification submitted to incremental re-training in the second session. Figure 2.6a represents the average classification results obtained with this strategy for the three different modalities individually and their average. 11 3. A New Practical Approach: Recognition of Feedback Potentials within Neurofeedback Training 3.2 Results have been proved to be not accurate with such a low number of electrodes [24], tools sLoreta [21] and Dynamo [25] were used. The results obtained suggested that Brodmann Areas 24, 32 (ACC) and 23, 31 (PCC) were systematically activated in practically all subjects. These areas agree with previous studies of the same field [1, 12], and with the results presented in previous chapter. 3.2.2 Feature Analysis and Extraction The r2analysis was performed to identify the temporal and spatial areas with most statistical differences between positive and negative feedback potentials [8]. Figure 3.3a shows the r2 coefficient for each channel during one second after feedback presentation, averaged for the five participants. The results showed that the most significant information belonged to fronto-central channels at the time window [200-600] ms. Thus, this information was used to select the features used throughout the remainder of the work: channels FCz, C1, C2 and Cz during time window [200-600] ms were selected and subsampled to 64Hz. The data of each channel was concatenated in a single feature vector of dimension 104 for each response. One important aspect of this neurofeedback setting was that non-stationarities were present in the EEG, due to the fact that it was carried out in several sessions. The feature vectors were used as a reference to analyze this issue. Then the feedback potentials were grouped according to the acquisition day, totaling ten groups. For each participant, a r2analysis was carried out for the feature vectors belonging to each training session, in order to study how this measurement changed throughout days. If one session did not present examples of one class, the r2coefficients were assigned a null value. Figure 3.3b depicts the evolution of r2values in the selected features throughout the 10 sessions for participant 2. Note that some features remained (a) (b) Figure 3.3: (a) r2Coefficients computed for all channels, averaged for all participants. (b) Evolution of r2 coefficients computed for selected feature vectors throughout sessions, corresponding to Participant 2. 18 3. A New Practical Approach: Recognition of Feedback Potentials within Neurofeedback Training 3.2 Results stable throughout sessions, but others varied. This indicates that the representative features differentiating classes in one session might not be the same in other sessions. Another important aspect was the variability of features with the subjects. Principal Component Analysis (PCA) was applied to the feature vectors of each participant and to the pooled dataset of all participants to decorrelate the features [26]. Then, a second r2analysis was performed to identify the two decorrelated features with most discriminant information between classes. Finally, the distributions of both features were displayed for positive and negative feedback responses, separately for each subject as well as for the five subjects grouped (Figure 3.4). The figure suggested that the distributions of both types of potentials for each subject were separated. Additionally, the results showed that the distributions were different for each subject, and that the subjects seemed to be divided into two clusters. Features of subjects 2, 3 and 4 presented positive responses rightwards, while subjects 1 and 5 and the pooled dataset presented responses leftwards. Thus, using information of certain subjects to model the responses of other subjects could be unproductive, as the relative position of distributions can be interchanged. Figure 3.4: 2D distributions of the two most representative features to discriminate between positive feedback (PF) and negative feedback (NF), computed for each participant separately and for the pooled data. 3.2.3 Calibration and Classification The last step was to analyze the classification possibilities of the potentials. Note that there was a great difficulty imposed by the fact that the potentials were acquired during a neurofeedback training, which was not explicitly designed as a standard calibration session in BCI systems. This resulted in a low and unbalanced number of examples in addition to the inherent nonstationarities among sessions. 19 3. A New Practical Approach: Recognition of Feedback Potentials within Neurofeedback Training 3.2 Results The feature vectors defined in the previous section were normalized in the range [0-1] and used for classification. Two different classifiers were used: Linear Discriminant Analysis (LDA) and Support Vector Machine (SVM). Both are representative examples of linear and non-linear classifiers (commonly used for EEG classification [22]). The inclusion of LDA classifier in this work was a step further, since the initial results obtained with SVM and the dataset available suggested that the non-linear classifier did not work so well as with the multimodal feedback potentials. Moreover, the classifiers were compared in two different contexts: (i) an offline analysis of the classification accuracy that could be obtained for each subject by cross-validation; and (ii) a more realistic setup of supervised online classifiers, which took into account previous information of the subject or other subjects. Table 3.1: Classification rates LDA SVM Pos. F. Neg. F. Avg. Responses Pos. F. Neg. F. Avg. Responses P1 72.46 60.78 67.50 91.30 62.75 79.17 P2 74.60 68.42 71.67 92.06 80.70 86.67 P3 61.76 65.12 64.17 41.18 89.53 75.83 P4 60.98 65.79 62.50 98.78 60.53 86.67 P5 61.76 51.92 57.50 97.06 61.54 81.67 Average 66.31 62.41 64.67 84.08 71.01 82.00 The offline performance was calculated using a leave-one-out cross-validation strategy, providing a measure of the maximum performance that could be obtained for each subject individually. This cross-validation strategy was chosen due to the fact that the number of potentials available was considerably lower than in the previous study, and using 10-fold cross-validation could provide biased classification results. For each subject, being nthe number of examples available, nclassifiers were trained with n-1 potentials each. The potential left out in each case evaluated the performance of the trained classifier. These measures give information about the best recognition results that the classifiers are able to perform, using all the data available, without having into account any temporal information. Table 3.1 presents the mean accuracy for positive and negative responses as well as the average for both types of responses. Rows P1 to P5 show the results obtained for each of the five participants for each condition, while the Average row corresponds to the mean of them. The column Avg.Responses represents the global performance for each subject, i.e., the percentage of correctly detected feedback potentials. Note that, as the number of examples for each class was not balanced, this value did not represent the mean between positive and negative accuracies. Results for SVM show that all subjects presented an average classification accuracy higher than 75%, and the average result for the five participants reached 82%. However in four subjects, one of the classes provided a poor classification performance, probably due to the low number of examples available. LDA results were notably lower in practically all cases, indicating that the non-linear classifier SVM worked better to model this classification problem with all the available data. The complementary analysis was the supervised online classification. Sequential supervised classifiers were trained to study the behavior of LDA and SVM when dealing with a low number 20 3. A New Practical Approach: Recognition of Feedback Potentials within Neurofeedback Training 3.2 Results of examples and when training with several subjects to classify a different one. Three different strategies are compared in this subsection: Incremental Strategy The data obtained from sessions [1 . . . i −1] was used to train the classifier, whereas the test set was formed by the i-th session (i∈[2 . . . 9]). This strategy was performed for each subject separately. The results for the SVM classifier are shown in Fig. 3.5a. Several observations can be made. Firstly, the first session did not have previous data to train the classifier. Secondly, the SVM results provided high accuracies in almost all sessions. Thirdly, the LDA classifier required almost all sessions to estimate the empirical covariance matrix and consequently is not reported here. Inter-Subject Strategy In order to address the shortcomings of the previous strategy, the classifier was initially trained with information of four subjects. Then, the sessions of the fifth subject were tested sequentially using that classifier. The five possible combinations of subjects were tested. The average performance for all subjects is plotted in Figure 3.5a-b (dotted lines). The results show better than random accuracies since first session. Thus, it is possible to use this classifier since the beginning of the neurofeedback training. SVM provided lower accuracies than the incremental strategy for the remaining sessions. LDA showed higher accuracies than SVM. Furthermore, it exceeded the offline maximum accuracy obtained previously in almost all sessions. Inter-Subject plus Incremental Strategy Here the two previous approaches were combined. The classifier was initially trained using the examples from four subjects and then retrained after each session with the fifth subject (dashed lines in Fig. 3.5a-b). This strategy implemented using SVM did not produce any improvements with respect to the previous strategy. In opposition, LDA improved previous strategy in all sessions from third one forwards. Furthermore, it showed a continuous growing trend that began on fifth session. In summary, the best results for SVM corresponded to the incremental strategy, while the performance degraded almost to randomness for the second and third strategies. However, it is important to remark that even the results of the best strategy obtained with SVM were not stable with the feedback potentials of this study. That motivated the exploration of LDA classifier. LDA worked better than SVM in the second and third strategies, in opposition to the results obtained for the offline classification. Moreover, sequentially-retrained LDA classified almost all sessions with a better performance than the maximum offline accuracy. 21 3. A New Practical Approach: Recognition of Feedback Potentials within Neurofeedback Training 3.2 Results (a) (b) Figure 3.5: Mean performance for the three classification methods averaged for all the participants using SVM (a) and LDA (b). The shadowing represents the standard error. The x-axis indicates the session classified in each point, and the percentage of the total sessions added to the incremental classifiers is indicated in brackets. The y-axis shows the accuracy obtained when classifying each session. Horizontal lines in both plots represent the maximum accuracy obtained previously through leave-one-out cross-validation. The solid line in (a) shows the average performance of the per-subject incremental classifiers. Dotted lines show the performance of the classifiers built with data from the remaining four subjects. Dashed lines show the performance of the classifiers built with data from four subjects, retrained each session with data of the fifth subject. 22 4. Conclusions and future work This master thesis studied brain potentials evoked after the presentation of feedback in different sensory modalities. The author used an ad-hoc protocol to obtain the potentials of visual, auditive and tactile modalities, and also studied the potentials obtained during a neurofeedback training which provided audio-visual feedback to the subjects. This practical approach of studying brain potentials recorded during a protocol not specifically designed for ERP acquisition goes a step further respect to standard studies with brain-computer interfaces. Extra difficulties are found with this approach, such as the number of examples available is low and unbalanced, and that these examples suffer variations between sessions. The results obtained for the characterization of the potentials agree with previous studies in this field. This study found similar signal typologies and neural origins than the obtained for other ERP responses [1, 4, 12]. For the multimodal potentials, comparisons of four different strategies for online classification were performed with a SVM classifier. The results showed that potentials vary across subjects and modalities. The best classification results were obtained through a user-specific calibration approach, although multiple-user calibration for each modality also provided reasonable performances. The classification results obtained for within-subject calibration agree with other studies of ERP classification [11, 14, 15]. For the potentials acquired within neurofeedback training, strategies of subject-specific and multi-subject calibration also have been explored. SVM and LDA classifiers were tested, finding that the offline classification performance is very similar to the obtained with the multimodal potentials using SVM. Online classification shows best results when working with LDA, obtaining a mean performance of about 80% in the last session. This value also agrees with the mean performance obtained for multimodal potentials with an incremental, subject-specific classifier. Future lines of this work are two-fold. On the one hand, more sophisticated techniques to improve generalization across feedback modalities can be explored, with the objective to increase the low performances obtained when combining modalities. On the other hand, prospective lines of this work have to go towards the development of metrics that connect feedback brain potentials with compliance and adherence of the subject to the neurofeedback training. 23 Bibliography [1] S. Nieuwenhuis, C.B. Holroyd, N. Mol, and M.G.H. Coles. Reinforcement-related brain potentials from medial frontal cortex: origins and functional significance. Neuroscience & Biobehavioral Reviews, 28(4):441 – 448, 2004. [2] K.P. Tee, C. Guan, K.K. Ang, K.S. Phua, C. Wang, and H. Zhang. Augmenting Cognitive Processes in Robot-Assisted Motor Rehabilitation. In Proc. of 2nd Biennial IEEE/RASEMBS International, 2008. [3] X. Perrin, R. Chavarriaga, C. Ray, R. Siegwart, and J. del R. Mill´an. A comparative psychophysical and EEG study of different feedback modalities for HRI. In International Conference on Human Robot Interaction, pages 41–48, New York, NY, USA, 2008. [4] W.H.R. Miltner, C.H. Braun, and M.G.H. Coles. Event-Related Brain Potentials Following Incorrect Feedback in a Time-Estimation Task: Evidence for a Generic Neural System for Error Detection. Journal of Cognitive Neuroscience, 9(6):788–798, 1997. [5] T. Hinterberger, N. Neumann, M. Pham, A. K˜ A ¼ bler, A. Grether, N. Hofmayer, B. Wilhelm, H. Flor, and N. Birbaumer. A multimodal brain-based feedback and communication system. Experimental Brain Research, 154:521–526, 2004. [6] J.F. Lubar, M.O. Swartwood, J.N. Swartwood, and P.H. O’Donnell. Evaluation of the effectiveness of eeg neurofeedback training for adhd in a clinical setting as measured by changes in t.o.v.a. scores, behavioral ratings, and wisc-r performance. Applied Psychophysiology and Biofeedback, 20:83–99, 1995. 10.1007/BF01712768. [7] David Vernon. Can neurofeedback training enhance performance? an evaluation of the evidence with implications for future research. Applied Psychophysiology and Biofeedback, 30:347–364, 2005. 10.1007/s10484-005-8421-4. [8] J.R. Wolpaw, N. Birbaumer, D.J. McFarland, G. Pfurtscheller, and T.M. Vaughan. Brain Computer Interfaces for Communication and Control. Clinical neurophysiology, 113(6):767– 791, 2002. [9] D.J. Mc Farland and J.R. Wolpaw. Brain-computer interface operation of robotic and prosthetic devices. IEEE Computer Society, pages 52–56, 2008. [10] J. Kalcher and G. Pfurtscheller. Discrimination between phase-locked and non-phase-locked event-related eeg activity. Electroencephalography and Clinical Neurophysiology, 94(5):381 – 384, 1995. 25 [11] P.W. Ferrez and J. del R. Millan. Error-Related EEG Potentials generated during simulated brain-computer interaction. IEEE Transactions on Biomedical Engineering, 55(3):923–929, 2008. [12] M. Falkenstein, J. Hoormann, S. Christ, and J. Hohnsbein. ERP components on reaction errors and their functional significance: A tutorial. Biological Psychology, 51:87–107, 2000. [13] H.T. van Schie, R.B. Mars, M.G.H. Coles, and H. Bekkering. Modulation of activity in medial frontal and motor cortices during error observation. Nature Neuroscience, 7:549– 554, 2004. [14] I. Iturrate, L. Montesano, and J. Minguez. Robot Reinforcement Learning using EEG-based reward signals. In IEEE Int. Conference on Robotics and Automation (ICRA), 2010. [15] G. Visconti, B. Dal Seno, M. Matteucci, and L. Mainardi. Automatic recognition of error potentials in a P300-based brain-computer interface. In Proceedings of the 4th International Brain-Computer Interface Workshop & Training Course, pages 238–243, 2008. [16] Benedikt Zoefel, Ren´e J. Huster, and Christoph S. Herrmann. Neurofeedback training of the upper alpha frequency band in EEG improves cognitive performance. NeuroImage, 54(2):1427–1431, January 2011. [17] M.B. Sterman. Basic concepts and clinical findings in the treatment of seizure disorders with EEG operant conditioning. Clinical Electroencephalography, 31(1):45 – 55, 2000. [18] G. Schalk, D.J. McFarland, T. Hinterberger, N. Birbaumer, and J.R. Wolpaw. BCI2000: a general-purpose brain-computer interface (BCI) system. IEEE Transactions on Biomedical Engineering, 51(6):1034–1043, June 2004. [19] T. Handy, editor. Event-Related Potentials A Methods Handbook. The MIT Press, 2005. [20] K. Brodmann. Vergleichende Lokalisationslehre der Grosshirnrinde in ihren Prinzipien dargestellt auf Grund des Zellenbaues. Johann Ambrosius Barth Verlag, 1909. [21] R.D. Pascual-Marqui. Standardized low resolution brain electromagnetic tomography (sLORETA): Technical details. Methods and Findings in Experimental and Clinical Pharmacology, pages 5–12, 2002. [22] F. Lotte, M. Congedo, A. L´ecuyer, F. Lamarche, and B. Arnaldi. A review of classification algorithms for EEG-based brain computer interfaces. Journal of Neural Engineering, 4, June 2007. [23] J.d.R. Mill´an. On the need for On-line learning in Brain-Computer Interfaces. In Int. Joint Conference on Neural Networks, 2004. [24] C.M. Michel, M.M. Murray, G. Lantz, S. Gonzalez, L. Spinelli, and R.G. de Peralta. EEG source imaging. Clinical Neurophysiology, 115(10):2195–2222, 2004. [25] J.M. Antelis and J. Minguez. Dynamo: Dynamic multi-model source localization method for eeg and/or meg. In Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE, 31 2010. [26] A. Hyvarinen, J. Karhunen, and E. Oja. Independent Component Analysis. Wiley Interscience, 2001. 26