Repositorio Institucional de Documentos

Trabajo Fin de Master Estudio de caracter´ısticas frecuenciales de los Potenciales de Error para el control en continuo mediante Interfaces Cerebro-M´aquina Jason Omedes LLorente Director: Luis Montesano del Campo Master de Ingenier´ıa Biom´edica Departamento de Inform´atica e Ingenier´ıa de Sistemas Escuela de Ingenier´ıa y Arquitectura Universidad de Zaragoza Septiembre de 2013 Acknowledgment I would like these lines serve to express my deep and sincere gratitude to all those who have collaborated with their help in the realization of this work, specially to I˜naki Iturrate, lab partner and friend who has guided me through this year and taught me all the knowledge necessary to develop this thesis. I am also extremely thankful to Luis Montesano, director of this research, for giving me this opportunity and all the suggestions to make me improve. I would like to extend my gratitude to my laboratory colleagues of the University of Zaragoza for their friendship and collaboration. A very special thanks with a deep sense of reverence, my gratitude towards my parents and family, who have always supported me and have been patients and encouraging. At last but not least gratitude to all of my friends who directly or indirectly helped me to complete this project. To everyone, thank you very much. 3 Estudio de caracter´ısticas frecuenciales de los Potenciales de Error para el control en continuo mediante Interfaces Cerebro-M´aquina RESUMEN El registro, an´alisis y procesado de las se˜nales el´ectricas generadas por el cerebro tiene aplicaciones en diversos ´ambitos como la medicina, la rehabilitaci´on o el entretenimiento. En los ´ultimos a˜nos el campo de las interfaces cerebro-computador (BCI) ha experimentado grandes avances incluyendo el control multi-dimensional de dispositivos. En este contexto, desde la Universidad de Zaragoza se ha trabajado en la utilizaci´on de informaci´on relacionada con los errores para proporcionar informaci´on de retro-alimentaci´on durante el uso de la BCI. En particular, se han utilizado los potenciales de error, un tipo de potencial evocado (ERP) que aparece cuando ocurre un evento no esperado. Las interfaces cerebro-computador, incluyendo aquellas basados en potenciales de error, utilizan informaci´on en el dominio del tiempo y requieren una fase de calibraci´on previa al control de un dispositivo. Esto implica una gran dificultad para el desarrollo de esta tecnolog´ıa ya que la se˜nal cerebral depende tanto del usuario, como del d´ıa o de la tarea a realizar. Aunque se ha demostrado que los potenciales de error son estables a lo largo del tiempo, trabajos recientes se˜nalan que existen diferencias en la respuesta cerebral en funci´on de la tarea a realizar, en funci´on de la dificultad al evaluar la tarea. Otra dificultad asociada a este tipo de se˜nales es la necesidad de tener un evento muy marcado en el tiempo, o trigger, para elicitar el potencial. Esto dificulta el uso de estos potenciales en situaciones de control realistas como por ejemplo un robot m´ovil. En este caso, no est´a claro cu´ando el usuario va a percibir un error y si se va a generar el potencial de error correspondiente. Los objetivos de esta tesis de M´aster son analizar la posibilidad de eliminar el trigger de este tipo de se˜nales 1) estudiando un nuevo tipo de caracter´ısticas en el dominio de la frecuencia y analizando si estas ´ultimas son m´as robustas ante variaciones en la latencia de respuesta del potencial de error; y 2) evaluando la capacidad de estas caracter´ısticas para proporcionar informaci´on de retroalimentaci´on durante el control en continuo de un dispositivo. Para ello, este trabajo se divide en tres partes: 1) Estudio y comparaci´on de la generalizaci´on de las caracter´ısticas temporales y frecuenciales de los potenciales de error cuando se hace transferencia entre tareas en protocolos con un marcador bien definido, es decir, acciones discretas. Refiriendose con transferencia a entrenar un clasificador con las caracter´ısticas extra´ıdas de una tarea y emplearlo para reconocer eventos en una tarea distinta. 2) Dise˜no de un protocolo (en pantalla) para el estudio de los potenciales en continuo (acciones continuas donde no existe marcador del evento, o si lo existe no se conoce d´onde est´a). Adquisici´on de datos de EEG con varios sujetos. Procesamiento de datos para analizar la presencia de potenciales de error y su detecci´on en continuo. 3) Dise˜no de un protocolo experimental para el control en l´ınea de un robot m´ovil mediante el uso de potenciales de error y su clasificaci´on en continuo. Experimentaci´on preliminar con varios sujetos y an´alisis de los resultados obtenidos. Finalmente, se ha llevado a cabo dentro de la empresa Bit&Brain Technologies, la implementaci´on de un sistema de visi´on que permite realizar seguimiento de atenci´on ocular a partir de la se˜nal del EEG sin restringir los movimientos de la cabeza. 5 On the usage of frequency features from Error Potentials to the continuous control of devices using Brain Machine Interfaces ABSTRACT Recording, analysis and processing of electrical signals generated in the brain has applications in several fields such as medicine, rehabilitation or entertainment. In the last years, the field of brain-computer interfaces (BCI) has achieved significant advances including multi-dimensional control of devices. In this context, the University of Zaragoza has worked on the usage of error-related information to provide feedback information during the use of the BCIs. In particular, it has been used the error potentials (ErrP), a type of evoked potential (ERP) that appears when an unexpected event happens. Brain-computer interfaces, including those based on error potentials, use information in the time domain and require a calibration phase prior to the control of a device. This implies a great challenge for the development of this technology since the brain signal is highly dependent on both the user and the day or from the task to execute. Although it has been shown that the error potentials are stable over time, recent works point out that there exist differences in the brain response in function of the task to be performed, depending on the difficulty in assessing the task. Another compromise associated with this type of signal is the necessity of having a very marked event in time (trigger) to elicit the potential. This hinders the use of these potentials in realistic control situations such as a mobile robot. In this case, there is not clear when the user will notice an error and whether it will generate the corresponding error potential. The objectives of this Master’s thesis are to analyze the possibility of removing the trigger from this type of signals by 1) studying a new type of features in the frequency domain and discussing whether these are more robust to latency shifts in the responses of error potentials, and 2) evaluating the ability of these features to provide feedback for the continuous control of a device. To this end, this work is divided into three parts: 1) Study and comparison of the generalization of the temporal and frequency features of error potentials when executing transference between tasks, in protocols with a well-defined trigger (i.e. discrete actions). Referring with transfer to train a classifier with the features extracted from a given task and later on uses it to recognize events in a different task. 2) Design of an experimental protocol (on screen) to study continuous potential generation (continuous actions without the existence of a trigger, or if it exists, it is not known where it appears). EEG data acquisition from multiple subjects. Data processing to analyze the presence of error potentials and its continuous detection. 3) Design of an experimental protocol for on-line control of a mobile robot through the use of error potentials and its continuous classification. Preliminary experimentation with various subjects and analysis of obtained results. Finally, it has been carried out within the company Bit&Brain Technologies, the implementation of a vision system that allows performing eye-tracking from the EEG without restricting the user’s head movements. 7 Index 1 Introduction 15 1.1 Motivation ................................... 15 1.2 Structure .................................... 17 2 Frequency features: Characterization, classification and generalization capabilities for ErrP 19 2.1 Methods..................................... 20 2.1.1 Datarecording ............................. 20 2.1.2 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.1.3 Electrophysiology analysis . . . . . . . . . . . . . . . . . . . . . . . 22 2.1.4 Feature extraction for classification . . . . . . . . . . . . . . . . . . 22 2.1.5 Methods for single-trial classification . . . . . . . . . . . . . . . . . 24 2.2 Results...................................... 24 2.2.1 Results of the electrophysiology analysis . . . . . . . . . . . . . . . 25 2.2.2 Classification results . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3 Continuous detection of Error Potentials 27 3.1 Methods..................................... 28 3.1.1 Datarecording ............................. 28 3.1.2 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.1.3 Electrophysiology analysis of time-locked error potentials . . . . . 30 3.1.4 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.1.5 Single-trial continuous classification . . . . . . . . . . . . . . . . . 31 3.1.6 Post-hoc electrophysiology analysis for the event-less condition . . 32 3.2 Results...................................... 32 3.2.1 Electrophysiology time-locked analysis . . . . . . . . . . . . . . . . 32 3.2.2 Classification of time-locked error potentials (condition 1) . . . . . 34 3.2.3 Classification of the event-less condition . . . . . . . . . . . . . . . 36 3.2.4 Post-electrophysiology event-less analysis .............. 39 4 Control of a robot using Error Potentials 41 4.1 Methods..................................... 42 4.1.1 DataRecording............................. 42 4.1.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 42 9 1. Introduction 1.1 Motivation potentials (ErrP) are one class of ERP that occur after the observation of erroneous events. ErrP can be measured in the area over the anterior cingulate cortex (ACC), which has been reported as the brain area involved in error processing [7]. Three different kinds of ErrPs have been discovered: the response ErrP, when the subject has committed an error in a choice reaction task [8]; the observation ErrP, by observation of another user committing the error [9]; and the interaction ErrP, when the user observes that a machine has performed an erroneous action [10, 11]. Error-related potentials are characterized by three distinctive morphological components identified as N2, P3 and N4. The name of these components is related to their sign (i.e. P=Positive, N=Negative) and the instant of time where they appear (i.e. 2 makes reference to 200 ms, and so on). Error potentials has been mostly studied in psychophysiology and neuroscience. For example, the N2 and P3 are related with errorrelated negativity (ERN), that has been observed in choice reaction tasks [8], which consist in making fast choices under pressure, thus inducing errors in user’s decisions; or the feedback related negativity (FRN), which has been reported in gambling tasks [7]. Also the N4 has been suggested to appear as a visual semantic mismatch [12]. On the other hand, BCIs have used ErrPs with different objectives such as correct the commands executed by a device [10], adapt the classifier [13] or as feedback [11, 14]. This thesis will focus on the study of feedback ErrP as a promising tool to evaluate the actions executed by a device, using the extracted information of the ErrP to modify the machine behavior through reinforcement learning algorithms. However, two main drawbacks of this type of brain signal are the necessity of a calibration phase prior to train a classifier, and the existence of fixed points in time to trigger the event. Calibration is required for each subject and task in order to deal with the large EEG sources of variability [6]. Regarding to on-line detection of ErrP under protocol that use discrete actions, successful single-trial recognition has been achieved. To detect ErrP, mainly characteristics of the temporal domain have been used based on the fact that these potentials are locked to a discrete event [11, 15]. However, even when the trigger is clearly marked, ErrP have slight variations depending on the performed task [16], resulting in a remarkable downfall in the classifier performance when the tasks in the calibration and execution phase are different. These issues limit the use of ErrP to discrete protocols (e.g. grids) with instantaneous actions and a clear onset. However, during the continuous actions of a robot, there may not exist a clear event that elicits the error potential. Indeed, it is not clear whether such a potential will appear and whether it can be detected on-line. Furthermore, calibrating such a system is not trivial due to the unknown instant at which the user detects the error. In this sense, two possible situations are: 1) the erroneous action is clearly marked (i.e. the device changes direction suddenly and in a sharp way) which implies that the trigger exists but its occurrence time is not known; and 2) the error is executed gradually (i.e.the device performs a smooth change in direction), thus removing the trigger completely. These issues make it difficult to find ErrPs when the subject is observing continuous feedback. 16 1. Introduction 1.2 Structure The objective of this thesis is to study the existence of error potentials under situations where the trigger (and thus the event) is either unclear or inexistent. To do so, this work is divided into two main parts: •The first objective deals with a standard experimental protocol based on discrete actions. Here, frequency features are proposed as a solution to overcome the latency variations when the tasks in the calibration and execution phase are different. •The second objective is to apply the frequency features as a way to deal with continuous actions. A novel experimental protocol is proposed to analyze the two possible cases where the trigger is either unclear or does not exist. This study is carried out for both virtual and real devices. 1.2 Structure This thesis is organized in three main parts that deal with the different objectives: characterization of frequency features, classification in a continuous protocol and control of a real mobile device. Chapter 2: This chapter introduces a study of frequency features used to train a classifier to automatically detect error-related potentials. The analysis is carried out using recorded data from 3 standard protocols based on discrete actions. As a result, an electrophysiological analysis of the signal and the classification performance in single-trial are reported. Results have been submitted and presented in the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC13) [17]. Chapter 3: In this chapter, an experimental protocol is designed to study the generation of error potentials in continuous tasks. A new protocol consisting of a virtual cursor that performs continuous actions is proposed to take into account the two possible conditions given by the uncertainty of trigger in continuous (the trigger is either unclear or does not exist). An analysis of the brain signal, along with classification results for this new method are presented. Results have been submitted to the IROS13 Workshop on Neuroscience and Robotics, and currently are under review. Chapter 4: In this chapter, a new protocol was designed, where the actions are not longer performed by a virtual cursor but for a real small mobile robot. Here, it is described the experimental set-up and the preliminary results obtained for this continuous device controlled using error potentials. These preliminary results were published in the IJCAI13 2nd Workshop on Machine Learning for Interactive Systems [18]. 17 1. Introduction 1.2 Structure Chapter 5: In this final chapter, the main results and achievements of this thesis are summarized and discussed. Additionally, a camera calibration (used for the robot location of Chapter 4) was employed to collaborate in a BCI European Project (CORBYS) carried out by Bit&Brain Technologies. The work developed for this clinical environment consisted in the implementation of an eye-tracker from the EEG signal. The main contribution was the addition of a small camera spotted on top of the users head that allowed the subjects to move their head while recording experiments. This is a great advantage, since most of the eye-tracker techniques require the head to remain at the same pose during the whole experiment. 18 2. Frequency features: Characterization, classification and generalization capabilities for ErrP In all BCI there is a calibration phase, prior to the device control, to learn the mapping from EEG activity to the control signals that operate the device. This calibration has to be carried out for each subject to deal with the large inter-user EEG variability [6]. In addition, a common procedure is to recalibrate the BCI for each new task to deal with the EEG variability [19, 20]. This is a large shortcoming of current BCI technology as the calibration is a tedious and boring process that may take between 30 and 45 minutes. Calibration is dependent on the EEG signal used for the BCI. Specially, the BCIs that rely on external cues, such as those using error potentials (ErrP) [21], have a good generalization among sessions but do not generalize between different tasks. This is because the amplitude and latency of their components are affected by factors such as spatial attention [22]; stimuli contrast [21]; the inter-stimulus interval [23]; userdependent factors such as age and cognitive capabilities [24]; and other cognitive aspects as the stimulus evaluation time [25, 21]. The on-line detection of ErrP relies on the fact that these signals are phase-locked to a trigger event [21, 26]. Thus, successful single-trial detection has been carried out mainly in the temporal domain [11, 27], despite of the single-trial temporal variability. A recent study showed that different tasks of these BCIs induce a phase change in the components of error potentials [16]. As a result, the use of temporal features significantly degrades the detection rate when the tasks in the calibration and execution phase are different. In this chapter, available data from the standard paradigm of ErrP elicited by discrete actions on different tasks will be used to study the usage of frequency features. These features are less sensitive to phase shifts, and therefore, they are a possible solution to overcome this degradation of the detection rate. Furthermore, delays in latency can be seen as slight variations on the trigger of the event. Thus, overcoming this situation is the first step before completely removing the trigger and step forward into a continuous protocol. 19 2. Frequency features: Characterization, classification and generalization capabilities for ErrP 2.1 Methods Nz CPz Fpz AFz Fz FCz Cz Pz POz Oz Iz Fp1 Fp2 AF3 AF4 AF7 AF8 F7 F5 F3 F1 F2 F4 F6 F8 F9 FT9 FT7 FC5 FC3 FC1 FC2 FC4 FC6 FC8 F10 FT10 A1 T9 T7 C5 C3 C1 C2 C4 C6 T8 T10 A2 TP10 P10 TP8 P8 PO8 O2 PO4 P2 P4 P6 CP2 CP4 CP6 TP9 TP7 CP5 CP3 CP1 P9 P7 P5 P3 P1 PO7 PO3 O1 Figure 2.1: Location of the 16 selected electrodes (green), reference (red) and ground (yellow). 2.1 Methods In this section it is shown the experimental protocol employed to elicit the Error Potentials, the hardware and software used to record the EEG signal, and a brief introduction to the different modalities to evaluate the results: electro-physiological analysis and single-trial classification. 2.1.1 Data recording The EEG was recorded using a commercial gTec system, consisting of 16 active EEG electrodes (Fz, FC3, FC1, FCz, FC2, FC4, C3, C1, Cz, C2, C4, CP3, CP1, CPz, CP2, and CP4 according to the 10/10 international system). The ground and reference electrodes were placed on AFz and on the left earlobe respectively, Figure 2.1. The EEG was digitized at a sampling rate of 256Hz, power-line notch-filtered to remove the 50Hz line interference, and bandpass-filtered between 0.5 and 10Hz, as ErrPs are known to be relatively slow cortical potentials [28]. A common average reference (CAR) filter, where at each time step the channel-wise mean is subtracted from each channel, was applied to remove any background activity detected on the signal. The data acquisition and on-line processing was developed under a self-made BCI platform. 20 2. Frequency features: Characterization, classification and generalization capabilities for ErrP 2.1 Methods 2.1.2 Experimental setup The goal of this set of experiments was to study the behavior of temporal and frequency characteristics of the brain signal in presence of discrete events that elicit ErrP and how these characteristics are modified when the users observe tasks with different cognitive workloads. The data from these experiments was already recorded from a previous study [16] carried out in collaboration with the ´ Ecole Polytechnique F´ed´erale de Lausanne (EPFL), where six healthy volunteers (five males and one female, mean age 27.33 ±2.73 years) participated in the study. Participants were instructed to observe movements performed by a device and evaluate them as correct when they were towards a target position and as incorrect otherwise, evoking non-error and error potentials. The participants were asked to restrict eye movements and blinks to specific resting periods. Three experimental conditions were designed (see Figure 2.2) to elicit the error potentials. The experiments presented different setups (and devices) with progressively higher cognitive workloads in order to assess changes in the potentials. All of these interfaces were composed by a device that performed correct/incorrect movements with the final goal to reach a desired target. The device executed random actions with approximately 20% probability of performing an erroneous movement. The time between two actions was random and within the range [1.7,4.0] s. The target position was randomly changed after 100 actions. Each experiment lasted ∼2.5 hours. They were always executed in the same order as presented above, with a time between sessions of 17.58 ±10.09 days. For each subject and experiment, approximately 800 trials (around 160 and 640 error and non-error potentials) were acquired. Experiment 1, Virtual Moving Square [11]: The first experiment consisted of a squared cursor (blue) that could perform two actions (move one position left or right) in a 1D grid with 9 different positions (marked by a horizontal grid) to reach the goal position (red). Experiment 2, Simulated Robotic Arm: This second experiment displayed a simulated robotic arm (Barrett WAM) with 7 degrees of freedom [29] that could perform four actions (move one position left, right, up or down) in a 2D grid with 13 positions (marked in orange) upon goal reaching represented in green. The device’s movements between two positions were continuous, lasting ∼500 ms. Experiment 3, Real Robotic Arm: The third experiment followed the configuration of the second experiment, but using a real robotic arm Barret WAM. The user was seated two meters away from the robot. A transparent panel was used to mark the positions, and the distance between two neighbor positions was 15 cm. 21 2. Frequency features: Characterization, classification and generalization capabilities for ErrP 2.1 Methods Figure 2.2: From left to right, experiments 1 to 3. 2.1.3 Electrophysiology analysis In this analysis the grand averages of the cerebral responses for both domains (time and frequency) were carried out. The grand averages are computed as the mean of a set of EEG recorded trials corresponding to one channel, time-locked at the onset of the stimuli. This computation is done to study the mental process [21]. For the time analysis, and following previous studies [10] the time-locked averaged potentials were computed for the error, non-error and difference conditions (error minus non-error averages), and for all participants, separately for each task (Experiments 1, 2 and 3) at channel FCz [30]. For the frequency analysis, the power spectral density (PSD) of each one-second trial was first computed using the Welch’s method with a Hamming window and a window overlap of 250 ms. Then, the error, non-error and difference average PSDs for all participants, separately for each task, were computed at channel FCz. Additionally, the r2discriminability test [31] between error and non-error conditions was computed for each channel and time instant (time analysis), and each channel and frequency component (frequency analysis) to find the spatio-temporal and spatio-frequency areas with the most significant differences between positive/negative conditions. r2=correlation(labels, features)2(2.1) where r2is the square of the correlation coefficient between the features and their class label. 2.1.4 Feature extraction for classification Two different sets of features were extracted: temporal and frequency. 22 2. Frequency features: Characterization, classification and generalization capabilities for ErrP 2.1 Methods 0 100 200 300 400 500 600 700 800 900 1000 −2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 Amplitude (µV) Time (ms) Exp1 Exp2 Exp3 0 5 10 15 −1 0 1 2 3 4 5 6 Power (µV2 / Hz) Frequency (Hz) Exp1 Exp2 Exp3 0 100 200 300 400 500 600 700 800 900 1000 −2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 Amplitude (µV) Time (ms) Error NonError Difference Time (ms) Channels 0 100 200 300 400 500 600 700 800 900 1000 16.CP4 15.C4 14.FC4 13.CP2 12.C2 11.FC2 10.CPz 9.Cz 8.FCz 7.Fz 6.C1 5.FC1 4.CP1 3.CP3 2.C3 1.FC3 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 0 5 10 15 −1 0 1 2 3 4 5 6 Power (µV2 / Hz) Frequency (Hz) Error NonError Difference Frequency (Hz) Channels 0 5 10 15 16.CP4 15.C4 14.FC4 13.CP2 12.C2 11.FC2 10.CPz 9.Cz 8.FCz 7.Fz 6.C1 5.FC1 4.CP1 3.CP3 2.C3 1.FC3 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 0 100 200 300 400 500 600 700 800 900 1000 −2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 Amplitude (µV) Time (ms) Error NonError Difference Time (ms) Channels 0 100 200 300 400 500 600 700 800 900 1000 16.CP4 15.C4 14.FC4 13.CP2 12.C2 11.FC2 10.CPz 9.Cz 8.FCz 7.Fz 6.C1 5.FC1 4.CP1 3.CP3 2.C3 1.FC3 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 0 5 10 15 −1 0 1 2 3 4 5 6 Power (µV2 / Hz) Frequency (Hz) Error NonError Difference Frequency (Hz) Channels 0 5 10 15 16.CP4 15.C4 14.FC4 13.CP2 12.C2 11.FC2 10.CPz 9.Cz 8.FCz 7.Fz 6.C1 5.FC1 4.CP1 3.CP3 2.C3 1.FC3 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 0 100 200 300 400 500 600 700 800 900 1000 −2.5 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 Amplitude (µV) Time (ms) Error NonError Difference Time (ms) Channels 0 100 200 300 400 500 600 700 800 900 1000 16.CP4 15.C4 14.FC4 13.CP2 12.C2 11.FC2 10.CPz 9.Cz 8.FCz 7.Fz 6.C1 5.FC1 4.CP1 3.CP3 2.C3 1.FC3 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 0 5 10 15 −1 0 1 2 3 4 5 6 Power (µV2 / Hz) Frequency (Hz) Error NonError Difference Frequency (Hz) Channels 0 5 10 15 16.CP4 15.C4 14.FC4 13.CP2 12.C2 11.FC2 10.CPz 9.Cz 8.FCz 7.Fz 6.C1 5.FC1 4.CP1 3.CP3 2.C3 1.FC3 0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05 EXPERIMENT 1 EXPERIMENT 2 EXPERIMENT 3 TEMPORAL DOMAINFREQUENCY DOMAIN DIFFERENCES Figure 2.3: Electrophysiology results for experiments 1 to 3. First row shows the error, non-error and difference grand averages for channel FCz, and the last column the difference average compared for the three experiments. Second row shows the r2test of the temporal signals (x-axis: time, and y-axis: channels). Third and fourth row show the PSD averages (for channel FCz) and the r2test of the frequency signals. For each r2plot, the squared zone represents the window used for the extracted features. 23 2. Frequency features: Characterization, classification and generalization capabilities for ErrP 2.2 Results Temporal Features: The EEG was [1,10] Hz band-pass filtered. Temporal features were the EEG voltages of each trial of eight fronto-central channels (Fz, FC1, FCz, FC2, C1, Cz, C2, and CPz) [27] within a time window of [200,800] ms (being 0 the stimulus onset) subsampled to 64 Hz, leading to a vector of 312 features. Finally, the features were normalized within the range [0,1]. Frequency Features: For each of the channels used in the temporal features, the PSD was computed on one second of EEG after the stimulus onset as explained in subsection 2.1.3. The frequency features were the power values of each channel from the theta band ([4,8] Hz) ±1 Hz (as previous studies suggested that the error potentials are generated within this band [26]), which led to a vector of 200 features. Finally, the features were normalized within the range [0,1]. 2.1.5 Methods for single-trial classification Previous studies showed that the usage of temporal features provoke a degradation of performance when training with one experiment and testing with another one (i.e. generalization) [16]. The objective of the present classification study was to analyze whether the frequency features or the combination of both (temporal and frequency) are robust enough to generalize among different tasks (experiments). Single-trial classification was carried out using support vector machine (SVM) with a radial basis function (RBF) kernel, as this classifier presents high accuracies when classifying ERPs [32] and error potentials in particular [27]. One important drawback of SVM is its sensitivity to imbalanced datasets. To avoid this drawback, the minority class (i.e. the error class) was oversampled to match the number of trials of the majority class (i.e. the non-error class) [33]. To study the generalization capabilities of the different feature sets, each task data was divided into a training and a test set composed by 50% of the data, each. The classifier was evaluated in two different conditions. First, the baseline accuracy was obtained by using the training and test sets of the same experiment Ej(denoted EjEj). Second, the classifier was trained using the train set of an experiment Eiand tested on the test set of another experiment Ej. The most relevant train-test combinations considered in this study were E1E2,E1E3, and E2E3following [16]. 2.2 Results This section compares the results obtained for the electro-physiological analysis and the single-trial classification when using temporal and frequency domain of the signal. 24 2. Frequency features: Characterization, classification and generalization capabilities for ErrP 2.2 Results 2.2.1 Results of the electrophysiology analysis Figure 2.3 (first row) depicts the error, non-error and difference grand averages, for the three experiments. The three difference grand averages of the error potentials have an early negativity and two broader positive and negative components, in agreement with other studies [11, 27]. However, in line with previous works, the latencies of these peaks varied among the three experiments [16] (see figure 2.3, up-right-most plot). For instance, the latency of the broader negative peak was of 426, 492 and 535 ms for experiments 1 to 3. This variation in latency is also visible with the r2metric (Figure 2.3 second row). Notice how the r2patterns of fronto-central channels present a time shift among experiments. Regarding the frequency analysis, Figure 2.3 third row depicts the error, non-error and difference PSD averages for the channel FCz for the three experiments. The difference averages were similar in the theta band for the three experiments (see Figure 2.3 third row, fourth column). This supports the fact that the main variation of the signals was due to latency differences, but not to amplitude differences (as described in [16]). The r2discriminability patterns were in the theta band as suggested in [26]. Notice that the r2values were progressively higher among experimental conditions. Despite there is not a clear reason of this increase in the r2, it could be due to: a user habituation to the protocols (since the three experiments were always executed in the same order from 1 to 3); or a higher cognitive workload that generated stronger error components with greater r2values. 2.2.2 Classification results Figure 2.4 depicts the baseline accuracies of EjEj, and the generalization accuracies of the combinations EiEjfor the temporal and frequency feature sets, and for the concatenation of both sets (c.f. subsection 2.1.4), averaged for all subjects. Regarding the temporal features, the baseline of each experiment had high accuracies, being on average 78.78%, 77.54% and 79.04% for Experiments 1 to 3. However, these features provoked an accuracy degradation when generalizing the classifier to another experiment, mainly due to the latency variations observed in the electrophysiology analysis. In fact, the mean accuracy dropped to 21.09%, 24.36% and 10.21% for the E1E2,E1E3and E2E3cases. On the other hand, the use of frequency features resulted on lower baseline accuracies than the temporal ones: 67.29%, 71.33% and 69.67% for Experiments 1 to 3. However, the accuracy drop was substantially lower when generalizing the classifier: 3.91%, 4.52%, and 3.44% for E1E2,E1E3and E2E3. For the baseline classifiers that use the temporal and frequency features, the accuracies presented very similar results to those obtained using the temporal features: 76.17%, 79.31%, and 77.64% for experiments 1 to 3. More interestingly, the generalization classifiers presented a substantial drop in the accuracies of 13.11%, 16.23% and 6.35%; but the absolute 25 3. Continuous detection of Error Potentials 3.2 Results detected after the change of direction (either smooth or abrupt), where considered as true positives (TP). When the error was not detected the trial was considered a false negative (FN). Those correct trials where the classifier did not detect any error were considered true negatives (TN). When an error was detected on correct trials, they were considered false positives (FP). To obtain a more intuitive representation of the performance achieved, we also displayed the trajectories followed by the device for each one of the four possible cases. The goal position of each trial has been repositioned to the center of the image for a better representation. 3.1.6 Post-hoc electrophysiology analysis for the event-less condition The analysis of Section 3.1.3 can only be carried out when the onset of the error potential is known (i.e. only for sharp changes). In order to evaluate whether the detection of error potentials in the event-less condition uses brain activity related to the error, we performed a post-hoc analysis using the output of sliding window classifier as an artificial onset of the error potential. Only correctly detected erroneous trials (i.e. true positives) were used in the analysis that was identical to that of Section 3.1.3. 3.2 Results 3.2.1 Electrophysiology time-locked analysis Figure 3.4 shows the error, correct and difference grand averages for channel FCz averaged for the two subjects in temporal and frequency domain. The difference average is characterized by a sequence of a positive peak at 150 ms, followed by a negative peak at 210 ms and two larger positive and negative peaks at 280 and 400 ms, and finally a positive peak at 600 ms. The topographic interpolation of these peaks can also be seen on Figure 3.4, showing how they are localized on fronto-central scalp areas. These results agree with previous studies using error potentials under standard conditions (i.e., discrete device actions) studies [11, 10]. Regarding to the frequency averages, a power increment in the theta band was observed for erroneous trials with respect to the correct ones, which also agrees with previous works [40]. Finally Figure 3.4, Bottom shows the source localization results for the most prominent negative peak (400 ms) of the timelocked difference grand average. The main activation areas were Brodmann areas 6 and 24 (premotor cortex and ventral anterior cingulate cortex), which is in accordance with previous works simultaneously recoding error-related activity and fMRI [41]. 32 3. Continuous detection of Error Potentials 3.2 Results Figure 3.4: Electrophysiology results of the experimental condition 1 (sharp turns). Temporal and frequency averages of error and correct trials plus the difference (error minus correct) for channel FCz and scalp topographies at the occurrence of the most relevant peaks for the average of the two subjects. Subject 1 Subject 2 Events 40 TRIALS Figure 3.5: Representative example of the sliding window results for the sharp condition. The error events are plotted in red indicating a change in direction, while the probability of detecting an error (pe) for each of the 2 subjects is plotted in blue. Black dots over the probability values indicate the time instant when the classifier detected an error (pe>0.8). 33 3. Continuous detection of Error Potentials 3.2 Results Subject 1 Subject 2 Events 40 TRIALS Figure 3.6: Representative example of the sliding window results for the event-less condition. In this case, the error events (plotted in red) indicate the duration of the turn. 3.2.2 Classification of time-locked error potentials (condition 1) Regarding to the results of applying the sliding window, Figure 3.5 displays the detection level obtained for both subjects in a representative round of the sharp experimental condition. Here, the 40 trials that compose the round are concatenated removing the inter-trial resting periods. In the shown example, it can be seen that most of the trials are properly classified. Also notice that the correctly detected ErrPs are delayed with respect the onset of the event (the sharp change in direction of the device). This delay was on average 867.13 ±99.33 ms after the onset of the erroneous action. This delay was expected, corresponding to the time needed for the appearance of the most relevant peaks and the maximum spectral power activation used as features. The performance rates achieved for the entire test set are depicted in Table 3.1. It can be observed that the number of false positives was reasonably low, which was the main objective of setting a high threshold value (pe>0.8). At the same time the number of erroneous trials detected as such (true positives), reached 64%. This value was around 10% less performance than those obtained with standard protocols using discrete actions [11, 34]. Table 3.1: Confusion matrix containing the performance rate for the Experiment 1 Actual Class Correct Error Predicted Class Correct TN = 89.09% FN = 36.00% Error FP = 10.91% TP = 64.00% Finally, Figure 3.7 displays the trajectories executed by the device according to their classification. Here, it can be seen that correct trials are mostly well detected independently of the direction and distance covered by the device, and only few of them are detected as erroneous. More interestingly, the number of erroneous trials not detected was higher. This was done this way since it was preferable to miss the detection of an error than detect errors where was not intended. Additionally, it can also be observed that many of these trials detected as correct end up very close to the actual goal, which 34 3. Continuous detection of Error Potentials 3.2 Results Detected as CORRECT being CORRECT Detected as CORRECT being ERROR Detected as ERROR being CORRECT Detected as ERROR being ERROR Figure 3.7: Confusion matrix of the trajectories performed by the device (black) during the sharp condition. The goal positions have been reprojected to the center of the image (red). The starting position of the cursor with reference to the goal is marked in blue. 35 3. Continuous detection of Error Potentials 3.2 Results lead us to think that the subjects may have not interpreted them as erroneous. 3.2.3 Classification of the event-less condition The process to analyze this section is similar to the one followed in the previous section 3.2.2. Regarding to the results of applying the sliding window, Figure 3.6 displays the detection level obtained for both subjects in a representative round of the smooth experimental condition. Once again, the 40 trials composing the round were concatenated after the removal of the inter-trial resting periods. In the example, it can be observed that the classification is carried out successfully, properly detecting most of the events. In this case, it is also noticeable the presence of a delay for correctly detected ErrPs. However, since there is not a clear onset that elicits the error potential, this delay cannot be computed. Nonetheless, it was possible to compute the delay of detecting an error after the curve finished, which was 297.22±213.72 ms on average. The larger standard deviations obtained indicated that the moment of error detection had larger trial-to-trial variations. On the other hand, assuming a similar error delay as the obtained in the previous case (867.13 ms), the ErrPs always appeared at random points within the radial movements. Thus, the potentials were not elicited at the beginning or the end of the radial turn, but rather depended on when the users became aware of the error. The performance rates achieved for the entire test set are shown in Table 3.2. It can be observed that the number of false positives was even lower than for the previous case, which may be caused by the extrapolation between training and testing data. Surprisingly, the number of erroneous trials detected as error (true positives), reached 67.33%, which is a slightly higher than in the previous case. Finally, Figure 3.8 displays the trajectories executed by the device according to their classification. Once again, it can be seen that correct trials have a similar behavior to the previous condition, and comparable detection rates are achieved. Once more, it can be observed that many of the erroneous trials detected as correct have their trajectories very close to the goal, which could be due to the subjective user’s interpretation. Table 3.2: Confusion matrix containing the performance rate for the Experiment 2 Actual Class Correct Error Predicted Class Correct TN = 97.15% FN = 32.66% Error FP = 2.85% TP = 67.33% 36 3. Continuous detection of Error Potentials 3.2 Results Detected as CORRECT being CORRECT Detected as ERROR being CORRECT Detected as CORRECT being ERROR Detected as ERROR being ERROR Figure 3.8: Confusion matrix of the trajectories performed by the device (black) during the second experimental condition. 37 3. Continuous detection of Error Potentials 3.2 Results −1200−1100−1000 −900 −800 −700 −600 −500 −400 −300 −200 −100 0 −8 −6 −4 −2 0 2 4 6 8 Time (sec) Amplitud (µV) 2 4 6 8 10 12 14 −2 −1 0 1 2 3 4 5 6 7 8 Freq (Hz) Power (µ V2/Hz) Error Correct Difference Error Correct Difference -5 0 5 Figure 3.9: For the event-less condition, temporal and frequency averages of error and correct trials plus the difference (error-minus-correct) for channel FCz and scalp topographies at the occurrence of the most relevant peaks for the average of the two subjects. 38 3. Continuous detection of Error Potentials 3.2 Results 3.2.4 Post-electrophysiology event-less analysis Figure 3.9 shows the grand average of error (as detected by the sliding window classifier) and correct trials plus the difference (error minus correct) for channel FCz averaged for the two subjects, both in temporal and frequency domain. Notice that the signal is referenced to the instant of time where the ErrP has been detected. Thus, the peaks analysis leads to a first positive peak that appears -750 ms before the detection of the ErrP. This positive peak is analog to the positive peak that appeared at 150 ms after the onset in the previous condition. The following observed positive peaks are found at -570 and -300 ms, while negative peaks are seen at -640 and -500 ms before the ErrP detection. All these peaks correspond to the positive peaks located at 280 and 600 ms; and to the negative peaks found at 210 and 400 ms respectively, observed in Figure 3.4. Furthermore, the scalp topographies of the described peaks had a similar morphology to the ones observed in the previous condition. Nonetheless, despite the averages and topographies resembled a similar morphology to the one obtained in the standard electrophysiology analysis, the ErrP amplitudes in this case were lower, most likely due to an inaccurate signal averaging. Regarding the frequency information of the averaged signals, a slight power increment in the theta band was present for the erroneous trials. Finally, Figure 3.9(Bottom), displays the source localization results at the most prominent negative peak (-500 ms). The brain activity corresponding to this peak was located at the Brodmann area 6 and 32 (premotor cortex and dorsal anterior cingulate cortex). In sum, these results suggest that the error potentials are elicited within continuous motions performed by a device, even when there is not a clear event that trigger them. 39 4. Control of a robot using Error Potentials When learning complex tasks, robots are usually faced with vast action-state spaces which are difficult and expensive to explore without knowledge of the structure of the environment. Furthermore, there exist hazardous regions or configurations that should be avoided since they may be dangerous for the robot or people around the robot. Humans are naturally aware of the intrinsic structure and domain knowledge of a task and, consequently, they can provide feedback during robot operation for learning or control purposes. EEG-based brain-computer interfaces (BCI) have been proposed in the last years as a way of communicating with virtual or real devices using only brain activity [6]. A promising cognitive EEG signal are the, here studied, error potentials (ErrP). These potentials appear when the user perceives an error, or more interestingly, when he observes a machine committing an error [42, 11]. These facts make ErrP natural candidates as feedback for a robotic device [27]. However, these signals have only been studied using locked stimulus (clear visual cue synchronized with the EEG) which makes their detection easier than during the continuous operation of the robot. This chapter proposes a protocol that uses EEG error potentials as feedback for controlling a robot executing continuous actions. The detection process of ErrP in continuous is based on the same idea of the previous Chapter 3. In this case a small robot (e-Puck, [43]) performed a target reaching task. During the experiment, the role of the user was to evaluate the robot movements as correct or incorrect, while the device tried to learn and reach the intended user’s target. In order to cope with the limited information provided by ErrP, a shared-control strategy based on the inverse reinforcement learning was used. Here, the robot assigns a specific probability to a set of possible targets, and updates their probability of being the desired goal according to the feedback information extracted from brain activity during robot operation. 41 5. Conclusions and Future Work The on-line detection of ErrP relies on the fact that these signals are phase-locked to a trigger event. As a result, BCI applications developed with these potentials have the necessity of being generated via external and clear events. Furthermore, even when the trigger is clearly marked, ErrP have slight variations depending on the performed task, which may result in a degradation of performance. In particular, previous studies [16] have reported a shift in latency induced by an increased cognitive workload of the executed task. These variations in latency can be interpreted as uncertainty on the trigger that marks the event. To cope with these issues, in this master thesis it has been proposed to use frequency features in BCI based on error potentials. Interestingly, in chapter 2, it has been shown that even thought the traditional features extracted from the temporal domain have a high performance in phased-locked classification within the same task, its detection rate drops significantly when dealing with inter-task classification. On the contrary, it has been reported that there exists a significant power increase in the theta band (as mentioned by other studies [26]) that allows to generalize better among experiments. Nonetheless, the concatenation of temporal and frequency features seems to obtain the best of both domains. Regarding the uncertainty of the trigger, a new experimental protocol based on continuous actions was designed in Chapter 3. In this experiment it was intended to elicit error potentials through a device performing correct and erroneous continuous actions under two different situations. The objective of this protocol was to study the ErrP generation and characterization when the trigger was unclear or inexistent. The results obtained for the proposed experimental protocol show that the error potentials appear when the user monitors a target reaching task and that they can be detected in single trial. Furthermore, the electrophysiology analysis of the recorded signals indicated that the main components of the error-related potentials were similar to previous studies and they had their origin in the anterior cingulate cortex. In the last part of the present work, it has been introduced the experimental setup to transfer the knowledge obtained from the previous experiments into a real mobile robot. Even though some adjustments are required in order to improve its performance, some preliminary results have been presented indicating that it is possible to control a device 49 5. Conclusions and Future Work during continuous motion. These promising results are a first step towards the use of this type of cognitive information to control or teach robotic devices in realistic and complex tasks. There are plenty of opportunities for future work. First, we are currently extending the study to more users and more error conditions. Second, further studies are required to understand whether the error potential appears every time an error is detected or the frequency features allow detecting errors even when no potential is present. Finally, the long term goal is to understand what cognitive information related to error perception can be decoded and incorporated in a BCI to control a robotic device in realistic scenarios. 50 APPENDIX 51 52 Appendix A Bit&Brain Technologies A.1 EEG eye-tracker Over the last years, the issue of tracking eye movements have been addressed by different techniques such as electrooculography (EOG), corneal reflection, pupil tracking, or reflection imaging [45]. These methods vary in resolution, range, tolerance to head motion, ease of use, invasiveness, and cost. Thus, the application clearly determine the method to use. A recent alternative has been proposed in [46] consisting of a method to track a subject’s eye fixations by recording activity from four EOG electrodes around the eyes and complemented with a few EEG electrodes on the scalp. This method has the advantage of being relatively unobtrusive and inexpensive. In addition, this technique can be used when the subject is wearing special eye wear such as glasses or contact lenses. Furthermore, this method provides good resolution, with eye gaze estimations between 1 and 2 degrees, approximately. However, it lacks from tolerance to head movements. For the users, it is natural to move their head together with the eyes in the gaze direction. These unconscious movements cannot be restrain from being executed at some degree. In this case, a slight difference in angle of the eye movement due to the head adjustment turns into a large error when it is translated into the screen coordinates located at a further distance. In this context, Bit & Brain Technologies were developing a similar system under the scope of the Cognitive Control Framework for Robotic Systems (CORBYS) European project, using exclusively frontal EEG channels and dispensing of the EOG electrodes. As in the previous work, they had the problem of head movement restriction. In this sense, the work developed by the 3 weeks of internship was focus on the implementation of a way to solve this problem. Here, it is proposed the use of a camera spotted on top of the subjects head to record the monitor at every instant. From the recorded video, 53 A. Bit&Brain Technologies A.2 Methods 1 2 3 4 5 6 marker cursor pattern Figure A.1: (Left) Squared virtual cursor and an example of a possible pattern (trajectory) which the user has to follow with his eyes. The 4 markers are located at the corners of the image. (Right) Marker designed for its good properties to be detected. homographies were computed to obtain a virtual rectification of the screen according to the slight head movements performed by the user. A.2 Methods A.2.1 Data recording The EEG was recorded using a commercial TMSi system, consisting of 6 active EEG electrodes (Fp1, Fpz, Fp2, F7, Fz and F8 according to the 10/10 international system). The ground and reference electrodes were placed on the left wrist and on the left earlobe respectively. The EEG was digitized at a sampling rate of 256Hz, power-line notch-filtered to remove the 50Hz line interference, and bandpass-filtered between 0.05 and 15Hz. An smooth average moving filter was applied and blinks were removed by threshold. The data acquisition and on-line processing was developed under a self-made BCI platform. A.2.2 Experimental setup From a known set of eye movements it is possible to calibrate the associated EEG activity across the scalp, and this way be able to decode eyes fixation from changes in the electrical potentials. During calibration, subjects have to follow with their eyes the trajectories performed by a virtual cursor. An example of the trajectory pattern executed by the cursor can be seen in Figure A.1(left). To obtain a representative sample of brain signal, a set of trajectory patterns are displayed several times. The recordings of brain 54 A. Bit&Brain Technologies A.2 Methods activity are used to map the different pairs of xand ycoordinates of the screen with their respective potentials. The correspondence between eye movements and changes in the electrical potentials at the electrode sites are related to the x,ycoordinates using a linear regression filter. Once the filter is computed, a new sequence of trajectories can be displayed and the coordinates xand ywill be estimated from the corresponding brain activity. In the standard protocol, the subjects would be asked to restrict head movements as much as possible. However, in this setup, the subjects had a camera on the top of their heads, recording the position of the screen at every moment. Deviations of the head are recorded by the camera to later on rectify this variations through the use of homographies. This allows the users to feel free for moving their head along with their eyes. A.2.3 Filter model The filter F, is computed as the transformation that relate the surface potential data Dinto the x, y coordinates of the eye trajectories that the subject is following. Let the x, y coordinates of the eyes fixation over a trajectory of N discrete points be designated by a 2 ×Nmatrix O. The first row of matrix Ocontains the xcoordinates (horizontal movement) and the second row the respective ycoordinates (vertical movement). Then the filter Fsatisfies the following equation: O=FD (A.1) where Dis the matrix of the recorded brain signal over the length of the trajectory at the 6 electrode sites. The filter Pis hen computed as: F=OD+(A.2) where D+is the Moore-Penrose generalized inverse or pseudoinverse. Once the filter Pis learned, it can be used to find any eye movements on the proximity of the training data set. Let Dnew denote some new data, digitized over N time points. The 2×Nmatrix XY containing the eye fixation coordinates over N time points is then found as: XY =FDnew =OD+Dnew (A.3) Notice that if the head position changes between experiments or within the recording of the experiment, the filter Pis not valid. The reason is that the mapping of the eye 55 A. Bit&Brain Technologies A.2 Methods fixations changes over time. To solve this problem is propose the use of a homographybased rectification to reference the data to normalized coordinates independently of the head pose. A.2.4 Homography-based rectification A.2.4.1 Pattern detection Two perspective images can be geometrically related through an homography H∈R3×3. This projective transformation Hrelates points that belong to a plane and are seen from different points of view. In order to compute the homography, 4 correspondent points are required [47]. The 4 points must be coplanar (i.e. belong to the same plane). For this application, the plane corresponds to the monitor and the 4 correspondent points will be the corners of the screen. One set of points will be the coordinates in pixels referenced to the top left corner, and their correspondences will be the coordinates in pixels of the recorded images. Automatically detection of the recorded points is done using computer vision algorithms. In this sense, a marker is designed and displayed in the mentioned corners. Figure A.1(right) shows the selected marker, which is small to maximize to effective screen area, but big enough to be recognize by the camera. Also is rounded to avoid the issues of squared shapes when the camera is rotated. The computer vision algorithm proposed for marker recognition in the recorded images is based on normalized cross-correlation. Cross-correlation is defined as a measure of similarity between 2 signals or patterns. In this case, one of the inputs is the template (i.e. the marker) and the other input is the image where the markers have to be found. Since the template and the image are not of the same size, the normalized cross-correlation is performed by local sums following the next expression: correlation(u, v) = Σx,y[f(x, y)−f(u, v)][t(x−u, y −v)−t] √Σx,y[f(x, y)−f(u, v)]2Σx,y[t(x−u, y −v)−t]2 (A.4) where fis the image, tis the mean of the template, and f(u, v) is the mean of f(x, y) in the region under the template. Hence, the pair of indexes (u, v) which maximize the cross-correlation value correspond to the area of the image where the marker has the highest probability to be located. Since for this application it is needed to locate 4 markers corresponding to each of the 4 corners of the screen, the best 20 values are selected and then filtered by geometrical restrictions to obtain the 4 best matches. An additional issue is that the pattern appear to change size when the screen is recorded from different ranges. To make the algorithm robust to distance, the normalize cross-correlation procedure is iterated for different templates sizes, including only reasonable values for the working space. Then, 56 A. Bit&Brain Technologies A.3 Results from the set of 4 best matches corresponding to the different pattern sizes, that set with higher correlation values is selected. Once the markers are found on a recorded image, the homography that relates image and monitor is computed by: Pimage =HtrainPscreen (A.5) where Pscreen are the 4 corner of the screen (i.e. [(0,0); (w, 0); (0, h); (w, h)] ), being wthe width and hthe hight of the monitor; and Pimage are the markers of the image obtained by the recognition algorithm. The transformation matrix Hallows to project any point of one space to the other. In this way, for the calibration phase, the xand y coordinates of the moving cursor that represent the eye fixation is reprojected onto the image recorded by the camera. From here, the equation to obtain the filter Fis modified as follows: F= (HtrainO)∗D+(A.6) where His the homography matrix computed for each different instant of time, and the filter Pmaps the recorded brain signal in an artificial space (recorded by the camera) that is independent of head movements. On the other hand, when it is required to obtain the eye fixation points from the brain activity, the equation A.3 has to be modified as follows: XY =H− testFDnew =H− testHtrainPscreenD+Dnew (A.7) where Htest represents the homography matrix at each instant of time for a testing condition. A.3 Results The proposed work for pattern recognition and homography-based rectification was tested under different conditions. A.3.1 Marker recognition The first attempt to verify the robustness of the algorithm was to detect a single marker in a synthetic image adding different perturbations such as gaussian noise, blurring the pattern or rotating the image. The results are displayed in Figure A.2 where it can be 57 [9] H.T. van Schie, R.B. Mars, M.G.H. Coles, and H. Bekkering. Modulation of activity in medial frontal and motor cortices during error observation. Nature Neuroscience, 7:549–554, 2004. [10] PW Ferrez and JdR Mill´an. Error-related EEG potentials generated during simulated Brain-Computer interaction. IEEE Trans Biomed Eng, 55(3):923–929, March 2008. [11] R. Chavarriaga and J.d.R. Mill´an. Learning from EEG error-related potentials in noninvasive brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng, 18(4):381–388, 2010. [12] Phillip J Holcomb and Helen J Neville. Auditory and visual semantic priming in lexical decision: A comparison using event-related brain potentials. Language and Cognitive Processes, 5(4):281–312, 1990. [13] J Blumberg, J Rickert, S Waldert, A Schulze-Bonhage, A Aertsen, and C Mehring. Adaptive classification for brain computer interfaces. In International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), volume 2007, pages 2536–9, January 2007. [14] I. Iturrate, L. Montesano, and J. Minguez. Robot reinforcement learning using eeg-based reward signals. In Robotics and Automation (ICRA), IEEE International Conference on, pages 4822–4829. IEEE, 2010. [15] I. Iturrate, R. Chavarriaga, L. Montesano, J. Minguez, and J. del R Millan. Latency correction of error potentials between different experiments reduces calibration time for single-trial classification. In Int Conf of the IEEE Engineering in Medicine and Biology Society (EMBC), 2012. [16] I. Iturrate, R. Chavarriaga, L. Montesano, J. Minguez, and J.d.R. Mill´an. Latency correction of error potentials between different experiments reduces calibration time for single-trial classification. In Proc of the Annual Int Conf of the IEEE Engineering in Medicine and Biology Society (EMBC), 2012. [17] J. Omedes, I. Iturrate, L. Montesano, and J. Minguez. Using frequency-domain features for the generalization of eeg error-related potentials among different tasks. In Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 2013. [18] I Iturrate, J Omedes, and L Montesano. Shared control of a robot using eeg-based feedback signals. In Proceedings of the 2nd Workshop on Machine Learning for Interactive Systems (MLIS13) at IJCAI13, 2013. [19] C. Vidaurre, M. Kawanabe, P. von B¨unau, B. Blankertz, and K.R. M¨uller. Toward Unsupervised Adaptation of LDA for Brain-Computer Interfaces. IEEE Trans Biomed Eng, 58(3):587 –597, 2011. 64 [20] C. Vidaurre, C. Sannelli, K.R. M¨uller, and B. Blankertz. Co-adaptive calibration to improve BCI efficiency. J Neural Eng, 8(2):025009, April 2011. [21] S.J. Luck. An introduction to the event-related potential technique. The MIT Press, 2005. [22] L. Li, D. Yao, and G. Yin. Spatio-temporal dynamics of visual selective attention identified by a common spatial pattern decomposition method. Brain Res, 1282:84– 94, 2009. [23] E. Sellers, D. Krusienski, D. McFarland, T. Vaughan, and J. Wolpaw. A P300 event-related potential brain-computer interface (BCI): The effects of matrix size and inter stimulus interval on performance. Biol Psychol, 73(3):242–52, October 2006. [24] J. Polich. On the relationship between EEG and P300: Individual differences, aging, and ultradian rhythms. Int J Psychophysiol, 26(1-3):299–317, 1997. [25] M. Kutas, G. McCarthy, and E. Donchin. Augmenting mental chronometry: The P300 as a measure of stimulus evaluation time. Science, 197(4305):792, 1977. [26] J.F. Cavanagh, M.X. Cohen, and J.J.B. Allen. Prelude to and resolution of an error: EEG phase synchrony reveals cognitive control dynamics during action monitoring. J Neurosci, 29(1):98–105, January 2009. [27] I. Iturrate, L. Montesano, and J. Minguez. Single trial recognition of error-related potentials during observation of robot operation. In Proc of the Annual Int Conf of the IEEE Engineering in Medicine and Biology Society (EMBC), 2010. [28] W. J. Gehring, B. Goss, M. G. H. Coles, D. E. Meyer, and E. Donchin. A neural system for error detection and compensation. Psychological Science, 4:385–390, 1993. [29] E. Sauser. Robottoolkit. Available online: lasa.epfl.ch/RobotToolKit. [30] S. Nieuwenhuis, C.B. Holroyd, N. Mol, and M.G.H. Coles. Reinforcement-related brain potentials from medial frontal cortex: origins and functional significance. Neuroscience & Biobehavioral Reviews, 28(4):441 – 448, 2004. [31] J.R. Wolpaw, N. Birbaumer, D.J. McFarland, G. Pfurtscheller, and T.M.Vaughan. Brain-computer interfaces for communication and control. Clin Neurophysiol, 113(6):767–91, June 2002. [32] F. Lotte, M. Congedo, A. L´ecuyer, F. Lamarche, and B. Arnaldi. A review of classification algorithms for EEG-based brain-computer interfaces. J Neural Eng, 4(2):R1–R13, June 2007. 65 [33] R. Akbani, S. Kwek, and N. Japkowicz. Applying support vector machines to imbalanced datasets. In Proceedings of the 15th European Conference on Machine Learning (ECML), pages 39–50, 2004. [34] I. Iturrate, L. Montesano, and J. Minguez. Shared-control brain-computer interface for a two dimensional reaching task using eeg error-related potentials. In Int. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 2013. [35] RJ Croft and RJ Barry. EOG correction of blinks with saccade coefficients: a test and revision of the aligned-artefact average solution. Clinical neurophysiology, 111(3):444–51, March 2000. [36] A Schl¨ogl et al. A fully automated correction method of EOG artifacts in EEG recordings. Clinical neurophysiology, 118(1):98–104, January 2007. [37] Pascual-Marqui RD. Standardized low resolution brain electromagnetic tomography (sLORETA): Technical details. Methods Find Exp Clin Pharmacol, pages 5–12, 2002. [38] H Ramoser, J Muller-Gerking, and G Pfurtscheller. Optimal spatial filtering of single trial eeg during imagined hand movement. IEEE Trans Neural Syst and Rehab Eng, 8(4):441–446, 2000. [39] Guido Dornhege. Speeding up classification of multi-channel brain-computer interfaces: Common spatial patterns for slow cortical potentials. IEEE EMBS Conference on Neural Engineering, pages 595–598, 2003. [40] Michael X. Cohen. Error-related medial frontal theta activity predicts cingulaterelated structural connectivity. NeuroImage, 55(3):1373–1383, 2011. [41] Tobias U. Hauser, Reto Iannaccone, Philipp St¨ampfli, Renate Drechsler, Daniel Brandeis, Susanne Walitza, and Silvia Brem. The feedback-related negativity (FRN) revisited: New insights into the localization, meaning and network organization. NeuroImage (to be published), August 2013. [42] P.W. Ferrez. Error-Related EEG Potentials in Brain-Computer Interfaces. PhD thesis, ´ Ecole Polytechnique F´ed´erale de Laussane, 2007. [43] Francesco Mondada, Michael Bonani, Xavier Raemy, James Pugh, Christopher Cianci, Adam Klaptocz, Stephane Magnenat, Jean-Christophe Zufferey, Dario Floreano, and Alcherio Martinoli. The e-puck, a robot designed for education in engineering. In Proceedings of the 9th conference on autonomous robot systems and competitions, volume 1, pages 59–65, 2009. [44] RS Sutton and AG Barto. Reinforcement learning: An introduction. MIT Press, 1998. [45] Laurence R Young and David Sheena. Survey of eye movement recording methods. Behavior Research Methods & Instrumentation, 7(5):397–429, 1975. 66 [46] Carrie A Joyce, Irina F Gorodnitsky, Jonathan W King, and Marta Kutas. Tracking eye fixations with electroocular and electroencephalographic recordings. Psychophysiology, 39(5):607–618, 2002. [47] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision, volume 2. Cambridge Univ Press, 2000. 67