EEG-fMRI fusion analysis of attention and visual working memory

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 3.0 https://creativecommons.org/licenses/by/3.0/ EEG-fMRI fusion analysis of attention and visual working memory © Authors 2023 Published version Salmela, Viljami; Guo, Lijing; Alho, Kimmo; Ye, Chaoxiong Salmela, V., Guo, L., Alho, K., & Ye, C. (2023). EEG-fMRI fusion analysis of attention and visual working memory. In CCN 2023 : 2023 Conference on Cognitive Computational Neuroscience (pp. 678-680). Conference Management Services, Inc.. https://doi.org/10.32470/ccn.2023.1293- 0 2023 EEG-fMRI fusion analysis of attention and visual working memory Viljami Salmela ([email protected]) Department of Psychology and Logopedics, PO Box 21, FI-00014 University of Helsinki, Finland Lijing Guo([email protected]) Institute of Brain and Psychological Sciences, Sichuan Normal University, 610021, China Kimmo Alho ([email protected]) Department of Psychology and Logopedics, PO Box 21, FI-00014 University of Helsinki, Finland Chaoxiong Ye ([email protected]) Department of Psychology, PO Box 35, FI-40014 University of Jyväskylä, Finland Abstract: While the previous fMRI studies suggest that the contents of the visual working memory (VWM) are represented in a spatially widely distributed brain network and the previous EEG studies have revealed some temporal properties of the memory processes, the exact spatio-temporal dynamics of working memory processes are not yet understood. Here we used multivariate EEG-fMRI fusion analysis to combine spatially (fMRI) and temporally (EEG) precise information, separately measured (n = 29) during a cued and delayed orientation change detection. Representational dissimilarity matrices (RDMs) from EEG responses in 10 ms time bins and fMRI responses from 360 different brain regions were correlated with each other as well as with model RDMs. Both EEG and fMRI response patterns were almost fully explained by attention (left/right cue) and only minimally by memory (set size and change magnitude). EEG-fMRI fusion showed distinct temporal profiles in different regions containing sustained information and transient peaks. The results highlight the role of attentional processes during working memory tasks. Keywords: Visual working memory; change detection; EEG; fMRI; representational similarity analysis; RSA; EEG-fMRI fusion Introduction Working memory is one the fundamental components of our cognition, and it is used for maintaining and manipulating information in a time scale of seconds or a few tens of seconds. The research in the past 15 years has shown that visual working memory (VWM) contains dynamic and flexible processes, and VWM representations can vary in precision (Ma, Husain, & Bays, 2014). Functional magnetic resonance imaging (fMRI) studies have shown that the contents of VWM can be decoded from the pattern activity in several different brain areas (Christophel, Klink, Spitzer, Roelfsema, & Haynes, 2017), suggesting a widespread brain network controlling encoding, maintenance, and retrieval of memory representations. Electroencephalogram (EEG) measured during memory tasks has provided information on the temporal aspects of working memory processes (Adam, Vogel, & Awh, 2020). However, the precise spatio-temporal dynamics of VWM are not yet known. Here we used multivariate EEG-fMRI fusion analysis (Cichy & Oliva, 2020) to study information flow during a delayed change detection task. Methods Participants The sample included 29 participants (18– 25 years; 20 females), recruited from the participant pool of Sichuan Normal University. The experiments were reviewed and approved by the ethical committee of Sichuan Normal University (SCNU-221114). Stimuli Stimuli were white line segments presented on a mid-gray background, arranged on a circle (not visible to participants; Figure 1). Figure 1: Experimental setup. Example trial: left cue, two different orientations on each side, 30 deg change. Procedure The structure of each trial (Figure 1) was as follows: 1) left/right arrow cue (200 ms), 2) 100 ms fixation cross, 3) memory array of 8 lines (four lines on each side of the display), 4) 1000 ms memory interval, 5) one probe line on both sides for 1500 ms or until response, 6) 5000 ms inter-trial-interval. The participants’ task was to report with an appropriate button press whether the orientation of the probe on the cued side of the display was changed or not. The cue 678 This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0 (left/right), set size (1, 2, 4 orientations on each side), and the amount of change (15, 30, or 60 deg) were varied. In all conditions 4 + 4 lines were shown and only the number of different orientations was varied. EEG and Acquisition and Processing Standard EEG preprocessing was conducted (e.g., band-pass filtering (.1-50 Hz), removing ICA-based artefacts). Event related potentials (ERPs) were extracted from 200 ms before to 3300 ms after cue onset. fMRI Acquisition and Processing EPI sequences with simultaneous multislice was measured with isotropic 2 mm voxels and 2.0 second TR. The fMRI data was preprocessd with fMRIprep (version 21.1.1.) standard preprocessing pipeline (Esteban et al., 2019). Each experimental condition was modelled with a separate regressor in GLM analysis. EEG/fMRI Analysis Representational dissimilarity matrices (RDMs) were calculated from EEG in 10 ms time bins (correlation across all conditions across channels; 350 RDMs in total) and from fMRI in 360 brain regions (HCP parcellation (Glasser et al., 2016); across all conditions across voxels; 360 RDMs in total). Then, EEG and fMRI RDMs were correlated with each other and with model RDMs. Results Behavioral Performance As expected, participants’ performance accuracy (d’) varied from 0.4 to 3.2 depending on the condition, d’ values gradually decreasing with increasing set size (F(2,56) = 308.5, p < .001, η2p = .92) and decreasing change magnitude (F(2,56) = 144.7, p < .001, η2p = .84). Figure 2: Partial correlation of EEG RDM and attention model RDM (left middle) reached noise ceiling, while memory model RDM (left bottom) did not correlate with EEG data (RDM example, left top). EEG/fMRI Results Correlation of EEG RDM and attention model RDM (left/right cue) reached noise ceiling and thus explained almost all of the variability in the data (Figure 2). In incorrect trials, these correlations were attenuated and delayed compared to correct trials. The correlation with memory model RDM was low. In fMRI, correlation of the data and attention model RDM reached noise ceiling in visual and parietal areas. Correlation of the fMRI and memory RDMs was weaker, but slightly elevated in a few parietal and frontal regions. The EEG-fMRI fusion showed gradual progress of information from early visual areas to higher visual, temporal and parietal regions with different temporal profiles (Figure 3). Figure 3: A few example temporal profiles obtained with EEG-fMRI fusion in visual (V1, V4, LO2), parietal (PIT, IP0), and temporal (TPOJ3) areas. Discussion In EEG and fMRI data measured during cued and delayed change detection task, attention related processes explained most of the variance in the data. While the behavioral performance varied according to set size and the magnitude of change, as expected, not much systematic information related to these were found in EEG or fMRI. In contrast to previous EEG studies using similar experimental setup, our results indicate that attention effects mask memory related effects during the change detection task. Interestingly, during incorrect trials the correlations were delayed, suggesting that memory failures were due to lapses of attention. In the EEG-fMRI fusion, different response profiles were found in different regions: sustained information and several transient peaks, presumably related to maintaining and switching of attention during the memory task. Acknowledgments We thank Yuxin Cheng, Jinru Chen and Zhihu Pan for their valuable contribution to the data acquisition. This work was supported by grants from the National Natural Science Foundation of China (No. 31700948), and the Academy of Finland (No. 333649 to C.Y.). 679 References Adam, K. C. S., Vogel, E. K., & Awh, E. (2020). Multivariate analysis reveals a generalizable human electrophysiological signature of working memory load. Psychophysiology, 57(12), e13691. doi:10.1111/psyp.13691 Christophel, T. B., Klink, P. C., Spitzer, B., Roelfsema, P. R., & Haynes, J. D. (2017). The distributed nature of working memory. Trends Cogn Sci, 21(2), 111-124. doi:10.1016/j.tics.2016.12.007 Cichy, R. M., & Oliva, A. (2020). A M/EEG-fMRI fusion primer: Resolving human brain responses in space and time. Neuron, 107(5), 772-781. doi:10.1016/j.neuron.2020.07.001 Esteban, O., Markiewicz, C. J., Blair, R. W., Moodie, C. A., Isik, A. I., Erramuzpe, A., . . . Gorgolewski, K. J. (2019). fMRIPrep: a robust preprocessing pipeline for functional MRI. Nat Methods, 16(1), 111-116. doi:10.1038/s41592- 018-0235-4 Glasser, M. F., Coalson, T. S., Robinson, E. C., Hacker, C. D., Harwell, J., Yacoub, E., . . . Van Essen, D. C. (2016). A multimodal parcellation of human cerebral cortex. Nature, 536(7615), 171-178. doi:10.1038/nature18933 Ma, W. J., Husain, M., & Bays, P. M. (2014). Changing concepts of working memory. Nat Neurosci, 17(3), 347-356. doi:10.1038/nn.3655 680