Processing Mechanism of Chinese Verbal Jokes : Evidence from ERP and Neural Oscillations
Full text
This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY-NC-ND 4.0 https://creativecommons.org/licenses/by-nc-nd/4.0/ Processing Mechanism of Chinese Verbal Jokes : Evidence from ERP and Neural Oscillations © 2019 University of Electronic Science and Technology of China Published version Li, Xue-Yan; Wang, Hui-Li; Saariluoma, Pertti; Zhang, Guang-Hui; Zhu, Yong-Jie; Zhang, Chi; Cong, Feng-Yu; Ristaniemi, Tapani Li, X.-Y., Wang, H.-L., Saariluoma, P., Zhang, G.-H., Zhu, Y.-J., Zhang, C., Cong, F.-Y., & Ristaniemi, T. (2019). Processing Mechanism of Chinese Verbal Jokes : Evidence from ERP and Neural Oscillations. Journal of Electronic Science and Technology, 17(3), 260-277. https://doi.org/10.11989/JEST.1674-862X.80520017 2019
Copyright 2019 University of Electronic Science and Technology of China. Publishing Services provided by Elsevier B.V. on behalf of KeAi. This is an open access article under the CC BY-NC-ND License (http://creativecommons.org/licenses/by-nc-nd/4.0/ ). DigitalObjectIdentifier: Processing Mechanism of Chinese Verbal Jokes: Evidence from ERP and Neural Oscillations Xue-Yan Li | Hui-Li Wang* | Pertti Saariluoma | Guang-Hui Zhang | Yong-Jie Zhu | Chi Zhang | Feng-Yu Cong | Tapani Ristaniemi Abstract—Thecognitiveprocessingmechanismofhumorreferstohowthesystemofneuralcircuitryandpathways inthebraindealswiththeincongruityinahumorousmanner.Thepastresearchhasrevealeddifferentstagesand corresponding functional brain activities involved in humor-processing in terms of time and space dimensions, highlightingtheeffectsofthetimewindowsofabout400ms,600ms,and900ms.However,muchlessisknown abouthumorprocessinginlightofthefrequencydimension.Atotalof36Chineseparticipantswererecruitedinthis experiment,withChinesejokes,nonjokes,andnonsensicalsentencesusedasthestimuli.Theexperimentalresults showed that there were significant differences among conditions in the P200 effect, which signified that the incongruitydetectionhadalreadybeenintegratedandperceivedatabout200ms,priortothesemanticintegrationatabout 400 ms. This pre-processing is specific to Chinese verbal jokes due to the simultaneous involvement of both orthographic and phonologic parts in processing Chinese characters. The analysis on the frequency dimension indicated that beta’s power particularly reflected the characteristics of different stages in Chinese verbal humor processing.Jokes’andnonsensicalsentences’relativepowerchangesonthebetabandrankedsignificantlyhigher thanthatofnonjokesatabout200ms,whichsuggestedtheexistenceofmoredifficultiesinmeaningconstructionin pre-processingtheincongruities.Thisindicatedacontinuitybetweentheanalysisofeventrelatedpotential(ERP) componentsandneural oscillations and revealedthekey role of thebetafrequency band in Chineseverbaljoke processing. Index Terms—Betaband,humorprocessing,P200effect. 1. Introduction Humor is ubiquitous, happening in all individuals in all stages over a lifespan. It plays a very critical role in varioussocialcontexts,exertinggreatinfluencesonsocietalandculturaldevelopment.Humor,rootedinitssocial component, presented in its cognitive component, passes laughter effects by its affective component. Different componentsembeddedinhumordecidethecomplexityofhumor,causingthedevelopmentofagreatnumberof *Correspondingauthor Manuscriptreceived2018-05-20;revised2018-12-10. X.-Y.Li,H.-L.Wang,G.-H.Zhang,andC.ZhangarewiththeSchoolofForeignLanguages,DalianUniversityofTechnology, Dalian116024(e-mail:[email protected];[email protected]). P. Saariluoma, Y.-J. Zhu, and T. Ristaniemi are with University of Jyväskylä, Jyväskylä FI-40014 (e-mail: [email protected]). F.-Y. Cong is with the Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian 116024. Colorversionsofoneormoreofthefiguresinthispaperareavailableonlineathttp://www.journal.uestc.edu.cn. Publishingeditor:Yu-LianHe 260 JOURNALOFELECTRONICSCIENCEANDTECHNOLOGY,VOL.17,NO.3,SEPTEMBER2019
relevant theories. The incongruity theory[1], one of the cognitive theories of humor, highlighting the incongruity detection and resolution, has been the most frequently quoted theory in this field due to its application to the researchonthecognitiveprocessingmechanismofhumor. Thecognitiveprocessingmechanismofhumorreferstohowthesystemofneuralcircuitryandpathwaysinthe brain deals with the incongruities in a humorous manner, which involves a complicated high-level cognitive process,renderingitworthwhiletobeinvestigated.Advancedbrain-imagingtechniqueshaveprovidedresearchers with more possibilities to reveal more objective evidence on the mechanism of humor processing. Functional magneticresonanceimaging(fMRI)isemployedtoshowspecificbrainregionsactivatedduringdifferentstagesof humorprocessingduetoitsexcellentspaceresolution.Thecognitivestage(includingtheincongruitydetectionand resolution)involvesthebilateralactivationinthebrain,withinferiorfrontalgyrus,superiorfrontalgyrus,andmiddle temporalgyrusshowingastrongeractivationinverbalhumor[2],[3];theaffectivestage(mirth)iscloselyrelatedwith mesolimbicrewardregions,withthehighlightofamygdala,hippocampus,andinsularcortex[3]-[5].Theeventrelated potential (ERP) based technique, another frequently-used brain-imaging method, with its excellent temporal resolution,isusedtodeterminedifferentstagesinhumorprocessing.DifferentERPcomponentscorrespondingto different stages were extracted in a number of relevant studies. The N400 component has been found to be correlated with the incongruity detection[6]-[10]; the P600 component is related to the incongruity resolution[7],[8],[10],[11], andthecomponentoflatepositivepotentials(LPPs)isindicatedtobeconnectedwiththemirth[7],[8],[10]. TheP200component,peakingbetween150msand275ms,triggeredbythevisualstimuli,islargerforthe expected endings than for the unexpected endings[12]. In language processing, P200 could be modulated by the contextualinformation,suchassentence-levelconstraintsorcongruityassociatedwiththetargetword[13].Itispart of the cognitive matching system that compares sensory input with the stored memory[14]. All these previous findingsaboutP200presentitsgreatcorrelationswithhumorprocessing,becauseinessence,humorprocessingis justtofindthewayouttotheunexpectedendings,theincongruitiesorunmatchingsystem.However,byfar,very limitedevidencehasindicatedtherolesofP200inhumorprocessingresearch.Furthermore,anumberofstudies indicate there are significant differences in processing mechanisms between the alphabetic and pictograph languages. For example, the component of P200 was suggested to be closely related with the processing of Chinesecharactersinsteadoftheprocessingofthealphabeticlanguage[15],duetoitsinclusionofbothorthographic andphonologicpartsatthesametime.Therefore,theP200effectisworthfurtherinvestigationinverbalhumorprocessingstudiesforitsspecificrolesinbothwordsandperceptions. Neuraloscillationscanalsobereflectedonthefrequencydimensionapartfromthetimeandspacedimensions. Some frequency bands, such as alpha, delta, theta, beta, and gamma, are often detected and studied in some specificbrainregionsincognitiveprocessing.Thoughitiscurrentlypopularinlanguageprocessingstudiesdueto itsmeaningfulapplicationstoartificialintelligenceandclinicalstudies,thefrequencydimensionanalysishasbeen rarelyusedtoinvestigatetheverbalhumorprocessing. 2. Method and Materials 2.1. Participants Atotalof42right-handedadults(21maleand21female)withnormal,orcorrected-to-normalvision,aged from 19 years to 28 years (mean age: 23.75 years), from Dalian University of Technology and Liaoning NormalUniversity,wererecruitedaspaidvolunteerstotakepartinthecurrentexperiment.Noparticipanthad neurologicalorpsychiatricdiseasesandallofthemwerenativeChinesespeakers.Theiraverageschooling yearsrangedfrom14yearsto18years(meanschooling:16.30years).Allparticipantshadcompletedthe LIet al.:ProcessingMechanismofChineseVerbalJokes:EvidencefromERPandNeuralOscillations 261
written informed consent, agreeing with following the directions during the experiment, and had been informedoftheinstructionsofallprocedurespriortotheexperiment.Thecurrentexperimentwasapproved bytheResearchEthicsCommitteeofLiaoningNormalUniversity.Theparticipantswereaskedtominimize theirmovementsandeye-blinksduringtheexperiment.Inthephaseofdatapre-processing,6participants wereremovedduetotheinvalidityoftheirdata,leaving36participants’databeinganalyzed. Since each punchline matched up with three set-up conditions to avoid the reviewing effects caused by repeatedly reading the same punchline, each participant can only read one punchline for one time during the experiment. Thus, 36 participants were randomly grouped into 12 groups, with three participants in one group finishingonesetofthestimuli,andeachgroupwasregardedasonesubject. 2.2. Materials Prior to the experiment, 90 question-answer type Chinese jokes (homophonic jokes) were selected from the Internetandmagazines.Afterthepretestamong150people(differentfromtheparticipantsintheexperiment),top 60 jokes ranked as funny were subsequently sorted out, together with 60 nonjokes (normal statements of facts from the Internet and newspapers), and 60 produced nonsensical sentences (totally irrelevant questions and answers),wereusedasthestimuliintheexperiment,180experimentalsentencesintotal.Thesetupsentences instead of punchlines of jokes were controlled so that the neural activities were not triggered by different punchlines[10]. Every set of stimuli consisted of three different conditions: Jokes, nonjokes, and nonsensical sentences, sharing the same punchline. All stimulus sets were randomly divided into three blocks and each participantcanonlyseeoneblockforonestimulustype,sothesamepunchlinewouldnotbeseentwiceinthe experiment. Nonjokes were all related to semantic memories from daily life and nonsensical sentences were all kept at a very low semantic level. Each setup sentence was limited to about 15 Chinese characters and each punchlinewaslimitedto2to4Chinesecharacters.Fig.1showsoneoftheexamplesofasetofstimuli,which consists of three conditions: Joke, nonjoke, and nonsensical sentence. The setup for the joke is that the sheepstopsbreathing,andguessanidiom.Theanswerorthepunchlineis“扬眉吐气”.Because“扬(raise)” has the same pronunciation with the Chinese character “羊(sheep)” and “眉(eyebrow)” has the same pronunciationwiththeChinesecharacter“没(stop)”.Thepuchlinetothesetupforthejokeis“羊没吐气(the sheepstopsbreathing)”,whichcarriesthesamepronunciationwiththeidiom“扬眉吐气(raiseeyebrowsand feelelatedafterunburdeningoneselfofresentment)”.Forthenonjoke,thesetupisthatraiseeyebrowsand feelelatedafterunburdeningoneselfofresentment,andguessanidiom.Theanswerorthepunchlineof“扬 眉吐气” is just the correct answer to the setup question, which is a normal statement. The setup of the nonsensical sentence has no semantic relations with the punchline “扬眉吐气”. More examples see the appendix. 羊停止了呼吸，猜一个成语？ 扬起眉头，吐出怨气，猜一个成语？ 手机不可以掉在地上吗？ “扬眉吐气” Fig.1.Exampleofasetofstimuli. 262 JOURNALOFELECTRONICSCIENCEANDTECHNOLOGY,VOL.17,NO.3,SEPTEMBER2019
2.3. Procedure E-Prime 2.0 was used to design the procedure in this experiment. Participants were instructed about the experimental procedures prior to the experiment. Each of them was seated in a quiet room and a screen was placed approximately at a 100 cm distance. Three trials, similar to the formal experiment trials, were firstly presented to the participants to help them become familiar with the procedure. The flow of the stimulus presentationineachtrialwentasfollows:Firstly,theparticipantsawafixationpoint(+)(thedurationwas1000ms) in the center of the screen; then the setup of the joke/nonjoke/nonsensical sentence, in the question type, appeared;afterthesetup,theparticipantpressedanybuttontogoonwiththetrialbyseeinganotherfixation(the durationwasalso1000ms)andthenthepunchline,intheanswertype,appearedinthecenterofthescreen;after 3000ms, theparticipantwasrequired tomakethreeratings of R1,R2,and R3, respectively,onthe degrees of surprise (not surprised at all/not surprised/surprised/very surprised), comprehensibility (not comprehensible at all/notcomprehensible/comprehensible/verycomprehensible),andfunniness(notfunnyatall/notfunny/funny/very funny)onafour-pointLikertscale. 2.4. Data Acquisition Electroencephalograph(EEG)wasusedtocollectthedataofbrainelectricalactivitiesinthisexperimentforits hightemporalresolutionoverotherbrainimagingmethodstobetterillustratethemechanismofhumorprocessing fromtheperspectivesofdifferentdimensions.Duringtheexperiment,brainelectricalactivitieswererecordedfrom 64scalpsitesbyelectrodesfixedintheelectrodecapwiththereferencesontheleftandrightmastoids,usingthe ActiveTwosystem(BioSemi, theNetherlands).Allinterelectrodesimpedance wasmaintainedbelow5kΩ. ERP waveforms were time-locked to the onset of the punchline. Trials, contaminated with artifacts, such as the excessive vertical or horizontal electro-oculographic potentials, excessive muscle activity, bursts of electromyographicactivity,orpeak-to-peakdeflectionexceeding±100mV,wereexcludedfromtheaveraging.The averagedERPepochwas2000ms,includinga200mspre-solutionbaseline.Forthefirst-timeanalysis,allepochs were band-pass filtered in the range of 0.1 Hz to 50.0 Hz using digital zero-phrase shift filtering. After artifact correction,60validepochsperconditionwereobtainedforeveryparticipant. 3. Experimental Results 3.1. Behavioral Results Behavioraldatawereanalyzedonthebasisof36participantsinsteadof12subjectstoguaranteemore accurate behavioral results. Among the ratings, jokes’funninessscores were the highest compared with nonjokes’ and nonsensical sentences’; nonjokes’comprehensibilityratingswerehigherthan jokes’, and jokes’ comprehensibility ratings were higher than nonsensical sentences’; nonsensical sentences’ surprise ratings were rated the highest followed by jokes’ surprise ratings; nonjoke’s surprise ratings were rated the lowest (seeFig. 2). The paired samples test showed that there were significantdifferencesinsurpriseratingsbetweenjokes and nonjokes (Mean=1.001, Standard error mean=0.0695, t=14.401, and p=0.000), between 0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Joke Nonjoke Nonsensical sentence Ratings Surprise Comprehensibility Funniness Fig.2.Comparisonamongratingsofjokes,nonjokes,and nonsensicalsentences. LIet al.:ProcessingMechanismofChineseVerbalJokes:EvidencefromERPandNeuralOscillations 263
nonjokesandnonsensicalsentences(Mean=–1.557,Standarderrormean=0.086,t=–18.025,andp=0.000), and between jokes and nonsensical sentences (Mean=–0.556, Standard error mean=0.602,t=–9.236, and p=0.000); there are significant differences in comprehensibility ratings between jokes and nonjokes (Mean=–0.308,Standarderrormean=0.524,t=–5.882,andp=0.000),betweenjokesandnonsensicalsentences (Mean=1.343, Standard error mean=0.081, t=16.557, and p=0.000), and between nonjokes and nonsensical sentences(Mean=1.651,Standarderrormean=0.082,t=20.068,andp=0.000);therearesignificantdifferencesin funninessratingsbetweenjokesandnonjokes(Mean=0.968,Standarderrormean=0.074,t=13.062,andp=0.000), between nonjokes and nonsensical sentences (Mean=–0.506, Standard error mean=0.935,t=–5.410, and p=0.000),andbetweenjokesandnonsensicalsentences(Mean=0.463,Standarderrormean=0.104, t=4.451,and p=0.000). 3.2. ERP Results EEG data were processed and analyzed offline using MATLAB 2013. The component amplitudes were analyzed with one-way ANOVAs using the factors task conditions (jokes, nonjokes, and nonsensical sentences) andelectrodesites(Cz,Fz,andPz).Andthe0.05levelofsignificancewasadoptedthroughoutallERPanalyses. AllstimulielicitedtypicalERPcomponentsofvisualwords,suchasP1andN1,whichwereelicitedduringreading, probablyreflectingthevisualfeatureextractionnecessarytotheprocessingofvisualinformationinthememory[16]. Byusingthewaveletanalysis,theeffectsweresignificantonthetimewindowsof180msto240ms,320msto 450ms,and600msto900ms.Byusingtheconventionalanalysis,theeffectsweresignificantonthetimewindow of900msto1500ms. 3.2.1. 180 ms to 240 ms Thestatisticalanalysisofthe180msto240mstimewindowshowedthereweremaineffectsbetweenstimulus conditions and electrode sites Fz and Cz (F(1, 11)=4.37, p=0.0252, and η2=1.5432). Nonjokes elicited more positivewaveformsthan nonsensical sentences and jokes. Moreover, the observed brainwaves stimulatedatFz andCzfor the main effect of midline site were larger than that at Pz in this timewindow(seeFig.3).Maximal deflectionsinpositivityweremainlylocatedaroundthefrontalandcentralregionsofthescalpforallthree typesofstimuli,butforjokes,theBroca’sareawasactivatedmorethanthatfornonjokesandnonsensical sentences(seeFig.4). 3.2.2. 320 ms to 450 ms Thestatisticalanalysisofthe320msto450mstimewindowshowedthereweremaineffectsbetweenstimulus conditionsandelectrodesitesCz(F(1,11)=3.68,p=0.0417,andη2=2.005)andPz(F(1,11)=5.04,p=0.0157,and η2=0.823).Bothjokesandnonsensicalsentenceselicitedmorenegativewaveformsthannonjokes.Moreover,the observedbrainwavesatFzandCzonthemaineffectofmidlinesitewerelargerthanthatatthatatPzinthistime window(seeFig.3).Maximaldeflectionsweremainlylocatedaroundthefrontalandoccipitalregionsofthescalp forallthreetypesofstimuli,butforbothjokesandnonsensicalsentences,thereweremoreactivationsinnegativity inthetemporalpoleandrightfrontallobethanthatintheoccipitalregion(seeFig.4). 3.2.3. 600 ms to 900 ms The statistical analysis of the 600 ms to 900 ms time window showed there were main effects of stimulus conditionsandelectrodesites Fz (F(1, 11)=3.97, p=0.0337, and η2=1.65) and Cz (F(1, 11)=4.07, p=0.0313, and η2=0.952).Nonjokes elicited more positive waveforms thanjokes and nonsensical sentences. What ismore, the observedbrainwaveselicitedatFzandCzonthemaineffectofmidlinesitewerelargerthanthatatPz(seeFig.3). Forbothjokesandnonsensicalsentences,maximaldeflectionsinnegativityweremainlylocatedaroundthe 264 JOURNALOFELECTRONICSCIENCEANDTECHNOLOGY,VOL.17,NO.3,SEPTEMBER2019
right frontal lobe, or more specifically, close to the middle frontal gyrus for both jokes and nonsensical sentences,butforjokes,theBroca’sareawasdistinctivelyactivated(seeFig.4). 3.2.4. 900 ms to 1500 ms The statistical analysis of the 900 ms to 1500 ms time window showed there were main effects of stimulus conditions and electrode sites Fz (F(1, 11)=4.07, p=0.0313, and η2=3.06144). Jokes elicited more positive waveformsthannonsensicalsentencesandnonjokes.Whatismore,theobservedLPPsstimulatedatFzandCz −3 −2 −1 0 1 2 3 4 5 Amplitude (μV) −200 −100 0200 300100 400 500 600 700 800 900 1000 1100 1200 1300 −3 −2 −1 0 1 2 3 4 5 Amplitude (μV) −200 −100 0200 300100 400 500 600 700 800 900 1000 1100 1200 1300 Time (ms) Time (ms) −3 −2 −1 0 1 2 3 4 5 Amplitude (μV) −200 −100 0200 300100 400 500 600 700 800 900 1000 1100 1200 1300 Time (ms) Joke Nonjoke Nonsensical sentence Joke Nonjoke Nonsensical sentence Joke Nonjoke Nonsensical sentence (a) (b) (c) Fig.3.Brainwaveformsontheelectrodes(Cz,Pz,andFz)onthetimewindowsof180msto240ms,320msto450ms, and600msto900ms:(a)wavelet-Fz,(b)wavelet-Cz,and(c)wavelet-Pz. LIet al.:ProcessingMechanismofChineseVerbalJokes:EvidencefromERPandNeuralOscillations 265
forthemaineffectofmidlinesitewerelargerthanthat at Pz (see Fig. 5). Maximal deflections were mainly located around the frontal and parietal regions of the scalp for all three types of stimuli, but for jokes, right hemispherewasactivatedmorethanlefthemisphere. 3.3. Neural Oscillations Results Fieldtriptoolbox[17]isthemainanalysistoolusedin thisstudyandhigh-passfilterisusedtoeliminatethe slow drifts. For a proper analysis of oscillatory dynamics, different analytic tools had been used in this experiment, including the wavelet based timefrequency analysis (for quantifying amplitude changes)and event related coherenceanalysis(for quantifying changes in phase coherence between electrodes), being performed in the domain of the language comprehension. For each trial, the time horizonwasdeterminedfrom–4000msto6000ms to get rid of the border artifacts in the power spectrum[18].Thedatawereanalyzedina10mstime stepfrom–500msto1000ms,andina1Hzstep from 4 Hz to 100 Hz, using Morlet wavelets of seven cycles each[19]. Power in the stimulus interval was transformed to the percentage of change relative to the baseline. Four time windows (about 200 ms, 400 ms, 600ms,and800ms)closetoERPcomponentsanalyzedinthepreviouspartwereexaminedtoexplorewhether the oscillations in different conditions of stimuli in different time windows were qualitatively different and whether therewasacontinuitybetweentheanalysisofERPcomponentsandneuraloscillations.Alldependentvariables wereanalyzedbymultivariateANOVAs.Effectswithasignificancedifferenceofp<0.05werereported. At about 200 ms, jokes’ beta power ranked higher than nonsensical sentences’ and nonjokes’, with differencesbeingsignificantamongthreestimuli(p<0.05).Allthreestimuli’spowerdecreasedtotheirlowest points at about 400 ms, with jokes’ beta power being the highest and nonsensical sentences’ being the lowest,withdifferencesbeingsignificantamongthreestimuli(p<0.05).Fromtheviewoftheelectrodesactivated byjoke-nonjokeinthebetarange,thebetabandwassignificantlyactivated(p<0.05)intheelectrodesofC4,CP4, P4,PO4,andP7atabout200ms;FT7andC4atabout400ms;P3atabout600ms;FC1,FZ,FT8,T8,andP8at about 800 ms (see Fig. 6 (a)). From 400 ms, all three stimuli’s beta power increased, with both jokes’ and nonjokes’beinghigherthannonsensicalsentences.Atabout600ms,jokes’andnonjokes’betapowerwere at almost equally height, being higher than nonsensical sentences. Then, all three stimuli’s beta power continuedtoincreaseandsimilarsituationswerekeptuptoabout800ms(seeFig.6(b)). 4. Discussion 4.1. Behavioral Results The results of different ratings indicated that the stimuli selected in this study highlight the features of jokes,nonjokes,andnonsensicalsentences,respectively.Jokeshadthehighestratingsinfunninessdueto Joke Nonjoke Nonsensical Joke Nonjoke Nonsensical Joke Nonjoke Nonsensical 4 3 2 1 0 −1 Amplitude (μV) −0.5 −1.5 −2.5 −3.5 −4.5 Amplitude (μV) 0 −0.5 −1.0 −1.5 Amplitude (μV) (a) (b) (c) Fig. 4. Topographic maps of different stimuli (jokes, nonjokes,andnonsensicalsentences)onthedifferenttime windows:(a)180msto240ms,(b)320msto450ms,and (c)600msto900ms. 266 JOURNALOFELECTRONICSCIENCEANDTECHNOLOGY,VOL.17,NO.3,SEPTEMBER2019
their affective stage, mirth, which was aroused after the incongruity resolution. Though the incongruity resolutionwasnotachievedinnonsensicalsentences,theabsurdfeelingscausedbylowsemanticmeanings between the setups and the punchlines would also arouse slight emotional changes. That was why nonsensicalsentences’funninessratingswerehigherthannonjokes’andlowerthanjokes’.Thesetupsand punchlines of nonjokes were the materials from common sense, greatly related to semantic memories, so their comprehensibility ratings were the highest. In comparison, jokes’ comprehensibility ratings were relativelylowerthannonjokes’,becausejokes’comprehensionneedstheconversionoffixedmindsetandnot −3 −2 −1 0 1 2 3 4 5 Amplitude (μV) −200 0200 400 600 800 1000 1200 1400 1600 1800 −200 0200 400 600 800 1000 1200 1400 1600 1800 −200 0200 400 600 800 1000 1200 1400 1600 1800 −3 −2 −1 0 1 2 3 4 5 Amplitude (μV) Time (ms) Time (ms) −3 −2 −1 0 1 2 3 4 5 Amplitude (μV) Time (ms) Joke Nonjoke Nonsensical sentence Joke Nonjoke Nonsensical sentence Joke Nonjoke Nonsensical sentence (a) (b) (c) Fig.5.Brainwaveformsontheelectrodes(Cz,Pz,andFz)onthetimewindowof900msto1500ms:(a)conventionalFz,(b)conventional-Cz,and(c)conventional-Pz. LIet al.:ProcessingMechanismofChineseVerbalJokes:EvidencefromERPandNeuralOscillations 267
M.Shibata,Y.Terasawa,andS.Umeda,“Integrationofcognitiveandaffectivenetworksinhumorcomprehension,” Neuropsychologia,vol.65,pp.137-145,Dec.2014. [3] D. W. Campbell, M. G. Wallace, M. Modirrousta, et al., “The neural basis of humour comprehension and humour appreciation: The roles of the temporoparietal junction and superior frontal gyrus,” Neuropsychologia, vol.79,pp.10-20,Dec.2015. [4] Y.-C.Chan,T.-L.Chou,H.-C.Chen, et al.,“Towardsaneuralcircuitmodelofverbalhumorprocessing:AnfMRI studyoftheneuralsubstratesofincongruitydetectionandresolution,”Neuroimage,vol.66,pp.169-176,Feb.2013. [5] S. Coulson and R. F. Williams, “Hemispheric asymmetries and joke comprehension,” Neuropsychologia, vol. 43, no.1,pp.128-141,2005. [6] X.Du, Y. Qin, S. Tu, et al., “Differentiation of stages in joke comprehension: Evidence from an ERP study,” Intl. Journal of Psychology,vol.48,no.2,pp.149-157,2013. [7] Y.-J.Feng,Y.-C.Chan,andH.-C.Chen,“Specializationofneuralmechanismsunderlyingthethree-stagemodelin humorprocessing:AnERPstudy,”Journal of Neuolinguistics,vol.32,pp.59-70,Nov.2014. [8] B. Mayerhofer and A. Schacht, “From incoherence to mirth: Neuro-cognitive processing of garden-path jokes,” Frontiers in Psychology,vol.6,pp.550:1-19,May2015. [9] L.-C. Ku, Y.-J. Feng, Y.-C. Chan, et al., “A re-visit of three-stage humor processing with readers' surprise, comprehension,andfunninessratings:AnERPstudy,”Journal of Neurolinguistics,vol.42,pp.49-62,May2017. [10] M.Shibata,Y.Terasawa,T.Osumi, et al.,“Timecourseandlocalizationofbrainactivityinhumorcomprehension: AnERP/sLORETAstudy,”Brain Research,vol.1657,pp.215-222,Feb.2017. [11] K.D.Federmeier,“Thinkingahead:Theroleandrootsofpredictioninlanguagecomprehension,”Psychophysiology, vol.44,no.4,pp.491-505,2007. [12] S.Coulson and D. Brang, “Sentence context affects thebrain response to masked words,”Brain and Language, vol.113,no.3,pp.149-155,2010. [13] R. Freunberger, W. Klimesch, M. Doppelmayr, et al., “Visual P2 component is related to theta phase-locking,” Neuroscience Letters,vol.426,no.3,pp.181-186,2007. [14] M.Xie,Q.Yang,andQ.Wang,“TheP200componentinlexicalprocessing,”Advances in Psychology,vol.6,no.2, pp.114-120,2016. [15] M.Kutas,“Viewsonhowtheelectricalactivitythatthebraingeneratesreflectsthefunctionsofdifferentlanguage structures,”Psychophysiology,vol.34,no.4,pp.383-398,2007. [16] R.Oostenveld,P.Fries,E.Maris, et al.,“FieldTrip:OpensourcesoftwareforadvancedanalysisofMEG,EEG,and invasiveelectrophysiologicaldata,”Computational Intelligence and Neuroscience,2011,DOI:10.1155/2011/156869 [17] S.Regel,L.Meyer,andT.C.Gunter,“DistinguishingneurocognitiveprocessesreflectedbyP600effects:Evidence fromERPsandneuraloscillations,”PloS One,vol.9,no.5,pp.e96840:1-11,2014. [18] J. P. Lachaux, E. Rodriguez, J. Martinerie, et al., “Measuring phase synchrony in brain signals,” Human Brain Mapping,vol.8,no.4,pp.194-208,1999. [19] B.E.McDonough,C.A.Warren,andN.S.Don,“Event-relatedpotentialsinaguessingtask:Thegleamintheeye effect,”Intl. Journal of Neuroscience,vol.65,no.1-4,pp.209-219,1992. [20] C.D.Lefebvre,Y.Marchand,G.A.Eskes, et al.,“Assessmentofworkingmemoryabilitiesusinganevent-related brainpotential(ERP)-compatibledigitspanbackwardtask,”Clinical Neurophysiology,vol.116,no.7,pp.1665-1680, 2005. [21] K.D.FedermeierandM.Kutas,“Picturethedifference:Electrophysiologicalinvestigationsofpictureprocessingin thetwocerebralhemispheres,”Neuropsychologia,vol.40,no.7,pp.730-747,2002. [22] E. W. Wlotko and K. D. Federmeier, “Finding the right word: Hemispheric asymmetries in the use of sentence contextinformation,”Neuropsychologia,vol.45,no.13,pp.3001-3014,2007. [23] E.M.MorenoandI.C.Rivera,“Setbacks,pleasantsurprisesandthesimplyunexpected:Brainwaveresponsesina languagecomprehensiontask,”Social Cognitive and Affective Neuroscience,vol.9,no.7,pp.991-999,2013. [24] 274 JOURNALOFELECTRONICSCIENCEANDTECHNOLOGY,VOL.17,NO.3,SEPTEMBER2019
A.Zinchenko,P.Kanske,C.Obermeier, et al.,“Emotionandgoal-directedbehavior:ERPevidenceoncognitiveand emotionalconflict,”Social cognitive and Affective Neuroscience,vol.10,no.11,pp.1577-1587,2015. [25] A. Barnea and Z. Breznitz, “Phonological and orthographic processing of Hebrew words: Electrophysiological aspects,”The Journal of Genetic Psychology,vol.159,no.4,pp.492-504,1998. [26] B. Chen, W. Liu, L. Wang, et al., “The timing of graphic, phonological and semantic activation of high and low frequencyChinesecharacters:AnERPstudy,”Progress in Natural Science,vol.17,no.B07,pp.62-70,2007. [27] L.Zhou,M.C.M.Fong,J.W.Minett, et al.,“Pre-lexicalphonologicalprocessinginreadingChinesecharacters:An ERPstudy,”Journal of Neurolinguistics,vol.30,pp.14-26,Jul.2014. [28] V. Goel and R. J. Dolan, “The functional anatomy of humor: Segregating cognitive and affective components,” Nature Neuroscience,vol.4,no.3,pp.237-238,2001. [29] M.KutasandK.D.Federmeier,“Thirtyyearsandcounting:FindingmeaningintheN400componentoftheeventrelatedbrainpotential(ERP),”Annual Review of Psychology,vol.62,no.1,pp.621-647,2011. [30] M. Kutas and S. A. Hillyard, “Brain potentials during reading reflect word expectancy and semantic association,” Nature,vol.307,no.5947,pp.161-163,1984. [31] S. Coulson and M. Kutas, “Getting it: Human event-related brain response to jokes in good and poor comprehenders,”Neuroscience Letters,vol.316,no.2,pp.71-74,2001. [32] B. G. Lopez and J. Vaid, “Psycholinguistic approaches to humor,” in The Routledge Handbook of Language and Humor,S.Attardo,Ed.Oxon:Routledge,2017. [33] A.M.Bihrle,H.H.Brownell,J.A.Powelson, et al.,“Comprehensionofhumorousandnonhumorousmaterialsbyleft andrightbrain-damagedpatients,”Brain and Cognition,vol.5,no.4,pp.399-411,1986. [34] P. Shammi and D. T. Stuss, “Humour appreciation: A role of the right frontal lobe,” Brain, vol. 122, no. 4, pp.657-666,1999. [35] T. A. Dennis and G. Hajcak, “The late positive potential: A neurophysiological marker for emotion regulation in children,”Journal of Child Psychology and Psychiatry,vol.50,no.11,pp.1373-1383,2009. [36] G. Hajcak, J. S. Moser, and R. F. Simons, “Attending to affect: Appraisal strategies modulate the electrocortical responsetoarousingpictures,”Emotion,vol.6,no.3,pp.517-522,2006. [37] S.WeissandH.M.Mueller,“Toomanybetasdonotspoilthebroth:Theroleofbetabrainoscillationsinlanguage processing,”Frontiers in Psychology,vol.3,pp.201:1-15,Jun.2012. [38] L.A.Hald,M.C.M.Bastiaansen,andP.Hagoort,“EEGthetaandgammaresponsestosemanticviolationsinonline sentenceprocessing,”Brain and Language,vol.96,no.1,pp.90-105,2006. [39] M.vanElk,H.T.vanSchie,R.A.Zwaan, et al.,“Thefunctionalroleofmotoractivationinlanguageprocessing: Motorcorticaloscillationssupportlexical-semanticretrieval,”Neuroimage,vol.50,no.2,pp.665-677,2010. [40] H.J.Park,J.J.Kim,S.K.Lee, et al.,“Corpuscallosalconnectionmappingusingcorticalgraymatterparcellation andDT-MRI,”Human Brain Mapping,vol.29,no.5,pp.503-516,2006. [41] I.Fried,C.L.Wilson,K.A.MacDonald, et al.,“Electriccurrentstimulateslaughter,”Nature,vol.391,no.6668,pp. 650:1-2,1998. [42] Xue-Yan Liwasbornin1979.ShereceivedtheB.S.degreeinEnglishfromChangchunUniversity of Technology, Changchun in 2002 and the M.S. degree in English education from Maquarie University, Sydney in 2007. She received the Ph.D. degree from the Faculty of Information Technology, University of Jyväskylä, Jyväskylä in 2018. Now she is working with the School of Foreign Languages, Dalian University of Technology, Dalian, as an associate professor. Her researchinterestsincludeneurolinguisticsandtechnologydesign. LIet al.:ProcessingMechanismofChineseVerbalJokes:EvidencefromERPandNeuralOscillations 275
Hui-Li Wang was born in 1968. She received the B.S. degree in English from Shanghai International Studies University, Shanghai in 1989 and the M.S. degree in education from Dalian University of Technology in 2004. She received the Ph.D. degree in engineering from Dalian University of Technology in 2008. She is currently working with the Institute for Language and Cognition, School of Foreign Languages, Dalian University of Technology, as a professor. Her researchinterestsincludeneurolinguistics,cognitivelinguistics,andsecondlanguageacquisition. Pertti SaariluomareceivedbothM.A.andPh.D.degreesinpsychologyfromUniversityofTurku, Turkuin1978and1984,respectively.From1981to1982andin1996,hewasavisitingresearcher with University of Oxford, Oxford. In 1989, he was a visiting researcher with University of Cambridge,Cambridge. In2008, 2009,and 2011, he was a visiting researcher with University of Granada,Granada.Hehasintroducedanumberofindependentscientificparadigms.Heiscurrently workingwiththeFacultyofInformationTechnology,UniversityofJyväskylä.Hisresearchinterests includepsychologyandcognitivescienceandtechnologydesign. Guang-Hui Zhang was born in 1989. He received his B.S. degree in telecommunications engineering from Dalian Polytechnic University, Dalian in 2015, and the M.S. degree in signal processingfromDalianUniversityofTechnologyin2018.HeiscurrentlypursuinghisPh.D.degree with the School of Foreign Languages, Dalian University of Technology. His research interests include signal processing in electroencephalography, principal component analysis, independent componentanalysis,andtime-frequencyanalysis. Yong-Jie Zhuwasbornin1987.HereceivedtheB.S.andM.S.degreesinbiomedicalengineering fromDalianUniversityofTechnologyin2013and2016,respectively.NowheispursuingthePh.D. degree with University of Jyväskylä. His research interests include signal processing in electroencephalography,principalcomponentanalysis,independentcomponentanalysis,andtimefrequencyanalysis. Chi Zhang was born in 1987. He received his B.S., M.S., and Ph.D. degrees from Northeastern University,Shenyangin2010,2012,and2016,respectively.HeiscurrentlyworkingwiththeSchool ofForeignLanguage,DalianUniversityofTechnology,asalecturer.Hisresearchinterestsinclude biomedicalsignalprocessing,brain-computerinterface,andcognitivescience. 276 JOURNALOFELECTRONICSCIENCEANDTECHNOLOGY,VOL.17,NO.3,SEPTEMBER2019
Feng-Yu CongreceivedtheB.S.degreeinpowerandthermaldynamicengineeringandthePh.D. degreeinmechanicaldesignandtheoryfromShanghaiJiaoTongUniversity,Shanghaiin2002and 2007,respectively.HealsoreceivedthePh.D.degreeinmathematicalinformationtechnologyfrom Universityof Jyväskylä in 2010. SinceMarch 2007, hehas been working with theDepartment of Mathematical Information Technology, University of Jyväskylä, as a postdoctoral researcher from March2007toAugust2008,ajuniorlecturer(facultyposition)fromSeptember2008toJune2011, and a tenure-track faculty position from July 2011 to December 2013. In May 2012, he was conferredthetitleofdocent(adjunctprofessor,tenuredacademictitle,rankingbetweenlecturerand fullprofessor)insignalprocessingattheDepartmentofMathematicalInformationTechnology,UniversityofJyväskylä. SinceDecember2013,he hasbeenaprofessorwiththe DepartmentofBiomedicalEngineering,FacultyofElectronic InformationandElectricalEngineering,DalianUniversityofTechnology.HeisalsoanIEEESeniorMember(from2013), editorial board member of Journal of Neuroscience Methods (from 2013), and program committee member of LVA/ICA2012&2015, MLSP2013-2015. His research interests include signal processing, cognitive neuroscience, brain science,andmachinelearning. Tapani RistaniemireceivedhisM.S.degreeinmathematicsandPh.D.degreeinsignalprocessing for communications from University of Jyväskylä in 1995 and 2000, respectively. He is currently workingwithUniversityofJyväskylä,asaprofessor.Hisresearchinterestsincludesignalprocessing forwirelesscommunications,brainsignalprocessing,radioresourcemanagement,andoptimization forwirelessnetworks. LIet al.:ProcessingMechanismofChineseVerbalJokes:EvidencefromERPandNeuralOscillations 277