Staying at the front line of literature : How can topic modelling help researchers follow recent studies?

This is a sel -a chi ed e sion o an o iginal a icle. This e sion may di e om he o iginal in pagina ion and ypog aphic de ails. Au ho (s): Ti le: Yea : Ve sion: Copy igh : Righ s: Righ s u l: Please ci e he o iginal e sion: CC BY-NC-ND 4.0 h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/ S aying a he on line o li e a u e : How can opic modelling help esea che s ollow ecen s udies? © 2021 he Au ho s Published e sion Lämsä, Joni; Espinoza, Ca alina; Tuhkala, A i; Hämäläinen, Raija Lämsä, J., Espinoza, C., Tuhkala, A., & Hämäläinen, R. (2021). S aying a he on line o li e a u e : How can opic modelling help esea che s ollow ecen s udies?. F on line Lea ning Resea ch, 9(3), 1-12. h ps://doi.o g/10.14786/ l . 9i3.645 2021 F on line Lea ning Resea ch Vol. 9 No. 3 (2021) 1-12 ISSN 2295-3159 In o co esponding au ho : Joni Lämsä, P.O. Box 35, FI-40014, Uni e si y o Jy äskylä, Finland, [email p o ec ed] DOI: h ps://doi.o g/10.14786/ l . 9i3.645 S aying a he on line o li e a u e: How can opic modelling help esea che s ollow ecen s udies? Joni Lämsä1, Ca alina Espinoza2, A i Tuhkala3, & Raija Hämäläinen1 1Depa men o Educa ion, Uni e si y o Jy äskylä, Finland 2Cen e o Ad anced Resea ch in Educa ion, Uni e si y o Chile, Chile 3Finnish Ins i u e o Educa ional Resea ch, Uni e si y o Jy äskylä, Finland A icle ecei ed 23 June 2020 / A icle e ised 20 Decembe / Accep ed 26 Ma ch 2021 / A ailable online 14 Ap il Abs ac S aying a he on line in lea ning esea ch is challenging because many ields a e apidly de eloping. One such ield is esea ch on he empo al aspec s o compu e -suppo ed collabo a i e lea ning (CSCL). To ob ain an o e iew o hese ields, sys ema ic li e a u e e iews can cap u e pa e ns o exis ing esea ch. Howe e , conduc ing sys ema ic li e a u e e iews is ime-consuming and do no e eal u u e de elopmen s in he ield. This s udy p oposes a machine lea ning me hod based on opic modelling ha akes a icles om a sys ema ic li e a u e e iew on he empo al aspec s o CSCL (49 o iginal a icles published be o e 2019) as a s a ing poin o desc ibe he mos ecen de elopmen in his ield (52 new a icles published be ween 2019 and 2020). We aimed o explo e how o iden i y new ele an a icles in his ield and ela e he o iginal a icles o he new a icles. Fi s , we ained he opic model wi h he Resul s, Discussion, and Conclusion sec ions o he o iginal a icles, enabling us o co ec ly iden i y 74% (n = 17) o new and ele an a icles. Second, clus e isa ion o he o iginal and new a icles indica ed ha he ield has ad anced in i s new and ele an a icles because he opics conce ning he egula ion o lea ning and collabo a i e knowledge cons uc ion ela ed 26 o iginal a icles o 10 new a icles. New i ele an s udies ypically eme ged in clus e s ha did no include any speci ic opic wi h a high opic occu ence. Ou me hod may p o ide esea che s wi h esou ces o ollow he pa e ns in hei ields ins ead o conduc ing epe i i e sys ema ic li e a u e e iews. Keywo ds: au oma ic con en analysis; compu e -suppo ed collabo a i e lea ning; li e a u e e iew; empo al analysis; opic model ! Lämsä e al 2 | FLR 1. In oduc ion Resea ch in lea ning sciences has become mo e in e disciplina y because inc easingly complex da ase s and me hods may equi e he expe ise o compu e scien is s and signal p ocesso s. This in e disciplina y collabo a ion opens up he possibili y o new publica ion o ums in he lea ning sciences. Howe e , his could also make ho ough sys ema ic o hema ic li e a u e e iews (see G ube e al., 2020) e en mo e a duous. Thus, i would be use ul i he as amoun o wo k done by schola s when conduc ing sys ema ic li e a u e e iews could be exploi ed when moni o ing how a speci ic line o esea ch would p oceed. I ele an u u e s udies can be au oma ically iden i ied and ela ed o p e ious esea ch, his would dec ease he need o pe o m ecu ing sys ema ic li e a u e e iews on simila opics, hus a o ding esea che s mo e wo king hou s o ad ance in hei ields. To add ess hese aspi a ions, we p esen a machine lea ning–based me hod ha akes a icles om a manual sys ema ic li e a u e e iew as a s a ing poin o desc ibe he ecen de elopmen s in he ield. We illus a e he po en ial o ou inno a i e me hod in he con ex o esea ch ocusing on he empo al analysis o compu e -suppo ed collabo a i e lea ning (CSCL). This ield o esea ch o ms a pa icula ly p omising basis o s udying i s p og ess because he s udies ocusing on he empo al aspec s o CSCL a e inc easingly being published and in ol e in e disciplina y collabo a ion (e.g., Hadwin, 2021; Lämsä e al., 2021). In his s udy, we de ine he empo al analysis o CSCL as analysing he cha ac e is ics o e en s o he in e ela ions be ween hese e en s o e ime. The e en s may ela e o lea ne in e ac ion, hough s and ideas de eloped du ing he in e ac ion and he use o echnological esou ces o media e he in e ac ion (see Lämsä e al., 2021). A empo al analysis o CSCL may bene i bo h p ac i ione s and esea che s by e ealing how (no only wha ) lea ning occu s in CSCL se ings (Lämsä, 2020), pa icula ly now when COVID-19 highligh s he need o e ec i e CSCL mo e han e e (Jä elä & Rosé, 2020). When we manually e iewed he li e a u e ocusing on he empo al aspec s o CSCL (see sec ion 2 and Lämsä e al., 2021), we ound ha he in e disciplina y collabo a ion in his ield has caused challenges ega ding he commensu abili y and compa abili y o he s udies. Pa icula ly, he s udies seemed o be agmen ed in e ms o hei heo e ical amewo ks (c . Hew e al., 2019), me hodologies, and esul s and implica ions. This inding implies ha bo h p ac i ione s and esea che s may s uggle wi h s aying a he on line conce ning he big pic u e o CSCL and i s esea ch because o his agmen a ion. P ac i ione s may bene i om ou me hod i i can il e applicable esea ch o suppo hem in he design and implemen a ion o esea ch-based CSCL inno a ions. Simila ly, ou me hod can bene i esea che s because i can illus a e whe he and how he ecen esea ch has con ibu ed o p io s udies. We in es iga e he added alue o ou me hod o p ac i ione s and esea che s by add essing he ollowing esea ch ques ions: RQ1: How and o wha ex en can a machine lea ning–based me hod be used o iden i y new ele an a icles in he ield o manual sys ema ic li e a u e e iew? RQ2: How and o wha ex en can he machine lea ning–based me hod be used o ela e new and o iginal a icles o each o he ? 2. Me hodology When manually e iewing he li e a u e on he empo al aspec s o CSCL in Feb ua y 2019 (see Lämsä e al., 2021), we ca e ully selec ed he sea ch e ms conce ning empo ali y, collabo a i e lea ning, and compu e -suppo ed lea ning. We used he Educa ion Resou ces In o ma ion Cen e (ERIC), Scopus, and Web o Science da abases and iden i ied 436 a icles, o which we manually sc eened and assessed hei eligibili y. In his s udy, we included 49 pee - e iewed jou nal a icles ha ocused on he empo al analysis o CSCL o u he analysis (o iginal a icles). To ind new a icles, ! Lämsä e al 3 | FLR we epea ed he li e a u e sea ches wi h he same sea ch e ms and da abases in Feb ua y 2020 as o he o iginal a icles. The sea ches ound 88 a icles ha had been published be ween Feb ua y 2019 and 2020. F om hese 88 a icles, we excluded 36 a icles, o which 31 we e duplica es, h ee had no ull ex a ailable, one was a con e ence p oceeding a icle, and one was al eady included in he se o he o iginal a icles. In he ollowing analyses, we e e o hese included 52 pee - e iewed jou nal a icles as a se o new a icles. The u ilised machine lea ning–based me hod was g ounded on a na u al language p ocessing echnique known as opic modelling, which is based on s a is ical algo i hms ha ind opics in a collec ion o documen s (Boyd-G abe e al., 2017). These opics a e anked lis s o wo ds, whe e each wo d has a p obabili y o belonging o a opic (see Table 1), o mo e o mally, opics a e p obabili y dis ibu ions o e ocabula ies. In he ollowing sec ions, we desc ibe how he o iginal a icles we e exploi ed o build he opic models ha , in u n, we e used o iden i y he new ele an a icles (RQ1) and ela e hem o he o iginal a icles (RQ2). Figu e 1 summa ises ou p ocedu e. Figu e 1: P ocedu e o desc ibing o iginal and new a icles o add ess he esea ch ques ions (RQs) 2.1 Ex ac ing and p ep ocessing ex Fi s , we ex ac ed aw ex om he o iginal and new a icles and emo ed ables, igu es, o mulas, bulle poin s, oo no es, and page numbe s. Second, we sepa a ed he di e en sec ions o he a icles unde he ollowing headings: In oduc ion, Theo e ical F amewo k, Me hodology, Resul s, Discussion, and Conclusion. Howe e , because no all he a icles had all o hese sec ions (e.g., an a icle may ha e a combined Resul s and Discussion sec ion), we decided o combine he sec ions in o h ee wide sec ions: (1) In oduc ion and Theo e ical F amewo k, (2) Me hodology, and (3) Resul s, Discussion, and Conclusion. Then, we ca ied ou ex p ep ocessing, including common ex cleaning, such as ans o ming ex o lowe case and emo ing symbols and in equen wo ds. Finally, we u ilised ! Lämsä e al 4 | FLR he Na u al Language Toolki (Bi d e al., 2009) o pe o m wo d s emming ( educing wo ds o hei oo o m) and common English s op wo d emo al (e.g., he, a , is). 2.2 T aining opic models We used la en Di ichle alloca ion (LDA) (Blei e al., 2003) and he Gensim lib a y (Rehu ek & Sohjka, 2010) o ain opic models o each sec ion 1–3 o he o iginal a icles. The ou pu o aining opic models includes bo h a lis o opics and he ained opic model i sel . The ained opic model can p ocess new ex and measu e he p esence o he lis ed opics. We pe o med a sensi i i y analysis based on opic cohe ence alues (p o ided by he Gensim lib a y) o ind an app op ia e numbe o opics o each sec ion. As an ou come, we had ained h ee opic models, one o each sec ion, ha all included 17 opics. 2.3 Labelling opics om opic models The ained opic models con ained a lis o opics ound in each sec ion. We labelled he opics by analysing he mos ep esen a i e wo ds and u ilising expe knowledge om he manual sys ema ic e iew o he li e a u e. I possible, we labelled he opics based on he heo e ical amewo k o which he mos ep esen a i e wo ds e e . We demons a e his idea in Table 1 using opic models o sec ion 3 as an example, p esen ing he labels and he 10 mos ep esen a i e wo ds. Fo example, opic 1 ( empo al aspec s o CSCL) is a gene ic opic ha illus a es a s age in he empo al analysis p ocedu e. Namely, esea che s code messages o g oups o s uden s, a e which hey analyse he ypical sequences o messages. This kind o sequen ial analysis e eals wha kind o messages ollow each o he in a sho empo al con ex whose du a ion may be a ew messages ( he wo ds wi h i alics e e o he 10 mos ep esen a i e wo ds om opic 1). 2.4 Ob aining opic occu ence in o iginal a icles In LDA, a icles a e ep esen ed as lis s o opic p obabili ies; he goal is o ind he opic p obabili ies o a documen ha a e be e sui ed o ebuild he documen by andomly selec ing wo ds. Fo example, i an a icle has a highe opic p obabili y o opic 16 compa ed wi h o he opics (see Table 1), mos o he wo ds in he a icle can be selec ed om he op o opic 16. We e e o opic p obabili ies in an a icle as a opic occu ence o dis inguish hem om wo ds’ p obabili ies inside a opic. When we used opic models o sec ions 1–3, we ob ained 51 opic occu ences o each o iginal a icle (17 opic occu ences o each opic model). 2.5 Ob aining opic occu ence in new a icles The p ocess used o he new a icles was e y simila o he one applied o he old a icles (Figu e 1). The only di e ence was ha we di ec ly applied he ained opic models o sec ions 1–3 o ob ain he opic occu ences o he new a icles. We illus a e he opic occu ences o o iginal, new ele an , and new i ele an a icles using he opic model o sec ion 3 in Figu e 2. Fo each a icle, some opics ha e a highe p obabili y han he es (e.g., opic 16 is mo e ele an o an o iginal a icle han o a new i ele an a icle; see (a) and (c) in Figu e 2). The e o e, we expec o ind seman ic simila i y be ween opic occu ences ha ha e sho e dis ances. ! Lämsä e al 5 | FLR Table 1 Fi e opics and he assigned opic labels, including he 10 mos ep esen a i e wo ds om he opic model o sec ion 3 (Resul s, Discussion, and Conclusion). Topic 1: Tempo al aspec s o compu e - suppo ed collabo a i e lea ning Topic 7: Regula ion o lea ning and lea ning pe o mance Topic 8: Regula ion o lea ning Topic 11: Socially sha ed me acogni i e egula ion (SSMR) Topic 16: Collabo a i e knowledge cons uc ion NUMBER Model Regul SSMR Discuss S uden G oup Lea n P ocess G oup Signi ic Pe o m Collabo Phase S uden Code Focus Task Th ead Knowledg G oup S uden Social S udi Beha iou Analysi SSRL1 Sha e Resea ch Cons uc Show Collabo S uden Di e Lea n Sequenc Challeng G oup Inqui i Resul Knowledg Di e Resul No e P ocess Messag Indi idu Discuss Da a Pa e n 1Socially sha ed egula ion o lea ning (a) (b) (c) Figu e 2: The opic occu ence o (a) an o iginal a icle, (b) a new ele an a icle, and (c) a new i ele an a icle ob ained using he opic model o sec ion 3. The dis ance be ween (a) and (b) was 0.33, be ween (a) and (c) was 0.65, and be ween (b) and (c) was 0.49. To answe RQ1, he i s and second au ho s sc eened and labelled he new 52 a icles manually as ele an o i ele an ega ding he analysis o he empo al aspec s o CSCL. In he i s phase, we sc eened he jou nal and i le o he a icles and labelled he s udies ha did no ha e a lea ning o ! Lämsä e al 6 | FLR ins uc ional con ex as i ele an (n = 27, e.g., s udies om en i onmen al sciences). In he second phase, we also sc eened he abs ac o he a icles and labelled he s udies ha did no ocus on CSCL and analyse i s empo al aspec s as i ele an (n = 2, e.g., a s udy ha ocused me ely on lea ning pe o mance). We sol ed he disag eemen s be ween he i s and second au ho s in he common mee ings among all he au ho s. Al oge he , 23 new a icles ocused on he analysis o he empo al aspec s o CSCL ( ele an ), while 29 a icles did no (i ele an ). Nex , o each opic model, we measu ed he dis ance be ween he co esponding opic occu ences o he new a icles and o iginal a icles. The sho e he dis ance be ween wo a icles, he mo e simila he opic occu ences (Figu e 2). Fo each new a icle, we kep he dis ance o he closes o iginal a icle (i.e., he mos simila because a new ele an a icle migh no be ela ed o e e y a icle in he manual sys ema ic li e a u e e iew). Finally, we compa ed he dis ances be ween he ele an and i ele an a icles. We selec ed he mos sui able opic model so ha he opic occu ences o he new ele an a icles we e simila o he ones om he o iginal a icles. To answe RQ2, we used he a icles’ opic occu ences om he p e iously selec ed opic model (RQ1). We measu ed he simila i y be ween opic occu ences using he Euclidean dis ance, and we applied hie a chical clus e ing o ind g oups o simila a icles. We pe o med he clus e ing in h ee le els: he oo , wo subg oups, and he lea es. The oo o he clus e ing con ains all he a icles: 52 new a icles and 49 o iginal a icles. The oo was hen di ided in o wo subg oups, deno ing he g ea es dis ance be ween he a icles belonging o di e en subg oups. The lea es a e g oups o a icles o a ying sizes. We in e p e ed he clus e s by examining he opic occu ences (Figu e 2) and p e iously assigned opic labels (Table 1). 3. Resul s 3.1 The opic model ained wi h he Resul s, Discussion, and Conclusion sec ions iden i ied new ele an a icles mos accu a ely. We iden i ied new ele an a icles ela ing o he empo al aspec s o CSCL by measu ing he dis ance be ween a new a icle and he closes o iginal a icle. The esul s showed ha o he h ee opic models, he ele an new a icles we e close o he o iginal a icles han he i ele an a icles (Figu e 3). Pa icula ly, he opic model o sec ion 3, which we ained wi h Resul s, Discussion, and Conclusion sec ions, ga e he bes esul s because he dis ance be ween he new ele an a icles and he closes o iginal a icle o e lapped he leas wi h he dis ance be ween new i ele an a icles and he closes o iginal a icle [Figu e 3 (c)]. When we used he opic model o sec ion 3 and he dis ance o 0.27 as a h eshold, 71% o he new a icles, which we e close han he h eshold, we e ele an . Those ele an a icles ep esen 74% o he o al ele an a icles, which minimised he numbe o i ele an a icles. When we used opic models o sec ions 1 and 2, he dis ances be ween he o iginal a icles and new ele an a icles o e lapped mo e wi h new i ele an a icles [Figu e 3 (a) and (b)]. Table 2 summa ises ou esul s i he dis ance o 0.27 is conside ed o he h eshold. ! Lämsä e al 7 | FLR (a) (b) (c) Figu e 3: Boxplo s o he dis ances be ween ele an and i ele an new a icles and he closes o iginal a icle sepa a ely o (a) opic model o sec ion 1, (b) opic model o sec ion 2, and (c) opic model o sec ion 3. ! Lämsä e al 8 | FLR Table 2 The numbe s o ele an and i ele an a icles iden i ied and missed when using h ee opic models Topic model (sec ion 1): In oduc ion and Theo e ical amewo k Topic model (sec ion 2): Me hodology Topic model (sec ion 3): Resul s, Discussion, and Conclusion Rele an iden i ied ( ue posi i es) 10 17 17 Rele an missed ( alse nega i es) 13 6 6 I ele an iden i ied as ele an ( alse posi i e) 3 14 7 I ele an iden i ied as i ele an ( ue nega i es) 26 15 22 P ecision (p opo ion o he ue posi i es o he sum o ue and alse posi i es) 0.77 0.55 0.71 Recall (p opo ion o he ue posi i es o he sum o he ue posi i es and alse nega i es) 0.43 0.74 0.74 3.2 A ew opics wi h high opic occu ence ela e new ele an o o iginal a icles Figu e 4 shows he ou come o he hie a chical clus e ing. When in e p e ing Figu e 4, based on he CSCL heo e ical amewo ks, a ew opics conce ning collabo a i e knowledge cons uc ion and egula ion o lea ning ela e new ele an a icles o o iginal a icles. Topic 16 (see Table 1) ela es i e new ele an a icles o 17 o iginal a icles ( he lea es wi h double bo de s), and hese a icles mos ly belong o a smalle subg oup. Topics 7, 8, and 11 (see Table 1) ela e i e new ele an a icles o nine o iginal a icles ( he lea es wi h bold bo de s), and hese a icles belong o a la ge subg oup. Mos o he new i ele an a icles (n = 28) we e clus e ed in o h ee di e en lea es (Figu e 4). F om his se , 17 a icles appea ed in he lea es wi h di e en opics. Mo eo e , 11 a icles appea ed in he lea ha included only one new ele an a icle and one o iginal a icle. Mos o he o iginal a icles (n = 31) had a opic wi h a alue highe han 0.45. In con as , he clus e s o med by a ious opics con ained a icles in which he opic occu ence o he mos impo an opic was less han 0.2, meaning ha he e we e no p edominan opics. Because opic occu ence is a p obabili y dis ibu ion (mus sum up o one), opic occu ence is mo e sca e ed i no pa icula opic is mo e signi ican [see Figu e 2 (c)]; his ea u e clus e s oge he mos o he i ele an a icles, bu i also mixes i ele an a icles wi h ele an a icles ha ha e se e al impo an opics. In ou case, 12 new ele an a icles eme ged in he lea es wi h di e en opics.