Analyzing user reviews of messaging Apps for competitive analysis

Analyzing Use Re iews o Messaging Apps o Compe i i e Analysis Wenyi Liang Disse a ion p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Da a Science and Ad anced Analy ics i NOVA In o ma ion Managemen School Ins i u o Supe io de Es a ís ica e Ges ão de In o mação Uni e sidade No a de Lisboa ANALYZING USER REVIEWS OF MESSAGING APPS FOR COMPETITIVE ANALYSIS by Wenyi Liang Disse a ion p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Da a Science and Ad anced Analy ics Ad iso : P o . Mau o Cas elli Augus 2021 ii DEDICATION This disse a ion is dedica ed o my pa en s o hei cons an unde s anding, encou agemen , and suppo . This wo k is also dedica ed o many iends who ha e enligh ened me wi h hei d eams, c ea i i y, and b a e y. iii ACKNOWLEDGEMENTS I would like o exp ess my since e g a i ude o my supe iso P o . Mau o Cas elli o his a i ma ion o he esea ch opic, suppo i e encou agemen , p omp eedback, and pa ien guidance in comple ing his disse a ion. I would also like o hank he acul y in he p og am o Da a Science and Ad anced Analy ics. The aluable knowledge and skills hey had augh me laid he academic ounda ion o his wo k. My special hanks go o P o . Fe nando Pe es and P o . Nuno Alpalhão who ga e me ex a lessons and guidance on homewo k and p ojec s. I wish o acknowledge he assis ance p o ided by he s a o lib a y and documen a ion se ices as well as academic se ices o NOVAIMS in inding some esea ch ma e ials and sol ing he issue o hesis egis a ion. I would like o ex end my hanks o my classma es who had e e helped me wi h he s udy du ing he i s academic yea . They had inspi ed me wi h p ac ical knowledge and in e es ing hough s. Ad ices p o ided by E nes o on eading esea ch pape s we e eally help ul. In he end, my app ecia ion goes o my o me supe iso s in wo k who come om di e en coun ies bu coinciden ally assigned me he wo k o compe i i e analysis o pee companies o mobile apps. This wo k expe ience, o some ex en , con ibu ed o he esea ch opic o his disse a ion. i ABSTRACT The ise o a ious messaging apps has esul ed in in ensi ely ie ce compe i ion, and he e a o Web 2.0 enables business manage s o gain compe i i e in elligence om use -gene a ed con en (UGC). Tex -mining UGC o compe i i e in elligence has been d awing g ea in e es o esea che s. Howe e , ele an s udies mos ly ocus on indus ies such as hospi ali y and p oduc s, and ew s udies applied such echniques o e ec i ely pe o m compe i i e analysis o messaging apps. He e, we conduc ed a compe i i e analysis based on opic modeling and sen imen analysis by ex - mining 27,479 use e iews o ou iOS messaging apps, namely Messenge , Wha sApp, Signal and Teleg am. The esul s show ha he pe o mance o opic modeling and sen imen analysis is encou aging, and ha a combina ion o he ex ac ed app aspec -based opics and he adjus ed sen imen sco es can e ec i ely e eal meaning ul compe i i e insigh s in o use conce ns, compe i i e s eng hs and weaknesses as well as changes o use sen imen s o e ime. We an icipa e ha his s udy will no only ad ance he exis ing li e a u e on compe i i e analysis using ex mining echniques o messaging apps bu also help exis ing playe s and new en an s in he ma ke o sha pen hei compe i i e edge by be e unde s anding hei use needs and he indus y ends. KEYWORDS Compe i i e analysis; Topic modeling; Sen imen analysis; Tex mining; Use e iews; Messaging apps INDEX 1 In oduc ion ......................................................................................................................... 1 2 Rela ed wo ks ....................................................................................................................... 3 2.1 Compe i i e analysis by ex -mining UGC ................................................................................ 3 2.2 Topic modeling ......................................................................................................................... 4 2.3 Sen imen analysis ................................................................................................................... 8 3 Me hodology ...................................................................................................................... 11 3.1 Da a collec ion ....................................................................................................................... 12 3.2 Da a p ep ocessing ................................................................................................................ 13 3.2.1 Con ac ion expanding ....................................................................................................... 13 3.2.2 Tex cleaning ...................................................................................................................... 13 3.2.3 Wo d co ec ion and no maliza ion .................................................................................. 13 3.2.3.1 Spell check .................................................................................................................. 13 3.2.3.2 B i ish-Ame ican spelling no maliza ion .................................................................... 14 3.2.3.3 Spelling co ec ion and abb e ia ion expansion ....................................................... 14 3.2.4 POS agging and lemma iza ion ......................................................................................... 17 3.2.5 Non-English e iews il e ing ............................................................................................. 18 3.2.6 Fea u e ex ac ion .............................................................................................................. 18 3.2.7 Cus omized s op wo d emo al ......................................................................................... 19 3.2.8 Re iew p uning .................................................................................................................. 19 3.3 Topic modeling ....................................................................................................................... 20 3.3.1 NMF opic model ................................................................................................................ 20 3.3.2 E alua ion o ex ac ed opics ........................................................................................... 21 3.4 Sen imen analysis ................................................................................................................. 21 3.4.1 VADER compound sco e .................................................................................................... 21 3.4.2 Weigh ed sen imen sco e ................................................................................................. 21 3.4.3 E alua ion o sen imen analysis ....................................................................................... 22 3.5 Compe i i e analysis .............................................................................................................. 23 3.5.1 Re iew dis ibu ions and a e age sen imen sco es ......................................................... 23 3.5.1.1 Visual e iew dis ibu ion .......................................................................................... 24 3.5.1.2 Visual compa ison o a e age sen imen sco es ....................................................... 24 3.5.2 Sen imen e olu ion ........................................................................................................... 24 i 4 Resul s and discussion......................................................................................................... 25 4.1 Resul s .................................................................................................................................... 25 4.1.1 Topic modeling ................................................................................................................... 25 4.1.1.1 Ex ac ed opics .......................................................................................................... 25 4.1.1.2 E alua ion esul o ex ac ed opics ......................................................................... 29 4.1.2 E alua ion esul o sen imen analysis ............................................................................. 30 4.1.3 Resul s o compe i i e analysis .......................................................................................... 32 4.1.3.1 Re iew dis ibu ions and a e age sen imen sco es ................................................. 32 4.1.3.2 Visual e iew dis ibu ion .......................................................................................... 33 4.1.3.3 Visual compa ison o a e age sen imen sco es ....................................................... 36 4.1.3.4 Sen imen e olu ion ................................................................................................... 37 4.2 Discussion ............................................................................................................................... 44 4.2.1 Topic modeling ................................................................................................................... 44 4.2.2 Sen imen analysis ............................................................................................................. 46 4.2.3 Compe i i e analysis .......................................................................................................... 46 5 Conclusion .......................................................................................................................... 50 6 Limi a ions and ecommenda ions o u u e wo ks ............................................................. 51 7 Bibliog aphy ....................................................................................................................... 52 Appendix A. Sampled e iews wi h w ong opics assigned by he NMF opic model ..................... 59 Appendix B. Sampled e iews wi h inconsis en labels o sen imen pola i y on weigh ed sen imen sco es ........................................................................................................................................ 61 ii LIST OF FIGURES Figu e 1: The singula alue decomposi ion o LSA wi h k opics .............................................. 5 Figu e 2: Illus a ion o he aspec model .................................................................................. 6 Figu e 3: Illus a ion o he la en Di ichle alloca ion ............................................................... 6 Figu e 4: Illus a ion o he non-nega i e ma ix ac o iza ion ................................................. 7 Figu e 5: The esea ch amewo k o compe i i e analysis using opic modeling and sen imen analysis ............................................................................................................................. 11 Figu e 6: Non-nega i e ma ix decomposi ion on ea u e e ms ............................................ 20 Figu e 7: Topics ex ac ed om he NMF model ..................................................................... 25 Figu e 8: Re iew dis ibu ion by opic o Messenge .............................................................. 34 Figu e 9: Re iew dis ibu ion by opic o Wha sApp ............................................................... 35 Figu e 10: Re iew dis ibu ion by opic o Signal ..................................................................... 35 Figu e 11: Re iew dis ibu ion by opic o Teleg am ............................................................... 36 Figu e 12: Compa ison o a e age sen imen sco es by opic ................................................. 37 Figu e 13: Sen imen e olu ion by opic .................................................................................. 38 iii LIST OF TABLES Table 1: In o ma ion o app and use e iews ......................................................................... 12 Table 2: Sample o da ase wi h selec ed ields ....................................................................... 12 Table 3: Addi ional B i ish-Ame ican spellings ........................................................................ 14 Table 4: Manual spelling co ec ions ....................................................................................... 15 Table 5: Abb e ia ion expansions ............................................................................................ 17 Table 6: POS ags o ea u e ex ac ion .................................................................................. 18 Table 7: Remo al p ocess o cus omized s op wo ds .............................................................. 19 Table 8: Con usion ma ix o sen imen e alua ion ............................................................... 22 Table 9: Explana ion o TP, FP, TN and FN ............................................................................... 23 Table 10: Accu acies o opic ex ac ion .................................................................................. 30 Table 11: Con usion ma ix o sen imen e alua ion on weigh ed sen imen sco es ............. 31 Table 12: Con usion ma ix o sen imen e alua ion on use a ings...................................... 31 Table 13: Con usion ma ix o sen imen e alua ion on no malized VADER compound sco es .......................................................................................................................................... 31 Table 14: Pe o mance o sen imen analysis .......................................................................... 32 Table 15: Summa y o e iew dis ibu ions and a e age sen imen sco es ............................ 33 Table 16: S a is ical agg ega ion o a e age sen imen sco es by mon h, opic and app ....... 43 Table 17: Compa ison o keywo ds in ex ac ed opics ........................................................... 45 6 Figu e 2 Illus a ion o he aspec model No e. Adap ed om P obabilis ic la en seman ic analysis — PLSA by Se g Ka po ich, 2013, Wikimedia Commons (h ps://commons.wikimedia.o g/wiki/File:Вероятностный_латентно- семантический_анализ.png). CC BY-SA 3.0. Un o una ely, pLSA is no capable o assigning p obabili y o p e iously unseen documen s and is p one o o e i ing due o he g ow h in pa ame e s wi h he inc easing numbe o documen s and wo ds (Blei e al., 2003). To add ess hese p oblems, Blei e al. (2003) p esen ed he la en Di ichle alloca ion (LDA), which is a hie a chical Bayesian model o h ee le els, namely documen , opic and wo d. Acco ding o hei s udy, a documen con ains mul iple opics wi h di e en p obabili ies 𝜃, whose dis ibu ion ollows he Di ichle dis ibu ion, and ano he Di ichle dis ibu ion also applies in he p obabili y dis ibu ion o wo ds 𝜑 in a opic (Figu e 3). The pa ame e s o he p io dis ibu ions, i.e., Di ichle dis ibu ions, o opic dis ibu ion 𝜃 and wo d dis ibu ion 𝜑 a e 𝛼 and 𝛽 espec i ely. Compa ed wi h he pLSA app oach, he LDA me hod gene alizes easily o unseen ex s because i ob ains he pos e io dis ibu ion o he opic mix u e weigh s by combining hei p io dis ibu ion wi h he sample da a a he han ea s hese weigh s as a la ge se o indi idual pa ame e s (Blei e al., 2003). Figu e 3 Illus a ion o he la en Di ichle alloca ion No e. F om Pla e no a ion o he Smoo hed LDA Model by Slxu.public, 2009, Wikimedia Commons (h ps://commons.wikimedia.o g/wiki/File:Smoo hed_LDA.png). CC BY-SA 3.0. 7 Some de i a ions based on he LDA me hod we e p esen ed o speci ic asks and imp o emen s. Combining he adi ional LDA and a s a is ical p ocess, Blei e al. (2010) c ea ed he Hie a chical la en Di ichle alloca ion (hLDA) o lea n opic hie a chies om complex da a. In hLDA, he la en opics a e s uc u ed in a ee whe e each node ep esen s a opic wi h i s opic e ms. Aga wal and Chen (2010) p oposed he ma ix ac o iza ion h ough LDA ( LDA) o p edic use a ings in ecommende sys ems such as con en ecommenda ion, ad a ge ing and web sea ch, whose i ems a e a icles, ads and web pages espec i ely. The LDA me hod egula izes he i em ac o s h ough LDA p io s. In he domain o mobile apps, Pa k e al. (2015) p esen ed he AppLDA opic model ha e ie es sha ed opics om app desc ip ions and use e iews and disca ds e iews which only con ains misaligned opics wi h app desc ip ions. This opic model aims o ind ou he key aspec s o apps and in e ela e he ocabula y be ween app de elope s and use s. Mo eo e , o handle he es ic ion due o a single Di ichle p io dis ibu ion o opic p opo ions, Chien e al. (2018) in oduced he la en Di ichle mix u e model (LDMM), which alloca es mul iple Di ichle p io dis ibu ions o lea n he la en opics and hei p opo ions. Besides unsupe ised asks such as opic modeling and documen clus e ing, he LDMM was also ex ended o a supe ised LDMM o documen classi ica ion (Chien e al., 2018). Ano he common opic model is he non-nega i e ma ix ac o iza ion (NMF) (Lee & Seung, 1999), whose de elopmen is ela i ely independen o he a o emen ioned opic models. This model was ini ially in oduced o lea ning pa s o aces and seman ic ea u es o ex . The basic idea o NMF is inding wo non-nega i e ma ices whose p oduc app oxima es a gi en non-nega i e ma ix, he eby ac o izing he o iginal non-nega i e ma ix 𝑉 in o a basis ma ix 𝑊 and a coe icien ma ix 𝐻, such ha 𝑉 ≈ 𝑊𝐻 (Figu e 4). The column ec o o he o iginal ma ix 𝑉 is he weigh ed sum o all he column ec o s in he le basis ma ix 𝑊, and he weigh coe icien is he elemen o he co esponding column ec o in he igh coe icien ma ix 𝐻. Bo h LSA and NMF sha e he key idea o ma ix ac o iza ion and dimensionali y educ ion. Howe e , singula ec o s decomposed om LSA can con ain nega i e alues, while NMF has non-nega i e cons ain s and i s posi i e and ze o coe icien s a e mo e in line wi h human cogni i e p ocess when i comes o in e p e ing he impo ance o he wo ds in ex ac ed opics (Lee e al., 2009). Figu e 4 Illus a ion o he non-nega i e ma ix ac o iza ion No e. F om Illus a ion o app oxima e non-nega i e ma ix ac o iza ion (NMF) by Qwe yus, 2013, Wikimedia Commons (h ps://commons.wikimedia.o g/wiki/File:NMF.png). CC BY-SA 3.0. The LDA me hod is cu en ly conside ed o be one o he mos popula opic models and has been widely used o ex ac ing use ul in o ma ion om app- ela ed pos s and e iews (e.g., Iacob & Ha ison, 8 2013; Guzman & Maalej, 2014; Ouyang e al., 2019; Su e al., 2019). Howe e , NMF a ou pe o med LDA in e ms o execu ion ime, while he accu acy di e ence be ween hese wo models was i ial (T uică e al., 2016). Al hough he NMF app oach p eceded he in oduc ion o LDA, some s udies ha e shown i s e ec i eness in ex ac ing opics om an in o mal ex ual con en . O’Callaghan e al. (2015) analyzed opics ex ac ed by NMF-based me hods and LDA-based me hods in six co po a wi h a o al numbe o 501,743 ex ual documen s and ound ha opics p oduced by NMF egula ly ha e highe cohe ence han hose gene a ed by LDA, especially o niche o non-mains eam co po a. This esul is in line wi h Con e as-Piña and Ríos’s (2016) conclusion ha NMF ex ac ed mo e use ul and cohe en opics han LSA and LDA based on an expe imen on a da ase o 21,863 consume complain s abou depa men s o e’s c edi ca ds. Mo eo e , acco ding o an expe imen on 57,934 use e iews o a popula e-comme ce app eleased on Google Play, NMF ob ained a be e solu ion compa ed o LDA in modeling opics om app e iews (Sup ayogi e al., 2018). The ecen s udy o Albalawi e al. (2020) shows bo h LDA and NMF app oaches deli e ed mo e meaning ul opics han LSA, andom p ojec ion and p incipal componen analysis when dealing wi h sho ex s such as commen s and e iews. Use e iews o messaging apps usually con ain in o mal exp essions. Based on he exis ing s udies, we selec ed he NMF me hod as he opic model o ou esea ch comp ehensi ely conside ing e ec i eness and e iciency. Fo use e iews, he da a ma ix con ains non-nega i e alues when each e m in he e iew ex s is p ope ly ep esen ed, e.g., by sco es o e m equency-in e se documen equency (TF-IDF). 2.3 SENTIMENT ANALYSIS Sen imen analysis e e s o he compu a ional s udy o human opinions, a i udes and emo ions owa ds en i ies such as indi iduals, e en s and opics, and aims o disco e opinions in ex s, iden i y he emo ions hese opinions exp ess and classi y hei sen imen pola i y (Medha , 2014). To achie e sen imen classi ica ion, bo h supe ised lea ning me hods and unsupe ised app oaches can be applied (Liu, 2012). Supe ised lea ning me hods equi e aining da a wi h sen imen labels. In eal p ac ice, da a sou ces om use -gene a ed pos s and e iews do no include such labels. Classi ying he sen imen pola i y o s eng h scale o UGC has become a popula esea ch opic in ecen yea s. Acco ding o Liu (2012), sen imen wo ds domina e he sen imen pola i y, and hus hese sen imen wo ds and ph ases may be u ilized o sen imen classi ica ion in an unsupe ised manne . One o he unsupe ised app oaches o sen imen analysis o UGC is he lexicon-based me hod, which equi es a p ede ined lexicon. The Gene al Inqui e (S one e al., 1966) is a lexical se wi h syn ac ic, seman ic and p agma ic in o ma ion o pa -o -speech agged wo ds and gi es labels o he sen imen pola i y o mos o i s included wo ds. The Mul i-Pe spec i e Ques ion Answe ing (MPQA) subjec i i y lexicon (Wilson e al., 2005) also p o ides he same s uc u al in o ma ion as he Gene al Inqui e . Mo eo e , Wilson e al. (2005) included he addi ional subjec i i y le el o a wo d o a ph ase wi h a label o s ong o weak. B adley and Lang (1999) c ea ed he A ec i e No ms o English Wo ds (ANEW), a dic iona y in which 1,034 English wo ds a e a ed in e ms o alence, a ousal and dominance on a con inuous scale be ween 1 and 9. The h ee a ed dimensions a e based on Osgood e al.’s (1957) heo y o emo ions. In hei heo y, alence (o pleasan ness) and a ousal ( he in ensi y o exci emen ) 9 a e he p incipal dimensions, while dominance (o con ol) is a less s ongly- ela ed dimension when i comes o emo ions in oked by a wo d. Simila o ANEW (B adley & Lang, 1999), Sen iWo dNe (Esuli & Sebas iani, 2006; Baccianella e al., 2010) also assigns h ee sen imen sco es o each synonym se (synse ), albei om h ee di e en aspec s ega ding posi i i y, nega i i y and neu ali y. This lexical esou ce is cons uc ed on Wo dNe (Mille , 1995; Fellbaum, 1998) and publicly a ailable in he Na u al Language Toolki (NLTK) (Bi d e al., 2009) o esea ch pu poses. Fu he mo e, Linguis ic Inqui y and Wo d Coun (LIWC) (Pennebake e al., 2003) p o ides a p op ie a y dic iona y ha o ganizes each wo d o wo d s em ac oss one o mo e psychologically ele an ca ego ies, which can in e posi i e o nega i e emo ions. Wi h he apid de elopmen o social media, mo e and mo e cybe ela ed wo ds appea in mic oblogs. Nielsen (2011) s a ed ha some o he a o emen ioned Gene al Inqui e , MPQA subjec i i y lexicon, ANEW and Sen iWo dNe do no inco po a e s ong obscene wo ds and In e ne slangs, and hus he p esen ed a new ANEW, a Twi e -based sen imen wo d lis including cybe slangs and obscene wo ds. In his compa a i e expe imen , he new ANEW exceeded he Gene al Inqui e , MPQA subjec i i y lexicon and ANEW, bu all hese lexicons did no pe o m as well as Sen iS eng h (Thelwall, 2013), a lexicon-based ool o sen imen analysis. Acco ding o Thelwall (2013), Sen iS eng h is a sen imen analysis ool o classi ica ion o social web ex s. This ool uses wo ds and wo d s ems om he exis ing LIWC and Gene al Inqui e as well as special wo ds and ph ases widely used on social media, and sen imen sco es o hese wo ds a e anno a ed by humans and imp o ed wi h machine lea ning me hods. Also, some ules we e cons uc ed o cope wi h non-s anda d ex ual exp essions pa icula ly in social media such as emo icons, emphasized punc ua ions and in ended misspellings. The es ed cases show ha Sen iS eng h pe o med well on a wide ange o social media ex s, bu less well on ex s wi h i onic and sa cas ic exp essions (Thelwall, 2013). Sen iS eng h has been used o aspec -based sen imen analysis o use e iews o mobile apps (Guzman & Maalej, 2014), o anking p oduc aspec s om online e iews (Wang e al., 2016) and o sen imen measu emen o use commen s on social media (He e al., 2016). Simila o Sen iS eng h, he Valence Awa e Dic iona y o sEn imen Reasoning (VADER) (Hu o & Gilbe , 2014) is ano he lexicon-based sen imen analysis ool designed o social media con ex s. The human- alida ed sen imen lexicon o VADER exploi s he exis ing lexicons such as he Gene al Inqui e , LIWC and ANEW, inco po a es addi ional e ms commonly used in mic oblogs and is buil on i e g amma ical and syn ac ical ules such as punc ua ions, capi aliza ion and deg ee modi ie s. In hei expe imen s, VADER achie ed no able success in social media domain and gene ally ou pe o med a majo i y o he well- ega ded sen imen analysis ools including he Gene al Inqui e , ANEW, LIWC, Hu- Liu04 opinion lexicon (Hu & Liu, 2004), Wo d-Sense Disambigua ion (Akkaya e al., 2009), Sen iWo dNe and Sen icNe (Camb ia e al., 2012). Recen s udies ha e employed VADER o analyzing he sen imen s o use e iews o mobile apps (Huebne e al., 2018; Su e al., 2019) and mic oblogs on Twi e (Elbagi & Yang, 2019) wi h encou aging pe o mance. Sen iS eng h and VADER di e in he ou pu o sen imen sco es. Fo each inpu , Sen iS eng h epo s wo independen sco es o posi i e and nega i e scales based on he psychological heo y o humans’ 10 mixed emo ions (No man e al., 2011) (Thelwall, 2013), while VADER p o ides a compound sco e compu ed om he posi i e, neu al and nega i e sco es (Hu o & Gilbe , 2014). Ano he ea lie in oduced dic iona y-based ool o ex ac ing sen imen om ex s is he Seman ic O ien a ion CALcula o (SO-CAL) (Taboada e al., 2011). Di e en om wo d-based app oaches using adjec i es (e.g., Whi elaw e al., 2005) o adjec i es and ad e bs (e.g., Benama a e al., 2007) o in e he emo ional o ien a ion, SO-CAL exploi s wo ds including adjec i es, e bs, nouns, and ad e bs o calcula e he sen imen pola i y and s eng h. Apa om his ex ension o pa s o speech (POS), SO- CAL also inco po a es a dic iona y o in ensi ie s and a e ined nega ion app oach. Thei esul s show ha SO-CAL achie ed consis en pe o mance on comple ely unseen ex s ac oss domains, di e en om Sen iS eng h and VADER designed o social web ex s. Use e iews a e one ype o UGC, whose language exp essions a e simila o use pos s and commen s on social media. In he p esen wo k, we conside ed sen imen analysis ool wi h a ocus on social media ex s and included he ecen ly in oduced VADER o be pa o he sen imen analysis o ou esea ch since he VADER compound sco ing me hod is mo e app op ia e o ou u he calcula ion o he inal sen imen sco es. 11 3 METHODOLOGY This sec ion in oduces he gene ic a chi ec u e o ou esea ch ocusing on h ee key esea ch issues: opic modeling, sen imen analysis and compe i i e analysis (Figu e 5). As depic ed in Figu e 5, ou amewo k s a s wi h da a collec ion, ollowed by a se ies o domain-speci ic da a p ep ocessing s eps including con ac ion expanding, ex cleaning, spelling no maliza ion, POS agging, wo d lemma iza ion, non-English e iews emo al, app ea u e ex ac ion, emo al o cus omized s op wo ds and e iew p uning. A e his, a opic model akes he ex ac ed ea u e e ms as inpu o ind unde lying app aspec -based opics. In pa allel, pa ially p ep ocessed e iew sen ences we e used o compu e hei sen imen sco es. In he end, we summa ized he ex ac ed opics and he sen imen sco es o conduc a mul i- ace ed compe i i e analysis. In addi ion o s a is ical summa ies, we also c ea ed compa a i e isualiza ions o be e e eal compe i i e insigh s in o se e al aspec s. Figu e 5 The esea ch amewo k o compe i i e analysis using opic modeling and sen imen analysis 12 3.1 DATA COLLECTION The app e iew managemen pla o m AppFollow (App ollow, n.d.) agg ega es use e iews om iOS, And oid, Mic oso and Amazon app s o es. We selec ed he Uni ed S a es as he coun y and sepa a ely expo ed use e iews o ou iOS mobile apps, namely Messenge , Wha sApp Messenge , Signal - P i a e Messenge and Teleg am Messenge , be ween June 1, 2020 and May 31, 2021. Since he app names o Wha sApp Messenge , Signal - P i a e Messenge and Teleg am Messenge ca y he wo d “Messenge ”, which is also he name o he o he app Messenge , we eplaced he h ee app names wi h “Wha sApp”, “Signal” and “Teleg am” espec i ely o cla i y he app names and minimize edundancy. A o al numbe o 27,479 e iews we e collec ed. Table 1 lis s he numbe o e iews and e iew pe cen age o each app. Table 1 In o ma ion o app and use e iews App Name App Company # o Re iews (%) Messenge Facebook, Inc. 12,346 (44.93%) Wha sApp Wha sApp Inc. 10,513 (38.36%) Signal Signal Messenge , LLC 2,633 (9.58%) Teleg am Teleg am FZ-LLC 1,987 (7.23) TOTAL 27,479 (100%) The expo ed da ase s we e conca ena ed in o a single da ase wi h 26 ields. The ields ega ding he e iew da e ime, app name, use a ing and e iew con en we e selec ed o u he analysis. Table 2 p o ides a sample o he da ase wi h selec ed ields. Table 2 Sample o da ase wi h selec ed ields Da e AppName Ra ing Re iew 2020-06-01 01:10:35 Messenge 1 Can’ send pic u es o ideos 2020-06-01 00:06:08 Wha sApp 5 S ill awesome 2020-06-01 01:17:05 Signal 4 i lo e he app since i ealized i needed he p i acy om hangou s and zoom, bu whene e i y o add someone o a g oupcha , i shows up as “e o use al eady in g oup” when hey’ e no ? is anyone else ha ing his p oblem? 2020-06-01 04:14:59 Teleg am 5 The bes app I’ e e e seen 13 3.2 DATA PREPROCESSING 3.2.1 Con ac ion expanding Con ac ions a e widely used in in o mal w i ing, and use e iews a e no excep ion. The wo e ms “doesn' ” and “does no ” ha e exac ly he same meaning, bu hey a e wo comple ely di e en okens o a compu e . Such con ac ions inc ease he dimensionali y o he documen - e m ma ix o u he opic modeling. To educe he edundancy in he da a, we expanded he common con ac ions and slangs using he Con ac ions Py hon lib a y (Koo en, n.d.) such ha , e.g., “doesn' ” was expanded o “does no ”, “could’ e” o “could ha e”, and “wanna” o “wan o”. This s ep also p e en ed he subsequen ex cleaning om gene a ing many w ong spellings such as “doesn ” and “could e” a e emo ing he apos ophes. In addi ion, his Py hon lib a y is also capable o co ec ing common ypos ela ed o con ac ions such as “didn ” and “can ”. This kind o ypos would be au oma ically expanded. 3.2.2 Tex cleaning Use e iews a e usually mixed-cased and comp ise many non- ex ual cha ac e s such as punc ua ions and digi s as well as cha ac e s in o eign languages. These cha ac e s p o ide leas use ul in o ma ion abou he app ea u es and use a i ude bu in oduce many noises o he opic model. As a esul , we lowe cased all e iew ex s and hen emo ed all URL’s, newline cha ac e s, punc ua ions, emojis, digi s and non-English cha ac e s. Also, e iews wi h only one wo d such as “awesome” and “ugh” we e d opped. These sho e iews a e usually p aise, c i iques o modal pa icles wi hou men ioning any speci ic app ea u es, and hus con ey meaningless in o ma ion o compe i i e analysis. A o al o 1,952 such e iews was emo ed and 25,527 e iews emained a he end o his s ep. 3.2.3 Wo d co ec ion and no maliza ion Use e iews end o in ol e spelling e o s, a ian o ms o English spellings and in o mal abb e ia ions. Gu and Kim’s (2015) ypo lis only collec s he common ypos in use e iews o gene al apps, and mos o hei ypos could be co ec ed by he Con ac ions Py hon lib a y. Mo eo e , hei ypo lis does no inco po a e di e en o ms o English spellings, which exp ess he same meaning bu ep esen en i ely di e en hings o a compu e jus like con ac ions. Since we we e dealing wi h a speci ic ype o apps, we pe o med a domain-speci ic co ec ion and no maliza ion o wo ds o educe he noises o u he POS agging, opic modeling and sen imen analysis. This sub-sec ion includes checking he spelling p ope y o each unique wo d, no malizing B i ish spellings o Ame ican spellings, and co ec ing English misspellings and abb e ia ions. 3.2.3.1 Spell check Be o e pe o ming he spelling no maliza ion and co ec ion, we needed o iden i y he spelling p ope ies o each unique wo d. The PyEnchan (Kelly, 2011) spellchecke was adop ed o his ask. This Py hon lib a y is capable o ecognizing di e en a ie ies o he English language such as Ame ican 14 English, B i ish English and Canadian English. The same wo d in di e en a ie ies is unde s andable o humans bu noisy o compu e s. The wo common Ame ican and B i ish a ie ies we e conside ed in ou spell check. We compu ed he equency o each unique wo d in he co pus and so ed his equency in descending o de . The unique uppe case wo ds we e hen passed o PyEnchan o checking i each o hem is a co ec ly spelled English wo d, in bo h Ame ican and B i ish spelling. We used capi al wo ds because all wo ds we e lowe cased du ing he ex cleaning, and PyEnchan is case-sensi i e. Fo ins ance, “english” is no a co ec spelling, bu “English” o “ENGLISH” is. Finally, o each unique wo d, he PyEnchan spellchecke epo ed “T ue” o “False” in bo h Ame ican and B i ish spellings. 3.2.3.2 B i ish-Ame ican spelling no maliza ion Fo wo ds in T ue B i ish spelling bu False Ame ican spelling, we no malized hem o Ame ican spellings based on he B i ish spelling dic iona y o an Ame ican-B i ish English ansla o (Hype eali y, n.d.), e.g., “beha iou ” o “beha io ”. Some o he T ue B i ish spellings we e no ound in his dic iona y, so we manually added he Ame ican spellings (Table 3). Finally, we eplaced all he B i ish spellings in he cleaned e iew ex s wi h hei Ame ican coun e pa s. Table 3 Addi ional B i ish-Ame ican spellings B i ish spelling Ame ican spelling amongs among lea n lea ned cus omisa ion cus omiza ion acknowledgemen acknowledgmen 3.2.3.3 Spelling co ec ion and abb e ia ion expansion Wo ds wi h bo h False spellings a e usually misspelled English wo ds, in o mal English abb e ia ions o non-English wo ds. In his s ep, we only co ec ed he English misspellings and in o mal English abb e ia ions, e.g., “ ecie e” o “ ecei e” and “pls” o “please”. The de ailed s eps we e selec ing he bo h False wo ds, going h ough hose wi h a equency g ea e han wo and manually assigning he co ec o m o each wo d i he wo d ob iously esembles an English wo d o abb e ia ion (Table 4). 15 Table 4 Manual spelling co ec ions Wo d Manual co ec ion b acebook pls, plz, pleasee, pleaseee, plzzz, pleaseeee please idk ecei e do no know ui use in e ace message , messange , massenge , messnge , mesenge , mesengge messenge ppl, ppls people msgs, messeges, messenges messages cuz, coz, bcz, cus because ecei e a lo de s de elope s wa sapp, wha sup, wha app, wha sap, sapp, wha ssap, wha saap, wha spp, wha sapps, wa sapp, wha sapp, wha up, whas app, wa sap, wha supp wha sapp hx hanks mins minu es soooo, sooo, soo, sooooo, soooooo so ux use expe ience acc accoun apps o e app s o e useable usable de de elope un ill un il ne wo kmanage e o ne wo k manage e o iam ecei e am lly, ealy eally ecei e happened ecei e ecei e goin going aceid ace id ha , h ha e homesc een home sc een dosen does no awsome awesome 22 3.4.3 E alua ion o sen imen analysis Since we used he weigh ed sen imen sco es, a he han he use a ings o no malized VADER compound sco es, as he inal sen imen sco es o u he compe i i e analysis, we assessed he e ec i eness o he sen imen analysis om wo pe spec i es. Fi s ly, we e alua ed he pe o mance o he weigh ed sen imen sco es. Secondly, we compa ed he pe o mance o he weigh ed sen imen sco es wi h ha o he use a ings and o he no malized VADER compound sco es. Fo e alua ing he pe o mance o he weigh ed sen imen sco es, a label indica ing sen imen pola i y was au oma ically assigned o each e iew acco ding o a sco e benchma k o 4 based on he weigh ed sen imen sco e. Fo example, e iews wi h weigh ed sen imen sco es g ea e han o equal o 4 we e au oma ically assigned a “Posi i e” label, o he wise “Nega i e”. This sco e benchma k was based on he assump ion ha e iews wi h a sen imen sco e abo e 4 usually exp ess p aise o iendly ad ice, while below 4 end o epo some issues ega ding speci ic app ea u es, no necessa ily consis ing o wo ds linking o emo ions o ha e, ange o annoyance. A e au oma ically assigning he label o sen imen pola i y based on he weigh ed sen imen sco es o each e iew, we andomly sampled 1% o e iews om each pola i y and conduc ed manual labeling o “Posi i e” o “Nega i e” o each sampled e iew. The c i e ion o manual labeling con o med o he assump ion o he sco e benchma k. An addi ional ema k o he c i e ion was ha e iews exp essing p aise i s and hen a shi o epo issues ega ding speci ic app aspec s we e manually labeled as “Nega i e”. To compa e he pe o mance o he weigh ed sen imen sco es wi h ha o he o he wo sco ing me hods, we au oma ically assigned wo mo e labels indica ing sen imen pola i y on he use a ings and no malized VADER compound sco es o each sampled e iew acco ding o he same sco e benchma k o 4. A his poin , each o he sampled e iews had h ee au oma ic labels o sen imen pola i y on he weigh ed sen imen sco es, use a ings and no malized VADER compound sco es espec i ely acco ding o he sco e benchma k o 4 and one manual label. We summa ized he numbe o sampled e iews based on he labels o sen imen pola i y and p esen ed he s a is ical esul s in h ee con usion ma ices (Table 8). Table 9 explains he e minology o TP, FP, TN and FN. Table 8 Con usion ma ix o sen imen e alua ion Weigh ed sen imen sco es/Use a ings/no malized VADER compound sco es Nega i e (< 4) Posi i e (>=4) Nega i e (manual) TN FP Posi i e (manual) FN TP 23 Table 9 Explana ion o TP, FP, TN and FN Ac onym Te m Desc ip ion TP T ue Posi i e he numbe o e iews wi h bo h an au oma ically assigned label (based on he sco e benchma k o 4) and a manual label o be “Posi i e” FP False Posi i e he numbe o e iews wi h an au oma ically assigned label (based on he sco e benchma k o 4) “Posi i e” and a manual label “Nega i e” TN T ue Nega i e he numbe o e iews wi h bo h an au oma ically assigned label (based on he sco e benchma k o 4) and a manual label o be “Nega i e” FN False Nega i e he numbe o e iews wi h an au oma ically assigned label (based on he sco e benchma k o 4) “Nega i e” and a manual label “Posi i e” Finally, we used he ollowing classi ica ion me ics P ecision, Recall and F1-sco e o espec i ely e alua e he pe o mance o he weigh ed sen imen sco es, use a ings and no malized VADER compound sco es: 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑇𝑃 𝑇𝑃 + 𝐹𝑃 𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑇𝑃 𝑇𝑃 + 𝐹𝑁 𝐹1−𝑠𝑐𝑜𝑟𝑒 =2 ∗ 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∗ 𝑅𝑒𝑐𝑎𝑙𝑙 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙 3.5 COMPETITIVE ANALYSIS This sec ion summa izes he ex ac ed opics and he weigh ed sen imen sco es om di e en pe spec i es o e ealing compe i i e in elligence. The summa y o e iew dis ibu ions and a e age sen imen sco es indica ed he o e all e iew coun s and use sen imen s by opic o each app du ing he pe iod be ween June 1, 2020 and May 31, 2021, while he sen imen e olu ion e lec ed he changes o use sen imen s o e ime on a mon hly basis. 3.5.1 Re iew dis ibu ions and a e age sen imen sco es To unde s and he e iew dis ibu ion, we agg ega ed he numbe o e iews by opic o each app. The mos ly discussed opics a e usually he main app aspec s ha he use s a e mo e conce ned abou . Using he weigh ed sen imen sco es as he inal sen imen sco es, we also calcula ed he a e age sen imen sco es by opic o each app. The a e age sco e o he opic 𝑡 o he app 𝑎 was calcula ed using he ollowing o mula: (4) (5) (6) 24 𝐴𝑣𝑔.𝑆𝑒𝑛𝑡𝑖𝑆𝑐𝑜𝑟𝑒(𝑡𝑎) = 1 𝑛∑𝑠_𝑤𝑒𝑖𝑔ℎ𝑡𝑒𝑑(𝑟𝑡𝑎)𝑖 𝑛 𝑖=1 whe e 𝑛 is he o al numbe o e iews o he opic 𝑡 o he app 𝑎, and he weigh ed sen imen sco e o each e iew is deno ed by 𝑠_𝑤𝑒𝑖𝑔ℎ𝑡𝑒𝑑(𝑟𝑡𝑎)𝑖 whe e 𝑖 = 1,2,…,𝑛. 3.5.1.1 Visual e iew dis ibu ion To be e isualize he main app aspec -based opics discussed by use s, we c ea ed ou pie cha s which espec i ely show he pe cen ages o e iews ega ding each opic o he ou messaging apps based on he e iew agg ega ion by opic o each app. 3.5.1.2 Visual compa ison o a e age sen imen sco es We c ea ed a ba cha based on he a e age sen imen sco es by opic o each app o isually compa e he a e age sen imen sco es o he ou messaging apps a a opic le el. The a e age sen imen sco es we e assumed o e lec he use sa is ac ion owa ds speci ic app aspec -based opics. 3.5.2 Sen imen e olu ion We g ouped he e iews by mon h and compu ed he a e age sen imen sco es by opic o each app on a mon hly basis. Mo eo e , o be e isualize he mon hly changes o use sen imen s o he ou messaging apps om June 2020 o May 2021, we plo ed wel e line cha s, each o an app aspec - based opic. (7) 25 4 RESULTS AND DISCUSSION A e mul iple s eps o da a p ep ocessing, a o al numbe o 21,395 in o ma i e e iews emained o opic modeling and sen imen analysis, whose esul s a e demons a ed in his sec ion, along wi h he s a is ical and isual ou comes o compe i i e analysis. A he end, a discussion ega ding he easonings and in e p e a ions behind ou indings is p esen ed. 4.1 RESULTS 4.1.1 Topic modeling This sub-sec ion p esen s he opics ex ac ed om he opic model and he accu acies o opic ex ac ion in he manual e alua ion. 4.1.1.1 Ex ac ed opics Figu e 7 displays he wel e opics decomposed om he NMF opic model, each wi h i s opic label numbe and opic name based on he i een opic-wo ds and hei co esponding weigh alues. Mos opics ha e one o wo domina ing wo ds excep o opic 2 - (g oup) cha | add ea u e | p i acy and opic 10 - app/accoun dele ion | download | accoun /login. These wo opics hold a ai numbe o wo ds wi h mode a ely dec easing impo ance, and hus hey comp ise mo e aspec s ega ding app ea u es. The es o he opics, hough led by one o wo wo ds, a e no always me ely o in ui i ely ep esen ed by hei domina ing wo ds. The o he wo ds in hese opics also comple e hei domina ing wo ds o ep esen he aspec mo e clea ly o e en gi e in o ma ion abou addi ional app- ela ed aspec s depending on he weigh alues o he opic wo ds. Figu e 7 Topics ex ac ed om he NMF model 26 Figu e 7 (Con inued) 27 Figu e 7 (Con inued) 28 Figu e 7 (Con inued) 29 Figu e 7 (Con inued) 4.1.1.2 E alua ion esul o ex ac ed opics Table 10 shows he esul o he manual e alua ion in ex ac ed opics wi h hei accu acy sco es so ed by opic in descending o de . Based on he o al numbe o 215 samples, an o e all accu acy o 86.05% was achie ed ac oss all opics. The accu acy sco es o eigh opics (1 - app/link opening, 3 - upda e | ios e sion, 7 - social/ unc ional usage, 5 - ( ideo) call | connec ion quali y, 11 - pho o/ ideo/link/message sending, 9 - gene al p oblems o ix, 0 - message/no i ica ion ecei ing | ead eceip and 8 - s a us | sea ch con ac ) a e abo e he o e all sco e. In de ail, bo h opic 1 - app/link opening and opic 3 - upda e | ios e sion ecei ed a 100% accu acy, while opic 5 - ( ideo) call | connec ion quali y and opic 11 - pho o/ ideo/link/message sending ecei ed he same accu acy o 93.75%, opic 7 - social/ unc ional usage in be ween, wi h 94.74%. Topic 9 - gene al p oblems o ix also ob ained a sco e abo e 90%. In addi ion, he accu acy sco es o ou opics (0 - message/no i ica ion ecei ing | ead eceip , 8 - s a us | sea ch con ac , 2 - (g oup) cha | add ea u e | p i acy and 4 - app c ash) all be ween 80% and 90%, 30 while opic 10 - app/accoun dele ion | download | accoun /login ecei ed he lowes accu acy (68.97%), abou 2.5% lowe han ha o opic 6 - app/ ea u e wo king issue | apple wa ch. Table 10 Accu acies o opic ex ac ion Topic # o samples #. o incohe ence Accu acy OVERALL 215 30 0.8605 1 - app/link opening 16 1.0000 3 - upda e | ios e sion 14 1.0000 7 - social/ unc ional usage 19 1 0.9474 5 - ( ideo) call | connec ion quali y 16 1 0.9375 11 - pho o/ ideo/link/message sending 16 1 0.9375 9 - gene al p oblems o ix 14 1 0.9286 0 - message/no i ica ion ecei ing | ead eceip 10 1 0.9000 8 - s a us | sea ch con ac 8 1 0.8750 2 - (g oup) cha | add ea u e | p i acy 49 9 0.8163 4 - app c ash 10 2 0.8000 6 - app/ ea u e wo king issue | apple wa ch 14 4 0.7143 10 - app/accoun dele ion | download | accoun /login 29 9 0.6897 4.1.2 E alua ion esul o sen imen analysis Based on a o al numbe o 214 samples, he con usion ma ices (Table 11, Table 12, Table 13) p esen he s a is ical ou comes o he sen imen e alua ion on weigh ed sen imen sco es, use a ings and no malized VADER compound sco es espec i ely. Acco ding o he manual labels, he majo i y class is Nega i e in he sample e iews. The numbe s o co ec ly classi ied labels o weigh ed sen imen sco es, use a ings and no malized VADER compound sco es a e 202, 192 and 182 espec i ely. As o he inco ec ly classi ied labels, he numbe o alse Nega i e (7) is sligh ly g ea e han ha o alse Posi i e (5) o weigh ed sen imen sco es. On he con a y, he numbe o alse Nega i e is ob iously smalle han ha o alse Posi i e o use a ings and no malized VADER compound sco es. Fo use a ings, he numbe o alse Posi i e is 20 and o alse Nega i e is only 2. Fo no malized VADER compound sco es, he numbe o alse Posi i e is 22 and o alse Nega i e is 10. 31 Table 11 Con usion ma ix o sen imen e alua ion on weigh ed sen imen sco es Weigh ed sen imen sco es Nega i e (< 4) Posi i e (>= 4) Nega i e (manual) 170 5 Posi i e (manual) 7 32 Table 12 Con usion ma ix o sen imen e alua ion on use a ings Use a ings Nega i e (< 4) Posi i e (>= 4) Nega i e (manual) 155 20 Posi i e (manual) 2 37 Table 13 Con usion ma ix o sen imen e alua ion on no malized VADER compound sco es No malized VADER compound sco es Nega i e (< 4) Posi i e (>=4) Nega i e (manual) 153 22 Posi i e (manual) 10 29 The calcula ed P ecisions, Recalls and F1-sco es o weigh ed sen imen sco es, use a ings and no malized VADER compound sco es a e epo ed in Table 14. Gene ally, he weigh ed sen imen sco es ou pe o med he o he wo sco ing me hods, wi h an imp o emen o a leas 7% in F1-sco e. Al hough he highes Recall (94.87%) was achie ed when conside ing only use a ings as he sen imen sco es, he P ecision was abou 30% lowe han he Recall, esul ing in a educed F1-sco e. As o he no malized VADER compound sco es, he P ecision, Recall and F1-sco e we e he lowes . 38 app/link opening in Oc obe 2020 and o opic 4 - app c ash in July, Sep embe and Oc obe 2020. As o Teleg am, he lowes a e age sen imen sco e o opic 3 - upda e | ios e sion in Ap il 2021 and o opic 11 - pho o/ ideo/link/message sending du ing he pe iod be ween Feb ua y o Ap il 2021 we e eco ded. Also, he a e age sen imen sco es o opic 0 - message/no i ica ion ecei ing | ead eceip , opic 5 - ( ideo) call | connec ion quali y and opic 8 - s a us | sea ch con ac su e ed a mild downwa d endency in luc ua ion since No embe 2020. Figu e 13 Sen imen e olu ion by opic 39 Figu e 13 (Con inued) 40 Figu e 13 (Con inued) 41 Figu e 13 (Con inued) 42 Figu e 13 (Con inued) 43 Table 16 S a is ical agg ega ion o a e age sen imen sco es by mon h, opic and app 2020 2021 Topic App Jun Jul Aug Sep Oc No Dec Jan Feb Ma Ap May 0 - message/no i ica ion ecei ing | ead eceip Messenge 2.3638 2.2865 2.3089 2.2167 2.2289 1.9869 2.0386 2.0885 2.2776 2.1728 2.6446 2.0403 Wha sApp 2.8857 2.9078 4.2164 2.8947 2.8752 3.2145 3.3961 3.0906 2.7589 2.5521 3.2270 2.8640 Signal 3.2700 3.4304 3.6107 3.8720 3.4125 3.6586 3.1081 3.2882 2.5848 3.0972 Teleg am 2.8003 3.1956 3.3350 3.5797 3.4320 4.4391 3.5129 3.2835 3.0859 3.1130 3.6943 2.2172 1 - app/link opening Messenge 2.2296 2.1492 2.2384 2.7128 2.5070 2.1520 2.3345 2.2665 1.8650 2.3562 2.1972 1.9824 Wha sApp 2.4307 2.4722 2.8606 2.9740 2.4108 2.3967 2.3376 2.5232 2.6880 2.3361 2.4067 2.6644 Signal 2.9305 1.6972 4.9884 1.6426 3.1510 3.4535 3.4291 2.5772 3.9005 2.1907 2.9478 Teleg am 2.5985 2.9528 3.6636 3.0872 2.1297 3.1088 2.3652 1.8427 2.6891 2.6182 3.1803 2.0932 2 - (g oup) cha | add ea u e | p i acy Messenge 3.0310 2.8480 2.7479 2.7368 2.5593 2.3157 2.4096 2.4122 2.6191 2.6502 2.6389 1.9545 Wha sApp 3.2234 3.9343 3.9550 3.8854 3.5372 3.6712 3.5475 2.4167 3.1884 3.4263 3.3581 2.5006 Signal 3.7442 3.3481 3.3824 3.8916 3.9520 3.8022 3.9567 4.0685 3.8967 3.7720 3.7532 4.2418 Teleg am 3.5699 3.2739 3.5987 3.6435 3.5134 3.0989 3.1997 3.9236 3.4021 3.3949 3.4053 3.0457 3 - upda e | ios e sion Messenge 2.2068 2.4530 2.1444 2.4909 2.5920 2.1493 2.1288 2.2744 2.2140 2.3206 2.1211 2.3013 Wha sApp 2.4352 3.5214 3.3479 3.0506 2.7390 2.6399 2.8454 2.4424 2.3220 2.6074 2.6744 2.6440 Signal 3.2008 2.4639 3.9507 2.0728 2.9753 3.4823 4.0063 4.0979 1.8590 3.4727 1.7962 Teleg am 3.0714 3.4381 2.5976 2.9335 3.0537 3.4169 3.5884 3.6723 3.9401 2.5798 1.4411 3.4120 4 - app c ash Messenge 2.3294 2.1243 2.1728 2.3938 2.2769 2.0010 2.0616 1.9573 2.1220 2.0324 2.6809 2.0153 Wha sApp 3.5229 2.7573 2.6936 3.0294 2.5036 2.4253 3.4909 2.7442 2.6264 2.0921 2.5431 2.6759 Signal 3.3318 1.8848 1.8947 1.6301 3.3765 2.4432 3.5193 4.7975 3.9046 3.3431 2.8587 Teleg am 1.9491 2.8785 3.9922 2.9617 2.0750 3.3103 3.6608 3.6032 1.9837 4.7679 1.8805 5 - ( ideo) call | connec ion quali y Messenge 2.5149 2.8097 2.6443 2.8622 2.4097 2.2581 2.5174 2.9462 2.3197 2.4827 2.7789 2.1944 Wha sApp 3.1105 3.3307 3.1759 3.1883 3.0272 3.2134 3.0063 3.0316 2.8560 2.8901 2.7690 2.7159 Signal 3.9512 2.4058 4.2402 3.8803 3.4331 2.9188 4.0029 3.8020 3.3223 2.8761 3.0087 3.5840 Teleg am 3.4898 4.0502 3.5206 2.9378 3.4180 3.6611 3.0830 3.0953 2.8802 3.0982 3.3780 2.0426 6 - app/ ea u e wo king issue | apple wa ch Messenge 2.3616 2.5807 2.2183 2.4760 2.5502 2.1760 2.2896 2.4085 2.4399 2.2497 2.1487 1.9844 Wha sApp 2.9412 3.3373 3.1515 2.9493 2.7282 2.9534 2.9765 2.7456 3.2867 3.2279 3.2983 2.2039 Signal 3.6491 3.8341 3.0147 3.6614 3.3828 3.7963 3.7119 3.9542 4.3755 3.7583 3.6905 Teleg am 2.9731 3.1284 2.7934 2.8894 3.0254 3.6051 3.3794 3.7166 2.8662 2.3776 2.8964 2.5772 7 - social/ unc ional usage Messenge 2.4790 2.7226 2.2743 2.4416 2.6954 2.2755 2.1732 2.1569 2.2573 2.4060 2.3385 2.2482 Wha sApp 3.2299 3.6690 3.5868 3.6122 3.5031 3.5697 3.4347 2.4441 3.1933 3.2731 3.3438 2.7749 Signal 3.5961 3.2402 4.6192 3.7990 3.2291 4.1911 3.9825 4.0896 4.1760 4.2620 3.7053 3.4604 Teleg am 3.2884 3.8529 3.4983 3.3414 3.0613 2.4046 3.3098 3.6310 3.5288 3.6539 3.1035 2.9611 8 - s a us | sea ch con ac Messenge 2.9390 2.3639 2.4817 2.6576 2.6991 2.5556 2.2944 3.1056 2.3153 2.3309 2.8268 2.1508 Wha sApp 3.1580 3.0234 3.5846 4.0132 2.9351 2.7939 2.8955 2.5883 3.4636 2.8803 3.4870 2.7182 Signal 4.3933 2.3244 4.0403 3.2704 3.9059 4.5141 3.7817 Teleg am 2.8694 4.0243 4.5564 4.6439 4.6360 4.3597 3.9469 3.6535 3.8297 2.8916 3.2212 9 - gene al p oblems o ix Messenge 2.4882 2.3797 2.2799 2.2523 2.3556 2.2618 2.1195 2.3953 2.4116 2.6213 2.4298 2.6768 Wha sApp 2.5228 2.8119 2.8665 2.5707 2.6262 3.0854 2.7581 2.7761 2.7374 2.5640 2.4026 2.5425 Signal 2.6883 2.9739 3.0525 3.9976 3.1953 3.2859 2.9759 3.5683 4.1149 3.4264 3.2683 3.0846 Teleg am 2.9512 2.9466 3.6774 3.3128 2.6736 2.7513 2.3777 2.9252 2.2393 3.1167 2.7231 2.2709 10 - app/accoun dele ion | download | accoun /login Messenge 2.2822 2.4381 2.2573 2.5419 2.1552 2.1790 2.1359 2.1345 2.1019 2.1207 2.1478 2.0040 Wha sApp 2.7814 2.8880 2.8320 2.6501 2.6622 2.7375 2.6472 2.1988 2.5763 2.6369 2.8155 2.0763 Signal 3.0253 2.5741 2.9547 2.9875 2.1958 2.8753 2.4677 2.8672 3.2691 3.4435 3.0024 3.4426 Teleg am 2.8000 2.7102 2.5806 2.7885 2.8444 2.5920 2.8882 2.5882 2.9681 2.7222 2.5535 2.6028 11 - pho o/ ideo/link/ message sending Messenge 2.3336 2.3289 2.4697 2.3306 2.3327 2.1401 2.3234 2.2309 2.1760 2.2920 2.4548 2.1653 Wha sApp 2.6894 3.0979 3.2554 3.4077 3.0389 2.8928 3.3946 2.5569 2.7187 2.8015 3.0754 2.7571 Signal 3.6399 3.0006 4.1891 2.4700 3.0566 3.2743 3.9777 3.0949 3.2191 3.3313 3.0825 3.4782 Teleg am 2.6049 3.2188 2.9884 2.9781 3.0375 2.8413 2.1573 3.5336 1.8173 2.0580 2.1174 3.7092 44 4.2 DISCUSSION By ex -mining 27,479 use e iews o ou messaging apps eleased on Apple S o e, he p esen wo k combined opic modeling and sen imen analysis o pe o m compe i i e analysis. The esul s show ha he opic model ex ac ed comp ehensible opics wi h p omising accu acies, and ha he sen imen analysis pe o med well. Addi ionally, he compe i i e analysis based on he ex ac ed opics and he weigh ed sen imen sco es e ealed meaning ul compe i i e in elligence in e ms o se e al ace s. 4.2.1 Topic modeling In he p esen wo k, he accu acies anging om 69.97% o 100% in opic ex ac ion sugges he NMF opic model wo ks well wi h use e iews o messaging apps. The ela i ely lowe accu acies migh pa ially esul om a ai numbe o e iews ha exp ess p o es s agains a ecen a ack in Pales ine. The language exp ession o poli ical discussion is likely o con use he opic decomposi ion o he model. Mo eo e , we ollowed he app oach adop ed by Vu e al. (2015) o use nouns and e bs as app ea u es and ob ained an imp o emen o 2.94% in o e all accu acy compa ed wi h hei a e age accu acy o 83.11% using keywo d-based app oach o analyze use e iews om Google Play. This imp o emen migh due o a di e en app oach o mine he app aspec s o a mo e domain-speci ic ex p ep ocessing o messaging apps. Ano he possible explana ion can be ha he exp ession pa e n o use e iews om Google Play is di e en om ha om Apple S o e. Fu he mo e, Guzman and Maalej (2014) used he LDA app oach o ex ac opics om colloca ions o nouns, e bs and adjec i es in use e iews o mul iple non-compe i i e And oid and iOS apps including Wha sApp. Thei esul s show ha he p ecisions and ecalls a ied on an app basis, and he highes p ecision and ecall we e achie ed o Wha sApp, wi h F1-sco es o 0.781 and 0.813 espec i ely o inclusion and exclusion o sen imen wo ds in ea u e wo ds. These F1-sco es o Wha sApp in hei esul , combining he o e all accu acy in opic ex ac ion in ou esul , migh indica e ha opic models a e pa icula ly e ec i e o ex ac app ea u es om use e iews o messaging apps bu no any ype o mobile apps. A possible explana ion can be ha messaging apps usually ha e homogeneous unc ionali ies, and hus use s o such apps migh end o exp ess a speci ic app aspec in simila ways. Also, hei exclusion o sen imen wo ds p o ides a easible way o imp o e ou pe o mance o opic ex ac ion by p uning sen imen wo ds in app ea u e e ms. In hei s udy, he sen imen wo ds a e usually adjec i es (e.g., g ea , bad) and e bs (e.g., ha e, lo e), while in ou esea ch, ea u e e ms ha bea sen imen s a e mos ly e bs. The e a e some impo an indings in he ex ac ed opics. Fi s ly, domain-speci ic knowledge was shown in some o he opic names. The e m “app” is no in any o he opic wo ds in opic 1- app/link opening, opic 4 - app c ash, opic 6 - app/ ea u e wo king issue | apple wa ch o opic 10 - app/accoun dele ion | download | accoun /login, bu hei opic names s ill ela e o app due o he logical con ex o hei opic wo ds. In ac , i was impossible o ha e he e m “app” in any lis o opic-wo ds since his e m was expanded o “applica ion” and hen emo ed as a s op wo d due o i s ex emely high equency in he co pus. Secondly, opic-wo ds such as “unins all” and “ eins all” appea ed in se e al opics such as opic 1 - app/link opening, opic 4 - app c ash and opic 6 - app/ ea u e wo king issue | apple wa ch. One possible explana ion can be he use beha io o unins alling and eins alling he app o a emp ing o sol e an issue, e.g., app c ash, and he use s men ioned such beha io s along wi h 45 o he app ea u e issues in he e iews. Thi dly, a ew simila i ies in opic keywo ds we e ound be ween some ex ac ed opics in he p esen wo k and some opics in Su e al.’s (2019) s udy. They applied he LDA me hod o ind la en opics om use e iews o ou mobile apps on Google Play, and Messenge was one o hose apps. Thei s udy lis s some ex ac ed opics wi h op i e keywo ds o Messenge . As illus a ed in Table 17, some o hei op i e keywo ds can also be ound in he op i een keywo ds o h ee ex ac ed opics in he p esen wo k. I is possible ha mo e opic-wo ds could be ma ched i hey had shown mo e keywo ds o each opic, e.g., op i een keywo ds a he han only op i e keywo ds. The app aspec s discussed in each pai o he h ee pai s o opics a e e y close based on he logical connec ion o he keywo ds in each opic. One explana ion o his could be ha nea ly hal o he o al use e iews o opic modeling a e o Messenge in he p esen wo k (Table 15). I is also likely ha iOS use s and And oid use s end o discuss simila app aspec s o epo simila app issues o messaging apps in use e iews. Las ly, some o he ex ac ed opics comp ise mul iple incohe en aspec s. Fo ins ance, in opic 2, he h ee aspec s o (g oup) cha , add ea u e and p i acy could be h ee indi idual opics. This opic incohe ence migh esul om a signi ican numbe o e iews men ioning he h ee aspec s oge he . Ano he explana ion could be ha he numbe o opics k was de ined o be oo small, and hus he opic model gene a ed excessi ely b oad opics. Simila p oblem o he numbe o k migh also happen o opic 7 - social/ unc ional usage. This opic equi ed a comp ehensi e conside a ion o o he wo ds wi h smalle weigh s and a combina ion wi h i s abs ac p ima y wo d “use” o condensing he main issues discussed in ha opic. The wo ds such as “lo e”, “ iend”, “ amily” and “communica e” sugges social aspec s, while he wo ds li ke “iphone”, “da um”, “phone” and “messaging” indica e unc ional ea u es. Howe e , social and unc ional usage is an o e ly b oad opic, which can e e o many app aspec s. O e all, he ex ac ed opics sugges ha de ining he numbe o k is a majo challenge, bu he NMF opic model is s ill use ul o ex ac ing opics, mos o which e lec clea and sepa able app aspec s o messaging apps. Table 17 Compa ison o keywo ds in ex ac ed opics The p esen wo k - NMF Su e al. (2019) - LDA Messenge , Wha sApp, Signal, Teleg am Messenge Topic id Top 15 keywo ds Topic id Top 5 keywo ds 0 message, no i ica ion, show, ge , see, ecei e, ead, go, eques , say, check, oice, ype, eply, ma ke place 3 message, no i ica ion, dele e, gi e, eques 2 cha , see, people, ea u e, add, wan , lo e, op ion, g oup, need, p i acy, make, iend, change, go 8 cha , change, ea u e, add, g oup 5 call, ideo, oice, phone, make, quali y, connec , hea , ing, ecei e, audio, d op, issue, play, sound 0 call, sc een, u n, oice, connec 46 4.2.2 Sen imen analysis Th ough a compa ison o di e en sen imen sco ing me hods, we disco e ed he weigh ed sen imen sco es mi iga ed he bias o use a ings and bene i ed om he no malized VADER compound sco es, esul ing in a ela i ely objec i e sen imen sco ing o u he compe i i e analysis. In iguingly, Su e al. (2019) conduc ed a simila sen imen analysis using a e age weigh ed sco es o wo g oups o simila apps eleased on Google Play and ob ained on a e age a p ecision o 92.29%, a ecall o 71.41% and a F1-sco e o 80.24%. Thei p ecision- ecall adeo is con a y o hose o use a ings and no malized VADER compound sco es in ou esul , which ended o ecei e lowe p ecisions and highe ecalls (Table 14). This migh due o he di e en sco e benchma ks o sepa a ing posi i e and nega i e e iews, 2.5 in hei s udy bu 4 in he p esen wo k, p oducing a majo i y class o nega i e in ou sen imen classi ica ion. Compa a i ely, a sco e benchma k o 4 seems mo e app op ia e o sepa a ing he sen imen pola i y o use e iews o mobile apps. Ou esul shows ha he weigh ed sen imen sco es based on his sco e benchma k ob ained a be e F1-sco e wi h a balanced p ecision- ecall adeo . Fu he mo e, in he sen imen analysis o he p esen wo k, we a e aged he weigh ed sen imen sco es o use e iews by opic as he opic sen imen sco e, while in Su e al.’s (2019) s udy, he sen imen sco e o each opic ound by he LDA model akes he numbe o use s’ hi ing he like bu on in o conside a ion. Howe e , he e a e no only use s who click he like bu on bu also use s who click he dislike bu on. I is possible ha he coun o use dislikes ou numbe s he coun o use likes o speci ic e iews. Me ely conside ing he numbe o use likes migh bias he sen imen sco e o a opic. In spi e o his possible bias, hei s udy s ill encou ages a mo e elabo a e sen imen analysis, which could ake bo h he numbe o use likes and he numbe o use dislikes in o accoun when i comes o he calcula ion o he sen imen sco e o a opic. 4.2.3 Compe i i e analysis The compe i i e analysis e ealed meaning ul insigh s in o use conce ns, compe i i e s eng hs and weaknesses as well as changes o use sen imen s o e ime. Acco ding o Po e (1980), he componen s o compe i o analysis comp ise mul i-dimensional objec i es o he compe i o ’s manage ial pe sonnel, assump ions o he compe i o i sel and o he indus y, compe i i e s a egy and he compe i o ’s esou ces and capabili ies including s eng hs and weaknesses. The ein o, he objec i es and assump ions e eal wha d i es he compe i o , while he s a egy, esou ces and capabili ies e lec wha he compe i o is doing o is capable o doing. Based on hese componen s o compe i o analysis, he use conce ns, compe i i e s eng hs and weaknesses as well as changes o use sen imen s o e ime shown in he p esen wo k mee wha he compe i o is doing o is capable o doing. Speci ically, use conce ns, on he one hand, e lec in a way he capabili ies o a compe ing messaging app, and on he o he hand, use conce ns also p o ide legi ima e g ound o conjec u ing he compe i i e s a egy o he i al. Also, compe i i e s eng hs and weaknesses as well as changes o use sen imen s o e ime unco e he esou ces and capabili ies o a compe i o . 47 The e iew dis ibu ions by opic o he ou messaging apps e lec s he majo use conce ns abou speci ic app aspec s. In gene al, app use s we e e y conce ned abou cha ea u es and p i acy issues. Thei a en ion o cha ea u es con o ms o people’s pe cep ion mainly because cha ea u es a e he co e unc ions o all messaging apps. P i acy issues should a ouse he a en ion o p ac i ione s in he indus y o ins an messaging se ices. Da a secu i y and p i acy may become a key compe i i e ad an age o di e en ia ing one messaging app om he o he s. Ano he majo use concen a ion is he download and accoun issues anging om download, login p oblem and accoun dele ion. These issues a e usually no ela ed o he key ea u es o apps bu migh be he gene ic p oblems o many mobile apps, which bo he he app use s e y much. Sol ing hese p oblems by be e unc ional designs and echnical suppo s would imp o e he use sa is ac ion. Fo example, a e downloading, use s could choose mul iple ways o log in he app wi hou any login ailu e. Mo eo e , use s o Wha sApp, Signal and Teleg am had simila ocuses on he app aspec s pa icula ly ega ding oice and ideo call, connec ion quali y, social and unc ional usage, accoun and download issues. Aspec s such as oice call and ideo call, jus like he cha ea u es, a e also he c ucial ea u es o apps o daily communica ion. Thei use s migh do equen oice o ideo call apa om ex messages, and hus hey a ached g ea impo ance o he quali y o calls. An excellen and s able connec ion quali y would be a big a ac ion o use s who make lo s o oice calls o ideo calls. Also, hese h ee apps migh be e y impo an o hei use s o keep in ouch wi h hei amily and iends. Du ing he social communica ion including ex messages, oice calls and ideo calls, he unc ional usage, e.g., mobile da a usage, migh d aw he special a en ion o app use s. I is likely ha a speci ic ea u e o mobile da a sa ing mode in he app se ings can become a unique compe i i e ad an age in he e a o mobile In e ne . As o Messenge , in addi ion o he cha ea u es, p i acy, download and accoun issues, he p oblems ega ding app and link opening we e also men ioned by many use s. Ne e heless, his aspec was no highly discussed by use s o Wha sApp, Signal and Teleg am. This con as migh sugges ha use s o Messenge su e ed om mo e ailu e o incon enience in opening he app and he links in messages. Rega ding he app opening, he p oblem migh be no eac ion a e clicking on he app icon. The p oblem o app and link opening dese es he a en ion o hei esea ch and de elopmen pe sonnel, who migh u he in es iga e he speci ic issues by e ie ing use e iews o ha opic. Ano he in e es ing inding is ha use s o Messenge did no discuss he aspec o oice call, ideo call and connec ion quali y as much as he use s o Wha sApp, Signal and Teleg am. One possible explana ion can be ha he use s o Messenge did no encoun e ou s anding issues o equen ly exp ess hei emo ions abou his app aspec . I could also be ha use s o Messenge mainly ex ed messages a he han made oice calls o ideo calls, and hus hey did no pay much a en ion o his app aspec . In he la e si ua ion, he easons can be u he examined. The oice and ideo call o Messenge migh be incon enien o use, and he use migh need addi ional clicks o use his app ea u e. Las ly, aspec s o he messaging apps such as gene al p oblems o ix, app and ea u e wo king issues as well as app c ash mainly pe ain o echnical issues. O he app aspec s such as messaging sending and ecei ing, ead eceip , s a us, con ac sea ching and upda e we e also discussed by use s o hese messaging apps. These app aspec s migh ela e o no only echnical issues bu also design o he apps. The speci ic solu ions o hese p oblems should be de e mined based on he speci ic issues men ioned in he use e iews. The app aspec -based opics could p o ide he p ac i ione s o messaging apps wi h clea e di ec ions o use conce ns and oubleshoo ing. 54 Géma , G., & Jiménez-Quin e o, JA. (2015). Tex mining social media o compe i i e analysis. Tou ism & Managemen S udies, 11(1), 84–90. Re ie ed Ma ch 03, 2021 om h ps://www. edalyc.o g/pd /3887/388743883010.pd G eene, D., O’Callaghan, D., & Cunningham, P. (2014). How Many Topics? S abili y Analysis o Topic Models. In Calde s, T., Esposi o, F., Hülle meie , E., & Meo, R. (eds), Machine Lea ning and Knowledge Disco e y in Da abases. ECML PKDD 2014. Lec u e No es in Compu e Science, ol 8724. Sp inge , Be lin, Heidelbe g. h ps://doi.o g/10.1007/978-3-662-44848-9_32 Gu, X., & Kim, S. (2015). Wha Pa s o You Apps a e Lo ed by Use s? 2015 30 h IEEE/ACM In e na ional Con e ence on Au oma ed So wa e Enginee ing (ASE), 760–770. h ps://doi.o g/10.1109/ASE.2015.57 Guo, Y., Ba nes, S. J., & Jia, Q. (2017). Mining meaning om online a ings and e iews: Tou is sa is ac ion analysis using la en di ichle alloca ion. Tou ism Managemen , 59, 467–483. h ps://doi.o g/10.1016/j. ou man.2016.09.009 Guzman, E., & Maalej, W. (2014). How Do Use s Like This Fea u e? A Fine G ained Sen imen Analysis o App Re iews. 2014 IEEE 22nd In e na ional Requi emen s Enginee ing Con e ence (RE), 153–162. h ps://doi.o g/10.1109/RE.2014.6912257 He, W., Zha, S., & Li, L. (2013). Social media compe i i e analysis and ex mining: A case s udy in he pizza indus y. In e na ional Jou nal o In o ma ion Managemen , 33(3), 464–472. h ps://doi.o g/10.1016/j.ijin omg .2013.01.001 He, W., Tian, X., Chen, Y., & Chong, D. (2016). Ac ionable Social Media Compe i i e Analy ics Fo Unde s anding Cus ome Expe iences. Jou nal o Compu e In o ma ion Sys ems, 56(2), 145–155. h ps://doi.o g/10.1080/08874417.2016.1117377 Ho mann, T. (1999). P obabilis ic la en seman ic analysis. P oceedings o he Fi een h con e ence on Unce ain y in a i icial in elligence (UAI1999), 289–296. Re ie ed Ma ch 05, 2021, om h p://a xi .o g/abs/1301.6705 Hong, L., & Da ison, B. D. (2010). Empi ical s udy o opic modeling in Twi e . P oceedings o he Fi s Wo kshop on Social Media Analy ics, 80–88. h ps://doi.o g/10.1145/1964858.1964870 Hu, M., & Liu, B. (2004). Mining and summa izing cus ome e iews. P oceedings o he en h ACM SIGKDD in e na ional con e ence on Knowledge disco e y and da a mining (KDD '04), 168–177. h ps://doi.o g/10.1145/1014052.1014073 Hu, F., & T i edi, R. H. (2020). Mapping ho el b and posi ioning and compe i i e landscapes by ex - mining use -gene a ed con en . In e na ional Jou nal o Hospi ali y Managemen , 84. h ps://doi.o g/10.1016/j.ijhm.2019.102317 55 Huebne , J., F ey, R. M., Ammendola, C., Fleisch, E., & Ilic, A. (2018). Wha People Like in Mobile Finance Apps: An Analysis o Use Re iews. P oceedings o he 17 h In e na ional Con e ence on Mobile and Ubiqui ous Mul imedia, 293–304. h ps://doi.o g/10.1145/3282894.3282895 Hu o, C., & Gilbe , E. (2014). VADER: A Pa simonious Rule-Based Model o Sen imen Analysis o Social Media Tex . P oceedings o he In e na ional AAAI Con e ence on Web and Social Media, 8(1). Re ie ed Ma ch 10, 2021, om h ps://ojs.aaai.o g/index.php/ICWSM/a icle/ iew/14550 Hype eali y. (n.d.). B i ish spellings. Re ie ed June 17, 2021, om h ps://gi hub.com/hype eali y/Ame ican-B i ish-English- T ansla o /blob/mas e /da a/b i ish_spellings.json Iacob, C., & Ha ison, R. (2013). Re ie ing and analyzing mobile apps ea u e eques s om online e iews. 2013 10 h Wo king Con e ence on Mining So wa e Reposi o ies (MSR), 41–44. h ps://doi.o g/10.1109/MSR.2013.6624001 Johann, T., S anik, C., Alizadeh B., A. M., & Maalej, W. (2017). SAFE: A Simple App oach o Fea u e Ex ac ion om App Desc ip ions and App Re iews. 2017 IEEE 25 h In e na ional Requi emen s Enginee ing Con e ence (RE), 21–30. h ps://doi.o g/10.1109/RE.2017.71 Ka po ich, S. (9 No embe 2013). P obabilis ic la en seman ic analysis — PLSA. Wikimedia Commons. h ps://commons.wikimedia.o g/wiki/File:Вероятностный_латентно- семантический_анализ.png Kelly, Ryan. (2011). PyEnchan . h p://pyenchan .gi hub.io/pyenchan /. Accessed June 17, 2021. Kim, Y., & Jeong, S. R. (2018). Compe i i e in elligence in Ko ean Ramen Ma ke using Tex Mining and Sen imen Analysis. Jou nal o In e ne Compu ing and Se ices, 19(1), 155–166. h ps://doi.o g/10.7472/JKSII.2018.19.1.155 Koo en, P. V. (n.d.). Con ac ions. h ps://gi hub.com/koo enp /con ac ions. Accessed June 17, 2021. La e y, J. D., McCallum, A., & Pe ei a, F.C.N. (2001). Condi ional andom ields: p obabilis ic models o segmen ing and labeling sequence da a. P oceedings o he Eigh een h In e na ional Con e ence on Machine Lea ning, 282–289. Re ie ed Augus 9, 2021, om h ps:// eposi o y.upenn.edu/cgi/ iewcon en .cgi?a icle=1162&con ex =cis_pape s Lee, D. D., & Seung, H. S. (1999). Lea ning he pa s o objec s by non-nega i e ma ix ac o iza ion. Na u e, 401(6755), 788–791. h ps://doi.o g/10.1038/44565 Lee, JH., Pa k, S., Ahn, CM., & Kim, D. (2009). Au oma ic gene ic documen summa iza ion based on non- nega i e ma ix ac o iza ion. In o ma ion P ocessing & Managemen , 45(1), 20–34. h ps://doi.o g/10.1016/j.ipm.2008.06.002 56 Li, Y., Jia, B., Guo, Y., & Chen, X. (2017). Mining Use Re iews o Mobile App Compa isons. P oceedings o he ACM on In e ac i e, Mobile, Wea able and Ubiqui ous Technologies, 1(3), 1–15. h ps://doi.o g/10.1145/3130935 Liu, B. (2012). Sen imen Analysis and Opinion Mining. Syn hesis Lec u es on Human Language Technologies, 5(1), 1–167. h ps://doi.o g/10.2200/s00416ed1 01y201204hl 016 Liu, L., Tang, L., Dong, W., Yao, S., & Zhou, W. (2016). An o e iew o opic modeling and i s cu en applica ions in bioin o ma ics. Sp inge Plus, 5, 1608. h ps://doi.o g/10.1186/s40064-016-3252-8 Liu, Y., Jiang, C., & Zhao, H. (2019). Assessing p oduc compe i i e ad an ages om he pe spec i e o cus ome s by mining use -gene a ed con en on social media. Decision Suppo Sys ems, 123. h ps://doi.o g/10.1016/j.dss.2019.113079 Manning, C. D., Su deanu, M., Baue , J., Finkel, J., Be ha d, S. J., & McClosky, D. (2014). The S an o d Co eNLP na u al language p ocessing oolki . P oceedings o 52nd Annual Mee ing o he Associa ion o Compu a ional Linguis ics: Sys em Demons a ions, 55–60. h ps://doi.o g/10.3115/ 1/P14-5010 Medha , W., Hassan, A., & Ko ashy, H. (2014). Sen imen analysis algo i hms and applica ions: A su ey. Ain Shams Enginee ing Jou nal, 5(4), 1093–1113. h ps://doi.o g/10.1016/j.asej.2014.04.011 Mille , G. A. (1995). Wo dNe : a lexical da abase o English. Communica ions o he ACM, 38(11), 39–41. h ps://doi.o g/10.1145/219717.219748 Nielsen, F. Å. (2011, Ma ch 15). A new ANEW: E alua ion o a wo d lis o sen imen analysis in mic oblogs. Re ie ed Ma ch 15, 2021, om h ps://a xi .o g/abs/1103.2903 No man, G. J., No is, C. J., Gollan, J., I o, T. A., Hawkley, L. C., La sen, J. T., Cacioppo, J. T., & Be n son, G. G. (2011). Cu en Emo ion Resea ch in Psychophysiology: The Neu obiology o E alua i e Bi alence. Emo ion Re iew, 3(3), 349–359. h ps://doi.o g/10.1177/1754073911402403 O’Callaghan, D., G eene, D., Ca hy, J., & Cunningham, P. (2015). An analysis o he cohe ence o desc ip o s in opic modeling. Expe Sys ems wi h Applica ions, 42(13), 5645–5657. h ps://doi.o g/10.1016/j.eswa.2015.02.055 Osgood, C., Suci, G., & Tannenbaum, P. (1957). The measu emen o meaning. U bana, IL: Uni e si y o Illinois. Ouyang, Y., Guo, B., Lu, X., Han, Q., Guo, T., & Yu, Z. (2019). Compe i i eBike: Compe i i e Analysis and Popula i y P edic ion o Bike-Sha ing Apps Using Mul i-Sou ce Da a. IEEE T ansac ions on Mobile Compu ing, 18(8), 1760–1773. h ps://doi.o g/10.1109/TMC.2018.2868933 Pa k, D. H., Liu, M., Zhai, C., & Wang, H. (2015). Le e aging Use Re iews o Imp o e Accu acy o Mobile App Re ie al. SIGIR '15: P oceedings o he 38 h In e na ional ACM SIGIR Con e ence on Resea ch and De elopmen in In o ma ion Re ie al, 533–542. h ps://doi.o g/10.1145/2766462.2767759 57 Ped egosa, F., Va oquaux, G., G am o , A., Michel, V., Thi ion, B., G isel, O., Blondel, M., P e enho e , P., Weiss, R., Dubou g, V., Vande plas, J., Passos, A., Cou napeau, D., B uche , M., Pe o , M., & Duchesnay, É. (2011). Sciki -lea n: Machine lea ning in py hon. Jou nal o Machine Lea ning Resea ch, 12, 2825–2830. Pennebake , J., Mehl, M., & Niede ho e , K. (2003). Psychological aspec s o na u al language use: Ou wo ds, ou sel es. Annual Re iew o Psychology, 54(1), 547–577. h ps://doi.o g/10.1146/annu e .psych.54.101601.145041 Po e , M. E. (1980). Compe i i e S a egy: Techniques o Analyzing Indus ies and Compe i o s. New Yo k: F ee P ess. Qwe yus. (21 Oc obe 2013). Illus a ion o app oxima e non-nega i e ma ix ac o iza ion (NMF), Wikimedia Commons. h ps://commons.wikimedia.o g/wiki/File:NMF.png San o ini, B. (July 1990). Pa -o -Speech Tagging Guidelines o he Penn T eebank P ojec (3 d Re ision). Re ie ed May 08, 2021, om h ps:// eposi o y.upenn.edu/cgi/ iewcon en .cgi?a icle=1603&con ex =cis_ epo s Shah, F. A., Si s, K., & P ahl, D. (2019). Using app e iews o compe i i e analysis: ool suppo . P oceedings o he 3 d ACM SIGSOFT In e na ional Wo kshop on App Ma ke Analy ics (WAMA 2019), 40–46. h ps://doi.o g/10.1145/3340496.3342756 Slxu.public. (29 Sep embe 2009). Pla e no a ion o he Smoo hed LDA Model. Wikimedia Commons. h ps://commons.wikimedia.o g/wiki/File:Smoo hed_LDA.png S one, P. J., Dunphy, D. C., Smi h, M. S., & Ogil ie, D. M. (1966). The gene al inqui e : a compu e app oach o con en analysis. MIT P ess, Camb idge. Su, Y., Wang, Y., & Yang, W. (2019). Mining and Compa ing Use Re iews ac oss Simila Mobile Apps. 2019 15 h In e na ional Con e ence on Mobile Ad-hoc and Senso Ne wo ks (MSN), 338–342. h ps://doi.o g/10.1109/MSN48538.2019.00070 Sup ayogi, E., Budi, I., & Mahend a, R. (2018). In o ma ion Ex ac ion o Mobile Applica ion Use Re iew. 2018 In e na ional Con e ence on Ad anced Compu e Science and In o ma ion Sys ems (ICACSIS), 343–348. h ps://doi.o g/10.1109/ICACSIS.2018.8618164 Taboada, M., B ooke, J., To iloski, M., Voll, K., & S ede, M. (2011). Lexicon-Based Me hods o Sen imen Analysis. Compu a ional Linguis ics, 37(2), 267–307. h ps://doi.o g/10.1162/coli_a_00049 Thelwall, M. (2013). Hea and Soul: Sen imen S eng h De ec ion in he Social Web wi h Sen iS eng h. Re ie ed Ma ch 28, 2021, om h p://sen is eng h.wl .ac.uk/documen a ion/Sen iS eng hChap e .pd 58 T uică, C. O., Velcin, J., & Boicea, A. (20 15). Au oma ic Language Iden i ica ion o Romance Languages Using S op Wo ds and Diac i ics. 17 h In e na ional Symposium on Symbolic and Nume ic Algo i hms o Scien i ic Compu ing (SYNASC), 243–246. h ps://doi.o g/10.1109/SYNASC.2015.45 T uică, C., Rădulescu, F., & Boicea, A. (2016). Compa ing Di e en Te m Weigh ing Schemas o Topic Modeling. 2016 18 h In e na ional Symposium on Symbolic and Nume ic Algo i hms o Scien i ic Compu ing (SYNASC), 307–310. h ps://doi.o g/10.1109/SYNASC.2016.055 Vu, P. M., Nguyen, T. T., Pham, H. V., & Nguyen, T. T. (2015). Mining Use Opinions in Mobile App Re iews: A Keywo d-Based App oach (T). 2015 30 h IEEE/ACM In e na ional Con e ence on Au oma ed So wa e Enginee ing (ASE), 749–759. h ps://doi.o g/10.1109/ASE.2015.85 Wang, W., Wang, H., & Song, Y. (2016). Ranking p oduc aspec s h ough sen imen analysis o online e iews. Jou nal o Expe imen al & Theo e ical A i icial In elligence, 29(2), 227–246. h ps://doi.o g/10.1080/0952813X.2015.1132270 Wang, W., Feng, Y., & Dai, W. (2018). Topic analysis o online e iews o wo compe i i e p oduc s using la en di ichle alloca ion. Elec onic Comme ce Resea ch and Applica ions, 29, 142–156, h ps://doi.o g/10.1016/j.ele ap.2018.04.003 Whi elaw, C., Ga g, N., & A gamon, S. (2005). Using app aisal g oups o sen imen analysis. P oceedings o he 14 h ACM In e na ional Con e ence on In o ma ion and Knowledge Managemen - CIKM '05. h ps://doi.o g/10.1145/1099554.1099714 Wilson, T., Wiebe, J., & Ho mann, P. (2005). Recognizing con ex ual pola i y in ph ase-le el sen imen analysis. P oceedings o Human Language Technology Con e ence and Con e ence on Empi ical Me hods in Na u al Language P ocessing, 347–354. h ps://doi.o g/10.3115/1220575.1220619 Xu, K., Liao, S. S., Li, J., & Song, Y. (2011). Mining compa a i e opinions om cus ome e iews o Compe i i e In elligence. Decision Suppo Sys ems, 50(4), 743–754. h ps://doi.o g/10.1016/j.dss.2010.08.021 59 Appendix A Sampled e iews wi h w ong opics assigned by he NMF opic model Re iew Topic I ’s s ill ell you ha you go mail when you don’ 10 - app/accoun dele ion | download | accoun /login My game is ozen ka y60 help 2 - (g oup) cha | add ea u e | p i acy I go my s a ues saying showing ha I’m online when I had i change o o line bu s ill shows me being ac i e online I don’ like ha I’m appea ing o line o a eason o no be con ac 10 - app/accoun dele ion | download | accoun /login I can’ unde s and is ha a censo ship o jus s op esponding pe iod??? 6 - app/ ea u e wo king issue | apple wa ch The pas wo weeks i ’s been kicking me ou and aking me igh back in o my home sc een on my phone Facebook has no esponded o my epo o his happening I’m missing my messages! 10 - app/accoun dele ion | download | accoun /login I ’s now going o be my hi d ome ins alling his app and i keeps elling me ha I don’ ha e In e ne connec ion when I wan o login please help as o wha I should do ? 4 - app c ash Yeah e y disgus ed Messenge igh now wouldn’ le me in o my Messenge so I had o unins all i now I can’ eins all i keeps asking o paymen in o ma ion which I don’ no wha i has o do wi h i 4 - app c ash No longe ha e edi bu on op ion o upda e p o ile pic in messenge . When I sea ch online o help, sugges ions indica e o go h ough FB app. I do no use FB he e o e do no ha e ha app. 8 - s a us | sea ch con ac Well I ied upda ing my messenge bu i ’s been an hou and my in e ne is awesome I jus go i ixed and i ’s s ill loading idk i i ’s jus my phone o wha bu u don’ hink i is jus saying 10 - app/accoun dele ion | download | accoun /login Why does Facec ook need ano he app on you phone o any hing. So Ma k can ge pa id. T ha ’s why ., oh and he can collec in o ma ion abou you. 10 - app/accoun dele ion | download | accoun /login Hi I logged in o messenge wi h my Ins ag am accoun and I can’ ge logged in o i . Plz help me. You guys should ha e old people who use messenge wi h a Ins ag am accoun and ell hem we can’ use he Ins ag am login accoun anymo e o a leas you guys should gi e me my accoun o some hing 7 - social/ unc ional usage All acebook messenge use Please be awa e A menian employe s o Facebook used acebook o ban Aze baijani use s. They a e aking ad an age o wo king o Facebook o make censo ship. I s in ole an and shoud be s opped 6 - app/ ea u e wo king issue | apple wa ch Since wha e e upda e came h ough a ew weeks ago, Messenge will no longe le me open links. Pe iod. Doesn’ ma e i hey’ e ex e nal o link back o FB. One s a un il you guys ix his c ap. Mul i-billion dolla company, and you app is ash. 9 - gene al p oblems o ix 60 Appendix A (Con inued) I ha e used messenge o yea s, and ou o no whe e I no longe ecei e no i ica ions. I ’s in u ia ing as his is my main messaging app. I ha e ied e e y hing om oggling se ings o ese ing my phone. No hing wo ks. 6 - app/ ea u e wo king issue | apple wa ch ee Pales ine des oy Is ael. ee Pales ine des oy Is ael. ee Pales ine des oy Is ael. 2 - (g oup) cha | add ea u e | p i acy Ve y bad Bad because o his acis policy agains he Pales inians 2 - (g oup) cha | add ea u e | p i acy Facebook suppo e hnic cleansing agains mino i ies wo ldwide by banning people, shu ing, dele ing, limi ing audience o pos s abou opp essed people in Pales ine and o he a eas in he wo ld. 2 - (g oup) cha | add ea u e | p i acy #GazaUnde A ack #Pales ineUnde A ack #Sa e_Sheikh_Ja ah 2 - (g oup) cha | add ea u e | p i acy Because his is acebook’s hen I gi e 1 s a and also ee P ales ine 2 - (g oup) cha | add ea u e | p i acy The de elope s p e end o p omo e p i acy, bu sell sensi i e da a abou you calls and ex s o co po a ions. This is no shocking, as he app is owned by acebook. 5 - ( ideo) call | connec ion quali y I am no longe ge ing no i ica ions and sound when I ecei e ex . I’ e ese o ac o y on phone and wha app and s ill no hing. 10 - app/accoun dele ion | download | accoun /login Me ey phone mein chalna hai o wa na achi a ha chal nahi o esa sabaq sikhaonga k munh dikhane k laik nai ahoge. Ye me i p i acy hai au me i p i acy mein koi bhi en ess nhi ka na. Hukaayyy!! 2 - (g oup) cha | add ea u e | p i acy Te ible applica ion. I sends you in o ma ion o companies you don’ wan o o ha e. 11 - pho o/ ideo/link/message sending Wha sApp says i is end o end enc yp ed and no one can ead o hea no e en Wha sApp. Well no ue. I had someone hack my Wha sApp and ge ahold o all my messages. So sad ha you a e ad e ising his. I ’s no ue people clea you messages . 0 - message/no i ica ion ecei ing | ead eceip I al eady ge ing scammed and ha assed on i 10 - app/accoun dele ion | download | accoun /login #سدقلا_ضفتنت #عيبطتلا_هنايخ # ح_خيشلا_ حارج #لا_ديوهتل_ سدقلا #اوذقنأ_ ح_خيشلا_ حارج #نل_لحرن # اوذقنا_ ح_خيشلا_ حارج #sa esheikhja ah #ŞeyhJa ahmahallesinku a ın #Sal ailqua ie ediSheikhJa ah #Re edasVie elSheikhJa ah #sau ezlequa ie desheikhja ah 10 - app/accoun dele ion | download | accoun /login #gaza #sa e_pales ine 2 - (g oup) cha | add ea u e | p i acy acking my phone 10 - app/accoun dele ion | download | accoun /login None compa e 2 - (g oup) cha | add ea u e | p i acy Aw ul app. I can’ deac i a e i . And hey don’ wan you oo. S ay away om his app. I ied o deac i a e i . Impossible. Impossible o wo k, oo. 6 - app/ ea u e wo king issue | apple wa ch 61 Appendix B Sampled e iews wi h inconsis en labels o sen imen pola i y on weigh ed sen imen sco es Re iew au oAssigned_label Manual_label Rema k T ying o cancel a i de ha was o di ed inco ec ly, please help me cancel he o de I ha e ied e e y hing. O de is om Linda icke s in amoun o $29 . Please help Posi i e Nega i e Recen ly my messages won’ go h ough, i ’ll jus say sending bu will ake oughly 10-20 mins o inally send. My in e ne wo ks pe ec ly ine and so does my da a so I hink he app may need ano he ix/upda e asap! Posi i e Nega i e G ea app. Since he upda e he Blue oo h no longe wo ks wi h my esla speake s when ideo is on ( ine- - ideo p obably shouldn be on anyways). Fix i please Posi i e Nega i e Posi i e a i s , hen shi o app p oblems I need his app o calling Nega i e Posi i e Suppo anima ed s icke s Nega i e Posi i e Ad ice This is an app ha doesn disappoin . lo e i ! Nega i e Posi i e Ve y help ul o communica ion ac oss he wo ld o jus iPhone o and oid when elying on WiFi in dead zones Nega i e Posi i e i connec s me wi h my a o i e people. hank you. Lolinche Nega i e Posi i e I’ll s a by saying he app is p e y awesome. I only ha e a couple hings ha bug me pe sis en ly. The i s is a lack o imes amps. The ecen messages ead “10 hou s ago” o example. I’d like o see an ac ual ime so I don’ ha e o do ma h o he messages sen / ecei ed wi hin 24 hou s. Also, messages show as sen and ecei ed on my end, bu he ecipien ac ually doesn’ ge hem o hou s. And some imes he same hing happens wi h me on he ecei ing end. Posi i e Nega i e Posi i e a i s , hen shi o app p oblems Lo e he app. Use i daily. Access o pho os no a ailable in IOS 14. Goes o “ ecen ” olde o ind pho o album. No pic u es in his olde . Posi i e Nega i e Posi i e a i s , hen shi o app p oblems I swi ched om WeCha and Wha sApp o Signal. Nega i e Posi i e Could be be e i you guys gi e op ions o s a us o his o y. Also he p o ile pic u e op ion needs o be imp o ed. Nega i e Posi i e Ad ice 62