Analyzing Use Re iews o Messaging Apps
o Compe i i e Analysis
Wenyi Liang
Disse a ion p esen ed as pa ial equi emen o ob aining
he Mas e ’s deg ee in Da a Science and Ad anced Analy ics
i
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
ANALYZING USER REVIEWS OF MESSAGING APPS
FOR COMPETITIVE ANALYSIS
by
Wenyi Liang
Disse a ion p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Da a Science and
Ad anced Analy ics
Ad iso : P o . Mau o Cas elli
Augus 2021
ii
DEDICATION
This disse a ion is dedica ed o my pa en s o hei cons an unde s anding, encou agemen , and
suppo . This wo k is also dedica ed o many iends who ha e enligh ened me wi h hei d eams,
c ea i i y, and b a e y.
iii
ACKNOWLEDGEMENTS
I would like o exp ess my since e g a i ude o my supe iso P o . Mau o Cas elli o his a i ma ion
o he esea ch opic, suppo i e encou agemen , p omp eedback, and pa ien guidance in
comple ing his disse a ion.
I would also like o hank he acul y in he p og am o Da a Science and Ad anced Analy ics. The
aluable knowledge and skills hey had augh me laid he academic ounda ion o his wo k. My
special hanks go o P o . Fe nando Pe es and P o . Nuno Alpalhão who ga e me ex a lessons and
guidance on homewo k and p ojec s.
I wish o acknowledge he assis ance p o ided by he s a o lib a y and documen a ion se ices as
well as academic se ices o NOVAIMS in inding some esea ch ma e ials and sol ing he issue o
hesis egis a ion.
I would like o ex end my hanks o my classma es who had e e helped me wi h he s udy du ing he
i s academic yea . They had inspi ed me wi h p ac ical knowledge and in e es ing hough s. Ad ices
p o ided by E nes o on eading esea ch pape s we e eally help ul.
In he end, my app ecia ion goes o my o me supe iso s in wo k who come om di e en
coun ies bu coinciden ally assigned me he wo k o compe i i e analysis o pee companies o
mobile apps. This wo k expe ience, o some ex en , con ibu ed o he esea ch opic o his
disse a ion.
i
ABSTRACT
The ise o a ious messaging apps has esul ed in in ensi ely ie ce compe i ion, and he e a o Web
2.0 enables business manage s o gain compe i i e in elligence om use -gene a ed con en (UGC).
Tex -mining UGC o compe i i e in elligence has been d awing g ea in e es o esea che s.
Howe e , ele an s udies mos ly ocus on indus ies such as hospi ali y and p oduc s, and ew
s udies applied such echniques o e ec i ely pe o m compe i i e analysis o messaging apps.
He e, we conduc ed a compe i i e analysis based on opic modeling and sen imen analysis by ex -
mining 27,479 use e iews o ou iOS messaging apps, namely Messenge , Wha sApp, Signal and
Teleg am. The esul s show ha he pe o mance o opic modeling and sen imen analysis is
encou aging, and ha a combina ion o he ex ac ed app aspec -based opics and he adjus ed
sen imen sco es can e ec i ely e eal meaning ul compe i i e insigh s in o use conce ns,
compe i i e s eng hs and weaknesses as well as changes o use sen imen s o e ime. We
an icipa e ha his s udy will no only ad ance he exis ing li e a u e on compe i i e analysis using
ex mining echniques o messaging apps bu also help exis ing playe s and new en an s in he
ma ke o sha pen hei compe i i e edge by be e unde s anding hei use needs and he indus y
ends.
KEYWORDS
Compe i i e analysis; Topic modeling; Sen imen analysis; Tex mining; Use e iews; Messaging apps

INDEX
1 In oduc ion ......................................................................................................................... 1
2 Rela ed wo ks ....................................................................................................................... 3
2.1 Compe i i e analysis by ex -mining UGC ................................................................................ 3
2.2 Topic modeling ......................................................................................................................... 4
2.3 Sen imen analysis ................................................................................................................... 8
3 Me hodology ...................................................................................................................... 11
3.1 Da a collec ion ....................................................................................................................... 12
3.2 Da a p ep ocessing ................................................................................................................ 13
3.2.1 Con ac ion expanding ....................................................................................................... 13
3.2.2 Tex cleaning ...................................................................................................................... 13
3.2.3 Wo d co ec ion and no maliza ion .................................................................................. 13
3.2.3.1 Spell check .................................................................................................................. 13
3.2.3.2 B i ish-Ame ican spelling no maliza ion .................................................................... 14
3.2.3.3 Spelling co ec ion and abb e ia ion expansion ....................................................... 14
3.2.4 POS agging and lemma iza ion ......................................................................................... 17
3.2.5 Non-English e iews il e ing ............................................................................................. 18
3.2.6 Fea u e ex ac ion .............................................................................................................. 18
3.2.7 Cus omized s op wo d emo al ......................................................................................... 19
3.2.8 Re iew p uning .................................................................................................................. 19
3.3 Topic modeling ....................................................................................................................... 20
3.3.1 NMF opic model ................................................................................................................ 20
3.3.2 E alua ion o ex ac ed opics ........................................................................................... 21
3.4 Sen imen analysis ................................................................................................................. 21
3.4.1 VADER compound sco e .................................................................................................... 21
3.4.2 Weigh ed sen imen sco e ................................................................................................. 21
3.4.3 E alua ion o sen imen analysis ....................................................................................... 22
3.5 Compe i i e analysis .............................................................................................................. 23
3.5.1 Re iew dis ibu ions and a e age sen imen sco es ......................................................... 23
3.5.1.1 Visual e iew dis ibu ion .......................................................................................... 24
3.5.1.2 Visual compa ison o a e age sen imen sco es ....................................................... 24
3.5.2 Sen imen e olu ion ........................................................................................................... 24
i
4 Resul s and discussion......................................................................................................... 25
4.1 Resul s .................................................................................................................................... 25
4.1.1 Topic modeling ................................................................................................................... 25
4.1.1.1 Ex ac ed opics .......................................................................................................... 25
4.1.1.2 E alua ion esul o ex ac ed opics ......................................................................... 29
4.1.2 E alua ion esul o sen imen analysis ............................................................................. 30
4.1.3 Resul s o compe i i e analysis .......................................................................................... 32
4.1.3.1 Re iew dis ibu ions and a e age sen imen sco es ................................................. 32
4.1.3.2 Visual e iew dis ibu ion .......................................................................................... 33
4.1.3.3 Visual compa ison o a e age sen imen sco es ....................................................... 36
4.1.3.4 Sen imen e olu ion ................................................................................................... 37
4.2 Discussion ............................................................................................................................... 44
4.2.1 Topic modeling ................................................................................................................... 44
4.2.2 Sen imen analysis ............................................................................................................. 46
4.2.3 Compe i i e analysis .......................................................................................................... 46
5 Conclusion .......................................................................................................................... 50
6 Limi a ions and ecommenda ions o u u e wo ks ............................................................. 51
7 Bibliog aphy ....................................................................................................................... 52
Appendix A. Sampled e iews wi h w ong opics assigned by he NMF opic model ..................... 59
Appendix B. Sampled e iews wi h inconsis en labels o sen imen pola i y on weigh ed sen imen
sco es ........................................................................................................................................ 61
ii
LIST OF FIGURES
Figu e 1: The singula alue decomposi ion o LSA wi h k opics .............................................. 5
Figu e 2: Illus a ion o he aspec model .................................................................................. 6
Figu e 3: Illus a ion o he la en Di ichle alloca ion ............................................................... 6
Figu e 4: Illus a ion o he non-nega i e ma ix ac o iza ion ................................................. 7
Figu e 5: The esea ch amewo k o compe i i e analysis using opic modeling and sen imen
analysis ............................................................................................................................. 11
Figu e 6: Non-nega i e ma ix decomposi ion on ea u e e ms ............................................ 20
Figu e 7: Topics ex ac ed om he NMF model ..................................................................... 25
Figu e 8: Re iew dis ibu ion by opic o Messenge .............................................................. 34
Figu e 9: Re iew dis ibu ion by opic o Wha sApp ............................................................... 35
Figu e 10: Re iew dis ibu ion by opic o Signal ..................................................................... 35
Figu e 11: Re iew dis ibu ion by opic o Teleg am ............................................................... 36
Figu e 12: Compa ison o a e age sen imen sco es by opic ................................................. 37
Figu e 13: Sen imen e olu ion by opic .................................................................................. 38
iii
LIST OF TABLES
Table 1: In o ma ion o app and use e iews ......................................................................... 12
Table 2: Sample o da ase wi h selec ed ields ....................................................................... 12
Table 3: Addi ional B i ish-Ame ican spellings ........................................................................ 14
Table 4: Manual spelling co ec ions ....................................................................................... 15
Table 5: Abb e ia ion expansions ............................................................................................ 17
Table 6: POS ags o ea u e ex ac ion .................................................................................. 18
Table 7: Remo al p ocess o cus omized s op wo ds .............................................................. 19
Table 8: Con usion ma ix o sen imen e alua ion ............................................................... 22
Table 9: Explana ion o TP, FP, TN and FN ............................................................................... 23
Table 10: Accu acies o opic ex ac ion .................................................................................. 30
Table 11: Con usion ma ix o sen imen e alua ion on weigh ed sen imen sco es ............. 31
Table 12: Con usion ma ix o sen imen e alua ion on use a ings...................................... 31
Table 13: Con usion ma ix o sen imen e alua ion on no malized VADER compound sco es
.......................................................................................................................................... 31
Table 14: Pe o mance o sen imen analysis .......................................................................... 32
Table 15: Summa y o e iew dis ibu ions and a e age sen imen sco es ............................ 33
Table 16: S a is ical agg ega ion o a e age sen imen sco es by mon h, opic and app ....... 43
Table 17: Compa ison o keywo ds in ex ac ed opics ........................................................... 45
6
Figu e 2
Illus a ion o he aspec model
No e. Adap ed om P obabilis ic la en seman ic analysis — PLSA by Se g Ka po ich, 2013, Wikimedia
Commons (h ps://commons.wikimedia.o g/wiki/File:Вероятностный_латентно-
семантический_анализ.png). CC BY-SA 3.0.
Un o una ely, pLSA is no capable o assigning p obabili y o p e iously unseen documen s and is p one
o o e i ing due o he g ow h in pa ame e s wi h he inc easing numbe o documen s and wo ds
(Blei e al., 2003). To add ess hese p oblems, Blei e al. (2003) p esen ed he la en Di ichle alloca ion
(LDA), which is a hie a chical Bayesian model o h ee le els, namely documen , opic and wo d.
Acco ding o hei s udy, a documen con ains mul iple opics wi h di e en p obabili ies 𝜃, whose
dis ibu ion ollows he Di ichle dis ibu ion, and ano he Di ichle dis ibu ion also applies in he
p obabili y dis ibu ion o wo ds 𝜑 in a opic (Figu e 3). The pa ame e s o he p io dis ibu ions, i.e.,
Di ichle dis ibu ions, o opic dis ibu ion 𝜃 and wo d dis ibu ion 𝜑 a e 𝛼 and 𝛽 espec i ely.
Compa ed wi h he pLSA app oach, he LDA me hod gene alizes easily o unseen ex s because i
ob ains he pos e io dis ibu ion o he opic mix u e weigh s by combining hei p io dis ibu ion wi h
he sample da a a he han ea s hese weigh s as a la ge se o indi idual pa ame e s (Blei e al.,
2003).
Figu e 3
Illus a ion o he la en Di ichle alloca ion
No e. F om Pla e no a ion o he Smoo hed LDA Model by Slxu.public, 2009, Wikimedia Commons
(h ps://commons.wikimedia.o g/wiki/File:Smoo hed_LDA.png). CC BY-SA 3.0.

7
Some de i a ions based on he LDA me hod we e p esen ed o speci ic asks and imp o emen s.
Combining he adi ional LDA and a s a is ical p ocess, Blei e al. (2010) c ea ed he Hie a chical la en
Di ichle alloca ion (hLDA) o lea n opic hie a chies om complex da a. In hLDA, he la en opics a e
s uc u ed in a ee whe e each node ep esen s a opic wi h i s opic e ms. Aga wal and Chen (2010)
p oposed he ma ix ac o iza ion h ough LDA ( LDA) o p edic use a ings in ecommende sys ems
such as con en ecommenda ion, ad a ge ing and web sea ch, whose i ems a e a icles, ads and web
pages espec i ely. The LDA me hod egula izes he i em ac o s h ough LDA p io s. In he domain o
mobile apps, Pa k e al. (2015) p esen ed he AppLDA opic model ha e ie es sha ed opics om app
desc ip ions and use e iews and disca ds e iews which only con ains misaligned opics wi h app
desc ip ions. This opic model aims o ind ou he key aspec s o apps and in e ela e he ocabula y
be ween app de elope s and use s. Mo eo e , o handle he es ic ion due o a single Di ichle p io
dis ibu ion o opic p opo ions, Chien e al. (2018) in oduced he la en Di ichle mix u e model
(LDMM), which alloca es mul iple Di ichle p io dis ibu ions o lea n he la en opics and hei
p opo ions. Besides unsupe ised asks such as opic modeling and documen clus e ing, he LDMM
was also ex ended o a supe ised LDMM o documen classi ica ion (Chien e al., 2018).
Ano he common opic model is he non-nega i e ma ix ac o iza ion (NMF) (Lee & Seung, 1999),
whose de elopmen is ela i ely independen o he a o emen ioned opic models. This model was
ini ially in oduced o lea ning pa s o aces and seman ic ea u es o ex . The basic idea o NMF is
inding wo non-nega i e ma ices whose p oduc app oxima es a gi en non-nega i e ma ix, he eby
ac o izing he o iginal non-nega i e ma ix 𝑉 in o a basis ma ix 𝑊 and a coe icien ma ix 𝐻, such ha
𝑉 ≈ 𝑊𝐻 (Figu e 4). The column ec o o he o iginal ma ix 𝑉 is he weigh ed sum o all he column
ec o s in he le basis ma ix 𝑊, and he weigh coe icien is he elemen o he co esponding
column ec o in he igh coe icien ma ix 𝐻. Bo h LSA and NMF sha e he key idea o ma ix
ac o iza ion and dimensionali y educ ion. Howe e , singula ec o s decomposed om LSA can
con ain nega i e alues, while NMF has non-nega i e cons ain s and i s posi i e and ze o coe icien s
a e mo e in line wi h human cogni i e p ocess when i comes o in e p e ing he impo ance o he
wo ds in ex ac ed opics (Lee e al., 2009).
Figu e 4
Illus a ion o he non-nega i e ma ix ac o iza ion
No e. F om Illus a ion o app oxima e non-nega i e ma ix ac o iza ion (NMF) by Qwe yus, 2013,
Wikimedia Commons (h ps://commons.wikimedia.o g/wiki/File:NMF.png). CC BY-SA 3.0.
The LDA me hod is cu en ly conside ed o be one o he mos popula opic models and has been
widely used o ex ac ing use ul in o ma ion om app- ela ed pos s and e iews (e.g., Iacob & Ha ison,
8
2013; Guzman & Maalej, 2014; Ouyang e al., 2019; Su e al., 2019). Howe e , NMF a ou pe o med
LDA in e ms o execu ion ime, while he accu acy di e ence be ween hese wo models was i ial
(T uică e al., 2016). Al hough he NMF app oach p eceded he in oduc ion o LDA, some s udies ha e
shown i s e ec i eness in ex ac ing opics om an in o mal ex ual con en . O’Callaghan e al. (2015)
analyzed opics ex ac ed by NMF-based me hods and LDA-based me hods in six co po a wi h a o al
numbe o 501,743 ex ual documen s and ound ha opics p oduced by NMF egula ly ha e highe
cohe ence han hose gene a ed by LDA, especially o niche o non-mains eam co po a. This esul is
in line wi h Con e as-Piña and Ríos’s (2016) conclusion ha NMF ex ac ed mo e use ul and cohe en
opics han LSA and LDA based on an expe imen on a da ase o 21,863 consume complain s abou
depa men s o e’s c edi ca ds. Mo eo e , acco ding o an expe imen on 57,934 use e iews o a
popula e-comme ce app eleased on Google Play, NMF ob ained a be e solu ion compa ed o LDA in
modeling opics om app e iews (Sup ayogi e al., 2018). The ecen s udy o Albalawi e al. (2020)
shows bo h LDA and NMF app oaches deli e ed mo e meaning ul opics han LSA, andom p ojec ion
and p incipal componen analysis when dealing wi h sho ex s such as commen s and e iews.
Use e iews o messaging apps usually con ain in o mal exp essions. Based on he exis ing s udies, we
selec ed he NMF me hod as he opic model o ou esea ch comp ehensi ely conside ing
e ec i eness and e iciency. Fo use e iews, he da a ma ix con ains non-nega i e alues when each
e m in he e iew ex s is p ope ly ep esen ed, e.g., by sco es o e m equency-in e se documen
equency (TF-IDF).
2.3 SENTIMENT ANALYSIS
Sen imen analysis e e s o he compu a ional s udy o human opinions, a i udes and emo ions
owa ds en i ies such as indi iduals, e en s and opics, and aims o disco e opinions in ex s, iden i y
he emo ions hese opinions exp ess and classi y hei sen imen pola i y (Medha , 2014). To achie e
sen imen classi ica ion, bo h supe ised lea ning me hods and unsupe ised app oaches can be applied
(Liu, 2012). Supe ised lea ning me hods equi e aining da a wi h sen imen labels. In eal p ac ice,
da a sou ces om use -gene a ed pos s and e iews do no include such labels. Classi ying he
sen imen pola i y o s eng h scale o UGC has become a popula esea ch opic in ecen yea s.
Acco ding o Liu (2012), sen imen wo ds domina e he sen imen pola i y, and hus hese sen imen
wo ds and ph ases may be u ilized o sen imen classi ica ion in an unsupe ised manne .
One o he unsupe ised app oaches o sen imen analysis o UGC is he lexicon-based me hod, which
equi es a p ede ined lexicon. The Gene al Inqui e (S one e al., 1966) is a lexical se wi h syn ac ic,
seman ic and p agma ic in o ma ion o pa -o -speech agged wo ds and gi es labels o he sen imen
pola i y o mos o i s included wo ds. The Mul i-Pe spec i e Ques ion Answe ing (MPQA) subjec i i y
lexicon (Wilson e al., 2005) also p o ides he same s uc u al in o ma ion as he Gene al Inqui e .
Mo eo e , Wilson e al. (2005) included he addi ional subjec i i y le el o a wo d o a ph ase wi h a
label o s ong o weak. B adley and Lang (1999) c ea ed he A ec i e No ms o English Wo ds
(ANEW), a dic iona y in which 1,034 English wo ds a e a ed in e ms o alence, a ousal and dominance
on a con inuous scale be ween 1 and 9. The h ee a ed dimensions a e based on Osgood e al.’s (1957)
heo y o emo ions. In hei heo y, alence (o pleasan ness) and a ousal ( he in ensi y o exci emen )
9
a e he p incipal dimensions, while dominance (o con ol) is a less s ongly- ela ed dimension when i
comes o emo ions in oked by a wo d. Simila o ANEW (B adley & Lang, 1999), Sen iWo dNe (Esuli &
Sebas iani, 2006; Baccianella e al., 2010) also assigns h ee sen imen sco es o each synonym se
(synse ), albei om h ee di e en aspec s ega ding posi i i y, nega i i y and neu ali y. This lexical
esou ce is cons uc ed on Wo dNe (Mille , 1995; Fellbaum, 1998) and publicly a ailable in he Na u al
Language Toolki (NLTK) (Bi d e al., 2009) o esea ch pu poses. Fu he mo e, Linguis ic Inqui y and
Wo d Coun (LIWC) (Pennebake e al., 2003) p o ides a p op ie a y dic iona y ha o ganizes each wo d
o wo d s em ac oss one o mo e psychologically ele an ca ego ies, which can in e posi i e o
nega i e emo ions. Wi h he apid de elopmen o social media, mo e and mo e cybe ela ed wo ds
appea in mic oblogs. Nielsen (2011) s a ed ha some o he a o emen ioned Gene al Inqui e , MPQA
subjec i i y lexicon, ANEW and Sen iWo dNe do no inco po a e s ong obscene wo ds and In e ne
slangs, and hus he p esen ed a new ANEW, a Twi e -based sen imen wo d lis including cybe slangs
and obscene wo ds. In his compa a i e expe imen , he new ANEW exceeded he Gene al Inqui e ,
MPQA subjec i i y lexicon and ANEW, bu all hese lexicons did no pe o m as well as Sen iS eng h
(Thelwall, 2013), a lexicon-based ool o sen imen analysis.
Acco ding o Thelwall (2013), Sen iS eng h is a sen imen analysis ool o classi ica ion o social web
ex s. This ool uses wo ds and wo d s ems om he exis ing LIWC and Gene al Inqui e as well as
special wo ds and ph ases widely used on social media, and sen imen sco es o hese wo ds a e
anno a ed by humans and imp o ed wi h machine lea ning me hods. Also, some ules we e cons uc ed
o cope wi h non-s anda d ex ual exp essions pa icula ly in social media such as emo icons,
emphasized punc ua ions and in ended misspellings. The es ed cases show ha Sen iS eng h
pe o med well on a wide ange o social media ex s, bu less well on ex s wi h i onic and sa cas ic
exp essions (Thelwall, 2013). Sen iS eng h has been used o aspec -based sen imen analysis o use
e iews o mobile apps (Guzman & Maalej, 2014), o anking p oduc aspec s om online e iews
(Wang e al., 2016) and o sen imen measu emen o use commen s on social media (He e al.,
2016).
Simila o Sen iS eng h, he Valence Awa e Dic iona y o sEn imen Reasoning (VADER) (Hu o &
Gilbe , 2014) is ano he lexicon-based sen imen analysis ool designed o social media con ex s. The
human- alida ed sen imen lexicon o VADER exploi s he exis ing lexicons such as he Gene al Inqui e ,
LIWC and ANEW, inco po a es addi ional e ms commonly used in mic oblogs and is buil on i e
g amma ical and syn ac ical ules such as punc ua ions, capi aliza ion and deg ee modi ie s. In hei
expe imen s, VADER achie ed no able success in social media domain and gene ally ou pe o med a
majo i y o he well- ega ded sen imen analysis ools including he Gene al Inqui e , ANEW, LIWC, Hu-
Liu04 opinion lexicon (Hu & Liu, 2004), Wo d-Sense Disambigua ion (Akkaya e al., 2009), Sen iWo dNe
and Sen icNe (Camb ia e al., 2012). Recen s udies ha e employed VADER o analyzing he sen imen s
o use e iews o mobile apps (Huebne e al., 2018; Su e al., 2019) and mic oblogs on Twi e (Elbagi
& Yang, 2019) wi h encou aging pe o mance.
Sen iS eng h and VADER di e in he ou pu o sen imen sco es. Fo each inpu , Sen iS eng h epo s
wo independen sco es o posi i e and nega i e scales based on he psychological heo y o humans’
10
mixed emo ions (No man e al., 2011) (Thelwall, 2013), while VADER p o ides a compound sco e
compu ed om he posi i e, neu al and nega i e sco es (Hu o & Gilbe , 2014).
Ano he ea lie in oduced dic iona y-based ool o ex ac ing sen imen om ex s is he Seman ic
O ien a ion CALcula o (SO-CAL) (Taboada e al., 2011). Di e en om wo d-based app oaches using
adjec i es (e.g., Whi elaw e al., 2005) o adjec i es and ad e bs (e.g., Benama a e al., 2007) o in e
he emo ional o ien a ion, SO-CAL exploi s wo ds including adjec i es, e bs, nouns, and ad e bs o
calcula e he sen imen pola i y and s eng h. Apa om his ex ension o pa s o speech (POS), SO-
CAL also inco po a es a dic iona y o in ensi ie s and a e ined nega ion app oach. Thei esul s show
ha SO-CAL achie ed consis en pe o mance on comple ely unseen ex s ac oss domains, di e en
om Sen iS eng h and VADER designed o social web ex s.
Use e iews a e one ype o UGC, whose language exp essions a e simila o use pos s and commen s
on social media. In he p esen wo k, we conside ed sen imen analysis ool wi h a ocus on social media
ex s and included he ecen ly in oduced VADER o be pa o he sen imen analysis o ou esea ch
since he VADER compound sco ing me hod is mo e app op ia e o ou u he calcula ion o he inal
sen imen sco es.
11
3 METHODOLOGY
This sec ion in oduces he gene ic a chi ec u e o ou esea ch ocusing on h ee key esea ch issues:
opic modeling, sen imen analysis and compe i i e analysis (Figu e 5).
As depic ed in Figu e 5, ou amewo k s a s wi h da a collec ion, ollowed by a se ies o domain-speci ic
da a p ep ocessing s eps including con ac ion expanding, ex cleaning, spelling no maliza ion, POS
agging, wo d lemma iza ion, non-English e iews emo al, app ea u e ex ac ion, emo al o cus omized
s op wo ds and e iew p uning. A e his, a opic model akes he ex ac ed ea u e e ms as inpu o ind
unde lying app aspec -based opics. In pa allel, pa ially p ep ocessed e iew sen ences we e used o
compu e hei sen imen sco es. In he end, we summa ized he ex ac ed opics and he sen imen sco es
o conduc a mul i- ace ed compe i i e analysis. In addi ion o s a is ical summa ies, we also c ea ed
compa a i e isualiza ions o be e e eal compe i i e insigh s in o se e al aspec s.
Figu e 5
The esea ch amewo k o compe i i e analysis using opic modeling and sen imen analysis

12
3.1 DATA COLLECTION
The app e iew managemen pla o m AppFollow (App ollow, n.d.) agg ega es use e iews om iOS,
And oid, Mic oso and Amazon app s o es. We selec ed he Uni ed S a es as he coun y and sepa a ely
expo ed use e iews o ou iOS mobile apps, namely Messenge , Wha sApp Messenge , Signal - P i a e
Messenge and Teleg am Messenge , be ween June 1, 2020 and May 31, 2021. Since he app names o
Wha sApp Messenge , Signal - P i a e Messenge and Teleg am Messenge ca y he wo d “Messenge ”,
which is also he name o he o he app Messenge , we eplaced he h ee app names wi h “Wha sApp”,
“Signal” and “Teleg am” espec i ely o cla i y he app names and minimize edundancy.
A o al numbe o 27,479 e iews we e collec ed. Table 1 lis s he numbe o e iews and e iew
pe cen age o each app.
Table 1
In o ma ion o app and use e iews
App Name
App Company
# o Re iews (%)
Messenge
Facebook, Inc.
12,346 (44.93%)
Wha sApp
Wha sApp Inc.
10,513 (38.36%)
Signal
Signal Messenge , LLC
2,633 (9.58%)
Teleg am
Teleg am FZ-LLC
1,987 (7.23)
TOTAL
27,479 (100%)
The expo ed da ase s we e conca ena ed in o a single da ase wi h 26 ields. The ields ega ding he
e iew da e ime, app name, use a ing and e iew con en we e selec ed o u he analysis. Table 2
p o ides a sample o he da ase wi h selec ed ields.
Table 2
Sample o da ase wi h selec ed ields
Da e
AppName
Ra ing
Re iew
2020-06-01 01:10:35
Messenge
1
Can’ send pic u es o ideos
2020-06-01 00:06:08
Wha sApp
5
S ill awesome
2020-06-01 01:17:05
Signal
4
i lo e he app since i ealized i needed he p i acy
om hangou s and zoom, bu whene e i y o
add someone o a g oupcha , i shows up as “e o
use al eady in g oup” when hey’ e no ? is
anyone else ha ing his p oblem?
2020-06-01 04:14:59
Teleg am
5
The bes app I’ e e e seen
13
3.2 DATA PREPROCESSING
3.2.1 Con ac ion expanding
Con ac ions a e widely used in in o mal w i ing, and use e iews a e no excep ion. The wo e ms
“doesn' ” and “does no ” ha e exac ly he same meaning, bu hey a e wo comple ely di e en okens
o a compu e . Such con ac ions inc ease he dimensionali y o he documen - e m ma ix o u he
opic modeling. To educe he edundancy in he da a, we expanded he common con ac ions and
slangs using he Con ac ions Py hon lib a y (Koo en, n.d.) such ha , e.g., “doesn' ” was expanded o
“does no ”, “could’ e” o “could ha e”, and “wanna” o “wan o”. This s ep also p e en ed he
subsequen ex cleaning om gene a ing many w ong spellings such as “doesn ” and “could e” a e
emo ing he apos ophes. In addi ion, his Py hon lib a y is also capable o co ec ing common ypos
ela ed o con ac ions such as “didn ” and “can ”. This kind o ypos would be au oma ically expanded.
3.2.2 Tex cleaning
Use e iews a e usually mixed-cased and comp ise many non- ex ual cha ac e s such as punc ua ions
and digi s as well as cha ac e s in o eign languages. These cha ac e s p o ide leas use ul in o ma ion
abou he app ea u es and use a i ude bu in oduce many noises o he opic model. As a esul , we
lowe cased all e iew ex s and hen emo ed all URL’s, newline cha ac e s, punc ua ions, emojis, digi s
and non-English cha ac e s.
Also, e iews wi h only one wo d such as “awesome” and “ugh” we e d opped. These sho e iews a e
usually p aise, c i iques o modal pa icles wi hou men ioning any speci ic app ea u es, and hus
con ey meaningless in o ma ion o compe i i e analysis. A o al o 1,952 such e iews was emo ed
and 25,527 e iews emained a he end o his s ep.
3.2.3 Wo d co ec ion and no maliza ion
Use e iews end o in ol e spelling e o s, a ian o ms o English spellings and in o mal
abb e ia ions. Gu and Kim’s (2015) ypo lis only collec s he common ypos in use e iews o gene al
apps, and mos o hei ypos could be co ec ed by he Con ac ions Py hon lib a y. Mo eo e , hei
ypo lis does no inco po a e di e en o ms o English spellings, which exp ess he same meaning bu
ep esen en i ely di e en hings o a compu e jus like con ac ions. Since we we e dealing wi h a
speci ic ype o apps, we pe o med a domain-speci ic co ec ion and no maliza ion o wo ds o educe
he noises o u he POS agging, opic modeling and sen imen analysis.
This sub-sec ion includes checking he spelling p ope y o each unique wo d, no malizing B i ish
spellings o Ame ican spellings, and co ec ing English misspellings and abb e ia ions.
3.2.3.1 Spell check
Be o e pe o ming he spelling no maliza ion and co ec ion, we needed o iden i y he spelling
p ope ies o each unique wo d. The PyEnchan (Kelly, 2011) spellchecke was adop ed o his ask. This
Py hon lib a y is capable o ecognizing di e en a ie ies o he English language such as Ame ican
14
English, B i ish English and Canadian English. The same wo d in di e en a ie ies is unde s andable o
humans bu noisy o compu e s. The wo common Ame ican and B i ish a ie ies we e conside ed in
ou spell check.
We compu ed he equency o each unique wo d in he co pus and so ed his equency in descending
o de . The unique uppe case wo ds we e hen passed o PyEnchan o checking i each o hem is a
co ec ly spelled English wo d, in bo h Ame ican and B i ish spelling. We used capi al wo ds because all
wo ds we e lowe cased du ing he ex cleaning, and PyEnchan is case-sensi i e. Fo ins ance, “english”
is no a co ec spelling, bu “English” o “ENGLISH” is. Finally, o each unique wo d, he PyEnchan
spellchecke epo ed “T ue” o “False” in bo h Ame ican and B i ish spellings.
3.2.3.2 B i ish-Ame ican spelling no maliza ion
Fo wo ds in T ue B i ish spelling bu False Ame ican spelling, we no malized hem o Ame ican spellings
based on he B i ish spelling dic iona y o an Ame ican-B i ish English ansla o (Hype eali y, n.d.), e.g.,
“beha iou ” o “beha io ”. Some o he T ue B i ish spellings we e no ound in his dic iona y, so we
manually added he Ame ican spellings (Table 3). Finally, we eplaced all he B i ish spellings in he
cleaned e iew ex s wi h hei Ame ican coun e pa s.
Table 3
Addi ional B i ish-Ame ican spellings
B i ish spelling
Ame ican spelling
amongs
among
lea n
lea ned
cus omisa ion
cus omiza ion
acknowledgemen
acknowledgmen
3.2.3.3 Spelling co ec ion and abb e ia ion expansion
Wo ds wi h bo h False spellings a e usually misspelled English wo ds, in o mal English abb e ia ions o
non-English wo ds. In his s ep, we only co ec ed he English misspellings and in o mal English
abb e ia ions, e.g., “ ecie e” o “ ecei e” and “pls” o “please”. The de ailed s eps we e selec ing he
bo h False wo ds, going h ough hose wi h a equency g ea e han wo and manually assigning he
co ec o m o each wo d i he wo d ob iously esembles an English wo d o abb e ia ion (Table 4).
15
Table 4
Manual spelling co ec ions
Wo d
Manual co ec ion
b
acebook
pls, plz, pleasee, pleaseee, plzzz, pleaseeee
please
idk
ecei e do no know
ui
use in e ace
message , messange , massenge , messnge ,
mesenge , mesengge
messenge
ppl, ppls
people
msgs, messeges, messenges
messages
cuz, coz, bcz, cus
because
ecei e
a lo
de s
de elope s
wa sapp, wha sup, wha app, wha sap, sapp,
wha ssap, wha saap, wha spp, wha sapps, wa sapp,
wha sapp, wha up, whas app, wa sap, wha supp
wha sapp
hx
hanks
mins
minu es
soooo, sooo, soo, sooooo, soooooo
so
ux
use expe ience
acc
accoun
apps o e
app s o e
useable
usable
de
de elope
un ill
un il
ne wo kmanage e o
ne wo k manage e o
iam
ecei e am
lly, ealy
eally
ecei e
happened
ecei e
ecei e
goin
going
aceid
ace id
ha , h
ha e
homesc een
home sc een
dosen
does no
awsome
awesome
22
3.4.3 E alua ion o sen imen analysis
Since we used he weigh ed sen imen sco es, a he han he use a ings o no malized VADER
compound sco es, as he inal sen imen sco es o u he compe i i e analysis, we assessed he
e ec i eness o he sen imen analysis om wo pe spec i es. Fi s ly, we e alua ed he pe o mance o
he weigh ed sen imen sco es. Secondly, we compa ed he pe o mance o he weigh ed sen imen
sco es wi h ha o he use a ings and o he no malized VADER compound sco es.
Fo e alua ing he pe o mance o he weigh ed sen imen sco es, a label indica ing sen imen pola i y
was au oma ically assigned o each e iew acco ding o a sco e benchma k o 4 based on he weigh ed
sen imen sco e. Fo example, e iews wi h weigh ed sen imen sco es g ea e han o equal o 4 we e
au oma ically assigned a “Posi i e” label, o he wise “Nega i e”. This sco e benchma k was based on he
assump ion ha e iews wi h a sen imen sco e abo e 4 usually exp ess p aise o iendly ad ice, while
below 4 end o epo some issues ega ding speci ic app ea u es, no necessa ily consis ing o wo ds
linking o emo ions o ha e, ange o annoyance. A e au oma ically assigning he label o sen imen
pola i y based on he weigh ed sen imen sco es o each e iew, we andomly sampled 1% o e iews
om each pola i y and conduc ed manual labeling o “Posi i e” o “Nega i e” o each sampled e iew.
The c i e ion o manual labeling con o med o he assump ion o he sco e benchma k. An addi ional
ema k o he c i e ion was ha e iews exp essing p aise i s and hen a shi o epo issues
ega ding speci ic app aspec s we e manually labeled as “Nega i e”.
To compa e he pe o mance o he weigh ed sen imen sco es wi h ha o he o he wo sco ing
me hods, we au oma ically assigned wo mo e labels indica ing sen imen pola i y on he use a ings
and no malized VADER compound sco es o each sampled e iew acco ding o he same sco e
benchma k o 4.
A his poin , each o he sampled e iews had h ee au oma ic labels o sen imen pola i y on he
weigh ed sen imen sco es, use a ings and no malized VADER compound sco es espec i ely acco ding
o he sco e benchma k o 4 and one manual label. We summa ized he numbe o sampled e iews
based on he labels o sen imen pola i y and p esen ed he s a is ical esul s in h ee con usion
ma ices (Table 8). Table 9 explains he e minology o TP, FP, TN and FN.
Table 8
Con usion ma ix o sen imen e alua ion
Weigh ed sen imen sco es/Use a ings/no malized VADER compound sco es
Nega i e (< 4)
Posi i e (>=4)
Nega i e (manual)
TN
FP
Posi i e (manual)
FN
TP

23
Table 9
Explana ion o TP, FP, TN and FN
Ac onym
Te m
Desc ip ion
TP
T ue Posi i e
he numbe o e iews wi h bo h an au oma ically assigned label (based
on he sco e benchma k o 4) and a manual label o be “Posi i e”
FP
False Posi i e
he numbe o e iews wi h an au oma ically assigned label (based on
he sco e benchma k o 4) “Posi i e” and a manual label “Nega i e”
TN
T ue Nega i e
he numbe o e iews wi h bo h an au oma ically assigned label (based
on he sco e benchma k o 4) and a manual label o be “Nega i e”
FN
False Nega i e
he numbe o e iews wi h an au oma ically assigned label (based on
he sco e benchma k o 4) “Nega i e” and a manual label “Posi i e”
Finally, we used he ollowing classi ica ion me ics P ecision, Recall and F1-sco e o espec i ely
e alua e he pe o mance o he weigh ed sen imen sco es, use a ings and no malized VADER
compound sco es:
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑇𝑃
𝑇𝑃 + 𝐹𝑃
𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑇𝑃
𝑇𝑃 + 𝐹𝑁
𝐹1−𝑠𝑐𝑜𝑟𝑒 =2 ∗ 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∗ 𝑅𝑒𝑐𝑎𝑙𝑙
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙
3.5 COMPETITIVE ANALYSIS
This sec ion summa izes he ex ac ed opics and he weigh ed sen imen sco es om di e en
pe spec i es o e ealing compe i i e in elligence. The summa y o e iew dis ibu ions and a e age
sen imen sco es indica ed he o e all e iew coun s and use sen imen s by opic o each app du ing
he pe iod be ween June 1, 2020 and May 31, 2021, while he sen imen e olu ion e lec ed he
changes o use sen imen s o e ime on a mon hly basis.
3.5.1 Re iew dis ibu ions and a e age sen imen sco es
To unde s and he e iew dis ibu ion, we agg ega ed he numbe o e iews by opic o each app. The
mos ly discussed opics a e usually he main app aspec s ha he use s a e mo e conce ned abou .
Using he weigh ed sen imen sco es as he inal sen imen sco es, we also calcula ed he a e age
sen imen sco es by opic o each app. The a e age sco e o he opic 𝑡 o he app 𝑎 was calcula ed
using he ollowing o mula:
(4)
(5)
(6)
24
𝐴𝑣𝑔.𝑆𝑒𝑛𝑡𝑖𝑆𝑐𝑜𝑟𝑒(𝑡𝑎) = 1
𝑛∑𝑠_𝑤𝑒𝑖𝑔ℎ𝑡𝑒𝑑(𝑟𝑡𝑎)𝑖
𝑛
𝑖=1
whe e 𝑛 is he o al numbe o e iews o he opic 𝑡 o he app 𝑎, and he weigh ed sen imen sco e o
each e iew is deno ed by 𝑠_𝑤𝑒𝑖𝑔ℎ𝑡𝑒𝑑(𝑟𝑡𝑎)𝑖 whe e 𝑖 = 1,2,…,𝑛.
3.5.1.1 Visual e iew dis ibu ion
To be e isualize he main app aspec -based opics discussed by use s, we c ea ed ou pie cha s
which espec i ely show he pe cen ages o e iews ega ding each opic o he ou messaging apps
based on he e iew agg ega ion by opic o each app.
3.5.1.2 Visual compa ison o a e age sen imen sco es
We c ea ed a ba cha based on he a e age sen imen sco es by opic o each app o isually compa e
he a e age sen imen sco es o he ou messaging apps a a opic le el. The a e age sen imen sco es
we e assumed o e lec he use sa is ac ion owa ds speci ic app aspec -based opics.
3.5.2 Sen imen e olu ion
We g ouped he e iews by mon h and compu ed he a e age sen imen sco es by opic o each app on
a mon hly basis. Mo eo e , o be e isualize he mon hly changes o use sen imen s o he ou
messaging apps om June 2020 o May 2021, we plo ed wel e line cha s, each o an app aspec -
based opic.
(7)
25
4 RESULTS AND DISCUSSION
A e mul iple s eps o da a p ep ocessing, a o al numbe o 21,395 in o ma i e e iews emained o
opic modeling and sen imen analysis, whose esul s a e demons a ed in his sec ion, along wi h he
s a is ical and isual ou comes o compe i i e analysis. A he end, a discussion ega ding he easonings
and in e p e a ions behind ou indings is p esen ed.
4.1 RESULTS
4.1.1 Topic modeling
This sub-sec ion p esen s he opics ex ac ed om he opic model and he accu acies o opic
ex ac ion in he manual e alua ion.
4.1.1.1 Ex ac ed opics
Figu e 7 displays he wel e opics decomposed om he NMF opic model, each wi h i s opic label
numbe and opic name based on he i een opic-wo ds and hei co esponding weigh alues. Mos
opics ha e one o wo domina ing wo ds excep o opic 2 - (g oup) cha | add ea u e | p i acy and
opic 10 - app/accoun dele ion | download | accoun /login. These wo opics hold a ai numbe o
wo ds wi h mode a ely dec easing impo ance, and hus hey comp ise mo e aspec s ega ding app
ea u es. The es o he opics, hough led by one o wo wo ds, a e no always me ely o in ui i ely
ep esen ed by hei domina ing wo ds. The o he wo ds in hese opics also comple e hei domina ing
wo ds o ep esen he aspec mo e clea ly o e en gi e in o ma ion abou addi ional app- ela ed
aspec s depending on he weigh alues o he opic wo ds.
Figu e 7
Topics ex ac ed om he NMF model
26
Figu e 7 (Con inued)
27
Figu e 7 (Con inued)

28
Figu e 7 (Con inued)
29
Figu e 7 (Con inued)
4.1.1.2 E alua ion esul o ex ac ed opics
Table 10 shows he esul o he manual e alua ion in ex ac ed opics wi h hei accu acy sco es so ed
by opic in descending o de . Based on he o al numbe o 215 samples, an o e all accu acy o 86.05%
was achie ed ac oss all opics. The accu acy sco es o eigh opics (1 - app/link opening, 3 - upda e | ios
e sion, 7 - social/ unc ional usage, 5 - ( ideo) call | connec ion quali y, 11 - pho o/ ideo/link/message
sending, 9 - gene al p oblems o ix, 0 - message/no i ica ion ecei ing | ead eceip and 8 - s a us |
sea ch con ac ) a e abo e he o e all sco e. In de ail, bo h opic 1 - app/link opening and opic 3 -
upda e | ios e sion ecei ed a 100% accu acy, while opic 5 - ( ideo) call | connec ion quali y and opic
11 - pho o/ ideo/link/message sending ecei ed he same accu acy o 93.75%, opic 7 - social/ unc ional
usage in be ween, wi h 94.74%. Topic 9 - gene al p oblems o ix also ob ained a sco e abo e 90%. In
addi ion, he accu acy sco es o ou opics (0 - message/no i ica ion ecei ing | ead eceip , 8 - s a us |
sea ch con ac , 2 - (g oup) cha | add ea u e | p i acy and 4 - app c ash) all be ween 80% and 90%,
30
while opic 10 - app/accoun dele ion | download | accoun /login ecei ed he lowes accu acy
(68.97%), abou 2.5% lowe han ha o opic 6 - app/ ea u e wo king issue | apple wa ch.
Table 10
Accu acies o opic ex ac ion
Topic
# o
samples
#. o
incohe ence
Accu acy
OVERALL
215
30
0.8605
1 - app/link opening
16
1.0000
3 - upda e | ios e sion
14
1.0000
7 - social/ unc ional usage
19
1
0.9474
5 - ( ideo) call | connec ion quali y
16
1
0.9375
11 - pho o/ ideo/link/message sending
16
1
0.9375
9 - gene al p oblems o ix
14
1
0.9286
0 - message/no i ica ion ecei ing | ead eceip
10
1
0.9000
8 - s a us | sea ch con ac
8
1
0.8750
2 - (g oup) cha | add ea u e | p i acy
49
9
0.8163
4 - app c ash
10
2
0.8000
6 - app/ ea u e wo king issue | apple wa ch
14
4
0.7143
10 - app/accoun dele ion | download | accoun /login
29
9
0.6897
4.1.2 E alua ion esul o sen imen analysis
Based on a o al numbe o 214 samples, he con usion ma ices (Table 11, Table 12, Table 13) p esen
he s a is ical ou comes o he sen imen e alua ion on weigh ed sen imen sco es, use a ings and
no malized VADER compound sco es espec i ely. Acco ding o he manual labels, he majo i y class is
Nega i e in he sample e iews. The numbe s o co ec ly classi ied labels o weigh ed sen imen
sco es, use a ings and no malized VADER compound sco es a e 202, 192 and 182 espec i ely. As o
he inco ec ly classi ied labels, he numbe o alse Nega i e (7) is sligh ly g ea e han ha o alse
Posi i e (5) o weigh ed sen imen sco es. On he con a y, he numbe o alse Nega i e is ob iously
smalle han ha o alse Posi i e o use a ings and no malized VADER compound sco es. Fo use
a ings, he numbe o alse Posi i e is 20 and o alse Nega i e is only 2. Fo no malized VADER
compound sco es, he numbe o alse Posi i e is 22 and o alse Nega i e is 10.
31
Table 11
Con usion ma ix o sen imen e alua ion on weigh ed sen imen sco es
Weigh ed sen imen sco es
Nega i e (< 4)
Posi i e (>= 4)
Nega i e (manual)
170
5
Posi i e (manual)
7
32
Table 12
Con usion ma ix o sen imen e alua ion on use a ings
Use a ings
Nega i e (< 4)
Posi i e (>= 4)
Nega i e (manual)
155
20
Posi i e (manual)
2
37
Table 13
Con usion ma ix o sen imen e alua ion on no malized VADER compound sco es
No malized VADER compound sco es
Nega i e (< 4)
Posi i e (>=4)
Nega i e (manual)
153
22
Posi i e (manual)
10
29
The calcula ed P ecisions, Recalls and F1-sco es o weigh ed sen imen sco es, use a ings and
no malized VADER compound sco es a e epo ed in Table 14. Gene ally, he weigh ed sen imen sco es
ou pe o med he o he wo sco ing me hods, wi h an imp o emen o a leas 7% in F1-sco e. Al hough
he highes Recall (94.87%) was achie ed when conside ing only use a ings as he sen imen sco es,
he P ecision was abou 30% lowe han he Recall, esul ing in a educed F1-sco e. As o he
no malized VADER compound sco es, he P ecision, Recall and F1-sco e we e he lowes .
38
app/link opening in Oc obe 2020 and o opic 4 - app c ash in July, Sep embe and Oc obe 2020. As
o Teleg am, he lowes a e age sen imen sco e o opic 3 - upda e | ios e sion in Ap il 2021 and o
opic 11 - pho o/ ideo/link/message sending du ing he pe iod be ween Feb ua y o Ap il 2021 we e
eco ded. Also, he a e age sen imen sco es o opic 0 - message/no i ica ion ecei ing | ead eceip ,
opic 5 - ( ideo) call | connec ion quali y and opic 8 - s a us | sea ch con ac su e ed a mild downwa d
endency in luc ua ion since No embe 2020.
Figu e 13
Sen imen e olu ion by opic

39
Figu e 13 (Con inued)
40
Figu e 13 (Con inued)
41
Figu e 13 (Con inued)
42
Figu e 13 (Con inued)
43
Table 16
S a is ical agg ega ion o a e age sen imen sco es by mon h, opic and app
2020
2021
Topic
App
Jun
Jul
Aug
Sep
Oc
No
Dec
Jan
Feb
Ma
Ap
May
0 -
message/no i ica ion
ecei ing | ead eceip
Messenge
2.3638
2.2865
2.3089
2.2167
2.2289
1.9869
2.0386
2.0885
2.2776
2.1728
2.6446
2.0403
Wha sApp
2.8857
2.9078
4.2164
2.8947
2.8752
3.2145
3.3961
3.0906
2.7589
2.5521
3.2270
2.8640
Signal
3.2700
3.4304
3.6107
3.8720
3.4125
3.6586
3.1081
3.2882
2.5848
3.0972
Teleg am
2.8003
3.1956
3.3350
3.5797
3.4320
4.4391
3.5129
3.2835
3.0859
3.1130
3.6943
2.2172
1 - app/link opening
Messenge
2.2296
2.1492
2.2384
2.7128
2.5070
2.1520
2.3345
2.2665
1.8650
2.3562
2.1972
1.9824
Wha sApp
2.4307
2.4722
2.8606
2.9740
2.4108
2.3967
2.3376
2.5232
2.6880
2.3361
2.4067
2.6644
Signal
2.9305
1.6972
4.9884
1.6426
3.1510
3.4535
3.4291
2.5772
3.9005
2.1907
2.9478
Teleg am
2.5985
2.9528
3.6636
3.0872
2.1297
3.1088
2.3652
1.8427
2.6891
2.6182
3.1803
2.0932
2 - (g oup) cha | add
ea u e | p i acy
Messenge
3.0310
2.8480
2.7479
2.7368
2.5593
2.3157
2.4096
2.4122
2.6191
2.6502
2.6389
1.9545
Wha sApp
3.2234
3.9343
3.9550
3.8854
3.5372
3.6712
3.5475
2.4167
3.1884
3.4263
3.3581
2.5006
Signal
3.7442
3.3481
3.3824
3.8916
3.9520
3.8022
3.9567
4.0685
3.8967
3.7720
3.7532
4.2418
Teleg am
3.5699
3.2739
3.5987
3.6435
3.5134
3.0989
3.1997
3.9236
3.4021
3.3949
3.4053
3.0457
3 - upda e | ios e sion
Messenge
2.2068
2.4530
2.1444
2.4909
2.5920
2.1493
2.1288
2.2744
2.2140
2.3206
2.1211
2.3013
Wha sApp
2.4352
3.5214
3.3479
3.0506
2.7390
2.6399
2.8454
2.4424
2.3220
2.6074
2.6744
2.6440
Signal
3.2008
2.4639
3.9507
2.0728
2.9753
3.4823
4.0063
4.0979
1.8590
3.4727
1.7962
Teleg am
3.0714
3.4381
2.5976
2.9335
3.0537
3.4169
3.5884
3.6723
3.9401
2.5798
1.4411
3.4120
4 - app c ash
Messenge
2.3294
2.1243
2.1728
2.3938
2.2769
2.0010
2.0616
1.9573
2.1220
2.0324
2.6809
2.0153
Wha sApp
3.5229
2.7573
2.6936
3.0294
2.5036
2.4253
3.4909
2.7442
2.6264
2.0921
2.5431
2.6759
Signal
3.3318
1.8848
1.8947
1.6301
3.3765
2.4432
3.5193
4.7975
3.9046
3.3431
2.8587
Teleg am
1.9491
2.8785
3.9922
2.9617
2.0750
3.3103
3.6608
3.6032
1.9837
4.7679
1.8805
5 - ( ideo) call |
connec ion quali y
Messenge
2.5149
2.8097
2.6443
2.8622
2.4097
2.2581
2.5174
2.9462
2.3197
2.4827
2.7789
2.1944
Wha sApp
3.1105
3.3307
3.1759
3.1883
3.0272
3.2134
3.0063
3.0316
2.8560
2.8901
2.7690
2.7159
Signal
3.9512
2.4058
4.2402
3.8803
3.4331
2.9188
4.0029
3.8020
3.3223
2.8761
3.0087
3.5840
Teleg am
3.4898
4.0502
3.5206
2.9378
3.4180
3.6611
3.0830
3.0953
2.8802
3.0982
3.3780
2.0426
6 - app/ ea u e
wo king issue | apple
wa ch
Messenge
2.3616
2.5807
2.2183
2.4760
2.5502
2.1760
2.2896
2.4085
2.4399
2.2497
2.1487
1.9844
Wha sApp
2.9412
3.3373
3.1515
2.9493
2.7282
2.9534
2.9765
2.7456
3.2867
3.2279
3.2983
2.2039
Signal
3.6491
3.8341
3.0147
3.6614
3.3828
3.7963
3.7119
3.9542
4.3755
3.7583
3.6905
Teleg am
2.9731
3.1284
2.7934
2.8894
3.0254
3.6051
3.3794
3.7166
2.8662
2.3776
2.8964
2.5772
7 - social/ unc ional
usage
Messenge
2.4790
2.7226
2.2743
2.4416
2.6954
2.2755
2.1732
2.1569
2.2573
2.4060
2.3385
2.2482
Wha sApp
3.2299
3.6690
3.5868
3.6122
3.5031
3.5697
3.4347
2.4441
3.1933
3.2731
3.3438
2.7749
Signal
3.5961
3.2402
4.6192
3.7990
3.2291
4.1911
3.9825
4.0896
4.1760
4.2620
3.7053
3.4604
Teleg am
3.2884
3.8529
3.4983
3.3414
3.0613
2.4046
3.3098
3.6310
3.5288
3.6539
3.1035
2.9611
8 - s a us | sea ch
con ac
Messenge
2.9390
2.3639
2.4817
2.6576
2.6991
2.5556
2.2944
3.1056
2.3153
2.3309
2.8268
2.1508
Wha sApp
3.1580
3.0234
3.5846
4.0132
2.9351
2.7939
2.8955
2.5883
3.4636
2.8803
3.4870
2.7182
Signal
4.3933
2.3244
4.0403
3.2704
3.9059
4.5141
3.7817
Teleg am
2.8694
4.0243
4.5564
4.6439
4.6360
4.3597
3.9469
3.6535
3.8297
2.8916
3.2212
9 - gene al p oblems o
ix
Messenge
2.4882
2.3797
2.2799
2.2523
2.3556
2.2618
2.1195
2.3953
2.4116
2.6213
2.4298
2.6768
Wha sApp
2.5228
2.8119
2.8665
2.5707
2.6262
3.0854
2.7581
2.7761
2.7374
2.5640
2.4026
2.5425
Signal
2.6883
2.9739
3.0525
3.9976
3.1953
3.2859
2.9759
3.5683
4.1149
3.4264
3.2683
3.0846
Teleg am
2.9512
2.9466
3.6774
3.3128
2.6736
2.7513
2.3777
2.9252
2.2393
3.1167
2.7231
2.2709
10 - app/accoun
dele ion | download |
accoun /login
Messenge
2.2822
2.4381
2.2573
2.5419
2.1552
2.1790
2.1359
2.1345
2.1019
2.1207
2.1478
2.0040
Wha sApp
2.7814
2.8880
2.8320
2.6501
2.6622
2.7375
2.6472
2.1988
2.5763
2.6369
2.8155
2.0763
Signal
3.0253
2.5741
2.9547
2.9875
2.1958
2.8753
2.4677
2.8672
3.2691
3.4435
3.0024
3.4426
Teleg am
2.8000
2.7102
2.5806
2.7885
2.8444
2.5920
2.8882
2.5882
2.9681
2.7222
2.5535
2.6028
11 - pho o/ ideo/link/
message sending
Messenge
2.3336
2.3289
2.4697
2.3306
2.3327
2.1401
2.3234
2.2309
2.1760
2.2920
2.4548
2.1653
Wha sApp
2.6894
3.0979
3.2554
3.4077
3.0389
2.8928
3.3946
2.5569
2.7187
2.8015
3.0754
2.7571
Signal
3.6399
3.0006
4.1891
2.4700
3.0566
3.2743
3.9777
3.0949
3.2191
3.3313
3.0825
3.4782
Teleg am
2.6049
3.2188
2.9884
2.9781
3.0375
2.8413
2.1573
3.5336
1.8173
2.0580
2.1174
3.7092

44
4.2 DISCUSSION
By ex -mining 27,479 use e iews o ou messaging apps eleased on Apple S o e, he p esen wo k
combined opic modeling and sen imen analysis o pe o m compe i i e analysis. The esul s show ha
he opic model ex ac ed comp ehensible opics wi h p omising accu acies, and ha he sen imen
analysis pe o med well. Addi ionally, he compe i i e analysis based on he ex ac ed opics and he
weigh ed sen imen sco es e ealed meaning ul compe i i e in elligence in e ms o se e al ace s.
4.2.1 Topic modeling
In he p esen wo k, he accu acies anging om 69.97% o 100% in opic ex ac ion sugges he NMF
opic model wo ks well wi h use e iews o messaging apps. The ela i ely lowe accu acies migh
pa ially esul om a ai numbe o e iews ha exp ess p o es s agains a ecen a ack in Pales ine.
The language exp ession o poli ical discussion is likely o con use he opic decomposi ion o he model.
Mo eo e , we ollowed he app oach adop ed by Vu e al. (2015) o use nouns and e bs as app ea u es
and ob ained an imp o emen o 2.94% in o e all accu acy compa ed wi h hei a e age accu acy o
83.11% using keywo d-based app oach o analyze use e iews om Google Play. This imp o emen
migh due o a di e en app oach o mine he app aspec s o a mo e domain-speci ic ex p ep ocessing
o messaging apps. Ano he possible explana ion can be ha he exp ession pa e n o use e iews
om Google Play is di e en om ha om Apple S o e. Fu he mo e, Guzman and Maalej (2014) used
he LDA app oach o ex ac opics om colloca ions o nouns, e bs and adjec i es in use e iews o
mul iple non-compe i i e And oid and iOS apps including Wha sApp. Thei esul s show ha he
p ecisions and ecalls a ied on an app basis, and he highes p ecision and ecall we e achie ed o
Wha sApp, wi h F1-sco es o 0.781 and 0.813 espec i ely o inclusion and exclusion o sen imen
wo ds in ea u e wo ds. These F1-sco es o Wha sApp in hei esul , combining he o e all accu acy in
opic ex ac ion in ou esul , migh indica e ha opic models a e pa icula ly e ec i e o ex ac app
ea u es om use e iews o messaging apps bu no any ype o mobile apps. A possible explana ion
can be ha messaging apps usually ha e homogeneous unc ionali ies, and hus use s o such apps
migh end o exp ess a speci ic app aspec in simila ways. Also, hei exclusion o sen imen wo ds
p o ides a easible way o imp o e ou pe o mance o opic ex ac ion by p uning sen imen wo ds in
app ea u e e ms. In hei s udy, he sen imen wo ds a e usually adjec i es (e.g., g ea , bad) and e bs
(e.g., ha e, lo e), while in ou esea ch, ea u e e ms ha bea sen imen s a e mos ly e bs.
The e a e some impo an indings in he ex ac ed opics. Fi s ly, domain-speci ic knowledge was
shown in some o he opic names. The e m “app” is no in any o he opic wo ds in opic 1- app/link
opening, opic 4 - app c ash, opic 6 - app/ ea u e wo king issue | apple wa ch o opic 10 - app/accoun
dele ion | download | accoun /login, bu hei opic names s ill ela e o app due o he logical con ex
o hei opic wo ds. In ac , i was impossible o ha e he e m “app” in any lis o opic-wo ds since his
e m was expanded o “applica ion” and hen emo ed as a s op wo d due o i s ex emely high
equency in he co pus. Secondly, opic-wo ds such as “unins all” and “ eins all” appea ed in se e al
opics such as opic 1 - app/link opening, opic 4 - app c ash and opic 6 - app/ ea u e wo king issue |
apple wa ch. One possible explana ion can be he use beha io o unins alling and eins alling he app
o a emp ing o sol e an issue, e.g., app c ash, and he use s men ioned such beha io s along wi h
45
o he app ea u e issues in he e iews. Thi dly, a ew simila i ies in opic keywo ds we e ound be ween
some ex ac ed opics in he p esen wo k and some opics in Su e al.’s (2019) s udy. They applied he
LDA me hod o ind la en opics om use e iews o ou mobile apps on Google Play, and Messenge
was one o hose apps. Thei s udy lis s some ex ac ed opics wi h op i e keywo ds o Messenge . As
illus a ed in Table 17, some o hei op i e keywo ds can also be ound in he op i een keywo ds o
h ee ex ac ed opics in he p esen wo k. I is possible ha mo e opic-wo ds could be ma ched i hey
had shown mo e keywo ds o each opic, e.g., op i een keywo ds a he han only op i e keywo ds.
The app aspec s discussed in each pai o he h ee pai s o opics a e e y close based on he logical
connec ion o he keywo ds in each opic. One explana ion o his could be ha nea ly hal o he o al
use e iews o opic modeling a e o Messenge in he p esen wo k (Table 15). I is also likely ha iOS
use s and And oid use s end o discuss simila app aspec s o epo simila app issues o messaging
apps in use e iews. Las ly, some o he ex ac ed opics comp ise mul iple incohe en aspec s. Fo
ins ance, in opic 2, he h ee aspec s o (g oup) cha , add ea u e and p i acy could be h ee indi idual
opics. This opic incohe ence migh esul om a signi ican numbe o e iews men ioning he h ee
aspec s oge he . Ano he explana ion could be ha he numbe o opics k was de ined o be oo small,
and hus he opic model gene a ed excessi ely b oad opics. Simila p oblem o he numbe o k migh
also happen o opic 7 - social/ unc ional usage. This opic equi ed a comp ehensi e conside a ion o
o he wo ds wi h smalle weigh s and a combina ion wi h i s abs ac p ima y wo d “use” o
condensing he main issues discussed in ha opic. The wo ds such as “lo e”, “ iend”, “ amily” and
“communica e” sugges social aspec s, while he wo ds li ke “iphone”, “da um”, “phone” and
“messaging” indica e unc ional ea u es. Howe e , social and unc ional usage is an o e ly b oad opic,
which can e e o many app aspec s. O e all, he ex ac ed opics sugges ha de ining he numbe o k
is a majo challenge, bu he NMF opic model is s ill use ul o ex ac ing opics, mos o which e lec
clea and sepa able app aspec s o messaging apps.
Table 17
Compa ison o keywo ds in ex ac ed opics
The p esen wo k - NMF
Su e al. (2019) - LDA
Messenge , Wha sApp, Signal, Teleg am
Messenge
Topic id
Top 15 keywo ds
Topic id
Top 5 keywo ds
0
message, no i ica ion, show, ge ,
see, ecei e, ead, go, eques , say,
check, oice, ype, eply,
ma ke place
3
message, no i ica ion, dele e, gi e,
eques
2
cha , see, people, ea u e, add,
wan , lo e, op ion, g oup, need,
p i acy, make, iend, change, go
8
cha , change, ea u e, add, g oup
5
call, ideo, oice, phone, make,
quali y, connec , hea , ing, ecei e,
audio, d op, issue, play, sound
0
call, sc een, u n, oice, connec
46
4.2.2 Sen imen analysis
Th ough a compa ison o di e en sen imen sco ing me hods, we disco e ed he weigh ed sen imen
sco es mi iga ed he bias o use a ings and bene i ed om he no malized VADER compound sco es,
esul ing in a ela i ely objec i e sen imen sco ing o u he compe i i e analysis.
In iguingly, Su e al. (2019) conduc ed a simila sen imen analysis using a e age weigh ed sco es o
wo g oups o simila apps eleased on Google Play and ob ained on a e age a p ecision o 92.29%, a
ecall o 71.41% and a F1-sco e o 80.24%. Thei p ecision- ecall adeo is con a y o hose o use
a ings and no malized VADER compound sco es in ou esul , which ended o ecei e lowe p ecisions
and highe ecalls (Table 14). This migh due o he di e en sco e benchma ks o sepa a ing posi i e
and nega i e e iews, 2.5 in hei s udy bu 4 in he p esen wo k, p oducing a majo i y class o nega i e
in ou sen imen classi ica ion. Compa a i ely, a sco e benchma k o 4 seems mo e app op ia e o
sepa a ing he sen imen pola i y o use e iews o mobile apps. Ou esul shows ha he weigh ed
sen imen sco es based on his sco e benchma k ob ained a be e F1-sco e wi h a balanced p ecision-
ecall adeo .
Fu he mo e, in he sen imen analysis o he p esen wo k, we a e aged he weigh ed sen imen sco es
o use e iews by opic as he opic sen imen sco e, while in Su e al.’s (2019) s udy, he sen imen
sco e o each opic ound by he LDA model akes he numbe o use s’ hi ing he like bu on in o
conside a ion. Howe e , he e a e no only use s who click he like bu on bu also use s who click he
dislike bu on. I is possible ha he coun o use dislikes ou numbe s he coun o use likes o speci ic
e iews. Me ely conside ing he numbe o use likes migh bias he sen imen sco e o a opic. In spi e
o his possible bias, hei s udy s ill encou ages a mo e elabo a e sen imen analysis, which could ake
bo h he numbe o use likes and he numbe o use dislikes in o accoun when i comes o he
calcula ion o he sen imen sco e o a opic.
4.2.3 Compe i i e analysis
The compe i i e analysis e ealed meaning ul insigh s in o use conce ns, compe i i e s eng hs and
weaknesses as well as changes o use sen imen s o e ime. Acco ding o Po e (1980), he
componen s o compe i o analysis comp ise mul i-dimensional objec i es o he compe i o ’s
manage ial pe sonnel, assump ions o he compe i o i sel and o he indus y, compe i i e s a egy
and he compe i o ’s esou ces and capabili ies including s eng hs and weaknesses. The ein o, he
objec i es and assump ions e eal wha d i es he compe i o , while he s a egy, esou ces and
capabili ies e lec wha he compe i o is doing o is capable o doing. Based on hese componen s o
compe i o analysis, he use conce ns, compe i i e s eng hs and weaknesses as well as changes o use
sen imen s o e ime shown in he p esen wo k mee wha he compe i o is doing o is capable o
doing. Speci ically, use conce ns, on he one hand, e lec in a way he capabili ies o a compe ing
messaging app, and on he o he hand, use conce ns also p o ide legi ima e g ound o conjec u ing
he compe i i e s a egy o he i al. Also, compe i i e s eng hs and weaknesses as well as changes o
use sen imen s o e ime unco e he esou ces and capabili ies o a compe i o .
47
The e iew dis ibu ions by opic o he ou messaging apps e lec s he majo use conce ns abou
speci ic app aspec s. In gene al, app use s we e e y conce ned abou cha ea u es and p i acy issues.
Thei a en ion o cha ea u es con o ms o people’s pe cep ion mainly because cha ea u es a e he
co e unc ions o all messaging apps. P i acy issues should a ouse he a en ion o p ac i ione s in he
indus y o ins an messaging se ices. Da a secu i y and p i acy may become a key compe i i e
ad an age o di e en ia ing one messaging app om he o he s. Ano he majo use concen a ion is
he download and accoun issues anging om download, login p oblem and accoun dele ion. These
issues a e usually no ela ed o he key ea u es o apps bu migh be he gene ic p oblems o many
mobile apps, which bo he he app use s e y much. Sol ing hese p oblems by be e unc ional designs
and echnical suppo s would imp o e he use sa is ac ion. Fo example, a e downloading, use s
could choose mul iple ways o log in he app wi hou any login ailu e. Mo eo e , use s o Wha sApp,
Signal and Teleg am had simila ocuses on he app aspec s pa icula ly ega ding oice and ideo call,
connec ion quali y, social and unc ional usage, accoun and download issues. Aspec s such as oice call
and ideo call, jus like he cha ea u es, a e also he c ucial ea u es o apps o daily communica ion.
Thei use s migh do equen oice o ideo call apa om ex messages, and hus hey a ached g ea
impo ance o he quali y o calls. An excellen and s able connec ion quali y would be a big a ac ion
o use s who make lo s o oice calls o ideo calls. Also, hese h ee apps migh be e y impo an o
hei use s o keep in ouch wi h hei amily and iends. Du ing he social communica ion including ex
messages, oice calls and ideo calls, he unc ional usage, e.g., mobile da a usage, migh d aw he
special a en ion o app use s. I is likely ha a speci ic ea u e o mobile da a sa ing mode in he app
se ings can become a unique compe i i e ad an age in he e a o mobile In e ne . As o Messenge , in
addi ion o he cha ea u es, p i acy, download and accoun issues, he p oblems ega ding app and
link opening we e also men ioned by many use s. Ne e heless, his aspec was no highly discussed by
use s o Wha sApp, Signal and Teleg am. This con as migh sugges ha use s o Messenge su e ed
om mo e ailu e o incon enience in opening he app and he links in messages. Rega ding he app
opening, he p oblem migh be no eac ion a e clicking on he app icon. The p oblem o app and link
opening dese es he a en ion o hei esea ch and de elopmen pe sonnel, who migh u he
in es iga e he speci ic issues by e ie ing use e iews o ha opic. Ano he in e es ing inding is ha
use s o Messenge did no discuss he aspec o oice call, ideo call and connec ion quali y as much as
he use s o Wha sApp, Signal and Teleg am. One possible explana ion can be ha he use s o
Messenge did no encoun e ou s anding issues o equen ly exp ess hei emo ions abou his app
aspec . I could also be ha use s o Messenge mainly ex ed messages a he han made oice calls o
ideo calls, and hus hey did no pay much a en ion o his app aspec . In he la e si ua ion, he
easons can be u he examined. The oice and ideo call o Messenge migh be incon enien o use,
and he use migh need addi ional clicks o use his app ea u e. Las ly, aspec s o he messaging apps
such as gene al p oblems o ix, app and ea u e wo king issues as well as app c ash mainly pe ain o
echnical issues. O he app aspec s such as messaging sending and ecei ing, ead eceip , s a us,
con ac sea ching and upda e we e also discussed by use s o hese messaging apps. These app aspec s
migh ela e o no only echnical issues bu also design o he apps. The speci ic solu ions o hese
p oblems should be de e mined based on he speci ic issues men ioned in he use e iews. The app
aspec -based opics could p o ide he p ac i ione s o messaging apps wi h clea e di ec ions o use
conce ns and oubleshoo ing.
54
Géma , G., & Jiménez-Quin e o, JA. (2015). Tex mining social media o compe i i e analysis. Tou ism &
Managemen S udies, 11(1), 84–90. Re ie ed Ma ch 03, 2021 om
h ps://www. edalyc.o g/pd /3887/388743883010.pd
G eene, D., O’Callaghan, D., & Cunningham, P. (2014). How Many Topics? S abili y Analysis o Topic
Models. In Calde s, T., Esposi o, F., Hülle meie , E., & Meo, R. (eds), Machine Lea ning and
Knowledge Disco e y in Da abases. ECML PKDD 2014. Lec u e No es in Compu e Science, ol
8724. Sp inge , Be lin, Heidelbe g. h ps://doi.o g/10.1007/978-3-662-44848-9_32
Gu, X., & Kim, S. (2015). Wha Pa s o You Apps a e Lo ed by Use s? 2015 30 h IEEE/ACM In e na ional
Con e ence on Au oma ed So wa e Enginee ing (ASE), 760–770.
h ps://doi.o g/10.1109/ASE.2015.57
Guo, Y., Ba nes, S. J., & Jia, Q. (2017). Mining meaning om online a ings and e iews: Tou is
sa is ac ion analysis using la en di ichle alloca ion. Tou ism Managemen , 59, 467–483.
h ps://doi.o g/10.1016/j. ou man.2016.09.009
Guzman, E., & Maalej, W. (2014). How Do Use s Like This Fea u e? A Fine G ained Sen imen Analysis o
App Re iews. 2014 IEEE 22nd In e na ional Requi emen s Enginee ing Con e ence (RE), 153–162.
h ps://doi.o g/10.1109/RE.2014.6912257
He, W., Zha, S., & Li, L. (2013). Social media compe i i e analysis and ex mining: A case s udy in he
pizza indus y. In e na ional Jou nal o In o ma ion Managemen , 33(3), 464–472.
h ps://doi.o g/10.1016/j.ijin omg .2013.01.001
He, W., Tian, X., Chen, Y., & Chong, D. (2016). Ac ionable Social Media Compe i i e Analy ics Fo
Unde s anding Cus ome Expe iences. Jou nal o Compu e In o ma ion Sys ems, 56(2), 145–155.
h ps://doi.o g/10.1080/08874417.2016.1117377
Ho mann, T. (1999). P obabilis ic la en seman ic analysis. P oceedings o he Fi een h con e ence on
Unce ain y in a i icial in elligence (UAI1999), 289–296. Re ie ed Ma ch 05, 2021, om
h p://a xi .o g/abs/1301.6705
Hong, L., & Da ison, B. D. (2010). Empi ical s udy o opic modeling in Twi e . P oceedings o he Fi s
Wo kshop on Social Media Analy ics, 80–88. h ps://doi.o g/10.1145/1964858.1964870
Hu, M., & Liu, B. (2004). Mining and summa izing cus ome e iews. P oceedings o he en h ACM
SIGKDD in e na ional con e ence on Knowledge disco e y and da a mining (KDD '04), 168–177.
h ps://doi.o g/10.1145/1014052.1014073
Hu, F., & T i edi, R. H. (2020). Mapping ho el b and posi ioning and compe i i e landscapes by ex -
mining use -gene a ed con en . In e na ional Jou nal o Hospi ali y Managemen , 84.
h ps://doi.o g/10.1016/j.ijhm.2019.102317

55
Huebne , J., F ey, R. M., Ammendola, C., Fleisch, E., & Ilic, A. (2018). Wha People Like in Mobile Finance
Apps: An Analysis o Use Re iews. P oceedings o he 17 h In e na ional Con e ence on Mobile
and Ubiqui ous Mul imedia, 293–304. h ps://doi.o g/10.1145/3282894.3282895
Hu o, C., & Gilbe , E. (2014). VADER: A Pa simonious Rule-Based Model o Sen imen Analysis o Social
Media Tex . P oceedings o he In e na ional AAAI Con e ence on Web and Social Media, 8(1).
Re ie ed Ma ch 10, 2021, om h ps://ojs.aaai.o g/index.php/ICWSM/a icle/ iew/14550
Hype eali y. (n.d.). B i ish spellings. Re ie ed June 17, 2021, om
h ps://gi hub.com/hype eali y/Ame ican-B i ish-English-
T ansla o /blob/mas e /da a/b i ish_spellings.json
Iacob, C., & Ha ison, R. (2013). Re ie ing and analyzing mobile apps ea u e eques s om online
e iews. 2013 10 h Wo king Con e ence on Mining So wa e Reposi o ies (MSR), 41–44.
h ps://doi.o g/10.1109/MSR.2013.6624001
Johann, T., S anik, C., Alizadeh B., A. M., & Maalej, W. (2017). SAFE: A Simple App oach o Fea u e
Ex ac ion om App Desc ip ions and App Re iews. 2017 IEEE 25 h In e na ional Requi emen s
Enginee ing Con e ence (RE), 21–30. h ps://doi.o g/10.1109/RE.2017.71
Ka po ich, S. (9 No embe 2013). P obabilis ic la en seman ic analysis — PLSA. Wikimedia Commons.
h ps://commons.wikimedia.o g/wiki/File:Вероятностный_латентно-
семантический_анализ.png
Kelly, Ryan. (2011). PyEnchan . h p://pyenchan .gi hub.io/pyenchan /. Accessed June 17, 2021.
Kim, Y., & Jeong, S. R. (2018). Compe i i e in elligence in Ko ean Ramen Ma ke using Tex Mining and
Sen imen Analysis. Jou nal o In e ne Compu ing and Se ices, 19(1), 155–166.
h ps://doi.o g/10.7472/JKSII.2018.19.1.155
Koo en, P. V. (n.d.). Con ac ions. h ps://gi hub.com/koo enp /con ac ions. Accessed June 17, 2021.
La e y, J. D., McCallum, A., & Pe ei a, F.C.N. (2001). Condi ional andom ields: p obabilis ic models o
segmen ing and labeling sequence da a. P oceedings o he Eigh een h In e na ional Con e ence
on Machine Lea ning, 282–289. Re ie ed Augus 9, 2021, om
h ps:// eposi o y.upenn.edu/cgi/ iewcon en .cgi?a icle=1162&con ex =cis_pape s
Lee, D. D., & Seung, H. S. (1999). Lea ning he pa s o objec s by non-nega i e ma ix ac o iza ion.
Na u e, 401(6755), 788–791. h ps://doi.o g/10.1038/44565
Lee, JH., Pa k, S., Ahn, CM., & Kim, D. (2009). Au oma ic gene ic documen summa iza ion based on non-
nega i e ma ix ac o iza ion. In o ma ion P ocessing & Managemen , 45(1), 20–34.
h ps://doi.o g/10.1016/j.ipm.2008.06.002
56
Li, Y., Jia, B., Guo, Y., & Chen, X. (2017). Mining Use Re iews o Mobile App Compa isons. P oceedings
o he ACM on In e ac i e, Mobile, Wea able and Ubiqui ous Technologies, 1(3), 1–15.
h ps://doi.o g/10.1145/3130935
Liu, B. (2012). Sen imen Analysis and Opinion Mining. Syn hesis Lec u es on Human Language
Technologies, 5(1), 1–167. h ps://doi.o g/10.2200/s00416ed1 01y201204hl 016
Liu, L., Tang, L., Dong, W., Yao, S., & Zhou, W. (2016). An o e iew o opic modeling and i s cu en
applica ions in bioin o ma ics. Sp inge Plus, 5, 1608. h ps://doi.o g/10.1186/s40064-016-3252-8
Liu, Y., Jiang, C., & Zhao, H. (2019). Assessing p oduc compe i i e ad an ages om he pe spec i e o
cus ome s by mining use -gene a ed con en on social media. Decision Suppo Sys ems, 123.
h ps://doi.o g/10.1016/j.dss.2019.113079
Manning, C. D., Su deanu, M., Baue , J., Finkel, J., Be ha d, S. J., & McClosky, D. (2014). The S an o d
Co eNLP na u al language p ocessing oolki . P oceedings o 52nd Annual Mee ing o he
Associa ion o Compu a ional Linguis ics: Sys em Demons a ions, 55–60.
h ps://doi.o g/10.3115/ 1/P14-5010
Medha , W., Hassan, A., & Ko ashy, H. (2014). Sen imen analysis algo i hms and applica ions: A su ey.
Ain Shams Enginee ing Jou nal, 5(4), 1093–1113. h ps://doi.o g/10.1016/j.asej.2014.04.011
Mille , G. A. (1995). Wo dNe : a lexical da abase o English. Communica ions o he ACM, 38(11), 39–41.
h ps://doi.o g/10.1145/219717.219748
Nielsen, F. Å. (2011, Ma ch 15). A new ANEW: E alua ion o a wo d lis o sen imen analysis in
mic oblogs. Re ie ed Ma ch 15, 2021, om h ps://a xi .o g/abs/1103.2903
No man, G. J., No is, C. J., Gollan, J., I o, T. A., Hawkley, L. C., La sen, J. T., Cacioppo, J. T., & Be n son, G.
G. (2011). Cu en Emo ion Resea ch in Psychophysiology: The Neu obiology o E alua i e
Bi alence. Emo ion Re iew, 3(3), 349–359. h ps://doi.o g/10.1177/1754073911402403
O’Callaghan, D., G eene, D., Ca hy, J., & Cunningham, P. (2015). An analysis o he cohe ence o
desc ip o s in opic modeling. Expe Sys ems wi h Applica ions, 42(13), 5645–5657.
h ps://doi.o g/10.1016/j.eswa.2015.02.055
Osgood, C., Suci, G., & Tannenbaum, P. (1957). The measu emen o meaning. U bana, IL: Uni e si y o
Illinois.
Ouyang, Y., Guo, B., Lu, X., Han, Q., Guo, T., & Yu, Z. (2019). Compe i i eBike: Compe i i e Analysis and
Popula i y P edic ion o Bike-Sha ing Apps Using Mul i-Sou ce Da a. IEEE T ansac ions on Mobile
Compu ing, 18(8), 1760–1773. h ps://doi.o g/10.1109/TMC.2018.2868933
Pa k, D. H., Liu, M., Zhai, C., & Wang, H. (2015). Le e aging Use Re iews o Imp o e Accu acy o Mobile
App Re ie al. SIGIR '15: P oceedings o he 38 h In e na ional ACM SIGIR Con e ence on Resea ch
and De elopmen in In o ma ion Re ie al, 533–542. h ps://doi.o g/10.1145/2766462.2767759
57
Ped egosa, F., Va oquaux, G., G am o , A., Michel, V., Thi ion, B., G isel, O., Blondel, M., P e enho e ,
P., Weiss, R., Dubou g, V., Vande plas, J., Passos, A., Cou napeau, D., B uche , M., Pe o , M., &
Duchesnay, É. (2011). Sciki -lea n: Machine lea ning in py hon. Jou nal o Machine Lea ning
Resea ch, 12, 2825–2830.
Pennebake , J., Mehl, M., & Niede ho e , K. (2003). Psychological aspec s o na u al language use: Ou
wo ds, ou sel es. Annual Re iew o Psychology, 54(1), 547–577.
h ps://doi.o g/10.1146/annu e .psych.54.101601.145041
Po e , M. E. (1980). Compe i i e S a egy: Techniques o Analyzing Indus ies and Compe i o s. New
Yo k: F ee P ess.
Qwe yus. (21 Oc obe 2013). Illus a ion o app oxima e non-nega i e ma ix ac o iza ion (NMF),
Wikimedia Commons. h ps://commons.wikimedia.o g/wiki/File:NMF.png
San o ini, B. (July 1990). Pa -o -Speech Tagging Guidelines o he Penn T eebank P ojec (3 d Re ision).
Re ie ed May 08, 2021, om
h ps:// eposi o y.upenn.edu/cgi/ iewcon en .cgi?a icle=1603&con ex =cis_ epo s
Shah, F. A., Si s, K., & P ahl, D. (2019). Using app e iews o compe i i e analysis: ool suppo .
P oceedings o he 3 d ACM SIGSOFT In e na ional Wo kshop on App Ma ke Analy ics (WAMA
2019), 40–46. h ps://doi.o g/10.1145/3340496.3342756
Slxu.public. (29 Sep embe 2009). Pla e no a ion o he Smoo hed LDA Model. Wikimedia Commons.
h ps://commons.wikimedia.o g/wiki/File:Smoo hed_LDA.png
S one, P. J., Dunphy, D. C., Smi h, M. S., & Ogil ie, D. M. (1966). The gene al inqui e : a compu e
app oach o con en analysis. MIT P ess, Camb idge.
Su, Y., Wang, Y., & Yang, W. (2019). Mining and Compa ing Use Re iews ac oss Simila Mobile Apps.
2019 15 h In e na ional Con e ence on Mobile Ad-hoc and Senso Ne wo ks (MSN), 338–342.
h ps://doi.o g/10.1109/MSN48538.2019.00070
Sup ayogi, E., Budi, I., & Mahend a, R. (2018). In o ma ion Ex ac ion o Mobile Applica ion Use
Re iew. 2018 In e na ional Con e ence on Ad anced Compu e Science and In o ma ion Sys ems
(ICACSIS), 343–348. h ps://doi.o g/10.1109/ICACSIS.2018.8618164
Taboada, M., B ooke, J., To iloski, M., Voll, K., & S ede, M. (2011). Lexicon-Based Me hods o Sen imen
Analysis. Compu a ional Linguis ics, 37(2), 267–307. h ps://doi.o g/10.1162/coli_a_00049
Thelwall, M. (2013). Hea and Soul: Sen imen S eng h De ec ion in he Social Web wi h Sen iS eng h.
Re ie ed Ma ch 28, 2021, om
h p://sen is eng h.wl .ac.uk/documen a ion/Sen iS eng hChap e .pd
58
T uică, C. O., Velcin, J., & Boicea, A. (20 15). Au oma ic Language Iden i ica ion o Romance Languages
Using S op Wo ds and Diac i ics. 17 h In e na ional Symposium on Symbolic and Nume ic
Algo i hms o Scien i ic Compu ing (SYNASC), 243–246. h ps://doi.o g/10.1109/SYNASC.2015.45
T uică, C., Rădulescu, F., & Boicea, A. (2016). Compa ing Di e en Te m Weigh ing Schemas o Topic
Modeling. 2016 18 h In e na ional Symposium on Symbolic and Nume ic Algo i hms o Scien i ic
Compu ing (SYNASC), 307–310. h ps://doi.o g/10.1109/SYNASC.2016.055
Vu, P. M., Nguyen, T. T., Pham, H. V., & Nguyen, T. T. (2015). Mining Use Opinions in Mobile App
Re iews: A Keywo d-Based App oach (T). 2015 30 h IEEE/ACM In e na ional Con e ence on
Au oma ed So wa e Enginee ing (ASE), 749–759. h ps://doi.o g/10.1109/ASE.2015.85
Wang, W., Wang, H., & Song, Y. (2016). Ranking p oduc aspec s h ough sen imen analysis o online
e iews. Jou nal o Expe imen al & Theo e ical A i icial In elligence, 29(2), 227–246.
h ps://doi.o g/10.1080/0952813X.2015.1132270
Wang, W., Feng, Y., & Dai, W. (2018). Topic analysis o online e iews o wo compe i i e p oduc s using
la en di ichle alloca ion. Elec onic Comme ce Resea ch and Applica ions, 29, 142–156,
h ps://doi.o g/10.1016/j.ele ap.2018.04.003
Whi elaw, C., Ga g, N., & A gamon, S. (2005). Using app aisal g oups o sen imen analysis. P oceedings
o he 14 h ACM In e na ional Con e ence on In o ma ion and Knowledge Managemen - CIKM
'05. h ps://doi.o g/10.1145/1099554.1099714
Wilson, T., Wiebe, J., & Ho mann, P. (2005). Recognizing con ex ual pola i y in ph ase-le el sen imen
analysis. P oceedings o Human Language Technology Con e ence and Con e ence on Empi ical
Me hods in Na u al Language P ocessing, 347–354. h ps://doi.o g/10.3115/1220575.1220619
Xu, K., Liao, S. S., Li, J., & Song, Y. (2011). Mining compa a i e opinions om cus ome e iews o
Compe i i e In elligence. Decision Suppo Sys ems, 50(4), 743–754.
h ps://doi.o g/10.1016/j.dss.2010.08.021
59
Appendix A
Sampled e iews wi h w ong opics assigned by he NMF opic model
Re iew
Topic
I ’s s ill ell you ha you go mail when you don’
10 - app/accoun dele ion | download | accoun /login
My game is ozen ka y60 help
2 - (g oup) cha | add ea u e | p i acy
I go my s a ues saying showing ha I’m online when I had i
change o o line bu s ill shows me being ac i e online I don’ like
ha I’m appea ing o line o a eason o no be con ac
10 - app/accoun dele ion | download | accoun /login
I can’ unde s and is ha a censo ship o jus s op esponding
pe iod???
6 - app/ ea u e wo king issue | apple wa ch
The pas wo weeks i ’s been kicking me ou and aking me igh
back in o my home sc een on my phone Facebook has no
esponded o my epo o his happening I’m missing my
messages!
10 - app/accoun dele ion | download | accoun /login
I ’s now going o be my hi d ome ins alling his app and i keeps
elling me ha I don’ ha e In e ne connec ion when I wan o
login please help as o wha I should do ?
4 - app c ash
Yeah e y disgus ed Messenge igh now wouldn’ le me in o my
Messenge so I had o unins all i now I can’ eins all i keeps
asking o paymen in o ma ion which I don’ no wha i has o do
wi h i
4 - app c ash
No longe ha e edi bu on op ion o upda e p o ile pic in
messenge . When I sea ch online o help, sugges ions indica e o
go h ough FB app. I do no use FB he e o e do no ha e ha
app.
8 - s a us | sea ch con ac
Well I ied upda ing my messenge bu i ’s been an hou and my
in e ne is awesome I jus go i ixed and i ’s s ill loading idk i i ’s
jus my phone o wha bu u don’ hink i is jus saying
10 - app/accoun dele ion | download | accoun /login
Why does Facec ook need ano he app on you phone o
any hing. So Ma k can ge pa id. T ha ’s why ., oh and he can collec
in o ma ion abou you.
10 - app/accoun dele ion | download | accoun /login
Hi I logged in o messenge wi h my Ins ag am accoun and I can’
ge logged in o i . Plz help me. You guys should ha e old people
who use messenge wi h a Ins ag am accoun and ell hem we
can’ use he Ins ag am login accoun anymo e o a leas you
guys should gi e me my accoun o some hing
7 - social/ unc ional usage
All acebook messenge use
Please be awa e
A menian employe s o Facebook used acebook o ban
Aze baijani use s.
They a e aking ad an age o wo king o Facebook o make
censo ship.
I s in ole an and shoud be s opped
6 - app/ ea u e wo king issue | apple wa ch
Since wha e e upda e came h ough a ew weeks ago, Messenge
will no longe le me open links. Pe iod. Doesn’ ma e i hey’ e
ex e nal o link back o FB.
One s a un il you guys ix his c ap. Mul i-billion dolla company,
and you app is ash.
9 - gene al p oblems o ix

60
Appendix A (Con inued)
I ha e used messenge o yea s, and ou o no whe e I no longe
ecei e no i ica ions. I ’s in u ia ing as his is my main messaging
app. I ha e ied e e y hing om oggling se ings o ese ing my
phone. No hing wo ks.
6 - app/ ea u e wo king issue | apple wa ch
ee Pales ine des oy Is ael.
ee Pales ine des oy Is ael.
ee Pales ine des oy Is ael.
2 - (g oup) cha | add ea u e | p i acy
Ve y bad Bad because o his acis policy agains he Pales inians
2 - (g oup) cha | add ea u e | p i acy
Facebook suppo e hnic cleansing agains mino i ies wo ldwide
by banning people, shu ing, dele ing, limi ing audience o pos s
abou opp essed people in Pales ine and o he a eas in he wo ld.
2 - (g oup) cha | add ea u e | p i acy
#GazaUnde A ack
#Pales ineUnde A ack
#Sa e_Sheikh_Ja ah
2 - (g oup) cha | add ea u e | p i acy
Because his is acebook’s hen I gi e 1 s a and also ee P ales ine
2 - (g oup) cha | add ea u e | p i acy
The de elope s p e end o p omo e p i acy, bu sell sensi i e da a
abou you calls and ex s o co po a ions. This is no shocking, as
he app is owned by acebook.
5 - ( ideo) call | connec ion quali y
I am no longe ge ing no i ica ions and sound when I ecei e ex .
I’ e ese o ac o y on phone and wha app and s ill no hing.
10 - app/accoun dele ion | download | accoun /login
Me ey phone mein chalna hai o wa na achi a ha chal nahi o esa
sabaq sikhaonga k munh dikhane k laik nai ahoge. Ye me i p i acy
hai au me i p i acy mein koi bhi en ess nhi ka na. Hukaayyy!!
2 - (g oup) cha | add ea u e | p i acy
Te ible applica ion. I sends you in o ma ion o companies you
don’ wan o o ha e.
11 - pho o/ ideo/link/message sending
Wha sApp says i is end o end enc yp ed and no one can ead o
hea no e en Wha sApp.
Well no ue. I had someone hack my Wha sApp and ge ahold o
all my messages. So sad ha you a e ad e ising his. I ’s no ue
people clea you messages .
0 - message/no i ica ion ecei ing | ead eceip
I al eady ge ing scammed and ha assed on i
10 - app/accoun dele ion | download | accoun /login
#سدقلا_ضفتنت
#عيبطتلا_هنايخ
#
ح_خيشلا_ حارج
#لا_ديوهتل_ سدقلا
#اوذقنأ_
ح_خيشلا_ حارج
#نل_لحرن
# اوذقنا_
ح_خيشلا_ حارج
#sa esheikhja ah
#ŞeyhJa ahmahallesinku a ın
#Sal ailqua ie ediSheikhJa ah
#Re edasVie elSheikhJa ah
#sau ezlequa ie desheikhja ah
10 - app/accoun dele ion | download | accoun /login
#gaza
#sa e_pales ine
2 - (g oup) cha | add ea u e | p i acy
acking my phone
10 - app/accoun dele ion | download | accoun /login
None compa e
2 - (g oup) cha | add ea u e | p i acy
Aw ul app. I can’ deac i a e i . And hey don’ wan you oo. S ay
away om his app. I ied o deac i a e i . Impossible. Impossible
o wo k, oo.
6 - app/ ea u e wo king issue | apple wa ch
61
Appendix B
Sampled e iews wi h inconsis en labels o sen imen pola i y on weigh ed sen imen sco es
Re iew
au oAssigned_label
Manual_label
Rema k
T ying o cancel a i de ha was o di ed inco ec ly, please
help me cancel he o de I ha e ied e e y hing. O de is
om Linda icke s in amoun o $29 . Please help
Posi i e
Nega i e
Recen ly my messages won’ go h ough, i ’ll jus say sending
bu will ake oughly 10-20 mins o inally send. My in e ne
wo ks pe ec ly ine and so does my da a so I hink he app
may need ano he ix/upda e asap!
Posi i e
Nega i e
G ea app. Since he upda e he Blue oo h no longe wo ks
wi h my esla speake s when ideo is on ( ine- - ideo
p obably shouldn be on anyways). Fix i please
Posi i e
Nega i e
Posi i e a i s ,
hen shi o
app p oblems
I need his app o calling
Nega i e
Posi i e
Suppo anima ed s icke s
Nega i e
Posi i e
Ad ice
This is an app ha doesn disappoin . lo e i !
Nega i e
Posi i e
Ve y help ul o communica ion ac oss he wo ld o jus
iPhone o and oid when elying on WiFi in dead zones
Nega i e
Posi i e
i connec s me wi h my a o i e people. hank you. Lolinche
Nega i e
Posi i e
I’ll s a by saying he app is p e y awesome. I only ha e a
couple hings ha bug me pe sis en ly. The i s is a lack o
imes amps. The ecen messages ead “10 hou s ago” o
example. I’d like o see an ac ual ime so I don’ ha e o do
ma h o he messages sen / ecei ed wi hin 24 hou s. Also,
messages show as sen and ecei ed on my end, bu he
ecipien ac ually doesn’ ge hem o hou s. And some imes
he same hing happens wi h me on he ecei ing end.
Posi i e
Nega i e
Posi i e a i s ,
hen shi o
app p oblems
Lo e he app. Use i daily. Access o pho os no a ailable in
IOS 14. Goes o “ ecen ” olde o ind pho o album. No
pic u es in his olde .
Posi i e
Nega i e
Posi i e a i s ,
hen shi o
app p oblems
I swi ched om WeCha and Wha sApp o Signal.
Nega i e
Posi i e
Could be be e i you guys gi e op ions o s a us o his o y.
Also he p o ile pic u e op ion needs o be imp o ed.
Nega i e
Posi i e
Ad ice
62