Text Mining Techniques for Car Price Prediction
Abstract
Modern data sources routinely contain information both in unstructured and structured forms, combining text with the usual numerical and categorical data. For instance, in websites dedicated for selling and buying cars the listings typically include a textual description of the car. Others also include a detailed list of numerical or categorical attributes, such as the total number of kilometers the car has, or it´s model. In this work project we apply text mining techniques to create predictors for car price regression from unstructured data, the textual description in car listings. Two different types of predictors were studied, the tf-idf features obtained from the n-gram count matrix, or the singular vectors derived from the decomposition of the tf-idf matrix. In this work we also examine the performance of reducing the vocabulary dimension by applying stemming, lemmatization or not applying either of those. We also compare the effects of creating the initial n-gram count matrix with only unigrams, unigrams and bigrams or only bigrams. Our regression experiment shows that Support Vector Regression performs best at car price prediction using text data as predictors with R2 = 0.77, MSE = 0.19 and MAE = 0.32. These results can be seen as respectable given the complex nature of the task.
Full text
i
s
Tex Mining Techniques o Ca P ice P edic ion
Rica do Miguel Gal ão Gonçal es
P ojec Wo k p esen ed as pa ial equi emen o ob aining
he Mas e ’s deg ee in Ad anced Analy ics
ii
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
TEXT MINING TECHNIQUES FOR CAR PRICE PREDICTION
by
Rica do Miguel Gal ão Gonçal es
P ojec Wo k p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Ad anced
Analy ics
Ad iso / Co Ad iso : Robe o And é Pe ei a Hen iques, Phd
Sep embe 2021
iii
ABSTRACT
Mode n da a sou ces ou inely con ain in o ma ion bo h in uns uc u ed and s uc u ed o ms,
combining ex wi h he usual nume ical and ca ego ical da a. Fo ins ance, in websi es dedica ed o
selling and buying ca s he lis ings ypically include a ex ual desc ip ion o he ca . O he s also include
a de ailed lis o nume ical o ca ego ical a ibu es, such as he o al numbe o kilome e s he ca
has, o i ´s model.
In his wo k p ojec we apply ex mining echniques o c ea e p edic o s o ca p ice eg ession om
uns uc u ed da a, he ex ual desc ip ion in ca lis ings. Two di e en ypes o p edic o s we e
s udied, he -id ea u es ob ained om he n-g am coun ma ix, o he singula ec o s de i ed om
he decomposi ion o he -id ma ix.
In his wo k we also examine he pe o mance o educing he ocabula y dimension by applying
s emming, lemma iza ion o no applying ei he o hose. We also compa e he e ec s o c ea ing he
ini ial n-g am coun ma ix wi h only unig ams, unig ams and big ams o only big ams.
Ou eg ession expe imen shows ha Suppo Vec o Reg ession pe o ms bes a ca p ice p edic ion
using ex da a as p edic o s wi h R2 = 0.77, MSE = 0.19 and MAE = 0.32. These esul s can be seen as
espec able gi en he complex na u e o he ask.
KEYWORDS
Tex Mining; Reg ession Analysis; Ca P ice P edic ion.
i
INDEX
1. In oduc ion .................................................................................................................. 1
2. Li e a u e e iew .......................................................................................................... 3
2.1. Tex Mining ............................................................................................................ 3
2.1.1. Tex P ep ocessing ......................................................................................... 4
2.1.2. Fea u e Rep esen a ion ................................................................................. 5
2.2. Reg ession ............................................................................................................. 6
2.2.1. Linea Reg ession ........................................................................................... 6
2.2.2. Nonlinea Reg ession ..................................................................................... 7
2.3. P e ious wo k on ex eg ession .......................................................................... 8
3. Me hodology .............................................................................................................. 16
3.1. Da a Collec ion .................................................................................................... 17
3.2. Da a P epa a ion ................................................................................................. 17
3.2.1. Wo d Co ec ion and Vocabula y Building ................................................... 18
3.3. Da a Explo a ion .................................................................................................. 20
3.3.1. Desc ip ion ex da a .................................................................................... 20
3.3.2. Ta ge a iable (p ice) dis ibu ion .............................................................. 22
3.3.3. P ice in ela ion o ad wo d coun ................................................................ 24
3.3.4. P ice in ela ion o numbe o badly w i en wo ds..................................... 25
3.4. Fea u e Ex ac ion ............................................................................................... 26
3.4.1. Dimensionali y Reduc ion ............................................................................ 26
3.5. Modelling ............................................................................................................. 27
3.5.1. Pe o mance Indica o s ................................................................................ 28
4. Resul s and discussion ................................................................................................ 29
4.1. Linea Reg ession Resul s .................................................................................... 29
4.1.1. Linea Reg ession Va ian s Resul s............................................................... 31
4.2. Suppo Vec o Reg ession Resul s ..................................................................... 32
4.2.1. Nu Value hype pa ame e op imiza ion ...................................................... 33
4.3. Decision T ee Reg esso and G adien Boos ing Reg esso Resul s ................... 35
4.3.1. G adien Boos ing Reg esso hype pa ame e op imiza ion ....................... 36
4.4. Mul i-Laye Pe cep on Resul s ........................................................................... 37
4.4.1. Mul i-Laye Pe cep on Reg esso Hype pa ame e op imiza ion .............. 38
4.5. Resul s Summa y ................................................................................................. 40
5. Conclusions ................................................................................................................. 42
6. Limi a ions and ecommenda ions o u u e wo ks ................................................. 44
7. Bibliog aphy ................................................................................................................ 45
8. Appendix ..................................................................................................................... 48
8.1. Da a Collec ion Sc ip .......................................................................................... 48
8.2. Resul s Tables ...................................................................................................... 51
8.2.1. Linea Reg ession ......................................................................................... 51
8.2.2. Suppo Vec o Reg ession Resul s .............................................................. 54
8.2.3. Decision T ee Reg esso and G adien Boos ing Reg esso esul s ............. 60
8.2.4. Mul i-Laye Pe cep on esul s .................................................................... 63
i
LIST OF FIGURES
Figu e 1- No malized P o i o each candida e in di e en sys ems in (Le man e al. 2008) .. 9
Figu e 2- Ou line o he CNN in (Bi ai and Cohn 2015) .......................................................... 13
Figu e 3 – P ojec Flowcha .................................................................................................... 16
Figu e 4- Ad wo d coun his og am ......................................................................................... 21
Figu e 5 - Wo dcloud wi h only unig ams ................................................................................ 22
Figu e 6 – Wo dcloud wi h big ams ......................................................................................... 22
Figu e 7- Log P ice dis ibu ion ................................................................................................ 23
Figu e 8 – E o s in p ice lis ings .............................................................................................. 24
Figu e 9- Log P ice in ela ion o ad wo d coun ...................................................................... 24
Figu e 10- Log P ice in ela ion o numbe o badly w i en wo ds ......................................... 25
Figu e 11- Pe o mance compa ison be ween Linea Reg ession on aw coun s s on -id
ea u es. (No s emming o lemma iza ion used, unig ams only, 2000 mos equen
wo ds used as p edic o s) ................................................................................................ 30
Figu e 12 – In luence o a ying nu alue on NuSV o MSE .................................................. 34
Figu e 13 – Inl uence o a ying nu ale on NuSVR o MAE ................................................... 34
Figu e 14 – Inl uence o a ying nu alue on NuSVR o R2 ..................................................... 35
Figu e 15 – G adien Boos ing Reg esso pe o mance when a ying he numbe o es ima o s
.......................................................................................................................................... 36
Figu e 16 – G adien Boos ing Reg esso pe o mance when a ying he max dep h o he
indi idual eg ession ees ............................................................................................... 37
ii
LIST OF TABLES
Table 1- Lis o keywo ds ob ained in (Guo e al. 2020) .......................................................... 10
Table 2- Pe o mance o he di e en models used in (Guo e al. 2020) ................................. 11
Table 3- Concep ual models de eloped in (Ngo-Ye and Sinha 2014) ...................................... 14
Table 4 – Ad leng h s a is ics .................................................................................................... 20
Table 5- P ice S a is ics ............................................................................................................. 23
Table 6 – Linea Reg ession esul s on he 2000 mos equen unig ams o basic, s emming
and lemma iza ion con igu a ions ................................................................................... 29
Table 7- Linea Reg ession esul s using he op 500 singula ec o s e ained om he unig am
coun ma ix ..................................................................................................................... 30
Table 8 - Linea Reg ession esul s using he op 500 singula ec o s e ained om he -id
sco es ma ix o he unig am & big am coun ma ix ...................................................... 31
Table 9 – Resul s o he di e en ypes o Linea Reg ession, using he -id sco es o he
2000 mos equen unig ams as p edic o s .................................................................... 32
Table 10 – Resul s o he di e en ypes o Linea Reg ession, using he op 500 singula
ec o s o he -id sco e ma ix as p edic o s................................................................ 32
Table 11 – Bes Pe o ming Suppo Vec o Reg ession Con igu a ions ................................. 33
Table 12 – Bes pe o ming con igu a ions o MLPReg esso ................................................ 38
Table 13 – Compa ing a ious hidden laye sizes con igu a ions ............................................ 39
Table 14 – Compa ing he di e en ac i a ion unc ions ........................................................ 39
Table 15 – Compa ing sol e unc ions .................................................................................... 39
Table 16 – Compa ing Lea ning Ra es ...................................................................................... 39
Table 17 – Compa ing di e en ini ial lea ning a e alues..................................................... 40
Table 18 – Compa ing di e en alpha alues .......................................................................... 40
Table 19 – Compa ing he pe o mance on di e en numbe s o maximum i e a ions ......... 40
Table 20 – Applied me hods and hei espec i e bes esul s ............................................... 41
Table 21 – Linea Reg ession esul s o 1000 mos equen unig ams (unig ams & big ams)
.......................................................................................................................................... 51
Table 22 – Linea Reg ession esul s o he 3000 mos equen unig ams (unig ams &
big ams) ............................................................................................................................ 51
Table 23 - Linea Reg ession esul s o he op 1000 -id sco es conside ing only unig ams
(unig ams & big ams) ....................................................................................................... 51
Table 24 - Linea Reg ession esul s o he op 2000 -id sco es conside ing only unig ams
(unig ams & big ams) ....................................................................................................... 52
iii
Table 25 - Linea Reg ession esul s o he op 3000 -id sco es conside ing only unig ams
(unig ams & big ams) ....................................................................................................... 52
Table 26 – Linea Reg ession esul s o he op 100 singula ec o s e ained om he wo d
coun ma ix using only unig ams (unig ams & big ams) ................................................ 52
Table 27 - Linea Reg ession esul s o he op 200 singula ec o s e ained om he wo d
coun ma ix using only unig ams (unig ams & big ams) ................................................ 52
Table 28 - Linea Reg ession esul s o he op 300 singula ec o s e ained om he wo d
coun ma ix using only unig ams (unig ams & big ams) ................................................ 53
Table 29 - Linea Reg ession esul s o he op 400 singula ec o s e ained om he wo d
coun ma ix using only unig ams (unig ams & big ams) ................................................ 53
Table 30 - Linea Reg ession esul s o he op 500 singula ec o s e ained om he wo d
coun ma ix using only unig ams (unig ams & big ams) ................................................ 53
Table 31 – Linea Reg ession esul s o he op 500 singula ec o s e ained om he -id
sco e ma ix using only unig ams (unig ams & big ams)................................................. 53
Table 32 – Compa ing he di e en e ypes o Linea Reg ession on he op 2000 -id sco es
conside ing only unig ams (unig ams & big ams) all wi h basic p e-p ocessing............. 54
Table 33 – Compa ing he di e en e ypes o Linea Reg ession on he op 500 singula ec o s
e ained om he -id sco e ma ix conside ing only unig ams (unig ams & big ams) all
wi h basic p e-p ocessing ................................................................................................. 54
Table 34 - Compa ing di e en ypes o suppo ec o eg ession using he op 2000 -id
ea u es, unig ams only (unig ams & big ams) ................................................................ 54
Table 35 - Compa ing di e en ypes o suppo ec o eg ession using he op 3000 -id
ea u es, unig ams only (unig ams & big ams) ................................................................ 55
Table 36 - Compa ing di e en ypes o suppo ec o eg ession using he op 2000 -id
ea u es, unig ams only (unig ams & big ams) ................................................................ 55
Table 37 - Compa ing di e en ypes o suppo ec o eg ession using he op 3000 -id
ea u es, unig ams only (unig ams & big ams) ................................................................ 55
Table 38 - Compa ing di e en ypes o suppo ec o eg ession using he op 2000 -id
ea u es, unig ams only (unig ams & big ams) ................................................................ 56
Table 39 - Compa ing di e en ypes o suppo ec o eg ession using he op 3000 -id
ea u es, unig ams only (unig ams & big ams) ................................................................ 56
Table 40 – Compa ing di e en ypes o suppo ec o eg ession using he op 100 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams) ..... 56
Table 41 - Compa ing di e en ypes o suppo ec o eg ession using he op 200 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams) ..... 56
ix
Table 42 - Compa ing di e en ypes o suppo ec o eg ession using he op 300 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams) ..... 57
Table 43 - Compa ing di e en ypes o suppo ec o eg ession using he op 400 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams) ..... 57
Table 44 - Compa ing di e en ypes o suppo ec o eg ession using he op 500 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams) ..... 57
Table 45 - Compa ing di e en ypes o suppo ec o eg ession using he op 100 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams),
s emming p e-p ocessing ................................................................................................. 57
Table 46 - Compa ing di e en ypes o suppo ec o eg ession using he op 200 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams),
s emming p e-p ocessing ................................................................................................. 58
Table 47 - Compa ing di e en ypes o suppo ec o eg ession using he op 300 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams),
s emming p e-p ocessing ................................................................................................. 58
Table 48 - Compa ing di e en ypes o suppo ec o eg ession using he op 400 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams),
s emming p e-p ocessing ................................................................................................. 58
Table 49 - Compa ing di e en ypes o suppo ec o eg ession using he op 500 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams),
s emming p e-p ocessing ................................................................................................. 58
Table 50 - Compa ing di e en ypes o suppo ec o eg ession using he op 100 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams),
lemma iza ion p e-p ocessing ......................................................................................... 59
Table 51 - Compa ing di e en ypes o suppo ec o eg ession using he op 200 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams),
lemma iza ion p e-p ocessing ......................................................................................... 59
Table 52 - Compa ing di e en ypes o suppo ec o eg ession using he op 300 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams),
lemma iza ion p e-p ocessing ......................................................................................... 59
Table 53 - Compa ing di e en ypes o suppo ec o eg ession using he op 400 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams),
lemma iza ion p e-p ocessing ......................................................................................... 60
Table 54 - Compa ing di e en ypes o suppo ec o eg ession using he op 500 singula
ec o s e ained om he -id sco e ma ix, unig ams only (unig ams & big ams),
lemma iza ion p e-p ocessing ......................................................................................... 60
4
1. Rep esen aw ex D as a nume ical a ay C;
2. Map C o p edic ed alues 𝑽
o unknown ou comes V; and
3. Use 𝑽
in subsequen desc ip i e o causal analysis.
In his o e iew, some me hods o p edic ing a nume ical a ibu e 𝒗𝒊 om coun s 𝒄𝒊, which is called
ex eg ession, a e p esen ed. This will be he ocus o his wo k.
2.1.1. Tex P ep ocessing
Tex ual da a is no s uc u ed as nea ly as he mo e common ypes o da a used in Machine Lea ning,
making i a much mo e complex ask o ind use ul in o ma ion and pa e ns in he da a in an
au oma ed way. The mos impo an way i di e s om o he kinds o da a is ha ex in inhe en ly
high dimensional (Gen zkow e al. 2019).
This means ha he p ep ocessing s ep is c ucial o he pe o mance a he ask a hand. Usually, he
i s s ep in he p ep ocessing ask is he okeniza ion o he ex . Tokeniza ion consis s in spli ing a
documen in o a s eam o wo ds by emo ing all punc ua ion ma ks and by eplacing abs and o he
non- ex cha ac e s by single whi e spaces (Ho ho e al. 2005). In (Kogan e al. 2009) lowe casing was
also used, a common s ep in okeniza ion. (Fos e , Libe man, and S ine 2013) also eplaced a e wo ds
by an in a ian “unknown” oken.
In o de o educe he dimensionali y o he documen s in he co pus, he se o wo ds/ okens
desc ibing hose documen s can be educed by il e ing and lemma iza ion o s emming me hods.
Fil e ing is done on documen s o emo e some o he wo ds. A common il e ing is s op-wo ds
emo al. S op-wo ds a e wo ds ha equen ly appea in he ex wi hou ha ing much con en
in o ma ion (Allahya i e al. 2017), like conjunc ions o p eposi ions. Lemma iza ion me hods y o
map e b o ms o he in ini e ense and nouns o he singula o m. Howe e , in o de o achie e his,
he pa o speech o e e y wo d in he ex documen has o be assigned (Ho ho e al. 2005).
S emming me hods y o build he basic o m o wo ds, i.e. s ip he plu al ‘s’ om nouns, he ‘ing’
om e bs o o he a ixes. A s em is a na u al g oup o wo ds wi h equal meaning (Ho ho e al. 2005).
E en ough he p ep ocessing s age may ha e no iceable in luence on he success o he ask a hand,
he ype o p ep ocessing done ha p oduces he bes esul s is p oblem dependen . Fo ins ance, a
s udy by (Pak and Gunal 2017) on he impac o ex ep esen a ion and p ep ocessing on Tu kish
au ho iden i ica ion wi h wo di e en classi ica ion algo i hms, Mul inomial Naï e Bayes (MNB) and
Sequen ial Minimal Op imiza ion (SMO) , ound ha disabling s op-wo d emo al and s emming
achie ed be e pe o mance han o he combina ions o MNB. Fo SMO enabling jus s emming
achie ed he highes F-Sco e.
5
2.1.2. Fea u e Rep esen a ion
In he li e a u e, he mos common way uns uc u ed ex is ep esen ed is by a ec o space model.
In VSM, he alues o he elemen s a e de i ed om e en equencies, such as he numbe o imes a
ce ain wo d appea s in a pa icula documen (Ngo-Ye and Sinha 2014).
The Bag-o -wo ds (BOW) model is a speci ic ype o ec o space model. In BOW he o de o he wo ds
is igno ed, 𝒄𝒊 is a ec o whose leng h is equal o he numbe o wo ds in he ocabula y and whose
elemen s 𝒄𝒊𝒋 a e he numbe o imes wo d j appea s in documen i. This scheme can be ex ended o
encode a limi ed amoun o dependence by coun ing unique ph ases a he han unique wo ds. A
ph ase leng h o n is e e ed o as an n-g am (Gen zkow e al. 2019).
Ano he s anda d way o e m weigh ing is he Te m F equency – In e se Documen F equency (TF-
IDF) sco e. This imp o es on no mal Te m F equency weigh ing because i dec eases he weigh o
e ms occu ing mo e equen ly in he documen collec ion, making su e he ma ching o documen s
is mo e a ec ed by dis inc i e wo ds which ha e ela i ely low equencies in he collec ion (Allahya i
e al. 2017).
The TF-IDF sco e is calcula ed as ollows: 𝑡𝑓 ×𝑖𝑑𝑓, whe e 𝑡𝑓 is he e m equency o wo d j in
documen i and in e se documen equency (𝑖𝑑𝑓) is he log o one o e he sha e o documen s
con aining j : log(n/dj), whe e n is he o al numbe o documen s (Gen zkow e al. 2019).
Nassi oussi e al. (2014), in hei e iew o ex mining o ma ke p edic ion, concluded ha he e
a e, a leas , 5 ypes o sco es commonly used o ep esen ing ea u es as nume ic alues: In o ma ion
Gain, Chi-Squa e s a is ics, Documen F equency, Accu acy Balanced and TF-IDF.
Fos e e al. (2013) p oposed a new algo i hm o con e ing ex in o nume ical eg esso s. The
me hod consis s o 3 s eps:
1. Con e he sou ce ex in o lis s o wo d ypes. A wo d ype is a unique sequence o non-blank
cha ac e s. Wo d ypes a e no dis inguished by meaning o use.
2. Compu e ma ices ha (a) coun he o imes ha wo d ypes appea wi hin each documen
and (b) coun he numbe o imes ha wo d ypes a e ound adjacen o each o he .
3. Compu e unca ed singula alue decomposi ions o he esul ing ma ices o coun s. The
leading singula ec o s o hese decomposi ions a e he eg esso s.
Besides he no mal ea u e ep esen a ions (Kogan e al. 2009) also used a sco e called LOG1P:
log1+𝑓𝑟𝑒𝑞(𝑥𝑗;𝑑) , a he han no malizing wo d like TF, his sco e dampens hem wi h a
loga i hm.
Liang e al. (2017) expand on he con en ional me hods o ex ea u e ex ac ion and ou line some
equen ly used deep lea ning me hods o ex ea u e ex ac ion. The i s me hod desc ibed is he
au oencode , which is a eed o wa d ne wo k ha can lea n a comp essed, dis ibu ed ep esen a ion
o da a, usually wi h he goal o dimensionali y educ ion o mani old lea ning. An au oencode usually
con ains one hidden laye be ween he inpu and he ou pu laye . Also desc ibed is a s acked
au oencode , he deep coun e pa o he au oencode . I can be buil simply by s acking laye s. Fo
e e y laye , i s inpu is he lea ned ep esen a ion o he o me laye .
6
In hei expe imen s i was compa ed he e ec o h ee ypes o ea u e ex ac ion me hods –
p incipal componen analysis, a shallow spa se au oencode and a deep spa se au oencode – o
pa e n ecogni ion. The p oposed me hod o a deep spa se au oencode enabled highe ecogni ion
accu acy, p o ing ha he use o deep lea ning o ex mining can make signi ican achie emen s.
2.2. REGRESSION
P edic ing ex s o a ce ain class can be highly use ul and i can sepa a e la ge collec ions o ex in o
ele an g oups. Howe e , in some cases i can be mo e sui able o p edic a speci ic alue and no a
global ange. This ype o p edic ion can be done using eg ession.
2.2.1. Linea Reg ession
The basic eg ession model is he linea eg ession. The aim o linea eg ession is o s udy he e ec
o one o mo e ac o s on a quan i a i e a iable a ge . The heo e ical model is: 𝑦= 𝛽 + 𝛽𝑥 +
𝛽𝑥 +⋯+𝛽𝑥 +𝑢, whe e 𝑦 is he a ge a iable obse ed o indi idual i, 𝛽 is he pa ame e
associa ed wi h p edic o a iable k, 𝑥 is he p edic o a iable k o obse ed indi idual i and 𝑢 is
he e o o indi idual i (Lehmann 2020). Usually he unknown pa ame e s in linea eg ession a e
lea ned by minimizing he sum o squa ed e o s (Nassi oussi e al. 2014).
The mos popula s a egy o high-dimensional eg ession in con empo a y s a is ics and machine
lea ning is he es ima ion o penalized linea models, pa icula ly wi h L1 penaliza ion. Fo simple ex
eg ession asks wi h inpu dimension on he same o de as he sample size, penalized linea models
ypically pe o m close o he on ie in e ms o ou -o -sample p edic ion (Gen zkow e al. 2019). The
mos common cos unc ions a e Lasso (L1 penal y), Ridge (L2 penal y) and Elas ic ne (mix o L1 and
L2).
Lasso is a penal y eg ession wi h a quad a ic loss unc ion ha in oduces a penal y e m associa ed
wi h SSR (Guo e al. 2020). The cos unc ion o Lasso eg ession can be w i en as:
∑(𝑦− 𝑦
)=∑𝑦− ∑𝑤∗
𝑥+ 𝜆∑𝑤
. This ype o egula iza ion can lead o
ze o coe icien s, i.e. some o he ea u es a e comple ely neglec ed o he e alua ion o he ou pu
(Cas elli e al. 2020)
Ridge eg ession is ano he penal y eg ession wi h a quad a ic egula ize ha inse s ano he penal y
e m in o he o iginal SSR e m. The cos unc ion is: ∑(𝑦− 𝑦
)=∑𝑦− ∑𝑤∗
𝑥+
𝜆∑𝑤
and i minimizes he sum o squa ed esiduals and he he 𝜆* he slope^2 (Cas elli e al.
2020). The pa ame e 𝜆 can ange om 0 o posi i e in ini y, inc easing i will p omo e a smalle slope.
Ridge eg ession can sol e o pa ame e s when he e isn´ enough da a samples.
Elas ic ne is a mix u e o bo h L1 and L2 penaliza ion whe e he penal y e m, he one ha is mul iplied
by he pa ame e 𝜆, is gi en by: ∑
(1−𝛼)𝐵+𝛼𝐵
whe e 𝛼 𝜖 (0,1) de e mines he ade-o
be ween L1 and L2 egula iza ion (Joshi e al. 2010).
7
2.2.2. Nonlinea Reg ession
Howe e , linea models a e limi ing: ex eg ession p oblems will o en in ol e complex in e ac ions
be ween ex ual inpu s, hus equi ing a non-linea app oach o p ope ly cap u e such phenomena
(Bi ai and Cohn 2015).
2.2.2.1. Reg ession T ees
Reg ession ees ha e become one o he mos popula app oaches o inco po a ing mul i way
p edic o in e ac ions in o eg ession. A ee “g ows” by sequen ially so ing da a obse a ions in o
bins based on he alue o he p edic o a iables. This pa i ions he da a se in o ec angula egions,
and o ms p edic ions as he a e age alue o he ou come a iable wi hin each pa i ion. This
s uc u e is an e ec i e way o accommoda e ich in e ac ions and nonlinea dependencies (Gen zkow
e al. 2019). Two ex ensions o he simple eg ession ee ha e been highly success ul hanks o cle e
egula iza ion app oaches ha minimize he need o uning and a oid o e i ing: Random Fo es s
and Boos ed T ees.
Random Fo es s, de eloped by B eiman, a e a combina ion o ee p edic o s such ha each ee
depends on he alues o a andom ec o sampled independen ly and wi h he same dis ibu ion as
all ees in he o es (B eiman, 2001). The algo i hm can be desc ibed as ollows:
- F om he aining se c ea e boo s ap samples
- Fo he spli o each node, in each o he base lea ne s (Decision T ees), x a ibu es a e
andomly chosen ou o X (wi h x <= X)
- Then an a ibu e ou o x is chosen o spli he node
- Each ee is g own o he la ges ex en possible
- Calcula e he pe o mance om ou -o -bag obse a ions
Boos ed eg ession ees combine he s eng hs o eg ession ees and boos ing, an adap i e me hod
o imp o ing model accu acy based on he idea ha i is easie o ind and a e age many ough ules
o humb, han o ind a single, highly accu a e p edic ion. These ype o ees inco po a e impo an
ad an ages o ee-based me hods, handling di e en ypes o p edic o a iables and accommoda ing
missing da a (Eli h, Lea hwick, and Has ie 2008).
2.2.2.2. Suppo Vec o Reg ession
Al hough suppo ec o machines a e mainly used o classi ica ion asks, hey a e also used o
eg ession p oblems. The suppo ec o eg ession model is ained by sol ing he ollowing
op imiza ion p oblem (Kogan e al. 2009): min
∈
|𝑤|+
∑max(0,|𝑣−𝑓(𝑑,𝑤)|−𝜀)
, whe e
C is a egula iza ion cons an , 𝜀 con ols he aining e o . The aining algo i hm inds o weigh s w
ha de ine a pa ame e ized unc ion o he documen s d and hose weigh s o op imize he alue o a
con inuous a iable .
8
2.2.2.3. Deep Lea ning
Va ious o he machine lea ning echniques ha e been applied o ex eg ession. The mos common
one in he deep lea ning a ea a e neu al ne wo ks, which ypically allow he inpu s o ac on he
esponse h ough one o mo e laye s o in e ac ing nonlinea basis unc ions. A main a ac ion o
neu al ne wo ks is hei s a us as uni e sal app oxima o s, a heo e ical esul desc ibing hei abili y
o mimic gene al, smoo h nonlinea associa ions (Gen zkow e al. 2019).
2.3. PREVIOUS WORK ON TEXT REGRESSION
The majo i y o wo k in he li e a u e ega ding ex eg ession is ela ed o s ock ma ke p edic ion
using ex ual da a om news websi es o inancial epo s. (Nassi oussi e al. 2014) p o ide an
ex ensi e o e iew o he li e a u e ela ed o his subjec .
(Kogan e al. 2009) apply well known eg ession echniques o a la ge co pus o eely a ailable
inancial epo s, cons uc ing eg ession models o ola ili y, which is he inancial isk o in es ing in
ha company, o he pe iod ollowing a epo . Fo hose p edic ions, he models a e ained using a
suppo ec o eg ession and he pe o mance was epo ed using he mean squa ed e o be ween
he p edic ed and he ue log- ola ili ies. In hei esul s i ´s epo ed ha he models ha used only
ex o p edic ola ili y came e y close o he his o ical baseline in some yea s. A ex only model
(LOG1P wi h big ams) comes wi hin 5% o he e o o a s ong baseline. A combined model, wi h ex
and his o ical da a, imp o es subs an ially o e he baseline in ou ou o six yea s (2003-2006) and
his di e ence is obus o he ep esen a ion used.
(Le man e al. 2008) use compu a ional linguis ics o au oma ically p edic he impac o news on
public pe cep ion. This wo k uses he 2004 US P esiden ial elec ion ma ke s om Iowa Elec onic
Ma ke s and he goal o he p edic ion sys em is o o ecas he p ice p edic ion o he nex day (up
o down). The sys em ope a es in an i e a i e ashion, on each day he news o ha day a e used o
cons uc a new ins ance. A logis ic eg ession is ained on all p e ious days and he esul ing classi ie
p edic s he p ice mo emen o he new ins ance.
As a as ea u e ep esen a ion hey used BOW; news ocus ea u es, which ep esen s di e ences
be ween days o news co e age, he esul ing alue cap u es change is ocus on day , whe e a alue
g ea e han 0 means inc eased ocus and a alue less han 0 dec eased ocus; en i y ea u es, which
a e conjunc ions o a ce ain wo d and he en i y, de ined a p io i; and dependency ea u es, which
we e ex ac ed om dependency pa ses o he news a icles, o cap u e dependency in e ac ions.
In each ma ke , he baseline news sys em makes a small p o i , bu he o e all pe o mance o he
combined sys em is wo se han he ma ke his o y sys em alone, showing ha he news baseline is
ine ec i e. Howe e , all news ea u es imp o e o e he ma ke his o y sys em which means ha he
news in o ma ion helps o explain ma ke beha io .
9
(Jin e al. 2013) p esen a sys em which mines news a icles and makes o ecas s abou he mo emen
o o eign exchange cu ency ma ke s. The sys em uses a combina ion o language models, opic
clus e ing and sen imen analysis o iden i y ele an news a icles. These a icles along wi h he
his o ical s ock index and cu ency exchange alues a e used in a linea eg ession model o make
o ecas s.
A language model was de eloped ha classi ies he incoming news a icles as ele an o no ele an .
Using he La en Di ichle Alloca ion model hey classi y hese news a icles in o 30 opics and ob ain
each a icle’s opic dis ibu ion. Then, he op opics a e iden i ied by manually aligning news a icles
wi h cu ency luc ua ions, which a e hen iden i ied as ele an opics. In o de o classi y incoming
news a icles, hey es ima e he opic dis ibu ion o each a icle and hen decide whe he i s mos
p e alen opics all in o he se o ele an opics iden i ied ea lie .
Acco ding o he au ho s he sys em was able o o ecas mos o he s udied app ecia ions and
dep ecia ions. Fo ins ance, he yea 2012 saw he B azilian Real’s exchange a e signi ican ly al e ed
due o go e nmen in e en ions. On May 21s , he sys em p edic ed co ec ly ha he BRL would
con inue dep ecia ing as pe ends om he p e ious weeks. Howe e , on May 22nd, as he B azilian
go e nmen was in e ening o e e se Real’s all, he sys em was able o co ec ly o ecas he
e e sal o he cu ency mo emen .
Ano he opic o ex eg ession app oached in he li e a u e is house p ice p edic ion. (Fos e e al.
2013) con e ex , ob ained om house desc ip ions in eal es a e lis ings, and con e i in o
nume ical eg esso s by exploi ing me hods om compu a ional linguis ics. The ea u es a e buil
using he algo i hm p e iously explained and o modelling he esponse, which in his case is he log
o he lis ed p ices because he e was some skewness in he p ope y p ices, hey ain a simple linea
eg ession model.
A baseline model was p o ided o compa ison by simply eg essing y on he wo d coun s in W, he
ma ix o coun s, o he mos common 2000 wo ds. This model p oduces a i o 𝑅
= 0,681. The nex
model uses eg esso s c ea ed om he SVD o he ma ix W. A eg ession o log p ices on he 100
leading singula ec o s a ains 𝑅
=0,49. Adding mo e singula ec o s p oduces s a is ically
signi ican , hough diminishing imp o emen s. Wi h 500 singula ec o s i p oduces 𝑅
=0,61.
Mos o he main leading singula ec o s a e signi ican wi h an inc easing p opo ion o insigni ican
a iables as he posi ion in he decomposi ion inc eases. Acco ding o he au ho s hese singula alues
Figu
e
1
-
No malized P o i o each c
andida e in di e en sys ems in (Le man
e al. 2008)
10
a e mo e consis en ly p edic i e wi h less noise in compa ison o he signi icance o he coe icien s
o he wo d indica o s. A hi d eg ession uses ea u es de i ed om he SVD o he big am ma ix B,
showing a sligh ly be e pe o mance han he p e ious.
This s udy is concluded by sugges ing ha he in oduc ion o nonlinea i y in hese models could
imp o e he esul s, also a b ie analysis on he use o ig ams shows ha hey o e modes gains,
albei a a non i ial inc ease in compu a ion.
(Guo e al. 2020) s udy he pe o mance o some machine lea ning algo i hms associa ed wi h ex
mining om in e ne da a in p edic ing house p ices in China. To sea ch o all possible ep esen a ions
o housing p ices and consequen ly, ob ain all possible keywo ds ha a e di ec ly and indi ec ly ela ed
o housing p ices h ough wo channels: one is ex ac ed om Baidu, a Chinese sea ch engine, o
display he p e e ences o o dina y In e ne use s – o example, buye s and selle s o eal es a e; while
ano he is ob ained by c awling he CNKI, he la ges Chinese ull- ex da abase, co e ing academic
jou nals in o de o show he concep o housing p ices om he iewpoin o schola s and expe s.
Th ough his ex mining p ocess, hey ob ained 29 keywo ds which we e classi ied in o 4 g oups based
on economic iewpoin s: Mac o-policies, local a ibu es, housing ma ke cha ac e is ics and housing
cos s.
Economic aspec s Keywo ds
Mac o policies (
7
)
U baniza ion, ail anspo a ion, eal es a e policy,
pension und, mac o con ol, mone a y policy,
in la ion
Local a ibu
es (
6
)
Shanghai`s second-hand house, house web, house,
house p ice, en al house and school dis ic house
Housing ma ke cha ac e is ics (
9
)
Second-hand house, second-hand web, housing
p ice, housing enzies, ising p ices, p ice/income
a io, house, en house web
Housing cos s (
7
)
Housing ee, housing ax, mo gage calcula o ,
mo gage in e es , down paymen , p ope y ax,
deco a ion
Tabl
e
1
-
Lis o keywo ds ob ained i
n (Guo e al.
2020)
11
The machine lea ning models used we e Gene alized Linea Reg ession, Elas ic Ne Reg ession and
Random Fo es s. The pe o mance o he models a e as ollows:
The au ho s conclude ha his me hod, especially andom o es no only de ec s u ning poin s, bu
also o e s p edic ion abili y ha clea ly ou pe o ms adi ional eg ession analysis.
Ano he ex eg ession ask ound in he li e a u e is p edic ing mo ie e enues h ough c i ic e iews.
(Joshi e al. 2010) used he ex o ilm c i ics e iews om se e al sou ces o p edic opening weekend
e enue. Thei da a was ga he ed o mo ies eleased in 2005-2009. Fo hose mo ies, me ada a was
ob ained and a lis o hype links o mo ie e iews by c awling Me aC i ic. The me ada a e ie ed
includes he name o he mo ie, i s p oduc ion house, he se o gen es i belongs o, he
sc ip w i e (s), he p ima y ac o s s a ing, unning ime and i s MPAA a ing. The e iews we e
sc aped om he se en mos equen websi es on Me aC i ic.
Two esponse a iables we e conside ed, he o al e enue on opening weekend and he pe sc een
e enue. Bo h p edic ions we e e alua ed using mean absolu e e o and Pea son´s co ela ion
be ween he ac ual and p edic ed alue. A penalized linea eg ession, he elas ic ne model, was used
o p edic he esponse a iables. As a as ex ea u es, he au ho s used n-g ams o leng h 1,2 and
3, pa -o -speech n-g ams, ob ained h ough he S an o d POS agge and las ly dependency ela ions.
In hese expe imen s h ee ypes o p edic o s we e compa ed, p edic o s based on me ada a, which
was used as a baseline, p edic o s based on ex and p edic o s ha use bo h kinds o in o ma ion.
They epo ed ha ea u es om c i ic’s e iews by hemsel es imp o e co ela ion on bo h esponse
a iables, howe e imp o emen in MAE is only obse ed o he pe sc een e enue p edic ion ask.
A combina ion o he me a and ex ea u es achie es he bes pe o mance in e ms o MAE and
Pea son’s co ela ion. While he ex only models ha e some high nega i e weigh ea u es, he
combined models do no ha e any nega i ely weigh ed ea u es and only e y ew me ada a ea u es,
which leads o he conclusion ha ex ea u es om p e- elease e iews can subs i u e o and
imp o e o e a s ong me ada a-based opening weekend e enue p edic ion.
Model
Index
Gene alized
Reg ession
Random Fo es Elas ic Ne
MSE 0.1021 0.0190 0.122
R
2
0.91 0.98 0.90
Tabl
e
2
-
Pe o mance o he di e en models us
ed in (Guo
e al.
2020)
12
(Mishne and Glance 2006) s udied whe he applying sen imen analysis me hods o weblog da a
esul s in be e co ela ion han olume o discussion only. They analyzed he sen imen exp essed in
weblogs owa ds mo ies bo h be o e he mo ie’s elease and a e , and es whe he his sen imen
co ela es wi h he mo ie’s box o ice in o ma ion be e han a simple coun o he numbe o
e e ences in weblog does.
The au ho s show ha , in he domain o mo ies, he e is good co ela ion be ween e e ences o
mo ies in weblog pos s – bo h be o e and a e hei elease—and he mo ie’s inancial success.
Fu he mo e, hey demons a e ha shallow usage o sen imen analysis can imp o e his co ela ion.
Speci ically, he numbe o posi i e e e ences co ela es be e han aw coun s in he p e- elease
pe iod. In o i sel , he co ela ion be ween p e- elease sen imen and sales is no high enough o
sugges building a p edic i e model o sales based on sen imen alone. Howe e , i is p oposed ha
sen imen migh be e ec i ely used in p edic i e models o sales in conjunc ion wi h addi ional
ac o s, such as mo ie gen es and season.
E en ough he majo i y o ex eg ession p oblems ha e been ackled using linea models o machine
lea ning algo i hms, (Bi ai and Cohn 2015) p opose a nonlinea me hod based on a deep
con olu ional neu al ne wo k o p edic he u u e box-o ice akings o mo ies based on e iews by
mo ie c i ics and a ibu es.
The model ope a es o e unig ams, big ams and ig ams. They use wo d embeddings o ep esen
wo ds in a low dimensional space, a con olu ional ne wo k wi h max-pooling o ep esen documen s
in e ms o n-g ams, and se e al ully connec ed hidden laye s o allow o lea ning o complex
nonlinea in e ac ions. Including nonlinea i ies is c ucial o accu a e modelling. A me hod o
quan i ying he e ec o ex n-g ams on he p edic ion ou pu is also shown. This allows o
iden i ica ion o he mos impo an ex ual inpu s and in es iga ion o nonlinea in e ac ions be ween
hese wo ds and ph ases in di e en da a ins ances.
13
The ne wo k is ained wi h s ochas ic g adien descen and he Ada Del a upda e ule using andom
es a s. S ochas ic g adien descen is noisie han ba ch aining due o a local es ima ion o he
g adien , bu i can s a con e ging much as e . Ada Del a keeps and exponen ially decaying his o y
o g adien s and upda es in o de o adap he lea ning a e o each pa ame e . Regula iza ion and
hype pa ame e uning we e pe o med by ea ly s opping on he de elopmen se . The ou line o he
con olu ional ne wo k is shown:
The esul s show ha he neu al ne wo k pe o ms e y well, wi h a ound 40% imp o emen o e he
p e ious bes (Joshi e al. 2010). Nonlinea i ies a e clea ly help ul as e idence by he ANN ex model
bea ing he BOW linea ex model wi h a mean absolu e es e o o 6.0 s 8.0.
(Ngo-Ye and Sinha 2014) de elop and compa e se e al ex eg ession models o p edic ing he
help ulness o online e iews. The au ho s adop a aw numbe o posi i e help ul o es o a e iew as
he measu e o help ulness, he a ge a iable. A new hyb id model was p oposed, which inco po a es
Figu e
2
-
Ou line o he CNN i
n (Bi ai and Cohn
2015)
20
lis o he bes “good enough” ma ches acco ding o a ce ain cu o o he simila i y sco e (Anon n.d.).
The simila i y sco e be ween wo s ings is calcula ed as ollows: 2*M/T, whe e T is he o al numbe
o elemen s in bo h sequences, and M is he numbe o ma ches.
A e a manual analysis on he lis i p oduces, he inco ec wo ds ha di e om he co ec spelling
by 2 edi s o mo e a e added o ha dic iona y. Along wi h wo d co ec ion, he dic iona y was also
used o s anda dize some wo ds ha could ha e mul iple spellings o only one, i.e.: “quilome os”,
”kilome os”, ”km s”,e c. we e all changed o “km”. I would also sepa a e wo ds inco ec ly joined
oge he (whe he h ough inpu e o o he p ocess o p ep ocessing).
A e he p ep ocessing s eps desc ibed p e iously, he ini ial da a ame con aining 54573 ows was
educed o 18403 ows.
3.3. DATA EXPLORATION
In o de o ge a be e pe cep ion and unde s anding abou he da a in his p ojec , a b ie explo a o y
analysis was made. A isual comp ehension o he da a helps build essen ial domain knowledge be o e
he modelling applica ion. I also helps in inding inconsis encies in he da a and imp o ing he
p ocessing ask.
3.3.1. Desc ip ion ex da a
Fo a i s analysis, we checked he desc ip ion ex leng h dis ibu ion. In able 4 some s a is ics a e
p esen ed ela ed o he desc ip ion ex leng h ( “coun ” column). As we can see, he wo d coun
dis ibu ion seems o be somewha skewed o he igh gi en ha he median is smalle han he mean,
also he e is a big disc epancy be ween he hi d qua ile alue and he max alue o 876.
TOTAL NUMBER OF ADS
18043
MEAN AD
LENGTH
53.8148
STANDARD DEVIATION
47.3378
MINIMUM AD LENGTH
15
1
ST
QUARTILE
24
MEDIAN AD LENGTH
38
3
RD
QUARTILE
65
MAXIMUM AD LENGTH
876
Table 4 – Ad leng h s a is ics
21
Analyzing igu e 4, an his og am whose bins co espond o in e als o leng h 10, i s we can con i m
ha he ad leng h dis ibu ion is clea ly skewed o he igh . Addi ionally, we obse e ha he mos
common desc ip ion ex leng h is be ween 20 and 29 wi h a o al o 4015 ows ou o he o al 18043
obse a ions.
As a as ocabula y size, he cu en ocabula y a e he p e iously desc ibed p ep ocessing s eps
con ains a o al o 18440 di e en wo ds. This is a qui e la ge ocabula y size, howe e he e a e a
high numbe o ex emely low equency e ms. Fo ins ance, he e a e a o al o 6274 di e en wo ds
ha only appea once in all he co po a, and a o al o 8673 wo ds ha ha e a equency o 2 o less.
Fil e ing ou hese low equency e ms will educe he size o he ocabula y o almos hal and i can
also help imp o e he pe o mance o he ask a hand. He e he decision was o il e ou wo ds ha
ha e a equency o 1, wi h his he ocabula y con ains a o al o 12166 unique wo ds.
Ano he way o educe ocabula y size is he use o s emming and lemma iza ion. Applying s emming
and emo ing wo ds wi h equency 1 educes he ocabula y o 8476 unique wo ds. The same
p ocedu e bu wi h lemma iza ion educes i o 10451 wo ds.
Two wo dclouds we e also c ea ed o ge a isual ep esen a ion o he mo e equen wo ds in he
ocabula y. The wo ds wi h a highe equency a e he ones wi h a bigge size in he wo dcloud. Figu e
5 only conside s unig ams opposed o igu e 6 ha also conside s big ams in i s ep esen a ion i hey
ha e a high enough equency.
Figu e
4
-
Ad wo d coun his og am
22
As we can see, he mos emphasized wo ds in igu e 5 a e “km”, “es ado”, “ca o” and “no o”, ollowed
by “ e isao” and “ele ico”. When big ams a e conside ed in equency analysis we see ha some
big ams a e e y p ominen in he ocabula y, gi en ha big ams like “bom es ado”, “a
condicionado”, “c uise con ol” and “ id o ele ico” a e ep esen ed in igu e 6.
3.3.2. Ta ge a iable (p ice) dis ibu ion
The nex analysis was ela ed o he a ge a iable dis ibu ion. In he ollowing able we can see ha
he p ice da a is e y skewed o he igh . 75% o he obse a ions ha e a p ice o 13500€ o less,
howe e , he e is an ex eme max alue o 3333333€.
Figu e
5
-
Wo dcloud wi h only unig ams
Figu e
6
–
Wo dcloud wi h big ams
23
TOTAL NUMBER OF ADS
18043
MEAN PRICE
11047.9
€
STANDARD DEVIATION
31742.2
€
MINIMUM PRICE
1000
€
1
ST
QUARTILE
3500
€
MEDIAN PRICE
7499
€
3
RD
QUARTILE
13500
€
MAXIMUM PRICE
3333333
€
Table 5- P ice S a is ics
To ge mo e insigh on he dis ibu ion o he p ice a iable a boxplo g aph was also plo ed wi h a log
ans o ma ion applied o he a ge a iable because a dis ibu ion ha is symme ic o nea ly so is
o en easie o handle and in e p e han a skewed dis ibu ion. The na u al log ans o ma ion can
also imp o e he eg ession models p edic ion capabili ies and will be la e explo ed.
In igu e 7 i is possible o obse e ha almos all ca s a e being sold unde 100 000€, which is he
uppe ence o he boxplo (e^11.5). A e ha h eshold, he da a s a s o become spa se , and hose
Figu e
7
-
Log P ice dis ibu ion
24
alues may be iden i ied as ou lie s. To unde s and i he e a e any inconsis encies wi h hose alues,
we looked a all he ows in he da a ame ela ed o a p ice o 100 000€ o mo e.
A e checking hose obse a ions some e o s we e iden i ied which can be conside ed ou lie s and
elimina ed om he da ase , such as a e age ca s being lis ed a inc edibly high p ices, p obably due
o human e o .
In igu e 8 we see an example o ca s ha a e no mally sold a a much lowe p ice, being lis ed a a
e y high p ice. One o hose (“Mi subishi ou lande phe ”) is e en lis ed a he max p ice in he whole
da ase . The logical conclusion is ha hese inconsis encies we e due o human e o in lis ing he p ice
and he bes p ocedu e is o emo e hem om he da ase .
3.3.3. P ice in ela ion o ad wo d coun
The nex analysis made was o unde s and i he numbe o wo ds used in he desc ip ion ex , he
column “ ex ” in he da a ame, was co ela ed wi h he ca p ice. In o de wo ds i he ad leng h was
a good p edic o o he ca p ice, because i is no mal o suppose ha he highe p iced ca s would
ha e a mo e de ailed ex desc ip ion.
Figu e
8
–
E o s in p ice lis ings
Figu e
9
-
Log P ice in ela ion o ad wo d coun
25
Figu e 9 shows he leng h o he desc ip ion ex plo ed agains he log p ice. As we can see, he e
seems o be no di ec co ela ion be ween he numbe o wo ds used and he p ice o he ca . The
sca e plo doesn´ show an upwa d end along he x-axis ( he numbe o wo ds) which leads o he
conclusion ha he desc ip ion leng h isn´ a good p edic o o he ca p ice.
3.3.4. P ice in ela ion o numbe o badly w i en wo ds
Gi en ha he desc ip ion ex s we e human c ea ed and so, p one o con aining e o s, ano he
in e es ing analysis would be o check i he numbe o badly w i en wo ds in he desc ip ion ex
co ela e wi h he ca p ice. In his case, i would be a nega i e co ela ion, he highe he numbe o
badly w i en wo ds he lowe he ca p ice.
Fo his analysis, a new column was c ea ed, “w ong_wo d_coun ”, and he p e-p ocessing s eps ha
in ol ed wo d co ec ion we e no applied o he ini ial da a ame ( he use o he wo d_co ec o
dic iona y and he wo d co ec ion o he spell checke objec ), howe e all he o he p e-p ocessing
s eps we e aken. A unc ion was c ea ed ha would use he spell checke objec , wi h all he ex a
ocabula y added p e iously, and e e y ime i would iden i y a wo d as being poo ly w i en a +1 was
added o a coun e . The inal alue o he coun e would he alue o he “w ong_wo d_coun ” column
in he co esponding ow.
Figu e
10
-
Log P ice in ela ion o numbe o badly w i en wo ds
26
Figu e 10 shows no di ec co ela ion be ween he numbe o badly w i en wo ds and he log p ice,
o ins ance conside ing an x alue o 0 we obse e ca p ices anging om he minimum alue (1000€)
o e^12 (a ound 160 000€). A e obse ing he igu e abo e, we conclude ha he numbe o badly
w i en wo ds is no a good p edic o o ca p ice.
3.4. FEATURE EXTRACTION
Be o e applying eg ession models o p edic ca p ices, he ex used o he p edic ions needs o be
ep esen ed in a way ha allows hose models o be applied. The chosen ep esen a ion was he Bag-
o -wo ds model, p e iously explained. In his case, each documen is he desc ip ion ex w i en by
he selle and he ocabula y is he o al numbe o unique wo ds in all he desc ip ion ex s.
Using sklea n´s Coun Vec o ize a wo d coun ma ix is c ea ed, whe e each ow co esponds o a
ce ain ad and he columns a e he wo ds in he ocabula y. The elemen s in he ma ix a e he numbe
o imes a ce ain wo d appea s in a ce ain documen .
Coun Vec o ize allows his scheme o be ex ended o encode a limi ed amoun o dependence wi h
he op ion ng am_ ange, ex ac ing in o ma ion abou he n-g ams up un il he maximum de ined. Fo
his wo k, unig ams and big ams we e conside ed.
Ins ead o he aw equencies o occu ence o a oken in a gi en documen , one can also use he -
id e m weigh ing scheme p e iously explained. This helps o scale down he impac o okens ha
occu e y equen ly in he co pus and ha a e hence less in o ma i e han ea u es ha occu in a
small ac ion o he aining co pus (Anon n.d.). Fo his p ojec , bo h aw equencies and -id sco es
we e used and hei pe o mances compa ed.
3.4.1. Dimensionali y Reduc ion
The p e-p ocessing s eps p e iously desc ibed help o educe he ocabula y dimension, howe e he
wo d coun ma ix o igina ed om Coun Vec o ize is s ill a e y spa se ma ix. Using all he ea u es
om he ma ix o a alue eg ession can nega i ely impac he model pe o mance, seeing ha a
la ge pa o he p edic o s ha e a alue o 0 in almos all ows o he ma ix.
To add ess his p oblem wo ypes o dimensionali y educ ion we e used:
- The i s one was o only conside he op n mos equen ea u es om he wo d coun
ma ix;
- The second one was o apply a dimensionali y educ ion echnique a ailable in sklea n called
T unca edSVD.
The egula Singula Value Decomposi ion is a ac o iza ion o a eal o complex ma ix ha gene alizes
he eigen decomposi ion o a squa e no mal ma ix o any n × p ma ix. I is de ined as ollows: Le X
be a n × p da a ma ix, he singula alue decomposi ion o X is X=UDV´.
27
Whe e U is n × p, D is p × p and V is p × p symme ic ma ix. The columns o V gi e he eigen ec o s o
X´X ma ix and he diagonal alues o D ma ix gi e he squa e oo o he co esponding eigen alues
o he X´X ma ix (Mendes 2017).
T unca ed SVD is di e en om egula SVD in ha i p oduces a ac o iza ion whe e he numbe o
columns is equal o he speci ied unca ion. Only he columns ec o s o U and ow ec o s o V´
co esponding o he la ges singula alues a e calcula ed.
3.5. MODELLING
Be o e unning he eg ession models he da a is spli in o aining and es ing se s wi h 80-20 a io. In
o al we ha e 14429 ins ances o aining and 3607 ins ances o es ing ou models’ pe o mances.
In his wo k di e en ypes o eg ession algo i hms we e es ed: Linea Reg ession and i ´s a ian s
(Lasso, Ridge and Elas ic Ne ), Suppo Vec o Reg ession (SVR), Nu Suppo Vec o Reg ession (NuSVR),
Linea Suppo Vec o Reg ession (Linea SVR), Decision T ee Reg esso , G adien Boos ing Reg esso ,
Mul i-Laye Pe cep on Reg esso (MLP Reg esso ). All hese es ima o s a e a ailable in he SciKi –
Lea n module (Ped egosa e al. 2011). The e ec s on he eg ession pe o mance o applying
s emming, lemma iza ion o none we e also compa ed.
To summa ize, he analysis is conduc ed acco ding o he ollowing s eps:
1) Load he p e-p ocessed da ase in o a Pandas da a ame wi h wo columns ( he ex column
and he p ice column) and apply one o he ollowing o he ex :
a. S emming
b. Lemma iza ion
c. Basic (no s emming o lemma iza ion)
2) Spli he da a ame in o aining se and es ing se (80% aining, 20% es ing)
3) C ea e n-g am wo d coun ma ix conside ing:
a. Only unig ams
b. Unig ams and big ams
c. Only big ams
4) T ans o m n-g am coun s ec o s in o -id sco es
5) Fo dimensionali y educ ion apply one o he ollowing:
a. Keep a maximum o op n-a ibu es acco ding o e m equency (n=1000,2000,3000)
b. Compu e unca ed singula alue decomposi ions o he esul ing ma ix, keeping he
op k singula ec o s, wi h k<n (k=100,200,300,400,500)
6) Fi he esul ing aining da a in o an es ima o using he SciKi -Lea n module. Fi ing is done
using each eg ession algo i hm
7) Calcula e pe o mance indica o s on he log p ice eg ession using he SciKi -Lea n module.
28
3.5.1. Pe o mance Indica o s
To assess he quali y o ou eg ession models, we will use he coe icien o de e mina ion, deno ed
R2, he Mean Squa ed E o (MSE) and he Mean Absolu e E o (MAE) o e alua ion. All me ics a e
calcula ed using he SciKi -Lea n module.
R2 is he a io o he explained a ia ion compa ed o he o al a ia ion, i is in e p e ed as he ac ion
o he sample a ia ion in y ( he dependen a iable) ha is explained by x ( he independen
a iable(s)) (Damásio 2019). I is calcula ed as:
𝑅=1−𝑆𝑆𝑅
𝑆𝑆𝑇 , 𝑤ℎ𝑒𝑟𝑒 𝑆𝑆𝑇 = (𝑦−𝑦)
𝑎𝑛𝑑 𝑆𝑆𝑅 = (𝑦−𝑦
)
Values o R2 ou side he ange 0 o 1 can occu when he model i s he da a wo se han a ho izon al
hype plane. This would occu when he w ong model was chosen.
The MSE is calcula ed as he mean o he squa ed di e ences be ween p edic ed and expec ed a ge
alues in a da ase :
𝑀𝑆𝐸 =1
𝑛 (𝑦−𝑦
)
Whe e 𝑦 is he i´ h expec ed alue in he da ase and 𝑦 is he i´ h p edic ed alue. The di e ence
be ween hese wo alues is squa ed, which has he e ec o emo ing he sign, esul ing in a posi i e
e o alue.
The squa ing also has he e ec o in la ing o magni ying la ge e o s. This has he e ec o
“punishing” models mo e o la ge e o s when i is used as a me ic (B ownlee 2021).
The MAE sco e is calcula ed as he a e age o he absolu e e o alues. I is compu ed as ollows:
𝑀𝐴𝐸 = 1
𝑛 |𝑦− 𝑦|
While MSE punishes la ge e o s mo e han smalle e o s due o he squa e o he e o alue, he
MAE does no gi e mo e o less weigh o di e en ypes o e o s and ins ead he sco es inc ease
linea ly wi h he inc eases in e o (B ownlee 2021).
29
4. RESULTS AND DISCUSSION
4.1. LINEAR REGRESSION RESULTS
Fo he de aul linea eg ession model on he wo d coun ma ix he bes con igu a ion was o
conside only he op 2000 mos equen unig ams. Table 6 summa izes he esul s o he h ee p e-
p ocessing con igu a ions analyzed. As we can obse e om able 6, o 2000 okens, lemma iza ion
p oduces sligh ly be e esul s.
Basic
S emming
Lemma iza ion
R2
0.58
0.58
0.59
MSE
0.35
0.35
0.34
MAE
0.46
0.46
0.45
Table 6 – Linea Reg ession esul s on he 2000 mos equen unig ams o basic, s emming and
lemma iza ion con igu a ions
In e ms o he numbe o p edic o s used, conside ing 1000 mos common wo ds he esul s a e he
same o s emming, howe e o he o he 2 op ions i p oduces wo se esul s. Fo 3000 mos equen
wo ds he esul s we e wo se o all h ee p e-p ocessing echniques. (See appendix 8.2.1)
In e ms o he n-g am leng h used, he e seems o be no imp o emen in esul s by conside ing bo h
unig ams and big ams ins ead o only unig ams (See appendix 8.2.1). Using only big ams p oduces
wo se esul s.
The nex linea eg ession model ained g ea ly imp o ed on he baseline pe o mance, as i is
obse ed in igu e 11. Ins ead o simply eg essing he log ca p ice on he 2000 mos equen wo ds
aw coun , he ma ix o coun s is ans o med o a no malized -id ep esen a ion. The con igu a ion
ha achie ed he bes pe o mance was: No s emming o lemma iza ion applied, unig ams only and
conside ing he -id ea u es o he mos equen 2000 okens.
36
The bes con igu a ion ha was chosen o use in u he hype pa ame e op imiza ion was: 100
singula ec o s de i ed om he -id ma ix conside ing bo h unig ams and big ams and applying
basic p e-p ocessing. The espec i e me ic alues we e 0.61 o R2 , 0.33 o MSE and 0.44 o MAE.
4.3.1. G adien Boos ing Reg esso hype pa ame e op imiza ion
A e choosing he con igu a ion ha pe o med bes o G adien Boos ing Reg esso ,
hype pa ame e uning was pe o med. The wo pa ame e s analyzed we e he numbe o es ima o s
used and he max dep h o each indi idual eg ession ee es ima o .
F om igu e 15 we can see ha he pe o mance o all h ee me ics keeps inc easing he highe he
numbe o es ima o s used, un il 500 es ima o s, a e 500 i doesn´ imp o e on all h ee me ics
analyzed. Wi h 500 es ima o s he R2 alue imp o es o 0.65, he MSE o 0.30 and he MAE o 0.41.
The same ype o analysis was made o he max dep h alue, bu in his case he numbe o es ima o s
was se o 500 ins ead o he de aul alue o 100, seeing as i inc eases he model pe o mance. Figu e
16 shows ha he pe o mance o he es ima o keeps inc easing un il a max dep h alue o 5 o bo h
R2 and MSE. Fo MAE, he alue is lowes wi h a max dep h alue o 10, albei e y small dec ease
compa ed o max dep h o 5 (jus 0.01 o di e ence).
Figu e
15
–
G adien Boos ing Reg esso pe o mance wh
en a ying he numbe o
es ima o s
37
Seeing as ough wo o he h ee me ics analyzed in his wo k show be e esul s wi h a max dep h
alue o 5, his alue was chosen as he one ha op imized he pe o mance o G adien Boos ing
Reg esso o his wo k. The inal alues o he pe o mance me ics we e 0.67 o R2, 0.28 o MSE
and 0.40 o MAE.
4.4. MULTI-LAYER PERCEPTRON RESULTS
The inal eg esso es ed in his wo k was he Mul i-Laye Pe cep on Reg esso . The same
con igu a ions we e es ed (See appendix 8.2.4). Fo his eg esso using singula ec o s p o ides
be e esul s han using he -id ea u es as eg esso s. The op wo con igu a ions in e ms o
pe o mance on de aul hype pa ame e s can be seen in Table 12.
Figu e
16
–
G adien Bo
os ing Reg esso pe o mance w
hen a ying he max dep h o
he indi idual eg ession ees
38
The e is jus a small di e ence in R2 alue be ween using he basic p e-p ocessing and applying
s emming, o he o he wo me ics he e is no di e ence. In e ms o he numbe o singula ec o s
used, we can obse e ha using he smalle numbe o 100 singula ec o s p o ides he bes esul s.
(See appendix 8.2.4)
4.4.1. Mul i-Laye Pe cep on Reg esso Hype pa ame e op imiza ion
A e ob aining he con igu a ion ha achie es he be e pe o mance on his eg esso wi h he
de aul hype pa ame e s. In an a emp o imp o e he eg esso pe o mance, hype pa ame e
uning was pe o med.
Fo his wo k he ollowing Mul i-Laye Pe cep on pa ame e s we e analyzed:
- Hidden laye sizes
- Ac i a ion unc ion
- Sol e unc ion
- Lea ning a e
- Lea ning a e ini ial alue
- Alpha alue
- Max i e a ions
The ollowing ables show he esul s o each hype pa ame e , and he di e en alues es ed. The
pa ame e s we e es ed in he o de s a ed abo e, each ime upda ing he hype pa ame e
con igu a ion wi h he bes alue ob ained.
P e
-
p ocessing
Classi ie
Va iables
N
-
G am
leng h
R
2
MSE
MAE
Basic
MLPReg esso
1
00
singula
ec o s
ob ained
om -id
ma ix
Unig ams
&
Big ams
0.
6
7
0.
28
0.
40
S emming
MLPReg esso
1
00
singula
ec o s
ob ained
om -id
ma ix
Unig ams
&
Big ams
0.
66
0.
28
0.
40
Table
12
–
Bes
pe o ming con igu a ions o
MLPReg esso
39
Ac i a ion
unc ion
iden i y
logis ic
anh
elu
R2
0.59
0.59
0.59
0.67
MSE
0.35
0.34
0.34
0.28
MAE
0.45
0.45
0.45
0.40
Table 14 – Compa ing he di e en ac i a ion unc ions
Sol e
unc ion
lb gs
sgd
adam
R2
0.63
0.59
0.67
MSE
0.31
0.34
0.28
MAE
0.43
0.45
0.40
Table 15 – Compa ing sol e unc ions
Lea ning a e
cons an
in scaling
adap i e
R2
0.67
0.67
0.67
MSE
0.28
0.28
0.28
MAE
0.40
0.40
0.40
Table 16 – Compa ing Lea ning Ra es
Hidden
laye
sizes
(10,)
(20,)
(40,)
(60,)
(100,)
(10,10)
(20,20)
(40,40)
(60,60)
(100,100)
R2
0.63
0.62
0.65
0.66
0.67
0.65
0.66
0.66
0.66
0.64
MSE
0.31
0.32
0.29
0.29
0.28
0.29
0.29
0.29
0.28
0.31
MAE
0.42
0.42
0.41
0.40
0.40
0.41
0.40
0.41
0.40
0.42
Table
13
–
Compa ing a ious hidden l
aye sizes con igu a ions
40
Lea ning
a e ini
0.0001
0.001
0.005
0.01
0.05
0.1
R2
0.62
0.67
0.64
0.63
0.64
0.64
MSE
0.32
0.28
0.30
0.31
0.30
0.30
MAE
0.43
0.40
0.42
0.42
0.42
0.42
Table 17 – Compa ing di e en ini ial lea ning a e alues
Alpha
0.00001
0.0001
0.001
0.01
R2
0.66
0.67
0.67
0.67
MSE
0.29
0.28
0.28
0.28
MAE
0.40
0.40
0.40
0.40
Table 18 – Compa ing di e en alpha alues
Max
i e a ions
100
200
300
400
500
1000
2000
R2
0.65
0.67
0.66
0.66
0.66
0.66
0.66
MSE
0.29
0.28
0.29
0.29
0.29
0.29
0.29
MAE
0.41
0.40
0.40
0.41
0.41
0.41
0.41
Table 19 – Compa ing he pe o mance on di e en numbe s o maximum i e a ions
F om he esul s we can see ha ac oss all hype pa ame e s alues es ed, he de aul alue is he
be e pe o ming one. Lea ning a e and alpha also p oduce he same esul s wi h di e en alues
o he han he de aul , howe e o all o he hype pa ame e s ha is no he case.
Since no u he imp o emen s in pe o mance we e made wi h hype pa ame e uning, gi en ha
he de aul alues ou pe o med he alues es ed. The MLPReg esso model has a pe o mance o
0.67 o R2, 0.28 o MSE and 0.40 o MAE.
4.5. RESULTS SUMMARY
The ollowing able p o ides a concise summa y o all he me hods applied and hei espec i e bes
esul s in p edic ing he log ca p ice.
41
Classi ie
P e
-
p ocessing
Va iables
N
-
G am
leng h
R
2
MSE
MAE
Linea
Reg ession
Basic
Top 2000
-id
ea u es
Unig ams
only
0.70
0.25
0.39
Ridge
Reg ession
Basic
Top 2000
-id
ea u es
Unig ams
only
0.72
0.24
0.37
Lasso
Reg ession /
Elas ic Ne
Reg ession
Basic
Top 2000
-id
ea u es
Unig ams
only
0.00
0.84
0.75
NuSVR
Lemma iza ion/
S emming
Top 3000
-id
ea u es
Unig ams
only
0.77
0.19
0.32
SVR
Lemma iza ion/
S emming
Top 3000
-id
ea u es
Unig ams
only
0.77
0.
20
0.3
3
LSVR
Basic
Top 3000
-id
ea u es
Unig ams
only
0.71
0.24
0.37
Decision T ee
Reg esso
Basic
Top 2000
-id
ea u es
Unig ams
only
0.60
0.28
0.56
G adien
Boos ing
Reg esso
Basic
100
singula
ec o s
de i ed
om -
id ma ix
Unig ams
&
Big ams
0.67
0.28
0.40
MLP
Reg esso
Basic
100
singula
ec o s
de i ed
om -
id ma ix
Unig ams
&
Big ams
0.67
0.28
0.40
Table
20
–
Applied me hods and hei e
spec i e bes esul s
42
5. CONCLUSIONS
The esea ch objec i e o his wo k p ojec was o iden i y i he desc ip ion ex alone was a good
enough p edic o o ca ´s p ice. The expe imen on c ea ing eg esso s om uns uc u ed ex o his
eg ession p oblem showed di e en esul s ac oss he a ious eg ession models used.
Suppo Vec o Reg ession ou pe o med all o he models o eg ession es ed in his wo k p ojec .
NuSVR was he be e suppo ec o eg ession model used, wi h i ´s bes con igu a ion explaining
77% o he a ia ion in he log p ice and also achie ing good esul s in he MSE and MAE me ics. SVR
also achie es e y simila esul s o NuSVR, only p oducing sligh ly wo se esul s in MAE.
The wo s pe o ming models we e Lasso Reg ession and Elas ic Ne eg ession, bo h wi h an R2 alue
o 0.00, a MSE alue o 0.84 and a MAE alue o 0.75. Howe e , he o he Linea Reg ession a ian
es ed, Ridge Reg ession, showed be e pe o mance han he de aul Linea Reg ession model wi h
an R2 alue o 0.72, a MSE o 0.24 and a MAE o 0.37, being he second-bes pe o ming model.
As a as p e-p ocessing s a egies applied in his wo k, esul s sugges ha he e wasn´ one ha had
an inc eased ad an age in pe o mance compa ed o he o he s, he di e ence usually being o jus
a ound 0.01/0.02 in ei he R2, MSE o MAE. The pe o mance a ied acco ding o he eg esso used.
Fo ins ance, o Linea Reg ession, Decision T ee Reg esso , G adien Boos ing Reg esso and Mul i-
Laye Pe cep on Reg esso he be e con igu a ion was ound by no applying ei he s emming o
lemma iza ion. Howe e , o NuSVR he be e con igu a ion was ound by using
s emming/lemma iza ion.
The same conclusion can be de i ed o he n-g am leng h conside ed o c ea e he ini ial n-g am coun
ma ix, he e was no op ion ha consis en ly pe o med be e o all ypes o eg esso s s udied. Fo
example, o Linea Reg ession when using he -id sco es as p edic o s conside ing bo h unig ams
and big ams doesn´ imp o e he model pe o mance compa ed o only using unig ams. Howe e ,
when using he singula ec o s as p edic o s, he model pe o mance is sligh ly inc eased when
conside ing bo h unig ams and big ams. Using only big ams o c ea e he ini ial n-g am coun ma ix
p o ed o lead o wo se esul s o all models analyzed.
In e ms o p edic o s de i ed om he ini ial uns uc u ed ex , wo ypes we e analyzed: e aining
he op n (n=1000, 2000, 3000) -id ea u es acco ding o coun equency o using he op k (k=100,
200, 300, 400, 500) singula ec o s de i ed om he -id ma ix. Ou esul s sugges ha , o his
eg ession p oblem, he be e pe o ming op ion a ies wi h he eg ession model being used and no
op ion consis en ly ou pe o ms he o he . Fo ins ance, o Linea Reg ession and Suppo Vec o
Reg ession, which we e he be e pe o ming models, using -id ea u es as p edic o s ou pe o ms
using singula ec o s by a small ma gin. Fo G adien Boos ing Reg esso and Mul i-Laye Pe cep on
Reg esso i was ound ha using singula ec o s as p edic o s led o be e pe o ming models.
Ne e heless, ha di e ence in esul s was no g ea e han 0.05 in any o he h ee me ics analyzed,
when compa ed o he bes con igu a ion ha used he -id ea u es as p edic o s.
Applying he -id ec o ize o ans o m he ini ial ma ix o aw coun s was he s ep ha as ly
imp o ed he esul s, we can see om he baseline Linea Reg ession ha used he aw wo d coun s
as p edic o s compa ed o he Linea Reg ession ha e ained he same numbe o okens bu wi h
he -id sco es ins ead a g ea imp o emen . The me ics jumped om an R2 o 0.58 o 0.70, a MSE
43
o 0.35 o 0.25 and a MAE o 0.46 o 0.39. This imp o emen was also e i ied when using Suppo
Vec o Reg ession and Mul i-Laye Pe cep on Reg esso .
The analysis o he esul s ound in he p e ious sec ion, show ha some eg ession models we e mo e
success ul han o he s in managing o p edic he log p ice a ca being sold in an online ca , using as
p edic o s ea u es c ea ed om uns uc u ed ex . These esul s demons a e ha ex eg ession
models o p ice p edic ion may be applicable as a complemen o adi ional me hods o ca p ice
p edic ion h ough he use o he common a ibu es.
Abou he objec i es ini ially p oposed in his wo k, we belie e hey we e achie ed. A he end o his
wo k p ojec , we ha e a be e unde s anding o he ex en o he capabili ies ha ex ual desc ip ions
ha e in accu a ely p edic ing ca p ices, wi h some eg ession models being mo e capable in using he
ea u es de i ed om hose desc ip ion ex s o eg ess he p ice. Also, we deepened ou knowledge
on he ques ion o ex p e-p ocessing and ea u e ex ac ion om ex , ha lead o being able o
c ea e ea u es capable o being used in ou eg ession p oblem. Las ly, we inc eased ou heo e ical
and p ac ical knowledge on he add essed opics o Tex Mining, ex eg ession, he s udied
algo i hms and he ools used.
44
6. LIMITATIONS AND RECOMMENDATIONS FOR FUTURE WORKS
Th oughou he de elopmen o his wo k p ojec , we ha e encoun e ed some di icul ies. One o
hose p oblems and pe haps he one ha mos in luenced he p og ess was he ea men o he da a,
gi en ha i all consis ed o human inse ed ex i con ained a e y la ge amoun o badly w i en
wo ds. This caused he da a o be e y inconsis en , ha ing mul iple a ia ions o he same wo d,
making he p ocess o wo d co ec ion qui e ime consuming.
Ano he p oblem ela ed o wo d co ec ion was he lack o ools o wo d co ec ion ha included in
hei ocabula y he con ex speci ic wo ds, ela ed o ca desc ip ions, causing hose ools o w ongly
iden i y a la ge numbe o wo ds as badly w i en.
Also, in he da a collec ion p ocess we encoun e ed some deadlocks, gi en ha he da a was aken
om ads published by he use s o a websi e he amoun o da a collec ed was dependen on he
numbe o new ads published by he use s, which made he da a collec ion p ocess qui e i e a i e and
ime consuming.
Fo u u e wo ks in his a ea, i is ecommended o c ea e a p e-de ined ocabula y wi h he con ex
speci ic wo ds o acili a e he da a ea men p ocess. Also, al hough he ex was ob ained om a
Po uguese websi e, ca desc ip ion ex s con ain a g ea numbe o wo ds ha a e anglicisms, such
as “blue oo h” and “ai bag”. Wi h ha being said, i is ecommended ha he wo d co ec ion ool is
able o ecognize bo h languages, in his case Po uguese and English.
In e ms o modelling and pe o mance esul s, o u u e wo ks ins ead o i s inding he op imal
con igu a ion in e ms o wha ype o ea u es o use as p edic o s, wi h he de aul model
hype pa ame e s, and only a e p oceeding o doing hype pa ame e uning. An op ion wo h
conside ing is o apply a G id Sea ch, a ailable in Sci-Ki Lea n, o ind he op imal con igu a ion wi h
he co esponding hype pa ame e s ha esul in be e pe o mance. This could lead o an inc ease
in he model’s pe o mance, albei wi h an inc ease in compu a ion ime.
45
7. BIBLIOGRAPHY
Allahya i, Mehdi, Seyedamin Pou iyeh, Mehdi Asse i, Saied Sa aei, Elizabe h D. T ippe, Juan B.
Gu ie ez, and K ys Kochu . 2017. “A B ie Su ey o Tex Mining: Classi ica ion, Clus e ing and
Ex ac ion Techniques.” A Xi :1707.02919 [Cs].
Anon. n.d. “1.4. Suppo Vec o Machines — Sciki -Lea n 0.24.2 Documen a ion.” Re ie ed June 29,
2021a (h ps://sciki -lea n.o g/s able/modules/s m.h ml#s m- eg ession).
Anon. n.d. “Di lib — Helpe s o Compu ing Del as — Py hon 3.9.2 Documen a ion.” Re ie ed Ap il
2, 2021b (h ps://docs.py hon.o g/3/lib a y/di lib.h ml).
Anon. n.d. “Sklea n.Fea u e_ex ac ion.Tex .T id T ans o me — Sciki -Lea n 0.24.2 Documen a ion.”
Re ie ed Ap il 28, 2021c (h ps://sciki -
lea n.o g/s able/modules/gene a ed/sklea n. ea u e_ex ac ion. ex .T id T ans o me .h ml).
Ba us, Tyle . 2018. “Pyspellchecke — Pyspellchecke 0.6.1 Documen a ion.” Re ie ed Ma ch 24,
2021 (h ps://pyspellchecke . ead hedocs.io/en/la es /).
Bi ai, Zsol , and T e o Cohn. 2015. “Non-Linea Tex Reg ession wi h a Deep Con olu ional Neu al
Ne wo k.” Pp. 180–85 in P oceedings o he 53 d Annual Mee ing o he Associa ion o
Compu a ional Linguis ics and he 7 h In e na ional Join Con e ence on Na u al Language P ocessing
(Volume 2: Sho Pape s). Beijing, China: Associa ion o Compu a ional Linguis ics.
B ownlee, Jason. 2021. “Reg ession Me ics o Machine Lea ning.” Machine Lea ning Mas e y.
Re ie ed June 8, 2021 (h ps://machinelea ningmas e y.com/ eg ession-me ics- o -machine-
lea ning/).
Cas elli, Mau o, Ma ia Dob e a, Robe o Hen iques, and Leona do Vanneschi. 2020. “P edic ing Days
on Ma ke o Op imize Real Es a e Sales S a egy.” Complexi y 2020:1–22. doi:
10.1155/2020/4603190.
Damásio, B uno. 2019. “S a is ics o Da a Science - The Linea Reg ession Model.” 120.
Eli h, J., J. R. Lea hwick, and T. Has ie. 2008. “A Wo king Guide o Boos ed Reg ession T ees.” Jou nal
o Animal Ecology 77(4):802–13. doi: 10.1111/j.1365-2656.2008.01390.x.
Fos e , Dean P., Ma k Libe man, and Robe A. S ine. 2013. “Fea u izing Tex : Con e ing Tex in o
P edic o s o Reg ession Analysis.” 37.
Gegic, Enis, Beci Isako ic, Dino Keco, Ze ina Mase ic, and Jasmin Ke ic. n.d. “Ca P ice P edic ion
Using Machine Lea ning Techniques.” 8(1):6.
Gen zkow, Ma hew, B yan Kelly, and Ma Taddy. 2019. “Tex as Da a.” Jou nal o Economic
Li e a u e 57(3):535–74. doi: 10.1257/jel.20181020.
Gilleland, Michael. n.d. “Le ensh ein Dis ance.” Le ensh ein Dis ance, in Th ee Fla o s. Re ie ed
Augus 30, 2021
52
2000 okens
Basic
S emming
Lemma iza ion
R2
0.70 (0.69)
0.69 (0.69)
0.70 (0.69)
MSE
0.25 (0.26)
0.26 (0.26)
0.26 (0.26)
MAE
0.39 (0.39)
0.39 (0.39)
0.39 (0.39)
Table 24 - Linea Reg ession esul s o he op 2000 -id sco es conside ing only unig ams
(unig ams & big ams)
3000 okens
Basic
S emming
Lemma iza ion
R2
0.68 (0.69)
0.69 (0.68)
0.69 (0.67)
MSE
0.26 (0.26)
0.26 (0.27)
0.26 (0.27)
MAE
0.39 (0.39)
0.39 (0.40)
0.39 (0.40)
Table 25 - Linea Reg ession esul s o he op 3000 -id sco es conside ing only unig ams
(unig ams & big ams)
Top 100 singula
ec o s
Basic
S emming
Lemma iza ion
R2
0.49 (0.48)
0.48 (0.48)
0.48 (0.47)
MSE
0.43 (0.44)
0.43
(0.44)
0.44 (0.44)
MAE
0.52 (0.52)
0.52 (0.52)
0.52 (0.52)
Table 26 – Linea Reg ession esul s o he op 100 singula ec o s e ained om he wo d coun
ma ix using only unig ams (unig ams & big ams)
Top 200
singula
ec o s
Basic
S emming
Lemma iza ion
R2
0.53 (0.53)
0.53 (0.53)
0.52 (0.52)
MSE
0.40 (0.40)
0.40 (0.39)
0.41 (0.40)
MAE
0.49 (0.49)
0.49 (0.49)
0.50 (0.49)
Table 27 - Linea Reg ession esul s o he op 200 singula ec o s e ained om he wo d coun
ma ix using only unig ams (unig ams & big ams)
53
Top 300 singula
ec o s
Basic
S emming
Lemma iza ion
R2
0.55 (0.54)
0.54 (0.55)
0.54 (0.54)
MSE
0.38 (0.38)
0.39 (0.38)
0.38 (0.38)
MAE
0.48 (0.48)
0.49 (0.48)
0.48 (0.48)
Table 28 - Linea Reg ession esul s o he op 300 singula ec o s e ained om he wo d coun
ma ix using only unig ams (unig ams & big ams)
Top 400
singula
ec o s
Basic
S emming
Lemma iza ion
R2
0.56 (0.56)
0.56 (0.56)
0.56 (0.56)
MSE
0.37 (0.37)
0.37 (0.37)
0.37 (0.37)
MAE
0.47 (0.47)
0.47 (0.47)
0.47 (0.47)
Table 29 - Linea Reg ession esul s o he op 400 singula ec o s e ained om he wo d coun
ma ix using only unig ams (unig ams & big ams)
Top 500 singula
ec o s
Basic
S emming
Lemma iza ion
R2
0.57 (0.57)
0.58 (0.58)
0.58 (0.57)
MSE
0.36 (0.36)
0.36 (0.36)
0.36 (0.36)
MAE
0.47 (0.46)
0.47
(0.46)
0.47 (0.47)
Table 30 - Linea Reg ession esul s o he op 500 singula ec o s e ained om he wo d coun
ma ix using only unig ams (unig ams & big ams)
500 singula ec o s
Basic
S emming
Lemma iza ion
R2
0.68
(0.69)
0.68 (0.69)
0.68 (0.69)
MSE
0.27 (0.26)
0.27 (0.26)
0.27 (0.26)
MAE
0.40 (0.39)
0.40 (0.39)
0.40 (0.39)
Table 31 – Linea Reg ession esul s o he op 500 singula ec o s e ained om he -id sco e
ma ix using only unig ams (unig ams & big ams)
54
Linea
Reg ession
Ridge Reg ession
Lasso Reg ession
Elas ic Ne
Reg ession
R2
0.70 (0.69)
0.72 (0.71)
0.00 (0.00)
0.00 (0.00)
MSE
0.25
(0.26)
0.24 (0.24)
0.84 (0.84)
0.84 (0.84)
MAE
0.39 (0.39)
0.37 (0.38)
0.75 (0.75)
0.75 (0.75)
Table 32 – Compa ing he di e en e ypes o Linea Reg ession on he op 2000 -id sco es
conside ing only unig ams (unig ams & big ams) all wi h basic p e-p ocessing
500 singula
ec o s
Linea
Reg ession
Ridge Reg ession
Lasso Reg ession
Elas ic Ne
Reg ession
R2
0.68 (0.69)
0.68 (0.69)
0.00 (0.00)
0.00 (0.00)
MSE
0.27 (0.26)
0.27 (0.26)
0.84 (0.84)
0.84 (0.84)
MAE
0.40 (0.39)
0.40 (0.39)
0.75 (0.75)
0.75 (0.75)
Table 33 – Compa ing he di e en e ypes o Linea Reg ession on he op 500 singula ec o s
e ained om he -id sco e ma ix conside ing only unig ams (unig ams & big ams) all wi h basic
p e-p ocessing
8.2.2. Suppo Vec o Reg ession Resul s
Basic P e-P ocessing:
2000 ea u es
LS
SVR
NuS
R2
0.70 (70)
0.76
(0.75)
0.76
(0.75)
MSE
0.25 (0.26)
0.20
(0.21)
0.20
(0.21)
MAE
0.38 (0.38)
0.33
(0.34)
0.33
(0.33)
Table 34 - Compa ing di e en ypes o suppo ec o eg ession using he op 2000 -id ea u es,
unig ams only (unig ams & big ams)
55
3000 ea u es
LS
SVR
NuS
R2
0.71
(0.70)
0.76
(0.75)
0.77
(0.76)
MSE
0.24
(0.25)
0.20
(0.21)
0.20
(0.20)
MAE
0.37
(0.38)
0.33
(0.34)
0.32
(0.33)
Table 35 - Compa ing di e en ypes o suppo ec o eg ession using he op 3000 -id ea u es,
unig ams only (unig ams & big ams)
S emming P e-P ocessing:
2000 ea u es
SVR
NuS
R2
0.76 (0.75)
0.76 (0.76)
MSE
0.20 (0.21)
0.20 (0.21)
MAE
0.33 (0.34)
0.32 (0.33)
Table 36 - Compa ing di e en ypes o suppo ec o eg ession using he op 2000 -id ea u es,
unig ams only (unig ams & big ams)
3000 ea u es
SVR
NuS
R2
0.77
(0.75)
0.77 (0.76)
MSE
0.20 (0.21)
0.19 (0.20)
MAE
0.33 (0.33)
0.32 (0.33)
Table 37 - Compa ing di e en ypes o suppo ec o eg ession using he op 3000 -id ea u es,
unig ams only (unig ams & big ams)
56
Lemma iza ion P e-P ocessing:
2000
ea u es
SVR
NuS
R2
0.76 (0.75)
0.76 (0.75)
MSE
0.20 (0.21)
0.20 (0.21)
MAE
0.33 (0.34)
0.32 (0.33)
Table 38 - Compa ing di e en ypes o suppo ec o eg ession using he op 2000 -id ea u es,
unig ams only (unig ams & big ams)
3000
ea u es
SVR
NuS
R2
0.77 (0.75)
0.77 (0.76)
MSE
0.20 (0.21)
0.19 (0.21)
MAE
0.33 (0.34)
0.32 (0.33)
Table 39 - Compa ing di e en ypes o suppo ec o eg ession using he op 3000 -id ea u es,
unig ams only (unig ams & big ams)
Using Singula Vec o s and basic p e-p ocessing:
100 singula
ec o s
LS
SVR
NuS
R2
0.58
0.70
(0.70)
0.70
(0.70)
MSE
0.35
0.26
(0.25)
0.26
(0.25)
MAE
0.46
0.38
(0.37)
0.38
(0.37)
Table 40 – Compa ing di e en ypes o suppo ec o eg ession using he op 100 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams)
200 singula
ec o s
LS
SVR
NuS
R2
0.61
0.73
(0.73)
0.73
(0.73)
MSE
0.32
0.21
(0.25)
0.23
(0.23)
MAE
0.44
0.35
(0.35)
0.35
(0.38)
Table 41 - Compa ing di e en ypes o suppo ec o eg ession using he op 200 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams)
57
300 singula
ec o s
LS
SVR
NuS
R2
0.65
0.75
(0.75)
0.75
(0.75)
MSE
0.29
0.21
(0.21)
0.21
(0.21)
MAE
0.41
0.34
(0.33)
0.33
(0.33)
Table 42 - Compa ing di e en ypes o suppo ec o eg ession using he op 300 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams)
400 singula
ec o s
LS
SVR
NuS
R2
0.66
0.76
(0.76)
0.76
(0.76)
MSE
0.28
0.20
(0.20)
0.20
(0.20)
MAE
0.41
0.33
(0.33)
0.33
(0.33)
Table 43 - Compa ing di e en ypes o suppo ec o eg ession using he op 400 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams)
500 singula
ec o s
LS
SVR
NuS
R2
0.67
0.76
(0.76)
0.76
(0.77)
MSE
0.28
0.20
(0.20)
0.20
(0.20)
MAE
0.40
0.33
(0.33)
0.33
(0.32)
Table 44 - Compa ing di e en ypes o suppo ec o eg ession using he op 500 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams)
Using Singula Vec o s and s emming p e-p ocessing:
100 singula
ec o s
SVR
NuS
R2
0.69
(0.70)
0.69
(0.70)
MSE
0.26
(0.26)
0.26
(0.26)
MAE
0.38
(0.37)
0.38
(0.37)
Table 45 - Compa ing di e en ypes o suppo ec o eg ession using he op 100 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams), s emming p e-p ocessing
58
200 singula
ec o s
SVR
NuS
R2
0.73
(0.74)
0.73
(0.74)
MSE
0.22
(0.22)
0.22
(0.22)
MAE
0.35
(0.34)
0.35
(0.34)
Table 46 - Compa ing di e en ypes o suppo ec o eg ession using he op 200 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams), s emming p e-p ocessing
300 singula
ec o s
SVR
NuS
R2
0.75
(0.75)
0.75
(0.75)
MSE
0.21
(0.21)
0.21
(0.21)
MAE
0.34
(0.33)
0.33
(0.33)
Table 47 - Compa ing di e en ypes o suppo ec o eg ession using he op 300 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams), s emming p e-p ocessing
400 singula
ec o s
SVR
NuS
R2
0.76
(0.76)
0.76
(0.76)
MSE
0.20
(0.20)
0.20
(0.20)
MAE
0.33
(0.33)
0.33
(0.33)
Table 48 - Compa ing di e en ypes o suppo ec o eg ession using he op 400 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams), s emming p e-p ocessing
500 singula
ec o s
SVR
NuS
R2
0.76
(0.76)
0.77
(0.77)
MSE
0.20
(0.20)
0.20
(0.20)
MAE
0.33
(0.33)
0.32
(0.32)
Table 49 - Compa ing di e en ypes o suppo ec o eg ession using he op 500 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams), s emming p e-p ocessing
59
Using Singula Vec o s and lemma iza ion p e-p ocessing:
100 singula
ec o s
SVR
NuS
R2
0.69
(0.70)
0.69
(0.70)
MSE
0.26
(0.26)
0.26
(0.26)
MAE
0.38
(0.37)
0.38
(0.37)
Table 50 - Compa ing di e en ypes o suppo ec o eg ession using he op 100 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams), lemma iza ion p e-
p ocessing
200 singula
ec o s
SVR
NuS
R2
0.73
(0.73)
0.73
(0.73)
MSE
0.23
(0.22)
0.23
(0.23)
MAE
0.35
(0.35)
0.35
(0.35)
Table 51 - Compa ing di e en ypes o suppo ec o eg ession using he op 200 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams), lemma iza ion p e-
p ocessing
300 singula
ec o s
SVR
NuS
R2
0.75
(0.75)
0.75
(0.75)
MSE
0.21
(0.21)
0.21
(0.21)
MAE
0.34
(0.33)
0.34
(0.33)
Table 52 - Compa ing di e en ypes o suppo ec o eg ession using he op 300 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams), lemma iza ion p e-
p ocessing
60
400 singula
ec o s
SVR
NuS
R2
0.76
(0.76)
0.76
(0.76)
MSE
0.20
(0.20)
0.20
(0.20)
MAE
0.33
(0.33)
0.33
(0.33)
Table 53 - Compa ing di e en ypes o suppo ec o eg ession using he op 400 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams), lemma iza ion p e-
p ocessing
500 singula
ec o s
SVR
NuS
R2
0.76
(0.76)
0.76
(0.76)
MSE
0.20
(0.20)
0.20
(0.20)
MAE
0.33
(0.33)
0.32
(0.32)
Table 54 - Compa ing di e en ypes o suppo ec o eg ession using he op 500 singula ec o s
e ained om he -id sco e ma ix, unig ams only (unig ams & big ams), lemma iza ion p e-
p ocessing
8.2.3. Decision T ee Reg esso and G adien Boos ing Reg esso esul s
1000 ea u es
Basic
S emming
Lemma iza ion
R2
0.21 (0.25)
0.23 (0.19)
0.18 (0.18)
MSE
0.66 (0.63)
0.65 (0.68)
0.69 (0.69)
MAE
0.59 (0.58)
0.58 (0.59)
0.60 (0.60)
Table 55 – Decision T ee Reg esso esul s using he op 1000 -id ea u es, conside ing only
unig ams (unig ams & big ams)
2000 ea u es
Basic
S emming
Lemma iza ion
R2
0.28 (0.28)
0.22 (0.24)
0.19 (0.23)
MSE
0.60 (0.60)
0.65 (0.64)
0.68 (0.65)
MAE
0.56 (0.56)
0.58 (0.58)
0.60 (0.58)
Table 56 - Decision T ee Reg esso esul s using he op 2000 -id ea u es, conside ing only
unig ams (unig ams & big ams)
61
3000 ea u es
Basic
S emming
Lemma iza ion
R2
0.25 (0.26)
0.24 (0.20)
0.20 (0.28)
MSE
0.63 (0.62)
0.64
(0.67)
0.67 (0.61)
MAE
0.57 (0.57)
0.58 (0.59)
0.59 (0.56)
Table 57 - Decision T ee Reg esso esul s using he op 2000 -id ea u es, conside ing only
unig ams (unig ams & big ams)
Numbe o
singula
ec o s
100
200
300
400
500
R2
0.20
0.16
0.16
0.16
0.17
MSE
0.67
0.70
0.71
0.71
0.70
MAE
0.60
0.61
0.62
0.62
0.61
Table 58 – Compa ing he pe o mance o Decision T ee Reg esso on he numbe o singula ec o s,
using basic p e-p ocessing and only unig ams
G adien Boos ing Reg esso
1000 ea u es
Basic
S emming
Lemma iza ion
R2
0.54 (0.54)
0.55 (0.55)
0.55 (0.54)
MSE
0.38 (0.38)
0.38 (0.38)
0.38 (0.39)
MAE
0.48 (0.48)
0.48 (0.48)
0.48 (0.48)
Table 59 – Compa ing he pe o mance o G adien Boos ing Reg esso esul s using he op 1000 -
id ea u es, conside ing only unig ams (unig ams & big ams)
2000 ea u es
No mal
S emming
Lemma iza ion
R2
0.55 (0.55)
0.55 (0.56)
0.54 (0.55)
MSE
0.38 (0.38)
0.38 (0.37)
0.38 (0.38)
MAE
0.48 (0.48)
0.48 (0.47)
0.48 (0.48)
Table 60 - Compa ing he pe o mance o G adien Boos ing Reg esso esul s using he op 2000 -
id ea u es, conside ing only unig ams (unig ams & big ams)