i
Mas e Deg ee P og am in
Da a Science and Ad anced Analy ics
Applica ion o P edic ed Models in Deb Managemen
De eloping a Machine Lea ning Algo i hm o P edic Cus ome Risk a EDP
Come cial
Inês Pi es Melo
In e nship Repo
p esen ed as pa ial equi emen o ob aining he Mas e Deg ee P og am in Da a Science and Ad anced Analy ics
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
MDSAA
i
[ his page should no be included in he digi al e sion. I s pu pose is only o he p in ed e sion]
Ti le:
Sub i le:
S uden ull name
MDSAA
2022
i
NOVA In o ma ion Managemen School
Ins i u o Supe io de Es a ís ica e Ges ão de In o mação
Uni e sidade No a de Lisboa
APPLICATION OF PREDICTED MODELS IN DEBT MANAGEMENT
by
Inês Pi es Melo
In e nship epo p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Ad anced
Analy ics, wi h a Specializa ion in Da a Science
Supe iso : Mau o Cas elli
Feb ua y 2023
ii
STATEMENT OF INTEGRITY
I he eby decla e ha ing conduc ed his academic wo k wi h in eg i y. I con i m ha I ha e no used
plagia ism o any o m o undue use o in o ma ion o alsi ica ion o esul s along he p ocess leading
o i s elabo a ion. I u he decla e ha I ha e ully acknowledge he Rules o Conduc and Code o
Hono om he NOVA In o ma ion Managemen School.
Inês Melo
Lisboa, 2023
iii
ACKNOWLEDGEMENTS
I would like o exp ess my g a i ude o my in e nship eam a EDP o hei unwa e ing suppo and
posi i e spi i h oughou my in e nship, especially o Hélde Oli ei a, who was always a ailable o
help me and belie ed in me om he s a .
I also canno hank my amily enough, my pa en s and sis e , who ha e always suppo ed me and been
a g ea pilla , as well as my iends and housema es, Ma iana and So ia, who pu up wi h me e e y day
and ha e my ull hea .
Las ly, I would like o hank my p o esso s whose guidance and men o ship ha e been ins umen al in
shaping no only my knowledge bu also my cha ac e and o e all de elopmen as an indi idual.

i
ABSTRACT
This epo is a esul o a nine-mon h in e nship a EDP Come cial whe e he main p ojec o esea ch
was he applica ion o a i icial in elligence ools in he ield o deb managemen . Deb managemen
in ol es a se o s a egies and p ocesses aimed a educing o elimina ing deb and he use o a i icial
in elligence has shown g ea po en ial o op imize hese p ocesses and minimize he isk o deb o
indi iduals and o ganiza ions.
In e ms o moni o ing and con olling he c edi wo hiness and quali y o clien s, deb managemen
has mainly been esponsi e and eac i e, a emp ing o eco e losses a e a clien has become
delinquen . The e is a gap in he knowledge o how o p oac i ely iden i y a - isk accoun s be o e hey
all behind on paymen s.
To a oid he cons an eac i e esponse in he ield, i was de eloped a machine-lea ning algo i hm
ha p edic s he isk o a clien becoming in deb by analyzing hei sco eca d, which measu es he
quali y o a clien based on hei in ingemen his o y.
A e p ep ocessing he da a, XGBoos was implemen ed o a da ase o 3M cus ome s wi h a leas
one ac i e con ac on EDP, on elec ici y o gas. Hype pa ame e uning was pe o med on he model
o each an F1 sco e o 0.7850 on he aining se and 0.7835 on he es se . The esul s we e discussed
and based on hose, ecommenda ions and imp o emen s we e also iden i ied.
KEYWORDS
Deb Managemen ; Machine Lea ning; Sco eca d P edic ion; A i icial in elligence
INDEX
1. In oduc ion .................................................................................................................. 1
1.1. Company O e iew ............................................................................................... 1
1.2. The p ojec ............................................................................................................. 2
2. Li e a u e e iew .......................................................................................................... 3
2.1. A i icial in elligence (AI) ....................................................................................... 3
2.2. Da a Mining ........................................................................................................... 3
2.3. CRISP-DM ............................................................................................................... 4
2.4. Machine Lea ning .................................................................................................. 6
2.4.1. Decision T ees ................................................................................................ 6
2.4.2. G adien Boos ing ........................................................................................... 8
2.4.3. XGBoos .......................................................................................................... 8
2.5. Classi ica ion Model E alua ion .......................................................................... 10
2.5.1. Con usion Ma ix .......................................................................................... 10
2.5.2. Accu acy ....................................................................................................... 11
2.5.3. P ecision ....................................................................................................... 11
2.5.4. Recall ............................................................................................................ 11
2.5.5. F1 Sco e ........................................................................................................ 12
2.6. Deb managemen in he ene gy sec o ............................................................. 12
3. Me hodology .............................................................................................................. 14
3.1. Business unde s anding ....................................................................................... 14
3.2. Da a Unde s anding ............................................................................................ 14
3.2.1. Da a sou ces ................................................................................................. 14
3.2.2. Main Findings ............................................................................................... 16
3.2.3. Ta ge analysis .............................................................................................. 19
3.3. Da a P epa a ion ................................................................................................. 20
3.3.1. Fea u e enginee ing ..................................................................................... 20
3.3.2. Adding demog aphic ea u es ...................................................................... 20
3.3.3. Missing alues .............................................................................................. 21
3.3.4. Ou lie de ec ion .......................................................................................... 21
3.3.5. Ca ego ical ea u es ..................................................................................... 21
3.3.6. Fea u e scaling.............................................................................................. 21
3.3.7. Ta ge dis ibu ion ........................................................................................ 22
3.3.8. Dealing wi h co ela ed ea u es.................................................................. 22
i
3.3.9. Fea u e Selec ion .......................................................................................... 23
3.4. Model Selec ion And E alua ion ......................................................................... 23
4. Resul s and discussion ................................................................................................ 25
4.1. E alua ing i s esul s ......................................................................................... 25
4.2. Hype pa ame e uning ....................................................................................... 25
4.3. Discuss inal esul s ............................................................................................. 28
5. Conclusion .................................................................................................................. 30
6. Limi a ions and ecommenda ions o u u e wo ks ................................................. 31
7. Re e ences .................................................................................................................. 32
Appendix.......................................................................................................................... 34
ii
LIST OF FIGURES
Figu a 1 - Phases o CRISP-DM P ocess Model Sou ce: (Ma inez-Plumed, F. e al., 2000) ....... 5
Figu a 2- Elemen s o a Decision ee ......................................................................................... 7
Figu a 3 - L1 and L2 Regula iza ion o mula .............................................................................. 9
Figu a 4 - A e age expe ed annual Consump ion by ype o consump ion ............................. 16
Figu a 5 - Pie Cha o Types o Consump ion .......................................................................... 16
Figu a 6 - Clien dis ibu ion o yea s o consump ion ............................................................. 16
Figu a 7- Dis ibu ion o ypes o Con ac s ............................................................................. 17
Figu a 8 - Ta ge dis ibu ion ................................................................................................... 19
Figu a 9 - F1 Sco e s Lea ning a e ......................................................................................... 26
Figu a 10 - F1 Sco e s Numbe o es ima o s ......................................................................... 27
Figu a 11 - Con usion Ma ix o Tes se .................................................................................. 28
Figu a 12- Ou lie s Visualiza ion .............................................................................................. 35
Figu a 13- Oulie s Visualiza ions pa 2 ................................................................................... 36
Figu a 14- Pea son's co ela ion ma ix ................................................................................... 37
Figu a 15 - Spea man's ank co ela ion ma ix ...................................................................... 38
Figu a 16 - Fea u e Selec ion Decision T ees: Gini and En opy ............................................... 39
Figu a 17- Fea u e Selec ion using Ridge Classi ie .................................................................. 40
Figu a 18- Fea u e Selec ion using Lasso Classi ie .................................................................. 41
5
ha he CRISP-DM app oach is lexible, allowing o modi ica ions o be made o i he speci ic needs
o a p ojec .
As i is shown in he igu e, he six s eps o he CRISP-DM p ocess a e as ollows:
Business Unde s anding: In his s ep, he objec i es o he p ojec a e de ined and he equi emen s
o he s akeholde s a e ga he ed. The goals and expec ed ou comes o he p ojec a e iden i ied, and
he easibili y o he p ojec is assessed. I is impo an o e alua e he business si ua ion o unde s and
wha esou ces a e a ailable and wha esou ces will be equi ed. (Sch öe , K use, & Gómez, 2021).
Da a Unde s anding: he da a ha is o be used o he p ojec is collec ed and summa ized. Da a
explo a ion and isualiza ion a e pe o med o gain an unde s anding o he s uc u e and con en s o
he da a. (Sch öe , K use, & Gómez, 2021).
Da a P epa a ion: In his s ep, he da a is cleaned, ans o med, and in eg a ed o p epa e i o
analysis. This may in ol e emo ing duplica es, dealing wi h missing alues, and dealing wi h ou lie s.
To selec he app op ia e da a o analysis, speci ic c i e ia need o be es ablished o wha da a should
be included and excluded. I he da a is o low quali y, cleaning me hods can be employed o add ess
he issue. (Sch öe , K use, & Gómez, 2021).
Modeling: In his s ep, s a is ical and machine lea ning models a e applied o he da a o unco e
pa e ns and ela ionships. This s ep is whe e he da a is mined o insigh s.
In he da a modeling phase, he app op ia e modeling echnique is chosen and a es case and model
a e de eloped. The selec ion o he da a mining echnique o be used ypically depends on he business
p oblem and he da a a ailable. (Sch öe , K use, & Gómez, 2021).
E alua ion: In his s ep, he esul s o he modeling p ocess a e e alua ed o de e mine i hey mee
he goals and expec a ions o he p ojec . Du ing he e alua ion phase, he ou comes a e measu ed
agains he p ede e mined business goals. This equi es in e p e ing he esul s and iden i ying any
necessa y ollow-up ac ions. Addi ionally, i is impo an o conduc a gene al e iew o he en i e da a
mining p ocess. (Sch öe , K use, & Gómez, 2021).
Figu a 1 - Phases o CRISP-DM P ocess Model
Sou ce: (Ma inez-Plumed, F. e al., 2000)

6
Deploymen : In his inal s ep, he esul s o he p ojec a e deployed and he indings a e
communica ed o he s akeholde s. The esul s can be used o make decisions, sol e p oblems, o
suppo ope a ions.
2.4. MACHINE LEARNING
Machine lea ning is a subse o a i icial in elligence ha concen a es on de eloping algo i hms
capable o lea ning om da a and making p edic ions o decisions. These algo i hms ha e ound
nume ous applica ions, including image ecogni ion, na u al language p ocessing, and o ecas ing
cus ome beha iou . (James, Wi en, Has ie, & Tibshi ani, 2021).
The e a e ou main ypes o machine lea ning:
• Supe ised lea ning: a ype o machine lea ning in which he inpu da a used o ain he
algo i hm includes he co ec answe s, also known as labels. This ype o lea ning is o en
used o classi ica ion asks, such as dis inguishing be ween spam and non-spam emails in a
spam il e . The algo i hm is ed examples o bo h ypes o emails, along wi h hei
co esponding labels, and uses his in o ma ion o lea n how o accu a ely classi y new,
p e iously unseen emails. (Gé on, 2019)
• Unsupe ised lea ning: a ype o machine lea ning ha does no use labelled da a o aining.
Ins ead, i allows he algo i hm o ind pa e ns and ela ionships in he da a on i s own. In
o he wo ds, he algo i hm mus iden i y simila i ies and di e ences in he da a wi hou being
old wha hese pa e ns a e. Clus e ing is a common applica ion o unsupe ised lea ning,
whe e he algo i hm g oups simila da a poin s oge he based on hei cha ac e is ics. (Gé on,
2019)
• Semi-Supe ised lea ning: he model is ained using a mix u e o labelled and unlabeled da a.
The goal o semi-supe ised lea ning is o make he mos o he labelled da a while also aking
ad an age o he unlabeled da a o imp o e he model's pe o mance.
• Rein o cemen lea ning: he model lea ns h ough ial and e o . The model ecei es eedback
in he o m o ewa ds o penal ies o i s ac ions, and i uses his eedback o make decisions
abou which ac ions o ake in he u u e. Rein o cemen lea ning is a echnique used in
obo ics, gaming, and na iga ion, among o he a eas, o ain a model by allowing i o lea n
h ough ial and e o wi hou human guidance. (Gé on, 2019)
In he ene gy ield, machine lea ning is inc easingly being applied o imp o e he e iciency and
e ec i eness o ene gy p oduc ion, dis ibu ion, and consump ion.
The nex opics will discuss se e al machine lea ning algo i hms ha will be used o i is impo an o
know he concep o hem in his esea ch.
2.4.1. Decision T ees
Decision T ees a e a popula and widely used supe ised machine lea ning echnique. They a e used
o bo h eg ession and classi ica ion p oblems and hey a e ee-based models ha a e easy o
in e p e and implemen . Decision T ees a e used o de e mine a decision o p edic ion by b eaking
down a la ge da ase in o smalle and simple sub-p oblems un il eaching a conclusion.
7
As s a ed by (B eiman, F iedman, Olshen, & S one, 1984), A decision ee is a g aphical ep esen a ion
ha uses ee-like b anches o display di e en possible ou comes based on he alues o a ious
a ibu es o ea u es. Each b anch in he ee ep esen s a di e en pa h o choice, and each in e nal
node is a es o decision based on one o he a ibu es. The e minal nodes, o lea es, ep esen he
inal classi ica ion o p edic ion. An example o a g aphical ep esen a ion o a decision ee:
Cons uc ion o Decision T ee
The lea ning algo i hms o decision ees cons uc a ee by ecu si ely spli ing ins ances in o
subse s (James, Wi en, Has ie, & Tibshi ani, 2021). The spli s in he ee a e de e mined based on he
ea u e alues, wi h he objec i e o maximizing he educ ion in impu i y in he a ge a iable. The
selec ion o he me hod o di iding he da a a each le el o a decision ee a ec s bo h he shape o
he ee and he e ec i eness o he classi ica ion. Two popula me hods o his a e he Gini impu i y
and in o ma ion gain. (Has ie, Tibshi ani, & F iedman, 2008)
"Gini impu i y" is calcula ed as he sum o he squa ed p obabili ies o each class occu ing in he node,
and anges om 0 (pu e node) o 1 (equal dis ibu ion o classes). Suppose we ha e a da ase D wi h
samples om k di e en classes. The p obabili y o a sample belonging o a speci ic class i a a ce ain
node can be ep esen ed by 𝑝𝑖. The Gini Impu i y o D can be de ined as:
𝐺𝑖𝑛𝑖(𝐷)=1 − ∑𝑝𝑖2
𝑘
𝑖=1
The idea behind “in o ma ion gain” is ha he ea u e ha p o ides he mos in o ma ion abou he
a ge a iable, o he mos educ ion in he unce ain y o he a ge a iable, should be selec ed as
he spli ing ea u e. A ea u e wi h a high in o ma ion gain is conside ed o be mo e use ul o spli ing
he da a in o di e en classes o ca ego ies. In heo y, i ep esen s he educ ion in en opy achie ed
by pa i ioning he da a on a pa icula ea u e. En opy, on he o he hand, is a measu e o he amoun
o unce ain y o andomness in he da a. In decision ee models, i is used o e alua e he pu i y o a
se o examples. The o mula o calcula ing in o ma ion gain is as ollows:
In o ma ion Gain = En opy(pa en ) - [Weigh ed A e age]*En opy(child en)
Whe e En opy(pa en ) is he en opy o he pa en node, and En opy(child en) is he en opy o he
child nodes c ea ed by spli ing he pa en node. The weigh ed a e age is aken o e all child nodes,
Figu a 2- Elemen s o a Decision ee
8
wi h he weigh gi en by he p opo ion o examples in each child node. En opy is calcula ed as
ollows:
𝐸(𝑆)=∑𝑝𝑖
𝑐
𝑖=1 𝑙𝑜𝑔2𝑝𝑖
Whe e S ep esen s he sample in which we wan o calcula e he en opy and p is he p opo ion o
examples in a gi en class. The en opy is maximum when he da a is equally dis ibu ed among all
classes, and minimum when all he da a belongs o a single class.
2.4.2. G adien Boos ing
G adien boos ing is a popula algo i hm used in machine lea ning ha can sol e a ious classi ica ion
and eg ession p oblems. I wo ks by combining mul iple weak p edic ion models, o en decision ees,
in o a single s ong p edic ion model, known as an ensemble. This p ocess is done in a way ha
g adually imp o es he o e all pe o mance o he model. (F iedman, 2001)
In a g adien boos ing algo i hm, he base lea ne is ypically a decision ee. The algo i hm builds an
ini ial model, and hen in each subsequen i e a ion, a new decision ee is added o he ensemble ha
ies o co ec he e o s o he p e ious ees. As desc ibed by (Chen & Gues in, 2016), G adien
boos ing is an algo i hm ha c ea es a p edic ion model by adding mul iple weak p edic ion models
oge he in a s ep-by-s ep manne . This p ocess enables i o op imize any loss unc ion ha can be
di e en ia ed.
The main idea behind G adien Boos ing aims o minimize he e o o a model by adding new models
o he exis ing ones. This is done in a s ep-by-s ep manne , whe e each new model is added o minimize
he loss unc ion a each i e a ion. The me hod can be hough o as an app oxima ion o he s eepes
descen op imiza ion o a gene al loss unc ion. (Has ie, Tibshi ani, & F iedman, 2008).
One o he mos popula implemen a ions o g adien boos ing is XGBoos (Chen & Gues in, 2016),
which has become a de ac o s anda d o many machine lea ning asks. XGBoos p o ides a highly
e icien and scalable implemen a ion o g adien boos ing ha can handle la ge da ase s and high-
dimensional ea u e spaces.
2.4.3. XGBoos
XGBoos s ands o Ex eme G adien Boos ing and is an ad anced g adien boos ing algo i hm.
XGBoos has been a popula choice o machine lea ning and da a science asks because o i s high
le el o e iciency, obus ness, and accu acy, specially when handling p oblems ha in ol e la ge
da ase s and mul i-class classi ica ion.
In XGBoos , se e al decision ees a e combined oge he o make mo e accu a e p edic ions, which is
known as an ensemble lea ning me hod. A i s co e, XGBoos uses decision ees as weak lea ne s,
which a e combined o o m a s onge model. I c ea es hese decision ees i e a i ely, wi h each
new ee a emp ing o co ec he e o s made by he p e ious ee, his is known as g adien
boos ing. (Chen & Gues in, 2016)
9
XGBoos goes beyond he s anda d g adien boos ing amewo k by adding se e al enhancemen s,
including:
Regula iza ion:
To p e en he model om o e i ing, he XGBoos algo i hm uses egula iza ion echniques such as
weigh decay, L1 and L2 egula iza ion, and column subsampling.
L1 and L2 a e wo popula o ms o egula iza ion, a echnique used in machine lea ning o educe
o e i ing. L1 egula iza ion is also called Lasso egula iza ion and adds he absolu e alues o he
coe icien s o he loss unc ion. L2 egula iza ion is also called Ridge egula iza ion and adds he
squa es o he coe icien s o he loss unc ion. (Gé on, 2019).
L1 egula iza ion ends o p oduce spa se models whe e many o he coe icien s a e ze o, which can
be use ul o ea u e selec ion, while L2 egula iza ion p oduces models wi h smalle bu non-ze o
coe icien s. The s eng h o he egula iza ion penal y is con olled by a hype pa ame e , wi h highe
alues esul ing in mo e egula iza ion and smalle coe icien s.
Adjus ing he hype pa ame e s o L1 and L2 egula iza ion can help ind a balance be ween a model's
complexi y and i s capaci y o apply o new da a, esul ing in imp o ed pe o mance on he es se .
(Has ie, Tibshi ani, & F iedman, 2008).
Handling missing alues:
XGBoos can handle missing alues au oma ically. I can handle missing alues in he da a by lea ning
how o eplace hem wi h a de aul alue. I does his by lea ning a di ec ion o missing alue
impu a ion du ing he ee cons uc ion p ocess. A each node, he algo i hm checks i a ea u e is
missing a alue and decides on he di ec ion o ake based on he gain, which is calcula ed as he
di e ence be ween he pa en node's loss and he weigh ed sum o losses in he child nodes. This
p ocess helps o imp o e he pe o mance o he algo i hm on da a wi h missing alues. (Chen &
Gues in, 2016)
Pa allel p ocessing:
XGBoos can ake ad an age o mul iple CPUs o GPUs o ain and make p edic ions as e . XGBoos
is known o i s abili y o e icien ly use a ailable compu a ional esou ces, and one o i s main
s eng hs is i s abili y o pa allelize he cons uc ion and e alua ion o decision ees. This allows
XGBoos o ake ad an age o mul i-co e p ocesso s and dis ibu ed compu ing en i onmen s o
as e p ocessing. (Chen & Gues in, 2016)
Figu a 3 - L1 and L2 Regula iza ion o mula
10
Weigh ed ins ances:
XGBoos allows you o assign weigh s o ins ances, which can be use ul o imbalanced da ase s.
Weigh ing ins ances is a echnique ha can be used o balance he posi i e and nega i e weigh s o
imbalanced da ase s. By assigning weigh s o he ins ances, he algo i hm can ocus on he
unde ep esen ed class and imp o e i s pe o mance on he es se . (Chen & Gues in, 2016)
The inal model in XGBoos is a combina ion o all he decision ees c ea ed du ing aining. Each ee
ou pu s a p edic ion, and hese p edic ions a e combined using weigh ed a e aging. The weigh s
assigned o each ee depend on i s pe o mance du ing aining.
2.5. CLASSIFICATION MODEL EVALUATION
Classi ica ion me ics a e used o assess he accu acy and pe o mance o a classi ica ion model. These
me ics p o ide a way o measu e how well he model can co ec ly classi y ins ances in o di e en
classes based on he inpu ea u es. E alua ion o classi ica ion models is essen ial o build a eliable
and e ec i e model. Many o hese me ics a e based on he Con usion Ma ix, which summa izes he
ac ual and p edic ed classi ica ion ou comes, making i a use ul ool o analyze he model's
pe o mance. (G andini, Bagli, & Visani, 2020).
2.5.1. Con usion Ma ix
A con usion ma ix is a echnique used o e alua e he pe o mance o a classi ica ion model on a
da ase whe e he ou comes a e al eady known. I o e s a means o measu e he accu acy o he
model and is pa icula ly use ul when dealing wi h imbalanced ca ego ies. (Gé on, 2019)
A con usion ma ix displays he ac ual classes o he da a along one axis and he p edic ed classes along
he o he . Each cell o he ma ix shows he coun o ins ances ha we e classi ied as belonging o a
pa icula class. The cells along he diagonal co espond o he co ec ly classi ied ins ances, while he
cells o he diagonal co espond o he ins ances ha we e classi ied inco ec ly.
Ac ual alues
Posi i e (1)
Nega i e (0)
P edic ed Values
Posi i e (1)
TP
FP
Nega i e (0)
FN
TP
Tabela 1- Con usion Ma ix
In o he wo ds, is composed o ou main componen s: ue posi i es (TP), alse posi i es (FP), ue
nega i es (TN), and alse nega i es (FN). Acco ding o (Gé on, 2019), T ue posi i es e e o he
ins ances whe e he model co ec ly iden i ied he posi i e class, while alse posi i es a e he ins ances
whe e he model inco ec ly iden i ied a nega i e ins ance as posi i e. T ue nega i es e e o he
ins ances whe e he model co ec ly iden i ied he nega i e class, and alse nega i es a e he ins ances
whe e he model inco ec ly iden i ied a posi i e ins ance as nega i e.

11
2.5.2. Accu acy
Accu acy is a widely used me ic o e alua ing classi ica ion models ha measu es he numbe o
co ec p edic ions di ided by he o al numbe o p edic ions. I p o ides an o e all idea o he model's
pe o mance by compu ing he p opo ion o ue posi i es and ue nega i es in he p edic ed da a.
(Hossin & M.N, 2015)
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦= 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒+𝑇𝑟𝑢𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒
𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒+ 𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒+ 𝑇𝑟𝑢𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒+ 𝐹𝑎𝑙𝑠𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒
While accu acy is a use ul me ic o balanced da ase s, i can be misleading in he case o imbalanced
da ase s whe e he numbe o ins ances in each class is di e en . In such cases, a model ha always
p edic s he majo i y class will ha e high accu acy, bu will no necessa ily pe o m well in p edic ing
he mino i y class.
2.5.3. P ecision
P ecision is a classi ica ion me ic ha quan i ies he accu acy o posi i e p edic ions made by a model.
I is de ined by di iding he numbe o ue posi i es by he sum o ue posi i es and alse posi i es.
In o he wo ds, p ecision measu es he p opo ion o ue posi i e p edic ions ou o all he posi i e
p edic ions made by he model. (Gé on, 2019).
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛= 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒
𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒+ 𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒
A high p ecision means ha he model is p edic ing co ec ly a high p opo ion o posi i e cases.
P ecision is commonly used o e alua e he pe o mance o a model in si ua ions whe e he cos o a
alse posi i e is high, i.e., when he model alsely classi ies a nega i e ins ance as a posi i e class. In
such cases, he ocus is on minimizing he numbe o alse posi i es, and he p ecision me ic is used
o measu e he p opo ion o ue posi i es among he p edic ed posi i es.
2.5.4. Recall
Recall (o he wise known as sensi i i y o ue posi i e a e) is a measu e o he classi ie 's capabili y o
co ec ly iden i y posi i e occu ences. I is calcula ed by di iding he numbe o ue posi i e
p edic ions by he sum o ue posi i es and alse nega i es, ep esen ing he p opo ion o ac ual
posi i e ins ances ha a e co ec ly iden i ied by he classi ie . (Gé on, 2019).
This means ha ecall is a measu e o how well he classi ie is able o co ec ly iden i y ins ances o
he posi i e class (i.e., ue posi i es) among all ins ances ha ac ually belong o he posi i e class ( ue
posi i es + alse nega i es).
𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒
𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒+ 𝐹𝑎𝑙𝑠𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒
In si ua ions whe e he numbe o posi i e ins ances is signi ican ly lowe han he numbe o nega i e
ins ances, imbalanced da ase s o example, ecall is o en conside ed a mo e use ul me ic han
accu acy. This is because ecall measu es he abili y o he model o co ec ly iden i y impo an
ins ances, e en i hey a e a e, while accu acy only measu es he o e all numbe o co ec
p edic ions. (Fawce , 2006).
12
2.5.5. F1 Sco e
Acco ding o (Gé on, 2019), The F1 sco e is a me ic ha akes in o accoun bo h p ecision and ecall
o p o ide an o e all e alua ion o he pe o mance o a classi ica ion model.
𝐹1=2 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛×𝑟𝑒𝑐𝑎𝑙𝑙
𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑟𝑒𝑐𝑎𝑙𝑙
I is calcula ed as he ha monic mean o p ecision and ecall, and is use ul in si ua ions whe e bo h
me ics a e impo an . This me ic is pa icula ly help ul in scena ios whe e he e is an imbalance
be ween he numbe o posi i e and nega i e ins ances o when he cos o alse posi i es and alse
nega i es is no he same.
The F1 sco e anges om 0 o 1, wi h 1 being he bes possible sco e. A high F1 sco e indica es good
pe o mance in bo h p ecision and ecall.
2.6. DEBT MANAGEMENT IN THE ENERGY SECTOR
Deb managemen B2C in he ene gy sec o e e s o he s a egies and p ac ices ha companies in
he ene gy indus y use o manage hei deb s and inancial obliga ions owa ds hei cus ome s
(business- o-consume , o B2C).
Deb managemen is a c i ical aspec o c edi isk managemen o u ili ies and is essen ial o
main aining inancial s abili y and minimising de aul isk (Esgalhado, Higginson, Jacques, Ma ecsa, &
Selanda i, 2019). The epo highligh s he impo ance o e ec i e deb collec ion p ac ices, egula
communica ion wi h cus ome s, and he use o ad anced echnologies o au oma e and s eamline
deb managemen p ocesses.
One o he se e al goals o cus ome deb managemen in he ene gy sec o is o p o ide educa ion
and esou ces o help cus ome s be e unde s and hei ene gy usage and cos s. This can include
p o iding in o ma ion on ene gy e iciency and conse a ion on hei in oices, as well as assis ance
wi h iden i ying and accessing inancial assis ance p og ams.
Cus ome deb managemen in he ene gy sec o equi es also wo king wi h cus ome s o de elop
paymen plans o o he solu ions o manage and educe hei deb . This means nego ia ing ex ended
paymen plans, connec ing cus ome s wi h inancial assis ance p og ams, and e en p o iding
empo a y ene gy assis ance o help clien s keep hei ligh s on du ing di icul inancial imes. Ene gy
companies may also o e ene gy e iciency p og ams, on-bill inancing and o he solu ions o help
cus ome s educe hei ene gy usage and cos s. The pape om McKinsey also no es ha u ili ies can
bene i om using digi al echnologies and au oma ion o enhance hei deb managemen p ac ices,
including c edi sco ing, collec ions, and es uc u ing. This app oach can help educe ope a ing cos s
and inc ease he amoun o deb collec ed. Fu he mo e, digi al echnologies can p o ide signi ican
imp o emen s in cus ome expe ience h oughou he deb collec ion p ocess. Some u ili ies ha e
de eloped a comp ehensi e digi al pla o m o delinquen cus ome s, co e ing deb o e iew,
paymen suppo , enego ia ion, and cus ome suppo . (Esgalhado, Higginson, Jacques, Ma ecsa, &
Selanda i, 2019)
O e all, deb managemen in he ene gy sec o is a complex and dynamic ield ha equi es a
comp ehensi e unde s anding o inancial isk and consume beha io . Companies mus con inuously
13
inno a e and adap hei deb managemen s a egies o ensu e inancial s abili y and minimize
de aul isk.
14
3. METHODOLOGY
3.1. BUSINESS UNDERSTANDING
The i s s ep needed is o ga he all he impo an in o ma ion abou he clien da a a ailable on he
EDP da abase. The way o achie e ha success is by gaining an in-dep h unde s anding o he business
and i s ope a ions. Fo ha , CRISP-DM was he me hodology used.
I is clea ha he eam has a ew main obs acles o o e come in o de o imp o e he e ec i eness o
deb eco e y: ge an insigh in o who is mos likely o become in deb . And why? And a e knowing
ha , how can we imp o e ha ?
Objec i e: build a machine lea ning classi ica ion model ha iden i ies pa e ns ha can help p edic
which cus ome s a e mos likely o de aul on hei loans.
Scope:
- Iden i y he main a iables ha co ela e wi h a cus ome being a de aul e o paymen s
- Tes di e en models and hei pa ame e s o choose he one wi h he bes accu acy
- Lea n and assess pa e ns o each class
- Apply he model and help iden i y and mi iga e po en ial inancial isks
3.2. DATA UNDERSTANDING
3.2.1. Da a sou ces
I was collec all he impo an in o ma ion abou he clien da a a ailable on he EDP da abase, which
means all he a iables co esponding o he in o ma ion o he con ac (s) and consump ion o he
clien s. Un o ne lly, as EDP can no ha e access o in o ma ion such as pu chasing powe , annual
income o o he in o ma ion ha leads us he assume he p obabili y o a pe son becoming in deb , i
was collec ed as much in o ma ion a ailable as possible including hei demog aphic in o ma ion,
consump ion pa e ns, billing in o ma ion, and paymen his o y. The collec ion o he da a was based
on SAS ables connec ed o he da abase se e s o he company and pe o med on SAS En e p ise
Guide.
The da ase is composed o all ac i e clien s o he company, meaning, he ones ha had a leas one
ac i e con ac on elec ici y o gas. Tha makes a o al o 3M people. The ea u es used we e:
Column name
Meaning
NOME_PRIMEIRO
Name o he clien
COD_CAE
Type o consump ion: Domes ic o o he
COD_CATEGORIA_CLIENTE
Cus ome ca ego y: Pe son o o ganiza ion
COD_TIPO_CLIENTE
Clien ype: Domes ic o non-domes ic
COD_CLASSE_CONTA
COD_VALOR_PRESENTE
Accoun Class: Small business o esiden ial
Le el o p omo ional campaigns gi s
FLG_EDP_ONLINE
i i uses he App o no
21
Column
Meaning
0-14
numbe o people in he ange o age om 0 o 14 yea s old
15-24
numbe o people in he ange o age om 15 o 24 yea s old
25-64
numbe o people in he ange o age om 25 o 64 yea s old
65 e mais
numbe o people in he ange o age 65+
Homens
numbe o men
Mulhe es
numbe o women
Pode de comp a pe capi a
Pe capi a pu chasing powe
Tabela 5- New Demog aphic Fea u es and meaning
3.3.3. Missing alues
The bes app oach o dealing wi h missing alues depends on he speci ic da ase , he amoun o
missing da a, and he o e all goals o he analysis. A small numbe o missing alues we e iden i ied,
wi h a maximum o 3% o he o al and only on ca ego ical ea u es. To add ess his issue, he chosen
me hod was o ill hese missing alues wi h he mode, which means illing wi h he mos equen
alue in he espec i e ea u e column. This me hod was chosen because i is a s aigh o wa d way o
handle missing alues, especially o small amoun s o missing da a.
3.3.4. Ou lie de ec ion
Iden i ying and emo ing ou lie s is a c ucial s ep in da a p ep ocessing ha should be pe o med p io
o applying any s a is ical o machine lea ning echniques o he da a. Ou lie s can ha e a majo impac
on he ou come o analyses. The e o e, i is impo an o handle ou lie s be o e p oceeding wi h da a
analysis o ensu e accu a e and eliable esul s. (Ende lein, 1987).
A e a combina ion o isualiza ion h ough box plo s, knowledge o he business and he
In e qua ile Range (IQR), he di e ence be ween he 75 h and 25 h pe cen ile o he da a, meaning
he obse a ions ha a e less han Q1 - 1.5 * IQR o g ea e han Q3 + 1.5 * IQR a e conside ed ou lie s,
2.63% o he da a we e conside ed ou lie s and elimina ed.
3.3.5. Ca ego ical ea u es
To ep esen ca ego ical a iables, such as: COD_CAE, COD_CATEGORIA_CLIENTE,
COD_TIPO_CLIENTE, COD_CLASSE_CONTA, COD_VALOR_PRESENTE, FLG_EDP_ONLINE,
MEAN_o _FLG_TARIFA_SOCIAL, Flg_Tel, Flg_Email, Dis i o, MEAN_o _FLG_FATURA_ELETRONICA,
MEAN_o _FLG_DEB_DIRETO, Tipo Con a os, Mul ibanco, PONTOS DÍVIDA, gen, i was used he
common echnique o dummy a iables. This will help imp o e he pe o mance o he model, as hey
cap u e he ela ionship be ween he o iginal ca ego ical a iable and he a ge a iable.
3.3.6. Fea u e scaling
Fea u e scaling is an impo an p ep ocessing s ep in many machine lea ning algo i hms. I e e s o
he p ocess o ans o ming he a iables o ea u es in he da ase o a common scale. This helps o
ensu e ha each a iable o ea u e con ibu es equally o he model, a oiding bias and imp o ing he
pe o mance o he algo i hm. As no ed by (Gé on, 2019), machine lea ning algo i hms end o wo k
mo e e ec i ely o each op imal esul s mo e quickly when he ea u es ha e been s anda dized o
be on simila scales. Wi hou ea u e scaling, a iables o ea u es wi h la ge alues can domina e he
dis ances, leading o subop imal pe o mance. The e o e, he da ase was no malized using he

22
me hod o min-max scaling. Min-max scaling, also known as no maliza ion, scales he alues o a ange
be ween 0 and 1.
3.3.7. Ta ge dis ibu ion
Analysing he a ge dis ibu ion, i 's clea i 's an imbalanced a ge a iable, which could impac he
accu acy o he p edic ions. Ha ing an unbalanced da ase can ha m he accu acy o he p edic i e
model in se e al ways. This is because when he a ge classes a e imbalanced, he model will lea n o
p edic he majo i y class mo e o en, leading o a high accu acy sco e bu wi h a low p ecision and
ecall sco e. This means ha he model will no pe o m well in iden i ying he mino i y class, leading
o biased esul s. Addi ionally, an unbalanced da ase can lead o o e i ing on he majo i y class,
making i di icul o he model o gene alize o unseen da a. The e o e a e conside ing di e en
echniques o handle his issue, i was decided o s a i y he da a when spli ing i in o a aining and
es ing se , ha in ol es dis ibu ing he a ge a iable e enly among he wo se s, so ha he
p opo ion o a ge ca ego ies in bo h se s is simila o he p opo ion o a ge ca ego ies in he en i e
da ase . This was done o ensu e ha he aining se and es ing se a e ep esen a i es o he en i e
da ase and ha he e is a balance o he a ge ca ego ies in bo h se s.
Ano he echnique conside ed was unde sampling. I is a echnique ha in ol es educing he size o
he majo i y class o balance he a ge a iable. Al hough i is an e ec i e way o handle imbalanced
da ase s, i can also lead o a loss o impo an in o ma ion and may esul in a biased model. This was
he case since i was ound ha unde sampling esul ed in a signi ican educ ion in he size o he
da ase , which in u n impac ed he pe o mance o he p edic i e model.
The e o e, i was decided no o use unde sampling and ins ead use s a i ied sampling, as discussed
ea lie , o ensu e a balanced dis ibu ion o a ge a iables in he aining and es ing se s. This
app oach helped o p e en he o e - ep esen a ion o unde - ep esen a ion o any pa icula a ge
ca ego y, which could ha e esul ed in biased models o inaccu a e model pe o mance e alua ion.
3.3.8. Dealing wi h co ela ed ea u es
Co ela ion analysis is a aluable echnique in he p ocess o selec ing ele an ea u es o a a ge
a iable, as i can help iden i y p edic o s ha may ha e a s ong ela ionship wi h he a ge . (Kellehe ,
Mac, Aoi e, & A cy, 2015). In he con ex o da a analysis, co ela ion analysis helps iden i y ea u es
ha a e highly co ela ed be ween hemsel es and he a ge a iable. This in o ma ion can hen be
used o make in o med decisions abou which ea u es o include o exclude om a p edic i e model.
The co ela ion was calcula ed using a ious me hods, including Pea son's co ela ion coe icien and
Spea man's ank co ela ion coe icien . These me hods measu e he linea ela ionship be ween wo
a iables, wi h alues anging om -1 (s ong nega i e co ela ion) o 1 (s ong posi i e co ela ion).
Fea u es ha a e highly co ela ed wi h each o he (mul icollinea i y) o wi h he a ge (high
co ela ion) - co ela ion coe icien o mo e han 0.8 wi h each o he and mo e han 0.7 wi h he
a ge - we e emo ed du ing co ela ion analysis o a oid uns able and i ele an solu ions in he
model since high co ela ions be ween he ea u es and a ge can esul in o e i ing o edundan
in o ma ion.
23
3.3.9. Fea u e Selec ion
P epa ing da a o aining models o en in ol es ea u e selec ion, which is c ucial o imp o ing
model pe o mance and educing compu a ional ime, especially when dealing wi h la ge da ase s. By
educing he dimensionali y o he da a, ea u e selec ion helps o iden i y he ele an p edic o s and
a oid he p oblem o ha ing oo many po en ial p edic o s, which can make he modeling p ocess
mo e di icul and ime-consuming (Guyon & Elissee , 2003). The e o e, using p edic i e models o
ea u e selec ion can help in iden i ying he mos impo an ea u es ha ha e a s ong impac on he
a ge a iable. This leads o a mo e e icien and e ec i e model, as well as inc eased in e p e abili y.
The algo i hms expe imen ed o selec he ea u es o he model we e:
• Embedded me hods such as Decision ees, idge classi ie and lasso classi ie .
• Fil e -Based Me hods such as Co ela ion
• W appe me hods such as RFE ( ecu si e ea u e elimina ion), we e used o he ea u es
selec ed by he me hods abo e.
3.4. MODEL SELECTION AND EVALUATION
A e he da a is ully p epa ed and explo ed, i is eady o model selec ion. The i s s ep made was
o spli he da a in o ain and es , wi h he ain ep esen ing 70% and he emaining 30% he es
pa . To ensu e all classes a e being ep esen ed while he da a is being ained by he model, i was
applied 10- old c oss alida ion as well.
Since we a e dealing wi h a mul iclassi ica ion p oblem, a high numbe o ea u es, and he p esence
o bo h ca ego ical and nume ical ea u es, he ollowing models we e conside ed o aining:
Random Fo es Classi ie , Decision T ees Classi ie , Neu al Ne wo ks and XGBoos .
Gi en his esea ch is acing a e y la ge size da ase , which con ains millions o obse a ions and
nume ous ea u es, I need a model ha could handle such la ge and complex da a while s ill p oducing
accu a e p edic ions. The main obs acle he e in he selec ion and implemen a ion o he model is
compu a ional esou ces. Conside ing ha , and he balance be ween he weaknesses and s eng hs
o he model, I chose o use XGBoos p ima ily because o i s e iciency and scalabili y wi h
compu a ional esou ces.
S eng hs:
Weaknesses:
High p edic i e accu acy
Compu a ionally in ensi e, especially when
dealing wi h la ge da ase s o complex models.
Flexibili y: can be used o a wide ange o asks,
including ea u e selec ion, missing alue
impu a ion, and anomaly de ec ion.
Sensi i i y o hype pa ame e s, and uning hese
hype pa ame e s can be ime-consuming and
compu a ionally expensi e.
Scalabili y: highly scalable and can handle la ge
da ase s wi h millions o ea u es and millions o
samples.
In e p e abili y: XGBoos models can be di icul
o in e p e
Pa allel p ocessing, which can speed up aining
imes.
Po en ial o o e i ing
Regula iza ion, which can help p e en
o e i ing and imp o e gene aliza ion
pe o mance.
Tabela 6- S eng hs and Weaknesses o XGBoos
24
When dealing wi h imbalanced da ase s, accu acy may no be a eliable me ic as i only measu es he
p opo ion o co ec p edic ions. Ins ead, i is essen ial o use an e alua ion me ic ha conside s
bo h p ecision and ecall o assess he model's o e all pe o mance in a balanced manne . This
app oach is necessa y because adi ional me ics may no accoun o he alse posi i e and alse
nega i e a es, which can signi ican ly impac he e ec i eness o he model.
The F1 sco e is a popula me ic used in classi ica ion asks o measu e he balance be ween p ecision
and ecall. I is pa icula ly use ul when he da ase is imbalanced, meaning ha one class has a much
la ge numbe o samples han he o he s, which is he case.
P ecision and ecall a e wo impo an me ics in classi ica ion. P ecision measu es how many o he
p edic ed posi i e samples a e ac ually posi i e, while ecall measu es how many o he ac ual posi i e
samples a e co ec ly p edic ed. In o he wo ds, p ecision ocuses on he accu acy o posi i e
p edic ions, while ecall ocuses on he abili y o ind all posi i e samples.
The F1 sco e is he ha monic mean o p ecision and ecall. I is calcula ed as ollows:
𝐹1=2 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛×𝑟𝑒𝑐𝑎𝑙𝑙
𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑟𝑒𝑐𝑎𝑙𝑙
The F1 sco e anges om 0 o 1, wi h 1 being he bes possible sco e. A high F1 sco e indica es good
pe o mance in bo h p ecision and ecall. By choosing he F1 sco e, I am able o e alua e he ade-
o be ween p ecision and ecall and ge a single me ic ha summa izes he o e all pe o mance o
my model. This is pa icula ly impo an when he cos o alse posi i es and alse nega i es is no
equal, as i allows me o op imize he model o he pa icula applica ion a hand.
25
4. RESULTS AND DISCUSSION
This chap e p esen s he esul s o applying he model and discusses he implica ions o he indings.
I s a s by p o iding an o e iew o he selec ed model ollowed by he hype pa ame e uning
p ocess. Then, i epo s he inal pe o mance o he model wi h he selec ed e alua ion me ics and
compa ed i o baseline models. Finally, he limi a ions a e discussed and he po en ial u u e
di ec ions o his s udy a e men ioned.
4.1. EVALUATING FIRST RESULTS
The model was i s applied wi h he de aul pa ame e s, using di e en se s o a iables cons uc ed
du ing ea u e selec ion. I gene a ed he ollowing esul s:
F1 Sco e
Fea u es
T ain
Tes
Decision T ees
0.7836
0.7800
Co ela ions
0.7804
0.7772
Ridge Reg ession
0.7816
0.7788
Lasso Reg ession
0.7611
0.7590
RFE
0.7873
0.7839
Tabela 7- F1 Sco e esul s
The esul s show ha di e en ea u e selec ion me hods can ha e a signi ican impac on he
pe o mance o he XGBoos model. In his case, he RFE (Recu si e Fea u e Elimina ion) me hod
esul ed in he highes F1 sco es on bo h he aining and es ing da ase s. This indica es ha he
selec ed ea u es by his me hod we e he mos ele an and in o ma i e o he classi ica ion ask.
These ea u es we e he ollowing: Con a os A i os, Con a o a i o mais An igo,
MEAN_o _QTD_CONSUMO_ANUAL_ESP, Con a os Ence ados, An iguidade Clien e,
MEAN_o _Pack01, Va iedade Se iços, Tempo a é pagamen o, a g_sco e_clien ean ,
COD_CAE_o he , COD_CATEGORIA_CLIENTE_2, COD_VALOR_PRESENTE_A,
COD_VALOR_PRESENTE_B, COD_VALOR_PRESENTE_M, COD_VALOR_PRESENTE_MA,
COD_VALOR_PRESENTE_MB, MEAN_o _FLG_TARIFA_SOCIAL_1, Flg_Tel_1, Flg_Email_1,
Dis i o_BEJA, Dis i o_COIMBRA, Dis i o_SANTAREM, Dis i o_VISEU,
MEAN_o _FLG_DEB_DIRETO_1, Tipo Con a os_D, Mul ibanco_1.
On he o he hand, he Decision T ees and Co ela ions me hods p oduced F1 sco es ha we e lowe
han RFE, bu s ill ela i ely high. This sugges s ha he ea u es selec ed by hese me hods also ha e
some p edic i e powe , bu may no be as in o ma i e as hose selec ed by RFE. The Ridge and Lasso
Reg ession me hods p oduced he lowes F1 sco es, indica ing ha he selec ed ea u es may no be
as ele an o in o ma i e o he classi ica ion ask as hose selec ed by he o he me hods.
4.2. HYPERPARAMETER TUNING
The choice o hype pa ame e s can signi ican ly a ec he pe o mance o a model, and
hype pa ame e uning is he p ocess o inding he bes combina ion o hype pa ame e s ha esul s
in he bes model pe o mance and possibly less o e i ing.
26
Finding he bes hype pa ame e s is c ucial o c ea ing an accu a e machine lea ning algo i hm.
Howe e , he e is no uni e sal ule o selec ing hype pa ame e s, as di e en alues can ha e a
signi ican impac on he model's p edic i e abili y. Using c oss- alida ion, we can es ima e he es
e o o di e en hype pa ame e alues, and selec he bes one. Ye , wi h a la ge numbe o possible
alues o each hype pa ame e , inding he bes se o hype pa ame e s can be a daun ing ask. To
add ess his, echniques like g id sea ch, andomized sea ch, and Bayesian op imiza ion can be used
o na iga e he as hype pa ame e space and ind he bes combina ion o alues. (James, Wi en,
Has ie, & Tibshi ani, 2021)
Since he mos well-pe o med se o ea u es we e he RFE, i is he one ha is going o be used o
hype pa ame e uning. The i s s ep made o adjus pa ame e s was o analyse he lea ning a e and
he numbe o es ima o s o he model and how a ec s i s pe o mance.
To p e en o e i ing, i is used he lea ning a e, also called sh inkage, which educes he weigh s o
each ea u e du ing boos ing i e a ions p opo ionally o hei cu en weigh s. This helps achie e
be e gene aliza ion pe o mance, bu i may ake longe o con e ge. A smalle lea ning a e is
usually pai ed wi h a highe numbe o ees o compensa e o he slowe con e gence. (XGBoos
Documen a ion — xgboos 1.7.4 documen a ion)
This means ha a smalle lea ning a e may esul in longe aining imes since he model akes
smalle s eps o op imize he objec i e unc ion. On he o he hand, a la ge lea ning a e may esul
in as e aining imes, bu he model may con e ge o a subop imal solu ion and may equi e
addi ional s eps o imp o e he pe o mance, and clea ly, ha is shown in he igu e. The e o e, he
choice o he lea ning a e should be balanced be ween he success a e and pe o mance ime based
on he a ailable compu ing esou ces, meaning he main ocus o looking a he igu e is o ind he
op imal ade-o .
Figu a 9 - F1 Sco e s Lea ning a e

27
The numbe o es ima o s is a hype pa ame e in he XGBoos algo i hm ha e e s o he numbe o
decision ees o be used in he model. We can see in he igu e ha inc easing he numbe o
es ima o s can imp o e he pe o mance o he model, bu can also signi ican ly inc ease he aining
ime and memo y equi emen s. In gene al, inc easing he numbe o es ima o s beyond a ce ain
poin may lead o diminishing e u ns in pe o mance imp o emen , while signi ican ly inc easing he
ime and compu a ional esou ces equi ed o ain he model.
While choosing he me hod o use o explo e he hype pa ame e s, i was conside ed g id sea ch and
andomized sea ch. Gi en he la ge da ase , he big se o di e en pa ame e s o explo e, and he
limi ed compu a ional esou ces, i may no be easible o use g id sea ch, which exhaus i ely sea ches
o e all possible combina ions o hype pa ame e s, as i may ake a e y long ime o comple e.
The e o e, al e na i e me hods such as andomized sea ch, which andomly samples a subse o
hype pa ame e s, may be mo e p ac ical and e icien in his scena io. Acco ding o (Be gs a, Ca, &
Ca, 2012), hey ound ha andom sea ch can achie e he same le el o pe o mance o
hype pa ame e uning as g id sea ch, bu wi h ewe a emp s and less ime spen on compu a ion.
This is because he sea ch space o hype pa ame e s is complex and includes bo h con inuous and
disc e e pa ame e s, making he g id oo coa se o cap u e all he de ails. The s udy also ound ha
al hough andom sea ch may p oduce pe o mance es ima es wi h high a iance, inc easing he
numbe o a emp s can compensa e o his wi hou signi ican ly inc easing compu a ional cos s
compa ed o g id sea ch.
The e o e, i was decided o use andomized sea ch ins ead o g id sea ch o une he hype pa ame e s
o he model. A e unning 100 i e a ions o he sea ch, i was ound a se o pa ame e s ha we e
able o educe he o e i ing o he model, e en hough he sco e was no imp o ed. Tha lea es us
wi h he inal esul o 0.7850 on he aining se and 0.7835 on he es se .
Figu a 10 - F1 Sco e s Numbe o es ima o s
28
4.3. DISCUSS FINAL RESULTS
This chap e is dedica ed o in e p e ing he esul s and he p edic ions o he model. In he case o a
classi ica ion model, i is impo an o e alua e he pe o mance o he model in a meaning ul way,
aking in o accoun he speci ic equi emen s o he ask a hand. To do his, I ha e gene a ed a
classi ica ion epo ha p esen s p ecision, ecall, 1-sco e, and suppo o each class in he da ase .
This p o ides a de ailed o e iew o he model's pe o mance in each class, allowing us o e alua e i s
s eng hs and weaknesses. By conside ing hese me ics, we can gain insigh s in o how well he model
is able o iden i y di e en ypes o ins ances and we can make in o med decisions abou how o
imp o e i s pe o mance.
P ecision
Recall
F1-Sco e
Suppo
0
0,84
0,94
0,89
651070
[1-3]
0,65
0,46
0,54
218689
[4-6]
0,64
0,32
0,43
34352
[7-9]
0,57
0,09
0,15
2828
accu acy
0,8
906939
mac o a g
0,47
0,45
0,5
906939
weigh ed a g
0,78
0,8
0,78
906939
Tabela 8 - Classi ica ion epo
Looking a he esul s, we can see ha he model achie ed high p ecision (0.84) and ecall (0.94) o
class 0, showing ha i pe o med well in iden i ying ins ances o his class. Howe e , o classes 1-3
and 4-6, he p ecision and ecall a e lowe , indica ing ha he model did no pe o m as well o
co ec ly iden i y hese classes. Class 7-9 has he lowes p ecision, ecall, and F1-sco e, indica ing ha
he model s uggles he mos in iden i ying ins ances o his class.
The mac o-a e aged F1-sco e is 0.50, indica ing ha he pe o mance o he model is mode a e,
howe e , he mac o-a e age F1 sco e ea s all classes equally and calcula es he F1 sco e o each
class sepa a ely, and hen akes he unweigh ed a e age o hose sco es. This means ha he
con ibu ion o each class o he inal sco e is he same, ega dless o he numbe o samples in ha
class. The e o e, i is mo e app op ia e o use me ics ha conside he class dis ibu ion, such as he
weigh ed a e age o mic o-a e age F1 sco e. The weigh ed a e age F1 sco e calcula es he a e age F1
sco e, weigh ed by he numbe o samples in each class, gi ing mo e weigh o he pe o mance o he
classes wi h mo e samples.
As seen be o e we can obse e he weigh ed a e age F1-sco e is 0.78, indica ing he o e all
pe o mance o he model is good bu sugges ing ha he model pe o med be e on he la ge
Figu a 11 - Con usion Ma ix o Tes se
29
classes and s uggled wi h he smalle ones. To u he analyse he s eng hs and weaknesses o he
model i was cons uc ed a con usion ma ix.
A e cons uc ing and analyzing he con usion ma ix, he main obse a ions ha we e d awn a e:
As seen be o e i is clea ha class 0 has a bigge sample and i is co ec o assume ha he model
pe o med be e in his class by looking a he classi ica ion epo . We can u he con i m his by
looking a he con usion ma ix as well since i has he highes numbe o ue posi i es and he lowes
numbe o alse posi i es and alse nega i es. We can also see ha he class mos mis aken by class 0
is class 1-3. The same happens o class 1-3 in he sense ha is mos mis aken by class 0. Class 1-3 has
a la ge numbe o alse posi i es, which means ha many samples a e inco ec ly p edic ed o belong
o class 1-3 when hey ac ually belong o ano he class. On he o he hand, class 4-6 has a la ge numbe
o alse nega i es, which means ha many samples ha belong o class 4-6 a e inco ec ly p edic ed
o belong o ano he class. In class 7-9 we can also conclude ha is o en mis aken by class 4-6. Also
om class 7-9, we can see ha has he lowes numbe o ue posi i es, indica ing ha i is he mos
di icul class o p edic accu a ely. O e all, we can obse e om he con usion ma ix ha i is
common o a class o be misclassi ied as i s neighbou ing class. This pa e n can be expec ed, as he e
may no be a signi ican di e ence in he ea u es be ween hese neighbou ing classes. The e o e, a
classi ie may ha e a highe chance o making alse posi i e o alse nega i e e o s when dis inguishing
be ween hese classes.
30
5. CONCLUSION
This epo was based on he main p ojec de eloped du ing he in e nship and i explo ed he use o
machine lea ning algo i hms o p edic ing he isk o clien s in e ms o deb sco e a EDP Come cial.
The p ojec add esses he u gen need o e ec i e deb managemen in a compe i i e and g owing
ma ke , whe e da a-d i en echniques can signi ican ly imp o e deb eco e y. To achie e ha
demand i was c ea ed a p edic i e model ha is able o iden i y pa e ns in clien s based on hei
sco eca d, which measu es he quali y o a clien based on hei in ingemen his o y. The da a se was
cons i u ed o clien con ac s in o ma ion, se ices equi ed, paymen his o y and in o ma ion on he
places o consump ion. Be o e applying he model, he da a p epa a ion and ea u e enginee ing
p ocess we e c ucial o he pe o mance o he model, including adding ea u es o help he model
succeed. A e aining he model using XGBoos classi ie , was conduc ed hype pa ame e uning o
help educe he o e i ing o he model and he model ended up achie ing an o e all pe o mance o
0.78. I was concluded ha one o i s main weaknesses is co ec ly iden i ying he lowe classes, bu i
has good pe o mance in p edic ing he bigge classes.
Al hough mo e wo k is needed o op imize he model's pe o mance and expand he sample size o
he da a se and add mo e in o ma ion on he clien 's inancial p o ile, his p ojec shows p omising
esul s and p o ides a ounda ion o u u e wo k on deb eco e y managemen .
By being p oposed his p ojec he majo i y o his in e nship ga e me exposu e o he p ac ical
applica ion o da a-d i en echniques, speci ically he heo e ical and p ac ical knowledge acqui ed
du ing he Mas e 's p og am. This was achie ed by applying da a analysis, p ocess mining, and
machine lea ning o cus ome da a and he e o e i was possible o de elop a p edic i e model ha
can iden i y a - isk accoun s be o e hey all behind on paymen s, allowing he company o espond
p e en i ely and p oac i ely. The collec ion o he da a was made using SAS Guide bu all he es was
con inued on Jupy e No ebook, he main sys em used du ing he mas e in se e al cou ses. Du ing
he in e nship, I also had he oppo uni y o gain a weal h o new knowledge and insigh s, pa icula ly
in he a eas o business ope a ions and managemen o a la ge po olio o clien s in he ene gy
indus y. Th ough his expe ience, I was able o explo e new algo i hms and so wa e ools, which
helped me o be e unde s and and apply he concep s lea ned du ing my Mas e 's p og am.
O e all, he in e nship p o ided a aluable and ele an expe ience ha enhanced my unde s anding
o he ene gy indus y and solidi ied my knowledge o he subjec s al eady s udied in he mas e ’s
p og am.
37
Figu a 14- Pea son's co ela ion ma ix

38
Figu a 15 - Spea man's ank co ela ion ma ix
39
Figu a 16 - Fea u e Selec ion Decision T ees: Gini and En opy
40
Figu a 17- Fea u e Selec ion using Ridge Classi ie
41
Figu a 18- Fea u e Selec ion using Lasso Classi ie
1