Application of Predicted Models in Debt Management: Developing a Machine Learning Algorithm to Predict Customer Risk at EDP Comercial

i Mas e Deg ee P og am in Da a Science and Ad anced Analy ics Applica ion o P edic ed Models in Deb Managemen De eloping a Machine Lea ning Algo i hm o P edic Cus ome Risk a EDP Come cial Inês Pi es Melo In e nship Repo p esen ed as pa ial equi emen o ob aining he Mas e Deg ee P og am in Da a Science and Ad anced Analy ics NOVA In o ma ion Managemen School Ins i u o Supe io de Es a ís ica e Ges ão de In o mação Uni e sidade No a de Lisboa MDSAA i [ his page should no be included in he digi al e sion. I s pu pose is only o he p in ed e sion] Ti le: Sub i le: S uden ull name MDSAA 2022 i NOVA In o ma ion Managemen School Ins i u o Supe io de Es a ís ica e Ges ão de In o mação Uni e sidade No a de Lisboa APPLICATION OF PREDICTED MODELS IN DEBT MANAGEMENT by Inês Pi es Melo In e nship epo p esen ed as pa ial equi emen o ob aining he Mas e ’s deg ee in Ad anced Analy ics, wi h a Specializa ion in Da a Science Supe iso : Mau o Cas elli Feb ua y 2023 ii STATEMENT OF INTEGRITY I he eby decla e ha ing conduc ed his academic wo k wi h in eg i y. I con i m ha I ha e no used plagia ism o any o m o undue use o in o ma ion o alsi ica ion o esul s along he p ocess leading o i s elabo a ion. I u he decla e ha I ha e ully acknowledge he Rules o Conduc and Code o Hono om he NOVA In o ma ion Managemen School. Inês Melo Lisboa, 2023 iii ACKNOWLEDGEMENTS I would like o exp ess my g a i ude o my in e nship eam a EDP o hei unwa e ing suppo and posi i e spi i h oughou my in e nship, especially o Hélde Oli ei a, who was always a ailable o help me and belie ed in me om he s a . I also canno hank my amily enough, my pa en s and sis e , who ha e always suppo ed me and been a g ea pilla , as well as my iends and housema es, Ma iana and So ia, who pu up wi h me e e y day and ha e my ull hea . Las ly, I would like o hank my p o esso s whose guidance and men o ship ha e been ins umen al in shaping no only my knowledge bu also my cha ac e and o e all de elopmen as an indi idual. i ABSTRACT This epo is a esul o a nine-mon h in e nship a EDP Come cial whe e he main p ojec o esea ch was he applica ion o a i icial in elligence ools in he ield o deb managemen . Deb managemen in ol es a se o s a egies and p ocesses aimed a educing o elimina ing deb and he use o a i icial in elligence has shown g ea po en ial o op imize hese p ocesses and minimize he isk o deb o indi iduals and o ganiza ions. In e ms o moni o ing and con olling he c edi wo hiness and quali y o clien s, deb managemen has mainly been esponsi e and eac i e, a emp ing o eco e losses a e a clien has become delinquen . The e is a gap in he knowledge o how o p oac i ely iden i y a - isk accoun s be o e hey all behind on paymen s. To a oid he cons an eac i e esponse in he ield, i was de eloped a machine-lea ning algo i hm ha p edic s he isk o a clien becoming in deb by analyzing hei sco eca d, which measu es he quali y o a clien based on hei in ingemen his o y. A e p ep ocessing he da a, XGBoos was implemen ed o a da ase o 3M cus ome s wi h a leas one ac i e con ac on EDP, on elec ici y o gas. Hype pa ame e uning was pe o med on he model o each an F1 sco e o 0.7850 on he aining se and 0.7835 on he es se . The esul s we e discussed and based on hose, ecommenda ions and imp o emen s we e also iden i ied. KEYWORDS Deb Managemen ; Machine Lea ning; Sco eca d P edic ion; A i icial in elligence INDEX 1. In oduc ion .................................................................................................................. 1 1.1. Company O e iew ............................................................................................... 1 1.2. The p ojec ............................................................................................................. 2 2. Li e a u e e iew .......................................................................................................... 3 2.1. A i icial in elligence (AI) ....................................................................................... 3 2.2. Da a Mining ........................................................................................................... 3 2.3. CRISP-DM ............................................................................................................... 4 2.4. Machine Lea ning .................................................................................................. 6 2.4.1. Decision T ees ................................................................................................ 6 2.4.2. G adien Boos ing ........................................................................................... 8 2.4.3. XGBoos .......................................................................................................... 8 2.5. Classi ica ion Model E alua ion .......................................................................... 10 2.5.1. Con usion Ma ix .......................................................................................... 10 2.5.2. Accu acy ....................................................................................................... 11 2.5.3. P ecision ....................................................................................................... 11 2.5.4. Recall ............................................................................................................ 11 2.5.5. F1 Sco e ........................................................................................................ 12 2.6. Deb managemen in he ene gy sec o ............................................................. 12 3. Me hodology .............................................................................................................. 14 3.1. Business unde s anding ....................................................................................... 14 3.2. Da a Unde s anding ............................................................................................ 14 3.2.1. Da a sou ces ................................................................................................. 14 3.2.2. Main Findings ............................................................................................... 16 3.2.3. Ta ge analysis .............................................................................................. 19 3.3. Da a P epa a ion ................................................................................................. 20 3.3.1. Fea u e enginee ing ..................................................................................... 20 3.3.2. Adding demog aphic ea u es ...................................................................... 20 3.3.3. Missing alues .............................................................................................. 21 3.3.4. Ou lie de ec ion .......................................................................................... 21 3.3.5. Ca ego ical ea u es ..................................................................................... 21 3.3.6. Fea u e scaling.............................................................................................. 21 3.3.7. Ta ge dis ibu ion ........................................................................................ 22 3.3.8. Dealing wi h co ela ed ea u es.................................................................. 22 i 3.3.9. Fea u e Selec ion .......................................................................................... 23 3.4. Model Selec ion And E alua ion ......................................................................... 23 4. Resul s and discussion ................................................................................................ 25 4.1. E alua ing i s esul s ......................................................................................... 25 4.2. Hype pa ame e uning ....................................................................................... 25 4.3. Discuss inal esul s ............................................................................................. 28 5. Conclusion .................................................................................................................. 30 6. Limi a ions and ecommenda ions o u u e wo ks ................................................. 31 7. Re e ences .................................................................................................................. 32 Appendix.......................................................................................................................... 34 ii LIST OF FIGURES Figu a 1 - Phases o CRISP-DM P ocess Model Sou ce: (Ma inez-Plumed, F. e al., 2000) ....... 5 Figu a 2- Elemen s o a Decision ee ......................................................................................... 7 Figu a 3 - L1 and L2 Regula iza ion o mula .............................................................................. 9 Figu a 4 - A e age expe ed annual Consump ion by ype o consump ion ............................. 16 Figu a 5 - Pie Cha o Types o Consump ion .......................................................................... 16 Figu a 6 - Clien dis ibu ion o yea s o consump ion ............................................................. 16 Figu a 7- Dis ibu ion o ypes o Con ac s ............................................................................. 17 Figu a 8 - Ta ge dis ibu ion ................................................................................................... 19 Figu a 9 - F1 Sco e s Lea ning a e ......................................................................................... 26 Figu a 10 - F1 Sco e s Numbe o es ima o s ......................................................................... 27 Figu a 11 - Con usion Ma ix o Tes se .................................................................................. 28 Figu a 12- Ou lie s Visualiza ion .............................................................................................. 35 Figu a 13- Oulie s Visualiza ions pa 2 ................................................................................... 36 Figu a 14- Pea son's co ela ion ma ix ................................................................................... 37 Figu a 15 - Spea man's ank co ela ion ma ix ...................................................................... 38 Figu a 16 - Fea u e Selec ion Decision T ees: Gini and En opy ............................................... 39 Figu a 17- Fea u e Selec ion using Ridge Classi ie .................................................................. 40 Figu a 18- Fea u e Selec ion using Lasso Classi ie .................................................................. 41 5 ha he CRISP-DM app oach is lexible, allowing o modi ica ions o be made o i he speci ic needs o a p ojec . As i is shown in he igu e, he six s eps o he CRISP-DM p ocess a e as ollows: Business Unde s anding: In his s ep, he objec i es o he p ojec a e de ined and he equi emen s o he s akeholde s a e ga he ed. The goals and expec ed ou comes o he p ojec a e iden i ied, and he easibili y o he p ojec is assessed. I is impo an o e alua e he business si ua ion o unde s and wha esou ces a e a ailable and wha esou ces will be equi ed. (Sch öe , K use, & Gómez, 2021). Da a Unde s anding: he da a ha is o be used o he p ojec is collec ed and summa ized. Da a explo a ion and isualiza ion a e pe o med o gain an unde s anding o he s uc u e and con en s o he da a. (Sch öe , K use, & Gómez, 2021). Da a P epa a ion: In his s ep, he da a is cleaned, ans o med, and in eg a ed o p epa e i o analysis. This may in ol e emo ing duplica es, dealing wi h missing alues, and dealing wi h ou lie s. To selec he app op ia e da a o analysis, speci ic c i e ia need o be es ablished o wha da a should be included and excluded. I he da a is o low quali y, cleaning me hods can be employed o add ess he issue. (Sch öe , K use, & Gómez, 2021). Modeling: In his s ep, s a is ical and machine lea ning models a e applied o he da a o unco e pa e ns and ela ionships. This s ep is whe e he da a is mined o insigh s. In he da a modeling phase, he app op ia e modeling echnique is chosen and a es case and model a e de eloped. The selec ion o he da a mining echnique o be used ypically depends on he business p oblem and he da a a ailable. (Sch öe , K use, & Gómez, 2021). E alua ion: In his s ep, he esul s o he modeling p ocess a e e alua ed o de e mine i hey mee he goals and expec a ions o he p ojec . Du ing he e alua ion phase, he ou comes a e measu ed agains he p ede e mined business goals. This equi es in e p e ing he esul s and iden i ying any necessa y ollow-up ac ions. Addi ionally, i is impo an o conduc a gene al e iew o he en i e da a mining p ocess. (Sch öe , K use, & Gómez, 2021). Figu a 1 - Phases o CRISP-DM P ocess Model Sou ce: (Ma inez-Plumed, F. e al., 2000) 6 Deploymen : In his inal s ep, he esul s o he p ojec a e deployed and he indings a e communica ed o he s akeholde s. The esul s can be used o make decisions, sol e p oblems, o suppo ope a ions. 2.4. MACHINE LEARNING Machine lea ning is a subse o a i icial in elligence ha concen a es on de eloping algo i hms capable o lea ning om da a and making p edic ions o decisions. These algo i hms ha e ound nume ous applica ions, including image ecogni ion, na u al language p ocessing, and o ecas ing cus ome beha iou . (James, Wi en, Has ie, & Tibshi ani, 2021). The e a e ou main ypes o machine lea ning: • Supe ised lea ning: a ype o machine lea ning in which he inpu da a used o ain he algo i hm includes he co ec answe s, also known as labels. This ype o lea ning is o en used o classi ica ion asks, such as dis inguishing be ween spam and non-spam emails in a spam il e . The algo i hm is ed examples o bo h ypes o emails, along wi h hei co esponding labels, and uses his in o ma ion o lea n how o accu a ely classi y new, p e iously unseen emails. (Gé on, 2019) • Unsupe ised lea ning: a ype o machine lea ning ha does no use labelled da a o aining. Ins ead, i allows he algo i hm o ind pa e ns and ela ionships in he da a on i s own. In o he wo ds, he algo i hm mus iden i y simila i ies and di e ences in he da a wi hou being old wha hese pa e ns a e. Clus e ing is a common applica ion o unsupe ised lea ning, whe e he algo i hm g oups simila da a poin s oge he based on hei cha ac e is ics. (Gé on, 2019) • Semi-Supe ised lea ning: he model is ained using a mix u e o labelled and unlabeled da a. The goal o semi-supe ised lea ning is o make he mos o he labelled da a while also aking ad an age o he unlabeled da a o imp o e he model's pe o mance. • Rein o cemen lea ning: he model lea ns h ough ial and e o . The model ecei es eedback in he o m o ewa ds o penal ies o i s ac ions, and i uses his eedback o make decisions abou which ac ions o ake in he u u e. Rein o cemen lea ning is a echnique used in obo ics, gaming, and na iga ion, among o he a eas, o ain a model by allowing i o lea n h ough ial and e o wi hou human guidance. (Gé on, 2019) In he ene gy ield, machine lea ning is inc easingly being applied o imp o e he e iciency and e ec i eness o ene gy p oduc ion, dis ibu ion, and consump ion. The nex opics will discuss se e al machine lea ning algo i hms ha will be used o i is impo an o know he concep o hem in his esea ch. 2.4.1. Decision T ees Decision T ees a e a popula and widely used supe ised machine lea ning echnique. They a e used o bo h eg ession and classi ica ion p oblems and hey a e ee-based models ha a e easy o in e p e and implemen . Decision T ees a e used o de e mine a decision o p edic ion by b eaking down a la ge da ase in o smalle and simple sub-p oblems un il eaching a conclusion. 7 As s a ed by (B eiman, F iedman, Olshen, & S one, 1984), A decision ee is a g aphical ep esen a ion ha uses ee-like b anches o display di e en possible ou comes based on he alues o a ious a ibu es o ea u es. Each b anch in he ee ep esen s a di e en pa h o choice, and each in e nal node is a es o decision based on one o he a ibu es. The e minal nodes, o lea es, ep esen he inal classi ica ion o p edic ion. An example o a g aphical ep esen a ion o a decision ee: Cons uc ion o Decision T ee The lea ning algo i hms o decision ees cons uc a ee by ecu si ely spli ing ins ances in o subse s (James, Wi en, Has ie, & Tibshi ani, 2021). The spli s in he ee a e de e mined based on he ea u e alues, wi h he objec i e o maximizing he educ ion in impu i y in he a ge a iable. The selec ion o he me hod o di iding he da a a each le el o a decision ee a ec s bo h he shape o he ee and he e ec i eness o he classi ica ion. Two popula me hods o his a e he Gini impu i y and in o ma ion gain. (Has ie, Tibshi ani, & F iedman, 2008) "Gini impu i y" is calcula ed as he sum o he squa ed p obabili ies o each class occu ing in he node, and anges om 0 (pu e node) o 1 (equal dis ibu ion o classes). Suppose we ha e a da ase D wi h samples om k di e en classes. The p obabili y o a sample belonging o a speci ic class i a a ce ain node can be ep esen ed by 𝑝𝑖. The Gini Impu i y o D can be de ined as: 𝐺𝑖𝑛𝑖(𝐷)=1 − ∑𝑝𝑖2 𝑘 𝑖=1 The idea behind “in o ma ion gain” is ha he ea u e ha p o ides he mos in o ma ion abou he a ge a iable, o he mos educ ion in he unce ain y o he a ge a iable, should be selec ed as he spli ing ea u e. A ea u e wi h a high in o ma ion gain is conside ed o be mo e use ul o spli ing he da a in o di e en classes o ca ego ies. In heo y, i ep esen s he educ ion in en opy achie ed by pa i ioning he da a on a pa icula ea u e. En opy, on he o he hand, is a measu e o he amoun o unce ain y o andomness in he da a. In decision ee models, i is used o e alua e he pu i y o a se o examples. The o mula o calcula ing in o ma ion gain is as ollows: In o ma ion Gain = En opy(pa en ) - [Weigh ed A e age]*En opy(child en) Whe e En opy(pa en ) is he en opy o he pa en node, and En opy(child en) is he en opy o he child nodes c ea ed by spli ing he pa en node. The weigh ed a e age is aken o e all child nodes, Figu a 2- Elemen s o a Decision ee 8 wi h he weigh gi en by he p opo ion o examples in each child node. En opy is calcula ed as ollows: 𝐸(𝑆)=∑𝑝𝑖 𝑐 𝑖=1 𝑙𝑜𝑔2𝑝𝑖 Whe e S ep esen s he sample in which we wan o calcula e he en opy and p is he p opo ion o examples in a gi en class. The en opy is maximum when he da a is equally dis ibu ed among all classes, and minimum when all he da a belongs o a single class. 2.4.2. G adien Boos ing G adien boos ing is a popula algo i hm used in machine lea ning ha can sol e a ious classi ica ion and eg ession p oblems. I wo ks by combining mul iple weak p edic ion models, o en decision ees, in o a single s ong p edic ion model, known as an ensemble. This p ocess is done in a way ha g adually imp o es he o e all pe o mance o he model. (F iedman, 2001) In a g adien boos ing algo i hm, he base lea ne is ypically a decision ee. The algo i hm builds an ini ial model, and hen in each subsequen i e a ion, a new decision ee is added o he ensemble ha ies o co ec he e o s o he p e ious ees. As desc ibed by (Chen & Gues in, 2016), G adien boos ing is an algo i hm ha c ea es a p edic ion model by adding mul iple weak p edic ion models oge he in a s ep-by-s ep manne . This p ocess enables i o op imize any loss unc ion ha can be di e en ia ed. The main idea behind G adien Boos ing aims o minimize he e o o a model by adding new models o he exis ing ones. This is done in a s ep-by-s ep manne , whe e each new model is added o minimize he loss unc ion a each i e a ion. The me hod can be hough o as an app oxima ion o he s eepes descen op imiza ion o a gene al loss unc ion. (Has ie, Tibshi ani, & F iedman, 2008). One o he mos popula implemen a ions o g adien boos ing is XGBoos (Chen & Gues in, 2016), which has become a de ac o s anda d o many machine lea ning asks. XGBoos p o ides a highly e icien and scalable implemen a ion o g adien boos ing ha can handle la ge da ase s and high- dimensional ea u e spaces. 2.4.3. XGBoos XGBoos s ands o Ex eme G adien Boos ing and is an ad anced g adien boos ing algo i hm. XGBoos has been a popula choice o machine lea ning and da a science asks because o i s high le el o e iciency, obus ness, and accu acy, specially when handling p oblems ha in ol e la ge da ase s and mul i-class classi ica ion. In XGBoos , se e al decision ees a e combined oge he o make mo e accu a e p edic ions, which is known as an ensemble lea ning me hod. A i s co e, XGBoos uses decision ees as weak lea ne s, which a e combined o o m a s onge model. I c ea es hese decision ees i e a i ely, wi h each new ee a emp ing o co ec he e o s made by he p e ious ee, his is known as g adien boos ing. (Chen & Gues in, 2016) 9 XGBoos goes beyond he s anda d g adien boos ing amewo k by adding se e al enhancemen s, including: Regula iza ion: To p e en he model om o e i ing, he XGBoos algo i hm uses egula iza ion echniques such as weigh decay, L1 and L2 egula iza ion, and column subsampling. L1 and L2 a e wo popula o ms o egula iza ion, a echnique used in machine lea ning o educe o e i ing. L1 egula iza ion is also called Lasso egula iza ion and adds he absolu e alues o he coe icien s o he loss unc ion. L2 egula iza ion is also called Ridge egula iza ion and adds he squa es o he coe icien s o he loss unc ion. (Gé on, 2019). L1 egula iza ion ends o p oduce spa se models whe e many o he coe icien s a e ze o, which can be use ul o ea u e selec ion, while L2 egula iza ion p oduces models wi h smalle bu non-ze o coe icien s. The s eng h o he egula iza ion penal y is con olled by a hype pa ame e , wi h highe alues esul ing in mo e egula iza ion and smalle coe icien s. Adjus ing he hype pa ame e s o L1 and L2 egula iza ion can help ind a balance be ween a model's complexi y and i s capaci y o apply o new da a, esul ing in imp o ed pe o mance on he es se . (Has ie, Tibshi ani, & F iedman, 2008). Handling missing alues: XGBoos can handle missing alues au oma ically. I can handle missing alues in he da a by lea ning how o eplace hem wi h a de aul alue. I does his by lea ning a di ec ion o missing alue impu a ion du ing he ee cons uc ion p ocess. A each node, he algo i hm checks i a ea u e is missing a alue and decides on he di ec ion o ake based on he gain, which is calcula ed as he di e ence be ween he pa en node's loss and he weigh ed sum o losses in he child nodes. This p ocess helps o imp o e he pe o mance o he algo i hm on da a wi h missing alues. (Chen & Gues in, 2016) Pa allel p ocessing: XGBoos can ake ad an age o mul iple CPUs o GPUs o ain and make p edic ions as e . XGBoos is known o i s abili y o e icien ly use a ailable compu a ional esou ces, and one o i s main s eng hs is i s abili y o pa allelize he cons uc ion and e alua ion o decision ees. This allows XGBoos o ake ad an age o mul i-co e p ocesso s and dis ibu ed compu ing en i onmen s o as e p ocessing. (Chen & Gues in, 2016) Figu a 3 - L1 and L2 Regula iza ion o mula 10 Weigh ed ins ances: XGBoos allows you o assign weigh s o ins ances, which can be use ul o imbalanced da ase s. Weigh ing ins ances is a echnique ha can be used o balance he posi i e and nega i e weigh s o imbalanced da ase s. By assigning weigh s o he ins ances, he algo i hm can ocus on he unde ep esen ed class and imp o e i s pe o mance on he es se . (Chen & Gues in, 2016) The inal model in XGBoos is a combina ion o all he decision ees c ea ed du ing aining. Each ee ou pu s a p edic ion, and hese p edic ions a e combined using weigh ed a e aging. The weigh s assigned o each ee depend on i s pe o mance du ing aining. 2.5. CLASSIFICATION MODEL EVALUATION Classi ica ion me ics a e used o assess he accu acy and pe o mance o a classi ica ion model. These me ics p o ide a way o measu e how well he model can co ec ly classi y ins ances in o di e en classes based on he inpu ea u es. E alua ion o classi ica ion models is essen ial o build a eliable and e ec i e model. Many o hese me ics a e based on he Con usion Ma ix, which summa izes he ac ual and p edic ed classi ica ion ou comes, making i a use ul ool o analyze he model's pe o mance. (G andini, Bagli, & Visani, 2020). 2.5.1. Con usion Ma ix A con usion ma ix is a echnique used o e alua e he pe o mance o a classi ica ion model on a da ase whe e he ou comes a e al eady known. I o e s a means o measu e he accu acy o he model and is pa icula ly use ul when dealing wi h imbalanced ca ego ies. (Gé on, 2019) A con usion ma ix displays he ac ual classes o he da a along one axis and he p edic ed classes along he o he . Each cell o he ma ix shows he coun o ins ances ha we e classi ied as belonging o a pa icula class. The cells along he diagonal co espond o he co ec ly classi ied ins ances, while he cells o he diagonal co espond o he ins ances ha we e classi ied inco ec ly. Ac ual alues Posi i e (1) Nega i e (0) P edic ed Values Posi i e (1) TP FP Nega i e (0) FN TP Tabela 1- Con usion Ma ix In o he wo ds, is composed o ou main componen s: ue posi i es (TP), alse posi i es (FP), ue nega i es (TN), and alse nega i es (FN). Acco ding o (Gé on, 2019), T ue posi i es e e o he ins ances whe e he model co ec ly iden i ied he posi i e class, while alse posi i es a e he ins ances whe e he model inco ec ly iden i ied a nega i e ins ance as posi i e. T ue nega i es e e o he ins ances whe e he model co ec ly iden i ied he nega i e class, and alse nega i es a e he ins ances whe e he model inco ec ly iden i ied a posi i e ins ance as nega i e. 11 2.5.2. Accu acy Accu acy is a widely used me ic o e alua ing classi ica ion models ha measu es he numbe o co ec p edic ions di ided by he o al numbe o p edic ions. I p o ides an o e all idea o he model's pe o mance by compu ing he p opo ion o ue posi i es and ue nega i es in he p edic ed da a. (Hossin & M.N, 2015) 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦= 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒+𝑇𝑟𝑢𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒+ 𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒+ 𝑇𝑟𝑢𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒+ 𝐹𝑎𝑙𝑠𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒 While accu acy is a use ul me ic o balanced da ase s, i can be misleading in he case o imbalanced da ase s whe e he numbe o ins ances in each class is di e en . In such cases, a model ha always p edic s he majo i y class will ha e high accu acy, bu will no necessa ily pe o m well in p edic ing he mino i y class. 2.5.3. P ecision P ecision is a classi ica ion me ic ha quan i ies he accu acy o posi i e p edic ions made by a model. I is de ined by di iding he numbe o ue posi i es by he sum o ue posi i es and alse posi i es. In o he wo ds, p ecision measu es he p opo ion o ue posi i e p edic ions ou o all he posi i e p edic ions made by he model. (Gé on, 2019). 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛= 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒+ 𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 A high p ecision means ha he model is p edic ing co ec ly a high p opo ion o posi i e cases. P ecision is commonly used o e alua e he pe o mance o a model in si ua ions whe e he cos o a alse posi i e is high, i.e., when he model alsely classi ies a nega i e ins ance as a posi i e class. In such cases, he ocus is on minimizing he numbe o alse posi i es, and he p ecision me ic is used o measu e he p opo ion o ue posi i es among he p edic ed posi i es. 2.5.4. Recall Recall (o he wise known as sensi i i y o ue posi i e a e) is a measu e o he classi ie 's capabili y o co ec ly iden i y posi i e occu ences. I is calcula ed by di iding he numbe o ue posi i e p edic ions by he sum o ue posi i es and alse nega i es, ep esen ing he p opo ion o ac ual posi i e ins ances ha a e co ec ly iden i ied by he classi ie . (Gé on, 2019). This means ha ecall is a measu e o how well he classi ie is able o co ec ly iden i y ins ances o he posi i e class (i.e., ue posi i es) among all ins ances ha ac ually belong o he posi i e class ( ue posi i es + alse nega i es). 𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒+ 𝐹𝑎𝑙𝑠𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒 In si ua ions whe e he numbe o posi i e ins ances is signi ican ly lowe han he numbe o nega i e ins ances, imbalanced da ase s o example, ecall is o en conside ed a mo e use ul me ic han accu acy. This is because ecall measu es he abili y o he model o co ec ly iden i y impo an ins ances, e en i hey a e a e, while accu acy only measu es he o e all numbe o co ec p edic ions. (Fawce , 2006). 12 2.5.5. F1 Sco e Acco ding o (Gé on, 2019), The F1 sco e is a me ic ha akes in o accoun bo h p ecision and ecall o p o ide an o e all e alua ion o he pe o mance o a classi ica ion model. 𝐹1=2 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛×𝑟𝑒𝑐𝑎𝑙𝑙 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑟𝑒𝑐𝑎𝑙𝑙 I is calcula ed as he ha monic mean o p ecision and ecall, and is use ul in si ua ions whe e bo h me ics a e impo an . This me ic is pa icula ly help ul in scena ios whe e he e is an imbalance be ween he numbe o posi i e and nega i e ins ances o when he cos o alse posi i es and alse nega i es is no he same. The F1 sco e anges om 0 o 1, wi h 1 being he bes possible sco e. A high F1 sco e indica es good pe o mance in bo h p ecision and ecall. 2.6. DEBT MANAGEMENT IN THE ENERGY SECTOR Deb managemen B2C in he ene gy sec o e e s o he s a egies and p ac ices ha companies in he ene gy indus y use o manage hei deb s and inancial obliga ions owa ds hei cus ome s (business- o-consume , o B2C). Deb managemen is a c i ical aspec o c edi isk managemen o u ili ies and is essen ial o main aining inancial s abili y and minimising de aul isk (Esgalhado, Higginson, Jacques, Ma ecsa, & Selanda i, 2019). The epo highligh s he impo ance o e ec i e deb collec ion p ac ices, egula communica ion wi h cus ome s, and he use o ad anced echnologies o au oma e and s eamline deb managemen p ocesses. One o he se e al goals o cus ome deb managemen in he ene gy sec o is o p o ide educa ion and esou ces o help cus ome s be e unde s and hei ene gy usage and cos s. This can include p o iding in o ma ion on ene gy e iciency and conse a ion on hei in oices, as well as assis ance wi h iden i ying and accessing inancial assis ance p og ams. Cus ome deb managemen in he ene gy sec o equi es also wo king wi h cus ome s o de elop paymen plans o o he solu ions o manage and educe hei deb . This means nego ia ing ex ended paymen plans, connec ing cus ome s wi h inancial assis ance p og ams, and e en p o iding empo a y ene gy assis ance o help clien s keep hei ligh s on du ing di icul inancial imes. Ene gy companies may also o e ene gy e iciency p og ams, on-bill inancing and o he solu ions o help cus ome s educe hei ene gy usage and cos s. The pape om McKinsey also no es ha u ili ies can bene i om using digi al echnologies and au oma ion o enhance hei deb managemen p ac ices, including c edi sco ing, collec ions, and es uc u ing. This app oach can help educe ope a ing cos s and inc ease he amoun o deb collec ed. Fu he mo e, digi al echnologies can p o ide signi ican imp o emen s in cus ome expe ience h oughou he deb collec ion p ocess. Some u ili ies ha e de eloped a comp ehensi e digi al pla o m o delinquen cus ome s, co e ing deb o e iew, paymen suppo , enego ia ion, and cus ome suppo . (Esgalhado, Higginson, Jacques, Ma ecsa, & Selanda i, 2019) O e all, deb managemen in he ene gy sec o is a complex and dynamic ield ha equi es a comp ehensi e unde s anding o inancial isk and consume beha io . Companies mus con inuously 13 inno a e and adap hei deb managemen s a egies o ensu e inancial s abili y and minimize de aul isk. 14 3. METHODOLOGY 3.1. BUSINESS UNDERSTANDING The i s s ep needed is o ga he all he impo an in o ma ion abou he clien da a a ailable on he EDP da abase. The way o achie e ha success is by gaining an in-dep h unde s anding o he business and i s ope a ions. Fo ha , CRISP-DM was he me hodology used. I is clea ha he eam has a ew main obs acles o o e come in o de o imp o e he e ec i eness o deb eco e y: ge an insigh in o who is mos likely o become in deb . And why? And a e knowing ha , how can we imp o e ha ? Objec i e: build a machine lea ning classi ica ion model ha iden i ies pa e ns ha can help p edic which cus ome s a e mos likely o de aul on hei loans. Scope: - Iden i y he main a iables ha co ela e wi h a cus ome being a de aul e o paymen s - Tes di e en models and hei pa ame e s o choose he one wi h he bes accu acy - Lea n and assess pa e ns o each class - Apply he model and help iden i y and mi iga e po en ial inancial isks 3.2. DATA UNDERSTANDING 3.2.1. Da a sou ces I was collec all he impo an in o ma ion abou he clien da a a ailable on he EDP da abase, which means all he a iables co esponding o he in o ma ion o he con ac (s) and consump ion o he clien s. Un o ne lly, as EDP can no ha e access o in o ma ion such as pu chasing powe , annual income o o he in o ma ion ha leads us he assume he p obabili y o a pe son becoming in deb , i was collec ed as much in o ma ion a ailable as possible including hei demog aphic in o ma ion, consump ion pa e ns, billing in o ma ion, and paymen his o y. The collec ion o he da a was based on SAS ables connec ed o he da abase se e s o he company and pe o med on SAS En e p ise Guide. The da ase is composed o all ac i e clien s o he company, meaning, he ones ha had a leas one ac i e con ac on elec ici y o gas. Tha makes a o al o 3M people. The ea u es used we e: Column name Meaning NOME_PRIMEIRO Name o he clien COD_CAE Type o consump ion: Domes ic o o he COD_CATEGORIA_CLIENTE Cus ome ca ego y: Pe son o o ganiza ion COD_TIPO_CLIENTE Clien ype: Domes ic o non-domes ic COD_CLASSE_CONTA COD_VALOR_PRESENTE Accoun Class: Small business o esiden ial Le el o p omo ional campaigns gi s FLG_EDP_ONLINE i i uses he App o no 21 Column Meaning 0-14 numbe o people in he ange o age om 0 o 14 yea s old 15-24 numbe o people in he ange o age om 15 o 24 yea s old 25-64 numbe o people in he ange o age om 25 o 64 yea s old 65 e mais numbe o people in he ange o age 65+ Homens numbe o men Mulhe es numbe o women Pode de comp a pe capi a Pe capi a pu chasing powe Tabela 5- New Demog aphic Fea u es and meaning 3.3.3. Missing alues The bes app oach o dealing wi h missing alues depends on he speci ic da ase , he amoun o missing da a, and he o e all goals o he analysis. A small numbe o missing alues we e iden i ied, wi h a maximum o 3% o he o al and only on ca ego ical ea u es. To add ess his issue, he chosen me hod was o ill hese missing alues wi h he mode, which means illing wi h he mos equen alue in he espec i e ea u e column. This me hod was chosen because i is a s aigh o wa d way o handle missing alues, especially o small amoun s o missing da a. 3.3.4. Ou lie de ec ion Iden i ying and emo ing ou lie s is a c ucial s ep in da a p ep ocessing ha should be pe o med p io o applying any s a is ical o machine lea ning echniques o he da a. Ou lie s can ha e a majo impac on he ou come o analyses. The e o e, i is impo an o handle ou lie s be o e p oceeding wi h da a analysis o ensu e accu a e and eliable esul s. (Ende lein, 1987). A e a combina ion o isualiza ion h ough box plo s, knowledge o he business and he In e qua ile Range (IQR), he di e ence be ween he 75 h and 25 h pe cen ile o he da a, meaning he obse a ions ha a e less han Q1 - 1.5 * IQR o g ea e han Q3 + 1.5 * IQR a e conside ed ou lie s, 2.63% o he da a we e conside ed ou lie s and elimina ed. 3.3.5. Ca ego ical ea u es To ep esen ca ego ical a iables, such as: COD_CAE, COD_CATEGORIA_CLIENTE, COD_TIPO_CLIENTE, COD_CLASSE_CONTA, COD_VALOR_PRESENTE, FLG_EDP_ONLINE, MEAN_o _FLG_TARIFA_SOCIAL, Flg_Tel, Flg_Email, Dis i o, MEAN_o _FLG_FATURA_ELETRONICA, MEAN_o _FLG_DEB_DIRETO, Tipo Con a os, Mul ibanco, PONTOS DÍVIDA, gen, i was used he common echnique o dummy a iables. This will help imp o e he pe o mance o he model, as hey cap u e he ela ionship be ween he o iginal ca ego ical a iable and he a ge a iable. 3.3.6. Fea u e scaling Fea u e scaling is an impo an p ep ocessing s ep in many machine lea ning algo i hms. I e e s o he p ocess o ans o ming he a iables o ea u es in he da ase o a common scale. This helps o ensu e ha each a iable o ea u e con ibu es equally o he model, a oiding bias and imp o ing he pe o mance o he algo i hm. As no ed by (Gé on, 2019), machine lea ning algo i hms end o wo k mo e e ec i ely o each op imal esul s mo e quickly when he ea u es ha e been s anda dized o be on simila scales. Wi hou ea u e scaling, a iables o ea u es wi h la ge alues can domina e he dis ances, leading o subop imal pe o mance. The e o e, he da ase was no malized using he 22 me hod o min-max scaling. Min-max scaling, also known as no maliza ion, scales he alues o a ange be ween 0 and 1. 3.3.7. Ta ge dis ibu ion Analysing he a ge dis ibu ion, i 's clea i 's an imbalanced a ge a iable, which could impac he accu acy o he p edic ions. Ha ing an unbalanced da ase can ha m he accu acy o he p edic i e model in se e al ways. This is because when he a ge classes a e imbalanced, he model will lea n o p edic he majo i y class mo e o en, leading o a high accu acy sco e bu wi h a low p ecision and ecall sco e. This means ha he model will no pe o m well in iden i ying he mino i y class, leading o biased esul s. Addi ionally, an unbalanced da ase can lead o o e i ing on he majo i y class, making i di icul o he model o gene alize o unseen da a. The e o e a e conside ing di e en echniques o handle his issue, i was decided o s a i y he da a when spli ing i in o a aining and es ing se , ha in ol es dis ibu ing he a ge a iable e enly among he wo se s, so ha he p opo ion o a ge ca ego ies in bo h se s is simila o he p opo ion o a ge ca ego ies in he en i e da ase . This was done o ensu e ha he aining se and es ing se a e ep esen a i es o he en i e da ase and ha he e is a balance o he a ge ca ego ies in bo h se s. Ano he echnique conside ed was unde sampling. I is a echnique ha in ol es educing he size o he majo i y class o balance he a ge a iable. Al hough i is an e ec i e way o handle imbalanced da ase s, i can also lead o a loss o impo an in o ma ion and may esul in a biased model. This was he case since i was ound ha unde sampling esul ed in a signi ican educ ion in he size o he da ase , which in u n impac ed he pe o mance o he p edic i e model. The e o e, i was decided no o use unde sampling and ins ead use s a i ied sampling, as discussed ea lie , o ensu e a balanced dis ibu ion o a ge a iables in he aining and es ing se s. This app oach helped o p e en he o e - ep esen a ion o unde - ep esen a ion o any pa icula a ge ca ego y, which could ha e esul ed in biased models o inaccu a e model pe o mance e alua ion. 3.3.8. Dealing wi h co ela ed ea u es Co ela ion analysis is a aluable echnique in he p ocess o selec ing ele an ea u es o a a ge a iable, as i can help iden i y p edic o s ha may ha e a s ong ela ionship wi h he a ge . (Kellehe , Mac, Aoi e, & A cy, 2015). In he con ex o da a analysis, co ela ion analysis helps iden i y ea u es ha a e highly co ela ed be ween hemsel es and he a ge a iable. This in o ma ion can hen be used o make in o med decisions abou which ea u es o include o exclude om a p edic i e model. The co ela ion was calcula ed using a ious me hods, including Pea son's co ela ion coe icien and Spea man's ank co ela ion coe icien . These me hods measu e he linea ela ionship be ween wo a iables, wi h alues anging om -1 (s ong nega i e co ela ion) o 1 (s ong posi i e co ela ion). Fea u es ha a e highly co ela ed wi h each o he (mul icollinea i y) o wi h he a ge (high co ela ion) - co ela ion coe icien o mo e han 0.8 wi h each o he and mo e han 0.7 wi h he a ge - we e emo ed du ing co ela ion analysis o a oid uns able and i ele an solu ions in he model since high co ela ions be ween he ea u es and a ge can esul in o e i ing o edundan in o ma ion. 23 3.3.9. Fea u e Selec ion P epa ing da a o aining models o en in ol es ea u e selec ion, which is c ucial o imp o ing model pe o mance and educing compu a ional ime, especially when dealing wi h la ge da ase s. By educing he dimensionali y o he da a, ea u e selec ion helps o iden i y he ele an p edic o s and a oid he p oblem o ha ing oo many po en ial p edic o s, which can make he modeling p ocess mo e di icul and ime-consuming (Guyon & Elissee , 2003). The e o e, using p edic i e models o ea u e selec ion can help in iden i ying he mos impo an ea u es ha ha e a s ong impac on he a ge a iable. This leads o a mo e e icien and e ec i e model, as well as inc eased in e p e abili y. The algo i hms expe imen ed o selec he ea u es o he model we e: • Embedded me hods such as Decision ees, idge classi ie and lasso classi ie . • Fil e -Based Me hods such as Co ela ion • W appe me hods such as RFE ( ecu si e ea u e elimina ion), we e used o he ea u es selec ed by he me hods abo e. 3.4. MODEL SELECTION AND EVALUATION A e he da a is ully p epa ed and explo ed, i is eady o model selec ion. The i s s ep made was o spli he da a in o ain and es , wi h he ain ep esen ing 70% and he emaining 30% he es pa . To ensu e all classes a e being ep esen ed while he da a is being ained by he model, i was applied 10- old c oss alida ion as well. Since we a e dealing wi h a mul iclassi ica ion p oblem, a high numbe o ea u es, and he p esence o bo h ca ego ical and nume ical ea u es, he ollowing models we e conside ed o aining: Random Fo es Classi ie , Decision T ees Classi ie , Neu al Ne wo ks and XGBoos . Gi en his esea ch is acing a e y la ge size da ase , which con ains millions o obse a ions and nume ous ea u es, I need a model ha could handle such la ge and complex da a while s ill p oducing accu a e p edic ions. The main obs acle he e in he selec ion and implemen a ion o he model is compu a ional esou ces. Conside ing ha , and he balance be ween he weaknesses and s eng hs o he model, I chose o use XGBoos p ima ily because o i s e iciency and scalabili y wi h compu a ional esou ces. S eng hs: Weaknesses: High p edic i e accu acy Compu a ionally in ensi e, especially when dealing wi h la ge da ase s o complex models. Flexibili y: can be used o a wide ange o asks, including ea u e selec ion, missing alue impu a ion, and anomaly de ec ion. Sensi i i y o hype pa ame e s, and uning hese hype pa ame e s can be ime-consuming and compu a ionally expensi e. Scalabili y: highly scalable and can handle la ge da ase s wi h millions o ea u es and millions o samples. In e p e abili y: XGBoos models can be di icul o in e p e Pa allel p ocessing, which can speed up aining imes. Po en ial o o e i ing Regula iza ion, which can help p e en o e i ing and imp o e gene aliza ion pe o mance. Tabela 6- S eng hs and Weaknesses o XGBoos 24 When dealing wi h imbalanced da ase s, accu acy may no be a eliable me ic as i only measu es he p opo ion o co ec p edic ions. Ins ead, i is essen ial o use an e alua ion me ic ha conside s bo h p ecision and ecall o assess he model's o e all pe o mance in a balanced manne . This app oach is necessa y because adi ional me ics may no accoun o he alse posi i e and alse nega i e a es, which can signi ican ly impac he e ec i eness o he model. The F1 sco e is a popula me ic used in classi ica ion asks o measu e he balance be ween p ecision and ecall. I is pa icula ly use ul when he da ase is imbalanced, meaning ha one class has a much la ge numbe o samples han he o he s, which is he case. P ecision and ecall a e wo impo an me ics in classi ica ion. P ecision measu es how many o he p edic ed posi i e samples a e ac ually posi i e, while ecall measu es how many o he ac ual posi i e samples a e co ec ly p edic ed. In o he wo ds, p ecision ocuses on he accu acy o posi i e p edic ions, while ecall ocuses on he abili y o ind all posi i e samples. The F1 sco e is he ha monic mean o p ecision and ecall. I is calcula ed as ollows: 𝐹1=2 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛×𝑟𝑒𝑐𝑎𝑙𝑙 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑟𝑒𝑐𝑎𝑙𝑙 The F1 sco e anges om 0 o 1, wi h 1 being he bes possible sco e. A high F1 sco e indica es good pe o mance in bo h p ecision and ecall. By choosing he F1 sco e, I am able o e alua e he ade- o be ween p ecision and ecall and ge a single me ic ha summa izes he o e all pe o mance o my model. This is pa icula ly impo an when he cos o alse posi i es and alse nega i es is no equal, as i allows me o op imize he model o he pa icula applica ion a hand. 25 4. RESULTS AND DISCUSSION This chap e p esen s he esul s o applying he model and discusses he implica ions o he indings. I s a s by p o iding an o e iew o he selec ed model ollowed by he hype pa ame e uning p ocess. Then, i epo s he inal pe o mance o he model wi h he selec ed e alua ion me ics and compa ed i o baseline models. Finally, he limi a ions a e discussed and he po en ial u u e di ec ions o his s udy a e men ioned. 4.1. EVALUATING FIRST RESULTS The model was i s applied wi h he de aul pa ame e s, using di e en se s o a iables cons uc ed du ing ea u e selec ion. I gene a ed he ollowing esul s: F1 Sco e Fea u es T ain Tes Decision T ees 0.7836 0.7800 Co ela ions 0.7804 0.7772 Ridge Reg ession 0.7816 0.7788 Lasso Reg ession 0.7611 0.7590 RFE 0.7873 0.7839 Tabela 7- F1 Sco e esul s The esul s show ha di e en ea u e selec ion me hods can ha e a signi ican impac on he pe o mance o he XGBoos model. In his case, he RFE (Recu si e Fea u e Elimina ion) me hod esul ed in he highes F1 sco es on bo h he aining and es ing da ase s. This indica es ha he selec ed ea u es by his me hod we e he mos ele an and in o ma i e o he classi ica ion ask. These ea u es we e he ollowing: Con a os A i os, Con a o a i o mais An igo, MEAN_o _QTD_CONSUMO_ANUAL_ESP, Con a os Ence ados, An iguidade Clien e, MEAN_o _Pack01, Va iedade Se iços, Tempo a é pagamen o, a g_sco e_clien ean , COD_CAE_o he , COD_CATEGORIA_CLIENTE_2, COD_VALOR_PRESENTE_A, COD_VALOR_PRESENTE_B, COD_VALOR_PRESENTE_M, COD_VALOR_PRESENTE_MA, COD_VALOR_PRESENTE_MB, MEAN_o _FLG_TARIFA_SOCIAL_1, Flg_Tel_1, Flg_Email_1, Dis i o_BEJA, Dis i o_COIMBRA, Dis i o_SANTAREM, Dis i o_VISEU, MEAN_o _FLG_DEB_DIRETO_1, Tipo Con a os_D, Mul ibanco_1. On he o he hand, he Decision T ees and Co ela ions me hods p oduced F1 sco es ha we e lowe han RFE, bu s ill ela i ely high. This sugges s ha he ea u es selec ed by hese me hods also ha e some p edic i e powe , bu may no be as in o ma i e as hose selec ed by RFE. The Ridge and Lasso Reg ession me hods p oduced he lowes F1 sco es, indica ing ha he selec ed ea u es may no be as ele an o in o ma i e o he classi ica ion ask as hose selec ed by he o he me hods. 4.2. HYPERPARAMETER TUNING The choice o hype pa ame e s can signi ican ly a ec he pe o mance o a model, and hype pa ame e uning is he p ocess o inding he bes combina ion o hype pa ame e s ha esul s in he bes model pe o mance and possibly less o e i ing. 26 Finding he bes hype pa ame e s is c ucial o c ea ing an accu a e machine lea ning algo i hm. Howe e , he e is no uni e sal ule o selec ing hype pa ame e s, as di e en alues can ha e a signi ican impac on he model's p edic i e abili y. Using c oss- alida ion, we can es ima e he es e o o di e en hype pa ame e alues, and selec he bes one. Ye , wi h a la ge numbe o possible alues o each hype pa ame e , inding he bes se o hype pa ame e s can be a daun ing ask. To add ess his, echniques like g id sea ch, andomized sea ch, and Bayesian op imiza ion can be used o na iga e he as hype pa ame e space and ind he bes combina ion o alues. (James, Wi en, Has ie, & Tibshi ani, 2021) Since he mos well-pe o med se o ea u es we e he RFE, i is he one ha is going o be used o hype pa ame e uning. The i s s ep made o adjus pa ame e s was o analyse he lea ning a e and he numbe o es ima o s o he model and how a ec s i s pe o mance. To p e en o e i ing, i is used he lea ning a e, also called sh inkage, which educes he weigh s o each ea u e du ing boos ing i e a ions p opo ionally o hei cu en weigh s. This helps achie e be e gene aliza ion pe o mance, bu i may ake longe o con e ge. A smalle lea ning a e is usually pai ed wi h a highe numbe o ees o compensa e o he slowe con e gence. (XGBoos Documen a ion — xgboos 1.7.4 documen a ion) This means ha a smalle lea ning a e may esul in longe aining imes since he model akes smalle s eps o op imize he objec i e unc ion. On he o he hand, a la ge lea ning a e may esul in as e aining imes, bu he model may con e ge o a subop imal solu ion and may equi e addi ional s eps o imp o e he pe o mance, and clea ly, ha is shown in he igu e. The e o e, he choice o he lea ning a e should be balanced be ween he success a e and pe o mance ime based on he a ailable compu ing esou ces, meaning he main ocus o looking a he igu e is o ind he op imal ade-o . Figu a 9 - F1 Sco e s Lea ning a e 27 The numbe o es ima o s is a hype pa ame e in he XGBoos algo i hm ha e e s o he numbe o decision ees o be used in he model. We can see in he igu e ha inc easing he numbe o es ima o s can imp o e he pe o mance o he model, bu can also signi ican ly inc ease he aining ime and memo y equi emen s. In gene al, inc easing he numbe o es ima o s beyond a ce ain poin may lead o diminishing e u ns in pe o mance imp o emen , while signi ican ly inc easing he ime and compu a ional esou ces equi ed o ain he model. While choosing he me hod o use o explo e he hype pa ame e s, i was conside ed g id sea ch and andomized sea ch. Gi en he la ge da ase , he big se o di e en pa ame e s o explo e, and he limi ed compu a ional esou ces, i may no be easible o use g id sea ch, which exhaus i ely sea ches o e all possible combina ions o hype pa ame e s, as i may ake a e y long ime o comple e. The e o e, al e na i e me hods such as andomized sea ch, which andomly samples a subse o hype pa ame e s, may be mo e p ac ical and e icien in his scena io. Acco ding o (Be gs a, Ca, & Ca, 2012), hey ound ha andom sea ch can achie e he same le el o pe o mance o hype pa ame e uning as g id sea ch, bu wi h ewe a emp s and less ime spen on compu a ion. This is because he sea ch space o hype pa ame e s is complex and includes bo h con inuous and disc e e pa ame e s, making he g id oo coa se o cap u e all he de ails. The s udy also ound ha al hough andom sea ch may p oduce pe o mance es ima es wi h high a iance, inc easing he numbe o a emp s can compensa e o his wi hou signi ican ly inc easing compu a ional cos s compa ed o g id sea ch. The e o e, i was decided o use andomized sea ch ins ead o g id sea ch o une he hype pa ame e s o he model. A e unning 100 i e a ions o he sea ch, i was ound a se o pa ame e s ha we e able o educe he o e i ing o he model, e en hough he sco e was no imp o ed. Tha lea es us wi h he inal esul o 0.7850 on he aining se and 0.7835 on he es se . Figu a 10 - F1 Sco e s Numbe o es ima o s 28 4.3. DISCUSS FINAL RESULTS This chap e is dedica ed o in e p e ing he esul s and he p edic ions o he model. In he case o a classi ica ion model, i is impo an o e alua e he pe o mance o he model in a meaning ul way, aking in o accoun he speci ic equi emen s o he ask a hand. To do his, I ha e gene a ed a classi ica ion epo ha p esen s p ecision, ecall, 1-sco e, and suppo o each class in he da ase . This p o ides a de ailed o e iew o he model's pe o mance in each class, allowing us o e alua e i s s eng hs and weaknesses. By conside ing hese me ics, we can gain insigh s in o how well he model is able o iden i y di e en ypes o ins ances and we can make in o med decisions abou how o imp o e i s pe o mance. P ecision Recall F1-Sco e Suppo 0 0,84 0,94 0,89 651070 [1-3] 0,65 0,46 0,54 218689 [4-6] 0,64 0,32 0,43 34352 [7-9] 0,57 0,09 0,15 2828 accu acy 0,8 906939 mac o a g 0,47 0,45 0,5 906939 weigh ed a g 0,78 0,8 0,78 906939 Tabela 8 - Classi ica ion epo Looking a he esul s, we can see ha he model achie ed high p ecision (0.84) and ecall (0.94) o class 0, showing ha i pe o med well in iden i ying ins ances o his class. Howe e , o classes 1-3 and 4-6, he p ecision and ecall a e lowe , indica ing ha he model did no pe o m as well o co ec ly iden i y hese classes. Class 7-9 has he lowes p ecision, ecall, and F1-sco e, indica ing ha he model s uggles he mos in iden i ying ins ances o his class. The mac o-a e aged F1-sco e is 0.50, indica ing ha he pe o mance o he model is mode a e, howe e , he mac o-a e age F1 sco e ea s all classes equally and calcula es he F1 sco e o each class sepa a ely, and hen akes he unweigh ed a e age o hose sco es. This means ha he con ibu ion o each class o he inal sco e is he same, ega dless o he numbe o samples in ha class. The e o e, i is mo e app op ia e o use me ics ha conside he class dis ibu ion, such as he weigh ed a e age o mic o-a e age F1 sco e. The weigh ed a e age F1 sco e calcula es he a e age F1 sco e, weigh ed by he numbe o samples in each class, gi ing mo e weigh o he pe o mance o he classes wi h mo e samples. As seen be o e we can obse e he weigh ed a e age F1-sco e is 0.78, indica ing he o e all pe o mance o he model is good bu sugges ing ha he model pe o med be e on he la ge Figu a 11 - Con usion Ma ix o Tes se 29 classes and s uggled wi h he smalle ones. To u he analyse he s eng hs and weaknesses o he model i was cons uc ed a con usion ma ix. A e cons uc ing and analyzing he con usion ma ix, he main obse a ions ha we e d awn a e: As seen be o e i is clea ha class 0 has a bigge sample and i is co ec o assume ha he model pe o med be e in his class by looking a he classi ica ion epo . We can u he con i m his by looking a he con usion ma ix as well since i has he highes numbe o ue posi i es and he lowes numbe o alse posi i es and alse nega i es. We can also see ha he class mos mis aken by class 0 is class 1-3. The same happens o class 1-3 in he sense ha is mos mis aken by class 0. Class 1-3 has a la ge numbe o alse posi i es, which means ha many samples a e inco ec ly p edic ed o belong o class 1-3 when hey ac ually belong o ano he class. On he o he hand, class 4-6 has a la ge numbe o alse nega i es, which means ha many samples ha belong o class 4-6 a e inco ec ly p edic ed o belong o ano he class. In class 7-9 we can also conclude ha is o en mis aken by class 4-6. Also om class 7-9, we can see ha has he lowes numbe o ue posi i es, indica ing ha i is he mos di icul class o p edic accu a ely. O e all, we can obse e om he con usion ma ix ha i is common o a class o be misclassi ied as i s neighbou ing class. This pa e n can be expec ed, as he e may no be a signi ican di e ence in he ea u es be ween hese neighbou ing classes. The e o e, a classi ie may ha e a highe chance o making alse posi i e o alse nega i e e o s when dis inguishing be ween hese classes. 30 5. CONCLUSION This epo was based on he main p ojec de eloped du ing he in e nship and i explo ed he use o machine lea ning algo i hms o p edic ing he isk o clien s in e ms o deb sco e a EDP Come cial. The p ojec add esses he u gen need o e ec i e deb managemen in a compe i i e and g owing ma ke , whe e da a-d i en echniques can signi ican ly imp o e deb eco e y. To achie e ha demand i was c ea ed a p edic i e model ha is able o iden i y pa e ns in clien s based on hei sco eca d, which measu es he quali y o a clien based on hei in ingemen his o y. The da a se was cons i u ed o clien con ac s in o ma ion, se ices equi ed, paymen his o y and in o ma ion on he places o consump ion. Be o e applying he model, he da a p epa a ion and ea u e enginee ing p ocess we e c ucial o he pe o mance o he model, including adding ea u es o help he model succeed. A e aining he model using XGBoos classi ie , was conduc ed hype pa ame e uning o help educe he o e i ing o he model and he model ended up achie ing an o e all pe o mance o 0.78. I was concluded ha one o i s main weaknesses is co ec ly iden i ying he lowe classes, bu i has good pe o mance in p edic ing he bigge classes. Al hough mo e wo k is needed o op imize he model's pe o mance and expand he sample size o he da a se and add mo e in o ma ion on he clien 's inancial p o ile, his p ojec shows p omising esul s and p o ides a ounda ion o u u e wo k on deb eco e y managemen . By being p oposed his p ojec he majo i y o his in e nship ga e me exposu e o he p ac ical applica ion o da a-d i en echniques, speci ically he heo e ical and p ac ical knowledge acqui ed du ing he Mas e 's p og am. This was achie ed by applying da a analysis, p ocess mining, and machine lea ning o cus ome da a and he e o e i was possible o de elop a p edic i e model ha can iden i y a - isk accoun s be o e hey all behind on paymen s, allowing he company o espond p e en i ely and p oac i ely. The collec ion o he da a was made using SAS Guide bu all he es was con inued on Jupy e No ebook, he main sys em used du ing he mas e in se e al cou ses. Du ing he in e nship, I also had he oppo uni y o gain a weal h o new knowledge and insigh s, pa icula ly in he a eas o business ope a ions and managemen o a la ge po olio o clien s in he ene gy indus y. Th ough his expe ience, I was able o explo e new algo i hms and so wa e ools, which helped me o be e unde s and and apply he concep s lea ned du ing my Mas e 's p og am. O e all, he in e nship p o ided a aluable and ele an expe ience ha enhanced my unde s anding o he ene gy indus y and solidi ied my knowledge o he subjec s al eady s udied in he mas e ’s p og am. 37 Figu a 14- Pea son's co ela ion ma ix 38 Figu a 15 - Spea man's ank co ela ion ma ix 39 Figu a 16 - Fea u e Selec ion Decision T ees: Gini and En opy 40 Figu a 17- Fea u e Selec ion using Ridge Classi ie 41 Figu a 18- Fea u e Selec ion using Lasso Classi ie 1