Modelling Decisions in Banking Supervision A Machine Learning Approach

Doc o al P og amme - In o ma ion Managemen 2 Acknowledgemen s To my supe iso P o . Mau o Cas elli o his in aluable suppo , guidance and expe ience. To a g ea spa ing pa ne . Also o my co-supe iso and colleague P o . Nadine Cˆo e-Real o he imely insigh s and c i ical business iews. To my wonde ul pa en s, who g ew in o my closes iends, o eaching me when o walk and when o ly. To my belo ed and magni icen wi e, wi h whom I sha e he g ea es joys, o being key o my successes and my ha en in despe a e imes. She s ood by me h ough all my a ails, my absences, my i s o pique and impa ience. She ga e me suppo and help, discussed ideas and p e en ed se e al w ong u ns. To my child en, he cou ageous, ado able and ebellious Miguel and Tom´as, who each me e e y day o be pa ien and unde s anding. F om he e y beginning, wi h a ms wide open, I hope o li e up o hem and wa ch hem do be e han me. To my iends and amily, la c `eme de la c `eme, he e y e y ew ha p e ail laughing wi h me. A special hank you o my mos unde s anding bosses, Jo˜ao Ped o Gomes and Lu´ıs Cos a Fe ei a, who ga e me he luxu y o ime and p o ided he means o see his p ojec h ough. 3 Doc o al P og amme - In o ma ion Managemen 4 Publica ions Machine Lea ning Applied o Banking Supe ision: a Li e a u e Re iew Ped o Gue a and Mau o Cas elli Risks, 2021, 9, no. 7: 136 h ps://doi.o g/10.3390/ isks9070136 Machine lea ning o liquidi y isk modelling: A supe iso y pe spec i e Ped o Gue a, Mau o Cas elli, Nadine Cˆo e-Real Economic Analysis and Policy, 2022, Volume 74, Pages 175-187 h ps://doi.o g/10.1016/j.eap.2022.02.001 App oaching Eu opean Supe iso y Risk Assessmen wi h SupTech: A P oposal o an Ea ly Wa ning Sys em Ped o Gue a, Mau o Cas elli, Nadine Cˆo e-Real Risks, 2022, 10, no. 4: 71 h ps://doi.o g/10.3390/ isks10040071 5 Doc o al P og amme - In o ma ion Managemen 6 Begin a he beginning, he King said g a ely, “and go on ill you come o he end: hen s op.” —Lewis Ca oll, Alice in Wonde land I is a capi al mis ake o heo ize be o e one has da a. Insensibly one begins o wis ac s o sui heo ies, ins ead o heo ies o sui ac s. —Si A hu Conan Doyle, She lock Holmes Doc o al P og amme - In o ma ion Managemen 8 Con en s 1 In oduc ion 11 2 Machine Lea ning Applied o Banking Supe ision: a Li e a u e Re iew 15 2.1 In oduc ion................................ 15 2.2 Me hodology ............................... 16 2.2.1 Engines .............................. 16 2.2.2 Que y ............................... 16 2.2.3 S eps................................ 17 2.3 Resul s................................... 18 2.3.1 Dis ibu ion............................ 18 2.3.2 E olu ion ............................. 20 2.3.3 Da ase s.............................. 26 2.3.4 Rela edWo k........................... 26 2.3.5 GlobalAnalysis.......................... 27 2.4 Conclusion................................. 28 2.4.1 Limi a ions and u u e wo k . . . . . . . . . . . . . . . . . . . 29 3 Machine Lea ning o Liquidi y Risk Modelling: a Supe iso y Pe - spec i e 31 3.1 In oduc ion................................ 31 3.1.1 Risk assessmen measu es . . . . . . . . . . . . . . . . . . . . 31 3.1.2 Machine lea ning o isk assessmen . . . . . . . . . . . . . . 32 3.2 Me hodology ............................... 34 3.2.1 TheDa a ............................. 35 3.2.2 T ans o ma ions.......................... 36 3.2.3 Fea u e Selec ion . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.2.4 Expe imen s............................ 37 3.3 Resul s and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.4 Conclusion................................. 46 3.4.1 P ac ical and heo e ical implica ions . . . . . . . . . . . . . . 46 3.4.2 Limi a ions and u u e wo k . . . . . . . . . . . . . . . . . . . 47 4 App oaching Eu opean Supe iso y Risk Assessmen wi h SupTech: A P oposal o an Ea ly Wa ning Sys em 49 4.1 In oduc ion................................ 49 4.1.1 Rela edwo k ........................... 51 4.2 Me hodology ............................... 54 4.2.1 TheDa a ............................. 55 9 Doc o al P og amme - In o ma ion Managemen gies o isk assessmen . These can be he pilla s o hei nex decision suppo sys ems by laying down he echnologies suppo ing isk assessmen p ocesses. Fu he mo e, his wo k can also inci e su eys and case s udies on he use and adop ion o ML a cen al banks. 3. Consul ancy companies will bene i om a compendium o ML echniques and isk measu es, o be e suppo hei clien s. 4. Academia ecei es an impo an con ibu ion ha ga he s an ex ensi e num- be o pape s on isk assessmen and colla es he iden i ied me hodologies om a supe iso y pe spec i e. This will hope ully se e as a s epping s one o u u e de elopmen s in his a ea, and p o ide a baseline o es ing new me hodologies. This pape is o ganised as ollows: i s a s by jus i ying he me hodology and desc ibing how he e e ences we e selec ed. The esul s sec ion ga he s simila i ies among published scien i ic knowledge and p esen s he mos ele an wo ks ha in luence his ield. The las sec ion p o ides a space o discussing lessons lea ned and u u e wo k. 2.2 Me hodology This esea ch was conduc ed h ough a se ies o explo a o y s eps on he opics o machine lea ning, banking, isk assessmen , and banking supe ision. The ini ial objec i e was o e alua e how machine lea ning echniques we e being used a cen al banks. Addi ionally, we in ended o analyse how hese me hods we e in o ming he analy ical capabili ies o supe iso s. We hen e ined a sea ch que y b oad enough o e u n a se o a icles we could wo k on. The ollowing subsec ions desc ibe a s ep-by-s ep guide o he e e ence sea ch and selec ion. 2.2.1 Engines This li e a u e e iew elies on h ee sea ch engines: Sp inge Link,ScienceDi ec , and Google Schola , que ied un il June 2021. The i s and second sea ch engines a e ex ensi ely enowned o hei us wo hiness and o selec ing op jou nals o hei esul s. The las one p o ides an ex ensi e o e iew o all a icles published in English (Gusenbaue , 2019). 2.2.2 Que y Th ough ex ensi e addi ion and di e si ica ion o sea ch e ms, we e ined he sea ch que y o he ollowing: ”machine lea ning” and (“bank” o “banking” o “supe ision”). The unde lying easoning is ha machine lea ning echniques a e he ocal poin o his e iew a icle. The added alue comes om analysing hei po en ial ap- plica ions o he banking sec o , speci ically banking supe ision. No limi a ion conce ning he yea o publica ion was applied. O e lapping esul s a e add essed in ou seconda y analysis. Fu he mo e, no il e ega ding ype o place o publica- ion was applied, since he included pape s’ jou nals o publica ion we e e alua ed 16 Doc o al P og amme - In o ma ion Managemen and classi ied a e sc eening. Addi ionally, o keep up wi h new publica ions, we de ined an ale in Google Schola wi h his que y. Finally, we pay close a en ion o Mendeley’s ale s o a icles ela ed o he se ga he ed in his e iew. 2.2.3 S eps The ollowing subsec ions de ail e e y s ep o he selec ion p ocess summa ise in he ollowing PRISMA diag am 2.1. Figu e 2.1: PRISMA diag am de ailing he selec ion p ocess o he iden i ied a icles. Table A.3 lis s he selec ed pape s, p o iding a single-sen ence summa y o hei con en . Iden i ica ion The esea ch que y iden i ied 85 a icles and wo books, om he h ee sea ch en- gines. All he pape s we e published in English, in se e al di e en jou nals, and spanned om 2000 o 2021. This i s s ep in ol ed i le and abs ac analysis, and excluded 14 a icles o lack o ele ance. Sc eening In his phase, he main opics o each a icle we e analysed, esul ing in he exclusion o 21 pape s, based on he ollowing c i e ia: •Da ase : when he analysed pape used da a o he han he banking sec o , i was disca ded. We a e awa e ha applica ions o ML o he s ock ma ke a e a endy opic in he li e a u e, and ha he insu ance and pension unds sec o is o g ea impo ance in he Eu ozone. Ne e heless, he egula ion is subs an ially di e en , and hey would me i om a di e en s udy and app oach; 17 Doc o al P og amme - In o ma ion Managemen •Me hodology: isk assessmen exe cises a e his o ically based on quan i a i e da a, combined wi h expe judgmen . Fu he mo e, i is he quan i a i e da a ha holds he la ges amoun o in o ma ion ega ding isk exposu e p ac ices. The e o e, we ocus ou analysis on quan i a i e me hods, o which a isk assessmen classi ica ion has al eady been assigned (le e aging on p e ious knowledge h ough supe ised lea ning). We hus excluded wo ks conce ning unsupe ised lea ning me hods, o sen imen analysis (quali a i e); •Region: his c i e ion is closely ela ed o he i s , since egula ion changes acco ding o geog aphy. We chose o ocus mainly on wo ks based upon in- s i u ions ope a ing in he Eu ozone. None heless, ele an wo ks by o he cen al banks we e conside ed eligible. Eligibili y The nex s ep equi ed a ho ough analysis o each pape , o e i y i s sou ces and classi y he jou nal i was published in (qua ile o impac ). Pape s we e analysed om 2021 backwa d o iden i y any o e lapping esul s o new o imp o ed me hod- ologies, esul ing in he exclusion o en mo e a icles: nine being pe sonal loans ela ed and one duplica e esul . The scope o his e iew is he applica ion o ML echniques o isk assessmen om a supe iso y pe spec i e, which includes a bes how ins i u ions a e add ess- ing hei isk assessmen exe cises. The da a and p edic o s used o e alua e an indi idual c edi applica ion (pe sonal loan) di e subs an ially om he da a used by banks om a co po a e pe spec i e, and e en mo e om he da a collec ed in he egula o y con ex . As such, wo ks ega ding c edi isk o indi idual applican s we e also excluded. Conside ed pape s The inal a icle base consis s o 41 pape s and wo books, published om 2000 un il 2021, selec ed h ough he s eps men ioned. In he nex sec ion, we will desc ibe he simila i ies among he pape s, as well as he me hods applied and espec i e banking a eas. 2.3 Resul s 2.3.1 Dis ibu ion Based on he e iewed wo ks om he p e ious sec ion, he ollowing pa ag aphs desc ibe how machine lea ning echniques ha e been used in he banking sec o . Ou esea ch in ends o p o ide a u u e e e ence on how hese echnologies add ess and suppo he isk assessmen p ocess, in pa icula om a cen al bank’s pe spec i e. These esul s solely e lec he analysis o he pape s selec ed o his e iew. They ep esen nei he he o al o publica ions h oughou hese yea s no he dis ibu ion o opics o all publica ions. Table A.1 summa ises he selec ed a icles, e e enced by au ho , yea o publi- ca ion, a ilia ion and numbe o ci a ions. Addi ionally, able A.2 lis s he jou nals om he selec ed a icles. 18 Doc o al P og amme - In o ma ion Managemen The mos common opic on hese pape s is c edi isk ela ed (nea ly 34% o e e ences), as shown in Figu e 2.2. Figu e 2.2: Dis ibu ion o a icles acco ding o main opic. The second majo ca ego y ela es o ”ML applica ion” (su eys, in- ech and sup- ech, as pe he di ision sugges ed by B oede s and P enio (2018), he use o inno a i e echnologies by supe iso y agencies o suppo hei p ocesses) along wi h “s ess es s”. The emainde o he esul s ocuses ei he on ”bank isk” mo e b oadly, o on speci ic opics o supe ision such as liquidi y isk and o he banking isk pe spec i es. Ano he ele an aspec is he publica ion da e o hese a icles, anging om 2000 o 2021 and dis ibu ed as shown in Figu e 2.3. Figu e 2.3: Re e ences acco ding o yea o publica ion. Impo an ly, al hough ML applied o he inancial sec o has been p esen since 2000, by 2015 he in e sec ion o hese knowledge a eas gained a huge in e es . This ansla ed o inc easing numbe s o publica ions in his ield, wi h he majo i y o ele an a icles in his s udy being published om 2017 onwa d. Table A.4 lis s he machine lea ning me hods applied by each au ho as well as he da ase s ha suppo ed each esea ch. 19 Doc o al P og amme - In o ma ion Managemen 2.3.2 E olu ion The selec ed pape s we e o ganised by da e o publica ion. Publica ion in e als we e de ined based on ele an e en s in he banking sec o , echnological e olu ion, and he numbe o pape s pe in e al. The i s slo anging om 2000 o 2011 encompasses he e ec s o he inancial c isis o 1999 and 2008. The second ange ( om 2012 o 2016) s ill e lec s se e al s udies based on he 2008 c isis, bu wi h a mo e ma u e insigh . In his pe iod he e is also a ending inc ease o ANN models. The hi d slo encompasses he yea s o 2017-2018, which show a signi ican inc ease in publica ions in e sec ing ML and he banking sec o . The inal in e al (2019 o he cu en da e) depic s impo an ML applica ions o he inancial ma ke in gene al. S udies in his pe iod e eal an inc eased pon- de a ion o he uses and impac s o machine lea ning in banking supe ision, wi h se e al publica ions om banking au ho i ies. 2000-2011 Six pape s we e iden i ied om his pe iod. They mos ly ocus on s ess es s al- hough h ee o hem engage on he opic o c edi isk and de aul isk. Ea ly in his pe iod, Galindo and Tamayo (2000) iden i ied he isk assessmen ask as c ucial o an e icien use o esou ces. They used an e o cu e me hodol- ogy o compa e model p ecision and concluded ha ee-based models ou pe o m ANNs, KNN and p obi . This se s o wa d he inding ha ee-based models a e mo e app op ia e o s uc u ed da a, as opposed o ANNs. Hillegeis e al. (2004) p oposed a new me hod o assessing bank up cy p oba- bili y. Based on he Black–Scholes–Me on op ion-p icing model, his me hod was compa ed o he well-known Z-sco e (Al man, 1968) and O-sco e (Ohlson, 1980), ob aining supe io esul s. These au ho s s essed he need o a s anda dised isk assessmen measu e mainly o compa abili y pu poses. Min and Lee (2005) p esen ed a pape ha compa es s a is ical and a i icial in elligence me hods, wi h he la e ou pe o ming he o me in he classi ica ion o bank up cy. Al hough his s udy ocuses on c edi isk assessmen o hea y in- dus y i ms in Ko ea, we included i in ou sample o a compelling eason. I is a clea example o machine lea ning me hods ou pe o ming con en ional s a is ics and i uses a se o p edic o s ( inancial a ios) easily mapped o egula o y inan- cial epo ing since hey a e based on balance shee en ies. Angelini e al. (2008) based hei wo k on he Basel II capi al equi emen s and he need o a sys em o assess c edi isk. The main objec i e o his wo k is o e alua e he possibili y o using neu al ne wo ks o es ima e he p obabili y o de aul o a bo owe (I alian small companies). In spi e o some ANNs being used, he compa ison o classic machine lea ning models o con en ional s a is ical me hods was he mo e ecu en app oach. Fu he mo e, he isk de ini ion used o e alua e he da a se s was based on he p obabili y o de aul . This is explained by he ac ha he da ase s a e mos ly om loan applica ions, ei he om small and medium en e p ises o pe sonal loans (housing included). These indings con adic Galindo and Tamayo (2000) as well as mo e ecen de elopmen s in his a ea. ANNs ha e been p o ed o excel in ime-se ies, image, and oice ecogni ion, as opposed o hei pe o mance using s uc u ed da a. Addi ionally, some a icles used inancial a ios and CAMELS a ing model (an 20 Doc o al P og amme - In o ma ion Managemen in e na ional a ing sys em used by egula o y banking au ho i ies o a e inancial ins i u ions) o assess an ins i u ion’s pe o mance (s ess es ing and bank up cy p edic ion). Assessing he heal h o a bank is c ucial o p e en i s ailu e and con ain he sys emic isk i s ailu e o losses ep esen . The wo k o Boyacioglu e al. (2009) iden i ies his assessmen as an o iginal classi ica ion p oblem. The au ho s use he CAMELS me hod o selec he mos ele an p edic o s. Using his me hod, neu al ne wo ks we e shown o ou pe o m mul i a ia e s a is ical me hods o a Tu kish banking sec o use case. Chaudhu i and De (2011) conside s Basel II de ini ion o isk o selec ea u es o he models. In his case, ANNs a e no as equen ly used as o he con en ional ML echniques, such as suppo ec o machines and k-nea es neighbou s. As a consequence, he au ho s ocus on he op imisa ion o hose models o he p oblem a hand (i.e. na u e o he da ase ). 2012-2016 In his pe iod, a icles mos ly e lec he i s insigh s gained om he 2008 inancial c isis. Ha ing iden i ied he lack o a comp ehensi e me hod o inco po a e ci cums an- ial aspec s in o he banking de aul isk p edic i e models, Ribei o e al. (2012) e- po ed ha SVM+ ou pe o med o he me hods ha did no include non- inancial in o ma ion. Hamme e al. (2012) showed ha Logical Analysis o Da a (LAD) is an accu a e me hod by e e se-enginee ing Fi ch isk a ings. The au ho s s a ed ha LAD can be used as an in e nal a ing sys em ha is Basel complian . I u iaga and Sanz (2015) ook a di e en app oach o his ma e . Fi s , hey used sel -o ganising maps (SOM) o p o ile dis essed banks. This unsupe ised lea ning me hod is compe i i e so i h i es o each he igh pa e n, he ep- esen a ion o bank up cy o a bank. A e wa d, he au ho s applied mul i-laye pe cep ons o assess a bank’s isk in se e al ime ames, ob aining e y p omising esul s p edic ing bank up cy o comme cial banks. This wo-s ep app oach is he i s in his selec ion o pape s o ecognise he bene i s o a p e-p ocessing phase o map he bank up cy layou o a bank. Al hough p e ious esea ch has shown be e esul s using con en ional ML, he success shown by his pe cep on model sugges s i is adequa e o model he ime e olu ion o quan i a i e da a. A new app oach o c edi sco ing using an ensemble model was p oposed by Ala’ aj and Abbod (2016). These au ho s combine se e al da a il e ing and ea- u e selec ion me hods be o e e alua ing model pe o mance, and compa e he mos adi ional classi ie s wi h hei me hod. The esul s a e alida ed on se e al public da ase s and hei accu acy assessed unde se e al measu es: a e age accu acy, a ea unde he cu e (AUC), H-measu e, and B ie Sco e. This is he i s pape in ou sample showing ha ensembles ou pe o m single models o classi ica ion p oblems. 2017-2018 These wo yea s showed a mo e han 60% inc ease in publica ions in he in e sec ion o ML and banking sec o . As highligh ed by S ydom and Buckley (2019), he echnological e olu ion allowed o he de elopmen o deep lea ning (DL) models, as well as new ensemble me hods like ex eme g adien boos ing (XGBoos ). Al hough 21 Doc o al P og amme - In o ma ion Managemen he DL’s i s eappea ance happened in 2012 (K izhe sky e al.), i s applica ion o inancial isk only came o ligh in 2016-2017. T adi ional ML and classical s a is ical app oaches a e s ill he co ne s ones o mos o hese a icles. Howe e , an inc easing end is no iceable in he use o ANN-based models mainly due o bigge da ase s and enhanced compu ing powe . Abell´an and Cas ellano (2017) build on hei p e ious wo k showing how ensem- bles achie e be e esul s in c edi isk assessmen han single models, alida ing he indings o Ala’ aj and Abbod (2016). The au ho s s ess he impo ance o indi idual model pe o mance as a c i e ion o ensemble selec ion. Al hough he au ho s emphasize hei own ee-based model (C edal Decision T ee, CDT), he main inding o hei wo k is he co obo a ion o he hypo hesis ha ensembles ou pe o m single classi ie s. P omp ed by he 2008 Global Financial C isis and he need o o esee signals o inancial ins abili y, I alian au ho s Pompella and Dicanio (2017) de eloped an Ea ly Wa ning Sys em (EWS) o help unco e dis ess signs o banks. This c edi isk model allows use s o disc imina e s able om likely- o- ail banks and migh be use ul in adjus ing a ing assignmen s by Ra ing Agencies. The au ho s sugges i s implemen a ion in egula o s o suppo he supe iso y p ocess. Xia e al. (2017) p esen an ex eme g adien boos ing model (XGBoos by Chen and Gues in (2016)) ha consis en ly ou pe o ms baseline models. The au ho s s ess he impo ance o model-based ea u e selec ion as well as he use o Bayesian hype -pa ame e op imisa ion o achie e be e p edic i e esul s. Al hough pe - sonal c edi isk is no he main opic o in e es in his e iew, his s udy shows he ad an ages o boos ing echniques and he impo ance o an in e p e able model o decision making. This ype o models ha e won se e al Kaggle compe i ions and a e consis en ly showing excellen esul s wi h s uc u ed da a. Chak abo y and Joseph (2017) om he Bank o England in oduce a cen al bank pe spec i e on machine lea ning and i s applica ions. The au ho s p o ide an o e iew o machine lea ning models and model alida ion o suppo he p esen- a ion o h ee case s udies. As a inal no e, his wo k acknowledges he amoun o a ailable da a as an impo an ec o in decision suppo sys ems based on machine lea ning a cen al banks and o he o ices. As p e iously s a ed, agency pape s as his one a e pa amoun in unde s anding he use o machine lea ning in hese con ex s, p o iding use cases and a eas o in e es o u u e wo k. Alessi and De ken (2018) con ibu e wi h ano he EWS o de ec excessi e c edi g ow h. This phenomenon is usually a he oo o sys emic isk o inancial s abili y and i s ea ly de ec ion can help a oid cases o bank up cy. The au ho s use Random Fo es classi ie model wi h c edi and eal es a e p edic o s. Thei wo k pionee s in he domain o isk assessmen om he pe spec i e o cen al banks, hus se ing pee p ac i ione s in hei u u e pa h. Mo eo e , he wo k ein o ces ha ensem- bles consis en ly ou pe o m single models. O he au ho s success ully use ex eme g adien boos ing o de elop a c edi isk model o inancial ins i u ions (Chang e al., 2018). Those ools p omise signi ican suppo (i.e. low e o a e) o isk assessmen in loans. The Cen al Bank o G eece also p o ides a ho ough analysis based on pos -2008 c isis loan da a om G eek banks, by Pe opoulos e al. (2018). This s udy se s a miles one o he use o ad anced ML echniques om a supe iso y pe spec i e. Fu he mo e, i le e ages he esul ing model o c ea e an EWS ha will suppo 22 Doc o al P og amme - In o ma ion Managemen subsequen decisions in loan app o al. Simila o wha I u iaga and Sanz (2015) ha e shown, modeling a imeline e olu ion is whe e neu al ne wo ks (in his case deep neu al ne wo ks, DNN’s) excel. Ano he impo an esul is ha DNNs can pe o m jus as well as XGBoos , showcasing how p ecisely deep lea ning models adap o s uc u ed da a. Ta ana e al. (2018) p esen a s udy ha di ec ly add esses liquidi y isk, which is he mos apidly de as a ing isk a bank is exposed o. In his pape , he au ho s p esen an a i icial neu al ne wo k model combined wi h a Bayesian ne wo k (BN) o assess liquidi y isk using sol ency as a p oxy. This combined app oach models he liquidi y isk indica o h ough he ANN and he p obabili y o occu ence h ough he BN. The esul s show his app oach dis inguishes he mos c i ical ac o s o liquidi y in his da ase . B oede s and P enio (2018) conduc a s udy ha compiles he expe ience o ea ly use s o inno a i e echnology in inancial supe ision (sup- ech). The au ho s s uc u e a de ini ion o sup- ech and show how i is used o da a collec ion and analy ics. These wo applica ions ha e di e en ini ia o s in supe iso y agencies. Da a collec ion ends o be ini ia ed by managemen decisions and p ojec s whe eas analy ics usually s a ou as esea ch ques ions o analysis que ies om supe i- sion uni s. A conduc i e h ead o all use cases is he sha ing o he expe ience o some ea ly adop e s and he impac hose echnologies a e ha ing on he o ganisa- ion. Simila s udies, such as he one conduc ed by Chak abo y and Joseph (2017) a e essen ial o compiling, sha ing, con as ing he se e al app oaches h oughou cen al banks and o he agencies. The Fede al Rese e p o ides a b oade pe spec i e, analysing how he use o machine lea ning and big da a will impac compliance aspec s (Jag iani e al., 2018). The au ho s also s ess he need o iden i y he isks ha hese echnologies ca y when applied o he inancial ma ke . Gogas e al. (2018) p opose a me hodology ha sepa a es sol en and ailed banks, using machine lea ning models. The au ho s p esen an al e na i e ool o s ess- es ing ha ou pe o ms he O-sco e. Thei app oach is based on a suppo ec o machine model ha helps o de ine a bounda y be ween sol en and insol en banks, con e ing his issue in o a classi ica ion p oblem. Kupiec (2018) p esen s a ela ed s udy ha s esses he need o new me hodologies o alida e con en ional bank s ess es s. As a inal e e ence o his pe iod, Le and Vi iani (2018) also ackle he p oblem o bank ailu e p edic ion using machine lea ning and classical inancial a ios. One impo an aspec o his wo k is ha he au ho s use a ios om 5 di e en isk pe spec i es: Loan quali y, Capi al quali y, Ope a ions e iciency, P o i abili y, and Liquidi y. This wo k alida es ye again ha machine lea ning me hods ou pe o m adi ional s a is ics. Howe e , hese au ho s do no explo e he possibili y o using ensembles, which ha e al eady been p o en o be op pe o me s in classi ica ion p oblems. 2019-2021 C edi and banking isks a e essen ial o a balanced economy; ying o p e en sys emic epe cussions s emming om hem is conside ed o he u mos impo ance. Simila ly o ea lie pe iods, hese isks main ain a p i ileged spo in esea ch. S ill, i was on ML applica ion we saw he mos signi ican inc ease in publica ions. This 23 Doc o al P og amme - In o ma ion Managemen sugges s he demand o coo dina ion and a global pe spec i e on he de elopmen s conque ed so a in his a ea. Leo e al. (2019) p oduce a ho ough e iew on how machine lea ning has been used a banks o isk assessmen . This pape o se s he indus ial and academic claim o ML applica ion e sus eal-li e p ac ices, highligh ing a se ies o pe spec- i es whe e isk managemen has been poo ly applied. Climen e al. (2019) de elop an insigh ul s udy ha aims o iden i y a se o inancial p edic o s ha bes model a bank’s inancial dis ess. To his end, he au ho s apply an XGBoos based model o a se o indica o s ha migh p edic a bank ailu e in he Eu ozone. The se o se- lec ed indica o s (To al asse s, Loan loss p o isions/ne in e es e enue, Equi y/ne loans and In e bank a io) a e shown o bes help egula o s moni o inancial dis- ess o hose banks. F om a echnical pe spec i e, his wo k ein o ces he choice o XGBoos o classi ica ion p oblems using s uc u ed da a. A ecen s udy by Wang e al. (2021) decons uc s he use o logi as he base classi ie o EWS de- eloped o p edic banking c isis. In ac , he au ho s use andom o es classi ie o simula e expe decision, ob aining a gene alisa ion capabili y abo e 80% a ea unde he cu e (AUC). Kou e al. (2019) compa e se e al ongoing esea ches conce ning he applica ions o machine lea ning me hods o he de ec ion o sys emic isk e en s, ha is, inan- cial dis ess phenomena ha a ec se e al ma ke s o geog aphic egions. They also p opose he use o big-da a analysis o assess sys emic isk. Soui e al. (2019) add ess he issue o comp ehensibili y o machine lea ning models o c edi isk assessmen . In e es ingly, in his s udy, in e p e abili y was men ioned as one o he ba ie s o adop ing ML models in day- o-day decision making. In an a emp o ci cum en his p oblem, he au ho s p oceeded o de elop an e olu iona y algo i hm o app oach c edi isk assessmen as an op imisa ion p oblem: minimising complexi y while maximising accu acy. A ecen e iew by Das ile e al. (2020) compa ing s a is ical and ML lea n- ing models o c edi sco ing showed ha ensembles ou pe o m single classi ie s, con i ming he esul s o p e iously men ioned wo ks. The au ho s iden i y model explainabili y and he abili y o deal wi h imbalanced da ase s, as he main issues o deal wi h when modelling c edi isk. Deep lea ning models also show p omising esul s, al hough hey ha e no been ex ensi ely explo ed o c edi isk assessmen . The au ho s iden i y he lack o in e p e abili y as he main ba ie o adop ing deep lea ning o c edi isk assessmen . Banco de Espa˜na (Alonso and Ca bo, 2021) published a compa ison o se e al well-known machine lea ning algo i hms o c edi de aul p edic ion, showing signi - ican imp o emen s o e logi . The au ho s es ima e ha implemen ing XGBoos - media ed assessmen could lead o sa ings o up o 17% o capi al equi emen s unde cu en ECB egula ion. An unes (2021) om he Cen al Bank o B azil p esen s a solid a gumen o main ain supe iso y on-si e inspec ions. The au ho compa es wo machine lea ning models, one ained wi h po olio a ings assessed by he banks hemsel es, and he o he based on pas a ings ob ained h ough on-si e inspec ions. The esul s show ha he o e all pe o mance is consis en ly highe when using da a e ie ed h ough inspec ions. This is he pe iod wi h he mos ML applica ions pape s iden i ied (wi h a o al o 9 ou o 13). They span om insigh s on how AI will con inue o e olu ionise indus ies and change social beha iou (Dwi edi e al., 2021), o mo e p ac ical 24 Doc o al P og amme - In o ma ion Managemen app oaches on how o inco po a e ML in inancial se ices (Lee and Shin, 2020). Milian e al. (2019) also p o ide a lis compa ing in- ech de ini ions, how i is suppo ed by digi al ans o ma ion, and he inancial isks associa ed wi h he use o ML. A comp ehensi e s udy om 2019 by di Cas i e al. (2019) ocuses on he de ini- ion o sup- ech and highligh s he need o a mo e p ecise no ion o wha o include as ”inno a i e echnology” a he se ice o a inancial au ho i y. I p esen s se e al use cases and classi ies he echnologies on o ma u i y le els (named in he pape as ”gene a ions”), concluding ha he iden i ied ini ia i es (applica ions o inno a i e echnologies o suppo he ac i i ies ca ied ou by inancial egula o s and au ho - i ies) a e mos ly expe imen al. The au ho s sugges an in e na ional coo dina ion e o and alignmen o c ea e syne gies ha le e age sup- ech de elopmen . The Bank o I aly p esen ed a use case o a classi ica ion p oblem (deducing he ins i u ional sec o code o a company based on i s cha ac e is ics) (Massa o e al., 2020). Al hough his wo k is no ela ed o isk assessmen , i p o ides an excellen example o a p oduc ion- eady applica ion o ML o supe iso y asks. Alonso and Ca bo (2020) om Banco de Espa˜na s ess he need o a join s a egy o assess ML models o inc ease anspa ency and p omo e adhe ence o his echnology. The au ho s conclude ML models inc ease he p edic i e capabili y o a c edi de aul classi ie by 20%. The s udy also iden i ies ac o s in c edi isk managemen ha migh inc ease supe iso y cos s. D i en by he ecen p og ess in inancial echnology, Huang e al. (2021) ac- knowledge he complex and hie a chical na u e o inancial da a and he echnologi- cal ba ie s ound when using s a is ics and classic ML. The au ho s hen p oceed o apply ad anced deep lea ning me hods and make use o se e al g aphic p ocesso s o imp o e compu a ion. As a inal ema k ega ding ML applica ions, Doe e al. (2021), om he Bank o In e na ional Se lemen s, p esen ed a policy b ie ing on he Eu opean Money and Finance Fo um, e alua ing o wha ex en cen al banks a e making use o ML and big da a. The au ho s conclude ha al hough cen al banks a e acquain ed wi h big da a, he e exis s a pe sis en need o specialised knowledge on how o use ML h oughou hese o ganisa ions. S ess es s a e also e e enced in hese yea s. In a 2019 s udy, Kola i e al. (2019) hypo hesise ha s ess es s hemsel es a e mo e o an assessmen o a bank’s abili y o deal wi h he isks i is exposed o. This s a emen challenges he common concep ion o s ess es s as a ma ke o a bank’s esilience o ad e se al e na i e mac oeconomic scena ios. Fo his pu pose, he au ho s de elop an ea ly wa ning sys em o assess how Eu opean banks will pe o m on s ess es s. These au ho s sugges su i ing s ess es s depends la gely on he unde lying isk dimensions o indi idual banks. Mo eo e , his pape ea i ms boos ing echniques as winning solu ions, no only o his so o classi ica ion p oblems bu also when applied o s uc u ed da a. As a u u e wo k, he au ho s ecommend a simila app oach using egula o y da a. In he same line o in es iga ion, an EWS was de eloped by Filippopoulou e al. (2020) o p edic bank sys emic isks in he Eu ozone. This s udy s a s by analysing he impo ance o he indica o s ha a e usually applied and p esen s a model ha de ec s a sys emic c isis one o ou yea s be o ehand. In spi e o using a classic mul i a ia e bina y logis ic eg ession model, he me hodology adop ed o his EWS 25 Doc o al P og amme - In o ma ion Managemen classi ica ion, o ins ance, ” ailu e” o ”no ailu e” o a bank. This a ge a iable is de i ed om a se o inancial a ios, mos o en om public o p oxy da ase s. Fo his s udy, we conside he classi ica ion me hod p esen ed in a well-es ablished and widely app o ed me hodology o isk measu emen - he Supe iso y Re iew and E alua ion P ocess (SREP) (Bank) - de ined by he ECB in coope a ion wi h he Na ional Compe en Au ho i ies (NCAs). This is he p ocess h ough which supe - iso s pe iodically assess and measu e he isk o each bank om i e pe spec i es: liquidi y, c edi , ma ke , ope a ional, and p o i abili y. The au ho s suppo ou isk classi ica ion on he au oma ic Risk Assessmen Sys em (RAS), which is hen eclassi ied acco ding o expe judgemen . This me hodology uses eal supe iso y da a collec ed h ough he Eu opean Banking Au ho i y (EBA) di ec i e o Implemen ing Technical S anda ds (Au ho - i y, 2013), wi hin he scope o he Single Supe iso y Mechanism (SSM) (Commis- sion, 2015). Da a is used o classi y each ins i u ion in e ms o i s isk le el, acco ding o he au oma ic isk assessmen sys em om he SREP p ocess. These obse a ions ange om 2014 un il Ma ch 2021. The da a used in his esea ch is ex ensi ely alida ed, hus ensu ing a posi i e co ela ion wi h liquidi y isk assess- men capabili ies (Ng, 2011). 3.1.2 Machine lea ning o isk assessmen Risk assessmen is a p edominan ly quan i a i e exe cise, o en adjus ed h ough ex- pe judgemen . The use o machine lea ning me hods om a cen al bank pe spec- i e is a ecen opic o in e es , no only om NCAs and o he agencies’ pe spec i e, bu also om he academic poin o iew. Since he ea ly 2000s, isk assessmen has been iden i ied as a op p io i y o he e icien use o inancial esou ces (Galindo and Tamayo, 2000). Ea ly in ha decade, he same au ho s es ablished ha ee-based models a e mo e adequa e in p edic ion asks when compa ed o a i icial neu al ne wo ks (ANN), using s uc u ed da a. This esul is ein o ced by o he publica ions, h oughou he yea s. Kola i e al. (2019) speci ically add ess s ess es ing, sugges ing i is an assessmen o a bank’s abili y o deal wi h he isk i is exposed o, a he han he bank’s ac ual esilience. Recen echnological e olu ion has been suppo ing he de elopmen o mo e sophis ica ed models (S ydom and Buckley, 2019), like deep lea ning (DL) models, as well as new ensemble me hods like ex eme g adien boos ing (XGBoos ) (Abell´an and Cas ellano, 2017), due o hei capabili y o cap u e he complexi y o his ype o phenomenon. DL i s eappea ed in 2012 wi h ImageNe (K izhe sky e al.). Howe e , DL was applied o inancial isk assessmen only in 2016. Das ile e al. (2020) con i m DL as a p omising ool in isk assessmen , in pa icula o c edi isk. They hypo hesise ex apola ing his app oach o o he isk pe spec i es, al hough he lack o in e p e abili y o DL is seen by hese au ho s as he main ba ie o adop ing his app oach. A he same ime, se e al s udies showcase he le el o p ecision wi h which deep lea ning models adap o s uc u ed da a. Pe opoulos e al. (2018) expand on he use o ad anced ML echniques om a supe iso y pe spec i e. These au ho s de eloped an Ea ly Wa ning Sys em (EWS) o c edi isk p edic ion, using da a om G eek banks’ co po a e loans (Bank o G eece; 2005-2015).Al hough XGBoos eme ged as he bes model, DNNs also p esen ed p omising esul s. Simila ly o 32 Doc o al P og amme - In o ma ion Managemen wha I u iaga and Sanz (2015) ha e demons a ed, modelling a imeline e olu ion is whe e neu al ne wo ks excel (in his case, deep neu al ne wo ks - DNN’s). As in bank up cy p edic ion, using machine lea ning o model a isk assessmen usually sums up o a classi ica ion ask whe e he de eloped model assigns a bina y esul o a ce ain obse a ion o con ex : ” ail” o ”no ail”. This means ha o a se o independen a iables/indica o s, ha ep esen a bank’s con ex in a ce ain pe iod, he model will i s lea n, hen p edic , whe he ha bank will go bank up o no , wi h a pa icula deg ee o ce ain y. On he business side, i is c ucial o unde s and how banks, na ional compe en au ho i ies and o he agencies a e adap ing o his e olu ion. In pa icula , we a e in e es ed in how cen al banks use inno a i e echnologies o le e age hei analy ical capabili ies, namely o isk assessmen . Acco ding o wha S ock and Wa son (2001) o mula e ha mac oeconome i- cians a policy ins i u ions do, NCAs a e esponsible o : 1. Summa ising and analysing da a; 2. Fo ecas ing he key mac oeconomic a iables; 3. Conduc ing isk analysis and balance o unce ain ies; 4. Pe o ming s uc u al/causal analysis, as well as scena io analysis; 5. Making decisions, communica ing hem and jus i ying hese decisions is-a- is he public. A s udy conduc ed by B oede s and P enio (2018) showcases he expe ience o ea ly use s o inno a i e echnology in supe ision (sup- ech). This wo k p esen s a new de ini ion o sup- ech and shows how i is used o da a collec ion and analy ics. Chak abo y and Joseph (2017) published a simila s udy whe e he au ho s compile, p esen and compa e he app oaches adop ed by NCAs and o he agencies. As no ed be o e, he amoun o a ailable da a eme ges as an impo an ec o o he de elopmen o decision suppo sys ems based on ML. Massa o e al. (2020) p esen a p oduc ion- eady solu ion using ML o suppo a NCA’s e e yday asks. Al hough his wo k is no a isk assessmen ool, i p o es how hese NCAs can le e age on sup- ech. We ound only one pape add essing isk assessmen using ML, om a supe i- so y pe spec i e (Filippopoulou e al., 2020). The EWS de eloped by hese au ho s is o g ea ele ance o cen al banks. I add esses isk assessmen , bu mos im- po an ly, i uses eal da a ga he ed in he a e ma h o he 2008 economic collapse (Eu opean Cen al Bank Mac op uden ial Da abase). Pompella and Dicanio (2017) also p opose an EWS o ale o banks’ dis ess signs. The au ho s p opose a c edi isk model o help adjus ing a ing assignmen s by he esponsible agencies. Along wi h Filippopoulou e al. (2020), hese indings sugges EWS as eliable ins umen s suppo ing supe iso y p ocesses. The pauci y o s udies such as he one jus men ioned, is a gap we p opose o add ess. To he bes o ou knowledge, he e a e no pape s add essing liquidi y isk assessmen om a supe iso y pe spec i e. Addi ionally, his wo k uses eal-wo ld da a collec ed a a cen al bank in he con ex o supe iso y di ec i es. The ac ha his ype o da ase s a e p i ileged and he e o e con iden ial u he jus i ies he nonexis ence o simila s udies. 33 Doc o al P og amme - In o ma ion Managemen The ew s udies add essing isk assessmen wi h ML echniques use public o p oxy da ase s. These ea ly wo ks se he one o he pa icula use case o cen- al banks. In he supe iso y con ex , da a is con iden ial and he p ocesses a e suppo ed by Eu opean-wide legisla ion, hus making hese pape s mo e likely o s em om join wo ks wi h NCAs. Addi ionally, we do no use a sample da ase bu a he he en i e popula ion: he Po uguese banking sec o . Also suppo ing he no el y o his wo k is he isk assessmen me hodology used: he quan i a i e pilla o SREP, he Risk Assessmen Sys em (RAS). We model he isk assessmen ask h ough a classi ica ion p oblem. As opposed o he pape s ci ed abo e, we p opose expanding he usual bina y classi ica ion in o mul iple classes, acco ding o banks’ isk le el and as es ablished in he RAS me hodology: 1. low isk; 2. medium-low isk; 3. medium isk; 4. high isk. This app oach ensu es ha we can look h ough he same lenses a all banks in he Eu o-a ea, making hese assessmen s compa able, eplicable and anspa en . In his wo k, we decide o conside solely liquidi y isk due o i s high impo ance o a bank’s inancial heal h (Ven o and Ganga, 2009). A liquidi y c isis can lead a bank o bank up cy in less han a week (Shah e al., 2018). The e o e, i is o he u mos impo ance o deli e inno a i e ools ha inc ease he cu en analy ical capabili ies o cen al banks. We aim o p o ide a solid base o a scena io analysis ool. 3.2 Me hodology The undamen al pu pose o machine lea ning (ML) is ex ac ing p edic ions om unde lying da a (o Big Da a). Gene ally, Machine Lea ning algo i hms a e applied o da a o ge insigh s om i . In his case we a e using C oss Sec ional Da a, ha can be cap u ed a any poin in ime. Using in o ma ion om p e iously obse ed ci cums ances (c oss sec ional da a), ML algo i hms can p edic alues pe aining o e en s ha ha e ye o occu . Figu e 3.1: Me hodology p ocess o e iew. Figu e 3.1 displays he s eps pe o med in he expe imen al phase o his s udy. We s a ed by e ie ing he da a om he Banco de Po ugal p oduc ion da abase 34 Doc o al P og amme - In o ma ion Managemen o supe iso y da a. This da ase includes all he a ailable ea u es, as well as he p e-compu ed a ge – he RAS sco e o liquidi y isk. Da a ans o ma ion comp ises da a cleaning, implemen ing a s a egy o deal wi h missing alues, and he ea u e selec ion p ocess. In he expe imen phase, we compa e h ee di e en app oaches o e alua e he ML algo i hms o his ask: he classic ain- es spli , he mo e accu a e c oss- alida ion, and he TPOT Au oML amewo k (Olson e al., 2016). We hen use he 1-sco e and he con usion ma ices o compa e he esul s and inally, selec he bes model. In u u e use, his model can be deployed as an Ea ly Wa ning Sys em making p edic ions o he liquidi y isk le el. In his sec ion we will desc ibe he me hods used in his esea ch, om da a ga he ing o model pe o mance e alua ion. 3.2.1 The Da a This s udy elies on supe iso y da a collec ed by Banco de Po ugal (Po uguese Cen al Bank - BdP) wi hin he Capi al Requi emen s Regula ion (CRR) and Cap- i al Requi emen s Di ec i e IV (CRD IV) Pa liamen (2013). The da a anges om Ma ch 2014 un il Ma ch 2021. Depending on i s na u e, some da a is ga he ed mon hly while in o he cases i is ga he ed qua e ly (Au ho i y, 2013). Due o con iden iali y issues, he da ase used in his s udy canno be made a ailable o public consul . Da a is ex ac ed ia SQL que y om BdP’s p oduc ion da abase in o a comma- sepa a ed- alues (cs ) ile o be impo ed using he Py hon p og amming language. An ex ac ion ou ine was implemen ed o assu e consis ency and au oma ion in da a ga he ing. No il e is applied ega ding e e ence da e, ins i u ions o le el o consolida ion. The ex ac ion is s uc u ed in wo s eps: 1. Fi s , he ea u es a e selec ed om he epo ed da a. These belong o he 4 main epo ing amewo ks o banking supe ision: Financial Repo ing, Common Repo ing, Asse Encumb ance and Funding Plans. This se encom- passes all possible p edic o s. 2. The a ge a iables a e selec ed. These a e compu ed h ough a co po a e calcula ion p ocess bu all in e media e a iables a e disca ded, in o de o a oid any possible ma hema ical ela ion be ween ea u es and a ge . The da a esides in a ela ional da abase whe e each ow ep esen s a epo ed alue. This means ha in he da a sou ce, se e al ows ep esen a single obse - a ion. Du ing ex ac ion, da a is anonymised using MD5 algo i hm wi hin a hash unc ion. This s ep assu es he same iden i ie o e e y ow in he same obse a ion. The base da ase has he ollowing opology: 1. ID - a hash code ep esen ing each obse a ion’s iden i ie ; 2. a iable - a code wi h business meaning ha ep esen s each epo ed alue; 3. al - he ac ual nume ic alue o he a iable. 35 Doc o al P og amme - In o ma ion Managemen 3.2.2 T ans o ma ions A py hon ou ine impo s he CSV ile, p epa ing he da a o machine lea ning algo i hms. The i s s ep is pi o ing he da a se so ha each o he esul ing lines co esponds o an obse a ion. Subsequen ly, we go h ough he da a cleaning p ocess ha s a s by disca ding he a ge columns ha all ou o he liquidi y con ex . By his s age, each ow co esponds o a single obse a ion, and he las column ep esen s ou a ge a iable ( he RAS liquidi y isk sco e). The o he columns po ay all he ea u es a ailable in ou da ase . The nex s eps delinea e unde which ci cums ances a ow o column is disca ded om ou da ase : 1. Rows o which he a ge a iable is null. 2. Rows ha ha e a a ge a iable 0. This alue ep esen s a non-applicable obse a ion. 3. Rows whe e all ea u es/columns a e null. 4. Null columns: e e y column/ ea u e has a leas one epo ed alue. A e comple ing he p e ious s eps, we mus con i m ha e e y ea u e s ill has alues. Finally, we deal wi h missing alues o each ea u e. As poin ed ou by Madley- Dowd e al. (2019), mul iple impu a ions can a ain unbiased esul s up un il 90% o missing da a. Since in ou da ase we ha e a mos 20% o missing alues, we do no disca d obse a ions based on his c i e ia. Ins ead, we use he median o ill ou he missing alues, which is he mos adequa e s a egy o nume ic da ase s whe e he ea u es p esen di e en dis ibu ions (Acuna and Rod iguez, 2004). I wi hin he same ea u e/column we ha e simila mean and median i is indi e en which s a egy o use. The use o he median gi es a mo e app op ia e idea o da a dis ibu ion. A e unde going his p ocess, he inal sample included 5299 obse a ions. 3.2.3 Fea u e Selec ion The selec ion o he mos ele an p edic o s is an impo an s ep, no only o educing compu a ional ime, bu also o compa e and con as wi h he business pe spec i e, he ECB Risk Assessmen Me hodology. A e cleaning he da a and d opping some non- ep esen a i e ea u es we a e s ill dealing wi h he o al uni e se o a ailable da a. Fo he ea u e selec ion p ocess we used Random Fo es Classi ie wi h an 85% h eshold o he ea u e impo ance. This me hod was chosen due o i s abili y o ank he pu i y o each node (gini impu i y): g ea es impu i y dec ease occu s a he op o he ee (nea oo le el) whe eas smalle impu i y dec ease a e is obse ed a he end (nea lea nodes). When his algo i hm p unes below a pa icula node, i c ea es a subse o he mos impo an ea u es. Th ough his s a egy, we a e able o echnically assess he ele ance o each ea u e ega ding he a iable we wan o p edic and selec he ones ha explain 85% ( he impo ance h eshold de ined in he algo i hm) o ou a ge a iable. The 36 Doc o al P og amme - In o ma ion Managemen inal da ase has a o al o 3409 ea u es selec ed om a uni e se o 82559 p edic o s, and 5299 obse a ions. A e wa ds, we compa e he simila i y o he ob ained ea u es wi h he ones he me hodology highligh s. This, pe se, is a use ul analysis since i gi es hin s o he analys s on which indica o s o moni o mo e closely. Fo he pu pose o educing compu a ional ime we ha e also conside ed, a i s , he P incipal Componen Analysis (PCA). Al hough his me hod is associa ed wi h dimensionali y educ ion, i s use comp omises model explainabili y. A he same ime, PCA loses ack o he ea u es ha be e ep esen ou a ge a iable, by p ojec ing he ea u e space in o a lowe dimensional space. A he end o his p ocess we compu e he co ela ion ma ix o he da ase o assu e he e is no a high co ela ion be ween ea u es and a ge . This would sugges ha a ce ain ea u e ep esen s he same phenomena as he a ge . The co ela ion indices ange be ween a posi i e 26% and a nega i e 32%. 3.2.4 Expe imen s The expe imen s ca ied ou o assess and compa e he pe o mance o each model we e o ganised in h ee sepa a e phases, each o which is explained in he ollowing subsec ions. Fi s , we adop ed he mos s aigh - o wa d app oach o spli ing he da a in o wo se s, he ain and es se s. A e wa ds, we use c oss alida ion o measu e he a e age pe o mance o each model, conside ing e e y obse a ion o ei he aining o es ing. Finally, we use an au o-ml lib a y, TPOT (Olson e al., 2016), o ha e ano he e alua ion pe spec i e. Fo each o he h ee app oaches we calcula e a measu e o pe o mance/sco ing o bo h ain and es se s. Fu he mo e, we compu e he con usion ma ix o a p ecise pic u e o each model’s p edic ion. We ha e selec ed a lis o some o he mos common machine lea ning algo i hms used o classi ica ion p oblems. Fo he pu pose o hese expe imen s we ha e selec ed sciki -lea n implemen a ion o he ollowing models: 1. Logis ic Reg ession (LG) by Cox (1958), o Mul inomial Logis ic Reg ession, is an ex ension o he Bi a ia e Logis ic Reg ession p oposed by McCullagh and Nelde in 1989 (Glonek and McCullagh, 1995) o p oblems wi h mo e han wo disc e e ou comes. The o iginal app oach was designed o bina y p oblems, and he a ge a iable was modelled h ough a binomial p obabili y dis ibu ion unc ion. In i s mul iclass o m, he p obabili y is dis ibu ed by he numbe o classes o he p oblem a hand. In his pape , we used he sciki - lea n implemen a ion o he Logis ic Reg ession o mul i-classes (Ped egosa e al., 2011). 2. Suppo Vec o Machine Classi ie (SVC) - o Mul i-class Suppo Vec o Ma- chine - is a gene alisa ion p oposed by Wes on and Wa kins (1998) o he bina y classi ica ion Suppo Vec o . Ins ead o compu ing he p obabili y o an obse a ion co esponding o a ce ain class (like he Logis ic Reg ession), his me hod ep esen s all da apoin s in an n-dimensional space, and aims a c ea ing a bounda y, called a hype plane, ha sepa a es he da apoin s in o classes. The algo i hm ies o maximise he dis ance be ween he bounda y and he nea es da apoin s. Real-wo ld da a is seldom linea ly sepa able, so 37 Doc o al P og amme - In o ma ion Managemen i becomes compu a ionally expensi e o p ojec all da a in o a highe di- mensional space o calcula ing he dis ances o he op imal bounda y. To o e come his compu a ional hu dle, SVM uses he ke nel ick, a me hod ha uses a ke nel unc ion ha akes wo ec o s/da apoin s in he o iginal space and compu es hei do p oduc in he ea u e space. Since he ec o s a e no malised he esul is ela ed o he Euclidean dis ance o bo h ec o s - he dis ance we wan ed o compu e. In o he wo ds, his me hod sho cu s he compu a ion o he dis ances om he da apoin s o he possible hype planes, by pe o ming hem in he o iginal n-dimensional space, hus educing wall ime (Adankon and Che ie , 2009). We ha e used sciki -lea n implemen a ion o SVM based on libs m lib a y. 3. Nai e Bayes Classi ie (NBC) is a supe ised lea ning me hod based on Bayes heo em, based upon he s a is ical independence o ea u es. This simpli ied app oach o lea ning shows i is up o pa wi h mo e sophis ica ed classi ie s, namely when dealing wi h high dimensionali y and complex classi ica ion p ob- lems (Rish, 2001). Nai e Bayes algo i hms a e hus e y e icien o ain and equi e li le da a o con e ge. This de i es om he ac ha hey only e- qui e o compu e he p obabili y o each class, he condi ional p obabili ies o each inpu alue gi en a ce ain class, and he mean and s anda d de ia ion alues o each a ibu e o each class. In his pape , we use he Gaussian Nai e Bayes implemen a ion om sciki -lea n which p esupposes a Gaussian dis ibu ion o he ea u es. 4. Random Fo es Classi ie (RFC) is a lea ning me hod ha combines ee p e- dic o s wo king oge he o minimise he e o (B eiman, 2001). As ho oughly explained by Fawag eh e al. (2014), each decision ee in he o es is a base classi ie using a sample o he ins ances in-bag, hence he bagging echnique. The ees a e combined h ough a o ing sys em - one o e pe ee - whe e he o es chooses he class wi h mos o es. Ano he aspec ha imp o ed he andomness o he ees was he use o he Gini index - ea u es wi h he highes index a e used o spli he inne node o he ee. This algo i hm p esen s g ea esul s when dealing wi h da a noise and a oiding o e i , and handles la ge da ase s wi h high dimensionali y. He e again, we a e using i s sciki -lea n implemen a ion. 5. Ex eme G adien Boos ing (XGBC) Classi ie p oposed by Chen and Gues in (2016) is a machine lea ning algo i hm used o ee boos ing ha uses da a comp ession and sha ding (a da a pa i ion echnique) o scale o la ge amoun s o da a. Due o i s capabili y o a oid o e i ing and i s e icien use o la ge amoun s o da a, i has become one o he mos popula ML me hods in he las ew yea s (Sahin, 2020). Ha ing F iedman (2001) g adien boos ing ech- nique as i s pilla , XGBoos uses a di e en iable loss unc ion and op imises i wi h g adien descen algo i hm, in o de o build an ensemble o classi ica ion ees. Fo his algo i hm, we ha e used he au ho s’ implemen a ion package (Chen and Gues in, 2016). The TPOT au o-ml lib a y au oma ically selec s he bes model and we use ha esul o compa e wi h he o he s. 38 Doc o al P og amme - In o ma ion Managemen In o de o ha e all ea u es in a simila scale we ha e applied a scaling me hod when p ep ocessing he da a. MinMaxScale was he bes choice since i p ese es he shape o he o iginal dis ibu ion. I does no signi ican ly change he in o ma- ion embedded in he o iginal da a. No e ha MinMaxScale does no educe he impo ance o ou lie s. The de aul ange o he ea u e e u ned by MinMaxScale is 0 o 1. He e we p esen a lis o he main cha ac e is ics o he expe imen s’ en i onmen : 1. Leno o ThinkPad P50 wi h an In el Xeon p ocesso (2.8GHz), 32 GB o RAM, 1 TB SSD; 2. Windows 10 64-bi s; 3. Py hon 3.9.1 64-bi s; 4. Pandas 1.2.0; 5. sciki -lea n 0.24.0; 6. TPOT 0.11.7. Pe o mance measu es We used wo di e en ools o compa ison pu poses: he con usion ma ix and he 1-sco e. The con usion ma ix is he mos de ailed iew o how a pa icula machine lea ning model is pe o ming in a classi ica ion p oblem (Tha wa , 2018). Th ough his ool, we a e able o assess each o ou model’s p edic ions and compa e hem wi h he co ec alue. Figu e 3.2: Example o a con usion ma ix o a bina y classi ica ion p oblem. Figu e 3.2 shows a gene ic ma ix o a bina y classi ica ion p oblem whe e we can obse e each possible classi ica ion: •T ue posi i e (TP) co esponds o he model co ec hi s. •False nega i e (FN) ep esen s e e y missed case, whe e he model unde es i- ma ed. •False posi i e (FP) ep esen s alse ala ms, whe e he model o e es ima ed. •T ue nega i e (TN) co esponds o he co ec ejec ions made by he model. 39 Doc o al P og amme - In o ma ion Managemen Fo ou speci ic p oblem whe e we ha e ou classes ep esen ing he isk le els o liquidi y, we will ha e a 4X4 ma ix o each model, which is simply a gene alisa ion o he one jus p esen ed. The e a e se e al me ics ha one can ex ac om hese s a is ics. Howe e we will ocus on he 1-sco e and wo o he s de i ed om i , p ecision - o posi i e p edic i e alues, ha is he numbe o posi i e esul s ha a e ue posi i es - and ecall - also known as sensi i i y o ue posi i e a e, which measu es he numbe o posi i e hi s among all he posi i es: • 1-sco e ep esen s he ha monic mean o p ecision and ecall. I is mos sui ed o une en class dis ibu ions, as is he case o ou da ase . I is calcula ed as 1 = 2 ∗p ecision ∗ ecall p ecision + ecall (3.1) whe e p ecision =TP TP +FP (3.2) ecall =TP TP +FN (3.3) T ain- es spli Ou i s app oach o e alua ing he pe o mance o each model is h ough a ain- es spli o he a ailable sample da a. As a gene al p inciple, we used 80% o he da a o aining and 20% o es ing. The assessmen is o ganized as ollows: 1. Use he MinMaxScale , as speci ied abo e; 2. I e a e h ough all machine lea ning models; 3. Fi he model o he da a; 4. Assess ain and es sco es; 5. Compu e he con usion ma ix; 6. S o e he esul s. C oss- alida ion When we a e dealing wi h small o medium da ase s a simple ain- es spli will mos likely mis ep esen ou eal-wo ld p oblem by missing some classes. This is he main indica ion o using c oss- alida ion, whe e e e y single obse a ion is eligible o he ain and es se s. The echnique consis s o spli ing he da ase in o a speci ic numbe o olds, o pa i ions, and i e a ing h ough he pa i ions. In Figu e 3.3 we pic u e how a 5- old c oss- alida ion example would p ocess. Fi s , he da ase is spli in o 5 olds. Then, in each o he i e i e a ions, one o he olds assumes he ole o es old and he o he ou as aining old. In each i e a ion, he machine lea ning algo i hms a e ained on he aining old, and hei pe o mance is assessed on he es old. By he end o he i e i e a ions, he a e age 40 Doc o al P og amme - In o ma ion Managemen Figu e 3.3: Example o a 5- old c oss- alida ion p ocess. o he pe o mance ob ained on each i e a ion is he alue conside ed o compa ison. C oss- alida ion is he p e e ed me hod o assessing model pe o mance because i gi es models he oppo uni y o ain on mul iple ain- es spli s. This will be e indica e how well a model will pe o m on unseen da a. Con e sely, a simple ain- es spli is dependen on jus a single da a spli which can o e es ima e he o e all pe o mance. In his expe imen we used S a i iedKFold (a o m o c oss- alida ion) o p e- se e he pe cen age o samples among classes. The pu pose o his speci ic o m is o he es o be as close as possible o he whole da ase . The s a i ica ion ensu es class equencies o he pa i ions a e equal o he comple e da ase . This is pa icula ly ad an ageous in an imbalanced da ase scena io, whe e his me hod ensu es e e y class is ep esen ed. The use o c oss- alida ion can also aise some issues. Since we a e assessing pe o mance o a model on se e al spli s, si ua ions may a ise whe e da a leaks om one i e a ion o ano he . In o he wo ds, da a leakage can happen when we a e lea ning om bo h he es ing and aining se . I we do any p e-p ocessing ou side he c oss- alida ion algo i hm, we will bias ou esul s and mos likely o e i ou model. To a oid his common p oblem we eed ou c oss- alida ion cycle he en i e da ase and pe o m e e y ans o ma ion wi hin each i e a ion. Al hough he au ho s concede ha his epe i ion akes i s oll on pe o mance, he ex a s ep assu es no da a is leaking om each o he spli s o i e a ions. F1-sco e is used as a pe o mance measu e since i keeps a balance be ween p ecision and ecall. Fu he mo e, since we obse e une en class dis ibu ion in he da ase , F1-sco e is mo e app op ia e han AUC (F1 gi es a sco e o a speci ic h eshold, whe eas AUC a e ages o e all possible h esholds). Con usion ma ix was selec ed as he bes ool o desc ibing pe o mance on a classi ica ion model. This is an NxN ma ix whe e N is he numbe o classes in ou classi ica ion p oblem (as men ioned ea lie , classes 1, 2, 3, and 4 ep esen ing he isk ie s o any gi en inancial ins i u ion). 41 Doc o al P og amme - In o ma ion Managemen We also ind ele an o include expe judgemen o ein o ce o inal isk assess- men . To his end, quali a i e da a sou ces like in e nal no es and isk assessmen epo s, as well as isk sco es eassigned by he supe iso s should con ibu e o he model’s lea ning phase. Finally, we belie e he o he isk pe spec i es comp ised in he SREP me hod- ology should also be add essed using he same me hodology. Ul ima ely, combining all isk pe spec i es could be a s epping s one o egula o s as a suppo o he SREP exe cise. 48 Chap e 4 App oaching Eu opean Supe iso y Risk Assessmen wi h SupTech: A P oposal o an Ea ly Wa ning Sys em 4.1 In oduc ion In ecen yea s, he use o decision suppo sys ems has sky ocke ed, wi h machine lea ning (ML) spea heading he change. The inancial indus y has always been one o he main d i e s o ha de elopmen (Zopounidis e al., 1997). As he amoun o da a collec ed soa s and compu ing powe ises o mee he challenge, he use o classical s a is ics such as linea and logis ic eg essions is g adually declining. Al hough hey we e once he mains ay o decision suppo sys ems, nowadays hey end o be ecalled spo adically, and mainly o hei be e comp ehensibili y in compa ison o mos ML models (Yang and Wu, 2021). The cu en esea ch p oblem is how o le e age on ML o suppo isk assessmen p ocesses a cen al banks, using quan i a i e supe iso y da a. Recen uses o machine lea ning ha e un eiled da a pa e ns ha we e as o ye undisco e ed (Huang e al., 2021). These applica ions ha e also expanded o he ields o egula ion and supe ision, as desc ibed by He ig (2021). Fo supe iso y pu poses, he e has been a huge inc ease o in e es in de eloping sup- ech ools. As Bee man e al. (2021) epo ed, he numbe o ongoing ML p ojec s in his ield sky ocke ed om 12 in 2019 o 71 as o Decembe 2021. The pandemic o ced an o - si e app oach o wha was p e iously equi ed o be done in pe son. In he pas wo yea s we ha e seen an inc easingly highe numbe o p oduc ion- eady sys ems ap- plying ML o suppo cen al banks’ asks (Massa o e al., 2020). F om he speci ic s andpoin o supe ision, he wo k om Filippopoulou e al. (2020) is a wa e shed in EWS de elopmen a cen al banks, using EBC Mac o-p uden ial Da abase o ad- d ess c edi isk. This wo k, along wi h he EWS p oposed by Pompella and Dicanio (2017), suppo s he impo ance o hese sys ems o suppo a ing assignmen s and ale o dis ess signals. The amoun o da a e ie ed in he supe iso y amewo k is o e whelmingly high (Au ho i y, 2013). Addi ionally, supe iso s o en ask o complemen a y in- o ma ion, ei he quan i a i e o quali a i e. E en hough Na ional Cen al Banks 49 Doc o al P og amme - In o ma ion Managemen (NCBs) a e equipped wi h business in elligence sys ems ha allow hem o o ganise mos quan i a i e in o ma ion, da a analysis is mos ly done on a ad-hoc manne ha is imp ac ical o a p omp spo ing o isky e en s (B oede s and P enio, 2018). Besides, his me hod only looks a pas e en s, making i impossible o sys ema i- cally es al e na i e economic scena ios. Fu he mo e, we mus men ion ha isk me hodologies migh a y, making i di icul o compa e no only he assessmen s, bu also he e olu ion o he classi ica ions. T adi ional app oaches al eady se ou an o ganised pe spec i e o he epo ed da a, h ough dashboa ds and epo s ha p o ide agg ega ed and speci ic iews o key indica o s (di Cas i e al., 2019). Howe e , hese app oaches only conside pas e en s and hey a e cons ained by he egula o y amewo k (no o men ion, hey lack wha -i analysis and decision p ocesses buil on ha da a). The use o inno a i e echnologies o suppo supe iso y p ocesses is de ined by B oede s and P enio (2018); Doe e al. (2021) as sup- ech, and hese au ho s summa ise he ba ie s o adop ion in h ee main i ems: 1. F equen egula o y upda es; 2. Conse a i e indus y; 3. Lack o quali ied human esou ces. F om a da a pe spec i e, Ea ly Wa ning Sys ems o p edic ing banking c isis ha e also been in he spo ligh . Casabianca e al. (2019); Consoli e al. (2021) a e some o he many examples o landma k indings in ha a ea, along wi h he p e iously men ioned Filippopoulou e al. (2020). Howe e , none o hese au ho s explo e he in o ma ion a ailable in he Eu opean supe iso y amewo k. In a p e ious wo k, we ha e add essed he issues o using a single isk me hod- ology, selec ing li e a u e-suppo ed ML models o e alua e he isk le el o banks, and using up- o-da e eal-wo ld supe iso y da a om he Po uguese banking sec- o (Gue a e al., 2022). The p e ious wo k add essed he concep o liquidi y isk since i is c ucial o a bank’s abili y o ope a e (Ven o and Ganga, 2009) and i can ende a bank nonope a ional in a ma e o days (Shah e al., 2018). In ou pape we expand p e ious indings o he o he isk pe spec i es comp ised in he Supe iso y Re iew and E alua ion P ocess (SREP). In ou cu en s udy, we ha e ex ended he sample om Ma ch 2014 un il Augus 2021. This da a is ex ensi ely alida ed by Banco de Po ugal and Eu opean Cen al Bank (ECB) quali y assu ance p ocesses. The quali y o ga he ed in o ma ion allows o accu a e assessmen , hus ensu ing a posi i e co ela ion be ween isk p edic ion and he obse ed phenomena (Ng, 2011). Ano he key componen o ou app oach is he way we se up he classi ica ion p oblem. Con a y o wha is commonly ound in he li e a u e, we ei e a e he impo ance o conside ing a mul i- ie classi ica ion app oach o his p oblem. Ou da a being p o ided by eal wo ld con ex , we eel highly con iden in expanding om he ail/no- ail classes and adop ing he ou classes comp ised in he RAS me hodology, a Eu opean-wide isk assessmen me hodology: 1. Low- isk; 2. Medium-low isk; 50 Doc o al P og amme - In o ma ion Managemen 3. Medium isk; 4. High isk. Ou wo k also showcases li e a u e-backed ML models o s uc u ed inancial da a ha suppo he e iciency o supe iso y p ocesses. Based on he indings o ou s udy, we p o ide a comp ehensi e guidance o he de elopmen o aluable supe iso y use-cases enhanced by inno a i e echniques. The pu pose o his wo k is o le e age on he abo e-men ioned aspec s, and expand he academic body o knowledge o quan i a i e isk assessmen o p uden- ial supe ision. F om a supe iso ’s s andpoin , we aim o b ing be e insigh s in o he da a and a ain highe e iciency - au oma ing esou ce in ensi e asks and eeing up analys s o mo e in eg a i e analysis (Bee man e al., 2021). As poin ed ou , he e is oom o imp o emen in his ield, since less han 25% o sup- ech sys ems a e exclusi ely in ended o quan i a i e pu poses. Following in ha lead, his wo k de elops a me hodology o add ess each o he isk pe spec i es in he RAS me hodology: c edi , ma ke , ope a ional and p o i abili y. 4.1.1 Rela ed wo k The use o machine lea ning o isk assessmen has been a highly deba ed opic, bo h om an academic and indus y s andpoin . Since he 2000s (Galindo and Tamayo, 2000), isk assessmen has been ecu en ly iden i ied as a op-p io i y in es men o de eloping he da a li e acy o inancial ins i u ions. As ecen ly shown by An unes (2021), isk assessmen by cen al banks is pa amoun o accu a e supe ision and less biased hen he sel -assessmen s ca ied ou by he banks hemsel es. Addi ionally, Galindo and Tamayo (2000) es ablished ha ee-based models pe o m consis en ly be e han a i icial neu al ne wo ks (ANNs) conside ing s uc- u ed inancial da a. This inding is one o he pilla s o ou app oach and i has been con i med by se e al o he au ho s (Xia e al., 2017; Chen and Gues in, 2016; Climen e al., 2019). In hei li e a u e e iew, Leo e al. (2019) highligh ed he popula i y o machine lea ning applica ions o isk managemen in banking indus y, while also no ing he expe imen al na u e o mos app oaches. The au ho s also ema k he disc epancy be ween he high le el o academic esea ch conce ning his a ea e sus he de ac o indus y applica ions. This deba e has ocused a ound wo main issues: •Finding he igh isk assessmen measu e. •Finding he adequa e machine lea ning algo i hm o build a isk assessmen model. Gue a and Cas elli (2021) s udied bo h o hese aspec s app aising se e al me hodologies o assessing dis ess signals. This e iew spans om 2004, when Hil- legeis e al. (2004) u ned he page on wo landma k me hods ( he Z-sco e (Al man, 1968) and he O-sco e (Ohlson, 1980)) by p oposing he use o he Black–Scholes–Me on op ion-p icing model, up un il 2019, when Kou e al. (2019) lis ed he mos common me hodologies o assessing sys emic isk on he inancial sys em. 51 Doc o al P og amme - In o ma ion Managemen On he same opic, Climen e al. (2019) used XGBoos o iden i y he bes p edic o s o bank ailu e and de elop a classi ica ion model o label ailed and non- ailed banks in he Eu ozone. The da a used in hei s udy comp ised 25 annual inancial a ios o comme cial banks. The majo i y o cu en li e a u e con e s he isk assessmen p oblem in o a bina y classi ica ion ask, whe e each bank is labelled as ” ailed o likely o ail” o ”no ail” (Climen e al., 2019; Kola i e al., 2019; Leo e al., 2019; Filippopoulou e al., 2020; Wang e al., 2021). These s udies usually ely on public da ase s, whe e he a ge a iable is de i ed om a se o inancial a ios. A cen al banks, as clea ly poin ed ou by S ock and Wa son (2001), economis s a e esponsible o conduc ing isk analysis and pe o ming scena io es ing. Since he appea ance o he Single Supe iso y Mechanism (Commission, 2015) we a e bea ing wi ness o a s anda disa ion o epo ing equi emen s and me hod- ologies. The he e ogeneous landscape o inancial pe o mance measu es iden i ied in he li e a u e has been inc easingly eplaced by he use o he Supe iso y Re- iew and E alua ion P ocess (SREP) (Bank), leading us o le e age on his isk assessmen me hodology. SREP is an ongoing wo k by he Eu opean Cen al Bank (ECB) and he Na ional Cen al Banks (NCBs) ha p o ides an in eg a ed iew on each bank acco ding o i e isk pe spec i es: liquidi y, c edi , ma ke , ope a ional and p o i abili y. The Risk Assessmen Sys em (RAS) is he quan i a i e pilla o he me hodology, and i is he ocal poin o his wo k. Selec ing he adequa e machine lea ning me hods applied o cen al banking, we ound ha i ecen ly became a ho opic om bo h an academic and NCBs s andpoin (Lee and Shin, 2020; Huang e al., 2021; Wang e al., 2021; Alonso and Ca bo, 2020; An unes, 2021). Bee man e al. (2021) epo ha he pandemic p omp ed NCBs o ely on sup- ech solu ions in hei e e yday p ocesses. Se e al o he su eyed au ho i ies al eady ha e ope a ional sys ems. Fo ins ance, Cen al Bank o B azil has a ool ha examines he whole c edi po olio o a bank o de ec exposu es wi h un ecognised expec ed losses; Bank o Spain is applying in e ence maps o model he ela ionships be ween bo owe and e alua e he isk impac ; and he Mone a y Au ho i y o Singapo e is de eloping a ool o au oma e da a analysis so ha supe iso s can ely on comple e da ase s, ins ead o sampling. Fo his eason, we expanded ou esea ch o applica ions o ML o isk assessmen . By b oadening his esea ch, we can e alua e how ML has been used o inancial s uc u ed da a and hen ocus on he cen al bank case. S ess- es ing is one o he many o ms o isk assessmen ha is pa icula ly used a cen al banks. Kola i e al. (2019) challenged he concep o a bank’s esilience by sugges ing ha i mos ly ep esen s a bank’s abili y o deal wi h a speci ic isk suppo ed by i s own capaci y o abso b i . In such a se ing, applying a isk- ocused me hodology like SREP allows supe iso s o be e assess he oo causes o wha migh o he wise be pe cei ed as a gene al business model issue. Chak abo y and Joseph (2017) p esen ed a se ies o ML applica ions o inancial p oblems and hey analysed he mos equen ly used algo i hms, like ee-based ensembles, a i icial neu al ne wo ks and clus e ing echniques. The au ho s also showcase h ee use-cases a cen al banks, ha es ablish ML as a be e solu ion han adi ional s a is ics. The mos ele an o ou wo k is one ha de elops a se ies o ale s (EWS) based on he balance shee s uc u e o a bank, in a supe iso y con ex . This shows no only how ele an supe iso y da a is o a p oac i e isk 52 Doc o al P og amme - In o ma ion Managemen assessmen , bu also how i can be used o sense he isk p ocli i y o supe ised ins i u ions. Recen echnological de elopmen s ha e allowed newe and mo e complex models o eme ge (S ydom and Buckley, 2019), such as deep lea ning (DL) and ex eme g adien boos ing (XGBoos ) (Abell´an and Cas ellano, 2017). E idence shows hose analysis me hods ha e a unique capaci y o cap u ing he in icacies o inancial phenomena (Ribei o e al., 2012; Huang e al., 2021). I u iaga and Sanz (2015) showed ha modelling ime se ies is whe e DL excels. Also Pe opoulos e al. (2018) le e age on DL’s p ecision and de elop an Ea ly Wa ning Sys em (EWS) o p edic ing ailu e o G eek banks (da a in 2005-2015). This is a landma k epo on he use o ad anced ML in a daily supe iso y con ex . Wang e al. (2021) p oposed an add-on o he con en ional logi -based EWS, which in ol es simula ing expe o ing h ough a Random Fo es based sys em, and ha showed aluable esul s in p edic ing sys emic c ises. B oede s and P enio (2018) o ganise supe iso y inno a ion concep s and p esen a se ies o use-cases whe e ea ly adop e s a e implemen ing inno a i e app oaches (sup- ech), con e ing e ie ed da a in o p edic i e indica o s. These wo ks a e o g ea impo ance o sys ema ise how o implemen his echnology. The inc easing amoun o a ailable da a is one o he main d i e s o he de elopmen o ML-based sys ems, as Chak abo y and Joseph (2017) also ha e claimed. Banking supe ision acknowledges he bene i s o inno a i e echnologies and he impo ance o keeping up wi h he a ie y o sup- ech ini ia i es being de eloped. These ini ia i es ha e he po en ial o d ama ically change he supe iso y p ocess; an icipa ing he conse- quences o cu en beha iou s ins ead o bela edly eac ing o pas e en s. The same au ho s explo e se e al use-cases om he Cen al Bank o he Republic o Aus- ia (OeNB), Mone a y Au ho i y o Singapo e (MAS), Secu i ies and Exchange Commission (SEC), among o he s. Business p ocess e ec i eness, cos educ ion and inc eased analy ical capabili y, a e no ed as he main d i e s o he sup- ech endea ou . These supe iso y agencies epo se e al challenges in explo ing and implemen ing hese echnologies, such as: •The echnical know-how and app op ia e in as uc u e o suppo hese ana- ly ical solu ions; •The legal amewo k o suppo he use o he ele an in o ma ion; •The in e nal suppo om managemen o in es in hese ini ia i es and om he end-use s, o p o ide he expe knowledge and o use and p omo e he new analy ical ools. Boa d (2020) also shows how he balance o supply and demand igni ed he de- elopmen and use o sup- ech ools. F om he demand side, hese au ho s men ion, among o he aspec s, enhanced supe iso y and egula o y equi emen s and im- p o ed isk managemen capabili ies, whe e he au oma ion o da a e ie al and summa isa ion can d as ically imp o e supe iso y p ocesses. F om he supply side, he inc easing a ailabili y o da a and new analy ical me hods a e among he op suppo e s o he abo e-men ioned egula o y necessi ies. Lis ed bene i s o imple- men ing hese g ound-b eaking ools include: •Enhanced analy ical capabili ies; 53 Doc o al P og amme - In o ma ion Managemen •En iched isuals, s emming om s a e-o - he-a da a collec ion o sophis i- ca ed dashboa ds; •Reduced cos s, as a consequence o au oma ion. Ne e heless, adop ing new analy ical p ocesses ine i ably b ings on esh chal- lenges. Recognising his aspec , Jag iani e al. (2018) expand on he impac s o hese new analy ical solu ions and possible isks o adop ing hem, such as: •Thi d-pa y endo isk, whe e banks gi e access o ou side specialis s - da a scien is s and business use s in ol ed in se ing up he ool - ha can lead o da a b eaches. Addi ionally, i he endo is a dominan playe in he ma ke , ha ci cums ance can c ea e a single poin o ailu e in he inancial sys em. •Cybe -secu i y isk, which is ela ed o he p e ious opic, as endo s migh no comply wi h supe iso s’ secu i y equi emen s. Addi ionally, by allowing o ex e nal sou ces o da a, banks and cen al banks become exposed o ha channel and he in o ma ion he ein con ained. •Model isk, whe e sys ems based on complex machine lea ning models o e en black-boxes make decisions ha migh no make sense om a business pe - spec i e, hence p o iding w ong p edic ions. Ano he ac o wi h majo impac in ML use is he comp ehensibili y o he models. Al hough ML models a e seldom capable o explaining p edic ion, hey consis en ly ou pe o m he classic app oaches. Das ile e al. (2020) published a sys ema ic li e a u e e iew con as ing hese echniques o a c edi sco ing p ob- lem, and hey s ess he lack o in e p e abili y o DL as he main ba ie o adop ion in suppo ing inancial decisions. I is wo h bea ing in mind ha poin ing ou he di ec ion o u u e esea ch is as impo an as signalling isks associa ed wi h implemen ing ML models. Kou e al. (2019) p esen a ho ough epo on s a e-o - he-a applica ions and ML echniques o assess sys emic isk. Based on he exis ing echnology, hey sugges se e al u u e wo k a eas, like big-da a analysis, da a-d i en esea ch and policy analysis wi h da a science. 4.2 Me hodology De elopmen s in he a ea o da a science and machine lea ning usually all in o one o wo ca ego ies: de eloping a new compu a ional me hod o be e sol e an exis ing p oblem; o al e na i ely, using he exis ing me hods o add ess a new p oblem. In his wo k, we aim o add ess a p oblem ha was ye o be sol ed using machine lea ning: supe iso y isk modelling. Figu e 4.1 illus a es how we a ained ou objec i e in a s ep-by-s ep diag am, as a de elopmen o wha was p esen ed in Gue a e al. (2022). The i s s ep comp ises a da a e ie al p ocess om Banco de Po ugal supe iso y da a sys em, including a wide se o ea u es and he a ge a iables we wan o model. Nex , he e is a ans o ma ion p ocess ha is esponsible o cleaning he da a, dealing wi h missing alues and selec ing he mos signi ican ea u es. In he ollowing s ep, we compa e he ML models o his ask using ain- es spli , c oss- alida ion 54 Doc o al P og amme - In o ma ion Managemen Figu e 4.1: Me hodology p ocess o e iew. and he TPOT Au oML amewo k (Olson e al., 2016). The 1-sco e and con usion ma ices a e used o compa e he esul s and selec he bes model ha can hen suppo an Ea ly Wa ning Sys em o he RAS isk pe spec i es. In his sec ion we p esen he s eps ca ied ou in his esea ch, beginning wi h explaining how he da a was e ie ed, wha ans o ma ions we e equi ed, which ea u es we e selec ed and i s c i e ia, and inally, how he models’ pe o mance was e alua ed. 4.2.1 The Da a One o he main pilla s o his pape is he supe iso y da a collec ed by Banco de Po ugal (BdP) wi hin he Capi al Requi emen s Regula ion (CRR) and Capi al Requi emen s Di ec i e IV (CRD IV) (Pa liamen , 2013). Ou da a anges om Ma ch 2014 un il Augus 2021 and mos o he da a used o he pu poses o his pape is qua e ly (Au ho i y, 2013). Due o con iden iali y issues, he da ase used in his s udy canno be made a ailable o public consul . Da a is ex ac ed ia an SQL que y om BdP’s p oduc ion da abase (wi h no il e s ega ding e e ence da e, banks o hei consolida ion le el) in o a comma- sepa a ed- alues (cs ) ile. The esul se is impo ed using a Py hon sc ip wi hin Jupy e no ebooks. An ex ac ion ou ine was implemen ed o assu e consis ency and au oma ion in da a ga he ing. To accoun o all possible p edic o s, we ha e selec ed ou ea u e space om he ou main epo ing amewo ks o banking supe ision: Financial Repo ing, Common Repo ing, Asse Encumb ance and Funding Plans. The da a esides in a ela ional da abase whe e each ow ep esen s a epo ed alue. This means ha in he da a sou ce, se e al ows ep esen a single obse a- ion. Du ing ex ac ion, da a is anonymised using MD5 algo i hm wi hin a SQL’s hashing unc ion. This s ep assu es he same iden i ie o e e y ow in he same obse a ion. The ex ac ed da ase ollows his column schema: 1. ID - a hash code ep esen ing each obse a ion’s iden i ie ; 2. a iable - a code wi h business meaning ha ep esen s each epo ed alue; 3. al - he ac ual nume ic alue o he a iable. 4.2.2 T ans o ma ions P epa ing he da a o machine lea ning algo i hms is he single mos c i ical s age in such s udies and p ojec s. The i s s ep in ou s udy is o pi o he da a wi h 55 Doc o al P og amme - In o ma ion Managemen he aim o ha ing each ow co esponding o one obse a ion. This ans o ma ion unco e s he spa si y o ou ea u e space, equi ing null columns o be d opped. Ano he impo an s ep is o ocus he da ase on he isk pe spec i e o be e al- ua ed. In ou s udy we a e add essing c edi , ma ke , ope a ional and p o i abili y isks. When in es iga ing one isk pe spec i e - one speci ic a ge a iable - we d op all he o he s. This migh lead o in alid obse a ions, ha is, obse a ions ha only made sense o a ce ain isk. As a consequence, we disca d he ows o which he selec ed a ge alue is null. Dealing wi h missing alues is he inal s ep o he ans o ma ions phase. Ou da ase is exclusi ely nume ic and each column/ ea u e has i s own dis ibu ion. The e o e, we op ed o inpu ing he missing alues o each ea u e wi h he median, since i p o ides a mo e accu a e pe spec i e on he da a’s dis ibu ion when dealing wi h up o wen y pe cen o missing alues (Acuna and Rod iguez, 2004). By he end o hese s eps ou da ase consis s o 9262 ows and 82576 columns. 4.2.3 Fea u e Selec ion As we saw in he p e ious subsec ion, his da ase is ex emely spa se - he e he inaccu acy o he e m ”ex emely” endea ou s o cap u e he ac ha his is a wide da ase (mo e ea u es han obse a ions). Al hough we ha e conside ed using P incipal Componen Analysis (PCA), his me hod comp omises model comp ehen- sibili y. Since i p ojec s he o iginal ea u es in o a lowe dimensionali y ea u e space, he e is always in o ma ion loss om disca ding he componen s wi h less a iance/in o ma ion. The selec ion c i e ia is based on he co a iance ma ix, and does no accoun o he a ge a iable o be s udied. As his da ase comp ises i e di e en a ge a iables - one pe isk - PCA migh exclude ea u es ega dless o hei con ibu ion o a speci ic a ge . To add ess he abo e-men ioned issues we ha e used he Random Fo es ea u e selec ion algo i hm, wi h an 85% h eshold o ea u e impo ance. T ee-based mod- els a e bes o pe o m his ask since hey no only ake in o accoun he a ge a iable o be explained, bu also a p io i hey ank ea u es acco ding o how well hey imp o e he pu i y o nodes (gini impu i y). The close a node is o he oo , he g ea e impu i y dec ease occu s (i.e. he ”cleane ” da a becomes). Con a ily, lea nodes ha e smalle impu i y dec ease. Hence, p uning below a ce ain node esul s in a subse o he mos impo an ea u es. This me hod allowed us o echnically assess he lis o ea u es ha explain a leas 85% o ou a ge a iable. F om he o iginal o al o 82576 ea u es we selec ed 2608 ea u es - o c edi isk. This numbe a ied o di e en a ge a iables. As a inal check, we ha e compu ed he co ela ion ma ix o each a ge a i- able o assu e ea u es and a ge we e no highly co ela ed - Pea son’s co ela ion coe icien less ha 0.3. 4.2.4 Expe imen s In he ollowing subsec ions we lay ou he h ee app oaches ollowed o assess he bes machine lea ning model: •T ain- es spli : simply spli ing he da ase in ain and es se s. 56 Doc o al P og amme - In o ma ion Managemen •C oss- alida ion: using di e en pa i ions o he da a o es and ain he model on e e y obse a ion, i e a i ely. •TPOT Au o ML: an au o ML amewo k by Olson e al. (2016), o compa ison pu poses. These app oaches p o ide a pe o mance measu e ha summa ises he gene ali- sa ion capabili y o e e y model and allows o a eliable and as compa ison among models. F1-sco e was used as a pe o mance measu e since i keeps a balance be- ween p ecision and ecall. Fu he mo e, since we obse e une en class dis ibu ion in he da ase , F1-sco e is mo e app op ia e han he A ea Unde he Cu e (AUC) ( 1-sco e gi es a sco e o a speci ic h esholds, whe eas AUC a e ages o e all pos- sible h esholds). Fo a ull de ail o each e alua ion, he con usion ma ices a e also p o ided. Fo he pu poses o his s udy we selec ed and e alua ed he pe o mance o each o he ollowing models: •Logis ic Reg ession (LG); used only o benchma king; •k-Nea es Neighbou s Classi ie (kNN); •Random Fo es Classi ie (RFC); •Ex eme G adien Boos ing Classi ie (XGBC). The TPOT amewo k is an Au oML amewo k ha makes use o gene ic p og am- ming o op imise he p ocess o inding he bes model o he p oblem a hand. This is a ising end in he usage o machine lea ning and we ha e included i in o de o o e alua e i s adequacy o his p oblem. All h ee app oaches comp ise an op imisa ion phase, whe e we expe imen wi h a ange o alues o he hype -pa ame e s o each o he conside ed models. Fo bo h he ain- es spli and c oss- alida ion we ca ied ou a 5- old c oss alida ed g id sea ch o he speci ic pa ame e s o each model. The TPOT amewo k has an op imisa ion s ep wi hin i s pipeline ha is ully documen ed. Jus be o e eeding he da a o he ML algo i hms, we used he MinMaxScale o i he ea u es in he same scale. This app oach p ese es ou lie s and he o iginal dis ibu ion o each ea u e, hence conse ing he in o ma ion embedded in he da a. All he expe imen s we e execu ed a Banco de Po ugal using i s compu ing in as uc u e. The speci ica ions o he node assigned o hese expe imen s we e he ollowing: •4 In el(R) Xeon(R) CPUs E7-8891 4 @ 2.80GHz, 32 GB o RAM, 1 TB SSD; •Ubun u 20.04.3 LTS; •Py hon 3.8.10; •Pandas 1.2.0; •sciki -lea n 0.24.0; •TPOT 0.11.7. 57 Doc o al P og amme - In o ma ion Managemen The esul s o he ain- es spli e alua ion a e shown in igu e 4.5. He e he esul s show a dis ibu ion simila o wha we obse ed wi h c edi isk howe e , he sco es a e sligh ly be e . The Logis ic Reg ession esul s sugges ha we ace linea (o close o linea ) bounda ies be ween classes. This eading is also suppo ed by he ac ha i s sco e is close o k-Nea es Neighbou s’. S ill, he use o ensemble ee-based models show a signi ican inc ease in pe - o mance. The spike is no as p ominen as wi h c edi isk, and Random Fo es has again a simila , bu lowe , sco e han XGBoos - on he o de o he decimal pe cen age poin s. Figu e A.3 shows he con usion ma ices o he ain- es spli e alua ion. Figu e 4.6: F1-sco es o each model, using c oss- alida ion app oach. Howe e , a andom ain- es spli migh gi e an unde alued o o e alued pe spec i e o a model’s pe o mance. To alida e hese indings we applied c oss- alida ion wi h 1-sco e o he whole da ase . The esul s o his p ocess a e shown in igu e 4.6 along wi h he e alua ion o he TPOT amewo k. The models show simila sco es when compa ed o each o he , wi h Logis ic Reg ession a ing close o he k-Nea es Neighbou s. Con a ily o wha we obse ed in he ain- es spli , a mo e disce ning look a he esul s shows ha Random Fo es classi ie sligh ly ou pe o ms XGBoos . TPOT comes in hi d place in e ms o pe o mance, and i becomes e en less appealing i we conside i s wall ime. Figu e A.4 p esen s he con usion ma ices o his classi ica ion p ocess. 4.3.3 Ope a ional Risk The sample p o ided o e alua e ope a ional isk has 4819 obse a ions and 3447 ea u es. The wall ime needed o e alua e he models on his sample was: 1. T ain- es spli : 5 minu e and 19 seconds; 2. C oss- alida ion: 18 hou s, 13 minu es and 52 seconds; 3. TPOT amewo k: 2 days, 15 hou s, 31 minu es and 41 seconds. 64 Doc o al P og amme - In o ma ion Managemen The ain- es spli esul s shown in igu e 4.7 pain a di e en pic u e han he o he pe spec i es. Al hough we can obse e a simila dis ibu ion o esul s, he Logis ic Reg ession p esen s below-a e age esul s on unseen da a. Fu he mo e, he k-Nea es Neighbou s classi ie exhibi a sligh imp o emen o he p e ious model. Random Fo es and XGBoos classi ie s again come in o he spo ligh , wi h he la e showing a modes ad an age o less han wo pe cen age poin s. Figu e A.5 shows he con usion ma ices o he ain- es spli , o a de ailed iew o each classi ica ion. Figu e 4.7: F1-sco es o each model, using ain- es spli app oach. Applying c oss- alida ion o his sample e eals se e al pe o mance adjus - men s. Ou non- ee-based models - he Logis ic Reg ession, and k-Nea es Neigh- bou s classi ie - exp essed an inc ease in hei sco e, due o he op imisa ion p ocess. Fo Random Fo es and XGBoos we see mino adjus men s in he 1-sco e, howe e , hei pe o mance di e ence is consis en wi h he ain- es spli app oach. This inding con i ms he abili y o g asp he he e ogenei y o egula o y inancial da a. The TPOT amewo k is again in hi d place, e ealing o be a poo choice due o he mo e han wo and a hal days o p ocessing. Figu e A.6 shows he con usion ma ices o he c oss- alida ion p ocess, o a de ailed iew o he classi ica ions o each model. 65 Doc o al P og amme - In o ma ion Managemen Figu e 4.8: F1-sco es o each model, using c oss- alida ion app oach. 4.3.4 P o i abili y Risk As o ou inal isk pe spec i e - p o i abili y - we used a sample o 6448 obse a ions and 3177 ea u es. The p ocessing and e alua ion imes o each o he app oaches we e: 1. T ain- es spli : 9 minu e and 14 seconds; 2. C oss- alida ion: 1 day, 2hou s, 25 minu es and 58 seconds; 3. TPOT amewo k: 1 day, 11 hou s, 56 minu es and 42 seconds. This is he isk pe spec i e wi h he wo se o e all esul s. Figu e 4.9 shows he ain and es s sco es o each model. Logis ic Reg ession, k-Nea es Neighbou s p esen a pal y pe o mance. E en Random Fo es and XGBoos show some de- c ease in pe o mance, al hough s ill p esen ing good esul s. Figu e A.7 pic u es he de ailed classi ica ions o hese models h ough he con usion ma ices. 66 Doc o al P og amme - In o ma ion Managemen Figu e 4.9: F1-sco es o each model, using ain- es spli app oach. The c oss- alida ion p ocess co ec s o any misclassi ica ion esul ing om a un a ou able ain- es spli . In igu e 4.10 we show he 1-sco es o each model, including he TPOT amewo k. Jus as wi h ain- es spli , he Logis ic Reg ession, and k-Nea es Neighbou s p esen a low sco e, when compa ed o he o he algo i hms and hei pe o mance in o he isk pe spec i es. Al hough his seems no ela ed o class imbalance (see igu e 4.2), he complexi y o he decision bounda ies and he dependence o some o he ea u es migh be he oo cause o hese ounde ing esul s. E en so, he Random Fo es and XGBoos show good esul s, wi h he la e again ou pe o ming he o me . The TPOT amewo k, comes in hi d wi h a e age esul s and one and a hal day o p ocessing, again making i an unsa is ac o y al e na i e o his ask. See igu e A.8 o he con usion ma ices o he c oss- alida ion p ocess. Figu e 4.10: F1-sco es o each model, using c oss- alida ion app oach. 67 Doc o al P og amme - In o ma ion Managemen 4.3.5 Final ema ks Following ou p e ious wo k (Gue a and Cas elli, 2021), we clea ly de ined he equi ed elemen s o modelling he supe iso y isk assessmen p ocess comp ised in RAS. Fi s , we sugges ed he use o SREP’s quan i a i e pilla - Risk Assessmen Sys em - as a s anda d me hodology o compa e he banks a Eu opean le el. This me hodology is al eady es ablished ac oss he Eu o-a ea, hence making i he ideal choice o he ask. Mo eo e , mos wo ks in his a ea adop a bina y classi ica ion o he isk le el o he banks, limi ing he classi ica ion o ” ailu e” o ”no ailu e”. As men ioned be o e, his app oach lacks he lexibili y equi ed o cen al banks o de ec he e ec o dis ess e en s g adually and ea lie in ime. This is accomplished h ough he p og essi e mul i-class scale p o ided in he RAS. Addi ionally, we iden i ied a esea ch gap speci ically add essing he supe iso y use-case. Using eal- wo ld supe iso y da a, designed and e ie ed o egula o y pu poses, i has been p o en o p o ide he mos accu a e ou look (B oede s and P enio, 2018; di Cas i e al., 2019; Massa o e al., 2020; Filippopoulou e al., 2020). We es ed he abo e men ioned elemen s and success ully modelled he liquidi y isk o a bank (Gue a e al., 2022). Based on hose indings, we se ou o gene alise he me hodology and model he emaining isk pe spec i es comp ised in he RAS: c edi , ma ke , ope a ional and p o i abili y. F om a echnical s andpoin , we con i med ha an op imised XGBoos ou pe - o med he o he conside ed models. This is acco dance wi h p e ious li e a u e esul s sugges ing XGBoos pe o ms bes wi h s uc u ed inancial da a. In addi- ion o ha , we ha e es ed i agains he au o ML amewo k TPOT, a ising end in he ield. The esul s showed ha due o he cha ac e is ics o he da ase - la ge numbe o ea u es and spa se da ase - compu ing ime was ex emely axing, e en wi h low pa ame e s o he GP algo i hm. I migh be in e es ing o educe he numbe o ea u es o ewe han en, and see how TPOT pe o ms. F om a business pe spec i e, he no el y wi hin he p esen ed esul s is he ac ha we a e modelling a mul i-class decision p ocess wi h eal-wo ld supe iso y da a. Whe eas o he wo ks ha e no explo ed supe iso y da a, we ely on he Eu opean egula o y amewo k and he da a collec ed wi hin i . This da a is he pilla o supe iso y p ocesses and b ings he s uc u e and con ex o ou models. By elying on hese models, we can de elop ea ly wa ning sys ems capable o an icipa ing dis ess e en s, conside ing he isk measu es abo e, and also gi e supe iso s a ool o es al e na i e economic scena ios o p e en pi alls. 4.4 Conclusion S eamlining an e ec i e supe iso y me hodology equi es an in eg a ed iew o he isks a c edi ins i u ion is subjec o. In ou p e ious wo k we ha e success ully modelled liquidi y isk acco ding o SREP me hodology. Once ha pilla was se , we we e able o apply he same modelling echniques o he o he isk pe spec i es comp ised in he Supe iso y Re iew and E alua ion P ocess (SREP) and i s Risk Assessmen Sys em (RAS): c edi , ma ke , ope a ional and p o i abili y. Based on he quan i iable mains ay o ECB’s Risk Assessmen Me hodology, we classi ied c edi ins i u ions om he Po uguese banking sec o acco ding o hei isk le el on each o he pe spec i es encompassed in he me hodology. We used 68 Doc o al P og amme - In o ma ion Managemen eal-li e supe iso y da a and modelled his decision p ocess by compa ing se e al machine lea ning echniques, benchma ked agains a widely used s a is ical me hod. We ha e eached signi ican esul s clea ly es ablishing ha his decision p ocess can be modelled and ha he ML echniques used ou pe o m he classic s a is ical app oaches. Regula o y supe iso y da a is highly co ela ed and he e ogeneous, making he decision bounda ies o his exe cise a challenging ask. Addi ionally, eal-wo ld e en s a e seldom ep esen ed by balanced da a. All isk le els a e obse ed bu wi h occu ences ha a e subjec o e en s in a speci ic poin in ime. The complexi ies o such eali y we e bes ep esen ed by ensemble ee-based models, like Random Fo es and XGBoos classi ie s. These models can cap u e he he e ogeneous na u e o inancial da a and es ablish clea decision bounda ies wi h li le e o - 1-sco e be ween 87% and 94%. These esul s we e ob ained a e applying an op imisa ion p ocess wi hin he c oss- alida ion cycle. Gi en he compu a ional esou ces a ailable and he cu ing edge gene ic p o- g amming op imisa ion pipeline a ailable h ough TPOT, we expec ed i o ou - pe o m XGBoos . Howe e , TPOT consis en ly came in hi d ega ding 1-sco e, being ou pe o med by Random Fo es and XGBoos . I s long p ocessing imes can be explained by he dedica ed op imisa ion p ocess, and he ac ha ou da ase is spa se (82576 ea u es). The ea u e selec ion p ocess is cos ly in compu a ional sense and i migh accoun o a signi ican sha e o he wall ime. We i mly belie e his wo k is a meaning ul con ibu ion o a se o s akeholde s in ol ed in isk assessmen in he banking sec o : •Na ional Cen al Banks (NCBs) can le e age on he indings o his wo k and use hese models o de elop ea ly wa ning sys ems. These sup- ech ini ia i es a e cu en ly in he limeligh , wi h many p ojec s being de eloped in his a ea by he ECB, Bank o In e na ional Se lemen s (BIS) and wo ldwide NCBs. A decision suppo sys em like his would p o ide an enhanced isk assessmen pe spec i e o supe iso s. •Banks and he consul ing indus y can con ey hese p inciples in o hei own sys ems. Consul ancy companies can u he suppo hei clien s in imple- men ing hei decision suppo sys ems using he da a owned by he banks hemsel es. A bank can hen p oac i ely moni o and adjus hei isk p o ile and s a egy acco ding o he egula o y equi emen s. •Academia can use his wo k o ex end and apply his ype o ML me hodologies o expand i s usage on a egula o y pe spec i e. Fu he mo e, we s ess he pos ula es o using high quali y, highly alida ed ele an da a, and adop ing an uni e sal me hodology o isk assessmen , one ha s anda dises how o app aise a bank. Th ough his pape , we aim o con ibu e o he echnical unde s anding o ML ha can be applied o sup- ech use cases acco ding o he business needs. G ounded in his o ical supe iso y da a, we p opose a Sup-Tech ool ha imp o es he Eu opean supe iso y isk assessmen by p o iding ea ly wa nings on se e al isks. 69 Doc o al P og amme - In o ma ion Managemen 4.4.1 Limi a ions and u u e wo k The e a e se e al aspec s we ha e iden i ied o e he cou se o his s udy ha would me i e ision and imp o emen . The da ase we used in his wo k e lec s he Po uguese banking sec o . Ideally, expanding o he Eu opean le el and using da a om all cen al banks in he Eu o- a ea would p o ide a comple e supe iso y pe spec i e. Addi ionally, mo e di e se da a, wi h mo e business models would s eng hen he ML models p esen ed he e. Each o he isks would also me i om con ex speci ic da a, in o de o enhance he gene alisa ion o each model. This would also allow he supe iso s o access mo e imely decisions. Supe iso y da a is mos ly qua e ly which p e en s quick eac ions o ad e se e en s. By combining i wi h daily da a sou ces, such as ma ke da a, paymen s sys ems and c edi esponsibili ies da a, we migh be able o ob ain a daily signal o each aspec o a bank’s isk. Con i ming his decision pa h will s eng hen he a o emen ioned models and p o ide a unning isk assessmen on which supe iso s can ely on. The abili y o explain he easoning behind each model is o u mos impo ance, in pa icula o c i ical sys ems such as o c isis de ec ion. Explainable AI bene i s hold ue no only o expe s o alida e he decision p ocess ca ied ou by he ML models, bu also as common g ound language o epo any issue o banks. As such, in es ing in explainable models will deli e a be e unde s anding o he echnology, b inging supe iso s close o sup- ech, and will also se o h a clea e communica ion be ween ins i u ions and cen al banks. Combining ou quan i a i e da a wi h quali a i e expe judgemen , using Na - u al Language P ocessing (NLP), will allow o au oma ed sco e adjus men s based on in e nal supe iso y no es and isk assessmen epo s. As a inal ema k, consolida ing he esul s o each isk model wi h he ele- an quali a i e da a could p o ide a single in eg a ed bank sco e as an addi ional measu e o he SREP exe cise. 70 Chap e 5 Conclusion The e e -g owing amoun o da a e ie ed by o ganisa ions has led almos e e y indus y o in es in big da a echnologies. Machine lea ning is one o oday’s op choices o ha es da a, and he inancial sec o has been one o i s main d i e s. Simila ly, Na ional Cen al Banks also h i e by le e aging on inno a ion as a key pilla o hei mission: keeping p ices s able and banks heal hy. Howe e , indus ies a e adop ing hese new echnologies a di e en paces. The much-needed changes on how supe ision is accomplished ha e been delayed due o he inancial sec o being pa icula ly conse a i e, he cons an upda ing o he egula o y ou look, and he lack o quali ied p o essionals who can ca y ou hese changes in a sus ainable manne . This is he iden i ied gap ha igge ed ou esea ch. He e, we use machine lea ning in he supe iso y con ex wi h he pu pose o modelling isk assessmen p ocesses. We achie ed ou objec i e and con ibu ed o he o emen ioned esea ch gap in h ee s eps: 1. We i s e iewed he li e a u e on isk assessmen and machine lea ning a cen al banks. Rega ding his ecen in e sec ion o knowledge a eas, we dis- co e ed he need o a common isk assessmen me hodology. Addi ionally, we gleaned ha he e a e se e al cen al banks ca ying ou sup- ech ini ia i es, e idencing he po en ial o ML in he supe iso y ealm (B oede s and P enio, 2018; Bee man e al., 2021; He ig, 2021). 2. We hen p oceeded o success ully model he liquidi y isk o a bank as a clas- si ica ion p oblem, compa ing se e al ML models. The wo key componen s o his s ep we e he applica ion o ECB’s Risk Assessmen Sys em (RAS) used ac oss he Eu o-a ea, and he use o eal-wo ld supe iso y da a om he Po uguese banking sec o . 3. Finally, we compiled equally alid esul s when applying he same me hodology o he emaining isk pe spec i es: c edi , ma ke , ope a ional and p o i abil- i y. The indings we accomplished in he abo e men ioned s eps se e a as numbe o s akeholde s ha can be g ouped in h ee ca ego ies: •Banks and consul ancy companies, which can ely on he de eloped ML models o p oac i ely manage hei isk pos u e. This can posi i ely in luence and suppo how banks app oach compliance obliga ions and also, how hey adjus hei p ac ices o hei business model. 71 Doc o al P og amme - In o ma ion Managemen •Cen al Banks can uple el hei supe iso y p ocesses and inno a ion ini ia- i es wi h he ML echniques he e desc ibed o inancial da a. Ea ly Wa ning Sys ems a e among he bes use cases a supe iso y agencies and can al eady be ound and he ECB and BIS (He ig, 2021). •Academia will be able o suppo new in es iga ion on he applica ions o ML o he supe iso y a ea, and expand he usage o inno a i e echnologies in he egula o y ealm. The e a e some limi a ions o he wo k he e de eloped ha we would like o add ess as u u e de elopmen opics. In a i s ie we would imp o e model o- bus ness by ex ending he sample o all banks in he Eu o-a ea (app oxima ely 3150 o ganisa ions). This could be achie ed h ough a join p ojec wi h he ECB, whe e da a om all Cen al Banks is al eady e ie ed. A e wa ds, we would inc emen accu acy by complemen ing each ML model wi h isk speci ic da a. As a inal ec- ommenda ion, we would sugges in e sec ing he quan i a i e pe spec i e s udied in his hesis wi h he quali a i e in o ma ion p esen in in e nal documen s and e- po s. This would de elop an inno a i e use-case o Na u al Language P ocessing (NLP) in supe ision. 72 Bibliog aphy Abell´an, J., Cas ellano, J.G., 2017. A compa a i e s udy on base classi ie s in ensemble me hods o c edi sco ing. Expe Sys ems wi h Applica ions 73, 1–10. doi:10.1016/j.eswa.2016.12.020. Acuna, E., Rod iguez, C., 2004. The ea men o missing alues and i s e ec on classi ie accu acy. Adankon, M.M., Che ie , M., 2009. Encyclopedia o Biome ics - Suppo Vec o Machine. Sp inge US. URL: h ps://doi.o g/10.1007/978-0-387-73003-5 299, doi:10.1007/978-0-387-73003-5_299. Ala’ aj, M., Abbod, M.F., 2016. A new hyb id ensemble c edi sco ing model based on classi ie s consensus sys em app oach. Expe Sys ems wi h Applica ions 64, 36–55. doi:10.1016/j.eswa.2016.07.017. Alessi, L., De ken, C., 2018. Iden i ying excessi e c edi g ow h and le e age. Jou - nal o Financial S abili y 35, 215–225. doi:10.1016/j.j s.2017.06.005. Alonso, A., Ca bo, J.M., 2020. Machine lea ning in c edi isk: Measu ing he dilemma be ween p edic ion and supe iso y cos . SSRN Elec onic Jou nal doi:10.2139/ss n.3724374. Alonso, A., Ca bo, J.M., 2021. Unde s anding he pe o mance o machine lea ning models o p edic c edi de aul : A no el app oach o supe iso y e alua ion. SSRN Elec onic Jou nal doi:10.2139/ss n.3774075. Al man, E.I., 1968. Financial a ios, disc iminan analysis and he p edic ion o co po a e bank up cy. The Jou nal o Finance 23, 589–609. doi:10.1111/j.1540- 6261.1968. b00843.x. Angelini, E., di Tollo, G., Roli, A., 2008. A neu al ne wo k app oach o c edi isk e alua ion. Qua e ly Re iew o Economics and Finance 48, 733–755. doi:10. 1016/j.q e .2007.04.001. An unes, J.A.P., 2021. To supe ise o o sel -supe ise: a machine lea ning based compa ison on c edi supe ision. Financial Inno a ion 7. URL: h ps://doi.o g/ 10.1186/s40854-021-00242-4, doi:10.1186/s40854-021-00242-4. Au ho i y, E.B., 2013. Eba implemen ing echnical s anda ds (i s). URL: h p://www.eba.eu opa.eu/documen s/10180/532570/EBA-ITS-2013- 12+(Final+d a +ITS+on+Hypo he ical+Capi al+o +a+CCP).pd . 73 Doc o al P og amme - In o ma ion Managemen 80 Appendix A Tables Au ho s Yea A ilia ion Ti le Ci a ions Abellan e al. 2017 academia A compa a i e s udy on base classi ie s in ensemble me hods o c edi sco ing 88 Ala’ aj e al. 2016 academia A new hyb id ensemble c edi sco ing model based on classi ie s consensus sys- em app oach 66 Alessi e al. 2018 cen al bank Iden i ying excessi e c edi g ow h and le e age 135 Alonso e al. 2020 cen al bank Machine Lea ning in C edi Risk: Measu ing he Dilemma Be ween P edic ion and Supe iso y Cos 1 2021 cen al bank Unde s anding he Pe o - mance o Machine Lea ning Models o P edic C edi De aul : A No el App oach o Supe iso y E alua ion 0 Angelini e al. 2008 academia A neu al ne wo k app oach o c edi isk e alua ion 305 An unes 2021 cen al bank To supe ise o o sel - supe ise: a machine lea n- ing based compa ison on c edi supe ision 0 Con inued on nex page 81 Doc o al P og amme - In o ma ion Managemen Table A.1 – con inued om p e ious page Au ho s Yea A ilia ion Ti le Ci a ions Boyacioglu e al. 2009 academia P edic ing bank inancial ailu es using neu al ne - wo ks, suppo ec o ma- chines and mul i a ia e s a- is ical me hods: A compa - a i e analysis in he sam- ple o sa ings deposi in- su ance und (SDIF) ans- e ed banks in Tu key 272 B oede s e al. 2018 indus y FSI Insigh s Inno a i e echnology in inancial supe ision 23 Chak abo y e al. 2017 cen al bank Machine Lea ning a Cen- al Banks 62 Chang e al. 2018 academia Applica ion o eX eme g a- dien boos ing ees in he cons uc ion o c edi isk assessmen models o i- nancial ins i u ions 17 Chaudhu i e al. 2011 academia Fuzzy Suppo Vec o Ma- chine o bank up cy p e- dic ion 155 Climen e al. 2019 academia An icipa ing bank dis ess in he Eu ozone: An Ex- eme G adien Boos ing app oach 10 Das ile e al. 2020 academia S a is ical and machine lea ning models in c edi sco ing: A sys ema ic li e a u e su ey 0 Doe e al. 2021 indus y How do cen al banks use big da a and machine lea n- ing? 0 Dwi edi e al. 2019 academia A i icial In elligence (AI): Mul idisciplina y pe spec- i es on eme ging chal- lenges, oppo uni ies, and agenda o esea ch, p ac- ice and policy 39 Filippopoulou e al. 2020 academia An ea ly wa ning sys em o p edic ing sys emic banking c ises in he Eu ozone: A logi eg ession app oach 1 Con inued on nex page 82 Doc o al P og amme - In o ma ion Managemen Table A.1 – con inued om p e ious page Au ho s Yea A ilia ion Ti le Ci a ions Galindo e al. 2000 academia C edi isk assessmen us- ing s a is ical and machine lea ning: Basic me hodol- ogy and isk modeling appli- ca ions 213 Giudice e al. 2020 cen al bank Ins i u ional Sec o Classi- ie , a Machine Lea ning App oach 0 Gogas e al. 2018 academia Fo ecas ing bank ailu es and s ess es ing: A ma- chine lea ning app oach 20 Hamme e al. 2012 academia A logical analysis o banks’ inancial s eng h a ings 49 Hillegeis e al. 2004 academia Assessing he p obabili y o bank up cy 1393 Hohl e al. 2019 indus y FSI Insigh s on policy im- plemen a ion The sup ech gene a ions 3 Huang e al. 2021 academia In elligen FinTech Da a Mining by Ad anced Deep Lea ning App oaches 0 Jag iani e al. 2018 cen al bank The Roles o Big Da a and Machine Lea ning in Bank Supe ision 4 Kola i e al. 2019 academia P edic ing Eu opean bank s ess es s: Su i al o he i es 4 Kou e al. 2019 academia Machine lea ning me hods o sys emic isk analysis in inancial sec o s 47 Kupiec e al. 2018 indus y On he accu acy o al e na- i e app oaches o calib a - ing bank s ess es models 5 Le e al. 2018 academia P edic ing bank ailu e: An imp o emen by implemen - ing a machine-lea ning ap- p oach o classical inancial a ios 24 Lee e al. 2020 academia Machine lea ning o en e - p ises: Applica ions, algo- i hm selec ion, and chal- lenges 7 Con inued on nex page 83 Doc o al P og amme - In o ma ion Managemen Table A.1 – con inued om p e ious page Au ho s Yea A ilia ion Ti le Ci a ions Leo e al. 2019 (blank) Machine lea ning in bank- ing isk managemen : A li - e a u e e iew 11 Lopez I u iaga e al. 2015 academia Bank up cy isualiza ion and p edic ion using neu al ne wo ks: A s udy o U.S. comme cial banks 129 Milian e al. 2019 academia Fin echs: A li e a u e e- iew and esea ch agenda 31 Min e al. 2005 academia Bank up cy p edic ion us- ing suppo ec o machine wi h op imal choice o ke - nel unc ion pa ame e s 866 Pe opoulos e al. 2018 cen al bank A obus machine lea n- ing app oach o c edi isk analysis o la ge loan le el da ase s using deep lea ning and ex eme g adien boos - ing 6 Pompella e al. 2017 academia Ra ings based In e ence and C edi Risk: De ec ing likely- o- ail Banks wi h he PC-Mahalanobis Me hod 5 Ribei o e al. 2012 academia Enhanced de aul isk mod- els wi h SVM+ 57 Soui e al. 2019 academia Rule-based c edi isk as- sessmen model using mul i- objec i e e olu iona y algo- i hms 3 Ta ana e al. 2018 academia An A i icial Neu al Ne - wo k and Bayesian Ne wo k model o liquidi y isk as- sessmen in banking 30 Wang e al. 2021 academia A machine lea ning-based ea ly wa ning sys em o sys emic banking c ises 2 Xia e al. 2017 academia A boos ed decision ee app oach using Bayesian hype -pa ame e op imiza- ion o c edi sco ing 158 Table A.1: Lis o pape s collec ed h ough he esea ch que y, e e enced by au ho , yea o publica ion, a ilia ion, and numbe o ci a ions. 84 Doc o al P og amme - In o ma ion Managemen Qua ile / O igin Jou nal Numbe o Pape s A (ERA) Ad ances in Neu al In o ma ion P ocessing Sys- ems 1 Banca d’I alia Ques ioni di Economia e Finanza 1 Banco de Espa˜na SSRN Elec onic Jou nal 2 Bank o In e na ional Se lemen s FSI Insigh s on policy implemen a ion 2 Bank o England Bank o England 1 Bank o G eece Nin h IFC Con e ence on “A e pos -c isis s a is i- cal ini ia i es comple ed?” 1 Fede al Rese e Banking Pe spec i es, Fo hcoming 1 Q1 Applied So Compu ing Jou nal 3 Business Ho izons 1 Elec onic Comme ce Resea ch and Applica ions 1 Expe Sys ems wi h Applica ions 9 In e na ional Jou nal o Fo ecas ing 1 In e na ional Jou nal o In o ma ion Managemen 1 Jou nal o Business Resea ch 1 Jou nal o Economic Beha io and O ganiza ion 1 Jou nal o Financial S abili y 2 Neu ocompu ing 1 Resea ch in In e na ional Business and Finance 1 Re iew o Accoun ing S udies 1 Technological and Economic De elopmen o Economy 1 Q2 Applied Economics 1 Compu a ional Economics 2 Economic Modelling 1 Financial Inno a ion 1 Global Finance Jou nal 1 Qua e ly Re iew o Economics and Finance 1 Risks 1 SUERF SUERF - The Eu opean Money and Finance Fo- um 1 Table A.2: Jou nals o he selec ed a icles and hei qua ile. Whe e he jou nal is no indexed, he en i y esponsible o he publishing was included. Au ho s Yea Summa y sen ence Galindo e al. 2000 CART decision- ees ou -pe o m s a is ics o c edi isk assess- men , using a comme cial bank loans da ase Hillegeis e al. 2004 Black–Scholes–Me on op ion-p icing model is a be e indica o o bank up cy p obabili y han Z-Sco e and O-Sco e. Con inued on nex page 85 Doc o al P og amme - In o ma ion Managemen Table A.3 – con inued om p e ious page Au ho s Yea Summa y sen ence Min e al. 2005 Mo i a ed by he inc easing use o machine lea ning echniques, his pape aims o ou pe o m classical s a is ics in bank up cy p edic ion. An op imised SVM model pe o ms be e han MDA, logi and BPN o bank up cy p edic ion. Angelini e al. 2008 Regula ion-imposed capi al equi emen s inc ease he need o p e- cise c edi isk assessmen sys ems. This pape shows ANNs’ e y good esul s p edic ing he de aul endency o a bo owe . Boyacioglu e al. 2009 Mul i-laye pe cep ons and lea ning ec o quan iza ion a e he mos success ul models p edic ing bank ailu e as a classi ica ion p oblem, in a Tu kish case. Chaudhu i e al. 2011 Fuzzy-SVM sa is ies Basel II demands o de ec ing bank up cy p obabili y, ou pe o ming o he app oaches. This algo i hm also p o ed o ha e mo e clus e ing capabili ies han PNN. Hamme e al. 2012 The logical analysis o da a (LAD) is able o e e se-enginee Fi ch isk a ings o bank, showing be e esul s han suppo - ec o ma- chines and logis ic eg ession when e alua ing he c edi wo hiness o banks. Ribei o e al. 2012 This s udy es ablishes he limi a ions o using exclusi ely quan- i a i e inancial da a when de eloping de aul isk models. The au ho s p opose a new app oach ha includes con ex ual knowl- edge in an SVM model, showing be e p edic abili y pe o mance Lopez I u iaga e al. 2015 P o iling dis essed banks using sel -o ganising maps and modelling ailu e de ec ion wi h mul i-laye pe cep on ou pe o ms adi- ional models o bank up cy p edic ion. The esul ing model de- ec s 96% o ailu es, up o 3 yea s be o e he bank up cy e Ala’ aj e al. 2016 The p oposed hyb id ensemble model imp o es p edic ing capabil- i y compa ed o base classi ie s, using 7 eal-wo ld da ase s. I uses a classi ie consensus sys em o compa e his new app oach wi h he adi ional combina ion me hods. Abellan e al. 2017 Selec ion o he bes base classi ie in ensemble me hods o c edi sco ing p oblems. The indi idual pe o mance o classi ie s is no he only c i e ia o ensemble schemes. Chak abo y e al. 2017 An o e iew o he applica ions o machine lea ning o inancial p oblems, he mos popula modelling app oaches, and h ee case s udies o ele an wo ks o cen al banks. This s udy also es ab- lishes ha machine lea ning models usually ou pe o m adi Pompella e al. 2017 An EWS is p oposed o de ec likely- o- ail banks. This me hod is compa ed wi h isk agencies’ a ing and de ec s possibly w ongly a ed banks. The au ho s sugges he adop ion o his EWS by egula o s. Xia e al. 2017 The c edi sco ing p oblem is add essed using a XGBoos model wi h Bayesian hype -pa ame e op imisa ion, no only ob aining be e accu acy han baseline models, bu also p o iding ea u e impo ance and a decision cha o in e p e abili y. Con inued on nex page 86 Doc o al P og amme - In o ma ion Managemen Table A.3 – con inued om p e ious page Au ho s Yea Summa y sen ence Alessi e al. 2018 The use o andom o es o p edic banking c ises seconda y o excessi e c edi g ow h, using c edi and eal es a e p edic o s. B oede s e al. 2018 A su ey on he use o inno a i e echnologies in inancial supe - ision, he challenges aced by supe iso y agencies and he need o a clea sup ech s a egy. Addi ionally, he expe ience o ea ly adop e s is desc ibed. Chang e al. 2018 The de elopmen o a c edi isk model using XGBoos classi ie o add ess he he e ogeneous na u e o inancial da a. An unde - sampling me hod is applied o deal wi h he imbalanced da a. Gogas e al. 2018 Ou pe o ming he Ohlson’s sco e wi h s ess- es ing ool based on a suppo - ec o machine model o o ecas bank ailu es. The adop ed me hodology de ines a clea bounda y be ween sol en and insol en banks. Jag iani e al. 2018 The impac o machine lea ning in banking supe ision in e ms o new possible analy ical solu ions and isks in ol ed in hose new app oaches. Kupiec e al. 2018 Add essing he need o alida ion o bank s ess es models, by emphasising model o ecas accu acy. A Lasso model shows he bes o ecas ing capabili ies o de e mining capi al equi emen s in s ess ul condi ions. Le e al. 2018 A i icial neu al ne wo ks and k-nea es neighbou me hods a e mo e accu a e o p edic ing bank ailu e han adi ional s a is- ics. Pe opoulos e al. 2018 P edic ing he p obabili y o de aul o G eek banks using da a min- ing echniques o educe dimensionali y, wi h XGBoos eme ging as he bes model. The au ho s aim o ully cap u e he in o ma ion wi hin hese la ge da ase s o be e suppo he o e all Ta ana e al. 2018 Add essing liquidi y isk assessmen h ough a model ha uses neu- al ne wo ks and Bayesian ne wo ks. The models we e capable o dis inguishing he mos c i ical ac o s in liquidi y isk measu e- men . Climen e al. 2019 Using XGBoos o iden i y he bes p edic o s o bank ailu e and de elop a classi ica ion model o label ailed and non- ailed banks in he Eu ozone. The da a used in his s udy is composed o 25 annual inancial a ios o comme cial banks in he Eu ozo Dwi edi e al. 2019 Expe con ibu o s iden i y and compile a se ies o oppo uni ies, impac s and esea ch opics aised by he apid adop ion o AI. The inancial sec o shows eno mous po en ial in obo ad iso y and au oma ion, and bank up cy p edic ion. Hohl e al. 2019 A su ey o ac i i ies wi hin he scope o sup ech, classi ying he deg ee o echnological de elopmen , and he s a egies in place o implemen hem, highligh ing he expe imen al na u e o hese ini ia i es and he need o in e na ional coo dina ion. Con inued on nex page 87 Doc o al P og amme - In o ma ion Managemen Table A.3 – con inued om p e ious page Au ho s Yea Summa y sen ence Kola i e al. 2019 Success ully unde going Eu opean bank s ess- es s depends la gely on he isks a bank is exposed o, as opposed o being p epa ed o speci ic ad e se scena ios. Using Bankscope da a, he de eloped model accu a ely p edic s 90% o he ailing banks. Kou e al. 2019 A su ey depic ing he mos common me hodologies o assess sys- emic isk in he inancial sys em, using machine lea ning, big da a analysis, ne wo k analysis and sen imen analysis. The pape show- cases cu en esea ches on he use o machine lea ning in Leo e al. 2019 A li e a u e e iew e idencing machine lea ning use o isk man- agemen pu poses in he banking indus y, while also no ing he expe imen al na u e o mos app oaches. Milian e al. 2019 A li e a u e e iew aiming o ind consensus on a in ech de ini- ion, showing how banks and supe iso y agencies a e using hese inno a i e echnologies and dealing wi h he isks in ol ed. Soui e al. 2019 Using e olu iona y algo i hms o add ess c edi isk assessmen by conside ing i as an op imisa ion ( ule-based) sea ch p oblem: minimise complexi y, maximise accu acy and weigh ( ules impo - ance). Alonso e al. 2020 Compa ing machine lea ning models om c edi de aul p edic ion. Necessi y o a s uc u ed s a egy o assessing ML models o in- c ease anspa ency in he use o hese echnologies, and p omo e inno a ion in he inancial indus y. Das ile e al. 2020 A sys ema ic li e a u e e iew on how s a is ic and machine lea n- ing echniques ha e been used o add ess he c edi sco ing p ob- lem. Al hough machine lea ning is o en incapable o explaining p edic ions, hese models consis en ly ou pe o m he classic Filippopoulou e al. 2020 De eloping an EWS o de ec sys emic banking c isis based on he ECB Mac op uden ial da abase. Mos o he isk indica o s used in he da ase a e key o o ecas a sys emic isk c isis 1 o 4 yea s be o e he e en . Giudice e al. 2020 De eloping an au oma ic classi ica ion sys em o he sec o o eco- nomic ac i i y o I alian companies, using a mul i-s ep classi ie wi h g adien boos ing and suppo - ec o machine models. The de eloped model is al eady being used in a p oduc ion en i Lee e al. 2020 A s udy on ypes o machine lea ning applica ions, explo ing he accu acy-in e p e abili y ade-o , and h ee use cases in inancial indus y. Alonso e al. 2021 P edic ing c edi de aul p obabili y wi h machine lea ning su - passes adi ional s a is ic me hods, po en ially leading o sa ings o up o 17% in egula o y capi al equi emen s. An unes 2021 Es ablishing he need o supe iso y on-si e inspec ion by com- pa ing he esul s o wo machine lea ning models, one based on he banks’ own isk assessmen and he o he based on he indings om p e ious on-si e inspec ions. Con inued on nex page 88 Doc o al P og amme - In o ma ion Managemen Table A.3 – con inued om p e ious page Au ho s Yea Summa y sen ence Doe e al. 2021 Policy b ie showing cen al banks a e elying on big da a o daily asks, and iden i ying a clea need o specialised knowledge on how o adequa ely use machine lea ning, and ex ac g ea e alue om ha da a. Huang e al. 2021 This s udy is de eloped unde he assump ion ha he in ica e na u e o inancial da a canno be p ope ly explo ed h ough a- di ional me hods. An ad anced deep lea ning model o add ess he complex and hie a chical ea u es o inancial da a, ha ou p Wang e al. 2021 Random o es based EWS ou pe o ms he classic logi app oach as he p edic i e ool o p e en sys emic banking c ises. This pape shows an expe o ing app oach o model he mul i a ia e na u e o sys emic isk assessmen da a. Table A.3: Sho summa y o each analysed pape , e e enced by au ho s and yea . Au ho s ML Me hods Da ase Abellan e al. ada-boos ing, bagging, andom subspace, DECORATE, o a ion o es public: Aus alian, Ge man, and Japanese da ase s ob ained om UCI eposi o y o machine lea n- ing; I anian da ase om ”A com- pa ison be ween s a is ical and da a mining me hods o c edi sco ing in case o limi ed a ailable da a. (2007)”; Polish da ase Ala’ aj e al. neu al ne wo ks, suppo ec o machines, andom o es s, deci- sion ees, Nai e Bayes public: Aus alian, Ge man, and Japanese da ase s ob ained om UCI eposi o y o machine lea n- ing; I anian da ase om ”A com- pa ison be ween s a is ical and da a mining me hods o c edi sco ing in case o limi ed a ailable da a. (2007)”; Polish da ase Alessi e al. logi , decision ees, andom o - es public: c isis da ase by De ken e al. 2014, cap u ing sys emic banking c ises ela ed o domes- ic c edi cycle Alonso e al. logi , lasso, CART, andom o - es , xgboos , deep lea ning p i a e: anonymized da ase om Banco San ande , con ain- ing mo e han 75000 c edi ope - a ions Alonso e al. logi , lasso, CART, andom o - es , xgboos , deep lea ning, RL & ensemble me hods public: kaggle.com ”Gi e me some c edi ” da ase Angelini e al. ann p i a e: SME loans om na I al- ian bank Con inued on nex page 89 Doc o al P og amme - In o ma ion Managemen (a) Logis ic Reg ession. (b) k-Nea es Neighbou s classi ie . (c) Random Fo es classi- ie . (d) Ex eme G adien Boos ing classi ie . (e) TPOT classi ie au- oML amewo k. Figu e A.6: Ope a ional isk: Con usion ma ices gene a ed when e alua ing he abo e men ioned models, using c oss- alida ion app oach. (a) Logis ic Reg ession. (b) k-Nea es Neighbou s classi ie . (c) Random Fo es classi- ie . (d) Ex eme G adien Boos ing classi ie . Figu e A.7: P o i abili y isk: Con usion ma ices gene a ed when e alua ing he abo e men ioned models, using ain- es spli app oach. 96 Doc o al P og amme - In o ma ion Managemen (a) Logis ic Reg ession. (b) k-Nea es Neighbou s classi ie . (c) Random Fo es classi- ie . (d) Ex eme G adien Boos ing classi ie . (e) TPOT classi ie au- oML amewo k. Figu e A.8: P o i abili y isk: Con usion ma ices gene a ed when e alua ing he abo e men ioned models, using c oss- alida ion app oach. 97