Multi-Objective Lagged Feature Selection Based on Dependence Coefficient for Time-Series Forecasting

Mul i-Objec i e Lagged Fea u e Selec ion Based on Dependence Coefficien o Time-Se ies Fo ecas ing Ma ´ıa Lou des Lina es-Ba e a , Manuel J. Jim´enez Na a o , Jos´e C. Riquelme ,andMa ´ıa Ma ´ınez-Balles e os(B) Depa men o Compu e Languages and Sys ems, Uni e si y o Se ille, 41012 Se ille, Spain {mlina es,mjimenez3, iquelme,ma iama inez}@us.es Abs ac . In he as -e ol ing field o machine lea ning, he p ocess o ea u e selec ion is essen ial o educing model complexi y and enhanc- ing in e p e abili y. Wi hin his con ex , fil e me hods ha e gained ecogni ion o hei effec i eness in assessing ea u es h ough s a is- ical me ics. A ecen ly in oduced me ic, he Condi ional Dependence Coefficien , aims o assess he dependence be ween subse s o ea u es and a a ge a iable, enhancing ou unde s anding o ea u e ele ance. This pape p esen s a no el ea u e selec ion app oach ha in eg a es his s a is ical me ic wi h a mul i-objec i e e olu iona y algo i hm. This s a egy le e ages he flexibili y o e olu iona y algo i hms o efficien ly explo e he ea u e space and employs an in ui i e me ic o iden i ying pe inen ea u es. Unlike many fil e -based app oaches, ou me hod does no equi e h esholds o pe cen iles ela ed o he numbe o selec ed ea u es and e alua es he collec i e me i o ea u e subse s ins ead o he significance o indi idual ea u es. To add ess he o ecas ing chal- lenge o iden i ying he app op ia e ime lags and ea u es, we pe o med expe imen s on eigh dis inc da ase s con aining mul i a ia e ime-se ies da a. Compa ing ou me hod agains a baseline wi h no ea u e selec ion, ou esul s show solid pe o mance in efficacy and a no able educ ion in model complexi y. Keywo ds: Fea u e Selec ion ·Mul i-objec i e Op imiza ion · Gene ic Algo i hm ·Neu al Ne wo k ·Time-Se ies Fo ecas ing c  1 In oduc ion In oday’s da a-d i en landscape, whe e massi e and eal- ime in o ma ion becomes he no m, da ase s o en con ain a as numbe o a ibu es. This si u- a ion demands ad anced s o age and p ocessing capabili ies o handle inc eased da a olumes and apid and in e p e able esponses. In his con ex , ea u e selec ion [3] allows he iden ifica ion o he mos ele an subse o ea u es, enabling a ocus on a educed se o a iables while main aining o enhancing efficacy. The Au ho (s) A. Alonso-Be anzos e al. (Eds.): CAEPIA 2024, LNAI 14640, pp. 81–90, 2024. h ps://doi.o g/10.1007/978-3-031-62799-6_9 82 M. L. Lina es-Ba e a e al. This p ocess offe s se e al benefi s. Fi s ly, i enhances model in e p e abil- i y [7], enabling expe s o ocus on meaning ul insigh s. Secondly, i add esses he cu se o dimensionali y [17], educing compu a ional complexi y and o e - fi ing. Finally, elimina ing i ele an ea u es p e en s he need o sample da a equi ing ex ensi e esou ces. Fea u e selec ion emains a dynamic a ea o esea ch. Despi e he di e se ange o app oaches, i is possible o dis inguish h ee dis inc g oups: fil e , w appe and embedded me hods [4,10]. Wi hin his domain, fil e me hods a e gaining significan a en ion o hei abili y o quickly assess ea u e impo ance h ough s a is ical measu es, bu may o e look ea u e in e ac ions. This pape in oduces MOLS (Mul i-Objec i e Lag Selec ion), a no el fil e ea u e selec ion algo i hm ha combines he Condi ional Dependence Coeffi- cien (CODEC) [1] wi h a mul i-objec i e gene ic algo i hm. The CODEC me - ic e alua es he collec i e impo ance o a ea u e subse . This me ic add esses he limi a ions o nume ous fil e echniques ha p ima ily emphasize indi idual ea u e ankings. The mul i-objec i e gene ic algo i hm app oach aims o balance model sim- plici y and in e p e abili y by selec ing a concise se o highly ele an ea u es. Ou p oposal is based on he Pa e o-op imal se and he NSGA-II algo i hm [6], o e coming he limi a ions o adi ional gene ic algo i hms ha ely on weigh ed unc ions, leading o subop imal ade-offs and sensi i i y issues. Gi en he complex ela ionships be ween a iables and he significance o iden i ying key lags, ime se ies o ecas ing p o ides an op imal scena io o e al- ua e ou app oach. The e o e, he main con ibu ions o his pape a e: – In oduc ion o a mul i-objec i e fil e me hod o ea u e selec ion ha e al- ua es ea u e subse s collec i ely. Unlike se e al fil e app oaches, his me hod elimina es he need o hype pa ame e s ha dic a e he numbe o ea u es o be selec ed, such as h esholds, pe cen iles, o a specific numbe o ea u es. – The p oposal is applied o e a neu al ne wo k, le e aging he significan ben- efi s ha ea u e selec ion can p o ide. This model is u he op imized ia he Bayesian Op imiza ion. – E alua ion o he p oposal effec i eness ac oss eigh di e se ime-se ies da ase s, co e ing a ious domains such as ai pollu ion, elec ici y, and affic o ecas ing. The esul s demons a e compe i i e pe o mance and a educ- ion in model complexi y h ough ea u e selec ion. The pape is o ganized in o se e al sec ions. Sec ion 2 e iews p e ious esea ch on ea u e selec ion and Sec . 3desc ibes MOLS p oposal. Sec ion 4ou - lines he expe imen al me hodology applied and Sec . 5p esen s he ob ained esul s. Finally, Sec . 6summa izes he main conclusions and sugges s u u e esea ch di ec ions. MOLS: Mul i-Objec i e Lag Selec ion 83 2 Rela ed Wo ks As he demand o mo e in e p e able and efficien models g ows, he e is a no iceable shi owa ds in eg a ing ea u e selec ion me hods in o neu al ne - wo k a chi ec u es. A no able example o his end is he wo k o Liu e al. [14], who combined a mu ual in o ma ion-based ea u e selec ion me hod wi h a hyb id deep neu al ne wo k model o accu a ely o ecas wind speed. Simila ly, Kilince e al. [12] ha e made a significan con ibu ion by using ecu si e ea u e elimina ion wi h mul ilaye pe cep on op imiza ion o imp o e he de ec ion o cybe secu i y h ea s in heal hca e sys ems. Among ea u e selec ion s a egies in eg a ed on machine lea ning models, fil e me hods s and ou o hei e sa ili y and compu a ional efficiency. The CODEC me ic, in oduced in [1], ma ks a significan ad ancemen in fil e me h- ods by efficien ly de ec ing bo h linea and nonlinea dependencies. Despi e he po en ial o ea u e selec ion, op imizing sea ch echniques wi hin a as ea u e space p esen s a conside able challenge. To o e come his, he adop ion o bio-inspi ed algo i hmic echniques has been sugges ed. Zhou e al. [19] p oposed a gene ic algo i hm guided by a co ela ion ma ix as a ea u e selec ion s a egy. In a simila ein, Espinosa e al. [8] ha e inco po a ed a mul i- objec i e e olu iona y ea u e selec ion app oach in o an LSTM model, achie - ing no able imp o emen s in ai quali y p edic ion in I aly. Ou wo k simila ly le e ages a mul i-objec i e e olu iona y app oach, combined wi h he CODEC me ic, o enhance ea u e selec ion. The impo ance o ea u e selec ion is unde sco ed in a ious domains, pa ic- ula ly in he analysis o ime-se ies da a. The need o pinpoin ele an empo al poin s and ackle issues like empo al dependencies and seasonali y is pa amoun . Add essing his need, in [15] is de eloped a wo-s age ea u e selec ion p ocess (co ela ion analysis and w appe me hod u ilizing a shallow neu al ne wo k) o financial ime se ies o ecas ing. Ou esea ch pa allels hese ini ia i es, concen- a ing on igo ously assessing ou me hodologies wi hin he ime-se ies domain. 3 Desc ip ion o he P oposed MOLS Algo i hm We p opose an e olu iona y s a egy-based me hod o iden i y he op imal ea- u e subse . The main goal is o iden i y he subse o ea u es maximizing CODEC alue while keeping he numbe o ea u es as small as possible. The Condi ional Dependence Coefficien (CODEC) [1](seeEq.1) assesses he ela ionship be ween a a ge a iable Yand a ea u e se Zin a da ase o N ins ances. We ocus on he uncondi ional e sion o CODEC, which e alua es he di ec ela ionship wi hou condi ioning on addi ional a iables. The o mula ion is as ollows: CODEC(Y,Z):=N i=1 Nmin Ri,R M(i)−L2 i N i=1 Li(N−Li)(1) 84 M. L. Lina es-Ba e a e al. In his o mula, o each ins ance iwe conside he ollowing e ms. Ri ep e- sen s he ank o he Y alue o he i- h ins ance. Li ep esen s he in e se ank o he Y alue o he i- h ins ance. M(i) ep esen s he index o he nea es da a poin o he i- h ins ance, compu ed using only he dimensions defined by Zand Euclidean dis ance. The me ic quan ifies he s eng h o he ela ionship be ween Yand Zby assessing how Zinfluences anks and posi ioning. In [1], he au ho s demons a e ha CODEC is a nonlinea gene aliza ion o pa ial R2. Ou p oposal is based on a mul i-objec i e gene ic algo i hm based on Pa e o on inspi ed by he NSGA-II algo i hm. 1. Ini ializa ion. We s a by gene a ing an ini ial popula ion o μindi iduals a andom, whe e each is codified by a bina y ec o (“1” o inclusion, “0” o exclusion o each ea u e). 2. E olu iona y p ocess. The whole e olu iona y p ocess is epea ed un il he desi ed numbe o gene a ions (ngens) is eached. A each i e a ion, we gene a e a new popula ion h ough a se ies o s eps: (a) Offsp ing gene a ion. A each gene a ion’s ou se , λoffsp ing a e p o- duced h ough he ollowing e olu iona y ope a ions (only one a each ime): c osso e (chance cxpb), mu a ion (chance mu pb), o ep oduc- ion (chance 1 −mu pb −cxpb). In c osso e , wo pa en s a e andomly selec ed om he popula ion, p o- ducing a single offsp ing using a wo-poin c osso e mechanism. Du ing mu a ion, each gene om a andomly selec ed indi idual ha e a p obabil- i y (gmu pb) o flipping om 0 o 1 o ice e sa, in oducing a iabili y. In ep oduc ion, a andom indi idual is cloned and he clone is added o he offsp ing, p ese ing gene ic in o ma ion. (b) E alua ion and selec ion o subsequen popula ion. The offsp ing, combined wi h he cu en popula ion, unde go e alua ion. The op μ pe o me s a e selec ed o he nex gene a ion. Each indi idual is e alu- a ed based on wo c i e ia: i s CODEC alue ( eflec ing he dependence be ween he selec ed ea u es and he a ge a iable) and he numbe o ea u es i ep esen s. Indi iduals a e so ed in o laye s called Pa e o on s. The fi s on con- ains indi iduals ha a e no domina ed. Subsequen on s a e filled wi h indi iduals ha a e only domina ed by hose in he p e ious on (s). Fo he nex gene a ion, he op μindi iduals a e selec ed s a ing om he fi s on and mo ing on o he nex ones as needed. I including an en i e on would exceed he desi ed popula ion size, he c owding dis- ance wi hin ha on is used o choose indi iduals, ensu ing he selec ion p io i izes di e si y. 3. Bes solu ion selec ion. Once he e olu iona y p ocess is comple ed, indi- iduals wi hin he popula ion a e classified in o diffe en Pa e o on s. The fi s on comp ises non-domina ed solu ions, cha ac e ized by achie ing an op imal balance be ween high CODEC alues and a educed numbe o ea- u es. The bes indi idual on he fi s on is chosen based on he c i e ion o MOLS: Mul i-Objec i e Lag Selec ion 85 Table 1. MOLS Pa ame e Se ings. Pa ame e De aul Value Popula ion size (μ) 50 Offsp ing size (λ) 50 Numbe o Gene a ions (ngens) 20 C osso e P obabili y (cxpb)0.5 Mu a ion P obabili y (mu pb)0.5 Gene Mu a ion P obabili y (gmu pb)0.2 ha ing he minimum numbe o ea u es. This selec ion p o ides an op imal balance be ween main aining high dependency and ensu ing model simplici y. The pa ame e s o he mul iobjec i e gene ic algo i hm ha e been fixed (Table 1). Op ionally, hese pa ame e s could be uned, bu ou esul s (Sec . 5) sugges ha he pa ame e s a e good enough o ob ain compe i i e esul s. 4 Me hodology and Expe imen a ion This sec ion ou lines he expe imen al me hodology, de ails he lea ning model in Sec . 4.1 and he main phases o he app oach in Sec . 4.2. 4.1 Model As a base model, we selec ed a ully connec ed neu al ne wo k a chi ec u e (MLP). I ea u es one o wo hidden laye s, each con aining 50 o 100 neu ons. The Rec ified Linea Uni (ReLU) ac i a ion unc ion is used o i s efficiency in speeding up con e gence and educing he anishing g adien issue. The Adam op imize is used o ne wo k op imiza ion, wi h a lea ning a e se be ween 0.0001 and 0.01. This lea ning a e spec um allows o a balanced app oach o lea ning, adap able o he unique demands o he p oblem a hand. Finally, o p e en o e fi ing, an ea ly s opping mechanism is applied, s opping aining i he alida ion loss ails o imp o e o 10 consecu i e epochs. 4.2 Main Phases 1. P ep ocessing: The ini ial da ase is segmen ed in o h ee subse s: aining (70%), alida ion (15%), and es se s (15%); ensu ing unbiased model e al- ua ion. As he algo i hms equi e he lagged in o ma ion, each ime se ies is segmen ed in o fixed-size, non-o e lapping windows. Da a om all channels a e fla ened in o a one-dimensional ec o . S anda diza ion is also applied o add ess issues ha a ise om diffe en scales and dis ibu ions. 86 M. L. Lina es-Ba e a e al. 2. MLP hype pa ame e s uning: Fi s , we pe o m MOLS ea u e selec ion on he aining se and apply he selec ion o e ain and alida ion se s. To iden i y he op imal hype pa ame e s o he MLP, Bayesian op imiza ion [16] is employed. This in ol es aining models wi h a a ie y o configu a ions on he aining se and assessing hei pe o mance on he alida ion se . The configu a ion ha esul s in he lowes Mean Absolu e E o (MAE) is conside ed op imal. 3. Bes model aining and e alua ion: In his s ep, MOLS ea u e selec ion is applied. Then MLP is ained wi h he op imal hype pa ame e s and MAE is e alua ed o e he es se . The expe imen is ca ied ou wi h and wi hou ea u e selec ion and is epea ed using 5 diffe en andom seeds. 5 Resul s and Discussion This sec ion p esen s he esul s, in oducing he da ase s in Sec . 5.1 and ana- lyzing he effec i eness o he MOLS app oach in Sec . 5.2. 5.1 Da ase s The p oposal is e alua ed in eigh ime se ies da ase s. P ep ocessing echniques a e de ailed in Sec . 4.2, ocusing on windowing and channel fla ening. –The To neo da ase [11] consis s o ou sepa a e da ase s, each ocusing on a specific pollu an (CO,NO2,O3o PM 10), ga he ed by hou ly senso s in he To neo egion (Se ille). The da ase consis s o 7 a iables (4 pollu an s and 3 me eo ological a ibu es) and a pas his o y o 24 h ( he eby esul ing in a o al o 168 ea u es). The a ge a iable is he le el o pollu an o he nex hou . Each da ase con ains 2,798 ins ances and 168 ea u es. –The Elec ici y da ase [9] con ains measu emen s o elec ici y consump- ion in Po ugal o 321 diffe en clien s. A pas his o y o 24 h is aken in o accoun . The a ge is o p edic he nex hou o he fi s sampled clien . I comp ises 1,095 ins ances and 7,704 ea u es. –The T affic da ase [5] de ails he oad occupancy a es cap u ed be ween 2015 and 2016 by senso s placed on highways in he San F ancisco Bay A ea a 10-minu e in e als. I includes da a om 862 senso s and 24 h o pas his o y. The a ge consis s o p edic ing he occupancy o he fi s senso in he nex hou . The da ase includes 730 ins ances and 20,688 ea u es. –The ExchangeRa e da ase [13] co e s daily exchange a e da a o he pe iod be ween 1990 and 2016 in eigh coun ies: Aus alia, he Uni ed King- dom, Canada, Swi ze land, China, Japan, New Zealand, and Singapo e. This da ase conside s 8 coun ies, and a his o ical eco d o he 7 p e ious days is conside ed. The a ge o his da ase is o p edic he alue o he nex hou o he fi s coun y. I is made up o 1080 ins ances and 56 ea u es. MOLS: Mul i-Objec i e Lag Selec ion 87 –The ETTh2 da ase [18] comp ises da a collec ed om an elec ical ans- o me loca ed in China, wi h hou ly measu emen s spanning om July 2016 o July 2018. The da ase includes 7 a iables (High Use ul Load, High Use- less Load, Middle Use ul Load, Middle Useless Load, Low Use ul Load, Low Useless Load, and Oil Tempe a u e) and a his o ical eco d o 24 h is consid- e ed. The a ge is o p edic he ans o me oil empe a u e o he nex hou . I con ains 725 ins ances and 168 ea u es. 5.2 Resul and Analysis Table 2p esen s he effec i eness o ou p oposal compa ed o he esul s wi hou using ea u e selec ion, based on he pe o mance in e ms o MAE and selec ed ea u es. The en i e expe imen a ion p ocess was epea ed using 5 dis inc an- dom seeds o ensu e he obus ness and eliabili y o he esul s. The a e age MAE (Mean Absolu e E o ) and he a e age numbe o ea u es om he 5 execu ions we e eco ded and a e shown in Table 2. Table 2. A e age efficacy Resul s in Te ms o MAE and Numbe o Fea u es. MAE # Fea u es Da ase MOLS No Selec ion MOLS To al ExchangeRa e 0.007 0.006 9.8 56 To neoCO 1.415 1.291 50.4 168 To neoNO2 1.432 1.386 46.8 168 To neoPM10 1.225 1.274 49.6 168 To neoO3 1.362 1.375 48.6 168 ETTh2 1.875 1.897 50.2 168 Elec ici y 204.573 213.243 3611.6 7704 T affic 0.007 0.009 9925.8 20688 In e ms o MAE, ou p oposal exhibi s supe io pe o mance compa ed o he app oach wi hou ea u e selec ion in fi e da ase s, while ob aining a MAE close o he baseline me hod in he emaining h ee da ase s. This unde sco es he s eng h o he CODEC me ic in accu a ely cap u ing he dependence be ween subse s o ea u es and he a ge a iable. In pa icula , a significan enhancemen was obse ed in he Elec ici y da ase . Howe e , i should be men ioned ha he esul s o To neoCO we e sligh ly wo se. Fu he mo e, ou p oposal s ands ou by subs an ially simpli ying he model, selec ing ewe han hal o he o al ea u es a ailable in he da ase s, and mos cases selec ing app oxima ely 20–30% o he o al ea u es. Figu e 1illus a es he p og ession o he MOLS algo i hm h oughou he ea u e selec ion p ocess o one o he 5 andom seeds execu ions conside ed, using he To neoPM10 da ase . The fi s Pa e o on fi ness poin s is highligh ed 88 M. L. Lina es-Ba e a e al. Fig. 1. MOLS selec ion on To neoPM10 in one o he 5 andom seeds execu ions. in ed, while he es o indi iduals fi ness is ep esen ed in black. I is e iden ha , ac oss gene a ions, indi iduals end o con e ge owa ds he bo om- igh co ne o he g aph. This indica es a endency owa ds achie ing a balance o highe CODEC alues wi h a educed numbe o ea u es. A he end o he p o- cess, he MOLS algo i hm success ully achie es a obus balance, hus ensu ing effec i e and efficien ea u e selec ion. The obse ed well-balanced ela ionship be ween he Mean Absolu e E o (MAE) and he numbe o ea u es unde sco es he po en ial o he mul i- objec i e e olu iona y app oach o main ain o enhance effec i eness while educing complexi y. To ensu e he significance o he conclusions ob ained, a s a is ical es [2] was applied o he esul s. Figu e 2p esen s he esul s o he Bayesian analysis compa ing ou me hodology wi h he NoSelec ion (NS) me hod. Wi h a 77% confidence le el, we ound ha he NoSelec ion me hod esul s in a highe e o . This esul s allow us o iden i y ha he e a e significan diffe ences be ween he me hods compa ed. MOLS: Mul i-Objec i e Lag Selec ion 89 Fig. 2. Bayesian MAE compa ison esul . 6 Conclusions and Fu u e Wo ks The pape p esen s MOLS, a no el app oach ha in eg a es he CODEC me - ic in o a fil e ea u e selec ion algo i hm o ime-se ies o ecas ing. MOLS aims o balance he ea u e dimension wi h he efficacy, add essing he chal- lenges posed by da ase s wi h nume ous ea u es. The key con ibu ions include p oposing a mul i-objec i e fil e selec ion me hod, applying a neu al ne wo k model op imized ia Bayesian Op imiza ion, and e alua ing he p oposal ac oss di e se ime-se ies da ase s. Resul s indica e ha MOLS ou pe o ms he app- oach wi hou ea u e selec ion in e ms o MAE in se e al da ase s, achie ing a significan educ ion in he numbe o ea u es while main aining o imp o - ing model efficacy. S a is ical analysis suppo s hese findings. Howe e , mo e esea ch is needed o es ablish he significance conclusi ely. Fu u e wo k could ocus on efining he MOLS algo i hm, explo ing addi- ional da ase s and models, and in es iga ing o he s a is ical me ics o op i- miza ion echniques. Addi ionally, ex ending he applica ion o MOLS o diffe - en and mo e complex ime-se ies o ecas ing p oblems could p o ide aluable insigh s in o i s effec i eness and e sa ili y. Acknowledgemen s. This esea ch has been suppo ed by he g an PID2020- 117954RB-C22 unded by MICIU/AEI/10.13039/501100011033. This wo k has also been suppo ed by TED2021-131311B-C21 unded by MICIU/AEI/10.13039/ 501100011033 and by he Eu opean Union Nex Gene a ionEU/PRTR. Re e ences 1. Azadkia, M., Cha e jee, S.: A simple measu e o condi ional dependence. Ann. S a . 49(6), 3070–3102 (2021) 2. Bena oli, A., Co ani, G., Demsa , J., Zaffalon, M.: Time o a change: a u o ial o compa ing mul iple classifie s h ough Bayesian analysis. J. Mach. Lea n. Res. 18, 2653–2688 (2017)