Input variable selection for forecasting models

INPUT VARIABLE SELECTION FOR FORECASTING MODELS Manuel R. A ahal, Al onso Cepeda, Edua do F. Camacho Dep o. Ingenie ía de Sis emas y Au omá ica. Uni e sidad de Se illa Abs ac : The selec ion o inpu a iables plays a c ucial ole when modelling ime se ies. Fo nonlinea models he e a e no well de eloped echniques such as AIC and o he c i e ia ha wo k wi h linea models. In he case o Sho Te m Load Fo ecas ing (STLF) gene aliza ion is g ea ly in luenced by such selec ion. In his pape wo app oaches a e compa ed using eal da a om a Spanish u ili y company. The models used a e neu al ne wo ks al hough he algo i hms can be used wi h o he nonlinea models. The expe imen s show ha ha inpu a iable selec ion a ec s he pe o mance o o ecas ing models and hus should be ea ed as a gene aliza ion p oblem. Keywo ds: Au oco ela ion, Au o eg essi e models, Neu al ne wo ks, Time-se ies analysis. 1. INTRODUCTION Some imes a model is needed o o ecas he u u e beha io o a ime se ies. Pas alues o he ime se ies and pas o o ecas ed alues o o he a iables can po en ially be used as inpu s o he model. When a g oup o (mos ly) independen a iables a e a ailable, a iable selec ion has o be pe o med in o de o a oid using a iables ha ha e li le in luence on he o ecas and, a he same ime, do no neglec use ul ones. Fo wa d inclusion and Backwa d elimina ion by means o co ela ion ma ix a e widely used me hods. Al- hough i has been used success ully in nonlinea sce- na ios, i has o be no ed ha co ela ion seeks o almos linea ela ionships among a iables. When o he ela ionships a e p esen , such as quad a ic, he co ela ion es can yield w ong esul s. A b u e o ce app oach can in some cases p o ide he bes esul s. Ei he by s ep-wise inclusion o dele ion o a iables a each s age a ull model has o be de eloped. The pe o mance o he model is es ed using some con ol da a and allows o de e mine he bes se o a iables. This me hod has a numbe o ob ious d awbacks. Fi s , i is e y ime consuming, and does no scale well wi h bo h he size/complexi y o he model and he numbe o po en ial a iables. Second, la ge amoun s o con ol da a a e needed, o he wise he imp o emen s in he a iable selec ion p ocedu e may no ha e co espondence in he la e use o he model. In many si ua ions da a is sca ce disallowing he use o such me hod. Sho e m elec ical load o ecas ing (STELF) is a ask in which nonlinea i ies, model complexi y and sca ce da a a e combined. These p oblems ha e been add essed in (Yuan and Fine, 1998). The di e ence index p esen ed he e is used wi h some modi ica ions in his pape and compa ed agains he b u e o ce app oach. A compa ison is ca ied ou using ac ual da a om a Spanish u ili y company. Gene aliza ion issues a e ackled wi h special ca e in he expe imen s ha simula e he use o he models unde di e en condi ions. The algo i hms o a iable selec ion a e p esen ed nex . The o ecas ing p oblem and he a ailable da a used o he compa ison a e shown in sec ion 3. Fol- lowing his, he esul s will be de ailed and he conclu- sions p esen ed. Copy igh © 2002 IFAC www.else ie .com/loca e/i ac Copy igh © 2002 IFAC 15 h T iennial Wo ld Cong ess, Ba celona, Spain 463 2. INPUT VARIABLE SELECTION ALGORITHMS Two algo i hms o a iable selec ion will be de- sc ibed he e. The i s one consis s on s ep-wise in- clusion o a iables in neu al models. In bo h cases he a ailable pas da a is spli in wo se s: one o cons uc ing models and he o he o es ing hei goodness. E alua ion can be seen as a unc ion ha assigns a igu e o me i o each model, allowing hei classi ica ion. Le us deno e by E(M) he alue o he e alua ion unc ion o model M. Also, he se o po en ial a iables is i s p uned e- mo ing a iables ha ha e a clea quasi-linea ela- ionship among hem. Le us deno e by n he numbe o emaining po en ial a iables. Ou p oblem is o decide which o he po en ial a iables should be used as inpu s in he model. 2.1 S ep-wise inclusion o a iables This s a egy adds one a iable a a ime o he se o selec ed a iables. This a iable is selec ed p obing all possible models ha esul om he addi ion o any a iable o he se o po en ial a iables. The i s s age o he selec ion algo i hm is he con- s uc ion models M1 1 o M1 nwhe e de supe index deno es he numbe o inpu a iables used by he model and he subindex is he a iable las added o ha model. The model ha achie es he bes pe o - mance indica es which a iable should be i s se- lec ed. Ma hema ically 1=a gminjE(M1 j). Va iable 1is hen elimina ed om he se o po en ial a iables and included in he se o selec ed ones. In a second s ep n−1 models wi h wo inpu a iables a e conside ed and a new a iable selec ed. The p ocedu e can go on o e e because heo e ically a model wi h an ex a a iable can pe o m as well as he o iginal model. We know howe e ha gene aliza- ion deg ades when a unnecessa y complexi y is used. The p oblem o when o s op is some imes sol ed plo ing he goodness o he successi e models agains he numbe o a iables. Small changes a e expec ed when including dispensable a iables whe eas signi i- can imp o emen s occu when an impo an a iable is added. Fo his eason he cu e has a la ge slope when he i s a iables a e added and la ens succes- si ely. The elbow o he cu e has been many imes signaled as he co ec numbe o a iables. As we will see in he expe imen s, his choice is coupled wi h he pe o mance unc ion conside ed. When using neu al ne wo ks as models o he consid- e a ions ha e o be made. Fi s o all, he e ec s o andom ini ial alues o weigh s and aining s opping c i e ion cause ne wo ks wi h he same inpu s and same aining da a o pe o m di e en ly. Second, he numbe o nodes/connec ions also a ec he capaci y o esul ing model. To a oid hese p oblem a numbe o ne wo ks should be gene a ed and used as an en- semble a e aging hei ou pu s. The ne wo ks can be ained using di e en algo i hms and do no ha e o sha e he same s uc u e o size. 2.2 Index based selec ion The second algo i hm o be compa ed he e is based on a selec ion index ha can be ob ained di ec ly om da a wi hou he need o cons uc ing any model. The index p oposed in (Yuan and Fine, 1998), is a di e ence-based es ima o o esidual a iance. We will use i he e wi h mino modi ica ions. Gi en a se o a ailable da a o med by a pai s o inpu ec o s and a ge alues (xi, i). The inpu ec o is composed o ndi e en a iables. Le us deno e by xi,j he j- h componen o he i- h da a pai . Fo each po en ial inpu a iable jan indica ion o i s use ulness can be ob ained by means o index Ij. This index measu es he changes in he a ge associa ed wi h changes in he j- h a iable. To do so, all pai s (xi,j, i)o a iable and a ge a e conside ed and eo de ed so ha he new pai s (xp,j, p) e i y ha xp,j≤xq,j o any p<q. The index is hen calcula ed as Ij= N ∑ h=1 (h+1)− (h)(1) whe e Nis he numbe o da a pai s. The way he index is used o selec ion is desc ibed now. Po en ial inpu a iables a e i s di ided among g oups, con aining each g oup a iables ha a e co - ela ed. The index is ob ained o all he a iables in each g oup. The one wi h lowe index is conside ed he bes ep esen a i e o he g oup and is selec ed. P oceeding in his manne , he chance o selec ing edundan (“collinea ”) a iables is uled ou . This p ocedu e does no cons uc any model, speed- ing up he selec ion p ocess. 2.3 S ep-wise index based selec ion We p opose a modi ica ion o he index based p oce- du e. The me hod consis s on calcula ing he di e - ence index o all a iables as be o e. Now, a e he i s a iable has been selec ed i is elimina ed om he se o po en ial inpu a iables. A model is cons uc ed using his one a iable. The ou pu o he model ˆ 1is an es ima ion o he a ge a iable. We use he e o o he model ( he esiduals) as a new a ge : 2= − ˆ 1. New di e ence index a e hen calcula ed o he 464 elemen s o he se o po en ial inpu a iables, and he p ocess epea s i sel . The numbe o models o be cons uc ed is less han in he b u e o ce app oach ou lined in sec ion 2.1. 3. FORECASTING PROBLEM The p oblem o sho e m load o ecas ing e e s o he p edic ion o he ene gy demand wi hin a ho izon o 24 o 48 hou s. Di e en app oaches ha e been used o his p oblem anging om linea o neu al ne s and uzzy se s. Many pape s can be ound in he li e a u e, especially in he IEEE T ansac ions on Powe Sys ems (see o ins ance (Chow and Leung, 1997)). In some cases (A ahal and Camacho, 2001), he hou ly load o ecas ing p oblem is di ided in wo: he p edic- ion o he in eg a ed demand o a day and he no - malized hou ly load cu e o he day. No malized load cu es a e used o dis ibu e he in eg a ed p edic ed demand among he 24 hou s o he day. In his pape he o ecas ing o he load cu e o he day is no con- side ed. The algo i hms unde conside a ion a emp o p edic he o al (i.e. in eg a ed) ene gy demanded o e 24 hou s o he nex wo o h ee days. Some no a ion is needed in o de o p esen he algo- i hm. The hou ly load ld hdemanded a hou ho a day dis usually he a iable used o elec ical companies o o ecas ing pu poses. Howe e , he u ili y used as a es bed is mainly in e es ed in p edic ing he in e- g a ed load o a day; ha is: c(j)= h=24 ∑ h=1 lj h(2) This in eg a ed load will be e e ed o as "daily load". The sequence o alues {c(j)} o j=0,1,...,nd cons i u es he da a base o elec ical load. Simila da a base exis s o he empe a u es measu ed in he egion whe e he elec ical ene gy is supplied and o o he a iables o in e es . Suppose ha we wish o p edic he daily load o day k. Di e en cases appea : (1) I kis a Monday, he p edic ion has o be made he p e ious F iday k−3. The ad ance in he o ecas is d=3 days. (2) I kis a Sunday, he p edic ion has o be made he p e ious F iday k−2. The ad ance in he o ecas is d=2 days. (3) I kis he day a e a holiday, he p edic ion has o be made he wo king day be o e he holiday, like in he p e ious case, excep in Mondays ollowing a holiday in F iday, whe e he ad ance is d=4 days. (4) In he es o he cases he p edic ion is done he p e ious day k−1. The ad ance in he o ecas is d=1 day. F om he cases abo e i is clea ha he alues o c(j) a e known o all j<k−d. These pas loads can be used o gene a e a p edic ion ˆc(k). The ha allows o dis inguish be ween he p edic ion and he ac ual alue o he load c(k), which is only known he day k+1. The objec i e o a o ecas e is o p oduce ˆc(k) o some days ahead (see abo e cases). A igu e o me i o en used is he p edic ion e o de ined as: ep(k)=100c(k)−ˆc(k) c(k)(3) his quan i y can be measu ed a e day kand is an indica ion o how good he p edic ion was. Usually a se o pas days wi h known load demand a e used o e alua e he goodness o a p edic o using he oo mean squa ed e o E ms =1 nd nd ∑ k=1 ep2(k)(4) 3.1 Da a se The da a has been supplied by a Spanish elec ical company ha does no wish o be u he iden i ied. I consis s o hou ly powe demand o a as egion du ing se e al yea s (see (Pa ón and A ahal, 2000)). In o de o main ain he con iden iali y o he da a, he plo s gi en in he pape co espond o a uniden i ied pe iod o ime and he e ical scale is no malized. The a ailable da a se has been spli in h ee pa s (see igu e 1): •Wo king se .Con ains ou yea s o dayly load. I would be spli in wo disjoin se s, namely Cons uc ion Se (CS) used o he aining o he neu al ne wo ks and Valida ion Se (VS) used o signal when o s op he aining o he ne wo ks and o selec he mos app op ia e ne s among a g oup o ne s. •Tes se (TS). Con ains one yea o da a which is pos e io o WS ha will be used o compa e he di e en selec ion algo i hms. The selec ion o he a iables o he algo i hm will be ca ied ou using jus he da a in WS. On he o he hand, he selec ion algo i hms will be es ed using da a om he TS. I has o be ema ked ha da a om TS will no be used (o e en disclosed) a e he algo i hms ha e chosen hei se o inpu a iables. 4. EXPERIMENTAL RESULTS The a iables conside ed as po en ial inpu s a e: 1=c dam Load o he p e ious day o he same ype. 2=m7da A e age o e pas se en days. 3=m14da A e age o e pas ou een days. 465 200 400 600 800 1000 1200 1400 1600 0.75 0.8 0.85 0.9 0.95 1 1.05 1.1 1.15 1.2 1.25 days no malized daily load WS TS Fig. 1. No malized elec ical daily load in he wo king se WS=CS+VS and in he es ing se TS. 4=mdaa A e age o e a week lagged a yea . 5=mm2ma A e age o e a mon h lagged wo mon hs. 6=mm3ma A e age o e a mon h lagged h ee mon hs. 7=mm4ma A e age o e a mon h lagged ou mon hs. 8=mm6ma A e age o e a mon h lagged six mon hs. 9= Fo ecas ed a e age daily empe a u e. 10 =m 2da A e age empe a u e o e pas wo days. 11 = d Type o day: 0-holiday {1 – 7 } Sunday h ough Sa u day. 12 =da Day o he yea . In o de o es he selec ion o inpu a iables a numbe o models ha e been ob ained. Each model di e s jus in he inpu ec o used. The models a e neu al ne wo ks o one-hidden laye wi h 15 nodes ained wi h he Le enbe g-Ma qua d algo i hm in MATLAB. T aining is pe o med using jus da a om CS o adjus he ne wo ks pa ame e s. The E ms in he VS is moni o ed o s op he aining i e a ions. The CS consis s o a 20% o he da a poin s in WS andomly selec ed. The VS is he emaining 80%. Since he ini ial weigh s o he ne wo k can a ec he esul s 50 ne wo ks a e cons uc ed ins ead o jus one. Deno ing by JCS he E ms in he CS and by JVS E ms in he VSa new igu e o me i is in oduced JWS =0.2JCS +0.8JVS (5) The 15 ne s ha p o ide he smalle alue o JWS a e jus conside ed. The esul s shown in he ollowing a e an a e age o such 15 ne s. The numbe o possible a iables is wel e, so i would be necessa y o gene a e 12 +11 +... +2=78 di e - en models o ge he comple e ee o inpu a iables ele ance. This is excessi ely ime consuming. 4.1 S ep-wise inclusion The nex able shows he a e age alue o JWS o he bes 15 ne s ou o 50. Each ow co espond o an inpu a iable and each column o a s age o he inclusion algo i hm. I can be seen ha in he i s s age ( i s column) all inpu a iables a e conside ed one by one. The smalle alue appea s in he i s ow, consequen ly 1 is selec ed om he se o po en ial a iables and used in subsequen s ages. In he second s age 1is combined one by one wi h he es o po en ial a iables. The smalle alue o JWS appea s in he las ow, co esponding o 11. This is hus he second a iable o be selec ed. I is in e es ing o no e ha , a his s age, all models yield be e pe o mance ha in he p e ious, as expec ed (see sec ion 2.1). 12345 14.94 58.31 4.68 4.23 3.78 3.69 68.94 4.81 4.55 3.97 3.72 97.57 4.78 3.99 3.73 10 7.09 4.69 3.97 11 7.21 4.65 Table 1. JWS o he di e en models a each s age o he selec ion me hod. Using his me hod he a iables a e selec ed in his o de : 1, 11, 10, 9and 5 4.2 Index based selec ion Using linea co ela ion ma ix i is possible o educe he numbe o a iables analyzed making g oups o simila a iables as commen ed abo e. Doing so he 466 numbe o a iables can be educed o ou 1= c dam , 4mdaa, 9= and 11 = d. These a iables ha e p obed as he bes a iables om hei g oups. I has o be commen ed ha his me hod and he nex ha e a p ac ical p oblem wi h a iables ha ake dis- c e e alues such as d. Fo his eason da is no mally selec ed by his me hod. Howe e , in o de o compa e wi h he p e ious one d has been selec ed ins ead o da. The index is calcula ed o each o he abo e p e- selec ed a iables using da a o WS. The esul s a e shown in he nex able. 1 4 9 11 20.64 37.58 32.96 36.03 Table 2. Values o I o he p e-selec ed a iables. 4.3 S ep-wise index based selec ion The second index based me hod c ea es in e media e models wi h inpu a iables elec ed ia he Ij. The esul s a e shown in he nex able. Jus he signi ican po ion o he able is shown o he sake o cla i y o p esen a ion. 12345 121.2 231.6 23.8 21.0 21.3 17.6 333.5 22.6 20.6 834.2 23.5 21.0 20.9 17.4 932.3 22.4 20.6 20.6 11 36.4 21.9 Table 3. Index o he di e en a iables a each s age o he hi d selec ion me hod. In he i s s age he a iable wi h lowe index is 1 and hus is selec ed. Simila ly he es o a iables conside ed a e o de ed yielding he sequence: 1, 11, 3, 9and 2. 5. COMPARISON A di e en way o looking a he esul s o he he di e en algo i hms is o es he imp o emen s (i he e is any) in he models when a new inpu a iable is added. In igu es 2, 3 and 4 he a e age alue o JCS (da k) and JVS ( ai g ay) is plo ed o he h ee selec ion me hods agains he numbe o inpu a iables. Acco ding o he wo i s me hods he bes esul s a e ob ained wi h i e inpu a iables. The hi d me hod sugges s ha ou a iables a e bes . We ha e o keep in mind ha hese selec ion has been pe o med using pas da a. The goodness o he models when con on ed wi h new da a has ye o be es ed. To his end we disclose he TS se and use e e y Fig. 2. JCS (da k) and JVS ( ai g ay) o he models de i ed o he i s selec ion me hod. Fig. 3. JCS (da k) and JVS ( ai g ay) o he models de i ed o he second selec ion me hod. Fig. 4. JCS (da k) and JVS ( ai g ay) o he models de i ed o he hi d selec ion me hod. model o o ecas in his new condi ions. This es is a hough one since no addi ional aining was pe o med and he TS se span one whole yea in he u u e. Fo each model, he abo e selec ed 15 ne s a e used wi h he new da a se , ob aining E ms in he usual way. The a e age o hese oo mean squa ed e o s is JTS, which is plo ed in igu e 5. The squa e ma k 467 co espond o me hod 1, he iangle o me hod 2 and he ci cle o me hod 3. 1 1.5 2 2.5 3 3.5 4 4.5 5 3.5 4 4.5 5 5.5 6 6.5 7 no. o a iables RMS E o Fig. 5. JTS e sus he numbe o inpu a iables o each model using a iables selec ed by he i s me hod (squa es), he second ( iangles) and he hi d me hod (ci cles). I can be seen ha ca e has o be aken since he bes esul s a e ac ually ob ained wi h jus h ee inpu a iables. I is also in e es ing o no ice ha o ou a iables he h ee me hods ob ain e y close and accep able solu ions. A conclusion we d aw in he ligh o his compa ison is ha index-based inpu a iable selec ion is easy o pe o m and less ime consuming and can (as in his case) lead o success ul selec ion. Also, i is wo h no ing ha he way he neu al ne s ha e been ained and selec ed g ea ly a ec s gene al- iza ion. A he ime o closing his d a e sion we ha e ye no comple e esul s on how his selec ion should be pe o med so as o a oid (as much as ha is possible) he misma ch be ween E(M)and JTS.Fo ins ance, p io o he cu en ly used e alua ion unc- ion o he pe cen ages o CS and VS whe e used wi h poo e esul s. I seems ha he i s index-based me hods a oids he o e pa ame e iza ion ap by using geome ical in o ma ion whe eas model-based selec ion me hods need o be e y ca e ul in his espec . 6. CONCLUSIONS Th ee me hods o inpu a iable selec ion ha e been compa ed using eal da a in he p oblem o sho e m load o ecas ing and he p oblems associa ed wi h each app oach commen ed. F om he esul s gi en i is clea ha inpu a iable se- lec ion can se iously a ec gene aliza ion. Mo eo e , i has been shown ha he p oblem o inpu a iable selec ion, neu al model assessmen and gene aliza ion a e coupled. Fu he esea ch will be aimed a de eloping neu al model es ing p ocedu es ha would selec ne s ha ha e he bes gene aliza ion capabili ies. 7. REFERENCES A ahal, M.R. and E.F. Camacho (2001). Neighbo his o ies o sho e m load o ecas ing. P o- ceedings o he Eu opean Con ol Con e ence pp. 2796–2801. Chow, T.W.S. and C.T. Leung (1997). Neu al ne wo k based sho - e m load o ecas ing usign wea he compensa ion. IEEE T ansac ions on on Powe Sys ems 11, 1736–1742. Pa ón, F. and M.R. A ahal (2000). P edicción a co o plazo de la demanda de la ene gía eléc ica con siconel. (in spanish) Ac as de las XXI Jo nadas de Au omá ica, CD-ROM 1, ja00_002. Yuan, J.-L. and T.L. Fine (1998). Neu al-ne wo k de- sign o small aining se s o high dimension. IEEE T ansac ions on neu al ne wo ks 9, 266– 280. 468