Performance Criteria to Validate Simulation Models

FACULDADE DE ECONOMIA DA UNIVERSIDADE
DO PORTO
Disse a ion o he Mas e in Quan i a i e Me hods o
Economics and Managemen
Pe o mance C i e ia o Valida e
Simula ion Models
Au ho :
Joana da Ho a Ma ins
Supe iso :
P o esso Ped o Campos
Sep embe 28, 2012
Abs ac
This hesis explo es pe o mance c i e ia adequa e o alida e simula ion models. An
o e iew on he mos widely pe o mance c i e ia used in li e a u e is i s ly p o ided.
The hesis p oceeds wi h he p oposal o wo new c i e ia ha assess he dis o ion o he
wa ping pa h ob ained a e applying he Dynamic Time Wa ping algo i hm: he Wa p-
ing Pa h Dis o ion (WPD) and he Pe cen age Wa ping Pa h Dis o ion (PWPD). A case
s udy ocused on he demog aphic e olu ion o Po uguese i ms is p esen ed, whose e-
sul s a e used o pe o m a compa a i e analysis on all he c i e ia e ised. This wo k
concludes wi h a concise ou line on he c i e ia ad an ages and d awbacks. The c i e ia
WPD and PWPD e u ned adequa e e alua ions, ye u he applicabili y o hese mea-
su es o benchma k da a se s is necessa y o p o ide a p ope conclusion on i s quali y.
Keywo ds: pe o mance c i e ia, alida ion, simula ion models, Wa ping Pa h Dis-
o ion, Pe cen age Wa ping Pa h Dis o ion.
“We ha e desc ibed he p inciple o induc ion as he means whe eby science decides upon
u h. To be mo e exac , we should say ha i se es o decide upon p obabili y. Fo i is
no gi en o science o each ei he u h o alsi y . . . bu scien i ic s a emen s can only
a ain con inuous deg ees o p obabili y whose una ainable uppe and lowe limi s a e
u h and alsi y.”
Hans Reichenbach (in E kenn nis 1, 1930, pp. 186)
i
Acknowledgemen s
I am g a e ul o my supe iso , P o esso Ped o Campos, o he knowledge, pa ience and
mo i a ion he g an ed o me du ing he de elopmen o his hesis.
I wan o cong a ula e all P o esso s o his Mas e cou se, wi h whom I had he plea-
su e o imp o ing my knowledge in di e en subjec s.
I lea e a wo d o g a i ude o he ins i u ion whe e I wo k, INESC TEC, o he lexi-
bili y and suppo always demons a ed.
Finally, I wan o hank o my dea amily and iends, o hei lo e and a ec ion.
Con en s
1 In oduc ion 1
1.1 Maingoalspu sued............................. 3
1.2 Maincon ibu ions ............................. 3
1.3 S uc u e o his hesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Te minology and Da a aspec s 5
2.1 Te minology................................. 5
2.2 Calib a ion and Valida ion me hodology . . . . . . . . . . . . . . . . . . 9
2.2.1 k- old C oss Valida ion . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.2 Spli sample es .......................... 11
2.2.3 P oxy-sys em es .......................... 11
2.2.4 Di e en ial spli -sample es . . . . . . . . . . . . . . . . . . . . 12
2.2.5 P oxy-sys em di e en ial spli -sample es . . . . . . . . . . . . . 14
3 Pe o mance C i e ia O e iew 16
3.1 E o -basedmeasu es............................ 17
3.1.1 Scale-dependen measu es . . . . . . . . . . . . . . . . . . . . . 17
3.1.2 Pe cen age-e o measu es . . . . . . . . . . . . . . . . . . . . . 19
3.1.3 Rela i e-e o measu es . . . . . . . . . . . . . . . . . . . . . . 20
3.1.4 Scale- ee e o measu es . . . . . . . . . . . . . . . . . . . . . . 21
3.1.5 Theil’smeasu es .......................... 22
3.2 In o ma ion Theo y IT based measu es . . . . . . . . . . . . . . . . . . 23
3.2.1 En opy............................... 23
3.2.2 Kullback-Leible Di e gence - DKL ................ 25
3.2.3 No malized In o ma ion Theo y measu es . . . . . . . . . . . . . 26
3.3 Valida ion using In o ma ion C i e ia - IC . . . . . . . . . . . . . . . . . 28
3.3.1 Akaike In o ma ion C i e ion - AIC ................ 28
3.3.2 Bayesian in o ma ion c i e ion - BIC ............... 29
iii

3.4 Pa ame ic es s............................... 29
3.4.1 Coe icien o co ela ion - .................... 30
3.4.2 Coe icien o de e mina ion - R2................. 31
3.4.3 C oss co ela ion - ρ(n)...................... 32
3.5 Nonpa ame ic es s............................. 32
3.5.1 Sign es ............................... 33
3.5.2 Spea man’s ank co ela ion coe icien - S............ 34
3.5.3 Kendall’s au - τ.......................... 35
3.5.4 Wilcoxon Signed Rank Tes . . . . . . . . . . . . . . . . . . . . 36
3.6 Dis ance-based measu es . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.6.1 Minkowski dis ance - d ...................... 38
3.6.2 Sho Time Se ies Dis ance - dSTS ................. 39
3.6.3 Dynamic Time Wa ping - DTW .................. 39
3.7 Combinedmeasu es............................. 42
3.7.1 Sp ague & Gea e o - CS&G.................... 42
3.7.2 Russel’s e o - CR......................... 43
3.7.3 No malized In eg al Squa e e o - CNISE ............. 43
4 Measu ing he Wa ping Pa h Dis o ion 45
4.1 Wa ping Pa h Dis o ion- WPD ...................... 45
4.2 Pe cen age Wa ping Pa h Dis o ion - PWPD ............... 46
4.3 A p ac ical example o WPDand PWPD ................ 47
5 P ac ical Explo a ion o Pe o mance C i e ia 52
5.1 PoFi Cons uc ion.............................. 52
5.1.1 Algo i hm used when a i m is dead . . . . . . . . . . . . . . . . 53
5.1.2 Algo i hm used when a i m is ali e . . . . . . . . . . . . . . . . 54
5.2 Resul s.................................... 54
5.2.1 Numbe o i ms in No e ...................... 55
5.2.2 Numbe o i ms in Cen o ..................... 59
5.2.3 Numbe o i ms in Lisboa ..................... 62
5.2.4 Numbe o i ms in Alen ejo .................... 66
5.2.5 Numbe o i ms in Alga e .................... 67
6 Conclusions 70
i
Lis o Figu es
2.1 Te minology o he cons uc ion o compu a ional simula ion models
( om[41]). ................................. 6
3.1 Schema ic ep esen a ion o In o ma ion Theo y quan i ies. . . . . . . . . 24
4.1 Exempli ica ion o a wa ping pa h e e ing o he wo s i ing si ua ion. . 47
4.2 Vec o s applied unde he example. . . . . . . . . . . . . . . . . . . . . . 48
4.3 Visualiza ion o he wa ping pa h o he example gene al case....... 50
5.1 Diag am ep esen ing he algo i hm ollowed by PoFi. This diag am ex-
plo es he ansi ion o one i m du ing one i e a ion (be ween and +1).
S a e o he i m: X, sec o o he i m: S, age: A, geog aphical egion: N,
dimension: D, a ia ion o dimension: ∆d.................. 53
5.2 Simula ed and obse ed da a om he Po uguese case s udy. . . . . . . . 56
5.3 Wa ping pa h isualiza ion o he numbe o i ms in No e. ....... 57
5.4 Wa ping pa h isualiza ion o he numbe o i ms in Cen o........ 60
5.5 Wa ping pa h isualiza ion o he numbe o i ms in Lisboa........ 62
5.6 Wa ping pa h isualiza ion o he numbe o i ms in Alen ejo....... 65
5.7 Wa ping pa h isualiza ion o he numbe o i ms in Alga e. ...... 68
Lis o Tables
3.1 Nonpa ame ic es s o pai ed obse a ions (adap ed om [11]). . . . . . 33
5.1 Pe o mance C i e ia summa y o he numbe o i ms in No e. . . . . . 55
5.2 Pe o mance C i e ia summa y o he numbe o i ms in Cen o. . . . . 60
5.3 Pe o mance C i e ia summa y o he numbe o i ms in Lisboa. . . . . 63
5.4 Pe o mance c i e ia summa y o he numbe o i ms in Alen ejo. . . . . 65
5.5 Pe o mance C i e ia summa y o he numbe o i ms in Alga e. . . . . 67
i
Nomencla u e
ρ(n)C oss co ela ion
τKendall’s Tau
AIC Akaike In o ma ion C i e ion
AR Au o eg essi e
ARMA Au o eg essi e Mo ing A e age
BIC Bayesian In o ma ion C i e ion
CNISE No malized In eg al Squa e combined e o
CRRussel’s combined e o
CS&GSp ague and Gea combined e o
CR C i ical Region
d Minkowski dis ance
DKL Kullback-Leible Di e gence
d =1 Manha an dis ance
d =2 Euclidean dis ance
dST S Sho Time Se ies dis ance
DIC De iance In o ma ion C i e ion
DTW Dynamic Time Wa ping
EIC Ex ended In o ma ion C i e ion
ii
Chap e 2
Te minology and Da a aspec s
This chap e includes wo main sec ions. The i s sec ion p o ides a desc ip ion o el-
e an e minology o he cons uc ion o compu a ional simula ion models. The second
sec ion includes a summa y o e iew on he classic me hodologies o coo dina e he p o-
cesses o alida ion and calib a ion, alongside wi h he classic echniques o da a spli
in o calib a ion and alida ion da a se s. Whene e ele an , he suppo o he example
demog aphic e olu ion o Po uguese i ms is used.
2.1 Te minology
The wo k de eloped by Re sgaa d [41] e iews many pe spec i es on main concep s con-
ce ning compu a ional simula ion models, and p oposes an uni ying e minology. The
e minology p oposed in [41] is adop ed in his hesis, and is summa ily desc ibed. A
causal- ela ionship scheme including he ele an e ms is p esen ed in Figu e 2.1.
•Reali y
The eal sys em o be s udied and modeled.
Fo example, a se o eal Po uguese i ms could be a eali y o s udy. F om he
eal i ms, i is possible o collec his o ic da a conce ning ele an ea u es o s udy
u he , such as he numbe o li ing i ms in each geog aphic zone o e ime.
•Concep ual model
A concep ual model includes (i) a ma hema ical o mula ion (equa ions) and (ii)
a desc ip ion on he mos ele an ea u es o be simula ed. I aims o desc ibe
he eali y in e ms o e bal desc ip ions, equa ions, go e ning ela ionships o
5

Figu e 2.1: Te minology o he cons uc ion o compu a ional simula ion models ( om
[41]).
na u al laws, using he mos accu a e pe cep ion o he key ea u es o be modeled
(pe cep ual model) and he co esponding simpli ica ions and nume ical accu acy
limi s ha a e assumed accep able. A concep ual model cons i u es he scien i ic
hypo hesis o heo y assumed o he model unde s udy.
Fo example, he hypo hesis he bi h a e o i ms in a speci ic momen is expo-
nen ially dependen on he numbe o li ing i ms o ha same momen could be
o mula ed o be u he es ed. This hypo hesis can only be p ecisely speci ied
p o ided all a iables used (bi h a e o i ms, numbe o li ing i ms and ime
momen s) a e desc ibed unambiguously.
•Model code
A compu e p og am wi h he implemen a ion o he gene ic ma hema ical o mu-
la ion hypo hesized in he concep ual model. The gene ali y o a model code means
ha i can be used o c ea e dis inc models o di e en case s udies, using he same
elemen al equa ions and allowing dis inc inpu a iables and pa ame e alues.
•Model
The model is es ablished o a pa icula case s udy. I is cons uc ed om he model
6
code, using inpu da a o ind pa ame e alues.
•Code e i ica ion
The code e i ica ion is he ask which allows he subs an ia ion ha a model code
is a sui able ep esen a ion o a concep ual model, conside ing speci ied limi s o
anges o applica ion and co esponding anges o accu acy. This is no mally done
wi h a me hodological debug o model code, in o de o ensu e ha i pe o ms
exac ly he desi ed asks.
In he wo k de eloped by Gilbe and T oi zsch [16], ocused on simula ion o
social sys ems, he model e i ica ion i de ined as “ he p ocess o checking ha a
p og am does wha i was planned o do” ([16], pp.21). Mo eo e , he ollowing
insigh ul ecommenda ions conce ning he e i ica ion phase a e p o ided by [16]:
(i) he debug should be made ca e ully and p e e ably using a se o es cases, such
as simula ing ex emes si ua ions, in o de o easily check wea he he esul s o he
model a e coinciden wi h he expec able esul s; (ii) i is ad ised he e un o he
model each ime a majo change is made, in o de o easily check possible e o s
wi hin he change made; (iii) each e un o he model should au oma ically include
he whole lis o es cases, eco d he espec i e esul s and i possible, highligh
majo di e ences o e di e en uns; (i ) inally, a eco d o he esul s and he
code o each un should be s o ed wi hin a “ e sion con ol sys em”.
•Model con i ma ion
The assessmen on he concep ual model adequacy o p o ide an accep able le el
o ag eemen o he domain o in ended applica ion. When his assessmen is con-
side ed accep able, he heo ies / hypo heses included in he concep ual model a e
scien i ically con i med. O he wise, he heo ies / hypo heses a e ejec ed.
•Model calib a ion
The p ocess o adjus men o pa ame e alues included in he model in o de o ap-
p oxima e he model esul s o eali y, conside ing a speci ied ange o accu acy in
he pe o mance c i e ia used. The e m aining is equen ly applied wi h simila
meaning (e.g. [32]).
Some modeling app oaches di e en ia e wo asks unde he calib a ion p ocess:
he aining and he es (no ing ha he e m es can be used bo h as a pa o he
calib a ion p ocess and as he alida ion p ocess).
7
•Model alida ion
A model is alida ed when, unde i s domain o applicabili y, he model ou pu s
e u n a sa is ac o y ange o accu acy o a speci ied pe o mance c i e ion, ha
should be consis en wi h he in ended applica ion o he model. The e m es ing
is used wi h he same meaning as alida ion in some modeling app oaches (e.g. in
[32]).
•Model se -up
Es ablishmen o a model o a speci ic case s udy wi h base on he model code.
This es ablishmen is made wi h he de ini ion o (i) bounda y, (ii) ini ial condi ions
and (iii) pa ame e assessmen om ield and labo a o y da a.
•Simula ion
Simula ion is he use o a alida ed model o gain insigh abou eali y. Simula ion
can be used o p edic he e olu ion o some ea u es o o s udy how eali y is
expec ed o e ol e o e he change o speci ic ea u es included in o he model. I
is impo an o conside he unce ain y unde lying he model when de ining he
conclusions o he s udied eali y.
•Analysis
The assessmen o he model quali y conside ing (i) he eali y and (ii) he scien i ic
desc ip ion o eali y (i.e. he concep ual model wi h i s heo ies and equa ions).
The analysis is pe o med wi h analy ic ools, and can lead o conclude on a good
o bad esul o he model con i ma ion.
As explained by [16], when simula ion models a e cons uc ed o ep oduce s ochas-
ic p ocesses (when he eali y o be simula ed is a leas , pa ly based on andom
ac o s), i is app op ia e o pe o m a sensi i i y analysis. A sensi i e analysis o
a simula ion models aims (i) “ o answe ques ions abou he ex en o which he
beha io o he simula ion is sensi i e o he assump ions which ha e been made”
([16], pp.23), and (ii) o in es iga e he obus ness o he model. I is conduc ed by
unning he model unde di e en alues o he ini ial condi ions and pa ame e s
( o example by pe o ming small sequen ial changes o by andomizing hese al-
ues), wi h he analysis o he co esponding ou comes. This app oaches allows he
s udy o he beha io o he model unde di e en condi ions.
•P og amming
8
The c ea ion o he model code om he concep ual model, using compu a ional
ools.
Ano he impo an concep desc ibed in [48] e e s o he quali y o a simula ion
model, whe e h ee main quali y goals a e iden i ied: (i) pe o mance: he abili y o a
simula ion model o execu e i s oles, wi h e iciency and eliabili y; (ii) sa e y: he ade-
quacy o he simula ion model o p e en acciden s when applied o pe o m some p ocess
in he eal wo ld (e.g. con olling a machine); and (iii) secu i y: he adequacy o a sim-
ula ion model o comply wi h laws, no ms and s anda ds ( his goal is connec ed wi h he
opics o con iden iali y, in eg i y o au hen ici y).
A sho summing on he main concep s e ised in his sec ion is now p esen ed. A
simula ion model is designed o alida e o ejec speci ic heo ies and hypo hesis consid-
e ed in he concep ual model, always unde he speci ic condi ions and assump ions made.
The e o e, a simula ion model can be accep ed as alid o speci ic condi ions only, which
means ha unde he condi ions simula ed, he model ep oduces well he ask o which
i was designed. To p o e ha a model ep oduces well i s ask (i.e. o con i m a model)
i is impo an o ollow he s anda d p ocedu es [41], implying he use o da a om e-
ali y o con on he esul s ob ained, and use he calib a ion and alida ion ou comes as
aluable in o ma ion o con i m (i.e. he model co obo a es he hypo hesis de ined in he
concep ual model) o ejec a concep ual model.
2.2 Calib a ion and Valida ion me hodology
A lis on di e en echniques o alida e simula ion models is p o ided by [42]. This
lis includes he ollowing alida ion echniques: (i) Anima ion, (ii) Compa ison o O he
Models, (iii) Degene a e Tes s, (i ) E en Validi y, ( ) Ex eme Condi ion Tes s, ( i) Face
Validi y, ( ii) His o ical Da a Valida ion, ( iii) His o ical Me hods, (ix) In e nal Validi y,
(x) Mul is age Valida ion, (xi) Ope a ional G aphics, (xii) Pa ame e Va iabili y - Sensi-
i i y Analysis, (xiii) P edic i e Valida ion and (xi ) T aces and (x ) Tu ing Tes s. Also,
an empi ical alida ion app oach is sugges ed in [14]. Al hough he ex ensi e ange o
alida ion app oaches a ailable in li e a u e, his hesis is ocused on he His o ical Da a
Valida ion one. Acco dingly, he pe o mance c i e ia e iewed in u he chap e s a e
sui able o assess quan i a i ely pai ed da a se s.
The His o ical Da a Valida ion is applied when he e a e his o ical eco ds o collec ed
da a om eali y. These da a is he e e ence poin ha he model should be able o
9
ep oduce, and is no mally sepa a ed in (i) da a used o build he model (i.e. o use in
model calib a ion) and (ii) da a used o es whe he he model beha es as he sys em
does (i.e. o use in he model alida ion p ocess). Acco ding o [41], he measu emen
o he le el o accep able ag eemen be ween he model and he eali y is made wi h
pe o mance c i e ia, bo h o model calib a ion and model alida ion. I is possible o
use he same pe o mance c i e ia in calib a ion and in alida ion.
The alida ion p ocess is deeply connec ed wi h calib a ion. The me hodology used
o coo dina e hese wo p ocesses is ex emely impo an ( he selec ion o an app op ia e
pe o mance c i e ion would be wo hless i no p ope ly applied).
Conce ning he da a o be applied wi hin he p ocesses o alida ion and calib a ion,
h ee main a ibu es a e de ailed by [32]: (i) he calib a ion da a may p o ide he model
wi h di ec o indi ec in o ma ion conce ning speci ic aspec s he model is in ended o
lea n. No mally, a model lea ns i s asks easily when p o ided wi h di ec in o ma ion,
and he c edi assignmen p oblem may a ise when he model is p o ided wi h indi ec
in o ma ion (see [32], pp. 5); (ii) The sequence o he aining examples p o ided o
he model wi hin he calib a ion da a se is an impo an aspec o conside ( o example
he da a can be andomly o de ed); (iii) he hi d a ibu e is he dis ibu ion simila i y
assessmen be ween calib a ion and alida ion da a se s.
The nex sec ions summa ily desc ibe classic me hodologies used (i) o spli he da a
o be included wi hin he alida ion and calib a ion da a se s, and (ii) o coo dina e he
alida ion and calib a ion p ocesses in o de o achie e meaning ul esul s. Acco dingly,
i e me hods a e e ised. The i s me hod is he c oss alida ion es , which allows al-
ida ion on a sample used o model calib a ion. The second me hod is he classic spli
sample es , which de ines ha he calib a ion da a se is independen om alida ion da a
se . The o he h ee me hods a e a ian s o he spli sample es , all o hem p osed by
Klemeˇ
s [21] o pe o m spli sample es wi hin si ua ions wi h insu icien da a a ailable
and o es he model beha io o changes in ea u es.
No e 1: The me hodology on how o de e mine he amoun o da a needed o mean-
ing ul calib a e a model is no add essed in his wo k, u he eading on his subjec can
be ound in [5].
2.2.1 k- old C oss Valida ion
The k- old c oss- alida ion es di ides he o iginal sample in o ksubsamples. The al-
loca ion o each elemen o each o subsamples is made andomly. Then, one o he k
subsamples is selec ed o be he alida ion da a se . The emaining k−1subsamples a e
10

used as calib a ion da a se . The accu acy esul ob ained wi h his expe imen is s o ed.
This expe imen is epea ed k imes, each ime using a di e en subse as alida ion
da a se .
The kaccu acy esul s om each expe imen a e hen a e aged (o combined wi h
o he c i e ia) o p oduce a single accu acy es ima ion.
The k- old c oss- alida ion es uses all da a elemen s o bo h calib a ion and alida-
ion. Each da a elemen is used o alida ion once. In he gene al case, he numbe o
subsamples k o adop is an unde ined pa ame e .
As ou lined by [13], e-sampling s a egies ha e been commonly misused, o en e-
sul ing in highly biased es ima es o p edic ion.
2.2.2 Spli sample es
The a ailable da a om eali y should be spli in o wo da a se s o use in calib a ion and in
alida ion. Each da a se should be used in u n o calib a ion and o alida ion. Acco d-
ingly, one expe imen would be conduc ed using he i s pa o he da a o calib a ion
and he second pa o he da a o alida ion, using a speci ied pe o mance c i e ion.
Ano he expe imen would be conduc ed using he second pa o he da a o calib a ion
and he i s pa o he da a o alida ion, using he same pe o mance c i e ion.
The esul s ob ained om each expe imen should be compa ed. The con i ma ion o
he model would hen be assessed by: (i) he simila i y o he esul s om he wo expe -
imen s and (ii) he adequacy o he alida ion ou comes wi h he co esponding anges o
accu acy. This means ha he concep ual model may be alida ed i he alida ion ou -
pu s ob ained wi h bo h expe imen s comply wi h he speci ied anges o accu acy o he
pe o mance c i e ia used, and i hese wo alida ion ou pu s a e simila . O he wise, he
concep ual model should be ejec ed.
Fo he au ho knowledge, he e is no consensual de ini ion on he ideal a io used o
spli calib a ion and alida ion da a se s. Fo ins ance in he s udy [21], i is sugges ed ha
when he a ailable da a is su icien ly long, wo equal pa s should be conside ed. In he
s udy [13] se e al es s a e conduc ed o add ess his same ques ion, and he conclusions
indica e anges o a ios depending on he speci ic condi ions o he model and o sample
popula ion.
2.2.3 P oxy-sys em es
The P oxy-sys em es is applicable o models designed o be ans e able o e sys ems.
A ans e able model is use ul when he e is low o any da a a ailable conce ning he
11
sys em o be modeled.
Modelling a sys em wi h no da a a ailable
Le us conside a sys em Cwi h any da a a ailable, and ha we wan o cons uc and
alida e a simula ion model o simula e sys em C.
In his si ua ion i is possible o de elop a model wi h base on wo o he sys ems A
and B, bo h wi h da a a ailable and wi h cha ac e is ics simila o sys em C, allowing he
conjec u e ha hese wo sys ems a e bo h ep esen a i e o sys em C.
The model should be calib a ed on sys em Aand alida ed on sys em B( i s expe i-
men ) and ice e sa (second expe imen ). The model is con i med (i.e. i is concluded
ha he model encompasses a basic le el o c edibili y wi h ega d o i s abili y o sim-
ula e sys em Cadequa ely) only when he alida ion esul s om bo h expe imen s a e
simila and comply wi h he accu acy anges speci ied.
Modelling a sys em wi h low da a a ailable
Le us conside a sys em D, wi h sca ce da a a ailable (i.e. he e a e no su icien da a o
pe o m a spli as he one sugges ed in he spli sample es ). On one hand, he cons uc ion
o a simula ion model o simula e sys em Dcanno be accomplished wi h he sca ce da a
a ailable, as i is no su icien o pe o m calib a ion and alida ion. On he o he hand,
he sca ce da a a ailable a e he bes knowledge abou sys em D, and should be included
wi hin he simula ion p ocess.
In his case, he p ocedu e o adop unde he calib a ion and alida ion p ocesses is
exac ly he same desc ibed in he o me sec ion (P oxy-sys em es o model a sys em
wi h any da a a ailable) wi h he a achmen o a hi d expe imen . The hi d expe imen
consis s on using he sca ce da a se om sys em D o alida ion.
The model is accep ed when he alida ion esul s om he h ee expe imen s comply
wi h he accu acy anges speci ied and a e simila .
2.2.4 Di e en ial spli -sample es
The di e en ial spli -sample es is applicable o alida e he sensi i i y o he model o
espond accu a ely on changes o a speci ic ea u e ha in eg a es he model. This es
may ha e se e al a ian s depending on he speci ic na u e o he change o be simula ed.
12
Modeling a sys em sensi i e o changes on a ea u e, wi h a ailable da a on he ad-
jus able ea u e
Le ’s conside a sys em E o be simula ed, o which he e a e su icien da a o cali-
b a ion and alida ion p ocesses. The simula ion model o sys em Eis in ended o be
sensi i e o changes on a speci ied ea u e Y( he sys em Eincludes a se o dis inc ea-
u es, one o which is ea u e Y).
In his case, he da a a ailable should be spli in o wo pa s acco ding o he alues
obse ed in ea u e Y. Acco dingly, he i s da a se would be composed wi h obse a ions
whe e ea u e Y e u ned high alues (e.g. conside ing he adjus able ea u e bi h a e o
i ms, he i s da a se would include obse a ions whe e high bi h a es we e obse ed).
The second da a se would include obse a ions whe e ea u e Y e u ned low alues. This
implies he exclusion o hose obse a ions whe e ea u e Y e u ned mode a e alues.
•To es he model abili y o ep oduce he eali y o high alues o ea u e Y, he
da a se wi h low alues o Yshould be applied o calib a ion, and he da a se wi h
high alues should be used o alida ion. The model is con i med i he alida ion
esul s comply wi h he accu acy anges speci ied. O he wise, he model is ejec ed.
•To es he model adequacy o beha e wi h low alues o ea u e Y, he same ea-
soning is applied ollowing a e e se o de ( he da a se wi h high alues is used o
calib a ion, and he da a se wi h low alues o Yis applied o alida ion).
Modeling a sys em sensi i e o changes on a ea u e, wi hou signi ican ly di e en
da a on he adjus able ea u e
Le us conside a sys em H o be simula ed. The simula ion model o His in ended o be
sensi i e o changes on a ea u e Y(sys em Hinco po a es se e al ea u es, being Yone
o hem). Le us assume ha , in he gi en eco d, segmen s wi h signi ican ly di e en
alues o ea u e Ycanno be iden i ied ( his si ua ion may occu due o sca ci y o da a
a ailable o because he majo i y o obse a ions ela e o mode a e alues o Y).
In his case he model should be calib a ed and alida ed on wo subs i u e sys ems
Fand G. The wo subs i u e sys ems should (i) encompass su icien and signi ican ly
di e en da a on he adjus able ea u e (allowing he accomplishmen o he di e en ial
spli -sample es de ailed in he o me sec ion), and (ii) be composed o simila cha ac-
e is ics o sys em H(allowing he conjec u e ha sys ems Fand Ga e ep esen a i e o
sys em H).
13
•To es he model adequacy o beha e wi h high alues o ea u e Y, wo expe imen s
should be conduc ed. The i s expe imen uses da a om sys em F: he model is
calib a ed wi h da a ela i e o low alue o ea u e Y, and is u he alida ed wi h
da a whe e high alues o Ywhe e obse ed. The second expe imen uses da a
om sys em G, wi h simila easoning as expe imen 1. The model is accep ed
i he alida ion esul s comply wi h he speci ied accu acy anges and i hey a e
simila o e he wo expe imen s. The model is ejec ed o he wise.
•To es he model adequacy o ep oduce low alues o Y, he ollowing wo expe i-
men s should be conduc ed. The i s expe imen applies he da a wi h high alues
o ea u e Y om sys em F o calib a ion, and he da a wi h low alues o Y om
sys em F o alida ion. The second expe imen applies da a om sys em Gwi h
simila easoning as he i s expe imen . The model accep ance o ejec ion is de-
cided wi h base on he accu acy and simila i y o he alida ion esul s ob ained in
bo h expe imen s.
No e 2: when using wo subs i u e sys ems on di e en ial spli -sample es , he cali-
b a ion and alida ion is done on each subs i u e sys em independen ly, which is di e en
om he p oxy-sys em es whe e a model is calib a ed on one sys em and alida ed on
he o he .
No e 3: As epo ed in [21]: “A di e en ial spli -sample es can a ise by de aul om
a simple spli -sample es i he only meaning ul way o spli ing an a ailable eco d is
such ha he wo segmen s exhibi ma kedly di e en condi ions”.
2.2.5 P oxy-sys em di e en ial spli -sample es
The P oxy-sys em di e en ial spli -sample es is applicable o models designed o be
ans e able bo h be ween sys ems and be ween ea u es. The es may ha e di e en
o ms depending on he speci ic modeling ask pu sued.
Le us conside he sys em K wi h no a ailable da a eco ds. Sys em K includes se e al
ea u es, being one o hem he ea u e Y. I is in ended o es he model adequacy o
eplica e changes on ea u e Y wi hin sys em K.
In his si ua ion wo o he sys ems I and J, bo h ep esen a i e o sys em K and wi h
a ailable and signi ican ly di e en da a on ea u e Y, should be chosen.
The da a eco ds o sys ems I and J a e hen spli conside ing he alues o ea u e Y.
Acco dingly, he da a om sys em I is spli in o wo pa s, one e e ing o high alues
o Y, and he o he conce ning he low alues o Y (obse a ions e e ing o mode a e
14
clusions aken om applying a ela i e-e o measu e o compa ing he espec i e scale-
dependen measu e o bo h models is equi alen . Mo eo e , hese me hods a e no appli-
cable when he e o ob ained om he benchma king model is ze o, as in ha case he
alue o iwould be unde ined [18].
3.1.4 Scale- ee e o measu es
Acco ding o Hyndman [18], he main ad an age o scale- ee e o measu es is ha hey
p o ide mo e accu a e esul s han he me hods e iewed in he o me h ee sec ions
when he da a se s o be assessed a e non s a iona y, meaning ha he da a e ol e ac-
co ding o a pa e n such as end o seasonali y. This does no inculca e any inadequacy
o he me hods e iewed in he o me h ee sec ions o he assessmen o non-s a iona y
da a. Impo an o no e ha he scale- ee e o measu es a e ecen , and he e is s ill ew
compa a i e li e a u e.
Scale- ee e o measu es a e based on scale-dependen measu es, bu hey use a
scaled e o qiins ead o e o ei. The scaled e o is calcula ed as desc ibed in equa-
ion (3.11).
qi=ei
1
N−1
N
X
j=2 |xj−xj−1|
(3.11)
Mean Absolu e Scaled E o - MASE
Fo he au ho knowledge, i was possible o ind only one scale- ee e o measu e ap-
plica ion in li e a u e, e e ing o Mean Absolu e Scaled E o MASE, al hough i is
sugges ed ha he easoning o eplace a scaled e o qiby he classic e o eimay be
ex ended o o he measu es. The MASE is calcula ed as p esen ed in equa ion (3.12),
which is he same as using equa ion (3.3) eplacing eiby qi. This me hod is applied in
[18].
MASE =1
N
N
X
i=1 |qi|(3.12)
Hyndman [18, 19] claims ha MASE is he only a ailable accu acy measu emen
ha can be used in all o ecas ing si ua ions and o all ypes o se ies. No e ha his
claim is based on a compa ison wi h e o -based measu es only, and is jus i ied wi h he
ollowing ad an ages iden i ied o his me hod: i is scale ee and he e o e he alues
21

ob ained can be compa ed ac oss models wi h di e en uni a y sys ems, when applied
o non-s a iona y da a he esul s ob ained a e mo e accu a e han wi h o he e o -based
measu es, i does no incu in unde ined elemen s, i p e en s nega i e and posi i e e o s
om o se ing each o he .
3.1.5 Theil’s measu es
Theil’s measu e o o ecas accu acy - U1
U1was p oposed by Theil in 1966 [51] as a measu e o o ecas accu acy, as speci ied in
(3.13).
U1="1
N
N
X
i=1
(ei)2#0.5
"1
N
N
X
i=1
(xi)2#0.5
+"1
N
N
X
i=1
(bxi)2#0.5(3.13)
When U1= 0, i means ha he es ima ion is comple ely coinciden wi h he obse a-
ions (xi=bxi,∀i), indica ing a pe ec o ecas . The case U1= 1 indica es he maximum
inequali y (when he e is nega i e p opo ionali y o e he wo da a se s o when one o
he da a se s is iden ically o ze o) [8, 51].
Bliemel [8] analyzed bo h measu es p oposed by Theil, U1and U2(see nex sec ion),
and concluded ha U1is only in o ma i e o assess o ecas accu acy when applied o he
absolu e alues o he e o s.
Theil’s measu e o o ecas quali y - U2
The measu e U2was p oposed by Theil in 1965 [50] o assess he quali y o o ecas s.
This measu e is de ined in (3.14).
U2="1
N
N
X
i=1
(ei)2#0.5
"1
N
N
X
i=1
(xi)2#0.5(3.14)
The esul U2= 0 indica es a pe ec o ecas , meaning ha bo h da a se s a e coinci-
den (xi=bxi,∀i). The meaning o he case U2= 1 is cla i ied by [8], s a ing ha his is
obse ed “when he p edic ion me hod is nai e no-change ex apola ion o when i leads
22
o he same s anda d de ia ion o o ecas e o as ha me hod”.
The compa a i e s udy pe o med by Bliemel [8] o e he wo Theil’s measu es, U1
and U2, concludes ha U2p o ides mo e meaning ul in o ma ion on he accu acy o he
es ima ions unde assessmen , sugges ing ha U2should be p e e ably used han U1.
As cla i ied in [33], he s a is ic U2is no mally applied wi hou he o mal de ini ion
o hypo hesis. Ne e heless, i can be applied as he es s a is ic o pa ame ic es when
bo h da a se s come om bi a ia e no mal popula ions. The dis ibu ion Fis applicable
in ha case. Fu he eading on his opic can be ound in [33].
3.2 In o ma ion Theo y IT based measu es
The In o ma ion Theo y IT ield was i s ly de eloped by Shannon [46], wi h he de -
ini ion o he concep s o en opy and mu ual in o ma ion. The IT concep s ha e been
applied in se e al esea ch ields since hen, such as in biology [1], simula ion o agen
based models [52] and compu a ional cybe ne ics [31].
The measu es used in IT a e ma hema ical quan i ies ha eco d he amoun o in o -
ma ion con ained wi hin a da a se . In his sec ion he measu es en opy and Kullback-
Leible Di e gence a e e iewed as hey we e p oposed by Akaike [2] o be applied o
alida e simula ion models. Mo e ecen ly, no malized in o ma ion heo y measu es ha e
been sugges ed o assess he goodness o i o simula ion models, see [53]. These no -
malized measu es a e also e iewed.
The obse ed da a se Xis assumed o ha e, a leas , some andomness (o he wise
i would be a de e minis ic phenomena, and he espec i e simula ion model would e-
u n pe ec es ima ions). The e o e le s conside he andom a iable X, wi h P=
(p1, p2, ..., pN)being he p obabili ies co esponding o each disc e e obse a ion X=
(x1, x2, ..., xN), and
N
X
i=1
pi= 1.
3.2.1 En opy
En opy was i s ly de ined by Shannon in 1948 [46]. Acco ding o Shannon [46], en opy
is he quan i y o in o ma ion linked o he p obabili y o occu ence o a ce ain e en :
(i) en opy is null o e en s whose ou pu is comple ely known a s a and (ii) en opy is
highe he less p edic able an e en is. En opy was p oposed as a pe o mance c i e ion
by Akaike in 1974 [2] unde he con ex o ime se ies es ima ions, and was s udied in
[36] unde he con ex o Ma ko models.
23
Figu e 3.1: Schema ic ep esen a ion o In o ma ion Theo y quan i ies.
The gene al ma hema ical de ini ion o en opy was p oposed by Renyi, he Renyi’s
En opy - H α, which is p esen ed in (3.15), being αa eal pa ame e subjec o he e-
s ic ions speci ied in (3.15), and he e o e he en opy i s ly desc ibed by Shannon Hsis
a pa icula case o he gene al en opy desc ibed by Renyi H α.
H α(X) = 1
1−αlog N
X
i=1
pα
i!
Subjec o: α > 0
α6= 1
(3.15)
Figu e 3.2.1 p o ides a schema ic isualiza ion on in o ma ion heo y quan i ies o
en opy, join en opy and mu ual in o ma ion.
In Figu e 3.2.1 he en opy o he obse ed da a se H(X)is ep esen ed as he blue
ci cle, he en opy o he es ima ed da a se H(b
X)is ep esen ed as he ed ci cle. The join
en opy o e he wo da a se s H(X, b
X)is ep esen ed as he union H(X)∪H(b
X). The
mu ual in o ma ion (see sec ion 3.2.2) is ep esen ed as he in e sec ion H(X)∩H(b
X).
Quad a ic en opy o Renyi
In he case o α= 2 he exp ession (3.15) leads o he quad a ic en opy, due o he
quad a ic o m o he p obabili y, as de ined in (3.16).
H 2(X) = −log N
X
i=1
p2
i!(3.16)
The quad a ic en opy o Renyi has he ad an age o being easie o calcula e h ough
he applica ion o Gaussian con olu ion [7].
24
En opy desc ibed by Shannon - Hs
The en opy desc ibed by Shannon Hsis he weigh ed sum o loga i hms by p obabili ies.
I ep esen s he a e age amoun o in o ma ion included wi hin a single obse a ion o
X. Acco dingly, he Shannon’s en opy Hs(X)is de ined in (3.17).
Hs(X) =
N
X
i=1
pi·log 1
pi
Subjec o:
N
X
i=1
pi= 1
pi≥0
0·log2(0) = 0
(3.17)
The use o en opy as a alida ion c i e ion is howe e icky. The di ec compa ison
be ween he en opy o he his o ic da a se wi h he en opy o he simula ed da a se do
no allow o conclude on he adequacy o he esul s ob ained. As i can be pe cei ed in
image 3.2.1, i is possible o ha e wo independen da a se s e u ning simila alues o
en opy H(X)and H(b
X), as in ha case he mu ual in o ma ion would be ze o.
The use o en opy as alida ion c i e ia, in he ligh o his wo k, should be done
conside ing he wo da a se s simul aneously, using he join en opy o e he wo da a
se s H(X, b
X). A pe ec i would be ob ained o a alue o join en opy equal o he
alue o he obse ed en opy (H(X, b
X) = H(X)). The join en opy is calcula ed as
desc ibed in (3.15), using he a ached obse a ions o Xand b
Xins ead o using he
obse a ions Xsepa a ely.
3.2.2 Kullback-Leible Di e gence - DKL
The Kullback-Leible Di e gence DKL is a ma hema ic quan i y which measu es dissim-
ila i y be ween wo di e en p obabili y densi y unc ions p(x)and q(bx)([39] pp. 16).
This measu e is de ined as ollows:
DKL(pkq) = X
x
p(x)logp(x)
q(bx)(3.18)
The di e gence DKL may be in e p e ed as he dis ance be ween he wo p obabili y
densi y unc ions, howe e i does no obey he dis ance ma hema ical pos ula es ([39] pp.
16). The di e gence DKL is o en used o alida e simula ion models, as i is sugges ed
25
in [29]. An example is p o ided by [20], wi h he alida ion o he esul s ob ained wi h a
simula ed annealing model.
Mu ual In o ma ion - I
Mu ual in o ma ion Iis a special case o he di e gence DKL: when he wo p obabili y
densi y unc ions unde assessmen a e (i) he join p obabili y densi y unc ion o p(x, bx)
and (ii) he p oduc o he ma ginals o p(x)q(bx)([39] pp. 18). Mu ual in o ma ion Iis
de ined as ollows.
I(X, b
X) = DKL(p(X, b
X)kp(X)q(b
X)) (3.19)
In he scope o his hesis, he mu ual in o ma ion measu e may be use ul when applied
o he wo da a se s unde assessmen . This measu e would allow he measu emen o he
di e gence be ween he dis ibu ions. The measu e I(X, b
X) e u ns alues in he ange
[0, min(H(X), H(b
X))]. A pe ec i o he es ima ed da a se would e u n he maximum
alue o I(X, b
X), as in ha si ua ion I(X, Y ) = H(X) = H(b
X). A bad es ima ed da a
se would e u n I(X, b
X) alues close o ze o.
The s udy [28] compa es mu ual in o ma ion and co ela ion measu es, cla i ying ha
al hough bo h measu e dependence o e da a se s, mu ual in o ma ion measu es a gene al
dependence, while he co ela ion unc ion measu es a linea dependence. Thus, mu ual
in o ma ion is conside ed a be e quan i y han he co ela ion unc ion o measu e he
dependence o e wo da a se s.
3.2.3 No malized In o ma ion Theo y measu es
The no malized IT measu es we e cons uc ed wi h base on he IT measu es de ined
in he p e ious wo sec ions. These measu es a e applied in [53]. A summa y o no -
malized IT measu es is p o ided in (3.20) conce ning No malized Mu ual In o ma ion
NMIc i e ia, and in (3.21) conce ning No malized In o ma ion Dis ance Measu es Idc i e ia.
26

NMIjoin =I(X, b
X)
H(X, b
X)
NMImax =I(X, b
X)
max(H(X),H(b
X))
NMImin =I(X, b
X)
min(H(X),H(b
X))
NMIsum =2·I(X, b
X)
H(X)+H(b
X)
NMIsq =I(X, b
X)
√(H(X)·H(b
X))
(3.20)
All No malized Mu ual In o ma ion de ailed in 3.20 e u n alues wi hin he ange
[0,1]. Values close o ze o indica e a bad es ima ion, whils alues close o one indica e a
good es ima ion.
Idjoin = 1 −I(X, b
X)
H(X, b
X)
Idmax = 1 −I(X, b
X)
max(H(X),H(b
X))
Idmin = 1 −I(X, b
X)
min(H(X),H(b
X))
Idsum = 1 −2·I(X, b
X)
H(X)+H(b
X)
Idsq = 1 −I(X, b
X)
√(H(X)·H(b
X))
(3.21)
The No malized In o ma ion Dis ances Measu es e u n alues wi hin he ange [0,1],
wi h ze o indica ing a good es ima ion and one indica ing a bad es ima ion.
27
3.3 Valida ion using In o ma ion C i e ia - IC
The alida ion o simula ion models may be pe o med using In o ma ion C i e ia, when
he model o be alida ed is s a is ically de ined, such as a linea model o and au o-
eg essi e mo ing a e age ARMA model. When he model o be alida ed is no de in-
able s a is ically, meaning ha he alida ion is o be pe o med using only he da a se s
( ha is he case o Sys em Dynamics models), in o ma ion c i e ia a e no applicable.
In [23], se e al in o ma ion c i e ia a e p esen ed, including: (i) Akaike In o ma ion
C i e ion AIC, (ii) Bayesian In o ma ion C i e ion BIC (also known as Schwa z In o -
ma ion C i e ion SIC), De iance In o ma ion C i e ion DIC, (i ) Ex ended In o ma ion
C i e ion EIC, ( ) Focused In o ma ion C i e ion FIC, ( i) Gene alized In o ma ion
C i e ion GIC, ( ii) Ne wo k In o ma ion C i e ion NIC, and ( iii) Takeuchi’s In o ma-
ion C i e ion TIC. Acco ding o [23], he majo i y o he c i e ia included in his lis a e
modi ica ions o gene aliza ions o AIC and BIC, which a e he wo mo e commonly
used.
3.3.1 Akaike In o ma ion C i e ion - AIC
A widely used model-selec ion c i e ion is Akaike’s In o ma ion C i e ion (AIC) (see [3,
30, 23]). AIC is an asymp o ically unbiased es ima o o he expec ed Kullback-Leible
di e gence DKL (see equa ion 3.18) be ween wo p obabili y densi y unc ions [38, 23, 2],
no mally applied o assess he quali y o he esul s ob ained wi h di e en models when
his o ic da a is a ailable. Thus, he his o ic da a is used as he e e ence, and he esul s
ob ained wi h AIC allow o measu e he di e gence be ween he (i) p obabili y densi y
unc ions ob ained wi h he simula ed esul s and (ii) he p obabili y densi y unc ion o
he his o ic da a. The e o e, he es ima ed model will be close o eali y, he lesse he
dis ance be ween he wo p obabili y densi y unc ions conside ed. Thus, smalle alues
o AIC indica e a be e i o he model.
The AIC o a gi en model is a unc ion o i s maximized alue o he likelihood
unc ion o he es ima ed model (L) and he numbe o es imable pa ame e s (K), as
de ined in equa ion (3.22).
AIC = 2K−2 ln(L)(3.22)
Ano he simila c i e ia, he Minimum In o ma ion Theo e ic C i e ion Es ima e -
MAICE, was also p oposed by Akaike in [2], as an imp o emen o he AIC. Ac-
co dingly, he MAICE elimina es he “need o he subjec i e judgmen equi ed in he
28
hypo hesis es ing p ocedu e o he decision on he le els o signi icance”. In his wo k,
he AIC is adop ed as i is much widely used.
3.3.2 Bayesian in o ma ion c i e ion - BIC
The Bayesian in o ma ion c i e ion BIC, o Schwa z c i e ion (also SBC,SIC and
SBIC) is no mally applied o choose he bes simula ion model om a se o models.
I assesses he quali y o adjus men o a simula ion model. BIC is a widely applied
measu e, no mally alongside wi h AIC, see ([30] pp. 173, [23]). A de ailed desc ip ion
and c i ic on BIC can be ound in [54] and in [23]. BIC is de ined in equa ion (3.23),
whe e is he numbe o deg ees o eedom emaining a e i ing he model. Smalle
alues o BIC indica e a be e i o he model.
BIC =L2− ln(N)(3.23)
In he s udy [23], he c i e ia AIC and BIC a e examined alongside, and he use o
hese wo c i e ia simul aneously o base he assessmen o he quali y o models esul s
is ad ised, as hese c i e ia app oxima e wo di e en a ge quan i ies.
3.4 Pa ame ic es s
I is equen o apply a Pa ame ic es when he e is a p io assump ion ha a sam-
ple came om a dis ibu ion o a pa icula amily (examples o dis ibu ion amilies a e
Gaussian, Binomial, Exponen ial, Gamma, Be a o Weibull). The pa ame ic es s main
aim is o ind a s a is ically signi ican es ima ion o a speci ic pa ame e , using an hypo h-
esis app oach. In his sec ion he Pea son’s coe icien o co ela ion and he coe icien
o de e mina ion a e e iewed, as hese wo pa ame ic es s a e widely used o assess
simula ion models. The use o co ela ion-based measu es is c i icized in [25, 55]. These
s udies highligh ha co ela ion-based measu es a e o e sensi i e o ex eme alues (ou -
lie s) and insensi i e o addi i e and p opo ional di e ences be ween model p edic ions
and obse a ions. These limi a ions can induce in w ong accep ance o simula ion mod-
els, as documen ed in [55].
Le s conside he mean o he his o ic da a se and he mean o he es ima ed da a se
as de ined in (3.24).
29
X=1
N
N
X
i=1
xi
b
X=1
N
N
X
i=1 bxi
(3.24)
3.4.1 Coe icien o co ela ion -
The coe icien o co ela ion, o Pea son’s P oduc -Momen Co ela ion Coe icien ,
ep esen s he deg ee o linea associa ion be ween wo a iables and is ma hema ically
de ined in (3.25).
=
N
X
i=1
(xi−X)(bxi−b
X)
"N
X
i=1
(xi−X)2#0.5"N
X
i=1
(bxi−b
X)2#0.5(3.25)
The coe icien may assume alues be ween he ange [−1,1], and his alue is ab-
solu e and non-dimensional. The in e p e a ion o his coe icien is de ailed in [49]. Ac-
co dingly, (i) a co ela ion coe icien o ze o indica es ha no associa ion exis s be ween
he measu ed a iables, (ii) a posi i e co ela ion coe icien indica es ha an inc ease
in he i s a iable would co espond o an inc ease in he second a iable, and (iii) a
nega i e co ela ion indica es ha whe eas one a iable inc eases, he second a iable
dec eases.
As Taylo [49] explains, a s a is ically signi ican coe icien can only indica e ha
he obse ed sample da a p o ides e idence o ejec he null hypo hesis ha he popula-
ion co ela ion coe icien pa ame e is ze o. The ejec ion o he null hypo hesis allow
o conclude ha he co ela ion coe icien o he popula ion is no equal o ze o. Unde
he con ex o alida ion o simula ion models, i is desi able a alue o s a ically signi i-
can and posi i e, he e o e he hypo hesis es should be o mula ed as speci ied in (3.26).
The applica ion o his es can only be made unde he ollowing assump ions conside ing
bo h samples: he samples a e andom, quan i a i e, no mally dis ibu ed, linea ly ela ed
and ha e he same a iance (homoscedas ici y).
30
i= he ank assigned o (xi,bxi)i di>0
i= he nega i e o he ank assigned o (xi,bxi)i di<0
(3.38)
The assump ions unde lying he Wilcoxon Signed Rank Tes a e: (i) each di ollows
a symme ic dis ibu ion, (ii) all elemen s in Da e mu ually independen and ha e he
same median, (iii) he elemen s o Da e measu ed in an in e al scale. The es may be
o mula ed wi h one- ailed o wo- ailed o m. He e he wo- ailed o m is p esen ed in
(3.39).
H0:E(X) = E(b
X)
H1:E(X)6=E(b
X)
(3.39)
I he anking p ocess included ies, he es s a is ic TS o use in his es is p esen ed
in (3.40).
TS =
N0
X
i=1
i
u
u
N0
X
i=1
2
i
(3.40)
In case o no ies, he es s a is ic TS+should be adop ed, as de ined in (3.41).
TS+=
N
X
i=1
( iwhe e di>0) (3.41)
The c i ical egion is de ined as p esen ed in (3.42).
CR = [0, c1[∪]c2, N0(N0+ 1)/2] (3.42)
The alues c1and c2a e ound in he ables wi h he quan iles o he Wilcoxon Signed
Ranks Tes S a is ic (see [11], pp. 460) o a speci ied signi icance le el α. I TS o TS+∈
CR, he H0is ejec ed. I TS o TS+/∈CR, hen he H0is accep ed and he e is
s a is ical e idence o assume ha bo h da a se s ha e iden ical means wi h a con idence
coe icien o 1−α.
37

3.6 Dis ance-based measu es
The alida ion o simula ion models can also be pe o med wi h he measu emen o he
dis ance be ween he wo da a se s unde assessmen . Se e al dis ance-based measu es
exis in li e a u e, no all can be classi ied as me ics, and o ha eason he e m “mea-
su es” is p e e ed.
In his sec ion, each da a se is used as a ec o , as p esen ed in equa ion (3.43), and
all algo i hms e iewed aim o ound he dis ance be ween he wo ec o s xand b
x.
x= (x1, x2, x3, . . . , xN)T
b
x= (bx1,bx2,bx3,...,bxN)T(3.43)
3.6.1 Minkowski dis ance - d
Minkowski dis ance is de ined in (3.44).
d (X, b
X) = N
X
i=1 |xi−bxi| !1/
(3.44)
Minkowski dis ance has many a ian s, depending on he alue used o he pa ame e
, he wo mos known a ian s a e he Euclidean dis ance, and he Manha an dis ance.
Minkowski dis ance and i s a ian s ha e been widely used o assess simila i y o images
[26, 27].
Euclidean Dis ance - d =2
Euclidean Dis ance is a pa icula case o Minkowski dis ance, when he pa ame e
assumes he alue 2.
d =2(x,b
x) = N
X
i=1 |xi−bxi|2!1/2
(3.45)
Manha an Dis ance - d =1
Manha an Dis ance is a pa icula case o Minkowski dis ance, when he pa ame e
assumes he alue 1.
38
d =1(x,b
x) =
N
X
i=1 |xi−bxi|(3.46)
3.6.2 Sho Time Se ies Dis ance - dST S
The Sho Time Se ies Dis ance was i s ly p oposed by [34] as a measu e o simila i y
be ween ime se ies wi h small numbe o elemen s. This measu e eme ges unde he
con ex o compa ing DNA mic oa ay da a. The dST S is u he applied o measu e
simila i y o e ime se ies by [29], unde he s udy o clus e ing o ime se ies da a. The
ma hema ic de ini ion o dST S is p esen ed in equa ion (3.47).
dST S(x,b
x) =
u
u
N−1
X
i=1 bxi+1 −bxi
i+1 − i−xi+1 −xi
i+1 − i2
(3.47)
As explained by [34], his measu e “co esponds o he squa e oo o he sum o he
squa ed di e ences o he slopes ob ained by conside ing imese ies as linea unc ions
be ween measu emen s”.
3.6.3 Dynamic Time Wa ping - DTW
Dynamic Time Wa ping DTW is an algo i hm which allows he measu emen o disc ep-
ancy be ween wo da a se s. This algo i hm was de eloped unde he con ex o speech
ecogni ion, bu i may be applied in o he esea ch opics, such as he alida ion o sim-
ula ion models, as i is sugges ed in [43, 44].
DTW is a powe ul algo i hm o iden i y whe he wo da a se s “ma ch” wi h each
o he . As explained in [9], DTW aligns peaks and alleys as much as possible by expand-
ing and comp essing he ime axis acco dingly. This is made by inding he smalles pa h
o dis ances be ween wo da a se s. DTW is applicable o wo da a se s wi h di e en
leng hs, bu as his wo k is ocused on pai ed da a se s, DTW is he e conside ed o da a
se s wi h equal leng h.
Le Abe he NxNma ix whe e he (i h,j h) elemen o ma ix Acon ains he
dis ance be ween he wo poin s d(xi,bxj). The ma ix Ais called cos ma ix and is
de ined in (3.48). The dis ance used o calcula e each elemen o Ais called cos unc ion,
and di e en cos unc ions may be applied. Examples o possible cos unc ions a e he
euclidean dis ance d(xi,bxj) = d =2(xi,bxj)o he squa ed e o d(xi,bxj) = (xi−bxj)2
[43]. In his wo k, he euclidean dis ance (see 3.6.1) is always applied as cos unc ion.
39
A=




d(x1,bx1)··· d(x1,bxN)
.
.
.....
.
.
d(xN,bx1)··· d(xN,bxN)




(3.48)
Ma ix Ashould include alues o s eps ha a e conside ed alid, and no alid s eps
should be le in blank.
A new ma ix Bis hen cons uc ed, using he cos ma ix A. The (i h,j h) elemen
o ma ix Bs o es he minimum cumula i e cos o achie e he co esponding posi ion
conside ing he s a ing poin (i= 1,j= 1). The easoning applied o ind he minimum
cumula i e cos o e each combina ion o posi ions is he same as he classic algo i hms
o dynamic p og amming o he Sho es Pa h P oblem, such as he Dijks a algo i hm.
One way o calcula e ma ix Bis de ined in (3.49), as sugges ed in [43].



B11 =A11
Bij =Aij +min(Bi−1,j−1, Bi−1,j, Bi,j−1),i i∧j6= 1
(3.49)
Once he Ma ix Bis de ined, he cons uc ion o he wa ping pa h Wis s a ed.
A wa ping pa h is a se o Kbi a ia e elemen s, being K∈[N, 2N−1]. Each elemen
wk= [iw
k, jw
k](wi h iw
k, jw
k∈[1, N]) s o es he loca ion posi ions o he elemen s o ma ix
Bco esponding o he sho es cumula i e pa h be ween he i s (i= 1,j= 1) and las
(i=N,j=N) posi ions. The wa ping pa h is hus de ined as W=hw1, w2,··· , wKi,
subjec o he ollowing condi ions:
•Bounda y condi ions:w1= [1,1] and wK= [N, N]. This condi ion obliga es he
algo i hm o s a in he i s pai o homologous elemen s, and o inish in he las
pai .
•Con inui y: This condi ion ensu es ha all cells chosen om Aa e adjacen . The
condi ion is o mula ed conside ing wk−1= [iw
k−1, jw
k−1]and wk= [iw
k, jw
k], hen
iw
k−iw
k−1≤1and jw
k−jw
k−1≤1.
•Mono onici y: The las condi ion ensu es he algo i hm o e ol e o e he ma ix
A. Conside ing wk−1= [iw
k−1, jw
k−1]and wk= [iw
k, jw
k], hen (iw
k−iw
k−1>0∧jw
k−
jw
k−1≥0) ∨(iw
k−iw
k−1≥0∧jw
k−jw
k−1>0).
The dis ance DTW is gi en by he squa e oo o he elemen BNN , as de ined in
40
(3.50). No e ha he DTW dis ance does no sa is y he iangle inequali y: (DTW(a,b)+
DTW(b,c)is no always ≥DTW(a,c)). The e o e, DTW is no a me ic.
DTW(x,b
x) = pBNN (3.50)
No e: i is possible o ind in li e a u e ambigui ies on he meaning o he in o ma ion
s o ed wi hin he wa ping pa h elemen s wk. Fo example, in he s udy [10], he elemen s
wka e i s ly de ined o s o e he posi ions wk= [iw
k, jw
k](as he e de ined), and la e hey
a e di ec ly applied o calcula e he DTW dis ance, ins ead o apply he cumula i e cos
s o ed in he ma ix Bco esponding o he posi ion s o ed in wK.
As e e enced in [10], when he wo da a se ies unde assessmen ha e di e en num-
be o elemen s, he alue ob ained in (3.50) should be di ided by K( he numbe o
elemen s ound in he wa ping pa h), o compensa e he leng hs di e gence. In his wo k,
he da a se ies a e conside ed o be o equal leng hs. Fo ha eason, he DTW is de ined
wi hou conside ing his compensa ion, as sugges ed in he s udy [43] (pp.5).
Lowes alues o DTW e e o a be e i be ween he wo da a se s. DTW is a scale
dependen dis ance, which does no enable he compa ison o his measu e o e di e en
case s udies. Mo eo e , DTW may e u n e y high alues, as hey e e o he sho es
cumula ed dis ance om A11 o ANN , he alues e u ned by DTW inc ease wi h he
numbe o elemen s wi hin he da a se s o be analyzed.
When DTW is calcula ed wi h he cos unc ion d(xi,bxj)=(xi−bxj)2 o each el-
emen o ma ix A, he alue ob ained is compa able wi h he euclidean dis ance d =2
de ined in sec ion 3.6.1. A di ec compa ison wi h he Manha an Dis ance d =1 would
ne e be possible e en i he cos unc ion d(xi,bxj) = |(xi−bxj)|was adop ed o each el-
emen o ma ix A, as he DTW includes he squa e oo on he cumula i e cos achie ed
in he elemen BNN , as i was p esen ed in equa ion (3.50).
The e o e, he DTW can only be compa ed wi h he euclidean dis ance dis ance d =2
when: he wo da a se s ha e he same leng h, he calculus is made exac ly as de ined
in equa ion (3.50) (i.e. wi hou di iding √BNN by K), and he cos unc ion adop ed
o cons uc each elemen o ma ix Ais he squa ed e o d(xi,bxj) = (xi−bxj)2. The
in e p e a ion o his compa ison is explained by [10]: (i) when DTW =d =2 i means
ha he sho es pa h ound in ma ix B ela es o he diagonal elemen s, and so he e
is no e idence ha he es ima e da a se is lagged om he o iginal da a se ; (ii) when
DTW < d2, i means ha he wa ping pa h Whas elemen s ou side he diagonal o
ma ix B, and he e is e idence o pa e n dissimila i y be ween he wo da a se s. In he
41
second case, he isual in e p e a ion o ma ix Bmay help he iden i ica ion o calib a ion
imp o emen s in he model.
3.7 Combined measu es
Le s conside he ollowing h ee quan i ies in (3.51).
λXX =1
N
N
X
i=1
x2
i
λb
Xb
X=1
N
N
X
i=1 bx2
i
λXb
X=1
N
N
X
i=1 bxi·xi
(3.51)
The accep able alues o he me ics p esen ed in his sec ion a e no es ablished
in li e a u e ye . In [45] he ollowing e e ence alues a e indica ed: 0in any o he
componen s is he pe ec i be ween wo da a se ies, alues below 20% a e eally good,
alues be ween 20% and 30% a e conside ed ai and alues abo e he 30% a e coside ed
poo .
The measu es u he de ailed a e no mally used o analyze wa e o m se ies, om
whe e he componen Phase (some imes called ime o a i al) is so impo an [45, 43].
The meaning o he Phase componen is ques ionable o non wa e o m se ies.
3.7.1 Sp ague & Gea e o - CS&G
Sp ague & Gea measu e conside s e o s due o magni ude and phase di e ences. The
e o in magni ude MS&Gand he e o in phase PS&Ga e i s ly calcula ed. These wo
componen s a e hen used o calcula e CS&Gusing he squa e oo o he sum o he squa e
o he wo componen s. All o mula ions a e de ailed in (3.52).
42

MS&G=qλXX
λb
X
b
X−1
PS&G=1
πcos−1λX
b
X
√λXX ·λb
X
b
X
CS&G=pM2
S&G+P2
S&G
(3.52)
The applica ion o his measu e can be ound in he s udies [45, 43, 44]. The alues
o CS&Gmay a y be ween he ange [0,1], and e u n asymme ic alues (meaning ha
CS&G(X, b
X)6=CS&G(b
X, X)). Lowe alues o CS&Gindica e a be e i o he es ima ed
da a se .
3.7.2 Russel’s e o - CR
Russel’s e o CRhas wo main componen s o e o : magni ude and phase. The phase
componen PRis calcula ed exac ly as PS&G. The magni ude componen is MRis calcu-
la ed di e en ly, as p esen ed in (3.53). Again, he combina ion o he wo componen s is
made using he squa e oo o he sum o he squa e o he wo componen s.
MR=sign(λXX −λb
Xb
X)·log10 1 + 
λXX −λb
X
b
X
√λXX ·λb
X
b
X
PR=1
πcos−1λX
b
X
√λXX ·λb
X
b
X
CR=pM2
R+P2
R
(3.53)
The Russel’s e o is de ailed in he s udies [43, 44]. The ad an age o CRo e CS&G
e o is he o e coming o he asymme y d awback.
3.7.3 No malized In eg al Squa e e o - CNISE
No malized In eg al Squa e e o CNISE conside s h ee main aspec s: magni ude,phase
and shape. The c oss-co ela ion ρ(n), which was de ined in sec ion 3.4.3, is used o
calcula e he es ima ed lag nlag. Once nlag is es ima ed, a shi o nlag is induced o one
43
o he da a se s, and wi h ha change he quan i y λXb
X(nlag)is calcula ed as de ined in
(3.51). Nex he e o measu es o magni ude MNISE, phase PNISE and shape SNISE
a e calcula ed. The combined e o CNISE is hen calcula ed as he sum o he h ee
componen s, as i is de ailed in exp essions (3.54).
MNISE =ρ(nlag)−2λX
b
X(nlag)
λXX +λb
X
b
X
PNISE =2λX
b
X(nlag)−2λX
b
X
λXX +λb
X
b
X
SNISE = 1 −ρ(nlag)
CNISE =MNISE +PNISE +SNISE = 1 −2λX
b
X
λXX +λb
X
b
X
(3.54)
The applica ion o NISE e o can be ound in he s udies [43, 44]. A be e es i-
ma ion o he model is obse ed o lowe alues o CNISE. A sepa a ed analysis o he
componen s is use ul o in es iga e he e o a i ing om each componen , which may
endow he esea che wi h mo e in o ma ion on how o imp o e he model calib a ion.
44
Chap e 4
Measu ing he Wa ping Pa h Dis o ion
This chap e includes he p oposal o wo new pe o mance c i e ia: Wa ping Pa h Dis o ion-
WPD and he Pe cen age Wa ping Pa h Dis o ion- PWPD. The heo e ical de ini ion
o WPD and PWPD is i s ly p o ided. PWPD is de ined as a pe cen age o WPD.
The chap e p oceeds wi h he applica ion o bo h measu es o a p ac ical example.
4.1 Wa ping Pa h Dis o ion- WPD
The Wa ping Pa h Dis o ion WPD is a new pe o mance c i e ion p oposed in his he-
sis, based on he Dynamic Time Wa ping DTW algo i hm. WPD e u ns he a e age
dis ance be ween he posi ions s o ed wi hin he wa ping pa h Wwi h he co esponding
nea es posi ions o he diagonal o ma ix B(3.49).
When he wo da a se s ollow simila pa e ns o e ime, he posi ions s o ed wi hin
he wa ping pa h Wa e expec ed o coincide wi h he posi ion o he diagonal o ma ix
B. In his case, he alue e u ned by WPD is ze o ( he dis ance be ween he posi ions
s o ed in Wand he posi ions co esponding o he diagonal o ma ix Bis null).
When he wo da a se s ha e di e en pa e ns, WPD is expec ed o e u n a le el
on he dis o ion be ween he wo pa e ns. The e o e, WPD may be in e p e ed as he
dissimila i y be ween he wo pa e ns unde analysis.
Nex , he ma hema ical o mula ion o WPDis p esen ed. The nea es posi ion o he
diagonal o ma ix B o an a bi a y posi ion s o ed wi hin he wa ping pa h wk= [iw
k, jw
k]
is de ined as pk= [ip
k, ip
k]. The dis ance be ween an a bi a y wk o he co esponding pk
may be isualized by acing a line amid hese wo posi ions, pe pendicula ly o di ec ion
o ma ix Bdiagonal, see Figu e 4.3 isualiza ion 2.
The calculus o ip
kis p esen ed in (4.1). No e ha all elemen s in diagonal a e sym-
45
me ic, hus he elemen s in eg a ing pka e bo h designa ed as ip
k. A diagonal posi ion
pkmay be in eg a ed wi h alues mul iples o 0.5. The se o nea es diagonal posi ions
ela ing o an a bi a y wa ping pa h Wis de ined as P=hp1, p2,··· , pKi.
ip
k=
iw
k−jw
k
2+min(iw
k, jw
k)(4.1)
The dis ance be ween an a bi a y wk o he co esponding nea es posi ion on di-
agonal o ma ix B,pk, is hen calcula ed using he Manha an Dis ance d =1 (see sec-
ion 3.6.1) as desc ibed in (4.2).
d =1(wk, pk) = |iw
k−ip
k|+|jw
k−ip
k|(4.2)
The Wa ping Pa h Dis o ion WPDis es ima ed as he a e age o he dis ances d =1(wk, pk),
conside ing k= (2, ..., K −1), as desc ibed in equa ion (4.3). The ex emi y ele-
men s w1and wKa e excluded. These wo elemen s include always he same posi ions
(w1= [1,1], wK= [N, N]) due o he bounda y condi ions, as speci ied in sec ion 3.6.3.
The e o e, hey do no e lec any in o ma ion on he dissimila i y be ween he wo da a
se s unde assessmen .
WPD =1
K−2
K−1
X
k=2
d =1(wk, pk)(4.3)
WPD is a scale independen measu e. The e o e, WPD may be compa ed o e
di e en case s udies p o ided he leng h o he ec o s is he same. This es ic ion
occu s as WPDdepends on he size o he da a se s analyzed N.
The alues e u ned by WPD a e included wi hin he ange WPD ∈[0,(N−1)2
K−2[. A
lowe alue o WPD is associa ed wi h a be e i o he es ima ed model. The pe ec
i would e u n a WPD alue o ze o, and a comple ely dis o ed pai o ec o s would
e u n he wo s alue o WPD: asymp o ically (N−1)2
K−2. An example o “ he wo s case”
is p o ided in Figu e 4.1.
4.2 Pe cen age Wa ping Pa h Dis o ion - PWPD
The WPD may be used o calcula e he co esponden pe cen age measu e, he Pe cen -
age Wa ping Pa h Dis o ion PWPD. This measu e indica es he pe cen age dis o ion
o a wa ping pa h conce ning he diagonal o he espec i e ma ix B. This measu e is
calcula ed wi h he di ision o WPD by he maximum possible alue ha WPD may
46
Figu e 5.1: Diag am ep esen ing he algo i hm ollowed by PoFi. This diag am explo es
he ansi ion o one i m du ing one i e a ion (be ween and +1). S a e o he i m:
X, sec o o he i m: S, age: A, geog aphical egion: N, dimension: D, a ia ion o
dimension: ∆d.
Alen ejo and Alga e.
The hi d dimension e e s o he ime e olu ion in yea s. The ou h dimension includes in-
o ma ion on he ollowing ca ego ies: (i) s a e o i ms - ali e o dead, (ii) age in yea s, (iii) sec o
including a) ag icul u e and ishe ies, b) indus y, c) cons uc ion, d) se ices, (i ) dimension ha
encompasses ou le els o i ms’ size.
The model e ol es acco ding o mic o ules es ablished o each cell on he ou dimensional
a ay, ollowing an annual pe iodici y. The Conway’s Game o li e [15] was used as a s a ing
poin o PoFi, om whe e mo e a iables and ules we e conside ed. The ules speci ied o each
cell wi hin an a bi a y i e a ion (i.e. be ween ime and ime + 1) a e desc ibed in Figu e 5.1,
which is o ganized in ou main phases.
The i s phase iden i ies he cu en s a e o a i m. When he i m is ali e i s s a e is 1
(X = 1), and when he i m is dead i s s a e is 0(X = 0). Depending on he cu en s a e, he
model ollows di e en pa hs.
5.1.1 Algo i hm used when a i m is dead
Conside ing ha he cu en s a e is dead, he second phase will decide he po en ial success o
he i m o be bo n o o con inue dead, conside ing he numbe o li ing i ms close o he i m in
s udy. The close i ms conside ed a e hose wi hin a adius o one cell. I he i m’s s a e is kep in
0, null alues a e kep o all cha ac e is ics. In case he i m’s s a e is changed o 1, he algo i hm
e ol es o he nex phase.
53

The hi d phase selec s he ac i i y sec o o he bo n i m. The model includes di e en
p obabili ies o he selec ion o each sec o o ac i i y conside ing he NUTS o he i m. These
p obabili ies a e he a e age p opo ion o obse ed i ms o each sec o wi hin each zone consid-
e ing he agg ega ion o all yea s. In his phase, a uni o m andom numbe is gene a ed, and he
selec ion o he sec o is ca ied ou wi h he compa ison o he andom numbe ob ained and he
p obabili y alues ound.
The ou h phase conside s wo ea u es o de ine he p obabili y associa ed o he dimension
de ini ion: NUTS and he sec o o ac i i y (i.e. i conside s he a e age p opo ion o obse ed
i ms o each dimension wi hin each sec o wi hin each zone, in eal da a). This phase also uses
uni o m andom numbe s o pe o m he decision o he dimension o he new i m.
A his s age, all cha ac e is ics o a new i m a e de ined. The espec i e in o ma ion is sa ed
and used on a new i e a ion.
5.1.2 Algo i hm used when a i m is ali e
When he ini ial s a e o a i m is 1, meaning ha he i m is ali e, he second phase decides on he
su i al o his i m. The su i al decision is made conside ing he numbe o i ms li ing wi hin
he adius o one cell o he cell unde assessmen . I he i m dies, all cha ac e is ics a e se o
ze o. I he i m su i es, he nex phase is pu sued.
The hi d phase de ines in o ma ion on wo cha ac e is ics: sec o and age. The sec o o
a su i ing i m is no changed. The gene ali y o he de ined classes is so pe asi e ha i is
conside ed e y unlikely ha a i m pe o ms his change. The a iable age is inc emen ed in one
uni e e y ime a i m su i es (no e ha when a i m is bo n, i s age is s ill ze o, and ha a i m
only ha e he i s yea comple ed when i achie es he i s yea o su i al).
The ou h phase de ines he new alue o he i m’s dimension. This is o mula ed conside ing
he assump ion ha a i m will ha e a maximum absolu e size a ia ion o one pe i e a ion (i.e. in
each i e a ion he dimension can be inc emen ed by −1,0o 1). This a ia ion is de e mined wi h
base on he p opo ion o obse ed i ms changing i s dimension as a p obabili y.
5.2 Resul s
The model p o ided esul s conce ning each o he cha ac e is ics simula ed. Nex , he esul s
ob ained conce ning he numbe o li ing i ms in each zone a e analyzed. Figu e 5.2 p esen s he
numbe o li ing i ms simula ed alongside wi h he espec i e his o ic eco ds.
The nex sec ions include he alida ion assessmen o each zone sepa a ely. The Pe o mance
c i e ia u he analyzed we e calcula ed conside ing alues in 103uni s, using he R so wa e o
s a is ics [40]. The Pa ame ic es s should no be calcula ed o his example, as he samples ha e
a size o N= 25, and he condi ion o no mali y is no ensu ed. Ne e heless, and knowing ha
54
E o Based Measu es
ME −9.199
MAE 26.863
MSE 911.899
RMSE 30.198
U10.174
U20.356
MP E −0.283
MAP E 0.433
MASE 6.568
In o ma ion Theo y Measu es
HSHS(X) 3.138
HS(X, b
X) 3.865
I(X, b
X) 2.478
NMI(X, b
X) 0.789
Id(X, b
X) 0.210
Dis ance Based measu es
Euclidean dis ance 150.988
Manha an dis ance 671.573
dST S 64.476
DTW 21.045
Dis o ion Pa h Measu es
DWP 5.432
PDWP 0.415
Pa ame ic Tes s
Pea son 0.396
p- alue 0.049
R20.157
ρ(n)ρ(nlag) 0.415
nlag 2
Nonpa ame ic Tes s
Sign es TS 14
CRα=0.05 [0,7.6[∪]17.4,25]
Spea man S0.397
CRα=0.05 ]0.337,+∞[
Kendall τ0.29
p- alue 0.049
Wilcoxon TS 105
p- alue 0.127
Combined measu es
S&G MS&G−0.048
PS&G0.318
CS&G0.322
Russel MR−0.041
PR0.318
CR0.320
NISE MNISE −0.525
PNISE 0.001
SNISE 0.585
CNISE 0.060
Table 5.1: Pe o mance C i e ia summa y o he numbe o i ms in No e.
his es ic ion is elaxed, he alues on hese c i e ia a e p esen ed and discussed.
Only wo no malized in o ma ion measu es a e u he p esen ed, as hey all lead o simila
conclusions and he analysis o one NMI and one Idis conside ed su icien o demons a e
how hese measu es beha e. Following he same easoning on he calculus o Pe cen age E o
PE (always in o de o he obse ed alues), he NMI u he p esen ed e e o he di ision o
he mu ual in o ma ion MI(X, b
X)by he en opy o he obse ed da a se H(X). The second
no malized in o ma ion measu e analyzed is he co esponding in o ma ion dis ance Id= 1 −
NMI.
5.2.1 Numbe o i ms in No e
The pe o mance c i e ia ob ained o he numbe o i ms in No e a e p esen ed in Table 5.2.1,
and he Wa ping pa h ob ained is illus a ed in Figu e 5.2.1.
The analysis o he numbe o i ms in No e is made wi h a single es ima ed da a se . Con-
sequen ly, isola ed alues o ME,MAE,MSE and RMSE do no endow a p ope conclusion
on he goodness o i o his ea u e. These measu es a e no compa able o e he emainde ex-
55
Figu e 5.2: Simula ed and obse ed da a om he Po uguese case s udy.
56
Figu e 5.3: Wa ping pa h isualiza ion o he numbe o i ms in No e.
pe imen s p esen ed in his sec ion, hey would only be compa able o di e en o ecas s o he
same ea u e. The same easoning is applicable o he emainde expe imen s on Cen o,Lisboa,
Alen ejo and Alga e.
The alue ob ained wi h he Theil’s coe icien o o ecas accu acy U1 e u ned he wo s
alue o he No e o ecas o e he i e expe imen s conduc ed. The alue o 0.174 canno be
conside ed as a good es ima e, as i is no close he e e ence alue o 0. The coe icien U2
e u ned a alue lowe han 1, which indica es ha he es ima ion ob ained has lowe s anda d
e o as he nai e no-change ex apola ion. I he alue ob ained wi h U2was la ge han 1, he
model should be ejec ed as in ha case he s anda d e o ob ained wi h he o ecas would be
wo s han he simples no-change ex apola ion.
The nega i e sign ob ained wi h ME and MP E indica es ha he es ima ion o No e ea u e
e u ns in a e age highe alues han he ones obse ed. This conclusion may be e i ied wi h he
isualiza ion o Figu e 5.2, and is impo an o u u e calib a ion o he model. These measu es,
alongside wi h MS&Gand MRallow o d aw simila conclusions.
The pe cen age-based e o measu es a e mo e in ui i e as hey allow he goodness o i as-
sessmen o a single es ima ion and hey a e compa able o e expe imen s on di e en ea u es.
The mean pe cen age e o o −28.3% indica es ha b
X alues a e in a e age 28.3% abo e X(as
i is obse ed ill he 15 h yea conside ed). The mean absolu e pe cen age e o indica es ha
in a e age he es ima ions b
Xa e dis anced om he his o ic eco d Xin 43.3%, which is a high
alue showing he bad i ness ob ained o his ea u e.
The No e case has he wo s MASE alue o e he i e expe imen s unde assessmen in
his sec ion. As he ange o MASE is [0,+∞[, he alue 6.568 he e ob ained pu po s mo e
57
in o ma ion when compa ed wi h o he MASE alues. As MASE is scale- ee, he alues o his
measu e a e compa able o e esul s on di e en ea u es.
The en opy alue ob ained o he obse ed da a se is simila o he join en opy o e he
wo da a se s. Howe e , he mu ual in o ma ion shows ha he wo se ies a e no simila , since he
mu ual in o ma ion is conside ably lowe han he join en opy. The no malized mu ual in o ma-
ion allows a mo e in ui i e assessmen on he quan i y o in o ma ion sha ed by he wo da a se s,
ha can be in e p e ed as a pe cen age o 78.9%. This means ha he wo da a se sha e 78.9% o
in o ma ion, and his alue is oo low o be conside ed a good esul ( he pe ec i would e u n
he alue 100.0%). As i was expec ed, he no malized in o ma ion dis ance, ap oxima ely 21.0%,
is oo high o be conside ed a good es ima ion ( he pe ec i would e u n he alue o 0.0%.
These measu es indica e ha he simula ed da a se is a bad ep esen a ion o eali y conce ning
he quan i y o in o ma ion.
The pa ame ic es on he Pea son’s co ela ion coe icien indica es a posi i e co ela ion
o = 0.396, wi h a s a is ical signi icance o 0.049. The coe icien o de e mina ion is low,
meaning ha he simula ed da a explains only a small po ion o he a iabili y obse ed in eali y.
The c oss co ela ion shows ha he simula ed da a would be e i he his o ic eco ds wi h a lag
o wo empo al uni s.
The Sign es does no ejec he null hypo hesis o equali y o Xand b
Xmedians, hus, con-
side ing a s a is ical signi icance o α= 0.05, hese medians a e simila .
The Spea man co ela ion e u ns a posi i e and s a is ically signi ican co ela ion, consid-
e ing α= 0.05. This e alua ion may concluded as S∈CRα=0.05, hus he null hypo hesis is
ejec ed. No e ha o an α= 0.01, his co ela ion is no longe signi ican . Fo his case, he
Spea man’s co ela ion es should be adop ed ins ead o Pea son, as he size o he da a se s is
small.
The Kendall’s τob ained indica es a mode a e and posi i e co ela ion, wi h a p- alue o
0.049.
The Wilcoxon es o means does no allow he ejec ion o he null hypo hesis E(X) =
E(b
X), as he p- alue ob ained is highe han α= 0.05. The e o e, he e is no s a is ical e idence
o assume ha he means ollowed by Xand b
Xa e di e en , and he null hypo hesis is kep .
The alues e u ned by he dis ances d =2 and d =1 seem o be high, bu wi hou an equi alen
compa ison, he in o ma ion hese c i e ia ha e is limi ed. High alues o hese dis ances indica e
ha he wo da a se s a e in a e age sepa a ed om each o he by high ampli udes. In ac , he
isualiza ion o Figu e 5.2 o he No e case indica es his same conclusion.
The dis ance dST S p o ides a di e en assessmen . I indica es a measu emen on he a e age
ampli ude di e gence o e consecu i e empo al momen s, be ween he wo da a se s. In his case,
he squa e oo o he sum o he squa ed di e ences o he slopes be ween empo al momen s
e u ned he alue dST S = 64,476. The complex in e p e a ion o his measu e is one o i s
d awbacks.
58

The alues ob ained wi h DT W does no include much in o ma ion wi hou a compa ison o
he same measu e wi h a di e en o ecas o he same case s udy. One o he main d awbacks o
DT W is i s dependency on he obse a ions scale and on he leng h o he da a se s. No e ha he
cos unc ion used o cons uc he ma ix Awas no he squa ed e o , and he e o e he DT W
alues he e p esen ed a e no compa able wi h he euclidean dis ance d =2.
Conce ning he combined measu es, he phase componen s PS&Gand PRa e sui able o assess
lag on se ies wi h sinusoidal beha io , o wa e o m, as explained in [45]. This is no he case
o he expe imen s conduc ed in his sec ion, and so, hese phase indica o s a e meaningless.
Conce ning he phase componen o NISE, i ela es wi h he esul s ob ained o he numbe o
pe iods o lag wi h ρ(n). Fo he expe imen No e, as nlag was se o 2, he PNISE e u ns a alue
sligh ly di e en om ze o. Fo he emainde expe imen s, whe e nlag = 0, he PNISE = 0 as
well.
The magni ude componen s MS&Gand MRwe e analyzed alongside wi h he sign e u ned
by ME and MPE. These magni ude componen s e u n alues wi hin he ange ]−1,1[, and a e
equen ly in e p e ed as a pe cen age o magni ude disc epancy on he se ies unde analysis. Con-
ce ning he MNISE, i e u ns alues wi hin he ange [−1,0], om whe e i p o ides in o ma ion
ha should be in e p e ed as he absolu e magni ude pe cen age de ia ion. Fo he No e case, he
MNISE e u ns a pe cen age de ia ion o 25,5%, ha is a eally bad es ima ion, and mo e se e e
han he MAPE es ima ion.
The shape componen SNISE e u ns alues wi hin [0,1], being 0 he bes possible esul indi-
ca ing a good pa e n simila i y and 1 he wo s possible esul indica ing a bad simila i y. Fo he
No e es ima ion SNISE = 0.585, ha is a bad pa e n simila i y. This measu e may be compa ed
wi h PDW P , as hey assess he same cha ac e is ic al hough using di e en me hodologies. No e
ha PDW P e u ns alues wi hin he same ange and wi h simila in e p e a ion as SNISE. The
esul ob ained wi h P DW P is howe e less se e e han SNISE, and hese wo measu es indica e
a conside ably bad pa e n simila i y.
The alue e u ned by WPD was 5,432 ou o a maximum o 13.091. This esul indica es
a bad pa e n simila i y be ween he wo da a se s. This dissimila i y is also obse able in Figu e
5.2.1, showing ha he Wa ping Pa h is highly di e gen om he e e ence posi ion om diagonal.
The alue P DW P quan i ies his di e gence in 41.5%.
5.2.2 Numbe o i ms in Cen o
The pe o mance indexes ob ained o he numbe o i ms in he zone Cen o a e p esen ed in
Table 5.2.2, and he wa ping pa h ob ained o he wo ime se ies is shown in Figu e 5.2.2.
The measu es ME, MAE, MSE and RMSE a e lowe o he es ima ed numbe o i ms in
Cen o han in No e. Al hough his compa ison is emp ing, i should no be made, as scale-
based e o s a e no sui able o assess es ima es o e di e en ea u es. The e o -based measu es
59
E o Based Measu es
ME −5.643
MAE 14.749
MSE 289.460
RMSE 17.014
U10.156
U20.319
MP E −0.289
MAP E 0.411
MASE 5.757
In o ma ion Theo y Measu es
HSHS(X) 3.120
HS(X, b
X) 3.852
I(X, b
X) 2.465
NMI(X, b
X) 0.790
Id(X, b
X) 0.210
Dis ance Based measu es
Euclidean dis ance 85.068
Manha an dis ance 368.722
dST S 28.981
DTW 14.101
Dis o ion Pa h Measu es
DWP 5.163
PDWP 0.385
Pa ame ic Tes s
Pea son 0.688
p- alue 0.000
R20.473
ρ(n)ρ(nlag) 0.688
nlag 0
Nonpa ame ic Tes s
Sign es TS 15
CRα=0.05 [0,7.6[∪]17.4,25]
Spea man S0.791
CRα=0.05 ]0.337,+∞[
Kendall τ0.628
p- alue 0.000
Wilcoxon TS 99
p- alue 0.090
Combined measu es
S&G MS&G−0.039
PS&G0.318
CS&G0.321
Russel MR−0.034
PR0.318
CR0.318
NISE MNISE −0.264
PNISE 0.000
SNISE 0.312
CNISE 0.049
Table 5.2: Pe o mance C i e ia summa y o he numbe o i ms in Cen o.
Figu e 5.4: Wa ping pa h isualiza ion o he numbe o i ms in Cen o.
60
sui able o add ess his compa ison a e MPE, MAPE and MASE.
The alue ob ained wi h U1is sligh ly be e han he co esponden alue ob ained o No e,
bu i canno be conside ed a good es ima e since i is no close o he e e ence 0. The alue
ob ained wi h U2is lowe han he e e ence 1, indica ing ha his o ecas has lowe s anda d
e o han he simple nai e no-change ex apola ion.
The nega i e sign ob ained wi h ME, MPE, MS&Gand MRindica e ha his es ima ion e u ns
in a e age highe alues han he espec i e his o ic eco ds. Figu e 5.2.2 shows ha he numbe
o es ima ed i ms is always highe han he his o ic eco d ill he 16 h yea simula ed. Fu u e
calib a ion o his model should be conduc ed o imp o e his misalignmen .
The MPE o Cen o shows a di e gence be ween es ima ed and obse ed da a sligh ly highe
han he one obse ed in No e case, wi h a esul o −28.9%, indica ing ha b
Xis in a e age 28.9%
abo e X. The e o e, wi h he MPE c i e ion, Cen o would be a wo s es ima ion in compa ison
wi h No e.
MAPE shows he con a y conclusion. The MAPE alue o Cen o is be e han he MAPE
alue o No e. This disag eemen is pe ec ly unde s andable as MPE is a measu e ha o se s
posi i e and nega i e alues o e o . A be e compa ison o e di e en es ima ions is achie ed
wi h he MAPE measu e, which does no o se s posi i e o nega i e alues.
The conclusion aken wi h MASE is simila o he one ob ained wi h MAPE, he es ima ion
o Cen o a e be e han he ones o No e. This conclusion is aken as he MASE alue o
Cen o is lowe han he one obse ed o No e. MAPE has he ad an age o being de ined wi hin
he closed ange [0,1], which is a mo e in ui i e assessmen .
The esul s ob ained o Cen o conce ning he in o ma ion heo y measu es a e simila o
he ones d awn o he No e expe imen . The no malized mu ual in o ma ion is pe haps he
mos in ui i e in o ma ion measu e o e he ou analyzed, which indica es ha only 79.0% o he
in o ma ion con ained in Xis con ained in b
X.
The Pea son’s co ela ion es s indica es ha bo h es ima ed and his o ic da a se s o Cen o
a e posi i ely co ela ed, wi h a high s a is ical signi icance (p- alue= 0.000). The coe icien o
de e mina ion indica es ha he a iabili y explained by he model is 0.473 ou o 1. The c oss
co ela ion indica es ha he simula ed da a would no p o ide a be e i wi h any lag on empo al
uni s.
The TS ob ained wi h he Sign es is no included wi hin he c i ical egion de ined, om
whe e he null hypo hesis ha es s he equali y o Xand b
Xmedians is no ejec ed, meaning ha
he wo medians a e conside ed simila o a s a is ical signi icance o α= 0.05.
The Spea man’s co ela ion e u ns a posi i e and s a is ically signi ican co ela ion, o α=
0.05. This conclusion is aken as S∈CRα=0.05, and consequen ly he null hypo hesis es ing
no co ela ion is ejec ed. Fo an α= 0.01 his co ela ion keeps o be signi ican (CRα=0.01 =
]0.466,+∞[).
The Kendall’s τob ained indica e a s ong and posi i e co ela ion, wi h a p- alue o 0.000,
61
Figu e 5.5: Wa ping pa h isualiza ion o he numbe o i ms in Lisboa.
meaning ha he wo da a se ies a e posi i ely co ela ed o any signi icance coe icien αcon-
side ed.
The Wilcoxon es o means do no ejec he null hypo hesis E(X) = E(b
X), as he p- alue
ob ained is highe han α= 0.05. The e o e, he null hypo hesis o equali y o he means ollowed
by he wo da a se ies is kep .
The dis ances d =2 and d =1 e u ned lowe alues o Cen o in compa ison wi h No e. This
means ha he es ima ed and obse ed da a se s a e close o he Cen o expe imen han o he
No e expe imen . This esul sugges s he Cen o es ima ions o be be e han he No e ones.
The dis ance dST S indica es ha he slope di e gence be ween each pai o consecu i e em-
po al momen s in Cen o is lowe han in No e. The e o e, his c i e ion sugges s Cen o o be a
be e es ima ion han No e. The compa ison o he DT W alues indica e simila conclusions.
As o he Cen o expe imen , nlag = 0, he phase componen o NISE is 0as well. The
MNISE indica es ha he wo da a se ies a e de ia ed om one ano he in 26,4%, which indica es
high di e ences conce ning his cha ac e is ic.
The shape componen SNISE o his case was 0.312, indica ing ha he pa e ns a e no
simila , al hough i is a be e esul han he one ob ained o No e. A simila conclusion is
ob ained wi h PDW P , wi h a alue o 0.385.
5.2.3 Numbe o i ms in Lisboa
The pe o mance indexes ob ained o he numbe o i ms in Lisboa a e p esen ed in Table 5.2.3.
The Wa ping pa h ob ained o his expe imen is illus a ed in Figu e 5.2.3.
62
bo h cases, he null hypo hesis S= 0 is always ejec ed.
The Kendall’s τob ained sugges s ha he wo da a se s a e posi i ely co ela ed, wi h τ=
0.774. This co ela ion is s a is ically signi ican o any αconside ed.
The Wilcoxon es o means e u ned a p- alue o 0.200, ha is highe han α= 0.05 and
consequen ly he null hypo hesis o E(X) = E(b
X)is no ejec ed. Thus he assump ion o
equali y o he means ollowed by he wo da a se s unde analysis is kep .
Conce ning he dis ance measu es, he Alga e e u ned he lowes alues o e he i e expe -
imen s. This means ha conside ing he dis ance measu es, Alga e would be he bes o ecas
among he i e analysed.
Fo he Alga e expe imen , he PNISE alue is ze o as he nlag is ze o as well. The as-
sessmen o magni ude de ia ion wi h MNISE e u ned he bes esul o e he i e cases, wi h
MNISE = 7.2%. This esul is in acco dance wi h he MASE esul .
The measu es o shape componen SNISE and PDW P e u ned o Alga e he bes esul s
o e all expe imen s, sugges ing ha his was he bes es ima ion p o ided by he model conce n-
ing he pa e n adjus men .
69

Chap e 6
Conclusions
A compu a ional simula ion model is designed o alida e o ejec he hypo hesis o mula ed in
he concep ual model. A simula ion model can be accep ed as alid o speci ic condi ions only,
which means ha unde he assump ions speci ied, he model ep oduces well he ask o which i
was designed. To p o e ha a model ep oduces well i s ask, he s anda d p ocedu es o con i m
scien i ic heo ies should be ollowed [41]. These p ocedu es include he use o da a om he
eali y unde s udy o con on he esul s ob ained wi h he model. This con on happens wi hin
he phases o calib a ion and alida ion. The ou comes om alida ion de e mine wea he he
hypo hesis o mula ed in he concep ual model should be accep ed o ejec ed.
Each phase in eg a ing he p ocess desc ibed o cons uc ing a simula ion model, encompasses
many complex aspec s. This wo k ied o in es iga e one o hese aspec s: he adequa e pe o -
mance c i e ia o use unde a alida ion p ocess. The pe o mance c i e ia conside ed in his wo k
a e hose adequa e o assess he goodness o i be ween pai ed da a samples, always conside ing
he pai ed da a sample o be he obse ed and simula ed da a se s.
The main con ibu ions o his hesis a e (i) he bibliog aphic o e iew on pe o mance c i e ia,
(ii) he p oposal o wo new pe o mance c i e ia, he Wa ping Pa h Dis o ion WPD and he
Pe cen age Wa ping Pa h Dis o ion PWPD, and (iii) he compa a i e analysis o he c i e ia
e iewed unde p ac ical expe imen s.
Nex he main ad an ages and d awbacks iden i ied in his wo k o he c i e ia e iewed a e
de ailed.
The measu es Mean E o ME, Mean Absolu e E o MAE, Mean Squa e E o MSE and
Roo Mean Squa e E o RMSE a e use ul when applied o assess di e en uns on he same
ea u e o he same case s udy. Only in ha si ua ion can hese measu es be compa ed.
The Theil’s coe icien o o ecas accu acy U1 e u ns alues included wi hin he ange [0,1]
and indica es good es ima es o alues close o ze o. This is a scale independen measu e ha
can be di ec ly compa ed o e di e en case s udies. Howe e , U1does no ha e an accu acy
on ie o p ecisely de ine wea he a alida ion esul should be ejec ed o accep ed. The e o e,
70
he analysis made wi h his measu e a e always subjec i e when assessing a single es ima ion. U1
is mo e ad an ageous when applied o compa e he accu acy o e di e en es ima es.
The coe icien o o ecas quali y U2 e u ns alues wi hin he ange [0,+∞[, being he alue
0indica i e o a pe ec o ecas . Values lowe han 1indica es ha he es ima ion ob ained has
lowe s anda d e o as he nai e no-change ex apola ion. I a alue la ge o equal o 1is ob ained
o he U2c i e ion, he model should be ejec ed as in ha case he s anda d e o ob ained wi h
he o ecas would be wo s han he simples no-change ex apola ion.
The sign ob ained wi h he measu es ME, Mean Pe cen age E o MP E, Sp ague and Gea
magni ude e o MS&Gand Russel’s magni ude e o MRis nega i e when he es ima ed da a
e u ns in a e age highe alues han he ones obse ed, and posi i e o he wise. The iden i ica ion
o magni ude di e gences is a good s a ing poin o imp o e u u e calib a ion on models. The
calib a ion o he model conce ning he magni ude should be made oge he wi h he isualiza ion
o he co esponding plo s. The isual obse a ion is impo an as, o example, in he No e
expe imen , nega i e alues we e ob ained o hese measu es, and he calib a ion needed in his
case is mo e ela ed wi h an imp o emen on he esul an slan han he blind dec ease o he
es ima ed alues.
The c i e ion Mean Absolu e Scaled E o MASE e u ns alues wi hin he ange [0,+∞[.
Since his measu e is no no malized, i s applicabili y is mo e use ul o compa e esul s o e
di e en es ima ions han o assess he goodness o i ness o an isola ed es ima ion.
MASE is based on he absolu e e o s incu ed by he es ima ion, while Mean Absolu e Pe -
cen age E o MAPE is based on he pe cen age e o s ob ained. Fo his eason, he compa ison
o he quali y o adjus men be ween Alen ejo and Alga e e u ned di e en conclusions wi h
MASE and MAP E.
The in o ma ion heo y measu es e u ned he se e es alida ion esul s, indica ing ha all
he i e es ima ed da a se s a e e y poo . The No malized Mu ual In o ma ion NMI and No -
malized in o ma ion dis ance Id showed o be aluable pe o mance c i e ia due o i s easiness
o in e p e a ion: bo h e u n alues wi hin he ange [0,1] and can be in e p e ed as in o ma ion
pe cen ages.
The pa ame ic es o Pea son’s co ela ion , and he pa ame ic measu es Coe icien o
de e mina ion R2and C oss co ela ion ρ(n)should no be conduc ed wi hou demons a ing he
no mali y o he samples’ popula ion. These c i e ia we e p esen ed o he i e expe imen s wi h
he elaxa ion o his main assump ion, in o de o p o ide a ulle compa a i e c i e ia assessmen .
As explained by [25], and R2a e insensi i e o addi i e and p opo ional di e ences be ween
simula ed and obse ed homologous elemen s. This is he main d awback o hese wo c i e ia.
The dis ance be ween he wo da a se s (conside ed as wo poin s poin s de ined wi hin a
N-dimensional space) is a aluable pe o mance c i e ia. The Euclidean dis ance d =2 and he
Manha an dis ance d =1 indica e be e es ima ions o alues close o ze o. When a single expe -
imen is p o ided, he dis ance alue does no p o ide much in o ma ion, simila ly o he easoning
71
explained o ME,MAE,MSE and RMSE.
The Sho Time Se ies dis ance dSTS e u ns a measu emen on he a e age ampli ude di e -
gence o e consecu i e empo al momen s be ween wo da a se s. To be p ecise, i e u ns he
squa e oo o he sum o he squa ed di e ences o he slopes be ween empo al momen s. The
complex in e p e a ion o his measu e is ound o be i s main disad an age. Mo eo e , dST S is
insensi i e o addi i e di e ences be ween he wo da a se s unde assessmen . Fo example, con-
side ing he si ua ion whe e bxi=xi+ 1,∀i, he alue dSTS = 0 would be ob ained, which is he
e e ence alue o a pe ec i .
Ano he dis ance measu e e ised is he Dynamic Time Wa ping DT W . The compa ison
be ween he measu es DT W and d =2 is possible. This compa ison can only be made when: (i)
he wo da a se s Xand b
Xha e he same leng h, (ii) DT W is calcula ed as de ined in equa ion
(3.50), and (iii) he cos unc ion used o calcula e ma ix A(3.48) is he squa ed e o d(xi,bxj) =
(xi−bxj)2. When his h ee condi ions a e ensu ed, DT W alues close o d =2 indica e a highe
simila i y be ween he wo pa e ns. Howe e , his compa ison would no indica e how much
simila he wo pa e ns a e, due o he scale-dependency o bo h measu es.
The DT W alues analyzed in chap e 5 used he euclidean dis ance d(xi,bxj) = d =2(xi,bxj)
as he cos unc ion o calcula e he elemen s o ma ix A(3.48), om whe e he alues o DT W
epo ed we e no compa able wi h he d =2(X, b
X).
The main d awback iden i ied o he DT W is ha i is dependen on he scale and on he
leng h o he obse a ions analyzed. This cons ains he assessmen o o ecas s ela ing o di e -
en case s udies wi h DT W .
The Wa ping Pa h Dis o ion WPD is based on he wa ping pa h cons uc ed wi hin he
DT W algo i hm. WPD e u ns he a e age dis ance be ween he Wa ping Pa h and he co -
esponding diagonal posi ions. The measu e WPDhas he ad an age o being scale independen ,
om whe e i may be used o compa e he accu acy o di e en o ecas s. WPD main disad an-
age is ha i depends on he numbe o obse a ions wi hin he da a se s N. This means ha i
should only be used o compa e di e en case s udies p o ided he leng h o he da a se s used is
simila . Values o WPD close o ze o indica e a be e i . The maximum alue obse able wi h
WPD a y depending on Nand on he numbe o elemen s in eg a ing he espec i e Wa ping
Pa h K.
The disad an age iden i ied wi h WPD is su passed by he Pe cen age Wa ping Pa h Dis o -
ion PWPD.PWPDis no dependen on he obse a ions scale no he numbe o obse a ions.
The alues e u ned by PWPDa e easily in e p e ed as pe cen ages, as hey a y wi hin he ange
[0,1]. The pe ec pa e n simila i y e u ns a PWPD alue o ze o. The wo s pa e n dissimila -
i y would e u n he asymp o ic PWPD alue o one, as i was illus a ed in Figu e 4.1. Al hough
he wo measu es p oposed beha ed as expec ed unde he expe imen s analyzed, u he applica-
bili y o hese measu es o benchma k da a se s is necessa y o p o ide a p ope conclusion on i s
quali y.
72
The Sp ague and Gea phase e o PS&Gand Russel’s phase e o PRa e sui able o assess lag
on se ies wi h sinusoidal beha io , o wa e o m, as explained in [45]. The e o e, he applica ion
o hese measu es o he case s udy o Po uguese i ms a e meaningless.
The No malized In eg al Squa e phase e o PNISE is ela ed wi h he esul s ob ained wi h
ρ(n). When he numbe o empo al pe iods o lag nlag ob ained wi h ρ(n)is di e en om 0,
PNISE e u ns alues di e en om ze o. When nlag = 0, he PNISE = 0 as well.
The Sp ague and Gea magni ude e o MS&Gand Russel’s magni ude e o MR e u n alues
wi hin he ange ]−1,1[. They a e in e p e ed as he pe cen age magni ude disc epancy be ween
he se ies unde analysis, wi h sensi i i y o he sign o he disc epancy (simila easoning as
MPE).
Conce ning he No malized In eg al Squa e magni ude e o MNISE, i e u ns alues wi hin
he ange [−1,0], and is in e p e ed as an absolu e magni ude pe cen age de ia ion (besides he
nega i e sign). The alues ob ained wi h MNISE ha e simila in e p e a ion o he MAPE mea-
su e.
The shape componen No malized In eg al Squa e shape e o SNISE e u ns alues wi hin
he ange [0,1], being 0 he bes possible esul indica ing a good pa e n simila i y and 1 he wo s
possible esul indica ing a bad simila i y. The measu e PDW P e u ns alues wi hin he same
ange and wi h he same in e p e a ion as SNISE.
I would be almos impossible o encompass all pe o mance c i e ia e e enced in li e a u e.
Ne e heless his hesis is a leas a s a ing poin o guide he choice o an adequa e pe o mance
c i e ion o a speci ic alida ion model.
Fu u e esea ch may include he s udy on how o apply he Simple S ing Dis ance Me ic,
sugges ed in [12], o a gene al p ocess o alida ion. This measu e was sugges ed unde he con ex
o alida ion o DNA simula ed pa e ns. Ano he in e es ing issue o explo e unde his con ex ,
is he use o combined g aphical and s a is ical app oaches, ha is summa ily desc ibed in he
e iew pape on alida ion [6]. Ano he ele an opic o u u e esea ch would be he compa a i e
assessmen on he compu a ional e o o pe o mance c i e ia.
73
Bibliog aphy
[1] C. Adami. The use o in o ma ion heo y in e olu iona y biology. Annals o he New Yo k
Academy o Sciences, 1256:49–65, 2012.
[2] Hi o ugu Akaike. A new look a he s a is ical model iden i ica ion. IEEE T ansac ions on
Au oma ic Con ol, AC-19(6):716–723, 1974.
[3] Hi o ugu Akaike. Likelihood o a model and in o ma ion c i e ia. Jou nal o Econome ics,
16:3–14, 1981.
[4] J.S. A ms ong and F. Collopy. E o measu es o gene alizing abou o ecas ing me hods:
Empi ical compa isons. In e na ional Jou nal o Fo ecas ing, 8:69–80, 1992.
[5] Vic Ba ne . Sample Su ey P inciples and Me hods. John Wiley & Sons, 2003.
[6] Gianni Bellocchi, Mike Ri ing on, Ma cello Dona elli, and Kei h Ma hews. Valida ion o
biophysical models: issues and me hodologies. a e iew. Ag onomy o Sus ainable De el-
opmen , 30(1):109–130, 2009.
[7] R. J. Bessa. T eino on line de edes neu onais com c i ´
e ios de in o mac¸˜
ao aplicado `
a p e-
is˜
ao e´
olica. Mas e ’s hesis, Faculdade de Economia da Uni e sidade do Po o, 2008.
[8] F iedhelm Bliemel. Theil’s o ecas accu acy coe icien : A cla i ica ion. Jou nal o Ma -
ke ing Resea ch, 10:444–446, 1973.
[9] F anky Kin Pong Chan and Ada Wai chee Fu. Haa wa ele s o e icien simila i y sea ch
o ime se ies: Wi h and wi hou ime wa ping. IEEE T ansac ions on Knowledge and Da a
Enginee ing, 15(3):686–705, 2003.
[10] Selina Chu, Eamonn Keogh, Da id Ha , and Michael Pazzani. I e a i e deepening dynamic
ime wa ping o ime se ies. In P oceedings o he 2nd SIAM in e na ional con e ence on
da a mining., 2002.
[11] W. J. Cono e . P ac ical Nonpa ame ic S a is ics. John Wiley & sons, 2nd edi ion, 1980.
[12] J.E. Cook and A.L. Wol . Towa d me ics o p ocess alida ion. In P oceedings o he Thi d
In e na ional Con e ence on he So wa e P ocess, pages 33–44, 1994.
74

[13] Ke in K. Dobbin and Richa d M. Simon. Op imally spli ing cases o aining and es ing
high dimensional classi ie s, olume 4. 2011.
[14] Gio gio Fagiolo, Alessio Mone a, and Paul Wind um. A c i ical guide o empi ical alida-
ion o agen based models in economics: Me hodologies, p ocedu es, and open p oblems.
Compu a ional Economics, 30:195–226, 2007.
[15] M. Ga dne . Ma hema ical games he an as ic combina ions o John Conway’s new soli ai e
game o li e. Scien i ic Ame ican, 223, 1970.
[16] Nigel Gilbe and Klaus G. T oi zsch. Simula ion o he Social Scien is . Open Uni e si y
P ess, Buckingham, Philadelphia, 1999.
[17] D. Guja a i. Econome ia. McG ow Hill, 2nd edi ion, 1996.
[18] Rob J. Hyndman. Ano he look a o ecas accu acy me ics o in e mi en demand. Fo e-
sigh , 4:43–46, 2006.
[19] Rob J. Hyndman and Anne B. Koehle . Ano he look a measu es o o ecas accu acy.
In e na ional Jou nal o Fo ecas ing, 22:679–688, 2006.
[20] X. Jiang and S. Mahade an. Bayesian c oss en opy me hodology o op imal design o
alida ion expe imen s. Measu emen Science and Technology, 17(7):1895–1908, 2006.
[21] V. Klemeˇ
s. Ope a ional es ing o hyd ological simula ion models. Hyd ological Sciences,
31(1):13–24, 1986.
[22] L.F. Konikow and J.D. B edehoe . G ound wa e models canno be alida ed. Valida ion o
Geo-hyd ological Models, 15(1):75–83, 2003.
[23] Jouni Kuha. AIC and BIC compa isons o assump ions and pe o mance. Sociological
Me hods & Resea ch, 33(2):188–229, 2004.
[24] Thomas S. Kuhn. The S uc u e o Scien i ic Re olu ions. Chicago: Uni e si y o Chicago
P ess, 1962.
[25] Da id R. Lega es, McCabe J ., and G ego y J. E alua ing he use o goodness o i measu es
in hyd ologic and hyd oclima ic model alida ion. Wa e Resou ces Resea ch, 35(1):233–
241, 1999.
[26] Bei ao Li, Edwa d Chang, and Ching-Tung Wu. DPF a pe cep ual dis ance unc ion o
image e ie al. In IEEE 2002 In e na ional Con e ence on Image P ocessing, 2002.
[27] Bei ao Li, Edwa d Chang, and Yi Wu. Disco e y o a pe cep ual dis ance unc ion o mea-
su ing image simila i y. Mul imedia Sys ems, 8, 2003.
75
[28] Wen ian Li. Mu ual in o ma ion unc ions e sus co ela ion unc ions. Jou nal o S a is ical
Physics, 60(5–6):823–837, 1990.
[29] T. Wa en Liao. Clus e ing o ime se ies da a a su ey. Pa e n Recogni ion, 38(11):1857–
1874, 2005.
[30] Iain L. MacDonald and Wal e Zucchini. Hidden Ma ko and O he Models o Disc e e-
alued Time Se ies. Sp inge , 1997.
[31] M.D. McDonnell, S. Ikeda, and J.H. Man on. An in oduc o y e iew o in o ma ion heo y
in he con ex o compu a ional neu oscience. Biological Cybe ne ics, 105(1):55–70, 2011.
[32] Tom M. Mi chell. Machine Lea ning. The McG aw Hill Companies, Inc.
[33] B uce Miz ach. The dis ibu ion o he Theil U s a is ic in bi a ia e no mal popula ions.
Economics Le e s, 38(2):163–167, 1992.
[34] C.S. Molle Le e , F. Klawonn, K.H. Cho, and O. Wolkenhaue . Fuzzy clus e ing o sho
ime se ies and une enly dis ibu ed sampling poin s. In P oceedings o he 5 h In e na ional
Symposium on In elligen Da a Analysis, Be lin, Ge many., 2003.
[35] N. O eskes, K. Sh ade -F eche e, and K. Beli z. Ve i ica ion, alida ion, and con i ma ion
o nume ical models in he ea h sciences. Science, New Se ies, 263(5147):641–646, 1994.
[36] Sanghyun Pa k and Vijay S. Pande. Valida ion o Ma ko s a e models using shannon’s
en opy. The Jou nal o Chemical Physics, 124(054118):1–5, 2006.
[37] K. Poppe . The Logic o Scien i ic Disco e y. London: Hu chingson Co, 1959.
[38] D. Posada and T.R. Buckley. Model selec ion and model a e aging in phylogene ics: Ad an-
ages o Akaike In o ma ion C i e ion and Bayesian app oaches o e likelihood a io es s.
Sys ema ic Biology, 53(5):793–808, 2004.
[39] J. C. P incipe. In o ma ion Theo e ic Lea ning Renyi’s En opy and Ke nel Pe spec i es.
Sp inge , 2010.
[40] R Co e Team. R: A Language and En i onmen o S a is ical Compu ing. R Founda ion o
S a is ical Compu ing, Vienna, Aus ia, 2012.
[41] J.C. Re sgaa d and H.J. Hen iksen. Modelling guidelines e minology and guiding p inci-
ples. Ad ances in Wa e Resou ces, 27:71–82, 2004.
[42] R.G. Sa gen . Valida ion and e i ica ion o simula ion models. In P oceedings o he 2004
Win e Simula ion Con e ence, 2004.
76
[43] H. Sa in, M. Kokkola as, G. Hulbe , P. Papalamb os, S. Ba ba , and R. J. Yang. A comp e-
hensi e me ic o compa ing ime his o ies in alida ion o simula ion models wi h empha-
sis on ehicle sa e y applica ions. In P oceedings o DETC 08, ASME 2008 In e na ional
Design Enginee ing Technical Con e ences and Compu e s and In o ma ion in Enginee ing
Con e ence, 2008.
[44] H. Sa in, M. Kokkola as, G. Hulbe , P. Papalamb os, S. Ba ba , and R. J. Yang. Compa ing
ime his o ies o alida ion o simula ion models: E o measu es and me ics. Jou nal o
Dynamic Sys ems, Measu emen , and Con ol, 132(6):1–10, 2010.
[45] Leona d E. Schwe . Valida ion me ics o esponse his o ies: pe spec i es and case s udies.
Enginee ing wi h Compu e s, 23(4):295–309, 2007.
[46] C.E. Shannon. A ma hema ical heo y o communica ion. The Bell Sys em Technical Jou nal,
27:379–423,623–656, 1948.
[47] P. Sp en . Applied Nonpa ame ic S a is ical Me hods. Chapman & Hall, 2nd edi ion, 1993.
[48] Leon S. S e ling and Kulda Ta e e . The A o Agen O ien ed Modeling. The MIT P ess.
[49] Richa d Taylo . In e p e a ion o he co ela ion coe icien : A basic e iew. Jou nal o
Diagnos ic Medical Sonog aphy, 6(1):35–39, 1990.
[50] Hen y Theil. Economic Fo ecas s and Policy. No h Holland Pub. Co., 1s edi ion, 1965.
[51] Hen y Theil. Applied Economic Fo ecas s. No h Holland Pub. Co., 1s edi ion, 1966.
[52] C. Tu kay, E. Koc, and S. Balcisoy. In eg a ing in o ma ion heo y in agen based c owd
simula ion beha io models. The Compu e Jou nal, 54(11):1810–1820, 2011.
[53] Nguyen Xuan Vinh, Julien Epps, and James Bailey. In o ma ion heo e ic measu es o clus-
e ings compa ison: Va ian s, p ope ies, no maliza ion and co ec ion o chance. Jou nal
o Machine Lea ning Resea ch, 11:2837–2854, 2010.
[54] Da id L. Weakliem. A c i ique o he bayesian in o ma ion c i e ion o model selec ion.
Sociological Me hods & Resea ch, 27(3):359–397, 1999.
[55] Co J. Willmo . On he alida ion o models. Physical Geog aphy, 2(2):184–194, 1981.
[56] Co J. Willmo and Kenji Ma suu a. Ad an ages o he mean absolu e e o (MAE) o e he
oo mean squa e e o (RMSE) in assessing a e age model pe o mance. Clima e Resea ch,
30:79–82, 2005.
[57] J. H. Za . Signi icance es ing o he Spea man ank co ela ion. Jou nal o he Ame ican
S a is ical Associa ion, 67, 1972.
77