1
This is he accep ed manusc ip o he a icle ha appea ed in inal o m in En i onmen al Modelling & So wa e
63 : 123-136 (2015), which has been published in inal o m a h ps://doi.o g/10.1016/j.en so .2014.09.019.
© 2014 Else ie unde CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/)
Mul i-objec i e en i onmen al model e alua ion by means o mul idimensional ke nel densi y
es ima o s: e icien and mul i-co e implemen a ions
Unai Lopez-No oa1, Jon Sáenz2,3, Alexande Mendibu u1, Jose Miguel-Alonso1, Iñigo E as i4,
Ganix Esnaola2, Agus ín Ezcu a2, Gab iel Iba a-Be as egi4
Co esponding au ho : Unai Lopez-No oa {E-mail: [email p o ec ed], Phone: +34 943 018 012}
1In elligen Sys ems G oup, Dep . o Compu e A chi ec u e and Technology, Uni e si y o he
Basque Coun y (UPV/EHU), P. Manuel La dizabal 1, 20018, Donos ia-San Sebas ian, Spain 2Dep .
Applied Physics II, Uni e si y o he Basque Coun y (UPV/EHU), Sa iena Auzoa z/g, 48940-
Leioa, Spain.
3Plen zia Ma ine S a ion (PIE-UPV/EHU), A ea za Pasealekua, 48620, Plen zia, Spain
4Dep . Nuclea Enginee ing and Fluid Mechanics, Uni e si y o he Basque Coun y (UPV/EHU),
Alda. U kijo, s/n, 48013, Bilbao, Spain
Abs ac
We p opose an ex ension o mul iple dimensions o he uni a ia e index o ag eemen be ween
PDFs used in clima e s udies. We also p o ide a se o high-pe o mance p og ams a ge ed bo h o
single and mul i-co e p ocesso s. They compu e mul i a ia e PDFs by means o ke nels, he op imal
bandwid h using smoo hed boo s ap and he index o ag eemen be ween mul idimensional PDFs.
Thei use is illus a ed wi h wo case-s udies. The i s one assesses he abili y o se en global
clima e models o ep oduce he seasonal cycle o zonally a e aged empe a u e. The second case
s udy analyzes he abili y o an oceanic eanalysis o ep oduce global sea su ace empe a u e and
sea su ace heigh . Resul s show ha he p oposed me hodology is obus o a ia ions in he
op imal bandwid h used. The echnique is able o p ocess mul i a ia e da ase s co esponding o
di e en physical dimensions. The me hodology is e y sensi i e o he exis ence o a bias in he
model wi h espec o obse a ions.
Keywo ds: Mul i a ia e ke nel densi y es ima ion, mul idimensional ke nel densi y es ima ion,
mul i-co e implemen a ion, en i onmen al model e alua ion
2
So wa e a ailabili y:
Name o so wa e o da ase : densi y-pa allel
De elope s: Unai Lopez-No oa (1) Jon Sáenz (2) Alexande Mendibu u (1) Jose Miguel-Alonso (1)
(1) In elligen Sys ems G oup, Dep . o Compu e A chi ec u e and Technology, Uni e si y
o he Basque Coun y (UPV/EHU), P. Manuel La dizabal 1, 20018, Donos ia-San
Sebas ian, Spain. [email p o ec ed], Phone: +34 943 018 012 ,
alexande .mendibu [email protected], Phone: +34 943015020 and [email p o ec ed] Phone: +34 943
018019
(2) EOLO G oup, Depa men o Applied Physics II, Uni e si y o he Basque Coun y,
(UPV/EHU), Ba io Sa iena s/n, 48940-Leioa, Spain. [email p o ec ed], Phone: +34
946012445.
Yea i s a ailable: 2014
Ha dwa e equi ed: any single-co e o mul i-co e p ocesso .
So wa e equi ed:
•C Compile (e.g. GCC). OpenMP compiling capabili ies equi ed o mul i- h eaded
implemen a ions
•MESCHACH ma h lib a y. A ailable a
h p://homepage.ma h.uiowa.edu/~ds ewa /meschach/
•ne CDF da a handling lib a y. A ailable a h p://www.unida a.uca .edu/so wa e/ne cd /
P og am language: ANSI-C
P og am size, including example da a: 102 KB in a single . a .gz ile.
License: Open so wa e, a ailable unde a “New BSD 3-clause license”.
A ailabili y: h p://www.sc.ehu.es/ccwbayes/isg/index.php?op ion=com_ emosi o y&I emid=13 O
go o “Downloads” sec ion o h p://www.sc.ehu.es/isg
3
1 In oduc ion
Clima e models a e he bes ools ha scien is s cu en ly ha e in o de o assess he impac o
inc easing concen a ions o g eenhouse gases and o he an h opogenic in luences on he obse ed
clima e o he Ea h. These ools allow scien is s o unde s and clima ic changes om a dynamical
poin o iew and o gi e quan i a i e answe s o ques ions abou u u e clima e. They make easible
he assessmen o he cha ac e is ics (de e minis ic e sus s ochas ic) o some clima ic a ia ions a
di e en empo al o spa ial scales. Finally, hey a e undamen al in he a ibu ion phase o he
s udy o he clima e change p oblem, since hey allow o con iden ly disca d compe ing hypo hesis
such as whe he he clima e change is oo ed in na u al o an h opogenic causes (Beng sson, 2013;
Knu i, 2008).
Clima e models mus be e alua ed agains di e en obse a ions (O o e al., 2013) o paleoclima e
da a (B aconno e al., 2012; Hind e al., 2012; Mobe g, 2013; Sundbe g e al., 2012) in o de o ge
a quan i a i e indica ion on he con idence ha we can pu in o hei ou pu s depending on he
e icacy o he models o eliably ep esen he clima ic p ocesses and eedbacks. Con a y o
ope a ional wea he o ecas models, he e is no way o p ope ly e alua e models agains u u e
clima e, since u u e clima e does no exis ye (Randall e al., 2007; S ocke e al., 2013).
Addi ionally, since pa ame e iza ions o sub-g id scale p ocesses a e no ully independen om
cu en clima e, i is clea ha e alua ion agains cu en clima e is he only easible, albei no
pe ec , solu ion in e ms o e alua ing he p ojec ions o u u e clima e (E as i e al., 2011, 2013;
Radić and Cla ke, 2011; Reichle and Kim, 2008).
The a ionale behind his hypo hesis is ha models ha a e able o be e simula e cu en clima e
a e he ones ha we expec will also be he bes ones in e ms o he simula ion o u u e clima e. I
is well known ha his is no necessa ily ue due o he di e en beha io o models in e ms o
hei in e nal eedback mechanisms (And ews e al., 2012; Dessle , 2013). These eedbacks lead o
di e ing alues o he clima e sensi i i y o models and, hence, u u e wa ming is dependen on
hese di e en sensi i i ies, leading o he ques ion whe he all he models a e equally alid (Knu i,
2010).
Pa icula ly o downscaling applica ions and egional impac analysis, an e alua ion o he
adequacy o models is a common s ep (B ands e al., 2011; Radić and Cla cke, 2011; Walsh e al.,
2008). Conside ing ha nume ical downscaling is compu a ionally expensi e, i canno be
4
pe o med o e all he models a ailable om a big expe imen such as CMIP3 o CMIP5, and i is
usually only pe o med on a subse o he models (Hewi and G iggs, 2004). Then, he bes models
a e only used o downscaling in egional clima e assessmen , since hey ha e been p o en o be
he mos sui able o e a pa icula egion. This s a egy o selec ing he bes subse o all he
a ailable models has been con es ed by some s udies (Rei en and Toumi, 2009) and de ended by
o he s (Macadam e al., 2010), since his esul depends on he emo al o he seasonal cycle and
he use o anomalies ins ead o aw ou pu om he models.
The e a e se e al in e -compa ison expe imen s ha ha e been se -up in o de o d i e he models
wi h common bounda y condi ions so ha esul s be ween model uns can be compa ed. As an
example o his kind o s anda d expe imen al se ups, ha in se e al cases ha e hei o igins in he
nine ies, we can ci e he A mosphe ic Model In e compa ison P ojec (Ga es e al., 1998), he
P ojec o In e compa ison o Land Su ace Pa ame e iza ion Schemes (PILPS) (Hende son-Selle s
e al., 1995) o he Palaeoclima e Modelling In e compa ison P ojec (PMIP) (Kageyama e al.,
1999), he A mosphe ic Chemis y and Clima e Model In e compa ison P ojec (ACCMIP),
desc ibed by Lama que e al. (2013), o he one ha is mos ele an o ou s udy, since we use da a
om his expe imen , he Coupled Model In e compa ison P ojec (CMIP) (Meehl e al., 2007;
Taylo e al., 2012). These p ojec s ha e co e ed se e al phases h ough he yea s and o he case
o he CMIP da a, CMIP5 can al eady be used. In gene al, models g ouped unde a simila
expe imen al se -up such as CMIP3 o CMIP5 a e conside ed as an ensemble o oppo uni y (Annan
and Ha g ea es, 2010). The e a e some limi a ions because e en o coo dina ed expe imen s such
as CMIP3, he e is some eedom in he way he ex e nal bounda y condi ions a e applied ( hey a e
no 100% equal o all he models), see Table 1 in Wang e al. (2007). The numbe o ealiza ions
om e e y model in he ensemble o oppo uni y is no he same, nei he and, he e o e, he
in luence o each model in he beha iou o he ensemble is no he same. Addi ionally, models a e
less independen han hey should be, since c i ical algo i hms o componen s a e sha ed by se e al
models (Fe nández e al., 2009; Knu i e al., 2013; Masson and Knu i, 2011; Pennell and Reichle ,
2011).
In e ms o e alua ion o clima e models, i is well known ha clima e simula ions a e un, mos o
he imes, pas he limi o de e minis ic p edic abili y associa ed wi h p edic abili y o he i s kind,
acco ding o Lo enz's classi ica ion. Clima e models simula e clima e change unde a ying
bounda y condi ions in e ms o he P obabili y Densi y Func ions (PDF) o clima ic a iables. The
a ying bounda y condi ions consis o ex e nal o cings such as he a iabili y in sola i adiance,
5
o bi al pa ame e s o an h opogenic g eenhouse gas emissions, amongs o he po en ial d i ing
ac o s (Beng sson, 2013). Some aspec s o clima e simula ions a e de e minis ic, such as he
seasonal cycle a ex a opical la i udes (E as i e al., 2013). On he o he side, some p ope ies o
he a mosphe ic ci cula ion such as he blocking a ex a opical la i udes can no be p ecisely
o ecas wi h lead imes co esponding o se e al days wi hou he use o ensemble o ecas sys ems
(Ma shall e al., 2014) because o he sensi i i y o ini ial condi ions (F ede iksen e al., 2004) and
he model o mula ion (Pelly and Hoskins, 2003) oo. The e o e, clima e model e alua ions ha a e
un pas he limi o de e minis ic p edic abili y should no a p io i expec a close consis ency
be ween wea he -linked a ia ions o global o egional empe a u e o p ecipi a ion be ween
models and obse a ions, excep a he longe ime-scales esponding o ex e nal o cings (Gleckle
e al., 2008; San e e al., 2011). These di e ences e lec he well-known di e ence o he
sensi i i y o he esul s o e o s in he ini ial condi ions (p edic abili y o he i s kind) o o
e o s in he e ol ing bounda y condi ions (p edic abili y o he second kind) (Chu, 1999; Lo enz,
2006).
The e is no a uni e sally accep ed s a egy o clima e model e alua ion, since i is well known ha
clima e model e alua ion and co esponding skill sco es undamen ally depend on he a ge a ea,
a iable o in ended applica ion o he model e alua ion s udy (Knu i, 2008). Some s udies make a
ocus on he de e minis ic pa s o he model simula ions (basically, he seasonal cycle) (Boe and
Lambe , 2001; Taylo , 2001). O he s udies (o ien ed o he s udy o d ough s o loods) end o
ocus on ex eme pe cen iles, since hey a e much mo e meaning ul indica o s o clima e change
(DeAngelis e al., 2013).
This s udy by DeAngelis e al (2013) is cu en ly impo an o us, because i shows ha models
some imes p oduce accu a e alues o he a e age o some clima ic a iable due o e o
compensa ion e ec s. They can, o ins ance, unde es ima e he equency o high p ecipi a ion
e en s and o e es ima e he equency o low p ecipi a ion e en s. This poin s o he need o
e alua e addi ional cha ac e is ics o clima e model simula ions beyond he mean alue and
s anda d de ia ion. Consequen ly, a ew yea s ago, an index compu ed om he whole PDF o
clima ic a iables was de eloped (Maxino e al., 2008; Pe kins, 2007). I compa es wo PDFs and
compu es he minimum alue o bo h PDFs a e e y abscissa. The a ea below his minimum
ep esen s he a ea below bo h PDFs. As such, o a pe ec model, i s alue would be one, i bo h
PDFs ma ched pe ec ly. This PDF-index o index o dis ibu ional ag eemen is he one ha we
will gene alize o he mul idimensional case in his con ibu ion. The PDF-index analyses he
6
co espondence o he whole PDF bo h om a model and obse a ions (Maxino e al., 2008;
Pe kins e al., 2007). The one-dimensional PDF-index has e y o en been used in he li e a u e
h ough he las yea s (B ands e al., 2012; E as i e al., 2011, 2013; Fu e al., 2013; Maxino e al.,
2008; Pe kins e al., 2007; Schwalm e al., 2013; Ylhaïsi and Räisänen, 2013 o name a ew). In his
con ibu ion we p opose i s ex ension o mul iple dimensions, hus allowing o compa e se e al
ea u es o clima e o en i onmen al models a a single s ep. The use o he PDF-index shows some
ad an ages wi h espec o o he app oaches, in he sense ha i samples he ull PDF o he clima ic
a iables. The e o e, he PDF-index is a e y good index o he o e all e alua ion o he ag eemen
be ween clima e models and obse ed clima e. Howe e , he e a e o he sho comings, such as he
ac ha he analysis in e ms o PDFs does no conside he ime sequence o e en s, and he
numbe o os days o he numbe o con inuous days wi hou p ecipi a ion a e impo an in e ms
o impac s (B ands e al., 2012). The PDF-index gi es less weigh o he ails o he dis ibu ion
and, i is hus no adequa e as he single index o he analysis o ex emes (B ands e al., 2012).
In he case o he pape s men ioned p e iously, he PDF-index is compu ed by means o
unidimensional PDFs. Howe e , in se e al cases, s udies using he PDF-index o o he sco es
(Dessai e al., 2005; Reichle and Kim, 2008) e alua e he skill o clima e models acco ding o
se e al a iables ha may be o in e es o he impac communi y. The mos ob ious ins ances
migh be p ecipi a ion and empe a u e, bu i he scien is s a e in e es ed in downscaling s a egies,
o he a iables such as geopo en ial heigh o sea le el p essu e appea e y o en (B ands e al.,
2011; E as i e al., 2011, 2013; Fu e al., 2013; Maxino e al., 2008; Radic and Cla ke, 2011). In
hese p e ious e e ences, he skill o he models is compu ed on a pe - a iable basis by means o
uni a ia e diagnos ics. Thei inal skill sco e is compu ed by agg ega ing indi idual pe - a iable
e alua ions ei he by simple a e aging o anking o skill sco es. Howe e , he e is cu en ly a lack
o uni e sally accep ed way o pe o ming his combina ion o sco es o di e en a iables and he
me hodology ha we p opose in his con ibu ion is aimed o ill his oid, since a single index o
dis ibu ional ag eemen is e u ned om he mul idimensional PDF.
The main objec i e o his con ibu ion is, he e o e, o de elop a me hodology ha can be applied
o ge a mul idimensional sco e ha allows o e alua e in a single s ep di e en a iables om
clima e simula ions agains obse a ions. In o de o explain he ad an ages de i ed om using a
mul idimensional app oach, we show a simple example de i ed om a syn he ic da ase . We ha e
c ea ed h ee syn he ic da ase s, G1, G2 and G3, de i ed om wo-dimensional gaussian
7
dis ibu ions. Fo each case, he gaussians a e cen e ed
µ
i
=
0
bu he co esponding co a iance
ma ices used o c ea e hem a e gi en by
S
1
=S
2
=
(
1
0.750.75
1
)
o G1 and G2, whils o G3, he
hi d gaussian, he co a iance ma ix is gi en by
S
3
=
(
1
−0.75−0.75
1
)
. I can be seen ha , despi e
he di e ence in he s uc u e o he dis ibu ions o poin s (Figu e 1, le ), he uni a ia e PDFs and
co esponding indices show a good ag eemen (Figu e 1, middle and igh and Table 1), e en
hough he dis ibu ions a e di e en . This is qui e an a i icial example, bu i illus a es he poin
ha some pa s o he PDFs close o he diagonals can be p ojec ed on o simila a eas o e he axis
when using unidimensional indexes o ag eemen , masking he di e ences be ween he PDFs o he
model and he obse a ions. The e o e, i is in e es ing o analyse he ull s uc u e o he
mul idimensional PDF, since i yields a ealis ic di e ence be ween he sco e co esponding o G1
e sus G2 (good ag eemen ) and G1 e sus G3 (bad ag eemen ).
Figu e 1. Poin s c ea ed om G1 ( ed) and G3 (g een) dis ibu ions (le ), uni a ia e p obabili y
dis ibu ions co esponding o he X a iables (middle) om G1 ( ed) and G3 (g een) and uni a ia e
p obabili y dis ibu ions co esponding o he Y a iable om G1 ( ed) and G3 (g een).
Table 1. Indices o dis ibu ional ag eemen s o poin s de i ed om known gaussians using
uni a ia e sco es o X and Y a iables o wo-dimensional sco es.
Uni a ia e sco e 2D sco e
G1-X G1-Y G1
G2 X 0.998 0.999
Y 0.999
G3 X 0.998 0.463
Y 0.997
8
To ill in he gap illus a ed by he example, we gene alize he PDF-index by Pe kins e al. (2007) o
n-dimensional phase spaces wi h he inal aim o allowing an easy mul i-c i e ia e alua ion o
models. Tha way, a single PDF-based index can g oup he pe o mance o he models acco ding o
he mul idimensional phase-space spanned by all he a iables chosen o he e alua ion o he
model. In o de o make i easie o o he esea che s o use his me hodology, we p esen an
implemen a ion o his mul idimensional ex ension by means o a se o ools ha can p ope ly
add ess he compu a ional p oblems ha appea when making ke nel-based es ima ions o PDFs
wi h massi e da ase s. In he case s udies ha we show in his pape , we compu e se e al
ealiza ions o he PDF-index using one o ou dimensional phase spaces wi h up o 13 000 poin s
o e e y pa icula model/ ealiza ion. This is e y in ensi e compu a ionally, and o his eason we
eel ha an e icien implemen a ion o he es ima ion o mul idimensional PDFs could be o g ea
help o esea che s in his a ea. Thus, we ha e de eloped a gene al-pu pose ool o compu e
ke nel-based mul idimensional PDF es ima ions ha uns on s a e-o - he-a mul i-co e p ocesso s.
Ou p oposal has wo main cha ac e is ics: (1) a ine- uned algo i hm o calcula e he PDF ha
minimizes he numbe o compu a ions and (2) a pa allel implemen a ion o his algo i hm ha
allows i o e icien ly un in mul i-co e p ocesso s.
The me hod is applied o wo di e en case-s udies. The i s case s udy co esponds o a ealis ic
applica ion o clima e model e alua ion (E as i e al., 2013). Thei esul s a e e-analyzed using
his new me hodology. Addi ionally, a sensi i i y o he esul s o he selec ion o he bandwid h
pa ame e is ca ied ou . Finally, he obus ness o he esul s p o ided by he me hod o he
exis ence o biases in he models is also s udied. The second case s udy co esponds o he analysis
o he pe o mance o a coupled a mosphe e-ocean eanalysis in ep oducing he global scale Sea
Su ace Tempe a u e and Sea Su ace Heigh and i co esponds o a highe -dimensional p oblem.
This second case s udy will be used o s ess wo o he me i s a ibu ed o he p oposed
me hodology. On he one hand, his example in he con ex o he physical oceanog aphy will
demons a e he wide ange o he applicabili y o he me hodology. On he o he , he combina ion
o a iables wi h di e en physical dimensions will illus a e he abili y o he me hod o p ocess
mul i a ia e da a and i s abili y o be applied o a la ge amily o en i onmen al models.
The emaining o his pape is s uc u ed as ollows. Sec ion 2 p esen s he ma e ials and me hods
used in he pape . Resul s a e shown in sec ion 3. The discussion is p esen ed in sec ion 4, and he
pape inishes wi h conclusions in sec ion 5.
9
2 Ma e ial and me hods
2.1 Da a ep esen ing he daily seasonal cycle o zonally a e aged empe a u e om global
clima e models and eanalysis.
Fo he i s case s udy shown in his pape , we selec a educed dimensionali y ep esen a ion o he
daily seasonal cycle o empe a u e ha we al eady analyzed in E as i e al. (2013). We analyze
empe a u e o he ai a he su ace (TAS) daily da a om se en models (20C3M simula ions) o
he CMIP3 expe imen (Meehl e al., 2007) ha we e used by he Fou h Assessmen Repo o he
IPCC (Randall e al., 2007). The models used a e he BCCR-BCM2.0, GFDL-CM2.0, GFDL-
CM2.1, MIROC3.2-HR, MIROC3.2-MR, MPI-ECHAM5 and MRI-CGCM2.3, and he basis o he
selec ion o his subse o models and hei cha ac e is ics can be ound in E as i e al. (2013). The
same p ocedu e is used o TAS da a om ERA40 (Uppala e al., 2005) and NCEP/NCAR
Reanalysis 1 (Kalnay e al., 2996), e e ed o as NCEP onwa ds. The TAS da a om models and
eanalyses we e e-g idded o he same 2.5º x 2.5º g id by means o bilinea in e pola ion, since his
was he coa ses g id used by any o he eanalysis used ( he one used by NCEP). To educe he
dimensionali y o such a g idded da ase , he TAS da a we e zonally a e aged and p ojec ed on o
Legend e polynomials ha cons i u e an adequa e basis o e he sphe e. Those ha e e y o en been
used as a basis in one-dimensional ene gy balance models (No h e al., 1981) and a e he basis used
o he me idional componen s o sphe ical ha monics used in spec al decomposi ions o he
equa ions o mo ion (Washing on and Pa kinson, 2005). The Legend e polynomials a e o hono mal
in he se o unc ions o e he con inuous in e al [-1, 1], bu hei co esponding disc e e-g id
coun e pa s a e no o hogonal. Fo his eason, we applied he G am-Schmid o hogonaliza ion
p ocedu e o ob ain he leading disc e e o hogonal
P
0
(
µ
)
,
P
1
(
µ
)
and
P
2
(
µ
)
Legend e
polynomials. In he p e ious equa ions,
µ=sin
(
θ
)
e e s o he sine o la i ude. The zonally
a e aged TAS p o iles ha e been p ojec ed on o he o hogonal disc e e
P
0
(
µ
)
,
P
1
(
µ
)
and
P
2
(
µ
)
Legend e polynomials and his has p o ided us wi h he co esponding ime- a ying
coe icien s
c
0
(
)
,
c
1
(
)
and
c
2
(
)
. Due o he me idional shape o he o hogonal disc e e
polynomials shown in Figu e 1 in E as i e al., (2013), he physical meaning o he empo al
expansion coe icien s can be easily unde s ood. The coe icien
c
0
(
)
desc ibes he seasonal
e olu ion o global-mean empe a u e linked o he di e en dis ances om he ea h o he Sun
co esponding o he apogee o he pe igee posi ions,
c
1
(
)
desc ibes he seasonal e olu ion o
summe -win e om one Hemisphe e o he o he and
c
2
(
)
desc ibes he TAS di e ences be ween
16
In he i s case s udy used in his pape he phase-space is idimensional, bu he implemen ed
p og am allows he use o go up o any dimension (s a ing om one), as shown by he second case
s udy used in his pape . The one-dimensional case is also co e ed by ou p og am e en hough
he e a e o he implemen a ions ha can co e he one dimensional case. The no el y o ou
con ibu ion lays on he gene aliza ion o he one-dimensional sco e o highe dimensions.
In addi ion, s anda d OpenMP p og amming di ec i es (Dagum and Menon, 1998) ha e been also
included wi h he aim o exploi ing he mul i-co e capabili y o p esen compu e s. The se o
obse ed poin s will be equally dis ibu ed amongs he p ocesso s o hei compu a ion in pa allel.
This way, he wo kload is spli among he a ailable p ocesso s, educing he execu ion ime
(almos ) linea ly o he numbe o co es used.
2.3.2 Selec ion o he op imal bandwid h
The second s ep ha we desc ibe (al hough i should be he i s s ep in he applica ion o he
p og ams) is o ind he op imal bandwid h alue o he PDF compu ed in he i s s ep. I is well
known ha he compu a ion o he op imal bandwid h o be used in PDF es ima ions using ke nels
is a c i ical s ep in ob aining eliable PDFs. The e a e wo majo s a egies o he de e mina ion o
he op imal bandwid h (Sco , 1992; Sl e man, 1986): c oss- alida ion (Duong and Hazel on, 2005)
and smoo hed boo s ap (Fa away and Jhun, 1990).
The c oss- alida ion app oach leads o he con olu ion o he ke nel wi h i sel , a e y ough
ma hema ical p oblem o he Epanechniko ke nel wi h an open numbe o dimensions. I is
usually sol ed by means o gaussian mul iplica i e ke nels (Duong and Hazel on, 2005), bu his
wouldn' allow us o use he OPB app oach explained in sec ion 2.3.1 abo e. In ou case, we ha e
selec ed he use o smoo hed boo s ap es ima es o he op imal bandwid h, since i simpli ies he
gene aliza ion o he solu ion o an open numbe o dimensions in he mul idimensional case o he
non-mul iplica i e Epanechniko ke nel we a e using. In o de o p oduce he new es ima ions in
he mul idimensional case we ake ad an age o he ac ha he ke nel is sphe ically symme ic in
he space co esponding o he sphe ically symme ic p incipal componen s. Thus, he same s a egy
used by uni a ia e ke nel es ima ions is used o e e y di ec ion in he space spanned by he
sphe ically symme ic p incipal componen s (Sil e man, 1986). Su oga e samples a e c ea ed in
his space, and his p ocedu e gua an ees ha he s uc u e o he co a iance ma ix is p ope ly
p ese ed.
17
Following Fa away and Jhun (1990), we use a smoo hed boo s ap p ocedu e o es ima e he
squa ed e o be ween wo es ima es o he PDF. The smoo hed es ima e s a s om a e e ence
e alua ion o he PDF
(
x,h
0
)
compu ed using a e e ence bandwid h
h
0
. Then, se e al
es ima ions
n
(
x,h
)
o he PDF a e pe o med a a ying alues o he bandwid h pa ame e h and
a numbe o
n=
1
…
N
ealiza ions o e e y h. The boo s ap p og am checks he e o be ween
he “ e e ence” PDF used in he smoo hed boo s ap p ocedu e and he ac ual boo s ap samples by
e alua ing he squa ed e o
ε
n
(
h
)
=
∫
(
(
x,h
0
)
−
n
(
x,h
)
)
2
dx
. Then, he boo s ap-de i ed
dis ibu ion o he squa ed e o s is used o in e minimum, maximum, median,
P
0
.
025
(2.5%) and
P
0
.
975
(97.5%) pe cen iles o squa ed e o o e e y alue o h. This in o ma ion is epo ed o he
use a e e y h alue. The h alue p oducing he lowes alues o he e o es ima es (we use he
median o
ε
n
(
h
)
in he case s udies in his pape ) is he one selec ed as he op imum bandwid h.
This p ocedu e has been implemen ed in he mpd es ima o _boo s ap p og am. I akes as inpu all
he obse ed poin s and op ionally (1) a e e ence bandwid h alue, (2) a ange o bandwid h alues
o be e alua ed, (3) he bounda ies o he e alua ion space, and (4) he numbe o epe i ions o he
andom sampling. I (1) is missing, he de aul co esponding o a mul idimensional gaussian
dis ibu ion wi h he same sample size is applied. I (2) is missing a ange (+/- 20% a ound (1), wi h
a s ep such ha he maximum bandwid h in e al is di ided in 10 subin e als) is de ined. In case
(3) is missing, mpd es ima o _boo s ap de ines a ange ha ensu es a space ha su ounds all he
obse ed poin s. Finally, i (4) is missing, 500 ealiza ions a e pe o med. The p og am gene a es as
ou pu squa ed e o s o each o he p o ided bandwid h alues. The pseudo-code is shown in
Lis ing 2.
Compu e he PDF o he e e ence bandwid h h
0
o each h in he ange [hmin,hmax]{
o i e =1 o max_ epe i ions{
gene a e a andom sub-sample S
compu e PDF o S
compu e squa ed e o
}
gene a e s a is ics o squa ed e o
}
e u n s a is ics
18
Lis ing 2. Pseudo-code o he mpd es ima o _boo s ap p og am.
2.3.3 Compu a ion o he PDF sco e
The inal s ep o he me hodology is o compu e he PDF sco e. Once he use has compu ed he
PDFs (by means o he mpd es ima o p og am) co esponding bo h o he model and he
obse a ions using he op imal bandwid h alue epo ed by mpd es ima o _boo s ap, p og am
mpd _sco e has o be execu ed o ge he PDF sco e agains he e e ence model.
The p og am mpd _sco e akes as inpu wo n-dimensional PDFs s o ed as ne CDF iles, gene a ed
o he same domain by he i s p og am mpd es ima o , and p o ides as ou pu a PDF-index S by
means o he ollowing equa ion (adap ed in his case o a h ee-dimensional example):
S=
∑
min
(
Z
ijk
o
,Z
ijk
m
)
dx
i
dx
j
dx
k
, whe e
Z
ijk
o
and
Z
ijk
m
e e o he e alua ion o he PDF om
obse a ions and he model, espec i ely. Please no e ha o highe dimensions, he ex ension is
s aigh o wa d.
The equa ion closely ollows he one used by Pe kins e al. (2007) o Maxino e al. (2008), bu has
been ex ended in his case o i s use wi h a PDF de ined in a mul i-dimensional (n-dimensional)
space. Addi ionally, when wo king in se e al dimensions, he olume o he n-dimensional in e al
whe e he PDF is being compu ed mus be aken in o accoun o no maliza ion pu poses, and so,
he
dx
i
,
dx
j
and
dx
k
e ms accoun o he ac ha he ange o he di e en a iables can be
e y di e en ( he a e age s anda d de ia ions o he coe icien s in ou i s case s udy a e 1.9 K,
9.7 K and 2.1 K). The p og am wa ns he use in case he bias o any o he dimensions is g ea e
han 5% o he s anda d de ia ion o ha a ia e.
2.4 Rep esen a ion o ma ginalized PDFs o he in e p e a ion o esul s.
Finally, e en hough i is no pa o he me hodology we p opose, in o de o be able o iden i y he
di e ences in he index co esponding o indi idual models and o illus a ion pu poses o he
esul s co esponding o he i s case s udy, we ha e compu ed ma ginalized
2D
(
c
i
,c
j
,h
)
=
∫
(
x,h
)
dc
k
,i≠j≠k
wo-dimensional PDFs and p ojec ed hem on o he i-j C0-
C1, C0-C2 and C1-C2 planes, a e ma ginalizing k axes C2, C1 and C0, espec i ely. This will
allow us o show ha using a single mul idimensional sco e is be e han using a se o
unidimensional sco es. We only p esen ma ginalized PDFs o he i s case s udy in he pape and,
o he second case s udy we jus collec he agg ega ed alues o he sco e in a able.
19
3 Resul s
3.1 Applica ion o clima e model simula ion o he daily cycle o su ace empe a u e
Figu e 3 shows he e olu ion o he median o he squa ed e o s and he 95% con idence in e al
compu ed om he boo s ap analysis co esponding o he ERA40 da a when he e e ence PDF is
compu ed wi h wo conse a i e es ima es o bandwid h (h
0
=0.8 and h
0
'=0.67) agains he
bandwid h ha would co espond o he same sample size o a gaussian PDF, 0.637. I can be seen
ha he boo s ap es ima e sugges s a sligh ly lowe alue (0.55-0.60) o he bandwid h pa ame e
han he one ha would co espond o a gaussian mul idimensional PDF. As will be iden i ied in he
ma ginalized PDFs la e , his is o be expec ed, since he zonally a e aged su ace empe a u e is
e y non-no mal and pe iodic, so ha se e al ine scale ea u es o he PDF mus be esol ed, and
hey can only be p ope ly esol ed i he bandwid h is no e y high. The e o e, in he ollowing
s eps an op imum bandwid h o h=0.6 will be used unless o he wise explici ly s a ed.
In o de o es he sensi i i y o he classi ica ion o di e en alues o he bandwid h used, Table 2
p esen s he esul s o he mul idimensional PDF sco es o di e en alues o he bandwid h
pa ame e (e e y model is cen e ed and checked agains ERA40). Fo he op imum bandwid h
(h=0.6), he bes model a ailable is he al e na i e eanalysis ha is used in his s udy ( he NCEP).
This is some hing ha we expec ed om he beginning, since bo h eanalyses a e based on
obse a ions. This esul suppo s he use o he me hod, since he me hod yields be e esul s o
al e na i e obse a ion-based eanalyses. MIROC3.2-MR and HADGEM1 a e he models ha
ollow. Some o he model uns di e only on he ini ial condi ions and mos o hem a e g ouped
oge he , wi h he excep ion o HADGEM1. The in e p e a ion o his esul is ha he a iabili y o
he index o he use o di e en ini ial condi ions is e y low, as should be expec ed. MIROC3.2-
HR, GFDL, ECHAM5 and BCM2 ollow he p e ious models. The anking inishes ( o he subse
o models and diagnos ic a iable used in his s udy) by he i e andom uns co esponding o he
MRI model. All he uns co esponding o MRI a e g ouped, wi h low alues o he sco e ha do
no mix wi h alues co esponding o he es o he models. I seems, he e o e, ha he in a-
ensemble a iance is in gene al (wi hou he excep ion o MIROC3.2-MR and HADGEM1) smalle
han he in e -model a iance o he sco e. In gene al, he main cha ac e is ics o hese esul s a e
obus e en wi h changes in he bandwid h ha span a -33% o a +33% in e al om he op imum
alue ound by means o boo s ap. The models ha show he highes (lowes ) pe o mances wi h
he op imum alue o he bandwid h con inue showing a simila pe o mance o highe o lowe
alues o he bandwid h. The e a e occasional excu sions o a model o a mos one al e na i e
posi ion up/down o he anking, bu , on he whole, models end o main ain hei ela i e anks
20
e en when he bandwid h is changed by a +/-33% ela i e change a ound he op imum alue.
Figu e 3. Squa ed e o s (median and 95% con idence in e al as de i ed om he boo s ap
es ima es) be ween he andomly gene a ed PDFs and he e e ence PDF (le , h
0
=0.8 and igh ,
h
0
=0.67) used o he gene a ion o he smoo hed boo s ap.
Figu es 4, 5 and 6 show he plo s o he ma ginalized PDFs o he case o he NCEP (con ou s)
e sus ERA40 (shaded), a model showing a high alue o he sco e, MIROC-3.2-MR (con ou s)
e sus ERA40 (shaded) and a model wi h a lowe sco e, such as MRI (con ou s) e sus ERA40
(shaded). In o de o show simple numbe s in he plo s and scales, alues o he ma ginalized PDFs
a e mul iplied by one housand be o e plo ing. Be o e compu ing he PDFs, he biases be ween
e e y model and ERA40 ha e been emo ed by cen e ing all he se ies.
Table 2. Values o he mul idimensional S sco e co esponding o di e en alues o he bandwid h
pa ame e and associa ed ankings ha would co espond o he models, when compa ed wi h
ERA40 eanalysis da a.
21
Figu e 4. Ma ginal PDFs o NCEP (con ou ) and ERA40 (shaded) p ojec ed on o he planes de ined
by he C0-C1 coe icien s (le ), C0-C2 coe icien s (middle) and C1-C2 coe icien s ( igh ). Values
o he PDF ha e been mul iplied by 1000 in o de o imp o e he ep esen a ion o numbe s.
Figu e 4, le , shows ha on he C0-C1 plane, he PDF is clea ly bimodal, as should be expec ed
om a pe iodic de e minis ic signal such as he seasonal cycle o empe a u e. C0 ep esen s he
global a e age o su ace empe a u e and C1 ep esen s he di e ence in empe a u e be ween he
No he n and Sou he n Hemisphe es. The main clus e s o he C0-C1 PDF appea agg ega ed
a ound each Hemisphe e's summe . NCEP alues show a sligh ly wa me global empe a u e (C0)
du ing Sou he n Hemisphe e summe han he alues shown by ERA40. Figu e 4 (middle) shows
ha he ampli udes and phases o he mean global empe a u e (C0) and he equa o ial bulge (C2)
a e simila in bo h eanalyses. On he C1-C2 plane, he ma ginalized PDF shows ha he main
di e ence be ween bo h eanalyses appea s as a sligh ly highe di e ence o empe a u e be ween
hemisphe es (C1) in NCEP when he coe icien ep esen ing he equa o ial bell (C2) is posi i e
22
(summe in he No he n Hemisphe e). Howe e , he PDFs gene a ed by bo h eanalysis a e
ex emely simila , as e lec ed in he high alue o he S index be ween NCEP and ERA40 (0.82).
This is some hing ha we expec ed om he beginning, since hey co espond o obse a ional
da ase s.
Figu e 5. Ma ginal PDFs o MIROC3.2-MR ( un 2, con ou s) and ERA40 (shaded) p ojec ed on o
he planes de ined by he C0-C1 coe icien s (le ), C0-C2 coe icien s (middle) and C1-C2
coe icien s ( igh ). Values o he PDF ha e been mul iplied by 1000 in o de o imp o e he
ep esen a ion o numbe s.
Figu e 5 co esponds o he ma ginal PDFs o MIROC3.2-MR model (second un), one o he bes
CMIP3 models acco ding o he me ic selec ed in his s udy. O e he C0-C1 plane (le ), he e is
qui e a good ag eemen be ween bo h PDFs, since bo h clea ly ep esen he bimodal s uc u e o
he PDF. Howe e , he di e ences be ween MIROC3.2-MR and ERA40 a e highe han in he
p e ious case, bo h in e ms o he loca ion o he No he n Hemisphe e summe and also in
ansi ions be ween seasons ha appea in he a eas be ween he maxima in he ma ginal PDF. In he
case o he C0-C2 plane (middle), he highes disag eemen appea s a he p ecise loca ion o he
maxima o he ma ginal PDFs, pa icula ly du ing No he n Hemisphe e summe . A simila
diagnos ic can be de i ed om he ma ginal PDF o e he C1-C2 plane. Despi e bo h ma ginal
PDFs a e clea ly bimodal, sligh di e ences exis a he placing o he PDF maxima. The equa o ial
bell (C2) in MIROC3.2-MR is s onge han he one in ERA40 du ing nega i e phases (No he n
Hemisphe e win e ) o in e -hemisphe ic empe a u e di e ences (C1).
23
Figu e 6. Ma ginal PDFs o MRI ( un 1, con ou s) and ERA40 (shaded) p ojec ed on o he planes
de ined by he C0-C1 coe icien s (le ), C0-C2 coe icien s (middle) and C1-C2 coe icien s ( igh ).
Values o he PDF ha e been mul iplied by 1000 in o de o imp o e he ep esen a ion o numbe s.
Figu e 6 co esponds o MRI ( un 1) model. The e olu ion o he daily seasonal cycle o
empe a u e in e ms o C0 (global T) and C1 (in e -hemisphe ic empe a u e con as , le ) does no
p esen a bimodal s uc u e wi h he PDF maxima placed a he same poin s shown by he
eanalysis. The ansi ions be ween summe and win e egimes happen h ough ou es ha do no
co espond o he ones in he ERA40 Reanalysis. The s uc u e o he ma ginal PDF o he C0-C2
plane is ma kedly di e en be ween MRI and ERA40, wi h he cold maximum in he PDF du ing
summe in he Sou he n Hemisphe e qui e misplaced in he case o MRI. This is also appa en in he
ma ginal PDF co esponding o he C1-C2 plane, whe e maxima o he PDFs do no appea nei he
on he same places no e en wi h he same phases.
Finally, Figu e 7 shows ha he index is e y sensi i e o he exis ence o a bias be ween he models
and e e ence obse a ions. In his case, he PDFs a e compu ed wi hou p e iously emo ing he
bias be ween he su oga e model (NCEP da a) and he obse a ions (ERA40) and he S sco e index
ha we ge be ween ERA40 and NCEP eanalyses is ex emely low (S=0.075). The ma ginal PDFs
show ha in gene al he e is a e y good ag eemen in he s uc u e o he 3D PDFs, bu he cen e
o masses o bo h PDFs a e no loca ed a he same places. The biases o e e y coe icien a e no
e y high, conside ing hei a iances. The bias o he C0 componen is 0.7 K (0.2% ela i e e o ),
he bias in C1 is 0.4 K (7.5% ela i e e o ) and he bias in C2 is -0.34 K (-1.33% ela i e e o ).
Howe e , e en such low alues o he bias lead o a sco e index ha could be in e p e ed as poo
pe o mance o he su oga e model (NCEP eanalysis) e sus ERA40 due o he complex s uc u e
o he 3D PDF. Howe e , his is a alse imp ession ha can no be de ended i he spa ial pa e ns o
24
he ma ginal PDFs a e analyzed in de ail. This means ha he index should no be applied o model
esul s ha a e biased agains he e e ence obse a ions. The exis ence o biases in he models
leads o g ea e obse a ional unce ain y when he model and obse a ional da ase s a e no
cen e ed. The code does no o ce he cen e ing o he da ase s and, he e o e, he use mus ake
ca e o his when he dimensionali y educ ion s age o he da a analysis is done. In pa icula , i is
in e es ing o s ess ha , in e nally, when compu ing he n-dimensional PDFs, all he da ase s a e
cen e ed (each one using i s n-dimensional a e age) be o e compu ing he co esponding
sphe ically symme ic p incipal componen s. When he ou pu ne CDF iles holding he PDFs a e
sa ed, he o iginal uni s in he phase space o each da ase (model o obse a ions) a e eco e ed
and he a e age is added o he anomalies de i ed om he PDF in he p incipal componen space
s o ed in he memo y o he compu e . The e o e, he key poin he e is ha i he e exis s a cons an
bias be ween he model and he e e ence obse a ions ( i s and second ne CDF iles passed o
p og am mpd _sco e), i could lead o e y low alues o he sco e despi e he model ep esen ing
p ope ly he a iabili y (anomalies). This means ha he e alua ion o he models in e ms o a
cons an bias and he n-dimensional PDFs should be ca ied ou as di e en s eps.
Figu e 7. Ma ginal PDFs o non-cen e ed NCEP (con ou s) and ERA40 (shaded) p ojec ed on o he
planes de ined by he C0-C1 coe icien s (le ), C0-C2 coe icien s (middle) and C1-C2 coe icien s
( igh ). Values o he PDF ha e been mul iplied by 1000 in o de o imp o e he ep esen a ion o
numbe s. The bias be ween bo h eanalysis has been e ained.
F om he poin o iew o pe o mance, we ha e measu ed he execu ion ime needed by each
e sion o he p og am o comple e he boo s ap p ocedu e. On a e age, he se ial OPB app oach is
140 imes as e han he se ial GPB app oach and, mo eo e , he pa allel OPB p og am scales
25
linea ly wi h he numbe o co es, being 4.3 imes as e han i s se ial coun e pa when using 4
co es. This means ha an e alua ion o a single model ha akes app oxima ely 22 days wi h he
se ial GPB p og am, can be execu ed in less han one hou using he mos e icien and pa allel
implemen a ion p esen ed in his con ibu ion. These expe imen s ha e been conduc ed in a desk op
compu e wi h an In el i7 3820 p ocesso ( ou co es, 3.6GHz, Hype h eading enabled) wi h 8GB
o RAM. The e o e, he use o his echnique is no limi ed o he a ailabili y o specialized clus e s
o ha dwa e ha would limi i s p ac ical use.
3.2 E alua ion o Sea Su ace Tempe a u e and Sea Su ace Heigh
The i s wo PCs o he global co e age weekly ime-scale SST (T1, T2) and SSH (H1, H2)
a iables belonging o he ARMOR-3D (Guinehu e al., 2004; Guinehu e al., 2012) and CFSR
(Saha e al., 2010; Saha e al., 2014) da ase s will be used in he ollowing o e alua e he second
wi h espec o he o me . This means ha he ARMOUR-3D p oduc (blended sa elli e and in-si u
obse a ion p oduc ) is he e e ence o e alua e he CFSR p oduc (coupled a mosphe e-ocean
modelling p oduc ). Conside ing he i s wo PCs o each a iable in he e alua ion (T1, T2, H1,
H2), he global-scale main a iabili y modes o each a iable a e being ake in o accoun a a
glance. As he seasonal cycle was no explici ly emo ed om he anomalies used o deduce he
PCs, he ou conside ed a iables a e almos comple ely ela ed o he global-scale seasonal cycle
(H2 con ains some longe ime-scale a iabili y). Thus compa ing combina ions o di e en
a iables om CFSR wi h hose o ARMOR-3D he capaci y o he modelling p oduc o join ly
cha ac e ize di e en main global-scale a iabili y modes ( hei seasonal cycles) is e alua ed. Fo
example, i T1 and H1 a e conside ed a he same ime (case T1H1) he capaci y o CFSR o
simula e he main global-scale componen s o he seasonal cycle o he SST and SSH a iables is
being e alua ed in a single and mul i a ia e sco e.
Table 3 shows he op imal h and he sco e ob ained wi h he 6 analyzed cases going om he
uni a ia e T1 and H1 cases, he mul i-dimensional uni a ia e T1T2 and H1H2 ( ese ing he e m
mul i a ia e o he cases wi h a iables wi h di e en physical dimensions, i.e. Kel ins and me e s)
and he mul i a ia e T1H1 and T1T2H1H2 cases. All a iables ha e ze o mean so no bias ela ed
issues will be obse ed in his case. Like in he p e ious case s udy on he TAS, he same h ee-s ep
me hodology was applied in his case: o a gi en ow in Table 3, he op imal h using he boo s ap
p ocedu e is ini ially es ima ed. Nex , he PDF using he op imal h is compu ed and, inally, he
sco e (one dimensional, mul idimensional o mul i a ia e) is compu ed om he PDFs ob ained
om he CFSR and he ARMOR-3D a iables.
32
The use o a mul idimensional analysis p oduces a single index co esponding o e e y model e en
a e analyzing se e al a iables, and his esul makes i easy o pe o m e alua ion o he models
unde se e al a ge a iables. In he con ibu ion p esen ed he e, we ha e explo ed one case such as
h ee Legend e coe icien s ha expand he daily cycle o zonally a e aged empe a u e. Howe e ,
he same app oach could be applied o he join analysis o empe a u e, ou going longwa e
adia ion o cloud co e ( o name a ew) such ha he s uc u e o he mul idimensional PDFs
(p obably p ope ly ma ginalized as in his con ibu ion) could shed ligh o e he beha iou o
models acco ding o known physical mechanisms.
E en hough a ool o he se desc ibed in his con ibu ion allows o make an objec i e selec ion o
he op imal bandwid h o be used in he gene a ion o he PDFs by means o smoo hed boo s ap,
he case s udy in his pape shows ha he anking o he models is qui e obus e en unde se e e
(+/- 30% o he op imal bandwid h) changes in he bandwid h used o he gene a ion o he
mul idimensional PDFs. Thus, he esul s ob ained h ough he use o he ools p esen ed in his
con ibu ion a e eliable.
Howe e , he index is ex emely sensi i e o he exis ence o a cons an bias be ween models and i
should no be used wi hou p e iously cen e ing he da a, a inding in ag eemen wi h p e ious
s udies using uni a ia e PDFs (B ands e al., 2011; B ands e al., 2012). The p og ams do no
eques ha he da ase s a e cen e ed, bu a cons an bias be ween he model and obse a ions could
lead o unphysical diagnos ics in se e al dimensions. A po en ial solu ion is o pe o m he
e alua ion on o n-dimensional cen e ed da a, so ha he bias is au oma ically emo ed.
Al e na i ely, he analysis can be pe o med emo ing he bias om he model esul s wi h espec
o obse a ions. The e o e, he analysis o he bias mus s ill be kep independen om he analysis
o he shape o he PDF p esen ed in his con ibu ion. The cu en implemen a ion o he
mpd _sco e p og am p o ides a wa ning i he bias a any o he dimensions is g ea e han 5% o
he s anda d de ia ion o he co espoding a ia e.
The second case s udy demons a ed he applicabili y o he p oposed me hodology o mul i a ia e
and mul idimensional da a using da a om oceanog aphic SST and SSH a iables oo. In addi ion,
and al hough i is no pa o he p oposed echnique, his case s udy also demons a ed he po en ial
o he use o a p ep ocessing s ep o he educ ion o he dimensionali y o he da a, based on a
PCA analysis o he o iginal da ase in his case. This shows ha he me hod can po en ially be
applied o a la ge amily o en i onmen al p oblems.
33
The o e all e alua ion o en i onmen al models is a complex ask and di e en pe o mance sco es
de ec di e en weak o s ong poin s o he a ailable global models. We hope ha he addi ion o a
new me hodology and ools ha allow i s easy applica ion by o he esea che s make i easie he
iden i ica ion in u u e expe imen s o a eas o models ha can be imp o ed.
Acknowledgemen s: Au ho s acknowledge cons uc i e commen s by h ee e e ees and he edi o
o his pape . These commen s ha e lead o an imp o ed e sion o he manusc ip . We
acknowledge he modeling g oups, he P og am o Clima e Model Diagnosis and In e -compa ison
(PCMDI) and he WCRP's Wo king G oup on Coupled Modeling (WGCM) o hei oles in
making a ailable he WCRP CMIP3 mul i-model da ase . Suppo o his da ase is p o ided by he
O ice o Science, U.S. Depa men o Ene gy. ECMWF ERA-40 da a used in his s udy ha e been
p o ided by ECMWF. NCEP eanalysis da a p o ided by he NOAA/OAR/ESRL PSD, Boulde ,
Colo ado, USA, om hei Web si e a h p://www.es l.noaa.go /psd/ ha e been used. CFSR and
CFS 2 da a was p o ided by he Resea ch Da a A chi e a he Na ional Cen e o A mosphe ic
Resea ch, Compu a ional and In o ma ion Sys ems Labo a o y, Boulde , Colo ado. ARMOUR-3D
da a was ob ainned om MyOcean (h p://www.myocean.eu/). Au ho s hank inancial unding by
p ojec CGL2013-45198-C2-1-R (MINECO, Na ional R+D+i plan), he SAIOTEK p og am om
he Basque Go e nmen (p ojec S-P11UN137). Addi ional unding om di e en calls om he
Uni e si y o he Basque Coun y (UFI 11/55, PPM12/01 and GIU 11/01) has allowed his pape o
be inished. This wo k has also been pa ially suppo ed by he Saio ek and Resea ch G oups 2013-
2018 (IT-609-13) p og ams (Basque Go e nmen ), TIN2010-14931 (Minis y o Science and
Technology), COMBIOMED ne wo k in compu a ional bio-medicine (Ca los III Heal h Ins i u e).
U. Lopez-No oa holds a g an om he Basque Go e nmen . J. Miguel-Alonso and A. Mendibu u
a e membe s o he HiPEAC Eu opean Ne wo k o Excellence. Au ho con ibu ions: JS, AM and
JMA designed he esea ch; ULN, JS, AM and JMA w o e he code dis ibu ed wi h his
con ibu ion; IE, AE, GIB and JS pe o med he compu a ions ha lead om da a in he CMIP3
eposi o y o he Legend e coe icien s used in he i s case s udy; GE pe o med he compu a ions
o he second example, ULN, JS, AM and JMA p epa ed he speci ic compu a ions, g aphics and
esul s used in his con ibu ions a e he Legend e coe icien s and ULN, JS, AM, IE, GE and JMA
w o e he pape .
Re e ences
34
And ews, T., G ego y, J. M., Webb, M. J., and Taylo , K. E., 2012. Fo cing, eedbacks and clima e
sensi i i y in CMIP5 coupled a mosphe e-ocean clima e models. Geophysical Resea ch Le e s, 39,
L09712, doi: 10.1029/2012GL051607.
Ahamada, I. And Flachai e, E. 2010. Non-pa ame ic Econome ics. Ox o d Uni e si y P ess, 176
pages, Ox o d.
Annan, J. D., and Ha g ea es, J. C. 2010. Reliabili y o he CMIP3 ensemble, Geophysical
Resea ch Le e s, 37, L02703, doi:10.1029/2009GL041994.
Bellman, R., 1961. Adap i e Con ol P ocesses: A Guided Tou . P ince on Uni e si y P ess, 255
pp.
Beng sson, L., 2013. Wha is he clima e sys em able o do “on i s own”? Tellus B65, 20189,
doi:10.3402/ ellusb. 65i0.20189.
Benne , N.D., C oke, B.F.W., Gua iso, G., Guillaume, J.H.A., Hamil on, S.A., Jakeman, A.J.,
Ma sili-Libelli, S., Newham, L.T.H., No on, J.P., Pe in, C., Pie ce, S.A., Robson, B., Seppel , R.,
Voino , A.A., Fa h, B.D., And eassian, V., 2013. Cha ac e ising pe o mance o en i onmen al
models. En i onmen al Modelling & So wa e, 40, 1-20, doi:10.1016/j.en so .2012.09.011.
Boe , G. J. and Lambe , S. J., 2001. Second-o de space- ime clima e di e ence s a is ics. Clima e
Dynamics 17, 213-218, doi: 10.1007/PL00013735.
B aconno , P., Ha ison, S. P. , Kageyama, M., Ba lein, P. J. , Masson-Delmo e, V., Abe-Ouchi, A.,
O o-Bliesne , B., Zhao, Y., 2012. E alua ion o clima e models using palaeoclima ic da a , Na u e
Clim. Change , 2, 417-424, doi: 10.1038/nclima e1456
B ands, S., He e a, S., San-Ma ín, D., Gu ié ez, J., 2011. Valida ion o he ENSEMBLES global
clima e models o e sou hwes e n Eu ope using p obabili y densi y unc ions, om a downscaling
pe spec i e. Clima e Resea ch, 48, 145-161, doi: 10.3354/c 00995.
B ands, S. Gu ié ez, J.M., He e a, S., Co iño, A. S., 2012. On he use o eanalysis da a o
35
downscaling. Jou nal o Clima e 25, 2517-2526, doi: 10.1175/JCLI-D-11-00251.1
Chu, Pe e C., 1999. Two Kinds o P edic abili y in he Lo enz Sys em. Jou nal o A mosphe ic
Sciences, 56, 1427–1432. doi: 10.1175/1520-0469(1999)056<1427:TKOPIT>2.0.CO;2
Dagum, L.; Menon, R., 1998 OpenMP: an indus y s anda d API o sha ed-memo y p og amming,
Compu a ional Science & Enginee ing, IEEE , 5, 46-55, doi: 10.1109/99.660313
DeAngelis, A. M., B occoli, A. J., Decke , S. G., 2013. A Compa ison o CMIP3 simula ions o
p ecipi a ion o e No h Ame ica wi h obse a ions: Daily s a is ics and ci cula ion ea u es
accompanying ex eme e en s, Jou nal o Clima e, 26, 3209-3230, doi: 10.1175/JCLI-D-12-00374.1
Dessai, S., Lu, X. and Hulme, M., 2005. Limi ed sensi i i y analysis o egional clima e change
p obabili ies o he 21s cen u y. Jou nal o Geophysical esea ch, 110, D19108, doi:
10.1029/2005JD005919.
Dessle , A. E., 2013. Obse a ions o Clima e Feedbacks o e 2000–10 and Compa isons o
Clima e Models , Jou nal o Clima e 26, 333-342, doi: 10.1175/JCLI-D-11-00640.1
Duong, T. and Hazel on, M. L., 2005. C oss- alida ion bandwid h ma ices o mul i a ia e ke nel
densi y es ima ion, Scandina ian Jou nal o S a is ics 32, 485-506, doi:10.1111/j.1467-
9469.2005.00445.x
E as i, I., Ezcu a, A., Sáenz, J., Iba a-Be as egi, G., 2011. E alua ion o IPCC AR4 models o e
he Ibe ian Peninsula, Theo e ical and Applied Clima ology, 103, 61-79, doi: 10.1007/s00704-010-
0282-y
E as i, I., Ezcu a, A., Sáenz, J., Iba a-Be as egi, G., Zo i a, E., 2013. Compa ison o he main
cha ac e is ics o he daily zonally a e aged su ace ai empe a u e as ep esen ed by eanalysis
and se en CMIP3 models, Theo e ical and Applied Clima ology, 114, 417-436, doi:
10.1007/s00704-013-0842-z
Fa away, J. J., Jhun, M., 1990. Boo s ap choice o bandwid h o densi y es ima ion. Jou nal o he
Ame ican S a is ical Associa ion, 85, 1119-1122
36
Fasano, G., F anceschini, A. 1987. A mul idimensional e sion o he Kolmogo o –Smi no es ,
Mon hly No ices o he Royal As onomical Socie y, 225, 155-170, doi: 10.1093/mn as/225.1.155.
Fe nández, J., P imo, C., Co iño, A. S., Gu ié ez, J. M., Rod íguez, M. A. 2009. MVL
spa io empo al analysis o model in e compa ison in EPS: applica ion o he DEMETER mul i-
model ensemble. Clima e Dynamics, 33, 233-243, doi: 10.1007/s00382-008-0456-9
F ede iksen, J. S., Collie , M. A., Wa kins, A. B. 2004. Ensemble p edic ion o blocking egime
ansi ions. Tellus, 56A, 485-500, u l: h p://www. ellusa.ne /index.php/ ellusa/a icle/ iew/14460.
Fu, G., Liu, Z., Cha les, S. P., Xu, Z., Yao, Z., 2013. A sco e-based me hod o assessing he
pe o mance o GCMs: A case s udy o sou heas e n Aus alia. Jou nal o Geophysical esea ch,
118, 4145-4167, doi: 10.1002/jg d.50269
Ga es, W. L. , Boyle, J. S., Co ey, C., Dease, C. G., Dou iaux, C. M., D ach, R. S., Fio ino, M.,
Gleckle , P.J., Hnilo, J. J., Ma lais, S. M., Phillips, T. J., Po e , G. L., San e , B. D., Spe be , K. R.,
Taylo , K. E., Williams, D. N. 1999. An O e iew o he Resul s o he A mosphe ic Model
In e compa ison P ojec (AMIP I). Bull. Ame . Me eo . Soc., 80, 29–55,
doi: 10.1175/1520-0477(1999)080<0029:AOOTRO>2.0.CO;2
Gleckle , P. J., Taylo , K. E., Dou iaux, C., 2008. Pe o mance me ics o clima e models. Jou nal
o Geophysical Resea ch 113, D22105, doi: 10.1029/2007JD008972
Guinehu S., Le T aon, P.-Y., La nicol, G., Philipps, S., 2004. Combining A go and emo e-sensing
da a o es ima e he ocean h ee-dimensional empe a u e ields - A i s app oach based on
simula ed obse a ions. Jou nal o Ma ine Sys ems, 46 (1-4), 85-98, doi:
10.1016/j.jma sys.2003.11.022
Guinehu S., Dhomps, A.-L., La nicol, G., Le T aon, P.-Y., 2012. High esolu ion 3D empe a u e
and salini y ields de i ed om in si u and sa elli e obse a ions. Ocean Science, 8(5):845–857,
doi:10.5194/os-8-845-2012
Hende son-Selle s, A., Pi man, A. J., Lo e, P. K., I annejad, P., Chen, T. 1995. The p ojec o
37
In e compa ison o land su ace pa ame e isa on schemes PILPS) Phases 2 and 3. Bull. Ame .
Me eo . Soc., 76, 489-503, doi: 10.1175/1520-0477(1995)076<0489:TPFIOL>2.0.CO;2
Hewi , C. D. and G iggs, D. J. 2004. Ensembles-based p edic ions o clima e changes and hei
impac s, Eos T ansac ions o he AGU, 85, 566–566, doi:10.1029/2004EO520005.
Hind, A., Mobe g, A. Sundbe g, R. 2012. S a is ical amewo k o e alua ion o clima e model
simula ions by u se o clima e p oxy da a om he las millennium – Pa 2: A pseudo-p oxy s udy
add essing he ampli ude o sola o cing, Clima e o he Pas , 8 1355-1365,
doi: 10.5194/cp-8-1355-2012.
Jus el, A., Peña, D., Zama , R. 1997. A mul i a ia e Kolmogo o -Smi no es o goodness o i .
S a is ics and P obabili y Le e s, 35, 251-259, doi: 10.1016/S0167-7152(97)00020-5
Kageyama, M., Valdes, P. J., Rams ein, G., Hewi , C., Wypu a, U. 1999. No he n Hemisphe e
S o m T acks in P esen Day and Las Glacial Maximum Clima e Simula ions: A Compa ison o he
Eu opean PMIP Models. Jou nal o Clima e, 12, 742–760,
doi: 10.1175/1520-0442(1999)012<0742:NHSTIP>2.0.CO;2
Kalnay, E. , Kanami su, M. , Kis le , R. , Collins, W. , Dea en, D. , Gandin, L. , I edell, M. , Saha,
S. , Whi e, G. , Woollen, J. , Zhu, Y. , Lee maa, A. , Reynolds, R. , Chelliah, M. , Ebisuzaki, W. ,
Higgins, W. , Janowiak, J. , Mo, K. C. , Ropelewski, C. , Wang, J. , Jenne, R., Joseph, D., 1996. The
NCEP/NCAR 40-yea eanalysis p ojec , Bulle in o he Ame ican Me eo ological Socie y, 77, 437-
470. doi: 10.1175/1520-0477(1996)077<0437:TNYRP>2.0.CO;2
Knu i, R., 2008. Should we belie e model p edic ions o u u e clima e change? Philosophical
T ansac ions o he Royal Socie y A, 366, 4647-4664, doi: 10.1098/ s a.2008.0169.
Knu i, R., 2010. The end o model democ acy? An edi o ial commen , Clima ic Change 102, 395-
404, doi: 10.1007/s10584-010-9800-2
Knu i, R., Masson, D., Ge elman, A., 2013. Clima e model genealogy: Gene a ion CMIP5 and
how we go he e. Geophysical Resea ch Le e s, 40, 1194-1199, doi:10.1002/g l.50256.
38
Lama que, J.-F., Shindell, D. T., Josse, B., Young, P. J., Cionni, I., Ey ing, V., Be gmann, D.,
Came on-Smi h, P., Collins, W. J., Dohe y, R., Dalso en, S., Falu egi, G., Folbe h, G., Ghan, S. J.,
Ho owi z, L. W., Lee, Y. H., MacKenzie, I. A., Nagashima, T., Naik, V., Plumme , D., Righi, M.,
Rumbold, S. T., Schulz, M., Skeie, R. B., S e enson, D. S., S ode, S., Sudo, K., Szopa, S.,
Voulga akis, A., and Zeng, G. 2013. The A mosphe ic Chemis y and Clima e Model
In e compa ison P ojec (ACCMIP): o e iew and desc ip ion o models, simula ions and clima e
diagnos ics, Geosci. Model De ., 6, 179-206, doi:10.5194/gmd-6-179-2013.
Lopes, R. H. C., Hobson, P. R., Reid, I. D. 2008. Compu a ionally e icien algo i hms o he wo-
dimensional Kolmogo o –Smi no es . Jou nal o Physics: Con e ence Se ies, 119, 042019,
doi:10.1088/1742-6596/119/4/0420
Lo enz, E. N. 2006. P edic abili y, a p oblem pa ly sol ed, Chap e 3 in Palme , T. and Hagedo n,
R. (eds.) P edic abili y o Wea he and Clima e, Camb idge Uni e si y P ess, Camb idge, 702 pp.
Macadam, I., Pi man, A. J., Whe on, P. H., Ab amowi z, G., 2010. Ranking clima e models by
pe o mance using ac ual alues and anomalies: Implica ions o clima e change impac
assessmen s. Geophysical Resea ch Le e s 37, L16704, doi: 10.1029/2010GL043877.
Ma shall, A.G., Hudson, D., Hendon, H.H., Pook, M. Al es,O., Wheele , M. 2013. Simula ion and
p edic ion o blocking in he Aus alian egion and i s in luence on in a-seasonal ain all in
POAMA-2. Clima e Dynamics doi: 10.1007/s00382-013-1974-7.
Masson, D., Knu i, R., 2011. Clima e model genealogy, Geophysical Resea ch Le e s, 38, L08703,
doi: 10.1029/2011GL046864.
Maxino, C. C., McA aney, B. J., Pi man, A. J., Pe kins, S. E., 2008. Ranking he AR4 clima e
models o e he Mu ay-Da ling Basin using simula ed maximum empe a u e, minimum
empe a u e and p ecipi a ion. In e na ional Jou nal o Clima ology, 28, 1097-1112, doi:
10.1002/joc.1612.
Meehl, G. A., Co ey, C., Taylo , K. E., Delwo h, T., S ou e , R. J., La i , M., McA aney, B.,
Mi chell, J. F. B., 2007. The WCRP CMIP3 mul imodel da ase : A new e a in clima e change
esea ch. Bulle in o he Ame ican Me eo ological Socie y, 88, 1383-1394. doi: 10.1175/BAMS-
39
88-9-1383
Menke, W. and Menke, J., 2012, En i onmen al Da a Analysis wi h Ma lab, 263 pages, Else ie ,
Ox o d.
Mobe g, A. 2013. Compa isons o simula ed and obse ed No he n Hemisphe e empe a u e
a ia ions du ing he pas millennium – selec ed lessons lea ned and p oblems encoun e ed: Tellus
B 65, 19921, doi: 10.3402/ ellusb. 65i0.19921.
Nie o S. and Rod íguez-Puebla C. 2006. Compa ison o P ecipi a ion om obse ed da a and
gene al ci cula ion models o e he Ibe ian Peninsula. Jou nal o Clima e, 19: 4254-4275.
No h, G. R., Cahalan, R. F., Coakley, J. A., 1981. Ene gy balance clima e models. Re iew o
Geophysica and Space Physics, 19:91-121, doi: 10.1029/RG019i001p00091
O o, A., O o, F., Bouche , O., Chu ch, J., Hege l, G., Fo s e , P. M., Gille , N. P., G ego y, J.,
Johnson, G. C., Knu i, R., Lewis, N., Lohmann, U., Ma o zke, J., Myh e, G., Shindell, D., S e ens,
B., Allen, M.R. 2013. Ene gy budge cons ain s on clima e esponse, Na u e Geoscience, 6, 415–
416, doi:10.1038/ngeo1836
Peacock, J. A. 1983. Two-dimensional goodness-o - i es ing in as onomy, Mon hly No ices o he
Royal As onomical Socie y, 202, 615-627, doi: 10.1093/mn as/202.3.615.
Pelly, J. L., Hoskins, B. J. 2003. How well does he ECMWF Ensemble P edic ion Sys em p edic
blocking?. Qua e ly Jou nal o he Royal Me eo ological Socie y, 129, 1683–1702. doi:
10.1256/qj.01.173
Pennell, C., Reichle , T., 2011. On he E ec i e Numbe o Clima e Models. Jou nal o Clima e, 24,
2358–2367, doi: 10.1175/2010JCLI3814.1
Pe kins, S. E., Pi man, A. J., N. H. Holb ook, McAneney, J., 2007. E alua ion o he AR4 clima e
models' simula ed maximum empe a u e, minimum empe a u e and p ecipi a ion o e Aus alia
using p obabili y densi y unc ions. Jou nal o Clima e 20, 4356-4376, doi: 10.1175/JCLI4253.1
40
Radić, R., Cla ke, G. K., 2011. E alua ion o IPCC Model's pe o mance in simula ing La e-
Twen ie h-Cen u y clima ologies and wea he pa e ns o e No h Ame ica. Jou nal o Clima e 24,
5257-5274, doi: 10.1175/JCLI-D-11-00011.1
Randall, D. A., Wood, R. A., Bony, S., Colman, R. Fiche e , F., Fy e, J., Ka so , V., Pi man, A.,
Shukla, J., S ini asan, J., S ou e , R. J., Sumi, A., Taylo , K. E., 2007. Clima e models and hei
e alua ion. In: Solomon, S., Qin, D., Manning, Chen, Z., M., Ma quis, M., A e y , K., Tigno , M.
M. B., Mille , H. L., Clima e Change 2007, The Physical Science Basis, Con ibu ion o Wo king
G oup I o he Fou h Assessmen Repo o he In e go e nmen al Panel on Clima e Change,
Camb idge Uni e si y P ess, Camb idge , UK and New Yo k, NY, USA, 996pp.
Reichle , T. and Kim, J. K., 2008. How well do coupled models simula e oday's clima e? Bulle in
o he Ame ican Me eo ological Socie y 89, 303-311, doi: 10.1175/BAMS-89-3-303
Rei en, C. and Toumi, R., 2009. Clima e p ojec ions: pas pe o mance no gua an ee o u u e skill?
Geophysical Resea ch Le e s, 36, L13704, doi: 10.1029/2009GL038082.
Russell J., S ou e R.J. and Dixon K.W. 2006. In e compa ison o he Sou he n Ocean ci cula ions
in IPCC coupled model con ol simula ions. Jou nal o Clima e, 19: 4560–4575.
Saha, S., Moo hi, S., Pan, H., Wu, X., Wang, J., Nadiga, S., T ipp, P., Kis le , R., Woollen, J.,
Beh inge , D., Liu, H., S okes, D., G umbine, R., Gayno, G., Wang, J., Hou, Y., Chuang, H., Juang,
H. H., Sela, J., I edell, M., T eadon, R., Kleis , D., Van Dels , P., Keyse , D., De be , J., Ek, M.,
Meng, J., Wei, H., Yang, R., Lo d, S., Van Den Dool, H., Kuma , A., Wang, W., Long, C., Chelliah,
M., Xue, Y., Huang, B., Schemm, J., Ebisuzaki, W., Lin, R., Xie, P., Chen, M., Zhou, S., Higgins,
W., Zou, C., Liu, Q., Chen, Y., Han, Y., Cucu ull, L., Reynolds, R. W., Ru ledge, G., Goldbe g, M.,
2010. The NCEP Clima e Fo ecas Sys em Reanalysis. Bulle ion o he Ame ican Me eoological
Socie y, 91, 1015–1057, doi: 10.1175/2010BAMS3001.1
Saha, S., Moo hi, S., Wu, X., Wang, J., Nadiga, S., T ipp, P., Beh inge , D., Hou, Y., Chuang, H.,
I edell, M., Ek, M., Meng, J., Yang, R., Mendez Malaquías, P., an den Dool, H., Zhang, Q., Wang,
W., Chen, M., Becke , E., 2014. The NCEP Clima e Fo ecas Sys em Ve sion 2. Jou nal o
Clima e, 27, 2185–2208. doi: 10.1175/JCLI-D-12-00823.1
41
San e , B. D., Mea s, C., Dou iaux, C., Caldwell, P., Gleckle , P. J., Wigley, T. M. L., Solomon, S.,
Gille , N. P., I ano a, D., Ka l, T. R., Lanzan e, J. R., Meehl, G. A., S o , P. A., Taylo , K. E.,
Tho ne, P. W., Wehne , M. F., Wen z, F. J., 2011. Sepa a ing signal and noise in a mosphe ic
empe a u e changes: The impo ance o imescale. Jou nal o Geophysical esea ch, 116, D22105,
doi: 10.1029/2011JD016263.
Schwalm, C. R., Hun inzge , D. N., Michalak, A. M., Fishe , J. B., Kimball, J. S., Muelle , B.,
Zhang, K. and Zhang, Y., 2013. Sensi i i y o in e ed clima e model skill o e alua ion decisions: a
case s udy using CMIP5 e apo anspi a ion. En i onmen al Resea ch Le e s, 8, 024028, doi:
10.1088/1748-9326/8/2/024028
Sco , D. W. 1992. Mul i a ia e Densi y Es ima ion: Theo y, P ac ice and Visualiza ion, John Wiley
and Sons, New Yo k, 336 pp, doi: 10.1002/9780470316849
Sil e man, B. W., 1986. Densi y Es ima ion o S a is ics and da a Analysis, Chapman and Hall,
London, 175pp.
S ewa , D. E. and Leyk, Z., 1994. Meschach: Ma ix Compu a ions in C, Cen e o Ma hema ics
and i s Applica ions, he Aus alian Na ional Uni e si y, Canbe a, Aus alia.
S ocke , T.F., Qin, D., Pla ne , G. K., Tigno , M., Allen, S. K., Boschung, J., Nauels, A., Xia, Y.,
Bex, V., Midgley, P. M. (eds.). 2013. IPCC, 2013: Clima e Change 2013: The Physical Science
Basis. Con ibu ion o Wo king G oup I o he Fi h Assessmen Repo o he In e go e nmen al
Panel on Clima e Change. Camb idge Uni e si y P ess, Camb idge, Uni ed Kingdom and New
Yo k, NY, USA, 1535 pp.
Sundbe g, R., Mobe g, A., and Hind, A. 2012. S a is ical amewo k o e alua ion o clima e model
simula ions by use o clima e p oxy da a om he las millennium – Pa 1: Theo y, Clima e o he
Pas , 8, 1339-1353, doi: 10.5194/cp-8-1339-2012.
Taylo , K. E., 2001. Summa izing mul iple aspec s o model pe o mance in a single diag am,
Jou nal o Geophysical Resea ch, 106, 7183–7192, doi: 10.1029/2000JD900719.
Taylo , K. E., S ou e , R. J., Meehl, G. A., 2012. An o e iew o CMIP5 and he expe imen
