Uncertainty and Variability Analysis of Agent-Based Transport Models

ScienceDi ec A ailable online a www.sciencedi ec .com T anspo a ion Resea ch P ocedia 62 (2022) 719–726 2352-1465 © 2022 The Au ho s. Published by ELSEVIER B.V. This is an open access a icle unde he CC BY-NC-ND license (h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0) Pee - e iew unde esponsibili y o he scien i ic commi ee o he 24 h Eu o Wo king G oup on T anspo a ion Mee ing (EWGT 2021) 10.1016/j. p o.2022.02.089 10.1016/j. p o.2022.02.089 2352-1465 © 2022 The Au ho s. Published by ELSEVIER B.V. This is an open access a icle unde he CC BY-NC-ND license (h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0) Pee - e iew unde esponsibili y o he scien i ic commi ee o he 24 h Eu o Wo king G oup on T anspo a ion Mee ing (EWGT 2021) A ailable online a www.sciencedi ec .com T anspo a ion Resea ch P ocedia 00 (2021) 000–000 www.else ie .com/loca e/p ocedia 24 h Eu o Wo king G oup on T anspo a ion Mee ing, EWGT 2021, 8-10 Sep embe 2021, A ei o, Po ugal Unce ain y and Va iabili y Analysis o Agen -Based T anspo Models Lasse Bienzeisle a,∗, To ben Lelkea, Oska Wageb, Lena-Ma ie Hucka, Be nha d F ied icha aIns i u e o T anspo a ion and U ban Enginee ing, TU B aunschweig, He mann-Blenk-S . 42, 38108 B aunschweig, Ge many bIns i u e o Ca og aphy and Geoin o ma ics, Leibniz Uni e si y Hanno e , Appels . 9a, 30167 Hanno e , Ge many Abs ac This pape p esen s an analysis o he ou pu a iabili y o agen -based anspo models. We simula ed a MATSim model o he ci y o Hano e mul iple imes wi h iden ical inpu and e alua ed he esul ing a el imes on di e en le el o agg ega ion. On a global le el, we obse ed mino a ia ions o a el imes. Howe e , he esul s show an inc eased a ia ion when examining he ou pu on he le el o dis ic s o o indi idual agen s. A ecommenda ion o es ima ing he equi ed numbe o simula ion uns o a s able ou pu o a el ime o he pu posed agg ega ion le el is de i ed om ou case s udy. ©2021 The Au ho s. Published by Else ie B.V. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/) Pee - e iew unde esponsibili y o he scien i ic commi ee o he 24 h Eu o Wo king G oup on T anspo a ion Mee ing. Keywo ds: MATSim; Ou pu Va ia ion; Agen -based T anspo Simula ion 1. In oduc ion Wi h he inc easing ele ance o agen -based simula ions, a ious app oaches ha e been de eloped, wi h MATSim (Ho ni e al.,2016) eme ging as one o he mos equen ly used open-sou ce simula ion amewo ks. MATSim is based on u ili y maximiza ion. Indi idual mobili y decisions on ip pu pose, des ina ion, mode, and ime choice a e calcula ed by econome ic disc e e choice models o ep oduce a ine-g ained a ic demand. The abili y o simula e each agen indi idually enables he conside a ion o complex linkages ac oss mul iple ips. While compe ing wi h all o he agen s o space- ime slo s on he anspo in as uc u e, each agen epea edly op imizes i s daily ac i i y schedule. Op imiza ion is pe o med in an i e a i e cycle wi h a p ede ined ac ion o agen s andomly changing hei plans a each i e a ion. The amewo k e alua es he new plan using a sco ing unc ion a e he subsequen simula ion s ep (Ho ni e al.,2016). ∗Co esponding au ho . E-mail add ess: [email p o ec ed] 2352-1465 ©2021 The Au ho s. Published by Else ie B.V. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/) Pee - e iew unde esponsibili y o he scien i ic commi ee o he 24 h Eu o Wo king G oup on T anspo a ion Mee ing. A ailable online a www.sciencedi ec .com T anspo a ion Resea ch P ocedia 00 (2021) 000–000 www.else ie .com/loca e/p ocedia 24 h Eu o Wo king G oup on T anspo a ion Mee ing, EWGT 2021, 8-10 Sep embe 2021, A ei o, Po ugal Unce ain y and Va iabili y Analysis o Agen -Based T anspo Models Lasse Bienzeisle a,∗, To ben Lelkea, Oska Wageb, Lena-Ma ie Hucka, Be nha d F ied icha aIns i u e o T anspo a ion and U ban Enginee ing, TU B aunschweig, He mann-Blenk-S . 42, 38108 B aunschweig, Ge many bIns i u e o Ca og aphy and Geoin o ma ics, Leibniz Uni e si y Hanno e , Appels . 9a, 30167 Hanno e , Ge many Abs ac This pape p esen s an analysis o he ou pu a iabili y o agen -based anspo models. We simula ed a MATSim model o he ci y o Hano e mul iple imes wi h iden ical inpu and e alua ed he esul ing a el imes on di e en le el o agg ega ion. On a global le el, we obse ed mino a ia ions o a el imes. Howe e , he esul s show an inc eased a ia ion when examining he ou pu on he le el o dis ic s o o indi idual agen s. A ecommenda ion o es ima ing he equi ed numbe o simula ion uns o a s able ou pu o a el ime o he pu posed agg ega ion le el is de i ed om ou case s udy. ©2021 The Au ho s. Published by Else ie B.V. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/) Pee - e iew unde esponsibili y o he scien i ic commi ee o he 24 h Eu o Wo king G oup on T anspo a ion Mee ing. Keywo ds: MATSim; Ou pu Va ia ion; Agen -based T anspo Simula ion 1. In oduc ion Wi h he inc easing ele ance o agen -based simula ions, a ious app oaches ha e been de eloped, wi h MATSim (Ho ni e al.,2016) eme ging as one o he mos equen ly used open-sou ce simula ion amewo ks. MATSim is based on u ili y maximiza ion. Indi idual mobili y decisions on ip pu pose, des ina ion, mode, and ime choice a e calcula ed by econome ic disc e e choice models o ep oduce a ine-g ained a ic demand. The abili y o simula e each agen indi idually enables he conside a ion o complex linkages ac oss mul iple ips. While compe ing wi h all o he agen s o space- ime slo s on he anspo in as uc u e, each agen epea edly op imizes i s daily ac i i y schedule. Op imiza ion is pe o med in an i e a i e cycle wi h a p ede ined ac ion o agen s andomly changing hei plans a each i e a ion. The amewo k e alua es he new plan using a sco ing unc ion a e he subsequen simula ion s ep (Ho ni e al.,2016). ∗Co esponding au ho . E-mail add ess: [email p o ec ed] 2352-1465 ©2021 The Au ho s. Published by Else ie B.V. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/) Pee - e iew unde esponsibili y o he scien i ic commi ee o he 24 h Eu o Wo king G oup on T anspo a ion Mee ing. 720 Lasse Bienzeisle e al. / T anspo a ion Resea ch P ocedia 62 (2022) 719–726 2L. Bienzeisle e al. /T anspo a ion Resea ch P ocedia 00 (2021) 000–000 The a iabili y o he agen s’ choice in MATSim is based on a se ies o pseudo- andom numbe s de e mined by a andom seed (Paulsen e al.,2018). Howe e , due o he coe olu iona y algo i hm, he choices a e no execu ed pseudo- andomly in e e y un o he simula ion (Ho ni e al.,2011). Thus, he esul ing models a e non-de e minis ic. The unce ain y in he e alua ion o di e en simula ion uns is a well-known p oblem ha needs o be conside ed when conduc ing simula ion case s udies (Rasouli and Timme mans,2012). In es iga ing di e en model pa ame e s on di e en le els o agg ega ion (LOA), we obse ed subs an ial a ia ions o di e en measu es. Howe e , he e is a lack o sys ema ic unce ain y analysis o MATSim simula ions o acili a e mo e educa ed decision-making. The p e equisi es o s able simula ion esul s wi h a desi ed eliabili y ha e no ye been in es iga ed o di e en agg ega ion le els. 2. Rela ed Wo k Quan i ica ion o he eliabili y o decisions based on ma hema ical models is a p ominen opic in he ield o anspo modelling. Wi h he eme gence o models o inc easingly complex p oblems ha a e ha d o in e p e , esea che s ha e s a ed e o s o gene alize de ini ions o unce ain y and analysis me hods. Walke e al. (2003) p esen ed a heo e ical amewo k o sys ema ic unce ain y analysis in model-based decision suppo . The ein, un- ce ain y is de ined as ’any depa u e om he unachie able ideal o comple ely de e minis ic knowledge o he ele- an sys em’. The au ho s di e en ia e be ween h ee dimensions o unce ain y, which hey de ine as unce ain y o loca ion,na u e, and le el. Mul iple s udies ha e examined a ia ions in ac i i y-based mic o-simula ions, such as he es ablished Alba oss (A en ze and Timme mans,2004) o Fea he s models (Bao e al.,2015). Acco ding o Baus e (2021), he mos commonly analyzed unce ain y loca ion in hese models is he simula ion e o . Reasons o his a e he ela i e ease wi h which his loca ion can be add essed and he o en s ochas ic na u e o hese models. Cas iglione e al. (2003) s udied he minimum numbe o uns needed o achie e obus a e age esul s. Cools e al. (2011) assessed he impac o mic o-simula ion e o s on he a e age daily numbe o ips pe pe son as well as he a e age daily dis ance a eled pe pe son. Thei esul s show minimal a ia ion, especially o agg ega ed alues. Agen -based mic o-simula ions, such as MATSim, a e pa icula ly p one o model unce ain ies since hey o en ely on disc e e choice models o pe o m mode choice and ip assignmen . Acco ding o Ho ni e al. (2016) and Ho ni e al. (2011), he coe olu iona y algo i hm o MATSim is he majo loca ion o unce ain ies in he simula ion and in e s di e en ypes o unce ain y in oduced by ime, ou e, and des ina ion choice modules. Caused by he andom seed, dis inc unce ain y is in oduced o e e i e a ion o he simula ion. Di e en andom numbe s may lead he op imiza ion algo i hm o ind o he local op ima. Mo eo e , MATSim con ains a andom a iabili y in how he e- planning o plans is handled. Ho ni e al. (2011) demons a ed ha he esul s o simula ions can change signi ican ly be ween mul iple uns. In hei wo k, hey s udied he impac o a ying andom seeds wi h a ocus on link loads in wo di e en MATSim scena ios. They conside ed ha he a ia ion in daily link loads is gene ally low. Howe e , conside ing hou ly alues, he coe icien o a ia ion inc eases. In hei li e a u e e iew hey also concluded ha a e age esul s gene a ed om mic o-simula ions become s able a e ’a ela i ely small numbe o simula ion uns’ (Ho ni e al.,2011, p. 8). These indings we e p oba ed by Paulsen e al. (2018). Chap e 48 o he MATSim book (Fl¨ o e ¨ od,2016) also desc ibes he challenges in MATSims ou pu e alua ion due o he in luence o he choice o one speci ic andom seed and elabo a es he need o u he esea ch in his pa icula a ea. Thus, we s i e o add addi ional le els o in es iga ion o his discussion by analyzing he ou pu ’s a el ime a ia ion o MATSim and expanding ou s udies o ega d di e en LOA. 3. Me hodology To in es iga e he a iabili y o MATSim simula ion ou pu s, we se up a simula ion case s udy o an 10 % model o he ci y o Hano e , Ge many (Bienzeisle e al.,2020). Using he e e enced con igu a ion pa ame e s, we epea edly simula ed he Hano e inpu model wi h 750 i e a ions. The public anspo sys em was implemen ed as a ne wo k mode. In addi ion, comme cial a ic was included in he model using he eigh ex ension o MATSim (Zilske e al., 2012) sepa a ed by di e en b anches. We simula ed 16 simula ion uns wi h he same inpu pa ame e s o explo e inconsis encies ac oss he simula ion ou pu s. Lasse Bienzeisle e al. / T anspo a ion Resea ch P ocedia 62 (2022) 719–726 721 L. Bienzeisle e al. /T anspo a ion Resea ch P ocedia 00 (2021) 000–000 3 A e Paulsen e al. (2018) concen a ed hei wo k on he a ia ion o link loads using di e en andom seeds, we ocused on he a ia ion o a el imes. T a el ime dis ibu ions a e a model cha ac e is ic ha can be used o calib a ion o alida ion. Thus, he e alua ed dimension o he a el ime pe p i a e agen (p) o comme cial a ic ehicle (c ) was de ined as he sum o all ip du a ions pe day. We conside ed he changing a el imes pe un in he se o 16 uns Rpe agen a∈all agen s A o explo e he e ec s o he unce ain ies om he andom choice pa s o he MATSim algo i hm. We assigned he co esponding home dis ic d∈all dis ic s o Hano e D o each agen ap. Th ee agg ega ion le els o he analyzed a el imes we e in oduced as a se o a el imes ∈LOA . The e alua ion o a el imes was ca ied ou sepa a ely o each LOA and each simula ion un ∈R: •LOA1: Global a e age a el ime o Hano e : wi h o a∈A •LOA2: A e age a el ime o each dis ic do Hano e : wi h o a∈d •LOA3: T a el ime o each agen ao Hano e : o all a∈A To quan i y he a ia ion o he a el ime, we applied he coe icien o a ia ion c ( ), which is de ined as he s anda d de ia ion o he sample di ided by he sample mean, on ou de ined LOA. 4. Analysis o he Va ia ion o T a el Times To ob ain a i s unde s anding o he a ia ion o a el imes ac oss he simula ion uns, we s a ed ou wo k by compa ing he equency dis ibu ions o all occu ing a el imes pe agen o he p i a e a ic o each simula ion un sepa a ely. T a el imes we e g ouped in bins o 1 minu e, each wi h hei co esponding equency pe un. Fo a be e compa ison o he esul ing 16 a el ime dis ibu ions, we ha e combined he his og ams in he 3D ba plo shown in Figu e 1. Each ba ep esen s he equency o occu ence o a a el ime g oup pe simula ion un. Fig. 1: Combined His og ams o a el ime in bins o 1 minu e wi h ∈R. The ough su ace depic ed in he igu e p o ides a isual indica ion ha he simula ion gene a es di e en dis- ibu ions o a el imes. Especially in he ange o equen ly occu ing alues, di e en pa e ns can be obse ed. 722 Lasse Bienzeisle e al. / T anspo a ion Resea ch P ocedia 62 (2022) 719–726 4L. Bienzeisle e al. /T anspo a ion Resea ch P ocedia 00 (2021) 000–000 Howe e , he diag am also highligh s ha he e a e uns wi h a simila dis ibu ion o a el imes, whe e he su ace o he plo is cons an and smoo h. This can be obse ed o uns 11 and 12. To de e mine he de ia ion o he simula ion uns, we calcula ed he co esponding Roo Mean Squa e E o (RMSE) and hus compa ed all uns o each o he (see Figu e 2). In mos cases, he RMSE a ies om 12.73 o a maximum o 20.20. No ice ha he e a e uns wi h a RMSE o 0. This indica es an iden ical dis ibu ion o he a el imes o he combina ion o hese speci ic uns. This obse a ion is consis en wi h he i s isual analysis. The simula ions eplica ed exac ly he same a el imes o un 4, 9, 11, 12, 14, and 16. Fig. 2: RMSE-Analysis o he a el ime dis ibu ion o all simula ion uns R. Ano he inding om ou da a is ha he simula ion ou pu s can se le in se e al disc e e s a es o each agen . The numbe o s a es spe agen ais de ined as he numbe o di e en a el ime alues o an agen occu ing o e all simula ion uns. Figu e 3shows he numbe o di e en s a es saoccu ing o e all simula ion uns as a cumula i e dis ibu ion plo . A la ge g oup o p i a e agen s, 21.9 % (n=16.312) has one s a e, i.e. one cons an a el ime o e all uns. In hese cases, he a el ime dis ibu ion does no oscilla e and he speci ic s a e eoccu s in e e y simula ion un. Fo comme cial ehicles, his applies o 3.4 % (n=211) o he agen s. Fig. 3: Empi ical cumula i e dis ibu ion wi h he numbe o di e en s a es saac oss all agen s g ouped by agen ype. As shown, he ou pu o comme cial a ic ehicles di e s mo e. In pa icula cou ie s, exp ess, and pa cel se ice (CEP) ehicles do no se le in disc e e s a es, bu show indi idual a el imes o each un. Howe e , he sample size is signi ican ly smalle (np=74.394, nc =6.148, nCEP =98 ). To explo e his cha ac e is ic o he eigh a ic in MATSim in de ail, we plo ed he equency dis ibu ion o a el imes pe simula ion un in Figu e 4. The mo e homogeneous dis ibu ion o a el imes o comme cial a ic ehicles compa ed o CEP ehicles pe un is e iden . Lasse Bienzeisle e al. / T anspo a ion Resea ch P ocedia 62 (2022) 719–726 723 L. Bienzeisle e al. /T anspo a ion Resea ch P ocedia 00 (2021) 000–000 5 Fig. 4: Dis ibu ion o a el ime o comme cial agen s di e en ia ed by ype wi h ∈R. To analyze he a iabili y o a el imes in de ail, we in oduced he coe icien o a ia ion c ( ) o di e en LOA as a measu e o a iabili y and applied i o ou da a se . Figu e 5illus a es he cha ac e is ics ha led o a pa icula ly high c ( )LOA3in ou simula ion case s udy. Fig. 5: Dis ibu ion o he a ia ion coe icien c ( ) in ela ion o he a e age a el ime combined wi h he empi ical cumula i e dis ibu ion unc ion o he a ia ion coe icien c ( ) wi h ∈R. The a ious a el ime dis ibu ions obse ed p e iously a e e iden in he a ia ion o measu ed a el ime alues ac oss all agen s and simula ion uns. The sca e plo indica es ha highe c ( ) alues usually occu a lowe a e age a el imes. Since he analysis o a el ime equencies shows ha mos o he agen s’ a el ime end o decline wi hin his ange o lowe a el imes, he obse ed clus e ing can be pa ly explained by he co espondingly la ge sample size. I is appa en ha se e al agen ’s a el imes a ies conside ably be ween he simula ion uns. The maximum alues c ( ) di e signi ican ly be ween agen ypes, i.e. c ( p)(max)=1.212 , c ( c )(max)=0.917 and c ( cep)(max)=0.230. In o al, only nine agen s show a alue o c ( )>1. A c ( )>0.5 can be obse ed o 741 agen s (0.1 %). Compa ing agen ypes, he a el imes o p i a e agen s show up he highes a e o de ia ion. This end is also e iden in he cumula i e equency dis ibu ion o c ( ). Analogous o he numbe o di e en s a es pe agen , 71.0 % o he agen s o he indi idual a ic(n=52.820) ha e a alue o c ( p)>0.1 ac oss all simula ion uns. Fo CEP- ehicles, i is close o 89.6 % (n=88). The co esponding esul s indica e ha all o hese ehicles change hei a el ime in e e y simula ion un. Howe e , his a iance is smalle compa ed o he o he agen ypes and he esul ing a el imes a e mo e consis en . 724 Lasse Bienzeisle e al. / T anspo a ion Resea ch P ocedia 62 (2022) 719–726 6L. Bienzeisle e al. /T anspo a ion Resea ch P ocedia 00 (2021) 000–000 Table 1: Resul ing dis ibu ion o he coe icien o a ia ion c ( ) wi h ∈R. Le el o agg ega ion Coe icien o a ia ion S anda d de ia ion σc ( )Q1c ( )Medianc ( )Q3c ( ) c ( )|c ( ) LOA1 To al p i a e agen s o Hano e 0.0015 - - - - - To al comme cial agen s o Hano e 0.0009 - - - - - LOA2 P i a e agen s pe dis ic s - 0.0058 0.0031 0.0037 0.0049 0.0064 LOA3 Indi idual agen s o p i a e a ic - 0.0818 0.1074 0.0028 0.0449 0.1156 Indi idual agen s o comme cial a ic - 0.0437 0.0579 0.0092 0.0250 0.0546 A e we we e able o show ha di e en a el ime dis ibu ions occu using iden ical simula ion inpu , we s a ed in es iga ing he h esholds o s able simula ion esul s. We de e mined he alue o c ( ) o each agen on LOA3 di e en ia ed by ype. CEP ehicles we e included in comme cial a ic. Fo a be e compa abili y, we a e aged c ( ) ac oss all agen s. E alua ing LOA2on dis ic le el, we only included p i a e agen s because comme cial ehicles usually s a a speci ic companies wi h hei ou and a e he e o e no so widely dis ibu ed o e he simula ion a ea. The a el imes o he co esponding agen s we e a e aged pe dis ic and he a iabili y o his a e age alue was examined and a mean alue o c ( ) wi h ∈dwas calcula ed ac oss all dis ic s. LOA1is he a ia ion o he global a e age a el ime ac oss all simula ion uns di e en ia ed by indi idual and comme cial agen s. The esul s a e summa ized in Table 1. Fo LOA2and LOA3s a is ical pa ame e s o he dis ibu ion o c ( ) a e p o ided since a single c ( ) alue was calcula ed o each dis ic o agen o Hano e . Ou esul s suppo ou ini ial assump ions and he indings om he li e a u e e iew. The global mean o he a e age a el imes o all p i a e agen s om Hano e emains almos cons an o e all simula ions uns c ( p)LOA1= 0.0015. The a ia ion o he comme cial agen a el imes a e smalle wi h a c ( c )LOA1=0.0009. Obse ed a ia ion a dis ic le el inc eases sligh ly c ( p)LOA2=0.0058 and he analysis o each agen indi idually esul s in he highes obse ed a ia ion o a el imes c ( p)LOA3=0.0818. As a compa ison o he a ia ion o he agg ega ed a el imes pe dis ic and he co esponding a ia ion o he agen s li ing in his dis ic , we g ouped he c ( p)LOA3 alues by he agen ’s home dis ic (Figu e 6). Fig. 6: Boxplo o a ia ion o indi idual a el imes C ( ) g ouped by he agen ’s home loca ion wi h ∈R. Subsequen ly, we compa ed he indi idual agen a el ime a iabili y on LOA3wi h he a iabili y o he agg e- ga ed a el imes on LOA2. The mean dis ibu ion o a el imes o all agen s li ing in he co esponding dis ic a ies be ween c ( p)LOA1(min)=0.048 and c ( p)LOA1(max)=0.114. The co esponding esul s o c ( p)LOA2a e 0.007 Lasse Bienzeisle e al. / T anspo a ion Resea ch P ocedia 62 (2022) 719–726 725 L. Bienzeisle e al. /T anspo a ion Resea ch P ocedia 00 (2021) 000–000 7 and 0.005. Al hough he agen s o a dis ic show a a ia ion o hei co esponding a el imes o e all simula ion uns, he agg ega e a el ime o all esiden s o he dis ic a ies less. The esul s o ou case s udy show a com- pensa ion o he a ia ions o a el imes o indi idual agen s on he agg ega e dimension o dis ic s. Thus, he mo e agg ega ed e alua ion alues a e s abilizing a he as a one le el. The esul s imply ha a p edic ion abou hese global pa ame e s, especially on LOA1and LOA2, can be made using an a e age alue o only a ew simula ion uns. Fo p ac ical wo k wi h MATSim, i is o in e es how many simula ion uns a e necessa y o de e mine he adequa e alue o he co esponding LOA wi h a desi ed accu acy. 5. P edic ion o Requi ed Numbe o Simula ion Runs To allow he de i a ion o gene ally alid indica ions om ou esul s, we in es iga ed how many simula ions a e necessa y o a i e a obus mean alues a he h ee agg ega ion le els de ined. We applied he con e gence o subsequen mean alues n−→ c o ou da a se by o ming a mo ing mean alue nwi h a p og essing numbe o simula ions. As soon as he de ia ion o he calcula ed mean alue o he con e gence mean alue cwas less han one pe cen , we conside ed he ob ained mean alue o be obus . Howe e , he 16 simula ion uns we pe o med we e no su icien o achie e a obus mean alue. Despi e his, o p edic he numbe o simula ions a which a obus mean is eached, we used ou obse ed a el ime dis ibu ions o each agen o gene a e a i icial simula ion esul s. This p ocess was con inuously epea ed o eplica e he obse ed a el ime dis ibu ion. We conside his me hodology o be alid because unning a la ge se o simula ions wi h MATSim o explo e he needed numbe o simula ion uns is no p ac ical due o he compa a i ely long compu a ion imes. The calcula ed dis ibu ions a e summa ized in Figu e 7. Ou i s in es iga ions indica e ha he a el ime alues on LOA1and LOA2a e al eady obus a e one i e a ion. Thus, his obus ness occu s o agg ega ed esul s. Fo LOA3,p31 simula ions we e in a e age su icien o each a obus mean. A he maximum 36 uns we e necessa y. Addi ionally, he g aph shows he de elopmen o he mean alue con e gence o comme cial ehicles and, as a subse o his, o CEP- ehicles. The unc ion o LOA3,c de elops simila o LOA3,pwi h a wide ange o a ia ion, e en hough he he obus mean alue was in a e age eached ea lie a e 20 uns. Fig. 7: Rela i e de ia ion om calcula ed con e gence mean alues o di e en LOA. 6. Conclusion and Fu u e Wo k The a el imes o a MATSim simula ion a y despi e cons an inpu pa ame e s. We de eloped a ecommenda ion o he needed numbe o simula ion uns acco ding o di e en agg ega ion le els. Ou aim was o ob ain esul s wi h a desi ed eliabili y o one pe cen de ia ion om he p edic ed a el ime alues. Acco dingly, he a ia ion o he a el imes dec eases as he agg ega ion le el inc eases, while global agg ega ed pa ame e s such as he a e age a el ime emain app oxima ely cons an h oughou he simula ion. By analyzing he con e ging a e age, we we e able o show ha a single simula ion is su icien o an agg ega ed e alua ion o a el imes. These esul s a e in 726 Lasse Bienzeisle e al. / T anspo a ion Resea ch P ocedia 62 (2022) 719–726 8L. Bienzeisle e al. /T anspo a ion Resea ch P ocedia 00 (2021) 000–000 line wi h p e ious con ibu ions in his esea ch a ea, since agg ega ed mac oscopic da a is o en used o alida e MATSim model esul s (Kagho e al.,2020). Howe e , he analysis o he coe icien o a ia ion also showed a ying a el imes o indi idual agen s pe simula ion un. This can be pa icula ly impo an when e alua ing simula ions ocused on speci ic popula ion g oups wi h compa a i ely small sample sizes. A possible e alua ion case applies o CEP a ic. These ehicles a e pa o comme cial anspo and hus ha e a small numbe o ehicles compa ed o he p i a e a ic. Ou simula ions illus a ed ha he a el imes o he eigh agen s end o be ela i ely cons an , al hough he a el imes o he CEP ehicles s ill a ies. Ou lie s can change he o e all esul due o he small size o he sample. Fo hese sample sizes ou esul s lead us o ecommend o a e age a leas he esul s o wo simula ion uns o educe he a iabili y o he e alua ed a el imes. MATSim simula ion uns a e compu a ionally expensi e. Due o his, MATSim models a e o en scaled down. The a ia ion o he indi idual agen a el imes on LOA3 hus has a highe in luence on he agg ega ed alues and leads o an inhe en e o . Consequen ly, ou indings suppo he wo k o Llo ca and Moeckel (2019), who obse ed di e en a el ime dis ibu ions o smalle scale ac o s. The objec i e o ou u u e wo k is o p o ide an o e iew o he a iance o a MATSim model in ela ion o he de ined le el o agg ega ion o allow mo e accu a e e alua ions wi h MATSim. T a el imes a y depending on he agen ypes. Thus, i is app op ia e o in es iga e a ibu es causing a co esponding a iabili y and inally p edic ing he expec ed e o o ce ain g oups o agen s. In addi ion, ou a i icial gene a ion o a el ime dis ibu ions, espec- i ely simula ion uns, mus be alida ed wi h u he simula ion uns in o de o be able o de e mine he p edic ed alues mo e p ecisely. Acknowledgemen The scien i ic esea ch published in his a icle is g an ed by he Fede al Minis y o Educa ion and Resea ch Ge many o p ojec s USE UL and USE UL XT (g an ID 03SF0547 & 03SF0609). The au ho s co dially hank he pa ne s and unding agency. Re e ences A en ze, T., Timme mans, H., 2004. Alba oss: A lea ning based anspo a ion o ien ed simula ion sys em. T anspo a ion Resea ch Pa B: Me hodological 38, 613–633. Bao, Q., Kochan, B., Bellemans, T., Janssens, D., We s, G., 2015. In es iga ing mic o-simula ion e o in ac i i y-based a el demand o e- cas ing: a case s udy o he ea he s amewo k. T anspo a ion Planning and Technology 38, 425–441. Baus e , P., 2021. De elopmen o an unce ain y analysis amewo k o model-based consequen ial li e cycle assessmen : Applica ion o ac i i y-based modelling and li e cycle assessmen o mul imodal mobili y. Ph.D. hesis. Buil En i onmen . Bienzeisle , L., Lelke, T., Wage, O., Thiel, F., F ied ich, B., 2020. De elopmen o an agen -based anspo model o he ci y o hano e using empi ical mobili y da a and da a usion. T anspo a ion Resea ch P ocedia 47, 99–106. Cas iglione, J., F eedman, J., B adley, M., 2003. Sys ema ic in es iga ion o a iabili y due o andom simula ion e o in an ac i i y-based mic osimula ion o ecas ing model. T anspo a ion Resea ch Reco d 1831, 76–88. Cools, M., Kochan, B., Bellemans, T., Janssens, D., We s, G., 2011. Assessmen o he e ec o mic o-simula ion e o on key a el indices: e idence om he ac i i y-based model ea he s. P oceedings o he 90 h Annual Mee ing o he T anspo a ion Resea ch Boa d . Fl¨ o e ¨ od, G., 2016. Ma sim as a mon e ca lo engine. The Mul i-Agen T anspo Simula ion MATSim , 327–336. Ho ni, A., Cha ypa , D., Axhausen, K.W., 2011. Va iabili y in anspo mic osimula ions in es iga ed wi h he mul i-agen anspo simula ion ma sim. A bei sbe ich e Ve keh s- und Raumplanung 692. Ho ni, A., Nagel, K., Axhausen, K.W., 2016. The Mul i-Agen T anspo Simula ion MATSim. Ubiqui y P ess, London, GBR. Kagho, G., Balac, M., Axhausen, K., 2020. Agen -based models in anspo planning: Cu en s a e, issues, and expec a ions. P ocedia Compu e Science 170, 726–732. Llo ca, C., Moeckel, R., 2019. E ec s o scaling down he popula ion o agen -based a ic simula ions. P ocedia Compu e Science 151, 782–787. Paulsen, M., Rasmussen, T.K., Nielsen, O.A., 2018. Ou pu a iabili y caused by andom seeds in a mul i-agen anspo simula ion model. P ocedia Compu e Science 130, 850–857. Rasouli, S., Timme mans, H., 2012. Unce ain y in a el demand o ecas ing models: li e a u e e iew and esea ch agenda. T anspo a ion Le e s 4, 55–73. Walke , W., Ha emo¨ es, P., Ro mans, J., Sluijs, J., Assel , M., Janssen, P., K aus, M., 2003. De ining unce ain y: A concep ual basis o unce ain y managemen in model-based decision suppo . In eg a ed Assessmen 4. Zilske, M., Sch ¨ oe , S., Nagel, K., Lied ke, G., 2012. Adding eigh a ic o ma sim. VSP Wo king Pape 12-02 .