An evaluation methodology for reliable simulation based studies of routing protocols in VANETs

An Evaluation Methodology for Reliable Simulation Based Studies of Routing Protocols in VANETs Authors: J.M. García-Campos, J. Sánchez-García, D. G. Reina, S. L. Toral, F. Barrero University of Seville (Spain) Escuela Superior de Ingenieros. Avda. Camino de los Descubrimientos, s/n 41092 Sevilla (Spain) email: josga[email protected], [email protected], [email protected], [email protected], [email protected] phone: +34 954 48 12 93; fax: +34 954 48 73 73 Corresponding author: S. L. Toral 1 An Evaluation Methodology for Reliable Simulation Based Studies of Routing Protocols in VANETs Abstract: Vehicular Ad hoc networks (VANETs) have attracted much attention in the last decade. Many routing protocols have been proposed for VANETs and their performance is usually evaluated and compared using simulation-based studies. However, conducting reliable simulation studies is not a trivial task since many simulation parameters must be configured correctly. The selected parameters configuration can considerably affect the simulation results. This paper presents a methodology for conducting reliable simulations of routing protocols in VANETs urban scenarios. The proposed methodology includes relevant simulation aspects such as measurement period, selection of source-destination pairs for the communication traffic flows, number of simulations, mobility models based on road city maps, performance metrics and different analyses to evaluate routing protocols under different conditions. The proposed methodology is validated by comparing the simulation results obtained for Ad Hoc On-Demand Distance Vector (AODV) routing protocol with and without using the proposed methodology. The obtained results confirm that by using the proposed methodology, we can achieve more reliable simulations of VANETs routing protocols. Keywords: Simulation studies, Performance evaluation, Routing protocols, VANETs. 2 1. Introduction An ad hoc network is an autonomous system of wireless nodes that cooperatively form a network without any specific administration [1]. Each node in an ad hoc network is in charge of routing information among its neighbours. When nodes are free to move randomly and organize themselves arbitrarily, we refer to them as Mobile Ad Hoc NETworks (MANETs) [2]. As an evolution of traditional MANETs, VANETs (Vehicular Ad hoc NETworks) [3] include communications between vehicles (Vehicle to Vehicle, also known as V2V communications) on the roads and with the road communication infrastructure (Vehicle to Infrastructure, also known as V2I communications). In these dynamic networks in which the topology is continuously changing, it is crucial to analyze how routing protocols react against reach possible changes in the network topology. Many routing protocols have been proposed and evaluated for both MANETs and VANETs [4]. The evaluation of MANETs and VANETs routing protocols by simulation is the most common approach for testing the protocols performance so far. The reason is that real experimentation in multi-hop ad hoc networks is costly in terms of hardware requirements [5][6]. Testing routing protocols with real MANETs and VANETs prototypes require a high number of wireless devices, and consequently, there are only a few available testbeds in the world [7]. Therefore, conducting reliable simulation studies is an important requirement to validate the performance of routing protocols for VANETs. MANET simulations have suffered from credibility problems for the last decades [8], mainly because of bad simulation practices conducted by the research community. This lack of credibility is also observable in VANET simulations since most MANET routing protocols researchers are also working on VANET routing protocols design [9]. The goal of this 3 paper is to propose an evaluation methodology to improve the reliability of VANET routing protocols simulation results. The main contributions of this paper are as follows:  A methodology for simulation-based studies of routing protocols for VANETs based on good simulation practices. It includes many simulation aspects such as Warm Up periods, selection of communication pairs, number of simulations, mobility models, performance metrics and simulation analyses. The selection of these simulation parameters is very important to make a fair and unbiased comparison of routing protocols.  The validation of the proposed methodology by the comparison of the performance of several well-known reactive routing protocols such as AODV [10], LAR [11], and DYMO [12] with and without using the proposed methodology. This paper is organized as follows: section 2 presents relevant related works on the evaluation methodology of routing protocols for MANET and VANET scenarios. Section 3 describes the proposed methodology. Section 4 includes the validation of the proposed methodology using AODV, LAR and DYMO routing protocols. Finally, section 5 contains the main conclusions of this work. 2. Related work One of the pioneering works highlighting the importance of conducting reliable simulations in wireless multi-hop ad hoc networks can be found in [13]. The authors report the bad practices followed by researchers in a high number of simulation studies for MANETs. The main findings of this study are, i) there is a high dissimilarity among 4 the simulation scenarios in terms of density, size of the scenarios and mobility, and all of them impact significantly on the obtained simulation results; ii) the mobility of nodes do not model real scenarios since most studies use the random waypoint mobility model, which has been reported to be harmful for simulation studies [14], and iii) the execution of the simulations and data collection are not properly carried out for a statistical analysis. As a conclusion, the authors in [13] provide some guidelines to run reliable simulations. As a solution for the dissimilarity in the simulation scenarios, the same authors provide in [15] a mechanism to generate standard scenarios for MANETs based on the number of nodes and the size of the scenarios. However, this mechanism is only valid for random waypoint mobility model. In [16], the authors also review the main issues in MANET simulations studies. Additionally, the authors put in evidence the discrepancies in the simulation results obtained from different network simulators. Moreover, they indicate an issue related to the traffic pattern generation and its impact in the obtained simulation results. This issue is specifically addressed by the evaluation methodology proposed in this paper. On this line, in [8] the authors evaluate the performance of several well-known routing protocols with different traffic patterns. They highlight the impact of the repetition of the source nodes in the communication pairs. In this paper, we also address this issue, but we take a step forward and we evaluate the impact of the repetition of the destination nodes in the selected communication source-destination pairs. Actually, we demonstrate that the repetition of destination nodes has a higher impact on the simulation results than the repetition of the source nodes. In a more recent work presented in [17], the authors state that still a high number of simulation studies about routing protocols for wireless multi-hop ad hoc networks do not follow good simulation practices. Among the bad practices described in 5 [17], the authors underline the random selection of source-destination pairs in the traffic patterns as an issue for the evaluation of routing protocols. This paper addresses many of the issues indicated in the previous works on the evaluation of routing protocols and proposes an evaluation methodology to solve many of them. It includes simulation aspects such as Warm Up period, selection of source-destination pairs, mobility models, number of simulations, and selection of simulation analyses. All the aforementioned simulation parameters must be selected carefully in order to obtain reliable simulation results. 3. The proposed methodology for reliable simulations in VANETs This section describes the proposed methodology that evaluates how the simulation parameters must be selected for obtaining reliable simulation results. Figure 1 shows the main points of the proposed methodology such as the communication pair selection, the measurement period, the selection of number of simulations, the selection of a mobility model, the performance metrics and the analyses. The main objectives of the proposed methodology are, 1) firstly, to highlight relevant simulation parameters that affect the simulation results, and 2) to provide guidelines on the selection of these parameters to obtain a low disperse set of samples in order to obtain reliable statistics. In the next subsection, we describe in more detail each point of the proposed methodology. 6 Figure 1 Main aspects of the proposed methodology 1) Measurement period In this subsection we introduce an important concept to carry out good simulations for VANETs. The Warm Up period (W.P. in Figure 2) is the time frame which ensures the stability of three relevant simulation aspects such as:  All communication pairs have started transmitting application packets.  The mobility model has achieved a stable state.  The buffers of the nodes have stabilized. Notice that the establishment of communications among a source-destination pair can start at different times. Normally, the starting times are selected randomly, so some pairs could have more time to transmit data packets than others. This fact can influence the simulation results if the selected pairs do not have the same properties in terms of average number of hops and path availability between the source and the destination nodes. By using a Warm Up period, we avoid discrepancies among the measurement period of the performance metrics during the simulation time. To obtain reliable and 7 non-dispersed simulation results, performance metrics must be measured from W.P. value to the end of the simulation period, which is named after Measurement Simulation Period (M.S.P. in Figure 1). In order to select the Warm Up period, we have to consider two aspects, the first one is the period during which the communication flows are established, and the second aspect is the mobility model, since we need to guarantee that the mobility of nodes is stable. Figure 2 Status time bar . . . 8 In order to show the importance of the Warm Up period values on the simulation results, we depict in Figure 3 the throughput performance metric for different Warm up values. We consider an urban scenario (4000x4000 m2) corresponding to a fragment of the city of Washington D.C with 125 nodes moving with the IDM (Intelligent Driver Model) [18] mobility model. From now on, we will refer to this scenario as the scenario under test that will be used to test several features of the proposed methodology. We use the throughput performance metric as standard metric. It is defined as the number of application packets delivered within the simulation time. In the simulations, all source nodes start generating application packets from the period between 0 and 50 seconds until the end of the simulation. The results shown in Figure 3 have been obtained using 25 different pairs selected randomly in the scenario under test. We include the 95% confidence intervals in Figure 3. From now on, similar confidence intervals are shown in this paper for the rest of figures. We can observe in Figure 3 that when the Warm Up value increases, the throughput also increases and this is because the number of source nodes that have started to transmit application packets is higher. When the Warm Up value is higher than 50 (See 75 and 100 in Figure 3) the throughput values are very similar (See Table 1 for more details). Regarding the confidence intervals for each Warm Up value, they are high because the source and destination pairs are selected randomly. We highlight the importance of the communication pair selection in the next subsection. In consequence, the Warm Up value selection only affects the mean while the dispersion does not depend on this parameter (the dispersion is the descriptive statistic used by researchers in order to compare the performance of routing protocols). From now on, we consider 50s as the Warm Up value recommended for the simulation of the scenario under test. 15 least 20 % in order to have enough number of source-destination pairs to be selected. This condition is marked in Figure 6 with black points. Figure 6 Inverse cumulative distribution function of the APA for different density levels Table 3 Target APA values Scenario Target APA 50 nodes 0.370 75 nodes 0.452 100 nodes 0.811 125 nodes 0.846 150 nodes 1.0 175 nodes 1.0 Following the APA condition described above, the Table 3 contains the minimum target APA values that accomplish such condition for each density level. The next step in the communication pair selection is to select those that are separated by the same number of hops on average. Notice that the APA value provides an idea about the availability of a communication path between the source node and the destination 16 node, but it does not take into account the separation in terms of the number of hops in the scenario under test (See Figure 7). Figure 7 Number of hops distribution for the scenario under test The minimum number of hops to reach a destination node is 1, this situation corresponds to the situation in which the destination node is within the source node coverage, and the maximum number of hops is 7. To ensure a reliable and fair evaluation of the routing protocols, we should fix a similar number of hops for each selected communication pair since the number of hops impacts significantly in the obtained simulation results as shown in Figure 8. Figure 8 shows that the throughput of the network is reduced (See Table 4 for more details) as the number of hops in the selected communication pairs increases. In order to illustrate the importance of the separation in number of hops, we use another well-known performance metric such as the NRL (Normalized Routing Load) (red line in Figure 8). This metric is defined as the 17 ratio between routing packets and the total number of delivered application packets. In Figure 8, the NRL increases (See Table 4 for more details) as the number of hops increases because the routing protocol needs to generate a larger number of control packets to discover/maintain the routes. This metric is directly related to the energy consumption [21]. The confidence intervals for both metrics also increase (blue and red vertical lines in Figure 8 respectively) as the number of hops between source and destination nodes increase. If we select source and destination pairs randomly, we could pick pairs separated by very different hop numbers. Consequently, the simulation results will be more disperse. Let us consider an example of this situation in which we focus on the NRL metric. If we select randomly two pairs separated by very different numbers of hops such as 2 and 7, the NRL values obtained for these values are 1.24 and 9.96 respectively (See Table 4 for more details). The mean of both values will be 5.6 and the confidence interval 8.54. Notice that the obtained confidence interval is too high compared with the mean. Consequently, the obtained mean does not reflect the performance of AODV in the scenario under test. 18 Figure 8 Throughput and NRL vs number of hops in the scenario under test Table 4 Statistics measures for Throughput vs number of hops Number of hops 1 2 3 4 5 6 7 Throughput (Kbps) Mean 0.8835 0.8812 0.8629 0.8484 0.8189 0.8111 0.7820 Confidence interval 0.0033 0.0031 0.0063 0.0066 0.0104 0.0128 0.0143 NRL Mean 0.5006 1.2448 3.4239 4.1562 7.2428 8.4652 9.9593 Confidence interval 0.0169 0.0251 0.0449 0.0390 0.0704 0.0858 0.3664 Following the same procedure for the APA selection, we need a selection mechanism to choose communication pairs with similar separation in terms of the number of hops. Again, we use the inverse cumulative distribution (1-CDF) for different density levels in order to get more insight into the distribution of the number of hops. The obtained results are shown in Figure 9. 19 Figure 9 Inverse cumulative distribution function of the number of hops for different density levels In this case, we should select the highest possible number of hops since we are interested in evaluating the performance of routing protocols, but we must also ensure that there are enough source destination pairs. For this reason, we should focus on the most restrictive case, which corresponds to the density level of 50 nodes, as it is very difficult to select communication pairs that are separated by more than 2 hops (See Figure 9). As with the APA value, we state that the 1-CDF value of the target number of hops for a given scenario should be at least 20 % in order to have enough number of source-destination pairs. This condition is marked in Figure 9 with black points. Table 5 shows the target number of hops values that accomplish a minimum number of pairs with that condition in the scenario under test. From here on, when we refer to the proposed methodology we must ensure that the two above conditions (APA and number of hops) are fulfilled. 20 Table 5 Target hops values Scenario Target Hops 50 nodes 2 75 nodes 2 100 nodes 4 125 nodes 6 150 nodes 7 175 nodes 9 Another important feature that should be controlled in the selection of communication pairs is the possibility of several communication pairs having the same source and destination nodes. Although the actual source and destination nodes will depend on the underlying application in the VANET, for a reliable evaluation, we should guarantee that the selected pairs have similar properties in terms of repetitions of source and/or destination nodes. A great variability in the repetition of selected nodes can impact significantly on the simulation results. For instance, if destination nodes are very frequently repeated then the queue of these nodes can saturate, and this fact can cause dropped packets. To highlight the importance of the repetition of the source and destination nodes, we depict in Figure 10 and Figure 11 the throughput for different numbers of repeated source and destination nodes in the scenario under test. 21 Figure 10 Throughput vs repeated destination nodes Table 6 Throughput vs repeated destination nodes values in the scenario under test Number of Repeated Destination Nodes 1 2 3 Throughput (Kbps) Mean 0.8529 0.9012 0.8690 Confidence interval 0.0034 0.0399 0.0504 As we can see in Figure 10, when the number of repeated destination nodes increases the confidence intervals are higher (See Table 6 for more details) because the number of application packets lost in the destination nodes buffers are higher. 22 Figure 11 Throughput vs repeated source nodes Table 7 Throughput vs repeated source nodes values in the scenario under test Number of Repeated Source Pairs 1 2 3 Throughput (Kbps) Mean 0.8529 0.8289 0.7535 Confidence interval 0.0034 0.0035 0.0070 We can observe in Figure 11 that as the number of repeated source nodes increases the results are more disperse (See Table 7 for more details) because there are some application packets lost in the intermediate nodes buffers due to traffic congestion. However, the results are lesser scattered than in the case of repeated destination nodes (See figure Figure 10). This is due to the fact that the destination nodes buffers are more congested than the intermediate nodes, causing more collisions and contention in the shared wireless medium. 23 When random selection of communication pairs is used, we cannot control whether the destination and source nodes are repeated or not. Such repetition does not mean that the simulated scenario is unrealistic. However, this situation can favor the performance of a routing protocol with respect to others. For example in the case that one routing protocol is specifically designed to take into account this network condition. By applying the proposed selection we can guarantee that all routing protocols are evaluated under the same network conditions. Finally, to illustrate the importance of the APA, the number of hops, the Warm Up period, and the repetition of source/destination nodes, we compare the simulation results in the scenario under test with and without considering the aforementioned simulation practices. Table 8 details the nomenclature that will be used to describe the following simulation results. Table 8 Chosen nomenclature Acronym Meaning R.P. Randomly selected pairs R.P with M.P. Randomly selected pairs with measurement period S.P. Selected pairs based on APA and number of hops S.P with M.P. S.P with measurement period S.R. Randomly selected pairs with repeated source D.R. Randomly selected pairs with repeated destination S.R. with M.P. + S.P. Repeated source nodes selected based on target APA and number of hops with measurement period D.R. with M.P. +S.P. Repeated destination nodes selected based on APA and number of hops with measurement period Figure 12 shows the throughput obtained in the scenario under test with 25 different source-destination pairs. We depict these results with boxplot graphs and we also highlight with a green point the obtained mean of the samples. The boxplot graphs are used to better understand how values are spaced out in different sets of data. The bottom line of the box represents the first quartile (Q1), the top line represents the third quartile 24 (Q3) and the distance between them is the interquartile range (IQR=Q3-Q1). Another important aspect in the boxplot graphs are the whiskers which depend on the IQR. The upper whisker is determined by the equation Q3 + 1.5*(IQR) while the bottom one is determined by Q1 – 1.5*(IQR). Therefore if we have a set of data bits scattered, the IQR and the whiskers will be low. In this paper we will focus on the IQR to measure the level of dispersion. Figure 12 includes results for R.P., R.P. with M.P., S.P. and S.P. with M.P. cases (See Table 8 for more details). Figure 12 Throughput vs source destination pair selection Using the Warm Up period (See R.P with M.P in Figure 12) we can observe in Figure 12 that the distance between first and third percentile is smaller than without these periods (R.P in Figure 12). This means that we have less scattered measures. To highlight the importance of Warm Up period we focus on the selected pairs based on the APA and number of hops case (S.P in Figure 12). We can observe that in this case the IQR is lower than the case when the selection technique is not used (See R.P and R.P with M.P in Figure 12) because the set of values is lesser scattered. If we also use the 31 Next step is to study the inverse of the cumulative distribution function for the APA values. In Figure 17, we can observe that we obtain similar results for the three layouts. It means that for the same APA, the connectivity of the network is very similar and does not depend on the layout. This might be explained by the fact that the simulations do not consider obstacles like buildings or other vehicles. Figure 17 1-CDF for APA distribution for the three considered layouts The last step in this analysis of the mobility of nodes is to study the cumulative distribution function (1-CDF) for the separation of nodes in terms of the number of hops. As we can observe in Figure 18, the results for the three layouts are very similar. However, for the Complex layout case the maximum number of hops is higher than for the other layouts. From now on in this study, we will use a city with a regular layout 32 such as Washington because it has an intermediate behavior in terms of the APA and number of hops distribution. Figure 18 1-CDF for the number of hops separation for the three considered layouts 33 Figure 19 depicts the area of Washington that will be used in the simulation results section. The left-hand part of Figure 19 shows a real capture of the area used in our simulations, the right-hand part shows the model obtained from C4R and the bottom part depicts the movements of the vehicles over the selected area, with each color representing a different node. Figure 19 Washington layout and node movements in the scenario under test with 125 nodes To model the behavior of the drivers we use the IDM model [18]. The chosen values for each parameter are those that ensure a maximum speed of 30Km/h, which is the speed limit in urban environments. We also ensure a minimum security distance between two vehicles trying to model the real driving conditions in urban scenarios. In this case we 34 are simulating a normal driving behaviour as described in [18] with a minimum security distance of 2 meters. 5) Performance metrics Another important aspect to be considered when evaluating routing protocols is which performance metrics should be used in order to represent an unbiased performance of the routing protocols. It is important to use metrics that exhibit the performance of the routing protocols in different conditions. The following performance metrics are some of the most used in simulation-based studies [34]. Throughput (THR): It is the sum of the data packets in the simulation period. 𝑇𝐻𝑅 (𝐾𝑏𝑝𝑠) = ∑𝐷𝑒𝑙𝑖𝑣𝑒𝑟𝑒𝑑 𝐴𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑃𝑎𝑐𝑘𝑒𝑡𝑠 𝑆𝑖𝑚𝑢𝑙𝑎𝑡𝑖𝑜𝑛 𝑡𝑖𝑚𝑒 (1) Average End-to-End Delay (E2E): It is defined as the time taken for a data packet to be transmitted across an ad hoc network from the source to the destination node. 𝐸2𝐸 (𝑠) = ∑(𝐷𝑒𝑙𝑖𝑣𝑒𝑟𝑒𝑑 𝑇𝑖𝑚𝑒−𝑇𝑟𝑎𝑛𝑠𝑚𝑖𝑡𝑡𝑒𝑑 𝑇𝑖𝑚𝑒) 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑎𝑐𝑘𝑒𝑡𝑠 𝑠𝑢𝑐𝑐𝑒𝑠𝑓𝑢𝑙𝑙𝑦 𝑑𝑒𝑙𝑖𝑣𝑒𝑟𝑒𝑑 (2) Normalized Routing Load (NRL): It is the ratio of the total routing packets to the total delivered data packets. 𝑁𝑅𝐿 = ∑Routing 𝑃𝑎𝑐𝑘𝑒𝑡𝑠 ∑𝐷𝑒𝑙𝑖𝑣𝑒𝑟𝑒𝑑 𝐴𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑃𝑎𝑐𝑘𝑒𝑡𝑠 (3) Packet Delivery Fraction (PDF): It is the ratio of the number of packets delivered to the receiver, to the number of packets sent by the source. 𝑃𝐷𝐹 (𝐾𝑏𝑝𝑠) = ∑𝐷𝑒𝑙𝑖𝑣𝑒𝑟𝑒𝑑 𝐴𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛 𝑃𝑎𝑐𝑘𝑒𝑡𝑠 ∑𝑆𝑒𝑛𝑡 𝑃𝑎𝑐𝑘𝑒𝑡𝑠 (4) 35 Jitter (JIT): It is the delay between two consecutive packet deliveries at a node. 𝐽𝐼𝑇 = ∑𝑑𝑒𝑙𝑎𝑦𝑠𝑝𝑎𝑐𝑘𝑒𝑡 𝑑𝑒𝑙𝑖𝑣𝑒𝑟𝑒𝑑 𝑃𝑎𝑐𝑘𝑒𝑡𝐴𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛𝑑𝑒𝑙𝑖𝑣𝑒𝑟𝑒𝑑 (5) Additionally, in this paper we propose a new performance metric, the Route Activity Time (RAT), which is aimed at evaluating the capability of a routing protocol to maintain an active route between the source and the destination nodes. The formal definition of RAT is as follows: Route Activity Time (RAT): It is the period of time during which a communication path is available between the source and the destination nodes. In routing protocols based on request, reply and error messages, such as AODV, DSR, and DYMO, it is the period elapsed between the time at which the reply message arrives at the source node and the time at which an error message of such route is generated. 𝑅𝐴𝑇(𝑠) = Errortime − 𝑅𝑒𝑝𝑙𝑦𝑡𝑖𝑚𝑒 (6) Notice that the RAT metric measures how the routing protocols manage the path availability. In theory, we control the APA values by selecting the communication pairs, however, the real time in which a communication path is established between a source and a destination node will depend on the underlying routing protocol and the network conditions. Table 11 summarizes the desirable values for each metric used to evaluate the routing protocols performance. Table 11 Desirable values for the performance metrics Although we have described six different performance metrics (1)-(6) to evaluate the performance of the routing protocols, we will only use four of them, THR(1), E2E(2), Metric THR E2E NRL PDF JIT RAT Desirable values High Low Low High Low High 36 NRL(3) and RAT(4) in the next section, since the rest of them provide equivalent information. By using the THR metric, we measure the performance of the routing protocols in terms of the number of delivered packets. With E2E we evaluate the average delay of the application packets. Using the proposed RAT metric, we measure how the routing protocols maintain the communication routes between the source and destination nodes. The NRL metric measures the number of routing packets used by the routing protocols and provides an idea about the power consumption of the routing protocol. Regarding PDF and JIT metrics, we do not use these metrics for the following reasons. First, with the PDF metric we also measure the number of delivered packets so this metric will show us similar results of THR metric. Second, we do not use the JIT metric because it gives us an idea about the network delay and we are actually using E2E to measure this performance. 6) Simulation analyses The objective at this point is to decide which analyses we should carry out for obtaining a good performance evaluation of routing protocols. In general, the number of nodes is a common parameter to vary in simulation-based studies in order to evaluate routing protocols under different density levels (connectivity). However, there are other parameters that also affect considerably the performance of routing protocols. For instance, the congestion is a common issue in multi-hop networks because nodes should share the wireless medium, and consequently, routing and application packets should compete for the wireless medium. The congestion in the network can be modified by varying some parameters of the communication flows between the source and destination nodes such as data rate, size of packets, and number of flows. Among the mentioned parameters, we focus on the number of flows since in this paper we are 37 giving relevance to the selection of source-destination pairs. Therefore, we propose two different analyses. First, a density analysis based on varying the number of nodes while maintaining the same number of communication flows. Second, a congestion analysis, focused on varying the number of communication flows while maintaining the same number of nodes. With the first analysis, we evaluate the routing protocols under different connectivity levels and with low congestion conditions. In the second analysis, we set a medium-high value of density (high connectivity) and vary the congestion of the network to observe how routing protocols perform under different levels of congestion. To summarize the procedure described in this section, Table 12 provides the most important values of the proposed methodology and also the benefits of each of them. Table 12 Summary of the simulation parameters used in the proposed methodology Simulation parameter Selection Benefit obtained S.P. = 300 s W. P.= 50 s (M.S.P.=250 s) Using W.P. we improve mean of the used performance metrics Selection Pairs based On APA Depending on the scenario and based on APA target Applying the proposed methodology based on APA and Number of hops we reduce the dispersion. Selection Pairs based On Hop Number Depending on the scenario and based on number of hops target Performance Metrics THR(1), E2E(2), NRL(3), RAT(6) We can evaluate different features of the evaluated routing protocol. Scenario Washington (Regular layout) We can emulate real scenarios. Analysis Congestion and density We evaluate the routing protocols under different network conditions. 38 4. Evaluation of the proposed methodology This section includes the results of the proposed methodology when evaluating routing protocols in VANETs. The aim is to show that the proposed methodology leads to more reliable simulation results. For this purpose, we compare the obtained results of several well-known and widely used routing protocols for multi-hop ad hoc networks such as AODV, LAR and DYMO with and without the proposed methodology. 1) Simulation environment settings We use NS-2.34 [35] under a Debian Linux operating system. NS-2 is a simulation tool for replicating real life networking environments. To simulate urban mobility of vehicles, we use CityMob for Roadmaps (C4R) [25]. Table 13 summarizes the general simulation settings used. It is important to highlight some specific aspects of VANET simulations such as the MAC protocol used, which is the IEEE 802.11p [36]. This is based on the 802.11a standard and has the same structure. The main difference, compared to 802.11a, is its bandwidth, which is narrower in order to make the signal more resistant to fading and multipath propagation in the automotive environment. Another important difference is the operating frequency; in the case of 802.11p standard is 5.9 GHz, as opposed to the 802.11a, which is a standard operating at 5 GHz in Europe [37]. Regarding the propagation model, we use the two-ray ground reflection model [28] because it gives more accurate prediction for long distances than the free space model. We select a transmission range of 500 m, which is a typical transmission range for VANET scenarios (the standard IEEE 802.11p can reach up to 1000 m). The simulation time is 300 s, it is high enough to guarantee a good evaluation of routing protocols. The warm up period has been selected according the study conducted in the 39 previous section 3.1. The type of traffic is Constant Bit Rate (CBR), which is typically used in multi-hop scenarios with UDP transport layer. CBR traffic is suitable for real time applications. The transmission rate of application packets is 1 packet per second and the size of each packet is 512 bytes. The simulation scenario is the one described in section 3.4. The maximum speed of nodes is 30 km/h. This maximum value is suitable for urban scenarios, where the limited speed is about 50 km/h. Table 13 Simulation parameters Parameter Value Simulation Time 300 s Warm Up period 50 s Routing Protocols AODV,DYMO, LAR Transmission range 500 m MAC Protocol Type IEEE802.11p Number of Nodes 50, 75, 100, 125, 150, 175 Numbers of Sources 5, 10, 15, 20, 25 Traffic Types CBR Transport Layer UDP Maximum Packet in Queue 50 Packet Size 512 bytes Packet Rate 1 packet/s Area Size 4000*4000 m2 Mobility model Washington Layout Propagation model Two-ray ground Maximum speed of nodes 30 km/h 2) Simulation results In this subsection we show the obtained results of the two different proposed analyses, the density analysis and the congestion analysis. Density analysis In this analysis we vary the number of nodes for each of the three used routing protocols, in steps of 25 each, within the interval [50, 175]. The number of communication flows is fixed to 5 (low congested scenario), the APA values are selected according to Table 3 and the maximum number of hops according to Table 5. 40 Figure 20, Figure 21 and Figure 22 show the simulation results for THR, RAT, E2E, NRL performance metrics, with their confidence intervals for each number of nodes. Figure 20, Figure 21 and Figure 22 include the results using the proposed methodology (Results with P.M in Figure 20, Figure 21 and Figure 22) and without using the proposed methodology (Results without P.M in Figure 20, Figure 21 and Figure 22). We depict the results for 5 and 30 simulations (seeds in Figure 20, Figure 21 and Figure 22) with and without applying the proposed methodology to check the differences between them in terms of the mean and the confidence interval. By using the proposed methodology, the results are in general, very similar in terms of mean. Nevertheless, the confidence intervals are low for both numbers of simulations. But if we randomly select the source destination pairs, there are important differences in the mean values of some metrics and also in the confidence intervals which are large in spite of running many simulations. Figure 20 Results of the density analysis – AODV 47 Congestion analysis In this study we vary the number of connections, in steps of 5 connections each, within the interval [5, 25]. The number of nodes of the network is fixed to 125 nodes (high density scenario) and the APA values and the number of hops are the same than those selected in the previous analysis (See Table 3 and Table 5 for more details). Figure 23 shows the simulation results for THR, RAT, E2E, NRL performance metrics with their confidence intervals for each number of nodes. Figure 23 includes the results using the proposed methodology (Results with P.M. in Figure 23) and without using the proposed methodology (Results without P.M. in Figure 23). Once again, we depict the results for 5 and 30 simulations and we verify that by using the proposed methodology more reliable results are obtained despite of using a lower number of simulations (See Table 17 for more details) Figure 23 Results of the congestion analysis - AODV 48 Table 17 Statistics measures for performance metrics vs number of pairs - AODV Number of Pairs 5 10 15 20 25 AODV THR (Kbps) With P.M. 5 seeds Mean 0.8582 0.8392 0.8557 0.8448 0.8346 C.I. 0.0119 0.0160 0.0098 0.0082 0.0086 30 seeds Mean 0.8526 0.8464 0.8546 0.8461 0.8359 C.I. 0.0063 0.0061 0.0036 0.0036 0.0035 Without P.M. 5 seeds Mean 0.5355 0.6735 0.6664 0.6187 0.6193 C.I. 0.1504 0.0898 0.0700 0.0637 0.0590 30 seeds Mean 0.5262 0.6615 0.6608 0.6104 0.6173 C.I. 0.0591 0.0360 0.0292 0.0261 0.0238 RAT (s) With P.M. 5 seeds Mean 206.923 211.773 232.659 235.301 239.376 C.I. 17.771 14.754 10.157 8.420 7.762 30 seeds Mean 193.780 216.385 226.776 232.539 235.134 C.I. 8.019 5.125 4.387 3.316 3.182 Without P.M. 5 seeds Mean 124.043 173.201 168.820 161.880 155.708 C.I. 38.520 26.256 20.532 18.419 16.549 30 seeds Mean 125.124 166.788 169.387 157.127 156.794 C.I. 15.413 10.331 8.578 7.420 6.730 E2E (s) With P.M. 5 seeds Mean 0.1269 0.1505 0.0850 0.0966 0.07975 C.I. 0.0758 0.0459 0.0327 0.0409 0.0272 30 seeds Mean 0.1173 0.1436 0.0831 0.0894 0.0844 C.I. 0.0274 0.0202 0.0135 0.0130 0.0131 Without P.M. 5 seeds Mean 0.3297 0.3092 0.2902 0.4364 0.5598 C.I. 0.2257 0.2650 0.1108 0.1386 0.2744 30 seeds Mean 0.4817 0.2895 0.4189 0.4846 0.4668 C.I. 0.1710 0.0802 0.0922 0.0723 0.0766 NRL With P.M. 5 seeds Mean 4.1438 4.2499 3.1861 4.2388 5.3047 C.I. 0.1047 0.0385 0.0181 0.0222 0.01427 30 seeds Mean 4.6115 3.8982 3.3883 4.0784 5.2930 C.I. 0.0623 0.0186 0.0093 0.0069 0.0055 Without P.M. 5 seeds Mean 8.1798 6.3340 6.5440 10.8139 11.4277 C.I. 0.0938 0.0269 0.0368 0.01402 0.0282 30 seeds Mean 8.6758 6.5426 6.4049 10.9307 11.2774 C.I. 0.0649 0.0224 0.0113 0.0107 0.0101 49 Figure 24 shows again that a smoother tendency is obtained for the congestion analysis when the proposed methodology is used. Similarly, the confidence intervals obtained with the proposed methodology are lower than those obtained when it is not applied. In this analysis (See Figure 24) the proposed methodology also provides better results as the network congestion increases (See Table 18 for more details). Figure 24 Results of the congestion analysis - LAR 50 Table 18 Statistics measures for performance metrics vs number of pairs – LAR Number of Pairs 5 10 15 20 25 LAR THR (Kbps) With P.M. 5 seeds Mean 0.8645 0.8601 0.8712 0.8688 0.8581 C.I. 0.0199 0.0133 0.0112 0.0098 0.0093 30 seeds Mean 0.8610 0.8595 0.8628 0.8611 0.8549 C.I. 0.0080 0.0054 0.0043 0.0036 0.0041 Without P.M. 5 seeds Mean 0.6477 0.6585 0.6656 0.6603 0.6436 C.I. 0.0539 0.0513 0.0440 0.0365 0.0339 30 seeds Mean 0.6356 0.6601 0.6669 0.6591 0.6396 C.I. 0.0224 0.0199 0.0157 0.0132 0.0141 RAT (s) With P.M. 5 seeds Mean 202.442 211.412 206.878 208.050 204.882 C.I. 15.102 8.350 9.819 8.958 7.529 30 seeds Mean 198.321 213.621 204.071 205.322 207.927 C.I. 6.601 3.203 3.925 3.100 3.084 Without P.M. 5 seeds Mean 180.722 150.339 158.320 163.162 155.743 C.I. 31.306 19.054 15.669 13.654 11.530 30 seeds Mean 168.474 155.804 165.221 159.049 156.168 C.I. 11.681 7.366 5.427 4.736 4.811 E2E (s) With P.M. 5 seeds Mean 0.1914 0.1224 0.1837 0.1839 0.3057 C.I. 0.0529 0.0405 0.0629 0.0483 0.0574 30 seeds Mean 0.1757 0.1148 0.1840 0.2163 0.3102 C.I. 0.0256 0.0146 0.0228 0.0207 0.0241 Without P.M. 5 seeds Mean 0.3338 0.4709 0.4289 0.4231 0.4204 C.I. 0.1161 0.1082 0.0889 0.0680 0.0630 30 seeds Mean 0.3486 0.4558 0.3957 0.4167 0.4104 C.I. 0.0510 0.0414 0.0290 0.0251 0.0259 NRL With P.M. 5 seeds Mean 5.3808 4.2594 3.8384 3.6584 3.8578 C.I. 0.061 0.047 0.015 0.006 0.006 30 seeds Mean 5.8295 4.3325 3.8238 3.6221 3.7453 C.I. 0.1699 0.0357 0.0063 0.0047 0.0035 Without P.M. 5 seeds Mean 10.4852 11.5379 10.1581 11.5247 12.8672 C.I. 0.2965 0.0612 0.0271 0.0124 0.0283 30 seeds Mean 10.539 11.216 10.192 11.598 13.123 C.I. 0.0804 0.0252 0.0144 0.0080 0.0072 51 Regarding the DYMO routing protocol (See Figure 25), the results are also not dispersed. These are better (in terms of mean) and we can see a smoother tendency of the performance metrics when applying the proposed methodology (For more details see Table 19). Figure 25 Results of the congestion analysis – DYMO 52 Table 19 Statistics measures for performance metrics vs number of pairs – DYMO Number of Pairs 5 10 15 20 25 DYMO THR (Kbps) With P.M. 5 seeds Mean 0.8614 0.8636 0.8587 0.8551 0.8461 C.I. 0.0214 0.01693 0.0164 0.0134 0.0146 30 seeds Mean 0.8585 0.8554 0.8620 0.8502 0.8354 C.I. 0.0071 0.0047 0.0053 0.0045 0.0056 Without P.M. 5 seeds Mean 0.5729 0.5400 0.5398 0.5315 0.4884 C.I. 0.1195 0.0664 0.0620 0.0535 0.0484 30 seeds Mean 0.5780 0.5368 0.5495 0.5290 0.4703 C.I. 0.0360 0.0154 0.0200 0.0176 0.0159 RAT (s) With P.M. 5 seeds Mean 224.562 227.734 216.916 210.874 206.888 C.I. 4.745 5.709 11.521 11.972 10.794 30 seeds Mean 222.868 226.217 211.986 209.079 207.642 C.I. 1.628 1.573 4.525 4.117 3.691 Without P.M. 5 seeds Mean 149.915 144.814 142.371 139.053 112.827 C.I. 29.379 16.855 15.374 13.429 14.962 30 seeds Mean 152.285 143.882 144.351 139.452 113.793 C.I. 8.972 3.864 4.987 4.419 4.910 E2E (s) With P.M. 5 seeds Mean 15.9048 13.8787 22.3367 27.0592 34.3470 C.I. 3.6844 2.5246 6.1019 5.4334 5.0076 30 seeds Mean 16.8680 14.7454 22.4142 27.8414 34.8339 C.I. 1.2619 0.7011 1.9091 1.7882 1.6750 Without P.M. 5 seeds Mean 66.6104 63.5052 59.2219 61.2661 57.1043 C.I. 21.3251 10.8777 9.2749 7.7797 6.4067 30 seeds Mean 69.0914 64.6261 59.3388 59.4877 58.4515 C.I. 6.6084 2.4876 2.9497 2.5537 2.1852 NRL With P.M. 5 seeds Mean 11.8661 10.1422 9.2064 9.7781 10.0558 C.I. 0.4036 0.0658 0.0688 0.0222 0.0162 30 seeds Mean 12.8203 10.4430 8.7722 9.8621 10.6552 C.I. 0.1274 0.0199 0.0185 0.0102 0.0092 Without P.M. 5 seeds Mean 56.2519 54.0320 50.4579 57.7146 64.3516 C.I. 1.1956 0.3271 0.1449 0.0851 0.1476 30 seeds Mean 53.6853 55.5944 48.6554 58.1439 66.5199 C.I. 0.4110 0.0921 0.0678 0.0417 0.0497 53 3) Comparison of the routing protocols using the proposed methodology This subsection is aimed at providing a fair an unbiased comparison between the three used routing protocols such as AODV, DYMO, and LAR. We evaluate the mentioned routing protocols under different network conditions using the proposed APA metric and number of hops. In Figure 26, we depict the throughput and NRL metrics for AODV, LAR and DYMO for different APA values. We vary the APA value in steps of 0.2. As it is shown in Figure 26, the throughput of the three routing protocols increases as the APA value also increases. The reason is that the routes between source and destination nodes are available during more time. As a consequence, the number of delivery packets is higher. According to the results, DYMO has the best performance metrics for high APA values while LAR is the best one for low APA values. However the NRL decreases when the APA value increases because is not necessary to initiate new discovery phases. The reason is that the routes are more time available so the number of routing packets decreases. In terms of NRL, DYMO presents the worst behavior because the necessary routing information (routing packets) is higher. 54 Figure 26 Throughput and NRL vs APA Now we evaluate the routing protocols versus the number of hops between the source and destination nodes (See Figure 27). We also evaluate the three routing protocols in terms of throughput and NRL metrics. For the three routing protocols, the throughput metric is better for low numbers of hop values (See Figure 27). This situation corresponds to the one in which the destination node is near the source node (in terms of hops). Consequently, the probability of losing data packets is lower. Yet, when the number of hops increases the throughput decreases for the three routing protocols. DYMO has the best performance for low number of hops because this one is able to generate routes entries for each intermediate hop. However, when the distance between source and destination node is higher the routing data packets are also higher. In consequence, the delivery of packets decreases. Regarding the NRL, it increases for the three used routing protocols because the number of discovery phases is higher when the number of hops increases. As we mentioned before DYMO has the worst behavior because the routing information is higher. 55 Figure 27 Throughput and NRL vs number of hops 5. Conclusions Simulation-based studies of routing protocols for VANETs involve selecting, adjusting, and setting a large number of simulation parameters, which can affect significantly the simulation results. In this paper, we propose a methodology to conduct reliable simulations of routing protocols based on a set of good simulation practices for VANET scenarios. The proposed methodology includes aspects such as: i) Measurement period, to ensure that all the simulation measurements begin and end at the same time, consequently we improve the performance metric mean. ii) Source destination pair selection, to avoid discrepancies in terms of path availability and number of hops between the source and destination nodes and to obtain results lesser scattered, iii) The number of trials, to obtain reliable measurements. iv) Mobility models, based on maps 56 to emulate mobility of vehicles in urban scenarios. v) The importance of the repetition of source and destination. vi) Performance metrics and simulation analyses, to evaluate routing protocols under different conditions. We have shown the importance of selecting these simulation parameters carefully in order to obtain reliable simulation statistics and make a fair and unbiased evaluation of routing protocols in VANET scenarios. Finally, we have validated the proposed methodology by conducting a comparison of the AODV, LAR and DYMO routing protocols with and without using the proposed methodology. The obtained simulation results demonstrate that the proposed methodology provides better results in terms of reliability (confidence intervals), and a smoother tendency of the performance metrics. Acknowledgements This work was supported in part by the University of Seville under the V plan propio de investigación and by the Ministerio Español de Economía y Competitividad under the contract of Jesús Sánchez García associated to the project DPI2013-44278-R. References [1] R. Jurdak, C.V. Lopes, P. Baldi. A survey, classification and comparative analysis of medium access control protocols for ad hoc networks. Communications Surveys & Tutorials, IEEE. Vol.6, pp. 2-16. 2004 [2] L. Junhai, Y. Danxia, X. Liu, F. Mingyu. A survey of multicast routing protocols for mobile Ad-Hoc networks IEEE. Communications Surveys & Tutorials. Vol 11, pp. 78-91. 2009.